Why Story Points are the Worst (and why we started using them again)

About a year ago, we made the switch from Scrum to Kanban. One of the reasons we made the switch was finally being able to abandon Story Points.

First, some background on Story Points:
Story Points are a key element of planning in Scrum. For every story (or feature), the development team is asked to rate how difficult the story will be. The rating is relative to previous projects and is given on a scale, usually "0, ½, 1, 2, 3, 5, 8, 13, 20, 40, 100, ?." The team arrives at their estimate by playing planning poker, which often involves a few rounds of negotiation.

Because Scrum is based on iterations or sprints, you need to predict how many stories your team will be able to accomplish during the sprint, which, for our team, was a two week period. So, you look at how many Story Points you completed in previous sprints and commit to completing stories that add up to the same number of Story Points for the new sprint.

Besides using Story Points to predict how much work you can complete in the next sprint, they are also used to measure how productive the team is. In Scrum, this is called Velocity. Velocity is measured by tracking how many Story Points the team can complete each sprint, with the hope that over time the team's velocity will go up.

In the years that we were doing Scrum, we experienced two key pain points with Story Points:

1. Estimating Story Points Takes a lot of Time and Money.

Every sprint we would need to provide story points for roughly a dozen stories. Cumulatively, this took a couple hours per sprint and about 50 hours per year. That amount of time cost our company many tens of thousands of dollars annually. This feeling of immense waste is part of what sparked the #noestimates movement.

2. Story Points don't deal with change very well.

One of our team's guiding principles is continuous improvement. We regularly invest time in improving our tools, environment, and codebase to make it easier and faster for us to develop new features in the future. So, while adding menu might have taken us a week in the past, now it would only take us a few days.

Let's say the last time we made a menu (before our velocity improvements) we gave it 7 story points. Now that we've made our improvements and we're being asked to add another menu, it's not clear how many story points it should get. There are basically two ways it could go down:

  1. The team will give it fewer story points, let's say 3.5, because it won't take as long

  2. The team will give it a 7 since that's was what the last menu project got.

Which of these do you think is most likely to happen? For us, #1 happened more often, but for Scrum really work properly, it needs scenario #2. Let me explain. Let's say that before the velocity improvements the team could complete the menu project (7pts) and a gallery project (13pts). At this point, the team can complete 20pts per sprint. This is the team's original velocity.

Now that the team has made menus twice as easy to build. In the next sprint, they will have time to build two menus and one gallery.

In scenario #2, the story points look like
Menu (7pts)
Menu (7pts)
Gallery (13pts)
Team Velocity = 27pts per sprint! Hooray, the team's velocity has improved! Our investment improving our code base paid off!

In scenario #1, the story points look like
Menu (3.5pts)
Menu (3.5pts)
Gallery (13pts)
Team Velocity = 20pts per sprint. Uh oh, our velocity stayed the same. It's going to be hard to convince anyone that we're going faster. They probably won't let us spend anymore time refactoring.

Kanban to the Rescue!

Kanban skirts the issue of estimating entirely. There are no sprints. Instead, work is organized into a continuous flow, so there is no need to predict ahead of time how many projects you can fit into the next two weeks.

But, without story points, how can you measure how productive the team is? Well, Kanban has a different metric for that. While Scrum asks, "How much work can get done in two weeks?" Kanban turns that question on its head and asks, "How long does it take to get a feature done?" This is called cycle time, and it's wonderful. Cycle time is entirely objective. It's simply the date that the project completed minus the date that the project started. There's no gaming it. It doesn't get confused by velocity improvements. It just is what it is. No need for the team to spend hours and hours estimating.

There and Back Again.

How Story Points have crept back into our process.

We get a number of requests from our partners for enhancements to our platform. We can't do everything, so we have to choose wisely. A focus of ours in the last few months has been to make sure that we work on the requests that give us the biggest bang for our buck. We were essentially looking for the classic ROI calculation of Value/Cost. We had a good handle on value by looking at how many people had requested the enhancement and how valuable their relationship was to us. Cost, though, meant estimating Story Points again. Because we're not using them to commit to deadlines, it's not important that they be perfectly accurate and arrived at via careful consensus of the whole team. So now, instead of hour-long meetings, when we have a new request that needs estimating, we simply say "Hey, Pat, how many story points do you think this is?" The whole thing takes about 20 seconds per request.

So, that's our where we're at now. We do use story points to help us prioritize some projects, but they are a small investment of time now, and never used to commit to deadlines or to measure the team's performance.

Does your team use story points? Let us know in the comments.