Lies, Damn Lies, and Estimates

If you've ever been involved with a software development project, big or small, you’ve probably learned one of the central tenets of software engineering: time estimates are wrong. Always.  Anyone who tells you that they can consistently estimate how long a project will take with a high degree of accuracy is either naive or arrogant.

 

How could a field seemingly based on science, mathematics, and logic be so terrible at estimating its efforts?

 

It’s terribly counterintuitive, but it shouldn’t be scary.  Software design and engineering isn’t much like the fields to which it is often compared.  Unlike construction, farming, or manufacturing, it’s not innately repeatable.  No two projects are the same, regardless of the level of similarity in their requirements.  Software relies on people, communication, knowledge, creativity and evolving process.  It is messy.

 

Despite that, we have financials to prepare, executives to brief, and investors to reassure.  So when we’re considering a software project, we need to know how much it will cost in time and money.  But, rather than jumping to estimation, let’s consider the power of the humble budget.  The concepts of estimation and budgeting seem related, but in they are actually opposite ends of the same process.  Let me show you what I mean.

 

A “budget” simply means the amount of money needed or available for a purpose.  We’ll get to the needed part later on, for now I want to focus on available.  Available implies that there is a fixed amount that can be used, once it’s gone, it’s gone.  In software terms we’re talking about time and materials versus fixed bid engagements.  This means that we can separate cost from scope, allowing scope to be fluid and cost to be set by the budget.  I’ll address the evils of fixed bids in a later post.

 

 

When a project is estimated up front, progress looks a lot like putting items from the grocery store shelves into a cart.  The only thing you can really see is how much is in there, and perhaps that the cost of potatoes has really gone up.  This viewpoint is entirely rearward facing, it only has clarity in hindsight.

 

Instead of big and complicated estimates, when we develop a reasonable budget instead our view is forward looking.  Budgets are concerned with the intrinsic potential of the remaining resources and less so with what’s already been spent.

 

 

In any creative endeavor, constraints are incredibly valuable.  Without constraints, the lines of effort wander and waste seeps into the process.  Estimates don’t give you constraints - they merely quantify the options, and when we use them, we stop asking, "is this actually the best way to spend our resources?"

 

Further, estimates atrophy, much like documentation gets out of date.  Original estimates become less and less valuable as the realities of a project veer away from the initial understanding.

 

Establishing a budget, however, shines a bright light on prioritization.  If funding is limited, then we always need to be focusing on the next most important thing.  Even if the budget is artificial, having that constraint provides a valuable lens through which to evaluate scope and possible tradeoffs.

 

 

Nobody is ever happy about an estimate being wrong.  Whether it was too high or too low, an incorrect estimate throws off all our predictions.   And, if estimates aren't good predictors of effort, why bother?  

 

When tracking against a budget, however, a team has a much higher chance of reaching some success. Here success is defined as anything at (or better than) the allocated amount.  This gives us a wide target to hit, and the flexibility to change things as we go, without our estimates going stale.

 

At this point, I'm sure you're asking yourself how do we set a reasonable budget if we don't do estimation?  The answer is, “you don't.”  Estimation still has to take place, but it's when you estimate, and what you do with the estimates, that changes.  Our projects typically employ two different types of estimation: sizing and story points.

 

 

We always start a project (or a milestone within a project) with a sizing exercise.  There's nothing scientific or formulaic about sizing.  Sizing is more of an art, which relies on the experience of senior folks from the various disciplines on a software team.

 

Sizing involves developing enough understanding of the high-level requirements to allows us to break down the project into feature units.  These aren't as granular as user stories, but are smaller than what you'd put on a bullet list of features in a marketing campaign.  Each unit is then roughly "estimated" at a specified number of days or weeks.

 

It’s important to keep in mind that we’re not just throwing numbers out.  Sizing requires thought and experience.  It’s key to have several different people weigh in and ask questions, a healthy debate should develop during these discussions.

 

Once all of the rough estimates for modules are tallied up, then we adjust for staffing.  How many developers do we think we'll need?  How many designers?  What about project management?  After factoring in staff and rates, we have our budget.  Pretty simple.

 

One last point to make here.  Once we have our budget, it’s important to discard the details of the sizing.  If we don’t it’s tempting to just view the sizing document as an estimate breakdown, which invalidates the whole budgeting approach.

 

 

The second type of estimation is done in real-time and is focused on helping us make smart decisions with how to spend the remaining budget.  Working in one week iterations, we hold a planning meeting at the beginning of each week that determines what user stories will be worked on during the cycle.  As part of this meeting, we add point estimates to each story.  These are not focused on time, but focused on complexity.  A simple story might get a single point, a more complex story may get 3 to 5 points.

 

After a few weeks, the project team begins to develop a velocity, the average number of points likely to be completed in a given iteration.  Velocity helps us project where we'll be  within our budget at any given time.  Rather than always looking backwards to assess whether we are keeping pace with our initial estimate, we can look forward and have rational and motivated conversations on how to spend the remaining budget.

 

Ultimately, how you and your teams choose to create budgets, or estimates, should reflect what works best for you.  Whatever process you implement, it needs to leverage the experience and intuition of your team, rather than being based in methodologies or formulas.  Software, despite it's roots in mathematics and engineering, is a collaborative art.  As such, it cannot accurately be predicted by an equation.  What we can do however, is set realistic goals, in regards to time and money, and provide our teams the necessary tools to measure and track a project’s progress.