Metrics & Predictive Analytics

@t_magennis1Five Estimation and Forecasting #TipsBy Troy Magennis@t_magennisFocusedObjective.com

This presentation lists my top 5 tips for forecasting and estimating software projects. My name is Troy Magennis and I run a consulting company focused on helping organizations plan more reliably. 1

@t_magennis2

All models are wrong. Some are useful.Just has to be better than what is currently used and intuition alone.

The famous quote from George Box succinctly says that no matter how hard we try, or how much money we spend, all forecasting models are wrong in some way.

Being wrong doesnt mean useless. The guidance we get from a model just has to be superior and cost effective against the status quo methods in software development estimation, thats a very low bar.

Our industry uses flawed methods at worst, a gut instinct at best. Even with crude models we will see an improvement in outcomes.2

@t_magennis

3

Expert GuessExpert Range GuessRegression Forecast

Trend based extrapolation(mathematical)Probabilistic Forecast

Distribution based(statistical) Guesses may be based on data and/or experience

Not About Use of Data

Starting with definitions of estimation and forecasting models. There is often a lot of debate about this on Twitter, so Im going to give you my definitions.

Guess or Estimate is what it sounds like someone, hopefully someone with historical context gives an opinion. Its not always in-expensive though. Getting an expert or a team of experts to give an estimate can take considerable time. Its also common for expert guesses to be in the form of a range. This helps convey some uncertainty and risk of each guess. It turns out although due to cognitive biases humans perform poorly at guessing. However, we can be trained by practicing. Dylan Evans in his book Risk Intelligence, and Douglas Hubbard in How to Measure Anything teach ways to calibrate estimators.

The difference between estimate and forecast isnt if data is used. Good experts will call upon data to help them weigh in on a decision. We earn the right to use the word forecast when we are estimating the future using some more formal mathematical or statistical model or technique. Trend based extrapolation is common, this is called regression. Velocity based burn-downs/ups are a regression forecast.

As models get more complex, they begin to provide a set of results, and this allows statistical probability to be computed. This is called probabilistic forecasting. 3

@t_magennis4

Freedom of choice by Krzysztof PoltorakIf there are no choices, there is no need to estimate or forecast

Best option

If there are no choices that will cause a change in direction then there is little or no need to estimate. Sometimes though, the ability to change investment levels once work has begun is limited. Sometimes the cost of missing a certain date is excessive. For these cases, some forecasting is necessary in order to find what investment in people and resources is required to achieve an objective date.

Estimation and forecasting help provide clarity over which option is best to pursue. However, just focusing on choosing the best option often leads to elaborate methods that still fail. 4

@t_magennis5

http://en.wikipedia.org/wiki/Brian_KnightRuling Out an option is a success to celebrate

The ability to eliminate poor or un-economical options is a key reason to estimate. We often forget the disasters we avoid by even the minutest of research and estimation about a potential option.5

@t_magennis6

Option1

> 2x

Opt 1 ~= Opt2Option2Option3

For example, The companies goal is to get from point A to point B. This could be a financial goal, or a customer retention goals. Its some increase in value.

The teams identify three options. By even a crude estimate in effort (displayed here by line length), Option 3 is at last 2x option 1 and 2. By quickly ruling out Option 3, we are still left with Option 1 and 2, but since they are so close we could just as easily toss a coin than go into days of estimation deliberation. 6

@t_magennis7

use the least precision that supports making the decision G. Dinwiddie via Twitter

Often to rule out options is to just know one option is an outlier. Use a calendar quarter if it makes a decision. Then months, and once you are blow that make it clear options are pretty much equal given the uncertainty of risk. Looks for some factor OTHER than calendar date that separates these options.

I hear estimation takes a long time, and almost every time I look its because of this premature detail estimation when some other glaring factor was overlooked. For example, speed and reliability of the final solution.7

@t_magennis8

Expert Judgement Requires Context and Practice

Expert judgement is called upon to estimate when we need to. Context matters. Einstein may not have been great at giving you the recovery time from certain surgery, a surgeon understands that through seeing historical trends of recovery time. But that same surgeon may not be the right person to delve into space and time calculations.

One point of caution here is that its not raw intelligence, or the act of doing the equations or surgery that counts here. Its the exposure to prior samples. Its seeing the feedback from experiences that helps calibrate the experts against their cognitive biases.8

@t_magennis9

versus

2 + 2 = 5

Given equal access to estimates by calibrated experts and historical data, we are faced with a decision of which one to trust. It comes down to how much you trust the expert and the data. Now it is up to you to use your intuition and experience to choose. My advice: do both and hope they agree. When they dont look at the assumptions used by each and take the one that accounts for more assumptions you can corroborate in some way. 9

@t_magennis10

Have an achievable planhttp://en.wikipedia.org/wiki/Mario_G%C3%B6tzeSet a goal.

I tend to avoid giving calendar dates as a forecast. I give a credible plan for hitting a desired date. I fix the date, or a few dates and work out what it takes to hit those targets. Often this highlights what is humanly viable versus dreamland. By approaching forecasting in this way, hard decisions about team size and scope size is dealt with upfront. A solution to hitting a planned date is a collaborative process not a combative one. Its harder to say just work harder when you can say, we actually dont have any team until December.10

@t_magennis11Source: NYT: http://www.nytimes.com/interactive/2014/04/22/upshot/100000002835671.mobile.html?abt=0002&abg=1&_r=0

1.4x more likely 3:59 than 4:01

Never underestimate the power of arbitrary BUT achievable and credible goals. For example, in the NY Marathon historical finish times, there are huge spikes just prior to round numbers. This doesnt mean someone who finishes around 5 hours will finish at 4 hours; it just means that people who would have finished just after 4 hours, somehow manage to get those few extra minutes. 1.4 times more likely to finish in the minute prior to 4 hours than the minute after.

What Im really saying here is work out the plan, and round it up or down to the nearest month. Its easier to communicate. And its likely that people make informed prioritization decisions to hit that target.11

@t_magennis12Dont estimate if you dont have toSet achievable dates, backed by a planUsed data (only) when it exceeds judgementEstimate in the largest units practicalTrack missed assumptions (not status)

To summarize the five tips for forecasting,

Dont estimate if you can get away with itUse the largest units of measure that gets a decisionsBalance your trust of data versus expertsFix an achievable date and work a plan to achieve itAnd track what assumptions you make in your plan. Use these as binary indicators of hitting or missing the plan.12

@t_magennis13

THEORYPRACTICEBit.ly/SimResources

13