Embed Size (px)
DESCRIPTION
Product development is inherently risky. While lean and agile methods are praised for supporting rapid feedback from customers through experiments and continuous iteration, teams could do a lot better at prioritizing using basic modeling techniques from finance. This talk will focus on quantitative risk modeling when developing new products or services that do not have a well understood product/market fit scenario. Using modeling approaches like Monte Carlo simulations and Cost of Delay scenarios, combined with qualitative tools like the Lean Canvas and Value Dynamics, we will explore how lean innovation teams can bring scientific rigor back into their process.
Citation preview
Stop Flying Blind!Quantifying Startup Product Risks with Monte Carlo Simulation.Sam McAfee, Principal - @sammcafee
Slides available at bit.ly/riskiest_assumption
Who is this guy?• Dot-com boom survivor. • Ran a dev shop for 10 years after that. • Agile has always been the “norm”. • Read ‘4 Steps’ before #LeanStartup. • Participated in 3 different startups. All dead. • Ran Change.org engineering. • Learned LeanUX from @clevergirl. • Wore plaid and facial hair before it was cool.
Tweet every word: @sammcafee
A Typical ScenarioMaybe you’ve experienced something like this?
The CEOA typical scenario
Our competitors have it, so we should
have it too!
“ ”
Product ManagementA typical scenario
It takes too long to get new features out.
“ ”
EngineeringA typical scenario
We have to refactor all the things we rushed out the door last time.
“ ”
How do we prioritize competing objectives?
Understanding RiskIt’s kind of important for startups.
Why Is Risk Important?Understanding Risk
Startups are risky!You are participating in one of the highest risk
business endeavors there is.
Risk is a business term.You are in business. Let’s use terminology that
business people use to make decisions.
Ignore it at your peril.Choosing to ignore risk will not protect you from it.
What Is Risk?Understanding Risk
Risk = Impact * Probability
Stop Using Your Gut!Understanding Risk
All people, including experts and managers,
are very bad at assessing the probabilities of events.
— Doug Hubbard
“ ”
Risk = Impact * Probability
How Do We Define Impact?
Hint: For startups, it can probably be found on the bottom line.
Economic framework.Allows you to compute the impact of any change in
the system into a single unit of measure.
Start with your P&L.Your CEO or finance people already have a
framework for you to start with. Include them.
Apply risk scenarios.Calculate the impact of likely scenarios on the total
product life-cycle profits, or similar KPI.
Cost of DelayScenario 1
Cost of Delay Scenarios• Projects will have different cost of delay curves.
• What is the effect on total life-cycle profits for each unit of delay?
• Delay in product development is overwhelmingly affected by queues between steps rather than job duration at a given step.
• Cost of Delay enables you to calculate the cost of queues.
Sequence of WorkScenario 2
Work Sequence Scenarios• The order in which you do work can have a substantial impact on
the total cost of queues.
• If cost of delay is homogeneous, do the shortest job first.
• If job duration is homogeneous, do the highest cost of delay first.
• You can combine job duration and cost of delay using weighted shortest job first.
Tangent: which is the shortest job?
• Job duration is an estimate.
• Estimates are probabilistic, not deterministic.
• Use cycle time and throughput to compute a probability distribution for likelihood of job duration.
Capacity vs. Queue CostScenario 3
Capacity Utilization Scenarios• You can reduce the cost of queues by adding additional capacity.
Obviously, additional capacity has a cost too.
• Additional capacity increases transaction costs, but lowers holding (delay) costs.
• Is the cost of adding additional capacity justified by the gains in product development throughput and lower queue costs?
How Do We Define Probability?
Use your metrics, Luke!
Use business metrics.You are probably already tracking tons of data about
how you acquire new customers and how they enter
and leave your product funnels.
Value each conversion.Use your historical data to calculate values for each
step in a product funnel.
Note areas of variance.The steps in your funnel that have consistent,
regular conversion rates are unlikely to change.
They represent lower information content.
How Do We Value Information?
Hint: It’s probabilistic too.
Information Has Value
• 10,000 visitors, 10% sign-up, $20/month.
• Each visitor is worth $2/month (10% x $20/month).
• A test to increase rate to 15% is worth:
• An additional $10/month * probability of test success.
Experiments Have Value
• Experiments attempt to capture new information.
• Experiment value = expected benefit - cost of running the test.
• Expected benefit = increase in KPI * probability of success.
• Cost of running the test = cost of delay * job duration.
Your backlog is a prioritized list of
experiments.
Monte Carlo SimulationsIt’s not just for insurance geeks anymore!
Monte Carlo Simulation• A series of dependent variables, each with probability
distributions.
• Randomly selects a value from each variable, and computes output. Repeats thousands of times.
• End result is another probability distribution, in the unit of measure that you care about.
• It’s 2014. You can do this in a spreadsheet, in about an hour.
Basic funnelUsing time series data from your funnel, create
probability distributions for each step toward
capturing revenue.
Basic histogramAfter 5,000 or so simulations, display a histogram of
the expected output.
1
2
1
2
References
+
Thank you!
Slides available at bit.ly/riskiest_assumption
