Business model innovation by experimentation

  • Published on

  • View

  • Download

Embed Size (px)


How to maximize learning and minimize risk. All new products start as a series of unvalidated assumptions. The most critical assumptions are usually implicit and relate to the purpose of the product and the value it is intended to deliver. The more key assumptions involved, the greater the risk. It is enough to have 7 key assumptions about which you are 90% certain for the combined odds of success to be below 50%. Contrary to popular belief, when we know very little about a situation, it only takes a small amount of new data to realise significant insights. Unfortunately, people often underestimate the value of information and misunderstand risk. As Product Owners we are often afraid to test our assumptions. We routinely pile on additional risk without a second thought. Do we have a death wish or are we simply masochists? Risk management is the bread and butter of the finance and insurance industries. Isnt it time we evolved? In this fast paced and practical session we will explore answers to the following questions: - What is risk and how do we quantify and manage it? - How do we assess the value of information? - How can experimentation reduce risk and where does it fit in the product development cycle? - What makes a good experiment? - How to run experiments in a cost effective manner? - What are good metrics? - How to obtain Zen like focus and prioritisation? New concepts will be introduced, examples will be given and we will then point out where to seek further information. Hold onto your hats.


  • 1. Business Model Innovation by Experimentation Yoav Aviram Energized Work @yobo
  • 2. IT Projects 60% dont meet schedule, budget or quality goals - IBM 50% failed to achieve what they set out to achieve - KPMG 17% go so badly that they can threaten the very existence of the company. - Calleam Consulting
  • 3. Startups 95% of technology startups fail - Allmand Law 93% of the Angel investments never achieve expected ROI - University of Washington 80% of the VC investments never achieve expected ROI - National VC Association
  • 4. WHY?
  • 5. Too late Poor quality Missing functionality Software: Tip of the Iceberg
  • 6. An iterative and incremental approach to development, where requirements and solutions evolve through collaboration between selforganizing, cross-functional teams - Wikipedia Agile
  • 7. ...the Product Iceberg Who is the customer? How do we acquire new users? How do we convert users to paying customers? How should the product look and behave? How is our product used? What problems does the product attempt to solve? What features provide the greatest value?
  • 8. A combination of business-hypothesis-driven experimentation, iterative product releases, and "validated learning" - The Lean Startup, Eric Ries Lean Startup
  • 9. ...the Business Model Iceberg Who are our customers? Who is the competition? How do we reach our customers? What are the revenue streams? What is the investment needed? What is the operational cost? What is our unique selling point? What team members to recruit? What are our business goals?
  • 10. Business Model Innovation - Business Model Generation, Alexander Osterwalder
  • 11. There is a poorly met need a reachable market faces There is a solution people are happy to pay for There is a way to package and deliver the solution in a cost effective manner Product-Market Fit
  • 12. This is all About Risk
  • 13. The lack of complete certainty, that is, the existence of more than one possibility. The "true" outcome/state/result/value is not known. - How to Measure Anything, Douglas Hubbard Uncertainty
  • 14. A set of probabilities assigned to a set of possibilities Quantifying Uncertainty There is a 60% chance this market will double in five years
  • 15. A state of uncertainty where some of the possibilities involve a loss, catastrophe, or other undesirable outcome. - How to Measure Anything, Douglas Hubbard Risk
  • 16. A set of possibilities each with quantified probabilities and quantified losses Quantifying Risk There is a 40% chance the proposed oil well will be dry, with a loss of $12 million in exploratory drilling costs
  • 17. The Problem with Risk & Technology Technology is used to solve complex problems Our mind is wired to oversimplify complexity (we are not very good at understanding probabilities) We are solution oriented We are optimistic (overconfident) by nature People (grossly) underestimate risk & uncertainty
  • 18. Combination of two or more related risks Compound Risk Compound probabilities are (very) counter intuitive
  • 19. Contestants on a game show are given the choice of three doors. Behind one door is a car, behind the others, goats. After a contestant picks a door, the host, who knows what's behind all the doors, opens an unchosen door, which reveals a goat. He then asks the contestant, "Do you want to switch doors?" Monty Hall
  • 20. Should the contestant switch doors?
  • 21. Yes! The odds of winning are 2 out of 3 if you switch
  • 22. This answer is so counter intuitive, most people get it wrong Its why they made a game show off of it: the trick works every time Compound Probabilities The lesson: We need to give serious consideration to risk We must not rely on our intuition when factoring risk & uncertainty We need a better accounting system
  • 23. Risk Management for Techies Avoid by eliminating the situation or activity that presents it Transfer through insurance or through other types of contracts Reduce by hedging your bets or reducing uncertainty Retain, because some risks are worth assuming
  • 24. Risk Reduction
  • 25. Spread your bets on a portfolio of investments instead of one Decide on an investment strategy Split the investment into small incremental bets: abort bad products quickly and ramp up investment in the good ones Investor Mindset
  • 26. Uncertainty Reduction
  • 27. Value of Information Information reduces uncertainty Reduced uncertainty improves decisions Improved decisions have observable consequences with measurable value - Information Theory (1948)
  • 28. Sources of Information Existing within the company Others research Experiments Live product
  • 29. A scientific procedure undertaken to make a discovery, test a hypothesis, or demonstrate a known fact - Wikipedia Experiment
  • 30. Experimental Discovery
  • 31. Prioritising Experiments Uncertainty around making a decision is high (little existing information) The value of the opportunity, or the cost of a mistake, is high Experimentation is least expensive in terms of cost and time
  • 32. Attempt to falsify a hypothesis Are objective, measurable and repeatable Are controlled (variables tested in isolation) Are cost effective Influence a decision Designing Good Experiments Experiments aren't free It's an investment decision Information may be cheaper elsewhere
  • 33. A measurement is an observation that results in information (reduction of uncertainty) about a quantity Measurement
  • 34. The soft stuff Descriptive, subjective, messy and hard to quantify Customer surveys, focus groups, interviews But provide insights about Perceived Value Qualitative Measurements
  • 35. Good Qualitative Questions Avoid leading phrases such as "do you agree that..." Make customers part with money or sign up, rather than asking them whether they would Use follow up questions to get to the why? Qualitative data can easily be influence by cognitive biases
  • 36. Start by decide on the right Metric (what to measure) Then set lines in the sand: what success and failure look like Always validate results with qualitative data Quantitative Measurements
  • 37. Good Metric Influences a decision Comparative Rates and Ratios are easier to compare and act upon Measure whats important to your customers
  • 38. Putting it all Together Invest wisely, through small incremental bets to reduce risk and discover opportunities Information reduces uncertainty & risk Experiments are not the only source of information Experiments have the biggest impact in a high value, high uncertainty and little data situations Disprove hypotheses, dont confirm them Think in terms of Risk and account for it The distinction between business and technology is anachronistic
  • 39. The Lean Startup by Eric Ries Business Model Generation by Alexander Osterwalder How To Measure Anything by Douglas W. Hubbard Lean Analytics by Alistair Croll and Benjamin Yoskovitz Reading List
  • 40. Questions? Yoav Aviram Energized Work @yobo
  • 41. Frame your business and product as a set of hypotheses Declare your assumptions and how you can prove them wrong (falsifiable) Evaluate your results ruthlessly, and be prepared to change course Accounting for Risk
  • 42. Assumption Backlog Assumption Certainty Risk ($) Critical People like shiny apps 90% $100K (cost of beta) Yes Market size for shiny apps at least 200 million people 50% $1M (cost of a full product) Yes I can build a shiny app 95% $50K (cost of development) Yes People share shiny apps with friends 45% $100K (cost of beta) No 5% of users will upgrade to shiny+ for $5 a month 20% $1M (cost of a full product) Yes 10% of active users will spend $2 a month on shiny accessories 75% $100K (cost of beta) No


View more >