Lean Product Analytics by Dan Olsen

  • Published on
    19-Sep-2014

  • View
    6

  • Download
    3

DESCRIPTION

Talk on how to use analytics to optimize your product that I gave at Yelp on February 5, 2014.

Transcript

<ul><li><p>Lean Product Analytics Dan Olsen Olsen Solutions Feb 5, 2014 </p></li><li><p>Copyright 2014 Olsen Solu7ons </p><p>My Background n Educa7on </p><p>n BS, Electrical Engineering, Northwestern n MS, Industrial Engineering, Virginia Tech n MBA, Stanford n Web development and UI design </p><p>n 20 years of Product Management Experience n Managed submarine design for 5 years n 5 years at Intuit, led Quicken Product Management n Led Product Management at Friendster n CEO &amp; Cofounder of YourVersion, Pandora for your news n Consultant: Box, YouSendIt, Chartboost, One Medical </p><p> Will post slides at hUp://slideshare.net/dan_o </p></li><li><p>Copyright 2014 Olsen Solu7ons </p><p>What does Lean mean? n Lean Startup </p><p>n Achieving product-market t n Tes7ng hypotheses &amp; learning n Valida7ng MVP with users n Improving &amp; itera7ng your product quickly </p><p>n Minimizing waste = using resources eec7vely </p></li><li><p>Copyright 2014 Olsen Solu7ons </p><p>Whats the Formula for Product-Market Fit? </p><p>n A product that: n Meets customers needs n Is beUer than other alterna7ves n Is easy to use n Has a good value/price </p></li><li><p>Dans Model for the Causality Underlying Product-Market Fit </p><p>Copyright 2014 Olsen Solu7ons </p><p>Target Customer </p><p>Product </p><p>Customer Needs </p><p>Customer has needs </p><p>You design &amp; build product to meet needs </p><p>Customer decides how well product meets needs (sa7sfac7on) </p></li><li><p>What are Customers Reac7ng To When They Use Your Product? </p><p>Feature Set </p><p>UX Design Messaging </p><p>Copyright 2014 Olsen Solu7ons </p></li><li><p>Valida7ng New vs. Exis7ng Products New Product Qualita7ve interviews </p><p>Exis0ng Product Quan7ta7ve data </p><p>Oprah Spock </p></li><li><p>How to be a Lean Product Ninja </p><p>+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+ </p><p>+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+ </p><p>slideshare.net/dan_o/ </p></li><li><p>Copyright 2014 Olsen Solu7ons </p></li><li><p>Iden7fy highest ROI idea </p><p>Design and Implement </p><p>Analyze How the Metric Changes </p><p>Brainstorm Ideas to </p><p>Improve Metric </p><p>Copyright 2014 Olsen Solu7ons </p><p>Lean Product Analy7cs Process </p><p>Iden7fy What Your Metrics Are </p><p>Measure Metrics Baseline Values </p><p>Evaluate Metrics Upside Poten7al </p><p>Global Level </p><p>Metric Level </p><p>Select Top Metric </p><p>Learn &amp; Iterate </p></li><li><p>Copyright 2014 Olsen Solu7ons </p><p>n Net Promoter Score </p><p>Valida7ng Product-Market Fit: Surveys </p><p>Key follow-up ques7ons: Why did you give the score you did? What do we need to do to improve? </p></li><li><p>Qualita7ve Compliments Quan7ta7ve </p><p>Copyright 2014 Olsen Solu7ons </p><p>QualWhy? </p><p>QuantWhat? </p></li><li><p>Copyright 2014 Olsen Solu7ons </p><p>n Survey.io / Qualaroo.com n How would you feel if you could no longer use Product X? </p><p>n Very disappointed n Somewhat disappointed n Not disappointed </p><p>n General guideline: 40% or more very disappointed = product-market t </p><p>Valida7ng Product-Market Fit: Surveys </p></li><li><p>Copyright 2014 Olsen Solu7ons </p><p>n Asking a user ques7ons in an interview or survey n Valuable, but n Theyre telling you what they think they would do n Measurement bias (because youre with them) </p><p>n Observing behavior n See what users actually do n Without you there </p><p>n Behavioral metrics for Product-Market Fit: n Prospects sign up = High conversion rate n They keep using it = High reten7on rate n They use it omen = High frequency of use n Theyre deeply engaged with it = Long session 7mes n They pay for it = Revenue per customer </p><p>Product-Market Fit: Actual User Behavior Trumps Opinions </p></li><li><p>Valuable to Have a Holis7c Analy7cs Framework </p><p>Dave McClures Startup Metrics for Pirates </p><p>AARRR</p><p>Focus on right metric at right 7me </p></li><li><p>Using Analy7cs for Op7miza7on </p><p>n In addi7on to Product-Market Fit, you can apply the Lean Product Analy7cs Process to op7mize: nYour Business Results nYour User Experience </p><p>Copyright 2014 Olsen Solu7ons </p></li><li><p>Prot = Revenue - Cost </p><p> Unique Visitors x Ad Revenue per Visitor </p><p> Impressions/Visitor x Eec7ve CPM / 1000 </p><p> Visits/Visitor x Pageviews/Visit x Impressions/PV </p><p> New Visitors + Returning Visitors </p><p> Invited Visitors + Uninvited Visitors </p><p> # of Users Sending Invites x Invites Sent/User x Invite Conversion Rate </p><p>Dene the Equa7on of your Business Peeling the Onion </p><p>Adver7sing Business Model: </p><p>Copyright 2014 Olsen Solu7ons </p></li><li><p>Copyright 2014 Olsen Solu7ons </p><p> ( SEO Visitors + SEM Visitors + Viral Visitors ) x Trial Conversion Rate </p><p> Paying Users x Revenue per Paying User </p><p> New Paying Users + Repeat Paying Users </p><p> Previous Paying Users x ( 1 Cancella7on Rate ) </p><p> Trial Users x Conv Rate </p><p>Prot = Revenue - Cost </p><p>Equa7on of your Business: Subscrip7on Business Model </p></li><li><p>How to Track Your Metrics n Track each metric as daily 7me series </p><p>n Create ra7os from primary metrics: X / Y n Example: How good is your registra7on page? n Okay: # of registered users per day n BeUer: registra7on conversion rate = # registered users / # uniques to reg page </p><p> Date </p><p>Unique Visitors </p><p>Page views </p><p>Ad Revenue </p><p>New User Sign-ups </p><p>4/24/08 10,100 29,600 25 490 </p><p>4/25/08 10,500 27,100 24 480 </p><p>Copyright 2014 Olsen Solu7ons </p></li><li><p>Registra7on Page Conversion Rate </p><p>Daily Signup Page Yield vs. TimeNew Registered Users divided by Unique Visitors to Signup Page</p><p>0%</p><p>10%</p><p>20%</p><p>30%</p><p>40%</p><p>50%</p><p>60%</p><p>70%</p><p>80%</p><p>90%</p><p>100%</p><p>1/31 2/14 2/28 3/14 3/28 4/11 4/25 5/9 5/23 6/6 6/20 7/4 7/18 8/1 8/15 8/29 9/12 9/26 10/10</p><p>Dai</p><p>ly S</p><p>ignu</p><p>p Pa</p><p>ge Y</p><p>ield</p><p>Changedmessaging</p><p>Added questionsto signup page</p><p>Started requiringregistration</p><p>Copyright 2014 Olsen Solu7ons </p><p>Registration Page Conversion Rate vs. Time </p><p>Reg</p><p>istra</p><p>tion </p><p>Pag</p><p>e C</p><p>onve</p><p>rsio</p><p>n R</p><p>ate </p></li><li><p>View Each Business Metric as a Gauge </p><p>Copyright 2014 Olsen Solu7ons </p><p>Minimum Possible Value </p><p>Maximum Possible Value </p><p>Current Value </p></li><li><p>Copyright 2014 Olsen Solu7ons </p><p>Priori7zing Product Ideas by ROI </p><p>Investment (developer-weeks) </p><p>Return (V</p><p>alue</p><p> Created</p><p>) </p><p>Idea C </p><p>Idea B </p><p>Idea D </p><p>Idea A </p><p>Idea F </p><p> 1 </p><p> 1 </p><p>2 3 4 </p><p>2 </p><p>3 </p><p>4 ? </p></li><li><p>Iden7fying the Cri7cal Few Metrics n What is the upside poten7al of each metric? n How many resources will it take to move the needle? </p><p>n Developer-days, 7me, money n How much will the needle move? Revenue impact? n Which metrics have highest ROI opportuni7es? </p><p>Return </p><p>Investment </p><p>Return </p><p>Investment Re</p><p>turn </p><p>Investment </p><p>Metric A Good ROI </p><p>Metric B Bad ROI </p><p>Metric C Great ROI </p><p>Copyright 2014 Olsen Solu7ons </p></li><li><p>Case Study from Intuit </p><p>q Improving UX q Improving Business Results </p><p>-&gt; Sign-up Conversion Rate </p><p>Copyright 2014 Olsen Solu7ons </p></li><li><p>Abandonment Rate (7 Day Moving Average)</p><p>0%</p><p>10%</p><p>20%</p><p>30%</p><p>40%</p><p>50%</p><p>60%</p><p>70%</p><p>80%</p><p>10/7</p><p>/02</p><p>10/1</p><p>4/02</p><p>10/2</p><p>1/02</p><p>10/2</p><p>8/02</p><p>11/4</p><p>/02</p><p>11/1</p><p>1/02</p><p>11/1</p><p>8/02</p><p>11/2</p><p>5/02</p><p>12/2</p><p>/02</p><p>12/9</p><p>/02</p><p>12/1</p><p>6/02</p><p>12/2</p><p>3/02</p><p>12/3</p><p>0/02</p><p>1/6/</p><p>03</p><p>1/13</p><p>/03</p><p>1/20</p><p>/03</p><p>Aba</p><p>ndon</p><p>men</p><p>t Rat</p><p>e (7</p><p> Day</p><p> Mov</p><p>ing </p><p>Ave</p><p>rage</p><p>)</p><p>Steps 1-2</p><p>Copyright 2014 Olsen Solu7ons </p><p>Case Study: Account Signup Process Redesign </p></li><li><p>Copyright 2014 Olsen Solu7ons </p><p>Analyzed Drop-O at Each Major Sec7on </p><p>100%</p><p>62.3%58.8%</p><p>50.9%</p><p>34.4% 32.7%</p><p>0%</p><p>20%</p><p>40%</p><p>60%</p><p>80%</p><p>100%</p><p>% of U</p><p>sers </p><p>Sign in / Registra7on </p><p>Account Type Cash vs. Margin </p><p>5 Partner Pages </p><p>3 Partner Pages </p><p>Focus on biggest drop </p></li><li><p>Copyright 2014 Olsen Solu7ons </p><p>Open Account </p><p>Sign in </p><p>Account Selec7on </p><p>Register </p><p>56% </p><p>44% </p><p>Forget Password </p><p>Registra7on Process </p><p>45% drop o (20% of total) </p><p>36% overall drop o for this step </p><p>70% (32% of Total) </p><p>17% drop o (10% of total) </p><p>20% drop o (6% of total) </p><p>30% (14% of Total) </p><p>80% (26% of Total) </p><p>55% (24% of Total) </p><p>64% of Total </p><p>Analysis of Sign In/Registra7on Flow </p><p>Change Password </p><p>83% (46% of Total) </p></li><li><p>Abandonment Rate (7 Day Moving Average)</p><p>0%</p><p>10%</p><p>20%</p><p>30%</p><p>40%</p><p>50%</p><p>60%</p><p>70%</p><p>80%</p><p>10/7</p><p>/02</p><p>10/1</p><p>4/02</p><p>10/2</p><p>1/02</p><p>10/2</p><p>8/02</p><p>11/4</p><p>/02</p><p>11/1</p><p>1/02</p><p>11/1</p><p>8/02</p><p>11/2</p><p>5/02</p><p>12/2</p><p>/02</p><p>12/9</p><p>/02</p><p>12/1</p><p>6/02</p><p>12/2</p><p>3/02</p><p>12/3</p><p>0/02</p><p>1/6/</p><p>03</p><p>1/13</p><p>/03</p><p>1/20</p><p>/03</p><p>Aba</p><p>ndon</p><p>men</p><p>t Rat</p><p>e (7</p><p> Day</p><p> Mov</p><p>ing </p><p>Ave</p><p>rage</p><p>)</p><p>Steps 1-2</p><p>Copyright 2014 Olsen Solu7ons </p><p>Redesigned User Flow Improved Registra7on Conversion Rate </p><p>37% improvement in conversion rate </p><p>Released New Design </p></li><li><p>Case Study from Friendster </p><p>q Improving Business Results -&gt; Viral New User Growth </p><p>Copyright 2014 Olsen Solu7ons </p></li><li><p> Which metric has highest ROI opportunity? </p><p>Case Study: Op7mizing Friendsters Viral Loop </p><p>Active Users </p><p>Prospective Users </p><p>Invite Click </p><p>Succeed </p><p>Invite click-through rate </p><p>Conversion rate </p><p>Dont Click </p><p>Fail </p><p>Invites per sender </p><p>% of users sending invites </p><p> Mul7plied together, these metrics determine your viral ra7o </p><p>Users </p><p>% of users who are active </p><p>= 15% = 2.3 </p><p>= 85% </p><p>Registration Process </p><p>Copyright 2014 Olsen Solu7ons </p></li><li><p>The Upside Poten7al of a Metric </p><p>0 </p><p>100% </p><p>0 </p><p>100% </p><p>0 </p><p>? </p><p>Registra7on Process Yield </p><p>% of users sending invita7ons </p><p>Avg # of invites sent per sender </p><p>2.3 </p><p>85% </p><p>15% </p><p>Max possible improvement </p><p>0.15 / 0.85 = 18% 0.85 / 0.15 = 570% ? / 2.3 = ?% </p><p>Copyright 2014 Olsen Solu7ons </p></li><li><p>Okay, so how can we improve the metric? </p><p>n How do we increase the average number of invites being sent out per sender? </p><p>n For each idea: n Whats the expected benet? (how much will it improve the metric?) </p><p>n Whats the expected cost? (how many engineer-days will it take?) </p><p>n You want to iden7fy highest ROI idea </p><p>Copyright 2014 Olsen Solu7ons </p></li><li><p>Amer Launching Address Book Importer </p><p>Copyright 2014 Olsen Solu7ons </p></li><li><p>Amer Launching Address Book Importer </p><p>Copyright 2014 Olsen Solu7ons </p></li><li><p>Amer Launching Address Book Importer </p><p>Copyright 2014 Olsen Solu7ons </p></li><li><p>If you could only track 1 metric to measure your Product-Market Fit: </p><p>Which metric would it be? </p><p>Copyright 2014 Olsen Solu7ons </p></li><li><p>Reten7on Rate nReten7on rate tracks what % of your customers are s7ll ac7ve over 7me </p><p>Copyright 2014 Olsen Solu7ons </p><p>~80% never use app again </p><p>Curve eventually flattens out </p></li><li><p>Cohort Analysis </p><p>Copyright 2014 Olsen Solu7ons </p></li><li><p>Cohort Analysis: Data </p><p>Copyright 2014 Olsen Solu7ons </p></li><li><p>Improving Reten7on Rate Over Time= Increasing Product-Market Fit </p><p>David Skok, Matrix Partners hUp://www.forentrepreneurs.com/saas-metrics-2/ </p></li><li><p>Alternate Ways to Track Reten7on nHaving lots of cohort curves is hard to read nWould be great to have a 7me series metric = one metric we can track over 7me </p><p>n% Users Retained who signed up X days ago nCan use single or mul7ple X (30 &amp; 90 days) </p><p>nAnother metric: Returning users nGood summary metric: # of users locking in nGives a sense of scale (not a %) nRecommend 7-day average (can do others too) </p><p>Copyright 2014 Olsen Solu7ons </p></li><li><p>Protability, anyone? </p><p>Two key metrics: Customer Life7me Value (LTV) Customer Acquisi7on Cost (CAC) </p><p> You want: </p><p>LTV CAC &gt; 0 </p></li><li><p>Protability, anyone? </p></li><li><p>Protability, anyone? </p><p>Two key metrics: Customer Life7me Value (LTV) Customer Acquisi7on Cost (CAC) </p><p> You want: </p><p>LTV CAC &gt; 0 </p></li><li><p>Life7me Value (LTV) n Life7me value of a customer = how much value your average customer will generate </p><p>n LTV = ARPU x Avg Customer Life7me x Gross Margin n ARPU (Avg Revenue / User) = Total Revenue / # of Users n Average Customer Life7me </p><p>n How long your average customer generates revenue n Equals 1 / churn rate (5% monthly churn = avg life 20 months) </p><p>n Gross Margin: the % of revenues lem over amer subtrac7ng the cost of providing the product/service </p><p>Copyright 2014 Olsen Solu7ons </p><p>Note: for simplicity, this LTV equa7on ignores the cost of capital </p></li><li><p>Customer Acquisi7on Cost (CAC) </p><p>nCAC is the average cost for you to obtain a revenue-genera7ng customer </p><p>nSo it takes into account both your cost of acquiring a prospect and your conversion rate for conver7ng prospects to revenue-genera7ng customers </p><p>nCAC=Cost per Acquisi7on / Conversion Rate </p><p>Copyright 2014 Olsen Solu7ons </p></li><li><p>What Youd Like to See Over Time </p><p>Copyright 2014 Olsen Solu7ons </p><p>n LTV increasing as you improve your value proposi7on, customer reten7on, &amp; pricing </p><p>n CAC decreasing as you op7mize your marke7ng: segments, channels, messaging </p></li><li><p>Ra7o of LTV to CAC: Real data from HubSpot </p><p>Copyright 2014 Olsen Solu7ons </p></li><li><p>Iden7fy highest ROI idea </p><p>Design and Implement </p><p>Analyze How the Metric Changes </p><p>Brainstorm Ideas to </p><p>Improve Metric </p><p>Copyright 2014 Olsen Solu7ons </p><p>Lean Product Analy7cs Process </p><p>Iden7fy What Your Metrics Are </p><p>Measure Metrics Baseline Values </p><p>Evaluate Metrics Upside Poten7al </p><p>Metric Level </p><p>Select Top Metric </p><p>Learn &amp; Iterate </p><p>Global Level </p></li><li><p> Questions? olsensolutions.com </p><p>linkedin.com/in/danolsen98 </p><p>@danolsen </p><p>Copyright 2014 Olsen Solu7ons </p></li></ul>

Recommended

View more >