Transcript
Page 1: Lean Product Analytics by Dan Olsen

Lean Product Analytics Dan Olsen Olsen Solutions Feb 5, 2014

Page 2: Lean Product Analytics by Dan Olsen

Copyright  ©  2014  Olsen  Solu7ons  

My  Background  n  Educa7on  

n  BS,  Electrical  Engineering,  Northwestern  n  MS,  Industrial  Engineering,  Virginia  Tech  n  MBA,  Stanford  n  Web  development  and  UI  design  

n  20  years  of  Product  Management  Experience  n  Managed  submarine  design  for  5  years  n  5  years  at  Intuit,  led  Quicken  Product  Management  n  Led  Product  Management  at  Friendster  n  CEO  &  Cofounder  of  YourVersion,  “Pandora  for  your  news”  n  Consultant:  Box,  YouSendIt,  Chartboost,  One  Medical  

 Will  post  slides  at  hUp://slideshare.net/dan_o    

Page 3: Lean Product Analytics by Dan Olsen

Copyright  ©  2014  Olsen  Solu7ons  

What  does  “Lean”  mean?  n  Lean  Startup  

n  Achieving  product-­‐market  fit  n  Tes7ng  hypotheses  &  learning  n  Valida7ng  MVP  with  users  n  Improving  &  itera7ng  your  product  quickly  

n Minimizing  waste  =  using  resources  effec7vely  

Page 4: Lean Product Analytics by Dan Olsen

Copyright  ©  2014  Olsen  Solu7ons  

What’s  the  Formula  for  Product-­‐Market  Fit?  

n  A  product  that:  n Meets  customers’  needs  n  Is  beUer  than  other  alterna7ves  n  Is  easy  to  use  n Has  a  good  value/price  

Page 5: Lean Product Analytics by Dan Olsen

Dan’s  Model  for  the  Causality  Underlying  Product-­‐Market  Fit  

Copyright  ©  2014  Olsen  Solu7ons  

Target  Customer  

Product  

Customer  Needs  

Customer  has  needs  

You  design  &  build  product  to  meet  needs  

Customer  decides  how  well  product  meets  needs  (sa7sfac7on)  

Page 6: Lean Product Analytics by Dan Olsen

What  are  Customers  Reac7ng  To  When  They  Use  Your  Product?  

Feature  Set  

UX  Design   Messaging    

Copyright  ©  2014  Olsen  Solu7ons  

Page 7: Lean Product Analytics by Dan Olsen

Valida7ng  New  vs.  Exis7ng  Products  New  Product  Qualita7ve  interviews  

Exis0ng  Product  Quan7ta7ve  data  

Oprah   Spock  

Page 8: Lean Product Analytics by Dan Olsen

How to be a Lean Product Ninja

+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+

+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+

slideshare.net/dan_o/

Page 9: Lean Product Analytics by Dan Olsen

Copyright  ©  2014  Olsen  Solu7ons  

Page 10: Lean Product Analytics by Dan Olsen

Iden7fy  highest  ROI  idea  

Design  and  Implement  

Analyze  How  the  Metric  Changes  

Brainstorm  Ideas  to  

Improve  Metric  

Copyright  ©  2014  Olsen  Solu7ons  

Lean  Product  Analy7cs  Process  

Iden7fy  What  Your    Metrics  Are  

Measure  Metrics  Baseline  Values  

Evaluate  Metrics  Upside  Poten7al  

Global  Level  

Metric  Level  

Select  Top  Metric  

Learn  &  Iterate  

Page 11: Lean Product Analytics by Dan Olsen

Copyright  ©  2014  Olsen  Solu7ons  

n  Net  Promoter  Score  

Valida7ng  Product-­‐Market  Fit:  Surveys  

Key  follow-­‐up  ques7ons:  •  Why  did  you  give  the  score  you  did?  •  What  do  we  need  to  do  to  improve?  

Page 12: Lean Product Analytics by Dan Olsen

Qualita7ve  Compliments  Quan7ta7ve  

Copyright  ©  2014  Olsen  Solu7ons  

QualWhy?

QuantWhat?

Page 13: Lean Product Analytics by Dan Olsen

Copyright  ©  2014  Olsen  Solu7ons  

n  Survey.io  /  Qualaroo.com  n  “How  would  you  feel  if  you  could  no  longer  use  Product  X?”  

n  Very  disappointed  n  Somewhat  disappointed  n  Not  disappointed  

n  General  guideline:    40%  or  more  “very  disappointed”  =  product-­‐market  fit  

Valida7ng  Product-­‐Market  Fit:  Surveys  

Page 14: Lean Product Analytics by Dan Olsen

Copyright  ©  2014  Olsen  Solu7ons  

n  Asking  a  user  ques7ons  in  an  interview  or  survey  n  Valuable,  but…  n  They’re  telling  you  what  they  think  they  would  do  n  Measurement  bias  (because  you’re  with  them)  

n  Observing  behavior  n  See  what  users  actually  do  n  Without  you  there  

n  Behavioral  metrics  for  Product-­‐Market  Fit:  n  Prospects  sign  up  =  High  conversion  rate  n  They  keep  using  it  =  High  reten7on  rate  n  They  use  it  omen  =  High  frequency  of  use  n  They’re  deeply  engaged  with  it  =  Long  session  7mes  n  They  pay  for  it  =  Revenue  per  customer  

Product-­‐Market  Fit:  Actual  User  Behavior  Trumps  Opinions  

Page 15: Lean Product Analytics by Dan Olsen

Valuable  to  Have  a  Holis7c  Analy7cs  Framework  

Dave  McClure’s  “Startup  Metrics  for  Pirates”  

AARRR

Focus  on  right  metric  at  right  7me  

Page 16: Lean Product Analytics by Dan Olsen

Using  Analy7cs  for  Op7miza7on  

n  In  addi7on  to  Product-­‐Market  Fit,  you  can  apply  the  Lean  Product  Analy7cs  Process  to  op7mize:  n Your  Business  Results  n Your  User  Experience  

Copyright  ©  2014  Olsen  Solu7ons  

Page 17: Lean Product Analytics by Dan Olsen

Profit  =  Revenue  -­‐  Cost    

     

         

         

     

     

   

   

 Unique  Visitors    x    Ad  Revenue  per  Visitor    

     

     

     

     

   

   

     

     Impressions/Visitor    x    Effec7ve  CPM  /  1000    

         

     

     

   

   

     

         

     Visits/Visitor    x    Pageviews/Visit    x    Impressions/PV    

     

     

   

   

     

         

         

 New  Visitors  +  Returning  Visitors    

     

   

   

     

         

         

     

 Invited  Visitors  +  Uninvited  Visitors    

   

   

     

         

         

     

     

 #  of  Users  Sending  Invites    x    Invites  Sent/User    x    Invite  Conversion  Rate  

Define  the  Equa7on  of  your  Business  “Peeling  the  Onion”  

Adver7sing  Business  Model:  

Copyright  ©  2014  Olsen  Solu7ons  

Page 18: Lean Product Analytics by Dan Olsen

Copyright  ©  2014  Olsen  Solu7ons  

               

 (  SEO  Visitors  +  SEM  Visitors  +  Viral  Visitors  )    x    Trial  Conversion  Rate      

   

 Paying  Users    x    Revenue  per  Paying  User    

           

       

 New  Paying  Users    +    Repeat  Paying  Users      

       

           

         Previous  Paying  Users    x    (  1  –  Cancella7on  Rate  )    

   

           

 Trial  Users    x    Conv  Rate        

   

       

     

     

   

Profit  =  Revenue  -­‐  Cost    

     

           

         

     

     

   

Equa7on  of  your  Business:  Subscrip7on  Business  Model  

Page 19: Lean Product Analytics by Dan Olsen

How  to  Track  Your  Metrics  n  Track  each  metric  as  daily  7me  series  

n  Create  ra7os  from  primary  metrics:    X  /  Y  n  Example:  How  good  is  your  registra7on  page?  n Okay:  #  of  registered  users  per  day  n  BeUer:  registra7on  conversion  rate  =      #  registered  users  /  #  uniques  to  reg  page  

 Date  

Unique  Visitors  

Page  views  

Ad  Revenue  

New  User  Sign-­‐ups   …  

4/24/08   10,100   29,600   25   490  

4/25/08   10,500   27,100   24   480  

…  

Copyright  ©  2014  Olsen  Solu7ons  

Page 20: Lean Product Analytics by Dan Olsen

Registra7on  Page  Conversion  Rate  

Daily Signup Page Yield vs. TimeNew Registered Users divided by Unique Visitors to Signup Page

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

1/31 2/14 2/28 3/14 3/28 4/11 4/25 5/9 5/23 6/6 6/20 7/4 7/18 8/1 8/15 8/29 9/12 9/26 10/10

Dai

ly S

ignu

p Pa

ge Y

ield

Changedmessaging

Added questionsto signup page

Started requiringregistration

Copyright  ©  2014  Olsen  Solu7ons  

Registration Page Conversion Rate vs. Time

Reg

istra

tion

Pag

e C

onve

rsio

n R

ate

Page 21: Lean Product Analytics by Dan Olsen
Page 22: Lean Product Analytics by Dan Olsen

View  Each  Business  Metric  as  a  Gauge  

Copyright  ©  2014  Olsen  Solu7ons  

Minimum  Possible  Value  

Maximum  Possible  Value  

Current  Value  

Page 23: Lean Product Analytics by Dan Olsen

Copyright  ©  2014  Olsen  Solu7ons  

Priori7zing  Product  Ideas  by  ROI  

Investment  (developer-­‐weeks)  

Return  (V

alue

 Created

)  

Idea  C  

Idea  B  

Idea  D  

Idea  A  

Idea  F  

 1  

 1  

2    3   4  

2  

3  

4  ?

Page 24: Lean Product Analytics by Dan Olsen

Iden7fying  the  “Cri7cal  Few”  Metrics  n  What  is  the  upside  poten7al  of  each  metric?  n  How  many  resources  will  it  take  to  “move  the  needle”?  

n  Developer-­‐days,  7me,  money  n  How  much  will  the  needle  move?  Revenue  impact?  n  Which  metrics  have  highest  ROI  opportuni7es?  

Return  

Investment  

Return  

Investment  Re

turn  

Investment  

Metric  A  Good  ROI  

Metric  B  Bad  ROI  

Metric  C  Great  ROI  

Copyright  ©  2014  Olsen  Solu7ons  

Page 25: Lean Product Analytics by Dan Olsen

Case  Study  from  Intuit  

q  Improving  UX  q  Improving  Business  Results  

-­‐>  Sign-­‐up  Conversion  Rate  

Copyright  ©  2014  Olsen  Solu7ons  

Page 26: Lean Product Analytics by Dan Olsen

Abandonment Rate (7 Day Moving Average)

0%

10%

20%

30%

40%

50%

60%

70%

80%

10/7

/02

10/1

4/02

10/2

1/02

10/2

8/02

11/4

/02

11/1

1/02

11/1

8/02

11/2

5/02

12/2

/02

12/9

/02

12/1

6/02

12/2

3/02

12/3

0/02

1/6/

03

1/13

/03

1/20

/03

Aba

ndon

men

t Rat

e (7

Day

Mov

ing

Ave

rage

)

Steps 1-2

Copyright  ©  2014  Olsen  Solu7ons  

Case  Study:  Account  Signup  Process  Redesign  

Page 27: Lean Product Analytics by Dan Olsen

Copyright  ©  2014  Olsen  Solu7ons  

Analyzed  Drop-­‐Off  at  Each  Major  Sec7on  

100%

62.3%58.8%

50.9%

34.4% 32.7%

0%

20%

40%

60%

80%

100%

%  of  U

sers  

Sign  in  /  Registra7on  

Account  Type   Cash  vs.  Margin  

5  Partner  Pages  

3  Partner  Pages  

Focus  on  biggest  drop  

Page 28: Lean Product Analytics by Dan Olsen

Copyright  ©  2014  Olsen  Solu7ons  

Open  Account  

Sign  in    

Account  Selec7on  

Register    

56%  

44%  

Forget  Password  

Registra7on  Process  

45%  drop  off  (20%  of  total)  

36%  overall  drop  off  for  this  step  

70%  (32%  of  Total)  

17%  drop  off  (10%  of  total)  

20%  drop  off  (6%  of  total)  

30%  (14%  of  Total)  

80%  (26%  of  Total)  

55%  (24%  of  Total)  

64%  of  Total  

Analysis  of  Sign  In/Registra7on  Flow  

Change  Password  

83%  (46%  of  Total)  

Page 29: Lean Product Analytics by Dan Olsen

Abandonment Rate (7 Day Moving Average)

0%

10%

20%

30%

40%

50%

60%

70%

80%

10/7

/02

10/1

4/02

10/2

1/02

10/2

8/02

11/4

/02

11/1

1/02

11/1

8/02

11/2

5/02

12/2

/02

12/9

/02

12/1

6/02

12/2

3/02

12/3

0/02

1/6/

03

1/13

/03

1/20

/03

Aba

ndon

men

t Rat

e (7

Day

Mov

ing

Ave

rage

)

Steps 1-2

Copyright  ©  2014  Olsen  Solu7ons  

Redesigned  User  Flow  Improved  Registra7on  Conversion  Rate  

37% improvement in conversion rate

Released New Design

Page 30: Lean Product Analytics by Dan Olsen

Case  Study  from  Friendster  

q  Improving  Business  Results  -­‐>  Viral  New  User  Growth  

Copyright  ©  2014  Olsen  Solu7ons  

Page 31: Lean Product Analytics by Dan Olsen

•   Which  metric  has  highest  ROI  opportunity?  

Case  Study:  Op7mizing  Friendster’s  Viral  Loop  

Active Users

Prospective Users

Invite Click

Succeed

Invite click-through rate

Conversion rate

Don’t Click

Fail

Invites per sender

% of users sending invites

•   Mul7plied  together,  these  metrics  determine  your  viral  ra7o  

Users

% of users who are active

= 15% = 2.3

= 85%

Registration Process

Copyright  ©  2014  Olsen  Solu7ons  

Page 32: Lean Product Analytics by Dan Olsen

The  Upside  Poten7al  of  a  Metric  

0  

100%  

0  

100%  

0  

?  

Registra7on  Process  Yield  

%  of  users  sending  invita7ons  

Avg  #  of  invites  sent  per  sender  

2.3  

85%  

15%  

Max  possible  improvement  

0.15  /  0.85  =  18%   0.85  /  0.15  =  570%   ?  /  2.3  =  ?%  

Copyright  ©  2014  Olsen  Solu7ons  

Page 33: Lean Product Analytics by Dan Olsen

Okay,  so  how  can  we  improve  the  metric?  

n  How  do  we  increase  the  average  number  of  invites  being  sent  out  per  sender?  

n  For  each  idea:  n What’s  the  expected  benefit?  (how  much  will  it  improve  the  metric?)  

n What’s  the  expected  cost?  (how  many  engineer-­‐days  will  it  take?)  

n  You  want  to  iden7fy  highest  ROI  idea  

Copyright  ©  2014  Olsen  Solu7ons  

Page 34: Lean Product Analytics by Dan Olsen

Amer  Launching  Address  Book  Importer…  

Copyright  ©  2014  Olsen  Solu7ons  

Page 35: Lean Product Analytics by Dan Olsen

Amer  Launching  Address  Book  Importer…  

Copyright  ©  2014  Olsen  Solu7ons  

Page 36: Lean Product Analytics by Dan Olsen

Amer  Launching  Address  Book  Importer…  

Copyright  ©  2014  Olsen  Solu7ons  

Page 37: Lean Product Analytics by Dan Olsen

If  you  could  only  track  1  metric  to  measure  your  Product-­‐Market  Fit:  

Which  metric  would  it  be?  

Copyright  ©  2014  Olsen  Solu7ons  

Page 38: Lean Product Analytics by Dan Olsen

Reten7on  Rate  n Reten7on  rate  tracks  what  %  of  your  customers  are  s7ll  ac7ve  over  7me  

Copyright  ©  2014  Olsen  Solu7ons  

~80% never use app again

Curve eventually flattens out

Page 39: Lean Product Analytics by Dan Olsen

Cohort  Analysis  

Copyright  ©  2014  Olsen  Solu7ons  

Page 40: Lean Product Analytics by Dan Olsen

Cohort  Analysis:  Data  

Copyright  ©  2014  Olsen  Solu7ons  

Page 41: Lean Product Analytics by Dan Olsen

Improving  Reten7on  Rate  Over  Time=  Increasing  Product-­‐Market  Fit  

David  Skok,  Matrix  Partners  hUp://www.forentrepreneurs.com/saas-­‐metrics-­‐2/  

Page 42: Lean Product Analytics by Dan Olsen

Alternate  Ways  to  Track  Reten7on  n Having  lots  of  cohort  curves  is  hard  to  read  n Would  be  great  to  have  a  7me  series  metric  =  one  metric  we  can  track  over  7me  

n %  Users  Retained  who  signed  up  X  days  ago  n Can  use  single  or  mul7ple  X  (30  &  90  days)  

n Another  metric:  Returning  users  n Good  summary  metric:  #  of  users  “locking  in”  n Gives  a  sense  of  scale  (not  a  %)  n Recommend  7-­‐day  average  (can  do  others  too)  

Copyright  ©  2014  Olsen  Solu7ons  

Page 43: Lean Product Analytics by Dan Olsen

Profitability,  anyone?  

Two  key  metrics:  •  Customer  Life7me  Value  (LTV)  •  Customer  Acquisi7on  Cost  (CAC)  

 You  want:  

LTV  –  CAC  >  0  

Page 44: Lean Product Analytics by Dan Olsen

Profitability,  anyone?  

Page 45: Lean Product Analytics by Dan Olsen

Profitability,  anyone?  

Two  key  metrics:  •  Customer  Life7me  Value  (LTV)  •  Customer  Acquisi7on  Cost  (CAC)  

 You  want:  

LTV  –  CAC  >  0  

Page 46: Lean Product Analytics by Dan Olsen

Life7me  Value  (LTV)  n  Life7me  value  of  a  customer  =  how  much  value  your  average  customer  will  generate  

n  LTV  =  ARPU  x  Avg  Customer  Life7me  x  Gross  Margin  n  ARPU  (Avg  Revenue  /  User)  =  Total  Revenue  /  #  of  Users  n  Average  Customer  Life7me  

n  How  long  your  average  customer  generates  revenue  n  Equals  1  /  churn  rate    (5%  monthly  churn  =  avg  life  20  months)  

n  Gross  Margin:  the  %  of  revenues  lem  over  amer  subtrac7ng  the  cost  of  providing  the  product/service  

Copyright  ©  2014  Olsen  Solu7ons  

Note:  for  simplicity,  this  LTV  equa7on  ignores  the  “cost  of  capital”  

Page 47: Lean Product Analytics by Dan Olsen

Customer  Acquisi7on  Cost  (CAC)  

n CAC  is  the  average  cost  for  you  to  obtain  a  revenue-­‐genera7ng  customer  

n So  it  takes  into  account  both  your  cost  of  acquiring  a  prospect  and  your  conversion  rate  for  conver7ng  prospects  to  revenue-­‐genera7ng  customers  

n CAC=Cost  per  Acquisi7on  /  Conversion  Rate  

Copyright  ©  2014  Olsen  Solu7ons  

Page 48: Lean Product Analytics by Dan Olsen

What  You’d  Like  to  See  Over  Time  

Copyright  ©  2014  Olsen  Solu7ons  

n  LTV  increasing  as  you  improve  your  value  proposi7on,  customer  reten7on,  &  pricing  

n  CAC  decreasing  as  you  op7mize  your  marke7ng:  segments,  channels,  messaging  

Page 49: Lean Product Analytics by Dan Olsen

Ra7o  of  LTV  to  CAC:  Real  data  from  HubSpot  

Copyright  ©  2014  Olsen  Solu7ons  

Page 50: Lean Product Analytics by Dan Olsen

Iden7fy  highest  ROI  idea  

Design  and  Implement  

Analyze  How  the  Metric  Changes  

Brainstorm  Ideas  to  

Improve  Metric  

Copyright  ©  2014  Olsen  Solu7ons  

Lean  Product  Analy7cs  Process  

Iden7fy  What  Your    Metrics  Are  

Measure  Metrics  Baseline  Values  

Evaluate  Metrics  Upside  Poten7al  

Metric  Level  

Select  Top  Metric  

Learn  &  Iterate  

Global  Level  

Page 51: Lean Product Analytics by Dan Olsen

Questions? olsensolutions.com

linkedin.com/in/danolsen98

@danolsen

Copyright  ©  2014  Olsen  Solu7ons