Conversion  Op,miza,on:    The  World  Beyond    

Headlines  and  Bu:on  Color  Patrick  McKenzie  Founder  

Kalzumeus  So>ware  @pa,o11  

 

Low  hanging  fruit  tastes  great,  but  the  rest  is  worth  the  work,  too.  

Test  Idea  #1  Quotas  /  Tier  Levels  /  etc  

@user.card_limit  =  ab_test("user_card_limit_May_2012",  [8,  15],  :conversion  =>  "purchase")

What  Did  We  Learn  From  That  Test?  

•  ~60%  increase  in  sales  going  from  15  -­‐>  8  cards  •  Synergis,c  effects  can  occasionally  actually  happen.  – More  on  that  later.  

•  You  don’t  know  what  you  don’t  know  un,l  you  test  for  it.    This  post  on  HN  was  me  losing  $100,000:  

Key  Points  With  TesPng  Quotas  

•  Grandfather  in  exis,ng  users.  •  Set  quota  levels  intelligently.  – Quotas  should  segment  people  by  willingness  to  pay  – Break  out  a  histogram  of  exis,ng  accounts  

•  Be  ready  to  tolerate  small  up,ck  in  support  load.  •  Your  product  is  s%ll  too  cheap.  

Test  Idea  #2  Payment  Methods  /  Checkout  Flow  

Threw  This  Together  In  An  Hour  (Oops)  

What  We  Learned  From  This  Test  

•  A  single  A/B  test  can  help  your  boZom  line  more  than  a  product  launch.  •  Dealing  with  synergis,c  effects  in  tes,ng  can  occasionally  be  maddening.  –  First  a:empts  at  implemen,ng  on-­‐site  checkout  had  null  result  a>er  null  result  

–  Combina,on  of  on-­‐site  checkout  plus  redesign  plus  aforemen,oned  card  limit  test  plus  hillclimbing  won  big  

•  The  data  is  never  wrong,  but  some,mes  the  data  isn’t  telling  you  what  you  think  it  is.  

Key  Points  With  TesPng  Checkout  Flow  

• Make  absolutely  sure  you  don’t  break  this!  •  Have  UX-­‐focused  team  (product  /  marke,ng  /  etc)  go  over  every  step  of  checkout  funnel  with  zero  tolerance  for  bad  UX.  •  Checkout  flows  go  “stale”  at  many  successful  companies.    Schedule  a  deep  dive  on  them  twice  a  year,  at  least.  •  Upsells  and  cross  selling  are  classics  for  a  reason!  

Test  Idea  #3  Offline  Business  Processes  

TesPng  degree  of  sales  which  maximizes  conversion  of  leads  

What  We  Learned  From  This  Test  

•  Important  to  communicate  with  sales/CS.  – Allay  fears  about  being  replaced,  leads  creamed,  etc.  – Demonstrate  it  directly  makes  them  money.  

•  Lead  to  resource-­‐intensive  but  hugely  successful  lead-­‐scoring  efforts.  •  Jus,fied  very  expensive  interven,ons  like  e.g.  onboarding  help  for  everybody.  

Key  Points  With  TesPng  Offline  Processes  

•  Your  so>ware/workflow  has  to  close  the  loop  back  to  the  tes,ng  so>ware.    This  can  be  tricky.  •  Virtually  no  CRM  or  A/B  tes,ng  so>ware  accurately  models  enterprise  decisionmaking  process.    You  might  be  forced  to  op,mize  for  proxy  conversions.  •  Volumes  required  for  sta,s,cal  significance  are  really  punishing,  so  some,mes  we  “proxy  and  pray.”  

Test  Idea  #4  Product  On-­‐boarding  /  FRE  

What  Engineers  Will  Ship  If  You  Let  Them  

Key  Goals  of  Onboarding  Process  

•  Get  user  to  a  concrete  success  as  soon  as  possible.  •  For  more  complex  sales,  lay  groundwork  for  future  success  by  e.g.  invi,ng  in  team  or  gepng  permission  to  follow-­‐up.  •  For  B2C,  especially  in  social,  you  live  and  die  on  virality  during  the  onboarding  experience.  

What  We  Learned  From  This  Test  

•  Users  who  start  onboarding  tour  25%  to  100%+  more  likely  to  purchase  so>ware  •  Users  some,mes  have  visceral  reac,on  to  elements  of  the  onboarding  experience.  – The  wise-­‐alecky  Kiwi  – “Beau,ful  So>ware  Lady  Susan”  

•  Engineers  prefer  in-­‐app  onboarding  but  it  is  expensive  as  heck  to  build  and  does  not  always  jus,fy  itself  versus  e.g.  a  Youtube  video.  

Key  Points  With  TesPng  Offline  Processes  

•  Your  so>ware/workflow  has  to  close  the  loop  back  to  the  tes,ng  so>ware.    This  can  be  tricky.  •  Virtually  no  CRM  or  A/B  tes,ng  so>ware  accurately  models  enterprise  decisionmaking  process.    You  might  be  forced  to  op,mize  for  proxy  conversions.  •  Volumes  required  for  sta,s,cal  significance  are  really  punishing,  so  some,mes  we  “proxy  and  pray.”  

Test  Idea  #5  Core  Product  InteracPons  

Too  Complicated

80%  of  Users  Use  20%  of  Features

Try  Radically  Simplifying

What  We  Learned  From  This  Test  

•  Just  hiding  the  advanced  features  increased  task  success  by  10%,  which  directly  improved  sales  •  Further  jump  of  15%  when  we  did  the  redesign  – Absolutely  no  increase  in  sales…  – …  but  we  saw  in  the  funnel  that  this  was  solely  due  to  less  people  hipng  the  trial  limits  …  

– …  which  we  were  able  to  profitably  exploit  with  the  quota  limit  test  described  earlier  

Key  Points  With  TesPng  Core  InteracPons  

•  Test  the  en,re  funnel.    Some,mes  task  success  doesn’t  (unfortunately)  help  bo:om  line  metrics.  •  A  good  understanding  of  where  people  spend  their  ,me/energy  is  vital  –  test  the  90%  interac,on,  not  the  2%.  

ROI  on  ~$500  A  Month  TesPng  Budget

0.00%  

0.50%  

1.00%  

1.50%  

2.00%  

2.50%  

3.00%  

3.50%  February,  2010  

March,  2010  

April,  2010  

May,  2010  

June

,  2010  

July,  2010  

August,  2010  

Septem

ber,  2010  

Octob

er,  2010  

Novem

ber,  2010  

Decembe

r,  20

10  

Janu

ary,  2011  

February,  2011  

March,  2011  

April,  2011  

May,  2011  

June

,  2011  

July,  2011  

August,  2011  

Septem

ber,  2011  

Octob

er,  2011  

Novem

ber,  2011  

Decembe

r,  20

11  

Janu

ary,  2012  

February,  2012  

March,  2012  

April,  2012  

May,  2012  

June

,  2012  

July,  2012  

August,  2012  

Septem

ber,  2012  

Octob

er,  2012  

Novem

ber,  2012  

Decembe

r,  20

12  

Janu

ary,  2013  

February,  2013  

March,  2013  

April,  2013  

May,  2013  

June

,  2013  

July,  2013  

August,  2013  

Septem

ber,  2013  

Octob

er,  2013  

Novem

ber,  2013  

Decembe

r,  20

13  

Janu

ary,  2014  

February,  2014  

Let’s  Talk  TesPng  Challenges  

A/B  TesPng  Across  System  Boundaries  

Not  A  Hard  Problem  

Our  Marke,ng  Site

A/B

Gets  A  Wee  Bit  Trickier  With  2  Sites  

Our  Marke,ng  Site

A/B

The  Web  Applica,on

Total  Freaking  Madness  

Our  Marke,ng  Site

A/B

The  Web  Applica,on

iPhone  app

Offline  processes Android  app

Partner  site

Content  minisite

Offsite  checkout Japanese  subsidiary

Sanity  Restored  

Consumer  Applica,ons Iden,ty  &  A/B  Manager

Lessons  Learned  With  This  

•  Think  about  strategy  for  transi,oning  people  between  systems  –  easier  to  build  into  design  earlier  than  later  •  Cookies  with  unique  iden,,es  are  a  stopgap,  not  a  solu,on.    You  need  to  understand  the  sales  process.  •  This  is  a  good  opportunity  to  standardize  on  an  A/B  tes,ng  tool  and  workflow…  or  you’ll  hate  your  life  •  This  is  a  “high  class  problem”  which  can  be  deferred  un,l  $50M+  in  revenue,  but  when  you  solve  it,  it  is  going  to  cost  serious  ,me/money.  

Increasing  A/B  TesPng  AdopPon  In  Engineering  and  MarkePng  Teams  

Single  Most  Important  Lesson:    One  Line  of  Code  

@user.card_limit  =  ab_test("user_card_limit_May_2012",  [8,  15],  :conversion  =>  "purchase")

Other  Lessons  Learned  

•  Organiza,on  should  learn  rather  than  guess  be:er.  – Best  investment  ever:  organiza,on-­‐wide  wiki.  – Next  best:  weekly  “What  we  tried  this  week”  email.  

•  Engineers  love  A/B  tes,ng  when  you  describe  it  like  a  puzzle  or  a  video  game,  with  bragging  rights.  • Marketers  love  A/B  tes,ng  when  it’s  posi,oned  as  a  way  to  increase  their  status  in  the  organiza,on  rather  than  a  replacement  for  their  talents.  

Things  To  Never  Be  Without  

•  CS  should  be  able  to  force-­‐override  a  user’s  par,cipa,on  in  an  A/B  test  (and,  relatedly,  “see  what  users  is  seeing”  is  great  if  you  can  get  it).  •  A/B  tests  deeper  than  surface  level  need  unit/etc  tests.    Make  sure  your  framework  supports  it.  •  Test  switcher  UI  in  development/staging  is  a  huge  win.  •  URL  query  parameter  to  flip  tests  a  huge  win.  

Thanks  for  Listening!  

•  I  love  talking  about  this  and  you  cannot  possibly  waste  my  ,me:  @pa,o11  or  patrick@kalzumeus.com  •  Free  email  mini-­‐course  on  CRO  in  so>ware  companies:  h:p://bit.ly/patrick-­‐op,mizely  •  Liked  this  presenta,on?    I  deliver  something  like  this  twice  a  month,  free,  to  my  email  list.    h:p://training.kalzumeus.com  to  sign  up  

Conversion  Op,miza,on:    The  World  Beyond    

Headlines  and  Bu:on  Color  Patrick  

McKenzie  Founder  

Kalzumeus  So>ware  @pa,o11