17
Rajesh Parekh Controlled Experimentation to Guide Product Innovation

Expt panel hive_data_rp_20130320_final-1

Embed Size (px)

Citation preview

!Rajesh Parekh!

Controlled Experimentation to Guide Product Innovation!

Controlled Experimentation (A/B Testing)!•  Method to study effects of a treatment#

•  Concept:!- Randomly split users into two groups#

➥ A : Control#

➥ B: Treatment#- A and B are identical to each other except

for the treatment being evaluated#- Collect performance metrics from the

experiment#- Run statistical tests to determine if

differences between A and B are purely by chance#

2

Randomly  Divide  

A  (Control)   B  (Treatment)  

Measure  &  Evaluate  

Controlled  Experimenta=on  Panel  

Why Run Controlled Experiments?!•  Commonly used approach in clinical trials!- What is the effect of a particular drug / treatment?#

•  Systematically validate hypotheses with data!!•  Concurrently run the treatment and control!- The difference (if any) is#➥ Because of the treatment OR#➥ Due to random chance#

•  Determine if a treatment is causal in nature!- E.g., Making the search box bigger causes increase in queries / user#

3 Controlled  Experimenta=on  Panel  

Controlled Experimentation: Use Cases! #

4 Controlled  Experimenta=on  Panel  

A  B  Stract  Widget  Company  

_________________  _________________  _________________  _________________   BUY  NOW  

A  B  Stract  Widget  Company  

_________________  _________________  _________________  _________________   BUY  NOW  

Website  Variants  

Controlled Experimentation: Use Cases! #

5 Controlled  Experimenta=on  Panel  

Free  Trial   Play  Now  

Mobile  Call  to  Ac=on  

Controlled Experimentation: Use Cases! #

6 Controlled  Experimenta=on  Panel  

Top  deal  highlighted  

Email  Template  Design  

Controlled Experimentation: Use Cases! #

7 Controlled  Experimenta=on  Panel  

Backend  changes  (e.g.,  Personaliza=on  Algorithm)  

8

Controlled Experimentation: Use Cases!

# •  Follow-up message for users who previously clicked on an ad#

•  Incentive campaign to re-engage lapsed users#

•  Think of this as placing filters / guards on a randomly chosen user population#

Controlled  Experimenta=on  Panel  

Custom  Defined  User  Segments  

9

Key Components of an Experimentation Platform!Hashing function!

!

!

!!

!

Logging!

!

!

!

!

Metrics – suite of KPI!

!

!

!!

!

Dashboard!

F(                        )  Group  0  

Group  1  

Group  N-­‐1  

Time  Spent  

Revenue  

Click-­‐Through  Rate  

Session  Length  

Abandonment  

Purchase  Rate  

•  Metric  improvements  and  Sta=s=cal  Significance  in  a  central  place  

•  Detailed  logging  of  all  user  interac=ons  

Controlled  Experimenta=on  Panel  

Ensure Identical Control and Treatment!

•  Custom Segments#

10

CONTROL   TREAMENT  

Gender    

Male  

Female  

CONROL   TREATMENT  

Region  Size  

Small  

Medium  

Large  

CONROL   TREATMENT  

Prior  Exposure  

No  

Yes  

δ%  

Controlled  Experimenta=on  Panel  

•  Frequency Distribution#

•  Large Difference in Prior Exposure Rate violates assumptions#

A/A Tests!•  Run an experiment with two identical variants#

•  Helps to determine if:#- Users are being split uniformly at random#- Correct data is being logged#- Variance between identical populations of users is acceptable#

•  Challenge:!- Few purchases of high value deals render statistically significant

difference between treatment and control#

11

SPAIN  TRIP  $1,999  

Controlled  Experimenta=on  Panel  

Monitor Each Variant!•  Place yourself in each variant

to validate the experience#!

!

12

Carefully  inspect  each  variant  

Controlled  Experimenta=on  Panel  

•  Wrong sort order!!

!

Objective Function!#

#

Conversion   Revenue  

P(conversion)    

•  Favors  lower  price  deals  

E(rev)  =  P(conversion)  *  price    

•  More  expensive  deals  can  dominate  

Need  to  balance  mul=ple,  oZen  conflic=ng  objec=ves    

13 Controlled  Experimenta=on  Panel  

Measure Overall Impact!•  Test focuses on#- A particular area of

the website#- A sub-population of

users#

•  Measure!- Improvement on the

sub-segment AND#- Entire population!#

14

Measure  overall  impact  to  guard  against  cannibaliza=on  

Controlled  Experimenta=on  Panel  

Panel Discussion: Questions!

15 Controlled  Experimenta=on  Panel  

Thanks to many talented individuals at Groupon I am privileged to work with!#•  Data Science#•  Engineering#•  Marketing / Market Research#

Acknowledgements#

16 Controlled  Experimenta=on  Panel  

Rajesh [email protected]!!

17 Controlled  Experimenta=on  Panel