Embed Size (px)
Citation preview
IMPLEMENTING AND ANALYZING ONLINE EXPERIMENTSSEAN J. TAYLOR 28 JUL 2015 MULTITHREADED DATA
WHO AM I?
• Core Data Science Team at Facebook
• PhD from NYU in Information Systems
• Four academic papers employing online field experiments
• Teach and consult on experimental design at Facebook
http://seanjtaylor.comhttp://github.com/seanjtaylorhttp://facebook.com/seanjtaylor@seanjtaylor
I ASSUME YOU KNOW
• Why causality matters
• A little bit of Python and R
• Basic statistics + linear regression
SIMPLEST POSSIBLE EXPERIMENT
user_id version spent
123 B $10
596 A $0
456 A $4
991 B $9
def get_version(user_id): if user_id % 2: return 'A' else: return 'B'
> t.test(c(0, 4), c(10, 9))
Welch Two Sample t-‐test
data: c(0, 4) and c(10, 9) t = -‐3.638, df = 1.1245, p-‐value = 0.1487 alternative hypothesis: true difference in means is not equal to 0 95 percent confidence interval: -‐27.74338 12.74338 sample estimates: mean of x mean of y 2.0 9.5
FIN
COMMON PROBLEMS
• Type I errors from measuring too many effects
• Type II and M errors from lack of power
• Repeated use of the same population (“pollution”)
• Type I errors from violation of the i.i.d. assumption
• Composing many changes into one experiment
POWEROR
THE ONLY WAY TO TRULY FAIL AT AN EXPERIMENT
OR
THE SIZE OF YOUR CONFIDENCE INTERVALS
ERRORS
• Type I: Thinking your metric changed when it didn’t. We usually bound this at 1 or 5%.
• Type II: Thinking your metric didn’t change when it did. You can control this through better planning.
HOW TO MAKE TYPE I ERRORS
$ SpentTime Spent
Survey Satisfaction
oMale, <25
Female, <25
Male, >=25
Female, >=25
Measure a ton of metrics
Find
a su
bgro
up it
wor
ks o
n
AVOID TYPE II ERRORS WITH POWER
1. Use enough subjects in your experiment.
2. Test a reasonably strong treatment. Remember: you care about the difference.
POWER ANALYSIS
First step in designing an experiment is to determine how much data you’ll need to learn the answer to your question.
Process:
• set the smallest effect size you’d like to detect.
• simulate your experiment 200 times at various sample sizes
• count the number of simulated experiments where you correctly reject the null of effect=0.
TYPE M ERRORS
• Magnitude error: reporting an effect size which is too large
• happens when your experiment is underpowered AND you only report the significant results
IMPLEMENTATION
PLANOUT: KEY IDEAS
• an experiment is just a pseudo-random mapping from (user, context) → parameters, and is serializable.
• persistent randomizations implemented through hash functions, salts make experiments orthogonal
• always log exposures (parameters assignment) to improve precision, provide randomization check
• namespaces create ability to do sequential experiments on new blocks of users
https://facebook.github.io/planout/
A/B TESTING IN PLANOUTfrom planout.ops.random import * from planout.experiment import SimpleExperiment
class ButtonCopyExperiment(SimpleExperiment): def assign(self, params, user_id): # `params` is always the first argument. params.button_text = UniformChoice( choices=["Buy now!", "Buy later!"], unit=user_id )
# Later in your production code: from myexperiments import ButtonCopyExperiment
e = ButtonCopyExperiment(user_id=212) print(e.get('button_text'))
# Event later: e = ButtonCopyExperiment(user_id=212) e.log_event('purchase', {'amount': 9.43})
PLANOUT LOGS → DATA
{"inputs": {"user_id": 212}, "name": "ButtonCopyExperiment", "checksum": "646e69a5", "params": {"button_text": "Buy later!"}, "time": 1437952369, "salt": "ButtonCopyExperiment", "event": “exposure"}
{"inputs": {"user_id": 212}, "name": "ButtonCopyExperiment", "checksum": "646e69a5", "params": {"button_text": "Buy later!"}, "time": 1437952369, "extra_data": {"amount": 9.43}, "salt": "ButtonCopyExperiment", "event": "purchase"}
user_id button_text
123 Buy later!
596 Buy later!
456 Buy now!
991 Buy later!
user_id amount
123 $12
596 $9
Exposures
Purchases
ADVANCED DESIGN 1: FACTORIAL DESIGN
• Can use conditional logic as well as other random assignment operators: RandomInteger, RandomFloat, WeightedChoice, Sample.
class FactorialExperiment(SimpleExperiment): def assign(self, params, user_id): params.button_text = UniformChoice( choices=["Buy now!", "Buy later!"], unit=user_id ) params.button_color = UniformChoice( choices=["blue", "orange"], unit=user_id )
ADVANCED DESIGN 2: INCREMENTAL CHANGES
## We're going to try two different button redesigns. class FirstExperiment(SimpleExperiment): def assign(self, params, user_id): # ... set some params
class SecondExperiment(SimpleExperiment): def assign(self, params, user_id): # ... set some params differently class ButtonNamespace(SimpleNamespace): def setup(self): self.name = 'button_experiment_sequence' self.primary_unit = 'user_id' self.num_segments = 1000
def setup_experiments(): # allocate and deallocate experiments here # First gets 100 out of 1000 segments. self.add_experiment('first', FirstExperiment, 100) self.add_experiment('second', SecondExperiment, 100)
ADVANCED DESIGN 3: WITHIN-SUBJECTS
Previous experiments persistently assigned same treatment to user, but unit of analysis can be more complex:
class DiscountExperiment(SimpleExperiment): def assign(self, params, user_id, item_id): params.discount = BernoulliTrial(p=0.1, unit=[user_id, item_id]) if params.discount: params.discount_amount = RandomInteger( min=5, max=15, unit=user_id ) else: params.discount_amount = 0
e = DiscountExperiment(user_id=212, item_id=2) print(e.get('discount_amount'))
ANALYSIS
THE IDEAL DATA SET
Subject / User
Gender Age Button Size
Button Text
Spent Bounce
Erin F 22 Large Buy Now!
$20 0
Ashley F 29 Large Buy Later!
$4 0
Gary M 34 Small Buy Now!
$0 1
Leo M 18 Large Buy Now!
$0 1
Ed M 46 Small Buy Later!
$9 0
Sam M 25 Small Buy Now!
$5 0
Independent Observations
Randomly Assigned MetricsPre-experiment
Covariates{ { {
{
SIMPLEST CASE: OLS
> summary(lm(spent ~ button.size, data = df))
Call: lm(formula = spent ~ button.size, data = df)
Residuals: 1 2 3 4 5 6 10.0 -‐0.5 -‐4.5 -‐10.0 4.5 0.5
Coefficients: Estimate Std. Error t value Pr(>|t|) (Intercept) 10.000 5.489 1.822 0.143 factor(button.size)s -‐5.500 6.722 -‐0.818 0.459
Residual standard error: 7.762 on 4 degrees of freedom Multiple R-‐squared: 0.1434, Adjusted R-‐squared: -‐0.07079 F-‐statistic: 0.6694 on 1 and 4 DF, p-‐value: 0.4592
DATA REDUCTION
Subject Xi Di Yi
Evan M 0 1
Ashley F 0 1
Greg M 1 0
Leena F 1 0
Ema F 0 0
Seamus M 1 1
X D Y Cases
M 0 1 1
M 1 1 1
F 0 1 1
F 1 1 0
M 0 0 0
M 1 0 1
F 0 0 1
F 1 0 1
N # treatments X # groups X #outcomes
source('css_stats.R')
reduced <-‐ df %>% mutate(rounded.spent = round(spent, 0)) %>% group_by(button.size, rounded.spent) %>% summarise(n = n())
> lm(rounded.spent ~ button.size, data = reduced, weights = n) %>% + coeftest(vcov = sandwich.lm)
t test of coefficients:
Estimate Std. Error t value Pr(>|t|) (Intercept) 7.43137 0.45162 16.4548 7.522e-‐14 *** button.sizes -‐2.45178 0.59032 -‐4.1533 0.0004149 *** -‐-‐-‐ Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
DATA REDUCTION + WEIGHTED OLS
FACTORIAL DESIGNS
• Identify two types of effects: marginal and interactions. Need to fix one group as the baseline.
> coeftest(lm(spent ~ button.size * button.text, data = df))
t test of coefficients:
Estimate Std. Error t value Pr(>|t|) (Intercept) 6.79643 0.62998 10.7884 < 2.2e-‐16 *** button.sizes -‐2.43253 0.86673 -‐2.8066 0.006064 ** button.textn 2.11611 0.86673 2.4415 0.016458 * button.sizes:button.textn -‐2.57660 1.27584 -‐2.0195 0.046219 * -‐-‐-‐ Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
USING COVARIATES TO GAIN PRECISION
• With simple random assignment, using covariates is not necessary.
• However, you can improve precision of ATE estimates if covariates explain a lot of variation in the potential outcomes.
• Can be added to a linear model and SEs should get smaller if they are helpful.
NON-IID DATA
• Repeated observations of the same user are not independent.
• Ditto if you ‘re experimenting on certain items only.
• If you ignore dependent data, the true confidence intervals are larger than you think.
Subject / User
Item Button Size
Spent
Erin Shirt Large $20
Erin Socks Large $4
Erin Pants Large $0
Leo Shirt Large $0
Ed Shirt Small $9
Ed Socks Small $5
THE BOOTSTRAP
R1
All Your Data
R2
…
R500
Generate random sub-samples
s1
s2
s500
Compute statistics or estimate model
parameters
…
}0.0
2.5
5.0
7.5
-2 -1 0 1 2Statistic
Count
Get a distribution over statistic of interest
(e.g. the treatment effect)
- take mean - CIs == 95% quantiles - SEs == standard deviation
USER AND USER-ITEM BOOTSTRAPS
source('css_stats.R') library(broom) ## for extracting model coefficients
fitter <-‐ function(.data) { lm(summary ~ opposed, data = .data, weights = .weights) %>% tidy }
iid.replicates <-‐ iid.bootstrap(df, fitter, .combine = bind_rows) oneway.replicates <-‐ clustered.bootstrap(df, c('user_id'), fitter, .combine = bind_rows) twoway.replicates <-‐ clustered.bootstrap(df, c('user_id', 'item_id'), fitter, .combine = bind_rows)
> head(iid.replicates) term estimate std.error statistic p.value 1 (Intercept) 0.4700000 0.04795154 9.801561 6.296919e-‐18 2 button.sizes -‐0.2200000 0.08003333 -‐2.748855 6.695621e-‐03 3 (Intercept) 0.4250000 0.05307832 8.007036 5.768641e-‐13 4 button.sizes -‐0.1750000 0.08456729 -‐2.069358 4.049329e-‐02 5 (Intercept) 0.4137931 0.05141050 8.048805 4.118301e-‐13 6 button.sizes -‐0.1429598 0.08621804 -‐1.658119 9.965016e-‐02
DEPENDENCE CHANGES CONFIDENCE INTERVALS
DATA REDUCTION WITH DEPENDENT DATA
Subject Di Yij
Evan 1 1
Evan 1 0
Ashley 0 1
Ashley 0 1
Ashley 0 1
Greg 1 0
Leena 1 0
Leena 1 1
Ema 0 0
Seamus 1 1
Create bootstrap replicates
R1
R2
R3
reduce the replicates as if they’re i.i.d.
r1
r2
r3
s1
s2
s3
compute statistics on reduced data
THANKS! HERE ARE SOME RESOURCES:
• Me: http://seanjtaylor.com
• These slides:http://www.slideshare.net/seanjtaylor/implementing-and-analyzing-online-experiments
• Full Featured Tutorial: http://eytan.github.io/www-15-tutorial/
• “Field Experiments” by Gerber and Green
