183
Measuring the Effects of Advertising: The Digital Frontier* Randall A. Lewis, Google, Inc. Justin M. Rao, Microsoft Research David Reiley, Google, Inc.

Measuring the Effects of Advertising: The Digital Frontier*

  • Upload
    rangle

  • View
    61

  • Download
    9

Embed Size (px)

DESCRIPTION

Measuring the Effects of Advertising: The Digital Frontier*. Randall A. Lewis, Google, Inc. Justin M. Rao, Microsoft Research David Reiley , Google, Inc. * Opinions expressed are our own, not our huge employers. Introduction. advertising is a $200+ billion per-year industry - PowerPoint PPT Presentation

Citation preview

Page 1: Measuring the Effects of Advertising:  The Digital Frontier*

Measuring the Effects of Advertising: The Digital Frontier*Randall A. Lewis, Google, Inc.Justin M. Rao, Microsoft ResearchDavid Reiley, Google, Inc.* Opinions expressed are our own, not our huge employers

Page 2: Measuring the Effects of Advertising:  The Digital Frontier*

Introduction

Page 3: Measuring the Effects of Advertising:  The Digital Frontier*

advertising is a $200+ billion per-

year industry

(~1.5-2.0% of GDP)

Page 4: Measuring the Effects of Advertising:  The Digital Frontier*

supports “free” services that

constitute a majority of American’s leisure

time

Page 5: Measuring the Effects of Advertising:  The Digital Frontier*
Page 6: Measuring the Effects of Advertising:  The Digital Frontier*

yet effects of advertising are

poorly understood

Page 7: Measuring the Effects of Advertising:  The Digital Frontier*

~$100,000 per month

Page 8: Measuring the Effects of Advertising:  The Digital Frontier*

~$25,000,000 per year

Page 9: Measuring the Effects of Advertising:  The Digital Frontier*

~$3-5,000,000 per year

Page 10: Measuring the Effects of Advertising:  The Digital Frontier*

~$10-25 CPM

Page 11: Measuring the Effects of Advertising:  The Digital Frontier*

~$1-8 CPM

Page 12: Measuring the Effects of Advertising:  The Digital Frontier*

I have no idea

Page 13: Measuring the Effects of Advertising:  The Digital Frontier*

impact of these ads?

Page 14: Measuring the Effects of Advertising:  The Digital Frontier*

digital measurement era two key advances

Page 15: Measuring the Effects of Advertising:  The Digital Frontier*

1) ad delivery and purchase data can be linked at individual level

Page 16: Measuring the Effects of Advertising:  The Digital Frontier*

2) ad delivery can be randomized essential exogenous variation to measure causal effects

Page 17: Measuring the Effects of Advertising:  The Digital Frontier*

plan for the talk

Page 18: Measuring the Effects of Advertising:  The Digital Frontier*

what methods should we use to measure adfx?

Page 19: Measuring the Effects of Advertising:  The Digital Frontier*

where can things go wrong?

Page 20: Measuring the Effects of Advertising:  The Digital Frontier*

what type of precision can we

expect?

Page 21: Measuring the Effects of Advertising:  The Digital Frontier*

we’ll go through a case study

Page 22: Measuring the Effects of Advertising:  The Digital Frontier*

what metrics are used to measure

adfx?

Page 23: Measuring the Effects of Advertising:  The Digital Frontier*

what metrics are should be used to

measure adfx?

Page 24: Measuring the Effects of Advertising:  The Digital Frontier*

what metrics are should be used to

measure adfx?

Page 25: Measuring the Effects of Advertising:  The Digital Frontier*

specific biases that can arise in online

settings

Page 26: Measuring the Effects of Advertising:  The Digital Frontier*

specific biases that can arise in online

settings

Page 27: Measuring the Effects of Advertising:  The Digital Frontier*

what’s in reach? what’s out of reach?

Page 28: Measuring the Effects of Advertising:  The Digital Frontier*

what’s in reach? what’s out of reach?

Page 29: Measuring the Effects of Advertising:  The Digital Frontier*

how does computational

advertising help?

Page 30: Measuring the Effects of Advertising:  The Digital Frontier*

Estimating the Causal Effects of Advertising

Page 31: Measuring the Effects of Advertising:  The Digital Frontier*

a few useful facts on advertising

Page 32: Measuring the Effects of Advertising:  The Digital Frontier*

average American sees ~$1.35 worth

of ads per day

Page 33: Measuring the Effects of Advertising:  The Digital Frontier*

universe of advertisers needs to

net $1.35 in marginal profits

Page 34: Measuring the Effects of Advertising:  The Digital Frontier*

universe of advertisers needs to

net $1.35 $5-6 in marginal profits incremental sales

Page 35: Measuring the Effects of Advertising:  The Digital Frontier*

(to break-even)

Page 36: Measuring the Effects of Advertising:  The Digital Frontier*

any given campaign will be a small

fraction of daily ad exposure

Page 37: Measuring the Effects of Advertising:  The Digital Frontier*

with this in mind…

Page 38: Measuring the Effects of Advertising:  The Digital Frontier*

25 display advertising

field experiments run at Yahoo!

Page 39: Measuring the Effects of Advertising:  The Digital Frontier*

experiment: exposure

determined by flip of coin

Page 40: Measuring the Effects of Advertising:  The Digital Frontier*

results here taken from

Lewis and Rao (2012)

Page 41: Measuring the Effects of Advertising:  The Digital Frontier*

data sharing gives us sales records paired with ad

delivery

Page 42: Measuring the Effects of Advertising:  The Digital Frontier*

Notation:: an individual=sales for person (online + offline)=1 if is treated with firm’s ad=0 if is treated with placebo ad: vector of covariates (we’ll ignore for now, all results go through by just adding “condition on ”)

Page 43: Measuring the Effects of Advertising:  The Digital Frontier*

Regression: average sales difference between exposed (E) and unexposed (U) groups

Page 44: Measuring the Effects of Advertising:  The Digital Frontier*

Experimental study:E: treatment groupU: control group

Observational study:E: endogenously exposedU: pseudo control

Page 45: Measuring the Effects of Advertising:  The Digital Frontier*

=s.d. of sales at individual level=treatment-control (sales impact)

Page 46: Measuring the Effects of Advertising:  The Digital Frontier*

let’s calibrate with medians from the 19

retail sales experiments

Page 47: Measuring the Effects of Advertising:  The Digital Frontier*

retailers ranging from budget to high-

end

Page 48: Measuring the Effects of Advertising:  The Digital Frontier*

(weekly)= (weekly)cost= $0.14 per customer (20-100) ads @$1-5 CPMROI goal=25% increase sales by $0.35 (based on margins)

Page 49: Measuring the Effects of Advertising:  The Digital Frontier*

goal is to increase baseline sales by 5%

Page 50: Measuring the Effects of Advertising:  The Digital Frontier*

problem: standard deviation is 10x the

mean

Page 51: Measuring the Effects of Advertising:  The Digital Frontier*
Page 52: Measuring the Effects of Advertising:  The Digital Frontier*

(for a successful

campaign!)

Page 53: Measuring the Effects of Advertising:  The Digital Frontier*

“not for scatter plots”

Page 54: Measuring the Effects of Advertising:  The Digital Frontier*

some observations

Page 55: Measuring the Effects of Advertising:  The Digital Frontier*

need very large samples to

distinguish a profitable campaign

from zero effect

Page 56: Measuring the Effects of Advertising:  The Digital Frontier*

even experiments with 1M subjects in each group can be

underpowered

Page 57: Measuring the Effects of Advertising:  The Digital Frontier*

even bigger concern

Page 58: Measuring the Effects of Advertising:  The Digital Frontier*

selection bias in observational

studies

Page 59: Measuring the Effects of Advertising:  The Digital Frontier*

omitted factors with partial

full order of magnitude more than treatment

effect

Page 60: Measuring the Effects of Advertising:  The Digital Frontier*

selection bias can swamp true adfx

Page 61: Measuring the Effects of Advertising:  The Digital Frontier*

and this is a concern

Page 62: Measuring the Effects of Advertising:  The Digital Frontier*

ads are by design not randomly

delivered!

Page 63: Measuring the Effects of Advertising:  The Digital Frontier*

(or why would be here!)

Page 64: Measuring the Effects of Advertising:  The Digital Frontier*

can an observational method ever be sure all selection effects have been eliminated down to such a low fraction of variance in dependent variable?

Page 65: Measuring the Effects of Advertising:  The Digital Frontier*

experiments are unbiased, but very noisy

observational methods can be dangerous

Page 66: Measuring the Effects of Advertising:  The Digital Frontier*

Advertising Experiment Case Study

Page 67: Measuring the Effects of Advertising:  The Digital Frontier*

1,577,256 customers of a retailer matched

to Y! ID

Page 68: Measuring the Effects of Advertising:  The Digital Frontier*

(allows tracking of offline sales too)

Page 69: Measuring the Effects of Advertising:  The Digital Frontier*

81% saw retailers ads,

19% held out

Page 70: Measuring the Effects of Advertising:  The Digital Frontier*

81%: 48 display ads per person on

average

Page 71: Measuring the Effects of Advertising:  The Digital Frontier*

treatment group had +5% sales

Page 72: Measuring the Effects of Advertising:  The Digital Frontier*

93% of impact occurred offline

Page 73: Measuring the Effects of Advertising:  The Digital Frontier*

total effect not significant at

10% level

Page 74: Measuring the Effects of Advertising:  The Digital Frontier*

power calculations: 21% chance of

rejecting ad had zero effect if true effect was break-

even

Page 75: Measuring the Effects of Advertising:  The Digital Frontier*

throw out the control group and use observational

techniques comparing exposed

vs. unexposed

Page 76: Measuring the Effects of Advertising:  The Digital Frontier*

would find statistically

significant and negative effect of

the adROI ~-700%

Page 77: Measuring the Effects of Advertising:  The Digital Frontier*

The Click and Related Metrics

Page 78: Measuring the Effects of Advertising:  The Digital Frontier*

CTR (click thru rate) has become second

nature

Page 79: Measuring the Effects of Advertising:  The Digital Frontier*

why?

Page 80: Measuring the Effects of Advertising:  The Digital Frontier*

cleanly defined

Page 81: Measuring the Effects of Advertising:  The Digital Frontier*

easily measurable

Page 82: Measuring the Effects of Advertising:  The Digital Frontier*

occur relatively frequently

Page 83: Measuring the Effects of Advertising:  The Digital Frontier*

an ad-click cannot occur with an ad

Page 84: Measuring the Effects of Advertising:  The Digital Frontier*

but a click can!

Page 85: Measuring the Effects of Advertising:  The Digital Frontier*
Page 86: Measuring the Effects of Advertising:  The Digital Frontier*

ads can crowd out “organic” clicks

Page 87: Measuring the Effects of Advertising:  The Digital Frontier*

need “baseline arrival rate”

Page 88: Measuring the Effects of Advertising:  The Digital Frontier*

can be solved with experiments

Page 89: Measuring the Effects of Advertising:  The Digital Frontier*
Page 90: Measuring the Effects of Advertising:  The Digital Frontier*

what does CTR miss?

Page 91: Measuring the Effects of Advertising:  The Digital Frontier*

brand advertising

Page 92: Measuring the Effects of Advertising:  The Digital Frontier*

offline sales

Page 93: Measuring the Effects of Advertising:  The Digital Frontier*

it is an intermediate metric

Page 94: Measuring the Effects of Advertising:  The Digital Frontier*

cost-per-acquisition (CPA)

advertising/bidding

Page 95: Measuring the Effects of Advertising:  The Digital Frontier*

move upstream from the click

Page 96: Measuring the Effects of Advertising:  The Digital Frontier*

sale with “qualifying link” to

advertisement

Page 97: Measuring the Effects of Advertising:  The Digital Frontier*

but what qualifies?

Page 98: Measuring the Effects of Advertising:  The Digital Frontier*

last ad? last click? first click? weighted

average?

Page 99: Measuring the Effects of Advertising:  The Digital Frontier*

known as the “attribution

problem”

Page 100: Measuring the Effects of Advertising:  The Digital Frontier*

exchanges use ad-hoc rules to “solve” the problem from an

accounting perspective

Page 101: Measuring the Effects of Advertising:  The Digital Frontier*

rules are inherently flawed

Page 102: Measuring the Effects of Advertising:  The Digital Frontier*

and widespread re-targeting has

complicated matters

Page 103: Measuring the Effects of Advertising:  The Digital Frontier*

attribution problem more generally

Page 104: Measuring the Effects of Advertising:  The Digital Frontier*

large firms advertise:- out-of-home- television- online display- online search

Page 105: Measuring the Effects of Advertising:  The Digital Frontier*

large firms advertise:- out-of-home- television- online display- online search

Page 106: Measuring the Effects of Advertising:  The Digital Frontier*

large firms advertise:- out-of-home- television- one point in global

spend- online search

Page 107: Measuring the Effects of Advertising:  The Digital Frontier*

large firms advertise:- out-of-home- television- cross-channel

effects?- online search

Page 108: Measuring the Effects of Advertising:  The Digital Frontier*

large firms advertise:- out-of-home- television- global concavity?- online search

Page 109: Measuring the Effects of Advertising:  The Digital Frontier*

not obvious what to conclude from “one

point”

Page 110: Measuring the Effects of Advertising:  The Digital Frontier*

Activity Bias

Page 111: Measuring the Effects of Advertising:  The Digital Frontier*

activity bias: a temporal selection bias present on the

webLewis, Rao and Reiley (2011)

WWW

Page 112: Measuring the Effects of Advertising:  The Digital Frontier*

in an observational study:

exposed generally more active on web

during campaign period than unexposed

Page 113: Measuring the Effects of Advertising:  The Digital Frontier*

why?

Page 114: Measuring the Effects of Advertising:  The Digital Frontier*

unexposed didn’t see campaign ad

so…

Page 115: Measuring the Effects of Advertising:  The Digital Frontier*

either they did not fit target dimensions

Page 116: Measuring the Effects of Advertising:  The Digital Frontier*

(classic heterogeneity bias)

Page 117: Measuring the Effects of Advertising:  The Digital Frontier*

or they are browsing less on average

Page 118: Measuring the Effects of Advertising:  The Digital Frontier*

(activity bias)

Page 119: Measuring the Effects of Advertising:  The Digital Frontier*

why does browsing activity matter

Page 120: Measuring the Effects of Advertising:  The Digital Frontier*

browsing ad exposure

Page 121: Measuring the Effects of Advertising:  The Digital Frontier*

browsing online purchases

Page 122: Measuring the Effects of Advertising:  The Digital Frontier*

browsing sign-ups

Page 123: Measuring the Effects of Advertising:  The Digital Frontier*

browsing clicks

Page 124: Measuring the Effects of Advertising:  The Digital Frontier*

2 examples

Page 125: Measuring the Effects of Advertising:  The Digital Frontier*
Page 126: Measuring the Effects of Advertising:  The Digital Frontier*

2 examples

Page 127: Measuring the Effects of Advertising:  The Digital Frontier*

without control group what we have

concluded?

Page 128: Measuring the Effects of Advertising:  The Digital Frontier*

activity bias difficult to control for

Page 129: Measuring the Effects of Advertising:  The Digital Frontier*

cross-validation could be promising

Page 130: Measuring the Effects of Advertising:  The Digital Frontier*

Local vs. Global Optimization

Page 131: Measuring the Effects of Advertising:  The Digital Frontier*

“a principled way to find the best match between a given user in a given context and a suitable advertisement“

Andrei Broder, 2008

Page 132: Measuring the Effects of Advertising:  The Digital Frontier*

traditional media: who

Page 133: Measuring the Effects of Advertising:  The Digital Frontier*

computational adv: what

Page 134: Measuring the Effects of Advertising:  The Digital Frontier*

automated targeting

Page 135: Measuring the Effects of Advertising:  The Digital Frontier*

replacing demographic

“portfolio building”

Page 136: Measuring the Effects of Advertising:  The Digital Frontier*

great things about computational advertising:

Page 137: Measuring the Effects of Advertising:  The Digital Frontier*

shifts conversation from “who to hit” to

“what did I get”

Page 138: Measuring the Effects of Advertising:  The Digital Frontier*

if machines beat humans

improved

performance

Page 139: Measuring the Effects of Advertising:  The Digital Frontier*

key parameters that govern spend still

set by humans(in general, for now)

Page 140: Measuring the Effects of Advertising:  The Digital Frontier*

bid, valuation, budget

Page 141: Measuring the Effects of Advertising:  The Digital Frontier*

bid, valuation, budget

Page 142: Measuring the Effects of Advertising:  The Digital Frontier*

crowd-in, crowd-out, etc. still in play

Page 143: Measuring the Effects of Advertising:  The Digital Frontier*

still hard!

Page 144: Measuring the Effects of Advertising:  The Digital Frontier*

maximizing conversions!=

maximizing profit

Page 145: Measuring the Effects of Advertising:  The Digital Frontier*

maximizing incremental

conversions=maximizing profit

Page 146: Measuring the Effects of Advertising:  The Digital Frontier*

careful not to just “advertise to the

purchasers”

Page 147: Measuring the Effects of Advertising:  The Digital Frontier*

Moving Forward

Page 148: Measuring the Effects of Advertising:  The Digital Frontier*

cost of experiments decreasing

Page 149: Measuring the Effects of Advertising:  The Digital Frontier*

ease of experimentation

increasing

Page 150: Measuring the Effects of Advertising:  The Digital Frontier*

ghost ads

Page 151: Measuring the Effects of Advertising:  The Digital Frontier*

90: treatment10: control

Page 152: Measuring the Effects of Advertising:  The Digital Frontier*

same power as:

Page 153: Measuring the Effects of Advertising:  The Digital Frontier*

10: treatment90: control

Page 154: Measuring the Effects of Advertising:  The Digital Frontier*

pre-experiment matching

Page 155: Measuring the Effects of Advertising:  The Digital Frontier*

linked data sources

Page 156: Measuring the Effects of Advertising:  The Digital Frontier*

cross-channel effects

Page 157: Measuring the Effects of Advertising:  The Digital Frontier*

what’s still out of reach?

Page 158: Measuring the Effects of Advertising:  The Digital Frontier*

long-run effects have a fundamental

difficulty

Page 159: Measuring the Effects of Advertising:  The Digital Frontier*

any study on adfx

Page 160: Measuring the Effects of Advertising:  The Digital Frontier*

Campaign

8/6/122:23pm

Page 161: Measuring the Effects of Advertising:  The Digital Frontier*

Campaign

8/6/122:23pm

Page 162: Measuring the Effects of Advertising:  The Digital Frontier*

Campaign

8/6/122:23pmAdfx

measured

Page 163: Measuring the Effects of Advertising:  The Digital Frontier*

Campaign

8/6/122:23pmAdfx

measured

Page 164: Measuring the Effects of Advertising:  The Digital Frontier*

why not just…

Page 165: Measuring the Effects of Advertising:  The Digital Frontier*

Campaign

8/6/122:23pmAdfx

measured

Page 166: Measuring the Effects of Advertising:  The Digital Frontier*

two problems(at least)

Page 167: Measuring the Effects of Advertising:  The Digital Frontier*

1) value to acting on information

Page 168: Measuring the Effects of Advertising:  The Digital Frontier*

2) statistical challenges increase with time

Page 169: Measuring the Effects of Advertising:  The Digital Frontier*

more data !=more precise

inference

Page 170: Measuring the Effects of Advertising:  The Digital Frontier*

why?

Page 171: Measuring the Effects of Advertising:  The Digital Frontier*

with time: cumulative effect

increases

Page 172: Measuring the Effects of Advertising:  The Digital Frontier*

with time: per time period effect decays

Page 173: Measuring the Effects of Advertising:  The Digital Frontier*

(otherwise )

Page 174: Measuring the Effects of Advertising:  The Digital Frontier*

datasignal

Page 175: Measuring the Effects of Advertising:  The Digital Frontier*

signal-to-noise

Page 176: Measuring the Effects of Advertising:  The Digital Frontier*

Campaign

8/6/122:23pmAdfx

measuredjudgment call

Page 177: Measuring the Effects of Advertising:  The Digital Frontier*

Campaign

8/6/122:23pmAdfx

measurednecessary decision

Page 178: Measuring the Effects of Advertising:  The Digital Frontier*

Conclusion

Page 179: Measuring the Effects of Advertising:  The Digital Frontier*

1) advertising and sales linked at individual

2) randomization of ad-delivery possible

Page 180: Measuring the Effects of Advertising:  The Digital Frontier*

whowhat

Page 181: Measuring the Effects of Advertising:  The Digital Frontier*

performance metrics part of culture of

advertising

Page 182: Measuring the Effects of Advertising:  The Digital Frontier*

many concerns still make measuring adfx very difficult