Debunking Ad Testing

Mart in RöttgerdingH e a d o f S E M Logo Here

Learner Outcomes

• Adtesting best practices don‘t work

• What actually happens when you followbest practices

• What you should be doing instead

MartinRöttgerding

• Headof SEM@BloofusionGermany

• Blogger@PPC-Epiphany.com

• Judge @SEMY(GermanSearchAwards)

• Dad@Home

Who’s Talking?

@bloomarty

First off: No Worries

• I’mnotamathematician.

• Thisisnotamathlecture.

• Ibrought some data,though.

iStock.com/digitalgenetics

In the Beginning: Optimize for Clicks

% Served Impressions Clicks CTR

Ad1 96% 12,323 594 4.82%

Ad2 4% 536 37 6.90%

Alternative Approaches

• Gutfeeling

• Rules(„wait 100clicks,then decide“)

• StatisticalSignificance (usually:95%)

Statistical Significance in a NutshellImpressions Clicks

Ad1 200 10

Ad2 200 20

Statistical Significance in a Nutshell

0% 1% 2% 3% 4% 5% 6% 7% 8% 9% 10% 11% 12% 13% 14% 15% 16%

CTR

Ad1

Impressions Clicks

Ad1 200 10

Ad2 200 20

Ad2

Statistical Significance in a Nutshell

0% 1% 2% 3% 4% 5% 6% 7% 8% 9% 10% 11% 12% 13% 14% 15% 16%

CTR

Ad1

Impressions Clicks

Ad1 200 10

Ad2 200 20

Ad2

Statistical Significance in a Nutshell

0% 1% 2% 3% 4% 5% 6% 7% 8% 9% 10% 11% 12% 13% 14% 15% 16%

CTR

Ad1

Impressions Clicks

Ad1 200 10

Ad2 200 20

Ad2

Statistical Significance in a Nutshell

0% 1% 2% 3% 4% 5% 6% 7% 8% 9% 10% 11% 12% 13% 14% 15% 16%

CTR

Impressions Clicks

Ad1 2000 100

Ad2 2000 200

Ad1 Ad2

Pros and Cons

• You have the power.

• You know what to do.

• You look good.

• It doesn‘t work.

Waiting for SignificanceProblem #1

Is it significant yet?

no

no

no

no

no

no

no no

no yes

„We are 95%certain that this adis better than the other one.“

An Experiment

• 576adpairs

• Rotated evenly

• Left untouched for 12months

Analyzing the Data

• Scriptto evaluate the data

• Calculate level of significance for each day

• Visualization:– Ad1reaches statistical significance (95%)

– Ad2reaches statistical significance (95%)

The Result (a small part)

statistically significantno longer significant

stillsignificant…

waiting for significance

The Result (zoomed out)

The Result (zoomed out)

Results

• Mosttests reached asignificance level of 95%atsome point

MinimumtotalImpressions Teststo reach significance Stillsignificant inthe end

1,000 55% 13%

10,000 62% 12%

100,000 81% 11%

Not looking at all the piecesProblem #2

So this is what we see…Impressions Clicks CTR

Ad1 2,000 200 10%

Ad2 3,000 240 8%

How we tend to think of CTR

CTRYour Ad

What drives CTR?

CTR

Ad

Network(Googlevs.SearchPartners)

Search Partners?

Search Partners

Segmented by NetworkImpressions Clicks CTR

Ad1 2,000 200 10%Google

SearchPartners

Ad2 3,000 240 8%Google

SearchPartners

Segmented by NetworkImpressions Clicks CTR

Ad1 2,000 200 10%Google 1,000 180 18%

SearchPartners 1,000 20 2%

Ad2 3,000 240 8%Google 1,000 220 22%

SearchPartners 2,000 20 1%

Also possible…Impressions Clicks CTR

Ad1 2,000 200 10%Google 1,000 180 18%

SearchPartners 1,000 20 2%

Ad2 3,000 240 8%Google 1,000 220 22%

SearchPartners 2,000 20 1%

Also possible…Impressions Clicks CTR

Ad1 2,000 200 10%Google 1,000 180 18%

SearchPartners 1,000 20 2%

Ad2 3,000 270 9%Google 1,000 220 22%

SearchPartners 2,000 50 2.5%

How common is this?

Based onastudy of 6,500adpairs,compared with anAdWordsScript

• OverallwinnerlosesonGoogle • OverallwinnerlosesonGoogle&SearchPartners

32.74%

12.23%

Quick Win: Ignore Search Partners

• Well…

What drives CTR?

CTR

Ad

Network(Googlevs.SearchPartners)

Slot(topvs.other)

Same thing with slots

Based onastudy of 6,500adpairs,compared with anAdWordsScript

• Overallwinnerlosesinthetopslot • Overallwinnerlosesontop&other

18.46%

6.30%

What drives CTR?

CTR

Ad

Device

AdpositionNetwork(Googlevs.SearchPartners)

Slot(topvs.other)

CTR by avg. Position (Google top)

0%

2%

4%

6%

8%

10%

12%

14%

16%

18%

20%

1.0 1.1 1.2 1.3 1.4 1.5 1.6 1.7 1.8 1.9 2.0 2.1 2.2 2.3 2.4 2.5 2.6 2.7 2.8 2.9 3.0 3.1 3.2 3.3 3.4 3.5 3.6 3.7 3.8 3.9 4.0

Avg.Pos.CTR

Exact Positions‘ CTR‘s (Google top)

0%

2%

4%

6%

8%

10%

12%

14%

16%

18%

20%

1.0 1.1 1.2 1.3 1.4 1.5 1.6 1.7 1.8 1.9 2.0 2.1 2.2 2.3 2.4 2.5 2.6 2.7 2.8 2.9 3.0 3.1 3.2 3.3 3.4 3.5 3.6 3.7 3.8 3.9 4.0

Avg.Pos.CTR ExactPos.CTR

18.05%

13.64%12.05%

10.09%

Interpolated CTR‘s

0%

2%

4%

6%

8%

10%

12%

14%

16%

18%

20%

1.0 1.1 1.2 1.3 1.4 1.5 1.6 1.7 1.8 1.9 2.0 2.1 2.2 2.3 2.4 2.5 2.6 2.7 2.8 2.9 3.0 3.1 3.2 3.3 3.4 3.5 3.6 3.7 3.8 3.9 4.0

Avg.Pos.CTR ExactPos.CTRinterpolated ExactPos.CTR

18.05%

13.64%12.05%

10.09%

What happens at the impression level…

• Thisis justaglimpse into impression level data

• How would you assess the following scenarios?

Scenario 1

Your ad

Userclicks onthis ad…and never scrolls down

Scenario 2

Your ad

user clicks onthis ad…after0.2seconds

Scenario 3

Your ad

user clicks onyour ad…and onallother ads as well

Scenario 4

user clicks onyour ad…afterspending 35secondsreading allsearch results

Your ad

How would you evaluate these?

• 4impressions

• 2clicks

• CTR:50%

Ignoring FeedbackProblem #3

What drives CTR?

CTR

Ad

Slot(topvs.other) Device

AdpositionNetwork(Googlevs.SearchPartners)

AdAuction

Adranking

CTR

The position feedback

• Positivefeedback

• No loop:position effects donotaffect QS

Adposition

AdAuction

AdrankingHigherCTRHigherQualityScore

BetterPositionHigherCTR

The impressions feedback

The impressions feedback

• Negativefeedback:

• No loop:low relevance impressionsevaluated separately

HigherCTRHigherQualityScore

Morelessrelevant

impressionsLower CTR

CTR

Ad AdAuction

CTR Feedback

CTR

Ad

Slot(topvs.other) Device

AdpositionNetwork(Googlevs.SearchPartners)

AdAuction

Adranking

mostly invisible

Thinking you are better motivatedthan Google

Problem #4

Googledoesn‘tunderstand my

business Googledoesn‘t care

Adtesting isn‘timportant enough forGoogleto get it right

The AdWords business model

„How much would you give us if we gave you the click?“

„How much would you give us if we showed your ad?“

Selladclicks

Selladimpressions

Convert bids

No control over clicks

Advertisers want clicks

The ad auction

• 𝐴𝑑𝑅𝑎𝑛𝑘 = 𝐶𝑃𝐶 ∗ 𝑄𝑢𝑎𝑙𝑖𝑡𝑦𝑆𝑐𝑜𝑟𝑒

• = 𝐶𝑃𝐶 ∗ 𝐶𝑇𝑅

• = 89:;8<=>?:

∗ 8<=>?:@ABCD::=9E

• = 89:;@ABCD::=9E =„How much would you give us if we showed your ad?“

Getting CTR wrong…

The concept itselfProblem #5

Conflicting Mindsets

Have adedicated adfor everything.

Findthe best adand use only that.

[keywords]

? ?

??

?

??

?

Example: Location Context

• Arethey outor athome?

• Arethey moving or standing still?

• Arethey atafamiliar place?

Example: Search History

• Have they searched for this before?

• Did they interact with ads?

• Did they interact with organic results?

• Have they seen our adbefore?

Example: Personality

• Dothey take their timeto read the entire ad?

• How dothey respond to– discounts– reassurances– testimonials

Testing for Things We Can‘t See?

• Manually:Impossible

• Otherwise:

What should we do?So…

New Mindset

• You don‘t have control over adtesting.Let it go.

• There can be multiplewinners.

• Use Google‘s optimized adrotation by default.

Let the Machines Do Their Job

• Googleis well motivated

• Googleis really good with data and algorithms[citation needed]

• Let Googledecide which ads to show

Provide Meaningful Ad Variations

• Createads for client personas

• TryoutdifferentUSP‘s

• Bigstuff first

Know Google‘s Limits

• CTR&conversion rate

• Historicaldata

• Semantics

Keep an Eye on the Machines

• If necessary,force data collection

• Rotate atadgroup level

• Consider the cost of even rotation

• Alternative:Addthe adagain

iStock.com/RichVintage

To Sum Up…

• Nomoremicromanagingadtests

• Focusonmessagingandsupervisingthemachines

Thank You!

• AgencyBlog:DieInternetkapitäne

• Advanced AdWordsBlog:PPC-Epiphany.com

@bloomarty

Martin RöttgerdingHead of SEABloofusion Germany GmbHmartin@bloofusion.de