Keynote Digital Data Tips Tuesday - Amsterdam - June 24th 2014 - A/B-testing analytics errors

< H a v e m o r e f u n A / B - t e s t i n g >

To n W e s s e l i n g – C E O Te s t i n g . A g e n c y # D D T T A m s t e r d a m 2 0 1 4 / 0 6 / 2 4

A / B - t e s t i n g E r r o r s

Email: [email protected] Twitter: @TonW #DDTT

Why me?

ü  10 years+ A/B-test experience

ü  I always loved the numbers side

ü  I’ve made all these mistakes



We opt imize


Nature


A better example


Groundhog Day – The A/B-test movie


Trying to get the gir l


Constant test ing environment

A / B Testing


Better Dialogue Optimizat ion

Winner!"

Variation B

Variation A

Default


This is why you’re A/B-test ing

Small steps to learn how

to make the next big step


Test ing is not free

Use it to LEARN


50% of the A/B-tests I get to see is wrong


Look at your A/B-test ing results!

+24% +52%

+83% +12%

+126% +16%


Did you push the winners l ive to your si te?

You must be rich now!


Or is this real i ty?


Or even worse:

A/B-test winners are letting you down!


Are these real winners?

+24% +52%

+83% +12%

+126% +16%


20 reasons why

Your A/B-test analyzing sucks


Get your stat ist ics r ight!


1: low signif icance levels

ü  Big chance the winner is not a real winner


2: low power levels

ü  Big chance a real winner is not recognized


Example test set-up

ü  90% significance level

ü  100 conversions per variation

ü  Average conversion rate: 2%

ü  20 out of 100 ideas is a winner

ü  With an average uplift of 10%


Average: 75% signif icance & 40% power


So:

ü  100 tests will give 10 false positives (90% significance)

ü  40% of the 20 winners are recognized (40% power)

ü  8 real winners & 10 false positives

ü  NOT 18 improvements of 15%+


3: Not calculat ing traff ic up front

ü  You thought the page had thousands of

visitors


4: Not calculat ing what you can improve

ü  You thought the page had lots and lots

conversions


ABTestGuide.com/calc


5: Not choosing your segments up front

ü  You just go and dig unitil you find something

(you will always find something that seems true BUT is not!)


6: You stop the test once i t is signif icant


6: You stop the test once i t is signif icant


7: You just let i t run unt i l i t ’s signif icant


7: You just let i t run unt i l i t ’s signif icant

Your test should have a fixed lenght!


8: Your test takes too long

ü  Cookie deletion (you will loose 10% in two weeks)

They will re-enter the the and pollute your samples!


9: You just choose a test length

ü  At least 1 full purchase cycle!


10: You don’t test ful l weeks

ü  At least 1 full week (if your not testing for time of day / week effects)


11: You think they always use one decive

ü Only test 1 device at a time!


12: You’re test ing broken stuff

ü  Always check browser compatibility!

ü Dont’ break dynamic stuff!


13: You have mult iple tests that overlap

ü  Your variation did not won (it was a combination of variations)


14: You thought this page matters

ü  But it did not (so you decide to use some other metric to declare the winner)


15: You forgot to leave out some visi tors

ü  Only test on those who can be changed (if you test a sales promotion – leave out your current clients who just logon)


16: You’re not slowing the control down

ü  Add the same code to the control


17: You’re not test ing on fresh people

ü  Start with control for a full purchase cycle (and then start sending traffic to your variations – setting the control to 0%)


18: You forgot to give them t ime to buy

ü  Stop getting fresh visitors after your test time (send new traffic to the control, but give you tested people time to finish converting)


19: You’re not diving into your analyt ics

ü  Always measure with analytics software (You will find conversions that are not supposed to be there!)


20: You’re not measuring the r ight stuff

ü  Use unique visitors

ü  Against lifetime value predictors!


I f you keep on making mistakes:

Stuff like this happens


Test ing is not free

Use it to LEARN


Bonus: You did not re-test your winners

ü  Always re-test to be sure it was not a glitch


So when you implement winners

This will happen!

< H a v e m o r e f u n A / B - t e s t i n g >

To n W e s s e l i n g – Te s t W i n n e r # D D T T A m s t e r d a m 2 0 1 4 / 0 6 / 2 4

A / B - t e s t i n g E r r o r s


ABTestGuide.com/calc

Marketing

Keynote Digital Data Tips Tuesday - Amsterdam - June 24th 2014 - A/B-testing analytics errors