A/B-testing #leanconf Manchester 2014: WHEN - HOW - EXAMPLES - PITFALLS

H O W , W H E N , E X A M P L E S & P I T T F A L L S

L e a n C o n f e r e n c e – N o v e m b e r 1 7 t h - M a n c h e s t e r To n W e s s e l i n g – C E O – Te s t i n g . A g e n c y

A/B-testing

Ton Wesseling | @ TonW #leanconf Manchester 2014

Manchester


Should I go for City?


Should I go for United?


Manchester


A/B-test ing

A/B-testing != CRO


A/B-test ing

CRO != user knowledge


A/B-test ing

A/B-testing ==

user knowledge


A/B-test ing

HOW & WHEN


A/B-test ing

EXAMPLES


A/B-test ing

PITTFALLS


Ton Wessel ing | @ TonW

ü  More testing ü  Quality Assurance ü  Lower pricing


Onl ine Exper iments – A/B-test ing

Winner!"

Variation B"

Variation A"

Default"


Hyper l ink junky


Digging through log fi les


Trying to improve


Over and over again


What most start-ups do


That is:

0 / 1 Testing


Why can this lead to fa i lures?


0/1 Test ing: weekly change?

26 out of 44 weeks: over 5% change!


Test ing should be l ike:


Scient ific approach

A / B Testing Live Online Research


Exper iments

“If you double the number of experiments you do per year

you’re going to double your inventiveness”

Jeff Bezos, CEO Amazon – 2004!


Growth l ike Amazon G

erm

an E

com

mer

ce m

arke

t, th

ank

you

Andr

é M

orys


Exper iments per year

1000+ (but it did not start like that)


Growth curve


Phase 1: test solut ions


Fake adwords test ing

Do they click

on your idea?


Fake facebook test ing

Do they like your idea?


Fake landing page test ing

Do they subscribe to your idea?


Make sure i t ’s not

0 / 1 Testing


Once you br idged the gap


The growth phase!


Dupl icate?


I t ’s the web, i t ’s scalable!


You need to opt imize!


Over and over again


I t takes many exper iments to win


Maximize your test capacity

Time

Conversion%


Speed up – more people



erm

an E

com

mer

ce m

arke

t, th

ank

you

Andr

é M

orys


Real growth


I t takes heavy exper iment ing

250+ in a year


Not every exper iment is successful


I t takes a fu l l stadium a week for 250+


Or roughly

1000+ conversion points


Too slow? Energy wi l l run out!


And you wi l l never get th is


You need the real fast paced culture


How does Bi l l win?


Emotion vs Rat io



Why is your user here?


How can you

help your user’s brain

to fullfil it’s needs?


I t ’s not about A/B-test ing

Growth is about learning how to serve brains!


Test ing costs t ime

Use it to LEARN


FA C T & A C T – 7 s t e p s t o l e a r n

&

Tell Conclude Analyze

Test Create Analyze Find


N o w : f o c u s o n C TA

Analyze

Test

Create


Create = Data & Psychology

Digital Data Persuasion Psychology

Analyze &

Experiment


Wheel-of-Persuasion.com


Persuasion on forms

+25% sales


Same technique

Significant more submits, but..

Less registrations


A smal l sentence

+18% sales


But also

+10% applications


Motivat ion

Winner (sales)


Win your own form A/B-test

http://testing.agency/manchester


And socia l proof?

Many went before you In november 1.675 people applied for a loan at MoneYou.

Winner (sales)


Chunk your USP’s

+14% sales


With learning: Phase 3 becomes easier


Test ing becomes user knowledge

Conversion)Op,miza,on) Market'Research'


Pitt fa l ls

How NOT to test


A/B-test ing can be


Don’t t rust your A/B-test analyt ics


I t keeps you away from this fa i lure


Enough traffic & conversions?


Low significance levels

ü Big chance the winner is not a real winner


Low power levels

ü Big chance a real winner is not recognized


Tested percentage is a spread!


Example test-setup


Average power: 40%


So:


Another error warning:

ü You just go and dig unitil you find something (you will always find something that seems true BUT is not!)


OK, results are in


Not s ignificant

ü It’s not a winner, stop

ü But, it’s does not mean it’s a loser!

ü Average: 30% of your tests will have winners


More?

http://ondi.me/contentverve

http://ondi.me/conversionxl


Conclude

ü Do nothing? or

ü Re-test? or

ü Implement? or

ü Scale? and

ü Continue or change test roadmap


Knowledge l ibrary: ABtestguide.com


I t ’s not about A/B-test ing

Growth is about learning how to serve brains!


So you wi l l get th is curve



erm

an E

com

mer

ce m

arke

t, th

ank

you

Andr

é M

orys


A/B-test ing

HOW & WHEN


A/B-test ing

EXAMPLES


A/B-test ing

PITTFALLS


Sl ides & more

http://testing.agency/manchester


Be l ike Bi l l , and win!

H O W , W H E N , E X A M P L E S & P I T T F A L L S

L e a n C o n f e r e n c e – N o v e m b e r 1 7 t h - M a n c h e s t e r To n W e s s e l i n g – C E O – Te s t i n g . A g e n c y

ht tp : / /testing.agency/manchester

Business

A/B-testing #leanconf Manchester 2014: WHEN - HOW - EXAMPLES - PITFALLS