Keynote Digital Elite Camp Estonia 2015 - ROAR - when to run more experiments

D i g i t a l E l i t e C a m p E s t o n i a , J u n e 1 3 t h 2 0 1 5 To n W e s s e l i n g – C E O – Te s t i n g . A g e n c y

# E l i t e C a m p - @ To n W

When to run more experiments?

Ton Wesseling | @ TonW Run more experiments #TonW

http://test ing.agency


Today

the

ROAR model


Methods used to improve conversion


Methods companies plan to use


A/B-test ing is

HOT


Everyone is pushing to test more


Never sat isfied with conversion rates


Start test ing or test more


NO Growth problems wi l l d issapear…


A/B-test ing is not a solut ion

It’s a methodology


I t ’s part of a:

Continuous Optimization

Program


And you need to know:

When

How What


Growth curve – how i t was teached


the

ROAR model


Optimizat ion phases - ROAR

Time span!

Con

vers

ions

per

mon

th!



Time span!

Con

vers

ions

per

mon

th!



Time span!

Con

vers

ions

per

mon

th!



Time span!

Con

vers

ions

per

mon

th!

Risk!



Time span!

Con

vers

ions

per

mon

th!

Risk! + Optimization!



Time span!

Con

vers

ions

per

mon

th!

Risk! + Optimization! + Automation!



Time span!

Con

vers

ions

per

mon

th!

Risk! + Optimization! + Automation! Re-think!



Time span!

Con

vers

ions

per

mon

th!


1.000 conversions!per month!



Time span!

Con

vers

ions

per

mon

th!


10.000 conversions per month!



Stat ist ics



Time span!

Con

vers

ions

per

mon

th!




15% impact needed!

5% impact!needed!

Impact needed: all conversions with an average conversion rate of 2%, a test length of 3 weeks and a power of 80%!


Low number test ing?

Why can it hurt you?


Warning: low significance levels

ü  Big chance the winner is not a real winner


Warning: low power levels

ü  Big chance a real winner is not recognized


Example test set-up

ü  90% significance level

ü  100 conversions per variation

ü  Average conversion rate: 2%

ü  20 out of 100 ideas is a winner…

ü  …with an average uplift of 10%


Average: 75% significance & 40% power


So:

ü  100 tests will give 10 false positives (90% significance)

ü  40% of the 20 winners are recognized (40% power)

ü  8 real winners & 10 false positives

ü  NOT 18 improvements of 10%+


Yes, I ’ve got those 1000+ conversions

Time span!

Con

vers

ions

per

mon

th!





With 1000+ conversions a month

Your tes t capac i ty i s max .

20+ tests a year

wi th a s ta t i s t i ca l power o f 80% on pred ic ted up l i f t s o f 15%


Tests with measureable impact

On average

1 out of 3

Simply truth: you’re just not always able to create a winner with enough impact


So in th is case, you’ve got impact

Every 7 or 8 weeks


Slower? Energy wi l l run out! And your continous optimization program will die


What to test? Business cases!

Delivery –test with in stock. 4 weeks

or 2 days?!


What to test? Business cases!

Service charge, do it or not? Include in

price or not?!


What to test? Big design changes

Yes this can be done with an A/B-test tool, you only need A/B-test coders!


ROAR – Opt imizat ion – moving up

Time span!

Con

vers

ions

per

mon

th!





Final ly you can test detai ls…

Is Multi Variate Testing (which is also A/B-testing but with more varations) a good idea? Almost never!!


You should grow to

Customer Buying Reasons


Reasons behind the customer journey


A cont inuous opt imizat ion team

Analyst Psychologyst

Designer Developer

Team Lead


They keep on asking:

Why are these users here?


And they want to know

How they can

help these users’ brains

to fullfil their needs?


Data & Psychology

Digital Data Persuasion Psychology

Analyze &

Experiment


A lack of resources


Test ing takes resources

Use it to LEARN


FA C T & A C T

&

Tell Conclude Analyze

Test Create Analyze Find


Know the company

Mission, Vision, Stategy & Goals


Know your user ( journeys)


Based on: asking – why are you here?


FA C T & A C T

&


Test Create Analyze Find Create

And yes, your team needs an economic / consumer Psychologyst and a UX designer!


Tests that can be done: A var iat ion


Tests that can be done: B var iat ion

Need help with coding A/B-test variations like this one? – http://testing.agency!







Need help with coding A/B-test variations like this one? – http://testing.agency!


Tests that can be done: A & B var iat ion










FA C T & A C T

&


Test Create Analyze Find

Analyze

Your analyst will analyze the results!


We measure in Analyt ics software


Calcu late resu l ts : ABTestGuide.com/calc


Not s ignificant

ü It’s not a winner, stop

ü But, it does not mean it’s a losing one!

ü Average: 33% of your tests will have winners If you are optimizing dialogues!


Choose what to analyze up front

ü  You just go and dig until you find something

(you will always find something that seems true BUT is not!)


Don’t stop the test once i t ’s s ignificant!


You just let i t run unt i l i t ’s s ignificant


You just let i t run unt i l i t ’s s ignificant

Your test has a fixed lenght!

It’s even a fixed amount of test population, but we love to test business cycles and/or weeks!


Be aware: Your test takes too long

ü  Cookie deletion (you will lose 10% in two weeks)

They will re-enter the and pollute your samples!


Test length: ABTestGuide.com/calc


T ip: Don’t test broken stuff

ü  Always check browser compatibility!

ü  Don’t break dynamic stuff!


T ip: Slow down the control down

ü  Add the same code to the control


T ip: leave out s o m e v is i tors

ü  Only test on those who can be changed (if you test a sales promotion – leave out your current clients who just logon)


T ip: Only test on fresh people

ü  Start with control for a full purchase cycle (and then start sending traffic to your variations – setting the control to 0%)


T ip: give them t ime to buy!

ü  Stop getting fresh visitors after your test time (send new traffic to the control, but give you tested people time to finish converting)


+ Optimization! + Automation!

Back to the ROAR model

Time span!

Con

vers

ions

per

mon

th!

Risk! Re-think!




ROAR – Opt imizat ion – moving up

Time span!

Con

vers

ions

per

mon

th!





Test capacity

maximum number of

experiments per year that your site or app

can handle


Regret

amount of money you are losing per year by

not using your full test capacity


Calculate your test capic i ty

http://ondi.me/size!


ROAR – Automation

Time span!

Con

vers

ions

per

mon

th!





Automation


You tested which locat ions are cr i t ical


You know which dia logues are cr i t ical


Automation

Contextual bandits

If you are ready – give me a call!!


Now you’ve moved to

Algorithm testing

Which internal and external influencers do have an impact on which segment of users?!


+ Optimization! + Automation!

ROAR – Automation

Time span!

Con

vers

ions

per

mon

th!

Risk! Re-think!




ROAR – You don’t stop opt imiz ing

Time span!

Con

vers

ions

per

mon

th!





ROAR – Maybe you’ l l never automate

Time span!

Con

vers

ions

per

mon

th!





ROAR – Re-think

Time span!

Con

vers

ions

per

mon

th!





The ul t imate Re-think resource


CRO br ings user knowledge


You wi l l make big impact on Discover


Customer knowledge center

Making money f rom convers ion improvements

Creat ing more and more cus tomer ins igh ts Which can be monet ized

on even a la rger sca le


Now, that’s

HOT


Done test ing? No!

TESTING NEVER STOPS

Okay, end of life, cash cow, no need for any further growth accelaration or user knowledge!


A/B-test ing

Don’t start to soon


A/B-test ing

Test the right way


A/B-test ing

Minimize regret

Run more experiments, maximize your capacity!


http://test ing.agency

D i g i t a l E l i t e C a m p E s t o n i a , J u n e 1 3 t h 2 0 1 5 To n W e s s e l i n g – C E O – Te s t i n g . A g e n c y

# E l i t e C a m p - @ To n W

When to run more experiments?

Data & Analytics

Keynote Digital Elite Camp Estonia 2015 - ROAR - when to run more experiments