Keynote Digital Elite Camp Estonia 2015 - ROAR - when to run more experiments

D i g i t a l E l i t e C a m p E s t o n i a , J u n e 1 3 t h 2 0 1 5 To n W e s s e l i n g – C E O – Te s t i n g . A g e n c y

# E l i t e C a m p - @ To n W

When to run more experiments?

Ton Wesseling | @ TonW Run more experiments #TonW

http://test ing.agency

ROAR model

Methods used to improve conversion

Methods companies plan to use

A/B-test ing is

Everyone is pushing to test more

Never sat isfied with conversion rates

Start test ing or test more

NO Growth problems wi l l d issapear…

A/B-test ing is not a solut ion

It’s a methodology

I t ’s part of a:

Continuous Optimization

Program

And you need to know:

How What

Growth curve – how i t was teached

ROAR model

Optimizat ion phases - ROAR

Time span!

Risk! + Optimization!

Time span!

Risk! + Optimization! + Automation!

Time span!

Risk! + Optimization! + Automation! Re-think!

Time span!

1.000 conversions!per month!

Time span!

10.000 conversions per month!

Stat ist ics

Time span!

15% impact needed!

5% impact!needed!

Impact needed: all conversions with an average conversion rate of 2%, a test length of 3 weeks and a power of 80%!

Low number test ing?

Why can it hurt you?

Warning: low significance levels

ü  Big chance the winner is not a real winner

Warning: low power levels

ü  Big chance a real winner is not recognized

Example test set-up

ü  90% significance level

ü  100 conversions per variation

ü  Average conversion rate: 2%

ü  20 out of 100 ideas is a winner…

ü  …with an average uplift of 10%

Average: 75% significance & 40% power

ü  100 tests will give 10 false positives (90% significance)

ü  40% of the 20 winners are recognized (40% power)

ü  8 real winners & 10 false positives

ü  NOT 18 improvements of 10%+

Yes, I ’ve got those 1000+ conversions

Time span!

With 1000+ conversions a month

Your tes t capac i ty i s max .

20+ tests a year

wi th a s ta t i s t i ca l power o f 80% on pred ic ted up l i f t s o f 15%

Tests with measureable impact

On average

1 out of 3

Simply truth: you’re just not always able to create a winner with enough impact

So in th is case, you’ve got impact

Every 7 or 8 weeks

Slower? Energy wi l l run out! And your continous optimization program will die

What to test? Business cases!

Delivery –test with in stock. 4 weeks

or 2 days?!

What to test? Business cases!

Service charge, do it or not? Include in

price or not?!

What to test? Big design changes

Yes this can be done with an A/B-test tool, you only need A/B-test coders!

ROAR – Opt imizat ion – moving up

Time span!

Final ly you can test detai ls…

Is Multi Variate Testing (which is also A/B-testing but with more varations) a good idea? Almost never!!

You should grow to

Customer Buying Reasons

Reasons behind the customer journey

A cont inuous opt imizat ion team

Analyst Psychologyst

Designer Developer

Team Lead

They keep on asking:

Why are these users here?

And they want to know

How they can

help these users’ brains

to fullfil their needs?

Data & Psychology

Digital Data Persuasion Psychology

Analyze &

Experiment

A lack of resources

Test ing takes resources

Use it to LEARN

FA C T & A C T

Tell Conclude Analyze

Test Create Analyze Find

Know the company

Mission, Vision, Stategy & Goals

Know your user ( journeys)

Based on: asking – why are you here?

FA C T & A C T

Test Create Analyze Find Create

And yes, your team needs an economic / consumer Psychologyst and a UX designer!

Tests that can be done: A var iat ion

Tests that can be done: B var iat ion

Need help with coding A/B-test variations like this one? – http://testing.agency!

Tests that can be done: A & B var iat ion

FA C T & A C T

Test Create Analyze Find

Analyze

Your analyst will analyze the results!

We measure in Analyt ics software

Calcu late resu l ts : ABTestGuide.com/calc

Not s ignificant

ü It’s not a winner, stop

ü But, it does not mean it’s a losing one!

ü Average: 33% of your tests will have winners If you are optimizing dialogues!

Choose what to analyze up front

ü  You just go and dig until you find something

(you will always find something that seems true BUT is not!)

Don’t stop the test once i t ’s s ignificant!

You just let i t run unt i l i t ’s s ignificant

Your test has a fixed lenght!

It’s even a fixed amount of test population, but we love to test business cycles and/or weeks!

Be aware: Your test takes too long

ü  Cookie deletion (you will lose 10% in two weeks)

They will re-enter the and pollute your samples!

Test length: ABTestGuide.com/calc

T ip: Don’t test broken stuff

ü  Always check browser compatibility!

ü  Don’t break dynamic stuff!

T ip: Slow down the control down

ü  Add the same code to the control

T ip: leave out s o m e v is i tors

ü  Only test on those who can be changed (if you test a sales promotion – leave out your current clients who just logon)

T ip: Only test on fresh people

ü  Start with control for a full purchase cycle (and then start sending traffic to your variations – setting the control to 0%)

T ip: give them t ime to buy!

ü  Stop getting fresh visitors after your test time (send new traffic to the control, but give you tested people time to finish converting)

+ Optimization! + Automation!

Back to the ROAR model

Time span!

Risk! Re-think!

ROAR – Opt imizat ion – moving up

Time span!

Test capacity

maximum number of

experiments per year that your site or app

can handle

Regret

amount of money you are losing per year by

not using your full test capacity

Calculate your test capic i ty

http://ondi.me/size!

ROAR – Automation

Time span!

Automation

You tested which locat ions are cr i t ical

You know which dia logues are cr i t ical

Automation

Contextual bandits

If you are ready – give me a call!!

Now you’ve moved to

Algorithm testing

Which internal and external influencers do have an impact on which segment of users?!

+ Optimization! + Automation!

ROAR – Automation

Time span!

Risk! Re-think!

ROAR – You don’t stop opt imiz ing

Time span!

ROAR – Maybe you’ l l never automate

Time span!

ROAR – Re-think

Time span!

The ul t imate Re-think resource

CRO br ings user knowledge

You wi l l make big impact on Discover

Customer knowledge center

Making money f rom convers ion improvements

Creat ing more and more cus tomer ins igh ts Which can be monet ized

on even a la rger sca le

Now, that’s

Done test ing? No!

TESTING NEVER STOPS

Okay, end of life, cash cow, no need for any further growth accelaration or user knowledge!

A/B-test ing

Don’t start to soon

A/B-test ing

Test the right way

A/B-test ing

Minimize regret

Run more experiments, maximize your capacity!

http://test ing.agency

D i g i t a l E l i t e C a m p E s t o n i a , J u n e 1 3 t h 2 0 1 5 To n W e s s e l i n g – C E O – Te s t i n g . A g e n c y

# E l i t e C a m p - @ To n W

When to run more experiments?

Keynote Digital Elite Camp Estonia 2015 - ROAR - when to run more experiments

Data & Analytics

NYPL ROAR!

Though Waters Roar

Estonia Estonia 21

Katy Perry - roar

ROAR by gnancy

Mik Roar Ray

ROAR OF THE LION”ROAR OF THE LION” · 2017. 6. 25. · ““ROAR OF THE LION”ROAR OF THE LION” Lions District 4-A2 “ROAR OF THE LION” Page 3 Hello Lions, Lionesses and

VIGOR ROAR

ROAR Issue 2

Missing The Roar?

The Morning Roar - Nov. 13 - 2009 Road to the Roar

Roar! UpRoar

The Lion's Roar

Lion's Roar

LambsThat Roar

Lions roar test

Tiger Roar

ROAR ISSUE 3

Roar 87 Prog

Tigers roar. Tigers Tigers roar Swallows will return.Swallows