SUBTITLE BELOW
Optimize for €€€HOW WE OPTIMIZE AT THE BIGGEST HOTEL CHAIN IN THE NETHERLANDS
@AM_Klaassen / @tomvdberg1
How we got here…
@AM_Klaassen / @tomvdberg1
The quest for female speakers…
@AM_Klaassen / @tomvdberg1
Female speakers…
?
@AM_Klaassen / @tomvdberg1
Our lovely colleagues
What we do…
@AM_Klaassen / @tomvdberg1
Conversion rate optimization
Analytics Psychology
Lots of A/B-tests
@AM_Klaassen / @tomvdberg1
Our clients
@AM_Klaassen / @tomvdberg1
Adding
direct value
Learning
user behavior
@AM_Klaassen / @tomvdberg1
@AM_Klaassen / @tomvdberg1
@AM_Klaassen / @tomvdberg1
Van der Valk
Unique website for each hotel
1 centralized team
Paid by % of their turnover
@AM_Klaassen / @tomvdberg1
Optimization of Valk
Increase revenue through CRO
Reduce costs per experiment
“We’re not in the business of
science, we’re in the business
of making money”
3 Tactics
1.
Free test software
@AM_Klaassen / @tomvdberg1
GTM testing
Easy setup
Free
Preview mode
2.
Change of statistics
@AM_Klaassen / @tomvdberg1
Focus on finding proof
@AM_Klaassen / @tomvdberg1
Example test: Order flow
@AM_Klaassen / @tomvdberg1
Example test: Set completion
A B
@AM_Klaassen / @tomvdberg1
Test result
@AM_Klaassen / @tomvdberg1
• No effect on conversion
Don’t implement
Re-test with higher volumes
Conclusion
@AM_Klaassen / @tomvdberg1
@AM_Klaassen / @tomvdberg1
What’s the alternative?
Frequentist
statistics
Bayesian
statistics
@AM_Klaassen / @tomvdberg1
Example test: Set completionA B
@AM_Klaassen / @tomvdberg1
Bayesian Test evaluation
@AM_Klaassen / @tomvdberg1
Focus on risk assessment
88,4%
A test result is the probability that B outperforms A:
ranging from 0% - 100%
@AM_Klaassen / @tomvdberg1
IMPLEMENT B PROBABILITY * EFFECT ON REVENU
Expected risk 11,6% - € 69.791
Expected uplift 88,4% € 213.530
Contribution € 180.733
* Based on 6 months and an average order value of € 175
Make a risk assessment
@AM_Klaassen / @tomvdberg1
Implement BB
@AM_Klaassen / @tomvdberg1
abtestguide.com/bayesian/
Roy Schieving
Annemarie Klaassen
@AM_Klaassen / @tomvdberg1
Conclusion Bayesian
Easier to understand and
communicate
Better suits the business
Don’t throw away good ideas (indicatively significant)
Higher implementation rate and
revenue
€ 0
€ 1.000.000
€ 2.000.000
€ 3.000.000
€ 4.000.000
€ 5.000.000
€ 6.000.000
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39
Expected impact on revenue in 1 year
€ Frequentist € Bayesian
3.
Change of test method
@AM_Klaassen / @tomvdberg1
50%
50%
A/B-test
@AM_Klaassen / @tomvdberg1
Traffic allocation over time
A
B
Tra
ffic
Time
Profit or loss?
@AM_Klaassen / @tomvdberg1
What’s the alternative?
A/B-test Bandits
If you have multiple options:
which bandit
will pay-out the
most?
@AM_Klaassen / @tomvdberg1
Explore Exploit
?
Explore / Exploit dilemma
@AM_Klaassen / @tomvdberg1
Bandit
100%
@AM_Klaassen / @tomvdberg1
Bandit
80%
20%
10%
90%
@AM_Klaassen / @tomvdberg1
Traffic allocation over time
A
B
Tra
ffic
Time
More Profit!
@AM_Klaassen / @tomvdberg1
Limited time frame Automation
When use Bandits?
Earn over learn
How?
@AM_Klaassen / @tomvdberg1
Example 1
@AM_Klaassen / @tomvdberg1
Example 1
Older target group
Comparing price on
other sites
Behavior of others
@AM_Klaassen / @tomvdberg1
Example 1
Control No message
Variation B Copy default
Variaton C
Variation D
11,1%
10,6%
10,4%
10,5%
@AM_Klaassen / @tomvdberg1
Example 1 – Effect on revenue
A
B
Tra
ffic
Time
+ € 6.637
@AM_Klaassen / @tomvdberg1
Example 2
Control New control
Variation B Copy control
Variation C
Variation D
9,2%
8,9%
7,8%
8,4%
@AM_Klaassen / @tomvdberg1
Example 2 – Effect on revenue
A
B
Tra
ffic
Time
+ € 19.862
@AM_Klaassen / @tomvdberg1
Conclusion Bandits
Less regret during testing
No implementation costs
Optimizing per user segment
1. Free test software
2. Bayesian statistics
3. Bandit testing
revenue
Maximize
@AM_Klaassen / @tomvdberg1
“We’re not in the business of
science, we’re in the business
of making money”
TÄNAN VÄGA!
Bayesian calculator: abtestguide.com/bayesian/
Slide deck: ondi.me/elitecamp2016
@AM_Klaassen
nl.linkedin.com/in/amklaassen
@tomvdberg1
nl.linkedin.com/in/vandenbergtom