Upload
webanalisten-nl
View
78
Download
0
Embed Size (px)
Citation preview
Speaker
#DDTT
Hubert Wassner Chief Data Scientist
@hwassner
Christopher Broque Senior Business Developer [email protected]
#DDTT
Paris Londres Cologne Sydney Madrid New-York
120+ Passionate people
450+ references
21+ awards
9+ billions Tested visitors
23ème
107ème
AB Tasty : CRO specialist !
First step
Variation B has 10 % (measured) gain & 95 % confidence index... but 1 % gain in production is likely!
The frequentist Approach
Provides a Pvalue = « The probability to see such data without any (real)
difference between A &B. »
Confidence index = (1-Pvalue)*100
Problem : It only qualifies the existence of a difference, not it's size! Example :
#DDTT
Smarter step
Bayesian Approach
give confidence interval on the gain Example : gain (A->B) is in [1%:15 %]
1. No bad surprise in production
&
2. Better choice when facing implementation cost. Example: testing a product recommender system
#DDTT
What about media ?
E-commerce
N visits for a “unique” conversion => CTR [0:1]
Media
N visits for n “multiple” conversions (where n>N) => CTR [0:∞]
#DDTT
E-commerce ≠ media
E-commerce
1 visitor x 10 conversions =
10 visitors x 1 conversion
Media
1 visitor x 10 conversions <
10 visitors x 1 conversion
#DDTT
Multiple conversion Analysis
A - Original 12 000 visitors 2 000 conversions CTR : 16,6%
Gain : [-3% : 28%] => 7,5% [0.7% : 15%] => 7,3% Chances of winning : 87% 98%
B A + 1 visitor, with 150 conversions CTR : 17,9%
C A + 15 visitors, with 10 conversions each CTR : 17,9%
12 000 visitors 2 000 conversions CTR : 16,6%
A + 1 visitor, with 150 conversions CTR : 17,9%
A + 15 visitors, with 10 conversions each CTR : 17,9%
Multiple conversion Analysis
A - Original
[-3% : 28%] => 7,5% [0.7% : 15%] => 7,3%
87% 98%
B C
Gain :
Chances of winning : #DDTT
We need to go faster !!
Exploitation time
Paper news < 8 days Private sale 2-4 days Classified ad < 8 days
No time for an A/B test...
Need to do a “smart” test
#DDTT
“Multi-armed bandit”
Imagine you have :
• N different slot machines • P coins
Objective :
• Win a maximum with minimum coins
#DDTT
Application to web data
Imagine you have :
• N variations • P visitors
Objective
• Making the more transaction you can
#DDTT
Application to web data
Intérêts : • Limiting regrets (lower T.C.O.) • Optimizing short living objects, like flash sales, newspaper article, ...
• Testing simultaneously more variations • Classic A/B test ~= testing A/B/C/D/E with « bandits »
#DDTT
Conclusion
A/B test can be more complex that you might think
It has evolved with specificity of web businesses.
And that's only the beginning! (Artificial Intelligence is on the road...)