Upload
orly-amrany
View
110
Download
3
Embed Size (px)
Citation preview
Orly Amrany
General Manager, Wix
When the question ‘why?’ is not relevant
The Unexpected Facts of Life
Hi, I’m Orly Amrany General Manager, Head of Premium Services @ Wix
Question #01
• Assume you have 3 Million Subscription (Monthly/Yearly)
• Each recurring charge is done on it’s purchase date
• Re-try upon failure is done every 7 days
• Success rate of recurring charge is ~90%
• Which day do we have the best changes to charge ?
24th or 25th?
85.25%
81.94%
84.14%
79.80%
82.79%
85.67%
85.13%
83.24%
81.59%
86.80%
83.55%
85.48%
86.49%86.96%
79.00%
80.00%
81.00%
82.00%
83.00%
84.00%
85.00%
86.00%
87.00%
AU BR CA ES FR GB US
24th Vs. 25th, initial recurring
85.25%
81.94%
84.14%
79.80%
82.79%
85.67%
85.13%
83.24%
81.59%
86.80%
83.55%
85.48%
86.49%86.96%
79.00%
80.00%
81.00%
82.00%
83.00%
84.00%
85.00%
86.00%
87.00%
AU BR CA ES FR GB US
24th Vs. 25th, initial recurring
+0.5% increase in approval rate
85.25%
81.94%
84.14%
79.80%
82.79%
85.67%
85.13%
83.24%
81.59%
86.80%
83.55%
85.48%
86.49%86.96%
79.00%
80.00%
81.00%
82.00%
83.00%
84.00%
85.00%
86.00%
87.00%
AU BR CA ES FR GB US
24th Vs. 25th, initial recurring
+0.5% increase in approval rate
+15k saved subscriptions
85.25%
81.94%
84.14%
79.80%
82.79%
85.67%
85.13%
83.24%
81.59%
86.80%
83.55%
85.48%
86.49%86.96%
79.00%
80.00%
81.00%
82.00%
83.00%
84.00%
85.00%
86.00%
87.00%
AU BR CA ES FR GB US
24th Vs. 25th, initial recurring
+0.5% increase in approval rate
+15k saved subscriptions
+1.5M
And the winner is…
The 24th!
You’ve won 1,500,000million dollars
Question: Why checking the 24th vs. 25th?
Question: Why checking the 24th vs. 25th?
Answer: Because we could
• We had data of million of transactions
• We actually checked all days of the month
• A lot of other assumptions failed..(15th Vs 16th)
• Should we AB test? (NO)
On the 24th there is more changes for charges to succeed
We gained +1.5M USD with a simple change!
When you have tones of data, you can find out a lot of interesting and unexpected facts
You should do it as soon as possible: Time is money
Make all recurring charges on the 24th ? (NO) Improve re-try logic (Which Tool)
So what did we learn?
So what did we learn?
Don’t ask yourself
‘why?’
Ask yourself
‘why didn’t I check this before?’
Some more fun examples
Question #02
• Same data
• Which day do we have the best changes to charge?
Weekdays or weekends?
And the winner is…
Weekday!
4.41%
-1.85%
6.36%
2.42%
5.38%
8.00%
-0.59%
1.18%
5.67%
-4.84%
6.39%
0.45%
2.97%
5.94%
-6.67%
0.10%
-0.62%
5.21%
7.62%
1.50%
2.59%3.22%
-5.88%
3.41%
0.66%
11.80%
6.81%
-0.35%
8.05%
7.64%
2.39%
-10.00%
-5.00%
0.00%
5.00%
10.00%
15.00%
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31
Approval rate difference - weekdays vs. weekends
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31
4.41%
-1.85%
6.36%
2.42%
5.38%
8.00%
-0.59%
1.18%
5.67%
-4.84%
6.39%
0.45%
2.97%
5.94%
-6.67%
0.10%
-0.62%
5.21%
7.62%
1.50%
2.59%3.22%
-5.88%
3.41%
0.66%
11.80%
6.81%
-0.35%
8.05%
7.64%
2.39%
-10.00%
-5.00%
0.00%
5.00%
10.00%
15.00%
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31
Approval rate difference - weekdays vs. weekends
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31
+1.3% increase in approval rate
• Same data
• Which hour of the day would be the best? (in Brazil for example)
Time or Day?
And the answer is…
8am – 10pm
20.00%
25.00%
30.00%
35.00%
40.00%
45.00%
50.00%
55.00%
60.00%
65.00%
70.00%
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
Approval Time, Recurring/UTC time
all
Thanks