Why do we need A/B Testing?
• Tests takes the guesswork out
• Enables data-backed decisions
• Enhances engagement and retention
Define the test
• Hypothesis – The layout with the Google+ button at the left will increase the number of total registered users.
• Goal – A registered user (the user can skip registration).
• View event
- Login fragment onCreate().
- Login activity onCreate().
• Variables – Facebook button position (left or right).
• Participants – New users.
Amazon A/B Testing SDK
• Very customizable, you can do just about anything as long as you know what it is you want to do
• It’s free
• Drippler created an open source library that simplify the process
Setup the A/B test
• Setup identifier
– https://developer.amazon.com/al/index.html
Dive into the code
• https://github.com/Drippler/ABTester
public class MyApplication extends Application {
@Overridepublic void onCreate() {
super.onCreate();ABTester.init(getApplicationContext(),
"my_public_key", "my_private_key");
}}
private void initLoginActivityTest() {
try {
ABTester.syncPreFetch(
TimeUnit.SECONDS.toMillis(15),
new ABTest("Login page test", false, "Facebook is
first”) );
} catch (TimeoutException e) {
// Couldn't reach amazon servers
}
}
Fetch the test
Login Fragment
@Override
public void onCreate(Bundle savedInstanceState) {
super.onCreate(savedInstanceState);
setRetainInstance(true);
ABTester.recordEvent(
"Login fragment shown",
false);
}
Login Fragment@Override
public View onCreateView(LayoutInflater inflater,
ViewGroup container, Bundle savedInstanceState) {
boolean shouldShowFacebookFirst =
ABTester.getBoolean("Login page test", "Facebook is
first", false);
if (shouldShowFacebookFirst)
return inflater.inflate(R.layout.facebook_first, null);
else
return inflater.inflate(R.layout.google_first, null);
}
Report goal event
public void onUserLoggedIn() {ABTester.recordEvent("Sign in", false);
}
@Overrideprotected void onPause() {super.onPause();/* Submit the events that were previously stored locally. * asynchronously * call it in the onPause() method of an activity */ABTester.submitEvents();
}
Analyze the results
Variation Views Conversions Conversion rate
Change
Google+first
1064 320 30.08%
Facebook first
1043 250 23.97% -20.30%
Analyze the results
Variation Views Conversions Conversion rate
Change
Hat off 200 31 15.50%
Hat on 200 38 19.00% +22.58%
Conversion is never a single number.
Confidence level
• Measure the reliability of an estimate
– The confidence levels help us understand if the results are different merely by chance or by reason
• 95% confidence level is considered good
Analyze the results
Variation Views Conversions Conversion rate
Change
Google First
1064 320 30.08% ±2.32%
Facebook First
1043 250 23.97% ±2.18%
-20.30%
• Confidence level of 99%
Analyze the results
https://developer.amazon.com/public/apis/manage/ab-testing/doc/math-behind-ab-testing
Choose the best variation
• Launch
– Choose the winning variation
– Control the percentage of customers that receive a new feature
Road map
• What is an A/B test ?
• Segmentation
• Multiple Experiments
http://visualwebsiteoptimizer.com/split-testing-blog/wp-content/uploads/2010/10/2010.09.10.ab_.png
Define the test
• Hypothesis – Coloring the “Rate” button, will increase the button’s click rate
• Goal – Click event on the “Rate” button
• View event – RateUsDialogFragment show();
• Variables – “Rate” button color
• Participants – All users
Rate us DialogFragment
public class RateUsDialog extends DialogFragment {
public static void show(FragmentManager fm,
int color) {
RateUsDialog rateUs = new RateUsDialog();
Bundle extras = new Bundle();
extras.putInt(“color”, color);
rateUs.setArguments(extras);
rateUs.show(fm, “my tag”);
ABTester.recordEvent("Rate us dialog shown", false);
}
Rate us DialogFragment
@Override
public Dialog onCreateDialog(Bundle
savedInstanceState) {
int color = getArguments().getInt("color");
return createColoredDialog(color);
}
Rate us DialogFragment
private Dialog createColoredDialog(int color) {
...
.setPositiveButton("Rate", new OnClickListener() {
@Override
public void onClick(DialogInterface dialog, intwhich) {
ABTester.recordEvent("Rate button click", false);
}
});
return myDialog;
}
Rate us test1) Asynchronously prefetching SplashActivity
Default timeout is 60 seconds, and can be overriden by
using preFetch(long timeout, ABTest... Test)
ABTester.preFetch(
new ABTest("Rate us test", false, "Rate button color")
);
Rate us test2) Show the dialog
String fetchedColor = ABTester.getString(
"Rate us test", "Rate button color", "#F5F5F5");
int color = Color.parseColor(fetchedColor);
RateUsDialog.show(getFragmentManager(), color);
3) Submitting the results onPause()
ABTester.submitEvents();
Analyze the results
Variation Views Conversions Conversion rate Change Confidence
Control (white)
865 234 27.05%± 1.51%
Variation A (green)
904 250 27.65%± 1.49%
-0.2%
Variation B (red) 830 230
27.71%± 1.55% +2.4% 51%
What can I do with these results?
Segmentation
Variation Views Conversions Conversion rate Change Confidence
Control (white)
432 92 21.30% ± 1.97%
Variation A (green)
464 165 35.56% ± 2.22% +66.9% 98.7%
Variation B (red)
420 120 28.57% ± 2.20% +34.1%
Variation Views Conversions Conversion rate Change Confidence
Control (white)
433 142 32.79% ± 2.26% +22.2% 97.1%
Variation A (green)
440 85 19.32% ± 1.88% -27.9%
Variation B (red)
410 110 26.83% ± 2.19% -18.8%
Under 40
Over 40
Define the test
• Hypothesis – Coloring the “Rate” button, will increase the button click rate
• Goal – Click event on the “Rate” button
• View event – RateUsDialogFragment show();
• Variables – “Rate” button color
• Participants – All users
Define the test
• Hypothesis – Coloring the “Rate” button, will increase the button click rate
• Goal – Click event on the “Rate” button
• View event – RateUsDialogFragment show();
• Variables – “Rate” button color
• Participants – Age specific tests
Assign the segment
• In your code before the fetch
– ABTester.addDimension(”age", myAge);
• Dimension will be remembered forever
• ABTester library will automatically add a “percentile” dimension
– ABTester.addDimension(“percentile”,
new Random().nextInt(100));
Road map
• What is an A/B test ?
• Segmentation
• Multiple Experiments
http://unbounce.com/a-b-testing/shocking-results/
Multiple Experiments
• Serial tests– Run the tests one after the other, without the
need to redistribute your app
– More accurate but takes more time
ABTester.preFetch(
new ABTest("Rate us test", false, "Rate button
color", "Rate actionbar icon")
);
Multiple Experiments
• Parallel tests– Run the tests together, increasing the ‘noise’ for
dependent tests – Faster
ABTester.preFetch(new ABTest("Rate us button", false,
"Rate button color”)new ABTest("Rate us actionbar", false,
"Rate button icon”) );
Thanks, any questions ?
Don’t overthink it, use Drippler’s A/B test library
https://github.com/Drippler/ABTester
Nir Hartmann, [email protected]
Droidcon Tel-Aviv 2014