TEXT

SAIF Ignition

Bangalore

20th Sep 2013

A/B testing for mobile apps

WHAT IS A/B TESTING?

Version ‘A’ vs Version ‘B’

That’s how the name A/B testing came about.

How does it work?

WEB A/B TESTING - Etsy

• In-house A/B testing solution

• Results are sometimes surprising!

• Infinite scroll is presumed to be a better experience as you see more items faster

• Visitors clicked on

• fewer results (-11%)

• favourited less (-8%)

• fewer items purchased (-22%)

INFINITE SCROLL

Infinite

Scroll

Source – Design for continuous experimentation, http://www.slideshare.net/danmckinley/design-for-continuous-experimentation

WEB A/B TESTING

• What went wrong?

• Changed multiple things at once

• They should have broken-down the feature change into two hypothesis testing validations:

• More items is better

• Faster is better

INFINITE SCROLL

WEB A/B TESTING

• Broke-down the process into testing multiple short measurable tests

• Decrease the number of items in dropdown

• Default to ‘all items’

• Search bars for in-shop and in-page search

• Rich auto-suggest

• .. and many more such tests!

SEARCH DROPDOWN

WEB A/B TESTING

• Most people did not notice the major feature changes

• Search dropdown was subsequently withdrawn

• Iterative ‘Design – Develop – Measure’

SEARCH DROPDOWN

Design Develop Measure

Infinite Scroll

Search Dropdown

Design Develop Measure

MOBILE A/B TESTING – Why is it needed?

• Increase conversion, revenue, engagement metrics

• User-segmented and targeted app feature launches

• Most mobile apps struggle to make money - faster iterations will genuinely help developers in monetizing their apps

MOBILE A/B TESTING – Challenges

• Complex coding involved

• Delays in app store updates -

• slowing the pace of innovation

• Simplicity and ease of web A/B testing not prevalent on mobiles -

• thus devoid of 'iterative improvements‘

• key growth drivers on the web

WHY A/B TEST MOBILE APPS?

• Everyone A/B tests in real-life:

• People try two or more pieces of clothing before buying

• You have a test drive of two shortlisted cars

• The HiPPO (the Highest Paid Person’s Opinion) method of decision making

• Decisions based on such opinions, hunches, guesses and conjectures

• Use data in decision making

• A/B testing on mobile apps is a great way to increase conversions and improving your key mobile metrics.

WHAT TO A/B TEST IN NATIVE MOBILE APPS?

If it is there in your app, it can be A/B tested!

Example use cases:

• Call to Action buttons (these include buttons like ‘buy now’, ‘add to cart’, ‘sign up’)

• Set any value to the text, size, colors, fonts or its position

• App features – Test engagement metrics by having different number of free levels in games

• User experience and app navigation sequence tests

• Entire screen themes/layouts

• Different page opens on tap of a button

• Different ways of social media credential-based sign-ins

Aim was to increase

conversion on the following:

•Review button – User

engagement metric

•Price link – Increases

revenue in the form of

referral fees.

VARIATIONS: Control Variation

DESCRIPTION: A review button with longer

text No visual element (arrow) beside the price

A review button with shorter text A small arrow beside the

price

DISTRIBUTION: 50% 50%

RESULTS:

Price link clicks on 18.1% of screen opens

Price link clicks on 22.4% of screen opens

Longer text button got clicked 1.4% times of screen

opens

Shorter text button got clicked 1.1% times of screen opens

The app lets users compare prices of multiple cross-category products from online stores.

Hypotheses:

•A short or a long text on a

‘review’ button

•Impact of a small visual

element (arrow) beside the

price

Price link with a visual element resulted in an increase of 23.4% in revenues for the app!

The longer text on the review button is better at driving engagement by 23.1%

CASE STUDY: PRICE COMPARISON APP

TIME FOR DEPLOYMENT & ITERATION fast slow

in-code test (normal)

PR

E S

ET

UP

TIM

E

•Integrate SDK •Pre-specify tests •Pre-specify attribute values

mo

re

less

COMPARING TEST TYPES (1/3)

TIME FOR DEPLOYMENT & ITERATION fast slow

in-code test (normal)

data driven test

PR

E S

ET

UP

TIM

E

•Integrate SDK •Pre-specify tests •Pre-specify attribute values

mo

re

less

COMPARING TEST TYPES (2/3)

•Integrate SDK •Pre-specify tests •Change attribute values live

•Bypass app store updates •Drag-n-drop visual interface •Fast and easy deployment

wysiwyg

TIME FOR DEPLOYMENT & ITERATION fast slow

in-code test (normal)

data driven test

PR

E S

ET

UP

TIM

E

•Integrate SDK •Pre-specify tests •Change attribute values live

•Integrate SDK •Pre-specify tests •Pre-specify attribute values

mo

re

less

COMPARING TEST TYPES (3/3)

Live Tracking of User’s Interaction with the App

Content Management System Feature Launches and Segmented Rollouts

NEW FEATURES

“If genius is, as Edison said, 1% inspiration and 99% perspiration, then perhaps innovation is 1% invention and 99% iteration.”

www.appiterate.com