A/B Testing in Mobile

A/B TESTING IN MOBILE

MAURICIO T ZAQUIA@mtzaquia

MAURICIO TZAQUIA

APPLE DISTINGUISHED EDUCATOR, CLASS OF 2015

iOS DEVELOPER @

MOBILEMARKET

NEW USER RETENTION

0x

1x

2x

Web Apps

MOBILE TRANSACTIONS

Apps 54 %

Web 46 %

MOBILE CONVERSION

Web

Apps

0x 1x 2x 3x

”A/B TESTING IS A TERM FOR A RANDOMIZED EXPERIMENT WITH TWO VARIANTS, A AND B, WHICH ARE THE

CONTROL AND VARIATION IN THE CONTROLLED EXPERIMENT.”

Wikipedia, 2017

HYPOTHESIS EXPECTATION

IMPLEMENTATION

EXECUTION

ANALYSIS

DECISION

/* code */

button.setTitle("Buy", for: .normal)button.setTitleColor(UIColor.green, for: .normal)

// controller code...

// controller code...

// controller code...

if (Experiments.ios_button_urgency.variant == 1) { button.setTitle("Buy before it's too late!", for: .normal) button.setTitleColor(UIColor.red, for: .normal) } else { button.setTitle("Buy", for: .normal) button.setTitleColor(UIColor.green, for: .normal) }

// controller code...

35% 42%

button.setTitle("Buy before it's too late!", for: .normal) button.setTitleColor(UIColor.red, for: .normal)

// controller code...

// controller code...

Experiments.ios_button_urgency.variant

FEATURE FLAGS

Experiments.ios_button_urgency.variant

WHAT HAPPENS IF A WEBPAGE CRASHES?

WHAT HAPPENS IF

AN APP CRASHES?

”47% OF THE APPS CRASH AT LEAST 1% OF THE TIME, AND 32% OF THE APPS HAVE A

CRASH RATE OVER 2%”

Crittercism, 2016

EVERYBODY BUGS

FINDING CRASHES IS ”EASY”…

…BUT WITH THE RIGHT TOOLS

NETWORKLOAD

SCREENISSUES

MEMORYWARNING

NETWORKLOAD

Experiments.ios_deep_search

DAT

A E

XCH

AN

GE

OVE

R TI

ME

BASE VARIANT

NETWORKLOAD

Experiments.ios_deep_search

APP

ABA

ND

ON

MEN

T O

VER

TIM

E

BASE VARIANT

MEMORYWARNING

Experiments.ios_revamped_card

MEM

ORY

WA

RNIN

GS

OVE

R TI

ME

BASE VARIANT

SCREEN ISSUES

Experiments.ios_table_refactor

1.26% ± 1.64%

BASE

VARIANT

SCREEN FREEZE OVER 1,5 SECONDS

SCREEN ISSUES

Experiments.ios_layout_flow

-95.78% ± 7.19%

BASE

VARIANT

SCREEN VIEWS

SCREEN FREEZE

Experiments.ios_table_refactor

BASE VARIANT

FRAMEWORKS

HTTPS://GITHUB.COM/KEEPSAFE/SWITCHBOARD HTTPS://GITHUB.COM/MATTT/SKYLAB

COMPANIES

Q&A

THANK YOUMAURICIO T ZAQUIA@mtzaquia