31
A/B TESTING IN MOBILE MAURICIO T ZAQUIA @mtzaquia

A/B Testing in Mobile

Embed Size (px)

Citation preview

Page 1: A/B Testing in Mobile

A/B TESTING IN MOBILE

MAURICIO T ZAQUIA@mtzaquia

Page 2: A/B Testing in Mobile

MAURICIO TZAQUIA

APPLE DISTINGUISHED EDUCATOR, CLASS OF 2015

iOS DEVELOPER @

Page 3: A/B Testing in Mobile

MOBILEMARKET

Page 4: A/B Testing in Mobile

NEW USER RETENTION

0x

1x

2x

Web Apps

MOBILE TRANSACTIONS

Apps 54 %

Web 46 %

Page 5: A/B Testing in Mobile

MOBILE CONVERSION

Web

Apps

0x 1x 2x 3x

Page 6: A/B Testing in Mobile

”A/B TESTING IS A TERM FOR A RANDOMIZED EXPERIMENT WITH TWO VARIANTS, A AND B, WHICH ARE THE

CONTROL AND VARIATION IN THE CONTROLLED EXPERIMENT.”

Wikipedia, 2017

Page 7: A/B Testing in Mobile
Page 8: A/B Testing in Mobile

HYPOTHESIS EXPECTATION

IMPLEMENTATION

EXECUTION

ANALYSIS

DECISION

Page 9: A/B Testing in Mobile

/* code */

Page 10: A/B Testing in Mobile

button.setTitle("Buy", for: .normal)button.setTitleColor(UIColor.green, for: .normal)

// controller code...

// controller code...

Page 11: A/B Testing in Mobile

// controller code...

if (Experiments.ios_button_urgency.variant == 1) { button.setTitle("Buy before it's too late!", for: .normal) button.setTitleColor(UIColor.red, for: .normal) } else { button.setTitle("Buy", for: .normal) button.setTitleColor(UIColor.green, for: .normal) }

// controller code...

35% 42%

Page 12: A/B Testing in Mobile

button.setTitle("Buy before it's too late!", for: .normal) button.setTitleColor(UIColor.red, for: .normal)

// controller code...

// controller code...

Page 13: A/B Testing in Mobile

Experiments.ios_button_urgency.variant

Page 14: A/B Testing in Mobile

FEATURE FLAGS

Page 15: A/B Testing in Mobile

Experiments.ios_button_urgency.variant

Page 16: A/B Testing in Mobile

WHAT HAPPENS IF A WEBPAGE CRASHES?

Page 17: A/B Testing in Mobile

WHAT HAPPENS IF

AN APP CRASHES?

Page 18: A/B Testing in Mobile

”47% OF THE APPS CRASH AT LEAST 1% OF THE TIME, AND 32% OF THE APPS HAVE A

CRASH RATE OVER 2%”

Crittercism, 2016

Page 19: A/B Testing in Mobile

EVERYBODY BUGS

Page 20: A/B Testing in Mobile

FINDING CRASHES IS ”EASY”…

Page 21: A/B Testing in Mobile

…BUT WITH THE RIGHT TOOLS

NETWORKLOAD

SCREENISSUES

MEMORYWARNING

Page 22: A/B Testing in Mobile

NETWORKLOAD

Experiments.ios_deep_search

DAT

A E

XCH

AN

GE

OVE

R TI

ME

BASE VARIANT

Page 23: A/B Testing in Mobile

NETWORKLOAD

Experiments.ios_deep_search

APP

ABA

ND

ON

MEN

T O

VER

TIM

E

BASE VARIANT

Page 24: A/B Testing in Mobile

MEMORYWARNING

Experiments.ios_revamped_card

MEM

ORY

WA

RNIN

GS

OVE

R TI

ME

BASE VARIANT

Page 25: A/B Testing in Mobile

SCREEN ISSUES

Experiments.ios_table_refactor

1.26% ± 1.64%

BASE

VARIANT

SCREEN FREEZE OVER 1,5 SECONDS

Page 26: A/B Testing in Mobile

SCREEN ISSUES

Experiments.ios_layout_flow

-95.78% ± 7.19%

BASE

VARIANT

SCREEN VIEWS

Page 27: A/B Testing in Mobile

SCREEN FREEZE

Experiments.ios_table_refactor

BASE VARIANT

Page 28: A/B Testing in Mobile

FRAMEWORKS

HTTPS://GITHUB.COM/KEEPSAFE/SWITCHBOARD HTTPS://GITHUB.COM/MATTT/SKYLAB

Page 29: A/B Testing in Mobile

COMPANIES

Page 30: A/B Testing in Mobile

Q&A

Page 31: A/B Testing in Mobile

THANK YOUMAURICIO T ZAQUIA@mtzaquia