Getting it Booking right

Ge#ng it Booking right – Mobile apps

Iaroslav, Khramov

Team Leader & Android developer

Iaroslav Khramov TL & Android developer

About me

Making Booking right apps

ExperimenAng •  Why •  How •  What do you achieve

Developing TesAng Releasing

Booking.com

•  Founded and based in Amsterdam •  Biggest accommodaAon plaKorm in the world •  > 8,000 dedicated employees

Mobile apps: •  iOS & Android •  7 app development teams •  ~35 apps developers

Booking.com apps

Android •  > 10M downloads * •  raAng 4.6 out of 5 *

iOS •  raAng “4+” out of 5 **

* according to Google Play Store on 23/03/2015 ** according to Apple Store on 23/03/2015

Facebook Vkontakte Instagram YouTube Gmail Inbox WhatsApp

4.0 / 5 * 4.2 / 5 * 4.5 / 5 * 4.1 / 5 * 4.3 / 5 * 4.2 / 5 * 4.4 / 5 *

Our challenges

•  Users all around the world -‐  Millions of users -‐  Our user doesn’t have a unique profile -‐  High user expectaAons

•  40+ languages •  > 600k properAes •  expert opinion vs. users opinion

What do we want to achieve?

Product that is: •  User friendly •  TrusKul •  Reliable •  …

What do we want to achieve?

Product that is: •  User friendly •  TrusKul •  Reliable •  …

Provide our users the best experience before, during and after making a

booking

Are you sure?

•  Are you sure that new design is good? •  Are you sure this change will not break the app? •  Are you sure customers will like our new feature? •  Are you sure …. ?

Variant A Lek hand up

Variant B Right hand up

Can you please stand up

Lets play!

Which got more money on Kickstarter?

Smartphone Controlled Paper Airplane

Jot: CapaciAve Touch Stylus

A B

Winner: Version B

A B


Winner: Version B

A B


People with hand up - please take a seat

Which variation performed better?

A B


Winner: Version A

“The truth is, someAmes it’s bener to have no proof than low proof.”


Winner: Version A

“The truth is, someAmes it’s bener to have no proof than low proof.”


Which got more requests? A B

Winner: Version A

“Like most people, if you guessed that Version B won, you are wrong. Version A surprisingly had a 24% increase in submissions without using an image on the page.”

Which got more requests?

Winner: Version A

“Like most people, if you guessed that Version B won, you are wrong. Version A surprisingly had a 24% increase in submissions without using an image on the page.”

Which got more requests?


What is better for conversion? A B

What is better for conversion?

Winner: Version A

In this example removing the social bunons resulted in an increased conversion to the basket of 11.9%.

What is better for conversion?

Winner: Version A

In this example removing the social bunons resulted in an increased conversion to the basket of 11.9%.


More completions?

A B

More completions?

Winner: Version B

“…surely adding an eTrust image would improve form compleAons, it makes everyone feel safe and secure, doesn’t it? The answer is no. Although it’s only a slight rise of 12.6%, Version B, without the image, proved to be more effecAve.”

More completions?

Winner: Version B

“…surely adding an eTrust image would improve form compleAons, it makes everyone feel safe and secure, doesn’t it? The answer is no. Although it’s only a slight rise of 12.6%, Version B, without the image, proved to be more effecAve.”


Which Copy Increased Trial Sign-Ups? A B

Which Copy Increased Trial Sign-Ups?

Winner: Version A

“In this test, Version A increased sign-‐ups by 38% – a big rise. However, your gut feeling might iniAally be that version B is the bener design.”

Which Copy Increased Trial Sign-Ups?

Winner: Version A

“In this test, Version A increased sign-‐ups by 38% – a big rise. However, your gut feeling might iniAally be that version B is the bener design.”


Decision-making using data: A/B testing

•  When should you should run an experiment (use A/B tesAng)? Need to decide which opAon is bener

•  What are the alternaAves? Expert opinion / intuiAon Historical data

Why is experimentation better?

•  Our intuiAon is not very good (people are complex)

•  Historical data

(becomes outdated fast)

What do we get if using A/B testing?

•  Reduced design discussions •  Less dependent on expert opinions •  Make sure all changes are actually (commercial)

improvements •  AddiAonal safety net for new changes in the app

How complex is it?

if ( newFeatureEnabled == true ) { // new feature code

} else { // old reliable solu2on

}

+ add control from server-side

Today: {

Experiment A: ON Experiment B: ON

}

Server Control experiments

At any moment of Ame: {

Experiment A: ON Experiment B: OFF

}

It’s easy! Can I start experimenting now?

Yes, but first: •  Pick a metric •  Find users to include in an experiment

Metrics

Metrics are the main determinant of success!

1.  Pick your own metric! 2.  Diversify metrics!

Metrics

Metrics are the main determinant of success!

1.  Pick your own metric! 2.  Diversify metrics!

Metrics for Booking.com?

What is my metric?

Type of business Possible key metrics

Ecommerce: Booking.com, Amazon

items sold, conversion, loyalty

Content: Euronews, CNN visitors, loyalty, engagement, ad clicks

Social Networking visitors, engagement, ad clicks

Gaming: Angry birds purchases, subscripAons, ad clicks

Come up with new idea to improve your

app Run an

experiment to see which

variant is the most successful

Select the winner based on

metric

Remove non-‐winner from the

code

Experiment lifecycle

Example of an experiment A B

Tools for mobile that will allow you to start experimenting today?

Yes, just Google it! For example: •  Airlock •  Arise.io •  AppAmize •  Leanplum …

A/B testing in apps provides

•  Final changes are real improvements •  All changes are server-‐side controlled and can be

enabled \ disabled at any point of Ame

A/B testing in apps provides

•  Final changes are real improvements •  All changes are server-‐side controlled and can be

enabled\disabled at any point of Ame

Is it enough for our apps?

Booking apps development

•  Use Experiments •  Develop fast •  Be stable •  Release fast •  Monitor how you are doing

Source: XKCD 303 http://imgs.xkcd.com/comics/compiling.png

Fast, iterative development

Android – slow, > 3 min to build on big projects

•  Buck + exopackage •  Booking DCL (internal soluAon) •  others

iOS – usually not a problem

Trunk development

What & how: •  Everything goes into master •  Local branches up to a developer •  Tests keep master stable

Why: •  As a result saves Ame on merging

Making sure project builds and stable

•  Hooks •  CI •  TesAng

Hooks

Master can always build!

git push

Developer

pass

fail

git repository

Hooks:

build & run test

CI

git push git repository

CI system Great job

Developer

pass

fail

Email && jabber developer about the problem

•  all dev build •  special builds

•  TranslaAons •  UI tests

Testing

•  Dedicated tesAng team •  Easy maintainable tests on core funcAonality + UI tests •  Bugbash sessions •  Internal releases •  3rd party tesAng

Release model

Train release model •  Regular •  Pre-‐planned schedule

Release model

Train release model •  Regular •  Pre-‐planned schedule

Nobody waits: you take this train or the next one

Author: Maurits Vink http://commons.wikimedia.org/wiki/File:VIRM6.jpg

Why train model?

•  Everyone know when the next release will take place •  Team can plan the scope to be done •  Complexity of each release more or less the same

App roll out

Rolling out apps not as fast as web: Android: •  ~1 day to all users

iOS : •  Depends on Apple •  Apple review process 1-‐5 days

App roll out

Rolling out apps not as fast as web: Android: •  ~1 day to all users

iOS : •  Depends on Apple •  Apple review process 1-‐5 days

Hot fixes can’t be fast…

( Experiments can help! )

Safe app release

Android: •  staged rollouts •  alpha & beta tesAng iOS: •  Enterprise signing program •  TestFlight – beta tesAng for iOS 8

Staged rollout

Get thing done right for Android users •  Make sure that everything goes according to plan •  NEW! 'Halt' A Staged Rollout if found a problem •  When sure -‐ release to 100%

iOS: •  No support for staged rollouts

Crash & data monitoring

Monitor your crashes, errors and main metrics via: •  CrashlyAcs, •  Bugsense, •  Google AnalyAcs, •  Flurry AnalyAcs, •  ….

Booking apps

ExperimenAng •  Why to try •  How •  What do you achieve

Developing TesAng Releasing

References

1.  Smartphone Controlled Paper Airplanehnps://www.kickstarter.com/projects/393053146/powerup-‐30-‐smartphone-‐controlled-‐paper-‐airplane

2.  Jot: CapaciAve Touch Stylus by Adonit!hnps://www.kickstarter.com/projects/531383637/jot-‐capaciAve-‐touch-‐stylus/descripAon

3.  12 Surprising A/B Test Results to Stop You Making AssumpAonshnp://unbounce.com/a-‐b-‐tesAng/shocking-‐results/

4.  5 Landing Page A/B Tests (And Their Surprising Results) hnp://blog.hubspot.com/markeAng/5-‐landing-‐page-‐a/b-‐tests-‐and-‐their-‐surprising-‐results

5.  Removing Social Sharing Bunons Increases Conversions. Yes, You Heard That Right!hnps://vwo.com/blog/removing-‐social-‐sharing-‐bunons-‐from-‐ecommerce-‐product-‐page-‐increase-‐conversions/

6.  Source: XKCD 303 hnp://imgs.xkcd.com/comics/compiling.png 7.  Foto van twee VIRM VI stellen te Amsterdam Bijlmer ArenA. Author: Maurits Vink

hnp://commons.wikimedia.org/wiki/File:VIRM6.jpg

QuesAons?

yaroslav.wash

Iaroslav, Khramov

Team Leader & Android developer

[email protected]

Software

Getting it Booking right