Conversion 2017 - Benjamin Groenewoud (Travix International)

Persoonlijke overtuiging Wat betekent CRO naar EBG:

IPV alleen ideeën op roadmap valideren Experimenten moet ‘innovaties’ drijven en het exploreren dmv

experimenten bepaalt strategische richting ROAR model

Wie is de tweede in golf?

Wie is de tweede in de Formule 1?

The winner takes it all

The winner takes it all

“De nummer 1 wordt 52% vaker gekozen dan de nummer 2”

“89% van de consumenten ruilt een aanbieder in voor een ander wanneer de service tegenvalt”

Hoi!

Benjamin Groenewoud

Product Owner CRO @ Travix

Eerder: SNS Bank, ClickValue, Mirabeau

Hoi!

Benjamin Groenewoud

Product Owner CRO @ Travix

Eerder: SNS Bank, ClickValue, Mirabeau

Conversion Rate Optimization

Evidence Based Growth

CRO @ Travix Lessons Learned

Emerce Conversion April 13, 2017

Benjamin Groenewoud Product Owner CRO @ Travix

Onze merken

About

35 countries

550+ employees

3.5M+ passengers a year

50+ planes filled every day

Travix is a global online travel agency (OTA) operating from offices in the

Netherlands, Germany, India, Singapore, Hong Kong, the UK and the USA.

With our 5 brands and an extensive portfolio of 35 websites we are market leader

in several countries and growing rapidly on all continents.

Our footprint

Our market

Wide range of airlines Price Transparency Deep range of ancillary services Website Usability & Scalability High Quality Customer Service

OTA META AIRLINE TRAVEL AGENT

✓

✓

✓

✓

✓

✓

✓

✓

✓

✓

✓ ✓

SERVICES

1. CRO at Travix so far

2. The future of CRO: EBG

3. Q & A

How to guide you through our lesson learned

CRO at Travix so far

Lesson #1 CRO is not a person

August 2015 Our redesign

More and better testing

Our former process Lineair from step to step

1. Conduct research

2. Hypothesize 3. Create treatment

4. Run experiment

5. Analyze results

6. Run follow-up experiment

Me, myself and I

Begging for some CRO fellows A-Team Approach

Multidisciplinary CRO Team

Is there value? Proof value + collect learnings Experiment regret

Earn Earn more / try to learn Earn, learn, & turn (innovate business)

CRO Specialist Specialist team / in-house + outsourced Embedded in product teams / in-house

Just try something Structured earn and learn driven team Structured earn and learn driven company

Optimizely NH site / Google Optimize Automated server site testing framework

Preaching CRO Share learnings frequently across Travix Workshops and knowledge sessions

Steering

Earn vs. Learn

Capacity

Process

Technology

Knowledge sharing

MacGyver Approach

One man show - jack of all trades

What I wanted

A multi-disciplinary team to do:

● More analysis

● More experiments

● Better experiments

● Evidence Everything

“So you want to pull in an entire team for CRO?

Don’t think that’s gonna happen...”

Campaigning...

Begging...

Waiting...

Praying...

Gaze placement

1. Use mouse- or eye-tracking tools to find what

content draws attention (fovea’s focus)

2. Analyze the path of the fovea (the ‘order’)

3. Test which chunks of content are the most

persuasive

4. Place that content ‘within the gaze-path’, and in

the right order

✓ “We only notice what’s in our gaze”

Or this 99 Persuasion Techniques (1 - 34)

Needs & Motivation > Autonomy > Self-efficacy > Response-efficacy > Need for Relatedness > Need for meaning > Outcome expectancies > Social rewarding > Fudge factor > Present focus bias > Ambiguity aversion > Reward substitution

Attention & Perception > Visual cuing > Gaze cuing > Gaze distraction > Gaze placement > Attentional bias > Focussing effect > Von Restorff effect > Unnoticable differences > Confirmation bias > Visual fluency

Memory & Learning > Flashbulbing > Availability heuristic > Self-generation effect > Choice-supportive bias > Recognition over recall > Affective forecasting > Chunking > Operant conditioning > Procedural fluency > Peak-end rule > Direct Priming > Associative Priming > Cue dependent forgetting


Self control > Implementation Intention > Personal writing > Trigger words > Audaciousness > Headline Formula > Engaging Images

Persuasive marketing > Equivalence Framing > Emphasis Framing > Mental Accounting > Bait-and-Switch > Defaults > Front-Loading > Loss Aversion > Storytelling > Serial Positioning > Anchoring & Adjustments

> Hyperbolic Discounting > Immediacy Effect > Price Partitioning > Reference Pricing > Free Effect > Payment Consolidation > Bundled Pricing > Limited Time


Social & external influences > Comparison targeting > Positional targeting > Time targeting > Domestic Country Bias > Country of Origin Bias > Weather Effects > Groupthink > Normative social influence > Self-fulfilling prophecy > Injunctive norms > Descriptive norms

> Reference Group > Informational Influence > Social Facilitation > Mimicry > Vicarious Modeling > Upward Comparison > Downward Comparison

System 2 > Low Balling > Sparking Trigger > The law of distraction > Facilitating Trigger > The law of least effort > Reminders > Ego Depletion > Awareness Alerts > Base Rate Neglect > One-choice paradox > Choice paradox > Post-Decision Rationalization > Hindsight Bias > Focusing Effect


System 1 > Foot in the door > Decoy effect > Fear appeals > Ambiguity aversion > Self-generation effect > Endowment effect > Affect heuristic > Rhyme as a reason > HALO effect

‘System 2’

‘System 1’

Can system 1 be tricked?

Can system 1 be tricked?

A psychologist?!

Lesson #2 The proof is in the pudding

Showing examples makes more sense Hotjar click and scroll maps

10% reaches this line

A

Random experiments “Let’s try Trustpilot on our homepage”


B

WINNER

It required quite some patience

It required quite some patience

But finally it took off...

From McGyver to… A-Team Approach








Steering

Earn vs. Learn

Capacity

Process

Technology

Knowledge sharing

MacGyver Approach


From McGyver to… a real team! A-Team Approach








Steering

Earn vs. Learn

Capacity

Process

Technology

Knowledge sharing

MacGyver Approach


From 1 chair

Dreaming of my own party

To 9 chairs

Joost Fromberg Customer Behaviour

Specialist

Daan Bouwmeester Web Analyst

Nicolas Biglie Visual Designer

Gerard Stroomer UX Designer

Benjamin Groenewoud Product Owner

Pavel Fokin Front-end Developer

Aliaksandr Zanouski Front-end Developer

Jorrin Geurts UX Conversion Designer

Tom van den Berg Web Analyst

Please meet the CRO Party “Keep calm and crunch some numbers”

Run experiments to improve conversion / learn user behaviour

Support Product Owners in validating new features

The CRO Team works on two streams

Squeeze optimalisatie vs. innovatie Projecten worden bepaald van bovenaf (top-down)

Bij hogere maturity kan dit cyclus worden waarbij directors op basis van exploratie dmv experimenten de strategie fine-tunen (EBG)

Lesson #3 CRO should be about intelligence

Directe waarde vs. Lange termijn waarde Resultaat vs. Ethiek

Voorbeeld BIG

EBG: both value and intelligence

Adding Direct Value Short-term goal

Customer Intelligence Long-term goal

It’s all about the money, money, money

It’s all about the money, money, money No that’s not funny

- Placeholder -

Belang van mobile Groei mobile voor Travix brands

Krijgt minder aandacht vanwege minder ‘directe’ biz value

In the end EBG is a religion You have believe it’s the way to go

ACT CRO as a transaction machine

Mapping van thema’s op fase van journey Voorstudie + >30 experimenten leveren cruciale inzichten over wat klanten drijft

Homepage

Search Results

Checkout

A



B

WINNER

Our former process Lineair from step to step

1. Conduct research

2. Hypothesize 3. Create treatment

4. Run experiment

5. Analyze results

6. Run follow-up experiment

ACT CRO as a transaction machine

AC & ACT From transactional to customer intelligence

Starting point September 2017: Pre-study

Starting point September 2017: Pre-study

1. Scientific & market research

2. Qualitative research

● Customer Service data

● Online customer feedback

● Usability tests

1. Quantitative research

● Google Analytics

● Heatmaps / session recordings

● Previous experiments

Conclusion Booking a flight feels too difficult and too ‘tricky’

Price dispersion

of the market

Complex

products

Constant state of

‘high perceived risk’

“Industry is playing

tricky”

Klanten zijn extreem prijsgedreven

● Long time on page

● High exit rate

● Lot’s of users clicking

back to homepage to

refine their search

F FIND

Findings Search Results Google Analytics

Conclusions for search results

1. Users look quite ‘system 2’ (rational)

2. There is a lot of information

and there are a lot of tools

(filters, tabs, etc.)

1. This costs a lot of mental energy

(which system 2 doesn’t have)

A ANALYZE

Analyze the data From data to insights

When putting more focus on ‘Change your flight’

→ users will not be distracted by clicking back to the homepage

→ which will lead to more use of the Change button and a higher CR.

A ANALYZE

Analyze how to optimize Coming up with the hypothesis

A ANALYZE

Analyze how to optimize Coming up with the hypothesis

A

C CREATE

Create Create a design which challenges the hypothesis

B

C CREATE


B

WINNER for XL screens

C CREATE


INCONCLUSIVE

AC & ACT From transactional to customer intelligence

FACT & ACT From transactional to customer intelligence to transformational

1.Growth

1.Customer intelligence

1.A better decision making process

This will result in

The future of CRO: EBG Transformation

Conversion Rate Optimization

Evidence Based Growth

From A-Team to... A-Team Approach








Steering

Earn vs. Learn

Capacity

Process

Technology

Knowledge sharing

MacGyver Approach


From A-Team to... Army! A-Team Approach


Army Approach

Evidence Based Everything







Steering

Earn vs. Learn

Capacity

Process

Technology

Knowledge sharing

MacGyver Approach


Strategy

Board Members Business Owners

Product Development binnen Travix

Landing Pages & Mkt. Automation

Product Owner Front-end Devs Back-end Devs

Search & Shopping


Checkout & X-sell


My Account & OPS features


Design Team

Product Owner / UX Designer UX Designer

Visual Designer

CRO Team

Psycholoog Webanalisten (2)

Front-end Devs (3)

Product Owner UX Designer

Visual Designer

THINK

BUILD

MEASURE &

OPTIMIZE

DESIGN

MVP

MVP

MVP

Strategy

Board Members Business Owners

De toekomst

Landing Pages & Mkt. Automation


Search & Shopping


Checkout & X-sell


My Account & OPS features


Design Team

Product Owner / UX Designer UX Designer

Visual Designer

Customer Intelligence Team

Psycholoog Webanalisten

Product Owner UX Designer

Visual Designer

FACTS &

ANALYZE

BUILD

MEASURE &

OPTIMIZE

CREATE &

TEST

MVP

MVP

MVP

Product Development binnen Travix

Landing Pages & Mkt. Automation ● Product

Owner ● Front-end

Devs ● Back-end

Devs

Search & Shopping ● Product

Owner ● Front-end

Devs ● Back-end

Devs

Checkout & X-sell ● Product

Owner ● Front-end

Devs ● Back-end

Devs

My Account & OPS features ● Product

Owner ● Front-end

Devs ● Back-end

Devs

This feature will

deliver us €1

extra per

booking

A warning about

correct name details

will lower the

number of calls

We need to

accelerate

personalisation

MVP

MVP

MVP

MVP

MVP

MVP

MVP

MVP

MVP

MVP

MVP

MVP

Product Development binnen Travix Landing Pages & Mkt. Automation ● Product Owner ● Front-end Devs ● Back-end Devs

Search & Shopping ● Product Owner ● Front-end Devs ● Back-end Devs

Checkout & X-sell ● Product Owner ● Front-end Devs ● Back-end Devs

My Account & OPS features ● Product Owner ● Front-end Devs ● Back-end Devs

Design Team ● Product Owner / Sr. UX Designer ● UX Designer ● Visual Designer

CRO Team ● Product Owner ● UX Designer ● Visual Designer

● Psycholoog ● Webanalisten (2) ● Front-end Devs (3)

Innovation (roadmap)









Enablers









Optimization Enablers

Product Development binnen Travix Landing Pages & Mkt. Automation ● Product Owner ● Webanalist ● Designer ● Front-end Devs ● Back-end Devs

Search & Shopping ● Product Owner ● Webanalist ● Designer ● Front-end Devs ● Back-end Devs

Checkout & X-sell ● Product Owner ● Webanalist ● Designer ● Front-end Devs ● Back-end Devs

My Account & OPS features ● Product Owner ● Webanalist ● Designer ● Front-end Devs ● Back-end Devs







Innovation From RAT to MVP

Optimization From minimal to awesome

Intelligence From insights to strategy






Innovation From RAT to MVP

Optimization From minimal to awesome

Intelligence From insights to strategy

● Hoe blijven we competitief? ● Creëren we hiermee een effect? ● Hoe bouwen we een succesvolle pilot?

● Hoe ziet klantgedrag eruit per segment? ● Waar liggen verbeterkansen? ● Hoe maken we dit onderdeel van een 7 een 9,5?

● Wat drijft onze klanten? ● Wat houdt ze tegen om bij ons te boeken? ● Welke tools hebben de teams nodig om hier

achter te komen?






"Throw out your roadmap, and trust in the

process and your people. Go where the

tests take you. That may end up in a

different place than you envisioned but it

is where your customers are taking you."

Stuart Frisby - Director of Design,

Booking.com






From CRO to EBG From validation to exploring the strategy

CRO at Travix (status quo)

● ‘The business’ defines a roadmap based on

ideas and/or (some) data

● The ideas will be transformed into an MVP

● The MVP will be validated by the CRO Team

EBG at Travix (future)

● There is no roadmap

● Strategy is driven ‘bottom up’ solely by analysis

and experiments

● The outcomes are driving new hypothesis to

validate

● Hypothesis are tested fast and cheap (RAT)

before investing in an MVP

Lesson #4 Don’t sit on an island

Don’t keep sitting in the ivory tower Try to involve all colleagues in your data-driven process

Moving from party to army requires:

● Belief in the way of working

● The right skills in each team

● Standardized process

● One backlog

● One way of calculating prio and value

● The right KPIs

Don’t keep sitting in the ivory tower Try to involve all colleagues in your data-driven process

Moving from party to army requires:

● Belief in the way of working

● The right skills in each team

● Standardized process

● One backlog

● One way of calculating prio and value

● The right KPIs

Lesson #5 Speed over Quality

Increasing velocity will be our main focus

>1200 A/B tests per year Always 800 tests live 7.000 A/B tests per year

99% chance you’re in a test 502 A/B tests per year 100.000 A/B tests per year

Development is purely focussed on quality But the last 20% takes 80% of the time

The MVP is dead From MVP to RAT (Riskiest Assumption Test)

Every step still takes a lot of time Automation is key to make EBG scalable

Optimize testing bandwidth How can you run as many experiments as possible?

Bandit algorithm

When a message seems to work

better, the system will send more

traffic to the winning variant. Or a

message that performs worse will get

almost no traffic anymore. This

makes we’re earning more money

during the experiment.

How they work

WINNER

Automated experiment framework

Automatically...

● Creating experiments when deploying a new feature (server-side)

● Automatically calculate how long experiment needs to run

● Statistical checks / monitoring

● Automated analysis

● Rolling out winner to all visitors automatically

● Use unused bandwidth to run e.g. bandit tests

Pre-analysis Clusteranalyse: welke segmenten zijn het belangrijkst? Welke segmenten presteren boven- en benedengemiddeld? Creation Remains creative... Deployment

Create optimal test planning automatically (using Potential and Impact) Creating experiments when deploying a new feature (server-side) Integrated bandit system

No A/B-tests? → Launch bandit experiments automatically (Smart Notifications) → ‘RAT system for exploring assumptions / make money

Automatically calculate how long experiment needs to run Statistical checks / monitoring Post analysis

Automated analysis Rolling out winner to all visitors automatically

Optimization using ad-hoc testing

Level 1 Level 2

Level 3 Level 4

Level 5

Continuous optimization with clear goals & plan

Evidence Based Growth Team is driving our

roadmaps

Evidence Based Growth is the

way of working across Travix

Evidence Based Growth is in the DNA of ProdDev

Now

From Product Teams to

‘Product Result Teams’ CRO Specialist

Big data specialists, data

mining, customer journey

analysts

Dedicated and multi-

disciplinary CRO Team

Level 3

Knowing what segment is

driven by which determinants

AI experts

Level 1

CRO Basics

Level 4

Business Intelligence

Data Mining

Level 2

Advanced knowledge of user

behaviour and persuasion

Level 5

Artificial Intelligence

& Machine Learning

Experiment on profiles,

steer on NPS and CES

(3-6 experiments / week)

Testing random ideas

(0,5 experiments / week)

Algorithm testing, cross-

channel / -device

(8-10 experiments / week)

Testing strategies

based on data

(2 experiments / week)

Steer on CLV, evidence-

based everything

(15+ experiments / week)

Optimized process with

high velocity, involving

more colleagues

No process defined / ad-hoc

Super-optimized process,

used by all teams with same

workflow and definitions

Clear process with closed

feedback-loop

Front-end + Flight content

Focus on balance quality /

quantity

Only front-end experiments

Injecting javascript

Company roadmap is based

and validated on data-driven

analysis and learnings

Own server-side

testing platform

Only front-end experiments

Focus on max. quality

Predictive analytics tools,

recommendation engines,

etc.

People

Knowledge

Testing strategy and frequency

Process

Technology

Manager Product

Development VP Marketing CCO + CTO

All board members +

directors Entire company

To achieve in 2017

Sponsor

1. Involve all product teams in EBG

2. Start using RAT before MVP

3. Prioritise quantity over quality

4. Set experiment KPI’s (e.g. regret, reporting)

5. Start building a server-side experimentation platform

From CRO to EBG Key focus areas 2017 / 2018

Afsluiter - Grappige

notificatie o.i.d.

The End Thanks for your attention!

Any questions? Email me at [email protected]

Or find me on LinkedIn:

https://www.linkedin.com/in/benjamingroenewoud

mailto:[email protected]

mailto:[email protected]

https://www.linkedin.com/in/benjamingroenewoud

Data & Analytics

Conversion 2017 - Benjamin Groenewoud (Travix International)