Who, how much, and how often? The difficult questions answered around your A/B testing strategy

The Past, Present and Future of Website OptimisationIn association with Qubit

Linus Gregoriadis Research Director@LinusGreg

| The Past, Present and Future of Website Optimisation| 6

Introduction and Methodology

The Business Case

Foundations and Resourcing

Moving Towards Personalisation

The Business Case

Aims of the report● To find out how companies are approaching the issue of

testing and optimisation.

● To identify the barriers to optimisation.− Technological− Cultural

● To understand the processes and approaches used.

● To highlight any potential future challenges.

The Business Case

Photo credit: 123dan321 on stock.xchng

Why test anyway?

Testing is key to improving website performance

Source: Conversion Rate Optimisation Report 2014

● Companies seeing significant increases in sales typically run more A/B and multivariate tests

Testing is key to improving website performance

“If it is going to fail, fail fast…

“We see a positive benefit from around a third of tests.”

Chris Howard, Head of Digital, Shop Direct

Case Study: Topshop

● Wanted to improve US experience

● Tested using size formats more familiar to North American shoppers

● Measured not on conversion but revenue optimisation

● Increased revenue by 4.5% with 99% statistical significance

Photo credit: 123dan321 on stock.xchng

What to test?

What to aim for?

“The onus is on optimising the funnel and we can’t do it all at once. We’ve started having grown-up conversations about what is the best point to focus optimisation on.”

Daniel Sale, Head of Digital Marketing, Investec Private Bank

Advanced Website Optimisation is yet to get over the adoption curve…

The Business Case

How much to test?

“We’ve learned what degrees of success you can have. You can chip away and only move the dial so far unless you are doing something significantly wrong.”

Paul Wishman, Group Ecommerce Director, LV

Size matters

“Realistically we can run three to four tests a month simultaneously… we never get the volume to the extent of retail banks for example. We need statistically significant samples and that takes time.”

Daniel Sale, Head of Digital Marketing, Investec Private Bank

“Apart from choosing the right tools, most of the obstacles were related to internal culture and getting buy-in from internal stakeholders.”

David Paice, Group Ecommerce and Admissions Director, Merlin Entertainments

“Testing has been an internal communications piece and especially because A/B testing involves so many different teams.”

Rachel Waller, Senior Digital Marketing and Social Manager, Farfetch

Decisions on technology

• Bought-in software, either off the shelf or customised?

• Built in-house? • Managed services

(when needs are constantly changing)?

Can your vendor grow with you?

The Business Case

“Personalisation is the most exciting challenge and the hardest. There’s so much we can do but it’s finding out what you should and what will add value.”

Rachel Waller, Senior Digital Marketing and Social Manager, Farfetch

Case Study: Farfetch

● Farfetch hypothesised that people convert more when they understand the business model

● Segment of users: new visitors with over 10 page views in their session

● A message layer slides in, prompting users to read FAQs

● Result: 17% increase in conversion

93% of companies saw an uplift in conversion rates

since implementing personalisation.

Ensure incremental gain

“Testing by revenue is critical to our business. Conversion tells only part of the story.”

Simon Pritchard, Group Digital Director, Arcadia

One to One?

“It’s the nature of fashion for consumers largely to behave like the herd in many ways. Perhaps one-to-one conversion isn’t necessary. If you can identify groups of people, that’s good enough.”

Stuart McMillan, Deputy Head of Ecommerce, Schuh

Case Study: DFS ● Hypothesised: European

customers may need more information when shopping on UK websites

● Targeted users from outside the UK with a layer informing them that DFS shipped to their country

● Result: 23% uplift in conversions for segment

Over time the majority will adopt best practice

Final Takeaways

● Testing has seen a surge of growth but barriers still remain.

● People elements are often as significant as technological ones.

● A pragmatic yet continual approach is most likely to pay future dividends.

PEOPLE, PROCESS, TECHNOLOGYNAOSHI YAMAUCHI, CPO@Yamalytics

Brooks BellThe premier firm focused exclusively on

enterprise-level A/B testing, optimization and personalization services.

SOME OF OUR CLIENTS

TALENT, CULTURE, PROCESS ISSUES CAUSING

‘UTILIZATION GAP’ Optimization tools increasing in sophistication

• Tracking across platforms• Personalization• Mobile• More data!

Capabilities

TALENT, CULTURE, PROCESS ISSUES CAUSING

‘UTILIZATION GAP’ Optimization tools increasing in sophistication

• Tracking across platforms• Personalization• Mobile• More data!

Insufficient Team & Talent• Business savvy analysts• Technical talent• Data savvy executive influencers

Broken Process• Lack of standards• Minimal documentation

Resistant Culture• Gut-driven decision

making• Minimal training & sharing

UTILIZATION GAP

Capabilities

DRIVING ISSUES

Utilization ranks 20th out of 24 total categories in Brooks Bell Strategic AssessmentAssessment

Who, how much, and how often? The difficult questions answered around your A/B testing strategy

Presentations & Public Speaking

A Difficult Past: How the Americas Changed

How difficult can this be?

Mathematics Student Questionnaire2019 Operational Mathematics Student Grade 4 VH260313 . 1. How easy or difficult was this test? AExtremely difficult BQuite difficult CSomewhat difficult

How To Prosper In A Difficult Economy

How to Handle Difficult Customers

Difficult Employees Think of a difficult employee….characteristics, how was it handled?

How to handle difficult people

A difficult past how the americas change

6 th Grade Semester Exam 8Junko answered 80% of her test questions correctly. She answered 32 questions correctly. How many questions were

HOW TO DEAL WITH DIFFICULT PEOPLE What is a difficult person? Who is a difficult person?

How to Deal With Difficult People_sample chapter

Dealing with Difficult - coloradoschoolnurse.orgcoloradoschoolnurse.org/assets/dealing-with-difficult-people...Dealing with Difficult People ... (Problems or outcomes?) 2. How would

How to handle difficult interview questions

How to Deal With Difficult Past Bosnia

052619 How Did Jesus Deal With Difficult People?” Review of …hisplace.com › notes › 052619 - How Did Jesus Deal With Difficult Pe… · all called upon to deal with difficult

How to handle difficult situation in restaurant

How to Deal With Difficult Behaviors

How to Handle difficult and defiant behavior

HOW TO DEAL WITH DIFFICULT PEOPLE

Through the Lens of Learning: How Experiencing Difficult ...cs301/fa14/resources/Brookfield_Through... · Brookfield, Stephen, "Through the Lens of Learning: How Experiencing Difficult