Unleash the power of A/B testing - Sitecore summit

Unleash the power of AB- testing

Digital marketing Summit 2016

The Optimisation

Wheel

This is where most start“The most innovative

company in our industry”

“Amazing creativity is the core of everything

we do”

What’s missing?

We are data driven - Horay!

Add experiments! Done or?

What’s missing?

8

9

10

This way or that

A / B testing direct visitors at random to different versions of a page or a

component.

The experiment is performed on your regular site on your real visitors and they

are not aware that they are in an experiment.

11

The classic A/B -test

A B

12

We have a winner!

You measure the outcome against the experiment goals (for example sales) for the different variations.

When the test has reached statistical significance a winner can be

declared.

13

Överfralla eller Underfralla?

Are you a scientist or an accountant?

14

“Where the rubber hits the road”

A/B Testning is the point when our

optimization wheel hits the cruel reality.

Two things happen - both are important

Forward movement

Gravel spray

What to expectYou will see one of these three outcomes:

17

1. Winning 2.Losing 3.Indecisive

18

# Winning tests

Total # of testsWin rate =

19

What’s a realistic goal?

50%0% 100%

In-house Agencies

19

What’s a bad test?

The only bad test is the one you don’t learn anything from

20

Case -

Lifesum (ShapeUpClub)

The Original

22

Variation 1

23

Test 1

24

Results

25

Engagement

Conversions

Revenue

+96%

+ 12%

+ 20%

+21% -15% +11%

+141%

WTF?

“Don’t be afraid to prioritize”

26

Test 2

27

Results

28

Engagement

Conversions

Revenue

+ 153% (+96%)

+ 51% (+12%)

+ 67% (+ 20%)

+26% + 32% +121% +188%

Key takeaways

- The only failed test is the one you don’t learn from

- Best practices don’t always work - Use several data points and

segments

Case - Namnband

Conversions Revenue

A/B-test #1

-6 % -13 %

A/B-test #2

+16 % +15 %

-25 % -48 %

Conversions Revenue

Conversions Revenue

34

35

Case - Spotify Premium

Premium trial for US market High Dropoff rate

Original

You must leave your credit card details in order to start your 30 day free trial

WHY?

37

Input to the hypothesisWe made a short exit survey.

The main reason stated:

38

“I don’t want to leave my credit card details for something that is free”!

Hypothesis:Give the user “A reason why”

39

We only use this to verify your account, you won't be charged anything for your trial

We need this just in case you decide to stay Premium after your free month

We need this because our music deals only allow free trials for users that are credit card or PayPal holders

A

B

C

The result

40

Original

“We only use this to verify…”

Blame the record companies

“In case you decide to stay…”

41

I’m waiting for the test results

Or my retirement

Whatever comes first

44

Visual test duration calculator

apps.conversionista.se

Case - Software company

Promo 1

46

Original

47

Variation - By the book

48

The Results?

49

0%

Because visitors don’t make any decision on

this page!

Average E-Commerce Uplift per Template Type

50

Testing sitewide to reduce bounce rate

51

Test 1 - Functional tagline

52

Engagement: + 6%

Original

Variation

Test 2 - Authority badge

53

Engagement: ±0%

Variation

Original

Product page test

54

Case - Swedoffice - Stripped Checkout

55

Checkout test

56

Test # 2

44

Knverteringsgrad +6% Intäkt per besökare + 10%

57

57

Remove navigation

Change up sales

The result?

58

58

0!

Why?

A heatmap in visual website optimizer gave us the answer59

Test #2

60

Conversion rate +6% Revenue per visitor + 10%

The BEST global

speakers

We’ve sold out every

singer year + 600 Attendees

No sponsors on stage

Scandinavia’s #1 Conversion Event

conversionjam.se