Quick Tips for Better Testing

Jen BolandAnalytics and Optimization Lead

Sara HoffmanWeb Development Manager

Brian RogelAssociate Director of SEO and SEM

About us

#Bridge16

Avon Breast Cancer Crusade

Beaconfire REDBeaconfire RED

CONFIDENTIAL 3

Agenda• Creating a plan• Test strategy• Test setup• Things to do while your are testing• How to evaluate results• What to do after testing

Create a Plan

Just because someone else did it doesn’t mean it will work for you.

CONFIDENTIAL 5

Define Website Pain Points1) Determine goals

2) List problem areas

3) Rank list based on costs and benefits

#Bridge16

Review Metrics of the Proposed Test

Key Metrics: • Unique Visits• Pageviews• Conversion Rate• Average Gift

#Bridge16

Form Analytics & Error Messaging

#Bridge16

#Bridge16

FPO

Usability Testing

User Testing Tools

#Bridge16

Test Strategy

Determine Metrics that can be Tested

#Bridge16

TOP-FUNNEL (Landing Pages)

MID-FUNNEL(Transition Pages)

BOTTOM-FUNNEL(Conversion Forms)

Use

r’s Jo

urne

yDecision Tips:• Top-funnel tests run faster

than bottom-funnel tests• Testing conversions is not

always possible • Make sure the target

sample size can be reached in the proposed test duration

Determining Sample Size

Sample Size Tips:• Use sample size

calculators• Run tests until the

sample size is reached, not until statistical significance is reached

#Bridge16

A minimum detectable effect (MDE) is the smallest true effect that has a ‘good chance’ of being found to be statistically significant.

Test Across Thank You Pages & EmailsC: Honor GiftB: Sustainer

w/ PremiumA:Sustainer

Test Duration: MinimumMinimum Test Duration Tips:• Run tests for full week cycles• One week minimum is possible for websites

with exceptionally high traffic• A two week minimum is preferred for most

websites

#Bridge16

Test Duration: MaximumMaximum Test Duration Tips:• Clearing cookies can cause sample pollution

o Lifehacker Poll: 72.35% of people clear cookies or use private browsing

o Microsoft Report: studies show 30% to 40% of people clear cookies monthly

• Avoid running tests longer than four weeks

#Bridge16

Test Duration and Skewed Results

#Bridge16

Avoid running tests during holidays or seasonal anomalies(Ex: End-of-year and non-end-of-year traffic acts very differently)

Test Duration and Sample Size Recap

#Bridge16

• Determine target sample size • Run tests for full week cycles• 2 to 4 week test durations are best• Run tests until the target sample size is reached,

not until statistical significance is reached• Be aware of holidays and seasonal anomalies

#Bridge16

Where are you going?Make a map

January February March April May June JulyHomepage User Testing

Homepage Hero Slider A/B Homepage Static Hero

Build Hero Ask String High-LowSurvey - Legal

Survey

Rebuild overlay

Pre Year-End Overlay POC

possible extension

Action TY Story TitleSuggested Amount

inahat.co/bridge-roadmap

CONFIDENTIAL 25

TestVariation Name

Test Length Traffic % # of Visitors VWO Type Test Location

ETA for Results

Hero Homepage Donation Ask No Test 7 60% 587 ABCDCMS Homepage 1/12/2015

Hero Homepage Donation Ask Control 7 13% 1401 ABCDCMS Homepage 1/12/2015

Hero Homepage Donation Ask Var1 7 13% 1458 ABCDCMS Homepage 1/12/2015

Hero Homepage Donation Ask Var2 7 13% 1464 ABCDCMS Homepage 1/12/2015

Grey Overlay Homepage Real Control 7 50% 883 Split testCMS Homepage 1/20/2015

Grey Overlay Homepage Grey Overlay 7 50% 857 Split testCMS Homepage 1/20/2015

Hamburger Control 14 50% 1828 ABC

Homepage and Directory 6/8/2015

Hamburger Button 14 25% 920 ABC

Homepage and Directory 6/8/2015

Hamburger Left Logo 14 25% 943 ABC

Homepage and Directory 6/8/2015

Create a HypothesisCONFIDENTIAL 2

6

Hypothesis Creation

#Bridge16

Sections of a Hypothesis:1) Statement of Problem2) Proposed Solution3) Expected Results

Hypothesis Template

#Bridge16

Based on <data discovered> we believe <website issue> is causing <undesirable result>. We believe <change to be tested> will help resolve <website issue> and cause <desired result>.

Writing your Hypothesis

#Bridge16

Example Hypothesis:Based on a 31% below average conversion rate for paid search traffic we believe a lack of information on donation forms is causing a high drop-off rate. We believe sending people to an information rich splash page before the donation form will help resolve the high drop-off rate and cause an increase in conversion rate.

Paid Search Landing Page Test

#Bridge16

+47%Conversion rate

Less form fields

More information

Test Setup

CONFIDENTIAL 31

Testing Tools

#Bridge16

Test Setup • Preserve Query

Parameters• Link with Analytics

Tools

#Bridge16

https://testsite.org?Donation2?df_id=12362&12362.donation=form1&utm_source=newsletter&utm_medium=email&utm_content=link1&utm_campaign=fall

Beware of Peeping Toms

• Browser Tools like WAPPALYZER, dataSlayer, Pessimezely or WASP can expose your test and variant names!

#Bridge16

Segmentation

#Bridge16

Test Large to Small

#Bridge16

Test BroadlyTest VariationControl

#Bridge16

Don’t Work Late or on Weekends

#Bridge16

While You’re Testing…

Monitor Results

#Bridge16

Monitor Results

#Bridge16

Micro Conversions

#Bridge16

Micro Conversions

#Bridge16

Control

Variant

+9.7%Website sign-in clicks+4.9%Credit card sign-in clicks

+21%Registration clicks

Avoid Other Website Changes

#Bridge16

Results

#Bridge16

Tools to Help with Testing – Handout!

Calculators!

Tools to Help with Testing – Handout!Your Good Friend – Excel!Percent Change (Current Value – Previous Value) / Previous Value

#Bridge16

Unanticipated Results

#Bridge16

P-Value and Statistical Power

#Bridge16

“If you use p=0.05 to suggest that you have made a discovery, you will be wrong at least 30% of the time.” - David Colquhoun

Analyzing Data: What’s a Win?

#Bridge16

Checklist: Low P-Value High Power Test Reaches Target

Sample Size Test is Not an Industry

Outlier

What Tools Tell You

#Bridge16

Two-Tail T-Test

After Your Test

Validate – Iterate!

inahat.co/bridge-validate

Hypothesis

Create / Iterate

Continuous Testing

Measure Results

#Bridge16

#Bridge16

Modify Your MapJanuary February March April May June JulyHomepage User Testing

Homepage Hero Slider A/B Homepage Static Hero

Build Hero Ask String High-Low Paid Search Ask

DRTV Ask

Survey - Legal

Survey

Rebuild overlay

Pre Year-End Overlay POC

possible extension

Paid Search Overlay

Action TY Story TitleSuggested Amount

inahat.co/bridge-roadmap

Sharing Lessons Across Channels

#Bridge16

Using Google AdWords to test how effective different copy variations are at improving click-through-rates.

Control Variants

+25.2%CTR Improvement

-6.9%CTR Reduction

Sharing Lessons Across Channels

#Bridge16

The difference in “Don’t miss” and “Don’t miss out on” made a 34.4% difference in click-through-rates. The test was significant at 99% with 99.93% statistical power.

Variant+34.4%CTR Improvement

Control

Cross-Channel Implementations

#Bridge16

Copy tests help determine the best messaging for your audience. Improvements discovered can be leveraged in many channels:

Meta Content (Organic)

Ad Copy (Paid)

• Email• Direct Mail• DRTV

• Social Media• Organic Meta Content• Display Ad Copy

I’ve failed over and over and that is why I succeed.

CONFIDENTIAL 60

Michael Jordan

Questions?

Download our Free Testing Guide:http://inahat.co/testing-handout

Don’t forget to visit the Solutions Showcase!

Many of the ideas discussed today are on display at the

Solutions Showcase!