Quick Tips for Better Testing
Jen BolandAnalytics and Optimization Lead
Sara HoffmanWeb Development Manager
Brian RogelAssociate Director of SEO and SEM
About us
#Bridge16
Avon Breast Cancer Crusade
Beaconfire REDBeaconfire RED
CONFIDENTIAL 3
Agenda• Creating a plan• Test strategy• Test setup• Things to do while your are testing• How to evaluate results• What to do after testing
Create a Plan
Just because someone else did it doesn’t mean it will work for you.
CONFIDENTIAL 5
Define Website Pain Points1) Determine goals
2) List problem areas
3) Rank list based on costs and benefits
#Bridge16
Review Metrics of the Proposed Test
Key Metrics: • Unique Visits• Pageviews• Conversion Rate• Average Gift
#Bridge16
Form Analytics & Error Messaging
#Bridge16
#Bridge16
FPO
Usability Testing
User Testing Tools
#Bridge16
Test Strategy
Determine Metrics that can be Tested
#Bridge16
TOP-FUNNEL (Landing Pages)
MID-FUNNEL(Transition Pages)
BOTTOM-FUNNEL(Conversion Forms)
Use
r’s Jo
urne
yDecision Tips:• Top-funnel tests run faster
than bottom-funnel tests• Testing conversions is not
always possible • Make sure the target
sample size can be reached in the proposed test duration
Determining Sample Size
Sample Size Tips:• Use sample size
calculators• Run tests until the
sample size is reached, not until statistical significance is reached
#Bridge16
A minimum detectable effect (MDE) is the smallest true effect that has a ‘good chance’ of being found to be statistically significant.
Test Across Thank You Pages & EmailsC: Honor GiftB: Sustainer
w/ PremiumA:Sustainer
Test Duration: MinimumMinimum Test Duration Tips:• Run tests for full week cycles• One week minimum is possible for websites
with exceptionally high traffic• A two week minimum is preferred for most
websites
#Bridge16
Test Duration: MaximumMaximum Test Duration Tips:• Clearing cookies can cause sample pollution
o Lifehacker Poll: 72.35% of people clear cookies or use private browsing
o Microsoft Report: studies show 30% to 40% of people clear cookies monthly
• Avoid running tests longer than four weeks
#Bridge16
Test Duration and Skewed Results
#Bridge16
Avoid running tests during holidays or seasonal anomalies(Ex: End-of-year and non-end-of-year traffic acts very differently)
Test Duration and Sample Size Recap
#Bridge16
• Determine target sample size • Run tests for full week cycles• 2 to 4 week test durations are best• Run tests until the target sample size is reached,
not until statistical significance is reached• Be aware of holidays and seasonal anomalies
#Bridge16
Where are you going?Make a map
January February March April May June JulyHomepage User Testing
Homepage Hero Slider A/B Homepage Static Hero
Build Hero Ask String High-LowSurvey - Legal
Survey
Rebuild overlay
Pre Year-End Overlay POC
possible extension
Action TY Story TitleSuggested Amount
inahat.co/bridge-roadmap
CONFIDENTIAL 25
TestVariation Name
Test Length Traffic % # of Visitors VWO Type Test Location
ETA for Results
Hero Homepage Donation Ask No Test 7 60% 587 ABCDCMS Homepage 1/12/2015
Hero Homepage Donation Ask Control 7 13% 1401 ABCDCMS Homepage 1/12/2015
Hero Homepage Donation Ask Var1 7 13% 1458 ABCDCMS Homepage 1/12/2015
Hero Homepage Donation Ask Var2 7 13% 1464 ABCDCMS Homepage 1/12/2015
Grey Overlay Homepage Real Control 7 50% 883 Split testCMS Homepage 1/20/2015
Grey Overlay Homepage Grey Overlay 7 50% 857 Split testCMS Homepage 1/20/2015
Hamburger Control 14 50% 1828 ABC
Homepage and Directory 6/8/2015
Hamburger Button 14 25% 920 ABC
Homepage and Directory 6/8/2015
Hamburger Left Logo 14 25% 943 ABC
Homepage and Directory 6/8/2015
Create a HypothesisCONFIDENTIAL 2
6
Hypothesis Creation
#Bridge16
Sections of a Hypothesis:1) Statement of Problem2) Proposed Solution3) Expected Results
Hypothesis Template
#Bridge16
Based on <data discovered> we believe <website issue> is causing <undesirable result>. We believe <change to be tested> will help resolve <website issue> and cause <desired result>.
Writing your Hypothesis
#Bridge16
Example Hypothesis:Based on a 31% below average conversion rate for paid search traffic we believe a lack of information on donation forms is causing a high drop-off rate. We believe sending people to an information rich splash page before the donation form will help resolve the high drop-off rate and cause an increase in conversion rate.
Paid Search Landing Page Test
#Bridge16
+47%Conversion rate
Less form fields
More information
Test Setup
CONFIDENTIAL 31
Testing Tools
#Bridge16
Test Setup • Preserve Query
Parameters• Link with Analytics
Tools
#Bridge16
https://testsite.org?Donation2?df_id=12362&12362.donation=form1&utm_source=newsletter&utm_medium=email&utm_content=link1&utm_campaign=fall
Beware of Peeping Toms
• Browser Tools like WAPPALYZER, dataSlayer, Pessimezely or WASP can expose your test and variant names!
#Bridge16
Segmentation
#Bridge16
Test Large to Small
#Bridge16
Test BroadlyTest VariationControl
#Bridge16
Don’t Work Late or on Weekends
#Bridge16
While You’re Testing…
Monitor Results
#Bridge16
Monitor Results
#Bridge16
Micro Conversions
#Bridge16
Micro Conversions
#Bridge16
Control
Variant
+9.7%Website sign-in clicks+4.9%Credit card sign-in clicks
+21%Registration clicks
Avoid Other Website Changes
#Bridge16
Results
#Bridge16
Tools to Help with Testing – Handout!
Calculators!
Tools to Help with Testing – Handout!Your Good Friend – Excel!Percent Change (Current Value – Previous Value) / Previous Value
#Bridge16
Unanticipated Results
#Bridge16
P-Value and Statistical Power
#Bridge16
“If you use p=0.05 to suggest that you have made a discovery, you will be wrong at least 30% of the time.” - David Colquhoun
Analyzing Data: What’s a Win?
#Bridge16
Checklist: Low P-Value High Power Test Reaches Target
Sample Size Test is Not an Industry
Outlier
What Tools Tell You
#Bridge16
Two-Tail T-Test
After Your Test
Validate – Iterate!
inahat.co/bridge-validate
Hypothesis
Create / Iterate
Continuous Testing
Measure Results
#Bridge16
#Bridge16
Modify Your MapJanuary February March April May June JulyHomepage User Testing
Homepage Hero Slider A/B Homepage Static Hero
Build Hero Ask String High-Low Paid Search Ask
DRTV Ask
Survey - Legal
Survey
Rebuild overlay
Pre Year-End Overlay POC
possible extension
Paid Search Overlay
Action TY Story TitleSuggested Amount
inahat.co/bridge-roadmap
Sharing Lessons Across Channels
#Bridge16
Using Google AdWords to test how effective different copy variations are at improving click-through-rates.
Control Variants
+25.2%CTR Improvement
-6.9%CTR Reduction
Sharing Lessons Across Channels
#Bridge16
The difference in “Don’t miss” and “Don’t miss out on” made a 34.4% difference in click-through-rates. The test was significant at 99% with 99.93% statistical power.
Variant+34.4%CTR Improvement
Control
Cross-Channel Implementations
#Bridge16
Copy tests help determine the best messaging for your audience. Improvements discovered can be leveraged in many channels:
Meta Content (Organic)
Ad Copy (Paid)
• Email• Direct Mail• DRTV
• Social Media• Organic Meta Content• Display Ad Copy
I’ve failed over and over and that is why I succeed.
CONFIDENTIAL 60
Michael Jordan
Questions?
Download our Free Testing Guide:http://inahat.co/testing-handout
Don’t forget to visit the Solutions Showcase!
Many of the ideas discussed today are on display at the
Solutions Showcase!
Recommended
