A/B Testing with Website Optimizer
Brennan Heyde
Technical Director – Professional Services
3/10/2012
Today’s Overview
• What is A/B Testing?• Why You Should be A/B Testing• Real World Examples• Introduction to Google Website Optimizer• Walk though A Real Test Using Miva
Merchant• Other Tools / Resources / Q.A.
What is A/B Testing?
• Also called split testing• Complicated: Testing a baseline control sample
with a variety of single-variable tests to improve conversion rates.
• Easy: Testing two different versions of a page to see which one converts better.
What is A/B Testing?
• 3 Parts to Any A/B test• 1. Original Page (A)• 2. Variation Page (B)• 3. Conversion Page
What is A/B Testing
• Other Types of Testing:• Multivariate Testing – Testing
multiple changes on the SAME page
Examples:• Button Colors • Headlines• Placement of Calls to Action
Why you should be A/B testing
• Data Driven Decisions• Allows you to take personal preference (yours,
your developers, your designers) out of the equation and help you make a data driven decision.
• It will Increase your conversion rate!
Examples – WhichTestWon.com
A
Examples
B
A
B
A
B
What Should You Test?
• Different Product Page Layouts• Checkout Variations• Alternate Payment Methods• New Features• Promotions
Create a plan before you begin.
Scottevest.com • Inside Zoom Vs Outside Zoom
Scottevest.com • Inside Zoom Vs Outside Zoom
Scottevest.com • Category Quick View
Scottevest.com • Runway Videos – 32% Improvement
Scottevest.com • Paypal vs No Paypal
What is Google Website Optimizer?
• Free Tool from Google• Allows you to easily
perform A/B tests on your website.
• Track Results and make data driven decisions
• Increase your conversion rate
• Use your same Google Account Login
Our Test Experiment
• Question: What add to basket button color and text converts best for my customers?Original: Grey – Add to Basket
Test Cases1. Green – Add to Cart2. Red – Add to Cart3. Greem – Add to Basket4. Green – Add to Basket
Website Optimizer Walk-Thru
Website Optimizer Walk-Thru
A/B Checklist
How to Create Page Variations in Miva Merchant?
• Trick: Any parameters passed in the URL automatically become global variables in Miva.
• Ex: www.mystore.com/mm5/merchant.mvc?button=1• I can now use &mvt:global:button; OR
<mvt:if expr=“g.button EQ 1”> <img src=“/images/button1.jpg” /><mvt:elseif expr=“g.button EQ 2">
<img src=“/images/button2.jpg” /></mvt:if>
How to create page variations in Miva?
• This allows us to use these different URLs in Website Optimizer
Orginal: http://www.mystore.com/product/product_code.html
Variations:http://www.mystore.com/product/product_code.html?button=1
http://www.mystore.com/product/product_code.html?button=2
http://www.mystore.com/product/product_code.html?button=3
http://www.mystore.com/product/product_code.html?button=4
Edit PROD
Then Product Display Layout
Add the conditional statement where your add to cart button appears
Conversion URL: http://store20.coolcommerce.net/store/merchant.mvc?Screen=INVC
Adding the JavaScript Code
Tracking Code – Step 1
Original Page – Control Script
Head Tag Conditional Statement
<mvt:if expr="g.Screen EQ 'PROD' AND g.Product_Code EQ '3583' AND ISNULL(g.button)">// control script goes here
<mvt:elseif expr="NOT ISNULL(g.button)"> // Variation tracking code goes here
<mvt:elseif expr="g.Screen EQ 'INVC'">// Conversion tracking code goes here
</mvt:if>
Validate Your Pages, Start The Test!
View Results
Additional Notes
• You can run multiple tests at the same time• GWO will give you improvement on conversion
rate. However, you can use Google Analytics custom variables to also track:• Average Order Value• Number of Transactions• Per Visit Revenue
Additional Resources / Questions
• http://blog.kissmetrics.com/guide-google-website-optimizer/
• http://www.google.com/intl/en/websiteoptimizer/tutorials.html
• http://www.google.com/intl/en/websiteoptimizer/articles.html