Embed Size (px)
Citation preview
AB Testing: Getting Started
Jayme Suravicz
15. October 2015
Agenda• Introduction
• Team Setup
• Understanding your traffic
• Things to consider when AB testing
• Establishing a reporting system
• Key Takeaways
#OPTEX2015
3
4
5
6
7
Why do AB testing?
Test Hypotheses
Learn more about user behavior
Keep our site “fresh”
Measure the impact of changes
Drive Conversion & Engagement
#OPTEX20158
Agenda• Introduction
• Team Setup
• Understanding your traffic
• Things to consider when AB testing
• Establishing a reporting system
• Key Takeaways
#OPTEX2015
Identify your AB testing team and responsibilities
10
What do you want to test?
Who is the business owner?
How can you share synergies?
Project Manager
TrackingSupport
Tester Tester Tester
Tester Tester Tester
Department A
Department C
Department B
11
Identify your AB testing team and responsibilities
Agenda• Introduction
• Team Setup
• Understanding your traffic
• Things to consider when AB testing
• Establishing a reporting system
• Key Takeaways
#OPTEX2015
Understanding the different traffic sources to your site will help you better address the needs of visitors
Westwing receives visitors from a variety of different traffic sources and assumes these sources would perform differently and therefore require different landing pages (LPs)
Identify your main traffic buckets:Example:
Consider if buckets can be further broken down into subgroups:Examples:
SEM SEO Social Display Affiliate
SEMbrand
SEMnon brand
DisplayPublisher X
Display Publisher Y
SocialFB
SocialTwitter
SocialInstagram
13
Understanding the different traffic sources to your site will help you better analyze your test results
In addition to traffic sources, devices should also be considered as users have different experiences on different devices and therefore require different landing pages (LPs)
Distinguish between your main devices:Example:
As well as your different platforms:Examples:
Desktop Tablet Mobile
Desktop Site Mobile Site Mobile app Tablet app
14
Understanding the different traffic sources to your site will help you better analyze your test results
15
Agenda• Introduction
• Team Setup
• Understanding your traffic
• Things to consider when AB testing
• Establishing a reporting system
• Key Takeaways
#OPTEX2015
1. Consider Primary and Secondary Goals
3 main points to consider when AB testing Landing Pages
17
3 main points to consider when AB testing Landing Pages
18
Email Only Email + Password Name + Email + Password
+37% +145%
+84%
3 main points to consider when AB testing Landing Pages
19
USPs with discount [-‐70%]
USPs without discount
+15% +50%+30%
2. Think Big, Optimize Small
3 main points to consider when AB testing Landing Pages
20
3 main points to consider when AB testing Landing Pages
21
3 main points to consider when AB testing Landing Pages
22
3 main points to consider when AB testing Landing Pages
23
3 main points to consider when AB testing Landing Pages
24
3. Don’t ignore the different user experiences in devices and environments
3 main points to consider when AB testing Landing Pages
25
Agenda• Introduction
• Team Setup
• Understanding your traffic
• Things to consider when AB testing
• Establishing a reporting system
• Key Takeaways
#OPTEX2015
Take advantage of the Optimizely API for reporting
27
Export test performance via the Optimizely API
Build your custom report
Use a consistent namingconvention for all tests:Team / Country / Traffic Group / Subgroup / Device
Organize your projects in Optimizely
Agenda• Introduction
• Team Setup
• Understanding your traffic
• Things to consider when AB testing
• Reporting and Analyzing AB tests
• Key Takeaways
#OPTEX2015
Key Takeaways• Identify your team & establish responsibilities
• Understand your traffic & build your testing strategy from day 1
• Set clear goals, both primary and secondary
• Think big & optimize small
• Invest the time to setup proper reporting
#OPTEX2015
