Upload
pete-mauro
View
109
Download
0
Tags:
Embed Size (px)
DESCRIPTION
Primer on A/B testing for your startup.Presented to Mucker Labs, February 27, 2012
Citation preview
A/B Testing for Lean Startups
Pete Mauro @mauronicMucker Labs, February 27, 2012
Everything you ever wanted to know about split testing.
Secrets from the experts revealed
Watch Presentation Learn More
In 2009 Tropicana wanted a new juice box
So they redesigned the package
New box launched in thousands of stores nationwide
How did it go?
• “Ugly, stupid, cheap and generic looking”
• Ad agency defends it• Sales plunge 20%• Agency CEO fired• Total cost: $33 Million
8 Months +$33 Million Dollars =
This Package Sucks
Very un-lean, dude
Lean Startups are Data Driven
Lean Startups Listen to their Customers
Lean Startups are Scientific
The Scientific Method
Ask Question
Do research
Hypothesis
Analyze Data
Conclusion
Believe it or not, some people leave Facebook
Let’s pretend we work at FacebookAsk Question How can we stop more people from deactivating their
Facebook account?
Do research The reason why people use Facebook is because their friends are on Facebook.
Hypothesis If you remind people about their friends who are being left behind, we will lower the deactivation rate.
Facebook’s Actual A/B TestControl
Experiment
Source: Julie Zhou, Facebook
The guilt trip workedControl
Experiment
Reduced Deactivation Rate by 7%✔
Source: Julie Zhou, Facebook
Quiz: Guess the Winner
A B
Source: http://www.abtests.com/
Quiz: Guess the Winner
A B
✔80% improvement on sign ups
Source: http://www.abtests.com/
5Points
Quiz: Guess the WinnerA
B
Source: http://www.abtests.com/
Quiz: Guess the WinnerA
10Points
B
+60% signups
Source: http://www.abtests.com/
✔
Designing a Test
• Anyone at the company should contribute• Ideas can come from user testing• Use best practices• Be creative, try crazy ideas• Use common sense: don’t piss off your users
for the sake of metrics
The 41 Shades of Blue Test at Google
Link Color
The Local Maxima
Source: Joshua Porter
A/B Testing Tools
• Kissmetrics• Visual Website Optimizer• Optimizely• Unbounce• Mailchimp (email campaigns)• Google Website Optimizer• And more…
Mailchimp Split Test
Further Reading
• Metrics Driven Design (See Joshua Porter on Slideshare)
• The New Science of Product Development (See James Birchler on Slideshare)
• Quora: “What are the most unexpected things people have learned from A/B tests?”
• Abtests.com & whichtestwon.com
Thanks!
Follow me @mauronic
I’m on Twitter @mauronic