26
A/B Testing for Lean Startups Pete Mauro @mauronic Mucker Labs, February 27, 2012 Everything you ever wanted to know about split testing. Secrets from the experts revealed Watch Presentation Learn More

A/B Testing for Lean Startups

Embed Size (px)

DESCRIPTION

Primer on A/B testing for your startup.Presented to Mucker Labs, February 27, 2012

Citation preview

Page 1: A/B Testing for Lean Startups

A/B Testing for Lean Startups

Pete Mauro @mauronicMucker Labs, February 27, 2012

Everything you ever wanted to know about split testing.

Secrets from the experts revealed

Watch Presentation Learn More

Page 2: A/B Testing for Lean Startups
Page 3: A/B Testing for Lean Startups

In 2009 Tropicana wanted a new juice box

Page 4: A/B Testing for Lean Startups

So they redesigned the package

New box launched in thousands of stores nationwide

Page 5: A/B Testing for Lean Startups

How did it go?

• “Ugly, stupid, cheap and generic looking”

• Ad agency defends it• Sales plunge 20%• Agency CEO fired• Total cost: $33 Million

Page 6: A/B Testing for Lean Startups

8 Months +$33 Million Dollars =

This Package Sucks

Page 7: A/B Testing for Lean Startups

Very un-lean, dude

Page 8: A/B Testing for Lean Startups

Lean Startups are Data Driven

Page 9: A/B Testing for Lean Startups

Lean Startups Listen to their Customers

Page 10: A/B Testing for Lean Startups

Lean Startups are Scientific

Page 11: A/B Testing for Lean Startups

The Scientific Method

Ask Question

Do research

Hypothesis

Analyze Data

Conclusion

Page 12: A/B Testing for Lean Startups

Believe it or not, some people leave Facebook

Page 13: A/B Testing for Lean Startups

Let’s pretend we work at FacebookAsk Question How can we stop more people from deactivating their

Facebook account?

Do research The reason why people use Facebook is because their friends are on Facebook.

Hypothesis If you remind people about their friends who are being left behind, we will lower the deactivation rate.

Page 14: A/B Testing for Lean Startups

Facebook’s Actual A/B TestControl

Experiment

Source: Julie Zhou, Facebook

Page 15: A/B Testing for Lean Startups

The guilt trip workedControl

Experiment

Reduced Deactivation Rate by 7%✔

Source: Julie Zhou, Facebook

Page 16: A/B Testing for Lean Startups

Quiz: Guess the Winner

A B

Source: http://www.abtests.com/

Page 17: A/B Testing for Lean Startups

Quiz: Guess the Winner

A B

✔80% improvement on sign ups

Source: http://www.abtests.com/

5Points

Page 18: A/B Testing for Lean Startups

Quiz: Guess the WinnerA

B

Source: http://www.abtests.com/

Page 19: A/B Testing for Lean Startups

Quiz: Guess the WinnerA

10Points

B

+60% signups

Source: http://www.abtests.com/

Page 20: A/B Testing for Lean Startups

Designing a Test

• Anyone at the company should contribute• Ideas can come from user testing• Use best practices• Be creative, try crazy ideas• Use common sense: don’t piss off your users

for the sake of metrics

Page 21: A/B Testing for Lean Startups

The 41 Shades of Blue Test at Google

Link Color

Page 22: A/B Testing for Lean Startups

The Local Maxima

Source: Joshua Porter

Page 23: A/B Testing for Lean Startups

A/B Testing Tools

• Kissmetrics• Visual Website Optimizer• Optimizely• Unbounce• Mailchimp (email campaigns)• Google Website Optimizer• And more…

Page 24: A/B Testing for Lean Startups

Mailchimp Split Test

Page 25: A/B Testing for Lean Startups

Further Reading

• Metrics Driven Design (See Joshua Porter on Slideshare)

• The New Science of Product Development (See James Birchler on Slideshare)

• Quora: “What are the most unexpected things people have learned from A/B tests?”

• Abtests.com & whichtestwon.com

Page 26: A/B Testing for Lean Startups

Thanks!

Follow me @mauronic

I’m on Twitter @mauronic