ShrimpTest: A/B Testing for WordPress

ShrimpTestA/B Testing with WordPress

mitcho (Michael 芳貴 Erlewine)

Bocoup Loft, Boston, July 20, 2010

Today

• Introduction• A/B testing basics• ShrimpTest

Today

• Introduction• A/B testing basics• ShrimpTest

Introduction

Introduction

• Hi, I’m mitcho.

Introduction

• Hi, I’m mitcho.• Linguist, coder, teacher.

Introduction


• http://mitcho.com; @themitcho

http://mitcho.com

http://mitcho.com

http://twitter.com/mitchoyoshitaka


Introduction


• http://mitcho.com; @themitcho• Linguistics PhD student at MIT

http://mitcho.com

http://mitcho.com



Introduction


• http://mitcho.com; @themitcho• Linguistics PhD student at MIT• Automattic, Mozilla

http://mitcho.com

http://mitcho.com



Yet Another Related Posts Plugin (YARPP) YARP

P!

• smart “related posts” listings with lots of customization options


P!


• mitcho.com/code/yarpp or search for “YARPP”; @yarpp


P!

http://mitcho.com/code/yarpp


http://twitter.com/yarpp



• mitcho.com/code/yarpp or search for “YARPP”; @yarpp

• just under 600k downloads


P!





HookPress

HookPress

• opens up the WordPress hook system to remote scripts

HookPress


• extend WordPress with other languages

HookPress


• extend WordPress with other languages• now implemented on WordPress.com!

HookPress


• extend WordPress with other languages• now implemented on WordPress.com!• mitcho.com/code/hookpress; @hookpress





Set your sites on WordPress 3.0!


MIT WebPub


MIT WebPubTomorrow at 12 noon


MIT WebPubTomorrow at 12 noonN42 Demo Center, MIT

Today

• Introduction• A/B testing basics• Testing on the web?• Theory• Practice

• ShrimpTest

Testing on the web?

Testing on the web?

• Websites aren’t just for passive communication.

Testing on the web?


• Every website has a purpose.

Testing on the web?


• Every website has a purpose.• Common actions: register, sign up, log in, pay, donate, read more...

Testing on the web?


• Every website has a purpose.• Common actions: register, sign up, log in, pay, donate, read more...

• tinyurl.com/everywebsite

http://tinyurl.com/everywebsite

http://tinyurl.com/everywebsite

Example:

Example:• highrisehq.com

http://highrisehq.com


Example:• highrisehq.com• Action: sign up



Example:• highrisehq.com• Action: sign up• Tried many different button texts: “Free Trial”, “Sign-up for Free Trial”, etc.



Example:• highrisehq.com• Action: sign up• Tried many different button texts: “Free Trial”, “Sign-up for Free Trial”, etc.

• “See Plans and Pricing” increased sign-ups by 200%

thinkvitamin.com/business/how-to-increase-sign-ups-by-200-percent/



http://thinkvitamin.com/business/how-to-increase-sign-ups-by-200-percent/

http://thinkvitamin.com/business/how-to-increase-sign-ups-by-200-percent/

Another example:

Another example:

Another example:

+30% +27%

+7%+15%

37signals.com/svn/posts/1525-writing-decisions-headline-tests-on-the-highrise-signup-page

http://37signals.com/svn/posts/1525-writing-decisions-headline-tests-on-the-highrise-signup-page

http://37signals.com/svn/posts/1525-writing-decisions-headline-tests-on-the-highrise-signup-page

Others:

Others:

• Huffington Post titles

Others:

• Huffington Post titles

• Most everything at Amazon, Google

Why test?

Why test?

• We know that {design, content, layout, titles, copy, interface, load time} matters.

Why test?


• How do you know you’re making the right decision? What if you disagree?

Why test?


• How do you know you’re making the right decision? What if you disagree?

• Get the “hard data.”

The theory of A/B testing


• Every visitor is different.


• Every visitor is different.• But when aggregated, a random subset can be a representative sample.



• So we can do a randomized experiment.



• So we can do a randomized experiment.• Randomly assign each user to a different treatment and keep track of a metric.



• So we can do a randomized experiment.• Randomly assign each user to a different treatment and keep track of a metric.

• How can we interpret the data?

Is this coin a fair coin?

TTTTHHHHTHTTHHTTHTHHHHHTTHHHHTHTTTTHHHHTTHHTTHTHTHHHHHTTTHHHHHHTHTHHHHTHHTTTTHHHHHTHTHHTTTTTHTHHHTHTTTHHTTHHHHHTTHTHHHHHTTTHHHTHTHHHHHHHTTTHTTHHHHTHHHTTTHHHHHHHHHHTTHTHHHHHTTHHHTHTHHHHTHHHHHTHHTHHHHHHHTHTHTHHHHHHHHHTHHHTHTHHTHHTHHHHHTHHHHTHTHTHHHHTTTHHHTHHHHHHHHHHTTTHHHHTTHTTHHHHTHTHHHHHTHTHHTHHHHTHTTTHTTTTHHTHHHTTHTHTTHHTHTHHHHHTHHHHHHHHTHTTHTTHTTHHHHTTHTHTHHHHTTHTHTHTHTHTTHHTHTTHTTTHHHHHHHHHHTHTHTHHTHTHHTHTHHHHHTHHHHHHHTHTHTTHTHHTHHHHTHHHTTTTHTTHHTHHHHTTHHHHTHTHTTHTHTHHHHHTHTTTHTHHTHTHHTTHTHTH


N = 500, P(H) = 0.632


N = 250, P(H) = 0.568


N = 10, P(H) = 0.5

Not a fair coin:

P(H) = 0.6P(T) = 0.4

HHHHTHHHTTHTHHHTHTTHTHTTTTTHHTTTTHHTTHHHHHHHTHTHHHHHTTHHHHHHHTTHHHHTTHTTHHTHTHHTTHHTHHHTHHTTHHTTTHHHTHTHHTHHHTTTHTHHHHHHHHHHHHTTHTHHTTHHHTTHHTTHHHHTTHHTHHHHHHHHTTTHHTTTTHHHTHHHTHHTTTHHHTTHTHTHTHHTHHTHHHHHTTTHHTHHTHHTTHHHTHHTTHTTTTHTHTTHHTHHHTHTHTHTTTTTHHTHTTHHTTHHTHHTTHHTHHHTHTHHHHHHTTHTHTTHHHHHHHTHHHHTHTTHTHHHHHHHHTHTHTTTHHHHTHTHTTHHHHTHTHTHTTHTHHHHHHHHTTHTHTHHTTTTHHHHHHTTTTTHTHHTHTHTHTHHHTHHHTHHHHHTTTTTHTHHHTHHHHTTTHTHHTTTTHTTHTTTTHHTTTHTHTTHHTTHTHHTHHTTHHTHTHTHHHHHHHHTTTHHHTTTHTHTHHTTTHTHTTHHHHTHHHTHHHTHTHHHTHHHTHHTTHHHTHHTHHHHTHHTHHTHTTTTTHHHHTTHHHTHTTHHHHTHHTHHTHTHTHHTHHTTTHTTHHHTTHHTTHTHTTHTHTHHTHHHTTTHHTHHTHTHHHTHTTHTHHHHTHHTTTHTHHHHTHHTHHTTHHTHTTHTTHHTHTHTHHHHHTHHHHTHHHHHHTHTHHHHHHHHTHTTTTHTTHHHTHHHTHHTTHTTHTTHHHTHHHHHTHHHHTHHTHHTHHHHTTHHHHHHTTTHHHHHHHHTTHHTTHTTHHTTHTHHHTHTTTHHHTHHHHTTTHTHHHHHHHHHTTHHHTHHTHHHTHHTHTHHHTHHHTHHTTHTHHTTTTHHTHHHHTTTHHTHHTHHTTTTHHTTHTTHHTHHHHHHTHHTTHHHTHHTHTHTHTHHHHTTHTHHTTHHHTHTHHHHHTHTHHHHHTTTHHHTHHTHHHTHTHTHHHHHHTHHHTHTHHHHHTHTHHHHTHHHHHTHHHHTHTHHTTHHTHHHHTHHTTHHHHTHHTTHTHTTTHHHHHHHHHTHHHHTHHHTTHHHTTTHHHHTTHTHHHHTHHHTTHTTTHTTTTHTHTHTTHTHHTHTHHTTHHHHTTTHTHTHHHHTTHHTHTTTHTHTHHHTTHHHHHHHHHTTTHHTTHHHTHTHTTHHHTTHHHHHHHHHHTTHTHHHTTTHHTHHHTHTHHHTHTHTTHTHHTHTHHTHTTTTTHHTTTTHHTHHHHHHHTTHTTHHHHHTTTHTHHTTHTHHHHTHHTHTHTTHTHTTHHHTTTHTTTHHTTHHHHTTTHTHTTTTHHHTHTHHHTHHTHHHHHTHHHHHHHHTTTHHHTTTHHHTHHTTHTHHTHHHHHHTHTHTHTHHHTTHTHTHTHHTTTHHHTHHTHHTHHTTHTHTTHHHTHHHTHHTHTHTHHHTHTTTTTTHTTHHHHHTTTTTHTTHTHHTHTHTTHHHTHTHHTTTTTTHHHTTTHTTHHHHTHHHTHTHTHHHHHHTHHTTTHHHHHTHHHHTTTHTHHTHHHTHHTHTHHTHHTHHHHHHHTHHHHHTTTTTHHHHHTTHHHHHHHHTHHHHHHHTTTTTHHTHHHHTHHTHHTHTHHHTHHTHTHTHHTTHHTTHTHHHHTTTHTHTTHTHTHHTHHHTHHTHHHTHHHHHHHTHTHHHHHTTHTHHHHTHHTTTHHTTHHHHTTTTHHHTHHHTHTHTTTTHTHHHHTHHHHHTHHHHHHHHTHHTHHHHTHHTHTHTTTHTHHTTHHTHTTHHHHTHHHHHHHHTHHHHTHHHTTHHHTHHHHTTTTHHHHTHHHHHHHTTTHHTHHTTHHTTTTHHTHHHHHHTHTHTTHHTHHHHTTTHTHHTHTTHHHTHTHTTTTTHHTHHHHTHHHHHHHHHHHHHHHHHTTHHHHTHHHTTTTHTHHHHHHHHTHTHHTTTTHTTHTHHTHHHHHHHTHTHTHHHTTTTHHTHHHHHHTHTTHTHHTHTHHHHHHHTTHHHHTHTHTHTTTHTTHHHTHHHHHHTTHHTHTTHTTTHHHHHTTTHHTHHHTTHHHHHTHHHHHTTHTHTHHTHHHHTTHHTTTHTHHHTHHHHTTTTHTHHHHHHHTHTHHHHHHHHTTHTHHHHTHHHHTHHHHHTTHTTHHHHTTHTTHHHHTHHTHHHTTTHTHTTHTTTTHTTHTHHHHTTTHTTHHTTTTTHTHTTHTHTHTTHHTHTTHHHTHTTHTHHHHTHHTHHHHHHHHTTTHTHHTHTHTHHHHTHTTTHHHTHHHHHTHTTTHTTHHTHTTHHHHHTTTTTHHHHHTTTTHTHHHHTTHHHHHHHHHTHTHHHTTTTHTHTHTHHTHHTTTHHTHHTHHTHHTTHHHHHTHHHHTHTHHHHHTHTTHTHHHHTHHHTHHTHHTHHHTHHTHTTHHHHHTTTHHTHTHTHTHHTTHTHTTHHTTTHTHTHTHTTHHTHTHTHHHTHHHHHHHTTHHHHTHHTTHTHHTHTHTHHHHHHHHTTHHTHHHHTHHHTTHHHHHTTHTHHHHHHHHHTHTHHTTTHHTHHHHHTHTHHTHTHHHTTHHHHTTTHHTTHHTTTHTTTTTTTTHHHHTTHTHHHHHHTTHHHTHHHHTHTTHHHHHHTHHTTHHTTTHHHHTHHTTHTHTHHHHTTHTTTHHHTHTHHTHHHTTHTHHHTTHHTHHHHTTHHTHTTTTHHTTHTTTTTHHHHHHHHHHHTTTTHTTTHTHHTHHHHHHHHHHHTHHHTTHHHHTHTTHHHTTHHHTHHHTTTHTHHTHHHTTTTHTTHHHHTTHTTTHTHHHHHTHTHTHHTTTTTHTTHHTTHHTTTHHTHTHHTTTHHHHHHHHTTTTTTTHTHHHHTTTHHTHHHHTHHHTHHHTHHTHHHHTHHHHHHTHTTHHHHTTHHTHTTHHHTHTHTTHHTHHHTTTHHTTTHHHTHHHHTHHHTHTHTHHTHTTTTTTHTHTTHHTHHHHHHHHHHHHHTTHHTTHTHTTTTTTHHHHHTTHHHHHTTHTHHHHHTTTHHTHTHHTHHHTHTHHTHTHTHHHTHHHHHHHHHHHTHHHHHHHTTHHHHTTHHHHTHHHTHHHTTHTTTHHHHHHHHHTTTHHHTTHTHHTHTHTTHTTTHHHTTHHHHHHHHHHTTHHHTTHTTTHTHHHTTTHHTHHTTTHHHHTTHHTHTTTTHTTHHHHHHTHTTHHHHHTHHTTHHHTTHHHHHTTTHTTHHTTHTTHTTHHHHHHTTTHHHTTTHTHHTHHHHHTHTTHHHHHHHHTTHHTHHTTHHHHTTHHHTHHHHHHHTTTHHTHHHHTHHHHHHHHHTHHTTHHTHTTTHHHTTHTTTHHHHHHHTTTHHHHTHHHHHHHTTTTTHHHTTHTTTHHHHTHTTTHTHHHTHTHHHTTTTHHTTHHTTTHHTHHTHHHTHTHTHHHHTHHTHHHTHHHTTTHHHTHHTHTHHTTHHTTHTTHHHTTTHHHHHHHHHHHTTHHTTHHHHHHHHHHTHTTHHHHHHTTHHTTHTTHTTHHTTTHTTHTHTHHHHTHHHTHHTHHTTHTTHTHHTTTHTHHTHTHHHHHTHTTHHTTHTHHHHHHTHTHHTHTHHTTHHHHHTHTHHTHTHHTTHTTTHHTHHHTTTTTTTTHHHHTHHTHTHHHHTHTHHHHTHHTHTTTHHTTHHTTTHHHHTTHHHHHHTHTTHTHTTHHTHHHTTHHTTTTTHHHHHTTTTTHHHTTHHTHTTHTHTTHHTTHHHTHHHHHHHTTHTTHHHHHHTTHHHHTHTHHTTHHHTHHHHTHHTHTTHHHTTTHTHTHTTHHHHHTHHHHHTTHTHHTTTHTTHHHHHHHTHTHHHHHTTTTTTHHHTTTTTHHTTTHHHHTHHTTHTHTHHTTHHHHTHHHTHHHHHHTHHHHHHHTTTHHHHHTTHHTHHTTTTTHHTHTTTTHHHTHTTHHTTTHTHHHTHHTHTHHTTHHTHTTTHHHTHHHHHHHHHHHHTHTTTHHTTHHHTHTTTHHTHHTTHHHHHHTHTTHTHHHTHTTTTHHHTHTHHHHHHHHTHHHTHHHHTTHHHHTTTTHTTHTHTHHHHHTHHHHHHTHTTHHTHTHHHHTHHTHHTHHHTHTHTHTTHTHHHHTHHTHHHTHHTHHTTHTHHTHHHHHTHHTTHHTTHTTTHTTHHHTTTTHTHHHHHHHHTHTTHHHTHTHTHHTHTTTHHHHHHTHTTTHHHTHHHHHTTHTHHHHHHTTTHTTHTHTHHTHTTTHHTTHTHHTTHTTHHTHHTHHHHTHTHHHTHHHHHTTHTTTHHTHHHHTTHHHTTTHHHTTTTHTHTHTHHTTTTHTHTHHHHTHHTHHTHHTTHHTTHHHTHHTHHHTTTTHHTHHHHTTHHHHHTHTTHHHHTTHHTHHTHHTHHHHHHTHHHTTHHHHHTTHTTHHTHHHTTTTTHTTTHTTTHTTHTHTTTTHHTHHHHTHHHHHTTHTTHTHTHTTHHHHTTTHHHHHHHHHTHTHHHTTHHTHTHTTHHHTHTTHHHTTHHTTHHHHTHTHHHHHHTHTHTHHTTHTTTHHTHTHTHHHTTTTTHHHHHTHTHTTTTHTTHHHTHTTTTHTTHTTHHTHTTHHHHHTTTHHHHHHHHHTTHTTHTTHHTHTTHTTTTTTHTTTHHTHTTHTTHHHHHHTHHHHHHHHHTHHTHHTHHHTHHHHHTHHTHTHTHHTHHHTHTHHTHTHHTTHTHTHTTHTHTTTHHTHHHTHTHHHHHTHHHTHHHTHTTHHHHHHTHHHHTHHHTTHHTHHTTHTTHTHTTHHTTHHHTTHHHHHHHTHTHHHTHHTTHHHHHTHTHHHHTHHTTHTTTHHTHHHHHTTTHHHHHHHHTTHTTTTTTHHHTHTTHHHTTTHTTHHHHHHHTTHTHTHHTHHTHTHHTTTHHHTTHHHHTHHHHHHHHHTHHHTTTHTTHTHHTHHHTHTHHHTTHHHHHHHHHHHTHTHTT

HHTHHTHTHHTTTTTHTHHHHHHHTTHHHHHTTTTTTTHHTTHTTHHTHHHTHHHHHHTHHTTTHHTHTHHHTHHHHHHHHHTTTTHTHHTTHHHHTHTHHHHHHHHTTHHHHTHTT

But maybe we should run N = 5000 ?

Statistics to the rescue!


• The question: could this actually be a fair coin, and the variance is due to chance?


• The question: could this actually be a fair coin, and the variance is due to chance?• Or: could this variant be just as “effective” as my control, and the variance is due to chance?


• The question: could this actually be a fair coin, and the variance is due to chance?• Or: could this variant be just as “effective” as my control, and the variance is due to chance?• We can calculate this probability: p.



• Normally, if p < 0.05, we can be confident. If p < 0.01, we can be very confident.



• We want to be 95% sure that the different results we see are due to a real difference in “effectiveness”.



• We want to be 95% sure that the different results we see are due to a real difference in “effectiveness”.

• This 95% confidence interval is standard.

Can we say this is an unfair coin?


N = 10, P(H) = 0.5, p = 1 unclear!


N = 10, P(H) = 0.5, p = 1

N = 250, P(H) = 0.568, p ≈ 0.03

unclear!

confident!


N = 10, P(H) = 0.5, p = 1

N = 250, P(H) = 0.568, p ≈ 0.03

N = 500, P(H) = 0.632, p < 0.01

unclear!

confident!

very confident!

Best practices

CC-BY-NC flickr.com/photos/kylemacdonald/3199283481/

http://www.flickr.com/photos/kylemacdonald/3199283481/


Best practices

• Decide on a clear, measurable, long-term metric ahead of time.




Best practices

• Decide on a clear, measurable, long-term metric ahead of time.• Clear metric = actionable results.




Best practices


• Experiment early and often.




Best practices


• Experiment early and often.• Test bold changes.




Best practices


• Experiment early and often.• Test bold changes.• Keep the potential costs in mind.




Best practices: how you test


• Assignment of visitors to variations must be truly random.



• Each visitor must have a consistent experience across visits.




• The site must still work, both during and after the experiment!




• The site must still work, both during and after the experiment!

• Beware of interactions. Or go multivariate.

A/B testing on the web


• Get clear, actionable results with hard data.


• Get clear, actionable results with hard data.• Take the guesswork out of design.


• Get clear, actionable results with hard data.• Take the guesswork out of design.• Increase your bottom line with little cost.


• Get clear, actionable results with hard data.• Take the guesswork out of design.• Increase your bottom line with little cost.• You can’t afford not to!


• Get clear, actionable results with hard data.• Take the guesswork out of design.• Increase your bottom line with little cost.• You can’t afford not to!

• Unfortunately, if it’s not drop dead easy and convenient, it won’t happen...

ShrimpTest

CC-BY flickr.com/photos/gastev/2630047705/ CC-BY-SA flickr.com/photos/stuart_spivack/315481947/

A B

http://www.flickr.com/photos/gastev/2630047705/

http://www.flickr.com/photos/gastev/2630047705/

http://www.flickr.com/photos/stuart_spivack/315481947/

http://www.flickr.com/photos/stuart_spivack/315481947/

ShrimpTestA/B Testing for WordPress

ShrimpTest*

* free beer and pizza if you guess why it’s so named

A/B Testing for WordPress

ShrimpTest

• A/B testing right in WordPress:

ShrimpTest

• A/B testing right in WordPress:• More convenient and easier to use

ShrimpTest

• A/B testing right in WordPress:• More convenient and easier to use• More robust and extensible

ShrimpTest


• A project of Automattic, written by mitcho

ShrimpTest


• A project of Automattic, written by mitcho• To be released as a free plugin.

ShrimpTest


• A project of Automattic, written by mitcho• To be released as a free plugin.• Developed in the open. Dev blog:

ShrimpTest


• A project of Automattic, written by mitcho• To be released as a free plugin.• Developed in the open. Dev blog:

• shrimptest.wordpress.com

ShrimpTest

http://shrimptest.wordpress.com


Demo

Custom variant selection and metrics


$variant = shrimptest_get_variant( $my_experiment_id );// $variant is going to be an integer.switch ( $variant ) { case true: // if $variant > 0 echo "This is variant #{$variant}."; break; default: // if $variant is 0, or if ShrimpTest is down. echo "This is the control.";}


$variant = shrimptest_get_variant( $my_experiment_id );// $variant is going to be an integer.switch ( $variant ) { case true: // if $variant > 0 echo "This is variant #{$variant}."; break; default: // if $variant is 0, or if ShrimpTest is down. echo "This is the control.";}

// the user did something worth xshrimptest_update_metric( $my_experiment_id, $x );

Features

✓ Written as a WordPress plugin

Features

✓ Written as a WordPress plugin✓ Server-side (PHP) selection

Features

✓ Written as a WordPress plugin✓ Server-side (PHP) selection✓ Tools for easy setup of experiments: [ab] shortcode, variant viewer

Features


✓ API for custom metrics, variant types

Features


✓ API for custom metrics, variant types✓ “eat my own dogfood” approach

Features


✓ API for custom metrics, variant types✓ “eat my own dogfood” approach

✓ Works with caching plugins

Features

A work in progress...

• Documentation


• Documentation• UI improvements


• Documentation• UI improvements• Experiment automation tools


• Documentation• UI improvements• Experiment automation tools• More testing, particularly on higher traffic sites


Get involved!

• Try it out!

Get involved!

• Try it out!• tinyurl.com/shrimptesttrunk

Get involved!

http://tinyurl.com/shrimptesttrunk



• Check out and submit bug tickets:

Get involved!




• Check out and submit bug tickets:• tinyurl.com/shrimptesttrac

Get involved!





• Follow development:

Get involved!





• Follow development:• shrimptest.wordpress.com

Get involved!





• Follow development:• shrimptest.wordpress.com• @ShrimpTest

Get involved!



ShrimpTest

Thank you! Questions?mitcho (Michael 芳貴 Erlewine), Automatticmitcho.com, @themitchoshrimptest.wordpress.com, @ShrimpTest

A/B Testing with WordPress

http://mitcho.com

http://mitcho.com



Technology

ShrimpTest: A/B Testing for WordPress