ShrimpTest: A/B Testing for WordPress

ShrimpTestA/B Testing with WordPress

mitcho (Michael 芳貴 Erlewine)

Bocoup Loft, Boston, July 20, 2010

• Introduction• A/B testing basics• ShrimpTest

Introduction

• Hi, I’m mitcho.

Introduction

• Hi, I’m mitcho.• Linguist, coder, teacher.

Introduction

• http://mitcho.com; @themitcho

Introduction

• http://mitcho.com; @themitcho• Linguistics PhD student at MIT

Introduction

• http://mitcho.com; @themitcho• Linguistics PhD student at MIT• Automattic, Mozilla

Yet Another Related Posts Plugin (YARPP) YARP

• smart “related posts” listings with lots of customization options

• mitcho.com/code/yarpp or search for “YARPP”; @yarpp

• just under 600k downloads

HookPress

• opens up the WordPress hook system to remote scripts

HookPress

• extend WordPress with other languages

HookPress

• extend WordPress with other languages• now implemented on WordPress.com!

HookPress

• extend WordPress with other languages• now implemented on WordPress.com!• mitcho.com/code/hookpress; @hookpress

Set your sites on WordPress 3.0!

MIT WebPub

MIT WebPubTomorrow at 12 noon

MIT WebPubTomorrow at 12 noonN42 Demo Center, MIT

• Introduction• A/B testing basics• Testing on the web?• Theory• Practice

• ShrimpTest

Testing on the web?

• Websites aren’t just for passive communication.

Testing on the web?

• Every website has a purpose.

Testing on the web?

• Every website has a purpose.• Common actions: register, sign up, log in, pay, donate, read more...

Testing on the web?

• Every website has a purpose.• Common actions: register, sign up, log in, pay, donate, read more...

• tinyurl.com/everywebsite

Example:

Example:• highrisehq.com

Example:• highrisehq.com• Action: sign up

Example:• highrisehq.com• Action: sign up• Tried many different button texts: “Free Trial”, “Sign-up for Free Trial”, etc.

• “See Plans and Pricing” increased sign-ups by 200%

thinkvitamin.com/business/how-to-increase-sign-ups-by-200-percent/

Another example:

+30% +27%

+7%+15%

37signals.com/svn/posts/1525-writing-decisions-headline-tests-on-the-highrise-signup-page

Others:

• Huffington Post titles

Others:

• Huffington Post titles

• Most everything at Amazon, Google

Why test?

• We know that {design, content, layout, titles, copy, interface, load time} matters.

Why test?

• How do you know you’re making the right decision? What if you disagree?

Why test?

• How do you know you’re making the right decision? What if you disagree?

• Get the “hard data.”

The theory of A/B testing

• Every visitor is different.

• Every visitor is different.• But when aggregated, a random subset can be a representative sample.

• So we can do a randomized experiment.

• So we can do a randomized experiment.• Randomly assign each user to a different treatment and keep track of a metric.

• How can we interpret the data?

Is this coin a fair coin?

TTTTHHHHTHTTHHTTHTHHHHHTTHHHHTHTTTTHHHHTTHHTTHTHTHHHHHTTTHHHHHHTHTHHHHTHHTTTTHHHHHTHTHHTTTTTHTHHHTHTTTHHTTHHHHHTTHTHHHHHTTTHHHTHTHHHHHHHTTTHTTHHHHTHHHTTTHHHHHHHHHHTTHTHHHHHTTHHHTHTHHHHTHHHHHTHHTHHHHHHHTHTHTHHHHHHHHHTHHHTHTHHTHHTHHHHHTHHHHTHTHTHHHHTTTHHHTHHHHHHHHHHTTTHHHHTTHTTHHHHTHTHHHHHTHTHHTHHHHTHTTTHTTTTHHTHHHTTHTHTTHHTHTHHHHHTHHHHHHHHTHTTHTTHTTHHHHTTHTHTHHHHTTHTHTHTHTHTTHHTHTTHTTTHHHHHHHHHHTHTHTHHTHTHHTHTHHHHHTHHHHHHHTHTHTTHTHHTHHHHTHHHTTTTHTTHHTHHHHTTHHHHTHTHTTHTHTHHHHHTHTTTHTHHTHTHHTTHTHTH

N = 500, P(H) = 0.632

N = 250, P(H) = 0.568

N = 10, P(H) = 0.5

Not a fair coin:

P(H) = 0.6P(T) = 0.4

HHHHTHHHTTHTHHHTHTTHTHTTTTTHHTTTTHHTTHHHHHHHTHTHHHHHTTHHHHHHHTTHHHHTTHTTHHTHTHHTTHHTHHHTHHTTHHTTTHHHTHTHHTHHHTTTHTHHHHHHHHHHHHTTHTHHTTHHHTTHHTTHHHHTTHHTHHHHHHHHTTTHHTTTTHHHTHHHTHHTTTHHHTTHTHTHTHHTHHTHHHHHTTTHHTHHTHHTTHHHTHHTTHTTTTHTHTTHHTHHHTHTHTHTTTTTHHTHTTHHTTHHTHHTTHHTHHHTHTHHHHHHTTHTHTTHHHHHHHTHHHHTHTTHTHHHHHHHHTHTHTTTHHHHTHTHTTHHHHTHTHTHTTHTHHHHHHHHTTHTHTHHTTTTHHHHHHTTTTTHTHHTHTHTHTHHHTHHHTHHHHHTTTTTHTHHHTHHHHTTTHTHHTTTTHTTHTTTTHHTTTHTHTTHHTTHTHHTHHTTHHTHTHTHHHHHHHHTTTHHHTTTHTHTHHTTTHTHTTHHHHTHHHTHHHTHTHHHTHHHTHHTTHHHTHHTHHHHTHHTHHTHTTTTTHHHHTTHHHTHTTHHHHTHHTHHTHTHTHHTHHTTTHTTHHHTTHHTTHTHTTHTHTHHTHHHTTTHHTHHTHTHHHTHTTHTHHHHTHHTTTHTHHHHTHHTHHTTHHTHTTHTTHHTHTHTHHHHHTHHHHTHHHHHHTHTHHHHHHHHTHTTTTHTTHHHTHHHTHHTTHTTHTTHHHTHHHHHTHHHHTHHTHHTHHHHTTHHHHHHTTTHHHHHHHHTTHHTTHTTHHTTHTHHHTHTTTHHHTHHHHTTTHTHHHHHHHHHTTHHHTHHTHHHTHHTHTHHHTHHHTHHTTHTHHTTTTHHTHHHHTTTHHTHHTHHTTTTHHTTHTTHHTHHHHHHTHHTTHHHTHHTHTHTHTHHHHTTHTHHTTHHHTHTHHHHHTHTHHHHHTTTHHHTHHTHHHTHTHTHHHHHHTHHHTHTHHHHHTHTHHHHTHHHHHTHHHHTHTHHTTHHTHHHHTHHTTHHHHTHHTTHTHTTTHHHHHHHHHTHHHHTHHHTTHHHTTTHHHHTTHTHHHHTHHHTTHTTTHTTTTHTHTHTTHTHHTHTHHTTHHHHTTTHTHTHHHHTTHHTHTTTHTHTHHHTTHHHHHHHHHTTTHHTTHHHTHTHTTHHHTTHHHHHHHHHHTTHTHHHTTTHHTHHHTHTHHHTHTHTTHTHHTHTHHTHTTTTTHHTTTTHHTHHHHHHHTTHTTHHHHHTTTHTHHTTHTHHHHTHHTHTHTTHTHTTHHHTTTHTTTHHTTHHHHTTTHTHTTTTHHHTHTHHHTHHTHHHHHTHHHHHHHHTTTHHHTTTHHHTHHTTHTHHTHHHHHHTHTHTHTHHHTTHTHTHTHHTTTHHHTHHTHHTHHTTHTHTTHHHTHHHTHHTHTHTHHHTHTTTTTTHTTHHHHHTTTTTHTTHTHHTHTHTTHHHTHTHHTTTTTTHHHTTTHTTHHHHTHHHTHTHTHHHHHHTHHTTTHHHHHTHHHHTTTHTHHTHHHTHHTHTHHTHHTHHHHHHHTHHHHHTTTTTHHHHHTTHHHHHHHHTHHHHHHHTTTTTHHTHHHHTHHTHHTHTHHHTHHTHTHTHHTTHHTTHTHHHHTTTHTHTTHTHTHHTHHHTHHTHHHTHHHHHHHTHTHHHHHTTHTHHHHTHHTTTHHTTHHHHTTTTHHHTHHHTHTHTTTTHTHHHHTHHHHHTHHHHHHHHTHHTHHHHTHHTHTHTTTHTHHTTHHTHTTHHHHTHHHHHHHHTHHHHTHHHTTHHHTHHHHTTTTHHHHTHHHHHHHTTTHHTHHTTHHTTTTHHTHHHHHHTHTHTTHHTHHHHTTTHTHHTHTTHHHTHTHTTTTTHHTHHHHTHHHHHHHHHHHHHHHHHTTHHHHTHHHTTTTHTHHHHHHHHTHTHHTTTTHTTHTHHTHHHHHHHTHTHTHHHTTTTHHTHHHHHHTHTTHTHHTHTHHHHHHHTTHHHHTHTHTHTTTHTTHHHTHHHHHHTTHHTHTTHTTTHHHHHTTTHHTHHHTTHHHHHTHHHHHTTHTHTHHTHHHHTTHHTTTHTHHHTHHHHTTTTHTHHHHHHHTHTHHHHHHHHTTHTHHHHTHHHHTHHHHHTTHTTHHHHTTHTTHHHHTHHTHHHTTTHTHTTHTTTTHTTHTHHHHTTTHTTHHTTTTTHTHTTHTHTHTTHHTHTTHHHTHTTHTHHHHTHHTHHHHHHHHTTTHTHHTHTHTHHHHTHTTTHHHTHHHHHTHTTTHTTHHTHTTHHHHHTTTTTHHHHHTTTTHTHHHHTTHHHHHHHHHTHTHHHTTTTHTHTHTHHTHHTTTHHTHHTHHTHHTTHHHHHTHHHHTHTHHHHHTHTTHTHHHHTHHHTHHTHHTHHHTHHTHTTHHHHHTTTHHTHTHTHTHHTTHTHTTHHTTTHTHTHTHTTHHTHTHTHHHTHHHHHHHTTHHHHTHHTTHTHHTHTHTHHHHHHHHTTHHTHHHHTHHHTTHHHHHTTHTHHHHHHHHHTHTHHTTTHHTHHHHHTHTHHTHTHHHTTHHHHTTTHHTTHHTTTHTTTTTTTTHHHHTTHTHHHHHHTTHHHTHHHHTHTTHHHHHHTHHTTHHTTTHHHHTHHTTHTHTHHHHTTHTTTHHHTHTHHTHHHTTHTHHHTTHHTHHHHTTHHTHTTTTHHTTHTTTTTHHHHHHHHHHHTTTTHTTTHTHHTHHHHHHHHHHHTHHHTTHHHHTHTTHHHTTHHHTHHHTTTHTHHTHHHTTTTHTTHHHHTTHTTTHTHHHHHTHTHTHHTTTTTHTTHHTTHHTTTHHTHTHHTTTHHHHHHHHTTTTTTTHTHHHHTTTHHTHHHHTHHHTHHHTHHTHHHHTHHHHHHTHTTHHHHTTHHTHTTHHHTHTHTTHHTHHHTTTHHTTTHHHTHHHHTHHHTHTHTHHTHTTTTTTHTHTTHHTHHHHHHHHHHHHHTTHHTTHTHTTTTTTHHHHHTTHHHHHTTHTHHHHHTTTHHTHTHHTHHHTHTHHTHTHTHHHTHHHHHHHHHHHTHHHHHHHTTHHHHTTHHHHTHHHTHHHTTHTTTHHHHHHHHHTTTHHHTTHTHHTHTHTTHTTTHHHTTHHHHHHHHHHTTHHHTTHTTTHTHHHTTTHHTHHTTTHHHHTTHHTHTTTTHTTHHHHHHTHTTHHHHHTHHTTHHHTTHHHHHTTTHTTHHTTHTTHTTHHHHHHTTTHHHTTTHTHHTHHHHHTHTTHHHHHHHHTTHHTHHTTHHHHTTHHHTHHHHHHHTTTHHTHHHHTHHHHHHHHHTHHTTHHTHTTTHHHTTHTTTHHHHHHHTTTHHHHTHHHHHHHTTTTTHHHTTHTTTHHHHTHTTTHTHHHTHTHHHTTTTHHTTHHTTTHHTHHTHHHTHTHTHHHHTHHTHHHTHHHTTTHHHTHHTHTHHTTHHTTHTTHHHTTTHHHHHHHHHHHTTHHTTHHHHHHHHHHTHTTHHHHHHTTHHTTHTTHTTHHTTTHTTHTHTHHHHTHHHTHHTHHTTHTTHTHHTTTHTHHTHTHHHHHTHTTHHTTHTHHHHHHTHTHHTHTHHTTHHHHHTHTHHTHTHHTTHTTTHHTHHHTTTTTTTTHHHHTHHTHTHHHHTHTHHHHTHHTHTTTHHTTHHTTTHHHHTTHHHHHHTHTTHTHTTHHTHHHTTHHTTTTTHHHHHTTTTTHHHTTHHTHTTHTHTTHHTTHHHTHHHHHHHTTHTTHHHHHHTTHHHHTHTHHTTHHHTHHHHTHHTHTTHHHTTTHTHTHTTHHHHHTHHHHHTTHTHHTTTHTTHHHHHHHTHTHHHHHTTTTTTHHHTTTTTHHTTTHHHHTHHTTHTHTHHTTHHHHTHHHTHHHHHHTHHHHHHHTTTHHHHHTTHHTHHTTTTTHHTHTTTTHHHTHTTHHTTTHTHHHTHHTHTHHTTHHTHTTTHHHTHHHHHHHHHHHHTHTTTHHTTHHHTHTTTHHTHHTTHHHHHHTHTTHTHHHTHTTTTHHHTHTHHHHHHHHTHHHTHHHHTTHHHHTTTTHTTHTHTHHHHHTHHHHHHTHTTHHTHTHHHHTHHTHHTHHHTHTHTHTTHTHHHHTHHTHHHTHHTHHTTHTHHTHHHHHTHHTTHHTTHTTTHTTHHHTTTTHTHHHHHHHHTHTTHHHTHTHTHHTHTTTHHHHHHTHTTTHHHTHHHHHTTHTHHHHHHTTTHTTHTHTHHTHTTTHHTTHTHHTTHTTHHTHHTHHHHTHTHHHTHHHHHTTHTTTHHTHHHHTTHHHTTTHHHTTTTHTHTHTHHTTTTHTHTHHHHTHHTHHTHHTTHHTTHHHTHHTHHHTTTTHHTHHHHTTHHHHHTHTTHHHHTTHHTHHTHHTHHHHHHTHHHTTHHHHHTTHTTHHTHHHTTTTTHTTTHTTTHTTHTHTTTTHHTHHHHTHHHHHTTHTTHTHTHTTHHHHTTTHHHHHHHHHTHTHHHTTHHTHTHTTHHHTHTTHHHTTHHTTHHHHTHTHHHHHHTHTHTHHTTHTTTHHTHTHTHHHTTTTTHHHHHTHTHTTTTHTTHHHTHTTTTHTTHTTHHTHTTHHHHHTTTHHHHHHHHHTTHTTHTTHHTHTTHTTTTTTHTTTHHTHTTHTTHHHHHHTHHHHHHHHHTHHTHHTHHHTHHHHHTHHTHTHTHHTHHHTHTHHTHTHHTTHTHTHTTHTHTTTHHTHHHTHTHHHHHTHHHTHHHTHTTHHHHHHTHHHHTHHHTTHHTHHTTHTTHTHTTHHTTHHHTTHHHHHHHTHTHHHTHHTTHHHHHTHTHHHHTHHTTHTTTHHTHHHHHTTTHHHHHHHHTTHTTTTTTHHHTHTTHHHTTTHTTHHHHHHHTTHTHTHHTHHTHTHHTTTHHHTTHHHHTHHHHHHHHHTHHHTTTHTTHTHHTHHHTHTHHHTTHHHHHHHHHHHTHTHTT

HHTHHTHTHHTTTTTHTHHHHHHHTTHHHHHTTTTTTTHHTTHTTHHTHHHTHHHHHHTHHTTTHHTHTHHHTHHHHHHHHHTTTTHTHHTTHHHHTHTHHHHHHHHTTHHHHTHTT

But maybe we should run N = 5000 ?

Statistics to the rescue!

• The question: could this actually be a fair coin, and the variance is due to chance?

• The question: could this actually be a fair coin, and the variance is due to chance?• Or: could this variant be just as “effective” as my control, and the variance is due to chance?

• The question: could this actually be a fair coin, and the variance is due to chance?• Or: could this variant be just as “effective” as my control, and the variance is due to chance?• We can calculate this probability: p.

• Normally, if p < 0.05, we can be confident. If p < 0.01, we can be very confident.

• We want to be 95% sure that the different results we see are due to a real difference in “effectiveness”.

• This 95% confidence interval is standard.

Can we say this is an unfair coin?

N = 10, P(H) = 0.5, p = 1 unclear!

N = 10, P(H) = 0.5, p = 1

N = 250, P(H) = 0.568, p ≈ 0.03

unclear!

confident!

N = 10, P(H) = 0.5, p = 1

N = 250, P(H) = 0.568, p ≈ 0.03

N = 500, P(H) = 0.632, p < 0.01

unclear!

confident!

very confident!

Best practices

CC-BY-NC flickr.com/photos/kylemacdonald/3199283481/

Best practices

• Decide on a clear, measurable, long-term metric ahead of time.

Best practices

• Decide on a clear, measurable, long-term metric ahead of time.• Clear metric = actionable results.

Best practices

• Experiment early and often.

Best practices

• Experiment early and often.• Test bold changes.

Best practices

• Experiment early and often.• Test bold changes.• Keep the potential costs in mind.

Best practices: how you test

• Assignment of visitors to variations must be truly random.

• Each visitor must have a consistent experience across visits.

• The site must still work, both during and after the experiment!

• Beware of interactions. Or go multivariate.

A/B testing on the web

• Get clear, actionable results with hard data.

• Get clear, actionable results with hard data.• Take the guesswork out of design.

• Get clear, actionable results with hard data.• Take the guesswork out of design.• Increase your bottom line with little cost.

• Get clear, actionable results with hard data.• Take the guesswork out of design.• Increase your bottom line with little cost.• You can’t afford not to!

• Unfortunately, if it’s not drop dead easy and convenient, it won’t happen...

ShrimpTest

CC-BY flickr.com/photos/gastev/2630047705/ CC-BY-SA flickr.com/photos/stuart_spivack/315481947/

ShrimpTestA/B Testing for WordPress

ShrimpTest*

* free beer and pizza if you guess why it’s so named

A/B Testing for WordPress

ShrimpTest

• A/B testing right in WordPress:

ShrimpTest

• A/B testing right in WordPress:• More convenient and easier to use

ShrimpTest

• A/B testing right in WordPress:• More convenient and easier to use• More robust and extensible

ShrimpTest

• A project of Automattic, written by mitcho

ShrimpTest

• A project of Automattic, written by mitcho• To be released as a free plugin.

ShrimpTest

• A project of Automattic, written by mitcho• To be released as a free plugin.• Developed in the open. Dev blog:

ShrimpTest

• A project of Automattic, written by mitcho• To be released as a free plugin.• Developed in the open. Dev blog:

• shrimptest.wordpress.com

ShrimpTest

Custom variant selection and metrics

$variant = shrimptest_get_variant( $my_experiment_id );// $variant is going to be an integer.switch ( $variant ) { case true: // if $variant > 0 echo "This is variant #{$variant}."; break; default: // if $variant is 0, or if ShrimpTest is down. echo "This is the control.";}

Custom variant selection and metrics

$variant = shrimptest_get_variant( $my_experiment_id );// $variant is going to be an integer.switch ( $variant ) { case true: // if $variant > 0 echo "This is variant #{$variant}."; break; default: // if $variant is 0, or if ShrimpTest is down. echo "This is the control.";}

// the user did something worth xshrimptest_update_metric( $my_experiment_id, $x );

Features

✓ Written as a WordPress plugin

Features

✓ Written as a WordPress plugin✓ Server-side (PHP) selection

Features

✓ Written as a WordPress plugin✓ Server-side (PHP) selection✓ Tools for easy setup of experiments: [ab] shortcode, variant viewer

Features

✓ API for custom metrics, variant types

Features

✓ API for custom metrics, variant types✓ “eat my own dogfood” approach

Features

✓ API for custom metrics, variant types✓ “eat my own dogfood” approach

✓ Works with caching plugins

Features

A work in progress...

• Documentation

• Documentation• UI improvements

• Documentation• UI improvements• Experiment automation tools

• Documentation• UI improvements• Experiment automation tools• More testing, particularly on higher traffic sites

Get involved!

• Try it out!

Get involved!

• Try it out!• tinyurl.com/shrimptesttrunk

Get involved!

• Check out and submit bug tickets:

Get involved!

• Check out and submit bug tickets:• tinyurl.com/shrimptesttrac

Get involved!

• Follow development:

Get involved!

• Follow development:• shrimptest.wordpress.com

Get involved!

• Follow development:• shrimptest.wordpress.com• @ShrimpTest

Get involved!

ShrimpTest

Thank you! Questions?mitcho (Michael 芳貴 Erlewine), Automatticmitcho.com, @themitchoshrimptest.wordpress.com, @ShrimpTest

A/B Testing with WordPress

ShrimpTest: A/B Testing for WordPress

Technology

Growing Your WordPress Site Business · Growing Your WordPress Site Business Paul B. Taubman, II

A/B testing aka split testing, bucket testing, and multivariant testing

Advanced Testing: 410(b) Coverage Testing of Combo Plans ......Advanced Testing: 410(b) Coverage Testing of Combo Plans, ESOPs, and 403(b) Plans Richard S. Phillips, AIF, ERPA, CPC,

Optimizely: A/B testing

Towards Automated A/B Testing

Testing and updating WordPress - Advanced techniques for avoiding regressions

Testing Oilfield Technologies for Wellsite Operationsflashpointcomponents.com/wordpress/.../12/Testing-Oilfield-Technologies... · 58 Oilﬁeld Review Testing Oilﬁeld Technologies

Beyond A/B testing: Automation within testing in Adobe Target€¦ · Beyond A/B testing: Automation within testing in Adobe Target ... Automation within testing in Adobe Target White

3 Ways to Setup a WordPress Testing Environment

A/B Testing - GitHub Pagessaravanan-thirumuruganathan.github.io/cse5334... · A/B Testing • A/B Split Testing is a mechanism for passive crowdsourcing that allows web developers

From Good to Great: Functional and Acceptance Testing in WordPress

GUI Testing By Norbert Haché. Contents b What is GUI testing b Elements of GUI testing b Old Approach (TRUMP Project) b Scripting b Capture / Replay b

A/B Testing for WordPress & Drupal

Auto Responders 101 - Split Testing (A/B Testing) Your Email

Unit Testing in WordPress

A/B testing

Mutation Testing€¦ · • Unit Testing + Code Coverage. Unit Testing. Unit Testing int sum(int a, int b) { return a + b; } Unit Testing int sum(int a, int b) { return a + b; }

Automate Accessibility Testing with Tenon.io for any Websites or WordPress sites

LIMITBREAK · UX Strategy Funnel Strategy SEO Strategy Web Analytics Conversion Rate Optimization A/B Testing Digital Products: Site Design & Development Wordpress Integration

Beyond A/B testing: Automation within testing in …...A/B testing can prove or disprove your CEO’s hypothesis. The most common approach to A/B testing is best described in statistical