A/B Testing the WordPress Way

A/B Testingthe WordPress Way

mitcho (Michael 芳貴 Erlewine)

WordCamp Phoenix, January 29, 2011

Introduction

Introduction

• Hi, I’m mitcho.

Introduction

• Hi, I’m mitcho.• Linguist, coder, teacher.

Introduction

• Hi, I’m mitcho.• Linguist, coder, teacher.

• http://mitcho.com; @themitcho

Introduction

• Hi, I’m mitcho.• Linguist, coder, teacher.

• http://mitcho.com; @themitcho• Linguistics grad student at MIT

Introduction

• Hi, I’m mitcho.• Linguist, coder, teacher.

• http://mitcho.com; @themitcho• Linguistics grad student at MIT• Automattic, Mozilla

Introduction

• Hi, I’m mitcho.• Linguist, coder, teacher.

• http://mitcho.com; @themitcho• Linguistics grad student at MIT• Automattic, Mozilla• Yet Another Related Posts Plugin

Introduction

• Hi, I’m mitcho.• Linguist, coder, teacher.

• http://mitcho.com; @themitcho• Linguistics grad student at MIT• Automattic, Mozilla• Yet Another Related Posts Plugin• Lightning talk later today!

Today

• A/B testing in a nutshell• Testing on the web?• Theory• Practice

• The WordPress Way

Testing on the web?

Testing on the web?

• Websites aren’t just for passive communication.

Testing on the web?

• Websites aren’t just for passive communication.

• Every website has a purpose.

Testing on the web?

• Websites aren’t just for passive communication.

• Every website has a purpose.• Common actions: register, sign up, log in, pay, donate, read more...

Testing on the web?

• Websites aren’t just for passive communication.

• Every website has a purpose.• Common actions: register, sign up, log in, pay, donate, read more...

• tinyurl.com/everywebsite

Example:

Example:• highrisehq.com

Example:• highrisehq.com• Action: sign up

Example:• highrisehq.com• Action: sign up• Tried many different button texts: “Free Trial”, “Sign-up for Free Trial”, etc.

Example:• highrisehq.com• Action: sign up• Tried many different button texts: “Free Trial”, “Sign-up for Free Trial”, etc.

• “See Plans and Pricing” increased sign-ups by 200%

thinkvitamin.com/business/how-to-increase-sign-ups-by-200-percent/

Example:

Example:

Example:

+30% +27%

+7%+15%

37signals.com/svn/posts/1525-writing-decisions-headline-tests-on-the-highrise-signup-page

Others:

Others:

• Huffington Post titles

Others:

• Huffington Post titles

• Most everything at Amazon, Google

Why test?

Why test?

• We know that {design, content, layout, titles, copy, interface, load time} matters.

Why test?

• We know that {design, content, layout, titles, copy, interface, load time} matters.

• How do you know you’re making the right decision? What if you disagree?

Why test?

• We know that {design, content, layout, titles, copy, interface, load time} matters.

• How do you know you’re making the right decision? What if you disagree?

• Get the “hard data.”

The theory of A/B testing

The theory of A/B testing

• So we can do a randomized experiment.

The theory of A/B testing

• So we can do a randomized experiment.• Randomly assign each user to a different treatment and keep track of a metric.

The theory of A/B testing

• So we can do a randomized experiment.• Randomly assign each user to a different treatment and keep track of a metric.

• How can we interpret the data?

Is this coin a fair coin?

TTTTHHHHTHTTHHTTHTHHHHHTTHHHHTHTTTTHHHHTTHHTTHTHTHHHHHTTTHHHHHHTHTHHHHTHHTTTTHHHHHTHTHHTTTTTHTHHHTHTTTHHTTHHHHHTTHTHHHHHTTTHHHTHTHHHHHHHTTTHTTHHHHTHHHTTTHHHHHHHHHHTTHTHHHHHTTHHHTHTHHHHTHHHHHTHHTHHHHHHHTHTHTHHHHHHHHHTHHHTHTHHTHHTHHHHHTHHHHTHTHTHHHHTTTHHHTHHHHHHHHHHTTTHHHHTTHTTHHHHTHTHHHHHTHTHHTHHHHTHTTTHTTTTHHTHHHTTHTHTTHHTHTHHHHHTHHHHHHHHTHTTHTTHTTHHHHTTHTHTHHHHTTHTHTHTHTHTTHHTHTTHTTTHHHHHHHHHHTHTHTHHTHTHHTHTHHHHHTHHHHHHHTHTHTTHTHHTHHHHTHHHTTTTHTTHHTHHHHTTHHHHTHTHTTHTHTHHHHHTHTTTHTHHTHTHHTTHTHTH

TTTTHHHHTHTTHHTTHTHHHHHTTHHHHTHTTTTHHHHTTHHTTHTHTHHHHHTTTHHHHHHTHTHHHHTHHTTTTHHHHHTHTHHTTTTTHTHHHTHTTTHHTTHHHHHTTHTHHHHHTTTHHHTHTHHHHHHHTTTHTTHHHHTHHHTTTHHHHHHHHHHTTHTHHHHHTTHHHTHTHHHHTHHHHHTHHTHHHHHHHTHTHTHHHHHHHHHTHHHTHTHHTHHTHHHHHTHHHHTHTHTHHHHTTTHHHTHHHHHHHHHHTTTHHHHTTHTTHHHHTHTHHHHHTHTHHTHHHHTHTTTHTTTTHHTHHHTTHTHTTHHTHTHHHHHTHHHHHHHHTHTTHTTHTTHHHHTTHTHTHHHHTTHTHTHTHTHTTHHTHTTHTTTHHHHHHHHHHTHTHTHHTHTHHTHTHHHHHTHHHHHHHTHTHTTHTHHTHHHHTHHHTTTTHTTHHTHHHHTTHHHHTHTHTTHTHTHHHHHTHTTTHTHHTHTHHTTHTHTH

N = 500, P(H) = 0.632

TTTTHHHHTHTTHHTTHTHHHHHTTHHHHTHTTTTHHHHTTHHTTHTHTHHHHHTTTHHHHHHTHTHHHHTHHTTTTHHHHHTHTHHTTTTTHTHHHTHTTTHHTTHHHHHTTHTHHHHHTTTHHHTHTHHHHHHHTTTHTTHHHHTHHHTTTHHHHHHHHHHTTHTHHHHHTTHHHTHTHHHHTHHHHHTHHTHHHHHHHTHTHTHHHHHHHHHTHHHTHTHHTHHTHHHHHTHHHHTHTHTHHHHTTTHHHTHHHHHHHHHHTTTHHHHTTHTTHHHHTHTHHHHHTHTHHTHHHHTHTTTHTTTTHHTHHHTTHTHTTHHTHTHHHHHTHHHHHHHHTHTTHTTHTTHHHHTTHTHTHHHHTTHTHTHTHTHTTHHTHTTHTTTHHHHHHHHHHTHTHTHHTHTHHTHTHHHHHTHHHHHHHTHTHTTHTHHTHHHHTHHHTTTTHTTHHTHHHHTTHHHHTHTHTTHTHTHHHHHTHTTTHTHHTHTHHTTHTHTH

N = 250, P(H) = 0.568

TTTTHHHHTHTTHHTTHTHHHHHTTHHHHTHTTTTHHHHTTHHTTHTHTHHHHHTTTHHHHHHTHTHHHHTHHTTTTHHHHHTHTHHTTTTTHTHHHTHTTTHHTTHHHHHTTHTHHHHHTTTHHHTHTHHHHHHHTTTHTTHHHHTHHHTTTHHHHHHHHHHTTHTHHHHHTTHHHTHTHHHHTHHHHHTHHTHHHHHHHTHTHTHHHHHHHHHTHHHTHTHHTHHTHHHHHTHHHHTHTHTHHHHTTTHHHTHHHHHHHHHHTTTHHHHTTHTTHHHHTHTHHHHHTHTHHTHHHHTHTTTHTTTTHHTHHHTTHTHTTHHTHTHHHHHTHHHHHHHHTHTTHTTHTTHHHHTTHTHTHHHHTTHTHTHTHTHTTHHTHTTHTTTHHHHHHHHHHTHTHTHHTHTHHTHTHHHHHTHHHHHHHTHTHTTHTHHTHHHHTHHHTTTTHTTHHTHHHHTTHHHHTHTHTTHTHTHHHHHTHTTTHTHHTHTHHTTHTHTH

N = 10, P(H) = 0.5

Not a fair coin:

P(H) = 0.6P(T) = 0.4

HHHHTHHHTTHTHHHTHTTHTHTTTTTHHTTTTHHTTHHHHHHHTHTHHHHHTTHHHHHHHTTHHHHTTHTTHHTHTHHTTHHTHHHTHHTTHHTTTHHHTHTHHTHHHTTTHTHHHHHHHHHHHHTTHTHHTTHHHTTHHTTHHHHTTHHTHHHHHHHHTTTHHTTTTHHHTHHHTHHTTTHHHTTHTHTHTHHTHHTHHHHHTTTHHTHHTHHTTHHHTHHTTHTTTTHTHTTHHTHHHTHTHTHTTTTTHHTHTTHHTTHHTHHTTHHTHHHTHTHHHHHHTTHTHTTHHHHHHHTHHHHTHTTHTHHHHHHHHTHTHTTTHHHHTHTHTTHHHHTHTHTHTTHTHHHHHHHHTTHTHTHHTTTTHHHHHHTTTTTHTHHTHTHTHTHHHTHHHTHHHHHTTTTTHTHHHTHHHHTTTHTHHTTTTHTTHTTTTHHTTTHTHTTHHTTHTHHTHHTTHHTHTHTHHHHHHHHTTTHHHTTTHTHTHHTTTHTHTTHHHHTHHHTHHHTHTHHHTHHHTHHTTHHHTHHTHHHHTHHTHHTHTTTTTHHHHTTHHHTHTTHHHHTHHTHHTHTHTHHTHHTTTHTTHHHTTHHTTHTHTTHTHTHHTHHHTTTHHTHHTHTHHHTHTTHTHHHHTHHTTTHTHHHHTHHTHHTTHHTHTTHTTHHTHTHTHHHHHTHHHHTHHHHHHTHTHHHHHHHHTHTTTTHTTHHHTHHHTHHTTHTTHTTHHHTHHHHHTHHHHTHHTHHTHHHHTTHHHHHHTTTHHHHHHHHTTHHTTHTTHHTTHTHHHTHTTTHHHTHHHHTTTHTHHHHHHHHHTTHHHTHHTHHHTHHTHTHHHTHHHTHHTTHTHHTTTTHHTHHHHTTTHHTHHTHHTTTTHHTTHTTHHTHHHHHHTHHTTHHHTHHTHTHTHTHHHHTTHTHHTTHHHTHTHHHHHTHTHHHHHTTTHHHTHHTHHHTHTHTHHHHHHTHHHTHTHHHHHTHTHHHHTHHHHHTHHHHTHTHHTTHHTHHHHTHHTTHHHHTHHTTHTHTTTHHHHHHHHHTHHHHTHHHTTHHHTTTHHHHTTHTHHHHTHHHTTHTTTHTTTTHTHTHTTHTHHTHTHHTTHHHHTTTHTHTHHHHTTHHTHTTTHTHTHHHTTHHHHHHHHHTTTHHTTHHHTHTHTTHHHTTHHHHHHHHHHTTHTHHHTTTHHTHHHTHTHHHTHTHTTHTHHTHTHHTHTTTTTHHTTTTHHTHHHHHHHTTHTTHHHHHTTTHTHHTTHTHHHHTHHTHTHTTHTHTTHHHTTTHTTTHHTTHHHHTTTHTHTTTTHHHTHTHHHTHHTHHHHHTHHHHHHHHTTTHHHTTTHHHTHHTTHTHHTHHHHHHTHTHTHTHHHTTHTHTHTHHTTTHHHTHHTHHTHHTTHTHTTHHHTHHHTHHTHTHTHHHTHTTTTTTHTTHHHHHTTTTTHTTHTHHTHTHTTHHHTHTHHTTTTTTHHHTTTHTTHHHHTHHHTHTHTHHHHHHTHHTTTHHHHHTHHHHTTTHTHHTHHHTHHTHTHHTHHTHHHHHHHTHHHHHTTTTTHHHHHTTHHHHHHHHTHHHHHHHTTTTTHHTHHHHTHHTHHTHTHHHTHHTHTHTHHTTHHTTHTHHHHTTTHTHTTHTHTHHTHHHTHHTHHHTHHHHHHHTHTHHHHHTTHTHHHHTHHTTTHHTTHHHHTTTTHHHTHHHTHTHTTTTHTHHHHTHHHHHTHHHHHHHHTHHTHHHHTHHTHTHTTTHTHHTTHHTHTTHHHHTHHHHHHHHTHHHHTHHHTTHHHTHHHHTTTTHHHHTHHHHHHHTTTHHTHHTTHHTTTTHHTHHHHHHTHTHTTHHTHHHHTTTHTHHTHTTHHHTHTHTTTTTHHTHHHHTHHHHHHHHHHHHHHHHHTTHHHHTHHHTTTTHTHHHHHHHHTHTHHTTTTHTTHTHHTHHHHHHHTHTHTHHHTTTTHHTHHHHHHTHTTHTHHTHTHHHHHHHTTHHHHTHTHTHTTTHTTHHHTHHHHHHTTHHTHTTHTTTHHHHHTTTHHTHHHTTHHHHHTHHHHHTTHTHTHHTHHHHTTHHTTTHTHHHTHHHHTTTTHTHHHHHHHTHTHHHHHHHHTTHTHHHHTHHHHTHHHHHTTHTTHHHHTTHTTHHHHTHHTHHHTTTHTHTTHTTTTHTTHTHHHHTTTHTTHHTTTTTHTHTTHTHTHTTHHTHTTHHHTHTTHTHHHHTHHTHHHHHHHHTTTHTHHTHTHTHHHHTHTTTHHHTHHHHHTHTTTHTTHHTHTTHHHHHTTTTTHHHHHTTTTHTHHHHTTHHHHHHHHHTHTHHHTTTTHTHTHTHHTHHTTTHHTHHTHHTHHTTHHHHHTHHHHTHTHHHHHTHTTHTHHHHTHHHTHHTHHTHHHTHHTHTTHHHHHTTTHHTHTHTHTHHTTHTHTTHHTTTHTHTHTHTTHHTHTHTHHHTHHHHHHHTTHHHHTHHTTHTHHTHTHTHHHHHHHHTTHHTHHHHTHHHTTHHHHHTTHTHHHHHHHHHTHTHHTTTHHTHHHHHTHTHHTHTHHHTTHHHHTTTHHTTHHTTTHTTTTTTTTHHHHTTHTHHHHHHTTHHHTHHHHTHTTHHHHHHTHHTTHHTTTHHHHTHHTTHTHTHHHHTTHTTTHHHTHTHHTHHHTTHTHHHTTHHTHHHHTTHHTHTTTTHHTTHTTTTTHHHHHHHHHHHTTTTHTTTHTHHTHHHHHHHHHHHTHHHTTHHHHTHTTHHHTTHHHTHHHTTTHTHHTHHHTTTTHTTHHHHTTHTTTHTHHHHHTHTHTHHTTTTTHTTHHTTHHTTTHHTHTHHTTTHHHHHHHHTTTTTTTHTHHHHTTTHHTHHHHTHHHTHHHTHHTHHHHTHHHHHHTHTTHHHHTTHHTHTTHHHTHTHTTHHTHHHTTTHHTTTHHHTHHHHTHHHTHTHTHHTHTTTTTTHTHTTHHTHHHHHHHHHHHHHTTHHTTHTHTTTTTTHHHHHTTHHHHHTTHTHHHHHTTTHHTHTHHTHHHTHTHHTHTHTHHHTHHHHHHHHHHHTHHHHHHHTTHHHHTTHHHHTHHHTHHHTTHTTTHHHHHHHHHTTTHHHTTHTHHTHTHTTHTTTHHHTTHHHHHHHHHHTTHHHTTHTTTHTHHHTTTHHTHHTTTHHHHTTHHTHTTTTHTTHHHHHHTHTTHHHHHTHHTTHHHTTHHHHHTTTHTTHHTTHTTHTTHHHHHHTTTHHHTTTHTHHTHHHHHTHTTHHHHHHHHTTHHTHHTTHHHHTTHHHTHHHHHHHTTTHHTHHHHTHHHHHHHHHTHHTTHHTHTTTHHHTTHTTTHHHHHHHTTTHHHHTHHHHHHHTTTTTHHHTTHTTTHHHHTHTTTHTHHHTHTHHHTTTTHHTTHHTTTHHTHHTHHHTHTHTHHHHTHHTHHHTHHHTTTHHHTHHTHTHHTTHHTTHTTHHHTTTHHHHHHHHHHHTTHHTTHHHHHHHHHHTHTTHHHHHHTTHHTTHTTHTTHHTTTHTTHTHTHHHHTHHHTHHTHHTTHTTHTHHTTTHTHHTHTHHHHHTHTTHHTTHTHHHHHHTHTHHTHTHHTTHHHHHTHTHHTHTHHTTHTTTHHTHHHTTTTTTTTHHHHTHHTHTHHHHTHTHHHHTHHTHTTTHHTTHHTTTHHHHTTHHHHHHTHTTHTHTTHHTHHHTTHHTTTTTHHHHHTTTTTHHHTTHHTHTTHTHTTHHTTHHHTHHHHHHHTTHTTHHHHHHTTHHHHTHTHHTTHHHTHHHHTHHTHTTHHHTTTHTHTHTTHHHHHTHHHHHTTHTHHTTTHTTHHHHHHHTHTHHHHHTTTTTTHHHTTTTTHHTTTHHHHTHHTTHTHTHHTTHHHHTHHHTHHHHHHTHHHHHHHTTTHHHHHTTHHTHHTTTTTHHTHTTTTHHHTHTTHHTTTHTHHHTHHTHTHHTTHHTHTTTHHHTHHHHHHHHHHHHTHTTTHHTTHHHTHTTTHHTHHTTHHHHHHTHTTHTHHHTHTTTTHHHTHTHHHHHHHHTHHHTHHHHTTHHHHTTTTHTTHTHTHHHHHTHHHHHHTHTTHHTHTHHHHTHHTHHTHHHTHTHTHTTHTHHHHTHHTHHHTHHTHHTTHTHHTHHHHHTHHTTHHTTHTTTHTTHHHTTTTHTHHHHHHHHTHTTHHHTHTHTHHTHTTTHHHHHHTHTTTHHHTHHHHHTTHTHHHHHHTTTHTTHTHTHHTHTTTHHTTHTHHTTHTTHHTHHTHHHHTHTHHHTHHHHHTTHTTTHHTHHHHTTHHHTTTHHHTTTTHTHTHTHHTTTTHTHTHHHHTHHTHHTHHTTHHTTHHHTHHTHHHTTTTHHTHHHHTTHHHHHTHTTHHHHTTHHTHHTHHTHHHHHHTHHHTTHHHHHTTHTTHHTHHHTTTTTHTTTHTTTHTTHTHTTTTHHTHHHHTHHHHHTTHTTHTHTHTTHHHHTTTHHHHHHHHHTHTHHHTTHHTHTHTTHHHTHTTHHHTTHHTTHHHHTHTHHHHHHTHTHTHHTTHTTTHHTHTHTHHHTTTTTHHHHHTHTHTTTTHTTHHHTHTTTTHTTHTTHHTHTTHHHHHTTTHHHHHHHHHTTHTTHTTHHTHTTHTTTTTTHTTTHHTHTTHTTHHHHHHTHHHHHHHHHTHHTHHTHHHTHHHHHTHHTHTHTHHTHHHTHTHHTHTHHTTHTHTHTTHTHTTTHHTHHHTHTHHHHHTHHHTHHHTHTTHHHHHHTHHHHTHHHTTHHTHHTTHTTHTHTTHHTTHHHTTHHHHHHHTHTHHHTHHTTHHHHHTHTHHHHTHHTTHTTTHHTHHHHHTTTHHHHHHHHTTHTTTTTTHHHTHTTHHHTTTHTTHHHHHHHTTHTHTHHTHHTHTHHTTTHHHTTHHHHTHHHHHHHHHTHHHTTTHTTHTHHTHHHTHTHHHTTHHHHHHHHHHHTHTHTT

HHTHHTHTHHTTTTTHTHHHHHHHTTHHHHHTTTTTTTHHTTHTTHHTHHHTHHHHHHTHHTTTHHTHTHHHTHHHHHHHHHTTTTHTHHTTHHHHTHTHHHHHHHHTTHHHHTHTT

But maybe we should run N = 5000 ?

Statistics to the rescue!

Statistics to the rescue!

• After 500 trials, it’s very rare even for a known fair coin to give you exactly 50% heads.

Statistics to the rescue!

• After 500 trials, it’s very rare even for a known fair coin to give you exactly 50% heads.• The question: could this coin (variant) actually be just as “effective” as a fair coin (control), and any variance we see is due to chance?

Statistics to the rescue!

• After 500 trials, it’s very rare even for a known fair coin to give you exactly 50% heads.• The question: could this coin (variant) actually be just as “effective” as a fair coin (control), and any variance we see is due to chance?• We can calculate this probability: p.

Statistics to the rescue!

Statistics to the rescue!

• Normally, if p < 0.05, we can be confident. If p < 0.01, we can be very confident.

Statistics to the rescue!

• Normally, if p < 0.05, we can be confident. If p < 0.01, we can be very confident.

• We want to be 95% sure that the different results we see are due to a real difference in “effectiveness”.

Statistics to the rescue!

• Normally, if p < 0.05, we can be confident. If p < 0.01, we can be very confident.

• We want to be 95% sure that the different results we see are due to a real difference in “effectiveness”.

• This 95% confidence interval is standard.

Can we say this is an unfair coin?

Can we say this is an unfair coin?

N = 10, P(H) = 0.5, p = 1 unclear!

Can we say this is an unfair coin?

N = 10, P(H) = 0.5, p = 1

N = 250, P(H) = 0.568, p ≈ 0.03

unclear!

confident!

Can we say this is an unfair coin?

N = 10, P(H) = 0.5, p = 1

N = 250, P(H) = 0.568, p ≈ 0.03

N = 500, P(H) = 0.632, p < 0.01

unclear!

confident!

very confident!

Best practices

CC-BY-NC flickr.com/photos/kylemacdonald/3199283481/

Best practices

• Decide on a clear, measurable, long-term metric ahead of time.

CC-BY-NC flickr.com/photos/kylemacdonald/3199283481/

Best practices

• Decide on a clear, measurable, long-term metric ahead of time.• Clear metric = actionable results.

CC-BY-NC flickr.com/photos/kylemacdonald/3199283481/

Best practices

• Decide on a clear, measurable, long-term metric ahead of time.• Clear metric = actionable results.

• Test bold changes.

CC-BY-NC flickr.com/photos/kylemacdonald/3199283481/

Best practices

• Decide on a clear, measurable, long-term metric ahead of time.• Clear metric = actionable results.

• Test bold changes.• Keep the potential costs in mind.

CC-BY-NC flickr.com/photos/kylemacdonald/3199283481/

Best practices: how you test

Best practices: how you test

• Assignment of visitors to variations must be truly random.

Best practices: how you test

• Assignment of visitors to variations must be truly random.

• Each visitor must have a consistent experience across visits.

Best practices: how you test

• Assignment of visitors to variations must be truly random.

• Each visitor must have a consistent experience across visits.

• The site must still work, both during and after the experiment!

Best practices: how you test

• Assignment of visitors to variations must be truly random.

• Each visitor must have a consistent experience across visits.

• The site must still work, both during and after the experiment!

• Beware of interactions.

A/B testing on the web

A/B testing on the web

• Get clear, actionable results with hard data.

A/B testing on the web

• Get clear, actionable results with hard data.• Take the guesswork out of design.

A/B testing on the web

• Get clear, actionable results with hard data.• Take the guesswork out of design.• Increase your bottom line with little cost.

A/B testing on the web

• Get clear, actionable results with hard data.• Take the guesswork out of design.• Increase your bottom line with little cost.• You can’t afford not to!

A/B testing on the web

• Get clear, actionable results with hard data.• Take the guesswork out of design.• Increase your bottom line with little cost.• You can’t afford not to!

• Unfortunately, if it’s not drop dead easy and convenient, it won’t happen...

Today

• A/B testing in a nutshell• The WordPress Way

ShrimpTest

CC-BY flickr.com/photos/gastev/2630047705/ CC-BY-SA flickr.com/photos/stuart_spivack/315481947/

A B

ShrimpTestA/B Testing for WordPress

ShrimpTest

• A/B testing right in WordPress:

ShrimpTest

• A/B testing right in WordPress:• More convenient and easier to use

ShrimpTest

• A/B testing right in WordPress:• More convenient and easier to use• More robust and extensible

ShrimpTest

• A/B testing right in WordPress:• More convenient and easier to use• More robust and extensible

• A project of Automattic, written by mitcho

ShrimpTest

• A/B testing right in WordPress:• More convenient and easier to use• More robust and extensible

• A project of Automattic, written by mitcho• Released as a free plugin.

ShrimpTest

Demo

Custom variant selection and metrics

Custom variant selection and metrics

$variant = shrimptest_get_variant( $my_experiment_id );// $variant is going to be an integer.switch ( $variant ) { case true: // if $variant > 0 echo "This is variant #{$variant}."; break; default: // if $variant is 0, or if ShrimpTest is down. echo "This is the control.";}

Custom variant selection and metrics

$variant = shrimptest_get_variant( $my_experiment_id );// $variant is going to be an integer.switch ( $variant ) { case true: // if $variant > 0 echo "This is variant #{$variant}."; break; default: // if $variant is 0, or if ShrimpTest is down. echo "This is the control.";}

// the user did something worth xshrimptest_update_metric( $my_experiment_id, $x );

Features

✓ Written as a WordPress plugin

Features

✓ Written as a WordPress plugin✓ Server-side (PHP) selection

Features

✓ Written as a WordPress plugin✓ Server-side (PHP) selection✓ Tools for easy setup of experiments: [ab] shortcode, variant viewer

Features

✓ Written as a WordPress plugin✓ Server-side (PHP) selection✓ Tools for easy setup of experiments: [ab] shortcode, variant viewer

✓ API for custom metrics, variant types

Features

✓ Written as a WordPress plugin✓ Server-side (PHP) selection✓ Tools for easy setup of experiments: [ab] shortcode, variant viewer

✓ API for custom metrics, variant types✓ “eat my own dogfood” approach

Features

✓ Written as a WordPress plugin✓ Server-side (PHP) selection✓ Tools for easy setup of experiments: [ab] shortcode, variant viewer

✓ API for custom metrics, variant types✓ “eat my own dogfood” approach

✓ Works with caching plugins

Features

Start testing now!

• Download at http://shrimptest.com

Start testing now!

• Download at http://shrimptest.com• Stay up to date: @ShrimpTest

Start testing now!

Thank you! Questions?mitcho (Michael 芳貴 Erlewine), Automatticmitcho.com, @themitchoshrimptest.com, @ShrimpTest

A/B Testingthe WordPress Way