A/B Testing + Why You Should Do It

A/bout this webinar

● A/B testing overview: what and why?

● Approach to testing: behavioral framework

● Testing methodology: copy, visuals, layout

● A/B testing on website● A/B testing emails

What is A/B Testing?

A B

Descendant of controlled experiment

Popularized by Google around 2000

What is A/B testing?

A/B testing is comparing two or more versions of something to see which one performs better.

Why A/B Test?Everybody’s doing it

Designs = best guesses

Build

Measure

Learn

40.6% lift in conversion2.8m email addresses$60m +/-

... ...

40.6% lift in conversion2.8m email addresses$60m +/-

... ...

Netflix + chill

Benefits for Nonprofits

● Improve key engagement conversions (email signups, campaign participation, donations, etc)

● Improve site continuously● Try before you buy

Defined Goals

● $ or emails● Sign-ups● Desired/predictive Behavior● Digital conversions tied to

impact ● MVP modeling

Approach to TestingTheory & reality

What makes people do something?

Passive Engaged

Behavior = motivation*ability*trigger

● Motivation: Undying love for food

● Ability: Abs + toned shoulders● Trigger: Human command

Behavior = motivation*ability*trigger

A/bility - Can I do this easily?

● Give perception of ease○ Time: “quick

survey”○ Effort: “no credit

card required”● Minimize number

of taps

Goal: Donate Form Completion

Goal: Donate Form Completion

+5%

Motivation - Do I want to do this?

● Focus on need fulfillment ● Social proof - others are using it● Authority: 3rd Party Validation ● Scarcity: Leverage FOMO (missing out)● Reciprocity: give something away

Goal: Join a Network

Original (boring)

George

Ali

Dollar bet battle

vs.

George

Ali

Dollar bet battle

vs.

+99%

-32%

Winner

InvigoratedConfident

ChallengedJoyful

Connected

AngryFearfulAnxious

DefensiveResentful

RelaxedMellow

PeacefulTranquilSerene

DepressedExhaustedBurned outHopelessDefeated

Positive(pleasant)

Negative(unpleasant)

Highintensity

Lowintensity

Trigger - What do I click?

● Clear calls-to-action● Ask at the right time● Ask at different moments

“Sticky”CTA

Reality (data) check

Where to Test?So many pages, so little time

Types of Pages to Test

● Main landing pages ● Conversion (SEM) pages ● Pop-ups● Page templates

What to Test?Look good, sound good, feel good

What to test: copy

● Body copy● Headlines● Calls-to-action● Menu item name● Donation amount

http://headlines.sharethrough.com/

What to test: copy

● Body copy● Headlines● Calls-to-action● Menu item name● Donation amount

http://headlines.sharethrough.com/

1.84% email conversion rate

9.66% email conversion rate

Improvement over time

Child Trends example

Child Trends example

What to test: Visuals

● Hero images● Icons● Colors

ASpice it up

BKeep it simple

ASpice it up

BKeep it simple

4x

teacher signups

Audubon example

Audubon example

What to test: layout

● Order of elements● Position of elements● Element “staticness”● Form length

A2 CTAs

B1 CTA

A2 CTAs

38.3% Clickthrough rateimprovement

B1 CTA

America Recycles Day example

Pledge form

America Recycles Day example

Pledge form

Running experimentsReady, set, test

Some A/B testing tools

Build

Build

Measure

Measure

Learn

Where to start?

● What is the most important action users can take on your site?

● Which behavioral element (ability, motivation, trigger) can be improved?

● Which material element (copy, visuals, layout) can you test?

● What’s your hypothesis?

=> Don’t walk. Run the A/B test!

NewslettersGot your email, now what?

Email Anatomy 101

Preheader - small text that shows up in inbox list. Can help increase open rate!

Subject line Preheader

Sender

A/B Testing

Preheader text and sender

Time (will tell)

Test delivery time methods to optimize time of day:● Send in batches● Use Timewarp to cater to

users’ time zone● Send Time Optimization

gives a recommendation based on zillions of data points

Subject line

Test your subject line if you have more than 5,000 people.Which would you open?

“Typhoon Haiyan Support Slam” “What Would YOU Change?”

Subject line

Test your subject line if you have more than 5,000 people.Which would you open?

“Typhoon Haiyan Support Slam” “What Would YOU Change?”

9,300 People difference for Power Poetry (List 100k)

Subject line length

Resources

● Optimizely Academy● Whole Whale’s 9 steps

to run a/b test● Mailchimp A/B test

tutorial

Where to start?

You know it matters, so GET STARTED!

The only way to get it (completely) wrong is to not do it all.