1

newmediamentors.orgINTRO TO A/B TESTING

CULTURE OF TESTING

• Not just a set of rules or best practices

• A commitment to:– Measure the effectiveness of your choices– Make decisions informed by data, not just intuition

• So… how do we do that?

WHAT IS A CULTURE OF TESTING?

• Metric: numerical measure of performance

• Email metrics:– Open rate– Click rate– Action rate– Unsubscribe rate

• Web page metrics:– Conversion rate (online actions per visitor)

METRICS, METRICS, METRICS

• Q: You have two ideas for an email – which one do you choose?

• A: Both!

• Steps to run an email A/B test:– Select two randomized email groups from your list– Send email A to the first group and email B to the second– Look at email metrics to determine which email did better– Send better email to the remainder of your list

METRICS IN ACTION:EMAIL A/B TESTS

• Example test:– Email A subject line about

puppies, email B subject line about cats

– List of 100,000 emails; select 10,000 randomly for A, 10,000 randomly for B

– Send emails, measure results:• 1,276 open A and 1,403 open B• 267 donate to A, 195 donate to B

– Which do you choose?

METRICS IN ACTION:EMAIL A/B TESTS

REAL EMAIL A/B TEST EXAMPLE

71% more in contributions from Version 1

• Higher test performance doesn’t always mean better – needs to be statistically significant

• Make sure test groups large enough for significance

• Stay tuned for calculator demo!

STATISTICAL SIGNIFICANCE

• Emails aren’t free – lose people who unsubscribe, and hurt email deliverability if people aren’t engaging

• Need to balance benefit of actions against cost of sending

• “Trial balloon” test:– Send email to small, randomized group– Evaluate metrics and decide if benefits outweigh costs– If yes, send to remainder; if no, don’t send

TO SEND OR NOT TO SEND:DELIVERABILITY & TRIAL

BALLOONS

• Like email, possible to A/B test web pages

• Web page A/B test:– Create several variations of web page– For new users coming to page, randomly show one

variation– Record if visitor takes action– Calculate conversion rate for each variation – highest

conversation rate is best

• Need enough time for statistical significance

WEB PAGE A/B TESTING

29% increasedconversion rate

WEB PAGE A/B TESTING

• Can also test engagement for posting by supporters to Facebook and Twitter

• Social media A/B testing:– Create several variations of Facebook post (title,

description, thumbnail) or Twitter text– For users sharing from your website, randomly give

them one variation– Record if users’ friends click share link and take action– Calculate recruitment rate for each variation – share post

variation most likely to drive action is best

SOCIAL A/B TESTING

SOCIAL A/B TESTING

Facebook TitleShares Driving Action / Total

SharesSuccess

RateImprovem

ent

Monsanto shouldn’t be above the law 341/1041

32.8% (±2.9%) --

“One of the most outrageous special interest provisions in years”

322/96233.5%

(±3.0%) 2.2%

Tell the Senate: Repeal the Monsanto Protection Act

458/97247.1%

(±3.1%) 43.8%

Sign the Petition: Repeal the Monsanto Protection Act

555/1037 53.5% (±3.0%) 63.4%Facebook headline test by CREDO Action – 63% increased

recruitment rate from Version 4

• Emails and web pages

• Social sharing

• Online ads

• Even offline actions

CULTURE OF TESTING INEVERYTHING YOU DO

MAINTAINING ACULTURE OF TESTING

• Get buy-in across the organization

• Run at least one test for every campaign you do

• Schedule periodic check-ins on what you've learned and accomplished

• Email A/B testing:– Check your email

management system

• Webpage A/B testing:– Google Analytics– Optimizely

• Social A/B testing:– ShareProgress

• New Media Mentors A/B Testing Tool

HELPFUL TOOLS

QUESTIONS?

18

A/B TESTING TOOLNew Media Mentors’

19

SAMPLE FINDER

Very sure4Medium

TemplateTest

14,154

20

SAMPLE FINDER

Somewhat sure

15Medium

SubjectTest

1,363

21

SIGNIFICANCE INSPECTOR

1363

1363

205

273

Yes!Results are statistically significant.

22

SIGNIFICANCE INSPECTOR

1363

1363

205

232

No.Results are not statistically significant.

23

newmediamentors.orgINTRO TO A/B TESTING