2013 DMA Post Conference Certification

The Answer is Always, “It Depends” – So Test It!

Ryan PhelanVice President, Strategy

Acxiom

• Prior Industry Experience– Vice President, Strategy at BlueHornet

– Director, Email Marketing & Acquisition at Sears Holdings

– Responsible for East Coast Operations at Responsys

• Thought Leadership– DM News : Email Gets Personal (Cover Story)

– Keynote address – March 2012, EEC12

– Ranked as one of the top 40 Digital Marketing Strategists in the country by OMI

– Co-Chair of the EEC

– Member of:Ryan PhelanVice President, Global Strategic Services

Join the Email Evolution Council for education, events, networking and

resources!

“OMG Honey, look at that compelling subject line ….totally made me open it”

This is more like it…

Tier 3:Advanced

Tier 2:Medium Complex

Tier 1: Foundation programs

Existingprograms

Persona development, cluster analysis, next logical product, behavioral \ attitudinal segmentation, dynamic messaging, shopping cart abandonment,

preference center phase 2, social messaging

Video in email, creative testing, promotional optimization, triggers, win-back, preference center, social media

Testing & Reporting

Welcome, transactional messaging, opt-down, acquisition, promotional, attrition

Optimize programs based on easy changes (Low Hanging Fruit)

Testing & Reporting

Testing & reporting

Email Program Development

1,000 Consumers from across the United States

Nearly 49% of respondents have an email account for emails they rarely intend to open

Lesson: Ensure that when you ask for an email address, you make the reason very compelling

2013 Acxiom Digital Impact Consumer Digital Behavior Study n=1,006

9

When an email is saved to be read

later, 60% never read it

Lesson: Make your CTA immediate, urgent, laced with benefit and time sensitive in email


Lesson: 21% to get updates and 13% because they love the brand10

40% of consumers sign up for email to receive discounts


11

36% of respondents check email, social media and texts before doing anything else after they wake up


Lesson: Think about your message and test the optimal time of the day to send your message.

Lesson: Test when you send and be wary of complex CTA in the morning.12

21% check their email before breakfast


Consumers have shifted their consumption and are active at the very start of the day

Lesson: Try testing in the evening or even Friday evening for retail based business.2013 Acxiom Digital Impact Consumer Digital Behavior Study n=1,006

14

Things are tiny in the morning…

Lesson: If you have to pinch it, you’re doing it wrong. Track mobile opens and design a template.


15

Phone calls, texts, browsing the internet and email are the top uses for

smart phones

Lesson: Lesson: Track statistics for your consumers that are consuming email on a mobile device.


Lesson: Track statistics for your consumers that are consuming email on a mobile device. 16

The shift in consumption is increasing based on consumer need for and ease of access


17

Consumers don’t see silos, they see devices

Lesson: Consumers have adopted more devices and marketers have to be truly Omni-Channel


Lesson: Capture and track the mobile opens of your subscribers and implement a template18

91% of consumers check email on their mobile phones


19

…25% of those that use iPhone Passbook used it to access coupons

Of those that use iPhone Passbook (33%), 22% use it for Movie Tickets…

Lesson: Try linking a coupon for a sale or event to the Passbook functionality and then track use


20

72% of consumers read email when they are bored…

…29% read email while in the bathroom

Lesson: Don’t make your email creative boring – inspire, delight and amaze


Nothing anyone says in marketing is right(except me of course)

It depends…(the strategists mantra)

QUIZ: WHICH TEST WON?DMA2013 | Email Testing in the Digital Age

Which Test Won?

Up or Down Arrows

Which Test Won?

Up or Down Arrows

Which Test Won

Which Test Won

Which series won?Images Courtesy of:

Which series won?Images Courtesy of:

CHALLENGES OF THE EMAIL MARKETER

DMA2013 | Email Testing in the Digital Age

The email marketer’s challenge• More subscribers• More data• More email

• More targeting• More expectations• Poor attribution

What do we struggle with

You’re not alone…everyone struggles with email

Chart 4.21 Email campaign element testing and optimizationWhich of the following email campaign elements do you routinely test to optimize performance? Please select all that apply.

Subject line

Call-to-action

Message (eg greeting, body, closing)

Days of the week sent

Layout and images

Time of day sent

Landing page

Target audience

Personalization

From line

Layout and images specifically for mobile viewing

Other

None of the above

86%

62%

58%

48%

47%

46%

44%

44%

42%

32%

26%

Source: ©2013 MarketingSherpa Email Marketing Benchmark Survey Methodology: Fielded December 2012, N=264

2%

2%

Message (e.g. greeting, body, closing)

Why do we care?

• Campaign on 10/17– 166

individual email segments

– 84 were tests

Images Courtesy of:

Does this look bad???

The highlighting worked…for a while

Images Courtesy of:

Testing Challenges

We don’t (as an industry) know how to test

Types of TestingA|B Testing• Testing one element against a control

Multi-Variant• Testing multiple elements against a control

Local Control Groups• Isolation of a population on a campaign level to see % of lift

Universal Control Groups• Isolation of a permanent population to see percentage of lift over time

A|B Testing

Cons• Populations must be equal• Time periods must be

significant to judge results• Limited to one variable so

extended testing can be long

Definition• 2 email creative that are

identical but with one element changed in one version

Pros• Simple testing that is built into

most email platforms• Systems usually handle

division of population• Results are easy to

understand and act upon

Multivariate Testing

Cons• Populations in each group

must be equal• Most email populations do

not have enough equal parts to be statistically relevant

DefinitionUsing one region in multiple email creative with changes in each sent to equal populations to determine a better performing email

Pros• Multiple elements can be

tested at the same time

Local Control Groups

Cons• Sometimes hard to manage

from a population• Some people that are

active/buyers will not get a message

• Must involve pre-planning

DefinitionIsolation of a small but significant population from an email campaign to see what happens with their behavior against those that received an email

Pros• Can show the influence that

and individual email has on a campaign level

• Population must be reflective of the entire list

Universal Control Group

Cons• Isolation of a population

means that some customers don’t get an email

• Hard sell internally• Must educate various groups

internally

DefinitionIsolating a population over a longer period of time to see what their behavior is against those that receive email

Pros• Gives email a true ROI

number

CONFIDENCE INTERVALSDMA2013 | Email Testing in the Digital Age

Sample Size DeterminationThere is no perfect answer in determining sample size. It is a trade-off between sample size and the difference between A and B it is important for us to detect.

Acxiom presents a chart (as seen below) to help clients find their optimal balance.

43

The smaller the difference we want to be able to detect, the greater the required

sample size

Example Confidence Interval

• We have some results from an A/B Test:

oHow much confidence do we have in these estimates?

oDo we feel comfortable enough in the observed gain of .3% to switch to

B?

–It’s all about Sample Size when considering confidence in Response Rates.

44


• The 95% Confidence Interval indicates we are 95% certain the RANGE of the interval captures the True response rate

• We observed B as 0.3% greater than A.

• We can now use our confidence interval for the difference (B – A) to establish how tight that 0.3% difference is based on our sample sizes

• The range includes A being .8% greater than B all the way to B being 1.4% greater than A. With this wide range of possibilities, taking action based on our estimates becomes very dangerous

45


• Let’s say we have the same results, but instead of results based on samples of 1,000 each, they are now based samples of 50,000 each.

• We now have more evidence, due to our greater sample size. This results in greater belief in our results (estimates), and hence tighter intervals

• We still observe B as 0.3% greater than A, however now we can now conclude B is greater than A by between 0.1% and 0.5%.

46

THE RULES OF TESTINGDMA2013 | Email Testing in the Digital Age

buck·shot mar·ket·ing n. Marketing without a plan, clue, intelligent design or path toward success

Rules for Testing

1. What needs to be tested2. Get a plan3. Execute the test4. Report on the results

What needs to be tested

• To develop a plan, take time to define what’s broken– Mobile creative– Subjects– CTA– Lifestyle images vs action images– personalization– Audience/Segment– Landing page– Discount type

Get a Plan

1. Develop a plan that lasts from 1-3 months1. Set rules for populations

1. What determines a valid population

2. What is success in each test1. What is the KPI that will be judged2. Define from prior test/campaigns what the range of KPI success

exists

3. Determine the right amount of time to build a significance to achieve the desired KPI

4. What are your exclusions5. Get extra pairs of eyes – make it a team effort

One point to remember about the plan

• Does the result really prove the point?– Always work to validate

your testing– Sometimes thing “win”

because they’re different

• Recognize the “shiny objects”

Images Courtesy of:

Execute the Test

• Define equal populations• Define the optimal timeline for response and

adhere to it• Monitor results• Verify that the test has been carried out

Report on the Results

• Each test should have a post-mortem report– Reason– Goal– Creative– Population– Results

• Should be one page per report and stored• How does the result inform the next test

1. Track results1. You must be able to track testing results2. Determine who’s on the strike team to examine results

EXERCISESDMA2013 | Email Testing in the Digital Age

YOU NOW ALL WORK FOR

…AND GET PAID $1,000,000 A YEAR(SO DON’T SUCK)

• Come up with a testing plan for this creative

Images Courtesy of:

Small Groups• Define what you could test

(10 things over 3 months)– Cannot be subject line– Why are you testing it

• Define how you would test it• Define the audience• Define the exclusions

Assumptions• Customer is a male• 42 years old• Lives in Half Moon

Bay, CA• 20 miles south

of SF• Single

Teams will present their testing plan to the group. Best one, wins something gooooooood.

Go…now…be smart

Conclusion

• Think about your testing plan• Find equal populations• Test things that move the needle• Stop testing only subject lines

Questions?

© 2013 Acxiom Corporation. All Rights Reserved.

Ryan PhelanVice President, Strategic [email protected]

Thank you!

Keep the learning going. Follow me on twitter @ryanpphelan @acxiom

APPENDIX


64

How did we get the confidence intervals?

Confidence Interval for one proportion:

Test A = 0.015 +/- 1.96*SqRt ( (0.015 x 0.985) / 1,000 ) = 0.015 +/- 0.008 = (0.7%, 2.3%) Test B = 0.018 +/- 1.96*SqRt ( (0.018 x 0.982) / 1,000 ) = 0.018 +/- 0.008 = (1.0%, 2.6%)

Confidence Interval for difference between two proportions:

Test (B – A) = 0.003 +/- 1.96*SqRt ( ((0.015 x 0.985) / 1,000 ) + ((0.018 x 0.982) / 1,000) )Test (B – A) = 0.003 +/- 0.011 = Test (B – A) = (-0.8%, 1.4%)