DM 201 - Avalon Consulting - Testing Creative

1Avalon Consulting Group, Inc.

DIRECT MARKETING 201The Strategy and Metrics of Creative Testing

Avalon Consulting Group

May 14, 2014

www.avalonconsulting.net

Testing QUIZ!Which Test Won?

And the Winners Are…

In case you’ve forgotten, your opinion doesn’t matter…

Creative Strategy & Testing

TESTING STRATEGY

Testing is essential to have a healthy, productive direct marketing program.

1. Each organization is different and not all best practices are universal.

2. Used to determine what works for your organization.

3. Test making incremental or large changes, seeing their impact, while minimizing the overall risk.

4. Help find ways to improve performance, can save money and improve the bottom-line.

Things to remember about testing:

Test one element at a time

Test to save cost (formats, components) or boost response rate or average gift (offer, teasers, treatments)

Ensure the statistical significance of your test panel.

CASE STUDY: A Snapshot of the NPCA Acquisition Testing History

NPCA Acquisition Control Packages

DM ACQUISITION CONTROL PERFORMANCEJu

2($30.00)

($25.00)

($20.00)

($15.00)

($10.00)

($5.00)

Survey net/donor

Member card net/donor

• The FY12 overall results of

the acquisition controls have

shown a weakening of the

member card, with the

survey more cost effectively

acquiring donors.

• However, Avalon analysis of

the member card showed a

stronger ROI, so both

packages continue to be

mailed as co-controls in

Test Within Your Acquisition Control Packages

Premium (cost savings)CONTROL: Calendar and bucket hat TEST: Calendar only WINNER: Control - Calendar bucket hat

Ask String (boost avg. gift)CONTROL: $15 minimum gift to receive premiumTEST: $25 upgrade ask testWINNER: There was no statistical winner in the original test, but we mailed a panel of this package every month. Backend analysis shows ROI of $25 ask is stronger.

Combine elements of co-controlsCONTROL: Member card package or Survey packageTEST: Member card in Survey package WINNER: Test – Member card in the Survey package. Re-test confirmed results.

Combine tone of Survey and Member Card CONTROL: Member card package or Survey packageTEST: Combine the urgency of the member card with the emotion of the survey letter. WINNER: There was no statistical winner. Re-testing to get significant results.

Phrase-based formatting (boost response rate)CONTROL: Control acquisition packageTEST: Use ReadSmart to make subtle changes in the arrangement of the text on the page WINNER: There is no statistical winner.

Removal of insert (cost savings)CONTROL: Control acquisition packageTEST: Remove glossy color insert WINNER: Control with the insert.

Test New Acquisition Packages

Map package Included a foldout wall map and personalized teaser with a national park in the prospect’s state. The test has a 0.91% response rate – comparable to the control’s 0.92%, and a higher average gift than the control ($23.13 vs. $22.73), but the costs were too high.

Patriotic package Used a stirring, patriotic tone to highlight the bond Americans feel with their national parks, and included a lift note with a quote from Ken Burns. Avg. gifts are only $0.30 apart, the strong control had the higher response rate (0.99% vs. the test’s 0.83%) and revenue/M - $234 vs. the test’s $196.

ACQUISITION TESTS – NEW PACKAGES

Postcard Package (Merge K) This package was developed to try and incorporate more park images into the acquisition with the theory that if someone was reminded of the beauty they were helping to protect, they would be more responsive. The “Wish you were here!” package included a 4-color insert with four postcards showcasing the grandeur of the parks. The letter invited members to enjoy our parks now, and support them for generations to come, by joining NPCA.

The postcard test had a significantly weaker revenue/M, compared to the member card and the survey.

ACQUISITION TESTS – NEW PACKAGES

Magazine Package The magazine subscription package included a 4-color sample of the National Parks magazine, and emphasized the magazine as NPCA’s premier membership benefit.

In addition to a weaker average gift, the package currently has a 10% weaker RR than the survey package and a 30% weaker RR than the member card, as well as a weaker revenue/M.

Review Acquisition Results by Premium

Anorak Bison Blanket Bucket hat

Tote bag$0.00

$10.00

$15.00

$20.00

$25.00 Survey

Survey

Review back-end premium costs to acquire a donor. Include front-end as well as back-end costs.

The anorak, blanket and bucket hat all have comparable results in the member card package and can be mailed interchangeably, but in the survey, the blanket, tote bag and bucket hat are strongest because their costs to acquire a donor are lower.

Anorak Bison Blanket Bucket hat Tote bag$10.20

$10.40

$10.60

$10.80

$11.00

$11.20

$11.40

$11.60

$11.80 Member card

Member card

Online Activist Conversion Results

The Sept pre-email was the first contact in the series. Both Group A and Group C had a 0.05% response rate.

The Oct DM acquisition was sent to groups A and B – group A had received the pre-email and group B did not. Group A had a 1.03% response rate, while group B had a 0.98% response rate. By comparison, the overall mailing had a 0.86% response rate.

The Nov TM acquisition made 4,723 contacts and 286 members joined with an average gift of $27.83. Groups A and B had nearly identical results on the phone.

The Dec post-email was the last contact in the series. Group B had a 0.10% response rate, while Group C had 0.08%. The modeled groups had an average gift of $31.30. In addition, the non-modeled group had a $26.75 average gift.

Testing for Impact

Testing Carriers: Getting them into the envelope

Image vs. No Image

Plain Carrier Test

Tested against 4-color control

carrier

Plain vs. 4-Color Carrier

Teaser and Photo Testing

Testing Offers: Compelling them to Respond

Discount Offer Test

Added discount to carrier teaser/

changed PS discount

language from $3 to 10% discount

Discount offer

Ask string testing

JUNE: Control asking string was tested against descending asking string and askingstring with only three asks. In both cases, control is maintained with higher response rate and higher

gross/thousand.

Control

Test 1

Test 2

C3 vs. C4 DM & EM SURVEY APPEAL

JANUARY mailed annual survey in official-looking carrier with companion e-appeal. Messaging focused on state of our democracy.

TESTS: 501c3 version of appeal was tested.

DM: C3 test version was winner at 85% confidence with higher response rate (4.85% vs. 4.75%) and $1.46 higher average gift.

EM: C4 control had higher response rate (0.28% vs. ).24%), but $17.91 lower average gift. There was no statistical winner.

Testing Engagement Devices: Engaging Beyond Giving

A Petition as a Call to Action

The NPCA Mining Appeal exceeded budgeted revenue goal by 17% and net by 33%.

The drive to web test, with the URL listed on the reply and in the P.S of the letter earned more revenue due to a higher response rate and beat the control at 84% confidence.

Drive to Web Test

Testing Messaging

PILOT CAMPAIGN Package #1: From the Field/Mike Fay

PILOT CAMPAIGNPackage #2: RSVP/Invitation

Testing Tone and Language: Aggressive

Testing Tone and Language: Cooperative

Testing to Settle an Argument

Testing a Long-Standing Control

Matching Gift Appeal Test

A: Control

B: New Logo Test

C: Tagline Test

Logo Testing

2 p vs. 4 p

And Don’t Forget: Testing to Reduce Costs

Some of our Favorite Cost-Saving Tests:

Test 1st vs. 3rd Class postage

Test a modified or standard size or format

Test eliminating an insert

Test an E-R0 effort to invite people to renew online and not receive a DM renewal

The Analytics of Testing

• There are two phases to validating and analyzing direct marketing

testing. The first is ensuring that the test sample size is valid. • A sample size that is too small will not provide reliable results.

Part 1: Sample Size Calculator

Go to:

http://

www.avalonconsulting.net/

Sample-Size-Calculator.html

Recommendation: Perform this

calculation before the campaign is

conducted to guide the size of the

test and to ensure testing is

worthwhile.

• Step 1: How much do I need to mail?• Enter proposed volume for control or test

panels.• Step 2: Is there a valid percent return?

• Enter expected response (pre-campaign,

use a conservative estimate.)• Step 3: Is my confidence above 80%?

• If not, remove either quantity or response to

determine what values are needed.

Repeat with actual campaign results

if needed and if fluctuating from

proposed.

Part 1: Sample Size Calculator

Part 2: Testing Confidence Calculator

• After a valid sample size is achieved, the next step is to determine

if the results of the test are significant.

• Important: Just because there is a difference in performance

does not mean the performance can be replicated. Often, it is

within the margin of error – a flip of the coin.

• Only make decisions on valid results.

• Goal: Determine if the difference in net per thousand mailed

(other metrics can be used) is significant or within the margin

of error. Typically, a confidence level of 80% is a starting point,

with 95% being ideal.

Testing Confidence Calculator – Process

• Step 1: • Gather control and test panel quantities, gifts, revenue, and roll out costs.• Remove outlier gifts from quantity, gifts, and revenue. A typical rule of thumb is

$1,000 gifts or more.• Step 2:

• Calculate “upper bound” and “lower bound” for the response rate, revenue per

thousand, and/or net per thousand. It is typical to start with the standard normal

deviate of 1.96.• Step 3:

• If there is no overlap between the upper bound of one panel and the lower bound

of another, a confidence level will be found.

Upper LowerNet/M % # Bound Bound

Test Group Net/M Winner Conf. Conf. Net/M Net/M

Control $1,477 - $1,611 $1,343Test $1,826 WINNER $1,982 $1,670

95% 1.96

Thank you!

Allison PorterPresident

Avalon Consulting Group2030 M Street, NW Suite 700

Washington, DC 20036202-429-6080 x102

Allisonp@avalonconsulting.net

Follow Avalon on LinkedIn Today!

DM 201 - Avalon Consulting - Testing Creative

Marketing

The Avalon Group News Letter - Talking Avalon - Winter ... Avalon - Winter... · Page 3 Talking Avalon is changing Or more accurately, The Avalon Group is changing in 2016. As we

DM 201- MMI Direct - Data Processing by the Numbers

HemoCue Glucose 201 DM System - meddevicedepot.com€¦ · HemoCue® Glucose 201 DM System ... STAT test Duplicate sampling ... Venous EDTA samples measured on HemoCue® Glucose 201

DM 101 - Avalon Consulting Group - Online Fundraising

DM 201- Belardi/Ostroy - Trends, Metrics, Strategy, and Implementation

DM 201- Avalon Consulting + Donor Services Group - Multi-Channel Marketing

HemoCue Hb 201 DM Analyzer & HemoCue DM … · 1 Introduction 3 PART I 1 Introduction Note! Prior to starting the HemoCue Hb 201 DM Analyzer, read section 2 Safety, followed by section

@TOYOTA@ TOYOTA T-SB-0142-14 October 23, 201 4 Page 3 of 7 2015 Avalon/Avalon HV Pre-Delivery Service (PDS) Functional Operation Apply parking brake and cycle ignition "IG-ON," place

AVALON PRIVACY POLICY - Avalon Advisorsavalonadvisors.com/assets/avalon-advisors-llc-adv-part-2ab-2017.06... · AVALON PRIVACY POLICY ... Management LLC (“Avalon”) ... The Cynosure

Avalon Memory-Mapped Interface Specificationsedwards/classes/2007/4840/mnl_avalon_sp… · Avalon Memory-Mapped Interface Specification 1. Introduction The Avalon Memory-Mapped (Avalon-MM)

Avalon Ridgeview Neerana ! 9555599099 Avalon Neemrana ! Avalon New Project Neemrana

HemoCue Glucose 201 DM Analyzer · 2020. 8. 1. · 4.3 QC Test Procedure ... Thank you for choosing the HemoCue Glucose 201 DM system. The HemoCue Glucose 201 DM system is a compact,

doormats katalog enson SON · Size : 43x70cm • Piece weight : 1200gr • Box in pieces : 12 pcs • Box size : 0,028 m³ DOORMATS ORIENTALIST SERIES DM-201 DM-202 DM-203 DM-204

Sisterhood of Avalon - Week 3 - Avalon Family

DM-3551/DM-4551 DM DM DM DM DM DM D Network-Ready …files.sharpusa.com/.../MFPsPrinters/Brochures/cop_dow_DM3551_4551bro.pdf · DM DM DM DM DM DM DHigh Performance Features Do more

N Figure 2.1 9 · dm 20 37 dm 2037 dm 2037 dm 2037 dm 2037 dm 2037 dm 2037 dm 2037 dm 2037 dm 2037 dm 2037 dm 2037 dm 2037 dm 2037 dm 2037 dm 2037 dm 2037 dm 2037 dm 2037 dm 2037----7800

Avalon Advanced Materials Inc.avalonadvancedmaterials.com › ... › reports › ...Research_Report_Oct_… · Growing Lithium Market Avalon Advanced Materials Inc. (“Avalon”

AVALON MALL SITE PLAN · avalon mall site plan. avalon mall floor plan. avalon mall floor plan

Toyota avalon

AVALON -1 = Avalon - tommyemmanuel.files.wordpress.com€¦ · AVALON (CAAS 2009 - thumbpick - standard tuning-1 = 142) Avalon (live @ CAAS 2009) Tommy Emmanuel Transcribed by Daniele