Embed Size (px)
Citation preview
How Do You Know…
Designing Multi-Channel
Campaign Tests That
Optimize Your Budget
Jim DeLash
November 18, 2014
One version had a Conversion Rate 24.6% higher
than the other one
Page layout
Moving registration to the right side form
increased the site’s Conversion Rate by 24.6%
Page layout
How do you know?
The right frequency
The best subject line
The optimal channel mix
Designing successful multi-
channel tests requires….
Establishing hypotheses
Developing test strategies by
segment
Creating a measurement plan
Establishing hypotheses
Statements to be proven
All tests must support a hypothesis
Extrapolating the results to other brands
Low unaided recall scores for
advertising creative will require
additional frequency….
Establishing hypotheses
Low unaided recall scores for
advertising creative will require
additional frequency….
Establishing hypotheses
Low unaided recall scores for
advertising creative will require
additional frequency….
BUT HOW MUCH MORE?
Establishing hypotheses
TEST…
More frequency in one channel
More frequency in multiple channels
Different levels of frequency
Establishing hypotheses
TEST…
More direct mail or e-mail
More frequency in multiple channels
Different levels of frequency
Establishing hypotheses
TEST…
More direct mail or e-mail
More direct mail and e-mail
Different levels of frequency
Establishing hypotheses
TEST…
More direct mail or e-mail
More direct mail and e-mail
10 15 20
Establishing hypotheses
TEST…
More frequency in one channel
More frequency in multiple channels
Different levels of frequency
Establishing hypotheses
The best design is not always the best approach
Do not let design conflate the objective of an email or message
Every unnecessary piece of content is waste and reduces your
chances of getting a click
Designing successful multi-
channel tests requires….
Establishing hypotheses
Developing test strategies by
segment
Creating a measurement plan
Developing Test Strategies by
Segment
Specialist or Generalist
Current User or Prospect
Decile
Awareness & Trial Offer
Developing Test Strategies by
Segment
Multiple
Brands Cadence
What will we learn from this test….
1. Best frequency by Specialty and Quintile
2. One channel vs. Two channels (Segments 5,7,9)
3. Total frequency: 10 vs. 12 vs. 14
Version B, the one with the ‘Save Now’ button, increased opt-ins
for logged-in users by 333% at a 99.9% confidence level.
Designing successful multi-
channel tests requires….
Establishing hypotheses
Developing test strategies by
segment
Creating a measurement plan
Control Groups
Measurement Milestones
Test Analysis
Creating a Measurement Plan
There are only two ways to get
fired from Harrah’s:
stealing from the company,
or
failing to include a proper
control group in your business
experiment
Control Groups…
Not a ‘holdout’ group
Must include all relevant variables
Large enough to be statistically valid
Creating a Measurement Plan
Measurement Milestones…
Track progress against goal
Forecast test impact early
Connect KPIs with Rx impact
Creating a Measurement Plan
Test Analysis…
Evaluate overall campaign impact
Both TRx and NRx
Creating a Measurement Plan
How can an e-mail test be
successful with a low open rate?
MICRO TESTS
MICRO TESTS
Version B, the subject line broken into three consumable
sections, increased opens by 73.7%.
Version B, the ‘Get My Free Paper’ Call to Action,
increased downloads by 25% at a 95% confidence level.
Version B, the ambiguous subject line, increased
opens by 112% at a 95% confidence rate.
Designing successful multi-
channel tests requires….
Establishing hypotheses
Developing test strategies by
segment
Creating a measurement plan
What is the optimal number of
samples to offer in a multi-channel
campaign?
So, how will you know?
This is the last slide
The Influence of Color
21.4% Conversion
Rate improvement
with the red button
The Influence of Color
Headline Copy
Headline Copy
32.3%
Conversion Rate
improvement
A Picture is Worth 1000 Words… But Which Picture? (Images)
A Picture is Worth 1000 Words… But Which Picture? (Images)
The female model in the header convinced 108%
more visitors to fill out and submit the lead gen
form
