Embed Size (px)
DESCRIPTION
Citation preview
Getting Testing Right: A Practical Guide to Testing in Direct Marketing IoF DM and Fundraising - March 2013
About me
Richard Hughes Marketing Data Team Manager Previously: Data Planner at Bluefrog Data Analyst at Good Agency / Cascaid Database Administrator at Crusaid (AIDS/HIV charity)
Objectives
• Talk through some of the finer points of testing
• the strategy side and the data techie side
• Both are really important!
• A brain dump of everything I’ve learnt about testing
• Advice on best practice
• Show that it can be exciting
• Inspire you to think about testing you can do
Why talk about testing?
– Not enough people talk about how to do it
– There is little info on the web for DM
– I’ve seen it go wrong
– Used well, testing can be very powerful, requires some thought and planning!
Definitions
Split testing, or A / B testing, is when an audience is split into two or more groups and
given different treatments in order to determine the most effective treatment
Why Test Anyway?
Why Test Anyway? How many
communications should we send throughout the
year?
How much should we ask our
supporters to donate?
Which creative should we choose?
Which Email Subject gets the best open rate?
Why Test Anyway? What stationery
types perform the best? More money on more expensive
packs?
What’s the best time to send a pack?
Who is the best signatory?
How do cash appeals affect regular giving attrition rates?
Marketing Triangle
Audience
Message Timing
Elements that affect
results
The Data Pyramid
Gut Reaction Versus Evidence
• Sometimes as experienced marketeers we know intuitively know the answers to some of these questions
• But we want to move to situation with where we make evidence based decisions
Concepts
Concepts
We are trying to find out if one approach is more likely to get better results than another
•Testing is affected by probability
–This means there is no guarantee that an approach will always “win”
–We can say that it is more likely to win and we can say how confident we are
Sampling Distribution
• When we test we …
– measure a sample of our audience and use that to generalise about the rest of the database
The results can be put into a bell curve
If we sample data from our database many times and treat in certain way we get a normal distribution
Two Curves
Mathematical properties of the curve means we can use stats to determine how likely test a and B are different
Stats Summary
• The response rate for each test is a normally distributed
• We want to measure the difference in performance between a given treatment and the control.
• The difference itself is a normally distributed random variable.
Structured Approach: Testing Life Cycle
Testing Strategy
Design Test (Tactics)
Execute Evaluate
Insight
Annual Testing Strategy
• Good testing starts with careful thinking
• Document what you want to find out
• Check and reflect on your questions
• Ensure that tests will deliver actionable results
Annual Testing Strategy
•Build scenarios to understand where you are going to get the best value
•Prioritise – focus on the best outcomes
For UNICEF, this means focusing on the outcome that brings the best result for children
Select tests that will have most impact, e.g. in mail packs, focus on outers rather than copy buried inside.
Cautionary Tales 1
• Testing can be expensive
– Paying for different creative
– Paying for different stationery to be printed
– Ring fencing certain supporters from different
comms is all expensive
• This is an important consideration when
thinking about the value of the test
Designing Tests: Sample Sizes
• Think about volume for your test
– You need sufficient quantity in your test
• The sample needs have enough volume to
be able to generalise about the population
Calculating Sample Sizes
Deep Dark Statistics:
• www.lucidview.com/sample_size.htm
• The most useful online resource that has
quite a technical explanation
Calculating Sample Sizes
• Two things determine sample size
– Existing Response Rate
• Low number of responders means we need a bigger sample
– Uplift of test
• Small uplift means we need a bigger sample to see if there is a difference
Sample Sizes – Worked Example
Take from http://www.testsignificance.com/
Testing more than one thing at once
• Need to be careful, but can split by more
than one test
Treatment A Treatment B Totals X & Y
Treatment X Segment 1 Segment 2
Treatment Y Segment 3 Segment 4
Totals for A & B
Cautionary Tales 2
• Be careful about testing too
many things in one campaign
– They can be difficult to manage
– Cause confusion evaluating
Selected the Data
• Once you you’ve decided on the volumes
the next task is to make sure you split the
data fairly
– This means selecting two or more samples,
ordering by factors that are important and
selecting alternate rows
– Do not take top / bottom half of spreadsheet
Coding
• This might be a no brainer but ensuring
the coding of A and B is set up right is
important
Evaluating
• We need to determin if two different results are significant
• This means showing that we are 95% confident there is a significant difference
• Quite a few websites that can help
Evaluating
• If we are testing prompt amounts in packs
we also need to test to see if the average
gift is significantly different
• We can use a T-test for this
Cautionary Tales 3
• Testing sometimes don’t tell us anything interesting
• This is a lesson in setting expectations
• Don’t say “we’re going to find out which is better”
• Instead say “We’re going to find out if there is any difference”
Don’t forget to focus on Net Income
Mailed Cost Response RR Income Net Average RoI
10000 £7,500 800 8% £14,400 £6,900 £18
1.92
10000 £11,000 1100 11% £19,800 £8,800 £18
1.80
Building Insight
• Understanding what your tests means for your programme
• Updating your strategy
Final Thoughts
• Testing is about making incremental improvements
• If you need more dramatic change then think about your overall fundraising strategy
• Make sure you do lots of planning
Summary
•What are your marketing questions?
•What are your priorities? Testing Strategy
•Calculate testing volume
•Split data fairly, Code data appropriately Test Design
•Mail, email, phone Execute
•Evaluate significance of results Evaluate
•Update documentation on your audience insights Build Insight
Any questions?
