Chapter 13

Perry D. Drake, Drake Direct Page 1

Perry D. Drake, Drake Direct Page 2

Chapter 13

Assessing Marketing Test Results

Objectives

Assessing marketing test results based on confidence intervals and hypothesis tests

The Central Limit Theorem How to set the confidence level of your tests Making a business decision based on the results of your tests Gross versus net Multiple comparisons Calculating breakeven response rates Facts regarding confidence intervals and hypothesis tests

Introduction - Why, What, How & When We Test

Why we Test Testing is the foundation upon which one builds and grows a

direct marketing firm.

With a database, names can be selected for certain treatments

and comparisons on the customer’s reaction to these treatments

made.

Based on these results, in conjunction with marketing cost and

revenue figures, the most profitable decision can be made.

Introduction -Why, What, How & When We Test (Continued)

What we Test The types of things most often tested by direct marketers include:

Lists Customer Segments Payment Terms Guarantees Premiums

Offers Creative Promotional Formats Outer Envelopes Copy


How we Test We test a new format, creative concept or offering by selecting a sample of names of interest.

This sample is a subset of customer records and a random selection from the universe of interest on the database.


How we Test (cont.)As mentioned back in Chapter 6, to ensure the test results are meaningful, the sample must be representative of the entire population of concern.

A representative sample is a sample truly reflecting the population of interest from which the direct marketer draws inferences.

For a sample to be representative, no member of the population of interest are purposely excluded.

To determine the effectiveness of a new format test sent to a specific segment of customers residing on the database, for example, the direct marketer cannot restrict the sample to only those names living in New York. Doing so will yield results only reflective of New Yorkers


How we Test (Cont.) The only exceptions to this rule should be:

Names also eliminated in roll-out such as DMA do not promotes, known frauds, credit risk accounts, etc.

Names recently test promoted for other marketing tests. States or cities such as DC known to have strict promotional

restrictions (especially important if the new test has not been fully reviewed by legal prior).


How we Test (Continued)In addition, the sample must also be drawn randomly or the test will yield biased and misleading results.

A random sample is one in which every member of the sample are equally likely to be chosen, ensuring a composition similar to that of the population.

To ensure a sample is randomly draw, many direct marketers utilize what is called “nth selects.”

To draw a random sample of 10,000 names from a database of 10,000,000 the direct marketer will begin by selecting one name on the database, choosing every 1,000th (10,000,000/10,000) name thereafter.


When we Test Depending on the product or service being offered, you may be well advised to take into consideration seasonality.

For example, a travel product offered in October will yield a much lower response than one offered in April when families are planning their summer vacations.

Unfortunately we are not always able to test during the same time of year as the roll-out will occur. Therefore, you are advised to determine seasonality adjustments factors based on historical information. For example, a travel product offered in October is know to equal 80% of the response rate when offered in April.

With this information, you can make appropriate adjustments to your forecasts.

Once tests are conducted, a marketing manager has two options available for analyzing the test results:

Hypothesis Tests Confidence Intervals

Test Analysis

Hypothesis Testing: This procedure will allow a marketing manager to determine if (a) the percent of favorable responses from a single test is significantly different from a certain value, or (b) if one test panel is significantly different from the control panel.

Confidence Intervals: If you are interested in knowing specifics regarding how much different your test result is from, for example, the control you will be required to construct a confidence interval. Confidence intervals will allow you to:

– determine a range in which the response rate is likely to fall in roll-out based on the sample results, or

– determine a range in which the difference in response rates between your test and control package truly lies based on the sample test results.

Test Analysis

Test Analysis/Confidence Intervals

There are two types of confidence intervals that can be created.

A confidence interval around a single test result

You will use this formula when interest revolves around assessing the results of a single outside list, new product/service, or new house customer segment test.

A confidence interval around the difference between two test results

You will use this formula when interest revolves around assessing the difference between a your control and new format, offer or creative tests.

Test Analysis/Confidence Intervals/A Single Test Response Rate

You conduct a new outside list test to a sample of size 10,000 names and receive a response rate of 5.5%.

Can you run to the bank with the 5.5% response rate?


Absolutely not!

Because you did not test the whole universe available, but only a sample, the response rate obtained is only an estimate. In fact, each time you conduct such a test you will get a different response rate.

Let’s say you ran not 1, but 10 different tests to the same outside list with the following results:

Test # % Resp. Test # % Resp 1 5.1 6 4.7 2 4.8 7 5.2 3 5.7 8 4.5 4 4.3 9 4.9 5 5.5 10 5.3

Test Analysis/Confidence Intervals/A Single Test Response Rate (Continued)

Every time you test a list, you will get a different response rate.

Some tests will yield results above the true response rate of the entire list and some below the response rate of the entire list.


So, how can we possibly make any decisions with any level of confidence about what we can expect to receive in roll-out given the fact that there is the potential for so much variation in our testing results?

The “Central Limit Theorem” is the answer!

Test Analysis/Confidence Intervals/The Central Limit Theorem

Let’s assume that instead of repeating the experiment 10 times we did it 1,000 times.

And, assume you tallied the response rates received and created a histogram (bar chart) of the response rates using the Chart Wizard feature in ExcelTM.


The tally of response rates might look as follows:


Test Response Rate Frequency3.75% - 4.00% 174.00% - 4.25% 734.25% - 4.50% 1254.50% - 4.75% 1704.75% - 5.25% 2265.25% - 5.50% 1745.50% - 5.75% 1225.75% - 6.00% 746.00% - 6.25% 19TOTAL 1,000

Test Response Rate Frequency3.75% - 4.00% 174.00% - 4.25% 734.25% - 4.50% 1254.50% - 4.75% 1704.75% - 5.25% 2265.25% - 5.50% 1745.50% - 5.75% 1225.75% - 6.00% 746.00% - 6.25% 19TOTAL 1,000

And, if we produce the histogram from this data using the Chart Wizard feature of Excel, it will look as follows:


Histogram of Response Rates

0

50

100

150

200

250

300


0

50

100

150

200

250

300

Voila! The symmetric bell shaped or “normal” curve is revealed.



0

50

100

150

200

250

300


0

50

100

150

200

250

300

By way of The Central Limit Theorem, the following has been proven*: The distributional shape of sample averages (or response rates as in our case) will closely

resemble a bell shaped and symmetric normal distribution. The average of these sample averages (or response rates) will approximate the true population

average (or response rate). And, the sample standard deviation** when divided by the number of observations in the

sample will approximate the true population standard deviation (or spread).

Note: This information is not found in the text book.

________________________

* These statements are only true for large sample sizes (to be revealed later) and becomes more true as the sample sizes approaches infinity.

** The standard deviation is nothing more than an average of the deviations of each observation from their mean. Simply speaking, it is a measure of the spread of the data. The larger this number for a particular set of data, the more “spread out” the data values are.


For any set of data that is distributed normally (having a symmetric bell shaped curve), the following can be said:

99.7% of the observations in the data set will lie within 3 standard deviations of the mean, 99% within 2.575 standard deviations of the mean, 95% within 1.96 standard deviations of the mean, 90% within 1.645 standard deviations of the mean, and 68% within 1 standard deviation of the mean.


And, we have just derived the confidence interval formula based on The Central Limit Theorem.


meanmean-1sd

For symmetric and bell-shaped distributions,

the mean falls exactly in the middle.

For symmetric and bell-shapeddistributions, 50% of the observations fall to the right side of the mean and the remaining

50% fall to the left side of the mean.

mean-3sd mean-2sd mean+3sdmean+1sd mean+2sd


Constructing a confidence interval will allow you, the marketer, to assess the likely range in which the true response rate will lie based on your test.

To calculate a confidence interval around a single test response rate, the following information is required.

The sample response rate p obtained from the test The sample size n of the test The desired confidence level c

And, n x p and n x (1 - p) must both be greater than or equal to 5.

^


To determine the confidence level c, you must answer the following question:

“How confident do I want to be that the interval I construct around my test response rate will contain the true response rate I can expect

to achieve in roll-out?”

Do you need to be 90% confident? 95%? 99%?


The value chosen for c will guarantee, with the same level of probability, that the interval constructed around the test response rate will contain the true population proportion you can expect to receive in roll-out.

It is recommended all confidence intervals be constructed with a 90% or better confidence level. Employing lower levels will yield more risk than you should be willing to take.


For example, an 80% confidence level implies there will be an 80% probability the true response rate to be expected in roll-out will fall within the constructed interval bounds.

But more importantly, it also implies there is a 20% probability (calculated as 1 - c) the true response rate to be expected in roll-out will not be continued in the constructed interval bounds. We call this value the “error rate” associated with the constructed confidence interval.


In simplistic terms, the confidence interval around your test response rate is constructed by adding and subtracting from it a multiple of the “sampling error.” The “multiple” will depend on the desired confidence level chosen.

The formula for the lower bound of the confidence interval is:

p - (z)

And, the upper bound:

p + (z)

( p )( 1 - p )/n^ ^^

The “sampling error” associated with the test response rate. Also called the standard deviation of the test.

The “multiplier” which equals 1.645, 1.96 and 2.575 for a 90%, 95% and 99% confidence level respectively.

^ ( p )( 1 - p )/n^ ^


Consider the following example:

Assume the marketing director at ACME Direct, a direct marketer of books, music, videos, home product catalogues and magazines, tested their core magazine title to a new outside list of names. The size of the test was 10,000. The test results of this new list yielded a response rate of 3.42%.

To assess the potential for this new list in roll-out, the marketing director decides to construct a confidence interval. In particular, she needs to calculate an interval around this sample response rate such that she can be 95% confident the true response rate she can expect in a full blown roll-out will fall within the bounds.


The marketing director will construct the lower bound of the 95% confidence interval as:

p - (z)

= .0342 - (1.96) (.0342)(1-.0342)/10,000

= .0342 - (1.96) (.0342)(.9658)/10,000

= .0342 - (1.96) .0000033

= .0342 - (1.96) (.0018165)

= .0342 - .00356

= .03064 or 3.06%

( p )( 1 - p )/n^ ^^


The marketing director will construct the upper bound of the 95% confidence interval as:

p + (z)

= .0342 + (1.96) (.0342)(1-.0342)/10,000

= .0342 + .00356

= .03776 or 3.78%

( p )( 1 - p )/n^ ^^


The marketing director can be 95% confident, should she decide to roll-out with this new outside list, the response rate will not be:

less than 3.06%, or more than 3.78%.

She can use the lower bound to determine the worse case scenario in terms of profitability by running a P&L calculation.

Based on her findings, she will decide whether or not to roll-out with this new outside list.


Setting the Confidence Level

At what percent should you set the confidence level of your interval?

In order to answer this question you must ask yourself the following question:

How much risk am I willing to take in making an incorrect decision of assuming the true response rate falls within the constructed interval, when in reality, it falls outside of

the interval?

How much risk am I willing to take in making an incorrect decision of assuming the true response rate falls within the constructed interval, when in reality, it falls outside of

the interval?


When you are constructing a confidence interval around a single test estimate to gauge a new list’s or new product’s potential, you will need to determine which of the following situations best fit your testing circumstances:

If the costs associated with a new list or new product are significantly higher in comparison to other lists or products currently being promoted, then there is a major risk associated with rolling-out with the new list or product if, in reality, it ends up performing badly. In other words, you really need to have a handle on the worse case scenario in term of response so set your confidence level high (95% or 99%).

And, by setting the confidence levels at a high rate (95% or 99%) you will minimize the chance of concluding the list or product test has met your requirements when, in reality, it did not.


If the costs associated with a new list or new product are similar to other lists or products currently being promoted, then there is certainly less risk associated with rolling-out when, in reality, it ends up performing badly. As a result, you will set your confidence level at “industry standard” levels (90% or 95%) because the risk in making an incorrect decision is not as high, relatively speaking, as in the first scenario.


If costs associated with a new list or new product are lower in comparison to other lists or products currently being promoted, then there is also less risk associated with rolling-out with the new list or product if, in reality, it ends up performing badly. Therefore, you will set your confidence level at “industry standard” levels (90% or 95%) since the risk in making an incorrect decision is also not as high as in the first scenario.


Let’s assume for our ACME Direct example that this new list is more expensive than others currently being used for prospecting. In particular, this list costs $175 per thousand versus an average of $150 per thousand for lists currently being used.

What confidence level should the marketing director use?


Interpreting the Confidence Interval

Once the confidence level has been determined and the confidence interval constructed, you are ready to interpret the results.

Be forewarned, the interpretation of the interval is not “black or white.” Business experience and knowledge plays a major role in the final interpretation and decision made. A confidence interval will not provide you with a definitive answer to a business questions, but rather provide you with valid best and worse case scenarios to consider.


For example, going back to our previous example the resulting 95% confidence interval was (3.06%, 3.78%). With 95% confidence, she knows the lowest this test list will respond in roll-out is 3.06% and the highest is 3.78%.

In order to assess whether or not to roll-out with this new list, she will first examine if a slightly less aggressive confidence interval (90% in this case) reveals a different action be taken. If the lower bounds of both intervals reveal her “profit” criteria are met, then the answer is easy.


If the two intervals are telling her conflicting information, she will need to do further analysis. In particular:

First, determine how close her “go/no-go” response rate level for the outside lists is to the lower bounds of both intervals constructed. Next, consider the true upside and downside potential of a business decision by performing profit calculations using the upper and lower bounds of both confidence intervals. What are the worse and best case profit scenarios telling her?


Going back to our example, we determined a 95% confidence interval was the appropriate level, so let’s construct a slightly less aggressive interval at 90% and see what that is telling us.

Lower bound:

p - (z)

= .0342 - (1.645) (.0342)(1-.0342)/10,000

= .0342 - (1.645) (.0018165)

= .0342 - .00299

= .03121 or 3.12%

( p )( 1 - p )/n^ ^^


Upper bound:

p + (z)

= .0342 + (1.645) (.0342)(1-.0342)/10,000

= .0342 + (1.645) (.0018165)

= .0342 + .00299

= .03719 or 3.72%

( p )( 1 - p )/n^ ^^


So, we now have the following two intervals:

95% = (3.06%, 3.78%) A slightly less aggressive interval at 90% = (3.12%, 3.72%)

Question: If we assume the minimum response rate the marketing director can accept for this list is 3.00% (which relates to a lose per paid sub of $25 or less), do you advise she roll-out with this new list to larger quantities?

Question: What if the minimum response rate of 3.10% was required?

Question: What if the minimum response rate of 3.35% was required?


She can also run a P&L on the lower and upper bounds of both the 90% and 95% confidence intervals. For the chart below she assumed a projected payment rate adjusted a bit given experience.

Question: If she is currently only pursing lists that cost no more than $22 per paid sub what should she do here?

Question: What if she was willing to spend as much as $25 per paid sub, but no more?

Question: What if she was willing to spend as much as $30 per paid sub, but no more?

Resp Rate

Mail

Quantity

Observed

Pay Rate Net Rate

List Cost

Per 1,000

Prom Cost

Per 1,000

Avg Billing

Costs Per

Order

BRE Costs

Per Order

Average

Copy Cost

Per Bad Debt

Rev Per

Order

Total

Revenue Total Costs

Net Rev

(P&L)

P&L Per Net

Order

Observ ed Point Estimate 0.0342 10,000 50% 0.0171 $175 $360 $2.67 $0.38 $1.41 $14.97 $2,559.87 $6,634.21 ($4,074.34) ($23.83)

Low er Bound @ 95% 0.0306 10,000 50% 0.0153 $175 $360 $2.67 $0.38 $1.41 $14.97 $2,290.41 $6,499.03 ($4,208.62) ($27.51)

Upper Bound @ 95% 0.0378 10,000 50% 0.0189 $175 $360 $2.67 $0.38 $1.41 $14.97 $2,829.33 $6,769.39 ($3,940.06) ($20.85)

Low er Bound @ 90% 0.0312 10,000 50% 0.0156 $175 $360 $2.67 $0.38 $1.41 $14.97 $2,335.32 $6,521.56 ($4,186.24) ($26.83)

Upper Bound @ 90% 0.0372 10,000 50% 0.0186 $175 $360 $2.67 $0.38 $1.41 $14.97 $2,784.42 $6,746.86 ($3,962.44) ($21.30)


Luckily, you do not need to use the Luckily, you do not need to use the complicated formula previously given to calculate complicated formula previously given to calculate

confidence intervals.confidence intervals.

With the help of With the help of The Plan-alyzerThe Plan-alyzer, a software package , a software package created by Drake Direct, you can easily create created by Drake Direct, you can easily create

confidence intervals around a single test response rateconfidence intervals around a single test response rate

Test Analysis/Confidence Intervals/The Difference Between Two Test Response Rates

When interest revolves around determining if one test has beaten another, you will be interested in examining the difference in response rates.

ACME Direct conducts a new format test against the control format for their core magazine title based on samples of size 10,000 each. You receive a 5.85% response rate for the new format test and a 5.45% response rate for the control format.

Can you go to the bank assuming the new format has beaten the control?

Test Analysis/Confidence Intervals/The Difference Between Two Test Response Rates

Absolutely not!

Because both results are based on samples, each will have a certain amount of associated “sampling error.” The true difference is not .40% but something more or less than this percent.

A confidence interval constructed around the difference between the two test response rates will allow you to determine the range in which the true difference actually lies by taking into account the amount or error associated with both tests.

To calculate a confidence interval around the difference between two test response rates, the following information is required.

The sample response rates p1 and p

2 for both tests

The sample sizes n1 and n

2 for both tests

The desired confidence level c

And, n1 x p

1, n

1 x (1 - p

1), n

2 x p

2 and n

2 x (1 – p

2), must all be greater than or

equal to 5.

Test Analysis/Confidence Intervals/The Difference Between Two Test Response Rates (Continued)

^ ^

In simplistic terms, you construct the confidence interval around the difference in your test response rates by adding and subtracting from it a multiple of the “sampling error” associated with the difference in response rates. The “multiple,” as before, will depend on the desired confidence level chosen.

The formula for the lower bound of the confidence interval is:

( p1 - p2 ) - (z)

And, the upper bound:

( p1 - p2 ) + (z)

^

The “sampling error” associated with the difference in test response rates.

The “multiplier” which equals 1.645, 1.96 and 2.575 for a 90%, 95% and 99% confidence level respectively.


(( p1)(1 - p1)/n1) + (( p2)(1 - p2)/n2)^ ^ ^ ^^

^ ^ (( p1)(1 - p1)/n1) + (( p2)(1 - p2)/n2)^ ^ ^ ^


Assume the marketing manager at ACME Direct has conducted a new and more expensive direct mail format test. The results of the new format test and the current control format are shown below.

The marketing manager determined the new format test must yield 2 additional responses per 1,000 names mailed in order to cover its incremental costs. In other words, the new format must beat the control by at least 0.20% (two-tenths of one percentage point) in response.

In order to properly assess the value of the new format test, the marketing manager will construct a 95% confidence interval around the observed difference in test response rates.


Numberof Customers

Mailed

Number ofCustomers who

Responded Response RateControl Format 9,978 348 3.49%

New Format Test 10,002 416 4.16%

The marketing manager will construct the lower bound of the 95% confidence interval as:

( p1 - p2 ) - (z)

= (.0416 - .0349) - (1.96) ((.0416)(1-.0416)/10,002) + ((.0349)(1-.0349)/9,978))

= .0067 - (1.96) (.0000040 + .0000034)

= .0067 - (1.96)(.0027)

= .0067 - .0053

= .0014 or 0.14%


(( p1)(1 - p1)/n1) + (( p2)(1 - p2)/n2)^ ^ ^ ^^ ^

The marketing manager will construct the upper bound of the 95% confidence interval as:

( p1 - p2 ) + (z)

= (.0416 - .0349) + (1.96) ((.0416)(1-.0416)/10,002) + ((.0349)(1-.0349)/9,978))

= .0067 + .0053

= .0120 or 1.20%


(( p1)(1 - p1)/n1) + (( p2)(1 - p2)/n2)^ ^ ^ ^^ ^

With 95% confidence, the new format test is guaranteed to outperform the current format by anywhere from 0.14% to 1.20%.

Since the lower bound of this difference is less than the minimum required difference of 0.20%, the marketing manager should not change to the new format.

However, the marketing manager can feel confident that the new format has in fact beaten the control since “zero” is not contained in the interval.



What if the marketing manager had instead constructed a 90% confidence interval?

The lower bound would be:

= (.0416 - .0349) - (1.645) ((.0416)(1-.0416)/10,002) + ((.0349)(1-.0349)/9,978))

= .0023 or 0.23%

And, the upper bound would be:

= (.0416 - .0349) + (1.645) ((.0416)(1-.0416)/10,002) + ((.0349)(1-.0349)/9,978))

= .0111 or 1.11%

Notice how the decrease in the confidence level has caused the interval to lessen in width. The lower bound (0.23%) now exceeds the required minimum difference in response rates (0.20%) required to cover the incremental costs associated with the new format.

Based on this confidence interval, the marketing manager can feel confident that the new format test will outperform the control by the required minimum.


Should the marketing manager base her decision on the results of the 95% or 90% confidence interval? She will come to two totally different conclusions depending upon her choice.

We will answer this question shortly.


Setting the Confidence Level

As was the case with confidence intervals for a single test response rate, you must also ask yourself the following question:

How much risk am I willing to take in making How much risk am I willing to take in making an incorrect decision of assuming the difference in response an incorrect decision of assuming the difference in response

rates falls within the constructed interval, when in reality, rates falls within the constructed interval, when in reality, it falls outside of the interval? it falls outside of the interval?



To illustrate the process of determining the confidence level to use when interest revolves around the difference between two test results, reconsider the example in which the marketing manager was faced with not knowing whether or not to use a 90% or 95% confidence level.

Her decision will be based on the amount of risk she is willing to take in the decision she makes. She has three options to choose from as shown on the next slide.


If the costs associated with the new format are significantly higher than the costs associated with the control format, there is a major risk in changing to a new format if, in reality, it ends up performing worse or the same as the control format. In this case, if the new format’s results are no better or worse than the control format, erroneously changing to the new format will yield an increase in promotional costs with a zero to negative change in the overall response rate. Therefore you want to have a very good handle on the true difference you can expect in response rates. To minimize the chance of concluding the test format has outperformed the control format when, in reality, it did not you set the confidence level high (95% or 99%). Therefore, she should set the confidence level at 95% or 99%.


If the costs associated with both promotional formats are similar, there is certainly less risk if she decides to change to the new format when, in reality, it performs worse than the control format. In this case, if the new format’s results are worse than the control format, erroneously changing to the new format will yield a negative change in the overall response rate but leave promotional costs unchanged (unlike the first scenario). Since the risk in making an incorrect decision is not as high, relatively speaking, as in the first scenario we can lower the required confidence level. As a result, she can keep the confidence level at “industry standard” levels (90% or 95%).


If the costs associated with the new format are lower than the costs associated with the control format, there is also less risk if she decides to change to the new format when, in reality, it ends up performing worse than the control format. In this case, erroneously changing to a new format will yield a negative change in the overall response rate but this time promotional costs will decline offsetting this fact. Since, relatively speaking, the risk in making an incorrect decision is not as high as in the first scenario again the confidence level can be set at a less aggressive level when compared to scenario one. She can keep the confidence level at “industry standard” levels (90% or 95%).


If we assume, for our example, that the test format is only moderately more expensive that the control format, a 95% confidence interval seems appropriate.

Since this 95% confidence interval reveals that the difference can be as low as .14%, which is lower than the required minimum to break-even, the marketing manager is advised to not change to the new format.

Or should she change to the new format?


Interpreting the Confidence Interval

Remember, the interpretation of the interval is not “black or white.” A confidence interval will not provide you with a definitive answer to a business questions, but rather provide you with valid best and worse case scenarios to consider.


When comparing two tests and examining the difference in their response rates, you will want to closely examine the true upside and downside potential of your business decision.

The steps for assessing the difference between two response rates are basically the same as that for single estimates with minor exceptions:

First, examine if a slightly less aggressive confidence interval reveals a different action be taken. If the two confidence intervals are telling you the same action be taken, the answer is easy.


If the two confidence intervals are telling you conflicting information, you will need to do further analysis (as in our example). In particular:

First, determine how close the difference you are interested in is to the lower and upper bounds of both intervals constructed. Next, consider the true upside and downside potential of a business decision by performing profit calculations using the upper and lower bounds of both confidence intervals. What are the worse and best case profit scenarios telling you?


Reconsidering the example presented, the marketing manager was faced with a major problem:

She decided a 95% confidence interval was the appropriate choice The 95% confidence interval suggested she maintain the current control format. She constructed a slightly less aggressive 90% confidence interval which suggested she should change to the new format.

The marketing manager now needs to closely examine the bounds of both confidence intervals in terms of what they are telling her.


What does the marketing manager know?

The lower bound of the 95% confidence interval (.0014) is less than, but very close to, the required minimum of .0020. The lower bound of a less aggressive 90% confidence interval (.0023) is above the required minimum of .0020. The new format has definitely beaten the control since the difference in response rates is guaranteed to be greater than “zero” based on both the 90% and 95% confidence intervals (the lower bounds of both exceed 0.00%). The upside potential is quite high for both confidence intervals - both show the new format can exceed the control format by over 1% in response. And, the upside potential (greater than a 1% difference in response) appears to far exceed that of the downside potential (only a .14% difference in response) for both confidence intervals calculated.


Given this information, do you recommendGiven this information, do you recommend

that the marketing manager switch to the new format?that the marketing manager switch to the new format?


Had the required difference in response ratesHad the required difference in response rates

been 0.30% or higher rather than 0.20%, all else the same, been 0.30% or higher rather than 0.20%, all else the same, would you still recommend to the marketing managerwould you still recommend to the marketing manager

that she switch to the new format?that she switch to the new format?


Luckily, you do not need to use the complicated formula previously given to calculate this

confidence interval either.

With the help of The Plan-alyzer, a software package created by Drake Direct, you can easily create confidence intervals around the difference

between two test response rates

Test Analysis/Confidence Intervals/Facts

Regardless of the amount of time spent planning a test, the results of a confidence interval are only valid if nothing outside of your control occurs from the time of the test to roll-out which may impact your test universe in some way.

Various scenarios which could negatively impact the likelihood that the true response rate obtained in roll-out will not fall with the bound of your confidence interval include:

the timing of the test vs. roll-out (e.g. test was conducted in May with a roll-out the following January) a major competitor emerged on the market after the test


Scenarios continued include:

increased competition in the mail box from the time of the test bad press regarding direct marketing practices since the time of the test and before roll-out a major natural disaster in a specific region of the country since the time of the test and before roll-out changes in the promotional package or offer prior roll-out economic recession or boom


It is important to keep in mind the following regarding the creation of any confidence interval:

If you want to be more confident in your estimates, the resulting interval will (wider or tighter). If you increase your sample size of the test, the resulting confidence interval will become (wider or tighter). The more accuracy you will need in your test estimate, the (lower or higher) you should set your confidence live. The confidence level you set should depend on the risk you are willing to take in making an incorrect decision.

Test Analysis/Confidence Intervals

You can also construct confidence intervals around sample You can also construct confidence intervals around sample averages and the differences between averages. You can read averages and the differences between averages. You can read about these confidence intervals in Chapter 13, pages 253-256 about these confidence intervals in Chapter 13, pages 253-256

and pages 258-260.and pages 258-260.

Test Analysis/Hypothesis Testing

Hypothesis testing is another approach to assessing test results but are not as revealing as confidence intervals.

A hypothesis test will simply answer a yes/no question such as:

Can I conclude my new format test has a different response rate from the control format?

A hypothesis test will only answer this question for you with a “yes” or a “no.” It will not tell you anything more.

Test Analysis/Hypothesis Testing (Continued)

What you will be taught today is how to conduct a hypothesis test based on the results of the confidence interval only. It is quite simple:

If the value of zero is contained in the interval then you may conclude there is no difference in the response rates.

If the value of zero is not contained in the interval, then you can conclude there is a difference in response rates.

For more information on learning how to properly set up and conduct hypothesis tests, read Chapter 13, pages 268-280

Test Analysis/Hypothesis Testing (Continued)

Consider the following example previously discussed:

Assume the marketing manager at ACME Direct has conducted a new and more expensive direct mail format test. The results of the new format test and the current control format are shown below.

The marketing manager wishes to determine if the new format test result is identical to the control by conducting a hypothesis test with 95% confidence.

Numberof Customers

Mailed

Number ofCustomers who

Responded Response RateControl Format 9,978 348 3.49%

New Format Test 10,002 416 4.16%

Using the results of the previously calculated 95% confidence interval (0.14%, 1.20%), he can conduct a hypothesis test to answer the following question:

Can I assume the test format hasa different response rate from the control with 95% confidence?

Since zero is not contained in the interval, she will answer “yes” and conclude that the test format is different from the control with 95% confidence. And, because the test response rate is higher than the control she can actually say that it has beaten the control with 95% confidence.

What about 99%? Wouldn’t it be great if the Marketing Manager could not only tell her boss the new format has beaten the control with 95% confidence but also with 99% confidence? That would be even better, right?


Many times you may hear about the “p-value” associated with a hypothesis test. A “p-value” tells you the highest level of confidence you could possibly consider and still say the test and control formats are different.

Suppose our Marketing Manager wanted to determine just this for her boss. That is, she wants to determine the highest level of confidence she can consider and still conclude the test format has beaten the control.

How does she go about doing this?


She might begin by first constructing a 90% confidence interval and seeing if it contains zero. If it does not, she would then construct a 95% confidence interval and examine to see if it contains zero. If if does not contain zero she will then construct a 99% confidence interval and examine to see if it contains zero. If if does contain zero she will then perhaps construct a 98% confidence interval and examine to see if it contains zero. If it does not contain zero she will then know that the highest level of confidence she can possibly consider and still conclude the test has beaten the control is 98%.

This is the “p-value” of the hypothesis test. Well, actually it is one minus the “p-value.”


The Plan-alyzer will calculate the “p-value” associated with a hypothesis test for the difference in two test response rates.

Doing so for the data presented three slides earlier we will obtain an exact “p-value” of .0138 for the question posed:

Can I conclude the test format has beaten the control format in terms of response rate?

One minus the “p-value” is the confidence level for which you cannot exceed if you want to answer the above question with a “yes.”

With this information, the Marketing Manager can now tell her boss that the test has beaten the control with 98.62% confidence. A pretty strong statement about the test format! One cannot be much more certain than this about a test result.


Question: What if instead she ran a hypothesis test and the resulting “p-value” was .3476 – what does this tell her?

Question: What if instead she ran a hypothesis test and the resulting “p-value” was .0001 – what does this tell her?


Determining Breakeven

Two common types of breakeven calculations

conducted by marketing managers are:

1. The response rate required for a new list or product

test in order to break-even.

2. The increase in response required for a new and more expensive format or creative test versus the control in order to break-even.

Determining Breakeven (Continued)

Break-even for a new list test

The break-even response rate for a list test is the response required to ensure that costs are exactly offset by revenues. No profit or loss results in a break-even situation.

Break-even for a list occurs when:

Mail Quantity x Response Rate x Profit per Response =

Mail Quantity x Promotion Cost per Piece

Solving for the break-even response rate, we have the following formula:

(Promotion Cost Per Piece) / (Profit Per Response Prior Promotion Costs)



Promotion cost per piece = $0.875

Profit prior promotion costs per response = $75.00

Break-even = ($0.875) / ($75.00)

= 1.17%

Therefore, all the marketer requires to at least cover all promotional costs is a response rate of 1.17% or greater.


Increase in response required to break-even

The break-even response rate when contemplating a new and more expensive format is the increase in response required to ensure the profit or loss generated by the new format equals that of the current control format. No change in the “profit” picture occurs in such a situation.

This occurs when:

(Mail Qty x Test Resp Rate x Profit per Resp) – (Mail Qty x Test Promotion Cost per Piece) =

(Mail Qty x Control Resp Rate x Profit per Resp) – (Mail Qty x Control Promotion Cost per Piece) =

Solving for the test response rate that must be achieve to ensure a similar profit or loss generated by the control format yields the following formula:

Control Resp Rate – [(Test – Control Prom Cost Per Piece) / (Profit Per Response Prior Promotion Costs)]



A marketer sells vitamins by mail to his house file. The control format costs $500 per 1,000 names mailed and generates a 5.00% response rates. He is testing the addition of sample vitamins to the control package. This new package will cost an additional $200 per 1,000 names mailed. If the profit before promotion per order is $20, then what response rate must the marketer achieve on the test in order to break-even versus the control package?

Break-even = 5.00% + [($0.70 - $0.50) / ($20)]

= 5.00% + [$0.20 / $20]

= 5.00% + 1.00%

= 6.00%

Therefore, the marketer must obtain at least a 6.00% response rate on the new test in order to guarantee no change in net revenue versus the control. In other words, the

marketer must observe at least a 1 percentage point increase in response to break even.

Gross Versus Net

When your marketing tests have performance implications, you will want to examine net response rather than gross. This includes: Pricing tests Soft versus hard offer tests Payment/Installment option tests

You can further analyze these test for significant payment differences but keep in mind that if your sample sizes are not large enough you will never be able to substantiate any difference unless it is quite large - a topic we will discuss shortly.

Gross Versus Net (Continued)

Because of this fact, when wishing to run best and worse case P&L calculations on your testing results as we did earlier (see below) you will only calculate a confidence interval on gross and simply use the observed payment rate (or an average).

What Not To Do: Calculate a confidence interval around gross response and a confidence interval around the payment rate. Then take the lower bound of both and the upper bound of both and run best and worse case P&L calculations.

Resp Rate

Mail

Quantity

Observed

Pay Rate Net Rate

List Cost

Per 1,000

Prom Cost

Per 1,000

Avg Billing

Costs Per

Order

BRE Costs

Per Order

Average

Copy Cost

Per Bad Debt

Rev Per

Order

Total

Revenue Total Costs

Net Rev

(P&L)

P&L Per Net

Order

Observ ed Point Estimate 0.0342 10,000 50% 0.0171 $175 $360 $2.67 $0.38 $1.41 $14.97 $2,559.87 $6,634.21 ($4,074.34) ($23.83)

Low er Bound @ 95% 0.0306 10,000 50% 0.0153 $175 $360 $2.67 $0.38 $1.41 $14.97 $2,290.41 $6,499.03 ($4,208.62) ($27.51)

Upper Bound @ 95% 0.0378 10,000 50% 0.0189 $175 $360 $2.67 $0.38 $1.41 $14.97 $2,829.33 $6,769.39 ($3,940.06) ($20.85)

Low er Bound @ 90% 0.0312 10,000 50% 0.0156 $175 $360 $2.67 $0.38 $1.41 $14.97 $2,335.32 $6,521.56 ($4,186.24) ($26.83)

Upper Bound @ 90% 0.0372 10,000 50% 0.0186 $175 $360 $2.67 $0.38 $1.41 $14.97 $2,784.42 $6,746.86 ($3,962.44) ($21.30)

Question:Question:

Do you think a test of 10,000 names yielding Do you think a test of 10,000 names yielding a response rate of 1% will have more or less error a response rate of 1% will have more or less error

associated with it than a test of the same size associated with it than a test of the same size yielding a response rate of 25%?yielding a response rate of 25%?

The Biggest Misconception Regarding Test Response Rates:The higher the response rate the more reliable the estimate.

The Truth:Not necessarily. The closer the response rate is to 50% the less reliable it will be, all else being equal.

Test Analysis/The Biggest Misconception

Believe it or not, a test response rate of 25% will have more error surrounding it than a test response rate of only 1%, all else the same.

And, a test renewal rate of 80% will have less error surrounding it than a test renewal rate of 55% (as 55% is closer to 50% than 80%).


To help explain this phenomenon, consider the following two experiments (1) two people toss a coin 10 times each and observe the number of heads vs. (2) two people roll a die 10 times each and observe the number of sixes.

In the first experiment it would not seem too far fetched for one person to get 3 heads and another to get 7 heads. A difference in the number of successes equal to 4 – a large spread.

However, in the second experiment, it would seem a bit far fetched for one person to get 3 sixes and another to get 7 sixes. Just not realistic. More than likely each will roll 1, 2 or 3 sixes at most – a more narrow spread.

When the probability of success and failure are close to being equal, you will have much more variance in your experiments (i.e., the test mailings).


Documents

Chapter 13