Case Studies: Setting Up and Conducting Tests That Make A ...€¦ · Mail Hold Gain Mail Hold Gain...

Case Studies: Setting Up and Conducting Tests That Make A Difference

Presented by: Branden Slattery, Marketing Director, CohereOne and Travis Seaton,

SVP Strategic Services, CohereOne

April 24-25, 2019

2Define The Problem

Observe/Research the Problem

Form a Hypothesis

Conduct the Experiment

Analyze the Results

Form a Conclusion

Retest

Scientific

Method

3Things to Keep in Mind When Testing

If possible, use a double control on the test group

Creating two identical panels (one control, one test), will help to remove

noise and provide valid results

Make sure the test groups are large enough to analyze.

Typically like to get ~100 orders for statistical significance

Test

10,000Control

10,000

Control Balance

80,000

Groups are

even size and

same makeup

4Things to Keep in Mind When Testing

When setting up test panels, code segments so that they stay in original

source in order to not lose statistical significance in performance

NO YESMerge Out Code Select Merge Out Code Control Code Control Code Free Ship Code $ Off

Coop Model #1 25,000 XX1 7,143 Coop Model #1 25,000 XX1A 7,143 XX1B 5,952 XXC 5,952 XXD 5,952

Coop Model #2 25,000 XX2 7,143 Coop Model #2 25,000 XX2A 7,143 XX2B 5,952 XXC 5,952 XXD 5,952

Coop Model #3 30,000 XX3 8,571 Coop Model #3 30,000 XX3A 8,571 XX3B 7,143 XX3C 7,143 XX3D 7,143

Coop Model #4 40,000 XX4 11,429 Coop Model #4 40,000 XX4A 11,429 XX4B 9,524 XX4C 9,524 XX4D 9,524

Coop Model #5 20,000 XX5 5,714 Coop Model #5 20,000 XX5A 5,714 XX5B 4,762 XX5C 4,762 XX5D 4,762

OSL #1 25,000 XX6 7,143 OSL #1 25,000 XX6A 7,143 XX6B 5,952 XX6C 5,952 XX6D 5,952

OSL #2 30,000 XX7 8,571 OSL #2 30,000 XX7A 8,571 XX7B 7,143 XX7C 7,143 XX7D 7,143

OSL #3 15,000 XX8 4,286 OSL #3 15,000 XX8A 4,286 XX8B 3,571 XX8C 3,571 XX8D 3,571

Control YY1 50,000 Control

Free Shipping YY2 50,000 Free Shipping

$ Off YY3 50,000 $ Off

Total 210,000 210,000 Total 210,000 60,000 50,000 50,000 50,000

5Things to Keep in Mind When Testing

When evaluating results it is vitally important to calculate actual COGS on

the test groups rather than a blanket estimate

Example:

Offer is 20% off orders over $99 with unique code

Redemption rate is only 50% on offer resulting in 10% blended discount

Identifies the offer as the most profitable solution

20% 40%

Qty Orders Sales Discount COGS Net Sales NS/B

Offer 50,000 350 $200,000 $40,000 $80,000 $80,000 $1.60

No Offer 50,000 245 $140,000 $56,000 $84,000 $1.68

10% 40%

Qty Orders Sales Discount COGS Net Sales NS/B

Offer 50,000 350 $200,000 $20,000 $80,000 $100,000 $2.00

No Offer 50,000 245 $140,000 $56,000 $84,000 $1.68

Cadence Testing

7Audience Contact Test Case StudyTest over 7 month period: 7 vs 3 vs 0 contacts to customer audiences

Test audiences held constant over entire time frame

Contribution per Customer is the main metric to evaluate

Seg 1-3 (0-12) Cust Qty Orders Sales Margin Cost Cont $/Customer Cont/Cust

All Drops 17,482 4,649 $596,764 $298,382 $67,306 $231,076 $34.14 $13.22

1 per Season 17,505 3,460 $448,027 $224,014 $28,883 $195,130 $25.59 $11.15

None - Holdout 17,422 2,382 $254,723 $127,361 $0 $127,361 $14.62 $7.31

Seg 4-6 (13-35) Cust Qty Orders Sales Margin Cost Cont $/Customer Cont/Cust

All Drops 25,438 3,084 $362,952 $181,476 $97,936 $83,540 $14.27 $3.28 Orders Time Frame 9/6/17 to 3/26/18

1 per Season 25,459 2,103 $282,980 $141,490 $42,007 $99,483 $11.12 $3.91 All Drops 9/6, 9/26, 10/24, 11/15,1/10,2/14, 3/14

None - Holdout 25,373 927 $90,495 $45,248 $0 $45,248 $3.57 $1.78 1 Per Season 9/6, 11/15, 2/14

Seg 7-10 (36+) Cust Qty Orders Sales Margin Cost Cont $/Customer Cont/Cust Margin 50%

All Drops 21,338 1,267 $149,437 $74,719 $82,151 ($7,433) $7.00 ($0.35) Cat Cost $0.55

1 per Season 21,290 942 $100,107 $50,053 $35,129 $14,925 $4.70 $0.70

None - Holdout 21,229 589 $59,306 $29,653 $0 $29,653 $2.79 $1.40

Total Cust Qty Orders Sales Margin Cost Cont $/Customer Cont/Cust

All Drops 64,258 9,000 $1,109,153 $554,577 $247,393 $307,183 $17.26 $4.78

1 per Season 64,254 6,505 $831,114 $415,557 $70,679 $309,538 $12.93 $4.82

None - Holdout 64,024 3,898 $404,525 $202,262 $0 $202,262 $6.32 $3.16

Catalog Incremental

9Catalog Incremental Case StudyHoldout groups created for each segment, rolled up to major segment

Difference between mail and hold to determine incremental performance

10Catalog Incremental Case StudyIncremental performance between campaigns and channels and seasonality

Sept/Oct campaign not driving retail sales only direct sales

Sept/Oct campaign lower seasonality on mail and hold audiences

Nov/Dec campaign drove high cross channel engagement

9/10-10/22 Mail Hold 11/5-12/16 Mail Hold

Qty 342,132 742,609 Qty 480,439 749,163

Customers who Ordered 56,822 90,850 Customers who Ordered 89,166 84,836

Cust Order Rate 17% 12% Cust Order Rate 19% 11%

Mail Hold Gain Mail Hold Gain

Direct $ Both $2.86 $3.18 -$0.32 Direct $ Both $41.94 $28.96 $12.98

Retail $ Both $6.64 $7.00 -$0.36 Retail $ Both $22.52 $18.43 $4.09

Total $ Both $9.30 $10.18 -$0.87 Total $ Both $64.46 $47.38 $17.07

Direct $ Only $137.65 $121.98 $15.67 Direct $ Only $152.26 $125.68 $26.58

Retail $ Only $34.75 $42.77 -$8.02 Retail $ Only $32.21 $37.68 -$5.46

Total $ $181.70 $174.92 $6.78 Total $ $248.93 $210.74 $38.19

Digital Scoring Testing

Scr 10+ Cust Qty Orders Sales Margin Cost Cont $/Customer Cont/Cust

All Drops 10,987 2,231 $244,696 $122,348 $42,300 $80,048 $22.27 $7.29

1 per Season 10,990 1,608 $171,401 $85,701 $18,134 $67,567 $15.60 $6.15

None - Holdout 10,936 1,203 $117,816 $58,908 $0 $58,908 $10.77 $5.39

Scr 5-9 Cust Qty Orders Sales Margin Cost Cont $/Customer Cont/Cust

All Drops 15,790 2,042 $254,617 $127,308 $60,792 $66,517 $16.13 $4.21

1 per Season 15,801 1,438 $184,465 $92,233 $26,072 $66,161 $11.67 $4.19

None - Holdout 15,732 981 $98,192 $49,096 $0 $49,096 $6.24 $3.12

Scr 0-4 Cust Qty Orders Sales Margin Cost Cont $/Customer Cont/Cust

All Drops 15,028 2,270 $304,705 $152,352 $57,858 $94,495 $20.28 $6.29

1 per Season 15,005 1,316 $161,253 $80,627 $24,758 $55,868 $10.75 $3.72

None - Holdout 14,932 775 $81,468 $40,734 $0 $40,734 $5.46 $2.73

Scr (1)-(3) Cust Qty Orders Sales Margin Cost Cont $/Customer Cont/Cust

All Drops 12,084 995 $121,053 $60,526 $46,523 $14,003 $10.02 $1.16

1 per Season 12,065 746 $100,104 $50,052 $19,907 $30,145 $8.30 $2.50

None - Holdout 12,068 329 $39,615 $19,807 $0 $19,807 $3.28 $1.64

Scr (4)-(10) Cust Qty Orders Sales Margin Cost Cont $/Customer Cont/Cust

All Drops 10,369 750 $100,325 $50,163 $39,921 $10,242 $9.68 $0.99

1 per Season 10,393 509 $99,939 $49,970 $17,148 $32,821 $9.62 $3.16

None - Holdout 10,356 288 $33,393 $16,697 $0 $16,697 $3.22 $1.61

Total Cust Qty Orders Sales Margin Cost Cont $/Customer Cont/Cust

All Drops 64,258 8,288 $1,025,395 $512,698 $247,393 $265,304 $15.96 $4.13

1 per Season 64,254 5,616 $717,163 $358,581 $106,019 $252,562 $11.16 $3.93

None - Holdout 64,024 3,576 $370,484 $185,242 $0 $185,242 $5.79 $2.89

12Digital Behavior Scoring Case StudyDetermine if high scoring digital

customers can support reduced

or no postal contacts

Score 10+ Uber Digital to

Negative score with no digital

behavior and low RFM score

All customer groups had positive

contribution with postal contacts

Negative score benefited from

fewer contacts due low

engagement characteristics

Segmentation Testing

14Housefile Scoring Case StudyTesting the efficacy of multiple scoring methodologies

RFM stays consistent with only the scoring segments changing

Easy procedure with little additional processing necessary

15Housefile Segmentation Case Study

Testing the efficacy of multiple segmentation methodologies

Methodology #1 and #2 run independently including mail selections

Secondary merge completed to identify cross over and incremental names

Efficacy of methodology is determined by performance of incremental names

Active Email Testing

17Email Segmentation Case StudyActive Email as a variable when all other RFM variables are equal

In all case the presence of an active email + catalog contact is more productive

Testing Promotions/Offers

19Promotion and Format Case Study

Three panel: No Promo, Dot Whack Promo, Clean Release Card Promo

Contribution per Order is most important metric

Accounts for loss of margin on promo

Accounts for additional marketing expense

No Offer: active, Dot Whack: lapsing, Clean Release: lapsed

20Promotion Case Study - Retargeting

Three offers that equally equate to 33% off and one that is equivalent to 10%

Contribution per Order is most important metric

Blended discount ~ 13% for $ off offer and .05% for free shipping on redemption

Middle offer hit sweet spot on AOV to drive highest performance

TestingTrigger Postcard

22Postal Retargeting Case StudyClient tested triggered postcard to supplement online marketing

When evaluating the overall campaign, looked like a loss

When evaluating at the audience level results differed

Roll out with customer retargeting and drop prospecting

23Postal Retargeting Case StudyClient tested triggered postcard program along side catalog program

Program saw positive results among the customer groups,

Negative results with prospects even with 62% response lift

24Postal Retargeting Case StudyTriggered postcard program to supplement sales in between catalog seasons

Program saw positive results among all audience groups

TestingBook Size

26Catalog Format Case StudyTested a big book vs a smaller book

Opportunity for smaller book in the mail to reduce catalog spend?

Combined universe, the big book was the clear winner

Contribution of $.90/book vs the smaller book at $.76/book

Group Mail Qty $/Book AOV Resp % Cont/Bk

Big Book Balance 1,277,132 2.74$ 58$ 4.70% 0.92$

Big Book Control 80,403 2.69$ 59$ 4.60% 0.90$

Small Book 98,485 2.30$ 57$ 4.10% 0.76$

Big vs Small -18% 17% 4% 12% 18%

Total Mailed Universe

27Catalog Format Case StudyFurther audience detail: Buyers and Prospects

The big book was the clear winner with both groups, generating more

contribution per book for each group

Group Mail Qty $/Book AOV Resp % Cont/Bk

Big Book Balance 541,790 1.01$ 54$ 1.85% (0.11)$

Big Book Control 34,204 1.01$ 57$ 1.77% (0.12)$

Small Book 35,401 0.61$ 47$ 1.28% (0.26)$

Big vs Small -3% 66% 21% 38% -54%

Prospects

Group Mail Qty $/Book AOV Resp % Cont/Bk

Big Book Balance 575,094 4.87$ 58$ 8.33% 2.20$

Big Book Control 35,891 4.81$ 59$ 8.15% 2.16$

Small Book 53,761 3.72$ 58$ 6.43% 1.61$

Big vs Small -33% 29% 2% 27% 34%

Buyers

TestingPaper Weight

29Case Study #1Tested a lighter weight paper versus what they were using for years

Split test up among customers and prospects

Both groups show that the lighter paper had a higher

contribution/book, so they were able to reduce their catalog costs and

get the same response as the heavier paper

Test Mail Qty $/Bk Resp % AOV Cost/Book Cont/Bk

Standard Paper 119,996 4.62$ 4.58% 101$ 0.787 1.52$

Test Paper 20,019 4.58$ 4.61% 100$ 0.633 1.66$

Variance -83% -1% 1% -1% -20% 9%

Test Mail Qty $/Bk Resp % AOV Cost/Book Cont/Bk

Standard Paper 89,952 1.60$ 1.90% 84$ 0.787 0.02$

Test Paper 20,016 1.71$ 2.10% 82$ 0.633 0.22$

Variance -78% 7% 11% -1% -20% 1333%

Customers Prospects

No double control!!

Branden Slattery

bslattery@cohereone.com

415.322.699031

Thank You!

Travis Seaton

tseaton@cohereone.com

657.363.8694