Embed Size (px)
Citation preview
Case Studies: Setting Up and Conducting Tests That Make A Difference
Presented by: Branden Slattery, Marketing Director, CohereOne and Travis Seaton,
SVP Strategic Services, CohereOne
April 24-25, 2019
2Define The Problem
Observe/Research the Problem
Form a Hypothesis
Conduct the Experiment
Analyze the Results
Form a Conclusion
Retest
Scientific
Method
3Things to Keep in Mind When Testing
If possible, use a double control on the test group
Creating two identical panels (one control, one test), will help to remove
noise and provide valid results
Make sure the test groups are large enough to analyze.
Typically like to get ~100 orders for statistical significance
Test
10,000Control
10,000
Control Balance
80,000
Groups are
even size and
same makeup
4Things to Keep in Mind When Testing
When setting up test panels, code segments so that they stay in original
source in order to not lose statistical significance in performance
NO YESMerge Out Code Select Merge Out Code Control Code Control Code Free Ship Code $ Off
Coop Model #1 25,000 XX1 7,143 Coop Model #1 25,000 XX1A 7,143 XX1B 5,952 XXC 5,952 XXD 5,952
Coop Model #2 25,000 XX2 7,143 Coop Model #2 25,000 XX2A 7,143 XX2B 5,952 XXC 5,952 XXD 5,952
Coop Model #3 30,000 XX3 8,571 Coop Model #3 30,000 XX3A 8,571 XX3B 7,143 XX3C 7,143 XX3D 7,143
Coop Model #4 40,000 XX4 11,429 Coop Model #4 40,000 XX4A 11,429 XX4B 9,524 XX4C 9,524 XX4D 9,524
Coop Model #5 20,000 XX5 5,714 Coop Model #5 20,000 XX5A 5,714 XX5B 4,762 XX5C 4,762 XX5D 4,762
OSL #1 25,000 XX6 7,143 OSL #1 25,000 XX6A 7,143 XX6B 5,952 XX6C 5,952 XX6D 5,952
OSL #2 30,000 XX7 8,571 OSL #2 30,000 XX7A 8,571 XX7B 7,143 XX7C 7,143 XX7D 7,143
OSL #3 15,000 XX8 4,286 OSL #3 15,000 XX8A 4,286 XX8B 3,571 XX8C 3,571 XX8D 3,571
Control YY1 50,000 Control
Free Shipping YY2 50,000 Free Shipping
$ Off YY3 50,000 $ Off
Total 210,000 210,000 Total 210,000 60,000 50,000 50,000 50,000
5Things to Keep in Mind When Testing
When evaluating results it is vitally important to calculate actual COGS on
the test groups rather than a blanket estimate
Example:
Offer is 20% off orders over $99 with unique code
Redemption rate is only 50% on offer resulting in 10% blended discount
Identifies the offer as the most profitable solution
20% 40%
Qty Orders Sales Discount COGS Net Sales NS/B
Offer 50,000 350 $200,000 $40,000 $80,000 $80,000 $1.60
No Offer 50,000 245 $140,000 $56,000 $84,000 $1.68
10% 40%
Qty Orders Sales Discount COGS Net Sales NS/B
Offer 50,000 350 $200,000 $20,000 $80,000 $100,000 $2.00
No Offer 50,000 245 $140,000 $56,000 $84,000 $1.68
Cadence Testing
7Audience Contact Test Case StudyTest over 7 month period: 7 vs 3 vs 0 contacts to customer audiences
Test audiences held constant over entire time frame
Contribution per Customer is the main metric to evaluate
Seg 1-3 (0-12) Cust Qty Orders Sales Margin Cost Cont $/Customer Cont/Cust
All Drops 17,482 4,649 $596,764 $298,382 $67,306 $231,076 $34.14 $13.22
1 per Season 17,505 3,460 $448,027 $224,014 $28,883 $195,130 $25.59 $11.15
None - Holdout 17,422 2,382 $254,723 $127,361 $0 $127,361 $14.62 $7.31
Seg 4-6 (13-35) Cust Qty Orders Sales Margin Cost Cont $/Customer Cont/Cust
All Drops 25,438 3,084 $362,952 $181,476 $97,936 $83,540 $14.27 $3.28 Orders Time Frame 9/6/17 to 3/26/18
1 per Season 25,459 2,103 $282,980 $141,490 $42,007 $99,483 $11.12 $3.91 All Drops 9/6, 9/26, 10/24, 11/15,1/10,2/14, 3/14
None - Holdout 25,373 927 $90,495 $45,248 $0 $45,248 $3.57 $1.78 1 Per Season 9/6, 11/15, 2/14
Seg 7-10 (36+) Cust Qty Orders Sales Margin Cost Cont $/Customer Cont/Cust Margin 50%
All Drops 21,338 1,267 $149,437 $74,719 $82,151 ($7,433) $7.00 ($0.35) Cat Cost $0.55
1 per Season 21,290 942 $100,107 $50,053 $35,129 $14,925 $4.70 $0.70
None - Holdout 21,229 589 $59,306 $29,653 $0 $29,653 $2.79 $1.40
Total Cust Qty Orders Sales Margin Cost Cont $/Customer Cont/Cust
All Drops 64,258 9,000 $1,109,153 $554,577 $247,393 $307,183 $17.26 $4.78
1 per Season 64,254 6,505 $831,114 $415,557 $70,679 $309,538 $12.93 $4.82
None - Holdout 64,024 3,898 $404,525 $202,262 $0 $202,262 $6.32 $3.16
Catalog Incremental
9Catalog Incremental Case StudyHoldout groups created for each segment, rolled up to major segment
Difference between mail and hold to determine incremental performance
10Catalog Incremental Case StudyIncremental performance between campaigns and channels and seasonality
Sept/Oct campaign not driving retail sales only direct sales
Sept/Oct campaign lower seasonality on mail and hold audiences
Nov/Dec campaign drove high cross channel engagement
9/10-10/22 Mail Hold 11/5-12/16 Mail Hold
Qty 342,132 742,609 Qty 480,439 749,163
Customers who Ordered 56,822 90,850 Customers who Ordered 89,166 84,836
Cust Order Rate 17% 12% Cust Order Rate 19% 11%
Mail Hold Gain Mail Hold Gain
Direct $ Both $2.86 $3.18 -$0.32 Direct $ Both $41.94 $28.96 $12.98
Retail $ Both $6.64 $7.00 -$0.36 Retail $ Both $22.52 $18.43 $4.09
Total $ Both $9.30 $10.18 -$0.87 Total $ Both $64.46 $47.38 $17.07
Direct $ Only $137.65 $121.98 $15.67 Direct $ Only $152.26 $125.68 $26.58
Retail $ Only $34.75 $42.77 -$8.02 Retail $ Only $32.21 $37.68 -$5.46
Total $ $181.70 $174.92 $6.78 Total $ $248.93 $210.74 $38.19
Digital Scoring Testing
Scr 10+ Cust Qty Orders Sales Margin Cost Cont $/Customer Cont/Cust
All Drops 10,987 2,231 $244,696 $122,348 $42,300 $80,048 $22.27 $7.29
1 per Season 10,990 1,608 $171,401 $85,701 $18,134 $67,567 $15.60 $6.15
None - Holdout 10,936 1,203 $117,816 $58,908 $0 $58,908 $10.77 $5.39
Scr 5-9 Cust Qty Orders Sales Margin Cost Cont $/Customer Cont/Cust
All Drops 15,790 2,042 $254,617 $127,308 $60,792 $66,517 $16.13 $4.21
1 per Season 15,801 1,438 $184,465 $92,233 $26,072 $66,161 $11.67 $4.19
None - Holdout 15,732 981 $98,192 $49,096 $0 $49,096 $6.24 $3.12
Scr 0-4 Cust Qty Orders Sales Margin Cost Cont $/Customer Cont/Cust
All Drops 15,028 2,270 $304,705 $152,352 $57,858 $94,495 $20.28 $6.29
1 per Season 15,005 1,316 $161,253 $80,627 $24,758 $55,868 $10.75 $3.72
None - Holdout 14,932 775 $81,468 $40,734 $0 $40,734 $5.46 $2.73
Scr (1)-(3) Cust Qty Orders Sales Margin Cost Cont $/Customer Cont/Cust
All Drops 12,084 995 $121,053 $60,526 $46,523 $14,003 $10.02 $1.16
1 per Season 12,065 746 $100,104 $50,052 $19,907 $30,145 $8.30 $2.50
None - Holdout 12,068 329 $39,615 $19,807 $0 $19,807 $3.28 $1.64
Scr (4)-(10) Cust Qty Orders Sales Margin Cost Cont $/Customer Cont/Cust
All Drops 10,369 750 $100,325 $50,163 $39,921 $10,242 $9.68 $0.99
1 per Season 10,393 509 $99,939 $49,970 $17,148 $32,821 $9.62 $3.16
None - Holdout 10,356 288 $33,393 $16,697 $0 $16,697 $3.22 $1.61
Total Cust Qty Orders Sales Margin Cost Cont $/Customer Cont/Cust
All Drops 64,258 8,288 $1,025,395 $512,698 $247,393 $265,304 $15.96 $4.13
1 per Season 64,254 5,616 $717,163 $358,581 $106,019 $252,562 $11.16 $3.93
None - Holdout 64,024 3,576 $370,484 $185,242 $0 $185,242 $5.79 $2.89
12Digital Behavior Scoring Case StudyDetermine if high scoring digital
customers can support reduced
or no postal contacts
Score 10+ Uber Digital to
Negative score with no digital
behavior and low RFM score
All customer groups had positive
contribution with postal contacts
Negative score benefited from
fewer contacts due low
engagement characteristics
Segmentation Testing
14Housefile Scoring Case StudyTesting the efficacy of multiple scoring methodologies
RFM stays consistent with only the scoring segments changing
Easy procedure with little additional processing necessary
15Housefile Segmentation Case Study
Testing the efficacy of multiple segmentation methodologies
Methodology #1 and #2 run independently including mail selections
Secondary merge completed to identify cross over and incremental names
Efficacy of methodology is determined by performance of incremental names
Active Email Testing
17Email Segmentation Case StudyActive Email as a variable when all other RFM variables are equal
In all case the presence of an active email + catalog contact is more productive
Testing Promotions/Offers
19Promotion and Format Case Study
Three panel: No Promo, Dot Whack Promo, Clean Release Card Promo
Contribution per Order is most important metric
Accounts for loss of margin on promo
Accounts for additional marketing expense
No Offer: active, Dot Whack: lapsing, Clean Release: lapsed
20Promotion Case Study - Retargeting
Three offers that equally equate to 33% off and one that is equivalent to 10%
Contribution per Order is most important metric
Blended discount ~ 13% for $ off offer and .05% for free shipping on redemption
Middle offer hit sweet spot on AOV to drive highest performance
TestingTrigger Postcard
22Postal Retargeting Case StudyClient tested triggered postcard to supplement online marketing
When evaluating the overall campaign, looked like a loss
When evaluating at the audience level results differed
Roll out with customer retargeting and drop prospecting
23Postal Retargeting Case StudyClient tested triggered postcard program along side catalog program
Program saw positive results among the customer groups,
Negative results with prospects even with 62% response lift
24Postal Retargeting Case StudyTriggered postcard program to supplement sales in between catalog seasons
Program saw positive results among all audience groups
TestingBook Size
26Catalog Format Case StudyTested a big book vs a smaller book
Opportunity for smaller book in the mail to reduce catalog spend?
Combined universe, the big book was the clear winner
Contribution of $.90/book vs the smaller book at $.76/book
Group Mail Qty $/Book AOV Resp % Cont/Bk
Big Book Balance 1,277,132 2.74$ 58$ 4.70% 0.92$
Big Book Control 80,403 2.69$ 59$ 4.60% 0.90$
Small Book 98,485 2.30$ 57$ 4.10% 0.76$
Big vs Small -18% 17% 4% 12% 18%
Total Mailed Universe
27Catalog Format Case StudyFurther audience detail: Buyers and Prospects
The big book was the clear winner with both groups, generating more
contribution per book for each group
Group Mail Qty $/Book AOV Resp % Cont/Bk
Big Book Balance 541,790 1.01$ 54$ 1.85% (0.11)$
Big Book Control 34,204 1.01$ 57$ 1.77% (0.12)$
Small Book 35,401 0.61$ 47$ 1.28% (0.26)$
Big vs Small -3% 66% 21% 38% -54%
Prospects
Group Mail Qty $/Book AOV Resp % Cont/Bk
Big Book Balance 575,094 4.87$ 58$ 8.33% 2.20$
Big Book Control 35,891 4.81$ 59$ 8.15% 2.16$
Small Book 53,761 3.72$ 58$ 6.43% 1.61$
Big vs Small -33% 29% 2% 27% 34%
Buyers
TestingPaper Weight
29Case Study #1Tested a lighter weight paper versus what they were using for years
Split test up among customers and prospects
Both groups show that the lighter paper had a higher
contribution/book, so they were able to reduce their catalog costs and
get the same response as the heavier paper
Test Mail Qty $/Bk Resp % AOV Cost/Book Cont/Bk
Standard Paper 119,996 4.62$ 4.58% 101$ 0.787 1.52$
Test Paper 20,019 4.58$ 4.61% 100$ 0.633 1.66$
Variance -83% -1% 1% -1% -20% 9%
Test Mail Qty $/Bk Resp % AOV Cost/Book Cont/Bk
Standard Paper 89,952 1.60$ 1.90% 84$ 0.787 0.02$
Test Paper 20,016 1.71$ 2.10% 82$ 0.633 0.22$
Variance -78% 7% 11% -1% -20% 1333%
Customers Prospects
No double control!!
Branden Slattery
415.322.699031
Thank You!
Travis Seaton
657.363.8694
![Corrosion of Building Materials - eng.hokudai.ac.jp · Pressure is increased step by step up to 350 MPa ... Pore size [µm] Beschichtung A Porosität: 28.96 Beschichtung B 11.88 %](https://img.pdfslide.us/doc/110x75/5c80f96b09d3f2b9438bb403/corrosion-of-building-materials-eng-pressure-is-increased-step-by-step-up.jpg)
