1

Next-Generation Media Mix Modelling Identifies Creative and Long-Term Ad Revenue

TV Outdoor

Digital Print

2

Agenda

Background

Media Mix Model Development

More Queries Enabled

Implications

3

Goal

• Apply predictive modeling to determine the relative importance of three media effects for an HBA brand: – The impact and importance of media spend levels over the short-

term (<=1 year).– The impact of media creative (or message) in the short-term. – The importance and impact of advertising long-term effects, if any.

4

Advertising Testing Approach

To measure and monetize ad creative, need to:• Find a copy testing system that handles the

range of media channels and all ads:– TV, Radio, Digital, Online Video., Print, Out-of-

home, FSI • Provide results quickly enough to adapt

marketing plans• Include competitive benchmarking• And most importantly….

5

Defining Long-Term Ad Effects

Long-Term Ad Effects are the incremental sales generated from customers - who were initially acquired via media/advertising messages - over the lifetime of that customer. These sales are attributed through repeat purchases and revenue streams generated by these customers over their lifetime with a brand. Once a customer has been acquired, LTAE reflect the value these purchasers generated over their lifetime as a customer.

References:Leslie A. Wood, David F. Poltrack, Measuring the Long-Term Effects Of Television Advertising, Journal of Advertising Research, June 2015Ataman, Van Heerde, Mela, Building Brands, Marketing Science, Nov.-Dec, 2008Ataman, Van Heerde, Mela , The Long-term Effect of Marketing Strategy on Brand Performance, Emory University Zyman instititute of Brand Science, 3 July 2006

6

Inputs to the model for predicting weekly sales:Short-Term TV Media

Short-Term Digital Display & Video Ads

Short-Term Radio Media

Short-Term Print Media

Long-Term Ad Effects all Media

Retail SKU Distribution

Store Merchandising- Features

Store Merchandising-Displays

Store Merchandising-Features + Displays

Seasonality

Brand HBAWeekly Retail Sales by SKU

1/3/15 – 7/16/16

Retail SKU Price

Model Architecture for the HBA Brand

ABX Ad Copy Test Score, All Media

7

HBA Brand Retail Sales and Ad Spending

• For YE 7/16/16, HBA Brand generated $1.027 Billion in revenue and 106.4 million units, on $138 million of total ad spend

'2015.01

'2015.04

'2015.07

'2015.1

'2015.13

'2015.16

'2015.19

'2015.22

'2015.25

'2015.28

'2015.31

'2015.34

'2015.37

'2015.4

'2015.43

'2015.46

'2015.49

'2015.52

'2016.02

'2016.05

'2016.08

'2016.11

'2016.14

'2016.17

'2016.2

'2016.23

'2016.26 -

2,000,000

4,000,000

6,000,000

8,000,000

10,000,000

12,000,000

-

500,000

1,000,000

1,500,000

2,000,000

2,500,000

Brand HBA Unit Sales & Media Spend

Online Display Online Video PrintRadio TV Unit.Sales

Med

ia S

pend

Unit

Sale

s

8

1/5/2013

2/18/2

013

4/3/2013

5/17/2013

6/30/2

013

8/13/2

013

9/26/2

013

11/9/2013

12/23/2013

2/5/2014

3/21/2

014

5/4/2014

6/17/2014

7/31/2

014

9/13/2

014

10/27/2014

12/10/2014

1/23/2015

3/8/2015

4/21/2

015

6/4/2015

7/18/2015

8/31/2

015

10/14/2015

11/27/2015

1/10/2016

2/23/2

016

4/7/2016

5/21/2

016

7/4/2016

8/17/2

016

9/30/2

016 -

500,000

1,000,000

1,500,000

2,000,000

2,500,000

Actual and Model Sales HBA Brand

Model Actual Unit.Sales

Blind Holdout

Period of Fit

R2

Blind Holdout

R2 MAPEModel Stats. 95.5% 93.6% 0.5%

Model Development• Employed a 10% data holdout to validate the model’s ability to predict. • Overall results were excellent.

9

For the HBA Brand, long-term ad effects were among the highest we’ve seen at 15X that of the short-term media impact.

66.0%7.3%

1.8%0.5%

0.2%0.0%1.4%

22.7%

Incremental Contributions to Total HBA Brand Sales

Baseline Dist.@Min Mdsg.Ftr Mdsg.DisplMdsg.F + D ST Adv.Spend ST Adv.Creative Long-Term Adv

Incremental Sales Contributions

10

The HBA Brand grew sales 5.6% in the current quarter v. YA. Most of this growth can be ascribed to including long-term ad effects and better ad creative.

Baseline

Distbtn.@Min

Price

ST Adv.Spend

ST Adv.Creative

Long-Term Adv

Mdsg.Ftr

Mdsg.Displ

Mdsg.F + D

-2.0% -1.0% 0.0% 1.0% 2.0% 3.0% 4.0% 5.0%

1.1%

-1.1%

0.5%

0.0%

1.2%

3.8%

-0.1%

0.1%

0.0%

HBA Brand Current Quarter v. YA Variance Due-To

Current Quarter v YA Sales

11

The advertising return changed from -$100K to $2 .7MM with the inclusion of long-term effects.

Short-Term Effect Only With Long-Term Effects($500)

$0

$500

$1,000

$1,500

$2,000

$2,500

$3,000

($100)

$2,720

HBA Brand Annual Ad Net Returns

* Assumes 20% EBITDA/Sales Margin

Next-Level Media Mix ModellingEnables More Accurate Analysis

12

Next-level Media Marketing MixEnables More Queries

13

-40% -20% 0% 20% 40% 60% 80% -

5,000

10,000

15,000

20,000

25,000

R² = 0.128730325258725

Media Correlation to Sales

$ Sp

endi

ng p

er A

d (o

oo)

Next-Level Media Mix ModellingEnables More Queries

Question: Does spending per-ad predict incremental sales per ad?Learning: The level of spending behind this HBA Brand ad had little to

do with the sales uplift for each ad.

14

-40% -20% 0% 20% 40% 60% 80% -

20.00

40.00

60.00

80.00

100.00

120.00

140.00

160.00

R² = 0.434936643800919

Media Correlaton to Sales

FM A

BX A

ll C

TA

Next-Level Media Mix ModellingEnables More Queries

Question: Do individual ABX ad scores predict incremental sales per ad?Learning: ABX scores do correlate to individual ad sales uplifts.

Specifically, ABX “Call-to-Action” scores among Females (brand’s broad target) show significant (95%) correlation to ad sales performance.

15

1/3/2015

1/22/2

015

2/10/2

015

3/1/2015

3/20/2015

4/8/2015

4/27/2

015

5/16/2015

6/4/2015

6/23/2

015

7/12/2

015

7/31/2

015

8/19/2

015

9/7/2015

9/26/2

015

10/15/2015

11/3/2015

11/22/2015

12/11/2015

12/30/2015

1/18/2

016

2/6/2016

2/25/2

016

3/15/2

016

4/3/2016

4/22/2016

5/11/2

016

5/30/2016

6/18/2

016

7/7/2016

200,000

250,000

300,000

350,000

400,000

450,000

500,000

550,000

(10,000)

10,000

30,000

50,000

70,000

90,000

110,000

130,000

150,000

HBA Brand Sales due to Ad Creative & Long-Term Ad Effects

Long-Term Adv ST Adv.Creative

Long

-Ter

m A

d Eff

ects

Con

trbt

n

Ad C

reati

ve C

ontr

btn

Correlation = .6998

Question: Does improved ad creative impact long-term sales like it does short-term sales?

Learning: Improving ad creative boosted gains from Long-Term Ad Effects

Next-Level Media Mix ModellingEnables More Queries

1/5/2013

1/29/2

013

2/22/2

013

3/18/2

013

4/11/2

013

5/5/2013

5/29/2

013

6/22/2

013

7/16/2

013

8/9/2013

9/2/2013

9/26/2

013

10/20/2013

11/13/2013

12/7/2013

12/31/2013

1/24/2

014

2/17/2014

3/13/2

014

4/6/2014

4/30/2014

5/24/2

014

6/17/2

014

7/11/2014

8/4/2014

8/28/2

014

9/21/2014

10/15/2014

11/8/2014

12/2/2014

12/26/2014

1/19/2

015

2/12/2015

3/8/2015

4/1/2015

4/25/2015

5/19/2

015

6/12/2

015

7/6/2015

7/30/2

015

8/23/2

015

9/16/2015

10/10/2015

11/3/2015

11/27/2015

12/21/2015

1/14/2

016

2/7/2016

3/2/2016

3/26/2

016

4/19/2

016

5/13/2

016

6/6/2016

6/30/2

016

7/24/2

016

8/17/2

016

9/10/2

016

10/4/2016

-

500,000

1,000,000

1,500,000

2,000,000

2,500,000

Actual and Model Sales HBA Brand

Model Actual Unit.Sales

Model Validation

R2 Holdout R2 MAPETotal.Model 95.5% 93.6% 0.5%Removing ST AD Quality 86.3% 73.5% 2.3%

Holdout period

Model Validations

Adding ABX the model significantly improved that model

17

HUg

Implications

18

HBA Brand Implications

• Brand HBA’s current retail sales growth is quite strong, with over +5.6% gain versus the prior year.

• Gains were primarily driven by long-term ad effects and improved ad creative.

• The very high returns on Brand HBA suggests under-investment.

• The only negative for Brand HBA for the quarter was due to declining distribution and some SKU availability.

• Because ABX correlates to brand sales, allocating spend to ads according to ABX score significantly improves overall ad effectiveness

19

Incremental sales can be improved in nearly real-time by moving weaker ads out of rotation.

50 60 70 80 90 100 110 120 130 140 150 (8.0)

(6.0)

(4.0)

(2.0)

-

2.0

4.0

6.0

8.0

(6.3)

(5.1)

(3.8)

(2.6)

(1.3)

-

1.1

2.4

3.6

4.8

6.1

HBA Brand & ABX Incremental Dollars (Million)

Incremental Dollars Million

ABX Female Call-to-Action Score Index

incr

emen

tal R

etai

l $ M

il.

Media Mix Modelling Implications

20

Media Mix Modelling Implications• Long-Term Ad Effects are the key to positive ROI on advertising spending.

– The BA Brand had among the highest LTAE found: about 23% of annual sales; and was 15X the short-term ad effect.

• Ad creative effectiveness was a more significant driver of incremental sales than spend behind each ad for this HBA Brand.

• One ad test measure may be a proxy for overall creative quality for a brand.– Enables creative team to understand how the brand's ads drive sales

• Moving to an environment of higher creative effectiveness may amplify long-term effects:

Improved Ad Effectiveness

Improved Ad Rotation

Improved Investment Allocation

Jan-15 Feb-15 Mar-15 Apr-15 May-150

2

4

6

8

10

12

14

16

18

Exponential Growth Contribution of Long-term Effects

Long Term EffectsBaseline

reve

nue

Highest-scoring Print

Print Ad Image

Client, Product,Ad Claim Category, sub

category

Product shot, headline, claim with 2 support points