Cim & brilliant media introduction to econometrics

Maximising returns on yourcommunications investment

Marketing measurement and econometrics

1st November 2011

We know that we need to measure marketing

• To increase its effectiveness

• To reduce risk

• To justify the marketing budget

But marketing measurement is hard

• Marketing doesn’t always work quickly

• The effects are often not felt immediately

• So we end up not being sure if it’s working at all

Week-to-week sales movements are affected by many factors other than marketing

• Short-term sales movements due to advertising are difficult to pull apart from other factors

Distribution change (+30%)

Promotions and discounts

Seasonality

Weather

Above the line advertising

Competitor activity

Price change (+10%)

Random ‘noise’

5-10%

30-50%

30%

10-20%

5-6%

3- 5%

1-2%

~ 1%

Scale of weekly sales movements typically measured by an FMCG model

The full impact of marketing is also not felt immediately

0

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

0

20

40

60

80

100

120

140

07-J

an-0

828

-Jan

-08

18-F

eb-0

810

-Mar

-08

31-M

ar-0

821

-Apr

-08

12-M

ay-0

802

-Jun

-08

23-J

un-0

814

-Jul

-08

04-A

ug-0

825

-Aug

-08

15-S

ep-0

806

-Oct

-08

27-O

ct-0

817

-Nov

-08

08-D

ec-0

829

-Dec

-08

19-J

an-0

909

-Feb

-09

02-M

ar-0

923

-Mar

-09

13-A

pr-0

904

-May

-09

25-M

ay-0

915

-Jun

-09

06-J

ul-0

927

-Jul

-09

17-A

ug-0

907

-Sep

-09

28-S

ep-0

919

-Oct

-09

09-N

ov-0

930

-Nov

-09

21-D

ec-0

911

-Jan

-10

01-F

eb-1

022

-Feb

-10

15-M

ar-1

005

-Apr

-10

26-A

pr-1

017

-May

-10

07-J

un-1

028

-Jun

-10

19-J

ul-1

009

-Aug

-10

30-A

ug-1

020

-Sep

-10

11-O

ct-1

001

-Nov

-10

22-N

ov-1

013

-Dec

-10

03-J

an-1

124

-Jan

-11

14-F

eb-1

107

-Mar

-11

28-M

ar-1

118

-Apr

-11

09-M

ay-1

130

-May

-11

Indexed Impact on Sales

TVRs

Yorkshire TVRs Shape of Impact on Sales (Index)

0

20

40

60

80

100

120

140

160

180W

eek

1W

eek

2W

eek

3W

eek

4W

eek

5W

eek

6W

eek

7W

eek

8W

eek

9W

eek

10W

eek

11W

eek

12W

eek

13W

eek

14W

eek

15W

eek

16W

eek

17W

eek

18W

eek

19W

eek

20W

eek

21W

eek

22W

eek

23W

eek

24W

eek

25W

eek

26W

eek

27W

eek

28W

eek

29W

eek

30W

eek

31W

eek

32W

eek

33W

eek

34W

eek

35W

eek

36W

eek

37W

eek

38W

eek

39W

eek

40W

eek

41W

eek

42W

eek

43W

eek

44W

eek

45W

eek

46W

eek

47W

eek

48W

eek

49W

eek

50W

eek

51W

eek

52

TVRs

/ S

ales

Impa

ct

TVRs Effect on Sales

Illustrated impact of TV airtime on sales

Continuous TV airtime and shape of impact on sales (Brilliant Media client example)

0.00

50.00

100.00

150.00

200.00

250.00

01-O

ct-1

005

-Oct

-10

09-O

ct-1

013

-Oct

-10

17-O

ct-1

021

-Oct

-10

25-O

ct-1

029

-Oct

-10

02-N

ov-1

006

-Nov

-10

10-N

ov-1

014

-Nov

-10

18-N

ov-1

022

-Nov

-10

26-N

ov-1

030

-Nov

-10

04-D

ec-1

008

-Dec

-10

12-D

ec-1

016

-Dec

-10

20-D

ec-1

024

-Dec

-10

28-D

ec-1

001

-Jan

-11

05-J

an-1

109

-Jan

-11

13-J

an-1

117

-Jan

-11

21-J

an-1

125

-Jan

-11

29-J

an-1

102

-Feb

-11

06-F

eb-1

110

-Feb

-11

14-F

eb-1

118

-Feb

-11

22-F

eb-1

126

-Feb

-11

02-M

ar-1

106

-Mar

-11

10-M

ar-1

114

-Mar

-11

18-M

ar-1

122

-Mar

-11

26-M

ar-1

130

-Mar

-11

03-A

pr-1

107

-Apr

-11

11-A

pr-1

115

-Apr

-11

19-A

pr-1

123

-Apr

-11

27-A

pr-1

1

New

Cus

tom

ers

Immediate TV Contribution TV Carryover contribution

New Customers

The effect of a TV burst can last well beyond the timing of the spots

Example of a highly impactful TV burst

Trying to identify marketing impact by looking for increases in sales, works occasionally

Year on year sales. Client was looking for the reason that sales increased from June 2011

-15%

-10%

-5%

0%

5%

10%

15%

20%

25%

30%

35%

03-J

an-1

1

10-J

an-1

1

17-J

an-1

1

24-J

an-1

1

31-J

an-1

1

07-F

eb-1

1

14-F

eb-1

1

21-F

eb-1

1

28-F

eb-1

1

07-M

ar-1

1

14-M

ar-1

1

21-M

ar-1

1

28-M

ar-1

1

04-A

pr-1

1

11-A

pr-1

1

18-A

pr-1

1

25-A

pr-1

1

02-M

ay-1

1

09-M

ay-1

1

16-M

ay-1

1

23-M

ay-1

1

30-M

ay-1

1

06-J

un-1

1

13-J

un-1

1

20-J

un-1

1

27-J

un-1

1

04-J

ul-1

1

11-J

ul-1

1

18-J

ul-1

1

25-J

ul-1

1

01-A

ug-1

1

08-A

ug-1

1

15-A

ug-1

1

22-A

ug-1

1

29-A

ug-1

1

Year

on

Year

Cha

nge

Unfortunately, more often it leads to confusion.Sales increased here, after TV was returned to normal.

-40%

-20%

0%

20%

40%

60%

80%

100%

03-J

an-1

1

10-J

an-1

1

17-J

an-1

1

24-J

an-1

1

31-J

an-1

1

07-F

eb-1

1

14-F

eb-1

1

21-F

eb-1

1

28-F

eb-1

1

07-M

ar-1

1

14-M

ar-1

1

21-M

ar-1

1

28-M

ar-1

1

04-A

pr-1

1

11-A

pr-1

1

18-A

pr-1

1

25-A

pr-1

1

02-M

ay-1

1

09-M

ay-1

1

16-M

ay-1

1

23-M

ay-1

1

30-M

ay-1

1

06-J

un-1

1

13-J

un-1

1

20-J

un-1

1

27-J

un-1

1

04-J

ul-1

1

11-J

ul-1

1

18-J

ul-1

1

25-J

ul-1

1

01-A

ug-1

1

08-A

ug-1

1

15-A

ug-1

1

22-A

ug-1

1

29-A

ug-1

1

Year

on

Year

Cha

nge

Sales Sales Year on YearTV Ratings Year on Year

Year on year sales. Client was looking for the reason that sales increased from June 2011

Weekly sales tracking that tries to pin down marketing, is rarely effective unless marketing uplifts are very large

• Tracking can lead to a singular focus on trying to explain the previous week’s sales

– Trying to explain away the movements that aren’t marketing is very difficult

– Often we end up blaming everything on the weather

A weekly retail sales tracking dashboard with a singular focus on weather (Brilliant Media client example)

Total sales TY 2,534.42 3,347.83 2,321.96 2,384.24 2,320.83 2,623.11 3,429.65Total sales LY 2,167.29 3,138.10 3,061.70 3,110.99 3,446.56 3,739.47 4,942.17Total budget sales 2,194.87 3,357.76 3,249.18 3,275.26 3,657.56 3,955.02 5,119.55

Sales vs budget 15.5% -0.3% -28.5% -27.2% -36.5% -33.7% -33.0%Sales vs LY 16.9% 6.7% -24.2% -23.4% -32.7% -29.9% -30.6%

2011 Weather

Temp 7.1°C 6.7°C 6.1°C 5.6°C 5.9°C 5.9°C 5.7°C

2010 Weather

We can achieve a lot, without complex statistics

• Track what we can measure

– Take care not to only spend on what we can track…

• Acknowledge the issues with marketing measurement

Direct response tracking completes a part of the picture

EconometricsReturn on investment

Budget allocationBudget setting

Sales forecastingTest and learn

Direct response trackingOptimisation within a marketing

channel, including:Colour vs. B&W

Ad sizeNewspaper titles

Web display placement

Consumer ResearchBrand tracking

Brand perceptionCreative tuningTarget audienceSegmentation

Competitor benchmarks

Direct response has problems, but it’s a good step for advertisers with a product that’s suited to being tracked

• Several mechanics allow us to track response– Bespoke numbers– Direct mail– Competitions– ‘Where did you hear about us?’

• Compares within channels only

– What about brand TV, or if your brand has a memorable telephone number that you don’t want to change?

– Many brands – such as FMCG - have no response mechanic

Direct response data lets us optimise within press, search or other channels with response metrics

• Database technology makes this type of reporting quick and relatively easy– Tool for both agencies and clients

‘Brand’ might be driving a lot of your direct activity

• True cost per acquisition is a combination of brand and direct

– Most advertisers don’t analyse to this depth (yet)

Large numbers of search clicks can be driven by TV (Brilliant Media client example)

0 10,000 20,000 30,000 40,000 50,000 60,000

CPC

Organic

CPC

Organic

CPC

Organic

CPC

Organic

Prod

uct 1

Prod

uct 2

Chea

pBr

and

Number of clicks

Base Driven by TV

But if you’re not careful, it can all get a little complicated

There’s a simple rule of thumb that avoids a lot of brand measurement issues

• If direct channels (including search) bring in sales at a cost lower than TV, then they’re making your marketing more efficient

– TV generates the interest in your product, whether you run search ads to convert it, or not

• But we’ll need econometrics to find out the cost per acquisition from TV

Econometrics solves some of our direct response measurement issues

WARC case studies incorporating econometrics

1

4

5

14

16

18

20

20

27

29

33

33

39

43

54

67

146

Tobacco

Utilities and services

Business and industrial

Wearing apparel

Govt. and non-profit

Motor and auto

Media and publishing

Travel, transport and tourism

Leisure and entertainment

Telecomms

Financial services

Toiletries and cosmetics

Household and domestic

Pharmaceutical and healthcare

Drink and beverage

Retail

Food

Number of WARC case studies referencing econometrics (to October 2011)

Econometrics adds a new set of information, that we can’t get from direct response tracking alone











Econometrics measures and then improves the effectiveness of advertising

Econometrics uses statistical models of sales to…

– Measure the effectiveness (return on investment) of past advertising campaigns

– Split marketing campaigns into their individual parts(TV, radio, outdoor etc.) and measure the effectiveness of each part of the marketingmix

– Forecast the effectiveness offuture advertising campaigns

– Use forecasts to produce amore effective marketing mix

-50

0

50

100

150

200

250

Wee

k 1

Wee

k 4

Wee

k 7

Wee

k 10

Wee

k 13

Wee

k 16

Wee

k 19

Wee

k 22

Wee

k 25

Wee

k 28

Wee

k 31

Wee

k 34

Wee

k 37

Wee

k 40

Wee

k 43

Wee

k 46

Wee

k 49

Wee

k 52

Wee

k 55

Wee

k 58

Wee

k 61

Wee

k 64

Wee

k 67

Wee

k 70

Wee

k 73

Wee

k 76

Wee

k 79

Wee

k 82

Wee

k 85

Wee

k 88

Wee

k 91

Wee

k 94

Wee

k 97

Wee

k 10

0W

eek

103

Wee

k 10

6W

eek

109

Wee

k 11

2W

eek

115

Wee

k 11

8W

eek

121

Wee

k 12

4W

eek

127

Wee

k 13

0W

eek

133

Wee

k 13

6W

eek

139

Wee

k 14

2W

eek

145

Wee

k 14

8W

eek

151

Wee

k 15

4

Sale

s (£

'000

s)

Press RadioTV Store OpeningsSeasonality ActualModel

In marketing, econometrics usually means…

• Proving the effectiveness of advertising in driving sales

• Measuring return on investment (ROI)

• Building a mathematical model of two to three years of historical sales data

• Concentrating hardest on major above the line spends

• Aiming to produce a more efficient media budget allocation

A wider definition is much more useful

Econometrics is a toolbox that helps you to test theories about

your marketing

“”

What’s the output?

1. Measurement of past advertising campaigns, split into the different media channels that were used.

Proof that past advertising added to sales and (hopefully!) was profitable

Return on investment calculations showing the individual profitability of each marketing channel

2. Forecasting and improvement of future campaigns

The really useful bit and why it’s worth investing in econometrics

We can use the model to forecast the effectiveness of potential media schedules and then choose the one with the highest returns.

How a (standard) model actually works

• The maths that goes into a model is complicated…

• But you really don’t need to understand it, to get a feeling for how econometrics works

tiitii

ti

i

ti TTP

P

S

S )(21

We start with a sales history; two to three years of weekly sales data

-50

0

50

100

150

200

250

Wee

k 1

Wee

k 4

Wee

k 7

Wee

k 10

Wee

k 13

Wee

k 16

Wee

k 19

Wee

k 22

Wee

k 25

Wee

k 28

Wee

k 31

Wee

k 34

Wee

k 37

Wee

k 40

Wee

k 43

Wee

k 46

Wee

k 49

Wee

k 52

Wee

k 55

Wee

k 58

Wee

k 61

Wee

k 64

Wee

k 67

Wee

k 70

Wee

k 73

Wee

k 76

Wee

k 79

Wee

k 82

Wee

k 85

Wee

k 88

Wee

k 91

Wee

k 94

Wee

k 97

Wee

k 10

0W

eek

103

Wee

k 10

6W

eek

109

Wee

k 11

2W

eek

115

Wee

k 11

8W

eek

121

Wee

k 12

4W

eek

127

Wee

k 13

0W

eek

133

Wee

k 13

6W

eek

139

Wee

k 14

2W

eek

145

Wee

k 14

8W

eek

151

Wee

k 15

4

Sale

s (£

'000

s)

Sales

and a model, that at this stage doesn’t know anything at all about sales movements

-50

0

50

100

150

200

250

Wee

k 1

Wee

k 4

Wee

k 7

Wee

k 10

Wee

k 13

Wee

k 16

Wee

k 19

Wee

k 22

Wee

k 25

Wee

k 28

Wee

k 31

Wee

k 34

Wee

k 37

Wee

k 40

Wee

k 43

Wee

k 46

Wee

k 49

Wee

k 52

Wee

k 55

Wee

k 58

Wee

k 61

Wee

k 64

Wee

k 67

Wee

k 70

Wee

k 73

Wee

k 76

Wee

k 79

Wee

k 82

Wee

k 85

Wee

k 88

Wee

k 91

Wee

k 94

Wee

k 97

Wee

k 10

0W

eek

103

Wee

k 10

6W

eek

109

Wee

k 11

2W

eek

115

Wee

k 11

8W

eek

121

Wee

k 12

4W

eek

127

Wee

k 13

0W

eek

133

Wee

k 13

6W

eek

139

Wee

k 14

2W

eek

145

Wee

k 14

8W

eek

151

Wee

k 15

4

Sale

s (£

'000

s)

SalesModel

The aim of model building is to produce a model (red line) that tracks actual sales as closely as possible – explaining why sales have moved in the past

Step 1:Add large, easy to measure factors

to the model

-50

0

50

100

150

200

250

Wee

k 1

Wee

k 4

Wee

k 7

Wee

k 10

Wee

k 13

Wee

k 16

Wee

k 19

Wee

k 22

Wee

k 25

Wee

k 28

Wee

k 31

Wee

k 34

Wee

k 37

Wee

k 40

Wee

k 43

Wee

k 46

Wee

k 49

Wee

k 52

Wee

k 55

Wee

k 58

Wee

k 61

Wee

k 64

Wee

k 67

Wee

k 70

Wee

k 73

Wee

k 76

Wee

k 79

Wee

k 82

Wee

k 85

Wee

k 88

Wee

k 91

Wee

k 94

Wee

k 97

Wee

k 10

0W

eek

103

Wee

k 10

6W

eek

109

Wee

k 11

2W

eek

115

Wee

k 11

8W

eek

121

Wee

k 12

4W

eek

127

Wee

k 13

0W

eek

133

Wee

k 13

6W

eek

139

Wee

k 14

2W

eek

145

Wee

k 14

8W

eek

151

Wee

k 15

4

Sale

s (£

'000

s)

Seasonality

Actual

Model

The bars show how many sales are driven by each factor that the model is measuring. This variable captures the effect of Christmas

Step 2:Identify and model major trends in sales

-50

0

50

100

150

200

250

Wee

k 1

Wee

k 4

Wee

k 7

Wee

k 10

Wee

k 13

Wee

k 16

Wee

k 19

Wee

k 22

Wee

k 25

Wee

k 28

Wee

k 31

Wee

k 34

Wee

k 37

Wee

k 40

Wee

k 43

Wee

k 46

Wee

k 49

Wee

k 52

Wee

k 55

Wee

k 58

Wee

k 61

Wee

k 64

Wee

k 67

Wee

k 70

Wee

k 73

Wee

k 76

Wee

k 79

Wee

k 82

Wee

k 85

Wee

k 88

Wee

k 91

Wee

k 94

Wee

k 97

Wee

k 10

0W

eek

103

Wee

k 10

6W

eek

109

Wee

k 11

2W

eek

115

Wee

k 11

8W

eek

121

Wee

k 12

4W

eek

127

Wee

k 13

0W

eek

133

Wee

k 13

6W

eek

139

Wee

k 14

2W

eek

145

Wee

k 14

8W

eek

151

Wee

k 15

4

Sale

s (£

'000

s)

Store OpeningsSeasonalityActualModel

Step 3:Once the basic model is built, we can get a first estimate for larger marketing spends

-50

0

50

100

150

200

250

Wee

k 1

Wee

k 4

Wee

k 7

Wee

k 10

Wee

k 13

Wee

k 16

Wee

k 19

Wee

k 22

Wee

k 25

Wee

k 28

Wee

k 31

Wee

k 34

Wee

k 37

Wee

k 40

Wee

k 43

Wee

k 46

Wee

k 49

Wee

k 52

Wee

k 55

Wee

k 58

Wee

k 61

Wee

k 64

Wee

k 67

Wee

k 70

Wee

k 73

Wee

k 76

Wee

k 79

Wee

k 82

Wee

k 85

Wee

k 88

Wee

k 91

Wee

k 94

Wee

k 97

Wee

k 10

0W

eek

103

Wee

k 10

6W

eek

109

Wee

k 11

2W

eek

115

Wee

k 11

8W

eek

121

Wee

k 12

4W

eek

127

Wee

k 13

0W

eek

133

Wee

k 13

6W

eek

139

Wee

k 14

2W

eek

145

Wee

k 14

8W

eek

151

Wee

k 15

4

Sale

s (£

'000

s)

TV Store Openings

Seasonality Actual

Model

At every stage, diagnostic statistics tell us how well the model is working

• We get a lot of information from a model

1. The sales impact of each factor that we have included (ROI)

2. How sure we are that each individual measurement is accurate (confidence)

3. How sure we are that the overall model is robust

R2, t and F; diagnostic statistics that only econometricians find interesting

Step 4:The final model is a good fit for sales and includes all major marketing investments

-50

0

50

100

150

200

250

Wee

k 1

Wee

k 4

Wee

k 7

Wee

k 10

Wee

k 13

Wee

k 16

Wee

k 19

Wee

k 22

Wee

k 25

Wee

k 28

Wee

k 31

Wee

k 34

Wee

k 37

Wee

k 40

Wee

k 43

Wee

k 46

Wee

k 49

Wee

k 52

Wee

k 55

Wee

k 58

Wee

k 61

Wee

k 64

Wee

k 67

Wee

k 70

Wee

k 73

Wee

k 76

Wee

k 79

Wee

k 82

Wee

k 85

Wee

k 88

Wee

k 91

Wee

k 94

Wee

k 97

Wee

k 10

0W

eek

103

Wee

k 10

6W

eek

109

Wee

k 11

2W

eek

115

Wee

k 11

8W

eek

121

Wee

k 12

4W

eek

127

Wee

k 13

0W

eek

133

Wee

k 13

6W

eek

139

Wee

k 14

2W

eek

145

Wee

k 14

8W

eek

151

Wee

k 15

4

Sale

s (£

'000

s)

Press RadioTV Store OpeningsSeasonality ActualModel

It’s about asking the right questions

• There is a ‘standard’ econometric analysis, but modelling works much better, when we set it up from the start to answer specific questions

Can my media mix be made more efficient?

[this is a ‘standard’ analysis]

Should I transfer some of my ATL budget online? [High Street banking]

What will it cost to hit this sales target? [Automotive]

Finance are threatening to cut the ad budget; I need to prove that advertising is profitable.

[Online banking]Do I really need

‘brand’ TV, or can search and DM do

the job alone?[Car insurance]

How much budget do I need for a store re-launch?

[Supermarket retail]

Finding the most effective marketing mix is more than ROI measurement

• Response curves measured using econometrics forecast the effect of changing marketing budgets

0%

50%

100%

150%

200%

250%

300%

350%

100 200 300 400 500 600 700 800 900 1000

Add

ition

al S

ales

£'000s

TV

Press 1. The most effective marketing mix allocates budget first to TV…

2. …and then the remaining budget to press

Beyond a £400k campaign, additional TV spend generates few extra sales

Example response curves

Bringing together the three elements of marketing evaluation

1. Monitoring the market– Tracking competitor activity– Benchmarking (share of voice etc.)

2. Response tracking– Immediate indicators of consumer behaviour– Web traffic, Click through, Cost per click, Call volume, store footfall

and more…– Awareness & consideration tracking

3. Modelling– Filling in the gaps that can’t be measured by direct response

Direct response trackingCost per acquisition etc.Daily / Weekly updated

Market MonitoringShare of voice etc.Weekly / Monthly updated

The flow of campaign evaluation

Plan Campaign Post CampaignAnalysis

Long-termanalysis

Flow of resultsClicks (web response)Phone #sDirect sales

Econometrics

Budget setting, forecastingand optimisation

‘Multiplier’ adjustments

- Adjustments to directresponse Measures

- Click path analysis

In-campaign‘tuning’

0.00

200.00

400.00

600.00

800.00

1,000.00

1,200.00

1,400.00

01-O

ct-1

005

-Oct

-10

09-O

ct-1

013

-Oct

-10

17-O

ct-1

021

-Oct

-10

25-O

ct-1

029

-Oct

-10

02-N

ov-1

006

-Nov

-10

10-N

ov-1

014

-Nov

-10

18-N

ov-1

022

-Nov

-10

26-N

ov-1

030

-Nov

-10

04-D

ec-1

008

-Dec

-10

12-D

ec-1

016

-Dec

-10

20-D

ec-1

024

-Dec

-10

28-D

ec-1

001

-Jan

-11

05-J

an-1

109

-Jan

-11

13-J

an-1

117

-Jan

-11

21-J

an-1

125

-Jan

-11

29-J

an-1

102

-Feb

-11

06-F

eb-1

110

-Feb

-11

14-F

eb-1

118

-Feb

-11

22-F

eb-1

126

-Feb

-11

02-M

ar-1

106

-Mar

-11

10-M

ar-1

114

-Mar

-11

18-M

ar-1

122

-Mar

-11

26-M

ar-1

130

-Mar

-11

03-A

pr-1

107

-Apr

-11

11-A

pr-1

115

-Apr

-11

19-A

pr-1

123

-Apr

-11

27-A

pr-1

1

Sale

s

Sales minus Central TV burstActual sales (including Central TV burst)Projected Sales if Central burst had been run nationally

Sales Forecasts and Projections

National sales revenue projections

-15%

-10%

-5%

0%

5%

10%

15%

20%

25%

30%

35%

03-J

an-1

1

10-J

an-1

1

17-J

an-1

1

24-J

an-1

1

31-J

an-1

1

07-F

eb-1

1

14-F

eb-1

1

21-F

eb-1

1

28-F

eb-1

1

07-M

ar-1

1

14-M

ar-1

1

21-M

ar-1

1

28-M

ar-1

1

04-A

pr-1

1

11-A

pr-1

1

18-A

pr-1

1

25-A

pr-1

1

02-M

ay-1

1

09-M

ay-1

1

16-M

ay-1

1

23-M

ay-1

1

30-M

ay-1

1

06-J

un-1

1

13-J

un-1

1

20-J

un-1

1

27-J

un-1

1

04-J

ul-1

1

11-J

ul-1

1

18-J

ul-1

1

25-J

ul-1

1

01-A

ug-1

1

08-A

ug-1

1

15-A

ug-1

1

22-A

ug-1

1

29-A

ug-1

1

Year

on

Year

Cha

nge

A Brilliant client (with econometric models) asked why their recent performance had been so good

Strong year on year sales performance raised the question: What was going right? (Brilliant Media client example)

Year on year analysis provided strong evidence that the overall market was improving

• Year on year is good for evidence, but it doesn’t tell you what to track. We could only draw this chart because we already knew Google searches were important

-10%

-5%

0%

5%

10%

15%

20%

17-J

an-1

1

24-J

an-1

1

31-J

an-1

1

07-F

eb-1

1

14-F

eb-1

1

21-F

eb-1

1

28-F

eb-1

1

07-M

ar-1

1

14-M

ar-1

1

21-M

ar-1

1

28-M

ar-1

1

04-A

pr-1

1

11-A

pr-1

1

18-A

pr-1

1

25-A

pr-1

1

02-M

ay-1

1

09-M

ay-1

1

16-M

ay-1

1

23-M

ay-1

1

30-M

ay-1

1

06-J

un-1

1

13-J

un-1

1

20-J

un-1

1

27-J

un-1

1

04-J

ul-1

1

11-J

ul-1

1

18-J

ul-1

1

25-J

ul-1

1

01-A

ug-1

1

08-A

ug-1

1

15-A

ug-1

1

22-A

ug-1

1

Year

on

Year

Cha

nge

Year on Year Sales (5wk MA)

Google Searches for product term (non-brand)

Google search activity closely matched overall sales (Brilliant Media client example)

What about test and control?

What makes a ‘good’ media test?

1. Clear objectives

• What, exactly, are we trying to find out?

2. Designed to generate a measure• What uplift do we expect that the test

might generate …?

• … So what scale does the test need to be for this effect to be measurable?

3. Useful negative results

• If the test finds no significant uplifts, are we sure the activity doesn’t work?

Why is econometrics Important?

1. Controls for external factors

2. Lets us measure smaller effects

3. Helps to specify an appropriate scale for the test

A ‘bad’ media test… Real world example

• An advertiser wanted to find the effect of a combined TV and Radio campaign on various brand preference metrics

• The campaign was run over four weeks in the Central and Granada BARB regions

• Pre and Post survey dips in four cities provided theawareness data

– Three test cities: Manchester, Carlisle & Birmingham

– One control: Norwich

• 600+ respondents in the pre-campaign dip and 700+ in the post campaign

• What’s wrong with that… ?

What’s wrong with that?

1. No clear objective for the campaign

• ‘Run a campaign and see which metric moves’ is not an ideal starting point

2. No prior knowledge of whether the test is likely to be big enough to make a difference (and so be measured)

• In order to generate the 10% sales uplift that we will need to get a solid measure, should the test be run for longer? Or with more GRPs?

• One control city only is very, very risky

3. Post campaign measurement using a simple average vs. Control

• Leaves the test exposed to unforeseen events that have a different impact in the test and control regions

• Econometrics gives a much better chance of a useable result

What actually happened?The test was inconclusive…

Ove

rall

Te

st

Co

ntr

ol

Man

ch

este

r

Carl

isle

Bir

min

gh

am

53% 54%

63%

41%47%

57%61%

55%

68%

52%

Pre Post

Awareness Metric

Two test cities increased. One decreased.

The two that increased were in different BARB regions

Control also increased(by more than the test regions)

Inconclusive result.

Spontaneous Awareness

What actually happened?The test was inconclusive…

Ove

rall

Te

st

Co

ntr

ol

Man

ch

este

r

Carl

isle

Bir

min

gh

am

36%

15%

31%33%

43%

28%

8%

31% 31%

23%

Pre Post

Purchase Intent

Purchase intent fell in two test regions and fell very heavily in Birmingham

The control region declined

Manchester stayed at 31%, bucking the decline of the control region

Overall, were our test regions better? Or worse?

Purchase intent

Briefing an analysis

A variety of companies provide econometrics and bring different strengths to the analysis

Completely impartial

Little internal data

Smaller (riskier)

May not be mediameasurement specialist

Very weak ties to planning

Generally impartial

Strong internal data

Large analytical teams

Can be expensive

Weak ties to planning

Risk of conflict of interest

Strong internal data

Mid to large analytical teams

Media measurement specialist

Strong ties to planning

Independent Semi-Independent Media Agencies

Automated tools can help (and sometimes reduce analysis costs) but need a lot of care

• Automated tools are seductive, but we need to be aware of their limitations

• Analysis is only ever an aid to decision making

Briefing an econometric analysis

1. Why do you want the work done?

2. What data exists and who is responsible for it?

3. When is the decision deadline that the work informs?

4. How will the work inform future decisions?

5. Who are your project team?

6. Interim meetings

7. What was your marketing designed to achieve?

8. everyone who will use the results needs to be involved from the start

We haven’t mentioned consumer research, but it’s the final piece of the puzzle











Contact

Neil CharlesHead of Econometrics

Brilliant Media1 City SquareLeedsLS1 2FF

+44(0)113 394 0078+44 (0)7508 [email protected]