Counting what will Counting what will countcount

Does your dashboard Does your dashboard predict ?predict ?

Koen PauwelsKoen Pauwels

Amit JoshiAmit Joshi

Marketing dashboardsMarketing dashboards

• Marketing accountability & accelerating Marketing accountability & accelerating changechange

• Limits to human processing capacity still there: Limits to human processing capacity still there: MSIMSI: ‘separate signal from noise’, ‘dashboards’: ‘separate signal from noise’, ‘dashboards’

• Top management interest: ‘substantial effort’ Top management interest: ‘substantial effort’ by 40% of US and UK companies (Clark ea by 40% of US and UK companies (Clark ea 2006)2006)

• But current applications often fail to impress:But current applications often fail to impress:

Marketing Scientists can help raise the bar !Marketing Scientists can help raise the bar !

Academic research focusAcademic research focus

Reibstein ea (2005) discuss 5 development Reibstein ea (2005) discuss 5 development stages:stages:

1)1) Identify key metrics (align with firm goals)Identify key metrics (align with firm goals)2)2) Populate the dashboard with dataPopulate the dashboard with data3)3) Establish relation between dashboard itemsEstablish relation between dashboard items4)4) Forecasting and ‘what if’ analysisForecasting and ‘what if’ analysis5)5) Connect to financial consequencesConnect to financial consequencesMost dashboards yet to move beyond stage 2 !Most dashboards yet to move beyond stage 2 !Need research to select best metrics, relate to Need research to select best metrics, relate to

performanceperformance

You’ve got 33 x 3 = 99 You’ve got 33 x 3 = 99 variablesvariables

• Market variablesMarket variables: price national brand, store brand, : price national brand, store brand, ∆∆• AwarenessAwareness: top-of-mind, aided, unaided, ad awareness: top-of-mind, aided, unaided, ad awareness• Trial/usageTrial/usage: ever, last week, last 4 weeks, 3 months : ever, last week, last 4 weeks, 3 months • Liking/Satisfaction: Liking/Satisfaction: given aware and given triedgiven aware and given tried• Preference:Preference: favorite brand, will it satisfy future needs? favorite brand, will it satisfy future needs?• Purchase intentPurchase intent: given aware and given tried: given aware and given tried• Attribute ratingsAttribute ratings: taste, quality, trust, value, fun, feel: taste, quality, trust, value, fun, feel• Usage occasionUsage occasion: home, on the go, afternoon, entertain: home, on the go, afternoon, entertainNeed to reduce 99 to 6-10 metrics (US) or 10-20 Need to reduce 99 to 6-10 metrics (US) or 10-20

(UK)(UK)

Metric deletion rules Metric deletion rules (Ambler 2003)(Ambler 2003)

1)1) Does the metric rarely change?Does the metric rarely change?

2)2) Is the metric too volatile to be reliable? Is the metric too volatile to be reliable? Univariate tests on time series propertiesUnivariate tests on time series properties

3)3) Is metric leading indicator of market Is metric leading indicator of market outcome? outcome? Pairwise tests of metric with Pairwise tests of metric with performanceperformance

4)4) Does the metric add sufficient explanatory Does the metric add sufficient explanatory power to existing metrics? power to existing metrics? Econometric models to explain Econometric models to explain performanceperformance

This researchThis research• Univariate: st. dev., coef. of variation, evolution Univariate: st. dev., coef. of variation, evolution • Pairwise: Pairwise: Granger CausalityGranger Causality test with test with

performanceperformance• Explanatory powerExplanatory power: regression model : regression model

comparisoncomparison1)1) Stepwise regression Stepwise regression (Hocking 1976, Meiri ea 2005)(Hocking 1976, Meiri ea 2005)

2)2) Reduced Rank Regression Reduced Rank Regression (Reinsel and Velu 1998)(Reinsel and Velu 1998)

3)3) Forecast error variance decomposition, based Forecast error variance decomposition, based on Vector Autoregressive Model on Vector Autoregressive Model (Hanssens 1998)(Hanssens 1998)

• AssessmentAssessment: forecasting accuracy hold-out : forecasting accuracy hold-out samplesample

• Managerial controlManagerial control: impact size and lead time: impact size and lead time

Stepwise regressionStepwise regression

• Automatic selection based on statistical criteriaAutomatic selection based on statistical criteria

Objective: select set of metrics with highest RObjective: select set of metrics with highest R22

• Forward: add variables with lowest p-valueForward: add variables with lowest p-value

Backward: delete variables with highest p-valueBackward: delete variables with highest p-value• Unidirectional: considers one variable at a timeUnidirectional: considers one variable at a time

Stepwise: checks all included against criterionStepwise: checks all included against criterion

Combinatorial: evaluates every combinationCombinatorial: evaluates every combination

Reduced Rank Reduced Rank RegressionRegression

• Uses correlation of key metrics and Uses correlation of key metrics and performanceperformance

Yi = Xi’C + εi with Yi (m x 1) and Xi (n x 1)Yi = Xi’C + εi with Yi (m x 1) and Xi (n x 1)

C (m x n) has C (m x n) has rank r ≤ min (m, n)rank r ≤ min (m, n)

Restriction: m – r linear restrictions on CRestriction: m – r linear restrictions on C

Maximize explained variance under Maximize explained variance under restrictionrestriction

• Originally shrinkage regression (Aldrin Originally shrinkage regression (Aldrin 2002), now for selecting best combination 2002), now for selecting best combination variablesvariables

Forecast Forecast variance variance decompositiondecomposition

• Based on Vector Autoregressive ModelBased on Vector Autoregressive Model

A ‘dynamic RA ‘dynamic R22’, FEVD calculates the ’, FEVD calculates the percentage of variation in performance that percentage of variation in performance that can can be attributed to changes in each of can can be attributed to changes in each of the endogenous variables (Hanssens 1998, the endogenous variables (Hanssens 1998, Nijs ea 2006)Nijs ea 2006)

• Measures the relative performance impact Measures the relative performance impact over time of shocks initiated by each over time of shocks initiated by each endogenous varendogenous var

• We consider the FEVD at 26 weeksWe consider the FEVD at 26 weeks

Methods share 4/10 Methods share 4/10 metricsmetrics

StepwiseStepwise Reduced Reduced RR

FEVDFEVD

MarketMarket pricenb, -pricenb, -stst

pricenb, -pricenb, -stst

pricenb, -pricenb, -stst

AwarenesAwarenesss

awareunnawareunnbb

awunnb, -awunnb, -stst

awareunnawareunnbb

Usage/Usage/trialtrial

tried3mtried3m∆∆ tried3mtried3m∆∆ tried4wnbtried4wnb

Purch. Purch. IntentIntent

piawarenpiawarenbb

AffectAffect satistriedsatistriednbnb

satistriedsatistriednbnb

liketriednliketriednbb

Attribute Attribute RatingsRatings

satisnb, satisnb, feelstfeelst

funfun∆∆, , taste taste ∆∆, , trust trust ∆∆

satisnb, satisnb, trusttrust∆, ∆, qualqual∆∆

Usage Usage occasionoccasion

afternoonafternoonnb nb entertainentertainnbnb

entertainentertainnbnb

afternoonafternoonnb nb entertainentertainnbnb

Stepwise scores within Stepwise scores within samplesample

0

10

20

30

40

50

60

70

80

90

100

R-SQUARED ADJUSTED R-SQUARED

STEPWISE REGRESSION REDUCED RANK REGRESSION FORECAST ERROR VARIANCE DECOMPOSITION

But sucks out-of-sampleBut sucks out-of-sample

0

5

10

15

20

25

MEAN AVERAGE PERCENTAGE ERROR THEIL'S INEQUALITY COEFFICIENT

STEPWISE REGRESSION REDUCED RANK REGRESSION FORECAST ERROR VARIANCE DECOMPOSITION

Sales Impact Size and Sales Impact Size and Timing Timing

1% change 1% change inin

Short-termShort-term Long-termLong-term Wear-inWear-in Wear-outWear-out

PRICENBPRICENB -161,794-161,794 -84,417-84,417 00 88

PRICESTPRICEST 71,56171,561 121,997121,997 00 11

AFTERNOONAFTERNOON 32, 12932, 129 32, 12932, 129 00 00

TRUST∆TRUST∆ 23,70723,707 23,70723,707 00 00

LIKEgiventriednbLIKEgiventriednb 00 40,42040,420 11 00

QUALITY∆QUALITY∆ 00 66,96366,963 11 00

TRIED4WNBTRIED4WNB 00 72, 48172, 481 22 44

SatisfyingNBSatisfyingNB 00 46,78846,788 22 00

EntertainfriendsNBEntertainfriendsNB 00 60,07160,071 22 33

AWAREUNNBAWAREUNNB 00 68,53768,537 33 44

Value (price-quality) matters Value (price-quality) matters right now, Awareness and right now, Awareness and

Trial soon ! Trial soon !

0

10,000

20,000

30,000

40,000

50,000

60,000

70,000

80,000

90,000

0 1 2 3 4

Weeks till peak impact

Lo

ng

-term

Sale

s I

mp

act PRIC

EQUALITY

Satisfying

Entertain friends

Tried last month

Unaided Awareness

TRUST

Afternoon Lift

Like if tried

Summary dashboard Summary dashboard intuitionintuition

1) To increases sales immediately (0-1 1) To increases sales immediately (0-1 weeks)weeks)

a) promote on price and on afternoon lift usagea) promote on price and on afternoon lift usage

b) communication focus on quality, affect, trustb) communication focus on quality, affect, trust

2) To increase sales soon (2-3 weeks)2) To increase sales soon (2-3 weeks)

a) provide free samples (to up ‘tried last a) provide free samples (to up ‘tried last month’)month’)

b) focus on satisfying and entertainment useb) focus on satisfying and entertainment use

c) advertise for unaided awarenessc) advertise for unaided awareness

Conclusion: P-model for Conclusion: P-model for dashboardsdashboards

1.1. Which metrics are leading indicators?Which metrics are leading indicators?

Granger causality testsGranger causality tests

2.2. Explain most of performance dynamicsExplain most of performance dynamics

Forecast error Variance decomposition Forecast error Variance decomposition

3.3. Forecast multivariate baseline with Vector Forecast multivariate baseline with Vector Autoregressive or Error Correction modelAutoregressive or Error Correction model

4.4. Displays Displays timingtiming and and size size of sales impactof sales impact

Your Questions ?Your Questions ?