simulate your market.

Simulation: A new modeling approachHow businesses can benefit from simulation

2

Contents

Contents .......................................................................................................................................................................... 2

Introduction: Social Simulation ...................................................................................................................................3

How to Simulate a Consumer Market .....................................................................................................................10

How Data Fits into Simulation..................................................................................................................................12

How to Build Confidence in Attribution ..................................................................................................................13

How to Avoid Over-Specification When Calibrating an Agent-Based Model ..................................................17

How Marketing Mix Modeling and Simulation Account for Uncertainty .........................................................19

Over-Specification: Comparing Agent-based and Marketing Mix Models ......................................................22

Attribution to What? Understanding Cross-Channel Interactions ...................................................................25

From Optimization to Strategy: Deeper into the Attribution Stack ...................................................................29

Marketing Effectiveness Measures: Know Your Options ...................................................................................33

Marketing Mix Modeling and Agent-Based Simulation – What Is the Question? .........................................34

Advantages and Disadvantages of Simulation .................................................................................................... .. 5

© Concentric, Inc. 2016-2017. All rights reserved.

3

Simulation is an analytical approach that gained in prominence with the advent of computers

and greater computing power. It is most useful when the system of interest has the following

properties:

1. The macro-outcomes result from the interactions of many micro-components.

2. The rules of interaction between the micro-parts are known.

3. The macro-outcomes are not obvious, without the calculation of all micro-behaviors.

Examples of these types of systems are the Universe, the Solar System, the Climate, the human

body, chemical substances, and elementary particles. It is not surprising that simulation is an

essential tool in astronomy, meteorology, chemistry, and physics.

Another example of such a system is human behavior. As an example, in the area of consumer

markets, sales, brand equity, word-of-mouth, and consideration all are metrics of interest to a

marketer. These metrics are products of the interaction of many consumers interacting with

certain rules. Simulation approaches that analyze human behavior are called Social Simulation.

Social Simulation is different from data analysis methods like regression or Markov chain models,

which are common social science approaches today. These methods are empirical. They derive

insight from data. As such, they inspire confidence. At the same time, they lack the ability to

make inferences about new or future states of the system of analysis. Simulation, on the other

hand, is a deductive method. It needs data only to be validated, i.e., compared to observation to

test its validity. To produce insight a simulation needs two things:

1. Rules of interaction of the micro-components.

2. Initial conditions.

In a sense, the rules of interactions form a system of logic and the initial conditions are the

premises. The macro-outcomes emerge as the logical conclusion of the rules and the initial

conditions. For this reason, simulation helps us to think about new or future states of the system

of analysis. The common difficulty in simulation is validation, but the latest advances in various

fields have led to breakthroughs in this area.

Introduction: Social Simulation

4

Lately, Social Simulation has been gaining prominence in economics as a way to improve

macroeconomic models and forecasting. Despite this, it is not well established in the business

community.

This white paper discusses many areas of advancement of Social Simulation that we have

achieved through the development of Concentric Market®, our simulator of consumer markets.

The paper offers best practices, clarifies the role of Social Simulation, and compares it to other

approaches.

5

Simulation is an approach that is used most commonly in two situations.

The first situation is when uncertainty is high due to sparse data. One such example is a

simulation of an ancient Native American tribe, the Anasazi, a culture that lived between the 9th

and 14th centuries. It is hard to run typical analytics on the limited available data, so researchers

use simulation to understand what happened to the tribe.

A second common use of simulation is for experimentation in a low-cost, low-risk environment.

Researchers at CERN simulate particles colliding in the Large Hadron Collider, before they

validate their forecasts in the expensive real-world collider in Switzerland. More common

examples include airline pilots practicing on simulated flights and doctors learning on test

patients.

Both of these applications of simulation are helpful to scientists and researchers, but they come

with a set of advantages and disadvantages. We have grouped these advantages and

disadvantages into three broad areas related to technology, process, and socialization. The

following table gives a summary of the advantages and disadvantages, that we elaborate on

below.

Advantage Disadvantage

Technology Forecasting under uncertainty Good theories needed

Able to answer many questions No standardized approach

Process Low data requirements to model Challenging to validate

Easy what-if scenario analysis Potential scope creep in projects

Socialization Low cost High skepticism

Innovative approach Political implications

Advantages and Disadvantages of Simulation

6

Great forecasting power, but a good theory is needed Data analysis methods such as regression are limited to forecasting effects of events that are

similar to what has already happened in the past. For example, if a brand has been investing in

TV ads within a range of $50M to $100M in the past few years, a marketing mix model is excellent

at forecasting what would happen if spend is within those bounds. However, the model is likely to

produce nonsensical results once it extrapolates to forecast what would happen if TV spend is

doubled or if a new marketing channel is deployed.

Simulation has an advantage over these methods in that it allows us to forecast things that have

never happened before and to run scenarios outside of historical bounds. The caveat is that we

need a good theory and causal hypotheses about how the system we are interested in analyzing

works. Theories that have high predictive power, at least in social science, are hard to come by

and may take years to develop.

Flexible, but not standardized Simulations, and agent-based modeling in particular, provide highly flexible techniques for

answering a wide range of research questions. These questions include what happened in the

first moments of the Universe, how wind turbulence around aircraft works, how the World Wide

Web evolves, or how to better design hospitals. Although simulation can be applied in a variety of

contexts, a formalized set of rules and best practices is not always readily available. For this

reason, simulation modeling (especially in social science) is incredibly creative, but may be

daunting for new researchers who have no single reference to consult when starting out.

Building a simulation does not require data, but validation does Simulation is an excellent approach to analyze problems when the available data is limited, since

no data is necessary to construct a simulation. Validating a simulation, however, often requires

multiple data sources to achieve a great degree of confidence in its representation of real-world

dynamics. The process of validation is a disadvantage for simulation when comparing to data

analytics approaches, since validating simulations is often more difficult.

For example, if we wanted to simulate traffic on a road, we would not need any data to start. We

could construct a simulation that incorporates modeled cars, driver behaviors, and road

conditions and voila: we have a traffic simulator.

7

Analysis of this traffic simulation could provide surprising insights – such as the pattern of traffic

jams migrating in the opposite direction that automobiles are traveling. But to test whether such

insights are valid, we would need to use various data. We would seek information about road

conditions in a range of contexts – in cities, on highways, in the U.S. and other countries, and

under different weather conditions. We could then recreate all of those scenarios within the

simulation and see how well they match what actually happened in the real world. If the

simulation data and the actual “validation” data match, then we have confidence that our

simulated model of traffic is useful and valid.

To get the simulations to match real-world outcomes, we need to change the theoretical rules

guiding the simulation or test different assumptions until they do. Simulations have the benefit

of forecasting multiple metrics simultaneously, but this can make it challenging to get all of the

assumptions synchronized. One change may improve the forecast for one metric, but degrade the

fit for another. Fortunately, expanding computing power and improving algorithms continue to

reduce the time and effort to overcome the process barrier of calibrating and validating

simulations.

Run any what-if scenario, but stay within budget We build simulations with the goal of having a “Petrie dish” in which to experiment in a controlled

and low-risk environment, before taking action in the real world. At the outset of a project, a team

can often list off a broad range of hypotheses to test within the simulator. Once a simulation is

built and what-if scenarios can be run, the desire to keep testing more and more scenarios often

grows.

Going back to the traffic simulation example, the initial goal of the simulation might be to

determine whether to replace 4-way-stops with roundabout intersections in a particular section

of town. The questions may compound from there: What will the impact be of traffic lights in

other parts of town? How should the signage be placed? How will this impact traffic during the

farmer’s market? It’s easy to see how the number of questions and scenarios can multiply very

quickly.

Enabling a team to test and answer more questions is a great value-add that simulation provides.

Projects may start by focusing on a single research question, but often grow to incorporate more

8

complex ideas. This dynamic and creative process can build consensus by bringing more

stakeholders to the table and ultimately lead to better decisions.

However, it is important when time and resources are limited to ensure that the scope of a project

does not expand beyond the available budget. Reaching agreement with the working team on the

appropriate balance between focus on a specific deliverable and open-ended exploration is a good

step to take early in the planning process.

Low cost, but high skepticism Compared to the cost of experimenting in the real world, the use of simulation requires very little

time and resources. Think about marketing: if we were to run various experiments in which we

varied the amount we invest in different channels, we would have to go through dozens of

budgets over as many years to gather enough data to answer a question with certainty. In the

meantime, our brand and business may have gone in an undesirable direction. The alternative to

real world experimentation is to run simulations to test different marketing plans. Within

minutes we can test many ideas before acting on a plan and making decisions in the real world.

The disadvantage of this approach is that some audiences today are skeptical of simulation. Most

of today’s analysis, especially in marketing, is based on reporting and building deterministic

statistical models to describe what has happened in the past. Researchers often prefer these

descriptive approaches to methods that test theories about the future. We believe that this

skepticism is a result of the relative novelty of simulation in marketing analytics, and that with

more success stories and validated forecasts, this skepticism will subside.

Innovative, but political Simulation may be one of the most innovative approaches researchers engage in today. We have

seen people advance in their careers for their intra-preneurial spirit in introducing simulation

within their organization. Long-standing and thorny problems get tackled every day with social

simulation, but the socialization of simulation results often presents organizational challenges.

Because it identifies trade-offs between a range of metrics, simulated insights may bring

conflicting interests to the forefront in organizations where stakeholder incentives are not

aligned.

9

Building consensus around a simulation is done best when the process is organized from the

start. All stakeholders should agree on the simulation’s framework, assumptions, and questions

to be answered at the outset. In our experience, this is the best way to ensure that the simulation

findings are impartial.

10

Simulation is flexible and somewhat unconstrained by nature. This open-endedness makes

simulation useful for understanding real-world markets in a broad variety of contexts. But given

this open-endedness, it becomes important to remain focused on a specific objective and to bring

some structure to the simulation.

Accordingly, the first step in building any Concentric Market simulation is to construct a

simulation framework. This framework should be built based on the consumer decision we are

seeking to better understand, in line with some business objective. The framework consists of the

following elements:

• Alternatives: What set of brands or options are in the consumers’ choice set?

• Attributes: What factors do consumers consider when selecting among alternatives?

• Touchpoints: What marketing activities or events influence consumers’ decisions or

behavior?

• Segments: What different types of consumers exist in the market?

Most importantly, these elements should be selected according to the specific consumer decision

you wish to simulate. Here are some additional guidelines when building a simulation

framework, along with the typical number of framework elements in simulations:

Alternatives (3-10):

• Consider the market share of the competitors. It’s usually appropriate to include the

dominant competitors in the market and to exclude brands with very small shares.

• Factor in how directly the alternative competes with your brand. It’s a good practice to

consider not only the size of the competitor, but also whether or not it presents a threat or an

opportunity for your brand’s market share. You may choose to include a brand with low

market share if it is an emerging threat to your brand.

• Consider any substitutes in adjacent categories that would be helpful.

Attributes (2-6):

• Consider the factors that matter most to a consumer when making a decision. Typically, this

list can be summarized into a few key drivers of choice.

How to Simulate a Consumer Market

11

• Observe what consumers are talking about and what messages marketers are broadcasting.

This can provide insight into what factors matter the most.

• Think about both tangible attributes (physically experienced by the consumer) and

intangibles (more ethereal in nature, e.g. “cool,” brand reputation).

Touchpoints (5-150):

• Consider anything that can change consumer perceptions or consideration or stimulate

conversation in the marketplace. Touchpoints may include traditional media channels, and

also other types of interactions with the brand (for example, in-store touchpoints, events, PR,

salespeople, online blogs).

• Remember to consider not only your brand’s marketing activities, but also the activities of the

competitors.

• Think about the level of detail at which marketing strategy decisions will be made when

determining how granular to make the list of touchpoints. For example, if the simulation will

only be used to determine broadly the mix of television and online advertising, then it would

probably not be helpful to expand the touchpoint list to include splits for different TV

programs, 15 vs. 30 second ads, etc.

Segments (1-6):

• Consider how the population can be divided into groups based on differences in behavior,

media habits, drivers of choice, and perceptions/consideration about the alternatives.

• Select a set of segments that are mutually exclusive and encompass the relevant population.

A segment may be demographic, attitudinal, psychographic, or based on other factors so long

as that holds.

• Determine whether there are any key non-purchasing influencer groups (e.g.

doctors/pharmacists, children) that should be included in the simulation. Even if an

influencer group does not purchase any of the alternatives, they may still be active in the

social network and have an impact on outcomes.

12

Building a simulation does not require any data. Validating a simulation, on the other hand, does.

Building a simulation only requires that we make assumptions. We assume initial conditions,

constraints, and behaviors within the system we seek to understand. Qualitative or quantitative

data may be used to inform some of these assumptions, but often times we just apply insights or

judgment.

Once we have made assumptions and built a simulation, we run the simulation and see if it

accurately reflects real world dynamics. This validation requires data.

Calibrating a valid simulation is an iterative process. The test-and-learn nature of the process is

what frees us from being bound by data when initially building the simulation. However, the

validation points by which we judge the accuracy of our simulation are based on data.

Initial Conditions

+

The Right Rules

->

Simulated Outcomes = Real World Outcomes

How Data Fits into Simulation

13

Simulation, like any other modeling approach, is an imperfect representation of the real world. A

question that commonly comes up when performing analysis and forecasting is “how can we be

confident that what this model tells us is right?” Of course, all modeling results have some

uncertainty, but attribution is a particularly difficult finding to validate.

In a nutshell, attribution is the decomposition of a variable into the elements driving changes in

its value. Attribution results in a measurement of the impact of each element. For example,

marketing mix analysis often decomposes sales into base (an unexplained contribution) and

individual marketing channel contributions. The difficulty in building high confidence about

attribution is that the numbers reported are virtually impossible to measure in the real world. If

we say that TV contributes 30% to the increase in sales, we are making an accounting statement.

There is no way to go into the field and truly isolate that 30% figure. Getting to that 30% requires

assumptions about how we measure effects and how we quantify contribution.

Yet, while attribution is in this sense more art than science, there are ways to increase our

confidence in the attribution results that come from a simulation. More often than not, increasing

the confidence requires investing more time into validation of the model against metrics that are

actually observable and measurable in the real world. Only then, by proxy, is our confidence

boosted that our latent results are as valid as the results we can compare with measured

observations.

Below we outline 4 routes that can be taken to increase the confidence in the simulation

attribution:

1. Calibrate the simulation to multiple metrics at an aggregate level.

2. Perform a holdout forecast.

3. Compare individual consumer journeys to simulated agent journeys.

4. Conduct in-market tests to assess impacts.

Calibrating the simulation to multiple objectives Simulations are often built with a relatively large number of parameters and initial conditions.

This often raises a question of how one can be confident that they have not over-fit the

How to Build Confidence in Attribution

14

simulation. One way to avoid over-fitting risk and build confidence is to calibrate to more data

points.

Typically, regression approaches fit to one time series, for example, sales over time for one brand.

With a market simulation, one may calibrate to sales for the primary brand of interest as well as

sales for its competitors. The number of data points that build confidence in the fit is then quickly

scaled up by the number of competing brands.

In addition to calibrating to sales across brands, one may also calibrate to other metrics beyond

sales. Simulations can be calibrated to brand metrics such as consideration, perceptions, and

word-of-mouth volume. The number of calibration points can be scaled even further by the

number of KPIs that are fit.

Concentric Market® forecasts multiple metrics, all of which can be compared to data.

Fitting a simulation to data from multiple brands across KPIs builds confidence that it is not over-

specified and that it recreates the dynamics of the real-world.

15

Performing a holdout forecast Often stakeholders are interested in the predictive power of a model or simulation. In this case,

the evaluation of the approach goes beyond analytics to explain the past and into how well it

projects future outcomes. A holdout forecast is one approach for assessing predictive capabilities.

When conducting a holdout, the historical data is split between a calibration time frame and a

holdout period. The model or simulation parameters are tuned to the calibration time frame only.

This calibrated model is then used to make a forecast during the holdout period.

An example of a calibration and hold-out analysis.

Using a holdout, we see how effectively the simulation was able to forecast past outcomes and

this provides a better sense of what sort of predictive accuracy may be anticipated in forecasting

future outcomes.

Analyzing consumer journeys One of the advantages of simulation is that it recreates the dynamics that take place at an

individual level in the real world. Aggregate sales outcomes emerge from many individual

consumer decisions. One may be interested in seeing not only how well macro outcomes are fit

by a simulation or model, but also how well the approach recreates individual-level data. These

journeys can give us a better sense of how all the marketing activities and earned touchpoints

interact to lead to sales.

16

If data on individual consumer journeys is available, this can serve as another way to assess the

validity of a simulation. Simulated agent journeys can be recorded and compared to the data on

individual consumer journeys. If the simulated journeys align with the actual consumer journeys,

then the simulation is tuned to the micro-level dynamics of the simulation.

Conducting in-market tests Ultimately, an audience may determine that the best way to assess the impact of a marketing

strategy is to actually put it into practice in the real-world. An in-market test allows for data to be

gathered that can help measure impacts. Although running a test in the actual market allows one

to see how an outcome pans out in the complex real-world, it is difficult to run a controlled

experiment when the many variable factors of the real world come into play. Even a real-world

experiment has its limitations in terms of building confidence in a particular finding.

An example in-market forecast that results from calibration and hold-out analysis.

17

Practiced statisticians are very familiar with the problem of over-fitting when building traditional

econometric models. As the number of parameters approaches the number of data points fitted,

two things will happen:

1. The model will fit the history progressively more precisely.

2. The model will become progressively more over-specified, eventually to the point of losing

all statistical and predictive value.

This is why experienced model-builders place such high value on parsimony and generally avoid

complicating a regression model with extra variables unless they contribute significant

explanatory power. The classical “degrees of freedom” problem can eliminate a model’s

credibility.

To summarize, as the number of regression parameters increases with a number of data points,

over-fitting risk also increases.

Similar concerns may arise when building a simulation. While having the ability to adjust a

variety of initial conditions and settings in an agent-based model may provide a lot of flexibility,

may we encounter the problem of over-specification when calibrating an agent-based model?

The answer is yes, absolutely. With all the various model settings in Concentric Market® one

could simulate an annual sales figure that matches with historical values an array of input

assumptions. Many of these assumption sets would not align with reality and result in

simulations that are misleading and lacking in insight.

There are two ways to avoid the problem of over-specification in agent-based modeling.

1. Reduce input uncertainty: Decrease the number of freely floating inputs when calibrating your

model by building confidence in your input settings via research, data analysis, expertise, or

norms. This will cut down the model calibration search space to a more manageable size and

reduce your risk of over-fitting.

How to Avoid Over-Specification When Calibrating an Agent-Based Model

18

2. Increase the number of validation points: Calibrate to more metrics and trends based on what

you know to be true. Go beyond sales to assess consideration, word-of-mouth volume, and

brand equity outputs. Analyze the sales and brand metrics over time to see if the trends are

sensible. Look at brand equity attribution to see if the relative impacts of each touchpoint are

realistic. Carry out sensitivity analyses to see if the simulation mimics marketing responses

you would expect.

By implementing these two approaches, you are attacking the problem of over-fitting risk from

both sides: You are reducing the number of parameters in your search space and increasing the

number of data points in your validation.

Your efforts should always be geared towards the business question you are ultimately trying to

answer. This should help guide where to focus your energy in reducing input uncertainty and

selecting your validation points.

Concentric Market is meant to be used as a “thinking system.” The more thought that goes into

the system inputs and validation efforts, the more confidence you will have in the insights the

system generates.

So, keep this simple rule in mind:

As Thinking ↑, Over-fitting Risk ↓

Always.

19

“I can calculate the motion of heavenly bodies but not the madness of people.” said Sir Isaac

Newton. Almost four centuries later, the sentiment is still strong among practitioners of

predictive analytics that forecast human behavior.

It is true – we lack theories and empirical evidence to come close to predicting human behavior

with the accuracy found in the physical sciences, and making choices based on models of human

behavior is risky. But we have identified ways of quantifying and analyzing that risk.

One of my projects back when I worked in actuarial science was to quantify a range around the

expected paid losses (i.e., the portion of losses that the insurer actually pays). I set to work

analyzing the historical data and began attempting to quantify the range.

Process Risk After fitting a model to the data, my first approach was to estimate how much variability seemed

to be inherent in the process. I quantified the range by estimating the error term of the model –

i.e., the volatility observed in the actual data around the model fit – and assuming a distribution

for the error term. This type of variability is often referred to by actuaries as “Process Risk.”

Parameter Risk I proudly shared my results with the group, but then another issue was raised. In order to

compute an error term, I had to assume a model with specific, fixed parameters. What if those

parameters, which had been calibrated to a sample of data, were not accurate? On top of the

“Process Risk,” there is additional risk due to uncertainty surrounding the model parameters –

“Parameter Risk.” Adding a bit more statistical rigor to my analysis, I came up with an estimate of

this term.

Model Risk With both process and parameter risks taken into consideration, I was confident in the range

around the expected loss estimate. Then I was asked a final question. What about “Model Risk”?

What do we do if the assumed functional form or structure of the model does not reflect the

How Marketing Mix Modeling and Simulation Account for Uncertainty

20

reality of the underlying process? This one had me stumped. After months of analysis and

research, I could not solve this puzzle. Actually, it turns out the solution to the “Model Risk”

predicament had alluded the entire group.

An outline of the three types of risk.

In statistical models, the different types of risk can be quantified in a formulaic manner. In

simulations – different types of risk can be assessed in an empirical manner – based on more

iterative approaches.

So how does simulation deal with the various types of risk?

Monte Carlo analysis assesses Process Risk This involves running multiple iterations of a simulation with the same initial conditions. The

simulation can evolve differently each time due to the probabilistic and path-dependent nature of

the approach. We can then observe the inherent risk in a given strategy.

Sensitivity analysis assesses Parameter Risk This entails building a series of simulations with inputs that are tweaked in each run. Variation

in simulated outcomes may be observed as the inputs are adjusted. We can then assess how

sensitive outcomes are to shifts in initial conditions or assumptions for which we are uncertain.

Testing different rules of the simulation assesses Model Risk Assessing Model Risk requires critical thinking about the fundamental assumptions made in the

simulation framework. Have we considered all the relevant competitors or drivers of choice? Do

21

we understand the consumer segments and all the touchpoints that affect them? We should also

consider the rules of the simulation. Are the underlying algorithms relevant in our real world

market? Are our assumptions about how consumer decision making, paid media effects, and the

social network accurate? These questions raise challenges that cannot be addressed overnight.

Model Risk is best mitigated through collaborative, creative, and strategic thinking.

So, Newton is still right: We cannot account for all types of risk and uncertainty. This is not just

because of the limitations of our analytical methods, but due to the shear unpredictability of a

complex, ever-changing world. The best we can do is to decipher patterns and build insights

through data analysis and statistics, research and observation of the world around us, and our

intuition. Simulation can help in our efforts to better understand the uncertainties and risks

involved in markets and business strategies. And these efforts will be most successful with a

team that is open, adaptive, knowledgeable, persistent, and creative.

22

A common objection that we hear during discussions about agent-based modeling (ABM) is that

agent-based models are over-specified and therefore their findings are invalid. Although it is

possible with ABM — as with any other modeling technique — to encounter the issue of over-

fitting, building the simulation diligently will minimize the risk of over-specification (more details

on how to minimize the risk of over-specifying are given in an earlier essay). In fact, agent-based

models often have a lower risk of over-specification than marketing mix models. In this essay,

we will explore an illustrative example.

A metric that is commonly used to assess whether or not a model is over-specified is the ratio of

calibration data points to model parameters. If there are too few data points per model parameter,

the model is over-specified. In an extreme case where there is one parameter for every

calibration data point, the model can achieve a perfect fit. The parameters, in a sense, would just

be a transformed restatement of what happened historically. This model would not be very useful

for making any forward-looking projections or drawing any conclusions about what happened in

the past.

Let’s compute the ratio of data points per model parameter for example agent-based and

marketing mix models below:

Marketing Mix Model Calibration Data Points:

Marketing mix models are often fit to a time series of a brand’s sales. In this example, let’s

assume the model is fit to two years of historical weekly sales data. That gives us:

2 years x 52 weeks = 104 calibration data points

Calibrating Model Parameters:

Marketing mix models are typically fit by linking changes in sales to different marketing

activities. Let’s assume there are 8 touchpoints that are being analyzed. There will be a

coefficient that defines the impact of each touchpoint on sales. Additionally, each touchpoint‘s

Over-Specification: Comparing Agent-based and Marketing Mix Models

23

marketing activities can also be modulated by parameters that define the point of saturation and

decay rate of the marketing impact. Taken together, there are 3 parameters defining each of the 8

touchpoint effects plus a y-intercept – so a total of:

8 touchpoints x 3 parameters per touchpoint + 1 = 25 calibrating model parameters

Ratio of Data Points to Model Parameters:

104/25 = 4.2 data points per model parameter

Agent-Based Model Calibration Data Points:

Concentric Market® simulations are often fit to multiple time series. These time series

correspond to KPIs across competing brands. Let’s assume the simulation is fit to one year of

historical weekly sales data, but not just for one, but for six competing brands. Let’s also assume

that the simulation is fit to tracking data on consideration and perceptions for each of the six

brands. However, in this case, let’s say the tracking study was only run quarterly. The total

number of calibrating data points is then:

6 brands x 52 weeks of sales + 6 brands x 4 quarters of consideration + 6 brands x 4 quarters of

perception = 360 calibration data points

Calibrating Model Parameters:

Concentric Market simulations are built by integrating data from various sources and calibrating

a set of parameters to available calibration data. Let’s assume that the settings defining

consumer behaviors and initial preferences and consideration levels are provided from a

consumer survey or tracking study and are therefore constrained. The parameters that are used

to fit the simulation are then based on three types of settings:

1. Touchpoint impacts on consideration and perceptions: 4 per touchpoint

2. Consumer consideration and perception decays: 2 in total

3. Decision-making parameters: 2 in total

4. 8 touchpoints x 4 parameters per touchpoint + 2 + 2 = 36 calibrating model parameters

24

Ratio of Data Points to Model Parameters:

360/36 = 10 data points per model parameter

In the example above, the risk of over-specifying the ABM is lower than the risk of over-specifying

the marketing mix model. Despite the fact that the ABM is fitting a time period that is half the

length, the number of calibrating data points per parameter is over twice as high as the marketing

mix model. This is because an ABM fits an entire market as opposed to one particular brand in

isolation.

Ultimately, there are a number of approaches that may be taken to build confidence in the validity

and usefulness of a given model. These approaches include alternative specification formulas

like a log function, calibration against multiple time series, holdout forecasts, and in-market tests.

By diligently taking such approaches, our users consistently build valid and useful market

simulations with agent-based modeling.

25

Cross-channel effects occur in situations where the sum of the parts is not equal to the whole.

These effects can be positive (synergistic) or negative (inefficient).

For instance, imagine that you have two media channels at your disposal: TV and Facebook.

Should you run them separately or together? The answer to that question depends on the nature

of cross-channel effects between these two channels: whether there is a positive or negative

interactive effect from running these two channels together. If the sales lift you get from running

these two channels together is different from what you would get if you ran each channel

separately and added the sales lift you got from each, then there is a cross-channel effect. The

types of cross-channel effects that can occur are defined as follows:

Synergistic Cross-Channel Effect

A case when the cross-channel effect results in a synergy

In this case, the sum of the parts is less than the two considered as a whole. There is some extra

lift in sales that results from a synergistic interplay between both channels.

Attribution to What? Understanding Cross-Channel Interactions

26

Inefficient Cross-Channel Effect

A case when the cross-channel effect results in an inefficiency

In this case, the sum of the parts is greater than the two channels working together, which has

created some inefficiencies.

The cross-channel effect is therefore defined as the difference between the individual lifts and

the total lift:

[Lift in Sales from TV] + [Lift in Sales from Facebook] + [Cross-Channel Effect] =

[Lift in Sales from TV & Facebook].

The effect of marketing activity on sales is often analyzed using regression techniques using

sales as a dependent variable. Cross-channel interactions may be quantified with regression by

including an interaction term in the model specification. This analytical technique is useful for

measuring the effects of media, i.e., allowing for sales attribution and cross-channel interactions

to be computed. But the task of understanding what mechanism actually causes the cross-

channel effects remains a mystery.

To fully understand the mechanism that drives cross-channel interactions – we need to

recognize that the linkage between marketing activity and sales lies in the mind of a consumer.

The consumer’s decision-making behavior is what ultimately drives sales. So how do consumers

choose?

27

Consumer Decision-Making is the Key to Cross-Channel Effects The growth and acceptance of behavioral economics as a science behavior has been instrumental

in allowing us to take some simple rules to create simulated models of consumer decision-

making that take into account heuristics and rationality. This knowledge has allowed us to build

models to measure consumer behavior that go beyond notions of perfect equilibrium and neo-

classical economics that were never originally designed for this purpose.

In simple terms, consumers make decisions based on options they would consider, and select the

option they perceive most highly on attributes that are most important to them (based on the

behavioral principle of utility maximization). This view assumes a path in the consumer-decision

making process as follows:

The components of the consumer decision-making process lead to the different cross-channel effects

This approach means that both brand consideration and brand perceptions are important in

driving sales. Some touchpoints may be more effective at building consideration and others at

building perceptions. It won’t do the marketer much good to focus on building consideration if the

product is perceived very poorly – a perception or satisfaction bottleneck that may not be solved

by throwing more media at the problem. Likewise, it might be unhelpful to attempt to build

deeper engagement for a brand no one recognizes – a consideration bottleneck.

Synergies and inefficiencies may result from the consideration stages that exist in the complex

non-linear path to consumer decisions. Inefficiency may occur when activities focused on

consideration-building or perception-building continue when brand consideration or brand

perceptions are already maximized. We may saturate consideration and observe that any

additional marketing yields no incremental benefit. A synergy may occur when touchpoints that

28

build consideration and others that build perceptions are used in conjunction. We will get more

out of our perception-building media campaign, for instance, if more of the consumers are made

aware of the brand.

In short, inefficiencies may occur when we concentrate too much on one step in the consumer-

decision making process, and synergies may occur when we balance our efforts between different

steps. See below for a mapping of TV and Facebook from our example.

A mapping of two channels on two criteria: how well they affect consideration and perceptions

Building understanding of what drives synergies and inefficiencies within a marketing plan can

bring valuable additional insight to strategic decision-making. Thinking about the path in the

non-linear consumer-decision making process – and the various bottlenecks and saturation

points therein – may allow a team to consider new creative strategies that balance consideration

building and perception building activities. Simulations may then serve as an aid to testing these

hypothetical strategies prior to implementation to include interim metrics such as brand

consideration, brand perceptions, and word-of-mouth activity.

So, as you think about the attribution of all of your brand activities, ask yourself: attribution to

what?

29

Sales attribution is a common diagnostic used in evaluating marketing effectiveness. This

involves decomposing a brand’s sales into “Base” and “Media” components. Media is often further

divided to attribute sales by distinct touchpoints. The result can be visualized as a stacked

attribution chart as follows:

Stacked Attribution Chart

This chart provides an indication of how much touchpoints are lifting sales and therefore can be

used to assess the Return on Investment (ROI). A planner may use these outcomes to change the

mix, putting more weight into the touchpoint with greater ROI.

Simulation makes it possible to dive deeper into attribution analysis to provide more insights and

further guidance on marketing strategy. Here are four more detailed views of the stack that

simulation provides:

1. Understanding how creative and messaging impact the effectiveness of each touchpoint.

2. Identifying the synergies and inefficiencies that exist between channels.

3. Decomposing the base to understand the role of earned media and brand history.

4. Placing the attribution into a consumer-focused competitive context.

From Optimization to Strategy: Deeper into the Attribution Stack

30

Creative Execution and Messaging Emphasis The concepts of creative execution and messaging emphasis allow us to go beyond analysis of

levels of investment and reach to analyze the effectiveness of media. Creative execution defines

how well the ad is created — the production value and how effectively it communicates the

message. Messaging emphasis defines what attributes of the product are the focus of the ad. For

example, is the ad part of a promotional campaign or a branding campaign? Unlike most

marketing mix modeling, simulation allows us to test the effects of different creative executions

and messaging emphasis and decompose each individual touchpoint contribution further.

Creative Execution and Messaging Emphasis

Cross-Channel Effects Touchpoint impacts may interact with one another through duplication or social network effects.

Inefficiencies occur when reaching the same audience via multiple touchpoints leads to no gain.

Synergies occur when messaging on two touchpoints yields better results than using each

touchpoint on its own. Simulation can serve as a guide to better understanding and replicating

these interactions in the consumer’s mind and provide a platform to test different approaches to

fully leverage synergies and avoid inefficiencies.

31

Cross-Channel Effects

Decomposing the Base The “Base” is often the forgotten piece of the attribution stack – an unexplained variance that

does not yield any actionable insights. Running a simulation that incorporates the effects of

earned media – word-of-mouth (new restaurant opens up) and product experiences (the joy of

driving your automobile) - plays a major role in driving a brand’s performance. Sometimes the

base is simply a matter of consumer inertia (think about insurance), seasonality (canned soup), or

a brand’s history (Coca-Cola vs. Pepsi). Better understanding the base may provide strategic ideas

on how to avoid erosion of the base or how to keep it growing strong.

Decomposing the Base

32

Considering the Competition Sales outcomes are the aggregate result of many individual consumer decisions. Consumers

make choices among a set of alternatives – and so in many cases a brand’s performance is just as

much dependent on what its competitors do as what the brand does itself. Building a simulation

of an entire market where simulated consumers choose among competitive alternatives allows

for an assessment of how a brand will fare in different competitive environments. If a brand

positions itself appropriately, it may rapidly grow by cannibalizing competitive share. If not

positioned appropriately, the entire attribution stack may sink into oblivion (think about

Blockbuster Video with the emergence of Netflix and Redbox). Considering competition is key to

evaluating strategies to boost and maintain market share over the long run.

Considering the Competition

Conclusion – Moving beyond Optimization to Strategy The initial view of the attribution stack allows for an assessment of ROI that can guide decisions

around how much to invest in touchpoints. However, assessing the stack from the lens of

creative messaging, cross-channel synergies and inefficiencies, earned media and brand history,

and the competitive environment allows for a richer understanding of how a brand is likely to

perform in its market. Simulation provides flexibility in approach to plan and react strategically

in a variety of ways.

33

There are a variety of approaches used to quantify the impact of marketing activities and attribute

sales to touchpoint activities. Such methods include:

1. Marketing Mix Modeling

2. Test and Control Group Experiments

3. Attribution Modeling

4. Agent-based Modeling

In marketing mix models, sales or some other KPI of interest are regressed against marketing

activities to understand how changes in the KPI are related to changes in marketing activities.

The resulting coefficients define the sensitivity of the KPI to touchpoint activities. From this

model specification, the quantity of sales to attribute to a given marketing activity may be

derived.

In test and control experiments, one group (the test group) is exposed to a marketing treatment

while another group (the control group) is left unexposed. Ideally, these two groups are otherwise

completely the same. The difference in outcomes (sales, response rate, or other KPIs) between the

two groups is then attributed to the marketing treatment.

In attribution modeling, individual level data is used to assess what individual touchpoint

activities (commonly digital ads) led to a sale or some other action. These models typically link a

KPI (such as a sale or website visit) to one or some combination of activity that preceded it. These

accounting approaches include last touch, first touch, or some other weighting across touchpoint

activities.

In Concentric Market® agent-based models, the dynamics of a marketplace are recreated by

simulated consumers who decide between alternatives, interact in a social network, and

experience marketing. In this way, the impacts of marketing activities are considered in the

context of all interactions a consumer may have with brands in a market – including not just paid

marketing for a single brand, but competitive actions and earned media.

Marketing Effectiveness Measures: Know Your Options

34

Often, we are asked about how agent-based simulation compares to marketing mix modeling.

How do the methodologies compare? Although parallels between the two exist in the context of

planning paid investments in marketing, the question is flawed because the two methodologies

serve very different purposes.

Marketing Mix and Simulation for Planning Investments in Marketing Marketing mix modeling is a statistical approach that explains variation around some baseline

value in a dependent variable (often times sales) due to marketing activities. A regression is

constructed between time series of sales data and marketing to attempt to correlate changes in

one to another. The model’s coefficients are calibrated to minimize the error between actual and

modeled sales. The model is then applied to answer questions around marketing investment

level and mix.

Agent-based modeling is an approach based on simulation that recreates a market and its sales

from the bottom-up. Interdependent variables within the simulation recreate the dynamics of the

market. Each sale in the simulation results from the action of an individual simulated agent. The

initial conditions and parameters define how consumers behave, influence one another, make

decisions, and interact with media. The parameters of the simulation are adjusted to minimize

error between actual and simulated sales. The calibration process confirms that the simulation is

an accurate representation of the world. The simulation is then applied to answer a broad set of

questions about marketing and product development resource allocation.

A number of parallels between the two methodologies often create confusion about their

similarity. Each has coefficients or parameters that are adjusted to minimize error in an attempt

to better reflect reality. Each approach may be used to guide decisions on paid marketing

investments. The key distinction is that marketing mix analyzes a single brand’s activity in

aggregate, while agent-based simulation recreates a brand’s marketplace based on individual

consumer actions. The differences are summarized further below:

Marketing Mix Modeling and Agent-Based Simulation – What Is the Question?

35

Differences Marketing Mix Modeling Agent-Based Modeling

Approach Top-down regression Bottom-up simulation

Scope One brand Multiple brands, multiple

segments

Outputs One metric Multiple metrics

Forecasts Deterministic Probabilistic

Inputs Quantitative time series Quantitative or qualitative info

Marketing mix modeling is well-suited to answer questions related to the optimization of paid

marketing investments. Agent-based simulation provides the capability to answer a broader set

of strategic questions about resource allocation.

Marketing Mix and Simulation Work Well Together We have seen a number of cases where a decision maker has both a marketing mix model and an

agent-based simulation in their toolkit for making decisions. In this context, simulation is used to

extend marketing mix to answer more business questions. A typical process is described below.

The marketing mix model is calibrated to specify the impact of the different marketing channels

a brand uses. The coefficients of the marketing mix model are then used to guide the selection of

parameters for the agent-based simulation. The simulation is tuned to replicate a marketing mix

model’s response curves and marketing attribution results, while also calibrating to sales and

other KPIs. At this point the simulation is calibrated to multiple objectives and also incorporates

insights on consumers from research or other data.

The mix model or simulation may now be used to answer questions around marketing

investment and mix. The simulation may be applied to address a range of questions. Because a

simulation incorporates multiple KPIs, segments, attributes, competitors, and probabilistic

outcomes it may extend mix to do the following:

36

• Interdependent KPIs: Consider trade-offs between sales, perceptions, consideration, and

word-of-mouth.

• Consumer segments: Develop a segment targeting strategy.

• Attributes driving choice: Test creative messaging.

• Competing alternatives: Respond to competitive actions and/or optimize a portfolio.

• Probabilistic outcomes: Assess risk based on a distribution of likely outcomes.

In summary, marketing mix modeling is an approach for performing optimization of paid

marketing investments for one brand in a stable environment. Simulation explores a broader

range of strategic options that consider the competitive market and consumer behavior. So, the

question of how marketing mix compares to simulation always comes back to what questions you

need to answer as a decision-maker.