Conducting Market Simulations Copyright Sawtooth Software, Inc

Conducting Market Simulations

Copyright Sawtooth Software, Inc.

Why Conduct Market Simulations?

• Examining just utilities and importances only gets you so far

• Average utilities cannot tell the whole story

Which Color is Preferred?

• Consider the following utilities:

Blue Red YellowRespondent #1 50 40 10Respondent #2 0 65 75Respondent #3 40 30 20

---- ---- ----Average: 30 45 35

• Red has the highest average preference• But, does any one respondent prefer red?

Which Color is Chosen?

• Assume each respondent “chooses” preferred color:

Blue Red Yellow “Choice”Respondent #1 50 40 10 BlueRespondent #2 0 65 75 YellowRespondent #3 40 30 20 Blue

---- ---- ----Average: 30 45 35

• Blue “chosen” twice, Yellow once

Why Conduct Simulations? Competitive Effects Matter!

• Assume 80% of market prefers round widgets, and 20% prefers square ones

• Which should you take to market?

• In the absence of any other information, round would be the logical choice

• But what if there currently are 10 competitors in the market, ALL only offering round widgets?

Why Conduct Market Simulations?

• Simulations better reflect real-world behavior

– Represent idiosyncratic preferences of segments and individuals (remember, you don’t have to appeal to the “fat” part of the market to carve out a profitable business)

• A “choice laboratory” for testing multitude of real-world possibilities

• Results expressed in terms that make sense to management and are actionable

Three Important Qualities of Good Market Simulator

• Differential Impact

• Differential Substitution

• Differential Enhancement

Differential Impact

• Differential Impact: “Impact of a marketing action depends on the extent that the alternative is near the purchase threshold”

• A simulator that reflects Differential Impact focuses attention on customers on the cusp of buying company’s offer

• Respondents either extremely unlikely or extremely likely to purchase are less affected

First Choice and Differential Impact

• First Choice model reflects differential impact at the extreme: as stair-step probability function.

• Once product alternative reaches certain threshold of utility, it is “chosen” with certainty

• Moderate changes in product’s utility have no impact on respondents with relatively low or high probabilities of purchase

• Gain in market share reflects gains of respondents “on the cusp”

Differential Substitution

• A new product offering should take share disproportionately from similar ones

• Simulators that reflect Differential Substitution are necessary to accurately model actions like product line extensions, or “me too” product imitations

First Choice and Differential Substitution

• First Choice model reflects differential substitution at the extreme: identical products split share (that would be captured by just one of them) exactly in half; with near identical products, winner takes all

• The real world reflects some degree of random buyer behavior, and near substitutes should reflect some degree of “share inflation” beyond that suggested by the first choice model

Differential Enhancement

• Pairs of highly similar product alternatives should display extreme probabilities of choice

• The slightly better product should strongly dominate the other

Differential Enhancement Illustration (under First Choice)

• Consider different vacation opportunities and costs:

Utility

Probability of Choice Paris @$3,000

Paris @$3,001 Italy@$3,500

First Choice reflects extreme Differential Enhancement for difference in choice probability between the slightly dominated pair

First Choice and Differential Enhancement

• First Choice model reflects differential enhancement at the extreme: the slightly dominant alternative of the pair is always chosen

• “Trip to Paris costing $3,000” always preferred vs. “Trip to Paris costing $3,001”

• The real world reflects some degree of random buyer behavior, so we shouldn’t expect the slightly dominated alternative to have a zero probability of choice

First Choice: How Realistic?

• First choice model simple to do, easy to understand, but “patently false” model of consumer behavior

• Extreme model that assumes a product barely preferred over another chosen 100% of the time

• Less efficient use of data: we learn about which product is preferred, but don’t capture anything about relative preferences of not preferred options

– Standard errors of simulated shares relatively higher than logit

First Choice: How Realistic? (Cont.)

• Despite the theoretical problems, there are certain conditions under which First Choice can work quite well

• We’ll later show a more flexible, “tunable” approach

The Fickle Buyer

• Buyers never purchase with 100% certainty the product our model says is most preferred within a set

– Some “random” behavior occurs

– Other influences (e.g. out-of-stock, children in the cereal aisle) can alter choice

– Variety seeking

How to Model Uncertain Behavior?

• For each respondent, perhaps we can estimate continuous probabilities of purchase rather than either 0% or 100% each alternative (vote splitting)

First Choice “Share of Preference”A 0% 10%B 100% 60%C 0% 30%

• But how to do it?

Bradley-Terry-Luce (BTL) Choice Model

• Assume three product alternatives with independent choice probabilities as follows:

A 25%B 20%C 5%

• If these three are the only alternatives available, the probability of choosing A is equal to:

p(A) 0.25-------------------- = -------------------- = 0.50p(A)+p(B)+p(C) 0.25 + 0.20 + 0.05

BTL Caution

• Note that the BTL model requires that independent probability of the product alternative be known

• It is a mistake to use BTL model with conjoint part worths whose scaling do not reflect probability estimate for the product alternative

– e.g. a negative utility sum for an alternative cannot work

IIA

• Independence from Irrelevant Alternatives (IIA)

– Ratio of any two product’s shares should be independent of all other products

– All other product alternatives are irrelevant to that relationship

– An improved product gains share from all other products in proportion to their previous shares

• IIA is also known as “Red Bus/Blue Bus” property

BTL is an “IIA” Model

• Imagine a market with just three ways to get to work:

Red Bus 20%Bike 30%Car 50%

• Assume a new alternative appears (Blue Bus), with an independent choice probability equal to a Red Bus (also 20%)

• New Shares:

Blue Bus 16.67%Red Bus 16.67% (lost 16.67% of its original share)Bike 25.00% (lost 16.67% of its original share)Car 41.67% (lost 16.67% of its original share)

Is IIA Realistic?

• Not realistic to assume that Blue Bus takes share proportionally from all alternatives

• We should expect it takes disproportionately more share from Red Bus than from bikes or cars

• The resulting net share for both buses is 33.34%, rather than 20% when only Red Bus was offered

– Call this “Share Inflation” for similar products

The Logit Simulation Model

• Much more commonly used than BTL model (known as “Share of Preference” in Sawtooth Software)

• Permits estimating continuous probabilities of choice for alternatives, for each individual (vote splitting)

• More flexible than the BTL model, because product utilities aren’t required to be positive probabilities

• Doesn’t require part worth utilities computed under “logit”: i.e. Least Squares Regression, monotone regression, linear programming may be used

Logit Simulation Math

• Probability of choosing alternative A with utility Ua from set of product alternatives {A B C} is:

P(A) = exp(Ua)-------------------------------exp(Ua)+exp(Ub)+exp(Uc)

• Where “exp(Ui)” is the antilog of Ui, also known as raising the constant “e” (2.7183…) to the power Ui

Numeric Example: Logit Simulation

• Assume three product alternatives with the following utilities (after adding up their respective part worths):

A 0.75B 0.00C -1.25

• Share of A:exp(0.75)/[exp(0.75)+exp(0.00)+exp(-1.25)]

= 2.117 / [2.117 + 1.000 + 0.287]

= 62.2%

First Choice vs. Logit

Two Simulation Methods Compared

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Product Utility

Ch

oic

e P

rob

abili

ty

First Choice

Logit

Important Properties of Logit Simulation Model

• IIA

• Product utilities (U sub “i”s) can be negative or positive values

• You can add any constant to all product utilities, and the shares are unchanged

• You can multiply the utilities by a constant, and the relative shares can be “tuned” flatter or steeper

“Tuning” Logit Simulations

• Multiplying all part worth utilities by value greater than unity (1) causes relative shares to become steeper

• Recall previous example with utilities:A 0.75B 0.00C -1.25

• Shares under different multipliers:0.01 1.0 5.0

A 33.6 62.2 97.7B 33.4 29.4 2.3C 33.0 8.4 0.0

Scale Factor

• The multiplier applied to all utilities referred to at Sawtooth Software as the “Exponent”

• With very large scale factor applied within logit simulations, it is essentially equivalent to “First choice” (best alternative gets 100% share)

• Scale factor near zero distributes share essentially equally across alternatives

Scale Factor and Uncertain Behavior

• Recall our discussion that respondents don’t always choose the product alternative with the greatest utility

• “Exponent” (scale factor) within logit simulations allows us to adjust degree of uncertainty we should place on predicted probabilities of choice

• Analysts apply “exponent” after utilities estimated

• But, most utility estimation techniques already have a “scale factor” reflected in the part worth utilities

Different Respondents, Different Scale Factors...

• Consider the following ACA utilities for two respondents:

Respondent #11.535 0.37 2.612 -0.223 -2.050 . . .

Respondent #20.102 -0.097 0.117 -0.055 -0.132 . . .

• Respondent #1’s utilities reflect a much wider range (larger scale factor). His simulated shares will tend more toward 0% or 100%.

• Respondent #2 was less consistent and/or displayed less sensitivity to differences in products than #1 during the ACA survey. His simulated shares will be relatively “flat.”

Utility Estimation Techniques and Scale Factors

• Many utility estimation techniques yield utilities with different scale factors to reflect uncertainty of choices at the individual level:

– ACA (through its “Calibration Concepts” section--but the exponent to best predict choice shares is often > 1)

– HB (for ACA, CBC or traditional conjoint CVA)– CVA (rating-based yes, but rank-ordered data NO)

• Or at the aggregate level:

– Latent Class (segment based)– CBC logit

Tuning to Survey Data or Market Data

• The scale factor evident from utility estimation reflects the degree of uncertainty suggested by conjoint judgments within the questionnaire

• You may choose to adjust the scale factor (for all respondents) by a uniform additional degree– to better fit actual market share information– to better fit “holdout choices” within the questionnaire

• Sawtooth Software’s “exponent” does that

• Or, simply multiply all part worths by the desired scale factor (spreadsheet simulator)

Heterogeneity

• Heterogeneity refers to the fact that people have different preferences

• Market simulation models that reflect heterogeneity are usually more accurate and flexible than those that do not

Heterogeneity Continuum

• Aggregate Model: A single set of part-worth utilities reflects the average preferences of a population

• Disaggregate Models:

– Segment-based models: A set of part-worth utilities is developed to reflect the average preferences of each segment

– Individual-level models: A set of part-worth utilities is developed for each respondent

Heterogeneity Continuum

• Relative heterogeneity for “typical” study:

No Heterogeneity High Heterogeneity

CBC Logit CBC LClass

CBC ICE

CBC/HB

CVA

ACA/HB ACA

CVA/HB

HB/Sum

Aggregate Model(main-effects only)

Individual-Level Models

Segment-BasedModel

Why is Capturing Heterogeneity Good?

• Helps reduce IIA (Red Bus/Blue Bus) problems (displays more accurate differential substitution and cross-elasticities)

• Can reflect (to a degree) higher-order interaction effects even though explicit terms have not been estimated

• Better reflects appropriate Scale Factor for different respondents or sub-groups of the population

• More accurately reflects Differential Impact, Substitution and Enhancement in simulations

IIA and Logit Simulation Methods

• IIA troubles in simulations are significantly reduced by models that capture heterogeneity

– Aggregate Logit: IIA problems at their worst

– Latent Class: IIA problems still significant

– Individual-Level: IIA problems very much reduced

• IIA still holds within the element (i.e. segment or individual) of analysis, but when the choices are aggregated across elements, the resulting shares do not strictly adhere to IIA

“First Choice” vs. “Logit” Simulations

• Both methods reflect Differential Impact, but logit’s uptake smoother and usually more realistic (S-shaped rather than stair-step)

• First Choice often better reflects Differential Substitution (logit suffers from IIA), though sensitivity of the substitution effect may be too extreme

• As for Differential Enhancement, Logit often doesn’t reflect enough dominance for the preferred product (of a similar pair), and First Choice reflects absolute (usually too strong) dominance

Differential Enhancement Illustration (under logit)

• Consider different vacation opportunities and costs:

Utility

Probability of Choice Paris @$3,000

Paris @$3,001

Italy@$3,500 Logit doesn’t do a good job at reflecting appropriate Differential Enhancement (near equal probability of choice between slightly dominated pair)

“First Choice” vs. “Logit” Simulations (cont.)

• First Choice essentially immune to “Share Inflation” of similar products– But First Choice often too extreme, not tunable, and doesn’t use

data as efficiently as logit simulation method (higher standard errors of share estimates)

• Logit Simulation method tunable and more efficiently uses data (lower standard errors of share estimates)– But… Logit falls prey to IIA problems

• Isn’t there a way to get the benefits of both methods?

Randomized First Choice (RFC)

• A Relatively New Technique (1999) that shares much in common with both First Choice and Logit (Share of Preference) simulation techniques

• Can be used with aggregate or disaggregate utilities

• “Splits” shares within the element of analysis, but reflects more accurate substitution effects for similar products than Share of Preference

• Is tunable, both in terms of scale and in terms of appropriate degree of “share inflation” for similar products

Random Effects Model (REM)

Ui = Xi () + i

• Where:

– Xi is the design matrix (dummy coded)

is a vector of part-worths (betas)

i is a random error term

• Translation: Utility of alternative i is a function of the utility of that alternative [Xi()] plus some random error [I] for that alternative. Respondent “chooses” alternative with highest utility

Applying REM to Simulations

• Assume utilities [Xi ()] for Honda (0.70) and Ford (0.65) for one respondent (or element of analysis)

• If the error (i) is zero:

U Honda = 0.70 + 0 = 0.70U Ford = 0.65 + 0 = 0.65

– Then, respondent chooses product with highest utility (Honda)

• Isn’t that just the First Choice model? Yes! Respondent chooses Honda, every time!


• What if the error (i) is not zero?

• Let’s assume i is a random value from 0 to 1, and repeat the simulated choice process 10 times (10 replications) for that one respondent, adding new error each time


• Honda chosen 6/10 = 60%, Ford = 40%

Simulated Choices for 10 Replications

0

0.5

1

1.5

2

1 2 3 4 5 6 7 8 9 10

Replication #

Uti

lity

Plu

s E

rro

r

Honda

Ford


• If i is distributed as Gumbel, then share predictions (after many, many draws) are identical to logit

• What is a Gumbel distribution?

Y = -ln(-ln(x))

where x is rectangular distributed [0<x<1]

Some Random Numbers: Gumbel Distribution

-0.49 0.24 0.270.10 0.33 -1.070.44 2.66 0.29-0.02 0.08 0.110.10 1.02 -0.03-0.41 5.72 -0.270.10 -0.40 2.170.23 -1.13 -1.10

• Mean ~ 0.58, Standard Dev. ~ 1.28

• Right-skewed (not normal) distribution

Role of i in REM

• Variance of i tunes scale of shares in the same way that exponent tunes shares in logit

• The more error, the flatter the shares; the less error, the steeper the shares

Product Similarity Problem

• REM is cool, but we’ve still got the red bus/blue bus problem!

– Error added to product utilities is independent – Adding a duplicate product in a simulation increases the net

share to the twin products

• But if we could somehow make similar products have correlated error terms, we could correct for product similarity. . . .

Product Similarity Based on Level Definitions

• Consider the following products, composed of the following levels of each of 4 attributes:

Att1 Att2 Att3 Att4Product A: 1 3 2 5Product B: 2 3 1 5Product C: 3 1 2 2Product D: 4 2 3 4

• Which Product is the most similar to others?

• Which Product is the most unique?


Ui = Xi ( + a) + p

• Translation: Utility of alternative i is a function of the utility of that alternative (after adding some random error to the part-worths) plus some error added to the product alternative

• Note that a held constant for all alternatives, but p

unique for each alternative


• The a term introduces correction for product similarity

– Consider two products identical on 4 out of 5 levels

4/5 of error contributed by a identical between products for each replication

– These products will compete more closely (highly substitutable)


Ui = Xi ( + a) + p

• If a is zero, then RFC = logit model

• If p is zero, then duplicate products always get their share divided in two. Like a “tunable” First Choice model in that case.

All Three Methods

Three Simulation Methods Compared

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Product Utility

Ch

oic

e P

rob

ab

ility

First Choice

Logit

RandomizedFirst Choice


Summary:

• RFC lets you tune the amount of correction for product similarity between an IIA model and one that splits share in half for duplicate offerings

• It is tunable for scale

• It reflects relatively accurate Differential Impact, Differential Substitutability, and Differential Enhancement effects

Weaknesses of RFC

• Not as easily understood or described as First Choice

• Most correct use of RFC requires carefully defined holdout concepts that reflect differential similarities among the product concepts within the sets

– Default is to apply “attribute” type error only (split shares exactly in half for duplicated offerings)

– Best use of RFC results from properly tuned “attribute” and “product” type error--which is time consuming to do!

Weaknesses of RFC (cont.)

• Assumes every attribute equally treated in similarity measurements and corrections (including price)

• Sometimes possible for a product to be made “worse” (change from a “good” level to a less preferred level) and have its share actually increase (if becoming “worse” means also becoming much more unique than current offerings)

– This problem was much more prevalent and pronounced under the old “Share of Preference with Correction for Product Similarity” model

Weaknesses of RFC (cont.)

• Not a great choice for generating demand curves through sensitivity analysis when the competitive brands (products) all held at the same (i.e. average) price

– creates distortions in the demand curve due to severe product similarities of reference brands held all at the same price

• Also note that with CBC’s “conditional pricing” RFC may count two prices as identical, even though they were technically different due to the customized entry in the lookup table

Which Simulation Method Should I Use?

• If First Choice technique best fits holdouts or market shares, then use that (this is rarely the case)

• If all products are equally similar/dissimilar, then use logit (Share of Preference)

– Note that with a two-product market simulation, the products are always equally similar/dissimilar to one another

• If more than three products in simulation, and different degrees of similarity among them, then probably use RFC--especially if aggregate utilities (logit or Lclass)

Big Questions...• How can I tell which estimation technique (i.e. logit/Lclass/HB) is

best for my data?

• How can I tell which model specification (i.e. main effects only vs interactions, vs. linear terms for quantitative attributes) is best for my data?

• How can I tell which Market Simulation method is best for my data?

Holdouts!

“Holdout” Choice Scenarios• CBC-looking choice-tasks that you put in your questionnaire, but

don’t use to estimate utilities– Suggest include three or more

– See “Including Holdout Choice Tasks in Conjoint Studies” for more information

• You observe % of respondents that choose each product concept

• Use same product specifications as holdouts in market simulator, and predict shares of choice

• Compare observed vs. predicted share of choice

Conjoint Market Simulation Assumptions

• We have interviewed the right people

• Each person is in the market to buy

• We’ve used a proper measurement technique

• Respondents have answered reliably and truthfully

• All attributes that affect buyer choices in the real world have been accounted for

Conjoint Market Simulation Assumptions (cont.)

• Equal availability (distribution)

• Respondents are aware of all products

• Long-range equilibrium (equal time on market)

• Equal effectiveness of sales force

• No out-of-stock conditions

Shares of Preference Don’t Always Match Actual Market Shares

• Conjoint simulator assumptions usually don’t hold true in the real world

• But this doesn’t mean that conjoint simulators are not valuable!

• Simulators turn esoteric “utilities” into concrete “shares”

• Conjoint simulators predict respondents’ interest in products/services assuming a level playing field

Final Conjoint Simulator Suggestions

• Simulators based on individual-level conjoint data (a separate prediction for each respondent) usually more accurate and flexible than simulators based on average (aggregate) data

• Beware of “Red-Bus/Blue-Bus” problem (share inflation for similar products)

• Avoid temptation to make Shares of Preference look like market shares (using external “fudge” factors)

• Don’t forget the assumptions!

Documents

Conducting Market Simulations Copyright Sawtooth Software, Inc