expect great answers

Menu-Based Choice modeling (MBC): a practitioner’s comparison of different methodologies

Sawtooth Software Conference, March 2012

Carlo Borghi, Paolo Cordella, Kees van der Wagt and Gerard Looschilder

Menu-based choice modeling – the next big thing

Sawtooth Software has recently launched its new Menu Based

Choice modeling software. Although the idea of build-your-own

exercises has been around for a while, the launch of a new tool from

Sawtooth Software usually causes a lot of excitement and uptake of

use.

As practitioners, we at SKIM want to be ready for the avalanche of

projects, so we started to look into pros and cons of several analysis

approaches.

2

Menu-based choice modeling – the next big thing

Look around

Menu-based choices are everywhere and

are becoming increasingly common

3

4

Whopper $3.50 California W. $ 4.50

Omega3 $3.75 Chicken Deli $ 3.50

Cheddar $0.50 American cheese

$ 0.75

Crispy Onions

$1.50 Bacon

$1.50

Curly fries $1.25

French fries $1.05

✔ ✔

✔

✔

✔

Total price $ 8.50

Menu-based choice exercises are found in areas where combining items matters

• Menu optimization in fast food/branded restaurant chains

• Telecom services bundling

• BYO computers (e.g. Dell)

• Optional features pricing optimization in automotive market

• Add-on services in the financial and insurance services industry

Menu-based Choice Modeling exercises deliver item-level forecasts of performance in these markets

It can deliver:

• Demand curves on an item level among many items

• Forecast revenue and find the optimal price for all items on the menu

• Measure uptake and decide whether to add a new item to your portfolio

• Cross-effects price sensitivity and cannibalization effects

• Does decreasing the price of single items hurt full menu sales?

• Most often chosen combinations and their prices

• Suggesting which items to bundle

• Insight in budget constraints

• How many items can we stuff in a bundle before we exceed the decision

maker’s budget?

6

At SKIM, we’re practitioners. We would like to understand how MBC works in our practice

In particular, we would like to better understand the analysis

procedure. At first sight, we loved the beta version of the Sawtooth

Software tool, but we wanted to investigate more.

So we developed an alternative analysis approach, and applied it to

a study into the consumer’s willingness to pay for features of a

notebook computer.

This presentation contains a comparison of results on aspects of

internal validity.

7

SKIM’s Menu Based Choice exercise

We apply the approach to a study into

consumer features of computer notebook

8

9

No glare screen

Wireless speakers

Easy keys

On-screen keyboard

spotlight

DVORAK keyboard

External battery

indicator

High quality touch-

screen

External radio w/

speakers

Universal plug for

US, EU, UK

3D-ready HD

webcam

Eye scanner

Gold-plated jack

Technical advancement has

brought new vistas of safety

and security and today it is

very easy to make your

laptops and notebooks safe

and secure with technologies

such as fingerprint readers,

face recognition, eye

scanners etc.

Eye recognition ensured only you can access your laptop

through a fast and accurate scan of the retina.

The laser scanner is conveniently positioned on top of the

screen, next to the webcam

This pilot application had the following specifications:

• 12 consumer

features

(2 levels: On/Off)

• 12 price attributes

(3 levels)

• 1 notebooks core

attribute (3 levels)

• 1 none option

9 choice tasks:

• 7 random tasks

• 2 hold out tasks

to estimate

predictive validity

10

There are 3 price

levels per feature,

varied in accordance

with an orthogonal

research design

26 attributes

7

9 Choice tasks

3 price levels

Sample size: 1408

There are various models to analyze MBC data:

As presented in Bryan Orme’s paper

“Menu-Based Conjoint Modeling Using Traditional Tools” :

• Exhaustive Alternatives Model

• Serial Cross Effect Model

Both models have drawbacks that we thought we could solve using

SKIM’s method:

• Choice Set Sampling Model

11

Exhaustive Alternatives Model All possible ways to choose options are included

in the choice set.

• This model formally recognizes and predicts the

combinatorial outcomes of menu choices.

• The dependent variable is the choice of a

combination using a single logit-based (MNL)

model

• All possible combinations of options are coded as

one attribute where each level is a combination :

• e.g. with 3 on/off options, this attribute would have

2^3=8 levels

• Price: one price attribute for each option (or one

total price attribute)

12

Drawbacks

The number of possible

combinations grows

exponentially with the

number of options (2ⁿ

dichotomous choices),

transcending into a

problem of computational

feasibility.

Serial Cross-effect Model

The choice of each option is modeled

separately

• The dependent variable is the single

choice of a feature

• N different logit models predicting the

choice of option X as a function of:

• Price of option X

• All other significant cross effects

13

Drawbacks

Only significant

cross-effects should

be included -

meaning they have

to be detected

beforehand

We thought of solving it by introducing a ‘hybrid’ approach: Choice Set Sampling approach

• Like in the Exhaustive Alternatives Model, we consider the full choice set

with all possible combinations of options. However:

• we code each feature and its price as separate attributes (instead of a unique

attribute with all combinations as levels);

• we use importance sampling* – we consider a random sample from the set of

all chosen combinations

• Similarly to the Serial Cross-Effect Models, we also consider whether a

respondent chose an option at various price points.

14

* See importance sampling Ben-Akiva and Lerman (1985)

Coding the “sampling of alternatives” approach

1. In our model there are a total of 3*2^12=12888 possible combinations. However, “only” 4560 were chosen at least once.

Note: » Each feature is either included in the combination (1) or not (2)

» Option prices are alternative specific

2. We draw a random sample from this choice set. It is basically still a single logit-based (MNL) model where the dependent variable is the choice of the combination.

15

Combination # Core Feature 1 Price 1 Feature 2 Price 2 Feature 3 Price 3 ... Feature 12 Price 12 Choice

1 3 2 0 1 3 2 0 ... ... ... 0

2 1 2 0 2 0 1 2 ... ... ... 1

3 2 1 1 1 3 2 0 ... ... ... 0

... ... ... ... ... ... ... ... ... ... ... ...

4560 3 2 0 1 3 2 0 ... ... ... 0

Coding the “sampling of alternatives” approach

3. Each task is codified with 33 concepts/combinations drawn from

the sub-sample, with:

• The chosen alternative in each task

• 32 combinations randomly sampled from the choice set of all chosen

combinations

16

CASEID Task# Concept# Core Feature 1 Price1 Feature 2 Price2 ... Response

1 1 1 1 1 2 1 3 ... 0

1 1 2 1 2 0 1 3 ... 1

1 1 3 1 2 0 2 0 ... 0

... ... ... ... ... ... ... ... ... ...

1 1 33 2 2 0 1 3 ... 0

Coding the “sampling of alternatives” approach

4. In addition, our model is “hybrid” because we add extra dummy tasks for each respondent:

• For each choice task, we add 12 dummy tasks, one per feature

• We check whether a feature has been chosen at a specific price point

• No explicit modeling of cross effects between features

• This coding contains the information that respondent 1 in task 1 chooses feature 1 at price points 1, while she does not choose feature 2 at price point 3, and so on. Therefore we embed a price barrier in our model which amplifies accuracy in price sensitivity estimation.

17

CASEID Concept# Core Feature 1 Price1 Feature 2 Price2 Feature 3 Price3 ... Response

1 1 1 1 1 2 0 2 0 ... 1

1 2 1 2 0 2 0 2 0 ... 0

1 1 1 2 0 1 3 2 0 ... 0

1 2 1 2 0 2 0 2 0 ... 1

1 1 1 2 0 2 0 1 1 ... 0

1 2 1 2 0 2 0 2 0 ... 1

Analysis steps of SKIM’s Choice Set Sampling approach

• Using this setting we run HB estimation, so we can estimate utilities

for:

• Each feature (present/not present; 12 utility values and their mirrors)

• Each price level for each feature (3*12 utility values)

• None option (nothing is chosen; one utility value)

• We build a simulator in Excel, based on either Share of Preference

(SoP) or Share of First Choice (SoFC) with which we have:

• Single feature choice prediction

• Combinations choice prediction

18

Serial-Cross effect model

• Using Sawtooth Software’s MBC we build 12 different models for

each feature choice

• We could not find any significant cross-effects between the

features, both using counts and aggregate logit

• We use HB estimation and we simulate:

• Single Feature Choice predictions using Draws and Point Estimates

• Combinations choice predictions using Draws, Point Estimates and

Weighted Draws.

19

All approaches can be used to answer the same business question

That’s why we compare the approaches to see:

• Which approach delivers the highest validity?

And because as practitioners, we often find ourselves dealing with

demanding clients and strict deadlines, so that we don’t just need

approaches that work but that are also efficient and as easy to apply:

• Which approach is most efficient to a practitioner?

20

Which approach has the highest validity?

The results - Choice Set Sampling vs Serial Cross-Effect model

21

Results 1: Single Features Choice Predictions The Hold-out choice tasks suggest a similar performance

22

Hold-out 1 Hold-out 2

R-Squared MAE R-Squared MAE

Serial

Cross-Effect

Model

HB, Point Estimates 0.991 0.9% 0.990 1.0%

HB, Draws 0.991 0.9% 0.992 1.1%

Choice Set

Sampling Model

HB, First Choice 0.987 1.6% 0.989 1.7%

HB, Share of Preference 0.984 1.5% 0.981 1.5%

Both approaches have a very low MAE on the hold-out tasks

0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

Option 1 Option 2 Option 3 Option 4 Option 5 Option 6 Option 7 Option 8 Option 9 Option10

Option11

Option12

Observed Serial cross-effect model (Draws) Choice set sampling model (SoP)

23

Fre

q.

of

ch

oic

e

Holdout task - 1

No structural consistency in errors

0.0%

0.5%

1.0%

1.5%

2.0%

2.5%

3.0%

3.5%

4.0%

4.5%

5.0%

5.5%

6.0%

Option 1 Option 2 Option 3 Option 4 Option 5 Option 6 Option 7 Option 8 Option 9 Option10

Option11

Option12

Choice set sampling model (First Choice, MAE = 1.5%) Serial cross-effect model (Draws, MAE = 0.9%)

24

Ab

so

lute

err

or

Holdout task - 1

• Hit rate: % of respondents for which the choice on the option was predicted correctly

• 2 holdout tasks x 1408 respondents = 2816 observations for the hit rate

25

86.8% 91.1%

86.5% 84.4% 87.5%

84.9% 86.6% 86.5% 90.9% 89.0% 87.0% 87.2%

86.4% 90.7%

85.8% 84.1% 86.5% 85.1% 87.0% 86.5% 90.2% 88.9% 86.8% 86.7%

0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

70.0%

80.0%

90.0%

100.0%

Option 1 Option 2 Option 3 Option 4 Option 5 Option 6 Option 7 Option 8 Option 9 Option10

Option11

Option12

Choice set sampling model (First Choice) Serial cross-effect model (Weighted Draws)

Hit r

ate

The individual hit rate is almost the same across the two hold out tasks

Both models fit individual choices of combinations

41.0%

63.2% 66.2% 67.9%

83.6% 86.4%

41.3%

63.6% 66.2% 67.7%

83.5% 86.2%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

All 12 optionchoices predicted

correctly

At least 11choices predicted

correctly

At least 10choices predicted

correctly

At least 9 choicespredicted correctly

At least 7 choicespredicted correctly

At least 8 choicespredicted correctly

Choice set sampling model (First Choice) Serial cross-effects model (Weighted Draws)

26

Result 2: Feature combination predictions

So we can conclude that both approaches are viable tools for MBC analyses

Both models

• Are able to predict accurately hold-out choice tasks on aggregate

level

• Are extremely effective to predict individual choices of single

options and combinations

So both models are viable tools for analyzing MBC data.

• But which one is the most effective for practitioners?

27

Which approach is most efficient to a practitioner?

So both approaches work and it comes down to efficiency.

28

Choice Set Sampling approach – Benefits and Drawbacks

Benefits

• One model to estimate, one

model to simulate

• No need to make a call on

which cross-effects to

include

• Explicitly predicts choice of

combinations

Drawbacks

• Complex procedure: time

consuming set up for estimation

• Simulations are computationally

intensive

• Simulators are not very handy

tools for clients

29

Serial Cross Effect Model – Benefits and drawbacks Benefits

• Dedicated software

available

• Explicit inclusion of cross-

effects in the model

• Easy simulation tools

Drawbacks

• Learning curve of

understanding how to interpret

the significance of cross /

interaction effects and their

inclusion in the model – it

takes art and craft to build an

accurate model

• Once cross-effects are

included in the model, they

hold for all respondents

30

To conclude: Sawtooth Software’s Serial Cross-effect model is the practitioner’s choice

• We would recommend using Sawtooth Software’s Serial Cross-effect

model and software package,

• After the initial learning, it’s an easy to apply and time-effective solution, thanks

to its dedicated software

• One just needs to invest in the learning curve of making the call about the

significance and meaning of interaction/cross effects

• If you want to use the Choice Set Sampling model, be prepared to invest

time to create dedicated tools

31

twitter.com/

skimgroup

facebook.com/

skimgroup linkedin.com/

company/skim

youtube.com/

skimvideos skimgroup.com

contact us or follow us online!

Carlo Borghi, Paolo Cordella, Kees van der Wagt and Gerard Looschilder

www.skimgroup.com | +31 10 282 3535