Experimental Designs for CBC - Sawtooth SoftwareFractional factorial plans require fewer observations They assume no or very few interactions When they measure main effects only, we

© 2016 Sawtooth Software, Inc. | www.sawtoothsoftware.com

Webinar

Experimental Designs for CBC


Webinar

Outline

2

Background – designs for linear models

CBC design strategies

Based on orthogonal arrays

Using efficiency calculations

Sawtooth Software’s multi-objective optimization


Webinar

BACKGROUND

3


Webinar

In the absence of an experimental design bad and terrible things can happen:

Terrible: Some of your model coefficients (utilities) may not be possible to estimate, your research becomes a crash-and-burn disaster, dollars and careers can go up in flames

Bad: your utility estimates lose precision

If you don’t get precision through your experimental design, you have to buy it with sample size

Savings in $, €, ¥, £, Rs are good reasons to like precision

Why care?

4


Webinar

Attributes are the characteristics of products and levels are the discrete values those characteristics may take on

Attribute (levels):

Brand (Pepsi, Coke, Dr. Pepper)

Capacity (16GB, 32GB, 64GB)

Price ($90, $125, $159)

Flame-retardant fabric (yes, no)

Attributes and Levels

5


Webinar

Orthogonal designs

“Full factorial” designs

Show every possible combination of predictors

An experiment with three 4 level factors and two 5 level factors (43 x 52) would need observations for 4 x 4 x 4 x 5 x 5 = 1,600 combinations

Fractional factorial plans require fewer observations

They assume no or very few interactions

When they measure main effects only, we call them orthogonal main effects plans (OMEPs)

An OMEP for the (43 x 52) design above uses just 25 combinations or “runs”

Design for Linear Models

6


Webinar

34 OMEP

Example Design

7

Run X1 X2 X3 X4

1 3 3 3 3

2 3 1 1 2

3 3 2 2 1

4 1 3 1 1

5 1 1 2 3

6 1 2 3 2

7 2 3 2 2

8 2 1 3 1

9 2 2 1 3


Webinar

Orthogonality - each level occurs with levels from each other attribute a proportional number of times

Correlation matrix:

Two-way frequencies:

Properties of Example

8

X1 X2 X3 X4

X1 1

X2 0 1

X3 0 0 1

X4 0 0 0 1

X1

0 1 2

X2

0 1 1 1

1 1 1 1

2 1 1 1


Webinar

Addelman (1962) OMEP catalog

Designs account for main effects (of individual attributes) not interactions

Until 1994, the marketing researcher’s bible for creating experimental designs

Hahn and Shapiro (1966) designs allowed some interactions

Kuhfeld’s SAS Technical Report “Orthogonal Arrays” offers an extensive collection plus links to other design databases

Sources of Orthogonal Fractional Designs

9


Webinar

Traditional conjoint analysis used fractional factorial experimental designs to create a set of stimuli (concepts, profiles, cards)

Respondents evaluated these stimuli via ratings or rankings

Regression analysis decomposed the evaluations into utilities for each level of each attribute in the study

We can also use fractional factorials in

Menu-based choice experiments

Situational choice experiments

Choice-based conjoint (CBC) experiments

Using Orthogonal Arrays

10


Webinar

CBC DESIGNS FROM ORTHOGONAL

PLANS

11


Webinar

Adapting fractional factorial designs for CBC experiments

Louviere and Woodworth 1983 – 2J designs

4 more early strategies (Louviere, Hensher and Swait, 2000)

Mix and match 1

Mix and match 2

Labeled mix and match

LMN

Simple shifting (Bunch, Louviere and Anderson 1996)

Flexible shifting (Street and Burgess 2007, Street, Burgess and Louviere 2005)

Designs Based on Orthogonal Plans

12


Webinar

1. Use a fractional factorial design to create profiles

2. Use a second fractional factorial design to decide which profiles to assign to particular choice sets (levels are “absent” and “present”

3. Hadamard matrices or balanced incomplete block designs can be used for step 2 as well

2J Designs

13


Webinar

Pros

Conceptually and mechanically simple

Cons

Designs can get very large

Different choice sets can have different numbers of profiles

Pros and Cons

14


Webinar

Mix and Match 1

15

Algorithm

1. Generate a fractional factorial plan

2. Make some number of k exact copies of this fraction

3. From each of the k+1 equivalent fractions

Randomly select one profile without replacement

This set of K+1 profiles is choice set 1

Prevent duplicates within a set

4. Repeat step 3 until all profiles are assigned to choice sets

This strategy produces an experiment where all choice sets have k+1 profiles


Webinar

The same 9 profiles appear in both the first and second alternative

Example of Mix and Match 1

16

Alternative A Alternative B

Set X1 X2 X3 X4 X1 X2 X3 X4

1 3 3 3 3 1 3 1 1

2 3 1 1 2 2 2 1 3

3 3 2 2 1 2 1 3 1

4 1 3 1 1 1 2 3 2

5 1 1 2 3 3 2 2 1

6 1 2 3 2 3 3 3 3

7 2 3 2 2 1 1 2 3

8 2 1 3 1 2 3 2 2

9 2 2 1 3 3 1 1 2


Webinar

Mix and Match 2

17

Algorithm


2. Make some number of k equivalent (not exact) copies of this fraction*

3. From each of the k+1 equivalent fractions

Randomly select one profile without replacement

This set of K+1 profiles is choice set 1

4. Repeat step 3 until all profiles are assigned to choice sets

This strategy covers more of the design space than Mix and Match 1

*You can do this simply by switching columns, relabeling levels (1 2, 23 and 31) or both.


Webinar

In this experiment unique profiles appear in the two alternatives and we cover twice as many possible profiles

Example of Mix and Match 2

18

Alternative A Alternative B


1 3 3 3 3 2 3 2 1

2 3 1 1 2 1 2 3 1

3 3 2 2 1 1 1 2 2

4 1 3 1 1 2 1 3 3

5 1 1 2 3 3 1 1 1

6 1 2 3 2 1 3 1 3

7 2 3 2 2 2 2 1 2

8 2 1 3 1 3 3 3 2

9 2 2 1 3 3 2 2 3


Webinar

We could create multiple versions of a mix and match design

One could be the design on the previous slide

One could be the previous design where columns 1&2 and 3&4 are swapped in the left hand profile

One could swap codes 3 and 2 in the even numbered columns of the right hand profile and 1 and 2 in the odd columns of the right hand profile

We could make several other versions in a similar way, swapping columns and/or recoding levels

Then we could randomly assign different respondents to different versions

When we do this we explore more of the “design space”

We also allow estimation of some interactions

Versions

19


Webinar

A trick called “labeling” allows us to make alternative-specific designs In each choice set, label the profile chosen from the first of the K+1 designs

with the first brand name

Label the profile chosen from the second design with the second brand name

Etc.

Benefits Brand is now an attribute in your experiment that you didn’t have to build

into the fractional factorial experiment

In analysis, you can have brand-specific effects of price and all other attributes

In other words you can estimate interactions of brand with all other attributes

Alternative-Specific Designs

20


Webinar

Just label the sets of alternatives with brand names

Example of Alternative-Specific Design

21

Bose JBL


1 3 3 3 3 2 3 2 1

2 3 1 1 2 1 2 3 1

3 3 2 2 1 1 1 2 2

4 1 3 1 1 2 1 3 3

5 1 1 2 3 3 1 1 1

6 1 2 3 2 1 3 1 3

7 2 3 2 2 2 2 1 2

8 2 1 3 1 3 3 3 2

9 2 2 1 3 3 2 2 3


Webinar

LMN Designs

22

If we have L levels per attribute, M alternatives in our choice sets and N attributes, build the design from a fractional factorial with M*N columns

For example, if we have 4 alternatives with 3 attributes of 3 levels each we need a design with 12 columns worth of 3 level attributes – a 27 row design from the Addelman catalog would work

In this design variables are independent both within and across profiles

This feature allows measurement of cross-effects


Webinar

3 alternatives, 2 attributes of 3 levels each - this design uses an 18 run fractional factorial design with 7 columns of 3-level variables – we use two per alternative, with one unused column

Note we have an unused column. We need not use every column of a design. We may also use it for other purposes, like creating “blocks” of questions so that no respondent has to answer all 18.

A 33*2 Design

23

Set A1 A2 B1 B2 C1 C2 Unused

1 3 3 3 1 2 3 3

2 3 1 1 2 1 1 2

3 3 2 2 3 3 2 1

4 1 3 1 3 1 2 3

5 1 1 2 1 3 3 2

6 1 2 3 2 2 1 1

7 2 3 2 2 1 3 1

8 2 1 3 3 3 1 3

9 2 2 1 1 2 2 2

13 3 3 2 3 2 1 2

11 3 1 3 1 1 2 1

12 3 2 1 2 3 3 3

13 1 3 3 2 3 2 2

14 1 1 1 3 2 3 1

15 1 2 2 1 1 1 3

16 2 3 1 1 3 1 1

17 2 1 2 2 2 2 3

18 2 2 3 3 1 3 2


Webinar

Shifting Designs

24


2. Results of this plan are the first profile of each choice set

3. Second profile is just the first with all attributes “shifted”

1 becomes 2, 2 becomes 3 so on, with wraparound (highest level becomes 1 again)

Example:

Six 3-level attributes

If first profile is 331232

Second profile is 112313 and third is 223121


Webinar

Starting with a 34 fractional factorial we create additional alternatives through shifting

Note there is no level overlap

Note that a 4th profile would duplicate the first

Shifting Design Example

25

Alternative A Alternative B Alternative CSet X1 X2 X3 X4 X1 X2 X3 X4 X1 X2 X3 X4

1 3 3 3 3 1 1 1 1 2 2 2 22 3 1 1 2 1 2 2 3 2 3 3 13 3 2 2 1 1 3 3 2 2 1 1 34 1 3 1 1 2 1 2 2 3 2 3 35 1 1 2 3 2 2 3 1 3 3 1 26 1 2 3 2 2 3 1 3 3 1 2 17 2 3 2 2 3 1 3 3 1 2 1 18 2 1 3 1 3 2 1 2 1 3 2 39 2 2 1 3 3 3 2 1 1 1 3 2


Webinar

We could choose to shift different columns of the initial matrix differently

For example, we could shift the 1st and 3rd columns by one place and the 2nd and 4th by 2 places

Street and Burgess (2007) call this pattern of shifts, 1212, a “generator”

They show how to use generators to create a variety of designs for main effects and interactions

Like simpler shifting designs, these will not work for alternative-specific effects

Flexible Shifting Designs

26


Webinar

This design uses a 1212 generator

Flexible Shifting Example

27

Alternative A Alternative B Alternative C

Set X1 X2 X3 X4 X1 X2 X3 X4 X1 X2 X3 X4

1 3 3 3 3 1 2 1 2 2 1 2 1

2 3 1 1 2 1 3 2 1 2 2 3 3

3 3 2 2 1 1 1 3 3 2 3 1 2

4 1 3 1 1 2 2 2 3 3 1 3 2

5 1 1 2 3 2 3 3 2 3 2 1 1

6 1 2 3 2 2 1 1 1 3 3 2 3

7 2 3 2 2 3 2 3 1 1 1 1 3

8 2 1 3 1 3 3 1 3 1 2 2 2

9 2 2 1 3 3 1 2 2 1 3 3 1


Webinar

For generic main effects models, shifting and modified shifting give more efficient designs than do mix and match designs

Mix and match designs with enough versions allow estimation of interactions

For models with alternative-specific effects, use Mix and Match 2 or and LMN designs

For models with cross-effects, use LMN designs

Avoid shifting if you want to estimate interactions or alternative-specific effects

Shifting designs cause minimal overlap

Comparison of Orthogonal Designs

28


Webinar

Recipe-based design algorithms easy enough that you can create designs by hand or using simple Excel workbooks

While the columns of a fractional factorial are orthogonal with respect to one another, they may be perfectly correlated with un-modeled interactions

Focus on orthogonality can detract from efficiency

Prohibitions must be added manually and there is no good way to do this

Pros and Cons of Using Orthogonal Plans

29


Webinar

EFFICIENT DESIGNS

30


Webinar

We can calculate D-efficiency of a design mathematically

Rather than retrofitting orthogonal designs to create our CBC designs, we could search for groups of choice sets that constitute efficient designs

Computer search algorithms can assist this process

The covariance matrix itself depends on the model we plan to run, so different models require different efficiency calculations

Designs Based on Efficiency

31


Webinar

Kuhfeld, Tobias and Garratt (1994) show how to use search algorithms to make efficient designs for MNL models

Generate many sets of profiles with the right number of attributes and levels and let a search algorithm identify the most efficient way to combine them

SAS macros MKTEX and CHOICEFF work together to create a wide range of CBC experimental designs (Kuhfeld and Wurst 2012)

D-Efficiency for MNL Models

32


Webinar

You can further improve designs with iterative refinements (Huber and Zwerina 1996) called relabeling and swapping

If you have reasonably good estimates of respondents’ utilities, you can do better still (Sandor and Wedel 2001)

These are sometimes called Dp-efficient designs because we have a guess about their parameters

They are distinguished from D0-efficient designs that assume null parameters

Further Improvements for MNL Models

33


Webinar

Efficient designs for mixed logit models (Sandor and Wedel 2002)

Efficient designs for hierarchical Bayesian models (Sandor and Wedel 2005)

Efficient designs for two-stage models with a non-compensatory first stage (Liu and Arora 2011)

Efficient designs for repeated measures logit models (Bliemerand Rose 2010)

Other Model Specifications

34


Webinar

Relatively easy to implement the basic D0-efficient MNL model (using off-the-shelf software)

Complexity grows rapidly for different models and requires

Extreme programming skills and/or

Specialized software and/or

Prior estimates of utilities

Pros and Cons of Efficiency-Based Designs

35


Webinar

Efficiency based designs are more flexible than orthogonal designs (e.g. prohibitions are easier to accommodate)

Efficiency-based designs by their nature will be a little more efficient than orthogonal designs - a lot more efficient according to Bliemer and Rose (2010)

The differences may not matter much in practice

A lot of sniping and grousing happens anyway

Orthogonal versus Efficiency-Based Designs

36


Webinar

LIGHTHOUSE STUDIO DESIGNS

37


Webinar

These properties promote efficiency

(One-way) Level Balance: within each attribute, each level appears an equal number of times

Orthogonality, or two-way level balance: each level appears proportionally often with every level of other attributes

Minimal Overlap: minimizing the extent to which an attribute level repeats within a choice set

Sources of CBC Efficiency

38


Webinar

We can balance the sources of efficiency to produce four design strategies

Good designs at the respondent level that get better as more respondents answer more versions of the design

As the number of versions increases, the difference in D-efficiency between complete enumeration and efficient designs becomes minuscule (Kuhfeld and Wurst 2012)

Using many versions means

We can cover a larger portion of the design space

We can estimate interactions without pre-planning for them

Version effects can cancel out

Lighthouse Studio’s Design Strategies

39


Webinar

Generation process:

Maximize one- and two-way level balance

Profiles are nearly orthogonal for a given respondent

Features

One-way level balance (each level of an attribute occurs as equally often as possible)

Two-way level balance (each pair of levels occurs as equally often as possible)

Minimal overlap of levels within choice sets (promotes precision of main effects)

Complete Enumeration

40


Webinar

Generation process: Similar to Complete Enumeration, except:

Profiles are built using previously least-used attribute levels (from previous tasks and versions)

Two-way frequency balance is not explicitly controlled

Features

Minimal overlap of levels between profiles

One-way level balance (each level of an attribute occurs as equally often as possible)

Loose (not strict) orthogonality

Shortcut

41


Webinar

Generation process

Profiles sampled randomly, with replacement, from full factorial universe of profiles

No duplicate profiles allowed within a choice set

Features

Only loose one-way level balance (each level occurs approximately an equal number of times)

Only loose two-way level balance (each pair of levels occurs approximately an equal number of times)

Significant level overlap (an advantage for precision of interactions)

Random

42


Webinar

Similar to Complete Enumeration but with more level overlap

Features

About half as much level overlap as the purely Random strategy

One-way level balance (each level occurs an equal number of times)

Two-way level balance (each pair of levels occurs an equal number of times)

Much better for main effects than Random and much better for interaction effects than Complete Enumeration

Balanced Overlap is the default method in Sawtooth Software’s CBC software

Balanced Overlap

43


Webinar

CompleteEnumeration

Shortcut Random Balanced Overlap

Fast + +

Main Effect Efficiency + + -

Interaction Efficiency - - + +

Deep Processing + +

Extreme Prohibitions + +

Pros and Cons of Lighthouse Strategies

44


Webinar

QUESTIONS?

45

Megan PeitzIngenuity Ambassador

[email protected]

Keith ChrzanSVP, Sawtooth Analytics

[email protected]

www.sawtoothsoftware.com+1 801 477 4700@sawtoothsoft

mailto:[email protected]

mailto:[email protected]

http://www.sawtoothsoftware.com/

https://twitter.com/SawtoothSoft

https://twitter.com/SawtoothSoft


Webinar

Addelman, S. (1962) “Orthogonal Main-effects Plans for Asymmetrical Factorial Experiments, Technometrics, 4: 21-46.

Bliemer, M.C.J. and J.M. Rose (2010) “Construction of Experimental Designs for Mixed Logit Models Allowing for Corelation Across Choice Observations,” Transportation Research Part B, 44, 720-734.

Bliemer, M,C.J., J.M. Rose and S. Hess (2008) “Approximation of Bayesian Efficiency in Experimental Choice Designs,” Journal of Choice Modeling, 1:98-127.

Bunch, D.S., J.J. Louviere and D. Anderson (1996) “A Comparison of Experimental Design Strategies for Multinomial Logit Models: The Case of Generic Attributes,” accessed on January 26, 2013 at http://faculty.gsm.ucdavis.edu/~bunch/bla_wp_1996.pdf.

Chrzan, K. and B. Orme (2000) “Overview and Comparison of Design Strategies for Choice-Based Conjoint Analysis,” Sawtooth Software Conference Proceedings, 161-177)

Hahn GJ. Shapiro SS. (1966) A catalogue and computer program for the design and analysis of orthogonal symmetric and asymmetric fractional factorial experiments. Report 66-C-165, General Electric Research and Development Centre, Schenectady, New York.

References

46


Webinar

Huber, J. and K. Zwerina (1996) “The Importance of Utility Balance in Efficient Choice Designs,” Journal of Marketing Research, 33: 307-317.

Kanninen, B. (2002) “Optimal Designs for Multinomial Choice Experiments,” Journal of Marketing Research, 39: 214-227.

Kuhfeld, W.F. Orthogonal Arrays. Technical Report, SAS Institute. Accessed on January 26, 2013 at http://support.sas.com/techsup/technote/ts723.html.

Kuhfeld, W.F, R. Tobias and M. Garratt (1994) “Efficient Experimental Design with Marketing Research Applications,” Journal of Marketing Research, 31, 545-557.

Kuhfeld, W.F. and J.C. Wurst (2012) “An overview of the Design of Stated Choice Experiments,” Sawtooth Software Conference Proceedings. Orem: Sawtooth Software

Liu, Q. and N. Arora (2011) “Efficient Choice Designs for a Consider-Then-Choose Model,” Marketing Science, 30: 321-338.

Louviere, J.J., D.A. Hensher and J.D. Swait (2000) Stated Choice Methods: Analysis and Application. Cambridge: Cambridge University

References

47


Webinar

Louviere, J.J. and G.W. Woodworth (1983) “Design and Analysis of Simulated Consumer Choice or Allocation Experiments: An Approach Based on Aggregate Data,” Journal of Marketing Research, 20: 350-367.

Sandor, Z. and M. Wedel (2001) “Designing Choice Experiments Using Managers’ Prior Beliefs,” Journal of Marketing Research, 38: 430-444.

Sandor, Z. and M. Wedel (2002) “Profile Construction in Experimental Choice Designs for Mixed Logit Models,” Marketing Science, 21: 455-475.

Sandor, Z. and M. Wedel (2005) “Heterogeneous Choice Designs,” Journal of Marketing Research, 42: 210-218.

Street, D.J. and L. Burgess (2007) The Construction of Optimal Stated Choice Experiments. Hoboken: Wiley.

Street , D.J., L. Burgess and J.J. Louviere (2005) “Quick and Easy Choice Sets: Constructing Optimal and Nearly Optimal Stated Choice Experiments,” International Journal of Research in Marketing, 22: 459-470.

References

48

Documents

Experimental Designs for CBC - Sawtooth SoftwareFractional factorial plans require fewer observations They assume no or very few interactions When they measure main effects only, we