Measuring Whats Important to Your Customers Download

AMA Marketing Effectiveness

Online Seminar Series

Lynette Rowlands

American Marketing Association

Proprietary and Confidential, Maritz Inc. ©

A wealth of information is available for marketing professionals at

www.MarketingPower.com

The #1 marketing site on the web


Commonly Asked Questions

1. Will I be able to get copies of the slides after the event?

2. Is this web seminar being taped so I or others can view it after the fact?


Commonly Asked Questions

Yes

1. Will I be able to get copies of the slides after the event?

2. Is this web seminar being taped so I or others can view it after the fact?

Yes


Introducing Today’s Speaker

Using Importance Measurement to Drive Product Improvements

Keith Chrzan

Vice President, Marketing

Sciences Maritz Research


Agenda

Introduction Stated Importance Methods Derived Importance Methods Summary

Introduction


Why Measure Importance?

Products or services can be thought of as bundles of attributes (properties, features, benefits, etc.)

Marketers usually cannot afford to optimize all attributes at once, so they must prioritize

In order to prioritize, marketers must know the relative importance of the attributes, hence the need for importance measurement

May be part of customer satisfaction, image/positioning, brand choice, loyalty, concept testing or other studies

Introduction


Myers and Alpert’s Clarification

Salience – attribute is easily brought to mind Importance – attribute is important Determinance – attribute is important and

brands differ on it, so that it “determines” preference or choice

Introduction


Stated Importance

Direct questioning methods e.g. Open-ends, rating, ranking, sorts, etc.

Respondents’ answers directly tell us what attributes are more important than others

Introduction


Derived Importance

Indirect questions Respondent describes her actual experience Respondent evaluates that experience We infer importance by relating descriptions to the

evaluations We apply statistical predictive models to respondent-

supplied attribute and evaluative judgments and use statistical outputs as measures of importance

Introduction


Measuring Attribute Importance

Introduction Stated Importance Methods

Unconstrained methods Constrained methods

Derived Importance Methods Summary

Stated importance


Objectives

As a result of this section, you will be able to Describe two broad classes of stated importance

measures List seven specific kinds of stated importance

measurement Identify the strengths and weaknesses of the various

kinds of stated importance measurement Determine the best stated importance measure for

your project using a decision tree

Stated importance




Unconstrained methods Open-end questions Importance ratings

Constrained methods


Stated importance


Open-End Questions

Example: What attributes are important when choosing a <widget>?

Advantages Simple to ask in any survey modality (mail, phone, Web,

etc.) Question does not force response categories on

respondent – researcher may learn about new attributes

Disadvantages Importance and recall are confounded Open-ends may measure salience more than importance

Recommendation: Use for exploratory research, not for importance measurement

Stated importance


Importance Ratings

Example: When choosing a <widget> how important is—

Not at all ExtremelyImportant Important

Fast service 1 2 3 4 5

Wide variety 1 2 3 4 5Reliability 1 2 3 4 5Ease of use 1 2 3 4 5Country of origin 1 2 3 4 5Package color 1 2 3 4 5

Stated importance


Importance Ratings

Advantages Easy to ask in any survey modality Respondents are familiar and comfortable answering

importance ratings Clients are familiar and comfortable receiving

importance ratings

Stated importance


Importance Ratings

Disadvantages Respondents tend to rate most attributes very

positively – this reduces ability to discriminate among them and hampers subsequent analyses

Scale use heterogeneity Some respondents use high part of the scale others use

low part – positional heterogeneity Some respondents use a wider range of the scale than do

others – heteroskedasticity These response effects hampers interpretation and

multivariate analyses and harms cross-cultural comparisons

Recommendation: Use as a last resort

Stated importance


Importance Ratings

Empirical evidence against importance ratings Discriminant analysis with brand used as dependent

variable and importance ratings as predictors seldom makes sense, and it should, if importances are meaningful

Derived importance works like this: You model some dependent variable (Y) as a linear function of some performance measures (X) and you calculate coefficients that are proxies for importance

If you do the reverse, modeling Y as a function of importance ratings, the coefficients should be measures of performance, but they usually are uncorrelated with actual measures of performance

Importance ratings are next to worthless




Unconstrained methods Constrained methods

Rank order Q-sort Constant sum Method of paired comparisons Maximum difference scaling


Stated importance


Rank Ordering

Example: Please rank these 11 features of <widgets>. Give the most important feature a “1,” the next most important feature a “2,” and so on until the least important feature has a rank of “11.”

Advantages Works well on mail, Web or in-person surveys if

attribute list is short Response distribution is constrained so rank

information is standardized for use in cross-cultural comparisons

All respondents’ scores have the same mean All respondents’ scores have the same variance

Stated importance


Rank Ordering

Disadvantages Difficult to administer in phone surveys Difficult to administer if attribute list is long Non-parametric statistical tests for rank orders are

relatively weak, so less discriminating than even importance ratings

Recommendation: Do not use

Stated importance


Q-Sort

Example: If 17 attributes are to be evaluated, respondent is instructed to sort them so that– 1 is in a pile for the most important attribute 3 are in a pile for the next most important attributes 1 is in a pile for the least important attribute 3 are in a pile for the next least important attributes 9 are implicitly sorted into the pile of middle

importance

Resulting distribution is 1 : 3 : 9 : 3 : 1

Stated importance


Q-Sort

Advantages Forced distribution standardizes responses, making

this a viable technique for cross-cultural comparisons Discriminating Works in face-to-face, mail and Web surveys

Disadvantages Will not work in phone interviews Time consuming task if the number of attributes is

large

Recommendation: May use if there are not too many attributes and data collection is not phone

Stated importance


Constant Sum Allocation

Example: Please assign 11 points to these attributes according to how important they are to you when choosing a <widget>. You can assign some, none, or all 11 points to a given attribute, as long as the total number of points you assign is 11.Fast service _____Wide variety _____Reliability _____Ease of use _____Package color _____Country of origin _____ TOTAL 11

Stated importance


Constant Sum Allocation

Advantages Ratio measurement of importance Discriminating – trade-off prevents all attributes from

being important

Disadvantages Difficult to do in telephone interviews unless attribute

list is very short Difficult to administer at all if attribute list is long Unknown scale use heterogeneity Unknown ability to standardize cross-cultural studies

Recommendation: Perhaps use with short attribute lists

Stated importance


Method of Paired Comparisons

From Thurstone (1927) Updated design theory by David (1988) Updated analysis via hierarchical Bayesian

analysis Ask attributes two at a time, forcing respondent

to choose which is more important Ask 1.5 times as many pairs as there are

attributes

Stated importance



Example: Is ease of use or reliability more important

when choosing a <widget>? Is reliability or package color more important

when choosing a <widget>? Is variety or fast service more important

when choosing a <widget>?

Stated importance



Advantages Easy to administer in any survey modality VERY discriminating Automatically standardized for cross-cultural

comparisons (no scale-use bias) HB analysis produces individual level importances Ideal for input to needs-based segmentation If well balanced (all attributes occur equally often with

each other) analysis is simple: importance is proportional to percentage of times an attribute is chosen when it is available

Stated importance



Disadvantages A paired comparison question takes about 50% longer

for respondent to answer in a phone survey, so that a 2 minute importance rating battery becomes a 3 minute paired comparison battery, all else being equal

Recommendation: Good method, use when possible

Stated importance


Maximum Difference Scaling

Multiple choice extension of MPC 3+ attributes per question; respondent picks

most and least important Example:

Which of the following is least important when you buy a widget and which is most important?

Least Most[ ] Ease of use [ ][ ] Country of origin [ ]

[ ] Package color [ ][ ] Reliability [ ]

Stated importance



Advantages Handles more attributes with fewer questions than

MPC HB analysis produces individual level importances Even MORE discriminating than MPC Automatically standardized for cross-cultural

comparisons Ideal for input to needs-based segmentation If well balanced, analysis is simple: importance is the

log of the number of times chosen divided by the number of times available

Stated importance



Disadvantages Requires visual presentation of stimuli (i.e. paper and

pencil or Web survey) May require analysis to produce importances

Recommendation: Good method, use when possible

Stated importance


Stated Importance Decision Tree

Stated importance




Derived importance


Objectives

Following this section, you will be able to Identify four types of derived importance models, in

two broad classes Use a decision tree to settle on a derived importance

model for your project Identify the strengths and weaknesses of the various

kinds of derived importance measurement

Derived importance


Overview - Derived Importance Measures

Respondent evaluates a brand/product using A rating (satisfaction, performance, liking, purchase

intent) A share (market share, share of preference, an

allocation) A choice of one brand/product/therapy over others

Respondent rates product on multiple attributes Predictive statistical model relates attributes to

the evaluation Model yields coefficients as proxies for

importance Correlation and choice-based methods

Derived importance



Introduction Stated Importance Methods Derived Importance Methods

Correlation-based Methods Correlation Regression True Driver Analysis

Choice-based Methods

Summary

Derived importance


Variance

Correlation-based methods (correlation, regression, True Driver Analysis) depend on variance patterns Importance just means “shared variance with the

dependent variable” If an attribute doesn’t share variance with a dependent

variable, it can’t be important

Attributes which do not vary, or that vary only randomly, cannot share variance with the dependent variable, and so cannot be important e.g. ‘cost of entry’ attributes like “4 wheels on a car”

or “airline safety”


Correlation

Description: Bivariate correlations of evaluation with each attribute in turn

Advantages Easy to do Coefficients are unaffected by multicollinearity

Disadvantages Lack of statistical control because attributes are

analyzed in isolation Importance can be double (or triple or more) counted to

the extent attributes are correlated with one another No composition rule for coefficients, so model cannot

support simulation Scale use heterogeneity can spuriously inflate all

correlations

Derived importance


Correlation

What correlation does

Derived importance


Multiple Regression Description: All attributes are used simultaneously

to predict the evaluation; coefficients or standardized coefficients are importances

Advantages Accessible technique learned in school that clients are

familiar with Easy to do

Disadvantage - multicollinearity (intercorrelation of independent variables) is omnipresent in survey research and has a pernicious effect on regression coefficients

It can distort the size and even the sign of coefficients Makes coefficients unstable

Derived importance


Multiple Regression

What regression does

Derived importance


Multiple Regression Why multicollinearity is bad news for regression

Derived importance


A Middle Ground

Kruskal came up with a way of computing importance by doing regression analysis and “averaging over orderings”

Theil added an information-theoretic framework for this averaging over orderings

This idea works so well to solve the shortcomings of both correlation and regression that we’ve built it into a family of techniques for importance measurement that we call True Driver Analysis

Derived importance


True Driver Analysis (TDA)

Advantages Immune to multicollinearity Importances are . . .

Additive Ratio scaled Intrinsically meaningful

Disadvantages Complex programming

Derived importance



Introduction Stated Importance Methods Derived Importance Methods

Correlation-based Methods Choice-based Methods

MNL

Summary

Derived importance


Logit vs Regression

In regression, the unit of analysis is a brand and its ratings:

Obs. Overall Att 1 Att 2 Att 3 Att 4

1 4 5 3 2 5


Logit vs Regression

In MNL, the unit of analysis is a choice, so we collect multiple brands’ ratings:

Obs. Chosen Att 1 Att 2 Att 3 Att 4

1 0 5 3 2 5 1 0 3 2 2 3 1 1 4 5 1 1 1 0 5 4 4 2


Multinomial Logit (MNL)

Description Evaluation is brand preferred over others (choice or

share) Models this relative preference as a function of

attributes of all competing brands Between-brand differences drive the model (as they

drive choice in the real world) MNL coefficients are significant when differences

between brand ratings relate to brand choice – i.e. it measures determinance

Derived importance



Advantages Model built from between-brand differences in

attribute ratings, so scale use heterogeneity is not a problem

For the same reason, less likely to be affected by multicollinearity

Explicitly includes competitive context Specifically answers the oft-asked question “Why do

customers choose one product over another” (or “competitors’ products over mine,” or “mine over theirs”)

Derived importance



Disadvantages Because we need to ask every respondent about

several brands’ attributes, questionnaire can get long if there’s a lot more in it

Derived importance


Derived Importance Decision Tree




Derived importance


Stated Vs. Derived Importance

Fairly often, stated and derived measures give conflicting views of what is “important”


Back to Myers and Alpert

One reason stated and derived methods may not agree is that they are not even measuring the same thing Open-end measures confound the measurement of

salience and importance Other stated methods may measure importance,

though there is good reason to believe attribute importance ratings do not do so very well

MNL measures determinance, NOT just importance Regression-based derived importance methods

measure a cross-sectional version of determinance



Among stated importance methods— Simpler approaches, like importance ratings and

rankings, have serious shortcomings More complex methods (MPC, MaxDiff) overcome at

least some of the shortcomings MPC for phone surveys MaxDiff for mail, in-person or Web-based surveys



Among derived importance methods— Simple approaches (correlations, regression) just

aren’t good enough More complex methods (TDA, MNL) fix a lot of the

shortcomings TDA for customer satisfaction and concept testing MNL for brand choice/brand preference


Q & A


Thanks for your time and participation today!

To replay this webcast (available September 10): go to www.MarketingPower.com/ResearchSeries

For copies of today’s presentation:

www.maritzresearch.com/measurement or (877) 4 MARITZ

To contact today’s speaker:

Keith [email protected]

Questions for AMA: Lynette Rowlands

[email protected]

Documents

Measuring Whats Important to Your Customers Download