Embed Size (px)
Citation preview
The 4 most common mistakes in market research
September 2010
Mistake #1:Making generalizations about a population from a small sample
• Small sample sizes (<50) have a high degree of variability– More likely that inappropriate conclusions will be drawn about the population
• Large sample sizes (>200) are less variable– More likely that appropriate conclusions will be drawn about the population
Source: Hague P. A Practical Guide to Market Research. B2B International. 2007.
Small samples are best used for …
1. Generating a wide range of ideas– A small sample is capable of generating a range of ideas– With larger sample sizes, ideas will simply begin to be duplicated
2. Drawing “All” or “None” conclusions– If each member of a small sample says the same thing, and there was no
sampling bias, the conclusion is likely to be valid
3. Drawing “Some” conclusions– If testing an “All” or “None” conclusion, a small sample may be enough to
generate a single negative counterexample
4. Providing a gut check– Small sample sizes can mirror previously established empirical generalizations,
in which case some conclusions about the small group may be justified
Source: Bock T, Sergeant J. Small sample market research. International Journal of Market Research. 2002;44(2):235-244.
Mistake #2:Recruiting participants without regard for the desired conclusions
Do you plan to draw conclusions
about…
The larger population?
Yes No
Segments of the
population?
Yes
Recruit a very large random sample of the larger population where each
segment has at least 50 participants
Recruit random samples from each segment of more than 50
participants
NoRecruit a random sample of the
larger population of more than 50 participants
Recruit a targeted sample, focusing on characteristics of interest
(exploratory research)
• Always consider potential conclusions before recruiting participants– Failure to do so can result in samples of insufficient sizes to make appropriate
conclusions (see Mistake #1)
Source: Hague P. A Practical Guide to Market Research. B2B International. 2007.
Mistake #3:Selecting a method that poorly addresses the research objective
• Qualitative methods are best suited for exploration and development– Especially hypothesis development and in-depth probing
• Quantitative methods are best suited for evaluative research– Especially ratings, rankings, and other metrics
Objective Style Appropriate MethodTo identify problems or
opportunitiesExploratory
Brainstorming, qualitative research (IDIs or focus groups)
To uncover the rationale behind perspectives and
opinionsDiagnosis
Additional qualitative research (IDIs or focus groups)
To evaluate the validity of perspectives and opinions
Testing/DescriptiveQuantitative research (surveys or large, well-controlled IDIs or focus
groups)
Source: Hague P. A Practical Guide to Market Research. B2B International. 2007.
Mistake #4:Performing analyses without regarding the type of data
• Common errors in analysis– Performing averages on rankings or ordinal scales
• The distance between units in an ordinal scale is not consistent– For example, the difference between responding “agree” and “strongly agree” may not be the
same as the difference between “neither agree nor disagree” and “agree”
– Making comparisons on rankings, interval, or ordinal scales• Ex: 80°F is not twice as hot as 40°F
– 0°F does not represent zero heat
Data Type Description Example
NominalValues that are simply named and have an
arbitrary orderGeographic regions
OrdinalValues of a scale that can be placed in
orderRankings, some rating
scales
IntervalValues of a scale that can be placed in
order and whose units are equally spaced
Fahrenheit and Celsius, some rating
scales
RatioValues of a scale that have an order, units
of equal spacing, and a true zero valueHeight
Source: Cohen BH. Explaining Psychological Statistics. 2nd ed. New York, NY: John Wiley & Sons, Inc.; 2001.
How to avoid mistakes
• Increase efficiency– Prevent minor errors that can “snowball” into major problems– Draw stronger conclusions by comparing new findings to past research
• Increase quality– Research design and methodology have a major impact on the quality of research
• Identify the best research method
• Consider the sample size
• Do not underestimate the importance of the language in discussion guides and surveys
• Choose metrics that allow the audience to draw accurate conclusions from the data
• Engage and challenge researchers during each step of the process– Do not take methodology and research design for granted– Ask researchers to explain their methods, designs, and analyses
Source: Hague P. A Practical Guide to Market Research. B2B International. 2007.
If you’d like to find out more or have some research thoughts of your own to share, contact –
Carl Turner
VP, Research & Analytics - [email protected]
or
Bob Mason
SVP, Brand Strategy - [email protected]
