Embed Size (px)
Citation preview
A short course in Market Research with Ray Poynter
(English language)
Lesson 2 Tuesday, 8 July • Ch. 3, Quantitative Research • Ch. 5, Writing Questionnaires
@RayPoynter [email protected]
Dates and Modules
Thu 3 July
Introduction The context for market research Communicating results
Tue 8 July Quantitative research Writing questionnaires
Thu 10 July Qualitative research Analysing qualitative data
Tue 15 July Major applications of research Mobile market research
Thu 17 July
Emerging research methods Communities Social media research
Tue 22 July
Fri 25 July
How to analyse quantitative data Quantitative analysis techniques Pricing research
Thu 24 July
B2B (business to business) International research Political polling
Tue 29 July
Research ethics, Guidelines and laws Current areas of sensitivity Questions from new researchers
QUANTITATIVE RESEARCH Part A
What is Quantitative Research?
• Measuring things to: – Describe
– Monitor
– Explain
– Predict
• Seeks to generalise the measurements to a wider context
• Assumes things can be measured with numbers – How many children do you have?
– How much do you love your children?
Sources of Quantitative Data
• Surveys
• Store audit data
• Diary data
• Web analytics
• Transactional data (e.g. phone usage, purchases, downloads)
• Meters (e.g. TV viewing meters)
• Social media data
Quantitative Modes
Global Japan
Online 35% 52%
Telephone 17% 2%
Face-to-face 16% 13%
Postal 1% 12%
Other (mostly non-survey)
31% 21%
Source: ESOMAR Global Market Research Report, 2013
Population and Sample
Population
– Everybody we are interested in
• All adults in Japan
• Everybody with a Nissan car
• Every male over 40 who buys whiskey every week
Sample
– Some of the people from a population
– Used to estimate what the answer would have been if we had interviewed everybody
If we interview everybody then it is a census
Reasons why the Sample Might NOT Match the Population
1. The list of the population may be incomplete
2. The method of selecting people might not be ‘statistically fair’
3. Some people may decline to take part
4. Random chance
Significance Testing
• Significance test looks at the odds that something is just chance, i.e. sampling error / noise
• Significance testing helps us understand if differences are big enough to be interested in
• If a result is ‘significant’ – It MIGHT be a genuine finding
• If a result is NOT significant – Perhaps there is no difference
– Or, the difference is small
– Or, the sample was too small to be sure about the difference
Imagine Coin Game – You want a head
TAIL 50% chance it was bad luck, 50% chance coin is not fair
TAIL 25% chance it was bad luck, 75% chance coin is not fair
TAIL 12.5% chance it was bad luck, 87.5% chance coin is not fair
TAIL 6.3% chance it was bad luck, 93.7% chance coin is not fair Significant at 90% level to market researchers
TAIL 3.1% chance it was bad luck, 96.9% chance coin is not fair Significant at 95% level to market researchers
Coin Game
1. Everybody toss a coin 5 times and write down how many tails
2. Then, do it again, 2 numbers per person – To pretend we have twice as many people
3. How many people have 5 tails
– If you had 5 tails both times, put both hands up
Types of Sample
Random probability sample – Everybody in the population has a known
probability of being selected
Quota sample – controls the outcome – E.g. 50% male, 50% female, 25% North, 75% South
Convenience – People who can easily be sampled
Most online access panels are a mixture of 2 and 3
How Big Should the Sample Be?
• Size is not everything – Quality can be important
• There is a theory – read it in the books
• N = (z2σ2) / e2
• Three factors – Custom and practice
– The size of the smallest cell (usually 50 or 100)
– The budget
Sample size calculator at http://www.maritzresearch.co.uk/insights/maritz-stats.aspx
1936 US Presidential Election
• American magazine, Literary Digest, wanted to predict the Presidential election
• Mailed 10 million questionnaires – 2 million replies – in 1936
• Predicted Landon would beat Roosevelt – But Roosevelt won
• Why? – Sample phone owners, car owners, buyers of the
magazine – During an economic recession
Predicting the Population
Two key problems 1. Were the respondents representative?
2. Were respondents willing and able to tell the truth? E.g. “How many kilos of rice will you buy next month if the price goes up by 10%?”
Main link between survey data and the real world is modelling:
• Benchmarks • Regression • Econometrics
Weighting the Data Weighting Example Target Actual Weight Report
Men 450 500 0.9 450
Women 450 400 1.125 450
Total 900 900 900
Weighting Example Target Actual Weight Report
Men 450 500 0.9 450
Women 450 400 1.125 450
Total 900 900 900
Weighting Example Target Actual Weight Report
Men 450 500 0.9 450
Women 450 400 1.125 450
Total 900 900 900
Weighting the Data
Weighting Example Target Actual Weight Report
Men 450 500 0.9 450
Women 450 400 1.125 450
Total 900 900 900
2 Main Reasons • To make the data closer to expectations
• To remove sample effects • Too many men might result in different answers
The Quiz
• The quizzes are mostly to check the communication.
• If the communication has been clear
– Most students should score 100% or close to 100%.
• If people do not score 100%
– We will try to explain the material in a different way.
• Take Quiz 02, Part A (and then a short break)
Part B
Any questions before we re-start?
WRITING QUESTIONNAIRES Part B
Two Tips 1. Choose one textbook and stick to it
2. Watch Pete Cape’s video on NewMR
Ian Brace Questionnaire Design Published by Kogan Page http://www.koganpage.com/editions/questionnaire-design/9780749467791
http://newmr.org/play-again/a194cb09/
Two Tips 1. Choose one textbook and stick to it
2. Watch Pete Cape’s video on NewMR
Ian Brace Questionnaire Design Published by Kogan Page http://www.koganpage.com/editions/questionnaire-design/9780749467791
http://newmr.org/play-again/a194cb09/
Linking Questions to the Objectives
Objective 1
Objective 2
Objective 3
Question 1
Question 2
Question 3
Question 4
Question 5
Question 6
Main Types of Survey Questions
• Open / Closed
• Single / Multi
• Nominal, Ordinal, or Cardinal
• Special questions
– Timed questions, eye tracking, geolocation etc
Open/Closed Questions
Open How long does it take to get home, in minutes? (Please type your answer in the box)
Closed How long does it take to get home, in minutes? (Please select one answer) Less than 10 minutes 10 to 25 minutes 26 to 40 minutes 41 minutes to 60 minutes More than one hour
Single / Multi
Single Which is your favourite drink? (Please select one answer) ビール ワイン ブランデ一 None of these
Multi Which of these do you ever drink? (Please select all that apply) ビール ワイン ブランデ一 None of these
Nominal, Ordinal, Cardinal Nominal Which do you regularly eat? (Please select all that apply) ご飯 パスタ うどん None of these
Ordinal Please rank these. 1=most favourite 2=next favourite, 3=least favourite. ご飯 パスタ うどん
Cardinal Rate how much you like these. 10 = very much, 0 = do not like at all
ご飯 0 1 2 3 4 5 6 7 8 9 10
パスタ 0 1 2 3 4 5 6 7 8 9 10
うどん 0 1 2 3 4 5 6 7 8 9 10
How Long Should the Survey Be?
As short as possible – But no shorter
Longer surveys lead to: – More people dropping out
– More people not wanting to do surveys
– Satisficing
Satisficing – Speeding
– Straight lining
– Skipping questions
When to Use Don’t Know, NA, None of These … ?
When respondents might want to pick them
Which is your favourite drink? (Please select one answer) ビール ワイン ブランデ一 None of these
What is your favourite drink? (Please select one answer) ビール ワイン ブランデ一 Other
How long does it take to travel from your home by bus to the nearest major train station? Less than 30 minutes 30 to 60 minutes More than one hour Don’t know
Key Rules for Survey Questions
1. Respondents should understand what they are supposed to do.
2. Questions should NOT be ambiguous
o Was the train on time and clean today?
o Questions should minimise bias.
Do you sometimes drink too much wine or beer? Yes No
How often do you drink more than 4 glasses of wine or beer? More than 10 times a month 6 to 10 times a month 1 to 5 times a month Less often Never
Making Surveys More Engaging
1. Choose the right style for your audience
2. Narrative flow
3. Don’t annoy the respondents
– Grids
– Long lists
– Long introductions
– Asking the same question 2 or more times
Gamification? • Lots of interest
• But! Not proven, still experimental http://www.researchthroughgaming.com
How to Test Questionnaires
1. Compare with Research Objectives
2. Spelling and grammar
3. Software checks
– Routing
– Auto-responses / dummy respondents
4. Comparing a printed copy with the screen
5. Soft launch
– 20 to 50 completes
– Check responses, drop-outs, missing data etc
Questions
And Then Quiz B
