Embed Size (px)
Citation preview
What is meant by mode effect on measurement?
A research study to identify causes of mode effects
Gerry Nicolaas
Background• Increasing use of mixed modes
Falling response ratesRising costs of data collection
• Risk of reduced data comparabilityCoverage errorNon-response errorMeasurement error
• Need for practical advice to inform decisions about when to mix modes and how
Mixed Modes and Measurement Error• Funded under the ESRC Survey Design and
Measurement Initiative• 3-year contract starting 1 Oct 2007 • Collaboration between NatCen, ISER and
independent survey methods consultant
Research Team• National Centre for Social Research (NatCen)
Gerry NicolaasSteven HopeDavid Hussey
• Institute for Social & Economic Research (ISER)Peter LynnAnnette JäckleAlita NandiNayantara Dutt
• Independent Survey Methods ConsultantPam Campanelli
Main Objective
• Practical advice on how to improve portability of questions across modesWhich mode combinations are likely to produce
comparable responses?Which types of questions are more susceptible
to mode effects?
Research Design• A literature review & framework of mixed modes
develop a conceptual framework identify gaps in evidence base and formulate
hypotheses to address gaps• Quantitative data analysis
test hypotheses using existing datasets and new experimental data
• Cognitive interviewingexplore how respondents process questions in
different modes
Mode is defined by:
• Interviewer presence (face-to-face, phone, none)• Oral and/or visual transmission of information
Question deliveryDelivery of response optionsRecording of response
Face-to-Face Interview
ACASIWritten (CAI)Visual (CAI)Aural (CAI)
ACASIWritten (CAI)Aural (CAI)Aural (CAI)
SAQ in-int’wWritten (paper)
Visual (paper)Visual (paper)
CASIWritten (CAI)Visual (CAI)Visual (CAI)
FTF (card)OralVisualAural
FTF (no card)OralAuralAural
DescriptionResponseResponse Options
Question
Telephone interview
IVROral (CAI)Aural (rec)Aural (rec)
Phone with showcards
OralVisualAural
TDEWritten (CAI)AuralAural
TelephoneOralAuralAural
DescriptionResponseResponse Options
Question
Self-completion
A-WebWritten (CAI)Aur & Vis (CAI)
Aur & Vis (CAI)
Web / emailWritten (CAI)Visual (CAI)Visual (CAI)
SAQ (e.g. mail)
Written (paper)
Visual (paper)Visual (paper)
DescriptionResponseResponse Options
Question
Causality
• How, when and why to mix/choose modes• Need to better understand causal mechanisms• Development and testing of behavioural theory
Comprehension Retrieval Judgement Response
Depth of cognitive processing: Sufficient Effort?
Social norms: Willingness to disclose?
Cognitive demands
Aural/visualInterviewer presence
Interviewer presence Privacy/legitimacyI-R interaction
Context information: Influence on processing?
Time pressureR distractionR motivation
Add. explanationsI characteristics
Sequential/Simultaneous
Qs
Control Qaire
Social desirability
bias
Satisficing
Hypotheses• Short versus long response lists
Effects of interviewer presence on satisficing • Agree-Disagree scales
Is acquiescence caused by satisficing, cognitive ability, social desirability?• Ranking versus Rating
Effects of interviewer presence • Fully-labelled versus End-labelled scales
Effects of visual stimulus and interviewer presence• Showcards versus No Showcards (face-to-face interview)
Effects of visual stimulus• “Branched” versus “Non-branched” questions
Effects of item design (no mode differences expected)• “Yes/No” versus “Code all that apply”
Effects of item design (no mode differences expected)
Question selection
Questions were designed to vary by:• Task difficulty• Sensitivity• Question type
SatisfactionOther attitudinalBehaviouralOther factual
Mixed Modes experiment
• Follow-up surveys to NatCen Omnibus (& BHPS)Face-to-face, telephone and web comparisonsExperimental design with random allocation
• LimitationRestricted to respondents with web access
Acquiescence (1)• Hypothesis
Acquiescence is a form of satisficingMore satisficing in web than f2f & tel
(no interviewer to motivate, explain, probe, etc)• 12 Agree/Disagree questions
5-point agree/disagree scaleUse of opposite statements
Acquiescence (2)• Initial results from the experiment
More acquiescence in f2f & tel compared to web• Results from the cognitive interviews
Only 2 out of 23 cases of agreeing to opposite statements due to acquiescence
Justifiable explanations given for other 21 cases
Acquiescence (3) • Example of justifiable agreement with opposite
statementsN36: Compared to other neighbourhoods, this
neighbourhood has more properties that are in a poor statement of repair.
N38: Compared to other neighbourhoods, this neighbourhood has more properties that are well kept.
Respondent: In this village, … it’s like half and half…”
Acquiescence (4)
• Interim conclusionsUse of opposite statements to detect acquiescence
bias brought into questionWhy higher rate of acquiescence in f2f & tel
compared to web?
Other forms of satisficing (1)• Hypotheses:
More satisficing in web than f2f & tel (no interviewer to motivate, explain, probe, etc)
More satisficing in tel than f2f (lack of physical presence of interviewer, lack of non-verbal communication, distractions, etc)
• Indicators of satisficingPrimacy effects in visual modes, recency effects
in aural modes, middle category effects, item non-response
Other forms of satisficing (2)• Initial results from the experiment
Primacy & recency effects: inconsistent patterns Item non-response: no mode differences Middle category effects: web respondents more
likely to select middle categories than f2f & tel resps (also for agree/disagree scales – see acquiescence results)
• Results from cognitive interviewsResults suggest more satisficing in web & tel than
in f2f interview
Mode effect or question design effect?
• Example:“Code all that apply” frequently used in f2f
interviews and self-completion surveysAlternative format for tel interviews tends to be a
series of Y/N questions • Hypothesis
No mode effecti.e. No differences if series of Y/N questions used
across all modes
Mode effect or question design effect?
• Initial results from the experiment F2F & tel resps more likely than web resps to say
“Yes” in series of Y/N questions Similar mode effect not found for f2f and web
respondents in “code all that apply” format (nb this format not used in tel mode)
• Results from cognitive interviews Questions raised about validity of “Yes” answers
in Y/N series that may differ by mode
• Continue with analysis of experimental data• Papers
Causes of mode effects on survey measurementThe role of the interviewer in producing mode effectsThe role of visual/aural stimuli in producing mode effectsThe role of question format in producing ‘mode’ effectsUsing cognitive interviews to explore mode effects
• Training courseQuestionnaire design for mixed mode surveys
What next?
