Upload
braith
View
42
Download
0
Tags:
Embed Size (px)
DESCRIPTION
Issues of Coverage, Sampling and Participation in Mixed Mode Surveys Peter Lynn, University of Essex 6 th ESRC Research Methods Festival Oxford, 08-07-2014. Mixing Modes. “mixing modes gives an opportunity to compensate for the weaknesses of each individual mode at affordable cost” - PowerPoint PPT Presentation
Citation preview
An initiative by the Economic and Social Research Council, with scientific leadership by the Institute for Social and Economic Research, University of Essex, and survey delivery by NatCen Social Research and TNS BMRB
Issues of Coverage, Sampling and Participation in Mixed Mode Surveys
Peter Lynn, University of Essex
6th ESRC Research Methods FestivalOxford, 08-07-2014
Mixing Modes
“mixing modes gives an opportunity to compensate for the weaknesses of each individual mode at affordable cost”
- de Leeuw (2005)
Involves an explicit trade-off between costs and (multiple sources of) survey errors
Recent interest in mixed modes is particularly stimulated by the (possibly false) notion that the marginal cost of data collection by web is close to zero
Lynn | ESRC RMF, 8 July 2014
A Distinction
• Multi-mode (or multiple mode) data collection: Different modes used for different survey items, e.g.- CASI component within a CAPI survey;- CATI follow-up to a mail questionnaire;- etc
• Mixed mode data collection: The same survey items can be collected by different modes for different sample members:- sequential;- concurrent selective;- concurrent with respondent choice; etc
Lynn | ESRC RMF, 8 July 2014
Motivation for Mixing Modes
Motivation for mixed modes rather than a single mode:• Cost reduction; or• Coverage/ participation enhancement.(Designs to achieve both simultaneously are proving elusive.)Different motivations tend to suggest rather different designs:• Combinations of modes;• Sequence of modes (or non-sequential).The coverage, sampling and participation issues may differ
between these types of design
Lynn | ESRC RMF, 8 July 2014
Sequential and Concurrent Mixed Mode DesignsSequential design:• Use a number of modes in sequence;• Get as much response as possible in one mode, before trying remaining
non-respondents in the next mode• Cheaper modes first if motivation is to reduce costs• Higher response rate modes first if motivation is to maximise participation
Concurrent selective design:• Offer a different mode to each of 2 or more subsets of sample members
Concurrent elective design:• Offer each sample member a choice of mode
Combination: e.g. Concurrent selective sequential design
Lynn | ESRC RMF, 8 July 2014
Sequential Design: Example
Phase 1:• Mail an invite to a web survey to all sample members;• Mail a reminder to those who have not responded after a week or two
Phase 2:• Mail a paper self-completion q’re to those who have still not responded
after a further week (or include this with the reminder above)
Phase 3:• Approach for a face-to-face interview those who have still not responded
after a further periodKey design choices: Which modes? Which order? Criterion for switching to next phase?
Lynn | ESRC RMF, 8 July 2014
Coverage Issues with Mixed Mode Surveys IMixed modes (of approach) can help address frame quality
problems. Examples:• Inconsistent contact details on sampling frame: only address for
some people, only email for others, etc• Dual-frame approaches: good coverage is provided only by the
union of multiple frames, which have different contact details
Note: Mixed modes of approach need not necessarily imply mixed mode data collection
Lynn | ESRC RMF, 8 July 2014
Inconsistent Contact Details: ExampleDutch GPSs (e.g. Labour Force Survey, ESS experiment)• Select addresses from Postal Address Register• Match phone numbers to addresses (via names)• 70% match successfully: Can approach by phone• 30% do not match: First approach face-to-face• Some of the addresses with phone numbers also require face-to-face
follow-up
Lynn | ESRC RMF, 8 July 2014
Dual-Frame: Example
• RDD or list-assisted sampling to generate sample of phone numbers:- These numbers are screened to identify households (with phones)
• Supplementary sample of addresses:- These addresses are screened (face-to-face) to identify households with no (landline) phone
• The two samples combined give good coverage
Lynn | ESRC RMF, 8 July 2014
Coverage Issues with Mixed Mode Surveys IICoverage issues can introduce constraints on mixed mode data
collection. Example:• Desire is a sequential web → face-to-face design• Not all sample members are web users• Non-web users must skip the web phase, either explicitly or
implicitly
Lynn | ESRC RMF, 8 July 2014
Coverage Issues with Web as a Primary ModeA. Restrict survey to web users:• Obvious cost advantages;• Non-random under-coverage: requires evaluation and adjustment.Or B. Include non-web users in web mode:• Requires provision of hardware, software and training;• Various models, e.g. LISS, KnowledgePanel, GIP, ELIPSS.Or C. Include non-web users in a different mode…
Lynn | ESRC RMF, 8 July 2014
Coverage Issues with Web as a Primary Mode, ctd.Option C, including non-web users in a different mode:• May have cost advantages compared to providing equipment –
depends on frequency/nature of data collection, etc;• May have measurement disadvantages (see next two presentations!)Various designs possible:• Web + mail, based on mail-only approach;• Interviewer-administered recruitment, followed by web+mail,
web+phone, or other mixes;• More options are feasible in longitudinal context.Examples: GESIS panel and Gallup panel are both web+mail
Lynn | ESRC RMF, 8 July 2014
Sampling Issues with Single Mode WebNo general population frames with email addresses, therefore
first approach must be by a different mode.For frames without names, mail approach requires self-
administered respondent selection:• This is error-prone if paper-based and may cause dropout if
web-based.Interviewer-administration preferred but costly.Alternatively, use non-probability recruitment methods (opt-in
panels) and model-based inferential paradigm.
Lynn | ESRC RMF, 8 July 2014
Sampling Issues with Mixed Mode including WebFrames with partial information can be used (e.g. email
addresses for a subset) – but none yet in UK.Instead, single-mode initial approach needed:• Mail approach, with web+mail data collection, may offer low-
cost solution of reasonable quality in some situations;• In UK, this design may work with named-person frames such
as admin records, but is hampered by the need for respondent selection for general population surveys;
• Interviewer approach and respondent selection may be preferable, but only cost-effective for longitudinal surveys.
Lynn | ESRC RMF, 8 July 2014
Respondent Selection
Next / last birthday method:• European Social Survey (UK) Experiment (Villar, 2013)• Selection compared to birth date info from questionnaire;• Approx. 50% correct, 20% incorrect , 30% uncertain• (amongst households with 2+ adults)
Household roster / grid method:• Community Life Survey (Williams 2013)• Approx. 25% incorrect selections
→ Difficult / impossible to control who completes a self-completion questionnaire;
→ Chance of incorrect respondent may be greater with web
Lynn | ESRC RMF, 8 July 2014
Participation in Mixed Mode SurveysResponse rates in single-mode surveys:
Face-to-face > telephone > self-completion – typically
Amongst self-completion surveys:Mail > web – often, but not always
Composition of response:Broadly similar between modes – typicallyBut some differences
Lynn | ESRC RMF, 8 July 2014
Participation in Mixed Mode Surveys
Population
FTF Response
Web Response
Lynn | ESRC RMF, 8 July 2014
Participation in Mixed Mode Surveys
Lynn | ESRC RMF, 8 July 2014
Population
Mixed Modes Response?
Increasing Response Rates
• Success requires:- All (or most) people who would have responded in mode 1 continue to respond;- Additional people respond too (in mode 2)
• But it is generally the case that:- (Average) response propensity declines FTF → Phone → Mail → Web- Data collection costs decline in same order- Refusal in one mode reduces propensity to respond in next mode
Lynn | ESRC RMF, 8 July 2014
Examples of Increasing Response Rates• Examples are (all?) from surveys that use a subset of the
sequence FTF – Phone – Mail – Web
• And even then, it is also necessary to “exhaust” each mode
• E.g. 1 British Household Panel Survey: FTF → phone
- Approx. 93% response FTF alone; 96% FTF + phone (amongst previous wave respondents)
Lynn | ESRC RMF, 8 July 2014
Examples of Increasing Response Rates• Examples are (all?) from surveys that use a subset of the
sequence FTF – Phone – Mail – Web
• And even then, it is also necessary to “exhaust” each mode
• E.g. 1 British Household Panel Survey: FTF → phone
- Approx. 93% response FTF alone; 96% FTF + phone (amongst previous wave respondents)
• E.g. 2 British Crime Survey 2011 Follow-Up: Mail → web
- 60% response mail alone; 63% mail + web (amongst BCS respondents)
Lynn | ESRC RMF, 8 July 2014
Examples of Failing to Increase Response Rates• All (?) surveys that use a sequence which is not a subset of
FTF – Phone – Mail – Web
• E.g. 1 UKHLS-IP 2009 wave 2: Phone → FTF- 76% response FTF alone; 67% Phone + FTF (w1 resps)
Lynn | ESRC RMF, 8 July 2014
Examples of Failing to Increase Response Rates• All (?) surveys that use a sequence which is not a subset of
FTF – Phone – Mail – Web
• E.g. 1 UKHLS-IP 2009 wave 2: Phone → FTF- 76% response FTF alone; 67% Phone + FTF (w1 resps)
• E.g. 2 UKHLS-IP 2012 wave 5: Web → FTF - 84% response FTF alone; 79% Web + FTF (w4 resps)
Lynn | ESRC RMF, 8 July 2014
Examples of Failing to Increase Response Rates• All (?) surveys that use a sequence which is not a subset of
FTF – Phone – Mail – Web
• E.g. 1 UKHLS-IP 2009 wave 2: Phone → FTF- 76% response FTF alone; 67% Phone + FTF (w1 resps)
• E.g. 2 UKHLS-IP 2012 wave 5: Web → FTF - 84% response FTF alone; 79% Web + FTF (w4 resps)
• E.g. 3 UKHLS-IP 2012 wave 2: Web → FTF - 85% response FTF alone; 81% Web + FTF (w1 resps)
Lynn | ESRC RMF, 8 July 2014
Examples of Failing to Increase Response Rates• All (?) surveys that use a sequence which is not a subset of
FTF – Phone – Mail – Web
• E.g. 1 UKHLS-IP 2009 wave 2: Phone → FTF- 76% response FTF alone; 67% Phone + FTF (w1 resps)
• E.g. 2 UKHLS-IP 2012 wave 5: Web → FTF - 84% response FTF alone; 79% Web + FTF (w4 resps)
• E.g. 3 UKHLS-IP 2012 wave 2: Web → FTF - 85% response FTF alone; 81% Web + FTF (w1 resps)
• E.g. 4 NL-ESS 2009: Web → Phone/FTF (new sample)- 52% response FTF alone; 46% Web + Phone/FTF
Lynn | ESRC RMF, 8 July 2014
Non-Response Bias
• Most (of the few) studies to date either found no effect or a modest (assumed) positive effect
• Hope is that web may disproportionately add young, full-time employed, busy people – who are generally under-represented in surveys
• But very little evidence either way on this point:- NL-ESS 2009 found almost identical sample composition in the FTF-only and Web + Phone/FTF samples (age, employment, education, etc)- UKHLS 2012 too found no significant differences between FTF-only and Web+FTF (age, gender, household type, etc)
Lynn | ESRC RMF, 8 July 2014
Mixed mode including Web: UK ExamplesTwo cross-sectional surveys:
Community Life Survey (Williams 2013):• Random sample of addresses from PAF• Advance letter → Mail invitation → Mail reminder → Mail questionnaire• 16% responded online + 11% mail = 27% response (no incentive)• 19% responded online + 12% mail = 31% response (£5 conditional)• 22% responded online + 13% mail = 35% response (£10 conditional)• 25% responded online + 14% mail = 39% response (£5 unconditional)
European Social Survey (UK) Experiment (Villar, 2013):• Random sample of addresses from PAF• Advance letter → Mail invitation → Mail reminder → Face-to-face fieldwork• 21% responded online + 18% face-to-face = 39% response
Lynn | ESRC RMF, 8 July 2014
Mixed mode including Web: Longitudinal ExamplesBritish Crime Survey re-contact study (Fong & Williams 2011):• Issued sample = 30% of BCS respondents who gave an email address;• Email invitation → Email reminder → Postal q’re → Postal reminder • 35% responded online + 27% by mail = 62% response
Lynn | ESRC RMF, 8 July 2014
Web+Interviews: Examples1958 Birth Cohort (Brown et al 2014):• 9th wave (age 55) – 5 years after previous wave• Sequential web → telephone• 62% responded online + 21% phone = 83% responseUnderstanding Society (Jäckle et al, 2013):• Issued sample = wave 5/wave 2 of a household panel• Mail(+email) invitation → (email reminders) → Mail reminder → Face-to-face• 21% responded online + 55% face-to-face = 76% response (w5 sample)• 30% responded online + 50% face-to-face = 80% response (w2 sample;
higher incentives)• 23% responded online + 55% face-to-face = 78% response (w5 sample, £10)• 23% responded online + 51% face-to-face = 74% response (w2 sample, £10)
Lynn | ESRC RMF, 8 July 2014
Conclusions
Potential cost savings from mixed mode may erode if we:- Aim for full population coverage- Aim for response at least as high as could be achieved with
single-mode
Participation advantages of mixed mode likely to be costly
Most promising cost-quality trade-off in the UK currently may be:- Web → Mail → Face-to-face
But … measurement concerns (next presentations!)
Lynn | ESRC RMF, 8 July 2014
An initiative by the Economic and Social Research Council, with scientific leadership by the Institute for Social and Economic Research, University of Essex, and survey delivery by NatCen Social Research and TNS BMRB
Issues of Coverage, Sampling and Participation in Mixed Mode Surveys
Peter Lynn, University of Essex
6th ESRC Research Methods FestivalOxford, 08-07-2014