Causes of mode effects on survey measurement

One day symposium at the Royal Statistical Society, London11 October 2011

BackgroundBackground

• Increasing use of mixed modes• Falling response rates• Rising costs of data collection

• Risk of reduced data comparability• Coverage error• Non-response error• Measurement error

• Need for practical advice to inform decisions about when to mix modes and how

Mixing Modes and Measurement Error

• Funded under the ESRC Survey Design and Measurement Initiative

• 3-year contract starting 1 Oct 2007 • Collaboration between NatCen, ISER and

independent survey methods consultant

Core research team

• National Centre for Social Research (NatCen)

• Gerry Nicolaas• Steven Hope (now UCL)• Margaret Blake• Michelle Gray

• Institute for Social & Economic Research (ISER)

• Peter Lynn• Annette Jäckle

• The Survey Coach (Independent Survey Methods Consultant)

• Pamela Campanelli

Main objective

• Practical advice on how to improve portability of questions across modes• Which mode combinations are likely to produce

comparable responses?• Which types of questions are more susceptible to

mode effects?

Research design

• A literature review & framework of mixed modes• develop a conceptual framework • identify gaps in evidence base and formulate

hypotheses to address gaps• Quantitative data analysis

• test hypotheses using existing datasets and new experimental data

• Cognitive interviewing• explore how respondents process questions in different

modes

Presentations & Discussion

1. How and when does the mode of data collection affect survey measurement?

2. The use of cognitive interviewing methods to evaluate mode effects

3. The role of visual and aural stimuli in producing mode effects

4. The role of the interviewer in producing mode effects5. Is it a good idea to optimise question format for mode

of data collection?6. Designing questions for mixed mode surveys7. Discussant (Patten Smith)8. Open discussion