Version 1 | Public (DELETE CLASSIFICATION) Version 1 | Internal Use Only Version 1 | Confidential Version 1 | Strictly Confidential© Ipsos MORI ESRC Research

Version 1 | Public (DELETE CLASSIFICATION) Version 1 | Internal Use Only Version 1 | Confidential Version 1 | Strictly Confidential© Ipsos MORI

ESRC Research Methods Festival

8 July 2014, St Catherine’s College, Oxford

Understanding the causes of measurement differences by mode

Gerry Nicolaas


Contents

• Background• What causes measurement to be different by mode?• Question design strategies


Background


Different ways of mixing modes:

• Mixtures of means of communication• Data collection mixtures, e.g.

• Different data items from the same person

• Same data items from different people

• Same data items from the same person at different time points

Source: Edith de Leeuw (2005)


Focus of this presentation:

• Collecting the same data items using different modes• Risk of differential measurement error because people may

answer questions differently depending on the mode• Potential for reduced data comparability


What causes measurement to be different by mode?


Causes of mode effects (Jäckle et al, 2011)

Comprehension Retrieval Judgment Response



Extent of Interviewer involvement

Reporting situationPrivacy, legitimacy, interaction

Willingness to disclose?

► Social desirability bias









Sufficient effort?

►Satisficing

Respondent ability

Respondent motivation

Task difficulty


Visual - Aural stimulusQuestion

format








Sufficient effort?

►Satisficing

Respondent ability

Respondent motivation

Task difficulty

Context information


Visual - Aural stimulusQuestion

formatRespondent control over questionnaire

How is question processed?

►Response effects


To sum up the causes of mode effects:

• Extent of interviewer involvement• Visual versus aural stimulus• Question format• Respondent control over questionnaire


Mode is more complex than simple distinction between:

• Face-to-face interview• Telephone interview• Mail questionnaire• Online questionnaire


For example, face-to-face interview

No showcard With showcard With self-administered module

Audio-CASI

Question Aural Aural Visual Visual & Aural recording

Response options

Aural Visual Visual Visual & Aural recording

Answer Oral Oral Written Typed


Another example, a telephone interview

Traditional Response lists mailed in advance

Touchtone Data Entry

Interactive Voice Recognition

Question Aural (interviewer)

Aural (interviewer)

Aural (interviewer)

Aural (recording)

Response options

Aural (interviewer)

Visual Aural (interviewer)

Aural (recording)

Answer Oral Oral Key Oral


A final example, an online questionnaire

Simple With audio With audio & recording

With remote or virtual interviewer

Question Visual Visual & Aural Visual & Aural Visual & Aural

Response options

Visual Visual & Aural Visual & Aural Visual & Aural

Answer Typed Typed Oral Oral


Mode is a characteristic of a question, not a survey.


Question Design Strategies


Question design strategies:

• Uni-Mode Construction• Writing and presenting questions the same to ensure respondents

receive a common mental stimulus

Source: Dillman et al (2009); de Leeuw (2005)





• Mode-Specific Construction• Modifying the question structure, wording or presentation for

different modes based on the particular capabilities of each mode








• Mode-Enhancement Construction• Using features not available in all modes to improve quality of

responses in one particular mode – minimal evidence








• Mode-Enhancement Construction• Using features not available in all modes to improve quality of

responses in one particular mode – minimal evidence

• Generalised Mode Design• Purposively designing questions differently in different modes with

the aim of achieving cognitive equivalence – minimal evidence



Summing up:


To sum up:

• Causes of mode effects:• Extent of interviewer involvement

• Visual versus aural stimulus

• Question format

• Respondent control over questionnaire

• Mode is a characteristic of a question, not a survey


Thank [email protected] | 020 xxxx xxxx

11/11/13

Click here to insert cover image

© Ipsos MORI This work was carried out in accordance with the requirements of the international quality standard for market research, ISO 20252:2006 and with the Ipsos MORI Terms and Conditions which can be found here

http://www.ipsos-mori.com/terms


References

• Dillman, D., Smyth, J. & Christian, L.M. (2009). Internet, Mail and Mixed-Mode Surveys: The Tailored Design Method, 3rd edition. Hoboken, New Jersey: Wiley.

• Jäckle, A., Lynn, P., Campanelli, P., Nicolaas, G., & Hope, S. (2011). How and When Does the Mode of Data Collection Affect Survey Measurement? ESRA Conference, 21 July 2011.

• de Leeuw, E. (2005). To Mix or Not to Mix Data Collection Modes in Surveys. Journal of Official Statistics, 21(2), 233-255.

Documents

Version 1 | Public (DELETE CLASSIFICATION) Version 1 | Internal Use Only Version 1 | Confidential Version 1 | Strictly Confidential© Ipsos MORI ESRC Research