Question or Mode Effects in Mixed-Mode Surveys: A Cross ... · Probability based panel, established in 2007 Random assignment to mode CATI: assigned 2000, responded 1207 (60%) Web:

67th AAPOR Conference

Orlando, May 17-20, 2012

Question or Mode Effects in Mixed-Mode

Surveys: A Cross-cultural Study in the

Netherlands and the UK

Edith de Leeuw & Joop Hox

Department of Methodology & Statistics, Utrecht University

Pamela Campanelli & Gerry Nicolaas

The Survey Coach / NatCen Social Research, UK

Campanelli, De Leeuw, Hox, & Nicolaas

The

Survey

Coach


Orlando, May 17-20, 2012 NatCen Social Research

2

Mixing Modes

Mixed-Mode Surveys combine data, but

Are responses between modes comparable?

Modes differ on many mode-inherent factors,

e.g.

Privacy

Interviewer present or not

Computer vs paper

Interactivity

Interviewer help or not

Feedback

Information transmission

Audio vs visual vs both

3

Question Design Different modes have different traditions

Prime example:

Unfolding or two-step aka branching (telephone)

Complete question one-step (web, mail)

Numerous studies have shown that modes differ very

slightly when comparable questions are used, but

Question format influences answers given

Response options, question wording, etc

Early overview Sudman et al 1974

Effects found even within one mode

May data be combined?

Channel

Capacity

4

Mode Experiments in Two Countries

Mode Experiments in the Netherlands*

NL: CATI vs Web (CAWI)

Members Dutch LISS-panel (CenTERdata)

Probability based panel, established in 2007

Random assignment to mode

CATI: assigned 2000, responded 1207 (60%)

Web: assigned 6134, responded 4003 (65%)

* Courtesy of the LISS-panel. The LISS panel data were collected by CentERdata

(Tilburg University, The Netherlands) through its MESS project funded by the

Netherlands Organization for Scientific Research

5

Mode Experiments in Two Countries

Mode Experiments in Great Britain*

GB: CAPI vs CATI vs Web

Respondents to NatCen Omnibus survey

Agreed to be recontacted

Access to internet

Random assignment to mode

CAPI: responded 282 (73%)

CATI: responded 314 (69%)

Web: responded 349 (47%)

* Funded by the Economic and Social Research Council (award RES-175-25-0007).

6

Question Format NL

Eight questions on acceptability of usage of

advanced medical technology

If it will save a live, everything is permitted

It is not desirable to utilize every medical invention, just

because it is technologically possible

Five point scale: Totally Agree….Totally Disagree

Do-not-Know not offered, but accepted when given

Two question format experiments

Unfolding (two step) vs Direct Question (one step)

With vs without explicit ‘neutral’ category

7

Analyses NL

Variables of Interest

Independent variables (dichotomous)

Mode: CATI vs WEB

Format

Unfolding vs Complete question

Dependent variables

Mean score (Concept of interest: what do people think about

the topic)

Extremeness: Proportion extreme answers 1 & 5

Analyses done after appropriate recoding of responses

1: most conservative in acceptance modern technology

5: very prone to accept new medical technology

8

Differences in Means!

Two factors affect mean score (and thereby

conclusions on acceptance of medical

technology)

1. Mode

2. Unfolding

Note: no interactions

What may cause these shifts in mean?

Mean score sensitive for extremes!

More extreme scores?

Is there a primacy (Web) or recency (CATI) effect?

9

Summary of NL Results

Higher Overall Means:

CAWI

Complete Question (one step question)

Same effects for with and without neutral

More Extreme Answers

CATI

Unfolding (two step question)

Without neutral midpoint

More recency in CATI and with Unfolding

More primacy in Web+Unfolding (interaction)

10

Question Format GB

Two questions

Whether standard of living has increased, decreased,

or stayed the same since last election

Seven point scale: Increased by a large amount…. Decreased

by a large amount

Neutral category (i.e. ‘stayed the same’)

Whether respondent considered local shopping

facilities to be poor or good

Six point scale: Extremely poor….Extremely good

No neutral category

Two question format experiments

Unfolding (two step) vs Direct Question (one step)

11

Analyses GB

Variables of Interest

Independent variables (dichotomous) Mode: CAPI vs CATI vs WEB

Format Unfolding (two-step) vs Complete question

Dependent variables Mean score

The higher the mean score, the more respondents chose decrease in standard of living

The higher the mean score, the more respondents chose shopping facilities extremely good

Proportion of extreme answers Standard of living – categories 1 & 7

Shopping facilities – categories 1 & 6

12

Summary of GB results

Standard of living (7 point scale, with midpoint)

Unfolding increases mean score overall and within each mode

Unfolding increases extreme responses overall and just within web

Overall mode effect for extremes but not means

Almost interaction of format and CATI / Web for means

Rating of shopping facilities (6 point scale, no midpoint)

No overall format effect for means or extremes or within modes.

No overall mode effect for means or extremes but some mode

differences within formats

Focusing on last 2 categories, unfolding has consistent but non-

significant higher mean scores across all modes similar to standard

of living question

13

Compare GB & NL results

Consistent results:

Unfolding increases extremeness for both NL analyses of

extremes and 1 GB analysis of extremes

Inconsistent results:

Overall mode effect in NL data but only for 1 GB analysis

of extremes

Only one interaction of format and mode with NL data and

one almost present with GB data

Evidence recency effects in NL data but not in GB data

More extremeness without neutral category for NL data,

less extremeness without neutral for GB

Evidence of response contraction bias in GB results

14

Overall Conclusions

Small, statistically significant, effects of unfolding on extremes in both the Netherlands and Great Britain

Small, statistically significant, effects of mode in the Netherlands and in 1 case in Great Britain

Very few interactions

All effects found point to the same conclusion: Unfolding creates differences within and between modes

Don’t mix unfolding and complete question formats across modes


Orlando, May 17-20, 2012

THANK YOU!

Contact information

Edith de Leeuw: [email protected]

Joop Hox: [email protected]

Pamela Campanelli: [email protected]

Gerry Nicolaas: [email protected]

The

Survey

Coach


Orlando, May 17-20, 2012 NatCen Social Research

16

Appendix A: Statistical Results NL

Results for the Netherlands

Effects of mode and unfolding on

means and extremeness

Graphs, significance and effects

No covariate adjustment necessary

No selective nonresponse

17

Effect of Mode and Unfolding With Neutral Midpoint

Significant but small effect of mode and unfolding: R2= .008

No Interaction Format and Mode!

Means Overall CATI: 2.93

Web: 3.00

Unfolding: 2.94

Complete Q: 2.99

Mean Score

18

Effect of Mode and Unfolding Without Neutral Midpoint



Means Overall CATI: 2.93

Web: 2.97

Unfolding: 2.92

Complete Q: 2.97

Mean Score

19




Mean Prop 1 and 5 CATI: 0.26

Web: 0.19

Unfolding: 0.26

Complete Q: 0.19

Extremeness

20




Mean Prop 1 and 5 CATI: 0.31

Web: 0.25

Unfolding: 0.30

Complete Q: 0.25

Extremeness

21

Primacy? Effect Mode and Wording

Significant effect wording and

Interaction, R2=.08

Proportion 1’s Unrecoded

CATI Web Total

Unfolding .07 .10 .08

Complete

Question

.04 .03 .04

Total .06 .06 .06

Neutral Midpoint Without Neutral

Significant effect mode and wording

Interaction , R2=.14


CATI Web Total


Complete

Question

.05 .05 .05

Total .06 .10 .09

22

Recency? Effect Mode and Wording

Significant effect mode and wording

R2=.05


CATI Web Total


Complete

Question

.21 .15 .18

Total .24 .19 .20

Neutral Midpoint Without Neutral

Significant effect mode, wording,

and interaction , R2=.02


CATI Web Total


Complete

Question

.25 .20 .23

Total .26 .23 .24

23

Appendix B: Statistical Results GB

Results for Great Britain

Effects of mode and unfolding on

means and extremeness

Graphs, significance and effects

The GB results are based on analyses

with control variables carefully chosen to

adjust for differential nonresponse

between the 3 modes

24


Significant effect of unfolding, no overall effect of mode: R2= .043

Almost an interaction between format and CATI/Web (p = .101)

Means Overall CAPI: 4.73

CATI: 4.63

Web: 4.81

Unfolding: 4.97

Complete Q: 4.47

Mean Score

X is

unfolding!

25


Significant but small effect of unfolding and mode: R2= .020

No Interaction Format and Mode

Mean Prop 1 and 7 CAPI: 0.23

CATI: 0.16

Web: 0.16

Unfolding: 0.22

Complete Q: 0.15

Extremeness

X is

unfolded

26


No effect of unfolding, almost effect of mode: R2= .011


Means Overall CAPI: 3.97

CATI: 4.15

Web: 3.98

Unfolding: 4.03

Complete Q: 4.03

Mean Score

Y is

unfolding

27


No effect of unfolding or mode: R2= .009



CATI: 0.07

Web: 0.06

Unfolding: 0.06

Complete Q: 0.09

Extremeness Y is

unfolded

28

Primacy and Recency?

Neutral Midpoint Without Neutral Top category - Primacy - No

Very few cases:

Unfolding 1.3%

Complete question 2.0%

No differences by mode.

Last category - Recency – No Unfolding 21.8%

Complete Question 12.6%

Within complete question, CAPI

(with card) > Web = CATI.

No recency effects in CATI.

Top category - Primacy - No

Very few cases:

Unfolding 2.4%


Within Unfolding, CAPI (no card) >

CATI = Web .

No primacy effects for Web.

Last category - Recency – No Unfolding 4.8%


Within complete question, CAPI (no

card) >Web = CATI.

No recency effects in CATI. Not

clear why CAPI, but not also CATI.

29

Response contraction bias?

Tendency to avoid the most extreme response categories (see Tourangeau, et al, 2000)

No sign of this in top categories

But evidence in bottom categories

Combine last two categories

Standard of living – categories 6 & 7

Rating of shopping facilities – categories 5 & 6

Standard of Living Rating of shopping facilities

Unfolding

(two-step)

Complete

Question

Unfolding

(two-step)

Complete

Question

Category 6 34% 21% Category 5 26% 20%

Category 7 22% 13% Category 6 5% 7%

30


Significant effect of unfolding, no overall effect of mode: R2= .043

Almost an interaction between format and CATI/Web (p = .107)


CATI: 0.32

Web: 0.38

Unfolding: 0.46

Complete Q: 0.25

Extremeness:

Last 2 categories X is

unfolded.

31


No significant effects of unfolding or mode: R2= .005



CATI: 0.34

Web: 0.29

Unfolding: 0.34

Complete Q: 0.29

Extremeness:

Last 2 categories Y is

unfolded

Documents

Question or Mode Effects in Mixed-Mode Surveys: A Cross ... · Probability based panel, established in 2007 Random assignment to mode CATI: assigned 2000, responded 1207 (60%) Web: