11

How Much of Interviewer Variance

is Really Nonresponse Error Variance?

Brady T. WestMichigan Program in Survey Methodology

University of Michigan-Ann Arbor

Kristen OlsonSurvey Research and Methodology Program

University of Nebraska-Lincoln

June 14, 2010International Total Survey Error Workshop 2010

22

Interviewer Variance: The Problem

An undesirable product of the data collection process, given interpenetrated sample designs

Responses for same interviewer are more similar than responses for different interviewers

Leads to inflation of variance in survey estimates due to intra-interviewer correlation, ρint

ρint = 0.01, 30 cases per interviewer 13.6% increase in SE of estimates

ρint usually less than 0.02, but can be larger

33

Research Question

Does ρint arise from complex interviewer-respondent interactions / probing for hard items?

High estimates of ρint (0.03–0.12) for factual (easy) and self-completion items in literature…

There is also consistent empirical evidence of interviewer variance in response rates

One estimates ρint with respondent data only, ignoring contributions of NR error variance

How much of interviewer variance can be attributed to nonresponse error variance?

44

The Wisconsin Divorce Study (WDS)

SRS of divorce records from four Wisconsin counties in 1989 and 1993

Sampled divorce records included official information also collected in a survey

The present study focuses on data collected using CATI: interviewer effects are likely attenuated relative to CAPI

n = 733 cases randomly sampled, and 355 CATI interviews performed by 31 trained interviewers

55

WDS Data

Six Survey Variables of InterestLength of Marriage in MonthsTime since Divorce in MonthsTime since Marriage in MonthsNumber of Marriages including the DivorceAge at MarriageAge at Divorce

Date of Divorce was recorded by an official body; other frame measures were reported by one member of the couple (possible errors)

66

Assigning Nonrespondents

Ideally, cases would not be worked by multiple interviewers (e.g., Singer and Frankel, 1982)

WDS used refusal conversion, and there were frequent changes in interviewers working non-finalized cases

This complicates the process of assigning nonrespondents to interviewers

77

Assigning Nonrespondents

Assumption: Interviewers working a particular shift work a random subsample of cases

The focus of this study is on interviewer variance within a shift, rather than across shifts

Persons with different characteristics are likely to be contacted at different times of the day

Account for shift: avoids possible confounding of nonresponse error with differences across shifts

88

Assigning Nonrespondents

Definitions of shifts:Weekday, 9-5pm (Shift 1: 26.8% of calls) Weekday, after 5pm (Shift 2: 44.1% of calls)Weekend, any time (Shift 3: 29.1% of calls)

Similar to work of Stokes and Yeh (1988)Interviewers worked multiple shiftsAlternative shifts were also considered,

and the study results did not change

99

Assigning Nonrespondents

Respondents were assigned to the interviewer completing the interview

Contacted refusals were assigned to: 1) the first interviewer receiving a refusal, or

2) the last interviewer to make contact

Non-contacts were assigned to the last interviewer making a call to the case

Random assignment of non-contacts was also considered; no change in results

1010

Assigning Nonrespondents

Limited power: 19-24 interviewers worked each shift, based on assignments

Large variability in assigned workloads across interviewers within a shift

Between 8 and 15 cases per interviewer within a shift, on average

Response rates lowest during the week, and especially before 5pm (41.1%)

1111

Analytic Approach

Examine interpenetration assumptions Estimate ρint for each survey variable within each

shift, based on respondent dataTest interviewer variance for significance

Estimate all variance components of the MSE of the respondent mean (possible with WDS data)

Estimate interviewer effects on (and interviewer contributions to) the variance components

Compute the proportion of interviewer-contributed variance due to NR error variance

1212

Example Derivation (Groves and Magilavy, 1984)MSE of respondent mean (u = # of refusals):

Example: Refusal error component

2 222

Noncontact

[ ( )]

Sampling

[

Measur( ) + + Refusae l ment

[ ]

+

[ )]

( ]

r

r r cr f rr ne

MSE x

Eu

E yE yc

En

Y yy yx yn

2

22

2 2

2 Refusal B[ ( )] + Refusal error variance

( ) ( )

ias

( )

r ref

r efe r rr f

u uE Y

uE y yn

uE Var y E Va Y

nr y

n n

13

Example Derivation, cont’d

Estimate variance components using linearized variance estimatorsAccounts for clustering due to interviewers and

unequal workloads

Estimate interviewer effects on variances based on estimates of intra-interviewer correlations in true values (Census Bureau, 1985)

13

1414

Example Derivation, cont’d

Estimated total contribution of interviewers to refusal error variance:

The estimated contribution is a function of intra-interviewer correlations in true values, for respondents and refusals

Similar derivations for other components

2 2

ˆ ˆˆ ˆ( ) ( 1) ( ) ( 1)r noint r r ref noint ref ref

u uVar y m Var y m

n n

1515

Results: Interviewer Variance Based on Respondent Data

Interpenetration evident in each shiftFour variable / shift pairs were found to

have unusually large estimates of ρint:Age at Divorce, Shift 2 (ρint = 0.08, p = 0.05)

Age at Divorce, Shift 3 (ρint = 0.10, p = 0.11)

Age at Marriage, Shift 2 (ρint = 0.11, p = 0.01)

Mths. since Marr., Shift 2 (ρint = 0.05, p = 0.13)

1616

Results: Sources of Interviewer Variance in Age at Divorce (Shift 2)Estimated intra-interviewer correlations in

Response errors: -0.003True values for respondents: 0.092True values for refusals: 0.008True values for noncontacts: -0.055

Total estimated variance of R mean: 0.423Total estimated variance contributed by

interviewers: 0.033Additional variance contributed by

interviewers is due to intra-interviewer correlations in true values for respondents!

17

Additional Results

Similar findings for age at divorce in shift 3Response error variance was main

contributor for age at marriage in shift 2 and months since marriage in shift 2

Response error variance may arise from outliers, as shown in the following graph

18

-4-2

02

4E

rro

r

Nonresponse Errors Measurement Errors

Mean Age at Divorce - Shift 2

-10

-50

5E

rro

r

Nonresponse Errors Measurement Errors

Mean Age at Divorce - Shift 3

-10

-50

5E

rro

r

Nonresponse Errors Measurement Errors

Mean Age at Marriage - Shift 2

-100

010

020

030

040

0

Err

or

Nonresponse Errors Measurement Errors

Mean Months Since Marriage - Shift 2

Illustration of Variance Sources

1919

Conclusions

Interviewer variance on key survey variables may arise from nonresponse error variance among interviewers

Interviewers may successfully obtain cooperation from different pools of respondents (e.g., older vs. younger)

Liking theory could be one explanation: variance in interviewer ages, voices variance in respondent ages (F. Conrad)

2020

Implications for Practice / Future Work

Monitoring Strategies: managers can continuously compare available features of R and NR for each interviewer, and intervene when large differences arise

Findings need to be replicated in a face-to-face setting with interpenetrated subsamples assigned to interviewers

Access to interviewer features would also enable use of multilevel modeling

2121

Thank You!

Bob Groves, Mick Couper, Frauke Kreuter and Paul Biemer have provided very helpful feedback and comments

Thanks to Vaughn Call for providing access to the WDS data

Please email bwest@umich.edu for these slides or a draft of the paper