Assessment of Bias When Field Operations Are Curtailed in a Mixed-Mode Telephone and Face-

to-Face Survey of Persons with Disabilities

May 13, 2011

Presentation to the AAPOR Annual Conference

Eric Grau

Multi-mode designs are used to boost response rates while controlling costs

Single mode:– Greater potential for nonresponse bias

Multi-mode:– Greater potential for measurement error bias

May be more problematic in certain populations (e.g., persons with disabilities)

Introduction

2

Telephone (CATI) and face-to-face (CAPI) interviews are often used together in mixed- mode studies

Similar: both involve an interviewer

Different:– CAPI interviews allow for greater ability to develop

rapport and maintain the respondent’s interest– CAPI interviews tend to be easier to match pace and

communication style

Background

3

Literature:– Response errors in CATI are different than those that

occur in face-to-face interviews– Krosnick (2002): greater cognitive demand

more shortcuts by respondents– Respondents who require more effort to obtain an

interview provide poorer quality data, increasing measurement error

Background (continued)

4

CAPI is seen as necessary to adequately survey the population in studies of persons with disabilities

CATI is used to save money in studies with limited resources

Mixed-mode approaches (such as CATI/CAPI) can therefore be used in studies of persons with disabilities that have limited resources, where CAPI interviews are only used for some difficult-to-reach subpopulations

Background (continued)

5

This study: assume field operations reduce bias. Under this assumption: how much is bias reduced?

Is it possible to scale back or eliminate CAPI interviews without significant changes in the respondent population, or significant changes in outcomes?

Previous research: mode effects may not be trivial

Research Question

6

Ticket to Work (TTW) program provides participants with a coupon to obtain employment training and other employment-related services

Eligibility for TTW program:– Must be a beneficiary of one or both of two Social

Security Administration programs for persons with disabilities, ages 18-64:

• Supplemental Security Income (SSI)• Social Security Disability Insurance (SSDI)

Evaluation of the Ticket to Work (TTW) Program

7

Component surveys– Beneficiary Survey

• Target Population– All 10.4 million SSI/SSDI beneficiaries

– Participant Survey• Target Population

– Among 10.4 million SSI/SSDI beneficiaries, subpopulation of TTW participants (108,000)

Evaluation of the Ticket to Work (TTW) Program

8

Collects health and well-being, service use, and employment information from SSI and SSDI beneficiaries

Dual-mode survey (CATI/CAPI)

Length: 45 minutes

In 2006 (third round of survey), 6,600 respondents– 82% response rate– Approximately 5,100 by CATI and 1,500 by CAPI– 23% of respondents by CAPI interview

National Beneficiary Survey

9

Multiple-Stage Cluster Sample– For the beneficiary sample and most subpopulations

of the participant population

Sample Design

10

3,382 sample members– 1,945 resolved by phone (completes, ineligibles,

nonrespondents)– 1,437 fielded (42%)

2,508 completes– 1,645 completed by phone– 863 fielded (34% of completes)

• 741 fielded completes with 60 days of field effort• No fielded completes with no field efforts

Attributes of Beneficiary Sample: Number Fielded and Number Complete

11

Used paradata on respondents

Process details (in chronological order)– Codes describing each locating and/or interview

attempt– The date that entries were recorded– Notes about the locating or interview attempt

Current Study

12

For sample members that were assigned to field operations at some point, we reassigned a disposition code1. As if no field operations were conducted

2. As if field operations were limited to 60 days after the first field assignment

Once this was done, nonresponse adjustments were recalculated with new status codes

Disposition Code Reassignment

13

Number of Standard Errors

Respondent’s Disability Frame

5 Months of Field Effort

60 Days of Field Effort

No Field Effort

Deaf 0.9% 0.1 -0.4 -1.2

Blind 2.4% 1.6 1.6 1.6

Psychiatric Disability

30.4% -2.6 -2.7 -3.1

Intellectual Disability

13.5% 0.7 0.6 -0.6

Other Physical Disability

52.8% 1.3 1.5 2.9

Comparison with Frame: Representative Beneficiary Sample

14

Number of Standard Errors

Respondent’s Disability Frame

5 Months of Field Effort

60 Days of Field Effort

No Field Effort

Deaf 0.9% 0.1 0.3 -0.1

Blind 2.4% 1.0 1.2 1.0

Psychiatric Disability

30.4% -1.2 -1.1 -0.7

Intellectual Disability

13.5% 0.4 0.6 0.1

Other Physical Disability

52.8% 0.5 0.2 0.3

Comparison with Frame: Beneficiary Sample After NR Adjustments

15

Variables with Differences in– Item nonresponse

• More item nonresponse in CATI– Social desirability

• More likely to give socially desirable answers in CATI– Acquiescence

• CATI respondents “better informed”

Comparison of Survey Variables: Measures

16

Maximum Number of Standard Errors Away from Original Estimate

Survey Variable60 Days of Field Effort No Field Effort

Race 0.27 0.50

Father’s education 0.20 0.98

Mother’s education 0.38 0.72

Education level 0.36 0.56

General health 0.24 0.56

Health insurance 0.14 1.79

Comparison of Survey Variables: Representative Beneficiary Sample (Item NR Variables)

17

Maximum Number of Standard Errors Away from Original Estimate

Survey Variable60 Days of Field Effort No Field Effort

Goals include moving up 0.37 0.40

Used drugs in past year 0.20 0.01

Work for pay next year 0.44 0.90

Work for pay next 5 years 0.32 0.71

Household income (median)

0.03 1.12

Comparison of Survey Variables: Representative Beneficiary Sample (Social Desirability Variables)

18

Maximum Number of Standard Errors Away from Original Estimate

Survey Variable60 Days of Field Effort No Field Effort

Heard of impairment-related work expenses exclusion

0.29 1.39

Heard of expedited reinstatement

0.16 1.66

Heard of BPAO 0.36 2.36

Heard of TTW 0.28 1.01

Comparison of Survey Variables: Representative Beneficiary Sample (Acquiescence Variables)

19

Mode effects do exist for some key variables

Removing field effort is not recommended

Consistent with work by Sloan, Wright, and Barrett (2006)

Variables that are prone to “acquiescence” (that is, saying yes just to get through the interview) sensitive to reduced field effort

Conclusions

20

Reducing field efforts to 60 days did not have a major effect on estimates– Possible cost-saving measure

More research needed– Only three levels (no field effort, 60 days of field

effort, and 5 months of field effort)– Need to determine how much field effort is enough

to minimize bias

Conclusions (continued)

21

Eric Grauegrau@mathematica-mpr.com

Mathematica Policy Research600 Alexander ParkPrinceton, NJ 08540

www.Mathematica-MPR.com

Contact Information

22