Are the Hard-to-Cover Also Less Likely to Respond?

AAPOR 2015

Stephanie EckmanFrauke Kreuter

2

Motivation

Reducing undercoverage means:‐ Phone: calling cell numbers ‐ Face-to-face: including homeless, institutionalized‐ Web: providing tablet, internet access

Costly

Are the additional people included disproportionately nonresponders?

3

Examples of Trade-Off

RDD + mobile phone surveys (AAPOR Cell Phone Task Force Report, 2010)

‐ Lower RR among mobile only HHs

LISS online panel (Leenheer & Scherpenzeel, 2013)

Screener experiment (Tourangeau, Kreuter & Eckman, 2012)

Condition Coverage Rate Recruit. RateInternet not provided 90% 84%Internet provided ~ 100% 79%

Condition Coverage Rate Response RateDirect: “Is anyone there 35-55?” 32% 86%Roster: Age of all adults in HH 45% 72%

4

Simulation Setup

Coverage propensity CPi‐ Propensity for case to be included on frame‐ Determined by X variables: ‐ comfort with technology, attachment to HH

Response propensity RPi‐ Propensity for case to respond ‐ Determined by Z variables‐ at-home patterns, privacy concerns, topic interest

5

Path Diagram

X

Z

Y

CP

RP

6

Path Diagram

X

Z

Y

CP

RP

ρ

βX

βZ

γX

γZ

7

Simulation Setup

Y related to X, Z

Survey A:

Survey B:‐ CP increased by κ ‐ RP unchanged

8

Simulation Setup

Y related to X, Z

Survey A:

Survey B:‐ CP increased by κ ‐ RP unchanged

Parameter Values0.2, 0.6, 1-0.2, -0.6, -1

0.2, 2

0.2, 2

0.2, 2, -0.2, -2

0.2, 2, -0.2, -2κ 1, 4, 10

9

Outcome Measures

When we increase coverage (A →B), what happens?

Coverage rate, RR‐ Under what conditions do we see trade-off?

Nonresponse Undercoverage Bias Bias

‐ Under what conditions will bias decrease (in absolute value)?

10

Results: Rates

cov(X,Z) > 0; strong relationship between X, CP & Z, RP

Results: When Does Absolute Bias Decrease?

X

Z

Y

CP

RP

ρ

βX

βZ

γX

γZ

11

+

Results: When Does Absolute Bias Decrease?

X

Z

Y

CP

RP

ρ

βX

βZ

γX

γZ

12

+

Results: Bias

X

Z

Y

CP

RP

ρ

βX

βZ

γX

γZ

13

+

www.iab.de

stephanie.eckman@iab.de

Website: stepheckman.com

Thanks – comments & ideas welcome

16

Results: Rates

17

Results: Rates

18

Coverage, Response Rates for 2 Surveys

CaseLandline Only Landline + Mobile

Covered Response Propensity Covered Response

Propensity

1 Yes 0.48 Yes 0.482 Yes 0.60 Yes 0.603 No 0.15 Yes 0.154 No 0.36 Yes 0.365 No 0.26 Yes 0.26

Coverage Rate 40% 100%Response Rate 54% 37%