Estimating Measurement Effects of Survey Modes from

Estimating Measurement Effects ofSurvey Modes from Between- andWithin-Subject Designs

Thomas KlauschJoop HoxBarry Schouten

Presentation at the 68th annual conference of the AAPORMay 16-19 2013, Boston, MA

Department of Methodology and Statistics

7 augustus 2013

1ViaInvoegen|KoptekstenVoettekstinvoegenSubafdeling<2spaties>|<2spaties>Titelvandepresentatie

Common Mixed-Mode Redesign Options

7 augustus 2013 2Klausch, Hox, & Schouten | Estimating Measurement Effects in Within-Subject Designs

Mode A

1. Single-to-Single Mode

Mode B

Mode A

2. Single-to-Mixed Mode

B A

3. Single-to-Mixed Mode

Mode A

A B

Redesigns require the same expected answer under modes A and B

1. For respondents under mode B2. For respondents under mode B3. For respondents under mode A

Estimate average difference in answers (measurement effects)

How to do this in the presence of nonresponse in both modes?

Between-Subject Mode Experiments

7 augustus 2013 3Klausch, Hox, & Schouten | Estimating Measurement Effects in Within-Subject Designs

Marginal Measurement Effect:= ( ) − ( )PotentialOutcomes

Between-Subject Mode Experiments

7 augustus 2013 4Klausch, Hox, & Schouten | Estimating Measurement Effects in Within-Subject Designs

Marginal Measurement Effect:= ( ) − ( )Response

Nonresponse

Response

Nonresponse

Between-Subject Mode Experiments

7 augustus 2013 5Klausch, Hox, & Schouten | Estimating Measurement Effects in Within-Subject Designs

Marginal Measurement Effect:= ( ) − ( )Response(S=1)

Nonresponse(S=0) Conditional Measurement Effects

for respondents in mode a or b:

= | = 1 − | = 1= | = 1 − | = 1

Use of Cond. MEs in Mixed-Mode Redesign

7 augustus 2013 6Klausch, Hox, & Schouten | Estimating Measurement Effects in Within-Subject Designs

Mode A

1. Single-to-Single Mode

Mode B

Mode A

2. Single-to-Mixed Mode

B A

3. Single-to-Mixed Mode

Mode A

A B

It can be shown that, if:

Redesigns 1 and 2 are possible

Redesign 3 is possible

Because then there is no relative measurement bias between modes(respondents give in expectation the same answers)

Marginal MEs not needed for these decisions

0bRME

0aRME

Estimation in Between-Subject Designs

7 augustus 2013 7Klausch, Hox, & Schouten | Estimating Measurement Effects in Within-Subject Designs

Exploit information on ‘X’, available for allunits (frame information)

Matching, regression or weightingtechniques can be used

Available X normally weakly associatedwith Ys or response mechanisms (S), e.g.socio-demographics

Thus: implausible MAR assumptions

Estimates then underlie selection bias(aka ‘selection effects’)

We suggest a new approach allowing forweaker assumptions when estimatingcond. MEs

Within-Subject Designs

Within-Subject Mode Experiments

7 augustus 2013 8Klausch, Hox, & Schouten | Estimating Measurement Effects in Within-Subject Designs

Time

Observed ‘Potential’ Outcome Strong relation of to Strong auxiliary information

Occasion twoSwitch to mode a

Forward Estimation of MEs for respondents in B

7 augustus 2013 9Klausch, Hox, & Schouten | Estimating Measurement Effects in Within-Subject Designs

Time

The forward method toestimate

MAR of at occasion 2 onearlier Assumes is time-stable ⊥ | , , = 1

Backward Estimation of MEs for respondents in A

7 augustus 2013 10Klausch, Hox, & Schouten | Estimating Measurement Effects in Within-Subject Designs

Time

The backward method toestimate

MAR of at occasion 1 onrepeated Assumes and aretime-stable ⊥ | , , = 1

Within-Subject Design with Control Group

7 augustus 2013 11Klausch, Hox, & Schouten | Estimating Measurement Effects in Within-Subject Designs

Time

Time

No mode switch Adjust time related

change of Y and S in‘treatment’ condition

Assumes that change isequivalent in mode A andB (plausible)

Control

‘Treatment’

Example: Quality of Social Life (Index)

Data collection Within-subject mode experiment with control group

Telephone, Mail, Web (1st occasion) followed up by F2F (2nd) Additionally F2F at 1st occasion (control)

Statistics Netherlands in 2011

Crime Victimization Survey, QSL index central variable in reporting

Estimation Regression Estimation (aka GREG)

Other methods possible, e.g. propensity score methods

Standard errors using 1000 bootstrap samples

7 augustus 2013 12Klausch, Hox, & Schouten | Estimating Measurement Effects in Within-Subject Designs

Missing Data Pattern for F2F (a) and Mail (b)

7 augustus 2013 13Klausch, Hox, & Schouten | Estimating Measurement Effects in Within-Subject Designs

Control

‘Treatment’

ME estimates with bootstrapped CI’s

7 augustus 2013 14Klausch, Hox, & Schouten | Estimating Measurement Effects in Within-Subject Designs

Within-Subject Within-Subject-Control-Group

Est. 95% CI p (Adj.)Est.

95% CI p

-.766 [-.925,-.608] <.001 -.729 [-.928,-.531] <.001

-.434 [-.556,-.314] <.001 -.489 [-.680,-.302] <.001

-.444 [-.554,-.336] <.001 -.389 [-.525,-.255] <.001( ) -.332 [-.185,-.479] <.001 -.240 [-.068,-.409] <.01( ) -.323 [-.178,-.469] <.001 -.340 [-.134,-.547] <.01∆ - - - .055 [-.025,.135] .179∆ - - - .093 [.008,.180] .033

Use of Cond. MEs in Mixed-Mode Redesign

7 augustus 2013 15Klausch, Hox, & Schouten | Estimating Measurement Effects in Within-Subject Designs

Mode A

1. Single-to-Single Mode

Mode B

Mode A

2. Single-to-Mixed Mode

B A

3. Single-to-Mixed Mode

Mode A

A B

Example: Cond. Measurement Effects identified For this variable: redesigns are not possible (incomparable answers)

Advisable to redesign questions to unified mode design, if possible(Dillman et al., 2009)

Otherwise a correction method is needed (will be possible!)

Conclusion

New technique to estimate measurement effects (aka mode effects)using within-subject designs

Weaker assumptions than estimation using socio-demographicauxiliary information in between designs

Better estimates, useful for effective mixed-mode design

Could be particularly useful in (mixed mode) panel settings

We can adjust for time-related change using control groups

Within subject design: no extra cost (compared to between design)

Control group extension: extra cost

Based on our model, adjustment of measurement effects will becomepossible

7 augustus 2013 16Klausch, Hox, & Schouten | Estimating Measurement Effects in Within-Subject Designs

ViaInvoegen|KoptekstenVoettekstinvoegenSubafdeling<2spaties>|<2spaties>Titelvandepresentatie

17

Appendix

7 augustus 2013

MAR Assumptions of Between-Designs

7 augustus 2013 18Klausch, Hox, & Schouten | Estimating Measurement Effects in Within-Subject Designs

Marginal Measurement Effect:

Conditional Measurement Effects:

⊥ |⊥ |⊥ | , = 1⊥ | , = 1:

:

Available X are seldom closelyrelated to Y or S

Illustration of Interplay of all Effects

7 augustus 2013 19Klausch, Hox, & Schouten | Estimating Measurement Effects in Within-Subject Designs

| = 1

| = 1