Estimating Measurement Effects of Survey Modes from

Estimating Measurement Effects ofSurvey Modes from Between- andWithin-Subject Designs

Thomas KlauschJoop HoxBarry Schouten

Presentation at the 68th annual conference of the AAPORMay 16-19 2013, Boston, MA

Department of Methodology and Statistics

7 augustus 2013

1ViaInvoegen|KoptekstenVoettekstinvoegenSubafdeling<2spaties>|<2spaties>Titelvandepresentatie

Common Mixed-Mode Redesign Options

7 augustus 2013 2Klausch, Hox, & Schouten | Estimating Measurement Effects in Within-Subject Designs

Mode A

1. Single-to-Single Mode

Mode B

Mode A

2. Single-to-Mixed Mode

B A


Mode A

A B

Redesigns require the same expected answer under modes A and B

1. For respondents under mode B2. For respondents under mode B3. For respondents under mode A

Estimate average difference in answers (measurement effects)

How to do this in the presence of nonresponse in both modes?

Between-Subject Mode Experiments


Marginal Measurement Effect:= ( ) − ( )PotentialOutcomes



Marginal Measurement Effect:= ( ) − ( )Response

Nonresponse

Response

Nonresponse



Marginal Measurement Effect:= ( ) − ( )Response(S=1)

Nonresponse(S=0) Conditional Measurement Effects

for respondents in mode a or b:

= | = 1 − | = 1= | = 1 − | = 1

Use of Cond. MEs in Mixed-Mode Redesign


Mode A


Mode B

Mode A


B A


Mode A

A B

It can be shown that, if:

Redesigns 1 and 2 are possible

Redesign 3 is possible

Because then there is no relative measurement bias between modes(respondents give in expectation the same answers)

Marginal MEs not needed for these decisions

0bRME

0aRME

Estimation in Between-Subject Designs


Exploit information on ‘X’, available for allunits (frame information)

Matching, regression or weightingtechniques can be used

Available X normally weakly associatedwith Ys or response mechanisms (S), e.g.socio-demographics

Thus: implausible MAR assumptions

Estimates then underlie selection bias(aka ‘selection effects’)

We suggest a new approach allowing forweaker assumptions when estimatingcond. MEs

Within-Subject Designs

Within-Subject Mode Experiments


Time

Observed ‘Potential’ Outcome Strong relation of to Strong auxiliary information

Occasion twoSwitch to mode a

Forward Estimation of MEs for respondents in B


Time

The forward method toestimate

MAR of at occasion 2 onearlier Assumes is time-stable ⊥ | , , = 1

Backward Estimation of MEs for respondents in A


Time

The backward method toestimate

MAR of at occasion 1 onrepeated Assumes and aretime-stable ⊥ | , , = 1

Within-Subject Design with Control Group


Time

Time

No mode switch Adjust time related

change of Y and S in‘treatment’ condition

Assumes that change isequivalent in mode A andB (plausible)

Control

‘Treatment’

Example: Quality of Social Life (Index)

Data collection Within-subject mode experiment with control group

Telephone, Mail, Web (1st occasion) followed up by F2F (2nd) Additionally F2F at 1st occasion (control)

Statistics Netherlands in 2011

Crime Victimization Survey, QSL index central variable in reporting

Estimation Regression Estimation (aka GREG)

Other methods possible, e.g. propensity score methods

Standard errors using 1000 bootstrap samples


Missing Data Pattern for F2F (a) and Mail (b)


Control

‘Treatment’

ME estimates with bootstrapped CI’s


Within-Subject Within-Subject-Control-Group

Est. 95% CI p (Adj.)Est.

95% CI p

-.766 [-.925,-.608] <.001 -.729 [-.928,-.531] <.001

-.434 [-.556,-.314] <.001 -.489 [-.680,-.302] <.001

-.444 [-.554,-.336] <.001 -.389 [-.525,-.255] <.001( ) -.332 [-.185,-.479] <.001 -.240 [-.068,-.409] <.01( ) -.323 [-.178,-.469] <.001 -.340 [-.134,-.547] <.01∆ - - - .055 [-.025,.135] .179∆ - - - .093 [.008,.180] .033

Use of Cond. MEs in Mixed-Mode Redesign


Mode A


Mode B

Mode A


B A


Mode A

A B

Example: Cond. Measurement Effects identified For this variable: redesigns are not possible (incomparable answers)

Advisable to redesign questions to unified mode design, if possible(Dillman et al., 2009)

Otherwise a correction method is needed (will be possible!)

Conclusion

New technique to estimate measurement effects (aka mode effects)using within-subject designs

Weaker assumptions than estimation using socio-demographicauxiliary information in between designs

Better estimates, useful for effective mixed-mode design

Could be particularly useful in (mixed mode) panel settings

We can adjust for time-related change using control groups

Within subject design: no extra cost (compared to between design)

Control group extension: extra cost

Based on our model, adjustment of measurement effects will becomepossible


ViaInvoegen|KoptekstenVoettekstinvoegenSubafdeling<2spaties>|<2spaties>Titelvandepresentatie

17

Appendix

7 augustus 2013

MAR Assumptions of Between-Designs


Marginal Measurement Effect:

Conditional Measurement Effects:

⊥ |⊥ |⊥ | , = 1⊥ | , = 1:

:

Available X are seldom closelyrelated to Y or S

Illustration of Interplay of all Effects


| = 1

| = 1

Documents

Estimating Measurement Effects of Survey Modes from