RTI International

RTI International is a trade name of Research Triangle Institute. www.rti.org

The Effects of Pushing Web in a Mixed-mode

Establishment Data Collection

American Association for Public Opinion Research

68th Annual Conference

May 16, 2013

Chris Ellis*, Kim Aspinwall, Todd Heinrich, Scott Ginder, Hope Smiley McDonald

RTI International

Margaret Noonan

Bureau of Justice Statistics

1

RTI International

Overview of Presentation

Acknowledgements

Overview of Deaths in Custody Reporting Program

(DCRP)

Genesis of DCRP Web Push Experiment

The Problem

Research Questions

Experiment Methodology

Results

Limitations

Conclusions

Areas for Further Research

2

RTI International

Acknowledgements

Bureau of Justice Statistics (BJS)

– Daniela Golinelli, Chief Corrections Statistics Program

– Margaret Noonan, Program Manager for DCRP

RTI International

– Kim Aspinwall, DCRP Data Collection Task Leader

– Todd Heinrich, Systems and Programming Task Leader

– Scott Ginder, DCRP Analysis and Reporting Task Leader

– Hope Smiley McDonald, DCRP Jail Universe Task Leader

The views expressed in this presentation do not necessarily reflect the official policies of the U.S.

Department of Justice or the Bureau of Justice Statistics; nor does mention of trade names,

commercial practices, or organizations imply endorsement by the U.S. Government.

This presentation is sponsored by RTI International’s Survey Research Division.

3

RTI International

Deaths in Custody Reporting Program

Primary source of mortality statistics within the American

correctional system

– 50 state departments of correction (DOCs)

– Approximately 3,000 local jail jurisdictions

Multimode data collection

– Respondents are typically prison and jail administrators

– Self-reporting

– Two forms: individual death reports and post-hoc annual

summary

– Web, paper, fax, e-mail, bulk data file, and (during NRFU)

telephone

4

RTI International

Deaths in Custody Reporting Program (cont.)

“Dillman-esque” data collection protocol implemented

throughout post-reference year reporting period

– Initial lead mailing (including reporting/Web login instructions)

– Thank you/reminder e-mail or postcard

– Replacement forms mailing

– Nonresponse telephone prompts

– Data quality follow-up calls for critical missing/conflicting items

2011 response rates (AAPOR RR2):

– 100% for 50 state DOCs

– 96.7% for “Top 150” jail jurisdictions

– 96.8% for remaining jail jurisdictions

[RR2 % = (I+P) / (I+P) + (R+NC+O) + (UH+UO)]

5

RTI International

Genesis of DCRP Web Push Experiment

Study-specific insights

– Using concurrent multimode approach, DCRP respondents used

the Web for their primary response (Heinrich et al., 2012)

68.8% in 2009

69.4% in 2010

– An even greater number of respondents indicated willingness to

respond via Web

– Over 90% of agencies had one or more e-mail addresses on file

Additional factors

– Data quality objectives would be aided by increased Web uptake

– The reduction of paper form submissions would likely decrease

data collection costs

6

RTI International

The Problem

Study-specific concerns

– DCRP study population has known paper-only submitters

– Further, some remote jails are known to not have computers/IT

– Historically, DCRP has always offered paper forms

Other concerns from the literature

– Giving respondents “buffet-like” choice of mode can lead to (Medway & Fulton, 2012)

Perceived increased complexity, leading to

Lower response rates

– “Pushing” one mode at the onset of data

collection may (Mooney et al., 2012)

Increase respondents’ selection of mode, but

Lower overall response rates

7

RTI International

Research Questions

BJS and RTI embedded an experiment in the 2011 data

collection cycle (which occurred in 2012 following the CY)

– Treatment: withhold paper forms from the initial survey request

(i.e., “push” the Web)

– Control: continue to offer paper forms concurrent with other mode

invitations (e.g., Web, bulk data file)

The research questions:

1. Do overall response rates vary when the Web mode is pushed?

2. Does time-to-response (TTR) vary when the Web mode is

pushed?

3. What are the cost implications when the Web-push method

(and its outcomes) are applied to the entire sample?

4. Does pushing the Web have an impact on mode self-selection?

8

RTI International

Experiment Methodology

Treatment affected contents of initial January mailing

Cohort 2010 Mode Treatment or Control Sample Size

1 Paper Control (Paper Forms) 337

2 Paper Treatment (No Paper) 338

3 Web Control (Paper Forms) 887

4 Web Treatment (No Paper) 888

5 Mixed Control (Paper Forms) 118

6 Mixed Treatment (No Paper) 119

Total 2,687

• Randomization controlled for 2010 response mode and speed of response

• DOCs, 2010 nonresponders, and special situations were excluded 9

RTI International

Experiment Methodology (cont.)

The treatment cohort received

– A DOJ-signed cover letter, with Web login

credentials

– DCRP informational handout

– 2011 reporting instructions

– 2012 reporting instructions

– Enclosed in a 10 x 13 outer mailing envelope

The control cohort received

– All of the above, but

– A 2011-specific inner envelope with those instructions,

a business reply envelope, and 2011 paper forms

– A 2012-specific inner envelope with those instructions,

a business reply envelope, and a 2012 paper form

All mailings were simultaneous to measure TTR 10

RTI International

Results* – Response Rates

* Analysis included 2,653 of the original 2,687 cases

78% 73%

10% 14%

10%

98%

11%

98%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Paper Included (PR group)

Paper Excluded (WP group)

Resp

on

se R

ate

Cohort

Response Rates by Treatment Cohort

Gain Between Nonresponse Calling and End of Data Collection

Gain Between Replacement Forms Mailing and Nonresponse Follow-up

Gain Between Initial and Replacement Forms Mailing

11

RTI International

Results* – Time-to-Response (TTR)

The treatment (Web push) cohort responded 1.6

days faster than the control cohort on average

That said, the difference was only marginally

statistically significant (p = 0.07)

Withholding paper certainly did not negatively impact

TTR

* Analysis included 2,653 of the original 2,687 cases

2011 TTR (Days) 2010 TTR (Days)

Cohort Mean Median Mean Median

Paper

Included

19.0 10.0 36.5 20.5

Paper

Excluded

17.4 8.0 35.4 19.0

12

RTI International

Results* – Mode Selection

224 (22%)

20 (2%)

777 (75%)

921 (95%)

31 (3%) 32 (3%)

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Forms included Forms excluded

Perc

en

t

Treatment Cohort

Response Mode, by Treatment Cohort

2011 mode during experiment period Other (mixed, phone, etc.)

2011 mode during experiment period Web

2011 mode during experiment period Paper

* Analysis included 2,653 of the original 2,687 cases

13

RTI International

Results* – Mode Selection (cont.)

Not surprisingly, clear correlation between treatment

cohort and respondent mode selection

– Inclusion of paper forms increased the incidence of paper mode

response by 20 percentage points

– Withholding paper resulted in 95% of respondents choosing Web

– An equal amount responded using mixed or multiple modes (3%)

Paper inclusion led to higher selection of Web response

across both cohorts; upon receipt of the replacement

forms mailing

– 30.8% of Web push (treatment) members responded via paper

– 33.3% of Paper (control) members responded via paper

The propensity for Web response changed

* Analysis included 2,653 of the original 2,687 cases 14

RTI International

Results* – Costs of Data Collection

To compute the impact on costs:

– The performance of each cohort was measured according to key

data collection components,

– Noting how many agencies required which types of follow-up,

plus

– How many required paper forms processing

* Analysis included 2,653 of the original 2,687 cases

Cohort

Initial

Mailout

Replacement

Forms Mailout

Nonresponse

Telephone

Contact

Respondents

via Paper

Paper

Included

2,942 642 351 633

Paper

Excluded

2,942 795 385 212

15

RTI International

Results* – Costs of Data Collection (cont.)

$10,768

$19,182

$4,023

$3,249

$6,988

$6,371

$1,105

$3,298

$22,883

$32,099

$0 $10,000 $20,000 $30,000

Paper Excluded (WP

Group)

Paper Included (PR

Group)

Estimated Cost

Tre

atm

en

t

Estimated Component and Total Costs, by Treatment Cohort

Initial Mailing Costs

Replacement Forms Mailing Costs

Nonresponse Phone Contact Costs

Paper Response Mode Costs

16 * Analysis included 2,653 of the original 2,687 cases

RTI International

Results* – Costs of Data Collection (cont.)

The Web push (treatment) cohort incurred

– Fewer initial mailing costs (no printing, less postage and

handling)

– Slightly more replacement forms mailing costs

– Almost identical nonresponse telephone contacting costs, and

– Fewer paper processing costs (receipt and processing, data

entry)

Costs of the two cohorts were computed, as if applied to

the entire study

The Web push (treatment) approach would

effect a savings of $9,200 (~$3.47 per case)

* Analysis included 2,653 of the original 2,687 cases

17

RTI International

Limitations

Some limitations associated with experiment; the DCRP

study

– Is a time series collection that is well known to the field

– Was a mandatory collection until 2006

– Typically enjoys high response rates

– Employs a robust nonresponse prompting protocol

– Allows for a multi-month response period

18

RTI International

Conclusions

1. Web can be pushed at the onset without jeopardizing

overall response rates

2. Implementing this change does not adversely affect

TTR

3. Withholding paper forms can lead to cost savings,

despite some interim cohort increases

4. Pushing the Web mode clearly influences respondents’

selection

5. Including paper as an initial option led to higher rates of

paper being used to respond

19

RTI International

Areas for Further Research

What are the effects of withholding paper from later-

stage promptings, too?

What are the effects of not explicitly offering paper, but

instead providing it only upon request?

Can the results of this experiment be replicated across

other establishment surveys?

20

RTI International

References

de Leeuw, E. D. (2005). To Mix or Not to Mix Data Collection Modes in Surveys. The

Journal of Official Statistics, 21(2), 233-255.

Dillman, D. A. (2007 updated). Mail and Internet surveys: The tailored design method

(2nd ed.), Hoboken, NJ: John Wiley Co.

Heinrich, T. D., Ellis, C. S., Ham, M. W., Ginder, S. A., Smiley McDonald, H. M.,

Aspinwall, K. R., & Noonan, M. (2012, June). Have It Your Way: Managing Data

and Business Preferences in a Multimode Collection. Presented at Fourth

International Conference on Establishment Surveys, Montreal, Quebec, Canada.

Medway, R. L., & Fulton, J. (2012). When more gets you less: a meta-analysis of the

effect of concurrent web options on mail survey response rates. Public Opinion

Quarterly, 76(4), 733-746.

Messer, B. L. (2012). Mixed-Mode & Internet Surveys: Lessons from AAPOR 2012.

Presented at the Pacific Association for Public Opinion Research Mini-

Conference, San Francisco, CA June 22, 2012.

21

RTI International

References (cont.)

Minton, T.D. (2011). Jail Inmates at Midyear 2010 – Statistical Tables. Washington,

DC: U.S. Department of Justice, Office of Justice Programs, Bureau of Justice

Statistics, Washington DC. Available at

http://www.bjs.gov/index.cfm?ty=pbdetail&iid=2375.

Mooney, G., Lan, F., Lin, X., & Hurwitz, A. (2012). Influencing Mode Choice in a

Mixed Mode Survey. Presented at the American Association for Public Opinion

Research 67th Annual Conference. Orlando, FL. May 2012.

Noonan, M. E. (2012). Prison and jail deaths in custody, 2000-2010 – Statistical

tables. Bureau of Justice Statistics, Washington DC. Available at

http://bjs.gov/index.cfm?ty=pbdetail&iid=4558.

Schaefer, D. R., & Dillman, D. A. (1998). Development of a standard e-mail

methodology: Results of an experiment. Public Opinion Quarterly, 62(3), 378-

397.

22

RTI International

More Information

Chris Ellis

Senior Survey Director

RTI International

919.541.6480

ellis@rti.org

Presentation available at:

www.rti.org/aapor

23

SurveyPost blogs.rti.org/surveypost

@SurveyPost