Upload
others
View
0
Download
0
Embed Size (px)
Citation preview
Address Based Sampling:
Lessons Learned Using Address Based Sampling to Conduct Mail Surveys of Populations Randomly S l t d f A th G l P l tiSelected from Among the General Population
Kelly Elver, Jennifer Dykema, and John StevensonKelly Elver, Jennifer Dykema, and John Stevenson
International Field Directors and Technologies Conference, May 18, 2010
University of Wisconsin Survey Center
© 2010. Materials may not be reproduced without permission of the author.
Outline of today’s talk
• What is Address Based Sampling• Why we chose ABS as the sampling method for this mail
survey• The sample design process
The project’s implementation• The project’s implementation• Personalization: Experimental design and results• Priority Mail: Description of use for final mailingPriority Mail: Description of use for final mailing• Costs of ABS sampling compared to estimated RDD cost• Lessons learned and suggestions for future investigationgg g
University of Wisconsin Survey Center 2
What is Address Based Sampling
• The Delivery Sequence File (DSF) from the United States Postal Service is a database containing all delivery point addressesaddresses.
• Sampling vendors have access to this database, and in combination with other resources, use it to create an extremely accurate frame from which to draw a random sample of households.p
• Address based sampling provides an increasingly attractive alternative to random digit dial (RDD) methods of samplingalternative to random digit dial (RDD) methods of sampling.
University of Wisconsin Survey Center 3
Why we chose ABS for this study
• Topic – A self administered questionnaire preferred given the sensitive nature of questions about contraception use.C t A il i l i d th• Cost – A mail survey is a less expensive mode than a telephone survey.
• Coverage – Reaching a young and diverse sample using RDD telephone methodology would be hard to accomplish. The study design called for sampling in a very specific geographic area.g g p
• Timing – We had one month following agreement to conduct the survey to put it in the field. Given out production schedule and programming load at that time a mail survey was quickerand programming load at that time, a mail survey was quicker to prepare for field.
University of Wisconsin Survey Center 4
The Sample Design Process
• Worked with Genesys to learn about and order an address based sample for the project.C l l t d th t d t f h h ld ith• Calculated the expected percentage of households with a woman in the required age range (25%) based on Census -American Community Survey figures.
• Calculated an expected response rate of 40 to 45%.• Using the desired number of 1,200 completes, calculated a
required sample size of 12,500 households to get the neededrequired sample size of 12,500 households to get the needed completes.
• Researchers wanted an African-American oversample, as well as urban and suburban representation so a plan to sampleas urban and suburban representation, so a plan to sample half of the households from the city of St. Louis (50% African American), and half from the county of St. Louis (20% African American) was adopted to provide both
University of Wisconsin Survey Center
American) was adopted to provide both.
5
The Sample Design Process
• Worked with Genesys to decide on types of addresses to include in sample
Ch i i l d• Choices include• P.O. Boxes• Drop Addresses (apartment buildings where un-p ( p g
personalized mail is left outside of individual mailboxes)• Rural routes or other types of “simplified” addresses
D i i i l i f th b dd t h ld b• Decisions on inclusion of the above address types should be made based on the geographic area and population density of your target sample
University of Wisconsin Survey Center 6
The Project Implementation
• 4 waves of data collection• Full mailing: cover letter, postage-paid return
envelope $2 incentive questionnaire screenerenvelope, $2 incentive, questionnaire, screener postcard (allowing households with no eligible women to opt out)
• Reminder postcard• Full mailing without incentive to non-responders• Full mailing without incentive to non-respondersFull mailing without incentive to non responders
University of Wisconsin Survey Center 7
The Project Implementation
• Address based sample does not have names available for all households, only those that match the listed household telephone database estimated to be 60% to 80%telephone database - estimated to be 60% to 80%.
• How to address households was discussed, literature reviewed, and we decided to conduct experiments.
• Designed first experiment with three levels of personalization• Designed first experiment with three levels of personalization to the salutation.
University of Wisconsin Survey Center 8
The Project Implementation
• Wondered if addressing a household where a name is available (estimated to be wrong 15% of the time), by the wrong name would be worse than “or current resident” junkwrong name would be worse than or current resident junk mail approach.
• Client wanted to send out final mailing via Priority mail rather than first class, so we incorporated this into the experimental design at the final mailing.g g
University of Wisconsin Survey Center 9
Methods: Experimental treatments for salutation
Sample Sizep
Treatment Matched Unmatched Total
Surname The <Surname> Household 3,448 0 3,448, ,
Municipal St. Louis Resident 3,446 1,081 4,527
Neighbor Our Neighbor At 3,446 1,079 4,525g g , , ,
Total N%
10,34083%
2,16017%
12,500100%
University of Wisconsin Survey Center 10
Methods: Experimental treatments for Priority mailing
Sample SizeSample Size
Treatment Priority First Class Total
Surname The <Surname> Household 872 873 1 745Surname The <Surname> Household 872 873 1,745
Municipal St. Louis Resident 1,268 1,269 2,537
Neighbor Our Neighbor At 1 312 1 313 2 625Neighbor Our Neighbor At 1,312 1,313 2,625
Total N 3,452 3,455 6,907
University of Wisconsin Survey Center 11
Analysis of the effects of treatment on participation
• Before the final mailing:• Effects of salutations on
• returning questionnaires• returning screener postcards• returned as undeliverablereturned as undeliverable
• For the final mailing• Effects of salutations and mailing type on
• returning questionnaires• returning screener postcards
• Final response rates• Final response rates
University of Wisconsin Survey Center 12
Questionnaire Return Rate by Salutation within Sample Before Third Mailing
13.31 13.311411.26
8 25
11.46 12.07
9.5310.54
10
12
urne
d
N i hb8.25
6
8
rcen
t ret
u NeighborMunicipalSurname
0
2
4Per
0All Matched Unmatched
University of Wisconsin Survey Center 13
Screener Post Card Return Rate by Salutation within Sample Before Third Mailing
28.16 28.163024.8724.9
26.78
18 87
28.16 28.16
23.2
20
25
30
ned
17.89 18.87
15
20
ent r
etur
n
NeighborMunicipalSurname
5
10
Perc
e Surname
0All Matched Unmatched
University of Wisconsin Survey Center 14
Questionnaire Return Rate by Mailing Type, Third Mailing Only
5
3.914.27
3.54
4.5
ned
2.21 2.002.272.17
22.5
3
ent r
etur
n
1st Class
Priority
0.51
1.5
Perc
e y
00.5
All Matched Unmatched
University of Wisconsin Survey Center 15
Screener Card Return Rate by Mailing Type, Third Mailing Only
9 279.7710 9.27
6.97789
10
ed
6.135.56
4567
nt re
turn
1st Class
Priority2.83
1234
Perc
e Priority
01
All Matched Unmatched
University of Wisconsin Survey Center 16
Response rate by wave for matched sample
5 75% 8.54%50.00%
55.00%
8.16% 8.16%9.60% 9.60%
9.76% 9.76%4.70%9.35% 5.57% 8.29%
5.75%
30 00%
35.00%
40.00%
45.00%
29.57% 29.57% 30.79% 30.79%35.14% 35.14%
15.00%
20.00%
25.00%
30.00%
0.00%
5.00%
10.00%
FirstClass Priorit FirstClass Priorit FirstClass PrioritFirst Class Priority First Class Priority First Class Priority
Neighbor Municipal Surname
Series1 Series2 Series3
University of Wisconsin Survey Center 17
Response rate by wave for unmatched sample
50.00%
55.00%
5.80% 7.03% 3.82%6.97%
30 00%
35.00%
40.00%
45.00%
% %
8.23% 8.23% 9.81% 9.81%
15.00%
20.00%
25.00%
30.00%
22.82% 22.82% 23.24% 23.24%
0.00%
5.00%
10.00%
First Class Priority First Class Priority
Neighbor Municipal
Series1 Series2 Series3
University of Wisconsin Survey Center 18
Experiment results
•Surname condition yielded highest response rates•Next time address to•Next time address to
•“The <Surname> Household or Current Resident”•“The Woman in Your Household”
•Municipality somewhat better than Neighbor•Municipality somewhat better than Neighbor•Worth the costs?•Harder to implement with more geographically diverse samplessamples
•Personalization as indicated by salutation more effective for Screener Postcard
Interaction between personalization and burden of the•Interaction between personalization and burden of the request
University of Wisconsin Survey Center 19
Overall Study Outcomes
Outcomes
Sample TypeSample N
Completed Surveys
Confirmed Ineligible Undeliverable
Response Rate
City of 6 250 789 1 576 735 42 9%City of St. Louis
6,250 789 1,576 735 42.9%
County of St Lo is
6,250 892 2,068 399 50.6%St. LouisTotal 12,500 1,681 3,644 1,134 46.9%
University of Wisconsin Survey Center 20
Costs for implementing ABS mail survey
• Costs to think about include• Pre-incentives ($2 in each outgoing wave 1 survey)• Screening postcard, stamped rather than metered,
included in every waveLarge N for wave mail outs• Large N for wave mail outs
• Priority mailing for final wave• The first class mailing was sent for 58 per mailingThe first class mailing was sent for .58 per mailing• The priority mailing was sent for $4.60 per mailing
Th dditi l l t f th t iliThe additional completes from that mailing were therefore obviously quite expensive, and probably not cost effective
University of Wisconsin Survey Center 21
Costs for implementing ABS mail survey
• Overall project cost was $158,702
• Estimated RDD Telephone survey budget was $286,535
S f $ %• Savings of over $127,000, or 45%
University of Wisconsin Survey Center 22
Lessons learned and suggestions for future investigation
• ABS sample can be used to randomly sample from the general population, including local samples and
loversamples.
• Screening is a challenge and not as reliable as RDD• Screening is a challenge, and not as reliable as RDDhousehold screening, so a cost versus quality tradeoff needs to be weighed when considering this method.
• Conducting mail surveys with ABS sample may provide significant cost savings over an RDD telephone surveysignificant cost savings over an RDD telephone survey, depending on the project.
University of Wisconsin Survey Center 23
Lessons learned and suggestions for future investigation
• More research needs to be done on how best to address sample where names are not available.
• May want to consider adding “Do Not Forward” to geography based ABS studiesgeography based ABS studies.
• Response rate calculations are challengingp g g• Should you use an e estimator modified for mail• Should you use Census data to estimate prevalence
of sample you are looking for• Should matched names returned as undeliverable be
treated as ineligible or re-fielded with less
University of Wisconsin Survey Center 24
treated as ineligible, or re fielded with less personalization?
Lessons learned and suggestions for future investigation
• Researching ABS methods before attempting to order from a sampling firm is recommended to avoid sampling errors.
• Researchers who use ABS should be encouraged to share the results, and the methods, very clearly, to help untangle the best ways to use this methodology.
University of Wisconsin Survey Center 25
Thank You!
F i f thi t ti i f ti t tFor copies of this presentation or more information, contact:
Kelly Elver, Director of Project ManagementKelly Elver, Director of Project [email protected]
Please visit us at:www.uwsc.wisc.edu
University of Wisconsin Survey Center 26