Embed Size (px)
Citation preview
Techniques for GeneratingHigher Response Rates
Presented at the 2006 Presented at the 2006 American Educational Research Association Annual Meeting American Educational Research Association Annual Meeting
April 7 – 11 San Francisco, CAApril 7 – 11 San Francisco, CA
Kevin Kenward, PhDKevin Kenward, PhD
Director of ResearchDirector of Research
National Council of National Council of
State Boards of State Boards of NursingNursing
Lynn C. Webb, Ed.D.Lynn C. Webb, Ed.D.
Testing ConsultantTesting Consultant
“Increasingly, [people] are refusing to participate in consumer surveys.”
“A truly representative survey may be unattainable…”
The New York TimesJune 2004
Status of Survey Nonresponse
• Falling cooperation rates in the developed world (de Leeuw and de Heer, 2002)
• Greater decline in one-time surveys vs. longitudinal surveys (AAPOR, 2003)
• Greater decline in unit nonresponse than item nonresponse
An electronic survey was conducted to compare the roles of two types of Advanced Practice Registered Nurses
Nurse Practitioners (NP)Clinical Nurse Specialists (CNS)
Job Analysis
The professional sophistication of Advanced Practice RNs led us to believe that an electronic survey would be appropriate, and the electronic format offered advantages over paper.
The history of nurse response rates
40%
led us to believe we would have strong participation.
The response rate was disappointingly low. 13%
In desperation, nurses were randomly selected from the mailing list and asked why they did not participate in the study.
TOO BUSY was the recurring theme in the explanations offered.
How do We Get People to Complete Our Surveys and Provide us with Good Information ??
Techniques for GeneratingHigher Response Rates
Preliminary or Advance Notification
Follow-ups Appeals Postage and Mailing Classes Personalization Incentives
Decreasing Response Rates
QUESTIONNAIRE LENGTH - MAIL
• 4 PAGES OR LESS No Effect
• MORE THAN 4 PAGES - 8%
MONETARY INCENTIVES
Prepaid - Creates Trust and Obligation
Amount - Symbolic Token of Appreciation
Understanding the Techniques
Phone Call ReminderPhone Call Reminder
Mail SurveyMail SurveyLong FormLong FormVersion 1Version 1
Mail SurveyMail SurveyLong FormLong FormVersion 2Version 2
Mail SurveyMail SurveyShort FormShort Form
N=1,500
N=1,500
N=1,000
Mail SurveyMail SurveyLong FormLong FormVersion 1Version 1
+ + Photo on Cover LetterPhoto on Cover Letter
Mail SurveyMail Survey
+ + $1 Incentive$1 Incentive
Mail SurveyMail Survey
+ + $2 Incentive$2 Incentive
Mail SurveyMail Survey
+ + $5 Incentive$5 Incentive
Monetary Incentives
N=250
N=250
N=250
Mail SurveyMail Survey+ +
Starbucks $3 Starbucks $3 Gift CertificateGift Certificate
Mail SurveyMail Survey+ +
$250 Drawing$250 Drawing
N=250
N=1,000
2%
26%
Results
Mail SurveyMail SurveyLong FormLong FormVersion 1Version 1
Mail SurveyMail SurveyLong FormLong FormVersion 2Version 2
Mail SurveyMail SurveyShort FormShort Form
24%
26%
30%
Results
Increasing Response Rates
Monetary Incentives
RESPONSE
AMOUNT RATE
$1.00 43%
$2.00 42%
$5.00 54%
SOURCE: Yu and Cooper, JMR.
Increasing Response Rates
NON-MONETARY INCENTIVES
Starbucks $3 Gift CertificateStarbucks $3 Gift Certificate 42%42%
$250 Drawing$250 Drawing 28% 28%
Effective Nonresponse Rate Reduction Methods
• Incentives reduce nonresponse rates
higher effects with increasing incentives
prepaid incentives more powerful than promised
cash more powerful than in-kind
incentives can reduce total survey costs
Sample Size
Mailing cost
Response Rate
Cost per Complete
10,000 10,000
$14,000
$5 Incentive
No Incentive
$64,000
14% 54%
$10.00 $11.85
2nd Mailing
10,000 – 1,400 = 8,660
5% Response Rate
Cost = $12,040
Mailing cost
Response Rate
Cost per Complete
18,600 10,000
$26,040
$5 Incentive
No Incentive
$64,000
18% 54%
$14.23 $11.85
Total Mailed To
Returned Incentives
$1$2$5
Starbucks
# People1353
2
