Embed Size (px)
Citation preview
© 2016 Ipsos. All rights reserved. Contains Ipsos' Confidential and Proprietary information and may
not be disclosed or reproduced without the prior written consent of Ipsos.
1
29th June 2017
DECLINING RESPONSE RATES AND THEIR IMPACT
Keith Bolling, Kantar Public UK
Patten Smith, Ipsos MORI
Evidence and contributions supplied by: BY…
Doc Name | Month Year | Version 1 | Public | Internal Use Only | Confidential | Strictly Confidential (DELETE CLASSIFICATION) 2
CONTENTS
1 What has been happening to response rates
in face-to-face cross-sectional surveys?
2
3
Levels of effort needed to maintain response
Response rates and relationship with survey error
Conclusions and implications 4
Doc Name | Month Year | Version 1 | Public | Internal Use Only | Confidential | Strictly Confidential (DELETE CLASSIFICATION) 3
BACKGROUND : WHAT HAS BEEN HAPPENING TO RESPONSE RATES
01
4
SURVEY RESPONSE RATES BY YEAR: AVERAGE
British Social Attitudes
Average
30
35
40
45
50
55
60
65
70
75
80
2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015
Resp
on
se R
ate
ch
an
ge (
%)
5
SURVEY RESPONSE RATES BY YEAR: DETAIL
BARB
British Social Attitudes
Crime Survey for England and
Wales
Family Resources
Survey
Health Survey for England
Labour Force Survey
Living Costs and Food Survey
National Travel Survey
Average
30
35
40
45
50
55
60
65
70
75
80
2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015
Resp
on
se R
ate
ch
an
ge (
%)
6
SURVEY RESPONSE RATES: CHANGE OVER TIME
-30
-25
-20
-15
-10
-5
0
5
Resp
on
se R
ate
ch
an
ge (
%)
1998/2001* - 2015
7
POSSIBLE EXPLANATIONS
Less trust in government, brands and professions
Survey fatigue
Availability
Accessibility
8
MAJOR FACTORS AFFECTING RESPONSE
Response
rate
Messaging
Channels
Volume
Timing
PARTICIPANT
COMMUNICATIONS
Volumes & allocation
Management
Reissues
FIELDWORK
SURVEY DESIGN
Sample type
Geography
Clustering
Sample data
Selection method
Interview length
Topic
Mode
Training
Motivation
Management
INTERVIEWER/NURSE
Type
Value
INCENTIVESocietal
change
Doc Name | Month Year | Version 1 | Public | Internal Use Only | Confidential | Strictly Confidential (DELETE CLASSIFICATION) 9
LEVELS OF EFFORT TO MAINTAIN RESPONSE
02
10
CASE STUDY RESPONSE RATES
51%
46%
75%72%
69%
63%
30%
35%
40%
45%
50%
55%
60%
65%
70%
75%
80%
2007 2008 2009 2010 2011 2012 2013 2014 2015 2016
Resp
on
se r
ate
Survey year
BSA CSEW BARB
All three surveys have maintained
low non-contact rates
– CSEW c. 3-4%
– BSA c. 6-7%
– BARB c. 15% > c. 10%
Refusal rates have crept up
especially in last few years
– CSEW c. 20% > c. 24%
– BSA c. 40% > c. 43%
– BARB c. 17% > 25%
11
MORE EFFORT BEING PUT IN TO KEEP RESPONSE RATES UP
4.7
3.94.1
5.2
4.8
5.2
2.0
2.5
3.0
3.5
4.0
4.5
5.0
5.5
BSA BARB CSEW
2008-9
2015-16
12
MORE EFFORT BEING PUT IN TO KEEP RESPONSE RATES UP
2008/9 2015/16 % increaseExtra calls per
year
BSA (2008-2016) 4.7 5.2 11% c. 4,000
BARB (2009-2015) 3.9 4.8 23% c. 72,000
CSEW (2008-2016) 4.1 5.2 27% c. 55,000
13
INCREASING EFFORT TO MAINTAIN INTERVIEWER PANELS
Challenges include:
– Recruitment
– Retention
– Engagement
– Response rate targets 1444
1021 11171322
11801186
1593
11611371 1495
36983367 3271
30022797
0
500
1000
1500
2000
2500
3000
3500
4000
2012 2013 2014 2015 2016
Joiners Leavers Panel Size
32% 47% 35% 46% 53%Leavers as % Total
Panel
Interviewer turnover across agencies
14
IS IT WORTH THE EFFORT (AND COST) TO MAINTAIN RESPONSE?
28%
42%46%
57%
67%
67%
46%
63%
72%
0%
10%
20%
30%
40%
50%
60%
70%
80%
0 1 2 3 4 5 6 7 8 9 10 11 12+
Resp
on
se r
ate
After x visits
BSA BARB CSEW
Doc Name | Month Year | Version 1 | Public | Internal Use Only | Confidential | Strictly Confidential (DELETE CLASSIFICATION) 15
Response rates and relationship with survey error
03
16
We worry about non-response because it can cause survey error
We care most about non-response bias: difference between estimates for
survey respondents and overall population
Non-response bias (NR bias) for variable Y increases with:
1. correlation between Y and likelihood of responding
2. (inversely with) response rate
NR bias:
• is variable-specific: wide range of NR bias values in a single survey
• increases as response rate decreases only if non-zero correlation between
response rates and variables
• but not by much unless correlation is substantial
RESPONSE RATE AND NON-RESPONSE BIAS
17
Relationship between RR and NR bias open; requires empirical assessment
Two types of empirical study:
1. Where estimate of “true” value of variable – difference from estimate for
survey respondents = estimate of absolute NR bias
2. Assess how survey estimates change with increasing fieldwork effort (e.g.
number of contact attempts, extent of reissuing). Look at impact of FW
efforts on relative NR bias (strong assumption that estimate changes
reflect reduced NR bias – might be wrong!)
RESPONSE RATE AND NON-RESPONSE BIAS
18
Number of studies - generally point in same direction. We summarise two
illustrative studies:
US: Groves and Peytcheva (2008): meta-analysis of studies of absolute NR
bias
UK: Sturgis, et al (2016) – relative NR bias and FW effort in 541 non-
demographic variables in six surveys
EMPIRICAL EVIDENCE
19
Meta-analyses (59 studies
/ 959 estimates)
Measured absolute NR
bias
Very low correlation
between RR and NR bias
Greater variation within
studies than between
them
GROVES AND PEYTCHEVA (2008)
20
Six recent Kantar Public surveys; all (541) non-demographic items
Relative bias (distance from final estimate) measured after different FW effort
levels:
• 1 call (RRs=7-22%)
• 2 calls
• 3 calls
• 5 calls
• final estimates (RRs = 55-76%)
STURGIS, WILLIAMS & BRUNTON-SMITH (2016, UK)
21
Average error after call 1 = 1.1% pts after calibration weighting
Average error after call 2 = 1.0% pts after calibration weighting
Study estimated that a 4-call rule leads to lowest # of total calls (hence, lowest cost)
On average, questions about beliefs and attitudes tended to respond to FW effort
slightly more than questions about behaviour
Small number of variables with high relative bias (internet use, freq. of being in
during day, freq. of going to pub) – but considerable reduction after 3 calls
STURGIS, WILLIAMS & BRUNTON-SMITH (2016, UK)
22
RR - NR bias relationship generally weak
However, for some variables in some studies extended FW
efforts make bigger difference – eg volunteering rates
BROAD CONCLUSIONS FROM EMPIRICAL EVIDENCE
Doc Name | Month Year | Version 1 | Public | Internal Use Only | Confidential | Strictly Confidential (DELETE CLASSIFICATION) 23
Conclusions and implications
04
24
1: How vulnerable to NR bias are the key variables?
Weighted survey estimates for most variables largely unaffected by response
rate variations within observed ranges
However occasional exceptions where RR has greater impact on NR bias: e.g.
volunteering and web usage - do you have any such variables? (literature /
earlier survey data…)
2: Will marginal increases in NR bias compromise conclusions?
Even with highest achievable response rates, levels of absolute NR bias may still
be substantial
Often trends of greater interest than point estimates. NR bias probably
relatively constant over time - little impact on trend lines
HOW MUCH SHOULD WE WORRY ABOUT
DECLINING RESPONSE RATES?
25
3: Are there better ways of reducing NR bias?
Blanket increase in RR often poor method for reducing NR bias
Address NR bias in other ways? – e.g. target increased FW efforts on
selected non-responders identified through supplementary variables
THINGS TO CONSIDER BEFORE SETTING
RESPONSE RATE TARGETSTHINGS TO CONSIDER BEFORE WORRYINGABOUT DECLINING RESPONSHOW MUCH SHOULD WE WORRY ABOUT DECLINING RESPONSE RATES?
26
Response rates are declining
And fieldwork efforts and costs are increasing
Observed impact of response rates on survey estimates usually modest
Pursuit of highest possible response rates may not be best use of
limited survey resources – may be better ways of addressing survey
errors
Making considerable efforts to maximise response rates can be
justified where minimising NR bias critical to conclusions
But often it isn’t
OVERALL CONCLUSIONS
