Upload
katherine-higgins
View
218
Download
0
Tags:
Embed Size (px)
Citation preview
Stop the Madness:Use Quality Targets
Laurie Reedman
Scope Aspects of quality
• Timeliness and accuracy
Mechanisms to manage quality• Indicators and pre-set targets
Survey processes• Computer assisted interviewing
• Collection follow-up
• Manual processing
23-04-21Statistics Canada • Statistique Canada2
Statistics Canada’s dimensions of quality Relevance Accuracy Timeliness Accessibility Interpretability Coherence
23-04-21Statistics Canada • Statistique Canada3
Statistics Canada’s dimensions of quality Relevance Accuracy Timeliness Accessibility Interpretability Coherence
23-04-21Statistics Canada • Statistique Canada4
Statistics Canada’s dimensions of quality Relevance Accuracy Timeliness Accessibility Interpretability Coherence
23-04-21Statistics Canada • Statistique Canada5
How can a survey manager manage both process and product quality of data collection and manual processing activities?
Interviewer Monitoring
Computer assisted interviewing Monitor observes and grades samples of
interviewer work Frequency of monitoring sessions geared to
attain desired average outgoing quality level
23-04-21Statistics Canada • Statistique Canada6
Responsive Collection Design (RCD)
An adaptive approach to survey data collection Uses information prior to and during data
collection to adjust the strategy for the remaining in-progress cases (Groves and Herringa, JRSS 2006)
Can use RCD to:• Control quality (response rate, representativeness)
• Control cost (time and resources spent)
23-04-21Statistics Canada • Statistique Canada7
Responsive Collection Design (RCD) RCD was piloted on 2 surveys at Statistics Canada
(Laflamme and St-Jean, JSM 2011). Three distinct phases during data collection
• Early in collection – attempt all cases
• Mid collection – increase response rates
• Late collection – reduce variability of response rates between domains of interest
Key to success is changing from one phase to the next at the optimal time
23-04-21Statistics Canada • Statistique Canada8
Responsive Collection Design (RCD) The turning point decisions are based on the
comparison of quality indicators to pre-set target levels Indicators are derived from paradata from current and
previous collection activity• If targets are too high or too low the turning points will not be
effective at improving quality
• Targets need to reflect the priorities, for example to reduce costs, improve response rates, or optimize both simultaneously
23-04-21Statistics Canada • Statistique Canada9
Selective Editing and Top-Down Approach Data editing is a quality assurance activity, not a data correction
activity (John Kovar, 199?) Goals:
• Make data fit for use (not perfect) - effectiveness
• Use as few resources as necessary - efficiency Human resources to do telephone follow-up calls and manual
analysis and data modification are costly Managers need a mechanism to improve efficiency and
effectiveness of manual processes without significantly impacting accuracy
23-04-21Statistics Canada • Statistique Canada10
Selective Editing and Top-Down Approach Focus effort where it will do the most good (Hedlin, UNECE
2008) Tackle efficiency and effectiveness from two angles:
• Choose certain units or domains of units for further processing, cease processing of the rest
• Arrange the units requiring further processing in priority order
Pro-actively control the impact on quality by basing turning points and priorities on comparisons of quality indicators to pre-set targets
23-04-21Statistics Canada • Statistique Canada11
Selective Editing
When to stop processing• Quality indicator could be mean squared error,
coefficient of variation, response rate, calculated for key variables at cell or domain level
Targets need to be set carefully• If too high, might never be reached, end of processing
will never be triggered, costs will not be reduced
• If too low, resulting data might not be fit for use
23-04-21Statistics Canada • Statistique Canada12
Top-Down Approach
How to prioritize units needing more processing• Score function – to get a single rank incorporating
several different criteria simultaneously
• “Biggest” based on some size measure (prior knowledge)
• “Biggest” based on a measure of impact (relative to what has already been collected)
• Most outrageous errors (outlier detection)
23-04-21Statistics Canada • Statistique Canada13
An Example: Edit process Canadian Census of Population edit and imputation
process 110 modules grouped into 43 processes Underwent a “Quality Assurance Review” in 2013
(Reedman and Julien, FCSM 2013). 65-70% of time was spent on manual data verification Recommended increased use of automation, and pre-set
quality thresholds to limit activity that amounts to “polishing the apple”
23-04-21Statistics Canada • Statistique Canada14
An Example: Edit process Pro-active quality management could include:
• Derive quality indicators for key variables, compare to pre-set targets, and direct satisfactory records onwards to the next processing step, while only retaining unsatisfactory records for appropriate intervention
• Use a top-down prioritization method to further restrict manual intervention to only records having a significant impact
The effect on data accuracy and potential time (cost) savings could be estimated using Census 2011 data
23-04-21Statistics Canada • Statistique Canada15
Conclusions Many sources of error in statistical processes We looked at four ways to manage accuracy and timeliness in data
collection and manual processing
• Interviewer monitoring
• Responsive Collection Design
• Selective editing
• Top-down prioritization
• Using paradata• Feasibility and effectiveness demonstrated• Can be used separately or together
23-04-21Statistics Canada • Statistique Canada16
Thank-you!
For more information, please contact:
Laurie Reedman
Statistics Canada
23-04-21Statistics Canada • Statistique Canada17