Upload
jaden-bishop
View
213
Download
0
Tags:
Embed Size (px)
Citation preview
Quality Review of Key Indicators at Statistics Canada
ICES-III, June 2007Claude Julien and Don Royce
Outline
Quality Management at Statistics Canada
Quality review process
Recommendations
Observations on the review process
Conclusion
Quality Management at STC
1997 Quality Audit
Quality Assurance Framework (QAF)
Audit on 4 key surveys
Office of the Auditor-General recommended to examine a wider application of enhanced self-assessment
Quality Assurance Framework (QAF)
Quality definition Fitness-for-use: relevance, accuracy, timeliness, coherence, accessibility, interpretability
Management of each dimensionPolicies, standards, guidelines and practices
Five main systems: users liaison, planning, methods and standards, dissemination and reporting
(Brackstone 1999, Survey Methodology)
Integrated Program Reports (IPR)
Quadrennial report and biennial follow-up
Quality, costs, management of HR and strategic direction
Self-assessment and user survey or consultation
Focus on quality of outputs
Review and feedback from Policy Committee
Quality Review Process
Initiating the review
Some serious errors in the preceding 18 monthsInternational tradeProductivity estimatesConsumer Price Index
Questions from Finance, other key users and mediaPolicy Committee created Steering Committee of DGs to review quality assurance practices
Headed by DG-Methodology and assisted by head of Quality Secretariat
Objectives of the review
Identify “best practices” that should be promoted to other programs
Identify specific weaknesses where action is needed
Programs Reviewed
Monthly programsLabour Force SurveyRetail Trade SurveyManufacturingInternational TradeConsumer Price IndexGDP by industry
Quarterly programsIncome and Expenditure AccountsLabour ProductivityBalance of Payments
Challenges – Types of programs
Labour Force SurveyRetail Trade SurveyManufacturingInternational TradeConsumer Price IndexGDP by industryIncome and Expenditure AccountsLabour ProductivityBalance of Payments
Traditional
Administrative data
Mixture of data
Derived data
Mixture of types
Challenges – Program process
Process = complex series of stepsSet objectives, define population, create frame, select sample, collect data, etc.
Identify a reasonable number of recognizable steps
Focus on one or two
Challenges – Aspects of a step
Design
Implementation
Execution
Evaluation
Challenges – Dimensions of quality
Relevance
Accuracy
Timeliness
Coherence
Interpretability
Accessibility
Mandate of the review
Review the practices during production (execution) to prevent the release of erroneous data (accuracy)Particular focus on the certification and data dissemination of data
Review of Dissemination and Communications Program
Identify the underlying risk factors and their importance.
Review Teams
Identify and “volunteer” 10 reviewers
10 review teams
3 reviewers + member of Steering Committee per team
Lead reviewer
Organization
Policy Committee
Directors General
Assistant Directors
Directors
Mandate
Recruitment Program
Introductory meeting
Program director, lead reviewer and head of QS
Introduce the review process
Introduce the program
Identify main steps of program process
Main steps of program process
Preparation for certification
Data collection
Editing and transformation
Imputation and estimation
Certification
Data release
Post-release
Gather information
Semi-structured questionnaireCover the topics of the mandate
Not to be completed
Interviews with review team and program director and managers
Available documentationMeta-data
Program reports
Assessment
Individual reportsReport by lead reviewerReviewed by Steering CommitteeVetted by program director
One-day debriefing session with all reviewersSummary report
Report by Steering CommitteePresentation to Policy Committee
Taking action
Disposition of recommendations by Policy Committee
Funding for corrective actions
Follow-up in Integrated Program Reports
Recommendations
Main recommendations
Program managers are well aware of the risks factors
The programs are well-positioned to identify the improvements
Some programs would benefit more than others from additional investment
Human resources concerns dominate all other risk factors.
Main recommendations
The existence of a research and analysis capacity separate from the production operation is a key factor in assuring quality.There are numerous “best practices” in all programs that can usefully be shared within the Agency.Statistics Canada should develop a formal Quality Incident Response Plan Proposals to further increase the timeliness of the programs should be regarded with extreme caution Statistics Canada should establish an ongoing program of Quality Assurance Reviews
Observations on the review process
Aspects to bring forward
Clear mandate from Policy Committee to focus the reviewParticipation at the DG level to assure legitimacyStrong cooperation from program directors and managersConcurrent reviews to assure attention and team workKnowledge and experience of reviewersComposition of review teams to assure some standardizationExcellent learning, development and networking opportunity for reviewersOverall positive experience for the reviewersCollection approach and toolsOne-day debriefing session
Aspects to improve
Better assessment and reporting of level of various risk factorsTimeliness
10 reviews in five months (Oct-Mar)Reviewers spent around 15 daysMore interviews than expectedTime to write, review and translate reports
More timely process and disseminationMore time to schedule meetings around production
More standard program reports
Conclusion
The review was developed and carried out very quicklyThe review was a success thanks to all participantsThe review was not thorough; it complemented other key quality management mechanismsMandate to propose an on-going review program
Build on recent reviewCriteria for selecting surveysTool box / Flexible approachPart of learning and development program
Merci
For more information, or
to obtain a French copy
of this presentation,
please contact:
Pour de plus amples
informations ou pour obtenir
une copie en français du
document, veuillez
contacter :
Claude JulienEmail / Courriel: [email protected] number / Téléphone: 613-951-6937