Upload
may-golden
View
214
Download
0
Tags:
Embed Size (px)
Citation preview
1
Economic Directorate’sQuality Audit Program
Steven S. KlementOffice of Statistical Methods and
Research for Economic Programs
Agenda
• Origins of the Quality Audit Program (QAP)• Scope• Current Program• Some Results• Discussion
2
3
Origins of the QAP
Late 1990’s
Beginning of Census Bureau Standards
2001 Office of Management and Budget (OMB) Quality Guidelines
2004 Establishment of Economic Directorate’s quality management program
2006 OMB Standards and Guidelines for Statistical Surveys
Origins of the QAP
• What form should the Quality Management Program take?
• Quality Profile vs. Quality Audit• Developing the self-assessment tool• Administration of the audit process
4
Scope
• There are about 60 programs in the Economic Directorate that are subject to QAP
• Each program will undergo an audit every five years
• QAP must average twelve audits per year to meet this goal
5
Scope (cont.)
• The Quality Audit Program (QAP) looks for compliance with the OMB Standards and Guidelines for Statistical Surveys (Sep 2006)– Does the program’s documentation show
compliance?– Is there a plan to produce quality estimates?
6
Scope (cont.)
• Auditors do not look in depth to see if execution matches the plan– If the plan calls to compute a variance, the auditor
looks to see that there is a variance in the final data product
– QAP does not look at the actual computer program which calculates the number
7
Conduct of the QAP
• Currently run by the Lead Auditor for Economic Programs– New Assistant Lead Auditor position just created
• Overseen by the Research & Methodology (R&M) Assistant Division Chiefs (ADC)
• Census Bureau’s Quality Improvement Staff assists in training
• Volunteer auditors from the Divisions within the Economic Directorate
8
9
TimelineSep Year Prior
Set audit schedule Audit Week Meet face-to-face to resolve issues
Oct Year Prior
Assign auditors Friday Audit wk
Draft final report
9 wks prior Engagement notification sent out
1 wk after Final report due
7-9 wks prior Conduct training 2 months after PAP plan due
4-7 wks prior Program area fills in checklist
8 months after 1st PAP review
4 wks prior Program area presentation
20 months after
2nd PAP review
1-3 wks prior Auditors review checklist
Every year after
Review PAP until all items complete
Scheduling
• A five-year program is put together by the Lead Auditor and reviewed by the Divisions– Includes all programs not already subject to an
outside audit– There are limited exceptions– The Division’s comment on schedule and request
deferments• The Lead Auditor schedules the actual dates
– 3-4 months out plans dates for next year’s audits– 2-3 months out auditors are selected
10
Auditor Selection
• The Divisions within the Econ Directorate submit managers & supervisors names to be auditors
• The Lead Auditor:– Coordinates schedules– Assigns auditors to each audit.
11
Conduct of the Audit
• Engagement Notification sent nine weeks out• Training conducted 7-9 weeks out• Checklist is due 4.5 weeks out• Program area presentation is four weeks out
12
13
Checklist Example3.1 Data
Editing
Rating
(Auditor)
(Auditor) Summary of findings and
recommendations {Initials}
1. Agencies must edit data to mitigate or correct detectable errors.
Condition: (Program Mgr.) – Objective evidence for existing conditions that supports or refutes compliance with the criteria of the standard.{Initials}Concern: (Auditor) – Cause: The factors responsible for the difference between the condition and criteria. Effect: The consequences (benefits / risks) due to the difference between the condition and criteria. {Initials}Recommendation: (Auditor) – Recommendations to 1) bring the program area in compliance with the standard and for 2) alternative approaches to improve program performance. {Initials}Response: (Program Mgr) – The Program Manager’s position regarding the Auditors concerns or recommendations. {Initials}
Conduct of the Audit (cont.)
• Document review is 1-3 weeks prior– It is the first part of the actual audit– Done by auditors alone
• On-site week is when auditors meet with program area– Draft findings are discussed and then finalized– Issues are resolved in groups or one-on-one
14
Final Report
• The final report is authored by the Lead Auditor– Reviewed by the auditors– Program Area is given chance to respond
• After receiving input and completing revisions, report is sent to management
15
Program Action Plan (PAP)
• Conducting an audit is of little value in of itself• The value comes in when improvements are
made• Therefore, the QAP directs that the program
area develop a PAP– The PAP is the plan for corrective action– Due two months after the audit– The auditors comment on the feasibility of the PAP
to address audit concerns
16
Follow Up
• Finally, progress with the PAP is followed by the Lead Auditor
• Once the PAP is approved Lead Auditor:– Checks for progress after six months– Continues to check yearly until PAP is completed
17
18
Some Results
• Nonresponse Bias Study:– It was found that there was a general lack of NR
bias studies in the Directorate– Study required if response rate is below 80%– Planning for study required if response rate is near
80% OR falling towards 80%• Directorate sponsored a pilot NR study• The Directorate created a new team to help
programs conduct these studies
Some Results (cont.)
• Document storage:– Most programs are well-documented– However, the documentation is scattered
• Multiple document storage systems in use• Many programs did not use any system
– For a number of programs, the audit was the first time all documentation was in one location
• The Directorate will roll out a new centralized easy to use system in summer 2010.
19
20
Some Results (cont.)
• Holds people accountable for their documentation– It must be accessible and understandable– It must outline processes actually used– Both help in knowledge transfer over time
• Compliance with standards is brought to the forefront (instead of an afterthought)
21
Key Finding
• Almost everything one program is deficient or needs improvement on…
…another program has a best practice for!• The question: ‘How much is transferable?’• Auditors take good ideas back to their own
programs……and they share their good ideas with the audited program
QAP Going Forward
• Look at ways to spread Best Practices• Identify areas that require corporate solutions• Help identify areas requiring additional or
concentrated resources
22
Questions?
• Thank you for your time
Steven S. KlementLead Auditor for Economic [email protected](301)763-6598
23
URLs
• OMB Standards & Guidelines for Statistical Surveys (Sep 2006)
http://www.whitehouse.gov/omb/inforeg/statpolicy/standards_stat_surveys.pdf• Guidance on Agency Survey and Statistical
Information Collectionshttp://www.whitehouse.gov/omb/inforeg/pmc_survey_guidance_2006.pdf
24