Upload
wilfrid-burns
View
215
Download
0
Tags:
Embed Size (px)
Citation preview
Performance Measure Performance Measure RecommendationsRecommendations
Greg Pierce – Chair, Performance Measures Task Force
Compliance and Certification Committee Meeting
September 8-9, 2010
2TopicsTopics
PMTF Tasks and Status
Performance Measure Recommendations
Performance Measure Trends
Next Steps
3TasksTasks
Establish an open, consistent and repeatable metric development process to identify and evaluate metrics
Develop compliance improvement measures at the highest levels to provide insight in terms of a set of compliance principles
Coordinate and work with Regional Entities, RMWG, OC, PC and CIPC to ensure agreement
Recommend how to benefit from the compliance improvement measures
4StatusStatus
Adopted RMWG’s open metric development process
Reviewed data source on compliance and other key input factors
Proposed a set of performance measures at high levels
Gathered data and validated preliminary trends
Received feedback from CCC, OC/PC, RMWG, Regional Entities and NERC staff since June 2010
The final report is available at http://www.nerc.com/docs/compliance/pmtf/Compliance_Performance_Measure_Recommendations_Final.pdf
5Performance Measure RecommendationsPerformance Measure Recommendations
Self Disclosure Compare self disclosed versus discovered violations
Timeliness of CVI/CIQ Resolutions
Average duration of CVI/CIQ and number of lessons learned issued
Mitigation Plan Average duration for mitigation plan implementation
Violation Risk Risk factor and severity level weighted average based on unmitigated violations
Feedback Timeliness and Effectiveness
Actionable lessons learned guidance from audits, operating experience, violations and disturbances
Feedback to Standards Direct input to standards development, including results-based standards initiative
Audit Results Timeliness Average duration for no-violation audit report completion
Enforcement Results Timeliness
Average duration for audit reports with potential violations
.
6Self DisclosureSelf Disclosure
0
100
200
300
400
500
2007 2008 2009 2007 2008 2009 2007 2008 2009
Self-Report Compliance Audit Self-Certification
Co
un
t
Reported Violation Trend By Discovery Method and Submission Year(June 18, 2007 - December 31, 2009)
OTHER PRC005 CIP001
7Self Disclosure (cont’d)Self Disclosure (cont’d)
2007 2008 2009
Total 575 1121 797
Dismissed 327 441 117
Self Certification 111 154 183
Self Report 137 526 497
0
200
400
600
800
1000
1200
Co
un
tSelf Disclosed Violations - Including Dismissed
(June 18, 2007 - December 31, 2009)
8Timeliness of CVI/CIQTimeliness of CVI/CIQ
0 10 20 30 40
NERC CIQs
NERC CVIs
Regional CIQs
Regional CVIs
Count
Status of NERC/Regional CIQs and CVIs(June 18, 2007 - June 30, 2010)
Open
Closed
CIQ - Compliance Inquiry CVI - Compliance Violation Investigation
9Timeliness of CVI/CIQ (cont’d)Timeliness of CVI/CIQ (cont’d)
0
200
400
600
800
1000
2007 2008 2009 2008 2009 2008 2009 2008 2009
REGIONAL CVI REGIONAL CIQ NERC CVI NERC CIQ
Aver
age
Dur
ation
(Day
s)
Average Duration for Closed CVIs and CIQs(June 18, 2007 - December 31, 2009)
10
Average Duration of Average Duration of Mitigation Plan CompletionMitigation Plan Completion
0
50
100
150
200
250
1Q2008 2Q2008 3Q2008 4Q2008 1Q2009 2Q2009 3Q2009 4Q2009
Ave
rag
e D
ura
tio
n (
Day
s)
Average Duration for Completing Mitigation by Quarter (1Q2008 - 4Q2009)
11Violation Risk IndexViolation Risk Index
1Q 2008 2Q 2008 3Q 2008 4Q 2008 1Q 2009 2Q 2009 3Q 2009 4Q2009 1Q 2010 2Q2010
CIP-007 0.0 0.0 0.1 2.4 2.4 2.4 2.2 3.4 4.0 4.1
FAC-001 4.4 4.4 4.4 4.2 3.6 3.5 3.2 3.0 2.6 2.0
EOP-005 5.4 5.3 4.1 3.9 3.7 1.9 1.8 4.3 3.1 2.0
PER-002 6.5 6.1 5.3 5.1 4.1 4.0 2.4 4.4 4.4 3.4
PRC-005 26.4 25.6 24.6 22.6 20.9 19.5 16.3 16.1 13.5 13.0
Other Standards 57.2 66.2 64.9 68.6 65.8 61.0 61.3 44.2 45.5 46.2
0
20
40
60
80
100
120
Viol
atio
n R
isk
Inde
x
Top 5 High Risk Violation Trend by Quarter (1Q 2008 - 2Q 2010)
12
Timeliness of Lessons-LearnedTimeliness of Lessons-Learned
02468
10121416
0 0 12 14 13 10 4 0 0 0 2 2 1 12 4 0 0 5
2008 2009 2010 2008 2009 2010 2008 2009 2010 2008 2009 2010 2008 2009 2010 2008 2009 2010
Lessons Learned Advisories Recommendations Compliance Analysis Public Notice Compliance Application Notices
Co
un
t
Availability of Lessons Learned and Guidance (June 18, 2007- July 31, 2010)
Special Considerations: The speed by which the lessons learned are completed and issued may also relate to whether a particular lesson is common to the industry or some segment of the industry. The complete data to measure the Timeliness of Lessons Learned is currently not available.
13
Feedback to StandardsFeedback to Standards
Number of SARsNumber of
InterpretationsNumber of
Suggestions
2009 9 20 39
2008 9 7 13
2007 15 15 9
Cou
nts
Direct Input to Standards Development(June 18, 2007 - December 31, 2009)
14
Average Duration of Audit Report Average Duration of Audit Report Completion w/ No ViolationsCompletion w/ No Violations
0
30
60
90
120
150
180
210
240
3Q 4Q 1Q 2Q 3Q 4Q 1Q 2Q 3Q 4Q 1Q 2Q
2007 2008 2009 2010
Avera
ge D
ura
tio
n (
Days)
Year and Quarter
Average Duration for No-Violation Audit Report Completion(3Q2007 - 2Q2010)
15
Average Duration of Audit Report Average Duration of Audit Report Completion w/ Possible ViolationsCompletion w/ Possible Violations
0
100
200
300
400
500
600
700
800
3Q 4Q 1Q 2Q 3Q 4Q
2007 2008
Ave
rage
Dur
atio
n (D
ays)
Year and Quarter
Average Duration for Audit Report Completion with Possible Violations (3Q2007 - 4Q2008)
17Performance Measures Working GroupPerformance Measures Working Group
Continue to develop and improve performance measures in the areas:
• CMEP process efficiency
• Direct feedback to Standards
• Material timeliness
• Reliability enhancements
• Risk-informed guidance
18Scope (cont’d)Scope (cont’d)
Define data collection and reporting guidelines
Publish quarterly website updates on compliance performance measures
Provide relevant prioritized feedback to enhance the performance under the compliance program and standards development programs, as well as enhance the reliability of the bulk power system
Provide an annual report assesses trends and recommendations.
20Recommended Action by CCC Recommended Action by CCC
The PMTF recommends that the Compliance and Certification Committee
• Approve the report “Compliance Performance Measure Recommendations”
• Approve the scope of the Performance Measures Working Group