DRAFT Agenda Planning Committee Meeting Highlights and... · 2015-03-11 · Draft Agenda – Planning Committee Meeting March 10-11, 2015 2 c. Update on Physical Security Guidelines

DRAFT Agenda Planning Committee Meeting March 10, 2015 | 1:00-5:00 p.m. (EST) March 11, 2015 | 8:00 a.m. to 12:00 p.m. (EST) Hyatt Regency Jacksonville Riverfront 225 East Coastline Drive Jacksonville, Florida 32202 904-588-1234 Conference Room: TBD Introductions and Chair’s Opening Remarks NERC Antitrust Compliance Guidelines and Public Announcement

Agenda Items

1. Administrative ― Secretary

a. Arrangements

b. Safety Briefing

c. Announcement of Quorum

d. Planning Committee Members

e. Future Meetings

2. Consent Agenda ― Chair, David Weaver

a. December 9-10, 2014 Draft Meeting Minutes

b. March 10-11 , 2015 Meeting Agenda

3. Update on the February 12, 2015 Board of Trustees Meeting, Dave Weaver, PC Chair

4. Committee Business

a. 2014 Probabilistic Assessment– Kevan Jefferies, RAS ProbA Chair Objective: Present results, key findings and request PC endorsement

Presentation: Yes Duration: 15 minutes Background Item: Draft 2014 Probabilistic Assessment Report

Personal Notes:

b. Integration of Variable Generation Task Force (IVGTF) Report Status Update – Noha Abdel-Karim, NERC Staff

Objective: Present overview of the Final IVGTF Summary Report and the transition plan work

NERC requests PC approval to disband the IVGTF

Presentation: Yes Duration: 20 minutes Background Item: IVGTF - Summary and Recommendation of 12 Tasks Report

Personal Notes:

http://www.nerc.com/comm/PC/Related%20Files%202013/PC_Membership_2014-2016.pdf

http://www.nerc.com/comm/PC/Related%20Files%202013/2015%20PC%20Meeting%20Dates.pdf

http://www.nerc.com/comm/PC/Agenda%20Highlights%20and%20Minutes%202013/Draft%20PC%20Meeting%20Minutes%20-%20December%209-10,%202014%20(Atlanta%20GA).pdf

http://www.nerc.com/comm/PC/Reliability%20Assessment%20Subcommittee%20RAS%202013/2014ProbA%20Draft%20Report_PC%20Review_02-24%202015.pdf

http://www.nerc.com/comm/PC/Integration%20of%20Variable%20Generation%20Task%20Force%20I1/IVGTF%20%20Summary%20and%20Recommendation%20Report_Final%20Draft.pdf

http://www.nerc.com/comm/PC/Integration%20of%20Variable%20Generation%20Task%20Force%20I1/IVGTF%20%20Summary%20and%20Recommendation%20Report_Final%20Draft.pdf

Draft Agenda – Planning Committee Meeting March 10-11, 2015 2

c. Update on Physical Security Guidelines CIP-014, Steve Crutchfield, NERC Staff

Objective: Update on Guidance Development as well as current drafting team activities

Presentation: Yes Duration: 15 Minutes Background Item: none

Personal Notes:

d. Essential Reliability Services Task Force (ERSTF) Update – Brian Evans-Mongeon, Co-Chair of ERSTF

Objective: The ERSTF had 4 measures endorsed by the OC and PC in December for framework pilot effort. The task force received data from the entities and analyzed the measures. In addition, the task force also continued evaluating other 5 measures simultaneously. The Framework Report Version 1 was finalized and posted in January 2015. This presentation will review the results obtained from the framework pilot, the status of the remaining 5 measures, and way ahead for the task force.

Presentation: Yes Duration: 30 minutes Background Item: ERSTF Framework Report

Personal Notes:

e. Reliability Assessment Subcommittee (RAS) Update – Layne Brown, RAS Chair Objective: Review of the recent Reliability Assessment Subcommittee efforts. 2015 Assessment Report status

update and schedule. Improved peer review process and enhanced data presentation plans.

Presentation: Yes Duration: 15 minutes Background Item None Personal Notes:

f. 2015 Summer Reliability Assessments – Layne Brown, RAS Chair

Objective: The objective is to present the PC with updates for the 2015 Summer Reliability Assessment, including a review of proposed enhancements to data collection and presentation.

Presentation: Yes Duration: 15 minutes Background Item: None

Personal Notes:

g. EPA’s Proposed Clean Power Plan – Phase 1 Update – Noha Abdel, Thomas Coleman, Amir Najafzadeh, NERC Staff

Objective: Overview and a status update of Phase I report

Resource Adequacy Study – Status update on timeline and preliminary results

Transmission Adequacy Study: Overview of study objectives, assumptions and study output

Presentation: Yes Duration: 45 minutes Background Item:

December 18, 2014 – PC EPA Steering Committee meeting minutes*

Jan. 9, 2015 - PC EPA Steering Committee meeting minutes*

Personal Notes:

h. ERO Enterprise Data Use and Administration – Howard Gugel, NERC Staff

Objective: Discuss the ERO’s need for data from a strategic and tactical perspective. At its December 2014 meeting, the OC discussed the volume of data being requested from reliability entities to support various ERO projects.

Presentation: Yes Duration: 15 Minutes Background Item: none

Personal Notes:

i. Department of Energy Research and Development Program Status Report – Dr. Emmanuel Taylor, U.S. Department of Energy

Objective: Review and discuss the DOE’s goals and objectives of its Research and Development program regarding planning reliability and to investigate emergent issues that impact the reliability of the BES.

Presentation: Yes Duration: 20 minutes Background Item: none

Personal Notes:

http://www.nerc.com/comm/Other/essntlrlbltysrvcstskfrcDL/ERSTF%20-%20Framework%20for%20Measures%20Report%20January%202015%20-%20Final.pdf


j. Performance Analysis Subcommittee (PAS) Update – Melinda Montgomery, PAS Chair

Objective: Endorse a revision to ALR6-15 (Element Availability Percentage) and a white paper that contains recommendations for compliance metrics ALR CP-1 and ALR CP-2. Provide update on State of Reliability Report and request reviewers.

Presentation: yes Duration: 30 minutes Background Items

Revised ALR6-15 (Element Availability Percentage and Unavailability Percentage)*

Draft Compliance Metric White Paper*

Compliance Metrics Consideration of Comments*

Draft ALR CP-1*

Draft ALR CP-2*

Presentation: Yes Duration: 15 Minutes Background Items: none

Personal Notes:

k. AC Substation Equipment Task Force (ACSETF) - Michael Lombardi, ACSETF Chair

Objective: To provide status update as directed by PC to prioritize recommendations and make them actionable.

The AC Substation Equipment Task Force (ACSETF) was formed to analyze one of NERC’s top priority reliability issues – AC Substation Equipment Failures. As reported in the NERC 2013 State of Reliability Report, AC substation equipment failures have been observed to be a significant contributor to disturbance events, and have a positive correlation to increased transmission severity for outages associated with them. The task force developed a report for the industry which summarizes trends in disturbance events resulting from AC substation equipment failure, identification of root and contributing causes and recommendations for actions which was presented at the December 2014 Planning Committee and Operating Committee meetings. The PC endorsed the ACSETF final report, subject to the ACSETF refining the recommendations into actionable recommendations with a business plan to help prioritize implementation of each recommendation.

Presentation: Yes Duration: 15 Minutes Background Items: none

Personal Notes:

l. NERC FAC-003 Vegetation Management Research Work with EPRI - Neil Burbure, NERC Staff

Objective: To provide an update on:

1. Preliminary results from research work conducted by NERC and EPRI to empirically validate the gap factor used in the calculation of the Minimum Vegetation Clearance Distances specified in the currently enforceable version of the NERC vegetation management Reliability Standard, FAC-003-3, which is designed to avoid flashovers between conductors and vegetation. The work is being completed to fulfill the directive in paragraph 59 of Revisions to Reliability Standard for Transmission Vegetation Management, Order No. 777, 142 FERC ¶ 61,208 (2013).

2. Future steps

Presentation: Yes Duration: 20 Minutes Background Item: None

Personal Notes:

m. Transmission and Frequency Performance with High Levels of Renewables – Nick Miller, General Electric

Objective: Nick Miller of General Electric, will present an overview of the key findings of the recently-completed National Renewable Energy Laboratory (NREL) study Transient Stability and Frequency Response of the US Western Interconnection under conditions of High Wind and Solar Generation. The study results show the necessity of carefully engineering the integration of renewables in order to maintain transmission system voltage and stability performance. The study also provides insights to areas of analysis that must be pursued for successful integration. Key findings include:


1. Ability of WI to meet NERC FRO under high (>30% annual energy) from wind and solar

2. Efficacy of alternatives to synchronous generation (wind, solar, storage, demand based controls) to mitigate FR concerns

3. Impact on large-scale (e.g., California-Oregon Interface) and regional (e.g., eastern Wyoming/Colorado) transient stability

4. Stability impacts of high and very high levels of displacement of thermal (especially coal) generation

Presentation: Yes Duration: 20 Minutes Background Item: None

Personal Notes:

5. Planning Committee and Subcommittees Project Queue Review

a. Planning Committee Work Plan

*Background materials included. ** Background materials will be provided prior to the meeting.

http://www.nerc.com/comm/PC/Related%20Files%202013/PC%20Work%20Plan.pdf

Antitrust Compliance Guidelines I. General It is NERC’s policy and practice to obey the antitrust laws and to avoid all conduct that unreasonably restrains competition. This policy requires the avoidance of any conduct that violates, or that might appear to violate, the antitrust laws. Among other things, the antitrust laws forbid any agreement between or among competitors regarding prices, availability of service, product design, terms of sale, division of markets, allocation of customers or any other activity that unreasonably restrains competition. It is the responsibility of every NERC participant and employee who may in any way affect NERC’s compliance with the antitrust laws to carry out this commitment. Antitrust laws are complex and subject to court interpretation that can vary over time and from one court to another. The purpose of these guidelines is to alert NERC participants and employees to potential antitrust problems and to set forth policies to be followed with respect to activities that may involve antitrust considerations. In some instances, the NERC policy contained in these guidelines is stricter than the applicable antitrust laws. Any NERC participant or employee who is uncertain about the legal ramifications of a particular course of conduct or who has doubts or concerns about whether NERC’s antitrust compliance policy is implicated in any situation should consult NERC’s General Counsel immediately. II. Prohibited Activities Participants in NERC activities (including those of its committees and subgroups) should refrain from the following when acting in their capacity as participants in NERC activities (e.g., at NERC meetings, conference calls and in informal discussions):

• Discussions involving pricing information, especially margin (profit) and internal cost information and participants’ expectations as to their future prices or internal costs.

• Discussions of a participant’s marketing strategies.

• Discussions regarding how customers and geographical areas are to be divided among competitors.

• Discussions concerning the exclusion of competitors from markets.

• Discussions concerning boycotting or group refusals to deal with competitors, vendors or suppliers.

NERC Antitrust Compliance Guidelines 2

• Any other matters that do not clearly fall within these guidelines should be reviewed with NERC’s General Counsel before being discussed.

III. Activities That Are Permitted From time to time decisions or actions of NERC (including those of its committees and subgroups) may have a negative impact on particular entities and thus in that sense adversely impact competition. Decisions and actions by NERC (including its committees and subgroups) should only be undertaken for the purpose of promoting and maintaining the reliability and adequacy of the bulk power system. If you do not have a legitimate purpose consistent with this objective for discussing a matter, please refrain from discussing the matter during NERC meetings and in other NERC-related communications. You should also ensure that NERC procedures, including those set forth in NERC’s Certificate of Incorporation, Bylaws, and Rules of Procedure are followed in conducting NERC business. In addition, all discussions in NERC meetings and other NERC-related communications should be within the scope of the mandate for or assignment to the particular NERC committee or subgroup, as well as within the scope of the published agenda for the meeting. No decisions should be made nor any actions taken in NERC activities for the purpose of giving an industry participant or group of participants a competitive advantage over other participants. In particular, decisions with respect to setting, revising, or assessing compliance with NERC reliability standards should not be influenced by anti-competitive motivations. Subject to the foregoing restrictions, participants in NERC activities may discuss:

• Reliability matters relating to the bulk power system, including operation and planning matters such as establishing or revising reliability standards, special operating procedures, operating transfer capabilities, and plans for new facilities.

• Matters relating to the impact of reliability standards for the bulk power system on electricity markets, and the impact of electricity market operations on the reliability of the bulk power system.

• Proposed filings or other communications with state or federal regulatory authorities or other governmental entities.

Matters relating to the internal governance, management and operation of NERC, such as nominations for vacant committee positions, budgeting and assessments, and employment matters; and procedural matters such as planning and scheduling meetings.

DRAFT Minutes NERC PC EPA Steering Committee Meeting January 9, 2015 | 10:00 a.m. - 12:00 p.m. (EST) Conference RM #740 Dial-in: (866) 740-1260 | Access Code: 3250202 | Security Code: 0202 Webinar: http://www.readytalk.com/?ac=3250202 NERC Antitrust Compliance Guidelines and Public Announcement The NERC staff Ganesh Velummylum reviewed the Antitrust and Public Announcement policies of NERC.

Meeting Minutes The Steering Committee approved the December 18, 2014 meeting minutes.

Agenda Items

1. Phase 1 Assessment Study Scope Review ‐ Ganesh Velummylum, NERC Staff Ganesh reviewed the Phase 1 study outline consisting of three parts after which there was a robust discussion on the Phase 1 study outline document. Comments from Andrew Tudor and Herb Schrayshuen were incorporated into the document. Herb Schrayshuen asked the question of whether the Planning Committee or the Steering Committee will be doing the review of Part 3 of the phase 1 study which will be a summary of existing studies completed by industry participants. John Moura, NERC staff responded that NERC staff will be doing the review and will subsequently forward the write‐up to the Steering Committee for their input. John Moura stated that it would be helpful to NERC staff if stakeholders can highlight key areas in the report that would advance this endeavor.

2. Assessment of Gas Prices for NERC Scenario Analysis – Energy Ventures Analysis, Inc. (EVA) Steve Thumb from EVA gave the presentation on NERC gas price scenario analysis for the CPP. EVA’s presentation talked about three gas price scenarios, six NERC scenarios, assessment of high gas price scenario, background information on gas price forecast and a simplified assessment of NYMEX future prices. EVA made the recommendation to use EIA’s AEO 2014 High Case Price for the high gas case price forecast and at the same time recommended using EIA’s AEO 2014 gas price forecast for the reference case. According to EVA, EIA AEO 2015 has not yet been published, however early release information indicates AEO 2014 and AEO 2015 gas prices are very similar and is a reasonable proxy for a reference case.

Agenda – NERC EPA Steering Committee – 1/9/2015 2

3. Resource Adequacy Study Discussion – Ganesh Velummylum, NERC Staff Ganesh Velummylum reviewed the updated input assumption file based on comments received from the steering committee by COB 01/07/2015. Major input assumptions were reviewed with the stakeholders and changes were made particularly with respect to the MW size of new units to be modeled based on fuel type, new generation capital technology cost, the fixed O& M cost, non‐fuel variable O&M cost and heat rate based on comments received from Municipal Energy Agency of Nebraska, SPP, NextERA, Exelon and ERCOT. NextERA had requested clarification on how maximum allowable wind and solar capacity was modeled in the study. EVA believes a large amount of new renewable capacity will be constructed throughout the country. Therefore, EVA forces renewable capacity into its dispatch model by employing an internally developed state‐by‐state forecast of renewable capacity deployment that not only takes into account each state’s renewable portfolio standards (RPS), but also considers each individual state’s economically reasonable renewable resource limitations and the cost effectiveness of each type of renewable capacity. Andrew Tudor of Municipal Energy Agency of Nebraska raised the question of where did the five percent forced outage rate for coal plants come from? EVA assumes each coal plant will have an unforeseen outage of 1.5 days per month. There was also a robust discussion on gas prices to be used for the scenario. It was recommended by the steering committee that we also do some sensitivity analysis on low gas prices. Kathleen Robertson and John Hutchinson both from Exelon Corporation, recommended using the NYMEX gas price and futures for the scenario analysis. It was agreed by the steering committee after a lengthy discussion that we would run nine scenarios using EIA’s AEO 2014 Low case price, 2014 EIA AEO long term outlook gas prices (5/2014) and EIA’s AEO 2014 High Case Price for each of the following cases (reference base case with no CPP, CPP state implementation, CPP regional trading).

Table 1: Summary of Proposed Clean Power Plan Cases

Modeling Case

Reference Case

(Baseline without CPP)

EPA Case

Reference Case


NERC Case

High‐ Gas Price

Scenario

EPA Case

NERC Case High‐ Gas Price

Scenario

Reference Case


NERC Case Low‐ Gas Price

Scenario

NERC Case Low‐ Gas Price

Scenario

Implementation Assumptions

Rate‐to‐Mass Translations

N/A Yes N/A Yes Yes Yes N/A Yes Yes

Gas Price+++ EIA‐AEO 2014

EIA‐AEO 2014

EIA –High EIA–High EPA EIA –High EIA –Low EIA –Low EIA –Low

State/Regional

N/A State N/A State Regional Regional N/A State Regional

Expected Delivery End of January 2015 2nd Week of February

2015 End of February 2015

1st Week of March 2015

1st Week of March 2015

1st Week of March

2015

EVA presented the regional boundaries that would be used in the models for the regional trading case. Jay Caspary, SPP, suggested that South and North Dakota be kept as a separate region in the regional model. After a lengthy discussion, it was unanimously agreed by the steering committee


members to accept the regional boundaries presented by EVA for the regional trading case as follows:

WECC

SPP

TEXAS

MISO + PJM

RGGI

SERC + FLORIDA

4. Reliability First Corporation(RFC) Spreadsheet Analysis Feedback‐ Paul Kure, RFC Staff Paul Kure, RFC gave a quick overview of RFC’s preliminary spreadsheet analysis of the CPP. He also kindly requested feedback from the steering committee on a set of questions about the section 111(d) emissions rates and calculations at their earliest convenient that was forwarded to members.

5. Action Items a. The EPA Steering Committee members will give their final feedback and comments on the

input assumption files by COB 01/12/2015. Action item completed. b. The EPA Steering Committee will forward to NERC any studies that have been completed (or

studies with approved scopes that are underway) to their knowledge on the proposed Clean Power Plan.

c. Future Discussion Needed: NERC is requesting additional information on timelines (lead times)

associated with generators and transmission new‐builds and upgrades for each voltage class (230kV, 345kV, 500kV and 765kV) as described in part 2a of the NERC EPA Phase 1 Study outline document.


Meeting Attendees: Herb Schrayshuen – Small User Sector Gary Brinkworth ‐ TVA Jay Caspary ‐ SPP Mark Ahlstrom ‐ WindLogics, NextEra Energy Kathleen Robertson ‐ Exelon Corporation Paul McCurley ‐ NRECA Andrew Wade Tudor ‐ Municipal Energy Agency of Nebraska John Hughes ‐ Electricity Consumers Resource Council (ELCON) John Hutchinson ‐ Exelon Corporation Vijay Satyal ‐ WECC Brian Evans‐Mongeon ‐ Utility Service Inc. Dave Weaver ‐ Exelon Corporation Lewis De La Rosa ‐ ERCOT Dana Lazarus ‐ ERCOT

NERC Staff Mark Lauby Tom Burgess John Moura Tom Coleman

Ganesh Velummylum Noha Abdel Karim

EVA Staff

Tom Hewson Principal, Project Manager for NERC effort, Steve Thumb Principal‐ Directs EVA Gas Practice Wes Mitchell Senior Consultant‐ Gas Team Phillip Graeter Analyst‐ Environmental and Power Team Rob Jennings Analyst‐ Power Team Aqeel Adenwala Analyst‐ Power Team

Meeting Minutes NERC PC EPA Steering Committee December 18, 2014 | 10:00 a.m. - 12:00 p.m. ET Conference RM #740 Dial-in: (866) 740-1260 | Access Code: 3250202 | Security Code: 0202 Webinar: http://www.readytalk.com/?ac=3250202 NERC Antitrust Compliance Guidelines and Public Announcement The NERC staff Ganesh Velummylum reviewed the Antitrust and Public Announcement policies of NERC.

Agenda Items

1. Phase 1 Assessment Study Scope Review ‐ Ganesh Velummylum, NERC Staff Ganesh reviewed the Phase 1 study outline consisting of three parts. There was a robust discussion on the input assumptions and the output assumptions. NERC plans to send out the input assumptions used in the model to the Steering Committee for their review as soon as those assumptions are finalized and is requesting that the committee respond back by 01/07/2015. For the output assumptions, it is recommended we remove average state retail prices and natural gas power demand and commodity price projections

2. Resource Adequacy Study Discussion – Noha Abdel Karim, NERC Staff Noha Abdel Karim reviewed part 1 of the Phase1 study which is the resource adequacy analysis.

Major input assumptions were reviewed with the stakeholders ‐ primarily the replacement generation location and fuel source of the replacement generation. It was agreed that the replacement generation would be modeled in the region where it is needed since the model is zonal based rather than nodal based. As far as fuel type for the replacement generation is concerned, it was agreed that the model will dispatch the next available lowest cost unit to meet demand and it will be technology neutral.

The load growth assumption was based on states’ Gross Domestic Product (GDP), population growth and climate impact etc. More details are available in the input assumption document for load growth assumption used in the models. After a lengthy discussion, the group decided on the following scenarios:


Table 1: Summary of Proposed Clean Power Plan Cases

Modeling Case

Reference Case

(Baseline+ without CPP)

EPA Case

Reference Case

(Baseline+ without CPP)

NERC Case High- Gas

Price Scenario++

EPA Case

NERC Case High- Gas

Price Scenario++

Implementation Assumptions

Mass/Rate Base

N/A Rate-to-Mass Translations

N/A Rate-to-Mass Translations

Rate-to-Mass Translations

Rate-to-Mass Translations

Gas Price+++ EPA EPA High High EPA High

State/Regional

N/A State N/A State Regional Regional

Expected Delivery 2nd Week of January 2015 End of January 2015 2nd Week of February 2015

+ Baseline assumptions will use reference case actual and projected data. ++ For scenario cases above, sensitivity runs on natural gas price will use higher than expected actual and projected gas price. +++ Gas price supply model used in CPP reference and baseline cases is a calibrated EPA supply curve consistent with EPA’s Integrated Planning Model published in June, 2014.

3. Transmission Adequacy Study ‐ Amir Najafzadeh, NERC Staff Amir Najafzadeh reviewed part 2 of the Phase 1 study which is the transmission adequacy analysis. The contingency analysis will be on Extra High Voltage (EHV) facilities. The analysis would be an Alternating Current (AC) load flow analysis to identify thermal overload on a zonal level and also reactive power deficiency. The output would be transmission constraints on a zonal level.

The steering committee requested more time to review the transmission adequacy study before they provide additional feedback. This topic will be discussed in more detail at the next steering committee meeting in January, 2015.

4. Review of Existing Studies – Group Discussion Ganesh mentioned that in many areas, such as larger ISO/RTOs and larger utilities, a large amount of work has already been completed to highlight and demonstrate reliability challenges associated with the proposed CPP. Ganesh requested stakeholders to provide any studies that have been completed to their knowledge to be forwarded to NERC. John Moura stated that it would be helpful to NERC staff if stakeholders can highlight key areas in the report that would advance this endeavor. NERC will evaluate the assessment of industry studies and write a regional level summary and conclusion of the industry assessment to identify key issues and timelines associated with the impacts.


5. Action Items

a. NERC will forward the model input assumptions used in the Aurora model to the Steering Committee as soon as those input assumptions are finalized and the Steering Committee will respond back to NERC with comments by COB 01/07/2015.

b. The Steering Committee will forward to NERC any studies that have been completed (or studies with approved scopes that are underway) to their knowledge on the proposed Clean Power Plan.

c. Future Discussion Needed: NERC is requesting additional information on timelines (lead times) associated with generators and transmission new‐builds and upgrades for each voltage class (230kV, 345kV, 500kV and 765kV) as described in part 2a of the NERC EPA Phase 1 Study outline document.

Meeting Attendees: Herb Schrayshuen – Small User Sector Gary Brinkworth ‐ TVA Jay Caspary ‐ SPP Mark Ahlstrom ‐ WindLogics, NextEra Energy Kathleen Robertson ‐ Exelon Corporation Chuck Chakravarthi ‐ Southern Company Andrew Wade Tudor ‐ Municipal Energy Agency of Nebraska John Hughes ‐ Electricity Consumers Resource Council (ELCON) John Hutchinson ‐ Exelon Corporation Stan Holland ‐ WECC Brian Evans‐Mongeon ‐ Utility Service Inc. Dana Lazarus ‐ ERCOT

NERC Staff John Moura

Tom Coleman Ganesh Velummylum Amir Najafzadeh Elliott Nethercutt Noha Abdel Karim

1 The APC is defined on page 20 of the 2009 TADS Phase II Final Report, available at http://www.nerc.com/docs/pc/tadstf/TADS_Phase_II_Final_Report_091108.pdf.

M-16ALR6-15 Element Availability Percentage (APC) & Unavailability Percentage

Metric Number ALR6-15M-16

Submittal Date April 01, 2015

Sponsor Group (OC, PC or subgroup name)

SERC Reliability Corporation, revised by PAS

Short Title

Metric Description Part A: Availability (APC) - Overall percent of Bulk Electric System AC Transmission Elements operated at 200kV or above transmission system elements (i.e., AC lines and transformers operated at 200 kV and above) that isare available for service as influenced by outage durations from both Automatic and Non-Automatic events. Momentary outages are not considered in this metric. Part B: Unavailability - This metric also includes the overall percent of Bulk Electric System AC Transmission Elements operated at 200kV or above transmission system elements (i.e., AC lines and Transformers 200 kV and above) that isare unavailable for service (i.e., out of service) due to Sustained Automatic and Non-Automatic Outages. These outages will be broken down into Automatic (sustained) and Non-automatic (planned and operational) outages. Momentary outages are not considered in this metric.

Purpose To determine the percentage of Bulk Electric System AC Transmission Elements operated at 200kV or above transmission system Elements operated at 200 kV and above that arethat is available and unavailable when outages due to automatic and non-automatic events are considered.

How will it be suited to indicate performance?

The overall availability is the percentage of of Bulk Electric System AC Transmission Elements operated at 200kV or above transmission system elements that isare available (i.e., in service) for the transmission of electricity. The overall unavailability is the percentage of Bulk Electric System AC Transmission Elements operated at 200kV or above transmission system elements that isare not available (i.e., not in service) for the transmission of electricity. The relative percentage provides an indication of the overall availability, or unavailability, of the transmission system operated at 200 kV and abovethe Bulk Electric System. The various Elements, AC Circuits and Transformers, will be calculated and plotted separately.

Formula Part A: The percent of Bulk Electric System AC Transmission Elements operated at 200kV or above transmission system Elements (AC circuits and transformers) operated at 200 kV and above that isare available due to Sustained Automatic and Non-Automatic outages, is calculated as follows: Total Sustained Element Outage Hours APC (in %) = (1 – ------------------------------------------------------) X 100 Total Element Hours where, The APC, the Total Sustained Element Outage Hours and the Total Element Hours are defined and calculated in the TADS report.1 Part B: The percent of Bulk Electric System AC Transmission Elements operated at 200kV or above transmission system Elements (AC circuits and transformers) operated at 200 kV and above that isare unavailable due to Sustained Automatic and Non-Automatic outages, is calculated as follows: Unavailability (in %) = 100 – APC

1 The APC is defined on page 20 of the 2009 TADS Phase II Final Report, available at http://www.nerc.com/docs/pc/tadstf/TADS_Phase_II_Final_Report_091108.pdf.

Metric Start Time or Baseline

Year 2010, the first year of TADS data collection that includes Non-automatic outages

Time Horizon Historical perspective

Data Collection Interval and Roll Up

Data collection is through the NERC TADS procedure. Metric calculation is one value for each Interconnection (Eastern, Western, ERCOT, and Québec) for the aggregate of facilities 200 kV and above. The metric would will be reported on the same interval as TADS reports.

Ease of Collection The TADS database makes this metric easily reportable on a uniform basis.

Aggregation Reported on an aggregate basis by Regional Entity, Interconnection (Eastern, Western, Texas, and Québec) and NERC.

Linkage to NERC Standard

None

Linkage to Data Source

NERC TADS data base http://www.nerc.com/docs/pc/tadswg/Data_Reporting_Instr_Manual_09-29-09.pdf

Need for Validation or Pilot

No, the data and results will be reported via the TADS process when it becomes available. [Note: The former ECAR, MAIN, and MAPP regions had collected and reported similar data and statistics in the past and could be used for reference.]

Data Submitting Entity

Transmission Owners via TADS procedures.

Smart Rating Total Score

Specific/Simple Measurable Attainable Relevant Tangible/Timely

13 3 3 3 2 2

Style (look and feel)

Bar charts, with possible trend lines added in the future

Publications and Documentation (e.g., section of LTRA)

The statistics needed to compute this ALR metric are currently shown in the TADS reports. This metric may be included in the annual NERC LTRA report, at the discretion of the NERC Planning Committee.

ERO Compliance Metrics, Risk, and Reliability NERC CCC and RISC Recommendations to the NERC Board of Trustees

NERC | ERO Compliance Metrics, Risk, and Reliability| December 2014 i

Table of Contents

Table of Contents ....................................................................................................................................................... ii

Preface ....................................................................................................................................................................... iii

Executive Summary ....................................................................................................................................................4

Chapter 1 – Previous Global Metrics Efforts and Limitations ....................................................................................5

Background .............................................................................................................................................................5

Chapter 2 – Global Metrics .........................................................................................................................................7

Introduction ............................................................................................................................................................7

Caveats ....................................................................................................................................................................7

ALR CP-1 (Revised)-Risk Metric ...............................................................................................................................7

ALR CP-2 Impact Metric ..........................................................................................................................................9

Chapter 4 – Recommendations ............................................................................................................................... 14

NERC and Regions ................................................................................................................................................ 14

Registered Entities ............................................................................................................................................... 14

Appendix 1 – Leveraging Existing Compliance Metrics ........................................................................................... 15

Appendix 2 – Team Members ................................................................................................................................. 16

NERC | ERO Compliance Metrics, Risk, and Reliability| December 2014 ii

Preface The North American Electric Reliability Corporation (NERC) is a not-for-profit international regulatory authority whose mission is to ensure the reliability of the bulk power system (BPS) in North America. NERC develops and enforces Reliability Standards; annually assesses seasonal and long-term reliability; monitors the BPS through system awareness; and educates, trains, and certifies industry personnel. NERC’s area of responsibility spans the continental United States, Canada, and the northern portion of Baja California, Mexico. NERC is the electric reliability organization (ERO) for North America, subject to oversight by the Federal Energy Regulatory Commission (FERC) and governmental authorities in Canada. NERC’s jurisdiction includes users, owners, and operators of the BPS, which serves more than 334 million people. The North American BPS is divided into several assessment areas within the eight Regional Entity (RE) boundaries, as shown in the map and corresponding table below.

The Reliability Issues Steering Committee (RISC) is an advisory committee that triages and provides front-end, high-level leadership for issues of strategic importance to the BPS reliability and offers high-level stakeholder leadership engagement and input on issues that impact bulk power system reliability. The RISC advises the NERC Board of Trustees (Board), NERC standing committees, NERC staff, regulators, Regional Entities, and industry stakeholders to establish a common understanding of the scope, priority, and goals for the development of solutions to address these issues, including the use of solutions other than the development of new or revised reliability standards. In doing so, the RISC provides a framework for steering, developing, formalizing, and organizing recommendations to help NERC and the industry effectively focus their resources on the critical issues needed to best improve the reliability of the BPS. This report documents the result of the RISC’s continued work to define risks to the reliable operation of the BPS and provide guidance to the Board on activities that NERC should take to manage those risks.

FRCC Florida Reliability Coordinating Council

MRO Midwest Reliability Organization NPCC Northeast Power Coordinating

Council RF ReliabilityFirst SERC SERC Reliability Corporation SPP-RE Southwest Power Pool Regional

Entity TRE Texas Reliability Entity WECC Western Electricity Coordinating

Council

NERC | ERO Compliance Metrics, Risk, and Reliability| December 2014 iii

Executive Summary This report outlines the work performed by the NERC Compliance and Certification Committee (CCC) and others based on a request by the RISC in June 2014 to identify ways compliance data can inform actions to reduce the risk to the bulk electric system (BES). This is based on a NERC Board request of the RISC in February 2013;

FURTHER RESOLVED, the Board hereby directs NERC management to work with the RISC and, as appropriate, NERC committee leadership to consider how NERC should utilize a data-driven reliability strategy development process that integrates with budget development and overall ERO planning (e.g., Standing Committee planning, department, and employee goal-setting).

The team formed to support the RISC request of the CCC focused on the following objectives: • Identify one or two high level global metrics to track BES risk due to compliance violations. • Utilize these global metrics as well as currently available granular compliance metrics to reduce risk by:

o Identifying with some confidence which requirements pose greater risk to reliability, o Encouraging timely mitigation of Potential Violations (PV) and sharing of lessons-learned (mitigation

and controls), o Fostering a culture of self-inspection, self-correction, and self-reporting, o Modeling after other “lessons learned” focused industries, such as those whose foundation relies on

safety triage, and o Tailoring compliance monitoring and enforcement effort (both Industry and NERC) commensurate

with the risk of the infraction.

The team developed two global metrics; ALR-CP1 (Risk Focus), and ALR-CP2 (Impact Focus). These metrics should offer significant value in achieving the objectives above. Additionally, the team recommends a set of subordinate metrics and a dashboard to summarize the state and direction of the ERO Enterprise compliance monitoring and enforcement impact on reliability. Detailed recommendations are found in Chapter 4.

The metrics, in conjunction with the self-correction model presented in the report, should prove effective in finding and fixing small issues before they lead to larger reliability events.

NERC | ERO Reliability Risk Priorities | January 2015 4

Chapter 1 – Previous Global Metrics Efforts and Limitations

Chapter 1 – Previous Global Metrics Efforts and Limitations Background From 2010-2012, there were two prior attempts at creating global compliance metrics focused on compliance violations and the associated risk to reliability. Both of the metrics had some value and were a useful start, but each had closely-related limitations. The work was initiated in response to a vision expressed that showed the intersection of compliance, performance, and reliability. This is illustrated in Figure 1 below.

Figure 1 Conceptual Diagram of Risk Indices

The first proposal was an integrated metric to serve as a single performance measure on the state of the bulk power system. The Key Compliance Monitoring Index (KCMI) was the outcome metric from this effort. This was followed by a short-lived replacement metric, ALR CP-1. Key Compliance Monitoring Index (KCMI) The first attempt at a global compliance metric was KCMI (originally named SDI, or Standards Driven Index). This metric was based on a set of 26 high Violation Risk Factor (VRF) requirements that were deemed by Enforcement Staff to be most important to reliability. The set of requirements was selected by a team of subject matter experts and can be thought of as a “Dow Jones” average of important requirements. To measure the ongoing risk to the BES, KCMI tracked the number of unmitigated KCMI-requirement violations in the ERO’s queue. The limitations of KCMI were: • The standards that were selected as “bellwether” observations to track were largely the result of evaluating

recent compliance records, thus they had much to do with what standard requirements appeared to be either readily detected or more difficult to demonstrate compliance;

NERC | ERO Compliance Metrics, Risk, and Reliability | February 2015 5

Chapter 1 – Previous Global Metrics Efforts and Limitations

• There was no feedback mechanism defined to retire less impactful requirements and replace them with new more consequential bellwether requirements;

• The KCMI set did not include CIP requirements, which raised questions of a potential shortcoming; • By tracking unmitigated violations, the new observations in a recent quarter are masked by those already in

queue. The metric was overstated by old requirements that were nearly fully-mitigated, but had not progressed through the administrative process of closing out the mitigation plan. In short, the metric was inflated, was not sensitive to recent changes in recent violations, and did not vary much quarter to quarter due to administrative delay.

ALR CP-1 (Original) The second metric previously tested was called ALR CP-1. This metric was proposed but never accepted by the Planning Committee or Operating Committee for use in evaluating compliance impact on reliability by Registered Entities. The name was intended to conform to other NERC Adequate Level of Reliability (ALR) metrics. Rather than selecting a set of defined impactful requirements, this metric relied on the expert judgment of Enforcement Staff. As a violation progresses through Enforcement, it is assigned situation-specific risk based on the facts and circumstances of the case. The most egregious violations are deemed “Serious Risk” violations. Somewhat similar to KCMI, ALR CP-1 tracked unmitigated Serious Risk violations (as deemed by Enforcement) by quarter. There were limitations, some similar to the limitations of KCMI: • Including unmitigated violations inflated the metric due to administrative lag and also reduced the visibility

of near term changes in the number of violations. • As it can take years to close out a Serious Risk violation, the approach to assess risk in the past may not

reflect the current situation of the BES. • While relying on the judgment of Enforcement Staff was a useful start, the metric did not have a feedback

loop. A review of the data showed significant differences between Regions on which standards had Serious Risk violations. While there were likely legitimate reasons for the differences, there was no mechanism to compare approaches and refine the process.

• Compared to KCMI, there was also an additional administrative lag in the original ALR CP-1. KCMI was a straightforward derivation of a requirement’s Violation Risk Factor (VRF) and Violation Severity Level (VSL). These two values are immediately known when a violation is identified. With ALR CP-1, the final risk value assessed by Enforcement isn’t deemed official for many months, if not years.


Chapter 2 – Global Metrics

Chapter 2 – Global Metrics Introduction The team recommends the development of two global compliance metrics, focusing on risk to and impact upon the BES. The data to support these metrics would require the support of the Regional Entities and reporting entities, but would be relatively simple to gather. Caveats To avoid incorrect conclusions when reading about the suggested metrics, it is important to keep these caveats in mind: • Very few system events or disturbances are caused by compliance violations. • Even though a potential violation has no impact on the BES, it does not necessarily mean that it was not a

Serious Risk Violation. • A potential violation that causes a state change to the BES does not necessarily translate to a Serious Risk

violation. ALR CP-1 (Revised)-Risk Metric Definition ALR CP-1 is a quarterly count of newly reported potential violations initially determined by Enforcement Staff at the regions to be a Serious or likely Serious Risk Violation. Description The team recommends using some of the concepts of the previously discussed ALR CP-1 with two modifications. 1. Count Violations in the quarter they were detected As the bulk of mitigation occurs shortly after a violation occurs and nearly all mitigation is complete well before a mitigation plan is formally closed, the team recommends tracking violations in the quarter they are detected. This focuses the new metric more on newly discovered and reported potential violations. Additionally, as further discussed later in this document, there are some aggregate metrics that NERC already calculates that track the time elapsed until violations are fully mitigated, so the duration of mitigation aspect of compliance tracking will still be captured and tracked going forward. 2. Include “likely serious” potential violations as determined by Enforcement Staff as well as confirmed

serious violations There is one added modification required to the metric. In many cases, the evaluation of the risk imposed by a potential violation takes a significant amount of time, perhaps months. Therefore, a potential violation isn’t officially deemed serious until months after its occurrence. Still, regional Enforcement Staff does an initial triage of potential violations in order to determine the resources that will be committed to the review. The team proposes to track Serious Risk Violations and “likely Serious” Risk Potential Violations (as triaged by Enforcement) by quarter. Including “likely Serious” violations in the measure will result in some slight modifications as Potential Violations (PV) progress through the queue. As long as the two types of observations (confirmed Serious Risk, likely Serious Risk) are appropriately designated in the trends and charts, this is expected to pose little variability to the metric results.



Benefits This metric is anticipated to enable multiple benefits: • Including “likely Serious” violations in the metric brings the measure closer to real-time. It is expected the

metric can be posted with confidence with a lag of no more than one quarter. • Differences in the relative proportion of violations and those standards assigned Serious Risk between

Regions may point to o Unique Regional Risks that can then become part of the Focus Areas1 of the Regional CMEPs. o Learning opportunities for Enforcement Staff (some Regions may be overlooking a risk or it is

possible some Regions may be including factors that inflate the perceived risk). • The most commonly violated standard requirements assigned Serious Risk violations are likely starting

points for the development of requirement level internal controls. • Sharing lessons-learned on the Serious Risk Violations (root causes, mitigation steps, internal controls)

should help Registered Entities reduce the number and impact of future violations of these requirements. • Because this metric does not rely solely on a static set of requirements, but can evolve as the standards

change. Initial Observations Figure 2 depicts the trend in Serious Risk Violations with the data available to the team. It does not include recent Likely Serious PVs. As additional data becomes available, the team recommends adding a 12 month rolling average line to better track trends.

Figure 2 Serious Risk and Likely Serious Risk Violations

1 Focus Areas are standards that receive particular emphasis in a Region’s CMEP.



Figure 3 depicts those standards and requirements processed in 2013 and 2014 as Serious Risk violations. The figure is interesting since Critical Infrastructure Protection (CIP) requirements are some of the most often violated requirements as well as the violations most often assigned Serious Risk. The requirements in the figure provide useful information on one starting point for the development of internal controls.

Figure 3 Standards and Requirements with Most Occurrences of Serious Risk Violations

ALR CP-2 Impact Metric Definition ALR CP-2 is a quarterly count of the number of newly reported Compliance Exceptions or Potential Violations with observed impact by Impact Tier level. Description While it is expected the ALR CP-1 (Risk) metric will provide value to the ERO Enterprise, it has two primary limitations:

• While feedback loops will improve the quality of the risk assessments, there still is some subjectivity. • Serious Risk violations are relatively rare events.



Figure 4 Final Risk Assessments (2013-2014)

Figure 4 depicts the risk breakdown of violations processed in 2013 and 2014. Very few violations were deemed Serious Risk. Since BES impact is a combination of probability and risk magnitude, the determination of actual reliability impact of a potential violation is another aspect that could provide significant benefit in trending. The team’s proposed ALR CP-2 provides a measure of the observable BES impact due to compliance violations. A common business approach to address a problem is to learn from similar but more mature processes and procedures in other industries. Industrial and utility safety programs provide a useful model; reducing the magnitude and frequency of serious safety mishaps by addressing the causes of more common lower level safety incidents. The theory behind this safety model notes that major injuries are prevented by continuously finding and rooting out small issues. A parallel reliability model implies that reducing the serious impacts on the BES due to non-compliance can be achieved by addressing the causes of more frequent and more common lower level impacts. Figure 5 depicts the comparison of the two models on which ALR CP-2 is based.

Figure 5 Accident Pyramid and Compliance Equivalent

At NERC Board meetings, Commission Chairman LaFleur has noted that she believes in the theory of the Accident Pyramid. This theory notes that major events are prevented by continuously finding and rooting out

Safety Model Compliance Parallel



small issues. H.W. Heinrich2 developed the original concept that there is a domino effect whereby a cluster of small problems can form a chain of events that lead to major mishaps. He presented data that implies a direct relationship between the number of unsafe acts, minor accidents and major injuries. Some of the most effective industrial safety programs are built upon the pyramid’s approach by developing a culture in which all employees have a stake in safety and are expected to continuously look for, report, and correct problems. The culture is achieved if the program is focused less on punishment, but more on correction of unsafe acts, which supports aggressive correction actions and does not tolerate negligence. A common approach to track success in safety programs is by collecting simple “observation reports.” The flow of observation reports show that people are actively looking for problems. The reports provide useful data on patterns of problems as well as a means to document corrective action. Similarly, in the electric power industry, the flow of self-reports and compliance exception log entries speak to the maturity of the entity’s compliance culture, the attitude of self-monitoring, and self-correcting, and the transparency to share lessons learned with others in the industry. To the extent that minor problems are aggressively found and corrected, there should be a decline in the more consequential mishaps higher in the pyramid. In order to follow this approach to reduce the impact to the BES of standard violations, an assessment needs to be made as to what was the observable reliability impact. If the issue was a logged item, the observation would be by the Registered Entity. If the problem was handled as a Compliance Exception or a PV, the data would be captured by Regional Entity staff. If the guidelines for assessing impact are clear, it should not matter where the impact determination is made.

There was a Violation or PV and it lead to

Tier 3 • Caused or contributed to a System Event

Tier 2 • IROL exceeded • Non-IROL SOL exceeded for > 30 minutes • BES facilities tripped unexpectedly • Frequency beyond FTL > 15 minutes • Emergency action taken (e.g. reconfiguration, load shed) to mitigate

or prevent impact of the violation • Equipment damage • Major loss of visibility for > 30 minutes • Loss of state estimation or contingency analysis for > 30 minutes

Tier 1 • Observations similar to those in the table immediately above but of

lesser duration or magnitude • Loss of ability to monitor ACE • Inability to control or follow dispatch orders • Voltage excursion beyond limits • Loss of ability to monitor cybersecurity events

Tier 0 • No observed impact

Figure 6 Impact Observations Mapped to the Impact Pyramid Tiers

2 Heinrich, H.W. (1931) Industrial Accident Prevention. McGraw-Hill: New York.



Figure 6 maps the four data tiers that define the impacts used for ALR CP-2. This metric only requires capturing a small amount of data along with each Compliance Exception or PV. The observed impacts in the figure were used to advance the development of this report. It is expected the list will evolve slightly over time based on experience. It should be noted that Tier 2 and Tier 3 observations are rare events. Of the data reviewed for this report, there were no Tier 3 violations and only a handful of Tier 2 cases. The value in the metric comes from verification that the industry is looking for problems (Tier 0) and addressing the minor problems (Tier 1) before they can become larger events. Benefits Capturing this simple data (tier and type of impact) associated with the requirement for each violation or compliance exception / log entry would enable several potential benefits: • Tracking reduction of high impact non-compliance occurrences using approaches proven in other industries

and fields of study. • Identifying those Requirements believed to be most often associated with observed impacts provides added

value o Focus Areas for CMEPs. o Likely starting point for development of internal controls and sharing on underlying causes and

mitigation activities. • Providing feedback to the standards process

o Requirements that are “Paragraph 81” candidates could be compared to history to determine whether or not there is history of impacts associated with their violations.

o Requirements that are “most violated” yet have no impact may need adjustment of their measure or may need to be clarified.

• Identifying low/no impact requirements that can be more expeditiously handled in the Enforcement process.

• While beyond the scope of the deliverables for this project, conceptually the four tiers of the impact pyramid could be used as a substitute for the present Violation Severity Levels (VSLs). NERC and the industry spend significant resources crafting VSLs for the standards, only to find out they generally are not used in the Enforcement process as the facts and circumstances associated with each violation are unique. Data from this effort could prove that the pyramid tiers could be used as a substitute for VSLs.

Initial Observations While the source of data was limited (violations filed in 2014), the figures in this section provide initial observations with some indicative results on how ALR CP-2 could be applied. Figure 7 represents the occurrence dates of the violations filed in 2014 that had some observed impact on reliability. Tier 0 observations (no-impact violations) are not depicted. There were no Tier 3 cases where violations caused or contributed to a system event. This explains the need to use the minor observations as early warning indicators of reliability risk rather than waiting for system events that are triggered due to compliance violations.



Figure 7 ALR CP-2 Trend

Figure 8 shows the breakdown by requirement of impactful violations filed in 2014. Note there were no Tier 3 cases in 2014. . A refined listed of impactful requirements would be useful for Registered Entities seeking guidance on where to start with their internal controls program.

Figure 8 ALR CP-2 Impactful Requirements (Recently Filed Violations)


Chapter 4 – Recommendations

Chapter 4 – Recommendations The team offers the following recommendations to achieve the objectives of the RISC’s request as well as garner additional benefits.

NERC and Regions 1. Move forward with proof of concept of ALR CP-1 and ALR CP-2.

a. NERC and Regions establish the data stream in PVs and logs. b. Work with NERC CCC and PAS to refine the symptoms that qualify as “impacts” during the rollout c. Create quarterly trends of ALR CP-1 and ALR CP-2 from 2012 to present as the starting point d. “Mine” 2012 to present PV data to create “Top 10” High Impact and “Top 10” Serious Risk

Requirements. 2. To the extent there are differences in logging approaches among Regions, include common elements to

capture the necessary data to enable ALR CP-2. 3. NERC to share “common cause” information (patterns) for all impactful (ALR CP-2) violations as well as root

cause information on violations that caused or contributed to system events. 4. NERC and Regions work with the CCC to extract and post “case note” type data from self-reports, logs, and

mitigation plans for the “Top 10” lists (most violated, high impact, serious risk). 5. Use the ALR CP-1 and ALR CP-2 metrics as input to the CMEP’s Risk Elements and Focus Areas. 6. Several of the team members had not seen the presently published granular metrics. Raise the visibility of

the compliance trends information through inclusion in the State of Reliability Report. 7. NERC and Regions to periodically review differences among Regions’ Serious Risk violations as an input to

developing Risk Elements, identifying Regional specific risks, as well as increasing consistency in the risk assessment process.

8. Use ALR CP-1 and CP-2 and other currently collected compliance data to create a Reliability Assurance Dashboard or equivalent status summary of compliance information to assess trends. Refer to Appendix 1.

Registered Entities 9. Review the “Top 10” lists as possible starting points for the development of your standard and requirement

level controls: a. Most violated list. b. Serious Risk (ALR CP-1) requirements. c. Impactful requirements (ALR CP-2).

10. Pursue logging authority and aggressively self-inspect and self-correct. 11. Capture underlying causes and actions taken to correct compliance exceptions. 12. For new compliance self-reports and log entries, include an assessment of impact of the non-compliance by

level and describe the impact observed.


Appendix 1 – Leveraging Existing Compliance Metrics

Appendix 1 – Leveraging Existing Compliance Metrics NERC publishes quarterly compliance-related metrics, several of which can be leveraged to give a sense of the compliance maturity of the ERO enterprise as well as meet the goals of RISC request of the CCC:

• Encouraging timely mitigation of Potential Violations (PV) and sharing of lessons-learned (mitigation and controls).

• Fostering a culture of self-inspection, self-correction, and self-reporting. • Tailoring compliance monitoring and Enforcement effort (both Industry and NERC) commensurate with

the risk of the infraction. Figure 9 shows how the global and selected granular compliance metrics can be used to create a Reliability Assurance Dashboard. The stoplight colors could represent the trend of the metric or whether the metric was achieving a particular ERO goal or trending favorably if a formal goal is not set. The team has developed a coding approach for the stoplights that can be used if the dashboard is developed.

Figure 9 Reliability Assurance Dashboard (Example)

Additionally, the team recommends adding “Top 10” most-violated requirement graphs from both a risk (ALR CP-1) and impact (ALR CP-2) perspective to NERC’s quarterly compliance statistics.


http://www.nerc.com/pa/comp/CE/Pages/Compliance-Violation-Statistics.aspx

Appendix 2 – Team Members

Appendix 2 – Team Members

• Aaron Hornick (NERC) • Barb Kedrowski (NERC CCC) • Chris Sweeney (NERC) • Ed Kichline (NERC) • Farzaneh Tafreshi (NERC) • Gizelle Babik (NERC) • Heide Caswell (NERC PAS) • Howard Gugel (NERC) • James Stanton (NERC CCC)

• Margaret Pate (NERC) • Matthew Varghese (NERC) • Melinda Montgomery (NERC PAS) • Michael DeLoach (NERC CCC) • Paul Kure (NERC PAS, RF) • Peter Raia (NERC) • Stanley Kopman (NPCC) • Terry Bilke (NERC CCC)


ERO Compliance: Metrics, Risk, and Reliability RISC Recommendations to the NERC Board of Trustees Comments, 2/19/2015

Page | 1

Section

Original Language Comment Proposed ResolutionDecision on comment

Executive Summary Additionally, the team recommends a Reliability Assurance dashboard that leverages the global metrics as well as existing compliance data to produce a snapshot of the state and trend of the ERO Enterprise compliance maturity.

Isn’t NERC moving away from the term Reliability Assurance? This link jumps down into the report? Do you want to really do that here?

We agree with the removal of the links and will include the high level recommendations. We agree that NERC is moving away from the initiative related to Reliability Assurance, but expect the assurance effort will continue

Executive Summary The concepts presented related to these metrics parallels findings in multiple fields of study, all of which offer insights into how to reduce risk and impact to systems and processes.

This link jumps to appendix 2 – do we really want to do that here?

The link and the associated section have been removed.

Executive Summary Finally, the team developed a set of recommendations to implement the metrics approach as well as potential ways to glean the benefits envisioned under the Reliability Assurance Initiative.

The recommendations should be listed in short form in the Executive Summary. Not tied in by link which will distract the readier into moving into chapter 4 prematurely

The key recommendations are now presented in the Executive Summary without the link.

Executive Summary

Editorial Comment: I think you should not use links in the Executive Summary. If someone wants to extract the Executive Summary for some other “show and tell” purpose, the link approach will interfere.

The key recommendations are now presented in the Executive Summary without the link.

Chapter 1 – Background Page 5 of 19

There were two prior attempts at creating global compliance metrics focused on compliance violations and the associated risk to

Another sentence is needed to discuss when this occurred e.g. between 20xx to20yy. Or drop a footnote with link to the reports or other documentation of that

Additional wording was added to clarify the period when the prior work was done.


Page | 2

reliability. effort. Chapter 1 ‐ Previous Global Metrics Efforts and Limitations Background, Key Compliance Monitoring Metric, page 5 of 19

• The selection of significant metrics is correlated to the weighting ascribed by Enforcement staff, as evidenced by their determination of significance, in addition to risk impacts determined by the Regional Entity, thus it relies heavily on expert judgment applied within several venues, where that judgment is then translated to infer compliance;

The major shortcoming of the draft document is the intention to use 'expert opinion' to gage the two proposed metrics: ALR‐CP1 (Risk Focus) and ALR‐CP2 (Impact Focus). Rather than proposing expert opinion as the measure of a metric a standard process to quantify each metric would be helpful. Further, a standard process that uses previously collected data would minimize the work load burden. Comment repeated in section on ALR‐CP1 benefits.

The concept of expert opinion applies to ALR CP‐1. As is, it would have the same shortcoming as past efforts. The team is recommending two feedback loops to improve the quality of the risk assessment. First, we recommend that NERC and Regions periodically review differences in risk assignment outcomes between Regions as well as the methodology for the assignment. Wording was changed to better communicate this We believe the ALR CP‐2 metric provides additional objectivity in that it is based on some measure of observed impact to reliability.

Same as above Same as above Suggest you search on the number of times this critique is mentioned. It is too many in my view. Try to reduce the number of times you point to this concern. Reason: with some many references to it, it seems that the only reason for doing this is because Enforcement Staff is not to be “trusted” or is not technically qualified. People will fill in the blanks if something goes unsaid

We agree. Changes were made.

Same as above “expert judgment” Apparent contradiction: “expert judgment” is a limitation with KCMI and with the original CP‐1, but a benefit with the revised CP‐1?

We agree. Changes were made.

Same as above While a useful start, there Yes, but how will the proposed approach Rather than starting from a set of


Page | 3

was no assurance that the selected set of requirements used for KCMI was the most impactful to reliability

provide that “assurance”. For instance can it be said that the proposed approach is self‐steering toward most impactful measures?

impactful requirements, the new metrics starts with a case by case review of the facts of the non‐compliance to determine risk and observed impact separately.

Same as above The KCMI set did not include CIP requirements, which raised questions of a potential shortcoming;

Yes and this is a problem, but the other side of this is that so far there has been no measureable operational impact form CIP related violations. There is no Category 0‐5 measure for CIP issues like there is for disturbances in the events analysis world. Extrapolating from a CIP violation to MW of load potentially lost is difficult. A single CIP breach by a determined foe can easily turn into a Cat 5 event. Therefore should all CIP violations then be classified as significant? Do you conclude that a single inadvertent tailgating entry in the security perimeter is a potential 10,000 MW+ generations or load loss severity assessment.

We believe the combination of ALR CP‐1 and ALR CP‐2 will address this over time. We expect that if a CIP violation causes loss of visibility or control, it will have some impact on reliability. Additionally, there are some very rare events that are Serious Risk if left unmitigated. The metrics are set up so that they can evolve as the ERO and the Industry learn.

Same as above KCMI did not easily accommodate recalibration when a new requirement was designated as a bellwether requirement, or if another was to be demoted from its bellwether standing.

Who is “they” in this sentence? Needs to be more clear. You mean the KCMI metric? Singular or its components?

We agree. The language was clarified.

Chapter 1 ‐ Previous Global Metrics Efforts and Limitations

Including (changed from Tracking) unmitigated violations inflated the

Including is a better word here. But I also disagree with the sentence. The safety



Page | 4

Background, ALR CP‐1, first bullet, page 6 of 19

metric due to administrative lag and also reduced the visibility of near term changes in the number of violations.

triangle model calls for detection as the first step. Correction is really not a part of the reporting. Correction is however part of the process.

Chapter 1 ‐ Previous Global Metrics Efforts and Limitations Background, ALR CP‐1, second bullet, page 6 of 19

• CP‐1 relies on expert judgment of Enforcement staff,

Apparent contradiction: “expert judgment of Enforcement staff” is a limitation with the original CP‐1 and with KCMI but a benefit with the revised CP‐1?


Same as above “expert judgment” The major shortcoming of the draft document is the intention to use 'expert opinion' to gage the two proposed metrics: ALR‐CP1 (Risk Focus) and ALR‐CP2 (Impact Focus). Rather than proposing expert opinion as the measure of a metric a standard process to quantify each metric would be helpful. Further, a standard process that uses previously collected data would minimize the work load burden. Comment repeated in section on ALR‐CP1 benefits.


Same as above “Enforcement staff” So – form a team of broad based experts. In the end someone or some group has to make the call. You really need an independent panel. No stakeholders, no registered entities? But still knowledgeable? A court system with independent (unelected, unemployed) judges?

The team proposes a feedback loop to improve the ALR CP‐1 determinations. Additionally, there will be some transparency of Serious Risk violations. We would expect staff to work with the PAS and CCC if significant changes to the approach are made.

Chapter 2 – Global Metrics, Introduction, page 7 of 19

Introduction Was thought given to the concept of velocity of impact? There is the risk, the impact and then the speed (velocity) with which the impact/risk occurs. I would find that informative. That is just a high‐level observation/question though, not a critique.

This is an interesting observation. Human performance speaks to latent vs. active errors. This could be analyzed once the data stream was established.

Chapter 2 – Global Metrics, All bullets Cleaned up language of “Caveats” for clarity. Agreed. Additionally, we should


Page | 5

Caveats, page 7 of 19

add a few other caveats (or repeat them later) that cause confusion. For example, some people felt that this effort was intended to change the events analysis process, which it is not..

Chapter 2 – Global Metrics, ALR CP‐1 (Revised) – Risk Metric Page 7 of 19

Definition: “enforcement staff” Capitalize all uses of Enforcement Staff. We agree.

Same as above Bullet 2, second paragraph: This approach will result in some slight modifications as Potential Violations (PV) progress through the queue. As long as the two types of observations

"Serious" or "Likely Serious" PVs are relative. Criteria should be established to ensure RE consistency. I believe Recommendation 7 in Chapter 4 addresses this issue.

Serious is an existing term used by Enforcement staff. As noted in the feedback loops above, we believe the quality of the risk assignments will improve over time.

Same as above Bullet 2, second paragraph: This approach will result in some slight modifications as Potential Violations (PV) progress through the queue. As long as the two types of observations

Will they be trued up later in the process if they are found to not be serious? I referenced this statement in a comment below.

Yes, the intent is that if a PV originally thought to be Serious is later downgraded, it will be removed from the observations in the period in which the PV occurred. Conversely, if a PV is at some point upgraded to Serious, it will be added to the quarter in which it occurred.

Chapter 2 – Global Metrics, ALR CP‐1 (Revised) – Risk Metric: Benefits Page 8 of 19

First Bullet: Global tracking of risk by potential violation based on the expert judgment of Enforcement Staff.

The major shortcoming of the draft document is the intention to use 'expert opinion' to gage the two proposed metrics: ALR‐CP1 (Risk Focus) and ALR‐CP2 (Impact Focus). Rather than proposing expert opinion as the measure of a metric a standard process to quantify each metric would be helpful. Further, a standard process that uses previously collected data would minimize the work load burden.

We have made suggested changes with regard to expert opinion and ALR CP‐1. Figure 5 in the document is intended to provide the standard process for determining the tiers for ALR CP‐2.

Same as above First Bullet: Global tracking of risk by potential violation based on the expert judgment of Enforcement

How will consistency be maintained across the 8 regions? The requirements should apply the same way for all registered entities throughout

We believe this should be part of the training that occurs under NERC”s Risk Based Monitoring


Page | 6

Staff. NERC (with exception for regional variances). The metric will get skewed if one region is looking at a requirement differently. Maybe that's another benefit of the metric if you are able to drill down to the regional level.

and Enforcement efforts. Also, refer to the discussion above on the recommended feedback to improve the quality of the risk assessment.

Same as above First Bullet: Global tracking of risk by potential violation based on the expert judgment of Enforcement Staff.

Is expert judgment purely subjective? Is it regional in nature? How do we know that the expert judgment is not biased in some persistent way?

See responses on “expert opinion” above.

Same as above First Bullet: Global tracking of risk by potential violation based on the expert judgment of Enforcement Staff.

Apparent contradiction: “expert judgment of Enforcement Staff” is a benefit here but a limitation with KCMI and the original CP‐1?


Same as above After the bullet list Need discussion of “how” to apply these guidelines. Who is the panel to make the calls?


Chapter 2 – Global Metrics, ALR CP‐1 (Revised) – Risk Metric: Initial Observations Page 8 of 19

Figure 2, first paragraph of description that includes the phrase, “…most often assigned Serious Risk.”

(See green highlights above) The second paragraph under item 2 of the description discuss the possible modification as these "likely" serious PVs move through the process. There was a statement that identified the need to properly designate them but the charts group everything together. This is misleading. I suggest identifying those that are "likely" until they are confirmed.

The graphs in the report only include confirmed Serious Risk violations. We would need to start the data stream to get the Likely Serious risk violations from the Regions.

Chapter 2 – Global Metrics, ALR CP‐1 (Revised) – Risk Metric: Initial Observations Page 9 of 19

Figure 2, second paragraph of description that includes the phrase, “…average trend line.”

I'd recommend taking it down to the requirement level. Just listing TOP‐002 doesn't do registered entity much good.

We agree. This granularity will be provided as the data becomes available.

Chapter 2 – Global Metrics, ALR CP‐2 Impact Metric, Description Page 9 of 19

Paragraph 1: While the ALR CP‐1 (Risk) metric should provide value to the ERO Enterprise, it has a limitation in that it relies partially on opinion and judgment by Enforcement Staff in the assignment of risk. While the evaluation of potential or actual risk that is imposed by non‐compliance is important, the determination of

You mentioned that the expert judgment was a benefit and here state that it is limiting in nature. Just a thought.

We will change the wording to reflect that there still is some subjectivity with ALR CP‐1, but with the feedback loops, we expect it will be less than previous attempts. ALR CP‐1 provides the risk of the violation. ALR CP‐2 shows


Page | 7

actual reliability impact of a potential violation is another aspect that could provide significant benefit in trending.

another aspect of non‐compliance, one that reflects that observable impact of the non‐compliance on BES reliability.

Same as above Description, paragraph 1: While the ALR CP‐1 (Risk) metric should provide value to the ERO Enterprise, it has a limitation in that it relies partially on opinion and judgment by Enforcement Staff in the assignment of risk. While the evaluation of potential or actual risk that is imposed by non‐compliance is important, the determination of actual reliability impact of a potential violation is another aspect that could provide significant benefit in trending.

I suggest removing this. It doesn't add anything. This description should focus on this metric.

We agree. Wording has been changed.

Same as above At NERC Board meetings, Commission Chair LaFleur has noted that she believes in the theory of the Accident Pyramid.

I recommend you not invoke the Chairman in this report. I think while this may be accurate it really adds nothing to the argument for or against the approach. The proposal needs to stand on its own.

The team will discuss this more. Chairman LaFleur is well known for leading safety and reliability improvements at National Grid using this model. It also seems appropriate to acknowledge that the industry listens to its regulators.

Chapter 2 – Global Metrics, ALR CP‐2 Impact Metric, Description Page 10 of 19

Figure 3 Accident Pyramid and Compliance Equivalent

I like the safety pyramid comparison but Linking No Impact PVs with Unsafe Acts isn't "Apples to Apples". Seems like this is saying every PV is reliability related...which is not true. Are Compliance Exceptions addressed? The bucket of No impact PVs (looks like Tier 1) will be quite large and basically a metric that will mean little to Registered Entities.

We agree it’s not identical, but there is a strong parallel. Assuming the industry has developed standards correctly, there is some risk associated with each PV. Similar to driving safety, not all unsafe acts are equal. Driving 5 MPH over the speed limit is not as risky as going through red lights or through flashing railroad


Page | 8

semaphores. Still, always driving under the speed limit should result in fewer accidents over time. Additionally the fact that the industry is finding and fixing these lower level occurrences should help reduce not only the size of the “base” of the pyramid but also the number of occurrences of the upper tiers. The bottom level of the safety pyramid starts with unsafe acts that had no safety impact (no one was hurt). Likewise, the bottom levels of our pyramid include non‐compliances that had no reliability impact. We’ve provided clarifying language.

Same as above A common approach to track success in safety programs is by collecting simple “observation reports.”

Right and the mitigation of the instances does not enter into it.

Thank you for your comment. We agree. No change to the language in the document.

Same as above In order to follow this approach to reduce the impact to the BES of standard violations, an assessment needs to be made by someone or some group as to what was the observable reliability impact. Unlike the assessment of risk, this metric focuses on the actual impact, not the potential impact that the violation might have occurred.

Wording changes to text in these two paragraphs.

We tend to believe that if a fairly transparent approach is followed, along with a feedback loop between NERC and Regional Enforcement staff, the quality of the data should improve over time.


Page | 9

If the guidelines are clear, the initial assessment of impact could occur at the Registered Entity level if the entity had logging authority. This should be followed by an independent regional review on the basis that the region has through its internal risk analysis determined where its regional risks lie. This could be further screened by NERC. It is expected that the “official” determination of impact would be done by the Regional Entity with oversight by a multi disciple team at NERC.

Chapter 2 – Global Metrics, ALR CP‐2 Impact Metric Page 11 of 19

Figure 4 Impact Observations Mapped to the Impact Pyramid Tiers

The Tier levels should be included here. We agree and will make thechange.

Same as above Figure 4 Impact Observations Mapped to the Impact Pyramid Tiers – Caused or contributed to a System Event

Contributing to an event is a very wide open metric. I think those things are captured in the moderate and below

While conceptually the comment makes sense, a review of the data shows that very rarely does a non‐compliance cause or contribute to a system event (perhaps once every few years).

Same as above Figure 4 Impact Observations Mapped to the Impact Pyramid Tiers

A "System Event" should be qualified. When this was introduced during the webinar I was highly opposed to this metric. I believe after reading this description there is a miscommunication or misunderstanding in the meaning of system event. I now think what are looking for are major events and I would agree if a violation of a requirement causes a major system event (must be qualified or defined) then it is a high impact event. I hate to use the

We agree there are wide differences between the different levels of system events. Initial data implies that cases where non‐compliances cause system events are very rare. If additional granularity is needed once the data stream is established, the model can accommodate it.


Page | 10

EA categories here but it's obvious in the chart below that there were used. I would equate a major system event as a Category 4 or 5 event. I believe the moderate and minor impacts need work too. For instance one of the findings from collecting EMS reports is that no EMS event reported to date (this could easily change) has had an impact on the BES (i.e. no load loss, no unintended transmission outage, and/or no unintended generation outage). Based on that finding it's a little difficult to establish a moderate impact unless this is qualified as the violation having caused that outage and the impact is the loss of EMS but that's really more of a Risk than an impact and you capture that in CP1. Also, saying major loss of visibility is broad who will define major. If the team is going to reference items from the EA Process then I strongly suggest the team consult the Event Analysis Subcommittee

We agree that there has been no known cases as of this date where an EMS outage has caused an event. We also agree that momentary EMS outages are not uncommon. Recall that this is not saying we are tracking EMS outages or that an EMS outage of > 30 minutes is a violation. We suggest tracking those situations where a PV has been found to cause an extended EMS outage of a length that could preclude correcting an IROL or BAAL violation. Data implies that these are rare events (EMS outages caused by a violation). If actual practice shows that the inclusion of this impact is too sensitive, the scale can be changed..

Same as above IROL exceeded Non-IROL SOL exceeded for

> 30 minutes

There are two clear items on the list. Thank you for the comment.

Same as above BES facilities tripped unexpectedly

I expect BES facilities to trip all the time. The question is did the protection misoperate? We now have a more clear approach for misoperation determination

This is a situation where there has been a non‐compliance identified and the facts point to the non‐compliance causing the facilities to operate. We are not suggesting that if facilities operate, there has been a violation.

Same as above Equipment damage

Note for example every overload event takes life out of a transformer for instance. Is every transformer overload event to be included? Or only the last one that causes the failure?

We expect these to be very rare events and apparent when they occur.


Page | 11

Same as above Major loss of visibility for > 30 minutes

Losing sight of 10 stations? 20? 30? What voltage? Point is, what is major?

We expect the table in the report to be expanded slightly when applied in practice. A starting point would be something akin to loss of visibility of [10] substations. This can be tested once the data flow is established.

Same as above Observations similar to those in the table immediately above but of lesser duration

Less than what? Judgment required

The report is primarily a proof of concept. We expect further refinement when the data stream is established.

Same as above Failure to control or follow dispatch orders

Control what? Generation?

We will clarify the language. Inability to control could be either generation or transmission. Again, we expect more objectivity to be added once the data stream is established.

Same as above Voltage excursion or failure to follow voltage schedule

Excursion happens all the time. You might want to be more specific and say voltage above or below planning limits? Tie to TPL standard?

We will attempt to clarify the language. The table in the report is a starting point. The measure is related to non‐compliances that cause a voltage problem. This could be an excursion of +/‐ 5% of nominal or a locally more restrictive limit.

Same as above Loss of ability to monitor cybersecurity events

How do you know when you lost the ability? Not very crisp – very judgment based.

We would have discussions on better language.

Chapter 2 – Global Metrics, ALR CP‐2 Impact Metric, Benefits Page 11 of 19

Bullet 1, • Modeling reduction of high impact non‐compliance occurrences after approaches proven in other industries and fields of study.

Delete part of sentence We will edit this sentence to make it clearer.

Same as above Last bullet’ The pyramid tiers Why tie it at all to VSL. Say it is a more We will change the language as you suggest.


Page | 12

could be used as “pro forma” VSLs.

effective substitute

Chapter 2 – Global Metrics, ALR CP‐2 Impact Metric Page 12 of 19 Initial Observations:

Figure 5 ALR CP‐2 Trend The tiers are not identified on the chart above. I assume tier 1 is no impact.

Tier 1 are the minor impacts. There were many no‐impact violations in each quarter, some minor impact cases, very few moderate impact violations in each quarter and no high impact cases during this period. We will add clarifying language.

Same as above Figure 5 description, paragraph 1, phrase “…indicator of reliability risk trends…”

This is supposed to be an impact based trend not a risk based trend. Are we trying to show that events (in general) include violation of requirements and have impact to the BES? I think the metric needs to speak for it's self if there is no impact let show there is no impact. If event are not caused by compliance violations then it should show that too.

There are correlations between risk, reliability and impact. As the industry increases self‐inspection and self‐correction, we would expect to see a decline in the number of tier 1, 2, and 3 violations. Additionally, if we assume that risk assessments are done generally correctly, we would see fewer serious risk violations..

Same as above Figure 6 ALR CP‐2 Impactful Requirements (Recently Filed Violations)

Again, I'd like to see the specific requirement. Entities are audited to the requirement level and not Standard.

We will provide that granularity.

Same as above Same as above

PV or violations? We propose for ALR CP‐2 to track both PVs and confirmed violations with impact (with some means to provide clarity which is which). It can take many months if not years for a PV to become a confirmed Violation. While we expect some true‐up of the trends over time, the relative change in each quarter should be small. We believe having good data now is better than perfect


Page | 13

data with a two year lag.

Chapter 2 – Global Metrics, Interaction between ALR CP‐1 and ALR CP‐2 Page 13 of 19

Heading: change to Expected Interactions between ALR CP‐1 and ALR CP‐2

Wording change Based on other comments received, we’ve deleted this language.

Same as above The two proposed metrics are complementary in that there are numerous violations that pose significant risk to the BES, yet had no observable impact on system performance because the system is designed to be resilient. Failing to correct an IROL in 30 minutes is one example. Even misoperations of protection will ultimately trigger operation of a secondary (or beyond) protection system that successfully limits the extent of the disturbance if one occurs at all.

Proposed wording change to this section We have made wording changes that are hopefully clearer.

Chapter 2 – Global Metrics, Interaction between ALR CP‐1 and ALR CP‐2 Page 13 of 19

Paragraph 1, sentence, “Failing to correct an IROL in 30 minutes is an example.”

This is the second item on the chart for CP2 which contradicts the previous statement. There's a breakdown somewhere.

Based on other comments received, this has been deleted.

Same as above It is expected that over time, there will be overlap between the requirements that rise to the top in the two metrics.

Are we talking about standards requirements here? Not clear to me?

Based on other comments received, this has been deleted.

Same as above Feedback into the Standards Process

This seems to come out of know where. Why is this here?

This has been deleted.

Chapter 3 – Leverage the Global and Current Granular Metrics, page 14 of 19

Figure 7 Reliability Assurance Dashboard

Add an example watermark if possible. We will make it clearer that this is an example.


Page | 14

Same as above Chapter 3 – Leverage the Global and Current Granular Metrics

I recommend you skip this. Do not design the dashboard before you get approval of the metric.

We’ve moved the information to an appendix so that it flows better. We tend to believe there is value in providing a wider vision on how the data can be applied.

Same as above Same as above I like having four categories, but “nearly meeting” is the same as “missing” to me. It should be “meeting the NERC performance threshold with a declining trend”.

We’ve made note that the team will provide more details about defining the categories if the metrics are approved.

Chapter 4 – Recommendations: NERC and Regions, page 15 of 19

NERC and Regions Need an explicit discussion how the multidisciplinary review team will be formed and operated.

The things that occur within Enforcement probably should be left to NERC’s discretion. We believe a level of transparency that will exist in the data stream will provide ongoing improvement. Additionally, if significant changes to the model are proposed, we would expect NERC to work with the PAS and the CCC.

Same as above 1. Move forward with proof of concept of ALR CP‐1 and ALR CP‐2.

Yes, but first you need to tighten up the words around how to put the detected violations into the Tier buckets

We provided some clarifying language and will look for feedback from the technical committees on how to improve this further.

Same as above To the extent there are differences in logging approaches among Regions, include common elements to capture the necessary data to enable ALR CP‐2

Need regional vetting and training. Not every regions is staffed to do this.

Recent discussions at the NERC Board meetings imply that compliance exemptions and logging are the next logical step for improving the compliance monitoring and enforcement processes.

same as above Bullet 6. Several of the team members had not seen the presently published granular metrics. Raise the visibility of the compliance trends

Visibility of the metrics on NERC.com is important. It will likely drive internal control programs to focus on impactful standards.

We agree and will work with NERC to follow through on this recommendation.


Page | 15

information through inclusion in the State of Reliability Report.

Chapter 4 – Recommendations: Registered Entities, page 15 of 19

Bullet 9c. Impactful requirements (ALR CP‐1).

Should this reference be ALR‐CP‐2? Good catch.

Same as above Bullet 9, For those with less than fully mature compliance processes, review the “Top 10” lists as a starting point for the development of your standard and requirement level controls:

Who decides if a given regional entity has a “less than fully mature” compliance process? And is the reference the pre RAI compliance team? Or the future one that is internal risk and controls based?

We’ve changed the wording to hopefully clarify this.

Appendix 2 – Related Fields of Study, Background, page 17 of 19

Background:There are several fields of study that apply concepts similar to what is presented in this report to achieve improved performance and reduce impact and likelihood of major events.

This is good information but I'm not sure it'sneeded in this report.

We’ve removed this information.

ALR-CP-1 Serious Risk Violations

Metric Number ALR–CP-1

Submittal Date November 14February 20, 20143 Sponsor Group (OC, PC or subgroup name)

PAS and CCC

Short Title Count of Serious and Likely Serious Risk Violations

Metric Description

Segment all confirmed standard violations and those that have been triaged by enforcement staff prior to being confirmed by their risk assessment rating in the calendar quarter they are reported

Purpose

The goal is to provide the industry a meaningful trends of non-compliance with the Reliability Standards that pose actual or potential high-risk standards compliance to the Bulk Power System. This metric is to inform, increase transparency, and quantify effectiveness of risk reduction and/or mitigation actions. PAS and CCC recommends using the risk assessment level as selection criteria to identify the subset of confirmed violations to be included in this metric. The level is the assessment of risk to the bulk power system when a requirement is has been violated, as determined by the Regional Entity and NERC, and in agreement with agreed by the entity involved. The three four risk assessment levels are no impact, minimal impact, moderate impact and serious impact. The following factors are considered when assessing the risk level, but not limited to:

Standard requirement violation risk factor (VRF)

Violation severity level (VSL)

Time horizon

Relative size of the entity

Relationship to other entities

Possible sharing of responsibilities

Voltage levels involved

Size of generator or equipment involved

Ability to project adverse impacts beyond the entity’s own system


This metric measures reduction the number of confirmed and likely serious risk violations over a trending period. The violations included all non-compliances that have been confirmed or found to be likely to pose actual or potential serious risk to reliability.of the BPShave actual or

potential serious reliability impacts.

Formula Count each calendar quarter of how many unmitigated confirmed serious or likely serious violations that have been submitted that have pose actual or potential serious risk impact to BPS reliability

Time Horizon Every Quarter Metric Start Time or Baseline and Roll Up

Year 201308, or when data is first available.


Quarterly

Ease of Collection The data is available on the NERC public website. The risk assessment and the mitigation completion date can be found in the Notice Of Penalty (NOP) searchable spreadsheets.

Aggregation Data will be aggregated at the Regional Entity and NERC level.


All non-CIP standard requirements

Linkage to Data Source


NoYes

Data Submitting Entity Regional Entity and NERC, monthly Enforcement and Mitigation

SMART Rating Total Score

Specific/ Simple

Measurable Attainable Relevant Tangible/ Timely

14 3 2 3 3 3

Reporting


Line Bar chart per quarter for the number of serious risk violations (see sample chart below). Could also create bar chart showing most violated requirements of reliability standards as shown in Figure 2 below.


The metric trending will be published on the NERC reliability indicator web page and included in the annual NERC State of Reliability report.

Figure 2 Standards and Requirements with Most Occurrences of Serious Risk Violations

CP-2 Serious Risk Violations

Metric Number CP-2

Submittal Date February 20, 2015 Sponsor Group (OC, PC or subgroup name)

PAS and CCC

Short Title Potential Violations with Observable Reliability Impact

Metric Description

CP-2 is a quarterly count of the number of newly reported Compliance Exceptions or Potential Violations with observed impact by Impact Tier level. Impact Tier is defined by the association of the specific Potential Violation to an observed impact as outlined below.

Purpose

The goal is to provide the industry meaningful trends of potential compliance violations that had observable impact on reliability. This metric is to inform, increase transparency, and quantify effectiveness of requirements that when violated have an observable impact on reliability. The initial assessment of impact could occur at the Registered Entity level if the entity has logging authority. It is expected that the “official” determination of impact would be done by the Regional Entity Enforcement staff. PAS and CCC recommend using the impact assessment level as defined in this document for potential violations to determine the actual impact that the violation had upon the reliability of the BPS. Capture the following with each log entry and PV (by requirement):

Tier 3. Caused or contributed to a system event

Tier 2. Moderate Impact

Tier 1. Minor impact

Tier 0. No observed impact

It is expected the list below will evolve slightly over time based on experience; at that time implications to the metric (and historic performance) will need to be evaluated.

Tier 3: Caused or contributed to a System Event Tier 2: Moderate Impact IROL exceeded Non-IROL SOL exceeded for > 30 minutes BES facilities tripped unexpectedly Frequency beyond FTL > 15 minutes Emergency action taken (e.g. reconfiguration, load shed) to mitigate or

prevent impact of the violation Equipment damage Major loss of visibility for > 30 minutes Loss of state estimation or contingency analysis for > 30 minutes Tier 1: Minor Impact Observations similar to those in the table immediately above but of lesser

duration or magnitude Loss of ability to monitor ACE Inability to control or follow dispatch orders Voltage excursion beyond limits Loss of ability to monitor cybersecurity events Tier 0: No observable impact


This metric measures the number of potential violations and the impact tier that the violation has been determined to be in for a calendar quarter.

Formula Count each quarter of how many potential violations have observable impact to reliability and the tier associated with that observable impact.

Time Horizon Every Quarter Metric Start Time or Baseline and Roll Up

Year 2013, or when data is first available.


Quarterly

Ease of Collection The data will need to be added to the compliance reporting form.

Aggregation Data will be aggregated at the Regional Entity and NERC level.


All standard requirements

Linkage to Data Source The data will be provided with the Notice of Potential Violation, and will be verified and validated by Regional Entity Enforcement Staff.


Yes

Data Submitting Entity Registered Entity submitting non-compliance data and Regional Entity and NERC Enforcement

SMART Rating Total Score

Specific/ Simple

Measurable Attainable Relevant Tangible/ Timely

14 3 2 3 3 3

Reporting


Bar chart per quarter for the number of serious risk violations (see sample chart Figure 1 below). Also, a bar chart showing the most impactful non-compliances by standard and requirement violated, as shown in Figure 2 below.


The metric trending will be published on the NERC reliability indicator web page and included in the annual NERC State of Reliability report.

Figure 1 CP‐2 Trend

Figure 2 CP‐2 Impactful Requirements (Recently Filed Violations)

Documents

DRAFT Agenda Planning Committee Meeting Highlights and... · 2015-03-11 · Draft Agenda – Planning Committee Meeting March 10-11, 2015 2 c. Update on Physical Security Guidelines