ANNUAL REPORT

FY 2016

TABLE OF CONTENTS

From Our Chair, ____________________________________________________________________________________ i

From our President & CEO, _____________________________________________________________________ ii

The Risk Based Approach to Reliability ____________________________________________________________________________ 1

Risk Identification ___________________________________________________________________________________________________ 2

Risk Mitigation _______________________________________________________________________________________________________ 8

Risk Communication _________________________________________________________________________________________________ 12

i

From Our Chair,

During ReliabilityFirst’s recent 2016 Annual Meeting of Members,

NERC CEO Gerry Cauley recognized ReliabilityFirst’s innovative

approaches and strong technical work as a Region. I echo Mr. Cauley’s

sentiments, and I’m proud of the work done by ReliabilityFirst in 2016

to advance reliability. ReliabilityFirst’s work is particularly important

given that our Region contains numerous, densely populated

metropolitan centers (including Washington, D.C.) and the

interconnection of two of the world’s largest energy markets.

ReliabilityFirst’s goals for 2016 included being risk-informed,

analytical and reliability-focused; and providing value by identifying

risks and aiding industry in risk mitigation. I’m pleased to report that

we achieved these goals, and that in 2016, ReliabilityFirst’s major

efforts included numerous education and outreach activities (which

Tim discusses in more detail below); the 2016 Regional Risk

Assessment; and collaborative work with NERC and the other Regions

to further mature the Risk-Based Compliance Monitoring and

Enforcement Program. These efforts are discussed in this Report.

On behalf of the entire Board of Directors, thank you to all of those

who worked with ReliabilityFirst and the greater ERO Enterprise to

advance reliability this year. I look forward to the accomplishments of

2017, which will include the development of a new five-year strategic

plan for ReliabilityFirst. This new strategic plan will be finalized by

late summer, and it will reflect ReliabilityFirst’s growth from a start-

up organization into a more mature state.

I am confident that ReliabilityFirst, working closely with FERC, the

rest of the ERO Enterprise, and our entities, will continue to move

reliability forward in 2017.

Forward Together,

Lou

Lou Oberski

Chair of the Board of Directors

ii

From Our President & CEO,

2016 was a transformative year for ReliabiltyFirst as we both stabilized

and continued to mature many of our processes and departments in

order to facilitate our commitment to you to become a leading risk-

based organization. Accordingly, you will note that this report is

organized a bit differently than in the past. Similar to how we approach

our work, this report is now focused around the three core functions

that guide our activity: Risk Identification, Risk Mitigation, and Risk

Communication. Within this construct, the overarching theme for

ReliabilityFirst in 2016 focused on Risk Communication, such as

outreach and training. With CIP Version 5 becoming effective in July,

many entities reached out to us for preparation and transition assistance.

My staff rose to the challenge and accommodated over 35 productive

Assist Visits in 2016.

ReliabilityFirst also conducted various webinars and workshops,

including an enforcement webinar on CIP themes; the always well-

attended spring and fall compliance workshops; a protection system

workshop; and a new generator-focused workshop. I am extremely

proud of my staff’s dedication to assist our entities to further reliability.

Jim Keller retired from the Board of Directors this year, and I would

like to thank him for his foundational contributions to ReliabilityFirst.

During his tenure, Mr. Keller served as a Board Chair, Vice Chair, and

as an officer for the Nominating and Governance Committee. Likewise,

I want to thank Mohan Sachdeva, who recently stepped down from the

Board due to personal reasons, for his important contributions as a

Small LSE Sector Director.

I am pleased to report that Matthew Paul from DTE Electric Company

was recently elected to the Board as director for the Medium LSE

Sector. I am also pleased to report that Sue Ivey from Exelon

Corporation and James Haney from FirstEnergy Services Corporation

were reelected to the Board this year as industry sector directors, and

Brenton Greene and Kenneth Capps and were reelected to the Board as

Independent and At-Large Directors, respectively. Lou Oberski and

Lisa Barton provided valuable leadership over the course of the year as

our Board Chair and Vice Chair, and we are thankful that they have

committed to continue in these roles in 2017.

On behalf of ReliabilityFirst, we look forward to working with you all

in furtherance of reliability.

Forward Together,

Tim

Timothy R. Gallagher

President & CEO

1

The Risk Based Approach to Reliability

In early 2015, FERC approved the implementation of the ERO Enterprise’s Risk Based

Compliance Monitoring and Enforcement program, a risk-focused approach to compliance monitoring

and enforcement that focuses time and resources on higher-risk issues while still identifying and

addressing lesser-risk issues.1 Consistent with the goals of the Risk Based Compliance Monitoring and

Enforcement Program, in 2016, ReliabilityFirst focused its work around three primary functions to ensure

the reliability, security, and resiliency of the ReliabilityFirst footprint.

These three functions are: (1) Risk Identification, which concerns ReliabilityFirst’s work to

identify and prioritize risks relevant to its footprint; (2) Risk Mitigation, which concerns

ReliabilityFirst’s work to develop thoughtful approaches to work with entities to ensure the mitigation of

these identified and prioritized risks; and (3) Risk Communication, which concerns ReliabilityFirst’s

work to communicate the identified and prioritized risks and mitigation strategies to the ERO Enterprise,

across its footprint, and/or to targeted entities, as appropriate. To help reinforce the intent and purpose

behind ReliabilityFirst’s activities in 2016, this Annual Report is organized around these three functions.

1 See, Order on Electric Reliability Organization Reliability Assurance Initiative and Requiring Compliance Filing,

150 FERC ¶ 61, 108 (2015), P. 25-28.

Risk Identification

Risk Communication

Risk Mitigation

Figure 1: ReliabilityFirst’s Risk Based Approach

This figure depicts how ReliabilityFirst’s departments and activities naturally organize into

three primary functions of Risk Identification, Risk Mitigation, and Risk Communication, all

of which continually feed into and inform each other. The shaded areas demonstrate how

certain activities can constitute the performance of more than one of these key functions.

2

To ensure reliability, it is critical to continually identify, understand, and prioritize risks impacting

the ReliabilityFirst footprint. Effective risk identification is foundational to the ability to perform

thoughtful and targeted risk mitigation and risk communication.

ReliabilityFirst’s risk identification activities cross numerous departments and include various

important activities. They primarily include the Regional Risk Assessment (identifying and prioritizing

risks impacting the ReliabilityFirst Region); Inherent Risk Assessments (identifying risks impacting a

specific entity); Risk-Harm Assessments (identifying risks caused by a specific violation); event analyses

(determining root causes and risks associated with system events); voluntary maturity model engagements

(collaborating with stakeholders to identify performance and capabilities to manage and mitigate

identified risks); and reliability assessments and performance analysis (identifying risks associated with

transmission performance and resource adequacy).

2016 Regional Risk Assessment and Risk Elements

The process to identify risk in the ReliabilityFirst footprint starts at the NERC Level, where the

NERC Reliability Issues Steering Committee identifies continent or ERO-wide Risk Elements. The

NERC-identified 2017 ERO-wide Risk Elements are set forth in the chart below and mapped by their

likelihood of occurrence and potential impact to reliability.

Risk Identification

Figure 2: 2017 ERO-wide Risk Elements

Note that cybersecurity vulnerabilities, the changing resource mix, and resource adequacy have

a higher likelihood of occurrence, pose a higher impact to reliability, and have an increasing

risk trend.

3

Upon NERC’s release of the ERO-wide Risk Elements, ReliabilityFirst analyzes them, along with

numerous other sources of Region-specific information and data, to identify and prioritize the Risk

Elements most relevant to ReliabilityFirst. In 2016, ReliabilityFirst’s analysis yielded the following eight

Risk Elements to guide its work:

After identifying the Risk Elements most important to the ReliabilityFirst footprint,

ReliabilityFirst identifies the associated Reliability Standards that help mitigate the risks set forth in the

Risk Elements. ReliabilityFirst then uses the Risk Elements and associated Reliability Standards as

inputs to the Inherent Risk Assessment that it conducts for each entity. This in turn helps scope

thoughtful and customized monitoring engagements for each entity to help ensure the mitigation of the

identified risks.

1. Critical Infrastructure Protection

The protection of critical infrastructure remains an area of

significant importance. The Critical Infrastructure

Protection Risk Element includes threats and

vulnerabilities that result from: (1) system downtime, (2)

unauthorized access, and (3) corruption of operational

data.

2. Extreme Physical Events

The Extreme Physical Events Risk Element includes the

risk of extreme natural events or physical attacks.

Extreme natural events can cause equipment damage, lead

to fuel limitations, and disrupt telecommunications.

Localized physical attacks of significance or

electromagnetic pulse (EMP) attacks can, at their extreme,

can cause extensive interconnection-wide equipment

damage and disrupt telecommunications.

3. Maintenance and Management of Bulk

Power System (BPS) Assets

The Maintenance and Management of BPS Assets Risk

Element includes the risk of aging infrastructure and lack

of infrastructure maintenance. Deficiencies in

maintenance strategies create additional pressure on spare

equipment programs and the ability to replace aging

infrastructure. Transmission outages related to

inconsistent vegetation management pose an ongoing

reliability risk to the BPS. Related to this area is the risk

posed by misalignment between the design and actual

construction of BPS facilities (as highlighted in NERC’s

2010 Facility Ratings Alert).

4. Monitoring and Situational Awareness

The Monitoring and Situational Awareness Risk Element

includes the risk of lack of system information.

Situational awareness helps operators understand the

current state of their environment and helps them adapt

their behavior as necessary to make effective and efficient

decisions to safeguard reliability. The unexpected outages

of tools, or planned outages without appropriate

coordination or oversight, can leave operators without

visibility to some or all of the systems they operate.

5. Protection System Failures

The Protection System Failures Risk Element includes the

risk of protection systems that trip unnecessarily and

protection systems that are not coordinated properly.

Protection systems that trip unnecessarily can contribute

significantly to the extent of an event. When protection

systems are not coordinated properly, the order of

execution can result in either incorrect elements being

removed from service or more elements being removed

than necessary.

6. Event Response/Recovery

The Event Response/Recovery Risk Element includes the

risk of poor event response and recovery during

restoration activities. Poor event response and recovery

causes safety, operational, or equipment related risks

during restoration activities, and also contributes to

prolonged transmission outage durations.

7. Planning and System Analysis

The Planning and System Analysis Risk Element includes

the risk of uncoordinated planning, and includes multiple

evolving planning challenges (such as increased use of

demand-side management, integration of variable

generation, changes in load and system behavior, smart

grid, increased dependence on natural gas, fossil

requirements and retrofit outage coordination, nuclear

generation retirements and outages, and resource

planning). Uncoordinated planning can lead to instances

where generation or transmission resources, or

information concerning those resources, may be

inadequate to ensure that firm demand is served.

8. Human Performance

The Human Performance Risk Element concerns the risk

posed by situations in which a human being makes a

decision that contributes to operational and

communication errors. Stronger management and

organizational support greatly contribute to the reduction

and prevention of operational errors.

4

In addition to successfully completing the 2016 Regional Risk Assessment, ReliabilityFirst

continued to mature the Regional Risk Assessment program by improving the risk analysis techniques

that feed into the assessment; collaborating with the ERO Enterprise to benchmark and ensure all Regions

are performing accurate and consistent mapping of Reliability Standards to Risk Elements; and

identifying risk mitigation approaches.2 ReliabilityFirst’s work to mature its processes has positioned it

to release a public Regional Risk Assessment in 2017. ReliabilityFirst believes this will facilitate both

risk awareness and a productive stakeholder dialogue to further improve the Regional Risk Assessment.

Inherent Risk Assessments

During the Inherent Risk Assessment, ReliabilityFirst assesses the inherent risk an entity poses to

reliability, to determine the areas of focus and level of oversight required for that entity. ReliabilityFirst

performed 67 Inherent Risk Assessments in 2016, and has now completed Inherent Risk Assessments for

all Balancing Authorities, Reliability Coordinators, and Transmission Operators in its footprint. Similar

to the maturation of its Regional Risk Assessment program, ReliabilityFirst has matured its Inherent Risk

Assessment program by developing an entity-facing Inherent Risk Assessment Report. This report

facilitates an entity’s risk awareness by allowing it to better understand ReliabilityFirst’s: (1) perspective

on the risks the entity poses to the ReliabilityFirst’s footprint, and (2) rationale for the entity’s customized

compliance monitoring plan.

Risk-Harm Assessments

During the Risk-Harm Assessment, ReliabilityFirst quantifies the risk and potential harm posed

by a possible violation through a series of questions answered by ReliabilityFirst subject matter experts.3

The large majority of violations identified in 2016 were minimal risk, with some moderate risk violations,

and no serious risk violations. This is a positive trend, and ReliabilityFirst continues to work with entities

to drive down the occurrence of higher risk issues in the Region.

2 For example, one of the identified Risk Elements is Protection System Failures: the associated action items include

hosting a protection system workshop, encouraging entities with higher than average misoperations rates to

participate in an appraisal of their misoperations-related management practices, and increasing the activities of the

Protection Subcommittee. 3 ReliabilityFirst’s Risk-Harm Assessment Methodology and Process is available at

https://rfirst.org/enforcement/Pages/default.aspx. ReliabilityFirst also conducts training on its risk-harm assessment

process to entities upon request.

Figure 3: 2016 Risk Allocation of Violations

The majority of violations identified in 2016 were minimal risk.

0

5

10

15

20

25 2016 Risk Allocation

Minimal Moderate Serious

5

Event Analysis and Situational Awareness

In the area of event analysis, ReliabilityFirst works with entities to identify and analyze the root

causes of system events, complete event analysis reports, and communicate the resultant information and

lessons learned to the industry. The 97 events analyzed by ReliabilityFirst in 2016 were all either lower

level, Category 1 events with no loss of load, or non-categorized events (Category 0). The chart below

shows the high level root causes associated with the 2016 events in the Region.

Voluntary Maturity Model Engagements ReliabilityFirst conducts maturity model engagements on a voluntary basis, where it identifies

and evaluates: (1) high-level risks faced by the entity; and (2) the entity’s performance and capability to

manage those risks (see Figure 5 below). Maturity model engagements are valuable to entities because

they are proactive engagements that focus on continuous improvement in pursuit of reliability excellence.

This non-compliance focused venue provides a collaborative environment for the entity to work with

ReliabilityFirst staff to understand entity (and department) specific challenges, and discuss and apply best

practices.

In 2016, ReliabilityFirst worked with NERC and the other Regions to further enhance and mature

the maturity model engagement process. In connection with this work, ReliabilityFirst further developed

its staff expertise, and refined its process to provide a more user-friendly and pragmatic deliverable.

ReliabilityFirst also continued to educate entities on the value of the maturity model engagement process.

Progress will continue in 2017, as several entities have requested and are in the queue for maturity model

engagements. ReliabilityFirst will use these engagements to deliver value to entities, and to continuously

improve engagements going forward.

Figure 4: 2016 Events in ReliabilityFirst Region The most prevalent high-level root causes of events were (1) physical threats; (2) EMS

related; (3) control center evacuations; and (4) weather.

0 5 10 15 20 25 30 35 40 45

Control Center EvacuationEMS Related

Generation Loss - Equipment FailurePhysical Threat

Transmission Event - Breaker FailureTransmission Event - Inadvertent Operation

Transmission Event - MisoperationTransmission Event - Misoperation and Breaker Failure

Weather

2016 Events in ReliabilityFirst Region

6

Management Practices for Maturity Model Engagements

Reliability Assessment and Performance Analysis Activities

ReliabilityFirst’s reliability assessment and performance analysis activities identify key risks

facing the Region: these activities include summer and winter seasonal assessments; near-term and long-

term transmission assessments; a long-term resource assessment; and four confidential extreme power

flow analyses.

Seasonal Assessments

For each upcoming summer and winter season, ReliabilityFirst reviews the projected resource

adequacy and transmission performance for PJM Interconnection, LLC (PJM) and Midcontinent

Independent System Operator (MISO), the two Regional Transmission Organizations that operate within

the Region. In the 2016 Summer Seasonal Reliability Assessment of Demand, Resources, and

Transmission System Performance, ReliabilityFirst concluded that the Region was projected to have

sufficient resources and that the transmission system could be operated reliably in summer 2016.

Likewise, in the 2016/2017 Winter Seasonal Reliability Assessment of Demand, Resources, and

Transmission System Performance, ReliabilityFirst concluded that the Region was projected to have

sufficient resources and that the transmission system could be operated reliably in winter 2016/2017.

Additionally, ReliabilityFirst conducted confidential summer and winter transmission assessments on: (1)

the operating performance for the 2015 seasons, (2) the expected operating performance for the 2016

seasons, and (3) the projected operating performance for the 2017 seasons.

Figure 5: Management Practices Management practices are groupings of internal controls for which successful implementation would likely result

in enhanced reliability and operations. These are natural groupings of functional activities that entities already

perform to ensure the reliability, resiliency, and security of their respective systems. The management practices

are grouped by theme: for example, entities can follow the “Planning” management practice for managing

projects. For additional information on the Management Practices and the Maturity Model Evaluation Process,

contact ReliabilityFirst.

Managing Goals and

Metrics

Reliability Quality

Management

Measurement and Analysis

Managing Risks

Risk Management

External Inter-dependencies

Structured Decision Making

Managing Assets

Information Management

Asset and Configuration Management

Managing Projects

Work Management

Planning

Managing Technical

Work

Implementation

Integration

Verification

Validation

Managing People

Grid Operations

Grid Maintenance

Workforce Management

7

Near-Term and Long-Term Transmission Assessments

In 2016, ReliabilityFirst conducted its annual confidential near-term and long-term transmission

assessments which summarize the projected performance within ReliabilityFirst’s footprint, and leverage

MISO Transmission Expansion Planning efforts and PJM’s Regional Transmission Expansion Plan

efforts.

Long-Term Resource Assessment

In the annual long-term resource assessment, ReliabilityFirst reviewed the future demand and

capacity resource balance, and analyzed the amount of capacity resource reserves compared to the target

reserves to determine excess or shortage in expected planning reserves for the future summer peak

demands. The Long Term Resource Assessment 2017-2026 reflects significant retirements in the

ReliabilityFirst footprint. ReliabilityFirst concluded that the projected reserve margins for PJM are

31.1% in 2017 and 26.2% in 2026, which satisfy PJM’s reserve margin requirements due to projections of

new capacity to replace generation retirements and satisfy load growth.

The projected reserve margins for MISO are 18.1% in 2017 and 9.1% in 2026, which are

adequate to satisfy MISO’s reserve margin requirements through 2021. However, the projected reserve

margins are 1.3% below MISO’s reserve margin requirement in 2022 and continue to decline throughout

the remainder of the assessment period (the reserve margin is 1,641 MW below the target reserves in

2022), due to the amount of projected new generation not being sufficient to offset projected capacity

reductions and load growth. ReliabilityFirst believes that there are several ways some of this reserve

margin deficit can be mitigated in MISO, such as an increase in Demand Side Management programs and

import transactions. Additionally, 5,434 MW of Tier 2 generation is expected to be online by 2022 (only

50% of this Tier 2 generation is included in the prospective reserve margin as a conservative measure).

As generator projects are being built, they will help satisfy the reserve margin requirements.

ReliabilityFirst will continue to closely monitor these resource adequacy issues. Click on the reports below to review:

8

Risk mitigation is just as critical to ensuring reliability as risk identification – once reliability

risks are identified, ReliabilityFirst utilizes various tools to work with industry to mitigate these identified

risks. These tools include compliance monitoring and enforcement, Reliability Standard commenting and

development, and registration and certification activities.

Compliance Monitoring and Enforcement Activities

In 2016, ReliabilityFirst tailored its compliance monitoring activities to ensure entities were

successfully addressing those Reliability Standards that map to and mitigate the Risk Elements identified

in the Regional Risk Assessment. This work included performing guided self-certifications, audits, and

spot checks; resolving complaints; and conducting reliability and compliance assessments of system

events. These monitoring activities are key to reliability, as they ensure that entities have effective

controls in place and are following the requirements of the Reliability Standards, especially those that

map to the Risk Elements.

ReliabilityFirst completed 30 Operations & Planning Audits and eleven CIP audits in 2016, all of

which were specifically scoped and tailored around the identified Risk Elements. During these varied

monitoring engagements, ReliabilityFirst identified five possible violations of the Operations & Planning

Reliability Standards and 12 possible violations of the CIP Reliability Standards. It also investigated and

resolved two complaints and reviewed 133 compliance assessments of system events.

The chart below depicts the most frequently identified possible violations during compliance

monitoring engagements in 2017. Although none of the possible violations constituted a serious risk to

the BES, it is important to be familiar with the below chart because the violations concern activities that

map to identified Risk Elements, and were not self-detected by the entity. Accordingly, entities must

remain vigilant to ensure that small issues are not permitted to develop into larger risk issues negatively

impacting the reliability of the BES.

0

10

CIP-007 PRC-005 CIP-005 CIP-010 EOP-005 PRC-001

Most Violated Reliability Standards at2016 Compliance Audits and Spot Checks

Risk Mitigation

Figure 6: Most Violated Reliability Standards at 2016 Compliance Audits and Spot Checks CIP-007 was the most violated Reliability Standard, which concerns patch management. ReliabilityFirst encourages

entities to carefully review their internal controls concerning patch management given its importance in protecting

cyber systems from known vulnerabilities.

9

2016 marked the first year ReliabilityFirst began to regularly utilize Guided Self-Certifications in

a targeted manner. Guided Self-Certifications prompt entities to self-assess their state of compliance with

specific Reliability Standards and report any deficiencies discovered during their review. In 2016,

ReliabilityFirst conducted two CIP Guided Self-Certifications, for CIP-002-5.1 and CIP-014-2; and three

Operations and Planning Guided Self-Certifications, for EOP-010, PRC-005, and VAR-001. In an effort

to make the process as useful and simple as possible, ReliabilityFirst recently created a Guided Self-

Certification Guide for entities, which can be found here.

In addition to compliance monitoring, ReliabilityFirst is responsible for the appropriate, risk-

based resolution of all identified noncompliances in its Region, including compliance monitoring

findings, self-reports, self-logs, and complaints. Commonly referred to as “enforcement,” this activity

focuses on understanding the risks behind each noncompliance and how to effectively mitigate those

risks, as well as sending the appropriate message to the noncompliant entity and the broader regulated

community (whether deterrent-driven for undesired behavior or incentive-driven for desired behavior).

Noncompliance resolutions range from compliance exceptions for low risk matters to settlement

agreements that may include significant monetary penalties and sanctions, but also afford a collaborative

opportunity to address more serious risk challenges through the implementation of valuable reliability

enhancements and sustainable programmatic approaches.

In 2016, ReliabilityFirst processed 148 noncompliances, of which 100 were compliance

exceptions (including self-logged compliance exceptions). As shown in Figures 7 and 9 below, an

impressive 94% of noncompliances were self-reported by the entities and there has been a year-over-year

decrease in the time from the start of a noncompliance to the date it is reported, which demonstrates

improving detective controls.

Since the majority of the resolved violations concern the CIP Standards (see Figure 8), below are

figures illustrating identifiable trends from these violations. In general, the trends indicate the continued

maturation of compliance and security programs due to a year-over-year decrease in violation volume and

severity (see Figure 10), and improved self-detection and self-reporting, due to the year-over-year

decrease in the time from the commencement of a violation to the date it is reported (see Figure 11).

94%

6%

2016 Self-Reported/Audit Findings

Self-Reports/Self-Logging

Figure 7: 2016 Self-Report/Audit

Findings

94% of violations in 2016 were self-

reported. ReliabilityFirst applauds this

proactive behavior.

Figure 8: CIP vs. Operations &

Planning (O&P)

65% of violations concerned the CIP

Standards. O&P violations were a bit

higher than usual due to low risk

implementation challenges with new or

revised Standards (e.g., MOD-025,

PRC-019, and PRC-024).

65%

35%

2016 CIP/Operations & Planning

CIP Operations & Planning389.34

297.26

287.10

212.31

90.41

0 200 400 600

2012

2013

2014

2015

2016

YE

AR

ID

EN

TIF

IED

All Noncompliances: Average Days from

Noncompliance Start Date to Report Date

Figure 9: Duration of Violations

The overall duration of violations

(from start date to report date) is

decreasing over time.

10

As shown in Figure 13, the majority of CIP violations in 2016 were driven

by a few Standards that govern high frequency conduct. In certain situations, such

as entities with large workforces spread across numerous business units with

voluminous assets, these violations can be indicative of entities with strong

detective controls that are able to promptly detect, correct, and self-report issues.

However, they can also be indicative of entities with systemic programmatic

challenges negatively impacting their security posture. Either way, the data

demonstrates the need for entities to be vigilant in establishing strong preventative,

detective, and corrective controls to safeguard their systems and ensure that their

strengths do not become weaknesses, and weaknesses are not exploited.

437.46

241.31

277.19

220.57

103.67

0 200 400 600

2012

2013

2014

2015

2016

YE

AR

ID

EN

TIF

IED

CIP Noncompliances: Average Days from

NoncomplianceStart Date to Report Date

2010 2011 2012 2013 2014 2015

Minimal 144 84 100 59 77 50

Moderate 75 44 28 12 8 3

Serious 14 16 19 9 1 3

0

50

100

150

200

250Volume and Severity

of CIP Noncompliances*

Figure 13: CIP Violation Drivers High frequency conduct violations continue to drive CIP Violations, but are trending downward over time.

The CIP violations that did occur in 2016 were driven by violations involving high frequency conduct (i.e., CIP-

004, R4: lists for cyber and physical access; CIP-006, R1: physical access logging; and CIP-007, R5: passwords),

and by violations associated with general CIP V5 migration issues.

Figure 10: Volume and Severity of CIP Violations over Time

Both the volume and severity of CIP Violations has decreased.

*2016 data is not included, as risk determination for violations

that started in 2016 generally takes place in 2017.

Figure 11: Violation Start Date-Report Date

The average duration of violations is decreasing over

time, with entities identifying and self-reporting

violations more quickly.

0

50

100

150

2012 2013 2014 2015 2016

CIP Violation Drivers

CIP-004/CIP-006/CIP-007 Remaining CIP

Figure 12: Internal controls help to

detect and correct smaller, low risk

issues before they become higher

risk issues.

11

Regional Coordination Activities

The Multi-Regional Registered Entity (MRRE) Coordinated Oversight program is intended to

improve communication and streamline compliance monitoring and enforcement activities for entities

that cross regional boundaries. Under the program, a Lead Regional Entity is selected to coordinate the

performance of all activities for the MRRE. In 2016, ReliabilityFirst served as the Lead Region for 34%

of the 193 MRREs in North America, and successfully took the lead in processing 59 MRRE violations.

Standards, Registration, and Certification Activities

In the Standards area, ReliabilityFirst provides input to the NERC Reliability Standards

development process and maintains regional Reliability Standards as appropriate. The purpose of this

activity is to ensure that Reliability Standards adequately and thoughtfully mitigate the risks they are

intended to address. In 2016, ReliabilityFirst analyzed, voted on, and provided feedback on 34 NERC

Reliability Standards, and facilitated the process to revise the ReliabilityFirst Regional Standard BAL-

502-RF-03 (Planning Resource Adequacy Analysis, Assessment and Documentation) to address two

FERC directives. ReliabilityFirst also participated in the creation of a supply chain management Standard

to address the unique risks that arise in that area.

In the registration area, ReliabilityFirst facilitated the registration of 22 new entities on the NERC

Compliance Registry and deregistration of 21 entities in 2016.4 The purpose of this activity is to ensure

that entities necessary for the reliable operation of the BES are registered appropriately. The majority of

registration changes involved the Generator Owner/Operator functions, largely due to the addition of

wind and solar farms, or the transfer of generator assets due to larger entities selling, merging, or

consolidating assets. ReliabilityFirst processed three Self-Determined Notifications and one Exception

Request to the Revised BES Definition in 2016.5 While there was a decline in the volume of Self-

Determined Notification and Exception Requests 2016, the activity that did occur in this area was

resource-intensive.6

4 ReliabilityFirst also reviewed and updated 12 Coordinated Function Registration (CFR) agreements, and reviewed

and updated 7 Joint Registration Organizations (JRO). This responsibility includes working with the lead entity,

applicable Regions, and NERC to verify that the agreement provides for an allocation or assignment of

responsibilities consistent with the JRO or CFR. 5 On March 20, 2014, FERC approved the revised BES definition, which includes bright-line core criteria with

various enumerated inclusions and exclusions. At the same time, FERC approved a process for entities to request

proposed exceptions from the revised BES definition on a case-by-case and element by element basis (referred to as

Exception Requests), and a process for entities to notify Regions when they determine specific elements no longer

fall within the revised BES definition (referred to as Self-Determined Notifications). 6 ReliabilityFirst also performed four Certification reviews in 2016, to ensure that entities applying to perform the

critical Balancing Authority, Transmission Operator and/or Reliability Coordinator reliability functions are capable

3.45%11.03%

0.69%

4.83%

0.69%

4.83%

1.38%

60.69%

0.69% 4.83% 6.90%

2016 Multi-Regional Violations RF/SPP

RF/SPP/TEXAS

RF/SPP/WECC

RF/NPCC

RF/MRO

RF/MRO/SPP

RF/MRO/NPCC

Figure 14: Breakdown of Multi-Regional

Violations Processed in ReliabilityFirst

In 2016, ReliabilityFirst served as the Lead

Region for 34% of the 193 MRREs in North

America. Processing MRRE violations requires

additional internal coordination with the other

Affected Regions, as well as processing additional

violations that occur in the Affected Regions.

12

ReliabilityFirst believes it is critical to not only identify and mitigate risks, but also to timely and

effectively communicate these identified risks and corresponding mitigation strategies across the industry

to facilitate awareness. ReliabilityFirst is also committed to sharing its expertise, and leveraging the

expertise of its entities, to advance industry practices surrounding risk identification, mitigation, and

prevention. As such, ReliabilityFirst continued to focus on and expand its risk communication activities in

2016, which include the Assist Visit program; the ReliabilityFirst newsletter; compliance bulletins and

monthly outreach calls; and a variety of workshops and training events.

Assist Visit Program

The Assist Visit program is a voluntary program pioneered by ReliabilityFirst, which provides

tailored training centered on the needs of the entity and key risks they are facing.7 Because Assist Visits

are tailored to the needs of the entity, the engagements are dynamic and can range from recurring

engagements to address significant programmatic improvements, to multi- or single-day onsite

engagements, to a simple conference call. The Assist Visit program has received overwhelmingly

positive feedback from entities, including the following:

of performing those functions. It performed one certification for a newly registered Transmission Operator; two for

newly registered Transmission Owners in the PJM footprint; and one for a newly constructed Control Center. 7 Entities can seek Assist Visits for various reasons, such as requesting guidance on the Reliability Standards;

requesting feedback on how to improve their compliance program; or discussing how to best handle changes in

operations. Entities can request an Assist Visit here.

Risk Communication

“The guidance ReliabilityFirst

provided through their Assist Visits

has been very helpful in ensuring that

our organization took appropriate

measures while transitioning to the

CIP v5 Standards.”

- Anonymous Participant

“Through the Assist Visit program with

RF, NRG conducted a coordinated

Readiness Assessment of the NRG CIP

program in preparation for the V5 CIP

standards transition in 2016. The

opportunities for NRG to interface with RF

for these efforts were tremendously

valuable to NRG SMEs and provided key

understanding aspects to the

implementation of the NRG V5 NERC CIP

program...”

– NRG

13

In 2016, ReliabilityFirst performed 35 Assist Visits, most of which related to the CIP Standards.

ReliabilityFirst seeks to share generic and anonymized lessons learned from Assist Visits when

possible, and utilizes its newsletter, workshops, and other engagements to do so.

In addition to its Assist Visit Program, ReliabilityFirst conducted individualized, on-site entity

training sessions on its Risk-Harm Assessment process. The Risk-Harm Assessment is a uniform,

repeatable process by which ReliabilityFirst quantifies the risk and potential harm posed by a possible

violation through a series of questions answered by ReliabilityFirst subject matter experts. Understanding

the severity of the risk and harm posed by a particular violation is not only fundamental to accurately

identifying a risk, but also critical to understanding how to effectively mitigate it. Accordingly,

ReliabilityFirst shares its Risk-Harm Assessment process to not only strengthen its entities’ approaches,

but also to receive feedback from entities to strengthen its own process. An additional positive byproduct

of these engagements is that it allows entities to better understand ReliabilityFirst’s approach, which

facilitates enhanced dialogue in prospective engagements.

Newsletters, Workshops, and General Training Engagements

Throughout 2016, staff from across the organization worked together to

issue the bi-monthly ReliabilityFirst newsletter, in an effort to highlight identified

risks and leading practices to address those risks. The newsletter provides an

effective vehicle to extrapolate generic lessons learned from not only the Assist Visit

program, but also from the varied risk identification and mitigation activities and

engagements across the entire ReliabilityFirst organization. Also throughout 2016,

ReliabilityFirst issued monthly compliance bulletins and conducted monthly

outreach calls to discuss and provide guidance on current and emerging reliability

and compliance topics.

ReliabilityFirst continued to hold its annual fall and spring reliability workshops, and fall and

spring CIP workshops in an effort to share identified risks, mitigation approaches, and compliance

The RF Newsletter

CIP General

Cyber System Categorization

(CIP-002)

Security Management Controls (CIP-003)

Electronic Security Perimeters (CIP-005)

Physical Security (CIP-006)

System Security Management

(CIP-007)

Configuration Change Management and

Vulnerability Assessments (CIP-010)

Physical Security (CIP-014)

Operationsand Planning

Assist Visits Conducted in 2016

Figure 15: Breakdown of Assist Visits Conducted in 2016

Over 3/4th of Assist Visits conducted in 2016 concerned topics surrounding the CIP Standards. Nearly half

involved either general CIP-related questions, or questions about cyber system categorization under CIP-002.

14

strategies. Workshop attendance was strong, exceeding 250 attendees during each session. Workshop

highlights included:

ReliabilityFirst also conducted numerous in-person webinars, events, and workshops over the

course of the year. For example, in May, ReliabilityFirst held a “case study” webinar to share

anonymized themes and root causes associated with an entity’s recent systemic CIP compliance issues,

and pathways to successful CIP security and compliance. 362 attendees participated in this engagement

from across the country and around the world: its materials can be accessed here. 90% of surveyed

respondents from this engagement rated the case study webinar as excellent.

Targeted Outreach and Training Engagements

In 2016, ReliabilityFirst held various risk communication activities

in an effort to work with its entities to help mitigate the identified Risk

Element of Protection System Failures. For example, ReliabilityFirst held its

second annual Substation Protection Workshop for Field Personnel. In an

effort to influence a decrease in relay misoperations, the workshop focused

on protection system commissioning and testing, with topics including: (1)

protection system misoperations metrics; (2) how to use relay event analysis

during commissioning and following power system interruptions to

determine corrective and preventive actions; (3) relay testing techniques; and

(4) group breakout sessions to discuss the strategic use of contractors, field

familiarity with new relays and test processes, handling field revisions, and

effectively trip testing composite protection schemes. In a similar effort, ReliabilityFirst sponsored a

training session from Schweitzer Engineering Laboratories University on setting ground relays.

In October, ReliabilityFirst hosted its first annual Generator Owner/Operator Workshop for Plant

Personnel, which focused on generating plant issues. Presentation topics included: (1) generating plant

preparations for summer and winter weather; (2) generator frequency response issues and the ERO’s

actions to address these issues; (3) reporting via the NERC Generator Availability Data System (GADS);

and (4) cyber security for industrial control systems. Materials from the Generator Owner/Operator

workshop can be accessed here.

To assist entities in preparing for winter 2016-2017, which concerns the Risk Element of Extreme

Physical Events, ReliabilityFirst conducted several generating facility visits. Specifically, cross

functional winter preparedness teams across the organization visited generating facilities that experienced

PSEG Fossil sharing its practices surrounding generator plant winter readiness

A presentation on supply chain risk management challenges and internal controls

A presentation on common trends and themes in CIP violations

Vectren and FirstEnergy sharing their respective transitions and approaches to CIP V5 compliance

NERC presenting the auditing approach for Low Impact BES Cyber Systems

FERC staff leadership sharing its thoughts on the evolving and maturing approach to reliability compliance

PJM sharing how to implement a continuous improvement plan

EDPR Renewables North America sharing its CIP V5 program

Navigant Consulting sharing trends and leading practices for Physical Security Plans

Exelon sharing its strategy to foster a culture of compliance and reliability

15

megawatt losses of greater than 500 MW due to cold weather related issues

during the 2015-2016 winter period, and new facilities that had not yet

operated during cold weather periods. Each of these visits included

discussions with the entity of winter preparedness issues in further detail, a

review of records related to the entity’s winterization plan implementation,

and a walk-through of areas of the entity’s facilities that may be exposed to

extreme weather conditions.

As a result of these visits, ReliabilityFirst’s winter preparedness

teams collected lessons learned, best practices, and positive observations, a

summary of which can be found here. Examples of best practices observed

during the visits include: using a portable diesel generator; sharing lessons

learned across the generation fleet; utilizing lubricating oil with a lower

operating point; and ensuring facility-wide situational awareness of extreme

weather events.

ReliabilityFirst performed numerous other cold weather preparedness outreach activities in 2016,

including entity information sharing and presentations by PJM and MISO during ReliabilityFirst’s

Reliability Committee meetings; presentations by ReliabilityFirst staff during the spring reliability

workshop, PJM Operating Committee, and generator workshop; and ReliabilityFirst’s 2016-2017 Winter

Seasonal Reliability Assessment of Demand, Resources, and Transmission System Performance.

Creation of Generator Subcommittee

In response to overwhelming interest from Generator Owners

and Operators, ReliabilityFirst created the Generator Subcommittee in

2016. The mission of the Generator Subcommittee is to leverage and

communicate the expertise of Generator Owners and Operators in the

Region in an effort to strengthen their reliability, security, and

resiliency.

In 2016, the Generator Subcommittee’s activities focused on

cold weather readiness; understanding issues identified in the

Generator Availability Data System; and driving value from upcoming

NERC surveys and information sharing efforts.

Figure 16: During a winter preparedness

visit, ReliabilityFirst observed an entity’s

use of temporary heating and ducting, to

prevent the formation of ice and snow on air

inlet filters.

ReliabilityFirst Member Companies

Linden VFT, LLC

Pennsylvania Office of

Consumer Advocate

Hazelton Generation

LLC

Darby Energy, LLLP

Forward Together • ReliabilityFirst