Powering forward. Together.

October 16, 2014

CIP V5 Implementation Study

SMUD’s Experience

Tim Kelley

SMUD – Fast Facts

2

General Information

SMUD employs approximately

2,000 individuals

Service area of 900 square miles

Population served is 1.4 million

~625,000 customers

477 miles of transmission

Peak Load (MW):

3,300 (SMUD), 5,000 (BANC)

Generation Specifics

1,000 MW of thermal generation

(9 BES Units)

688 MW Hydro

(7 BES Units)

100 MW of solar generation

230 MW of wind generation within

the California ISO

NERC Registrations

TOP, TO, GO, GOP, TSP, TP, PA, RP, DP, PSE, LSE

- Also performs BA reliability compliance for the BANC

Study Participants

3

Overview of CIP Standards

Critical Infrastructure Protection (CIP) Standards:

CIP-002-5 BES Cyber System Categorization

CIP-003-5 Security Management Controls

CIP-004-5 Personnel and Training

CIP-005-5 Electronic Security Perimeter

CIP-006-5 Physical Security of BES Cyber Systems

CIP-007-5 System Security Management

CIP-008-5 Incident Reporting and Response Planning

CIP-009-5 Recovery Plans for BES Cyber Systems

CIP-010-1 Configuration Mgt. and Vulnerability Assessments (new, V5)

CIP-011-1 Information Protection (new, V5)

4

V3 to V5 Changes

Version 3 Version 5 Version 3 Version 5 High Impact (control centers)

*Primary Control Center

*Backup Control Center

*Distribution Control Center (new)

Medium Impact (substations)

*Substation #1 (new)

Substation #2 (new)

Substation #3 (new)

Substation #4 (new)

(* included in V5 Study scope)

5

V5 Major Impacts – Cyber Security

BES Cyber Assets increased from 119 to 391 devices (228% )

Evidence requirements for CIP-007 increased:

From 3,332 to 10,948 pieces

Firewalls and cyber monitoring at substations (PSP, ESP, EAP, EACMS)

Patch Management:

Assess all security patches for all assets every 35 days

Installed in test environment, security scans performed

In v3 - patches applied on 6-9 month cycle

Logging:

Review every 15 days

Configuration management every 30 days (annually in v3)

6

V5 Major Impacts – Physical Security

150 to 250 additional employees under CIP-004

training and PRAs now required

Substation relays and RTUs are now in scope

Badge readers at the substations

Dual authentication at the control centers – badge

readers and PIN-pads

Access to cyber assets removed within 24 hours instead of 7 days.

7

Study Timeline and Beyond

Key Dates and Goals:

July 31, 2014 Study Milestones Completed

Oct. 13, 2014 Study Report Released

January 1, 2015 V5 Compliant at PCC, BCC, DCC, (1) MI Substation

July 1, 2015 V5 Compliant at Remaining 3 Medium Impact Substation

April 1, 2016 V5 Effective & Enforceable

8

CIP-002-5

BES Cyber System Categorization

What is a BES Cyber Asset (BCA)?

BCA definition – “Cyber Asset that if rendered

unavailable, degraded, or misused would, within 15

minutes of its required operation, misoperation, or

non‐operation, adversely impact one or more Facilities,

systems, or equipment, which, if destroyed, degraded, or

otherwise rendered unavailable when needed, would

affect the reliable operation of the Bulk Electric System.

Redundancy of affected Facilities, systems, and

equipment shall not be considered when determining

adverse impact. [more…]”

10

What is a BES Cyber Asset (BCA)?

Recommend you define what is a Cyber Asset

What? Cyber Asset is already defined, right?

Cyber Asset definition – “Programmable electronic devices,

and communication networks including the hardware,

software, and data in those devices.”

Recommend you define what is a “programmable device”

Lots of discussion around differences of “programmable”

and “configurable”

SMUD’s definition of “programmable” = Anything with a

microprocessor in it

11

BCS Categorization Process

Keep It Simple! SMUD’s process includes 3 documents”

Procedure (only 5 pages)

Facilities Analysis (spreadsheet)

BES Cyber Asset List (spreadsheet)

Steps:

Complete a list of SMUD’s assets that impact BES

Apply Attachment 1 IRC to list to determine facility levels

For all High and Medium Impact control centers:

List all Cyber Assets (CA) in the host file used by the EMS

Scan each network in host file for devices not already listed

Perform physical inspection at each MI, HI control center

12

Facilities Analysis

13

BCS Categorization Process (cont’d)

MI facilities that are not control centers (substations and

generating plants) – inventory all CAs in control bldg.

Determine CAs from preliminary list that are BCA

Criteria used for this determination is the applicability of BES

Reliability Operating Services along with the definition of a

BES Cyber Asset – specifically “…that if rendered

unavailable, degraded, or misused would, within 15 minutes

adversely impact the reliable operation of the BES.”

Determine each CA from preliminary list that are:

PCA, EACMS, or PACS

Associate BCAs, PCAs, EACMSs and PACs to the

appropriate BES Cyber System (in following list:)

14

BCS Categorization Process (cont’d)

In general, BCS are

large groupings of Cyber

Assets

One BCS per asset (i.e.):

PCC BCS

Substation 1 BCS

Substation 2 BCS

Entity has flexibility to

create/group their

Cyber Assets into BCS

as they see fit

15

Non-BCA Examples – Pi Historian

Pi Servers “push data” (one direction only)

Pi data serves to augment functions within the control center, used to create other views and nice visualizations

Evidence stacking:

Real-time decisions are not made using Pi data

No alarm summaries on Pi

Everything displayed on Pi is already in the EMS

Operators trained to verify Pi displays with EMS console

Caution: Could be considered BCA if operators use the data for real-time decision-making or situational awareness

16

Non-BCA Examples – Control Room Wallboards

EMS servers push wallboard data to a server in DMZ

Data is then pushed to wallboard display servers on

corporate network

Operating procedures call for failures to be addressed on

“next-business day”

Not used for system control (no touch-screen capability,

cannot operate BES elements from the board)

Transmission system fits onto one EMS console screen

17

Non-BCA Examples – OATi webTRANS

SMUD does not utilize locally-staged scheduling software

– uses OATi webTRANS

All individual schedules are handled through e-tags

Operations does not enter any schedules; power

marketing group does

OATi in Minneapolis consolidates data they receive into

interchange numbers

OATi webTRANS is not a BCA

18

V5 Study Lessons Learned

Introducing CIP Compliance to Newbies

“Newbies” – substation and generation facilities with no prior

CIP experience (no Version 3 CCAs)

SMUD treated this as a separate project for CIP-004 & 006

Things to consider:

Communications – emails, signs, meetings, tailgates, intranet

Training – V5 revised, new assets, new personnel, role based

PRAs – Scheduling, labor agreements, communications

2 Factor Authentication – Installation, programming (PIN & thumbs)

Visitor Control Program – communications

Shared Facilities – communicate, vet outside personnel (how?)

Timing of Everything – create a detailed schedule

20

21

V5 Documentation - Procedure Template

EMS

Substation Real Time (RTUs and associated equipment)

Relays and Communication Processors

Jump Hosts (EACM to the listed BES Cyber Systems)

EACM devices, other than Jump Hosts (firewalls, routers and

switches, Ciscoworks, ACS)

IDS devices, SIEM collectors & associated Mgt. Consoles

Active Directory Servers at PCC and BCC

PACS System & Door Panel Controllers

Revenue Meters – No ERC

Emergency Backup System RTU – No ERC

22

Devices “Directly” Accessed through ERC

Background:

ERC (External Routable Connectivity)

Definition of Medium Impact BCS with ERC:

“Only applies to medium impact BES Cyber Systems with

External Routable Connectivity. This also excludes Cyber

Assets in the BES Cyber System that cannot be directly

accessed through External Routable Connectivity”.

23

Devices “Directly” Accessed through ERC

Question: For protection relays in a BES Cyber System that are serially connected to a router/protocol converter and the router/protocol converter has External Routable Connectivity, are the relays themselves considered “Cyber Assets in the BES Cyber System that can be directly accessed through External Routable Connectivity”?

Answer: Yes, the protection relays would be considered Cyber Assets with External Routable Connectivity (ERC). If they’re connected to the router/protocol converter and they can be accessed “outside of its associated Electronic Security Perimeter via a bi-directional routable protocol connection,” it doesn’t matter if they are serially connected. A protocol converter cannot be used to avoid compliance. If the relay can be accessed and its state can be changed through any means using a “bi-directional routable protocol connection,” then it is considered to have ERC.

24

Devices “Directly” Accessed through ERC

25

• If you can connect to

and change the relay

settings from a routable

protocol connection

(I/P), the relays are to

be treated as having

ERC

• CAUTION: Lesson

Learned is under review

by CIP V5 Advisory

Group

Impact Ratings of Cyber Assets and Facilities

Using a Shared EMS

Background:

The entity has a single Energy Management System

(EMS) that services both transmission and distribution

operations.

The Distribution Operations Control Center (DOCC)

located inside the entity’s Distribution facility does not

control any BES elements, however, the DOCC shares

the same EMS as the Primary Control Center (PCC)

which is classified as a High Impact facility.

The entity has identified its EMS at the PCC as a BCS.

26

Impact Ratings of Cyber Assets and Facilities

Using a Shared EMS

Question: In this case, are the EMS DOCC Human Machine

Interface (HMI) consoles classified as High impact BES Cyber

Assets as part of the main EMS?

Question: If so, how is the balance of the Distribution facility,

outside of the DOCC, evaluated?

Answer: In this case, the HMI consoles at the DOCC use the same

EMS as the PCC and it is only logical configuration that prevents a

distribution operator from performing transmission operations.

Therefore, due to the connectivity and possible misuse of the DOCC

HMI consoles, these Cyber Assets should be treated as High Impact.

The High Impact rating applies even though the Cyber Assets at the

DOCC and PCC have separate Physical and Electronic Security

Perimeters.

27

BES Cyber System (BCS) boundaries

Question: Can a BCS span multiple facilities and locations?

28

Simple rules for BCA, BCS, and PSP

Background:

An entity has a Medium

Impact substation that

contains a Protection

System BES Cyber

System (BCS) and a

single BES Cyber Asset

(BCA).

The “single BCA” has no

routable connectivity and

is not part of the Protection

System BCS.

29

Simple rules for BCA, BCS, and PSP

Question: Does the “single BCA” need to be associated

with a BES Cyber System (BCS)?

Answer: Yes. Every BCA must be associated with a

BCS. A BCS can also contain just one BCA. Therefore,

in this case, the entity may create a separate BCS that

only contains the “single BCA”, or it may associate the

“single BCA” with the Protection Systems BCS. If the

entity chooses the later option, the “single BCA” must be

protected as a BCA with no ERC and not as a Protected

Cyber Asset (PCA) inside the ESP.

30

Simple rules for BCA, BCS, and PSP

Question: Does the “single BCA” need to be inside an

Electronic Security Perimeter (ESP)?

Answer: No. A cyber device with no routable connectivity,

external or otherwise, cannot be inside an ESP.

Question: Does a BCS have to reside entirely within an

Electronic Security Perimeter (ESP)?

Answer: No. A BCS may have Cyber Assets outside of an

ESP. A BCS can contain BCAs in multiple ESPs. A BCS may

contain BCAs in multiple PSPs. However the BCS is defined,

it must meet the CIP V5 Standards at the system level for all of

its component BCAs.

31

Simple rules for BCA, BCS, and PSP

32

CIP-004 R3 – Existing PRAs

Question: Do existing Personnel Risk Assessments

performed under CIP-004 Version 3 need to be redone under

Version 5 by April 1, 2016 to meet compliance with the new

seven year criminal history records check requirements?

Answer: No. As long as the background check has not

exceeded the seven year requirement, there is no need to do

it again. All PRA completed prior to April 1, 2016 that are

compliant with CIP-004 Version 3 will be “grandfathered” in

under Version 5 as compliant.

33

Questions

34