CIP Workshop - Home - Southwest Power Pool spp re cip workshop... · SPP CIP Workshop . 2 . RELIABILITY ... •Reliability Standards Audit Worksheets (RSAWs) will be posted by June

CIP Workshop

Internet Join the SPPGuest network. Open your internet browser and enter your email address (no password required) on the Guest User page. Restrooms & Vending Machines From the auditorium, go left, then left again at the next hallway.

More restrooms are behind the atrium stairway. Break Room with Tables Other side of the vending machine wall Designated Smoking Area Outside the large break room Business Center Behind the reception desk. Ask a staff member for assistance with copies or faxes. A PC is set-up for printing boarding passes.

2

NERC 1Q 2014 Vegetation Management Report

• No reportable contacts in SPP RE footprint

• 4th consecutive quarter with no reportable contacts

3

NERC Facility Ratings Alert as of 12-31-13 SPP National

High Priority 345 kV+

~420 discrepancies 99% remediated

7,966 discrepancies 88% remediated

Medium Priority 230 kV – 345 kV

~1,980 discrepancies 38% remediated


Low Priority Below 230 kV

~3,630 discrepancies 24% remediated


4

Remediation to be completed within one year of discovery or end of 2014, whichever is earlier, unless under an extension request

SPP RE Misoperation Report as of 4Q 2013

5

6

Most Violated Standards Based on rolling 12 months through 5/31/14 [Represents ~ 84% of total violations]

* NERC Report Q4 2013 ** Not in NERC Rolling 12 month Top Ten

SPP RE

Rank

NERC 12 Month

Rank * Standard Description Number of

Violations Risk Factor

1 1 CIP-007 Systems Security Management 28 Medium

2 10 FAC-008 Facility Ratings (includes FAC-009) 23 Med./Lower

3 3 CIP-005 Electronic Security Perimeters 18 Medium

4 2 CIP-006 Physical Security - Critical Cyber Assets 13 Med./Lower

5 4 PRC-005 Protection System Maintenance 12 High/Lower

6 5 CIP-004 Personnel & Training 8 Med./Lower

7 6 CIP-002 Critical Cyber Asset Identification 6 High/Lower

8 8 VAR-002 Network Voltage Schedules 6 Med./Lower

9 7 CIP-003 Security Management Controls 6 Med./Lower

10 ** PRC-008 UFLS Relay Maintenance 5 Medium

SPP RE Regional Events - 1Q 2014 • Eight total events

• Four did not meet NERC Event Analysis threshold

• Four Category 1 Events were analyzed – Three Category 1h Events - Loss of monitoring or control

at a control center

– One Category 1a Event - Unexpected outage of three or more BPS facilities contrary to design

7

New BES Definition Effective July 1

• NERC BESNet tool now open for registration

• BESNet opens July 1 for submitting Exception Requests & Self-Determination Notifications

• SPP RE Webinar June 12

• Contact Greg Sorenson for more info

8

Upcoming Events

• June 12, BES Definition and BESnet Tool Webinar • June 17, Trustee Budget Meeting, Little Rock • July 28, Trustee Meeting, Omaha • Sept. 30-Oct. 1, Fall Workshop, Oklahoma City • Oct. 1-2, RTO Compliance Forum, Oklahoma City

9

http://www.spp.org/event_register2.asp?oID=5484

http://www.spp.org/event_detail.asp?dateID=Sep_30_2014

http://www.spp.org/event_detail.asp?dateID=Oct_1_2014

10

Watch 30 “SPP RE Basics” videos!

SPP.org > Regional Entity > Outreach

RAI Oversight Plan Framework

Scope Scope

Applicable Standards

Risk Elements

Controls Not Evaluated

CMEP Tools

I R A

I C E

Com

plia

nce

Ove

rsig

ht P

lan

Oversight Scoping

Inherent Risk Assessment

Internal Controls Evaluation

Input Input • RE Functions • Characteristics - ERO / Regional • Events • RISC

CIP Version 5 Revisions Standard Drafting Team Update Marisa Hecht, NERC Standards Developer Phil Huff, Arkansas Electric Cooperative June 3, 2014 SPP CIP Workshop

RELIABILITY | ACCOUNTABILITY 2

• Overview of FERC Directives • Overview of Development Activities • Overview of Revisions • Identify, Assess, and Correct – struck from 17 Requirements • Low Impact Assets – revised CIP-003 • Communication Networks – revised CIP-006 & CIP-007 • Transient Devices – revised CIP-004 & CIP-010 • Implementation Plan • Next Steps

Agenda


• FERC approved CIP standards in January 2013. • FERC directed NERC to modify certain aspects.

Identify, Assess, and Correct language. Communication Networks. Low Impact assets protections Transient Devices.

• Identify, Assess, and Correct and Communication Networks modifications must be filed at FERC by February 3, 2015.

Overview of FERC Directives


• Standard Drafting Team (SDT) appointed to address these revisions in Project 2014-02. Maggy Powell, Exelon Philip Huff, AECC David Revill, GTC Jay Cribb, Southern Company Forrest Krigbaum, BPA David Dockery, AECI Greg Goodrich, NYISO Christine Hasha, ERCOT Steve Brain, Dominion Scott Saunders, SMUD

Overview of Development Activities


• SDT and observers participated in aggressive development schedule from February to May Ten hours of conference calls per week, including subgroup calls

focused on each directive area Four face-to-face SDT meetings

• Participation from variety of stakeholders ensured that the SDT considered different perspectives from industry, government, and NERC

Overview of Development Activities


• Address all four directive areas by the filing deadline • Outreach crucial during development and during comment

period • Observer engagement critical to success

Key Objectives


• Identify, Assess, Correct language struck from all 17 requirements SDT determined that the requirements should state the performance

expectation and compliance language should be removed

• Substantive requirements remain the same • Violation Severity Levels revised accordingly

Identify, Assess, Correct


• SDT continues coordination with NERC Compliance and Enforcement Staff on supporting documents NERC presented draft RSAWs for CIP-002, CIP-007, and CIP-009 to the

SDT at its last meeting RSAW development team continuing to modify drafts to prepare for

concurrent posting with standards SDT suggested scenarios for NERC to address under RAI FAQ document on IAC and RAI

• NERC staff will offer a more comprehensive view of RAI beyond CIP standards in a June 19 webinar (following SDT webinar)



• SDT posed questions to NERC Enforcement and Compliance regarding Reliability Assurance Initiative (RAI)

• Questions are included in a Frequently Asked Questions document posted as a supplement to the revised standards

• Document focuses on how the RAI will handle high frequency low risk security obligations in the compliance and enforcement domains once IAC is removed



• CIP-003-6 Requirement R2 now in table format SDT kept all requirements applicable to low impact assets in this

requirement Technical areas same as CIP-003-5 passed by industry but with more

specificity to meet FERC directive Proposed requirement language borrowed from existing, FERC- and

industry-approved V5 standards but tailored to low impact

• Parent requirement above table parts is as follows:

Low Impact Assets


• CIP-003-6 Requirement R2, Part 2.1 Similar to Requirement R2 passed by industry in CIP-003-5

Low Impact Assets


• CIP-003-6 Requirement R2, Part 2.2

Low Impact Assets



Low Impact Assets


• CIP-003-6 Requirement R2, Part 2.4 SDT drew from CIP-005-5 when developing controls but tailored them to

low impact assets

Low Impact Assets


• CIP-003-6 Requirement R2, Part 2.5 SDT drew from CIP-008-5 for requirements but tailored them to lows

Low Impact Assets



Low Impact Assets


• CIP-006-6 Requirement R1, new Part 1.10 Restrict physical access to cabling and other nonprogrammable

communication components used for connection between applicable Cyber Assets within the same ESP in those instances when such cabling and components are located outside of a PSP.

Where physical access restrictions are not implemented, entity shall document and implement: o Encryption of data o Monitoring the status of the communication link and issuing an alarm o Equally effective logical protection

Applicable to High Impact BES Cyber Systems and PCAs and Medium BES Cyber Systems at Control Centers and PCAs

Communication Networks


• CIP-007-6 Requirement R1, Part 1.2: new applicability and incorporated new glossary term

Communication Networks


Transient Cyber Asset - A Cyber Asset directly connected for 30 consecutive calendar days or less, to: (1) a BES Cyber Asset, (2) a network within an ESP, or (3) a Protected Cyber Asset. Examples include, but are not limited to, Cyber Assets used for data transfer, vulnerability assessment, maintenance, or troubleshooting purposes.

Removable Media - Portable media, connected for 30 consecutive calendar days or less, that can be used to copy, move and/or access data. Examples include, but are not limited to, floppy disks, compact disks, USB flash drives, external hard drives, and other flash memory cards/drives that contain nonvolatile memory. A Cyber Asset is not Removable Media.

30-day exemption removed from BES Cyber Asset and Protected Cyber Asset definitions

Transient Devices – New and Modified Definitions


• CIP-010-2 new Requirement R4 New Table Requirement Addressed the FERC directive to consider the following security

controls: o device authorization as it relates to users and locations o software authorization o security patch management o malware prevention o detection controls for unauthorized physical access to a transient device o processes and procedures for connecting transient devices to systems at

different security classification levels Borrowed concepts from FERC- and industry-approved V5 standards

Transient Devices


• CIP-010-2, Requirement R4, Part 4.1 (Authorization)

Transient Device Protection


• CIP-010-2, Requirement R4, Parts 4.2 and 4.3 (Malware Protection)



• CIP-010-2, Requirement R4, Parts 4.4 and 4.5 (Malware Protection) (continued)



• CIP-010-2, Requirement R4, Parts 4.6 (Unauthorized Modifications)



• CIP-010-2, Requirement R4, Part 4.7 (Security Patches)



• CIP-004-6 – Adds Transient Cyber Assets and Removable Media to cybersecurity training program in Requirement R2, Part 2.1.

• CIP-007-6 – Added clarifying language in guidance for Requirement R3 (malware protection) to remind entities of the Transient Device Protections in CIP-010-2, Requirement R4.

• CIP-011-2 – Added qualifiers to guidance for entities to include Transient Cyber Assets and Removable Media in their information protection programs.

Transient Device Protection – Modification to Other Standards


• Builds from April 1, 2016 effective date of V5 • While the standard has an effective date, a compliance date

may differ for Requirements • Do not expect IAC language from V5 to go into effect • The following from V5 implementation remains the same:

Initial performance of certain periodic requirements Previous identity verification Planned or unplanned changes resulting in a higher categorization

Implementation Plan


• For those requirements and parts not listed below, compliance date would be effective date of standard, which is proposed to be later of April 1, 2016 or 3 months following govt. approval.

Implementation Plan


• NERC Standards Committee authorized posting for 45-day comment and ballot on May 30

• Comment period open June 2-July 16 • Join the ballot pool from June 2-July 2 • Reliability Standards Audit Worksheets (RSAWs) will be posted

by June 17 Not a part of the record for standards development.

• Ballot period open July 7-16 Will use the current ballot and commenting system.

Posting Schedule


• Industry SDT webinar June 19: 1-3pm ET Followed by RAI webinar from 3-5 pm ET

• SDT will consider all comments and make appropriate revisions • SDT meeting week of July 28 in St. Paul, MN • SDT meeting week of August 19 in San Francisco, CA

Next Steps


[email protected] 501-570-2444 [email protected] 404-446-9620

mailto:[email protected]


Westar Energy CIP V5 Transition Study & ESP Lessons Learned June 3, 2014

Taking Security to heart.

Transition Study Scope

• BES Cyber System Identification • Generation • Substation

• Technical Solutions • Configuration Baseline & Patch Management – BigFix • Security Information & Event Manager - Splunk • Privileged Access & Account Management - CyberArk

• Administrative Controls • Identify V3 to V5 Policy and Procedure Gaps • Develop Program Level Guidance

CIP V5 Transition Study Phase 1

• BES Cyber System Identification • Generation • Substation • EMS

• Applicability • At the device level

• Ownership • New SME’s • Begin Culture shift

Transition Study Wrap-up Week

• Attended by NERC and SPP • Reviewed Westar’s progress of transition • Toured Substation and Generation facilities • Q&A Session

CIP V5 Transition Study Phase 2

• Procedure Analysis • Transitioning existing procedures from V3 – V5 • Integrate technical solutions into procedures • Identify controls

• Technical Solutions • Configuration Baseline & Patch Management – BigFix • Security Information & Event Manager - Splunk • Privileged Access & Account Management - CyberArk

• Training • Informative • Required (role based) • Continue culture shift

ESP Design Lessons Learned

• Completed as part of CIP-002 identification process

• Complete detailed inventory of how cyber assets are communicating

• Determine essential connections to BES Cyber Assets

Challenges

• New CIP SMEs and affected employees • Substation Routable/Serial Protocol

• Business case still in progress • Complete Administrative Controls

• Documentation, implementation, education and outreach

Westar Transition Study Lessons Learned

• Establish “buffer” timeframe for compliance • There are different ways to achieve CIP V5 compliance • Consider reliability improvement opportunities in design • Involve Business Unit engineering resources early in the

design phase • Work with impacted Business Units to budget accordingly • Centralize Physical Security monitoring

Questions ?

Megan Wagner (785) 575-1852 [email protected]


CIP V3-V5 Transition Tom Hofstetter, CIP Compliance Auditor SPP-RE CIP Workshop June 3, 2014


Disclaimers

• I don’t speak for the Commission • I don’t speak for NERC • I don’t speak for any region • I can’t tell an entity how to comply, but I can share my thoughts

and recommendations • My opinions are my own, subject to change in light of facts,

threats, innuendo, complacency, ignorance, enlightenment, sorrow, wisdom, hunger, thirst, weariness, time of day, or whether I’ve hit my head recently. (And in certain circumstances, what my wife says I can say.)


Responsibilities of the ERO (NERC & Regional Entities) • Monitor the cyber security measures used to protect the

reliable operation of the Bulk Electric System (BES) • Recognize the challenges facing Responsible Entities

Establishing compliance with V5 will be an ongoing process No single point in time when they will move from compliance with V3 to

compliance with V5

CIP V3 – V5 Transition


Big Picture • The “Effective Date” of V5 was April 1, 2014

Based on the date that V5 Standards were approved by FERC Beginning of the implementation stage/transition period

• The “Compliance Enforcement Date” of V5 is April 1, 2016 Applies to Responsible Entities with High and Medium Impact Cyber

Systems

• Compliance with V3 remains mandatory and enforceable, and will be assessed until the V5 compliance enforcement date V4 is no longer in the picture (that includes use of V4 “bright-line”

criteria for identifying Critical Cyber Assets)



Mapping the path • Responsible Entities with High and Medium Impact

Cyber Assets need to prepare for April 1, 2016 • ERO focus during transition period is not “gotcha”,

but an effective and successful implementation of V5



Transition Guidance • Draft of next version is currently being vetted • Supersedes previous versions • Provides guidance and flexibility that will allow

Responsible Entities to focus on implementing changes to achieve compliance with V5

• During the transition period, there will be a flexible approach to the evaluation of V3 compliance



Next version of Transition Guidance • Narrative – similar to content of current version with

explanations, considerations, and recommendations. • Compatibility table – spreadsheet listing V5

requirements that are “Mostly Compatible” with V3 requirements



Mostly Compatible? • Many V5 requirements are mostly compatible (MC)

with Version 3 • In those cases, processes and documents that

demonstrate compliance are not significantly different from V3 to V5

• Requirement numbers may not always align, but the requirements themselves are considered “MC”

• Not necessarily one to one mapping; e.g., CIP-002-3 R4 and CIP-002-5 R2 both address senior manager approval of the asset list



Mostly Compatible! • As described in the guidance document, V5

compliance to designated requirements can be considered compliance with the “MC” Version 3 requirement

• For V5 requirements with no V3 counterpart, V5 compliance will not be a factor during the transition period



Similar V3/V5 requirements • CIP-003 – 90% • CIP-004 – 80 to 90% • CIP-005 – 85% • CIP-006 – 85% (Review required for new Assets) • CIP-007 – 80% • CIP-008 – 95% • CIP-009 – 90% (Review required for new Assets) • CIP-010 – 50% (new to V5) • CIP-011 – 50% (new to V5)



CIP V3 – V5 Transition Guidance

Narrative 1. Introduction 2. Background 3. Newly Identified BES Cyber Systems 4. Compliance during the Transition Period 5. Transition Period Audits 6. V5 Implementation Study 7. Technical Feasibility Exceptions (TFEs)



Narrative – Section 1. Introduction • Overarching philosophy: apply the appropriate cyber

security measures to protect the reliable operation of the Bulk Electric System (BES).

• Applies to Regional Entities and Responsible Entities • Provides guidance and flexibility for implementing

changes to achieve compliance with V5 without undue concerns regarding compliance status with V3

• Supersedes previous Cyber Security Standards Transition Guidance



Narrative – Section 2. Background • FERC Order 791 was issued on November 22, 2013 • Some directives referred to a specially formed

Standards Drafting Team for resolution • Version 4 CIP Reliability Standards (V4) will never be

mandatory and enforceable



Narrative – Section 3. Newly Identified BES Cyber Systems • When guidance is issued, a Responsible Entity with

newly identified systems and facilities shall begin implementing V5

• Applies to Responsible Entities that previously would have referred to Implementation Plan for Newly Identified Critical Cyber Assets and Newly Registered Entities in CIP V3 or used V5 Impact Ratings to identify new assets



Narrative – Section 3. Newly Identified BES Cyber Systems • Responsible Entities that have assets identified by an

acquisition or that receive a Registered Third-Party Designation (i.e., Impact Rating 2.3, 2.6, or 2.8) will be provided the latter of 12 calendar month implementation window from the time

of notification (per the V5 Implementation Plan) or Implementation date of April 1, 2016 for any newly

identified BES Cyber Systems



Narrative – Section 4. Compliance during the Transition Period • Responsible Entities are expected to take the

appropriate actions to become compliant with the V5 Standards by the compliance enforcement date

• Those that previously identified CCAs according to the bright-line criteria in V4 no longer have that approach available

• Regional Entities will exercise discretion when assessing compliance to V3 requirements during the transition period



Narrative – Section 5. Transition Period Audits • Compliance Monitoring and Enforcement Program

(CMEP) will continue to be the guiding directive for the conduct of CIP audits

• Regional Entity may use outreach and annual Self-Certifications in lieu of off-site audits for entities without V3 Critical Assets or Critical Cyber Assets

• Details regarding the scope and performance of audits during the transition period described in the guidance document



Narrative – Section 6. V5 Implementation Study • Six Responsible Entities participated in a study to

voluntarily implement the V5 Standards prior to the enforcement date of April 1, 2016

• Goals of the study include identification of processes, tools, and guidance for achieving V5 compliance

• V5 Implementation Study page at NERC’s website (http://www.nerc.com/pa/CI/Pages/Transition-Program-V5-Implementation-Study.aspx)

http://www.nerc.com/pa/CI/Pages/Transition-Program-V5-Implementation-Study.aspx



Narrative – Section 7. Technical Feasibility Exceptions (TFEs) • In general, TFEs will align with the overall transition process • They will be considered in context of underlying requirement(s)

V5 TFEs pertinent to V3 TFEs V5 V3

CIP-005-5 R2.3 CIP-005-3 R2.4 CIP-007-5 R1.1 CIP-007-3 R2.3 CIP-007-5 R4.3 CIP-007-3 R6.4 CIP-007-5 R5.6 CIP-007-3 R5.3.3



Narrative – Section 7. Technical Feasibility Exceptions (TFEs) • TFE requests will be needed for systems or devices unable to

meet strict compliance with a V5 requirement that has no associated TFE in V3

V5 TFEs not associated with V3 TFEs CIP-005-5 CIP-006-5 CIP-007-5 CIP-010-1

R1.4 R1.3 R5.1 R1.5 R2.1 R5.7 R3.2. R2.2



Narrative – Section 7. Technical Feasibility Exceptions (TFEs) • Existing TFEs that are no longer applicable per V5 requirements

will remain in effect until the V5 compliance enforcement date

V3 TFEs superseded when V5 is Implemented CIP-005-3 CIP-006-3 CIP-007-3

R3.1 R1.1 R3.2 R4 R5.3 R 5.3.1 R 5.3.2 R6


Compatibility Tables • Show the requirements from V5 that have been

deemed as “mostly compatible” with a V3 counterpart • Comparison is intended to help Responsible Entities

maintain adequate protection of BES assets as they move from compliance with V3 to compliance with V5 of the CIP Reliability Standards




Compatibility Tables



Compatibility Tables


Underlying philosophy • Still a lot to do, but leveraging appropriate V3 policies,

procedures, and processes will contribute to successful transition to V5

• ERO wants to facilitate the transition



Technical Feasibility Exception (TFE) Update

June 4, 2014

Shon Austin Lead Compliance Specialist - CIP [email protected] · 501.614.3273

TFE BASICS

2

Multiple Choice Question

• OMG!! My entity cannot strictly comply with certain CIP requirements on grounds of technical infeasibility or limitations, what do I do?

A. Cross my fingers and hope the auditors don’t notice when conducting an audit of my entity.

B. Replace all hardware that can not comply with the strict language of the requirements.

C. Request a TFE!!!!

3

What is a TFE and when do I need one? • Appendix 4D to the Rules of Procedure allows

Responsible Entity to request and receive exception from strict compliance with certain CIP standard requirements on grounds of technical infeasibility or limitations

• TFE provides “safe harbor” from possible violation – Unless the TFE is terminated or you fail to maintain

approved compensating measures

4

http://www.nerc.com/AboutNERC/Pages/Rules-of-Procedure.aspx

• CIP-005-3: – R2.4

– R2.6 (Paragraph 81)

– R3.1

– R3.2

• CIP-006-3c: – R1.1

• CIP-007-3 – R2.3

– R3

– R4

– R5.3

– R5.3.1

– R5.3.2

– R5.3.3

– R6

– R6.3

Requirements eligible for TFE

5

APPENDIX 4D REVISIONS

6

TFE Process Revisions

• Appendix 4D to NERC Rules of Procedure was simplified in September 3, 2013

• SPP RE applied the revised process to TFEs beginning January 1, 2014

• SPP RE reviews TFEs within 60 days of request – We no longer perform an extensive Part B review

– We will approve or disapprove request within 60 days of request

– “Safe Harbor” now begins when TFE is requested

7


TFE Process Revisions (cont.)

• No quarterly or annual reports required

• A completed Material Change Report (MCR) is required when requesting the initial TFE – MCR supplements the data entered into webCDMS for

each unique covered asset

• The MCR also replaces the Amendment process

8

Interim webCDMS TFE Processes

• webCDMS still supports “old” TFE process; enhancements expected 4th quarter 2014

• Requesting a new TFE – Input data in required fields

– Complete and upload MCR spreadsheet

• Requesting a Material Change to an existing TFE (as needed)

• Terminating an existing TFE

9

Requesting a New TFE 1. Download the “TFE Material Change Report Template”

from webCDMS document repository (Document management-> document repository)

2. Complete and save a local copy of “TFE Material Change Report Template” spreadsheet

10

Requesting a New TFE (cont.) 3. Follow the webCDMS process to “Create a TFE”. Enter

the following information and click save:

Device count Category (asset class) CIP Standard CIP Requirement Proposed TFE expiration date (if any) Basis for approval (not technically possible, etc.) Compensating/mitigating measures Completion date of compensating/mitigating measures Has this TFE been previously approved TFE ID of previously approved TFE

11

Requesting a New TFE (cont.) 4. Attach the following documents to “Entity Documents”:

• Completed TFE Material Change Report

• Evidence regarding why the device cannot be compliant

• Evidence regarding compensating and mitigating measures in place or planned

• Alternatively, the evidence can be submitted to the SPP RE EFT server

– The MCR spreadsheet still has to be uploaded to webCDMS

12

TFE Material Change Report Template

13

Completing TFE Material Change Report Template

14

• Asset Type (Dropdown)

– *Other - Explanation

• Unique Device ID Assigned by Entity

• Physical Location of Device (Critical Asset)

• Date Device Placed Into Production (Actual or Estimate)

• Self-Report or Self-Certification Filed? Y/N

• Date Device Removed From Production

• Comments

Requesting a Material Change to Existing TFE

• Material Changes replaced Amendment process

• Material Change: A change in facts that modifies Required Information in connection with an approved TFE

• All TFEs approved before January 1, 2014 do not require any immediate action unless a Material Change occurs – Begin filling out MCR templates now

• When there is a Material Change, you must: – update your TFE Material Change Report

– submit the change in webCDMS

15

Material Change Examples

• Could include, but are not limited to: - Addition of a Covered Asset

- Replacement of a Covered Asset

- Change in compensating measures

- Change in basis for TFE approval

- Change in TFE expiration date

- Responsible Entity achieving strict compliance with the Applicable Requirement

16

Requesting a Material Change to Existing TFE

1. Update your local copy of TFE Material Change Report 2. Log into webCDMS, click “Amend” for the applicable TFE 3. Update necessary information (i.e. device count, expiration

date, etc.) 4. Submit Amended TFE

5. Click down arrow under “Actions” for the TFE

6. Select “Attach Documentation” 7. Attach the following documents to “Entity Documents”:

– Updated TFE Material Change Report – Evidence regarding the Material Change

17

Reducing the number of covered assets? 1. Update your local copy of the TFE Material Change

Report

2. In webCDMS, select applicable TFE


4. Click “Reduce No. of Covered Assets”

5. Enter the new number of Covered Assets

6. Enter comments

7. Save

8. Upload your updated local copy of the TFE Material Change Report

18

Terminating a TFE 1. In webCDMS, select applicable TFE


3. Select “Attach Documentation” 4. Attach a letter signed by your senior manager or

delegate that includes the following : – TFE ID

– TFE Termination Date

– Reason the TFE is no longer required for the Covered Assets

5. Email Shon Austin ([email protected]) notifying SPP RE that you’ve submitted termination request

19


Future upgrades

• webCDMS will be modified to accommodate: – Streamlined webCDMS TFE process

– Additional required information fields

– Other layout and reporting functions

20

COMMON ISSUES

21

What about patches?

• If there are no patches available, no TFE is required

• If a patch is available and you have scheduled implementation, no TFE is required

• If a patch is available but cannot be implemented, a TFE is required

22

Lack of robust evidence

• Example: Cannot add anti-virus to an asset

• Not sufficient: Opinion document from an IT staffer

• We want to see: – Operating manual

– Attestation from vendor company

– Test results

23

Lack of planning

• Think about the need for TFEs while planning for CCAs – Not during or after implementing!

• Example: Entity installed Oracle DB server – Server had user account that could not conform to CIP-

007 R5.3

– Led to numerous self-reported violations

24

Compensating measures must resolve vulnerabilities

• Ensure measures “close the gap” – Can’t always solve logical vulnerabilities only with

physical measures

– Can’t always solve physical vulnerability only with logical measures

25

Example: USB vulnerability

26

• Appropriate physical measures:

– Disable port

– Control who has access

• Ineffective measures:

– Firewall

– IDS/IPS

– Anti-virus

Example: Unable to install Anti-Virus

• Appropriate logical measures: – Keep systems patched

– Scan removable media before use

27

• Ineffective measure: – Physical access controls

Questions and Summary

• TFE needed for “safe harbor”

• TFE process has been simplified

• Appendix 4D to the Rules of Procedure has been updated; webCDMS process changes are pending

• Provide through justification and compensating/ mitigating measures

• Be proactive and plan ahead when changing and adding assets

28


1

Substation CIP Change Management Program

SPP CIP Users Group Dawn Berndt June 2014

2

Agenda Background Substation CIP Change Management Program Challenges Lessons Learned

3

Xcel Energy Background Serves 3.4 million electric customers in 8 states Three Operating Companies

Northern States Power (NSP) Public Service Company (PSCo) Southwestern Public Service (SPS)

NERC Compliance Regions MRO (NSP) WECC (PSCo) SPP (SPS)

4

CIP Change Control Substations

78 CIP substations with 1,100 Critical Cyber Assets 16,000 work orders per year at CIP substations More than 100 change control forms processed

each year Opportunity for change control errors is large

Change Control Program continues to evolve

5

CIP Change Control Key Players

CIP Consultant

Substation Field Engineering

Project Initiator

Engineering

Operations & Maintenance

Field Technician

6

CIP Substation Change Control

Pro

ject

/Wor

k In

tiatit

edC

IP S

ubst

atio

n / C

yber

Ass

ets

7 D

AYS

23 D

AYS

CIP Consultant

Project InitiatorsSFE -Substation Field Engineering

SED – Substation Engineering DesignSCE – Substation Commissioning

EngineeringSPE – System Protection Engineering

CO&M

YesNo

Store Project Spreadsheet in

ProjectWise

CIP Sub <AND>Cyber Assets

Impacted?

Record all affected Cyber

Assets in Spreadsheet

Update CIP ESP Diagram

Done

CIP Project Spreadsheet

Change Control not needed,

inform InitiatorProcess Done

Update CIP Asset Inventory

Issue Project Spreadsheet

to field

PROJECTS Small Capital or Maintenance Large Capital Small Special O&M Install or Upgrades

Normal or Emergency Maintenance

YesCIP Project

SpreadsheetRecord device information on Spreadsheet

Perform required configuration and testing

activities

Return data to CIP

Consultant

Substation Work

Is Substation Work Complete?<OR>

Has remote connectivity been established for the first time?

Yes 7 DAYS Start

Complete Project Spreadsheet

documentation

No

CIP Substation Change ControlP

roje

ct/W

ork

Intia

tited

CIP Consultant




Substation O&M

YesNo

CIP Sub <AND> Cyber Assets

Impacted?




YesCIP Project



activities

7


Pro

ject

/Wor

k In

tiatit

edC

IP S

ubst

atio

n / C

yber

Ass

ets

7 D

AYS

23 D

AYS

CIP Consultant




Substation O&M

YesNo


ProjectWise


Impacted?




Done






to field



YesCIP Project



activities

Return data to CIP

Consultant

Substation Work



Yes 7 DAYS Start


documentation

No

C

IP S

ubst

atio

n / C

yber

Ass

ets

YesNo







to field

YesCIP Project



activities

Substation Work



Yes 7 DAYS Start

No

8


Pro

ject

/Wor

k In

tiatit

edC

IP S

ubst

atio

n / C

yber

Ass

ets

7 D

AYS

23 D

AYS

CIP Consultant




Substation O&M

YesNo


ProjectWise


Impacted?




Done






to field



YesCIP Project



activities

Return data to CIP

Consultant

Substation Work



Yes 7 DAYS Start


documentation

No

C

7

DA

YS23

DA

YS


ProjectWise


Done


Return data to CIP

Consultant

Yes 7 DAYS Start


documentation

9

10

11

12

13

14

15

16

CIP Change Control Challenges

Manual change control process Rely on personnel to initiate change control for CIP substations Multiple departments required to fill out information

Depend on human performance for documentation Time Accuracy Completeness

Large number of people had access to CIP Substations Almost 25% were external parties (other utilities, contractors) Paring this list down

Training and awareness for all affected personnel on correct procedure

17

CIP Change Control Lessons Learned

Workflow automation is crucial when dealing with many touch points in CIP Substations Identify process handoffs and consider controls

Importance of ongoing training and awareness Employee turnover Contractors/Consultants Face-to-face is valuable with field resources

Implementing compliance ‘controls’ can help prevent and detect issues

18

CIP Change Control Controls Implemented

Established controls to help identify substation work requiring change control upfront Automated biweekly report to review new work orders Established process for checking CIP status on new substation

design projects Implemented manual security testing of access points Conduct bi-weekly change control meetings in each Operating

Company with Engineering, Operations and CIP Compliance groups Quarterly monitoring program Implemented tagging of disconnected device cables Installed access point “CIP notice” labels

19

CIP Change Control Improved Field Awareness

20

Port plugs and locks to prevent inadvertent connections

Technician guide for connecting and making changes to cyber assets in the Electronic Security Perimeter

Restructuring process documentation Change control awareness posters in CIP substations

CIP Change Control Controls In Development

21

Implement additional controls Asset management solution (not Excel spreadsheets) Investigate automation solutions

Change control workflow Configuration management

Evaluate viability of additional testing facilities

CIP Change Control Preparation for CIP V5

22

Questions?

CIP-010-1 R1 & R2: Configuration Change Management June 3, 2014

Steven Keller Lead Compliance Specialist - CIP [email protected] 501.688.1633

Outline

• What is CIP-010-1?

• How it is different from CIP-003-3 R6?

• Focus on CIP-010-1’s Change Management Sections (R1 & R2)

• Evidence we like to see and best practices

2

What is CIP-010-1, and what does it bring?

• Updated Configuration Change Management and Vulnerability Assessments

• Development of Baseline

• Examples of Evidence

3

CIP-010-1 R1 Baselines

• Develop Baseline Configuration • All software on each machine or a group of machines

- High and Medium BES Cyber Systems

- “Audit trail” of changes to the baseline

• Change Management record trail

• Understand what is on your systems and prevent any unauthorized changes.

4

CIP-10-1 R1.1 Baseline • Develop Baseline for all High or Medium BES Systems

and associated Electronic Access Control and Monitoring Systems (EACMS), Physical Access Control Systems (PACS) and Protected Cyber Assets (PCA)

• Baselines must have: - OS or Firmware

- All Commercial software and/or Open Source software

- Any custom software

- Logical Network accessible ports

- List of all security patches applied to the software listed above

5

CIP-10-1 R1.1 Baseline

• Risk: Unauthorized software on BES Cyber System

• Internal Control Type: Supports detective controls

• Sample Evidence: – Spreadsheet identifying baseline for each asset/group

– Records from asset management system

– Documentation in change management system

6

CIP-10-1 R1.2 Document Changes

• Authorize/document deviations from existing baseline

• Risk: Having changes that are not authorized

• Internal Control Type: Preventive

• Sample Evidence: – Change Request record that shows what changed from

the Existing Baseline • Must be performed by person/group with authority

• Best practice*: Document how your change management system works

7

* Best practices are suggested by SPP RE staff but are NOT required by the standard

CIP-10-1 R1.3 30 Days

• Must update Baseline with in 30 DAYS of change

• Risk: Undetected changes to your system

• Internal Controls: Detective

• Sample Evidence: – Evidence that baseline was updated in 30 days of

change, such as ticket or revision history

• Best practice: Document how your baselines are updated. Maintain a version history.

• Update your baseline concurrently as you are testing the change.

8

CIP-10-1 R1.4 Deviations of Baseline • What could be impacted by the change?

- CIP-005-5 (access permissions, malicious communications, and remote access)

- CIP-007-5 (ports, malicious code, logging, shared accounts)

• Provide evidence that change will not adversely affect the BES Systems in question

• Risk: Undetected baseline deviations (You didn’t pre-identify an impacted CIP-005-5 and CIP-007-5 control & therefore did not test it)


• Sample Evidence: List of Cyber Security Controls that were verified/tested and date tested

9

Best Practices: CIP-10-1 R1.4 Deviations of Baseline • Test all controls every time you make a change, even

the controls you don’t think will be affected

• Automate testing if possible

• Have a different group or third party test changes or review test results to verify nothing gets missed

10

CIP-10-1 R1.5 Testing of Changes • Risk: Loss of High Impact systems due to faulty change

• Internal Controls: Preventive

• Sample Evidence: – List of Cyber Security Controls that were verified/tested

and successful test results

– If using a test environment, must document any differences between production

11

CIP-10-1 R1.5 Testing of Changes

• Only applies to High Impact BES Cyber Systems

• For each identified deviation from baseline, test changes in either test or production environment before implementing

• Document test results

• Show how cyber security controls for CIP-005-5 and CIP-007-5 are not adversely affected

12

Best Practices - CIP-10-1 R1.5 Testing of Changes

• Checklists are GOOD!

• Document your actual test results

• Test environment should replicate production, including fail-over capability

• Test PCAs, Medium, and Low-impacting BES Cyber Systems, not just High

13

CIP-10-1 R2 Configuration Monitoring

• Monitor changes to the baseline for each High Impact BES Cyber systems and associated EACMS and/or PCA - Must be done every 35 days

- Investigate unauthorized changes that are detected

• Risk: Undetected changes to your system


• Sample Evidence: – Monitoring system logs

– Investigation records

– Records of investigations (work orders or raw data)

14

Best Practices: CIP-10-1 R2 Configuration Monitoring • Daily monitoring is good, but real-time monitoring is

BETTER!

• Document how you are monitoring the baselines and detecting changes

• Document how you are will handle an unauthorized change

15

Summary

• Baseline configurations now required for all assets within your BES cyber system

• Know what’s on your system!

• Test your changes fully and “deeply” to ensure application functionality

• Changes that aren’t fully vetted can have major adverse impacts, such as to EMS systems

16

17

Steven Keller Lead Compliance Specialist – CIP 501-688-1633

ES-ISAC Update Fred Hintermister, Manager ES-ISAC


SECTOR STRATEGIC DIFFERENTIATORS

• Atop a cascading tier of critical interdependent sectors • Owner Operated • Proven reliability performance • Inherently resilient system of systems • Continental Bulk Power System (BPS) footprint • Collaborative standards development process • Comprehensive, mandatory and enforceable standards • Alert capability with compelled full feedback loop


WHAT IS ES-ISAC?

• Electricity Sector Information Sharing and Analysis Center

• Functional capability provided by North American Electric Reliability Corporation (NERC)

• Provides operational support to Electricity Subsector Coordination Council (ESCC), the highest policy level nexus between industry and government under the National Infrastructure Protection Plan (NIPP)

• Several other Critical Infrastructure sectors also have ISACs

• Communicates with members, government, and other ISACs about threat indications, vulnerabilities, and protective strategies

• ISACs work together to better understand cross-industry dependencies and to account for these in emergency response and planning


WHAT DOES ES-ISAC PROVIDE?

• Sector coordination (ES-ISAC is not operational but is operationally informed)

• Maintain grid, cyber security, physical security near real time wide area awareness

• Continuously assess North American Bulk Power System (BPS)/Bulk Electric System (BES) risk

• Rapidly develop and precisely deliver authoritative mitigation guidance


HOW DOES ES-ISAC DO IT?

• Maintains grid, cyber security, physical security near real time wide area awareness

• Continuously assesses North American Bulk Power System (BPS)/Bulk Electric System (BES) risk

• Maintains a centralized analytic capability, comprised of indicators of compromise (IOCs), watchlists (patterns of IOC relationships), and a tools suite to develop and manage these on a technical analysis level

• Maintains, refines and executes an information sharing process to apply this analytic content within the sector and with collaborative security partners


Our NERC ES-ISAC World

ES-ISAC -Sector level coordination -Cyber and physical security expertise -Mitigation development and delivery -ISAC -Sector level coordination -Cyber and physical

tGovernment Fusion -DHS Critical Infrastructure -DoD Mission Assurance -DOE SSA relationship u Mission Assurance -DOE SSA relationship

pBPSA Wide Area Operational Awareness -ESF 12 coordination -Entity relationship maintenance and coordination maintenance and coordination mIndustry Collaboration -Compliance reporting data -Noncompliance security trusted sharing -Collaboration with trade organizations


Information Sharing IT Architecture

Security Reporting &

Dialogue

Analytics for BPS Dynamic

Risk Assessment and Mitigation Development

Cross Sector Information

Sharing Solution


Portal


Indicators of Compromise

~20,000 IOCs and their relationships visualized 3-8 month opportunity gap between observable attack set and execution

Trend report analysis has shown: • Over80% of reports used email phishing as a

vector • Less than 5% of data were contributed by

electricity subsector • ysis (from September Trend report): • 83% of reports used email phishing as vector •Only 3% of the data were contributed by electricity sector

Since April 2012 • Well over 1K separate shares • Over 25K individual indicators

Our data set grows and shows! • Approx 25K plus IOCs • Visualization reveals 3-8 month

opportunity gap between observable attack set and execution

• That gap is our shared opportunity to break the kill chain


• Shifting gears –What’s happening, What we see • Physical Security, then Cyber Security • The Observations distilled • What they mean • Steps to consider

The Latest News & What it Means


Physical Security, Recent Observations

• Substation compromise does happen • Cross sector indicates ICS can be an issue • Holistic view underscores that some adversaries are advanced

or some events imply sophistication


What do these mean for you?

• Routine break-in or copper theft may have deeper meaning • Accessible enterprise networks and ICS remote accessible

layers are vulnerable • What appears simple at one entity, place and time might not

be so simple


Prudent operator steps to consider

• Register and utilize ES-ISAC portal and Experience Sharing Tool (EST)

• Use ES-ISAC Weekly Report and our Webinars, heed our Alerts • Engage with ES-ISAC in trusted collaboration with benefit of

attribution protection • Leverage NERC events – GRIDSECON, GRIDEX • Security is dynamic not static. Make sure your staff actions are

too… • If an observed physical security event falls below reporting

thresholds, share it- If in doubt, share it out!


Cyber Security, Recent Observations

• Robust phishing continues, and most breaches have a phishing genesis

• Cross sector has seen significant social engineering cases and scale financial crimes

• Cross sector has seen IT approaches which demonstrate the likely ability to create OT impacts


What do these mean for you?

• Cyber Hygiene matters! • Own systems awareness matters • Know your available resources • INFORMATION SHARING WITH ES-ISAC HAS NEVER BEEN MORE

IMPROTANT FOR YOU AND YOUR SECTOR!


Prudent operator steps to consider

• Cyber assessment for ground truth on own systems remote accessibility, equipment and S/W inventory, configuration, and corporate policy and process

• Understand the dynamics of your own operating environment, know activity levels at ports, pieces and network corners

• Ask if security has a seat at the table – is your organization structured to share and act effectively with respect to contemporary security challenges?

• Exercise smart • Pack all preps left of boom


Future Themes

• A lively Trusted Collaborative Environment • Automated Information Exchange • Predictive Analytics • Enhanced Cross Sector Resilience Management • Better Centralized Analytic Capability • ES-ISAC as a NERC functional capability, aligned to (the new!)

ESCC providing operational support • Key value delivery as threats and vulnerabilities change


Physical Security Reliability Standard Robert Rhodes (SPP) SPP RE CIP Workshop June 3-4, 2014


• Project Overview Standard overview Reliability Standard Audit Worksheet (RSAW) approach

• Review effort to-date • Final Proposed Standard

Agenda


Name Entity Susan Ivey (Chair) Exelon Corporation Lou Oberski (Vice Chair) Dominion John Breckenridge Kansas City Power & Light Ross Johnson Capital Power Kathleen Judge National Grid Mike O’Neil Florida Power & Light / NextEra, Inc. Stephen Pelcher Santee Cooper John Pespisa Southern California Edison Robert Rhodes Southwest Power Pool Allan Wick Tri-State Generation and

Transmission Manho Yeung Pacific Gas and Electric Company

Standard Drafting Team


• At a Glance: The key items to know about Reliability Standard CIP-014-1.

• The standard includes only those Transmission stations and Transmission substations (and associated primary control centers) that if rendered inoperable or damaged could result in widespread instability, uncontrolled separation, or Cascading within an Interconnection.

• Only a relatively small number of Transmission Owners and Transmission Operators will need to comply with the entire Standard.

• The Standards Committee (SC) approved waivers to shorten the initial comment/ballot period and the final ballot period.

Project Overview



• “To identify and protect Transmission stations and Transmission substations, and their associated primary control centers, that if rendered inoperable or damaged as a result of a physical attack could result in widespread instability, uncontrolled separation, or Cascading within an Interconnection.”

• Drafted as Critical Infrastructure Protection (CIP) family of standards.

Purpose


• The applicability of proposed CIP-014-1 starts with those Transmission Owners that own Transmission facilities that meet the bright line criteria in Reliability Standard CIP-002-5.1 for a “medium impact” rating.

• The SDT sought to ensure that entities could apply the same set of criteria to assist with identification of facilities under CIP Version 5 and proposed CIP-014-1.

• By application of the requirements, only certain Transmission Operators that are notified under the standard’s Requirement R3 have obligations under the standard.

Applicability


• The first three requirements of CIP-014-1 require Transmission Owners to: Perform risk assessments to identify those Transmission stations and

Transmission substations that meet the “medium impact” criteria from CIP-002-5.1, and their associated primary control centers Arrange for a third party verification (as directed in the order) of the

identifications; and Notify certain Transmission Operators of identified primary control centers

that operationally control the identified and verified Transmission stations and Transmission substations. The requirements provide the periodicity for satisfying these obligations.

Only an entity that owns or operates one or more of the identified facilities has further obligations in Requirements R4 through R6. If an entity identifies a null set after applying Requirements R1 through R2, the rest of the standard does not apply.

Requirements R1-R3


• The final three requirements of CIP-014-1 require: The evaluation of potential threats and vulnerabilities of a physical attack

to the facilities identified and verified according to the earlier requirements, The development and implementation of a security plan(s) designed in

response to the evaluation, and A third party review of the evaluation and security plan(s) (as directed in

the order).

Requirements R4-R6


Confidentiality

• Physical Security Order, Paragraph 10: • “…NERC should include in the Reliability Standards a procedure

that will ensure confidential treatment of sensitive or confidential information but still allow for the Commission, NERC and the Regional Entities to review and inspect any information that is needed to ensure compliance with the Reliability Standards.”

• Addressed in Requirement R2, Part 2.4 and Requirement R6, Part 6.4.


Confidentiality

• CIP-014-1, Section 1.4. Additional Compliance Information • Confidentiality: To protect the confidentiality and sensitive

nature of the evidence for demonstrating compliance with this standard, all evidence will be retained at the Transmission Owner’s and Transmission Operator’s facilities.


• RSAW team provided an RSAW that was posted concurrently with the CIP-014-1 standard. RSAW team consists of 12 members from NERC and the Regional Entities.

• Team members attended the technical conference and SDT meetings RSAW was reviewed with the SDT prior to posting

• Industry commented on the draft RSAW. • Special notes included in RSAW providing guidance regarding

third party verifications and reviews (see Page 4 of RSAW).

CIP-014-1 RSAW Development


• Note to Auditors Concerning Third Party Verifications and Reviews

• Requirements R2 and R6 prescribe, respectively, unaffiliated third party verifications for Requirement R1 and unaffiliated third party reviews for Requirements R4 and R5. Auditors are encouraged to rely on the verifications and reviews performed in cases where the verifying or reviewing entities are qualified, unaffiliated with the audited entity, and the scope of their verification or review is clear. The concept of reliance means using the work of others to avoid duplication of efforts and is consistent with recognized professional auditing standards, which are required for Compliance Audits per NERC’s Rules of Procedure.

CIP-014-1 RSAW


• Reliance in the context of this Reliability Standard means using the Requirement R2 verifications and Requirement R6 reviews to reduce audit risk and the related rigor of audit testing for Requirements R1, R4, and R5. However, in cases where the verifying or reviewing entity lacks the qualifications specified in Requirement R2 for verifications or Requirement R6 for reviewers, the required unaffiliation from the audited entity, or where the scope of the third party entity’s verification or review is unclear, auditors may need to apply audit testing of Requirements R1, R4, or R5.

CIP-014-1 RSAW


• For this reason, the Evidence Requested and Compliance Assessment Approach Sections are still present in this RSAW for Requirements R1, R4, and R5. The intent is that those sections will also facilitate expectations for entities and their unaffiliated third party verifiers and reviewers, assist Electric Reliability Organization (ERO) auditors to understand the audit procedures applied by unaffiliated third party verifiers and reviewers, and provide transparency between ERO auditors and Industry, should circumstances require audit testing of Requirements R1, R4, or R5. Further, it is an objective of the ERO to have transparent Evidence Requests and Compliance Assessment Approaches for every enforceable standard, whether they are in audit scope or not.

CIP-014-1 RSAW


• Draft standard posted for 15-day initial comment and 5-day ballot April 9, 2014

• Industry webinars (3) during initial comment and ballot • Standard received an 82% approval rating • Based on comments received the SDT made only minor

clarifying modifications • Posted for 5-day final ballot on May 1, 2014 • Standard received an 85.6% approval rating in final ballot • NERC Board of Trustees adopted the standard on May 13, 2014 • NERC Legal plans to file proposed standard with FERC on or

about May 23, 2014

Effort To-Date


• The following slides contain summaries of the specific requirements language and the associated Compliance Assessment Approach sections from the RSAW.

• This language was developed and posted for the final ballot on May 1, 2014.

CIP-014-1 Requirements


• Requirement R1 requires the Transmission Owner perform a risk assessment through transmission analysis to identify: Each transmission substation (existing and planned) that, if rendered

inoperable or damaged, could result in instability, uncontrolled separation, or cascading within an Interconnection Associated control centers

• Requirement R1 specifies that the assessment shall be conducted at least: Once every 30 months for a Transmission Owner that has identified in its

previous risk assessment (as verified according to Requirement R2) one or more Transmission stations or Transmission substations… At least once every 60 calendar months for a Transmission Owner that has

not identified in its previous risk assessment (as verified according to Requirement R2) any Transmission stations or Transmission substations…

Requirement R1


Requirement R1 - RSAW

The RSAW Developer will complete this section with a set of detailed steps for the audit process. See the RSAW Developer’s Guide for more information.

(R1) Review entity’s process for determining Transmission stations/substations subject to identification in accordance with Requirement R1, including weighting described in Section 4.1.1.2.

(R1) Review entity’s risk assessment process to determine the Transmission stations/substations that if rendered inoperable or damaged could result in widespread instability, uncontrolled separation, or Cascading within an interconnection.

(R1) Ensure entity’s risk assessment process includes Transmission stations/substations planned in the next 24 months.

(R1) Ensure risk assessment(s) covers each Transmission station/substation meeting applicability described in Section 4.1.

(R1 Part 1.1) If applicable, review any prior risk assessments and verify whether or not Transmission stations/substations were identified.

(R1 Part 1.1) Review evidence that risk assessment was performed and verify that it occurred within the past 30 months where items were identified in the previous risk assessment and 60 months where no items were identified in the previous risk assessment.

Note to Auditor: Review entity’s answer to the above Question and if the auditor can verify the answer is ‘no,’ Requirements R3-R6 do not apply and no further audit testing of Requirements R3-R6 is necessary, unless the entity performs the Transmission Operator function for a station/substation meeting the criteria of Requirment R1, Part 1.2. The 24 month period referenced for Transmission stations/substations planned to be in service is as of the date of the risk assessment not the date of the audit. See above Note Concerning Third Party Verifications for important details regarding audit risk assessment and related rigor of audit procedures to be applied for this Requirement.


• Requirement R2 requires the Transmission Owner to have an unaffiliated third party verify the risk assessment performed under Requirement R1. The verification may occur concurrent with or after the risk assessment performed under Requirement R1.

• This requirement includes confidentiality provisions.

Requirement R2




(R2) Review evidence of third party verification of the entity’s risk assessment and verify the following:

(R2 Part 2.1) The reviewing entity is registered in accordance with Part 2.1 or has transmission planning or analysis experience.

(R2 Part 2.2) Verification was completed within 90 calendar days of risk assessment. (R2 Part 2.3) Verifying entity’s recommendations, if any, were used to modify the entity’s Requirement

R1 identification or the technical basis for not modifying the Requirement R1 identification is documented within 60 calendar days of completion of the verification.

(R2 Part 2.4) Review non-disclosure agreement (or other evidence) to verify procedures for protecting sensitive or confidential information between the entity and third party were implemented.

Note to Auditor See Guidelines and Technical Basis section of the standard and Rationale for Requirement R2 associated with the Standard for additional details regarding the term ‘unaffiliated.’ The third party verification may occur concurrent with or after the risk assessment performed under Requirement R1.


• Requirement R3 requires the Transmission Owner notify the Transmission Operator that operates the primary control center identified in Requirement R1 and verified under Requirement R2 of such identification.

Requirement R3




(R3) For each applicable primary control center identified in Requirement R1 Part 1.2 not under the control of the entity’s registration, verify notification exists and contains the date of completion of Requirement R2.

(R3 Part 3.1) For each Transmission station/substation removed under Part 3.1, ensure the responsible Transmission Operator was notified of the removal within seven calendar days of removal from identification.

Note to Auditor: Note the entity’s response to the above Question. If auditor can verify the entity’s answer of ‘No,’ then Requirement R3 is not applicable and no further audit testing is required.


• Requirement R4 requires the Transmission Owner and Transmission Operator of identified facilities to conduct an evaluation of potential physical threats and vulnerabilities

• Evaluation must consider any unique characteristics of the facility, prior history or attack on similar facilities and intelligence or threat warnings.

Requirement R4




(R4) Review evidence of evaluation and verify it considers the following: (R4) Potential threats and vulnerabilities as described in Requirement R4. (R4 Part 4.1) Unique characteristics as described in Requirement R4 Part 4.1. (R4 Part 4.2) Prior history of attack on similar facilities taking into account the frequency, geographic

proximity, and severity of past physical security related events. (R4 Part 4.3) Intelligence or warnings as described in Part 4.3. Note to Auditor: See above Note Concerning Third Party Verifications for important details regarding audit risk assessment and related rigor of audit procedures to be applied for this Requirement. Auditor should cross reference the Transmission stations/substations and primary control centers identified in the risk assessment performed under Requirement R1 to the evaluation prescribed in Requirement R4 to ensure it is complete.


• Requirement R5 requires the Transmission Owner and Transmission Operator develop and implement a documented security plan covering each identified facility within 120-calendar days following completion of Requirement R2.

• The physical security plan is to include the following attributes: Resiliency or security measures designed collectively to deter, detect,

delay, assess, communicate, and respond to potential physical threats and vulnerabilities identified during the evaluation conducted in Requirement R4. Law enforcement contact and coordination information. A timeline for executing the physical security enhancements and

modifications specified in the physical security plan. Provisions to evaluate evolving physical threats and their corresponding

security measures.

Requirement R5




(R5) Review evidence and verify the physical security plan(s) covers the Transmission stations/substations and primary controls identified in Requirements R1 and/or R2, and verify plan was developed within 120 calendar days following the completion of Requirement R2 and executed according to the timeline specified in the physical security plan(s). In addition, verify the plan includes the following attributes:

(R5 Part 5.1) Resiliency or security measures designed collectively to deter, detect, delay, assess, communicate, and respond to potential physical threats and vulnerabilities identified in the evaluation conducted in Requirement R4.

(R5 Part 5.2) Law enforcement contact and coordination information. (R5 Part 5.3) A timeline for executing physical security enhancements and modifications specified in

the physical security plan. (R5 Part 5.4) Provisions to evaluate evolving physical threats, and their corresponding security

measures in accordance with R5 Part 5.4 (R5) Verify implementation of physical security plan(s). See ‘Note to Auditor’ for details. Note to Auditor: See above Note Concerning Third Party Verifications for important details regarding audit risk assessment and related rigor of audit procedures to be applied for this Requirement. Auditor should cross reference the Transmission stations/substations and primary control centers identified in the risk assessment performed under Requirement R1 to the evaluation prescribed in Requirement R4 and the security plan(s) prescribed in Requirement R5 to ensure the plan addresses vulnerabilities to physical attacks per the evaluation conducted in Requirement R4. Requirement R5 includes implementation of the security plan(s), which is not within the scope of the third party review described in Requirement R6. Auditors can gain reasonable assurance security plan(s) was/were implemented by determining if specific actions prescribed by the plan(s) have taken place within the timelines established by the plan(s). For example, if the plan calls for certain procedures to occur, then auditors could ask for evidence demonstrating the procedure has been implemented within the timeline established in the security plan. Also, if the plan calls for construction of a barrier, an auditor could verify evidence that such a barrier was constructed in accordance with the entity’s timeline. As auditors should obtain reasonable, not absolute, assurance the plan(s) was/were implemented, testing implementation on a sample basis may be appropriate.


• Requirement R6 requires the Transmission Owner and Transmission Operator obtain a review of the threat evaluation (Requirement R4) and security plan (Requirement R5)

• Transmission Owner/Transmission Operator shall select an unaffiliated third party reviewer

• Reviewer may suggest revisions. Transmission Owner/Transmission Operator can either modify its evaluation or security plan(s) consistent with the recommendation or document its reasons for not modifying its evaluation or security plan(s).

• This requirement includes confidentiality provisions.

Requirement R6




(R6) Review evidence and verify the physical security plan(s) and the Requirement R4 evaluation have been reviewed by an unaffiliated third party. Also, review evidence and verify the following:

(R6 Part 6.1) Reviewing party has the qualifications identified in Part 6.1. (R6 Part 6.2) Review is dated within 90 calendar days of completion of the Requirement R5 security

plan. (R6 Part 6.3) Reviewing entity recommended changes to security plan(s) were made by entity or the

reason(s) for not making the change(s) was/were documented within 60 calendar days of the completion of the unaffiliated third party review.

(R6 Part 6.4) Review non-disclosure agreement (or other evidence) to verify procedures for protecting sensitive or confidential information between entity and third party were implemented.

Note to Auditor: The third party review may occur concurrent with or after the evaluation performed under Requirement R4 or the security plan develop under Requirement R5. See Guidelines and Technical Basis associated with the Standard for additional details related to qualifications of reviewing entities that may inform audited entities selection of a reviewing entity.


• The following slides contains specific requirements language. • This language was developed and posted for the final ballot on

May 1, 2014.

CIP-014-1 Requirements


• R1. Each Transmission Owner shall perform an initial risk assessment and subsequent risk assessments of its Transmission stations and Transmission substations (existing and planned to be in service within 24 months) that meet the criteria specified in Applicability Section 4.1.1. The initial and subsequent risk assessments shall consist of a transmission analysis or transmission analyses designed to identify anythe Transmission station(s) and Transmission substation(s) that if rendered inoperable or damaged could result in widespread instability, uncontrolled separation, or Cascading within an Interconnection. [VRF: High; Time-Horizon: Long-term Planning]

Requirement R1


1.1 Subsequent risk assessments shall be performed: At least once every 30 calendar months for a Transmission Owner that has

identified in its previous risk assessment (as verified according to Requirement R2) one or more Transmission stations or Transmission substations that if rendered inoperable or damaged could result in widespread instability, uncontrolled separation, or Cascading within an Interconnection; or At least once every 60 calendar months for a Transmission Owner that has

not identified in its previous risk assessment (as verified according to Requirement R2) any Transmission stations or Transmission substations that if rendered inoperable or damaged could result in widespread instability, uncontrolled separation, or Cascading within an Interconnection.

Requirement R1


Requirement R1

1.2 The Transmission Owner shall identify the primary control center that operationally controls each Transmission station or Transmission substation identified in the Requirement R1 risk assessment.


R2. Each Transmission Owner shall have an unaffiliated third party verify the risk assessment performed under Requirement R1. The verification may occur concurrent with or after the risk assessment performed under Requirement R1. [VRF: Medium; Time-Horizon: Long-term Planning]

2.1 Each Transmission Owner shall select an unaffiliated verifying entity that is either: A registered Planning Coordinator, Transmission Planner, or Reliability

Coordinator; or An entity that has transmission planning or analysis experience.

Requirement R2


2.2 The unaffiliated verifying entitythird party verfication shall either verify the Transmission Owner’s risk assessment performed under Requirement R1, or recommendwhich may include recommendations for the addition or deletion of a Transmission station(s) or Transmission substation(s). The Transmission Owner shall ensure the verification is completed within 90 calendar days following the completion of the Requirement R1 risk assessment. 2.3 If the unaffiliated verifying entity recommends that the Transmission Owner add a Transmission station(s) or Transmission substation(s) to, or remove a Transmission station(s) or Transmission substation(s) from, its identification under Requirement R1, the Transmission Owner shall either, within 60 calendar days of completion of the verification, for each recommended addition or removal of a Transmission station or Transmission substation: Modify its identification under Requirement R1 consistent with the recommendation; or Document the technical basis for not modifying the identification in accordance with the

recommendation.

Requirement R2


Requirement R2

2.4 Each Transmission Owner shall implement procedures, such as the use of non-disclosure agreements, for protecting sensitive or confidential information exchanged withmade available to the unaffiliated third party verifier and to protect or exempt sensitive or confidential information developed pursuant to this Reliability Standard from public disclosure. verifying entity.


Requirement R3

R3. For a primary control center(s) identified by the Transmission Owner according to Requirement R1, Part 1.2 and that a) operationally controls an identified Transmission station or Transmission substation verified according to Requirement R2 that, and b) is not under the operational control of the Transmission Owner,: the Transmission Owner shall, within seven calendar days following completion of Requirement R2, notify the Transmission Operator that has operational control of the primary control center of such identification and the date of completion of Requirement R2. [VRF: Lower; Time-Horizon: Long-term Planning]


Requirement R3

3.1 If a Transmission station or Transmission substation previously identified under Requirement R1 and verified according to Requirement R2 is removed from the identification during a subsequent risk assessment performed according to Requirement R1 or a verification according to Requirement R2, then the Transmission Owner shall, within seven calendar days following the verification or the subsequent risk assessment, notify the Transmission Operator that has operational control of the primary control center of the removal.


Requirement R4

R4. Each Transmission Owner that owns or operatesidentified a Transmission station, Transmission substation, or a primary control center identified in Requirement R1 and verified according to Requirement R2, and each Transmission Operator notified by a Transmission Owner according to Requirement R3 that the Transmission Operator’s primary control center has operational control of an identified Transmission station or Transmission substation, shall conduct an evaluation of the potential threats and vulnerabilities of a physical attack to each of their respective Transmission station(s), Transmission substation(s), and primary control center(s) identified in Requirement R1 and verified according to Requirement R2. The evaluation shall consider the following: [VRF: Medium; Time-Horizon: Operations Planning, Long-term Planning]


Requirement R4

4.1 Unique characteristics of the identified and verified Transmission station(s), Transmission substation(s), and primary control center(s); 4.2 Prior history orof attack on similar facilities taking into account the frequency, geographic proximity, and severity of past physical security related events; and 4.3 Intelligence or threat warnings received from sources such as law enforcement, the Electric Reliability Organization (ERO), the Electricity Sector Information Sharing and Analysis Center (ES-ISAC), U.S. federal and/or Canadian governmental agencies, or their successors.


Requirement R5

R5. Each Transmission Owner that owns or has operational control ofidentified a Transmission station, Transmission substation, or primary control center identified in Requirement R1 and verified according to Requirement R2, and each Transmission Operator notified by a Transmission Owner according to Requirement R3 that the Transmission Operator’s primary control center has operational control of an identified Transmission station or Transmission substation, shall develop and implement a documented physical security plan(s) that covers their respective Transmission station(s), Transmission substation(s), and primary control center(s). The physical security plan(s) shall be developed within 120 calendar days following the completion of Requirement R2. and executed according to the timeline specified in the physical security plan(s). The physical security plan(s) shall include the following attributes: [VRF: High; Time-Horizon: Long-term Planning]


Requirement R5

5.1 Resiliency or security measures designed collectively to deter, detect, delay, assess, communicate, and respond to potential physical threats and vulnerabilities based on the results ofidentified during the evaluation conducted in Requirement R4. 5.2 Law enforcement contact and coordination information. 5.3 A timeline for implementingexecuting the physical security enhancements and modifications specified in the physical security plan. 5.4 Provisions to evaluate evolving physical threats, and their corresponding security measures, to the Transmission station(s), Transmission substation(s), or primary control center(s).


Requirement R6

R6. Each Transmission Owner that owns or operatesidentified a Transmission station, Transmission substation, or primary control center identified in Requirement R1 and verified according to Requirement R2, and each Transmission Operator notified by a Transmission Owner according to Requirement R3 that the Transmission Operator’s primary control center has operational control of an identified Transmission station or Transmission substation, shall have an unaffiliated third party review the evaluation performed under Requirement R4 and the security plan(s) developed under Requirement R5. The review may occur concurrently with or after completion of the evaluation performed under Requirement R4 and the security plan development under Requirement R5. [VRF: Medium; Time-Horizon: Long-term Planning]


Requirement R6

6.1 Each Transmission Owner and Transmission Operator shall select an unaffiliated third party reviewer from the following:

• 6.1.1 An entity or organization with electric industry physical security experience and whose review staff has at least one member who holds either a Certified Protection Professional (CPP) or Physical Security Professional (PSP) certification.

• 6.1.2 An entity or organization approved by the ERO. • 6.1.3 A governmental agency with physical security expertise. • 6.1.4 An entity or organization with demonstrated law enforcement,

government, or military physical security expertise.


6.2 The Transmission Owner or Transmission Operator, respectively, shall ensure that the unaffiliated third party review is completed within 90 calendar days of completing the security plan(s) developed in Requirement R5. The unaffiliated third party review may, but is not required to, include recommended changes to the evaluation performed under Requirement R4 or the security plan(s) developed under Requirement R5. 6.3 If the unaffiliated reviewing entitythird party reviewer recommends changes to the evaluation performed under Requirement R4 or security plan(s) developed under Requirement R5, the Transmission Owner or Transmission Operator shall, within 60 calendar days of the completion of the unaffiliated third party review, for each recommendation: Modify its evaluation or security plan(s) consistent with the recommendation; or Document the reason(s) for not modifying the evaluation or security plan(s) consistent

with the recommendation.

Requirement R6


Requirement R6

6.4 Each Transmission Owner and Transmission Operator shall implement procedures, such as the use of non-disclosure agreements, for protecting sensitive or confidential information exchanged withmade available to the unaffiliated third party reviewerreviewing entity and to protect or exempt sensitive or confidential information developed pursuant to this Reliability Standard from public disclosure.

Port Scans of Physical Access Control System Boards

The missing piece of the puzzle!

By Matt Bordelon [email protected] Thanks to Phillip Miller for his help

Presentation Targets What ‘s a PACS system?

Typical configuration (Server- Processor – I/O – Doors)

Things to consider/ concerns

Configuring a test system (boards-laptop-OS)

nmap command and parameters

Example output

UDP versus TCP

Considering what to disable

Physical Access Control System

BCS PSP

ENERGY OPS PSP CTOC PSP

PACS Server



Processor Board

Input - Output Board

PACS Server

Corporate Network

TCP/IP

Input - Output Board

Serial

Serial

Physical Access Point (i.e. Door)

Card Reader

Outside Inside

Magnetic Lock

Emergency Exit

Hard Wired

Located local to the PSP

Located Remote to the PSP

Within scope of

Vulnerability Assessment

Concerns

Running port scans on an embedded system board was not like running one on other machines

o Concerned over the lack of a high level operating system Afraid to “Brick” the system- essentially making it unusable. Would we possibly turn off something that was needed for the

system to run or communicate to it?

o How would we run the port scan on a live system without affecting the CIP requirements of the PSP (i.e. what happens to the doors)?

First Step

Vital to communicate with your vendor- these are embedded systems o If you have a contract for support, take advantage of it

Be specific in what you’re planning to do The vendor may not be familiar with customers performing port scans

on their boards The vendor may have to ask their board supplier about the effects of

running the scan- depends on who is writing the firmware Find out how to reset your board to factory defaults in case a mistake

is made You will need to know which ports and services are essential to the

system’s operation Also will need to know the effects of disabling the non-essential ports

and services. Some may not be able to be reset.

Live System

Weigh your options when it comes to performing a scan on a live system o It’s best if you can run your scan on a live system, but you may not be able.

o You may have to configure a test system which accurately represents the environment of the processor boards

Acquire the same model processor boards used for each of the PSPs

Get a power supply

Mounted processor boards to a piece of plywood- not elegant, but hey it works

Wire boards together and to the power supply

Configure the boards just like they are in the field

Example of a Test setup

Laptop

Computer cable

Power Supply

Processor Boards

Preparation

Read processor board documentation

o Need to find out how to connect to the processor boards via a laptop For this example:

Firmware allowed you to use a web browser over a http connection

Had to adjust the dip switches on board to connect and then switch them back again- be sure to power down each time

Configure the laptop to a specific IP Address

Also set a static IP Address of the board via the board’s web interface

Preparation

Laptop

o Disabled the wireless interface

o Set the IP Address according to the board’s documentation

o Used a version of Linux to boot-up on- Kali Linux Live CD Kali Linux is designed for use in Penetration Testing Comes with nmap (used for port scan) as part of the distribution Double checked the “eth0” network interface to make sure it matches

the IP Address that is required by the board

o Connected the laptop to the TCP/IP connectors with a CAT-5 patch/crossover cable.

Checking eth0 in Kali Linux

Right click on the network icon (red arrow). Then left click the edit connections menu

Terminal for nmap Network Connections


Network Connections window

Click on the wired connection name and choose edit. Choose the IPv4 Settings Tab and choose the Manual method and enter in the Laptop’s IP address, Netmask (typically 255.255.255.0), then the gateway (can be 0). Of course choose save at the end of the process.


Configuring the Linux eth0 network connection on laptop “eth0” is the first network device (Ethernet card) connected to a computer that a Linux operating system identifies.

Check this out if are getting problems with running the nmap command- giving you a can’t find target error.

1. Right click on the network icon in the upper right of the Kali Linux display 2. Left click on “Edit connections” 3. Select “wired connection 1” 4. Click on “edit” 5. Select the “IPv4 Setting” tab on the pop-up display 6. Type in under address, xxx.xxx.xxx.xxx, this is the IP Address that we will

give to eth0 7. Type in “255.255.255.000” for the subnet mask 8. Click on “add” 9. Click on “Save”

What’s nmap?

Nmap (www.nmap.org)

o Open source utility used in finding devices on your network and in device security audits- has many other uses

o Supported by most operating systems- Linux is the most popular

o Just type on a command line the nmap command and options- next slide

nmap command & breakdown

The nmap command used to run the port scan is as follows: nmap –n –T3 –sS –sU –p1-65535 xxx.xxx.xxx.xxx |tee filename.txt Breakdown: -n is no DNS resolution. No need because we are connecting directly to the board- speeds the process -T3 adjustment to the overall speed of the command 0- very slow to 5- very fast -sS runs a scan of TCP ports in a “stealthy” fashion. Very popular and fastest way to scan TCP ports and works against all functional TCP stacks -sU runs a scan of your UDP ports. So scan will be of your TCP & UDP ports –p1-65535 defines the port range of your scan. This one does from ports 1 through 65K which covers all your 16 bit addressing. Also can use –p- which covers all ports as well xxx.xxx.xxx.xxx is the Processor board IP Address the scan is running on

|tee “tee” will display on your screen the results of your scan and like a plumber’s tee will also direct it to a text file. In our example, it writes to the current directory.

filename.txt Is the name of the text file you want your output to go to. You can also use |tee –a filename.txt if you want to append to the original file (in case you are running more than one scan and want the output in one file)

Tee Pipe (shift \)

nmap example output Starting Nmap 6.40 ( http://nmap.org ) at 2014-02-19 09:33 UTC Stats: 0:00:17 elapsed; 0 hosts completed (1 up), 1 undergoing SYN Stealth Scan SYN Stealth Scan Timing: About 4.41% done; ETC: 09:39 (0:05:47 remaining) Stats: 0:00:20 elapsed; 0 hosts completed (1 up), 1 undergoing SYN Stealth Scan SYN Stealth Scan Timing: About 5.27% done; ETC: 09:39 (0:05:42 remaining) Stats: 0:22:56 elapsed; 0 hosts completed (1 up), 1 undergoing UDP Scan UDP Scan Timing: About 32.38% done; ETC: 10:32 (0:36:04 remaining) Stats: 0:22:59 elapsed; 0 hosts completed (1 up), 1 undergoing UDP Scan UDP Scan Timing: About 32.46% done; ETC: 10:32 (0:36:02 remaining) Stats: 0:23:00 elapsed; 0 hosts completed (1 up), 1 undergoing UDP Scan UDP Scan Timing: About 32.48% done; ETC: 10:32 (0:36:02 remaining) Stats: 0:23:00 elapsed; 0 hosts completed (1 up), 1 undergoing UDP Scan UDP Scan Timing: About 32.49% done; ETC: 10:32 (0:36:01 remaining) Stats: 0:23:00 elapsed; 0 hosts completed (1 up), 1 undergoing UDP Scan UDP Scan Timing: About 32.50% done; ETC: 10:32 (0:36:00 remaining) Stats: 0:23:00 elapsed; 0 hosts completed (1 up), 1 undergoing UDP Scan UDP Scan Timing: About 32.50% done; ETC: 10:32 (0:36:00 remaining) Stats: 0:23:00 elapsed; 0 hosts completed (1 up), 1 undergoing UDP Scan UDP Scan Timing: About 32.51% done; ETC: 10:32 (0:36:01 remaining) Stats: 0:23:01 elapsed; 0 hosts completed (1 up), 1 undergoing UDP Scan UDP Scan Timing: About 32.52% done; ETC: 10:32 (0:36:00 remaining) Stats: 0:23:01 elapsed; 0 hosts completed (1 up), 1 undergoing UDP Scan UDP Scan Timing: About 32.53% done; ETC: 10:32 (0:35:59 remaining) Stats: 0:23:01 elapsed; 0 hosts completed (1 up), 1 undergoing UDP Scan UDP Scan Timing: About 32.53% done; ETC: 10:32 (0:35:59 remaining) Stats: 0:23:01 elapsed; 0 hosts completed (1 up), 1 undergoing UDP Scan UDP Scan Timing: About 32.54% done; ETC: 10:32 (0:36:00 remaining) Stats: 0:23:31 elapsed; 0 hosts completed (1 up), 1 undergoing UDP Scan UDP Scan Timing: About 33.44% done; ETC: 10:32 (0:35:31 remaining)

nmap example output

Nmap scan report for 192.168.0.151 Host is up (0.00081s latency). Not shown: 131062 closed ports PORT STATE SERVICE 80/tcp open http 443/tcp open https 3001/tcp open nessus 4001/tcp open newoak 67/udp open|filtered dhcps 68/udp open|filtered dhcpc 161/udp open|filtered snmp 5353/udp open|filtered zeroconf MAC Address: 02:0E:E5:A2:5E:DA (Mercury Security) Nmap done: 1 IP address (1 host up) scanned in 3604.09 seconds

At the end of the scan we get what we were looking for

Possible Actions

Ensure that default user names and passwords are changed

Contact vendor about what you found and inquire about your options to disable non-essential services.

Make sure that you identify the port and its service that is used to communicate with the server. Obviously this one should not be disabled.

Consider leaving the port/service necessary to communicate with the board enabled

Test the board reset procedure from your vendor. Make sure you can go back to factory defaults.

Check with your vendor to see how and which services can be disabled. User Interfaces may be your only option

Run another scan after making changes to see that the ports/services have been disabled

nmap example output- Checking closed ports

Nmap scan report for 192.168.0.151 Host is up (0.00081s latency). Not shown: 131062 closed ports PORT STATE SERVICE 80/tcp open http 443/tcp open https 3001/tcp open nessus 4001/tcp open newoak 67/udp open|filtered dhcps 68/udp open|filtered dhcpc 161/udp open|filtered snmp 5353/udp open|filtered zeroconf MAC Address: 02:0E:E5:A2:5E:FA (Mercury Security) Nmap done: 1 IP address (1 host up) scanned in 3604.09 seconds

Original

Nmap scan report for 192.168.0.101 Host is up (0.00077s latency). Not shown: 131064 closed ports PORT STATE SERVICE 80/tcp open http 443/tcp open https 3001/tcp open nessus 4001/tcp open newoak 67/udp open|filtered dhcps 68/udp open|filtered dhcpc MAC Address: 02:0E:E5:A2:5E:FA (Mercury Security) Nmap done: 1 IP address (1 host up) scanned in 3605.13 seconds

Check

No, the I/O board is not running Nessus..

• As much as we would love for this PACS I/O board to run Nessus (CIP v5 compliant), this is not the case.. – Yes, the board has port 3001 open, but the listening service is not

Nessus

– Actually, the vendor’s service associated with this port is used to communicate with the PACS server

– /etc/services contains the “general” services listening on a particular port.. However this is not always correct

• /etc/services is a file that lists ports and services normally associated with that port. If the scan sees the port is open it assumes this service is listening

– You will need to work with your vendor in order to help you identify these ports and their services

No, the I/O board is not running Nessus..

Port 3001?

Scan

OK /etc/services file says you

should be NESSUS

/etc/services file

Hello!

Port 3001

Could be any service

Open |Filtered ??? PORT STATE SERVICE 67/udp open|filtered dhcps

• open|filtered means that Nmap didn’t get a response when the port was scanned – Ports are placed in this state when nmap doesn’t know if a port is open or filtered. – UDP open ports give no response – A “No response” could mean that a packet filter dropped the probe or any response it

elicited. – Nmap doesn’t know if the port is open or the packet is being filtered – In this example the UDP protocol is used

• Closed TCP or UDP ports will normally respond with an ICMP* message (Destination unreachable)

*Internet Control Message Protocol- used by networked devices for sending error messages, diagnostics and controls

TCP Packet Scan

UDP Packet Scan

What did we learn here? • How a Physical Access Control System works

• How one can setup an applicable test system

• How to use nmap to run the port scan

• What you see may not be what is actually running

• How nmap output responses differ between UDP and TCP ports

• Work with your vendor

Questions?

PSP Maintenance Laptop Configuration and Operation

By Matt Bordelon [email protected] Thanks to Phillip Miller for his help

So what’s a PSP Maintenance Laptop?

Laptop used to perform maintenance on your cyber assets within your PSPs even those connected to your ESP

Connected via the external maintenance port on your critical devices

Configured so that the cyber attack surface is as small as possible

Following your procedures, it can be transported from one PSP to another

Costs for this setup are relatively low

An old redeployed laptop Open-source operating system (LiveCD)

What’s the takeaway from this presentation?

• Learn how Cleco setup its PSP Maintenance Laptop.

• What hardware adjustments you have to make

• How to provide a read-only operating system

• Limiting the connection capability

• Show which operating system Cleco went with and how it’s used

• Guidance on how to connect to a device using the laptop via a serial connection

Cleco’s PSP Maintenance Laptop

Hardware

• Started with a used laptop • Didn’t have to worry much about its capabilities (e.g. hard drive

size, RAM, etc.)

• Removed the hard drive • Forces the unit to use an operating system other than the HD

• Cleco uses a CD for the operating system (read-only) • Have one less place to store anything malicious

• Your Bios boot sequence will try to boot off the Hard Drive first • You can alter the boot sequence to boot off of the CD first • Can Password protect the Bios (battery removal will reset though)

• Removed the wireless card • If you can’t, then you could disable it from the Bios screen • Reduces the number of external connections

Hardware – Removing HD and wireless card

Screws


Hard Drive

Wireless Card


Wireless Card

Hard Drive

Note the tape over connections

Software- Operating System

• Cleco used a Linux Live CD to boot off of • By using a CD-R nothing can write to it • You can either leave the CD in the laptop or secure it somewhere

• Using System Rescue CD (www.sysresccd.org) • Based on Gentoo Linux • Download the iso image from the download page • Use an application to burn the image to the CD • Has the “minicom” serial communication application with it

• You can preconfigure the Live CD ahead of time before burning • User accounts • Password • Setting up minicom • Can remove wireless drivers (in case you still have the card installed) • Tutorials on-line (Google: personalize system rescue cd)

Procedure

• USB Serial cable

USB

Serial

Procedure

• Device Serial connection

Procedure

• Stick the CD in the laptop and turn on machine

• Boot up on Linux Gentoo

• Disable the Ethernet port (if not disabled in the BIOS)

• Plug in the USB to serial adaptor

• Operating System will automatically create a device, usb tty file in the /dev folder. (/dev folder is where the device files are located)

Procedure – (continued)

• Use the Minicom application to connect serially to device • Texted based modem control & terminal emulation program • Configure the Serial settings (Buad rate, stop bits, etc.) • Modem initialization warning:

• Minicom sends out a modem initialization string which might cause unexpected actions on the connected device

• Allow the initialization to complete before connecting the laptop to the management port

• (could be disabled or removed when personalizing the LiveCD)

Procedure

• Will have to consider documentation of laptop usage in your security procedure.

• Document usage on a log sheet which stays with the laptop.

Procedure

• Cleco placed a sticker on the Laptop for awareness. • Prevents someone from mistakenly using the laptop

Considerations

• What’s the future of serial communications • Some firmware updates are so large now that doing it serially will

be very difficult • Will devices continue offering serial communication ports due to

the speed and throughput? • One alternative would be to use the Ethernet port

• How about configuring the Ethernet port with a Time-To-Live (TTL) of 1?

• You can plug into an Ethernet maintenance port and the packets can only go to the device connected. Packets would not be able to be routed to a different network.

• You can potentially still connect to the local subnet though • Setting the management port to a different subnet to prevent routing

Questions?

Vulnerability Assessment and Penetration Testing

Presenters:

Bruce Upton CISSP, CISA, C|EH

[email protected]

Jerry McClurg CISSP, CISA, C|EH

[email protected]

Agenda and Overview:

Vulnerability Scanning and Identification:

• What is a vulnerability assessment? • Internal versus external scanning an testing • How are vulnerability tests different from penetration tests? • Vulnerability scanning tools and reporting • Additional considerations:

• Understanding vulnerability assessments and penetration tests are only valid for a short period of time

• Continuous monitoring • Management oversight

What’s the Difference?:

Vulnerability Identification versus Penetration Testing: • Vulnerability Assessment: Generally, a vulnerability assessment is an

automated scan of network resources resulting in a detailed report of security vulnerabilities.

• Penetration Test: Penetration testing incorporates vulnerability scanning and identification, but additional effort is applied in an attempt to exploit identified vulnerabilities.

• Vulnerability assessments and penetration tests are both good security due-diligence.

What’s the Difference?:

Vulnerability Identification versus Penetration Testing: • A vulnerability assessment may identify the following security weaknesses:

• Users have local administrator rights on their Windows 7 computers. • Users can access most websites on the Internet. • Users have the authority to run programs from within Internet Explorer.

• A penetration test would identify the security weaknesses, but go quite a bit further:

• Users have local administrator rights on their Windows 7 computers. • Users can access most websites on the Internet. • Users have the authority to run programs from within Internet Explorer. • A user was convinced to visit a phishing website • The user ran a “connection test” application • Symantec Antivirus did not detect the “connection test” application • Unauthorized remote access was obtained into the network.

Internal versus External:

Generally speaking, there are two types of assessments: • Internal Assessment: The vulnerability scan or penetration test is

performed from inside the organization. The engineer(s) either physically visit the organization or gain secure remote access. The test simulates an attack from the inside-out. This overall approach will:

• Identify internal devices (enumeration) • Identify services and footprint internal devices • Identify internal security weaknesses in the following, at a minimum, categories:

• Patch management, network segregation, network access controls, data security, intrusion detection systems (IDS) testing, SCADA (if in scope), key management and crypto security (if in scope);

• Password practices and overall PC, server, and network device security due-diligence, etc.

Internal versus External:

Generally speaking, there are two types of assessments: • External Assessment: The vulnerability scan or penetration test is

performed from outside the organization. The engineer(s) test the organizations infrastructure using an outside-in approach. This overall approach will:

• Identify external devices (enumeration) • Identify services and foot print external devices • Identify external security weaknesses in the following, at a minimum, categories:

• Firewall security, remote access portals, database management system (DBMS) security, web application security, intrusion detection systems (IDS) and intrusion prevention services (IPS) testing.

Testing Quality:

Testing quality and effectiveness: • The overall effectiveness of your assessment is generally based on three

main factors: • The effectiveness and thoroughness of the scanning toolset(s) • The overall quality and talent of the internal and/or external security firm or

personnel • Certifications, experience, etc.

• How effectively the security firm and internal departments work together • Free flow of information between the firm and key departments is central to the success of

an assessment

Toolsets:

Vulnerability scanning toolsets: • The effectiveness and thoroughness of a vulnerability assessment is heavily based on

toolsets. Some effective toolsets include: • Qualys

• Internal and external vulnerability scanning • Low false-positive rates • Pay per IP model

• Rapid7 • Internal and external vulnerability scanning • Low false-positive rates • Pay per IP model

• Nessus • Internal and external vulnerability scanning • Effective pricing model • In our experience, Nessus tends to have a higher false-positive rate

• Nexpose • Internal and external vulnerability scanning • Community edition available • Low false-positive rates

Toolsets:

Vulnerability scanning toolsets: • Qualys

• Enterprise scanning tool with a number of compliance modules

Toolsets:


• Reporting, remediation tracking and a large knowledgebase:

Toolsets:


• Advantages: • Low false-positive rates • Detailed reporting • Remediation tracking • Most vulnerabilities identified will have resolution strategies

• Disadvantages • All scan data is stored in the cloud at Qualys • Pay-per-IP model makes scanning large IP blocks very expensive

Toolsets:

Vulnerability scanning toolsets: • Nexpose

• Enterprise-class with a low false-positive rate, strong reporting and numerous compliance templates

Toolsets:


• Extensive compliance and simulation testing:

Toolsets:


• Advantages: • Low false-positive rates • Detailed reporting • Many compliance and simulation scan templates • Most vulnerabilities identified will have resolution strategies • It’s not a pay-per-IP scanning solution, potentially making it a good fit for internal scanning

and testing • Disadvantages

• Like most vulnerability scanners it generates a lot of network traffic. It could cause network latency or denial-of-service if it’s not configured properly.

• Resource intensive • It’s pricy at about $10,000 to $15,000 depending on the scope and services needed

Tool Availability:

Overall, We’re seeing two concerning trends today: • The availability of hacking tools is unprecedented. Free tools to:

• Exploit websites • Identify vulnerabilities • Perform data mining • Hacking wireless networks, etc.

• Tools available to hide your tracks and/or become virtually invisible are at an all time high. Examples include:

• VPN solutions that don’t keep long-term logs • ProXPN - pro

• TOR • Tor is free software and an open network that helps you defend against traffic analysis, a form of

network surveillance that threatens personal freedom and privacy, confidential business activities and relationships, and state security – torproject.org

• The world has learned just how effective TOR is: the Snowden leaks demonstrated the NSA has not broken TOR (only circumvented it)

Why are we concerned?

The following attack was performed using publicly-available software and our origin was successfully masked: • First step, go dark (anonymous) to obscure where your Internet

traffic is originating from. In our case, it looks like we’re coming from Germany:


The following attack was performed using publicly-available software and our origin was successfully masked: • An outside-in approach was used and it starting with Google

hacking:

• The hacker identifies a target (neither of these websites were used in our demonstration):


The following attack was performed using publically-available software and our origin was successfully masked: • An escape string is used to look for an SQL-injection vulnerability:

• Inserting the string generated an error:


The error message tells us it’s likely vulnerable to an SQL-injection attack… • Further testing reveals it is, and the following information is

initially obtained via SQL-injection strings:


SQL-injection commands are further used to extract information: • Database table names are obtained using an SQL-injection string:

• Count(table_name) of information_schema.tables where table_schema=0x67656D656469615F7073 is 27

• Note the “users” table


SQL-injection commands are further used to extract information: • Table field names are extracted from the “users” table:

• Count(column_name) of information_schema.columns where table_schema=0x67656D656469615F7073 and table_name=0x7573657273 is 13


SQL-injection commands are further used to extract information: • Username and password field data are extracted from the

“users” table: • Count(*) of XXXXXXXX_ps.users is 4


Data extraction: • Testing revealed the passwords are encrypted

• How are they encrypted? • How do we find out? • Is there a way to decrypt them?


Password decryption: • There are a number of off-line tools such as Hashcat and

L0phtCrack that can be used to launch brute-force or dictionary attacks.

• A number of websites specialize in dictionary lookups • Cloudcracker.com • Crackstation.net • md5decrypter.co.uk


Password decryption: • We were able to successfully identify how the passwords were

encrypted, and we were able to decrypt two of three:


Malicious intent: • We stopped testing at this point, but unfortunately, most

blackhat hackers would not. • Web anonymity is a great way to encourage Internet privacy.

However, the tools to protect our Internet privacy are being used maliciously by hackers to cover their tracks.

Additional Considerations

Vulnerability and penetration testing frequency: • Vulnerability assessments and penetration tests are only valid for a

short period of time • For example, the second Tuesday of every month is known as

“Patch Tuesday”. The following Wednesday is known as “Exploit Wednesday”.

• To address these security gaps-in-time, continuous monitoring systems can be implemented:

• Bit9 Parity • Tripwire • FireMon


Assess and Identify All Ports, Programs, and Services : • The programs we have discussed should identify all active

ports/services • What About Software Programs or Inactive Processes?

• Tools that Identify All Installed Software • Microsoft Assessment and Planning (MAP) Toolkit • Emco Network Software Scanner

• Try to use at least two programs to check against each other



Assessment Plan: • Start from the External View

• View Information from a Hacker Viewpoint • Domain and DNS Registration • Gateway Routers, IDS, Firewalls • Email and DMZ Devices

• Internal Network • Internal/Rogue User Access • Vendor/Visitor Access • Remote Access


Document and Track Open Issues: • Easy to Lose Track Without Tracking and Follow-up • Schedule Regular Progress Updates and Report to Management

Summary and Take-Aways:

• Document and Follow an Assessment Plan • Maintain Multiple Software Toolsets • Stay Engaged—Technology Security is a Moving Target

Questions?