Emergency Restoration Plan...and Testing It!

1

Your Emergency Restoration Plan...and Testing It!

June 22, 2016

2

Today’s Objective

Vulnerability & Risk AssessmentDevelopment of the ERPExercising the ERPEvaluation and Critique

3

Requirements/Objectives of RUS

• Critical facilities and business functions:

– National security– Reliability and security of grid– Significant risk to safety and

health of public• Criticality and risk level associated

with assets• External impact (interdependency)• Threats and vulnerabilities• Existing & additional mitigation

procedures

Vulnerability andRisk Assessment

(VRA)

• Procedures for response and restoration efforts in event of major system outage

• Key contact information (external parties)

• Key utility management and other personnel

• Chain of command and delegation of authority responsibility

• Recovery procedures in case of loss of power to headquarters, key offices, or operation center facilities

• Business Continuity section• ERP available to key personnel

EmergencyRestoration Plan

(ERP)

• Verify, at least annually*:– Operability of alert and notification

systems– Efficacy of ERP– Employee competency with ERP

procedures– Points of Contact (POC) of key

personnel – internal and external– Contact numbers for POCs

*Per RUS, a borrower may exercise its ERP in a number of ways:

– After natural event that requires utilization of ERP

– Participation in joint exercises with other utilities

– Tabletop exercise

Annual ERP Exercise*

4

Vulnerability and Risk Assessment (VRA)

Recognized by RUS Bulletin 1730B-2 as an effective decision supporting tool for prioritizing and determining sites and functions within a business in need of security investment

and development of a meaningful ERP.

Risk = Threat x Vulnerability x Asset Value

5

VRA Objectives Understand the utility’s critical facilities and business

functions Identify threatening vulnerabilities of critical facility

systems Understand system design and operations in order to

determine likelihoods of failure Identify consequences of system failures Recommend facility improvements to reduce vulnerability

6

Core of the VRA MethodologyA

ll E

ntity

Ass

ets

Cri

tical

Ass

ets

Ass

ets a

t Ris

k

Miti

gatio

n Pr

iori

ties

Screen

Risk Mitigation

Threat

Vulnerability

Asset Value

Risk Assessment

• Universe of Potential Threats

• Likelihood of Event/Attack

• Infrastructure Interdependencies

• Existing Asset Protection

• Prioritization of Critical Facilities and Business Functions

= Risk Factor

times

times

Incremental Cost of Best Practices to Reduce/Limit:

• Vulnerability• Damage/Consequences• Restoration Time

versus

Existing Asset Protection/Mitigation

= Gap Analysisand

Cost/Benefit

Potential Debilitating or Material Financial Impacts on:

• National Security• Mission Critical

Operations• Stakeholders

Deemed to be Critical

7

Identifying New Threats

Are there any new threats to address in the ERP?‒ Coordinated Cyber or Physical Attacks‒ Insider Threat‒ Public & Media Relations Sensitivity‒ Changing Demographics

• Members• Staff

‒ Additional Critical Business Functions or Facilities

Is there potential for expansion or reorganization of the current ERP to better fit today’s landscape?

8

Emergency Restoration Plan (ERP)

Practical and functional tool which a utility can rely on for initial recovery efforts

Foundation under which a utility, under possible adverse and confusing conditions, can begin to restore its system including both physical and cyber components and efficiently utilizes its resources

In an emergency, all personnel should know:– What is my role?– Where should I go?

9

ERP Components

Executive Summary List of key contact emergency phone numbers List of key utility management and chain of command Procedures for recovery from loss of power to HW, key

offices and/or operation center facilities Business Continuity Section Plan to comply with eligibility requirements to qualify for

FEMA Public Assistance Grant Program Other items identified in VRA as essential for inclusion

10

ERP DevelopmentAnalysis of

Existing ERP

New ProcessDevelopmentand Review

ERP Draft ERP Review andAnnual Testing

• Review current ERP• Review existing

Complete Storm Plan• Identify procedural

shortcomings in existing ERP, if any

• Document relationship between VRA and existing ERP

– Critical assets identified in VRA that are not incorporated in existing ERP

– Potential threats/ consequences identified in VRA that are not anticipated in existing ERP

KeyActivities

• Review draft ERP with Board of Directors

• Finalize ERP report• Exercise ERP at least

annually to ensure operability and employee familiarity

• Draft ERP consolidate new processes, POCs and Business Continuity Plan (maintaining/restoring business operations during an event) into existing ERP

• On-site meeting to walk-through draft ERP and finalize document

• Develop restoration process and response timing for each new asset-threat combination

• Identify key points of contact (POCs), internal and external, for each asset-threat combination

• Define and document chain of command and delegation of authority in event of emergency

• Consult with department heads via conference call for clarification and additional information for draft ERP

11

Exercising the ERP

RUS borrowers are required to exercise their ERP at least annually in the following ways:– After natural event– Joint exercises with other utilities – Tabletop exercise

12

What is a Tabletop Exercise?From RUS Bulletin 1730b-2, Definitions, p. 4:

Tabletop Exercise: A hypothetical emergency response scenario in which participants get together around a table and identify the policy, communications, resources, data, coordination, and organizational elements associated with an emergency response.

13

How does a tabletop exercise work?RUS Bulletin 1730b-2, Exhibit C, p.13 :

Review, Conduct Training and Revise

“The tabletop exercise should be conducted with management and personnel who have a key emergency management responsibility. In a conference room setting, describe an emergency scenario and have participants discuss their responsibilities and how they would react to the situation. Based upon this discussion, identify areas of confusion and overlap, and modify the plan accordingly.”

14

Objectives of the Exercise• Test the existing ERP process and procedures, and document any

shortcomings• Resolve questions of coordination and responsibilities in the event

of an actual emergency• Familiarize participants with the administration of response

procedures• Demonstrate the need for communication and cooperation during

an emergency or disaster situation• Provide insights into what resources might be needed in the event

of an actual emergency• Use the results of the tabletop exercise to update and improve the

ERP

15

Selection of Tabletop Exercise Scenarios Three tabletop scenarios selected by key staff Participants given additional information within specific scenarios

prior to exercise Scenarios developed based on findings from Vulnerability and Risk

Assessment Three scenario levels which are designed to simulate an emergency

event with Scenario #1 being the least severe and Scenario #3 being the most severe

16

Example Scenarios Severe Weather Utility Outages Fires Terrorism (Foreign & Domestic) Geographic Vulnerabilities Human Error Technological Failure Widespread Virus

17

Evaluate and Critique Implement ERP into company operations Develop formal process to gather feedback in critical areas

following emergency activations:– Timeliness of activation– Damage assessment– Restoration– Communications equipment– Facilities– Media coverage– Coordination of materials and equipment– Effectiveness in handling contacts with government agencies

18

Benefits of Tabletop Exercise

Build and broaden awarenessEstablish or evaluate against a baselineIdentify vulnerabilities and develop responsesDevelop and build internal skills and expertisePromote action

19

Lessons Learned

Vulnerability & Risk Assessment Development of the ERP Exercising the ERP Evaluation and Critique

20

Questions