38
Oakville Hydro January 14, 2016 BUSINESS CONTINUITY

Oakville Hydro Business Continuity Edist 2016 · Disaster Recovery Plan (DRP) vs Business Continuity Plan (BCP) Information Technology: Ice Storm Response

Embed Size (px)

Citation preview

Page 1: Oakville Hydro Business Continuity Edist 2016 · Disaster Recovery Plan (DRP) vs Business Continuity Plan (BCP) Information Technology: Ice Storm Response

Oakville Hydro January 14, 2016

BUSINESS CONTINUITY

Page 2: Oakville Hydro Business Continuity Edist 2016 · Disaster Recovery Plan (DRP) vs Business Continuity Plan (BCP) Information Technology: Ice Storm Response

Oakville Enterprises Corporation

Page 3: Oakville Hydro Business Continuity Edist 2016 · Disaster Recovery Plan (DRP) vs Business Continuity Plan (BCP) Information Technology: Ice Storm Response

Oakville Hydro Electricity Distribution

13th largest electricity distributor in Ontario

OUR ASSETS Total asset base of $230 million

Distribution system consisting of

o Underground cable: 1,348 km

o Overhead circuit wires: 486 km

OUR CUSTOMERS 67,821

OUR SERVICE AREA 143 square km

Page 4: Oakville Hydro Business Continuity Edist 2016 · Disaster Recovery Plan (DRP) vs Business Continuity Plan (BCP) Information Technology: Ice Storm Response

Business Continuity vs Emergency Operation Plan 2013 Ice Storm Oakville Hydro’s Initiatives “Mock Disaster” of the Emergency Operations Plan “Casper Test” of the Business Continuity Plan Test

Lessons Learned Conclusions

Outline

Page 5: Oakville Hydro Business Continuity Edist 2016 · Disaster Recovery Plan (DRP) vs Business Continuity Plan (BCP) Information Technology: Ice Storm Response

Business Continuity Plans at OEC

Emergency Operation

Plan

IT DRP

Pandemic Plan

Emergency Communication

Plan Customer Service

Plan Affiliate

Emergency Plans

Page 6: Oakville Hydro Business Continuity Edist 2016 · Disaster Recovery Plan (DRP) vs Business Continuity Plan (BCP) Information Technology: Ice Storm Response

Background

Page 7: Oakville Hydro Business Continuity Edist 2016 · Disaster Recovery Plan (DRP) vs Business Continuity Plan (BCP) Information Technology: Ice Storm Response

Timeline of Events

2015

EOP Review

Q4

Impact Analysis & Mitigation

Casper Test

2014 2013 Q1 Q3 Q1 Q2 Q3 Q4 Q1 Q2

Ongoing Business Continuity Program

2016

Ice Storm

Q2 Q4

Mock Disaster

Page 8: Oakville Hydro Business Continuity Edist 2016 · Disaster Recovery Plan (DRP) vs Business Continuity Plan (BCP) Information Technology: Ice Storm Response

Potential Business Interruptions

Enterprise Critical Functions

Vulnerability? Interruption Tolerance?

Impacts

Mitigate Probability Mitigate Impact

Provide for Interruption

Emergency Incident

Response & Recovery Emergency Command

Communications

Continuous Enterprise

Risk Management

Risks

Tracking

BCP Framework Implementation

Page 9: Oakville Hydro Business Continuity Edist 2016 · Disaster Recovery Plan (DRP) vs Business Continuity Plan (BCP) Information Technology: Ice Storm Response

First Phase: The Mock Disaster

Scope: major ice storm increased customer calls loss of power from the Grid loss of feeder to office and control room

Mitigations to be tested – prepared simulations Departments’ preparations including I.T. support Test performed June 2015

Page 10: Oakville Hydro Business Continuity Edist 2016 · Disaster Recovery Plan (DRP) vs Business Continuity Plan (BCP) Information Technology: Ice Storm Response

Second Phase: The Casper Event

Why a Second test?

Major Interruption: Building unavailable

What did we want to achieve out of the test? Test Performed October 2015

Page 11: Oakville Hydro Business Continuity Edist 2016 · Disaster Recovery Plan (DRP) vs Business Continuity Plan (BCP) Information Technology: Ice Storm Response

Casper Event

Page 12: Oakville Hydro Business Continuity Edist 2016 · Disaster Recovery Plan (DRP) vs Business Continuity Plan (BCP) Information Technology: Ice Storm Response

Distribution Operations

Consists of: Line Operations

Maintenance

Construction

Warehouse

Page 13: Oakville Hydro Business Continuity Edist 2016 · Disaster Recovery Plan (DRP) vs Business Continuity Plan (BCP) Information Technology: Ice Storm Response

Distribution Operations

Page 14: Oakville Hydro Business Continuity Edist 2016 · Disaster Recovery Plan (DRP) vs Business Continuity Plan (BCP) Information Technology: Ice Storm Response

Operations: Ice Storm Response

Our Experience of the Ice Storm Customers Feedback was lost High volume of customer calls Met customer needs but took a large concerted effort

Used scouts to locate the worst hit areas

Constantly prioritized work – in a reactive nature

Paper based planning and reporting

Page 15: Oakville Hydro Business Continuity Edist 2016 · Disaster Recovery Plan (DRP) vs Business Continuity Plan (BCP) Information Technology: Ice Storm Response

Operations: Mock Disaster

The Strategy Leveraged the work order system

Moved away from paper

Trained staff in new process

Provide more accurate Estimated Time of Restoration

(ETR)

Page 16: Oakville Hydro Business Continuity Edist 2016 · Disaster Recovery Plan (DRP) vs Business Continuity Plan (BCP) Information Technology: Ice Storm Response
Page 17: Oakville Hydro Business Continuity Edist 2016 · Disaster Recovery Plan (DRP) vs Business Continuity Plan (BCP) Information Technology: Ice Storm Response

Operations: Casper Event

An event with significant preparation Challenges Find locations for: vehicles equipment material

Relocation of critical staff to “Hotel”

Page 18: Oakville Hydro Business Continuity Edist 2016 · Disaster Recovery Plan (DRP) vs Business Continuity Plan (BCP) Information Technology: Ice Storm Response
Page 19: Oakville Hydro Business Continuity Edist 2016 · Disaster Recovery Plan (DRP) vs Business Continuity Plan (BCP) Information Technology: Ice Storm Response

Operations: Casper Event

Lessons Learned Long preparation time Resource availability

Next Steps How can we shorten the lag Continually assess your tolerances for business

interruptions

Page 20: Oakville Hydro Business Continuity Edist 2016 · Disaster Recovery Plan (DRP) vs Business Continuity Plan (BCP) Information Technology: Ice Storm Response

Engineering & Construction

Consists of: Planning, Design & Standards

Construction Management

GIS & Records Management

New Customer Connections

Page 21: Oakville Hydro Business Continuity Edist 2016 · Disaster Recovery Plan (DRP) vs Business Continuity Plan (BCP) Information Technology: Ice Storm Response

Engineering: Ice Storm Response

All Staff Re-deployed to Emergency Operations

Prioritized Customer Calls

Managed Scout Deployment

Prepared Paper “Work Packages” for Operations team

Entered Trouble Calls into GIS

Page 22: Oakville Hydro Business Continuity Edist 2016 · Disaster Recovery Plan (DRP) vs Business Continuity Plan (BCP) Information Technology: Ice Storm Response

Engineering: Mock Disaster

Customer Calls Logged in Service Manager Dispatched Electronically to Scouts Service Calls Logged Manually into GIS Scouts: Assigned To Geographic Zones Redeploy as Required Updated Calls on Laptop

Page 23: Oakville Hydro Business Continuity Edist 2016 · Disaster Recovery Plan (DRP) vs Business Continuity Plan (BCP) Information Technology: Ice Storm Response

Service Manager Dispatch Board

Page 24: Oakville Hydro Business Continuity Edist 2016 · Disaster Recovery Plan (DRP) vs Business Continuity Plan (BCP) Information Technology: Ice Storm Response

GIS Overall Showing Trouble Calls

Page 25: Oakville Hydro Business Continuity Edist 2016 · Disaster Recovery Plan (DRP) vs Business Continuity Plan (BCP) Information Technology: Ice Storm Response

Home Office set up for Technical Staff

Mobile Office set up for Field Staff

Hoteling organized for critical Administrative Staff What About All of that Paper in the Office?

GIS/CAD – “The Need For Speed”

Engineering: Casper Event

Page 26: Oakville Hydro Business Continuity Edist 2016 · Disaster Recovery Plan (DRP) vs Business Continuity Plan (BCP) Information Technology: Ice Storm Response

Engineering: Future Considerations

Automate Interface Between Service Manager & GIS

Accelerate current project – Paperless Filing System Enhance Mobile & Off Site Access Capabilities Cloud Solutions For Access to Information Secure Offsite Facilities for “Heavy Users” - GIS

Page 27: Oakville Hydro Business Continuity Edist 2016 · Disaster Recovery Plan (DRP) vs Business Continuity Plan (BCP) Information Technology: Ice Storm Response

Customer Service

Customer Service: Providing our customers with a positive experience

when they interact with Oakville Hydro

Billing

Electricity and Water

Bi-monthly

Page 28: Oakville Hydro Business Continuity Edist 2016 · Disaster Recovery Plan (DRP) vs Business Continuity Plan (BCP) Information Technology: Ice Storm Response

Customer Service: Ice Storm Response

Issues Identified Limited phone capacity – simultaneous calls Limitations to Call Centre capabilities – few

options Limitations with after hours service Logging calls into excel – need for a common

database

Page 29: Oakville Hydro Business Continuity Edist 2016 · Disaster Recovery Plan (DRP) vs Business Continuity Plan (BCP) Information Technology: Ice Storm Response

Customer Service: Casper Event

Onsite – Staff available to serve customers

Offsite – Vendor Facility

Field Services

Home setup with Jabber software for phones

Page 30: Oakville Hydro Business Continuity Edist 2016 · Disaster Recovery Plan (DRP) vs Business Continuity Plan (BCP) Information Technology: Ice Storm Response

Customer Service: Enhancements and Future Considerations

1. Call Centre - Simultaneous calls and flow

2. After Hours Service – Power Assist

3. Data base – Outage Management System, Great Plains

4. Social Media – Twitter and Facebook

5. Interactive Voice Response (IVR)

Page 31: Oakville Hydro Business Continuity Edist 2016 · Disaster Recovery Plan (DRP) vs Business Continuity Plan (BCP) Information Technology: Ice Storm Response

Information Technology

IT Staff servicing Oakville Enterprises Corporation and responsible for: Infrastructure Core Business Applications Client services

Operation technology (OT)

Disaster Recovery Plan (DRP) vs Business Continuity Plan (BCP)

Page 32: Oakville Hydro Business Continuity Edist 2016 · Disaster Recovery Plan (DRP) vs Business Continuity Plan (BCP) Information Technology: Ice Storm Response

Information Technology: Ice Storm Response

Building availability, systems intact

Providing access to data externally, VPN, Citrix was available

Challenges Web site, phone lines capacity Communications to the customers Field Mobility

Page 33: Oakville Hydro Business Continuity Edist 2016 · Disaster Recovery Plan (DRP) vs Business Continuity Plan (BCP) Information Technology: Ice Storm Response

Information Technology: Mock Disaster

Mock Disaster Emergency Operating Plan (EOP) IT department fully engaged Innovation for new ways to provide for the business

needs Short term planning for next test

Facilities – Generator power requirements for IT

Infrastructure

Page 34: Oakville Hydro Business Continuity Edist 2016 · Disaster Recovery Plan (DRP) vs Business Continuity Plan (BCP) Information Technology: Ice Storm Response

Information Technology: Casper Event New services Jabber – VPN access to the CS queues Support of OMS initiatives New Citrix environment New business applications for field work order management Upgrade of mobile devices Casper IT enhances its understanding of the expectations of the

business Business’s understanding of IT team support

Page 35: Oakville Hydro Business Continuity Edist 2016 · Disaster Recovery Plan (DRP) vs Business Continuity Plan (BCP) Information Technology: Ice Storm Response

Information Technology: Future Considerations

Ongoing technical and system improvements

Building on Collaboration

Long term capital planning

Page 36: Oakville Hydro Business Continuity Edist 2016 · Disaster Recovery Plan (DRP) vs Business Continuity Plan (BCP) Information Technology: Ice Storm Response

Lessons Learned

Modular Testing Formal Preparation Clear Objectives & Goals Proper Planning Debrief and learn

Page 37: Oakville Hydro Business Continuity Edist 2016 · Disaster Recovery Plan (DRP) vs Business Continuity Plan (BCP) Information Technology: Ice Storm Response

Conclusions

Support from the leaders of the organization Significant commitment and effort required Goals and targets Reaching elevated levels of engagement and

collaboration Conducting “Test without warning” Journey of continual improvement

Page 38: Oakville Hydro Business Continuity Edist 2016 · Disaster Recovery Plan (DRP) vs Business Continuity Plan (BCP) Information Technology: Ice Storm Response