Business Continuity Planning for the Business Executive... · Business Continuity Planning for the Business Executive Ben F. Thornton DR/BCP Practice Director Optimus Solutions

Business Continuity Planning for the

Business Executive

Ben F. ThorntonDR/BCP Practice Director

Optimus SolutionsApril 23, 2007

Why Business Continuity Planning?Why Business Continuity Planning?

“Of all human lamentations, without doubt the most common is, If I had only known.But we can’t know, and so days of death and fire so often begin no differently than those of love and warmth.”

Tom ClancyDebt of Honor

Lessons Learned - What Worked?• Contingency Planning Works

– Incident Management was vital and adaptable– Information protection and alternate site strategies were essential – Things that were planned for and drilled - worked

• Lessons Of the Past Still Apply– Transportation and communication challenges – Soft people issues – Logistics planning– Voice recovery importance

Business Continuity for the Business ExecutiveUnderstanding and Assessing Your Business Risks

• What Risks Do We Face?• Are We Compliant?• Where Are The Holes?

Identifying Needs and Preparing Your BC Plan• Business Impact Analysis• Strategy and Solutions Design• Plan Development and Implementation• Crisis Management Planning

Performing Ongoing Management and Testing• Putting Your Plan to the Test• Using Metrics to Determine Success• Keeping Your Plans Current and Viable

So, Where Do We Start?

Key Points:� “Business Continuity is not a destination; it is a journey.”� For any productive journey, we need a good road map.

� Help us find the way� Keep us on course� Proceed quickly and efficiently� Easy to follow

� So, let’s build one.

Our Business Continuity Roadmap

Essential Elements:� Where are we now?

� Where do we want to go?

� What is the best route?

� What will we need along the way?

� What will the trip cost?

� When will we arrive?


Step #1 - Discovery

� Where are we now?

� What condition are we in?

A check-up is in order


Step #1 - Discovery

� What Risks Do Our Business Face?

• External factors

• Internal Factors

• Variable Factors

Our Business Continuity RoadmapStep #1 – Discovery - Is our Business Compliant?

• Federal White Paper for Financial Institutions• FFIEC updates to handbooks on Business Continuity,

Information Security, and Supervision of 3rd Party Outsourcers

• Gramm-Leach-Bliley for security of personal and private information

• HIPAA Security Standards• Sarbanes-Oxley Act, Section 404 on Internal Controls• Basel II• NASD Rule 3510/3520 and NYSE Rule 446

Our Business Continuity RoadmapStep #1 - Discovery

� Where are the Holes (Gaps)?• Belief or Desire - versus Reality

– Level of Preparation– Real Capabilities / Solutions

• External Issues– Pending Acquisition– Value System

• Changes Needed– Organizational– Technological

Our Business Continuity Roadmap� Step #1 – Discovery

� Step #2 – Planning the Route

• Business Impact Analysis

• Strategy and Solutions Design

• Plan Development and Implementation

• Crisis Management Planning

Our Definition Of A Disaster

“Any Condition That Prevents Me From Performing My Critical Business Functions In An Acceptable Period Of Time”

� Computer hardware failure� Data center flooded after a pipe bursts � Phone lines severed / damaged due to

construction � Bomb Threat causes corporate evacuation� Building sinks due to water erosion� Transformer fire causes power outage� Security Breach� Hazmat Situations

Business Impact Analysis• Prioritize business functions, processes &

applications• Quantify business impacts of disruption, lack of

compliance• Assess impact of computer systems and networks

failure on business continuity• Assess current preparedness and evaluate

options for reducing risks and exposures• Identify vendor dependencies and exposures• Analyze gaps and develop recommendations • Develop Roadmap to Pursue Solutions

Business Impact Analysis• Prioritize business functions, processes & applications• Quantify business impacts of disruption, lack of

compliance• Assess computer systems and networks impact on

business availability• Assess current preparedness and evaluate

options for reducing risks and exposures• Identify vendor dependencies and exposures• Analyze gaps and develop recommendations • Develop Solution Roadmap

Murphy was right:

“Left to themselves, things generally go from bad to worse!”

Road Map Check Point

We now understand what the problem is.

How do we solve the problem?

What are our options?

What is the right solution?

Parallel Operations or Load Balancing

Mirroring

Journaling

Vaulting

Co-location

Traditional HotSite

Quick ShipMobile

React

A Short

Reliability

Redundancy

Recoverability

DETOUR

Distance

What’s Recoverability?

Strategy Development – solution criteria

• How well will it satisfy the company business continuity requirements?

• How cost-effective is it, given the risks identified in the BIA?

• Is it technically feasible?• Is it operationally sound and reliable?• Can it be implemented with current staff?• Can it be managed with current staff?

Strategy Analysis & Solution DesignPart 1: Strategy Analysis• Analyze 2-3 Strategy Options, High-Level

Costing:– Develop evaluation criteria – Determine availability alternatives – Develop pros and cons and cost/benefit analysis– Recommend availability strategies– Obtain Consensus– Obtain management sign-off & funding

Strategy Analysis & Solution DesignPart 2: Solution Design

• Blueprint for Selected Strategy:– Technology design (sufficient to request vendor

bids)– Bill of materials required– Cost estimates– Performance metrics

• Recovery Solution Implementation:– Implementation timelines– Critical resource requirements– Funding requirements– Acceptance criteria

Strategy Analysis & Solution Design

It’s all about “balance”:

• Acceptable Level Business Protection• Acceptable Level of Business Risk• Acceptable Level of Associated Costs

DisasterRecoveryCosts AreExcessive

Impacts AreExcessive

Acceptable RangeSame Day 1-2 Days 3-7 Days Beyond

Time

What is in a Business Continuity Plan?• Who – team structure, contacts & responsibilities• What – action oriented tasks • When – duration and sequence• Where – primary site, alternate• How – procedures & logistics

Sample Development Phase SummarySchedule Month

Plan Development Activities 01 02 03 04 05 06

Development Project Planning- Ongoing Project Management- Develop Project Schedule- Conduct Orientation and Training Sessions

Project ExecutionData Gathering & Validation Activities- Conduct Requirements and Planning Sessions- Conduct Data Gathering- Review & Validate InformationPlan Development Activities- Conduct Workshops or Round 1 Interviews- Develop Draft Plans- Conduct Plan Walk-Through- Conduct Follow-up Interviews- Update Draft Plans- Conduct Operational Analysis (e.g., Info. Protection)

Project Close- Delivery of Final Draft- Management Review and Acceptance Activities

Plan Maintenance Phase

• Organization of Information– Addresses Familiarity of User– Logical Arrangement / Ease of Use– Appropriate Levels of Detail

• Time Phasing / Meets RTO• Provides Organization and Controls

Conventional 24 hour recovery

ITSystems

Data backup by tape stored

offsiteAction Set 11. …2. …3. …

Mobile recovery units

Resource Set 11. …2. …

BusinessFunctions

DependentApplications

Strategies

BusinessImpacts

Conventional 24 hour recovery

ITRequirements

Data backup by tape stored

offsiteAction Set 11. …2. …3. …

Mobile recovery units

Resource Set 11. …2. …

BusinessFunctions

ContactsProcedures

Strategy

Business Continuity Plan

Disaster Recovery Plans

(by Platform or Technology Area)

Business Continuity Plans (by Business Unit

or Location)

• Incident Manager

• Support FunctionsAuditCommunications FacilitiesFinanceInsuranceLegalHuman ResourcesInformation SystemsPublic / Government Relations

Incident Management Team Plan

Business Unit Examples:• Customer Services• Distribution• Financial Management• Field Operations • Human Resources• Manufacturing• Marketing &

Communications• Purchasing• Product Development• Retail

Tech Area Examples:• Mainframe• Mid-Range• Network• Desktop• Voice

Typical Enterprise Plan Structure






or Location)


or Location)


or Location)

Executive ManagementExecutive

Management

IncidentCommanderIncident

Commander

Response andRecovery

Coordinator(s)

Response andRecovery

Coordinator(s)Support TeamsSupport Teams

Public SpokespersonIMT Liaison

Public SpokespersonIMT Liaison

• Human Resources• Facilities• Finance• Communications• IT

• Operations• Customer Services • F & A – GL, AR, AP• Distribution• IT

Department Heads

Department Heads

Typical Incident Management Team

Department Heads

Department HeadsDepartment

HeadsDepartment

HeadsDepartment Heads

Department Heads

Support TeamsSupport TeamsSupport TeamsSupport Teams

Support TeamsSupport Teams

Incident Management – Key Concepts

• Dynamic role assignment• Flexible to Address Interdependencies• Integration to Evacuation, Notification, etc• Supporting (but not requiring) the ICS model• Capable of Supporting Multiple Events• Dynamic Communications Capability


• Step #1 – Discovery

• Step #2 – Planning the Route

• Step # 3 – Making The Trip– Testing Our Plans– Managing Our BC Program

Testing Our Plans – Key Concepts

• Engaging and Enjoyable Experience• Creative and Resourceful• Progressive in Complexity• Specific Objectives• Realism Encouraged – design & participation• Details, Details, Details

Testing Our Plans

• Testing Metrics• Estimated / Actual

Comparisons• Test Status and

Tracking• Historical Tracking

– When– What– Who– Results

• Graphical Comparisons

1.00 - .93 Exceeds Objectives

.92 - .80Meets Objectives

.79 - .50 NeedsImprovement

.49 - .00Needs Significant Improvement

0.75

0.80

0.85

0.90

0.95

1.00

M1 G2/G4 B2 D2 BSMS (UTS) BSMZ (UTS)

Operating System

Sub-System Control Region

Database Load

Technology Platform <8 Hours 8 – 12 hrs 12 – 24 hrs 2 – 7 days

Option A

Option B

Option C

Option D

Option E


• How does it all fit together?

• Remember – “a journey not a destination”

• Business Continuity Program

��

� ��

��

��

� �� ! �

��

" #$

� � �� % �& ��

' � ��' ��

" ��( � ��) ��* �&" � ��

��

� ��

��

��

� �� ! �

��

" #$

� � �� % �& ��

' � ��' ��

" ��( � ��) ��* �&" � ��

What is a Business Continuity Program?• Business Impact Analysis / Risk & Vulnerabilities Assessment

Business Areas

Analyzed

BusinessProcessesIdentified

& Prioritized

CriticalBusiness

Functions Identified

Application & System

DependenciesIdentified

Business Imperatives

& DriversBusiness plans

GLB, SOX, External auditors

Business risks /competition

Other regulations – Example: State regulations concerning payroll processing

Human resources

Risks & ThreatsIdentified

Impact DimensionsIdentified

ImpactsOver TimeDetermined

Revenue / Cash Flow Impacts

Fines & PenaltiesRecurring Fixed Costs – taxes, salaries, insurance, rent

Recovery & Mitigation / Management Costs

Customer Dissatisfaction / Lost Opportunity Costs

Inefficiency / Backlog Costs

Total Financial Exposure

$5,800,000

$10,675,000

$2,300,000

$12,250,000

$

$2,000,000

$4,000,000

$6,000,000

$8,000,000

$10,000,000

$12,000,000

$14,000,000

0-48 Hours 3 to 7 Days 8-15 Days 16-30 Days

$3,350,000$2,725,000$1,825,000$550,000Additional Costs (penalties, temps, overtime)

$4,600,000$4,600,000$2,850,000$1,200,000Current Revenue

$4,300,000$3,350,000$1,125,000$550,000Future Sales

16-30 Days8-15 Days3 to 7 Days0-48 HoursFinancial Impact

AnyCompany Inc. Financial Impacts Roll-up (Thousands)

$3,350,000$2,725,000$1,825,000$550,000Additional Costs (penalties, temps, overtime)

$4,600,000$4,600,000$2,850,000$1,200,000Current Revenue

$4,300,000$3,350,000$1,125,000$550,000Future Sales

16-30 Days8-15 Days3 to 7 Days0-48 HoursFinancial Impact

AnyCompany Inc. Financial Impacts Roll-up (Thousands)

What is a Business Continuity Program?

• Strategy

��

�Production site is protected from disaster

�Secondary site has standby failover server

�Secondary site offloads information processing

PrimaryCLARiiON

RemoteCLARiiON

Production Host(Site A)

Secondary Host (Site B)

Snapshot

SNAP

ProductionInformation

Snapshot

SNAP

Target

MirrorView Mirror

Report Generation

Decision Support Tools

Tape Backup

PrimaryCLARiiON

RemoteCLARiiON

Production Host(Site A)

Secondary Host (Site B)

Snapshot

SNAP

ProductionInformation

Snapshot

SNAP

Target

MirrorView MirrorMirrorView Mirror

Report Generation


Tape Backup

Report Generation


Tape Backup

��

SunGard Recovery Facility

ESCONCHANNEL

CONNECTIVITY

STK SILO

DS3 Fiber Optic Circuit

Alternate Hotsite

ChannelExtender

Production Processor

ChannelExtender

ChannelExtender

HotsiteSubscription

______

CRC SGN

SunGard Recovery Facility

ESCONCHANNEL

CONNECTIVITY

STK SILO

DS3 Fiber Optic Circuit

Alternate Hotsite

ChannelExtenderChannelExtender

Production Processor



HotsiteSubscription

______

CRC SGNCRC SGN

NO T erm in ate

Activ ation Dec isio n NO Term ina te

Ale rt /Notification

Initial Respons e Perso nnel Assembly

Initial Assessment

M obilize Reco very

Team

R ecovery

T eam Ass embly

YE S

M obilize Rec ov ery Teams

T rave l to Alte rnate Site

Restore Op erating Env ironmen t(s)/

Network

Restore C ritical Applic atio n

Data

Validate Resto red Sy stems/Apps

T urn Ov er Produc tio n Environ ment

EVENT Hour 1

Hour 2

Hour 3

Hour 3.5

Hour 5.5

Hour 9.5

Hour 15.5

Hour 19.5

Hour 20

Hour 21

Hour 4

Tim e Line

P lan Activation

Recovery Operations

Y ES

What is a Business Continuity Program?• Plans

Intro

Activation

Initial Logistics

(Travel, etc.)

Process-levelPlan for IT

Recovery

System Procedure-

levelRecovery

System Procedure-

levelRecovery

System Procedure-

levelRecovery

Process-level Plan for

Site RecoveryPlatform / OS Recovery

Standalone Application Recovery/Validation - Utility Systems

Environment Recovery High-Level Strategy / Process

Application Integration/Validation

Local & Wide Area Network Recovery

Standalone Application Recovery/Validation – Business Systems

Platform / OS Recovery

Standalone Application Recovery/Validation - Utility Systems

Environment Recovery High-Level Strategy / Process

Application Integration/Validation

Local & Wide Area Network Recovery

Standalone Application Recovery/Validation – Business Systems

EmergencyManagementTeam Plan

Sampleplatform / OS

scripts custom-ized by client

RecoveryProcedures

Application specific

recovery proceduresEnvironment

recoveryprocedures

Applicationvalidationprocedures

NO Terminate

Team Mobilization and

Backup Tape Shipment

EVENT 0 - 1 Hour

1 - 2 Hours

2 - 4 Hours

8 Hours

8 Hours

8 - 15 Hours

15 - 43 Hours

15 - 43 Hours

Declaration Decision

0 Hour

43 - 48 Hours

8 - 15 Hours

4 - 9 Hours

8 - 19 Hours

48 Hours

Initial IT Response Team Activation

Alert/Notification

Meet at Crisis Management Center

• Damage Assessment • Management Notification

• Recovery Team Notification • Vendor Alert

YES

Travel to SunGard

Restore: - LAN Servers - Desktop

Restore: - Data Network - Voice

Restore: - Mid-Range AS/400 (Public Safety & Non-

Public Safety

Backup Tape Delivery

Database Verification and Application Validation

Recovery/Turnover of Production Systems

SunGard System Initialization

8 - 11 Hours

Corporate Crisis ExecutiveLeadership Team Leader

Corporate CrisisExecutive Team

Member


Member


Member

Corporate CrisisResponse

ManagementTeam Leader

...


ManagementTeam Member

Crisis ResponseTeam Leader




...

Crisis ResponseTeam Member












Crisis Situation AssessmentTeam Members

6 Recovery Procedures 6.1 Restore Computer Services These Recovery Procedures are based upon a worst case disaster such as total destruction or loss of access to the home site, requiring relocation and recovery at an alternate site. This team acts as a central clearing house for coordinating available resource allocation among the teams members. Requests for resources and logistical support are funneled through this team to the appropriate support personnel. Report to the alternate site and coordinate implementation of the team plan strategies. Perform the following activities to reestablish computer processing: 1. Implement intercept messages and redirect calls normally routed to the Help Desk. 2. Utilizing the VENDORS REPORT, notify vendors that their application/software will be run on a different computer system. Refer to the

CPU numbers provided after declaration by the alternate site vendor.

a. Verify the vendor products will successfully run on a different computer serial number. b. Request technical support for any changes required to process the vendor products at the recovery facility.

3. Investigate the status of all development and maintenance work-in-process. Evaluate the impact of delaying development and maintenance activities.

4. Determine the status of operations and processing at the time of the disruption.

a. What is the general status of application data? b. Are onsite backups available for recovery? c. How current are the offsite backups? d. What is the anticipated time without operating and processing services?

NO Terminate

Team Mobilization and

Backup Tape Shipment

EVENT 0 - 1 Hour

1 - 2 Hours

2 - 4 Hours

8 Hours

8 Hours

8 - 15 Hours

15 - 43 Hours

15 - 43 Hours

Declaration Decision

0 Hour

43 - 48 Hours

8 - 15 Hours

4 - 9 Hours

8 - 19 Hours

48 Hours

Initial IT Response Team Activation

Alert/Notification

Meet at Crisis Management Center

• Damage Assessment • Management Notification

• Recovery Team Notification • Vendor Alert

YES

Travel to SunGard

Restore: - LAN Servers - Desktop

Restore: - Data Network - Voice

Restore: - Mid-Range AS/400 (Public Safety & Non-

Public Safety

Backup Tape Delivery

Database Verification and Application Validation

Recovery/Turnover of Production Systems

SunGard System Initialization

8 - 11 Hours

Corporate Crisis ExecutiveLeadership Team Leader


Member


Member


Member


ManagementTeam Leader

...


ManagementTeam Member





...













Crisis Situation AssessmentTeam Members

6 Recovery Procedures 6.1 Restore Computer Services These Recovery Procedures are based upon a worst case disaster such as total destruction or loss of access to the home site, requiring relocation and recovery at an alternate site. This team acts as a central clearing house for coordinating available resource allocation among the teams members. Requests for resources and logistical support are funneled through this team to the appropriate support personnel. Report to the alternate site and coordinate implementation of the team plan strategies. Perform the following activities to reestablish computer processing: 1. Implement intercept messages and redirect calls normally routed to the Help Desk. 2. Utilizing the VENDORS REPORT, notify vendors that their application/software will be run on a different computer system. Refer to the

CPU numbers provided after declaration by the alternate site vendor.

a. Verify the vendor products will successfully run on a different computer serial number. b. Request technical support for any changes required to process the vendor products at the recovery facility.

3. Investigate the status of all development and maintenance work-in-process. Evaluate the impact of delaying development and maintenance activities.

4. Determine the status of operations and processing at the time of the disruption.

a. What is the general status of application data? b. Are onsite backups available for recovery? c. How current are the offsite backups? d. What is the anticipated time without operating and processing services?

Business Continuity Program• Provides ability to manage risks & recoverability• Links business, technology, and continuity objectives• Provides periodic reporting of metrics to management

– Continuity scorecard, program progress, and resource consumption

• Provides required resources • Assures change management capabilities • Accomplishes effectiveness & cost control of program• Protects against skills obsolescence

Disaster Recovery Planning

Business Resumption Planning

Identification, Mitigation andPrevention of Risk

Planning

Quality Assurance

Assurance andManagement

1. Disaster Recovery/BusinessResumption Project Management

2. Lifecycle Program Management

Business Impact Analysis Analysis and Discovery

Business Continuance Assessments

Continuity Program OfficePolicies, Procedures, Guidance

Steering CommitteeComposed of Key Corporate Stakeholders / Customers / Resource Providers

Program Governance:Provides guidance, oversight,direction, unity of effort, and resources..

Certification Program

Testing Program

Crisis Management Program

Business Continuity Program

Enterprise Continuity Programs

Enterprise Continuity Programs

Disaster AvoidancePrograms

Disaster AvoidancePrograms

Emergency Response /Evacuation Plans

Safety / Security Programs

Insurance / Risk Management

Compliance / Legal

Records Management

Executive Management andStaff Support Departments

Management Agenda

Staff Support Procedures

Disaster Recovery PlansDisaster Recovery Plans

Essential Departments

Recovery Procedures

Business Continuity PlansBusiness Continuity Plans

Response Procedures

Typical Program StructureTypical Program Structure

Incident Management PlanIncident Management Plan

Loss Mitigation ProgramsLoss Mitigation Programs


• Step #1 – Discovery

• Step #2 – Planning the Route

• Step # 3 – Making The Trip

Business Continuity for the Business ExecutiveUnderstanding and Assessing Your Business Risks

• What Risks do we Face?• Are We Compliant?• Where Are The Holes?

Determining the Components of Your Business Continuity Plan

• Business Impact Analysis• Strategy and Solutions Design• Plan Development and Implementation• Crisis Management Planning

Performing Ongoing Management and Testing• Putting Your Plan to the Test• Using Metrics to Determine Success• Keeping Your Plans Current and Viable

• Key Learning Points - A Summary– “It’s a Journey, Not a Destination”– Planning Works – As Proven Time and Again– Solid, Executable Strategies are Essential– BIA: It is all About “Consensus Building”– Crisis Management is Vital– Interdependencies Must Be Addressed– Program Management Pulls it all Together

Business Continuity for the Business Executive

Please complete Your Evaluations!

In conclusion,

Phideaux says:

Business Continuity Planning for the

Business Executive

Ben F. ThorntonDR/BCP Practice Director

Optimus SolutionsApril 23, 2007

Documents

Business Continuity Planning for the Business Executive... · Business Continuity Planning for the Business Executive Ben F. Thornton DR/BCP Practice Director Optimus Solutions