New Clerk Academy · For minor disasters for important business functions, the recovery strategy is...

New Clerk Academy

May 8, 2014

OVERVIEW

Disaster Recovery

3

Presentation Agenda

Introduction and Definitions

DR Motivators and Drivers

Recovery Challenges

Scope of Disasters

Components of Recovery Plans

Business Continuity Planning Process

FACCSG Data Vaulting Solution

4

What is a disaster?

Definitions:

The interruption of business due to the loss or denial

of the information assets required for normal

operations.

A sudden, unexpected emergency requiring

immediate action.

An event that renders the system inoperable.

A loss or interruption to business functions.

A failure within the infrastructure which interrupts

business resulting in unplanned downtime.

5

DR Motivators and Drivers

Audit requirements

Government mandates

Insurance

requirements

Business impact

Legal liabilities

Protection of business

assets

Protection of business

relationships

Impact to customers

Recent disasters

Increased regulation

from external sources

6

Recovery Challenges

Why do planning efforts fail?

Degree of dependence not understood

Potential impact not recognized

Lack of management commitment

The “won’t happen here” syndrome

DR is never a priority

7

Recovery Challenges (cont’d)

Shrinking recovery windows

Seamless or transparent recoveries

Growth of E-commerce

Internet dependencies

“Too big to recover” syndrome

Regional and global recoveries

8

Why Move Forward?

Potential Impact

Tangible costs:

Lost revenue

Lost productivity

Legal implications

and fines

Lost wages

Intangible costs:

Political image

Reputation

Credibility

Employee morale

9

Planning Questions

What types of interruptions could we have?

Which of our business functions are critical?

How are we defining critical?

How would a disaster affect the quality of our

service?

If you had 15 minutes to vacate your office, do

you know what you would take with you?

10

Planning Questions (cont’d)

How long could your organization survive in

the event of a disaster?

Are recovery tasks adequately organized and

assigned?

What happens if there’s a disaster at 2:00

pm on a holiday weekend?

11

• terrorism

• theft

• tidal wave

• tornado

• volcano

• water damage

blizzard

bomb threat

chemical spill

civil disturbance

contractual failure

dam collapse

earthquake

epidemic

hardware failure

explosion

fire

flooding

freezing

heat

high winds

hostage taking

hurricane

network failure

nuclear war

power failure

riot

sabotage

strike

telecommunications failure

Types of Disasters

12

Region/Area

Building/Site

Data-Center

Network

System

Data

Application database

Scope – Layers of Protection

Emergency Operations Center (EOC) Crisis management

Internal hot-site

Commercial hot-site Remote fail-over Work area recovery

Redundant networks Reroutable networks Network backups

Database backups Database logging

Component redundancies Local fail-over systems

Data back-ups Off-site media storage Disc mirroring (remote) Electronic vaulting(remote)

13

Components of an IT Recovery Plan

System Recovery

Database Recovery

Application Recovery

Data Recovery

Network Recovery

Telecommunications Recovery

LAN Recovery

User Work Area Recovery

14

Components of a Business Continuity or Emergency Response Plan

Employee Safety and Awareness

Emergency Response

Crisis Management

Site or Facilities Recovery

Relocation of business units

Vendor-supplier recovery

Public Relations/Media Response

IT Recovery is a component of the BCP

15

Disaster Recovery Plan

Definitions

A predefined, tested set of procedures to

implement in the event of a disaster.

Documented procedures which outline the who,

why, what, and how to plan and recover from a

disaster.

Purpose is to increase the chances of survival

and to decrease the amount of loss.

16

Does provide:

Blueprint for survival of the business after a

disaster

Does not:

Duplicate a normal business environment

Provide business-as-usual, in survival mode

A Disaster Recovery Plan …

17

Business Continuity Planning Process

Management Sponsorship

Scope and Objectives

Risk Management and Disaster Avoidance

Recovery Requirements

Design and Development of the Plan

Rehearsal, Maintenance, and Reviews

18

Management Sponsorship

Strategies for selling senior management

Identify risks and vulnerabilities

Describe the benefits of having a plan in place

Identify impact in not having a plan

Substantiate downtime in dollars

Clarify insurance coverage boundaries

Increase awareness

19

Scope and Objectives

Defining scope What are you attempting to protect? Business continuity versus IT recovery Define scope of plan

Defining objectives

Define how much you are willing to lose Define disaster scenarios Classify disasters: minor, major, catastrophic Define acceptable, tolerable downtime Define project management plan Define project team members Define methodology to be used

20

Operating

System Database

System

Recovery Team

Site/ Building

Security

Clean-up / Restoration

Facilities

Tape

Librarian

Data Entry

Data Control

Personnel,

Insurance, Public

Relations, Legal

Petty Cash,

Legal Purchasing, Audit

Administration

Users

Application

Support

Application Development

User-Liaison

Business Continuity Recovery Coordinator

Outside Authorities Business Continuity Manager Senior Management

Data

Communications

Voice

Communications

LAN

Communication

Disaster Recovery Project Team Structure - Sample

21

BACKUP

ALTERNATE

PROCESSING

RECOVERY

PROCEDURES DISASTER

RECOVERY PLAN

REVISIONS AND UPDATES

TESTING

TRAINING

MAINTENANCE

DR Methodology - Sample

PROJECT

MANAGEMENT

CRITICAL

APPLICATIONS

MANAGEMENT PROCEDURES

INVENTORY

EVENT

DETECTION

22

Risk Management and Disaster Avoidance

Identify risks, threats, and exposures

Assess current environment

Conduct disaster avoidance review

Assess cost-benefit ratio for accepting risk

versus implementing controls

Implement controls to mitigate risk and

potential disasters

23

Recovery Requirements

Define recovery needs and requirements

Conduct Business Impact Analysis (BIA) define key functions

develop and validate questionnaire

conduct interviews

identifies tolerable, maximum amount of downtime

identifies potential workarounds

Categorize and prioritize business functions vital, critical, important, deferrable

disaster classifications: minor, major, catastrophic

Conclusions from the analysis

24

Recovery Requirements (cont’d)

Define Recovery Time Objectives (RTOs)

Define recovery strategies to support potential disasters

For example:

For minor disasters for important business functions, the recovery strategy is to do nothing for a period of 4 hours or less, not to exceed 24 hours.

For minor disasters for critical business functions, the strategy will be to alert stand-by processing and wait for 1 hour or less.

For major disasters affecting vital business functions, the strategy is to assemble the disaster management team and to invoke the recovery plan within the hour.

25

DISASTER

ELAPSED TIME

Backup Facility Activated

Full Recovery

Alternate Processing Interim

Processing Normal

Recovery Timeline

26

* If resolved, write incident report.

* If NOT resolved within allowable time,

escalate to next level.

?

OPERATOR:

- Attempt to diagnose

and resolve problem

REFER, AS NEEDED, TO:

- Shift Supervisor

- Operations Manager

15 MIN

PROBLEM

DECLARE DISASTER

INITIATE RECOVERY PLAN

DISASTER

NOTIFY:

- Dir, Computer Ctr Opns

- Mgr, Computer Ctr Opns

- Mgr, Info Sys Network

CALL IN:

- Outside resources

CONTINUE:

- Diagnosis and

problem solving

4 HRS

ALERT 1

NOTIFY:

- VP, Information Systems

- Mgr, Change Management

- User Liaison Team Leader

REPORT ON-SITE ASAP:

- Dir, Computer Ctr Opns

- Mgr, Op Supp & Control

- Disaster Recovery Manager

- Mgr, Computer Ctr Opns

- Mgr, Info Sys Network

CONTINUE:

- Resolution efforts

EVALUATE CRITICALITY

AND DECIDE ON

INITIATING RECOVERY

PLAN

MAXIMUM

TOLERABLE

DOWNTIME

(24 HRS)

ALERT 2

Problem Escalation Process

27

Define backup alternatives, options, workarounds for potential scenarios. Depending upon the type of disaster, will dictate which backup option is appropriate.

Some of include: do nothing

recover at time of disaster

recover using manual processing, service bureau

recover using stand-alone, PC-based system

recover at another company owned facility

recover at a commercial hot-site

recover at an internal hot-site

NONE OF THESE ARE MUTUALLY EXCLUSIVE combination of the above

Define Backup Alternatives or Options

28

Design and Develop Recovery Plan

Plan to include: Scope and objectives

Escalation and notification procedures

Recovery scenarios

Project team roles and responsibilities

Recovery priorities

Recovery procedures

Alternate processing

Notify key business units owners

Notify key vendor and supplier contacts

“Return to Normal” procedures

29

RECOVERY

DIRECTIVE

DAMAGE/IMPACT

ASSESSMENT

MANAGEMENT PROCEDURES

DISASTER MANAGEMENT TEAM

NOTIFICATION

& CONTROL

CENTER

ALTERNATE

PROCESSING

APPLICATION TEAMS

RECOVERY

PROCEDURES

INTERIM

PROCESSING

RECOVERY TEAMS

FULL

RECOVERY

DISASTER

EMERGENCY

PROCEDURES

Disaster Recovery Process Flow

30

Rehearsals, Maintenance and Reviews

Rehearsals Practice-practice-practice

Design test plans define evaluation criteria for success

define areas to be rehearsed (people, procedures)

Planned rehearsals versus unannounced

Use variety of techniques: partial to full-scale

evacuation

phone lines/email

table-top exercises

1-2 applications or business units

full scale rehearsals

31

Maintenance

Schedule regular updates

Incorporate feedback from the rehearsals

Incorporate into change control process

Reviews

Validate against Service Level Objectives (SLAs)

Conduct pre-audit of existing recovery plans

Conduct objective review

Rehearsals, Maintenance and Reviews (cont’d)

Civitek Data Vaulting Solution

Civitek has partnered with Hayes Computer Systems and EMC to provide a data vaulting solution to Clerks

Each participating Clerk will purchase or lease a local storage appliance manufactured by EMC

The local device will be networked with the central vault in Alpharetta, Ga

This device can be easily integrated into the existing IT environment

How does the service work?

Data is replicated from Clerks local EMC

appliance to a secure CAT 3 rated facility in

Alpharetta, Ga

Utilizes Clerk’s current WAN connected to all

67 counties throughout Florida

Data replication requiring more network

bandwidth can utilize a secure vpn

connection

34

Benefits

Improve Disaster Recovery- data off site,

replace tape based DR with replication

Reduce backup costs

Reduce backup and recovery risks

Speed up your backups

Simplify backup and recovery operations

Benefits

Improve Disaster Recovery- data off site,

replace tape based DR with replication

Reduce backup costs

Reduce backup and recovery risks

Speed up your backups

Simplify backup and recovery operations

Disaster Recovery

Replicate data offsite

Replace Tape based

DR

Reduce Backup Costs

Replace costly media

Tape drives

Backup application

licensing costs

Reduce Backup and Recovery Risks

Replace unreliable

media

Tapes get lost

Tapes get damaged

High percentage fail

RTO

RPO

Speed up backups

Efficient technology

DD Boost software

EMC Networker

EMC Avamar

NetBackup

Backup Exec

Dell NetVault

Dell vRanger

HP Data Protector

Simplify backup and recovery

Scalability

Less to manage

Less infrastructure

Options

Buy – server sized for your environment and

projected growth.

Lease- server sized for your environment ,

lease to buy, replace as growth requires

Cloud protect- server sized for your

environment, replaced with larger server as

needed, charged for amount of data stored.

Contacts

David Porter -Marketing

porter@flclerks.com

Earl Donaldson – Network Engineer

edonaldson@flclerks.com

Tab Bradford- TAP Administrator

bradford@flclerks.com