Disaster Recovery(Business Continuity Planning)

Tim Babco

AGENDA

• What is BCP?• Key BCP Components• How to Implement?• How to Maintain?• Effort Required• Value Obtained• Q&A

• World’s largest distributor of swimming pool supplies, equipment and related leisure products

• ~$2 billion in revenues

• >3,600 employees; 285 locations; 8 countries

• >100,000 products

• >70,000 customers

• Headquartered in Covington, Louisiana

Headquarters in “Hurricane Alley”

Who is Poolcorp?

• Task of identifying, developing, acquiring, documenting, and testing procedures and resources that will ensure continuity of a firm's key operations in the event of an accident, disaster, emergency, and/or threat. It involves:

(1)Risk mitigation planning (reducing possibility of the occurrence of adverse events)

(2)Business recovery planning (ensuring continued operation in the aftermath of a disaster)

Keep the business running successfully

What is BCP?

• Business Continuity Planning (BCP) and

Disaster Recovery (DR) are often used

synonymously

• Continuum: Enterprise Individual• Corporate functions• Remote locations• Succession Planning

• Prevent and Recover

Keep the business running successfully

What is BCP?

• Needs assessment• Employees• Communications • IT Infrastructure• Recovery site logistics• Third party information• Supplies• Pre-event protection steps• Trigger points• Municipal, state, federal interaction and updates

Many important facets

Key BCP Components

• RTO – Recovery Time Objective• Amount of down time for each critical function before

outage threatens company survival • RPO – Recovery Point Objective

• How old can the data be before it is so out of date that

recreation is not practical or possible

• Consider Time of Year, Month, etc.

• Economic Benefit• Cost of protection vs. cost of down time

• Hard costs and opportunity costs

Clearly define what’s important

Needs Assessment

• Current and complete contact information

Know how to reach employees

Employees

• Current and complete contact information

Employees know key BCP contacts

Employees

• Personal BCP plan• Home• Belongings• Immediate family members• Extended family members• Pets, livestock

Employees can’t be productiveif worried about personal items

Employees

• Roles• Executive team

• Make decisions• Delegations of authority

• Communications team• Internal• External

Execute quickly and correctly

Employees

• Roles• Core team

• Coordinate detailed plan execution• Tiered response teams

• Tier 1 – IT only; sent when disaster impact predicted• Tier 2 – Employees with critical functions; sent when disaster impact is imminent

• Tier 3 – Important functions; work better as group; sent after significant impact realized

• Tier 0 – Can work remotely as situation unfolds

Avoid the scattered workforce

Employees

• Voice• Potential issues

•Land lines may be out•Cell phones may be out

• Solutions•Satellite phones•Private 2-way radios• IP telephony virtual phone system

• Call centers• Key support teams• High risk locations

Communications

Ability to verbally communicate

• Voice• Dedicated toll-free BCP lines

• Employee information line• Command conference line• Regularly scheduled, daily conferences

Communications

Ability to verbally communicate

• Data

• Choose the right circuit provider

• Have redundant data circuits

• Different providers; different routes

• Broadband wireless capabilities

Ability to access business systems

Communications

• Data

• Portable satellite systems

Ability to access business systems

Communications

• Messaging

• Text messaging

• E-mail

• Web access from anywhere

• BCP web site – externally hosted

Remotely connect & send/receive updates

Communications

• Backup power – Battery, Generator and fuel

• Offsite tape rotations (e.g. Iron Mountain)

• Low risk data center location

• Redundant data centers

• Co-location

• Cold failover facility (e.g. Sungard)

• Full mesh network

Just like insurance policies

IT Infrastructure

Internet

Global CrossingEurope

Internet

Global CrossingUS

AT&T Network

MPLS MLPPP3 meg2 T-1s

MPLS OC3

155 meg

MPLST-1

1.5 megsPOOLCORP

GLOBAL WAN

100 megsVericenter

feed

Covington Anahiem Phoenix

VericenterDallasData

Center

US BranchesSCPSPP

HorizonHorizon South

Portugal BranchesSCP Europe

MPLSDS3

45 meg

MPLS MLPPP3 meg2 T-1s

EuropeanData

CenterMPLS

E325 meg

MPLSE-1

1.5 megs

Canadian BranchesSCP

European Remote UsersAnd Customers

US Remote UsersAnd Customers

VPN DSL1 - 2 megs

Spain BranchesSCP Europe

MPLSE-1

1.5 megs

France BranchesSCP Europe

MPLSE-1

1.5 megs

Italy BranchSCP Europe

MPLSE-1

1.5 megs

UK BranchesSCP Europe

MPLSE-1

1.5 megs

MPLSDS3

25 meg

12 megsGlobal

Crossing feed

PoolCorp GlobalWide Area Network

High speed, secure access from anywhere

IT Infrastructure

• Choose good partners• Corporate grade equipment and solutions

• High reliability

• Fast response time

• Available technical support staff

• Cost competitive

• Willingness to go “above and beyond”

Only as good as the weakest link

IT Infrastructure

• Central command/recovery center• Wireless

• Size

• Proximity

From chaos to recovery in hours

Recovery Site Logistics

• Laptops for key employees• Remote connectivity (VPN)• Transportation• Housing – hotels, apartments• Childcare• Schools• Kennels (house hold pets only)• Expense reporting• Cash advances

Recovery Site Logistics

Employees can quickly be productive

• Vendors• Consultants• Financial institutions• Investors• Governmental agencies• Media• Board members

Third Party Information

Fast access to key parties

• First aid• Portable generators• Extension cords• Flashlights and batteries• Tarps• Tools (e.g. chain saws)• Ice coolers• Bottled water and non-perishable food• Energy drinks

Supplies

Ability to ride out DR events

• Full equipment inventory• Protect equipment and information

• Unplug electronics• Move electronics off of floor• Safeguard important paperwork• Close blinds and doors

• Take critical items if planned evacuation• Focus on safety if unplanned event

Pre-event Protection Steps

Attempt to minimize loss

• Define for all predictable events• Example: 9-step hurricane process

1. Storm enters gulf2. Projections converge with New Orleans in cone3. Within 4 days of landfall; still in cone4. Within 3 days of landfall; still in cone5. Within 50 hrs of landfall; still in cone; material impact imminent6. Within 40 hrs of landfall; still in cone; material impact imminent7. Within 30 hrs of landfall; still in cone; material impact imminent8. Next 36 hrs during/after storm9. Authorities give “all clear” to return home

Trigger Points

Know what to do and when to do it

• Participate in municipal DR planning/testing• Get to know local and state officials• Know evacuation routes• Placards to re-enter impact areas

Municipal, State, Federal Interaction

Take advantage of available help

• Start with basics• Focus on critical systems, functions, people• Use available “free” help and templates• Hire consultants if needed• Train and communicate

How to Implement

Don’t be overwhelmed

Create

Up

dateT

est

Audit

• Annual testing• Validation of tape backups

• Failover to backup facility• User validation• Signed acceptance forms

• 300-400 hours can get you started• 100-200 hours annually to test/audit• 100-200 hours annually to enhance/update• 5 person “core team”

• Senior Management Sponsor• Project manager• Tech writer• IT manager• Logistics/facilities coordinator

Effort Required

It can be a reasonable effort

• Creating the initial plan: $15-20K

• Annual updates and testing: $10-20K• Annual infrastructure costs: $200K• Peace of mind during a disaster: Priceless

Value Obtained – Basic BCP Plan

The best insurance policy you’ll ever buy!

QUESTIONS ?QUESTIONS ?

Tim Babco(985)801-5230

tim.babco@poolcorp.com