Not the Unthinkable, But What We Didn’t Think Of: Preparing For and Recovering From Disaster

Simmons College April 8, 2006

Not the Unthinkable, But What We Didn’t Think Of

Preparing For and Recovering From DisasterCharles Patch

Director of SystemsThe Historic New Orleans Collection


Hurricane Katrina was the most destructive natural disaster in U.S. history. The overall destruction wrought byHurricane Katrina, which was both a large and powerful hurricane as well as a catastrophic flood, vastly exceededthat of any other major disaster, such as the Chicago Fire of 1871, the San Francisco Earthquake and Fire of 1906,and Hurricane Andrew in 1992.

– The Federal Response to Hurricane Katrina, Lessons Learned. Department of Homeland Security, February 2006

Disaster in the Real WorldThe Official Word


Living in a post disaster world





City population < half of pre-Katrina levels > 60% of city uninhabited 20 public schools in operation

15% of student body 5 of 14 branches of public library open Most museums and archives not open until late February


Living in a Post Disaster WorldCondition of Public Libraries in LA Post-Katrina


Disaster in the Abstract

Q: It’s Armageddon. You have 30 minutes to leave. You are carrying provisions for 3 days, your family and your pets. What is the one last thing you will take? From home? From Work? How long will it take you to find this thing?


Disaster in the Abstract

Increases in scope and intensity increase impact of a disaster. Impact can vary in regard to timing:

As intensity increases, the greater the need for fast recovery measures The recovery of materials may be time-dependent Severely damaged physical plants need immediate stabilization

As scope increases, recovery becomes more difficult to execute Common services and infrastructure may be unavailable

Scope vs. Intensity can affect how you recover Materials Human Resources Property

Both scope and intensity hamper the execution of business processes A widespread disaster can prevent access to facilities as much as

an intense but localized disaster


Responding to Disaster in the Abstract

Disaster Prep vs. Disaster Recovery vs. Business Recovery Disaster preparation = protection from uncontrollable

events: Protecting assets:

Materials Human Resources Property

Disaster Recovery is about the restoration of damaged assets and the recovery of lost resources

Disaster Preparation + Disaster Recovery Plan = Disaster Plan Disaster plans develop “Controls”:

Disaster Prep: install sprinkler system Disaster Recovery: stock conservation supplies



Disaster Prep vs. Disaster Recovery vs. Business Recovery Disaster preparation = protection from uncontrollable

events: Protecting assets: the “tiki hut” at the Museum of Natural History

in NYC – to protect network gear from an aging sprinkler system



Business Continuity is about the survival of basic business functions If you can’t function, you can’t recover

The larger/greater in scope/more intense the disaster, the more important business continuity becomes

Business Continuity is about managing the consequences of an adverse “risk event” in terms of basic business functions Specifically, business functions that have been interrupted



TERMS: Business Activities Business Continuity Plan Business Impact Analysis (BIA) Business Interruption Event Business Process Business Resource Business Resumption Plan Control Key Business Process Maximum Acceptable Outage (MAO) Outage Procedures Resources Risk Event Risk Management Plan Service Area Contingency Plan



Business Continuity Management (BCM) is part of a larger Risk Management Plan Disaster Recovery is a subset of BCM

Risk Management considers both negative and positive risk

The Prime Directive: Plan for the best, prepare for the worst Proactive design and implementation of

controls to prevent risks from occurring Reactive design of controls to mitigate effects

of adverse events that occur.


Responding to Disaster in the Abstract: Overview of Risk Management Process

Establish Context Determine key business objectives, processes and resources

Identify and Assess Risks

Identify, analyze, rate and prioritize risksEvaluate design of existing controls.Redesign controls if necessary

Implement Controls Establish plan. Implement controls

Monitor and Review Review operation and continuing suitability of controls


Responding to Disaster in the Abstract: Organizational Context

Organizational Objectives

Output Group Output GroupOutput Group

key business process





Business Support Process




Responding to Disaster in the Abstract: Organizational Context

Collect, Interpret, Display Cultural Materials for General

Public

AdministrationCollections

Management

Information Systems

Payroll

Facilities management

Accounts Payable

Acquisitions

Data storage

File hours with Paychecks Inc.

Create loan records in Collection Management

System

Create semi-weekly backups of all data


Responding to Disaster in the Abstract: Risk Management Process

No

Yes

No

Yes

Determine Possible Risk

Events using risk framework

Determine likelihood and consequence

without control in place

Determine risk level and

compare with acceptable risk

Acceptable?

Evaluate design of existing

controls

Determine likelihood and consequence with control in

place

Determine risk level and

compare with acceptable risk

Acceptable?

Redesign controls

Record in Risk Register

Identify

Analyze

Evaluate

Control

Document


Responding to Disaster in the Abstract: Risk Identification


Responding to Disaster in the Abstract: Analysis & Evaluation

Is a risk minor (acceptable) or major (unacceptable)? What is “acceptable”?

What are the consequences in terms of resources? Do they have a detrimental impact on

Staff Facilities Collections Telecommunications Information Systems

Rank the consequences on the degree of impact on these resources What is it worth to you?

A consequence can be minor if its fullest possible impact will not be detrimental to institutional resources.

Last question: will a consequence have an impact on normal business operations? Events which have detrimental impacts on resources are likely to

impact business operations <- remember this!


Responding to Disaster in the Abstract: Controls

Preventative Controls Stop the risk from occurring in the first place

E.g.: require passwords to access the computer system Ban flammable solvents from processing areas

Corrective Controls Minimize the consequence of a risk event once it has occurred

Sprinkler system Backup tapes for computer system

Evaluate current control mechanisms If the consequences remain unacceptable, redesign them


Responding to Disaster in the Abstract: Monitor and Review

Incorporate risk management into normal business operations Strategic planning Budget process Facilities Maintenance

Make risk management part of operational management Supervisors and key workers participate in planning and analysis Sign-offs by managers on risk controls

Practice procedures Structured walk-through Review for validity Confirmation of supplies needed


Responding to Disaster in the Abstract: The BCM

A Business Continuity Plan is a Corrective Control An interruption in a Business Process is an adverse consequence

of a risk event A complete interruption of this sort is an Outage.

How long can an Outage be tolerated before the viability of the organization is threatened? This duration is the Maximum Acceptable Outage (MAO)

Risk Management

Conrols

Preventative

Corrective

Disaster Recovery

Business Continuity



Business Continuity in 6 arduous steps1. Initiate the project:

Document Hypotheses, Scope and boundaries Establish a management committee Establish a budget and timetable

2. Identify Key Processes: Key institutional objectives Key “outputs” or end results of any process critical to the

institution Understand the key activities, resources and their

interdependencies Rank the importance and create an activity and resources

“schedule”3. Conduct a Business Impact Analysis:

Identify key personnel Conduct interviews Document Concerns Determine MAO



4. Design Continuity Controls Review existing controls Identify and evaluate options and alternatives Select the alternative activities and resources

5. Implement the controls Establish recovery teams Document “Service Area” action steps Establish event escalation process Obtain and consolidate contact and inventory lists Document recovery management process

6. Test and Maintain the plan Paper test Manual Verification Supply validation Supply, Service and equipment availability Structured walkthrough Unannounced team assembly


Responding to Disaster in the Abstract: Who Does All This?

Roles: Disaster Recovery Coordinator Service Area Teams Vendors / Service Providers


Responding to Disaster in the AbstractThe Inconvenience of the personal lives of personnel

“Normal” personnel issues Employees with dependents Employees with physical disabilities Extended absences of employees with critical knowledge

Disaster issues that exacerbate personnel problems People with dependents need more time to prepare for an

evacuation People with dependents cannot respond as quickly to emergency

calls as those without dependents People with dependents may not be able to return as early as

others E.g.: no school in New Orleans until January 2006 E.g.: inadequate medical care


Responding to Disaster in the Real World:

Background: The Yearly Drill Street flooding is common Hurricane / Tropical Storm alerts are yearly events in New Orleans


Responding to Disaster in the Real World: Cindy – Wednesday 7/6/05


Responding to Disaster in the Real World: Dennis –Sunday 7/10/05


Responding to Disaster in the Real World: Katrina – Monday 8/29/05


Responding to Disaster in the Real World: Living With the Past


Responding to Disaster in the Real World: Living With the Past


Responding to Disaster in the Real World: Preventative Controls

Keep Buildings maintained and up to code Fire alarms, fire extinguishers checked on semi-yearly basis Routine roof inspections Plumbing and electrical services kept to code or better Lighting and signage maintained properly

If you have the choice, don’t put things where they could be harmed


Responding to Disaster in the Real World: Preventative Controls

Follow best practices for data management and storage including offsite backups

Make duplicates or copies of important documentation and store them offsite E.g.: All accession records and documentation microfilmed

at the end of each year, including updates to active collections

Stored in two difference places,

one offsite


Responding to Disaster in the Real World: Corrective Controls

File plans with local fire and police departments

Be sure that room and floor designations are the same in all documentation, including insurance, collections management systems and fire department plans.

Keep current contact lists for staff

Consider requiring third-person contact numbers

Keep current contact lists for vendors, lenders, staff and emergency services


Responding to Disaster in the Real World: Corrective Controls


Responding to Disaster in the Real World: What We Did Right

Routine Best Practices Paper Accession records microfilmed and

stored offsite Computer records backed up routinely

and stored in separate buildings on a scheduled basis.

Payroll and benefits services handled offsite




What We Did Right: Disaster Preparation

Disaster preparedness planTHE HISTORIC NEW ORLEANS COLLECTIONDISASTER PREPAREDNESS PLANTABLE OF CONTENTSI. Mission Statement/Staff Assignments Page 4II. Emergency Prevention Strategies Page 8III. Communication in the Event of Page 10

an EmergencyIV. Visitor Safety

A. Fire Page 11B. Illness

Page 11C. Accidents Page 12D. Robbery Page 12E. Vandalism Page 12F. Bomb Threat Page 13

V. Emergency Resources Page 15Building Resources Page 16Collections Resources Page 17

VI. SuppliesPage 20

VII. Fire Prevention Page 21VIII. Hurricane Preparedness Page 22


What We Did Right: Staff Preparation

Regular testing of alarm system and drills Training of docent staff in emergency procedures All areas kept stocked with emergency supplies


What We Did Right: Staff Preparation


Responding to Disaster in the Real WorldWhat We Did Right 3: Disaster Preparation

Prioritization of materials



Storm monitored by Disaster Coordinator from first appearance early in the week when still a tropical storm

Emergency supplies checked, calls made to confirm that all staff has copy of Disaster manual

Preparation Meeting held Friday morning, August 26th in directors office Emergency message activated Staff put on alert Head count of who is available this day and the next

Initial clean-up of processing areas completed Storm monitored throughout day Saturday, escalation of probable hit, staff called in at 7:30 AM,

16 people respond.


Responding to Disaster in the Real World What We Did Right: Set Up

Saturday: Fresh computer backup tapes produced Collection Management System data and

Membership/Development data compressed and sent via FTP to vendors

Computer system shut down and disconnected from power source

Major servers moved to second floor of buildings and covered with plastic sheeting

Phone system shut down and disconnected from power source Art work removed from exterior walls


Responding to Disaster in the Real World What We Did Right: Set Up

Sunday: Remaining staff either evacuates city or takes protective

measures at their homes Hurricane raised to Category 5, then Category 4 status; Direct

hit procedures invoked. Five staff return to carry out remaining preparations Removal of all works from floors and surface areas Relocation of all art works on first to second floors


Responding to Disaster in the Real WorldWhat We Did Right: Set Up

Building Preparation


Responding to Disaster in the Real WorldWhat We Did Right: Set Up

Building Preparation In-place precautions


Responding to Disaster in the Real World: August 28, Sunday: The Evacuation

Mayor declares mandatory evacuation on Sunday morning

Extremely difficult travel conditions prevail 8 to 12 hours to reach Baton Rouge (80

mile) Director contacts all department heads by

phone to establish whereabouts Director and several other staff members

stay to ride out storm


Responding to Disaster in the Real World: 8/29 – 9/3 : Katrina “misses” New Orleans

Hurricane tracks to the East of the city Power and phone service throughout

Southern part of state lost during most of Monday

Monday night, most believe New Orleans has “dodged the bullet” again and plan return to the city on Tuesday morning.

Tuesday morning television reports 80% of city is under water

Phone service dysfunctional, cable and Internet access extremely rare

By Wednesday there are 100,000 extra people in Baton Rouge

Google Group for staff established on Thursday night

Initial reports suggest it will be months before the city can be inhabited again

THNOC arranges to participate in a state police convoy. Due to dangers in the city, the convoy is cancelled


Responding to Disaster in the Real World What we did right: The Evacuation

We learned to work the damaged phone system Lesson: cell phones worked via text message in the New Orleans area and

with voice when calling outside the New Orleans area code. First Contact: Tuesday night, August 30

Staff members in the area organize rescue of valuable materials. Initially planned for September 2, delayed because of conditions until

September 8 “High Priority” materials were clearly marked and were stored in easily

accessed locations Offsite location for materials established upstate Movers retained on special contract called in



Google Group created 9/1 Staff assured of continued employment, salary and benefits

This is the first message posted to the list that is not a response to the “Where are you?” query.



Staff web page created 9/5



Personnel Database created 9/13





Responding to Disaster in the Real WorldWhat Went Right: The Evacuation

Staff stayed of their own volition Our vendor becomes our go-between


Responding to Disaster in the Real WorldWhat Went Right: The Evacuation

Staff members found the Google Group Our web master took the site with him on his laptop Our network manager was close enough to join the

convoy Our vendor anticipated our need for a personnel database The French Quarter was spared and sufficient staff were

on hand to open the facility on October 3.


Responding to Disaster in the Real WorldWhat We Could Have Done Better

No designated essential personnel Compounded by timing issues

Inadequate communications procedures Dependence on phone system

for communications No working email addresses No common contact point for

reaching individual staff members

No alternative admin site And no policy for defining how

to recognize a workable alternative

No back-up network site Inadequate conservation supplies on

site Most suppliers for recovery

materials listed in handbook were local



Some things we didn’t think (that might have been caught in a risk analysis) Book Trucks The Real Truck:

Who was going to drive this?



Some things we didn’t think (that might have been caught in a risk analysis) The giant image files that weren’t backed up



Some things we didn’t think (that might have been caught in a risk analysis) Network “single point of failure” (actually 4 potential failure points) Magnetic Locks (no electricity, no security)


Responding to Disaster in the Real WorldWhat Didn’t Go Well?

A failure to communicate Administration chose to stay close to the physical institution rather than

communication hubs. Most were in rural areas with poor or no Internet access Nearly all communication from the administration was by cell phone,

which worked only sporadically Administration did not take advantage of communications channels

Tended to wait until staff contacted them rather than contacting staff Rarely posted information on the staff web page or the Personnel DB

bulletin board

Time lost to making ad hoc arrangements No pre-arranged alternative “home” for rescued collections No pre-arrangements for care of “essential staff” before or after

disaster No plan for setting up network in alternative location while staff was

disbursed (as opposed to securing backups)


Responding to Disaster in the Real WorldWhat Didn’t Go Well?


Responding to Disaster in the FutureLessons Learned?

Most forms of communication will fail No single source will provide adequate information concerning real

events Individuals or small groups working independently were most effective Data redundancy is highly desirable Professionals involved in activities unrelated to what authorities regard

as the central economic interests of the city and region will be marginalized



Communications 101 Mechanics:

Create multiple forms of asynchronous communication Out-of-Region phone bulletin board Web discussion groups/web pages/staff directories “Real” bulletin board on site for leaving paper messages to

other staff Develop policies for choosing administrative sites that

provide for access to the disaster area AND good communications



Communications 201 Semantics:

Create a protocol for communicating with staff Procedures for posting staff supplied information

Scheduled administrative updates and news “Test the channel”

No news is bad news


Responding to Disaster in the Future Information

“We heard stories about helicopters being shot. But you’ve been in helicopters, and you know how noisy they are. The only way you know you have been shot at is if there’s a bullet hole,” Landreneau said. “There were no shots fired at our helicopters.”

-Maj. Gen. Benny Landreneau, Louisiana National Guard commander , quoted in “What We Signed Up For” by John Hill, Louisiana Life Magazine, March 2006

Information: Who To Believe?



The New York Times has confirmed that one person was murdered at the Convention Center and one at the Superdome, and the Times-Picayune has confirmed that a National Guardsman was attacked by an assailant wielding a metal rod in the darkened Dome. The coroner’s early report implies that the murder rate among those stranded in Katrina’s aftermath was at least five times New Orleans’s normal murder rate. This real, not imagined, violence prevented New Orleans from getting the level of volunteer and professional help it needed after Katrina. - “Who’s Killing New Orleans?” Nicole Gelinas, City Journal, Autumn 2005, http://www.city-journal.org/index.html




If the images were to be reduced to a sentence in the minds ofUptown New Orleans, that sentence would be: Crazy black people withautomatic weapons are out hunting white people, and there's no bag limit!"The perspective you are getting from me," one of Fort Huger's footsoldiers said, as he walked around the living room with an M-16, "is theperspective of the guy who is getting disinformation and reactingaccordingly.“

-Michael LewisWading Toward HomeNew York Times Magazine, Oct 9, 2005




The media’s willingness to report thinly attributed rumors may . . . have contributed to a cultural wreckage that will not clean up easily. . . . Victims, officials and reporters all took one of the most horrific events in American history and made it worse than it actually was,” Lecture by New York Times media reporter David Carr, September 2005

“Four weeks after the storm, few of the widely reported atrocities have been backed with evidence,” - Times-Picayune September 26, 2005




Weigh all information carefully and whenever possible, rely on trusted sources in the area



Responding to Disaster in the FutureInformation: Local Discussion Lists


Responding to Disaster in the FutureInformation: Local Discussion Lists


Responding to Disaster in the Future Facilitate Independent Agency

Provide materials for short term survival as well as collections care.

Leverage the actions of individuals by providing some method of coordinating among disparate players

“Response to Katrina is less dependent on traditional disaster plans than on improvised actions as conditions permit.”-Report of Hurricane Katrina Damage Assessment:Debra Hess Norris (Heritage Preservation)Richard Pearce-Moses (Society of American Archivists)David Carmichael (Council of State Archivists)21 September 2005


Responding to Disaster in the Future Have Have More More Than One One

Off-site storage of data Meta-data tagging of individual digital

assets Cooperative arrangements for

network operations with sister institutions

Cooperative arrangements for physical storage with sister institutions


Responding to Disaster in the Future Have Have More More Than One One

Just a few miles west of Pass Christian, the Hancock County Historical Society in Bay St. Louis fared much better with very little water damage and a vault that held, protecting thousands of documents, including family diaries and thousands of local photographs.

Charles Harry Gray, the executive director, was prepared in case disaster struck. Over the years he had been making copies of all of the group's most treasured documents, including 30,000 pictures. Not one single photograph or record was lost.

They are the pieces of Bay St. Louis' 306-year history that made the town of 8,230 what it is today, he said. Many of the copies were on computer disks and hard drives, others were sent to the University of Southern Mississippi, two hours north in Hattiesburg.

"It is imperative that you have copies in other locations because you never know what's going to happen, what the next catastrophe is going to be, and there certainly will be one," Gray said.


Responding to Disaster in the Future Disenfranchisement

Though archivists began asking for re-entry into the city on August 31…(all) requests for attention to historic paper records were denied…The Department of Culture, Recreation and Tourism failed to include archivists on planned reconnaissance trips into the city or to include anyone from the archival community in its planning meetings

Susan TuckerCurator of Archives at the Newcomb College Center for Research on Women Email, 10/14/2005


Responding to Disaster in the Future Disenfranchisement


If this ever happens again…

All staff should have a web-based email address . Publish and update as part of the recovery documentation

All staff should provide a contact phone number outside of the region We should establish a telephone bulletin board outside of the institution We should consider a subscription to a satellite phone service All department heads should have lap-top computers Online forum should be set up permanently with a single instructional

message At least one institutional email address that is hosted on an off-site

server Staff-only emergency web page should be set up permanently Policy for alternative administrative site

Not necessarily choose an actual location but develop criteria for choosing one

Develop cooperative arrangements with other institutions for storing collections

Develop cooperative arrangements with other institutions to host our computing equipment and run our network

Contract for archiving of digital assets that are difficult to transmit or store easily to tape


Web Sites of Interest

Business Continuity Management NEDRIX - New England Disaster Recovery Information X-cha

nge http://www.nedrix.com/

Risk Management Standards http://www.incom.com.au/enterprise-risk-management-standards.htm

Better Practice Guide - Business Continuity Management - Keeping the wheels in motion http://www.anao.gov.au/WebSite.nsf/Publications/

4A256AE90015F69B4A2568EE0010062B National Archives of Australia - Business continuity plannin

g for digital records http://www.naa.gov.au/recordkeeping/er/guidelines/9-continuity.html

Business Continuity Planning & Disaster Recovery Planning World http://www.disasterrecoveryworld.com/

http://www.nedrix.com/800/802.htm

http://www.nedrix.com/800/802.htm

http://www.incom.com.au/enterprise-risk-management-standards.htm

http://www.anao.gov.au/WebSite.nsf/Publications/4A256AE90015F69B4A2568EE0010062B

http://www.anao.gov.au/WebSite.nsf/Publications/4A256AE90015F69B4A2568EE0010062B

http://www.naa.gov.au/recordkeeping/er/guidelines/9-continuity.html

http://www.naa.gov.au/recordkeeping/er/guidelines/9-continuity.html

http://www.disasterrecoveryworld.com/

http://www.disasterrecoveryworld.com/

Technology

Not the Unthinkable, But What We Didn’t Think Of: Preparing For and Recovering From Disaster