1

Lessons Learned from the April 16, 2007 Tragedy

at Virginia Tech

EDUCAUSE Live!November 5, 2007

Erv Blythe, VP for Information Technologies

2

Presentation Topics

• April 16, 2007• The impact – load on the communications

infrastructure• Web Communications• The notification issue• Radio communications interoperability• Identity management and privacy• Data preservation and computer forensics• A new perspective on IT

3

April 16, 2007

• 49 people shot

• 32 dead plus shooter

• 9 minutes

4

Crisis Response

• About 1500 first responders• 27 ambulances from 14 agencies and five hospitals

5

Figure 1: Communications Infrastructure Stress Factors April 16, 2007 System Normal April 16 Effect

University Web Site Access 455 gigabytes per MONTH (largest ever)

432 gigabytes in a DAY

3000 % increase

Virginia Tech Police Dispatch Center

400 -500 calls per day

2,027 calls 45 0% increase

Cellular Provider Capacity and Coverage

Designed for non -emergency peak load, limited in -building coverage

Added 3 COLTs, 2 in -building antenna systems, 200 phones

By April 17, temporary coverage/capacity added

Internet gateway capacity 500 Mbps Added 1 Gbps over 10GE research link

300% increase

University Switchboard 3,200 calls handled per week

9,878 calls handled 4/16 -4/21

300% increase

Telephone calls into Blacksburg Central Office

Reported by local provider Multifold Increase

Virginia Tech Telephone System Inbound Calls

25,000 calls inbound daily on average

75,000+ calls inbound on April 16

300% increase

Centralized Computing Systems Data Storage

Prior to 4/16, roughly 300 Terabytes/day

Since 4/16, over 600 Terabytes/day

100% increase

Data Preservation (12 week period)

3,000 tapes 11,700 tapes 390% increase

Communications

6

A humongous traffic spike …

7

Response by Private Carriers

• After the initial response period on April 16, cellular providers including AT&T, Sprint-Nextel, Verizon Wireless, and US Cellular all dispatched technicians to increase tower capacity.

• By April 17, Sprint-Nextel, Verizon Wireless, and US Cellular each had “Cell on Light Truck” systems operating on campus and each had provided emergency-use phones and accessories.

8

Infrastructure Facilities Duties

During the crisis, Virginia Tech IT faculty and staff were called upon to:

• Install telephone and data communications for– At least 9 geographically dispersed command centers – As well as media workrooms and counseling centers

• Perform network traffic balancing under unusual load

• Begin data collection and preservation activities

• Assist in obtaining location

information or class roster information

9

Lessons Learned:

Communications Infrastructure

• Public safety access point forwarding• Information centers• Command and control centers• Language barriers

10

Web Communications

• Due to unprecedented load, the webpage had to be stripped down to bare essentials from the normal design to a “Lite” design

Lessons Learned:

11

Normal home page before 4/16

12

“Lite” home page on 4/16

13

Lessons Learned:

Radio Communications• Responders from several jurisdictions, all with

different radios• Radio transmission was less than ideal in some

locations• Deficiencies in interoperability and coverage of

police, fire, and rescue radio communications are decades-old problems in the United States.

14

Notification System

A significant challenge during an emergency is providing mass notification of a threat and instructions for response.

All of these methods were used on April 16:• Broadcast e-mail to @vt.edu addresses (via LISTSERV)• Broadcast voicemail to campus phones (offices and residence

halls)• Recorded message on the WeatherLine/Hotline• VT.edu (www.vt.edu) and the Virginia Tech News website• University switchboard• Public media (TV, radio, news websites)• Siren system

15

VT Alerts• A short list of vendors for this service

was identified prior to 4/16/07

• Post-4/16 the vendor review process was expedited

• 3n was selected to provide hosted services for sending emergency messages

– Through cellular phones– PDAs and other wireless devices– SMS/ text messaging– Email– Or to voice services

16

• Virginia Tech students, faculty, and staff can “opt in” to subscribe to an automated event notification system that will notify them of an emergency situation. Each subscriber can select up to three different methods to be contacted. These contact methods include:

• Text message (SMS) to mobile devices

• Instant message (AOL, MSN, ICQ, and Yahoo)

• E-mail (including non-Virginia Tech addresses)

• Phone call to office

• Phone call to residence

• Phone call to mobile number

• Phone call to another destination

VT Alerts Continued

17

QuestionsQuestions

18

Lessons Learned: Privacy

• Maintaining vigilant attention to the restrictions of FERPA and HIPAA added significant complexity to the urgent data needs of April 16th and the activities of the days that followed.

• Immediately following the tragedy, it was recognized that information concerning victims should be restricted from inclusion in the public directory. The University Registrar’s office was asked to mark the student records as “confidential” in Banner to ensure the information would not be publicly accessible from the Enterprise Directory.

• For selected individuals identified by university, state, or federal authorities, copies of the e-mail content, Filebox content, ePortfolio content, and voicemail recordings were preserved on compact disk (CD) or digital video disk (DVD.)

19

Lessons Learned:

Identity Management• Across many of the response activities, one

overarching difficulty from a data perspective was the consideration of the diverse constituencies.

• An integrated, enterprisecentric identity management system would have reduced some of the complexity of this process.

20

• Support for investigators

• Support for families and victims

• e-Discovery requirements

Data Preservation

21

QuestionsQuestions

22

DC ATLANTA

DIVERSE OPTICALCORE

ADVANCED IP NETWORK

INTEGRATEDAPPLICATIONS

Pervasive Computing

Massive Data Storage and Access

High Performance Computing

Integrated Services

Inherent Resilience, COOP

Anywhere, Any Device, Any Service

SERVICES & GLOBAL ACCESS

CyberSecurity

Integrated Emergency Applications

Sensor Networks

Strategic RecommendationIntegrated IP -Based

Information ArchitectureLegacy Systems

CENTRAL IT SERVICES

EDUCATIONAL SYSTEMS

CATV

DATA

TELEPHONE

911 DISPATCH

EMERGENCY RADIO

Near TermRecommendations

VT Alerts Notification

Phone Features

CATV Alert System

Crestron Control

WUVT Alert

Police Dispatch

PSAP Forwarding

Blackboard/Scholar

Improve Cellular

Update Personal Info

Location Awareness

Local Interop/Data

RESPONSE

RECOVERY MITIGATION

PREPAREDNESS

Role of IT inEmergency Management

DCDCDC ATLANTAATLANTAATLANTA

DIVERSE OPTICALCORE

ADVANCED IP NETWORK

INTEGRATEDAPPLICATIONS

DIVERSE OPTICALCORE

ADVANCED IP NETWORK

INTEGRATEDAPPLICATIONS

Pervasive Computing

Massive Data Storage and Access

High Performance Computing

Integrated Services

Inherent Resilience, COOP

Anywhere, Any Device, Any Service

SERVICES & GLOBAL ACCESSSERVICES & GLOBAL ACCESS

CyberSecurity

Integrated Emergency Applications

Sensor Networks

Strategic RecommendationIntegrated IP -Based

Information ArchitectureLegacy Systems

CENTRAL IT SERVICESCENTRAL IT SERVICES

EDUCATIONAL SYSTEMS

CATV

DATA

TELEPHONE

911 DISPATCH911 DISPATCH

EMERGENCY RADIOEMERGENCY RADIO

Near TermRecommendations

VT Alerts Notification

Phone Features

CATV Alert System

Crestron Control

WUVT Alert

Police Dispatch

PSAP Forwarding

Blackboard/Scholar

Improve Cellular

Update Personal Info

Location Awareness

Local Interop/Data

RESPONSE

RECOVERY MITIGATION

PREPAREDNESSRESPONSE

RECOVERY MITIGATION

PREPAREDNESS

RESPONSERECOVERY

RESPONSERECOVERYRole of IT in

Emergency Management

A New Perspective on IT

23

24

QuestionsQuestions

25

Other Resources

Governor’s Review Panel final report

http://www.vtreviewpanel.org/report/index.html

Information and Communications Infrastructure Group report

http://www.vtnews.vt.edu/documents/2007-08-22_communications_infrastructure.pdf