VERMONT STATE COLLEGESWHO WE ARE

VERMONT STATE COLLEGESSHARED SERVICES

• From the data center: (located in a state-owned building)• Wide-area network & single Internet connection• ERP• Identity & Access Management• Portal• IT Help Desk• Data warehouse• Conference & events application

• Enterprise IT Staff:• 6 programmers; 3 system admins; 1 network; 1 security

& data center

VERMONT STATE COLLEGESDISASTER RECOVERY

• First plan in 2007• Approved by Council of Presidents• Reviewed twice a year - updated February, 2011 • Key feature: assumed local and business process

planning “to accommodate the absence of critical systems for a period of up to two weekstwo weeks”• In-scope and out-of-scope clearly articulated

VERMONT STATE COLLEGESBUSINESS CONTINUITY

• Plans for admissions, accounts payable, accounts receivable, finance, financial aid, payroll and registrars processes• Reviewed and updated annually• Some business areas could go for several weeks

without extraordinary measures• Some could only function for few hours or days

IRENEAS OF 427 AM EDT FRIDAY...HURRICANE IRENE REMAINS THE MAJOR PLAYER FROM SATURDAY NIGHT INTO MONDAY MORNING AS IT LIKELY TREKS NNE FROM NEAR CAPE HATTERAS TO LONG ISLAND/CT AND ACROSS CENTRAL/NRN NEW ENGLAND. LATEST MEDIUM RANGE MODELS NOW ALL COMING TOGETHER ON THIS SOLN INCLUDING THIS MORNING`S GFS AND ECMWF WHICH ARE AMONG THE MORE RELIABLE IN HANDLING TROPICAL SYSTEMS. THESE TRACKS ARE INCIDENTLY NEARLY IDENTICAL TO THE OFFICIAL NHC FORECAST SO CONCERN IS NOW INCREASING CONSIDERABLY FOR A VERY HIGH IMPACT EVENT FOR MUCH OF OUR AREA...THE LIKES WE`VE NOT SEEN HERE IN QUITE SOME TIME. THIS WOULD INCLUDE TORRENTIAL RAINFALL WITH A HIGH RISK OF FLOODING...AND VERY STRONG AND GUSTY WINDS FOR THE EASTERN 2/3RDS OF THE FORECAST AREA. INDEED...THE WIND COULD LIKELY BE STRONGER THAN IN TROPICAL STORM FLOYD BACK IN 1999 AS MODELS CONTINUE TO SUGGEST IRENE`S INNER CORE DYNAMICS WILL NOT DETERIORATE AS QUICKLY AS NORMAL AFTER LANDFALL. LIKELY AIDING IN IRENE`S SLOWER THAN AVERAGE WEAKENING WILL BE A POWERFUL 250-300 MB JET ACROSS SRN/CENTRAL QUEBEC ALLOWING ROBUST EVACUATION AND DIFFLUENCE ATOP THE SYSTEM AS IT NEARS AND CROSSES OUR REGION. BY MONDAY...IRENE SHOULD BE EXITING QUICKLY INTO QUEBEC THROUGH THE DAY WITH WINDS SLOWLY ABATING AND OVERALL IMPROVING WEATHER CONDITIONS EXPECTED. STORM SPECIFICS FOLLOW...THOUGH PLEASE NOTE THESE ARE MY BEST ESTIMATES VIA COORDINATION WITH SURROUNDING OFFICES AND USE OF BLENDED NHC/HPC GUIDANCE. CONDITIONS COULD PAN OUT SLIGHTLY DIFFERENT. RAINFALL... TIMING...WIDESPREAD HEAVY RAINFALL EXPECTED SUNDAY INTO SUNDAY NIGHT FROM THE ADIRONDACKS EASTWARD...ENDING BY MONDAY MORNING. AMOUNTS...LIKELY TO RANGE FROM LESS THAN AN INCH IN THE ST LAWRENCE VALLEY...1-3 INCHES IN THE ADIRONDACKS...2-5 INCHES IN THE CHAMPLAIN VALLEY AND WESTERN VT...AND 3 TO AS MUCH AS 7 INCHES FROM THE GREEN MTNS INTO ERN VT WITH LOCALLY HIGHER AMOUNTS. FLOODING... AREA OF CONCERN...BIGGEST CONCERN FOR VERMONT STREAMS AND RIVERS BUT SOME THREAT FOR THE EASTERN SLOPES OF THE ADIRONDACKS AS WELL. TIMING...HIGHEST FLOOD THREAT WOULD OCCUR SUNDAY NIGHT INTO MONDAY AND COULD BE QUITE SIGNIFICANT. WINDS...

VERMONT STATE COLLEGESWHAT HAPPENED

• On Aug 28, 2011, at 8:13 PM, "Pollak, Dianne M" <dianne.pollak@vsc.edu> wrote:•  • > Just got a text from michael... First floor of the

hospital is under water... They are evacuating randall st. We are on generator.• > • > Dianne Pollak• > Director of Administrative Information Systems

Vermont State Colleges

VERMONT STATE COLLEGESWHAT HAPPENED

VERMONT STATE COLLEGES-WHAT HAPPENED

• By 6 am on 8/29 – water had filled basement and up to 5 feet of the first floor• Generator under water = power gone• Entire state office complex flooded• Roads to town closed

VERMONT STATE COLLEGESBUSINESS IMPACT

• Two colleges had started classes• Two colleges were about to start classes on

Tuesday• One college was still in “marketing mode”

VERMONT STATE COLLEGESCOMMUNICATIONS

VERMONT STATE COLLEGESSHORT-TERM SOLUTIONS

• Monday - Emergency web site up by Monday • Monday - Alternate e-mail addresses collected• Tuesday - Wide-area network and Internet restored• Monday - Colleges using internal e-mail• Monday-Thursday – Some paper paychecks

handwritten• Tuesday-Friday – Air-conditioning and server issues• Friday morning – ERP and other critical systems

restored

VERMONT STATE COLLEGESEARLY LESSONS

• Even if you can’t afford what you really need, have a plan!• Business continuity plans had some issues• Never underestimate the power of denial• Good relationships sustain you; bad ones are

killer• Stay on the lookout for good ideas and be flexible• It’s true: never waste a good disaster• Keep track of who’s doing what• Communications are critical & difficult

Data center

DATA CENTER GENERATOR...

High WaterMark

VERMONT STATE COLLEGESEARLY LESSONS

• Take notes everywhere, in every conversation• Have a conference call system set up• Texting is critical (and set up those contacts!)• Have an alternate e-mail address for all critical

contacts – IT and priority business areas• Have a plan for alternate web site hosting• Staff check-in: food and sleep• Staff in different places on the resilience scale• As everyone gets tired, mistakes will be made• Danger in false sense of stability/reliability

UP NEXT: SNOWTOBER

From: The EDUCAUSE CIO Constituent Group Listserv [mailto:CIO@LISTSERV.EDUCAUSE.EDU] On Behalf Of Theresa RoweSent: Thursday, November 03, 2011 8:44 AMTo: CIO@LISTSERV.EDUCAUSE.EDUSubject: Re: [CIO] Snowtober

Thanks so much for taking the time to write out the details of your experiences.  We all learn from these posts.  Our experiences with electrical power failures and generators have been very difficult to manage.  Every situation has Every situation has been different, so you really need thinking, problem-been different, so you really need thinking, problem-solving people on hand, rather than a scripted actionsolving people on hand, rather than a scripted action. .  You've given us more to think about.

Theresa

“Thinking, problem –solving people”

VERMONT STATE COLLEGES OPPORTUNITIES

• Reinforce the importance of both disaster recovery and business continuity planning and underscore the differences• Make the case for disaster recovery resources-

less difficult in the reality of business interruption• Change things that were “unchangeable”• Focus on what’s important: people and mission

VERMONT STATE COLLEGESTODAY

• Access to building restricted to 2-3 staff• Off generator; data center is only room with

power• Legislature to determine future use of the

complex • IT Staff distributed across the state• Building two data centers – looking at co-location• Dealing with displaced employees• Re-projecting the annual project list• Getting back to “normal” - IT reporting cycle