IT disaster recovery,

communications &

facilities management

Simon Turner - IT Operations Manager

Day zero: 3:30am – 7am

Main activities

• Called by building supervisor – raised the alarm.

• Initiated DR plan.

• Contacted IT team.

• Contacted BT to reroute telephone numbers.

• Collected ‘battlebox’.

Lessons Learnt

• Battlebox on site, not where it should have been!

• Not all staff kept their work mobiles on or took them home.

Day zero: 7am – 11am

Main activities

• IT team arrive on site.

• DR plan reviewed.

• Existing infrastructure configured to start DR.

• Backup tapes retrieved.

• DR equipment delivered.

Lessons learnt

• Only one key to backup safe – lost in fire!

Day zero: 11am – 7pm

Main activities

• DR equipment installed and configured by Adam Continuity.

• Build of restore server.

• Initial run of backup tapes.

• Shared bandwidth with OCC, external access to web.

• 7pm team sent home to rest.

Lessons Learnt

• Tape drive delivered was wrong type.

Day one: Friday 16 January

Main activities

• Correct tape drive delivered –restores commenced.

• Emergency laptops purchased for frontline staff.

• Temporary website up and running.

• Contact suppliers for resource & help were needed.

Days 3 & 4: Saturday & Sunday

Main activities

• Restore process fully underway – some key systems back online.

• Migration to Office365 to establish email.

• Building of laptops ready for staff on Monday.

Key considerations

• Issue with available storage – re-factor SAN to create space for recovery.

• Problem with recovery of AD; took several attempts.

Days 4 and 5: Monday & Tuesday

Main activities

• Access to most systems – restores still running.

• Initiated VDI project to replace desktops.

• Building of laptops and desktops.

• Setting up desk spaces for users to work.

• Buying of laptops for staff.

Key consideration

• Remember file stores of unstructured data can take a long time to recover.

Wednesday 21 – Sunday 25 January

Main activities

• All systems and data recovered by Wednesday.

• More buying replacement equipment: laptops, servers, thin clients.

• Building/configuring/installing laptops & desktops & servers.

• Creating ‘gold’ image for VDI solution.

• Decision to purchase Surface Pro tablets for Planning team.

Lessons learnt

• Surface Pro decision to purchase delayed recovery and put pressure on IT.

• Procurement process was followed but was challenged later.

Major lessons learnt

Test your plans

• Test every year but cannot predict every eventuality.

• Makes sure people know what they should be doing.

Decision making and changes, be prepared

• to make decisions as the situation develops; and

• to change your plan as necessary to make recovery process successful.

Never assume anything!

• Only 1 key to safe??

• Manage expectations.

Communications

Main Activities

• Communication with staff, councillors and the public.

• Use of social media to get messages out quickly.

• Establish lines of communication to get messages out .

• Feed the media with up to date information.

Lessons Learnt

• Keeping the media up to date is a full time job.

• Timely and relevant information on services available.

• All lines of communication lost.

Facilities & Building Control

Main activities

• Liaising with Fire & Police services access to site.

• Providing advice to Fire & Police on safety of building – asbestos.

• Securing the site after blue light services handed over control.

• Full assessment of damage to building.

• Safety of officers accessing site.

Lessons learnt

• Who controls the building?

• Manage site access to all sites.

135 Eastern Avenue, Milton Park

What have we got?

• Reception area

• Cash office

• 3 interview rooms

• 7 meeting rooms

• 4 acoustic pods

• 13 individual offices

• 4 kitchen / break-out areas

• 380 desks

• Collaborative work space

• Store rooms