CHIME LEAD DC 2014 “Key Attributes for Success, Challenges and Critical Success Factors” with Paul Scheib, CISO and Senior Director IS Operations, Boston Children’s Hospital

A CHIME Leadership Education and Development Forum in collaboration with iHT2

Creating an Effective Cyber Security Strategy

________ Key Attributes for Success, Challenges and

Critical Success Factors

Paul Scheib

Senior Director Information Services & CISO

Boston Children’s Hospital

#LEAD14

Case Study: When Hacktivists

Attack Your Hospital


The Cyber Threat

Under attack

Our response

Lessons Learned

Who is Boston Children’s Hospital


• Regional medical center in Eastern Massachusetts with 13 satellite locations - 395 bed pediatric teaching hospital, affiliate of Harvard Medical School

• Approximately 25,000 inpatient admissions each year and 200+ specialized clinical programs schedule 557,000 visits annually

• One of the top rated pediatric institutions in the world (US News & World Report), World's largest research enterprise based at a pediatric hospital

• Over 8000 staff and ~14,000 users • Diverse user community

• Full-time employees and Foundation physicians • Residents, fellows, researchers and rotational staff

A Real Threat


• March 20, 2014 – notified by external cyber intelligence group about Twitter/ Pastebin posting by Anonymous, threatening attack - result of highly publicized child custody case

• “d0x” of staff and presiding judge posted • “Details” of BCH external web site posted

Who is Anonymous?


• Anonymous is a loosely associated international network of activist and hacktivists

• Resume includes attacks on Bank of America, Sony, Boston Police, CIA and Sarah Palin.

• Weapons of choice are Distributed Denial of Service, web site defacing, & exposing confidential information.

• Seeks publicity to rally their followers • Posted YouTube videos threatening

Boston Children’s Hospital

Was This the Real “Anonymous”?


• Convened Hospital’s general Incident Response Team • Inventoried potentially impacted applications • Began forming contingency plans - focused on potential

of loosing or cutting ourselves off from Internet • Message to entire organization emphasizing vigilance,

email security best practices • Contacted law enforcement • Redoubled our security efforts and prepared for possible

hacking attempts

Not hard to get details they posted Not hard to post a video on YouTube

Should we take this seriously or is it a hoax?

The Cyber Attack


• About 3 weeks later... low volume DDoS attack starts • Mitigated by network changes • Cat and mouse – we address attack, they change

tactic/increase volume • 1 week later, Easter/Patriot’ Day weekend (Boston

Marathon bombing 1 year anniversary) • Massive uptick in DDoS volume • Engaged 3rd party vendor’s Emergency Services and

within 8 hours began blocking DDOS attack

Internet Traffic During DDoS Attack


The Cyber Attack Evolves


• Direct attacks on exposed ports, web sites • Proactively took down virtually all externally facing

sites: research, philanthropy, patient and provider portals, etc…

• Massive influx of malware laden emails • Proactively shut down entire email system for ~24 hrs • Re-emphasized to staff to not open suspicious

mails/attachments • Ensured no malware made it through filters

What did we experience?


• DDOS attack created short periods of web site outage. • Attack reached 27 Gbps aimed at a 10Gbps connection. Congestion

affected Harvard’s ISP. • Additional attacks took down web sites of NStar, Wayside Youth, the

Mass. Medical Society, and the Town of Framingham. • Several attempts to deface BCH website. • Massive influx of malware laden emails

• Proactively shut down entire email system for ~24 hrs. to ensure no malware made it through filters

• Re-emphasized to staff to not open suspicious mails/attachments

• Attempts to compromise systems to potentially expose patient and confidential data, through brute-force attacks, SQL injections, buffer overflows, and the recent HeartBleed vulnerability.

Cyber Attack Response


• Initial attack mitigated by network architecture and changes

• Proactively shut down critical systems to reduce attack surface

• Projected likely attack escalations and formulated real time response plan

• Engaged outside security experts and law enforcement

• DDOS attack flitering

• Breach investigation services and penetration testing of our DMZ systems

• Web application firewall protection of DMZ ePHI systems

• Contingency plans developed to respond to extended Internet outage • Internal systems (EMR, ERP, etc) remain available while external services (ePrescribe, some

Pharmacy apps, etc) not available. • External communication disruption – email, payers, portals, supply orders, … • Impact across most functions – Finance, Supply Chain, HR, Clinical, Research.

• Staffed, and continue to staff, Intrusion Detection tools 24 by 7 to identify and block attacks


Cease Fire

• About 1 week after high volume DDoS started, it abruptly declined, to a low trickle

• Only gradually brought externally facing sites back online, after extensive 3rd party scanning and (re)penetration testing

What Did We Learn


• DDoS is a real threat and countermeasures are critical! • Know what systems (or features within systems) depend on Internet

access, and have contingency plans for those • Recognize importance of email, and need for alternate forms of

communication • Challenging to defend an extended cyber attack with “peace time”

staffing levels • Difficult to separating signal from noise - need a baseline to help

detect escalation of cyber activities

Q & A

Paul Scheib [email protected]


Insert Twitter handle(s) here