Getting Your Network Operations In Orderhealthsystemcio.com/presentation/networkops-webinar.pdfWebex Support 1-866-229-3239 Event #666 024 476 Fall 2011 •Mandate to consolidate data

Slide Deck: http://goo.gl/Fij8axWebex Support 1-866-229-3239

Event #666 024 476

“Getting Your Network Operations In Order”

A Complimentary Webinar From healthsystemCIO.com

Your Line Will Be Silent Until Our Event Begins at 12:00 ET

Thank You!


Event #666 024 476

Housekeeping

• Moderator – Anthony Guerra, editor-in-chief, healthsystemCIO.com

• Ask A Question• We will be holding a Q&A session after the formal presentations.

• You may submit your questions at any time by clicking on the QA panel located in the lower right corner of your screen, type in your questions in the text field and hit send. Please keep the send to default as “All Panelists.”

• Download the Deck • Go to Download today's deck at:

http://healthsystemcio.com/presentation/networkops-webinar.pdf

• Shortened URL at bottom of all slides

• View the Archive• You will receive an email when our archive recording is ready.

• Separate registration is required.

http://healthsystemcio.com/presentation/networkops-webinar.pdf


Event #666 024 476

Agenda — Approximately 40 Minutes

• 25-30 minutes: Scott MacLean, Deputy CIO and Director of IS Operations, Partners Healthcare

• 10-15 minutes: Q&A w/Scott MacLean


Event #666 024 476

“Getting Your Network Operations In Order”


Event #666 024 476

Fall 2011

• Mandate to consolidate data centers, mature disaster recovery options and improve information security posture

• Be able to quickly respond to events and incidents, particularly for major applications for clinical, revenue cycle, ERP, email and more

• Funding available, but daunting task


Event #666 024 476

About Partners HealthCare

• Founded in 1994 by Brigham and Women’s Hospital and Massachusetts General Hospital in Boston

• Physicians Organizations, Nine Acute Care Hospitals, Rehabilitation Network, Specialty Hospitals, Home Care, Insurer

• Currently implementing a single clinical and revenue cycle system across the enterprise

• Participating in Pioneer ACO and risk-based commercial contracts, so significant effort on Population Health Management and analytics

• Expect fluctuation in clinical affiliations


Event #666 024 476

Enterprise Growth

• Prior to the Massachusetts and Federal healthcare reform laws, we sought to have primary/specialty practices, community hospitals and non-acute facilities to support our AMCs. These entities were responsible for their bottom lines.

• Clinical coordination and use of IS to support those activities took place, but weren’t the main focus in a fee-for-service system.

• While the AMCs brought IS infrastructure under central management, community hospitals and other facilities maintained local support, largely for reasons of expense.

• Not apples to apples comparison. Board raised questions about disaster recovery and information security.


Event #666 024 476

Data Center Consolidation

• Funding was provided to consolidate community/non-acute data centers to our tier three facilities. Disaster Recovery equipment was funded as a carrot. Sites are expected to cover ongoing operating costs.

• Data Center Strategy• Multi-year study – investment/flexibility issues.

• Just completed new data center west of Boston; will retire our Charlestown data center this summer.

• Completed “cloud” strategy, but most applications still on premise. Using some SaaS, Office 365. Have BAAs with some cloud providers.

• Run critical workloads six months in one facility, then flip to other to practice DR operations.

• Do we want to be in the data center business?


Event #666 024 476

Marlborough Data Center – opened Fall 2014

9


Event #666 024 476

Data Center Plans

Needham Data Center Charlestown Data Center

Community

Site

Partners Network

New Data Center Online – 2015

9 Community Data Centers Consolidated

Cloud Services Expanded to Include Some

Production

Cloud Services

Non critical production

Test DR

Development

Secure

Access

Production

Proposed New Data Center

Community

Site

Marlborough Data Center

5


Event #666 024 476

New Affiliates

• Recent affiliations come with increased awareness about collaboration and security, but suffer from non-profit niceness.

• CIO given opportunity for input, but organizational structures and business priorities slow integration timelines.


Event #666 024 476

Command Center – opened Fall 2013

12


Event #666 024 476

Key Command Center ObjectiveThe overall objective of the Command Center is to have a proactive and Collaborative

functional monitoring and support center with SME capabilities. A consolidated group for

which Event and Incident Management can reduce MTTD, MTTR and impact to the

Business

13

UsersLevel 1 Support

Communications

Level 2 and 3 Support

Resolution

Monitoring Tools

InformationalNotification

Warnings

ExceptionHelpdesk

Request for Change

Forward Schedule Change

Current State

Reconciliation

Problem Records

Problems/Known Errors

Workarounds

Root Cause/Workarounds

Incident Management

Event Management

Problem Management

Change Management

Command

Center

CMS/CMDBKnowledge

Source: AT&T Consulting


Event #666 024 476

Command Center Video Wall View

14

Dedicated eCare console for monitoring the infrastructure

components

This is the main console view. Shows Events and Incidents being worked on by the teams

Dedicated SAN Console for Storage monitoring

Cisco CTC Console for monitoring fiber between major data centers

Connected Real time communication to the Helpdesk via Lync

Console to monitor the current active alarms and associated

Tickets


Event #666 024 476

Functional View – Target State

15

Command Center

CC NetEng

CC Telecom CC NetSec

CC Server CC Open Systems Tools Engineer CC Storage CC SQL

95+% tickets resolved within CC

TBD

HelpDesk

Legends

Incidents Only

Server SQL Exchange SAN Web

CC NetEngEast

CC NetEngWest

NetEng Telecom NetSec

Command Center Model - Target StateObjective – to resolve 90% of Events/Incidents within the CC

TSO Command Center Engineers

INC Command Center Engineers

CC Manager

Help Desk to assign all Incident tickets to CC queues for the skills present in the CC

Potential skills in CC – TBD*

Objective is to resolve 90% of the

tickets within Command Center

Objective is to resolve 90% of the

tickets within Command Center

PeopleSoft

SOC

eCare

Other GroupsDesktop, NetSec, etc.

TBD

PODSfor

services/technolog

ies

< 70 % tickets resolved within CC

< 50 % tickets resolved within CC

Source: AT&T Consulting


Event #666 024 476

Command Center – Fiber Cut 7/10/14

16

• Key Success –• Cisco CTC indicated a fiber issue• EMC SMARTS alerted of remote site issues

• The redundancy worked as designed and the remote sites failed over with minimum impact (a few users called HD) – Crisis averted

While performing proactive Event/Incident management activities, NetEng engineeridentified an alert on the optical gear and noticed a couple of remote sites had lostconnectivity.

• 12:02pm – CC engineer immediately notified the management of the potential outage and started troubleshooting

• 12:09pm – Further informed management that there is an outage and started collaborating with other engineers in the command center to determine the impact, resolve and restore the service

• 12:11pm – Joined the conference call with management• 12:14pm – Called RCN (vendor) and was informed that there was a fiber cut in

South Boston which is causing the outage• 12:18pm – Informed Helpdesk about the issue and the impact

Cisco Transport Controller Alert in the Command Center started 11:55 AM (visually showed link down)SMARTS Alerts in the Command Center started 11:55 AMJuly 10, 11:55:36 AM NOTIFICATION-SNMPTrap_ccores102.mgh.harvard.edu_clogMsg__LINK-SP__4 SNMPTrap 11:55:36 July 10, 11:57 AM NOTIFICATION-TrunkCable_LINK-PORT-ccores102.mgh.harvard.edu/2<->PORT-ecores102.partners.org/2_DownOrFlapping TrunkCable 11:57:04

EMC SMARTS Network Device alerts started 11:55 AM remained active till 3:34 PM the next day


Event #666 024 476

Q&A

Click on the Q&A panel located in the lower right corner of your screen, type in your questions in the text field and hit send. Please

keep the send to default as “All Panelists.”


Event #666 024 476

Thank You!

• Thanks to our featured speaker: Scott MacLean!

• You will receive an email when our archive recording is ready. (Separate registration is required)

• CHIME CHCIO Credits – Attending our Webinars = 1 CEU

• Questions/Comments – Anthony Guerra [email protected]

Go to www.healthsystemCIO.com/webinars to view our upcoming schedule and see the last 12 months of archived events.

mailto:[email protected]

http://www.healthsystemcio.com/webinars

Documents

Getting Your Network Operations In Orderhealthsystemcio.com/presentation/networkops-webinar.pdfWebex Support 1-866-229-3239 Event #666 024 476 Fall 2011 •Mandate to consolidate data