Survive Splitting an EHR into Private and Public Entities

March 3, 2016 — 2:30PM - 3:30PM PT

Marcus Hobgood, Chief Information Officer, University Health System

Marcus Hobgood Has no real or apparent conflicts of interest to report. Marcus Hobgood Chief Information Officer University Health System marcus.hobgood@uhssystem.com https://www.linkedin.com/pub/marcus-hobgood/b6/304/aa1

Marcus D. Hobgood, is University Health System Chief Information Officer and former Chief Information Officer of LSU Health Science Center Shreveport. As the state director of the LSU Health electronic health record project, Mr. Hobgood led LSU Health to became the first state run public health system to successfully implement a fully electronic health record system. Mr. Hobgood has over 15 years in healthcare information technology experience, is a member of the American College of HealthCare Executives, a HIMSS Analytics member, the Gartner CIO program, and a member of the Cyber Information Technology Advisory Board.

Conflict of Interest

• Introduction to University Health • Challenge presented by privatization of the LSU System • University Health’s mission • DRIVE Cost out of EHR platform (Initiative and Vision) • Summary Post Privatization

Agenda

• Describe how to successfully manage an organization through a major bifurcation of two hospitals

• Discuss how to create operational efficiency and EHR responsiveness despite a major transformation

• Identify steps needed to ensure improved performance after a major transformation

Learning Objectives

University Health System’s goals were to: 1. realize savings 2. increase satisfaction 3. expand patient engagement

http://www.himss.org/ValueSuite

Goals

Dividing One Epic System Serving Five Hospitals into Two Epic Systems Serving Two Hospitals Each

Mission: Impossible

University Health System is a northern Louisiana healthcare network, consisting of two hospitals, University Health Shreveport and University Health Conway in Monroe. Specialty Services and Centers of Excellence: Level 1 Trauma Center Regional Burn Center Feist-Weiller Cancer Center St. Jude Affiliate Clinic Children’s Hospital Center for Excellence for Arthritis and Rheumatology Facts:

Shreveport Conway

• Licensed Beds 459 247

• Inpatient Admissions 20,902 6,648

• Births 1,501 949

• Emergency Visits 60,425 35,618

• Outpatient Visits 362,308 105,662

• Medical Staff 307 45

• Residents 539

University Health System

University Health was once a part of the LSU Health state hospital system that included 10 hospitals. From Shreveport to New Orleans, our charity hospitals have provided care for its citizens for over 200 years. Of the 10 hospitals, 5 joined together as the first state hospital system to implement Epic for EHR. The first “go live” was in November 2011 and the last in March 2013.

University Health History

• Due to deep cuts in the state budget, our governor made the decision to privatize the Louisiana public hospital system.

• This made it possible for the hospitals to continue providing patient services.

Mandatory Privatization

Our Mission

• One EHR, 10 hospitals • Started with the Shreveport hospital, Nov 2011, 5th one done 2013 • Change of plans. State wanted no hospitals, started looking for organizations interested in buying • 2014 started plan to split Epic. Children’s bought NO, BRFF bought EAC/SHV, HPL closed and LAK still state • Completed in May 2015.

Our Mission

• The plan was new compute, SAN and Virtualization • One team was working on deciding hardware and getting ready to start GenIO testing on new SAN • My cache dba team, Epic TC(Sarah) and server TS(Jeff) and i were coming up with a plan on how

we could do this split. • Epic doesn’t have a lot of “split” project plans laying around so we did took a normal GoLive project

plan and tried to pick out the items that we knew we would have to do. • This was our existing setup that was supporting the 5 hospitals for the state(LSU). • We also have REL,RELBLD,VLD for upgrade purposes. • We had to take this and create two completely separate instances.

Current Environment Strategy

• On the left side, was our existing IBM compute and SAN plus UCS blades for NonProd • On the right side, was our new Dell compute and SAN plus the UCS blades for Nonprod • So to start, we created RedHat VMs for our new Prod and TST. Then we took backups and restored to

create UH PRD and UH TST • We decided that rather than build the new UH POC and have the analysts build there yet, we continued

to have the analysts build in LSU POC, build for the current live Epic would go on to PRD and be auto DC’d to UH TST. New build was manually DC’d to LSU TST then UH TST

• Created MST from LSU MST, Ace envs created later

Technical Practice Migration

For cutover Stop shadowing, finish build for UH PRD, Stop the DC from LSU TST TO UH TST

Cutover / Go-Live

• Started May 2014 with hardware discussions • Create a new UH(University Health)domain • Created new VM datacenter • Created new Citrix Farm • All UH users were create new domain IDs • Their EMPs in live Epic were flipped to these new IDs

Timeline

• Created all NonProd Epic servers, Xenapp/Hyperspace,BLOB,CareEverywhere…

• Jan 13th – had Epic NonProd build trip • At this point we had TST

Timeline - continued

Since our UH Prod env was a shadow of LSU env, we couldn’t connect the Windows client servers except during a cutover practice. The shadow was brought down, env brought live and testing would happen. Feb 25th had the production build trip which included Client, server and interface TS. Configurations on production windows servers were completed. In Mar tested document scanning, barcode scanning and printing April 13 our 3r and final cutover test. Went from taking a day ½ to run clinical scripts to remove LSU inpatients, down to a little over 4 hours After cutover, we changed our test interfaces to point to UH PRD and completed full integrated testing. Since our production interfaces were still live on the LSU instance, we couldn’t test with them.

Timeline - continued

Most interfaces were already divided between the LSU OL engine and the UH Corepoint engine but all connections to Epic flowed thru the CP engine. Had to change Ips for any interfaces going to the new UH production Since we were a new company, we got new AIP number. So all our interfaces went from 747 prefix to 582. Plus we renamed them from LSU_ to UH_ Handful of interfaces that had to be created, Surescripts, MdStaff, GE RIS/Pacs(same interfaces but new RIS/PACs) For the Ris/Pacs there were new servers and thus new Ips. Interfaces had to be connected to the new servers and then there was full integrated testing.

Interface Split

• These are just the highlights of the steps performed during each cutover and then the final GoLive.

• Along with the Epic split, we were also splitting off and bringing our own instance of GE RIS/PACS. This started at the same time.

• Stop shadowing

• Build UH PRD

• Disable UH employees access to the LSU Epic, LSU employees can continue to work

• Sarah was doing some configuration of new env, plus licenses

• Started a script to discharge all LSU inpatients – 4 hours

Cutover Steps

• Currently, our Production, Reporting, and Support instances are hosted on 2 Dell r920s running Red Hat 6.5 and VMWare ESXi v 5.5. The non-prod environments are living on Cisco UCS hosts. For storage we are using Dell Compellent SC8000 w/ all flash array drives. When we first implemented Epic in 2011, Intel was not supported. We went live with IBM Power 770/780 physical servers running AIX 6.1 w/ IBM DS 8300 15K storage. It worked well for the most part but we did see some I/O performance issues.

• We were the first Epic customer who attempted a simultaneous migration of both servers and storage, going from physical to virtual, AIX to Red Hat Linux as well as splitting the patient database all at one time. Both Epic and Dell had a huge stake in the success of this undertaking and were there throughout the project, providing support on every level, both remotely and onsite.

Simultaneous Migration

Pre-Migration Infrastructure

This is the documentation we used – most of which is available on Epic’s UserWeb.

• Preliminary Hardware Configuration Guide (created for our organization)

• Cache on Linux on VMware Architecture

• Operational Database Storage Layout Recommendations

• Epic’s Operating System Settings and Security Setup and Support Guide

• Cache 2013.1 Install and Upgrade Setup and Support Guide

• Dell server and storage migration checklists

Documentation

This is a diagram of our current layout. We had an excellent support team from both Epic and Dell. The Dell team made several trips to our site, guiding us through the initial install, testing the servers and storage to make sure that they met the performance criteria specified by Epic, as well as providing cutover support. Jeff Karle, our Epic TS was also immensely helpful and both were available day and night to help us no matter how big or small the issue.

Post-Migration Infrastructure

• The VM’s were designed around the environment functionality. The Production VM lives on a dedicated Dell host. We are using VMWare clustering for high availability (? Ask Huey). Reporting and Support live on the 2nd Dell host as separate VMs. We grouped the Development – POC and TST - together on the same VM on a Cisco UCS host. The Training instances – MST, REF, PREP, ACE, and PLY - are also grouped together on the same VM on a Cisco UCS host.

• Release Engineering - REL, RELBLD, and VLD - will be built as we get closer to our next major upgrade.

Post-Migration Flow

How Did We Get There?

• Created Gold Image VM • Created VM from Gold Image for Gen IO Testing • Determined Cache Migration method • Created UH TST environment by copying LSU TST • Created UH PRD environment by copying LSU PRD • Cutover Testing

• Before the new Dell server and storage hardware could be setup for the migration, we had to set up a VM on the new Dell host and test it. We followed Epic’s recommendations in the Operational Database Storage Layout document. Epic's Gen-IO tool was used to identify configuration and hardware issues. The critical factors that had to be reviewed were:

• Average write latency < 1ms

• Average random read latency < 15ms

• 99% of reads < 60ms

• 99.9% < 200ms

• 99.99% < 600ms

• Write cycle needs to be less than 80s

Gen IO Testing

Gen IO Test Results

After the faulty host bus adapter was replaced, our gen i/o test was successful and we given the green light to proceed with building the new production server.

Gen IO Test Results

Jason Jones Has no real or apparent conflicts of interest to report. Jason Jones National Director of Healthcare Strategy Dell Healthcare – Enterprise Solutions Group Jason_Jones1@DELL.com https://www.linkedin.com/in/jason-jones-3a35332b

Jason Jones is the National Director of Healthcare Strategy with Dell Healthcare and Life Sciences. He has implemented and managed an EHR in a clinical setting and currently works with leading clinical, academic, research and biotechnology organizations to integrate and implement healthcare IT (HIT) enterprise platforms. As the lead for Dell’s enterprise healthcare strategy practice, he is working to address the cost and complexity of HIT and ongoing management and maintenance in the data center.

Conflict of Interest

Cutover Testing

Cutover Data Migration

SUP Refresh

Physical I/O

Grefs/s

Spindles vs All-Flash Array

• For once printers were not the issues

• Because we created our new AD accounts and changed our Epic EMPs over to these prior to the split, all those issues were already taken care of by the time of the split.

• With the icons divided by locations prior to the split, we were able to block the UH users from access to the LSU Epic and allow LSU to continue working.

• Make sure you Order, receive and install all hardware long before the split. We were still setting up secondary SAN the day before.

• Some workstations were still pointing to the LSU Citrix URL for icons, Physicians were still logging into desktop with the LSU ID but logging into Epic with UH ID

• When you bring a shadow env live and interfaces were set to autostart on the original env, they will start up before you get connections straight. Live outgoing interfaces pointed to the production engine, can connect to the new environment. Thus sending orders and such to a non-production env.

• So what happens when you have a room full of geeks, hyped up on caffeine, who just pulled off a successful project? They start thinking again.

• Without testing, the tech team decided to do a Veeam backup of the live primary SAN for Epic. Veeam uses VMware native snapshots and there is a problem when attempting a snapshot on large systems. The IO latency gets so high VMware disconnects the disks which caused Prod to go down again.

• Then a little later that day, they decided to swap the live data from Prod over to the secondary SAN. Unfortunately there was a step missing on this process and is caused the storage to disappear from Prod. Never test on a live production.

Of all the exciting things we did with the split, this is the best. With the new hardware, compute and SAN, we brought our Response Time Exceptions percentages down from Around 1.20% to under .25%. Our users should be happier.

Response Times

Delivering Innovation in EHR

Engaging connections among people, their data and care community empower health ownership

Diagnosis based on individual genome for a targeted treatment plan

Seamlessly connect patients, clinicians and data for holistic proactive care across settings

Powerful & Trusted Computing, Storage and Communications Advances from Intel

Make It Personal! Consumer Health Life Sciences Healthcare IT & Medical Devices

Intel Health & Life Sciences | Make it Personal

A Vision for Healthcare

One architecture across connected devices

Enabling a Secure, High Performance, and Consistent User Experience

Workstation/ Desktop/All in One

2 in 1/ Ultrabook* Intelligent Systems Phone & Handheld Ruggedized Tablet Wearable /

Internet of Things

*Ultrabook™ products are offered in multiple models. Some models may not be available in your market. Consult your Ultrabook™ manufacturer.

Intel Health & Life Sciences | Make it Personal

Intel in Health & Life Sciences

A summary of how benefits were realized for the value of Health IT

Direct Savings: 40% lower TCO Satisfaction: Dropped response time from 1.20% to .25% Patient engagement: Expanded patients use of portal technology

http://www.himss.org/ValueSuite

Questions

• INFORMATION IN THIS DOCUMENT IS PROVIDED IN CONNECTION WITH INTEL PRODUCTS. NO LICENSE, EXPRESS OR IMPLIED, BY ESTOPPEL OR OTHERWISE, TO ANY INTELLECTUAL PROPERTY RIGHTS IS GRANTED BY THIS DOCUMENT. EXCEPT AS PROVIDED IN INTEL'S TERMS AND CONDITIONS OF SALE FOR SUCH PRODUCTS, INTEL ASSUMES NO LIABILITY WHATSOEVER AND INTEL DISCLAIMS ANY EXPRESS OR IMPLIED WARRANTY, RELATING TO SALE AND/OR USE OF INTEL PRODUCTS INCLUDING LIABILITY OR WARRANTIES RELATING TO FITNESS FOR A PARTICULAR PURPOSE, MERCHANTABILITY, OR INFRINGEMENT OF ANY PATENT, COPYRIGHT OR OTHER INTELLECTUAL PROPERTY RIGHT.

• UNLESS OTHERWISE AGREED IN WRITING BY INTEL, THE INTEL PRODUCTS ARE NOT DESIGNED NOR INTENDED FOR ANY APPLICATION IN WHICH THE FAILURE OF THE INTEL PRODUCT COULD CREATE A SITUATION WHERE PERSONAL INJURY OR DEATH MAY OCCUR.

• Intel may make changes to specifications and product descriptions at any time, without notice. Designers must not rely on the absence or characteristics of any features or instructions marked "reserved" or "undefined." Intel reserves these for future definition and shall have no responsibility whatsoever for conflicts or incompatibilities arising from future changes to them. The information here is subject to change without notice. Do not finalize a design with this information.

• Approved use: NDA only

• Do not post publicly

• In countries where restrictions on competitive comparison materials exists – PRC, Russia, Belarus, Kazakhstan, Ukraine, Vietnam – the material must be shown under NDA only with no leave behind.

• Intel, the Intel logo, Intel Atom, Intel Core, Intel. Experience What’s Inside, the Intel. Experience What’s Inside logo, Intel RealSense, Intel Unite, Intel vPro, Intel Xeon Phi, Ultrabook and Xeon are trademarks of Intel Corporation in the U.S. and/or other countries.

• Copyright © 2015, Intel Corporation. All rights reserved.

• *Other names and brands may be claimed as the property of others.

Legal Disclaimers