EHR Usability Test Report for EpicCare Inpatient EHR Suite · 2017-05-08 · EHR Usability Test Report for EpicCare Inpatient EHR Suite ... introduced the test and instructed participants

© 2017 Epic Systems Corporation.

1

EHR Usability Test Report

for EpicCare Inpatient

EHR Suite Epic 2017

Report based on NISTIR 7742 Customized Common Industry Format Template for Electronic Health Record Usability Testing

Dates of Usability Study: January, 2017

Date of Report: March 28, 2017

Report Prepared by: Epic

________________________________________________________________________________________________

© 2017 Epic Systems Corporation 2

Table of Contents

1 Executive Summary .........................................................................................................................................4

2 Introduction .......................................................................................................................................................5

3 Method.................................................................................................................................................................6

3.1 Participants................................................................................................................ 6

3.2 Study Design ........................................................................................................... 10

3.3 Tasks ....................................................................................................................... 11

3.4 Procedures............................................................................................................... 18

3.5 Test Location ........................................................................................................... 18

3.6 Test Environment .................................................................................................... 18

3.7 Test Forms and Tools ............................................................................................... 19

3.8 Participant Instructions ............................................................................................ 19

3.9 Usability Metrics...................................................................................................... 19

4 Results ............................................................................................................................................................... 21

4.1 §170.315(a)(9) Clinical Decision Support ................................................................... 21

4.2 §170.315(b)(2) Clinical Information Reconciliation and Incorporation ........................ 23

4.3 §170.315(a)(1) Computerized Provider Order Entry - Medications............................. 25

4.4 §170.315(a)(2) Computerized Provider Order Entry - Laboratory .............................. 27

4.5 §170.315(a)(3) Computerized Provider Order Entry – Diagnostic Imaging ................. 29

4.6 §170.315(a)(5) Demographics.................................................................................... 31

4.7 §170.315(a)(4) Drug-Drug, Drug-Allergy Interaction Checks ..................................... 33

4.8 §170.315(b)(3) Electronic Prescribing......................................................................... 35

4.9 §170.315(a)(14) Implantable Device List .................................................................... 37

4.10 §170 315(a)(8) Medication Allergy List .................................................................... 39

4.11 §170 315(a)(7) Medication List ................................................................................ 41

4.12 §170.315(a)(6) Problem List ..................................................................................... 43

5 Results Conclusion....................................................................................................................................... 45

Appendices .......................................................................................................................................................... 46

Appendix 1 ................................................................................................................... 46

Appendix 2 ................................................................................................................... 51

Appendix 3 ................................................................................................................... 52

Appendix 4 ................................................................................................................... 53

________________________________________________________________________________________________


Appendix 5 ................................................................................................................... 57

Appendix 6 ................................................................................................................... 58

Appendix 7 ................................................................................................................... 60

________________________________________________________________________________________________


1 Executive Summary Epic staff conducted a usability study of the Epic 2017 version of the EpicCare Inpatient EHR Suite1 in January

2017 at multiple healthcare organizations. The purpose of this study was to evaluate the usability of the user

interface and provide quantitative analysis of the usability of EpicCare Inpatient. During the usability test, 72

healthcare providers used EpicCare Inpatient in simulated, representative tasks. Each task was analyzed for

risk using the methods detailed in section 3.3.

This study collected performance data on various tasks typically conducted by physicians and nurses. The tasks correspond to certification criteria identified in 45 CFR Part 170 Subpart C of the Health Information

Technology: 2015 Edition Health Information Technology (Health IT) Certification Criteria, 2015 Edition Base Electronic

Health Record (EHR) Definition, and ONC Health IT Certification Program Modifications. For information about the

overall User-Centered Design (UCD) process at Epic, see the User Centered Design document as submitted

for 9.1.4.

Participants in the usability study had experience with a prior version of EpicCare Inpatient. The moderators

introduced the test and instructed participants to complete a series of tasks, given one at a time, using EpicCare

Inpatient, following the testing procedures outlined in section 3.4. After each task, the participants were asked

to complete an Ease of Task Completion rating for that task. The participants’ interactions with the screen,

facial expressions, and verbalizations were recorded electronically during the test and later analyzed to collect

time measurements and evaluate performance. The moderators did not assist the participants in completing

the tasks. All participant data was de-identified.

In accordance with the examples in the NIST 7742 Customized Common Industry Format Template for Electronic

Health Record Usability Testing, various recommended metrics were used to evaluate the usability of the

software. The following quantitative metrics were collected for each eligible participant:

Task completion

Time to complete each task

Number and type of unnecessary steps

Number and type of extra steps

Participant’s Ease of Task Completion ratings

System Usability Scale2 score

The System Usability Scale measures the subjective satisfaction with the system, based on the usability tasks

performed. The SUS scores for this study are 85.94 for inpatient physicians and 87.41 for inpatient nurses,

which represent an above average satisfaction rating (see footnote for score interpretation information). 3

In addition to the performance data, the following qualitative observations were made:

Post-test debrief comments

Major findings

Areas for improvement

1 ONC Health IT Certification (for Meaningful Use) information including pricing and limitations is available here: http://www.epic.com/Docs/MUCertification.pdf. 2 See Tullis, T. & Albert, W. (2008). Measuring the User Experience. Burlington, MA: Morgan Kaufman (p. 149). 3 See Tullis, T. & Albert, W. (2008). Measuring the User Experience. Burlington, MA: Morgan Kaufman (p. 149). Generally, scores under 60 represent systems with poor usability; scores over 80 would be considered above average.

http://www.epic.com/Docs/MUCertification.pdf

________________________________________________________________________________________________


2 Introduction The usability study was conducted on the Epic 2017 version of the EpicCare Inpatient EHR Suite, which

facilitates inpatient care workflows and presents healthcare providers in an inpatient setting with integrated

medical information documented in a single patient chart. This usability study included realistic scenarios

typically encountered by clinicians who use the EpicCare Inpatient EHR Suite. Scenarios were constructed in

collaboration with clinicians to ensure clinical accuracy.

The purpose of the usability study was to validate and provide quantitative evidence of the usability of the

EpicCare Inpatient EHR Suite. Accordingly, the testing data measured efficiency, effectiveness, and user

satisfaction through a collection of metrics including time spent on each task, extra and unnecessary steps

taken per task, and Ease of Task Completion ratings from participants.

For the purpose of reporting findings from this usability study, commonly used terms are defined as follows:

Participant: A clinician who has experience with a prior version of the EpicCare EHR Inpatient Suite,

is eligible for participation in the usability study as determined by the Recruiting Screener (see

Appendix 1), and has completed the usability test

Scenario: A patient synopsis, given to participants to provide clinical context for tasks

Task: A verbal and written clinical workflow that is provided to all participants in the usability study

and has a predefined desired outcome

Subtask: The portion of a task relating to a specific criterion for which data is analyzed

Test: The compilation of tasks specific to studied criteria given in a single sitting to a participant

Path: A series of actions that can be taken in the EpicCare Inpatient EHR Suite to reach an outcome

The study was performed on the Epic 2017 version of the EpicCare Inpatient EHR Suite, which includes the

following products:

EpicCare Inpatient

ASAP

MyChart

Care Everywhere

Reporting tools such as Reporting Workbench and Radar

E-Prescribing Interfaces (Outgoing Medication Orders to Retail Pharmacies, Incoming Refill Requests

from Retail Pharmacies, Outgoing Medication Dispense History Query)

Vaccination Interfaces (Outgoing Vaccination Administration, Outgoing Vaccination History Query)

Incoming QRDA Documents Interface

FHIR Services

________________________________________________________________________________________________


3 Method

3.1 Participants

A total of 36 inpatient physicians and 36 inpatient nurses participated in the study. Physicians were asked to

complete 15 tasks and nurses had 10 tasks. Participants were recruited by leaders at their organizations and

were not compensated by Epic for their time. The EpicCare Inpatient EHR Suite is designed to accommodate

physician and nurse specialists, therapists, and other specialized care providers, in addition to general

medicine providers, in an inpatient setting. The participants were actual users of a previous version of

EpicCare Inpatient. Participants were not directly connected to the development of EpicCare Inpatient, nor

were they employed by Epic.

Participants completed a recruiting screener that was used to gather demographic data and to verify eligibility

for participation in the study (see Appendix 1 for the Recruiting Screener used for this study). Recruited

participants represented a mix of backgrounds and demographic characteristics. Participants were assigned

an alphanumeric participant identifier at the time of the usability test to de-identify results.

Of the 46 physicians who were given the Recruiting Screener, 36 qualified to participate in the usability study.

Of the 58 nurses who were given the Recruiting Screener, 36 qualified to participate in the usability study. The

most common reason for ineligibility was that the participant did not provide patient care (see Appendix 1 for

a full list of eligibility criteria). Participants were scheduled for individual 30-minute testing sessions.

Demographic data4 is listed in Tables 1-2 and summarized in Appendix 3.

Table 1: Participant Demographics - Inpatient Physicians

ID Gender Role Education Leve l Epic User

Experience (Years)

Specialty (as reported by

user)

Age First

Language English?

Experience

in Fie ld (Years)

Computer

Use (Years)

1 A19 M Physician Doctorate(MD, DO,

PhD, DNP)

3 Surgery 50-59 Yes 26 20

2 A21 F Physician Doctorate (MD, DO,

PhD, DNP)

10 Obste trics and

Gynecology

30-39 Yes 5 20

3 A23 M Physician Doctorate (MD, DO,

PhD, DNP)

4 Pediatrics 30-39 Yes 4 15

4 A117 M Physician Doctorate (MD, DO,

PhD, DNP)

0.5 Internal Medicine 30-39 Yes 5 10

5 B05 M CEO,

CMIO, CIO,

e tc.

Doctorate (MD, DO,

PhD, DNP)

3.5 Pediatric Cardiac

Intensive Care

40-49 Yes 12 30

4 Note for readers referencing the “open data” CHPL: Experience in Field and Computer Use were originally gathered as ranges, w ith specific values

obtained later once “open data” CHPL specifications were re leased. Average values were applied to these ranges for participants with whom Epic was

not able to establish a second contact.

________________________________________________________________________________________________



Experience (Years)


user)

Age First

Language English?

Experience

in Fie ld (Years)

Computer

Use (Years)

6 B06 M Physician,

Director

Doctorate (MD, DO,

PhD, DNP)

4.5 Psychiatry 50-59 Yes 19 20

7 B10 M CMIO Doctorate (MD, DO,

PhD, DNP)

3 Family Medicine 40-49 Yes 10 25

8 B12 M Physician;

Associate

CMIO;

Information

Technology;

SVP, AVP, VP, e tc.

Doctorate (MD, DO,

PhD, DNP)

15 Internal Medicine 50-59 No 22 26

9 B101 F Physician Doctorate (MD, DO,

PhD, DNP)

6 Geriatrics 50-59 Yes 18 12

10 C05 M Physician Doctorate (MD, DO,

PhD, DNP)

9 Pulmonary, Critical

Care

30-39 Yes 6 30

11 C06 M Physician Doctorate (MD, DO,

PhD, DNP)

8 Pediatric Intensive Care 40-49 Yes 10 20

12 E04 F Physician Doctorate (MD, DO,

PhD, DNP)

5 Pediatrics 30-39 Yes 7 20

13 E10 M Physician Doctorate (MD, DO,

PhD, DNP)

8 Colorectal Surgery 30-39 Yes 8 18


PhD, DNP)

10 Family Medicine 40-49 Yes 13 30


PhD, DNP)

9 Infectious Diseases 30-39 Yes 3 18


PhD, DNP)

9 Physical Medicine and

Rehabilitation

30-39 Yes 5 20


PhD, DNP)

13 Hospitalist 50-59 Yes 12 25


PhD, DNP)

11 Internal Medicine,

Pediatrics

40-49 Yes 8 35


PhD, DNP)



PhD, DNP)

14 Internal Medicine,

Palliative Care

50-59 No 27 25


PhD, DNP)

6 Internal Medicine 30-39 Yes 4 25


PhD, DNP)

3 Family Physician,

Hospitalist

30-39 Yes 9 30

________________________________________________________________________________________________



Experience (Years)


user)

Age First

Language English?

Experience

in Fie ld (Years)

Computer

Use (Years)


PhD, DNP)

9 Cardiology 30-39 Yes 5 30

24 F107 M Physician Doctorate (MD, DO,

PhD, DNP)

6 ICU 50-59 Yes 20 20


PhD, DNP)

1.5 Family Medicine,

Hospitalist

60-69 Yes 30 39


PhD, DNP)


27 F114 F Physician Doctorate (MD, DO,

PhD, DNP)

8 Pediatric Internal

Medicine

30-39 Yes 8 16

28 G02 F Physician Doctorate (MD, DO,

PhD, DNP)

7 Hematology, Oncology 40-49 Yes 11 20


PhD, DNP)

6 Pediatric Hospitalist 30-39 Yes 5 20


PhD, DNP)

4.5 Pediatric Hospitalist 30-39 Yes 7 20

31 G10 M Physician Doctorate (MD, DO,

PhD, DNP)

4.5 Pediatrics 30-39 Yes 4.5 20


PhD, DNP)

5 Pediatric Physical

Medicine and

Rehabilitation

30-39 Yes 8 20


PhD, DNP)

2.5 Infectious Diseases 40-49 No 6 16

34 G111 M Director,

Information

Technology,

Physician

Doctorate (MD, DO,

PhD, DNP)

10 Internal Medicine 40-49 Yes 7 25


PhD, DNP)

9 Obste trics and

Gynecology

30-39 Yes 9 20


PhD, DNP)

5 Geriatrics 50-59 Yes 5 25

________________________________________________________________________________________________


Table 2: Participant Demographics - Inpatient Nurses


Experience

(Years)

Specialty (as reported

by user)

Age First

Language

English?

Experience

in Fie ld

(Years)

Computer

Use

(Years)

1 A05 F Nurse Bache lor's Degree 4 Adult Acute and

Intermediate

Medicine

30-39 Yes 4 18

2 A33 M Nurse ,

Director

Bache lor's Degree 2 Psychiatric Nursing 60-69 Yes 11 41

3 A39 F Nurse Bache lor's Degree 1.5 OR 40-49 No 2 15

4 A47 F Nurse Associate Degree 1 Direct Observation

Unit

40-49 No 6 15

5 A108 F Nurse Doctorate (MD,

DO, PhD, DNP)

3 Vascular, Plastics,

Orthopedics, Trauma

60-69 Yes 30 30

6 A109 F Nurse Bache lor's Degree 4 ICU 40-49 Yes 20 25

7 A112 F Nurse Bache lor's Degree 6 OR 50-59 Yes 36 25

8 A115 F Nurse Bache lor's Degree 4 Neurology 60-69 Yes 14 20

9 A116 F Nurse Master's degree

(MSN, MS)

4.5 Diabetes 50-59 Yes 18 35

10 A118 M Nurse Bache lor's Degree 1 Critical Care 40-49 Yes 3 22

11 B09 F Nurse Associate Degree 4 PACU 50-59 Yes 24 20

12 B11 F Nurse Bache lor's Degree 4 PACU 60-69 Yes 39 20

13 B13 F Nurse Bache lor's Degree 8 Cardiology, ICU 50-59 Yes 30 24

14 B105 F Nurse Bache lor's Degree 5 Surgical, Urology 30-39 Yes 11 12

15 B106 F Nurse Associate Degree 4 Med Surg 30-39 Yes 4 20

16 B109 M Nurse Associate Degree 5 Gastrointestinal 20-29 No 1.2 5

17 B110 F Nurse Associate Degree 2 Med Surg 20-29 Yes 5 15

18 B111 F Nurse Associate Degree 4 OB, Labor and

Delivery

50-59 Yes 29 10

19 B112 F Nurse Associate Degree 5 Med Surg, Renal 40-49 Yes 10 20

20 B113 M Nurse Associate Degree 4 Critical Care 20-29 Yes 2.5 10

21 B114 M Clinical

Nurse

Manager

Bache lor's Degree 5 Critical Care 20-29 Yes 6 20

22 E303 F Nurse Bache lor's Degree 6 Cardiology 20-29 Yes 6 20

________________________________________________________________________________________________



Experience (Years)

Specialty (as reported

by user)

Age First

Language English?

Experience

in Fie ld (Years)

Computer

Use (Years)

23 E304 F Nurse Bache lor's Degree 10 PACU 40-49 Yes 23 10

24 E308 F Nurse Bache lor's Degree 8 Trauma 40-49 Yes 8 20

25 E309 F Nurse Bache lor's Degree 15 Post-Partum,

Newborn Nursery

50-59 Yes 35 30

26 E310 F Nurse Bache lor's Degree 10 Labor and Delivery 30-39 Yes 16 20

27 E311 F Nurse Bache lor's Degree 12 Surgery, Trauma 30-39 Yes 12 20

28 E312 M Nurse Bache lor's Degree 15 Not reported 60-69 Yes 28 15

29 E314 F Nurse Bache lor's Degree 5 Med Surg 50-59 No 14 10

30 E315 F Nurse Bache lor's Degree 10 Med Surg 40-49 No 20 30

31 E316 F Nurse Bache lor's Degree 2 MICU 30-39 Yes 2 20

32 E318 F Nurse Associate Degree 10 Labor and Delivery 40-49 Yes 15 20

33 E404 F Nurse Master's degree

(MSN, MS)

5 Not reported 30-39 Yes Not

reported

Not

reported

34 E507 F Nurse Bache lor's Degree 4 ICU 40-49 Yes 12 20

35 F101 F Nurse Bache lor's Degree 7 PACU 50-59 Yes 39 30

36 F105 F Nurse Bache lor's Degree 5 Rehab 20-29 Yes 2 20

3.2 Study Design

The objective of this study was to demonstrate areas where the application suite performed well – that is,

effectively, efficiently, and satisfactorily – and identify areas where improvements can be made.

Participants interacted with the Epic 2017 version of the EpicCare Inpatient EHR Suite. Each participant used

the system in a designated location, usually a conference or training room at the site where the participant is

employed. All participants were provided with the same instructions by the test moderator. The system was

evaluated for effectiveness, efficiency, and satisfaction as defined by metrics collected and analyzed for each

participant:

Task completion

Time to complete each task

Number and type of unnecessary steps

Number and type of extra steps

Participant’s Ease of Task Completion ratings

System Usability Scale score

For additional information on usability metrics, see section 3.9.

________________________________________________________________________________________________


3.3 Tasks

3.3.1 Task List

Tasks were constructed to be realistic and representative of typical activities a user would complete using the

EpicCare Inpatient EHR Suite. The tasks were prioritized and selected through a combination of the risk

analysis framework outlined in section 3.3.2 and the priorities outlined in NISTIR 7804-015.

NISTIR 7804-01 is an industry standard that provides scenarios and guidelines for usability testing of

Electronic Health Records. Tasks for the usability study were constructed to emphasize priorities articulated

in NISTIR 7804-1 (e.g. identification of information, consistency of information, and integrity of information),

and incorporated the NIST test scenarios when applicable.

Tasks are split into subtasks that are measurable components related to criteria supplied by the ONC. See

Appendix 4 for full task wording.

Physician Tasks/Subtasks

Scenario 1: Gertrude is a 55-year-old female who has been admitted to your unit from the ED after

experiencing a fall at home. She is a diabetic patient being treated for dehydration, malnutrition, abrasions,

and a possible concussion.

Task 1: Reconcile a problem from a primary care provider at an outside organization .

§170.315(b)(2) Clinical information reconciliation and incorporation

Task 2: Add a problem to the problem list for a patient admitted from the ED.

§170.315(a)(6) Problem list

Task 3: Add a medication allergy to a patient’s allergy list and assess any interactions.

§170.315(a)(4) Drug-drug, drug-allergy interaction checks

§170.315(a)(8) Medication allergy list

Task 4: Modify an order for a diagnostic imaging procedure.

§170.315(a)(3) Computerized provider order entry – Diagnostic imaging

Scenario 2: Sheryl is a 68-year-old female who is recovering from a recent knee replacement and is currently

admitted to your unit.

Task 5: Review the patient’s problem list.


Task 6: Modify the details of a patient-controlled analgesic medication order.

§170.315(a)(1) Computerized provider order entry – Medications

Task 7: Modify the details of an inpatient laboratory order.

5 NISTIR 7804-1 Technical Evaluation, Testing, and Validation of the Usability of Electronic Health Records: Empirically Based Use Cases for Validating Safety-Enhanced Usability Guidelines for

Standardization

________________________________________________________________________________________________


§170.315(a)(2) Computerized provider order entry – Laboratory

Task 8: Respond to the system’s clinical decision support and place a vaccine order based on a quality

improvement initiative.

§170.315(a)(8) Clinical decision support

Task 9: Order an outpatient lab to monitor a discharge medication.

§170.315(a)(2) Computerized provider order entry – Laboratory

Task 10: Electronically prescribe a discharge medication and assess any interactions.


§170.315(b)(3) Electronic prescribing


Scenario 3: Arthur is a 65-year-old male with a history of type 2 diabetes mellitus, hypercholesterolemia, and

bradycardia. He came to the ED complaining of weakness in his right side and was admitted to the ICU for a

CVA.

Task 11: Modify a medication allergy on a patient’s allergy list.

§170.315(a)(8) Medication allergy list

Task 12: Place a medication order suggested by the system’s clinical decision support and assess any

interactions.




Task 13: Order an ultrasound to diagnose symptoms.

§170.315(a)(3) Computerized provider order entry – Diagnostic imaging

Task 14: Resolve a problem on the patient’s problem list.


Task 15: Electronically prescribe a discharge medication.


§170.315(b)(3) Electronic prescribing

Nurse Tasks/Subtasks

Scenario 1: Walter is a 79-year-old male with a complex medical history that includes CHF, osteoporosis,

dementia, hypertension, and hyperlipidemia. He is directly admitted to your hospital for a wound infection.

Task 1: Document the patient’s preferred language.

§170.315(a)(5) Demographics

Task 2: Reconcile allergy information from an outside urgent care facility.


Task 3: Review a complex medication list based on information provided by the patient .

________________________________________________________________________________________________


§170.315(a)(6) Medication list

Task 4: Add an implantable device to the patient’s chart using information provided by the patient.

§170.315(a)(14) Implantable device list

Task 5: Determine if the patient has any active devices on the implantable device list that may be contributing

to symptoms.


Task 6: Indicate that the patient has declined the intervention suggested by the system’s clinical decision

support.


Scenario 2: Robin is a 50-year-old who has been admitted for diabetic ketoacidosis.

Task 7: Update gender identity as reported by the patient.

§170.315(a)(5) Demographics

Task 8: Reconcile allergy information from an outside facility.


Task 9: Add a patient-reported medication to the medication list.

§170.314(a)(7) Medication list

Task 10: Update the information in a patient’s implantable device list.


Task selection was based on criticality of function and the risk analysis described in section s 3.3.2 and 3.3.3

below.

3.3.2 Risk Analysis Framework

Risk assessment for each task involves assessing the Likelihood of Risk Occurrence and the Significance of

Risk Materialization (referred to as Likelihood and Significance, respectively). Overall risk for each task is then

categorized as High, Moderate, Low, or Negligible.

Likelihood is determined by a combination of two factors: Frequency of Workflow and Possibility of

Alternative Outcome. Frequency of Workflow reflects the general prevalence of a specific or closely analogous

workflow. Possibility of Alternative Outcome is an assessment of the likelihood that a variation might occur

during a specific or closely analogous workflow. The way in which Frequency of Workflow and Possibility of

Alternative Outcome contribute to Likelihood is outlined in Appendix 7.

Significance is the measurement of the impact of possible outcomes that result from a variation from an

expected task workflow. Impact of possible outcomes of each task was determined in consult with clinicians

and other subject matter experts.

________________________________________________________________________________________________


Figure 1 lists representative factors used to evaluate Likelihood and Significance. Human and system factors

that affect the evaluation of each component were discerned from conceptual and historical analysis and

include, but are not limited to, those that appear in Figure 1. 6

Fig. 1: Factors which Influence Risk

Likelihood Significance

↑ Factors increasing Frequency of Workflow

Task involves a common sequence of events

↑ Factors increasing Possibility of Alternative Outcome

Clinician must make a decision that requires an increased cognitive load (i.e. the clinician must find information in multiple locations and must synthesize or remember high volumes of information to complete the workflow)

Clinician is limited in ability to recover from an issue in documentation or action

Workflow completes an event or otherwise closes an instance of care to any further documentation

High intrinsic complexity of information or information management

Clinician is likely to experience an interruption during the workflow

↑ Factors increasing Significance

Clinician is likely to take inappropriate action after the issue occurs

Issue affects mostly high-acuity patients, high-risk medication, or urgent/critical workflows

Affected data directly informs significant clinical decisions

↓ Factors decreasing Frequency of Workflow

Task involves an unusual sequence of events

↓ Factors decreasing Possibility of Alternative Outcome

Clinician does not make decisions in the course of the workflow or has robust decision-making support within the workflow

Clinician finds information in one location, or the workflow requires low degree of information synthesis or memory

Clinician has ample opportunity to recover from an issue in documentation or action

Task or workflow occurs in the midst of an event; documentation can easily be edited or added

Low intrinsic complexity of information or information management

Clinician is unlikely to experience an interruption during the workflow

↓ Factors decreasing Significance

Clinician is likely to take appropriate action regardless of the issue

Issue affects mostly low-acuity patients, low-risk medications, or non-urgent/non-critical workflows

Affected data does not inform significant clinical decisions

6 See Beasley, J. W., Wetterneck, R. B., Temte, J., Lapin, J., Smith, P., Rivera-Rodriguez, J., & Karsh, B. (2011). Information Chaos in Primary Care: Implications for Physician Performance and Patient Safety. J Am Board Fam Med. , 24(6), 745-751. doi:10.3122/jabfm.2011.06.100255; Carayon, P. Sociotechnical systems approach to healthcare quality and patient safety. Work, 4(1). doi:10.3233/WOR-2012-0091-38Carayon; Holden, R. J., Carayon, P., Gurses, A. P., Hoonakker, P., Hundt, A. S., Ozok, A., & Rivera-Rodriquez, A. (2013). SEIPS 2.0: A human factors framework for studying and improving the work of healthcare professionals and patients. Ergonomics, 53(11). doi:10.1080/00140139.2013.838643; Meeks, D. W., Takian, A., Sittig, D. F., Singh, H., & Barber, N. (2014). Exploring the sociotechnical intersection of patient safety and electronic health record implementation. Journal of the American Medical Informatics Association, 21, 2834. doi:10.1136/amiajnl-2013-001762. Epub 2013 Sep 19; O'Hara, R.,

& Et al. (2014). A qualitative study of decision-making and safety in ambulance service transitions. Health Services and Delivery Research, 2(56). doi:10.3310/hsdr02560; Vincent, C., Taylor-Adams, S., & Stanhope, N. (1995). Framework for Analysing Risk and Safety in Clinical Medicine. BMJ: British Medical Journal, 316(7138), 1154-1157; Wogalter, M. S., & Laughery, K. R. (1996). WARNING! Sign and Label Effectiveness. Current Directions in Psychological Science. doi:10.1111_1467-8721.ep10772712

________________________________________________________________________________________________


Figure 2 illustrates how Likelihood and Significance contribute to overall risk. Significance is weighted more

heavily than Likelihood when completing the overall risk assessment for each task. The relationship between

Significance and Likelihood was created in consult with literature on risk evaluation.7

Fig. 2: Overall Risk Assessment

Significance

Negligible Low Moderate High

Like

lih

oo

d

High Negligible Risk Moderate Risk Moderate Risk High Risk

Moderate Negligible Risk Moderate Risk Moderate Risk High Risk

Low Negligible Risk Low Risk Moderate Risk Moderate Risk

Negligible Negligible Risk Low Risk Moderate Risk Moderate Risk

3.3.3 Risk-Based Task Selection

Moderate to high-risk workflows were selected for physician and nurse tasks based on the framework

presented above. Risk analysis of inpatient physician and nurse tasks is summarized in Tables 3 and 4,

respectively.

Table 3: Risk Analysis of Physician Tasks

Physician Task

Criteria Likelihood Significance Risk

1 Reconcile a problem from a primary care provider at an outside organization.

Clinical information reconciliation and incorporation High Moderate Moderate

2 Add a problem to the problem list for a patient admitted from the ED.

Problem List Moderate Moderate Moderate

3 Add a medication allergy to a patient’s allergy list and assess any interactions.

Drug-drug, drug-allergy interaction checks; Medication allergy list

Moderate High High

4 Modify an order for an inpatient diagnostic imaging procedure.

Computerized provider order entry - Diagnostic imaging Moderate Moderate Moderate

7 NASA. (1994). Systems Engineering (EIA/IS-632). Electronic Industries Association (EIA); Ben-Asher, J. Z. (2004). Systems engineering aspects in theatre missile defense? Design principles and a case study. Systems Engineering. doi:10.1002/sys.10058; Ben-Asher, J. Z. (2008). Development Program Risk Assessment Based on Utility Theory. Risk Management, 10(4), 285-299. doi:10.1057/rm.2008.9; Ben-

Asher, J. Z., Zack, J., & Prinz, M. (2000). Development Program Risk Management. AIAA Progress in Aeronautics and Astronautics, 192, 341-351.; Blanchard, B. S., & Fabrycky, W. J. (1981). Systems engineering and analysis. Englewood Cliffs, NJ: Prentice-Hall.; Robertson, T. C. (Ed.). (2000). Systems Engineering Handbook (2000 ed.). INCOSE; Tummala, V. M., & Mak, C. L. (2001). A risk management model for improving operation and maintenance activities in electricity transmission networks. Journal of The Operational Research Society, 52, 125-134. doi:10.1057/palgrave.jors.2601044.

________________________________________________________________________________________________


Physician Task


5 Review the patient’s problem list.

Problem list Moderate Moderate Moderate

6 Modify the details of a patient-controlled analgesic medication order.

Computerized provider order entry - Medications Moderate High High

7 Modify the details of an inpatient laboratory order.

Computerized provider order entry - Laboratory Moderate Low Moderate

8 Respond to the system’s clinical decision support and place a vaccine order based on a quality improvement initiative.

Clinical decision support Moderate Low Moderate

9 Order an outpatient lab to monitor a discharge medication.

Computerized provider order entry – Laboratory Moderate High High

10 Electronically prescribe a discharge medication and assess any interactions.

Computerized provider order entry - Medications; Drug-drug, drug-allergy interaction checks; Electronic prescribing

Moderate High High

11 Modify a medication allergy on a patient’s allergy list.

Medication allergy list Moderate Moderate Moderate

12 Place a medication order suggested by the system’s clinical decision support and assess any interactions.

Clinical decision support; Computerized provider order entry – Medications; Drug-drug, drug-allergy interaction checks

Low Moderate Moderate

13 Order an ultrasound to diagnose symptoms.

Computerized provider order entry - Diagnostic imaging Moderate Moderate Moderate

14 Resolve a problem on the patient’s problem list.

Problem list Moderate Low Moderate

15 Electronically prescribe a discharge medication.

Computerized provider order entry - Medications; Electronic prescribing

Moderate High High

________________________________________________________________________________________________


Table 4: Risk Analysis of Nurse Tasks

Nurse Task


1 Document the patient’s preferred language.

Demographics Moderate Low Moderate

2 Reconcile allergy information from an outside urgent care facility.

Clinical information reconciliation Moderate Moderate Moderate

3 Review a complex medication list based on information provided by the patient.

Medication list Moderate Moderate Moderate

4 Add an implantable device to the patient’s chart using information provided by the patient.

Implantable device l ist Moderate Moderate Moderate

5 Determine if the patient has any active devices on the implantable device list that may be contributing to symptoms.

Implantable device l ist Low Low Low

6 Indicate that the patient has declined the intervention suggested by the system’s clinical decision support.

Clinical decision support Moderate Low Moderate

7 Update gender identity as reported by the patient.

Demographics Moderate Moderate Moderate

8 Reconcile allergy information from an outside facility.

Clinical information reconciliation Moderate Moderate Moderate

9 Add a patient reported medication to the medication list.

Medication list Moderate Moderate Moderate

10 Update the information in a patient’s implantable device list.

Implantable device l ist Moderate Low Moderate

________________________________________________________________________________________________


3.4 Procedures

Moderators were Epic employees with experience in usability testing who underwent training specific to the

type of usability testing being conducted.

Participants arrived at their organization’s testing location where they were greeted by a moderator and

oriented to the testing computer and equipment. Participants were then assigned an alphanumeric participant

identifier in order to de-identify results. Each participant was asked for verbal consent to record the testing

session.

To prepare the participant for testing, the moderator outlined the format of the test and gave general

instructions. The moderator then began recording the session using screen capture, a microphone, and a

portable camera. Before starting the tasks, the moderator showed the participant a brief instructional recording

representative of the training typically given to users before the implementation of new functionality. Pieces

of functionality were included in the instructional recordings when research indicated that not all users were

familiar with the testing setup used, due to organizational variations in configuration. (Epic provides

organizations with a high degree of latitude in setting up the software to meet specific organizational needs.)

Physicians viewed an instructional recording on updated screen layout and clinical information reconciliation.

Nurses viewed an instructional recording on updated screen layout, clinical information reconciliation, and

implantable device list. At this point, the participant was given an opportunity to ask any questions or express

any concerns. The moderator continued to administer general instruction and tasks during the session.

Participants were instructed to perform tasks:

At their normal pace

Without assistance; moderators were allowed to give immaterial guidance and clarification on tasks,

but not instructions on use of the software

Withholding comments until the test was completed

Before each task, the moderator gave participants a written copy of the task and oral instruction. Task timing

began and ended when the participant reached particular predetermined points in the task.

After the participants finished the tasks, they participated in individual debrief sessions. During these sessions,

the moderators solicited feedback from participants on any areas where the participants had ex tra steps,

unnecessary steps, or incomplete tasks or subtasks. The purpose of this session was to gain additional

information about the possible causes of the unnecessary steps, extra steps, or incomplete tasks or subtasks.

3.5 Test Location

Testing was conducted at healthcare organizations where participants were employed. Participants’

organizations provided testing rooms, typically small conference or training rooms. To ensure that the

surroundings were comfortable for participants, noise levels were kept to a minimum and the ambient

temperature kept within a normal range. See Appendix 2 for a table summarizing the testing dates and

locations.

3.6 Test Environment

The EpicCare Inpatient EHR Suite is typically used in an inpatient facility. The testing was conducted at a

variety of participating organizations, in rooms made available for this purpose. Testing workstations were

either a Lenovo T430, Intel Core i5-3320M processor (2.60GHz) with 8 GB RAM or a Lenovo T440P, Intel Core

________________________________________________________________________________________________


i5-4300M processor (2.60GHz) with 16 GB RAM. Workstations of both types used Windows 8 Enterprise x64,

Microsoft Internet Explorer 11 and a 23-inch flat panel display in landscape orientation, 1600 by 900 pixel

resolution, and set to thousands of colors.

The application was locally installed and used an InterSystems Caché 2013.1 database server on a loopback

connection.

The participants used a mouse and keyboard when interacting with the EpicCare Inpatient EHR Suite. The

overall system performance was comparable to what users would experience in a field implementation.

3.7 Test Forms and Tools

During the usability test, the following documents were used:

Recruiting Screener

Moderator Guide

Participant Packet

The participant’s interaction with the EpicCare Inpatient EHR Suite was captured and recorded digitally with

screen capture software running on the test workstation. Each participant’s facial expressions were recorded,

along with onscreen actions and verbal comments. Recordings were saved and used for further analysis.

3.8 Participant Instructions

The moderator read general introductory statements and instructions aloud to the participant before

administering the test. See Appendix 5 for the Sample Participant Orientation script.

The participant was then asked to complete a number of tasks that were read aloud by the moderator and

provided on paper to the participant for reference.

3.9 Usability Metrics

According to the NIST Guide to the Processes Approach for Improving the Usability of Electronic Health Records ,

EHRs should support a process that provides a high level of usability for all users. Th e goal is for users to

interact with the system effectively, efficiently, and with high satisfaction. As such, metrics for these measures

were captured during the usability testing.

The goals of the test were to assess:

1. Efficiency of the EpicCare Inpatient EHR Suite by measuring the average task time and extra steps

2. Effectiveness of the EpicCare Inpatient EHR Suite by measuring task completion rates and unnecessary

steps

3. Satisfaction with the EpicCare Inpatient EHR Suite by measuring Ease of Task Completion ratings

________________________________________________________________________________________________


Data Scoring

Table 5 details how metrics measuring efficiency, effectiveness, and satisfaction were scored.

Table 5: Usability Scoring Metrics

Measures Rationale and Scoring

Efficiency:

Average Task Time

Each task is timed from a predetermined starting point until the participant reaches the predetermined end point for the task or verbalizes completion.

Only task times for tasks that were successfully completed are included in the average task time analysis and standard deviation (reported in parentheses in the table below).

Efficiency:

Average Extra Steps per Task

The number of steps in a participant’s path through the application is recorded and compared to the number of steps in the closest acceptable path. An extra step is recorded if the participant performs a step that is not included in the defined path but is also not counterproductive to completing the task.

The total number of steps taken by a participant is counted and the difference between the steps in their path and the closest defined path is calculated. The average of the differences for the participants is calculated. Only extra steps for tasks that were successfully completed are included in the average extra steps per task analysis and standard deviation.

Effectiveness:

Binary Task Completion Rate

A task is considered a success if the participant achieves the defined task outcome without assistance.

The total number of successes was calculated for each task and then divided by the total number of times that the task was attempted. The results are presented as a percentage.

The task failure percentage can be calculated by subtracting the binary task completion rate from 100.

Effectiveness:

Partial Task Completion Rate

A task is considered 100 percent completed if the participant achieves the defined task outcome without assistance. For participants unable to successfully complete a task, the number of steps completed are counted and divided by the number of steps in the closest defined path to calculate the percentage of the task the participant completed. These results are, in turn, added together and divided by the number of participants who attempted the task to obtain the average partial task completion rate.

Effectiveness:

Average Unnecessary Steps per Task

Unnecessary steps are recorded each time a participant performs an action in the system that is not his intended action. Examples of unnecessary steps include typing mistakes and errant clicks that do not contribute to the completion of the task. The total number of unnecessary steps is calculated and divided by the number of participants to obtain the average number of unnecessary steps committed per participant.

Only unnecessary steps for tasks that were successfully completed are included in the average unnecessary steps per task analysis and standard deviation.

Satisfaction:

Ease of Task Completion Rating

The participant’s subjective impression of the ease of use of the application is recorded for each task. After each task was completed, the moderator asked the participant to rate the task on a 5-point Likert scale: 1 (Very Difficult), 2 (Somewhat Difficult), 3 (Neither Difficult nor Easy), 4 (Somewhat Easy), 5 (Very Easy).

These values are averaged across participants for each task with the calculated standard deviation reported in parentheses in the table in the Data Analysis and Reporting section for each criterion.

________________________________________________________________________________________________


4 Results The usability testing results for the EpicCare Inpatient EHR Suite are detailed below (see Tables 6-17). Results

are organized and analyzed by criteria. The results were calculated according to the methods specified in section

3.9 Usability Metrics.

4.1 §170.315(a)(9) Clinical Decision Support

4.1.1 Data Analysis and Reporting

Table 6: Clinical Decision Support Physician and Nurse Task Results

N=Number of participants

4.1.2 Discussion of the Findings

Clinical decision support testing with inpatient physicians covered two tasks:

Placing a vaccine order based on a quality improvement initiative (Task 8)

Placing a medication order based on an initiative and assessing any interactions (Task 12)

Testing with inpatient nurses covered one task:

Indicating that the patient declines intervention (Task 6)

Efficiency

Nurses completed Task 6 in 8.26 seconds and did so without taking any extra steps. Physicians completed the

tasks in an average of 5.66 seconds for Task 8 and 23.63 seconds for Task 12.

Clinical Decision Support

Efficiency Effectiveness Satisfaction

N Average Task Time


Binary Task Completion

Rate

Partial Task Completion

Rate

Average Unnecessary

Steps per Task

Task Ease Rating

# Mean (seconds)

(SD)

Mean (SD) Mean % (SD) Mean % (SD) Mean (SD) Mean (SD) 5 = very

easy

Physician Subtasks

Placing a vaccine order based on a

quality improvement initiative

35 5.66 (3.16) 0.20 (0.45) 100 (0) 100 (0) 0 (0) 4.86 (0.43)

Placing a medication order based on

an initiative and assessing any interactions

35 23.63

(16.38)

0.34 (0.59) 100 (0) 100 (0) 0 (0) 4.09 (1.07)

Nurse Subtasks

Indicating that the patient declines intervention

35 8.26 (3.62) 0 (0) 100 (0) 100 (0) 0 (0) 4.51 (0.56)

________________________________________________________________________________________________


Effectiveness

The binary task completion rate for physicians is 100 percent for both Tasks 8 and 12. The binary task

completion rate for nurses is 100 percent for Task 6. Both doctors and nurses completed the tasks with no

unnecessary steps.

Satisfaction

The average satisfaction ratings for all tasks are between Somewhat Easy an d Very Easy. The average task ease

ratings for physician Tasks 8 and 12 are 4.86 and 4.09, respectively. The average task ease ratings for nurse

Task 6 is 4.51.

Major Findings

Physicians and nurses completed clinical decision support tasks consistently across various scenarios and

reported that the tasks were between Somewhat Easy and Very Easy. Metrics between physicians and nurses

are similar, suggesting a consistent experience across user groups.

In Tasks 8 and 12, the extra step physicians most frequently took was attempting to select an option that was

already selected by default. This action had no impact on patient care, but increased the average task time

slightly and accounted for the majority of extra steps taken in clinical decision support tasks.

Areas for Improvement

Overall, both physicians and nurses were able to successfully assess the information provided by the system's

clinical decision support and appropriately respond in scenarios where the system suggested a course of

treatment. Based on observation and analysis, the majority of extra steps for clinical decision support tasks

occurred when physicians were uncertain whether a choice was selected by default or needed to be made

manually. Though the outcome of these extra steps taken had no clinical impact, making default selections

more clear by improving the appearance of selected options could further improve clinician efficiency.

________________________________________________________________________________________________


4.2 §170.315(b)(2) Clinical Information Reconciliation and Incorporation


Table 7: Clinical Information Reconciliation and Incorporation Physician and Nurse Task Results


Clinical information reconciliation and incorporation testing with inpatient physicians covered one task:

Reconciling a problem from a primary care provider at an outside organization (Task 1)

Testing with inpatient nurses covered two tasks:

Reconciling allergy information from an outside urgent care facility (Task 2)

Reconciling allergy information from a primary care provider at an outside facility (Task 8)

Efficiency

Thirty-five out of 36 nurses who completed Task 2 did so without any extra steps. Nurses completed Task 8

with a negligible number of extra steps. (Negligible is hereafter defined as within one confidence interval of

zero.) Physicians completed Task 1 with a negligible number of extra steps.

Effectiveness

The binary task completion rate for all clinical information reconciliation tasks is 100 percent. Both nurses

and physicians completed the clinical information reconciliation tasks without taking any unnecessary steps.

Satisfaction

The majority of participants rated the clinical information reconciliation tasks as Very Easy. For physicians,

the average satisfaction score for Task 1 is 4.80. For nurses, the average satisfaction scores for Tasks 2 and 8

are 4.75 and 4.86, respectively.

Clinical Information Reconciliation and

Incorporation


N Average Task Time



Rate


Rate

Average Unnecessary

Steps per Task

Task Ease Rating

# Mean (seconds)

(SD)


easy

Physician Subtasks

Reconciling a problem from an outside organization

35 13.26 (14.83)

0.06 (0.24) 100 (0) 100 (0) 0 (0) 4.80 (0.41)

Nurse Subtasks

Reconciling an allergy from an outside urgent care facility

36 14.33 (17.85)

0.19 (0.44) 100 (0) 100 (0) 0 (0) 4.75 (0.55)

Reconciling an allergy from an outside facility

36 9.83 (5.18) 0.06 (0.24) 100 (0) 100 (0) 0 (0) 4.86 (0.42)

________________________________________________________________________________________________


Major Findings

Between the two user groups, 100 percent of participants successfully completed the full clinical information

reconciliation workflow with an average satisfaction rating of Very Easy. Thirty-three out of 35 physicians

and 33 out of 36 nurses did not take any extra steps. Extra steps taken were to look in other areas of the

patient chart but all participants were able to complete the workflow.


A few participants looked through other areas of the chart to reconcile outside information before pursuing

the task workflow, which accounts for the longer task times. Even with these extra steps, all users were able

to complete the full reconciliation workflow with high task satisfaction ratings. This observation highlights

opportunities to improve the integration and visibility of the clinical information reconciliation functionality:

Optimize references to reconciling outside information alongside internally documented information

Explore the use of inline links to incorporate outside information to provide more direct paths to task

completion

________________________________________________________________________________________________


4.3 §170.315(a)(1) Computerized Provider Order Entry – Medications


Table 8: Computerized Provider Order Entry – Medications Physician Task Results

Computerized Provider Order Entry - Medications


N Average Task Time



Rate


Rate

Average Unnecessary

Steps per Task

Task Ease Rating

# Mean

(seconds) (SD)

Mean (SD) Mean % (SD) Mean % (SD) Mean (SD) Mean (SD)

5 = very easy

Physician Subtasks

Modifying the details of a PCA medication order

36 18.47 (17.61)

0 (0) 100 (0) 100 (0) 0 (0) 4.61 (0.73)

Ordering a discharge medication and assessing any interactions

34 36.47 (24.84)

0.26 (0.51) 100 (0) 100 (0) 0.06 (.24) 4.03 (1.00)

Placing a medication order based on an initiative and assessing any

interactions

35 23.63 (16.38)

0.14 (0.38) 100 (0) 100 (0) 0.03 (0.16) 4.09 (1.07)

Ordering a discharge medication 35 27.37 (13.92)

0.14 (0.38) 100 (0) 100 (0) 0 (0) 4.43 (0.95)


Computerized provider order entry of medications testing with inpatient physicians covered four tasks:

Modifying the details of a PCA medication order (Task 6)

Ordering a discharge medication and assessing any interactions (Task 10: Subtask A)

Placing a medication order based on an initiative and assessing any interactions (Task 12: Subtask B)

Ordering a discharge medication (Task 15: Subtask A)

Efficiency

For discharge medications, the average time taken to complete Task 10—which had a drug interaction for the

participant to assess—was 36.47 seconds. The average time for Task 15—which did not have a drug

interaction—was 27.37 seconds. The average times for Tasks 6 and 12, which involved placing or modifying

an inpatient order, are more consistent. The average time for these tasks ranged from 18.47 seconds to 23.63

seconds.

Effectiveness

All medication ordering tasks had a binary completion rate of 100 percent. Participants took no unnecessary

steps for Tasks 6 and 12, which had participants modify or place an inpatient order. Participants who

completed Tasks 10 and 15 by placing discharge medication orders did not take any unnecessary steps for

Task 15 and took a negligible number of unnecessary steps to complete Task 10.

________________________________________________________________________________________________


Satisfaction

The average satisfaction ratings for all tasks are between Somewhat Easy and Very Easy. The average task ease

ratings for ordering a medication and assessing interactions in Tasks 10 and 12 are 4.03 and 4.09, respectively.

The average task ease ratings while ordering a medication without any interactions, Tasks 6 and 15, are higher

at 4.61 and 4.43, respectively.

Major Findings

All participants were able to successfully complete the medication ordering tasks, suggesting a high degree of

effectiveness in these workflows. The differences in other metrics between tasks with and without drug

interaction components suggest that interactions can affect provider effectiveness and perceived ease of use in

ordering tasks. For further analysis of the drug interaction tasks, see section 4.7 §170.315(a)(4).


The medication ordering task presented participants with user interface components updated in Epic 2017.

Although all participants successfully completed all tasks, some extra steps were observed that may be related

to adjusting to the new UI components, suggesting that changes could be made to facilitate this transition to

the updated interface. This may be an appropriate area for further study, particularly:

Exploring potential options to orient users to new UI

Improving continuity of provider navigation in response to alternative courses of clinical decision -

making

________________________________________________________________________________________________


4.4 §170.315(a)(2) Computerized Provider Order Entry – Laboratory


Table 9: Computerized Provider Order Entry – Laboratory Physician Task Results

Computerized Provider Order Entry - Laboratory


N Average Task Time



Rate


Rate

Average Unnecessary

Steps per Task

Task Ease Rating

# Mean

(seconds) (SD)

Mean (SD) Mean % (SD) Mean % (SD) Mean (SD) Mean (SD)

5 = very easy

Physician Subtasks

Modifying the details of an inpatient lab order

36 10.14 (3.59)

0.03 (0.17) 100 (0) 100 (0) 0.03 (0.17) 4.83 (0.45)

Ordering an outpatient lab to monitor a discharge medication

36 9.28 (6.01) 0.11 (0.33) 100 (0) 100 (0) 0.03 (0.17) 4.64 (0.64)


Computerized provider order entry of laboratory tests with inpatient physicians covered two tasks:

Modifying the details of an inpatient lab order (Task 7)

Ordering an outpatient lab to monitor a discharge medication (Task 9)

Efficiency

The average task times for Tasks 7 and 9 are within one second of each other, suggesting a consistent

experience between inpatient and discharge lab ordering.

Effectiveness

The binary task completion rate for participants is 100 percent for Tasks 7 and 9. Participants took a negligible

number of unnecessary steps for both Tasks 7 and 9.

Satisfaction

The majority of participants rated both tasks Very Easy, with average satisfaction scores for Tasks 7 and 9 of

4.83 and 4.64, respectively.

Major Findings

One hundred percent of physicians completed the tasks, and usability metrics suggest that they did so with a

high degree of efficiency, effectiveness, and satisfaction.


User feedback indicates little difficulty with the entry of laboratory orders, which can in part be attributed to

the consistency with other inpatient ordering workflows. As industry best practices for placing laboratory

________________________________________________________________________________________________


orders, assessing relevant information, and communicating additional information to laboratories evolve, we

will continue to monitor this area for future enhancement opportunities, with specific focus on the following

areas:

The display of relevant clinical information at the point of ordering

The communication of additional details to laboratory staff

________________________________________________________________________________________________


4.5 §170.315(a)(3) Computerized Provider Order Entry – Diagnostic Imaging


Table 10: Computerized Provider Order Entry – Diagnostic Imaging Physician Task Results

Computerized Provider Order Entry – Diagnostic Imaging


N Average

Task Time

Average Extra

Steps per Task

Binary Task

Completion Rate

Partial Task

Completion Rate

Average

Unnecessary Steps per Task

Task Ease

Rating

# Mean (seconds)

(SD)


easy

Physician Subtasks

Modifying a diagnostic imaging

procedure

36 17.03

(5.63)

0.03 (0.17) 100 (0) 100 (0) 0.03 (0.17) 4.83 (0.38)

Ordering a diagnostic imaging

procedure

36 13.36

(10.96)

0.03 (0.17) 100 (0) 100 (0) 0 (0) 4.72 (0.57)


Computerized provider order entry testing for diagnostic imaging tests with inpatient physicians covered two

tasks:

Modifying a diagnostic imaging procedure to include a comment (Task 4)

Ordering a diagnostic imaging procedure (Task 13)

Effectiveness

Participants completed Task 4 in an average of 17.03 seconds with negligible extra steps. Task 13 was

completed in an average of 13.36 seconds with negligible extra steps.

Effectiveness

The binary completion rate for Tasks 4 and 13 is 100 percent. Participants completed Task 4 with a negligible

number of unnecessary steps. All participants completed Task 13 with no unnecessary steps.

Satisfaction

The majority of participants rated the tasks as Very Easy, with average scores of 4.83 and 4.72 for Tasks 4 and

13, respectively.

Major Findings

All of the imaging order tasks were completed with high levels of effectiveness, efficiency, and satisfaction,

suggesting that physicians are comfortable using this functionality.


________________________________________________________________________________________________


Physicians successfully completed all imaging orders tasks. Diagnostic image ordering continues to be an

important topic with implications for patient care and the financial health of organizations. Although the test

results did not indicate specific areas for improvement, as new industry practices arise, future testing should

focus on the following areas:

Decision support for Appropriate Use Criteria

Display of radiation exposure information

________________________________________________________________________________________________


4.6 §170.315(a)(5) Demographics


Table 11: Demographics Nurse Task Results

Demographics


N Average

Task Time

Average Extra

Steps per Task

Binary Task

Completion Rate

Partial Task

Completion Rate

Average

Unnecessary Steps per Task

Task Ease

Rating

# Mean (seconds)

(SD)

Mean (SD) Mean % (SD) Mean % (SD)

Mean (SD) Mean (SD) 5 = very

easy

Nurse Subtasks

Documenting preferred language 32 11.69 (9.75)

0.06 (0.25) 100 (0) 100 (0) 0.06 (0.25) 4.72 (0.58)

Updating sexual orientation and gender identity

36 10.26 (12.21)

0.11 (0.34) 97 (0.03) 97 (0.03) 0.03 (0.17) 4.61 (0.55)


Demographics testing with inpatient nurses covered two tasks:

Documenting the patient’s preferred language (Task 1)

Updating sexual orientation and gender identity as reported by the patient (Task 7)

Efficiency

Participants that completed Tasks 1 and 7 did so with an average time between 10 and 12 seconds and took a

negligible number of extra steps to complete Task 1. This suggests a consistently efficient experience when

documenting demographic information, such as sexual orientation, gender identity, and preferred language.

Effectiveness

Participants performed Tasks 1 and 7 with a negligible number of unnecessary steps and binary task

completion rates of 100 percent and 97 percent, respectively. Participants demonstrated they were able to

effectively document demographics information using the software.

Satisfaction

Participants on average rated Tasks 1 and 7 as between Somewhat Easy and Very Easy. Task 1, which involved

preferred language, had higher satisfaction metrics than Task 7, which involved gender identity.

Major Findings

Task 1, which involved preferred language, had slightly higher effectiveness and satisfaction metrics than Task

7, which involved gender identity. Based on debrief session feedback, some participants struggled with

________________________________________________________________________________________________


discussing gender identity in a medical context, which led to uncertainty when completing documentation for

Task 7.


Documenting gender identity is a relatively new workflow for most clinicians. Some nurses commented that

although they were comfortable with the demographics documentation tools in the system, they were

unfamiliar with the concepts and terminology for gender identity that they were asked to document. This

suggests the importance of training staff on gender identity prior to implementing the workflow for capturing

the information. In addition, given the ongoing evolution of standard terminology in the areas of sexual

orientation and gender identity, "just-in-time" training regarding the definition of unfamiliar terms could be

added to help facilitate user adoption.

________________________________________________________________________________________________


4.7 §170.315(a)(4) Drug-Drug, Drug-Allergy Interaction Checks


Table 12: Drug-Drug and Drug-Allergy Interaction Checks Physician Task Results

Drug-Drug, Drug-Allergy Interaction Checks


N Average Task Time



Rate


Rate

Average Unnecessary

Steps per Task

Task Ease Rating

# Mean (seconds)

(SD)


easy

Physician Subtasks

Adding a medication allergy and assessing any interactions

35 10.49 (7.74)

0 (0) 100 (0) 100 (0) 0 (0) 4.51 (0.74)

Ordering a discharge medication and assessing any interactions

34 12.44 (10.18)

0.03 (0.17) 100 (0) 100 (0) 0 (0) 4.03 (1.00)

Ordering an inpatient medication and assessing any interactions

35 23.63 (16.38)

0.03 (0.17) 100 (0) 100 (0) 0 (0) 4.09 (0.31)


Drug-drug and drug-allergy interaction checks testing with inpatient physicians covered three tasks:

Adding a medication allergy and assessing any interactions (Task 3: Subtask B)

Ordering a discharge medication and assessing any interactions (Task 10: Subtask C)

Ordering an inpatient medication and assessing any interactions (Task 12: Subtask C)

Efficiency

Participants took no extra steps while completing Task 3 and a negligible number of extra steps while

completing Tasks 10 and 12.

Effectiveness

All three tasks had a binary completion rate of 100 percent, and participants took no unnecessary steps while

completing the tasks.

Satisfaction

The average satisfaction ratings for all tasks are between Somewhat Easy to Very Easy. Task 3 had higher

satisfaction ratings than Tasks 10 and 12, which may be attributed to the lower complexity of that task —

assessing an interaction after adding an allergy rather than assessing an interaction during an ordering

workflow.

________________________________________________________________________________________________


Major Findings

Physicians did not perform any unnecessary steps while completing the tasks and all three tasks had perfect

completion scores. Overall, physicians completed the tasks efficiently and effectively. The ease of task

completion ratings highlight an opportunity for improvement regarding assessing interactions during

ordering.


Overall, physicians completed drug-allergy and drug-drug interaction tasks with high efficiency and

effectiveness. The ease of task completion ratings and debrief comments suggest that participants were

responding to the perceived frequency of alerts, not the functionality of the interaction alerts themselves. Alert

fatigue is an important topic in the healthcare industry which EHR vendors and organizations using EHR

software must address often. Organizations using Epic have a high degree of flexibility in configuring which

interaction warnings appear and to whom, which can be leveraged to address the concern of alert fatigue.

User satisfaction could be further addressed by investigating ways to display more visual cues of interactions

within the ordering display. However, sometimes more intrusive warnings are necessary to ensure they are

not overlooked, and any investigation will adhere to the principle that patient safety should always be

paramount, even at the expense of user satisfaction, as seen in section 4.3 §170.315(a)(1).

________________________________________________________________________________________________


4.8 §170.315(b)(3) Electronic Prescribing


Table 13: Electronic Prescribing Physician Task Results


Electronic prescribing (e-prescribing) testing with inpatient physicians covered two tasks:

E-prescribing a discharge medication and assessing any interactions (Task 10: Subtask B)

E-prescribing a discharge medication (Task 15: Subtask B)

Efficiency

Participants took no extra steps while completing Tasks 10 and 15, suggesting a consistently efficient

experience for e-prescribing tasks.

Effectiveness

The binary task completion rate for both tasks is 100 percent. No participants performed unnecessary steps

while completing these tasks.

Satisfaction

The average satisfaction ratings for both tasks is between Somewhat Easy and Very Easy with a rating of 4.03

for Task 10 and 4.43 for Task 15. Task 15 had higher satisfaction ratings than Task 10, which may be attributed

to the lower complexity of that task which does not include assessing an interaction.

Major Findings

Physicians did not perform any extra or unnecessary steps while completing the tasks, and both of the tasks

had perfect completion scores. Overall, physicians completed the tasks efficiently and effectively, with ease of

task completion highlighting an opportunity for improvement. Participant debrief comments suggest that the

satisfaction scores for these two tasks might be due to the intrinsic complexity of the discharge order

reconciliation process, requiring both the review of existing information and placing new orders.

Electronic Prescribing


N Average Task Time



Rate


Rate

Average Unnecessary

Steps per Task

Task Ease Rating

# Mean (seconds)

(SD)


easy

Physician Subtasks

E-prescribing a discharge medication and assessing any interactions

34 36.47 (24.84)

0 (0) 100 (0) 100 (0) 0 (0) 4.03 (1.00)

E-prescribing a discharge medication 35 27.37 (13.92)

0 (0) 100 (0) 100 (0) 0 (0) 4.43 (0.95)

________________________________________________________________________________________________



Physicians successfully completed tasks to e-prescribe discharge medication orders with perfect efficiency and

effectiveness metrics. Ease of task completion scores indicate satisfaction as a possible area for improvement ,

such as through the following:

Based on observed actions and participant debrief comments, the workflow could be further

streamlined by reducing the amount of navigation required

Task 10 involves interaction checking, and the ease score for this task is likely influenced by the

interaction checking portion of the task. Improvements for interaction checks are discussed in section

4.7 §170.315(a)(4)Drug-Drug, Drug-Allergy Interaction Checks

________________________________________________________________________________________________


4.9 §170.315(a)(14) Implantable Device List


Table 14: Implantable Device List Nurse Task Results

Implantable Device List


N Average Task Time



Rate


Rate

Average Unnecessary

Steps per Task

Task Ease Rating

# Mean (seconds)

(SD)


easy

Nurse Subtasks

Adding a historically implanted device 36 61.23 (27.77)

0.14 (0.38) 97 (0.03) 99 (0.02) 0.17 (0.41) 4.39 (0.69)

Verifying the information for a historically implanted device

34 13.53 (13.76)

0.03 (0.17) 100 (0) 100 (0) 0 (0) 4.74 (0.51)

Updating the information for a historically implanted device

35 8.74 (4.74)

0 (0) 100 (0) 100 (0) 0 (0) 4.94 (0.24)


Implantable device list testing with inpatient nurses covered three tasks:

Adding a historically implanted device based on information provided by the patient (Task 4)

Verifying the information for a historical entry on the implantable device list (Task 5)

Updating the information for a historical entry on the implantable device list based on information

provided by the patient (Task 10)

Efficiency

Participants who completed tasks verifying and updating historical implant information did so with zero or

negligible extra steps. As anticipated, Task 4, which involved adding a new implant to the chart, has a longer

task time than the other two tasks.

Effectiveness

The binary task completion rate for Tasks 5 and 10 is 100 percent, and 35 out of 36 participants completed Task

4. Participants that completed Tasks 5 and 10 did so without taking any unnecessary steps.

Satisfaction

The majority of participants rated the implants tasks as Somewhat Easy or Very Easy. The ease of task

completion rating is 4.39 for Task 4, 4.74 for Task 5, and 4.94 for Task 10.

________________________________________________________________________________________________


Major Findings

Nurses successfully verified and updated information for a historically implanted device, demonstrating a

strong effectiveness and satisfaction with the system even if they did not have prior experience with the

implantable devices functionality. The nature of adding a new historically implanted device required more

manual entry, which accounts for the longer average task time.


The complexity and volume of data that needs to be entered into the system in a given workflow inversely

correlates with high satisfaction and effectiveness metrics. Further development can aim to bolster the efficiency

of the implantable device list by:

Optimizing the organization of data in the implantable device list

Improving visual cues for specific input fields

Similarly, there are techniques that have increased efficiency in other workflows by providing expedited ways

to document common responses. Further study can determine the best way to further implement these

techniques in the area of implantable device documentation in order to minimize high-volume data entry tasks

and optimize the efficiency of the workflow. Areas to explore include:

Auto-completion of implantable device information where clinically appropriate

More advanced default settings that leverage information such as context of use and implant type

________________________________________________________________________________________________


4.10 §170.315(a)(8) Medication Allergy List


Table 15: Medication Allergy List Physician Task Results


Medication allergy list testing with inpatient physicians covered two tasks:

Adding a medication allergy and assessing any interactions (Task 3: Subtask A)

Entering new patient-reported information for an existing allergy (Task 11)

Efficiency

The average time taken to complete Task 3, which involved adding a new allergy, was 20.68 seconds, while the

average time for Task 11, which involved adding a reaction to an existing allergy, was 9.37 seconds.

Effectiveness

For Task 3, 34 out of 35 participants completed the task and for Task 11, 35 out of 36 participants completed the

task. No unnecessary steps were taken for either task.

Satisfaction

The majority of participants rated both allergy tasks as Very Easy. The ease of task completion rating is 4.51 for

Task 3 and 4.69 for Task 11.

Major Findings

The usability metrics suggest a consistent level of effectiveness and efficiency for adding a medication allergy

to a patient's chart and adding information to an existing allergy. The similar results for entering a new allergy

in Task 3 and for modifying an existing allergy in Task 11 suggest the consistency of the workflow for both

methods of updating the patient's medication allergy list.

Medication Allergy List


N Average Task Time



Rate


Rate

Average Unnecessary

Steps per Task

Task Ease Rating

# Mean (seconds)

(SD)

Mean (SD) Mean % (SD) Mean % (SD) Mean (SD) Mean (SD) 5 = very easy

Physician Subtasks

Adding a medication allergy and assessing any interactions

35 20.68 (16.53)

0.09 (0.30) 97 (0.03) 99 (0.02) 0 (0) 4.51 (0.74)

Entering new information for an existing allergy

36 9.37 (7.54) 0.06 (0.24) 97 (0.03) 97 (0.03) 0 (0) 4.69 (0.75)

________________________________________________________________________________________________



Comments from participants and other research indicate that modifying existing allergies is an infrequent task

for most inpatient physicians, but results suggest that existing guidance provided by the system allows users

to successfully complete this task, despite a lower degree of familiarity.

Overall, physicians successfully added medication allergies to patients' charts. The few extra and unnecessary

steps relate to variations in allergy terminology usage, such as the distinction between reaction and reaction

type. Clearer industry definitions of reaction and reaction type may improve efficiency and effectiveness in

accurately documenting medication allergies.

________________________________________________________________________________________________


4.11 §170.315(a)(7) Medication List


Table 16: Medication List Nurse Task Results


Medication list testing with inpatient nurses covered two tasks:

Reviewing a complex medication list based on information provided by the patient (Task 3)

Adding a patient-reported medication (Task 9)

Efficiency

Participants that completed Tasks 3 and 9 did so at a high level of efficiency, with no extra steps taken while

completing the tasks.

Effectiveness

Thirty-five out of 36 participants successfully reviewed a complex medication list in Task 3 and 35 out of 36

added a patient-reported medication in Task 9. No unnecessary steps were taken to complete Task 3 and the

number of unnecessary steps is negligible for Task 9.

Satisfaction

A majority of users rated the medication list tasks as Very Easy. The two tasks garnered task ease ratings of

4.81 for Task 3 and 4.69 for Task 9, suggesting participants had a consistent level of satisfaction with both

tasks.

Major Findings

As indicated by the usability metrics, nurses completed Task 3 efficiently and effectively, while giving the task

a high satisfaction rating. In Task 9, 35 of 36 users successfully added the new medication to the patient's chart.

Overall, these results demonstrate high proficiency among nurses using the medication list.

Medication List


N Average Task Time



Rate


Rate

Average Unnecessary

Steps per Task

Task Ease Rating

# Mean (seconds)

(SD)


easy

Nurse Subtasks

Reviewing a complex medication list 36 27.64 (19.19)

0 (0) 97 (0.03) 99 (0.01) 0 (0) 4.81 (0.47)

Adding a patient-reported medication 36 31.12

(13.76)

0 (0) 94 (0.04) 98 (0.02) 0.06 (0.24) 4.69 (0.58)

________________________________________________________________________________________________



Overall, efficiency and satisfaction scores were high for both tasks, with no extra steps taken and an average

satisfaction rating of Very Easy. Effectiveness metrics for Task 9 suggest improvements could be made to how

the medication list responds to new information. Effectiveness and ease of use could be further improved by

system updates to the medication list functionality to more explicitly indicate when the “last taken” date was

last updated and the prioritization of how recently added information is presented on the screen.

________________________________________________________________________________________________


4.12 §170.315(a)(6) Problem List


Table 17: Problem List Physician Task Results

Problem List


N Average Task Time



Rate


Rate

Average Unnecessary

Steps per Task

Task Ease Rating

# Mean (seconds)

(SD)


easy

Physician Subtasks

Adding a problem to a patient’s problem list

35 22.53 (10.98)

0.09 (0.30) 97 (0.03) 97 (0.03) 0 (0) 4.63 (0.49)

Reviewing a patient’s problem list 36 2.39 (1.81) 0 (0) 100 (0) 100 (0) 0 (0) 4.92 (0.37)

Resolving a problem on a patient’s problem list

35 3.14 (0.88) 0 (0) 100 (0) 100 (0) 0 (0) 4.83 (0.57)


Problem list testing with inpatient physicians covered three tasks:

Adding a problem to a patient’s problem list (Task 2)

Reviewing a patient’s problem list (Task 5)

Resolving a problem on a patient’s problem list (Task 14)

Efficiency

All participants completed Tasks 5 and 14 without any extra steps. Thirty-one out of 34 participants who

completed Task 2 did so without any extra steps.

Effectiveness

All participants completing the tasks did so without any unnecessary steps. The binary task completion rate

for both Tasks 5 and 14 is 100 percent, indicating a high level of effectiveness. The binary task completion rate

for Task 2 is 97 percent.

Satisfaction

The majority of participants rated the tasks as Very Easy. The average task ease rating for Task 2 was 4.63, for

Task 5 was 4.92, and for Task 14 was 4.83.

Major Findings

In Task 5 and 14 participants reviewed and resolved a problem on a patient's problem list efficiently and

effectively, while giving the tasks high ease of completion ratings. The higher ease of task completion rate

________________________________________________________________________________________________


average for Tasks 5 and 14 compared to Task 2 is likely due to the lower inherent complexity of reviewing or

resolving a problem from a patient's problem list compared to adding a problem to a patient's problem list.


In general, participants performed all problem list tasks with high effectiveness and satisfaction. Although the

results were positive and did not indicate specific areas for improvement, future study should focus on the

following areas as new industry practices arise:

• The evolution of problems over time to facilitate documentation of a longitudinal patient story

• Patients with a complex medical history and a wide variety of interrelated medical conditions

________________________________________________________________________________________________


5 Results Conclusion Physicians and nurses performed tasks that were selected to address areas of Moderate to High risk (see section

3.3.3 for details). The overall average number of extra steps is 0.07 per task, average binary task completion is

99 percent, and average unnecessary steps is 0.02 per task. Common areas for improvement include:

Consolidating navigation to support a variety of clinical decision-making approaches

Due to differences in background and clinical education, clinicians can manage clinical decisions

in a wide range of ways. As discussed in sections 4.3, 4.8, and 4.9, participants were sometimes

observed using navigation and screen elements in a non-linear order. Further research could be

done to ensure that navigation supports both the most common workflows and significant

variations.

Enhancing visual cues to increase visibility of system status

Visibility of system status is key to guiding users to the appropriate next steps in a workflow and

provides immediate visual feedback of the effects of their choices. While elements of these design

principles were already observed to positively affect usability in our testing, we also identified

further opportunities for their use, such as in sections 4.1, 4.7, and 4.11.

In addition to Ease of Task Completion Scores recorded per task, each participant completed the System

Usability Scale (SUS) at the end of each testing session. The SUS is a reliable industry standard for measuring

user satisfaction.

Inpatient physicians gave an average SUS score of 85.94. Inpatient nurses gave an average SUS score of 87.41.

According to usability research, both the physician and nurse scores correlate with an “excellent” user

experience,8 indicating that Epic users find that the software is easy to use overall.

8 See Bangor, A., Kortum, P. T., & Miller, J.T. (2009). Determining What Individual SUS Scores Mean; Adding an Adjective Rating Scale. Journal of Usability Studies, 4(3), 114-123.

________________________________________________________________________________________________


Appendices

Appendix 1

Recruiting Screener

Note: Italicized text indicates information used for internal determination of eligibility and was not included

on the copy given to the participant.

________________________________________________________________________________________________


Recruiting Screener Demographic Information

1. Name:

2. Credentials:

3. Highest Level of Education:

a. High school graduate/GED

b. Some college

c. College graduate

d. Postgraduate

e. Other

4. Organization:

5. Primary Work Location:

6. Contact method (please provide one of the following):

a. Work phone:

b. Cell phone:

c. Email:

7. What is your gender?

a. Male

b. Female

c. Other (please specify):__________________

8. Which of these best describes your current age?

a. <20

b. 20-29

c. 30-39

d. 40-49

e. 50-59

f. 60-69

g. 70-79

h. ≥80

________________________________________________________________________________________________


Additional Information

9. Is English your first language?

a. Yes

b. No

10. Are you a fluent English speaker? [if No, disqualify]

a. Yes

b. No

11. Due to logistical restraints and the parameters of this study, we cannot provide assistive technologies

during the testing session. Do you require any assistive technologies to use a computer? [if Yes,

disqualify]

a. Yes

b. No

12. Do you, or does anyone in your household, have a commercial interest in an electronic health record

software or consulting company? [if Yes, disqualify]

a. Yes

b. No

13. How many years of experience do you have using computers for personal and professional activities

(such as email, shopping, record keeping, etc.)?

a. <5 years

b. 5-10 years

c. 10-20 years

d. >20 years

________________________________________________________________________________________________


14. What is your current role? [if not Nurse or Physician, disqualify]

a. Analyst

b. Application Coordinator

c. Certified Nursing Assistant (CNA)

d. CEO, CMIO, CIO, etc.

e. Consultant

f. Director

g. Information Technology

h. Licensed Practical Nurse (LPN)

i. Marketing/Communications

j. Medical Assistant (MA)

k. Nurse

l. Nurse Practitioner (NP)

m. Office Manager

n. Pharmacist

o. Physician

p. Physician Assistant (PA)

q. Project Manager

r. SVP, AVP, VP, etc.

s. Trainer

t. Other (please specify)

15. Do you currently provide direct patient care? [if No, disqualify]

a. Yes

b. No

16. In which setting do you primarily work? [if Ambulatory or Emergency Department, disqualify]

a. Inpatient

b. Emergency Department

c. Ambulatory

17. What is your specialty? [if role is Physician and specialty is Radiology, Ophthalmology or Pathology,

disqualify]

18. How many years have you been working in your field?

a. <5 years

b. 5-10 years

c. 10-20 years

d. >20 years

19. Have you participated in Epic usability testing previously?

a. Yes

b. No

If yes, please describe.

________________________________________________________________________________________________


20. How do you capture patient data in your organization? [if Primarily on paper, disqualify]

a. Primarily on paper

b. Primarily electronically

21. Is Epic the EHR you use most often in your organization? [if No, disqualify]

a. Yes

b. No

22. How long have you been using Epic? [if <3 months, disqualify]

23. How frequently do you use Epic? (daily, weekly, monthly)

________________________________________________________________________________________________


Appendix 2

Testing Dates and Locations

Test Dates Locations

1 January 9, 2017 Ann Arbor, MI

2 January 9, 2017 Loma Linda, CA

3 January 10, 2017 Dearborn, MI

4 January 10, 2017 Royal Oak, MI

5 January 11, 2017 Los Angeles, CA

6 January 11, 2017 Tacoma, WA

7 January 12, 2017 Anaheim, CA

8 January 17, 2017 St. Louis Park, MN

9 January 17, 2017 St. Paul, MN

10 January 18, 2017 Minneapolis, MN

11 January 18, 2017 Kettering, OH

12 January 19, 2017 Maplewood, MN

13 January 19, 2017 Akron, OH

________________________________________________________________________________________________


Appendix 3

Participant Demographics

Following is a high-level summary of participants in this study.

Gender Men 32 Women 40 Total (participants) 72

Occupation/Role RN/BSN 36 Physician 36 Total (participants) 72

Years of Experience Years of experience with Epic (average)

6.26

________________________________________________________________________________________________


Appendix 4

IP Physician Scenario 1

Your first patient is Gertrude. Gertrude is a 55-year-old female who has been admitted to your unit from the

ED after experiencing a fall at home. She is a diabetic patient being treated for dehydration, malnutrition,

abrasions, and a possible concussion.

Task 1:

Gertrude mentions that she saw her PCP at an outside organization, River Hills Medical System. She

remembers that he diagnosed her with something new. Reconcile Gertrude’s problem list by adding and

accepting the information from River Hills.

Task 2:

You discuss Gertrude’s current symptoms with her, which include stiffness in both hip joints which makes

it difficult to walk or bend. You examine Gertrude and conclude she is experiencing osteoarthritis in both

hips. As there is no prior documentation of this diagnosis, add this problem to her chart.

Task 3:

Gertrude mentions that sulfa antibiotics give her a rash. Add sulfa antibiotics to her allergy list with a

reaction of rash. You know that any drug-allergy interaction due to her furosemide is unlikely. Use your

clinical judgment to respond to any warnings you receive.

Task 4:

Gertrude complains of pain in her right wrist and you see that it is bruised and swollen. You suspect that

she may have fractured it when she fell. The ED physician already ordered a right wrist x-ray. Update the

order with a comment to the radiologist to look for a possible scaphoid fracture. Sign the order when

complete.


Sheryl is a 68-year-old female who is recovering from a recent knee replacement and is currently admitted to

your unit.

Task 5:

In addition to osteoarthritis of the left knee, Sheryl mentions that her current conditions include high

cholesterol and constipation. Document that you have reviewed her problem list .

Task 6:

As you speak with Sheryl, you learn that she is still experiencing significant pain, despite constant use of her

PCA. Increase the dose of her existing HYDROmorphone (Dilaudid) PCA order slightly by decreasing the

lockout interval to 5 minutes and increasing the basal rate to 0.2 mg/hour. After modifying, sign the order.

Task 7:

There were concerns with how much blood Sheryl lost in surgery, so a CBC was ordered for every 8 hours.

Her hemoglobin levels are looking good, so the frequency of the CBC order can be decreased. Modify the

existing CBC order to have a frequency of daily. After modifying, sign the order.

________________________________________________________________________________________________


Task 8:

Your hospital has a quality improvement initiative that all patients should be up-to-date with a pneumonia

immunization before discharge. Sheryl is not up-to-date with her pneumonia vaccine. Using the information

provided by the system, place and sign the suggested order.

Task 9:

You’ve already completed most of Sheryl’s discharge order reconciliation. Since she will continue her current

warfarin dose, you would like to continue monitoring Sheryl’s INR. Enter an order for Protime-INR. You’ll

also be prescribing a medication, so WAIT to sign this order.

Task 10:

You want to prescribe Sheryl warfarin for post-surgical prophylaxis. Your system automatically sends

prescriptions electronically to the patient’s preferred pharmacy, which has already been specified as Epic

Apothecary. Place and sign an order for warfarin. Sheryl is stabilized on her current warfarin dose. Sign all

orders when complete. Use your clinical judgement to respond to any warnings you receive.


Arthur is a 65-year-old male with a history of type 2 diabetes mellitus, hypercholesterolemia, and bradycardia.

He came to the ED complaining of weakness in his right side and was admitted to the ICU for a CVA.

Task 11:

While reviewing Arthur’s chart, you notice that there is no reaction for his aspirin allergy. He tells you that

aspirin gave him hives. Update Arthur’s aspirin allergy with a reaction of hives.

Task 12:

Your organization has an initiative to increase antiplatelet utilization for stroke and CVA patients. Using the

suggestion provided by the system and your clinical judgement, select and sign an order for an appropriate

antiplatelet therapy.

Task 13:

You decide to order a carotid ultrasound to help diagnose Arthur’s condition. Place and sign an order for a

bilateral carotid ultrasound.

Task 14:

Several days have passed and Arthur is ready to be discharged. Shortly after admission Arthur was

diagnosed with hypokalemia (low potassium). He responded well to the prescribed potassium drip and

now his levels are much better. Resolve hypokalemia in the problem list.

Task 15:

You’ve already completed most of Arthur’s discharge order reconciliation. You would like Arthur to

continue taking Plavix for antiplatelet therapy. Your system automatically sends prescriptions

electronically to the patient’s preferred pharmacy, which has already been specified as Epic Apothecary.

Place and sign an order for Plavix.

________________________________________________________________________________________________


IP Nurse Scenario 1

Your first patient is Walter. Walter is a 79-year-old male with a complex medical history that includes CHF,

osteoporosis, dementia, hypertension, and hyperlipidemia. He is directly admitted to your hospital for a

wound infection.

Task 1:

Walter’s preferred language is Spanish and his PCP speaks Spanish, so he has never needed an interpreter

until now. You have already documented that Walter needs interpreter services. Add Spanish as Walter’s

primary language.

Task 2:

Walter tells you that he went to an outside organization’s urgent care clinic and they determined he has a

new allergy, but he can't remember what the allergy is. You discuss the allergy with Walter and his wife and

confirm what is displayed as accurate. Reconcile Walter’s allergy list by adding and accepting the

information from River Hills.

Task 3:

Walter’s wife gives you a list of Walter’s at-home medications and states that he took them all yesterday.

Review his medications, adding or changing as needed. Indicate that Walter last took doses for all his

medications yesterday. Document that you have reviewed his medication list.

Task 4:

You’re taking Walter’s vitals when he mentions he has a pacemaker. You see that the pacemaker is not on

his Implants List. Walter does not have his pacemaker card with him but his wife gives you some details.

Add Walter’s pacemaker to the Implants List.

Task 5:

While discussing implants, Walter also mentions he had his hip replaced 6 years ago. You remember that

Depuy Orthopaedics recalled several of their hip replacement systems due to the implants shedding metal

shards. You want to determine if Walter could be affected. Review Walter’s hip implant and state verbally if

the manufacturer is Depuy Orthopaedics.

Task 6:

When giving Walter a snack, you notice he has difficulty chewing. After documenting this, you see that a

nutritional consult is recommended by the system. You discuss this with Walter, but he refuses to see a

nutritionist. Document that Walter has difficulty chewing and move to the next section. Review the

advisory given by the system and indicate that Walter refused the nutritional consult.

IP Nurse Scenario 2

Your next patient is Robin. Robin is a 50-year-old who prefers male gender pronouns. He has been admitted

for diabetic ketoacidosis.

Task 7:

Your organization has an initiative to more accurately document gender identity. There is already

documentation present for Robin and you discuss this with him. Robin informs you that while his sex

assigned at birth was female, he identifies as a man. Update this information in the chart.

________________________________________________________________________________________________


Task 8:

Robin recently saw his PCP at an outside organization, River Hills Medical System, for an allergic reaction.

His PCP documented a new allergy which you discuss with Robin and confirm is accurate. Reconcile Robin’s

allergy list by adding and accepting the information from River Hills.

Task 9:

Robin mentions that he is taking Lasix (furosemide) 20 mg at home and last took it yesterday. You also

confirm he is still taking his other medications. Add Lasix to the medication list and indicate that he also

took his other medications yesterday.

Task 10:

You notice Robin touching his right knee as he mentions some tenderness. He confirms that he had a right

knee replacement last year. You review the implant and notice there is no laterality listed for it. Add the

appropriate laterality to Robin’s chart.

________________________________________________________________________________________________


Appendix 5

Sample Participant Orientation

1) We are not testing you or your ability to use the system. We are measuring the usability of the system

itself.

2) You will be taken to the appropriate starting point for each task.

3) You will have a written copy of the task to read.

4) Work at your normal speed and only do what you are specifically asked to do in the system.

5) Because we are testing specific pieces of functionality, you may not complete the entirety of your

normal clinical workflow with a patient.

6) There may be multiple ways to complete a task. You can complete the task in whichever way is

apparent to you or easiest for you.

7) Verbalize that you are done upon completion of each task.

8) Fill out the Ease of Task Completion rating after each task.

9) You will complete a survey about your experience after all tasks are complete.

10) At the end of the test, we may discuss your thought process during specific tasks.

11) Save your comments until all tasks are completed. The facilitator will not offer help or answer any

questions during the test.

12) All of the information you provide will be kept confidential and your name will not be associated with

the results of this session.

________________________________________________________________________________________________


Appendix 6

System Usability Scale Questionnaire

________________________________________________________________________________________________


Rate the Functionality

This scale measures your overall impression of the functionality you tested today.

1 - Strongly Disagree 2 – Disagree 3 – Neutral 4 – Agree 5 – Strongly Agree

I think that I would enjoy using this system if I had to use it frequently

I found the system unnecessarily complex

I thought the system was easy to use

I think I would need the support of a technical person to be able to use the system

I found the various functions in this system were well integrated

I thought there was too much inconsistency in this system

I would imagine that most people would learn to use this system very quickly

I found the system very cumbersome to use

I felt very confident using the system

I need to learn a lot about this system before I could effectively use it

System Usability Scale

________________________________________________________________________________________________


Appendix 7

Frequency of Workflow and Possibility of Alternative Outcome

Possibility of Alternative Outcome

Negligible Low Moderate High

Fre

qu

en

cy o

f W

ork

flo

w

High Negligible Likelihood Moderate Likelihood Moderate Likelihood High Likelihood

Moderate Negligible Likelihood Moderate Likelihood Moderate Likelihood High Likelihood

Low Negligible Likelihood Low Likelihood Moderate Likelihood Moderate Likelihood

Negligible Negligible Likelihood Negligible Likelihood Low Likelihood Low Likelihood