Test Results Summary for 2014 Edition EHR Certification 15 ......Test Results Summary for 2014 Edition EHR Certification 15-3332-R-0034-PRA V1.1, February 28, 2016 ©2016 InfoGard

Test Results Summary for 2014 Edition EHR Certification15-3332-R-0034-PRA V1.1, February 28, 2016

©2016 InfoGard. May be reproduced only in its original entirety, without revision 1

Part 1: Product and Developer Information1.1 Certified Product Information

1.2 Developer/Vendor Information

Part 2: ONC-Authorized Certification Body Information2.1 ONC-Authorized Certification Body Information

Test Type: Complete EHR

Developer/Vendor Name: EndoSoft, LLCAddress: 135 Broadway

ONC HIT Certification Program Test Results Summary for 2014 Edition EHR Certification

Product Name: EndoVault®Product Version: 3.0Domain: Ambulatory

Developer/Vendor Contact: Katie Tracy

ONC-ACB Name: InfoGard Laboratories, Inc.Address: 709 Fiero Lane Suite 25

Schenectady, NY 12305Website: www.endosoft.comEmail: [email protected]: (518) 831-8057

ONC-ACB Contact: Adam Hardcastle

This test results summary is approved for public release by the following ONC-Authorized Certification Body Representative:

Adam Hardcastle EHR Certification Body ManagerONC-ACB Authorized Representative Function/Title

San Luis Obispo, CA 93401Website: www.infogard.comEmail: [email protected]: (805) 783-0810

Signature and Date2/28/2016

http://www.endosoft.com/

mailto:[email protected]

http://www.infogard.com/




2.2 Gap Certification

(a)(1) (a)(19) (d)(6) (h)(1)(a)(6) (a)(20) (d)(8) (h)(2)(a)(7) (b)(5)* (d)(9) (h)(3)(a)(17) (d)(1) (f)(1)(a)(18) (d)(5) (f)(7)**

*Gap certification allowed for Inpatient setting only**Gap certification allowed for Ambulatory setting only

2.3 Inherited CertificationThe following identifies criterion or criteria certified via inherited certification

(a)(1) (a)(16) Inpt. only (c)(2) (f)(2) (a)(2) (a)(17) Inpt. only (c)(3) (f)(3)

The following identifies criterion or criteria certified via gap certification§170.314

No gap certification

§170.314

(d)(3) (a)(6) (b)(1) (d)(4)

(f)(6) Amb. only (a)(7) (b)(2) (d)(5)

(a)(3) (a)(18) (d)(1) (f)(4) Inpt. only

(a)(4) (a)(19) (d)(2) (f)(5) Amb. only

(a)(5) (a)(20)

(a)(10) (b)(5) (d)(8) (g)(2) (a)(11) (b)(6) Inpt. only (d)(9) Optional (g)(3)

(a)(8) (b)(3) (d)(6) (f)(7) Amb. Only

(a)(9) (b)(4) (d)(7) (g)(1)

(a)(14) (b)(9) (e)(3) Amb. only (h)(2) (a)(15) (c)(1) (f)(1) (h)(3)

(a)(12) (b)(7) (e)(1) (g)(4) (a)(13) (b)(8) (e)(2) Amb. only (h)(1)

No inherited certification



Part 3: NVLAP-Accredited Testing Laboratory Information

3.1 NVLAP-Accredited Testing Laboratory Information

3.2 Test Information

3.2.1 Additional Software Relied Upon for Certification

No additional software required

Accreditation Number: NVLAP Lab Code 100432-0Address: 709 Fiero Lane Suite 25

San Luis Obispo, CA 93401Website: www.infogard.com

Report Number: 15-3332-R-0034 V1.0Test Date(s): N/A

ATL Name: InfoGard Laboratories, Inc.

For more information on scope of accreditation, please reference http://ts.nist.gov/Standards/scopes/1004320.htm

Part 3 of this test results summary is approved for public release by the following Accredited Testing Laboratory Representative:

Milton Padilla EHR Test Body Manager ONC-ACB Authorized Representative Function/Title

Email: [email protected]: (805) 783-0810ATL Contact: Milton Padilla

MaxMD Configuration Mgr b1, b2 Transmit/receive CCDA Pop-Health c1-3 Upload QRDA files

Signature and Date

Additional Software Applicable CriteriaFunctionality provided by

Additional Software

First Data Bank a1-2, a6-7, a8, a10, Drug Database

2/28/2016

http://www.infogard.com/




3.2.2 Test Tools

Version

No test tools required

3.2.3 Test Data

3.2.4 Standards3.2.4.1 Multiple Standards Permitted

HL7 v2 Laboratory Restults Intervace (LRI) Validation ToolHL7 v2 Syndromic Surveillance Reporting Validation ToolTransport Testing ToolDirect Certificate Discovery Tool

Edge Testing Tool

Alteration (customization) to the test data was necessary and is described in Appendix [insert appendix letter ]

Test ToolCypressePrescribing Validation ToolHL7 CDA Cancer Registry Reporting Validation ToolHL7 v2 Electronic Laboratory Reporting (ELR) Validation ToolHL7 v2 Immunization Information System (IIS) Reporting Valdiation Tool

(a)(13)

§170.207(a)(3)IHTSDO SNOMED CT® International Release July 2012 and US Extension to SNOMED CT® March 2012 Release

§170.207(j)HL7 Version 3 Standard: Clinical Genomics; Pedigree

No alteration (customization) to the test data was necessary

The following identifies the standard(s) that has been successfully tested where more than one standard is permitted

Criterion # Standard Successfully Tested

(a)(8)(ii)(A)(2)

§170.204(b)(1)HL7 Version 3 Implementation Guide: URL-Based Implementations of the Context-Aware Information Retrieval (Infobutton) Domain

§170.204(b)(2)HL7 Version 3 Implementation Guide: Context-Aware Knowledge Retrieval (Infobutton) Service-Oriented Architecture Implementation Guide



None of the criteria and corresponding standards listed above are applicable

3.2.4.2 Newer Versions of Standards

No newer version of a minimum standard was tested

(a)(15)(i)

§170.204(b)(1) HL7 Version 3 Implementation Guide: URL-Based Implementations of the Context-Aware Information Retrieval (Infobutton) Domain

§170.204(b)(2)HL7 Version 3 Implementation Guide: Context-Aware Knowledge Retrieval (Infobutton) Service-Oriented Architecture Implementation Guide

(b)(7)(i)

§170.207(i) The code set specified at 45 CFR 162.1002(c)(2) (ICD-10-CM) for the indicated conditions


(b)(8)(i)



(a)(16)(ii) §170.210(g) Network Time Protocol Version 3 (RFC 1305)

§170. 210(g)Network Time Protocol Version 4 (RFC 5905)

(b)(2)(i)(A)



(e)(3)(ii) Annex A of the FIPS Publication 140-2

Common MU Data Set (15)


§170.207(b)(2)The code set specified at 45 CFR 162.1002(a)(5) (HCPCS and CPT-4)

(e)(1)(i) Annex A of the FIPS Publication 140-2

(e)(1)(ii)(A)(2) §170.210(g) Network Time Protocol Version 3 (RFC 1305)

§170. 210(g)Network Time Protocol Version 4 (RFC 5905)

The following identifies the newer version of a minimum standard(s) that has been successfully tested

Newer Version Applicable Criteria



3.2.5 Optional Functionality

No optional functionality tested

(a)(4)(iii) Plot and display growth charts

(b)(1)(i)(B) Receive summary care record using the standards specified at §170.202(a) and (b) (Direct and XDM Validation)

(b)(1)(i)(C) Receive summary care record using the standards specified at §170.202(b) and (c) (SOAP Protocols)

Criterion # Optional Functionality Successfully Tested

(f)(3)

Ambulatory setting only – Create syndrome-based public health surveillance information for transmission using the standard specified at §170.205(d)(3) (urgent care visit scenario)

(f)(7) Ambulatory setting only – transmission to public health agencies – syndromic surveillance - Create Data Elements


Express Procedures according to the standard specified at §170.207(b)(3) (45 CFR162.1002(a)(4): Code on Dental Procedures and Nomenclature)

(b)(2)(ii)(B) Transmit health information to a Third Party using the standards specified at §170.202(a) and (b) (Direct and XDM Validation)

(b)(2)(ii)(C) Transmit health information to a Third Party using the standards specified at §170.202(b) and (c) (SOAP Protocols)

(e)(1) View, download and transmit data to a third party using the standard specified at §170.202(d) (Edge Protocol IG version 1.1)


Express Procedures according to the standard specified at §170.207(b)(4) (45 CFR162.1002(c)(3): ICD-10-PCS)



3.2.6 2014 Edition Certification Criteria* Successfully Tested

TP** TD*** TP** TD***

(a)(1) (c)(3) (a)(2) (d)(1) (a)(3) (d)(2)

Criteria #Version

Criteria #Version

(a)(7) (d)(6) (a)(8) (d)(7) (a)(9) (d)(8)

(a)(4) (d)(3) (a)(5) (d)(4) (a)(6) (d)(5)

(a)(13) (e)(3) Amb. only

(a)(14) (f)(1) (a)(15) (f)(2)

(a)(10) (d)(9) Optional

(a)(11) (e)(1) (a)(12) (e)(2) Amb. only

(a)(19) (a)(20) (f)(6) Optional &

Amb. only (b)(1)

(a)(16) Inpt. only (f)(3) (a)(17) Inpt. only (f)(4) Inpt. only

(a)(18) (f)(5) Optional & Amb. only

(b)(5) (g)(3) (b)(6) Inpt. only (g)(4) (b)(7) (h)(1)

(b)(2) (f)(7) Amb. only

(b)(3) (g)(1) (b)(4) (g)(2)

*For a list of the 2014 Edition Certification Criteria, please reference http://www.healthit.gov/certification (navigation: 2014 Edition Test Method)**Indicates the version number for the Test Procedure (TP)***Indicates the version number for the Test Data (TD)

(b)(8) (h)(2) (b)(9) (h)(3) (c)(1) (c)(2)



3.2.7 2014 Clinical Quality Measures*Type of Clinical Quality Measures Successfully Tested:

CMS ID Version CMS ID Version CMS ID Version CMS ID Version2 90 136 155

22 117 137 15650 122 138 15752 123 139 15856 124 140 15961 125 141 16062 126 142 16164 127 143 16365 128 144 16466 129 145 16568 130 146 16669 131 147 16774 132 148 16975 133 149 17777 134 153 17982 135 154 182

CMS ID Version CMS ID Version CMS ID Version CMS ID Version9 71 107 172

26 72 108 17830 73 109 18531 91 110 18832 100 111 19053 102 11355 104 11460 105 171

Ambulatory Inpatient No CQMs tested*For a list of the 2014 Clinical Quality Measures, please reference http://www.cms.gov (navigation: 2014 Clinical Quality Measures)

Ambulatory CQMs

Inpatient CQMs



3.2.8 Automated Numerator Recording and Measure Calculation3.2.8.1 Automated Numerator Recording

Automated Numerator Recording was not tested

3.2.8.2 Automated Measure Calculation

Automated Measure Calculation was not tested

3.2.9 Attestation

(a)(3) (a)(12) (a)(19) (b)(8)(a)(4) (a)(13) (a)(20) (b)(9)

Automated Numerator Recording Successfully Tested(a)(1) (a)(11) (a)(18) (b)(6)

(a)(7) (a)(16) (b)(4) (e)(3)(a)(9) (a)(17) (b)(5)

(a)(5) (a)(14) (b)(2) (e)(1)(a)(6) (a)(15) (b)(3) (e)(2)

(a)(4) (a)(13) (a)(20) (b)(9)(a)(5) (a)(14) (b)(2) (e)(1)

Automated Numerator Recording Successfully Tested(a)(1) (a)(11) (a)(18) (b)(6)(a)(3) (a)(12) (a)(19) (b)(8)

Attestation Forms (as applicable) Appendix

(a)(6) (a)(15) (b)(3) (e)(2)(a)(7) (a)(16) (b)(4) (e)(3)

*Required if any of the following were tested: (a)(1), (a)(2), (a)(6), (a)(7), (a)(8), (a)(16), (a)(18), (a)(19), (a)(20), (b)(3), (b)(4), (b)(9)**Required for every EHR product

Safety-Enhanced Design* A Quality Management System** B Privacy and Security C

(a)(9) (a)(17) (b)(5)



An inaccurate description of the summative usability testing measures used for Effectiveness, Efficiency, and Satisfaction was provided in the "Results" section of the report. The information provided in the table of results data did not match the results as described for measures of Effectiveness, Efficiency, and Satisfaction.

Appendix A: Safety Enhanced Design

User-Centered Design (UCD) process and standard

ISO/TR 16982

SO/TR 16982:2002 provides information on human-centered usability methods which can be used for

design and evaluation. It details the advantages, disadvantages and other factors relevant to using

each usability method.

It explains the implications of the stage of the life cycle and the individual project characteristics for

the selection of usability methods and provides examples of usability methods in context.

https://www.iso.org/obp/ui/#iso:std:31176:en

ISO 9241-210 (2010)

ISO 9241-210:2010 provides requirements and recommendations for human-centered design principles and activities throughout the life cycle of computer-based interactive systems. It is intended to be used by those managing design processes, and is concerned with ways in which both hardware and software components of interactive systems can enhance human–system interaction.

https://www.iso.org/obp/ui/#iso:std:iso:9241:-210:ed-1:v1:en

• §170.314(a)(1) Computerized provider order entry

- Standard used – ISO/TR 16982

- Standard used – ISO 9241-210 (2010)

• §170.314(a)(2) Drug-drug, drug-allergy interaction checks



• §170.314(a)(6) Medication list

https://www.iso.org/obp/ui/#iso:std:31176:en

https://www.iso.org/obp/ui/#iso:std:iso:9241:-210:ed-1:v1:en



• §170.314(a)(7) Medication allergy list



• §170.314(a)(8) Clinical decision support



• §170.314(a)(16) Inpatient setting only –electronic medication administration record



• §170.314(a)(18) Optional – computerized provider order entry –medications



• §170.314(a)(19) Optional – computerized provider order entry –laboratory



• §170.314(a)(20) Optional – computerized provider order entry –diagnostic imaging



• §170.314(b)(3) Electronic prescribing



• §170.314(b)(4) Clinical information reconciliation



Page 1 of 11

EHR Usability Test Report of EndoVault

EHR Version 3.0

EndoVault EHR Version 3.0

Standard Used: ISO/IEC 25062: Common Industry Format for Usability Test Reports

Date of Usability Test: Usability Tests Conducted in December 2014- January 2015

Date of Report: December 12th 2014 to January 2nd 2015

Report Prepared By:

Abhishek Bajaj

Clinical Director

518-831-8064

[email protected]

135 Broadway, Schenectady, NY 12305

Page 2 of 11

EXECUTIVE SUMMARY

A usability test of EndoVault EHR, version 3.0, Inpatient, Outpatient and Ambulatory

was conducted on December 12th, 2014 through January 2nd 2015 at Jewish General

Hospital. Dr. Grossman and Price, Dr. Gaston and Segal Cancer Center Clinical Staff

participated. The purpose of this test was to test and validate the usability of the current

user interface, and provide evidence of usability in the EHR Usability Test (EHRUT).

During the usability test, 12 healthcare providers and clinical and non-clinical staff

matching the target demographic criteria served as participants and used the EHRUT in

simulated, but representative tasks.

This study collected performance data on 12 tasks typically conducted in an

EHR:

Find information in Patient Summary screen

Use Pt. chart to find lab results

Review Consults

Review Procedure Notes

Drug-drug / drug-allergy checks

Medication List

Allergy List

Clinical Decision Support

Reconciliation

eRx

CPOE

eMAR

During the test, clinical and non-clinical staff members along with new

employees were requested to participate (but not required) in the Usability test.

They were instructed that they could withdraw at any time. During the Usability

testing at different sites, some participants (physicians) had experience with

EndoVault 3.0’s EHR. Others along with ancillary staff at certain sites did not

have prior experience with the EHR.

The administrator introduced the test, and instructed participants to complete a

series of tasks (given one at a time) created by EndoSoft LLC using the

EHRUT guidelines. During the testing, the administrator used data logger(s)

Page 3 of 11

who recorded the participants’ performance data on paper. The administrator

did not give the participants assistance in how to complete the tasks.

NO training or help materials were provided.

Usability test scenarios for all eleven situations were designed based on real end-user needs to perform

the tasks needed to provide for the patient on a day to day basis.

The following types of data were collected for each participant:

• Number of tasks successfully completed.

• Number and types of errors

• Path deviations (Correct path, Minor deviations, Major Deviations)

• Participant’s satisfaction ratings of the system

• Ease of use. (Easily Completed, Completed with Difficulty or Help, Not

completed)

All participant data was de-identified – no correlation could be made from the identity of the

participant to the data collected. Following the conclusion of the testing, participants were

thanked for their help. No dollar amount was exchanged. Various recommended metrics, in

accordance with the examples set forth in the NIST Guide to the Processes Approach for

Improving the Usability of Electronic Health Records, were used to evaluate the usability of the

EHRUT. Following is a summary of the performance and rating data collected on the EHRUT.

4 of 11

Measure

Task

N Task Success Path Deviation Easily

Completed

Minor

Deviations

Completed

with difficulty

Mean(SD)

Non

Applicable

Task Ratings

Easy

# Mean Deviations

(Comp

w/difficulty

/Optimal/Minor

deviations)

Mean Mean Mean Mean

1.Find information in

Patient Summary

screen

12 24(12)/12

Mean=24

10 Optimal

2 Comp

w/difficulty

22(10)/12

=18.33

14(2)/12

=2.33

22(10)/12

=18.33

2.Use Pt. chart to find

lab results

12 24(12)/12

=24

11 Optimal

1 Minor deviation

23(11)/12

=21.08

13(1)/12

=1.08

24(12)/12

=24

3.Review Consults 12 24(12)/12

=24

9 Optimal

3 Comp

w/difficulty

21(9)/12

=15.75

15(3)/12

=3.75

21(9)/12

=15.75

4.Review Procedure

Notes

12 24(12)/12

=24

9 Optimal

3 Comp

w/difficulty

21(9)/12

=15.75

15(3)/12

=3.75

21(9)/12

=15.75

5.Drug-drug / drug-

allergy checks

12 24(12)/12

=24

7 Optimal

1 Minor Deviation

4 Non Applicable

16(7)/8

=14

16(7)/8

=14

16(4)/12

= 5.33

16(7)/8

=14

6.Medication List 12 24(12)/12

=24

7 Optimal

1 Minor Deviation

4 Non Applicable

16(7)/8

=14

16(7)/8

=14

16(4)/12

= 5.33

16(7)/8

=14

7.Allergy List 12 24(12)/12

=24

12 Optimal 24(12)/12

=24

24(12)/12

=24

5 of 11

8.Clinical Decision

Support

12 24(12)/12

=24

12 Optimal 24(12)/12

=24

24(12)/12

=24

9.Reconciliation 12 24(12)/12

=24

3 Optimal

2 Minor

Deviations

7 Non Applicable

10(3)/5

=6

10(2)/5

=4

19(7)/12

=11.08

10(3)/5

=6

10.eRx 12 24(12)/12

=24

2 Optimal

10 Non Applicable

4(2)/2

=4

22(10)/12

=18.33

4(2)/2

=4

11.CPOE 12 24(12)/12

=24

8 Optimal

1 Minor Deviation

3 Non Applicable

20(8)/12

= 13.33

13(1)/12

= 1.08

15(3)/12

= 3.75

21(9)/12

= 15.75

12.eMAR 12 24(12)/12

=24

2 Optimal

10 Non Applicable

4(2)/2

=4

22(10)/12

=18.33

4(2)/2

=4

The results from the System Usability Scale scored the subjective satisfaction with the

system based on performance with these tasks to be: Easily completed by the majority of the

clinical and non-clinical staff.

In addition to the performance data, the following qualitative observations were made:

o Major findings

The majority of the people who participated in the Usability test found the application to be quite

straight forward and easy to use. There were occasional users that had minor deviations from the

optimal path mostly due to the lack of experience with EHR’s. Some just didn’t notice certain

obvious and non-obvious choices and had reactions such as: “Oh yeah, missed that for sure” as

reported by the administrator of the usability test. The more inexperienced usability testers had

some difficulty navigating through certain sections as they were unaware of the clinical flow of a

patient and completed certain tasks with difficulty.

o Areas for improvement

According to the feedback received certain users mentioned that there were multiple clicks

required in certain sections.

Some users found certain sections to be too complex for a couple of reasons:

o They had used a different system in the past and had a different workflow in

mind.

o They never had experience with certain sections because it was out of the scope

of their practice.

o Reduction of clicks was a major suggestion and we will be working on it for

sections indicated by the users.

6 of 11

INTRODUCTION

The EHRUT(s) tested for this study was EndoVault, Version 3.0. Designed to present medical information to

healthcare providers in Inpatient/Outpatient and Ambulatory facility types and specialties such as Oncology (Medical,

Surgical, Hematology, Gynecology and Dermatology), Breast Surgery, Gastroenterology, Pulmonology, and

Administration (placing orders such as medication renewal). The EHRUT consists of realistic scenarios and we were

asked to demonstrate the uses of application as a trained user would use in real life patient care. The usability testing

attempted to represent realistic exercises and conditions. The purpose of this study was to test and validate the usability

of the current user interface, and provide evidence of usability in the EHR Usability Test (EHRUT). To this end,

measures of effectiveness, efficiency and user satisfaction, such as ease of use, flexibility of the application, most used

section being the highlight and were asked to complete the tasks necessary for any facility using the EHR. Tasks such

as: Reviewing patient summary, Checking Lab results, Reviewing Procedure notes, Reviewing Consults, CPOE, Drug-

Drug/ Drug –Allergy check, Medication List, Allergy List, Clinical Decision Support, Reconciliation, eRx,eMAR etc.

were captured during the usability testing.

METHOD

PARTICIPANTS

A total of 12 participants were tested on the EHRUT(s). Participants in the test were Physicians, Nurses,

Medical Technicians, Clerical Staff and Non-Clinical staff. Participants were recruited by facility

administrators where EndoVault is available and installed currently. Participants were not influenced or

contacted by EndoVault other than providing a Usability Testing document. There was no compensation

provided to the users in efforts to reduce biased opinions. In addition, participants had no direct

connection to the development of or organization producing the EHRUT(s). Participants were not from

the testing or supplier organization. Participants were given the opportunity to have the same orientation

and level of training as the actual end users would have received. In certain User testing cases the users

were provided no training other than given a “User ID and password for login and a test patient on the

Schedule Screen already scheduled for testing purposes.” For the test purposes, end-user characteristics

were not requested by the developing company but were released by the administrators after the testing

was complete.

ID Age Gender Education Profession Occupational Role Computer

Experience

Yrs of

Experience

Product

Experience

101 32 Female Bachelors Practice Admin Administrator 15 + yrs 8 yrs 2 years

102 21 Female Bachelors Student Office Assistant 5 + yrs 2 yrs 0 yrs

103 25 Female Bachelors Medical Tech Medical Assistant 8 + yrs 7 yrs 6 months

104 67 Male Doctorate Physician Gastroenterologist Unknown 39 yrs 6 yrs

105 35 Male Doctorate Physician Breast Surgery Unknown 10 yrs 2 yrs

106 31 Female Doctorate Physician Medical Oncologist 15+ yrs 5 yrs 2yrs

107 36 Female Doctorate Physician GYN Oncologist 20+ yrs 8 yrs 2yrs

108 27 Male Unknown Medical Tech Technician 10 + yrs 6 yrs 0 yrs

109 55 Male Doctorate Physician Gastroenterologist Unknown 25 Yrs 2 Yrs

110 40-45 Female Unknown Admin Asst. Assistant Unknown 15 + yrs 0 yrs

111 25 Male Medical

Student

Student Student 10 + yrs 0 yrs 0 yrs

112 72 Male Doctorate Physician Breast Surgeon Unknown 47 + yrs 1 yrs

Page 7 of 11

STUDY DESIGN

Overall, the objective of this test was to uncover areas where the application performed well –

that is, effectively, efficiently, and with satisfaction – and areas where the application failed to

meet the needs of the participants. The data from this test may serve as a baseline for future

tests with an updated version of the same EHR and/or comparison with other EHRs provided

the same tasks are used. In short, this testing serves as both a means to record or benchmark

current usability, but also to identify areas where improvements must be made.

During the usability test, participants interacted with EndoVault 3.0 EHR. Each participant

used the EndoVault 3.0 under the supervision of the site administrator. The Usability test was

conducted in Medical University Teaching Hospitals, ASC’s, and Stand-alone physician

practices, and were all provided with the same instructions. The system was evaluated for

effectiveness, efficiency and satisfaction as defined by measures collected and analyzed for

each participant:

Number of tasks successfully completed without assistance

Number and types of errors

Path deviations

Participants’ verbalizations (comments)

Participants’ satisfaction ratings of the system

TASKS

A number of tasks were constructed that would be realistic and representative of the kinds of

activities a user might do with this EHR, including:

1. Find patient’s information in the patient summary screen

2. Find results of recent blood work

3. Check and review Consult Notes.

4. Check and review Procedure Notes

5. Drug-drug / drug-allergy checks

6. Medication List

7. Allergy List

8. Clinical Decision Support

9. Reconciliation

10. eRx

11. Place Orders from the CPOE Screen

12. eMAR

Page 8 of 11

Tasks were selected based on their frequency of use, criticality of function, and those that may be most

troublesome for users. Tasks should always be constructed in light of the study objectives.

PROCEDURES

Participants were selected at random from the site administrators where the Usability tests

were conducted. Participants were asked to volunteer to take the Usability Test therefore

consenting verbally and then the time and date were confirmed by the administrators on site.

To ensure that the test ran smoothly, the site administrator did one on one testing therefore

data logging and recording any verbalizations from the Testers. The usability testing staff

conducting the test were experienced usability practitioners with medical experience, a

practice management background and a number of years of experience in the field of

medicine.

The administrator moderated the session including administering instructions and tasks. The administrator

also monitored, obtained post-task rating data, and took notes on participant comments. They also served as

the data logger and took notes on task success, path deviations, number and type of errors, and comments.

Participants were instructed to perform the tasks as quickly as possible making as few errors and deviations

as possible. (See specific instructions below):

All participant data has been de-identified and kept confidential.

• Without assistance; administrators were allowed to give immaterial guidance and clarification on

tasks, but not instructions on use.

• Without using a think aloud technique.

For each task, the participants were given verbal commands for each of the

tasks.

Tasks began once the administrator finished reading the question.

Following the session, the administrator gave the participant the post-test

verbal questionnaire (e.g., the System Usability Scale, suggestions), and

thanked each individual for their participation.

Participants' demographic information, task success rate, time on task, errors,

deviations, verbal responses, and post-test questionnaire were recorded into the

testing script for the participants.

Page 9 of 11

TEST LOCATION

The test facility included a waiting area and a quiet testing room with a table,

computer for the participant, and recording computer for the administrator.

Only the participant and administrator were in the test room. All observers and

the data logger worked from a separate room where they could see the

participant’s screen and face, and listen to the audio of the session. To ensure

that the environment was comfortable for users, noise levels were kept to a

minimum with the ambient temperature within a normal range. All of the

safety instruction and evacuation procedures were valid, in place, and visible to

the participants.

TEST ENVIRONMENT

The EHRUT would typically be used in a healthcare office or facility.

In this instance, the testing was conducted in Medical University Teaching Hospitals, ASC’s,

and Stand-alone physician practices. For testing, participants used PC’s, Tablets (Surface Pros

or IPad’s) and laptops depending on the facility running Windows XP, Windows 7, and

Windows 8.

The participants used a mouse and keyboard, touch screen pens or finger touch

when interacting with the EHRUT.

The EHRUT used various screens and resolutions based on the availability of the facility.

The application was set up by the vendor in advance according to the vendor’s

documentation describing the system set-up and preparation. The application itself was

running on a windows platform using a test database on a LAN / WAN connection.

Technically, the system performance (i.e., response time) was representative to what actual

users would experience in a field implementation. Additionally, participants were instructed

not to change any of the default system settings (such as control of font size).

TEST FORMS AND TOOLS

During the usability test, various documents and instruments were used, including:

1. Moderator’s Guide (testing script)

2. Post-test Satisfaction Verbal Questions by the administrator

Page 10 of 11

The participant’s interaction with the EHRUT was captured and recorded digitally with screen

capture software running on the test machine. Reaction and verbal comments were recorded

by administrators and conveyed to the developing company EndoSoft.

PARTICIPANT INSTRUCTIONS

The administrator reads the following instructions aloud to the each participant:

Thank you for participating in this study. Your input is very important. Our session today will

last about 15-20 minutes. During that time you will use an instance of an electronic health

record. I will ask you to complete a few tasks using this system and answer some questions.

You should complete the tasks as quickly as possible making as few errors as possible. Please

try to complete the tasks on your own following the instructions very closely. Please note that

we are not testing you we are testing the system, therefore if you have difficulty all this means

is that something needs to be improved in the system. I will be here in case you need specific

help, but I am not able to instruct you or provide help in how to use the application.

Overall, we are interested in how easy (or how difficult) this system is to use, what in it would

be useful to you, and how we could improve it. I did not have any involvement in its creation,

so please be honest with your opinions.

Following the procedural instructions, participants were shown the EHR and as their first task, were given time- 5

minutes to explore the system and make comments.

Participants were then given 12 tasks to complete as listed above. An analysis following the

results was performed.

DATA SCORING

RESULTS

DATA ANALYSIS AND REPORTING The results of the usability test were calculated according to the methods specified in the

Usability Metrics section above. NONE of the participants who failed to follow session and task

instructions were excluded from the analyses. No Exclusions to report. There were no

irregularities during testing process or interpretation of the data provided by testers.

DISCUSSION OF THE FINDINGS

EFFECTIVENESS

In light of the testing performed, the application seemed to be quite effective in providing

the medical practices what they need and what the medical professionals require.

Page 11 of 11

EFFICIENCY

According to the data collected during the EHRUT it seems that the application was quite

efficient overall, although some users had minor deviations from the optimal path, most

testers navigated through the application with ease.

SATISFACTION

Based on the data collected the satisfaction rate was a range from 75-100% in each category.

Overall the satisfaction with the EHR seemed to be high.

MAJOR FINDINGS

Overall based on the findings provided by the administrators the application seemed to be

easy to use and simple for the users and non-clinical users. The majority of them were able

to navigate quite efficiently. There were some users who were not so tech savvy and had

difficulties navigating.

AREAS FOR IMPROVEMENT

Common suggestions from a few of the testers were to reduce the number of clicks

navigating to specific sections.

Usability Test Reports (Additional)

EndoVault EHR 3.0

Descriptions of the steps for each measures:

§ 170.314(a) (1) (CPOE);

Step 1: Find patient by using search routine.

Step 2: Find Item on patient summary Screen

Step 3: Go to order tab

Step 4: using CPOE routine to add orders.

Assistive Technology used: None

Optimal Time: 30 seconds

Average Actual Time: 25 seconds

§ 170.314(a)(2) (Drug- drug, drug-allergy interaction checks);

Step 1: Find Patient by using search routine.

Step 2: Find Item on patient summary screen

Step 3: Add Drug to the existing chart of the Patient

Step 4: find Drug-drug, drug-allergy interaction




§ 170.314(a)(6) (Medication list);



Step 3: Click on Medication list Tab




§ 170.314(a)(7) (Medication allergy list);



Step 3: Click on Medication allergy Tab




§ 170.314(a)(8) (Clinical decision support);


Step 2: Go To patient chart

Step 3: Clinical decision support will be triggered.

Step 4: Cancel or take action.




§ 170.314(a)(16) (Electronic medication administration record);


Step 2: Go To Medication list tab

Step 3: use scan routine to search meds from order list.

Step 4: add meds to the list.

Assistive Technology used: Bar code Scanner



§ 170.314(b)(3) (Electronic prescribing);



Step 3: go to prescription tab

Step 4: add meds to the prescription list.

Step 5: click send button to send out eprescription.




§ 170.314(b)(4) (Clinical information reconciliation).


Step 2: Go to patient chart

Step 3: click on clinical reconciliation button

Step 4: take action.




TEST ENVIRONMENT

Facility: Jewish General Hospital

Computers: Desktop/Laptop/Tablet with Windows 7 Professional/Windows 8 with 4GB

of RAM

Screen sizes: 19 inches

Screen resolutions: 1920/1200

Color settings: true color (32 bit)

Facility: Dr. Grossman and Price

Computers: Desktop/Laptop with Windows 7 Professional with 4GB of RAM



Color settings: 256 color

Facility: Dr. Gaston

Computers: Desktop/Laptop with Windows 7 Professional/Windows XP with 4GB of

RAM



Color settings: true color (32 bit)

USABILITY METRICS

The goals of the test were to assess:

1. Effectiveness of EHRUT by measuring participant

success rates and errors

2. Efficiency of EHRUT by measuring the average

task time and path deviations

3. Satisfaction with EHRUT by measuring ease of

use ratings

DATA SCORING

The following table details how tasks were scored, errors

evaluated, and the time data analyzed.

Measures Rationale and Scoring

Effectiveness:

Task Success

A task was counted as a “Success” if the participant was able to achieve the correct outcome, without assistance, within the time allotted on a per task basis.

The total number of successes were calculated for each task and then divided by the total number of times that task was attempted. The results are provided as a percentage.

Task times were recorded for successes. Observed task times divided by the optimal time for each task is a measure of optimal efficiency.

Optimal task performance time, as benchmarked by expert

performance under realistic conditions, is recorded.

Effectiveness:

Task Failures

If the participant abandoned the task, did not reach the correct answer or performed it incorrectly, or reached the end of the allotted time before successful completion, the task was counted as a “Failures.” No task times were taken for errors.

The total number of errors was calculated for each task and

then divided by the total number of times that task was

attempted.

Efficiency:

Task

Deviations

The participant’s path (i.e., steps) through the application was

recorded. Deviations occur if the participant, for example, went

to a wrong screen, clicked on an incorrect menu item, followed

an incorrect link, or interacted incorrectly with an on-screen

control. This path was compared to the optimal path. The

number of steps in the observed path is divided by the number

of optimal steps to provide a ratio of path deviation.

Efficiency:

Task Time

Each task was timed from when the administrator said “Begin”

until the participant said, “Done.” If he or she failed to say

“Done,” the time was stopped when the participant stopped

performing the task. Only task times for tasks that were

successfully completed were included in the average task time

analysis. Average time per task was calculated for each task.

Variance measures (standard deviation and standard error)

were also calculated.

Satisfaction:

Task Rating

Participant’s subjective impression of the ease of use of the application was measured by administering both a simple post-task question as well as a post-session questionnaire.

After each task, the participant was asked to rate “Overall, this task was:” on a scale of 1 (Very Difficult) to 5 (Very Easy). These data are averaged across participants.

Common convention is that average ratings for systems judged easy to use should be 3.3 or above.

To measure participants’ confidence in and likeability of the

EHRUT overall, the testing team administered the System

Usability Scale (SUS) post-test questionnaire. Questions

included, “I think I would like to use this system frequently,” “I

thought the system was easy to use,” and “I would imagine that

most people would learn to use this system very quickly.



Appendix B: Quality Management System



Appendix C: Privacy and Security



Test Results Summary Document History Version

V1.0V1.1 Updated Safety-Enhanced Design report February 28, 2016

Initial release July 6, 2015

END OF DOCUMENT

Description of Change Date

Documents

Test Results Summary for 2014 Edition EHR Certification 15 ......Test Results Summary for 2014 Edition EHR Certification 15-3332-R-0034-PRA V1.1, February 28, 2016 ©2016 InfoGard