Embed Size (px)
Citation preview
This may be the author’s version of a work that was submitted/acceptedfor publication in the following source:
Georgiou, Andrew, McCaughey, Euan, Tariq, Amina, Walter, Scott, Li,Julie, Callen, Joanne, Paoloni, Richard, Runciman, William, & Westbrook,Johanna(2017)What is the impact of an electronic test result acknowledgement systemon Emergency Department physicians’ work processes? A mixed-methodpre-post observational study.International Journal of Medical Informatics, 99, pp. 29-36.
This file was downloaded from: https://eprints.qut.edu.au/102740/
c© Consult author(s) regarding copyright matters
This work is covered by copyright. Unless the document is being made available under aCreative Commons Licence, you must assume that re-use is limited to personal use andthat permission from the copyright owner must be obtained for all other uses. If the docu-ment is available under a Creative Commons License (or other specified license) then referto the Licence for details of permitted re-use. It is a condition of access that users recog-nise and abide by the legal requirements associated with these rights. If you believe thatthis work infringes copyright please provide details by email to [email protected]
Notice: Please note that this document may not be the Version of Record(i.e. published version) of the work. Author manuscript versions (as Sub-mitted for peer review or as Accepted for publication after peer review) canbe identified by an absence of publisher branding and/or typeset appear-ance. If there is any doubt, please refer to the published source.
https://doi.org/10.1016/j.ijmedinf.2016.12.006
1
What is the impact of an electronic test result acknowledgement system on Emergency
Department physicians’ work processes? A mixed-method pre-post observational study
Andrew Georgiou a, Euan J McCaughey a, Amina Tariq a,b, Scott R Walter a, Julie Li a, Callen
Joanne Callen a, Richard c,d, William B Runciman e-h, Johanna I Westbrook a
a Centre for Health Systems and Safety Research, Australian Institute of Health Innovation,
Macquarie University, Sydney, Australia
b School of Public Health and Social Work, Faculty of Health, Queensland University of
Technology, Brisbane, Australia
c Emergency Department, Concord Repatriation General Hospital, Sydney, Australia
d Sydney Medical School, The University of Sydney, Sydney, Australia
e Centre for Population Health Research, School of Health Sciences, The University of South
Australia, Adelaide, Australia
f Visiting Professor - Macquarie University, Sydney, Australia
g Clinical Professor - The Joanna Briggs Institute, Department of Anaesthesia and Intensive
Care, University of Adelaide, Adelaide, Australia
h Australian Patient Safety Foundation, Adelaide, Australia
Correspondence to
Professor Andrew Georgiou
Centre for Health Systems and Safety Research, Australian Institute of Health Innovation
6/75 Talavera Road
Macquarie University NSW Australia
Tel: +61 2 9850 2424
Email: [email protected]
2
ABSTRACT
Objective: To examine the impact of an electronic Results Acknowledgement (eRA) system
on emergency physicians’ test result management work processes and the time taken to
acknowledge microbiology and radiology test results for patients discharged from an
Emergency Department (ED).
Methods: The impact of the eRA system was assessed in an Australian ED using: a) semi-
structured interviews with senior emergency physicians; and b) a time and motion direct
observational study of senior emergency physicians completing test acknowledgment pre and
post the implementation of the eRA system.
Results: The eRA system led to changes in the way results and actions were collated, stored,
documented and communicated. Although there was a non-significant increase in the average
time taken to acknowledge results in the post period, most types of acknowledgements (other
than simple acknowledgements) took less time to complete. The number of acknowledgements
where physicians sought additional information from the Electronic Medical Record (EMR)
rose from 12% pre to 20% post implementation of eRA.
Conclusions: Given that the type of results are unlikely to have changed significantly across
the pre and post implementation periods, the increase in the time physicians spent accessing
additional clinical information in the post period likely reflects the greater access to clinical
information provided by the integrated electronic system. Easier access to clinical information
may improve clinical decision making and enhance the quality of patient care. For instance, in
situations where a senior clinician, not initially involved in the care process, is required to deal
with the follow-up of non-normal results.
Keywords: information systems; test result follow-up; laboratory; medical imaging;
emergency department
3
1. INTRODUCTION
The usability of health information technology (IT) systems is a key factor affecting the ability
of health IT to contribute to the delivery of safe, high quality health care [1]. Numerous health
IT applications across healthcare systems globally have been developed to support test result
management processes, including systems that can track pending test results at hospital
discharge [5], deliver result notifications to clinicians [6] and utilise tracking systems to
document test result acknowledgement and subsequent clinical actions [6]. Nevertheless,
research continues to point to challenges associated with the use of electronic test result
management systems [7]. These include problems related to the potential for information
overload [8], negative effects on the efficiency of work processes [9, 10] and challenges in
establishing appropriate escalation procedures to ensure test result follow-up [11].
Inefficiencies are often related to the time taken to use computers and the complexity of
computer tasks [10, 13]. It is therefore necessary to ensure that electronic test result
management systems are developed to complement existing clinical work processes [14]. The
aim of this study was to examine the impact of the introduction of an electronic results
acknowledgement (eRA) system on emergency physicians’ work processes and the time taken
to manage microbiology and radiology test results for patients discharged directly from an ED.
2. MATERIAL AND METHODS
2.1 Setting
The study was conducted in the ED of a 550-bed metropolitan teaching hospital in Sydney,
Australia. The ED manages over 34,000 patient attendances per annum. Test result
acknowledgements for all radiology and microbiology results for patients discharged from the
ED are managed daily from Monday-Friday by emergency physicians who perform one
4
administrative shift per week on a rotating roster. During this time they review and
acknowledge all radiology and microbiology results received since the conclusion of the
previous administrative shift. Prior to August 2013, paper copies of radiology and
microbiology results were reviewed and acknowledged by the emergency physician on
administrative duty using a date stamp and signature on the hard copy report, or in a patient’s
medical record [15]. An eRA system (Message Centre, Cerner Millennium, MO, USA) was
implemented as an additional module to the existing EMR system, and replaced the manual
eRA process in August 2013.
2.2 Intervention
The Cerner Millennium Message Centre is an electronic inpatient and outpatient workflow
management module which comprises an Inbox containing documents and notifications
requiring review, attention, or signature, including test result reports. Results of all
microbiology and radiology tests ordered from the ED arrive electronically in the Inbox, and
can be viewed by all authorised ED physicians through their personal account. Available
results are displayed as a list of electronic reports, which may be sorted by a range of
parameters including date, test type (microbiology or radiology), result status if flagged
(normal, abnormal, critical) and whether the patient was admitted or discharged from the ED.
Message Centre can be accessed from any computer in the hospital via the Cerner Millennium
network, or remotely if clinicians have obtained secure access software for their computer or
other device.
2.3 Study Design
The impact of the eRA system, including effects on physician test acknowledgement work
processes, and the amount of time taken to acknowledge results was assessed pre and post
implementation of the eRA system using mixed methods comprising qualitative semi-
5
structured interviews with all emergency physicians involved in the test acknowledgement
process, and direct observation of emergency physicians undertaking the result
acknowledgment process.
2.3.1 Semi-structured interviews
Semi-structured, in-depth interviews relating to physician test acknowledgement work
processes pre and post eRA implementation were conducted with emergency physicians (n=12)
by two researchers (JC and JL) between December 2013 and April 2014. Audio recordings
were made of interviews during which physicians were asked to describe their result
acknowledgement work process pre and post eRA implementation (interview guide in
Appendix A). Participants were recruited following an email disseminated by the Director of
the Department and were selected purposively based on their use of the eRA system and
responsibility for test result acknowledgement.
2.3.2 Observation sessions
Researchers were provided with a roster and contacted each participant ahead of their
scheduled administration shift to arrange a time to undertake observation. All participants
provided written consent. A subset of seven emergency physicians were recruited purposively
for direct observation based on their availability during the data collection periods. Five
physicians participated in the pre-eRA data collection phase and six physicians were observed
during the post-eRA data collection phase; four physicians were observed in both the pre and
post intervention phases. Each participant took part in one pilot session, followed by two
observation sessions in either the pre or post eRA phase, totalling 10 observations sessions pre-
and 12 sessions post-eRA.
The Work Observation Method by Activity Timing (WOMBAT) method was utilised to
conduct a direct observational time and motion study of physicians performing their test result
6
acknowledgement process pre and post eRA. The WOMBAT tool is a validated data collection
software that is adaptable to different study settings [16-18]. The WOMBAT data collection
template was modified for the purposes of this study and finalised during pilot sessions in the
pre-eRA data collection phase. Using a tablet computer, observers collected information about
specific work tasks performed by the emergency physician (e.g., endorsing either radiology or
microbiology reports), any additional information physicians required if the report warranted
further investigation (e.g., an abnormal result may require review of past results in the patient’s
electronic health record to determine its clinical significance), actions taken and subsequent
communication (e.g., contacting the patient or the patient’s GP if the result was clinically
significant), what documentation resulted due to the action taken (e.g., documentation in the
patient notes), and who the doctor undertook this task with (if anyone) or if they used an
artefact such as a telephone to undertake the task. Figure 1 provides a screenshot of the data
collection tool. Detailed definitions for tasks reported in this paper are available in Appendix
B.
Each process of test acknowledgement began in the pre-eRA phase with the physician
beginning to review individual reports from the paper test result reports collected from the
dedicated fax/printer through which the radiology and microbiology departments transmitted
result reports. Conclusion of the acknowledgement process was defined as the moment the
physician placed an individually reviewed report into a designated tray for filing into the
medical record. In the post eRA phase, the test acknowledgement process was defined as the
period of time between the physician clicking open an electronic result report from the list in
their Message Centre account, to when the same report is “clicked off” (electronically
acknowledged by the click of a specific button), and the physician was taken back to the list
view. Items under the Documentation section of the tool was modified in the post phase of the
study to reflect the electronic locations for documenting follow up action or notes in the
7
electronic test result endorsement system. Pre eRA sessions took place between 21 May and 17
June 2013 and post eRA sessions between 24 June and 21 August 2014, with the duration of
each session determined by how long each participant took to endorse the results that they had
been allocated for that shift (from 2 minutes to 70 minutes). As information was entered into
the WOMBAT tool, all tasks were automatically time-stamped, including interruptions and
tasks conducted in parallel (multi-tasking).
Figure 1 Work Observation Method by Activity Timing (WOMBAT) data collection tool.
2.4 Analysis
All interviews were recorded and transcribed. Information obtained during interviews was used
to facilitate the development of process models of the acknowledgement process pre and post
eRA. An activity diagram modelling approach using Business Process Model and Notation
Refers to the type of report the observed doctor is reviewing, or other activities that may take place during the results review
period
To record sources of additional information required for abnormal reports. Multiple sources of information may be selected.
Refers to the action the observed doctor takes after reviewing a report
Documentation completed by the observed doctor to signify that the review process has been completed
Used to specify other individuals with whom certain tasks may have taken place
Replaced by “Reminder Function” button in the POST data collection phase as the Red Folder is no longer used after eRA implementation
Drop down menus in some categories contain further options
8
(BPMN) [19] was used to develop these models. The models were shared with the study
participants, providing an opportunity for feedback and validation of the findings.
Linear regression was used to assess the change in test acknowledgement time pre and post
eRA for test results relating to patients not admitted to wards at the time of acknowledgement.
The times were log-transformed so that they were approximately normally distributed.
Microbiology and radiology acknowledgements were modelled separately. In addition, the
models allowed for differential changes in acknowledgement time depending on the type of
acknowledgement and were also adjusted for interruption and multitasking status, day of week
and time of day. A random intercepts approach was used to account for correlation within
individual participants and to provide a subject-specific estimate of the change in
acknowledgement time. All analyses were performed using SAS (version 9.4, SAS Institute,
North Carolina, USA).
2.5 Ethics approval
The study was approved by the ethics committee of the hospital. Each participant provided
written informed consent.
3 RESULTS
3.1 Test Acknowledgement- Pre and Post Workflow Analysis
Test result acknowledgement work processes pre and post eRA are represented in Figure 2 and
Figure 3, respectively.
3.1.1 Pre eRA Workflow
Results of all radiology and microbiology tests ordered through the ED were faxed through to a
dedicated fax machine located within the ED. Results were received as soon as they were
9
authorised/finalised, and picked up daily by the emergency physician on administrative duty.
Results for admitted patients could be either reviewed or disregarded and culled (shredded).
Normal reports for all patients would be initialled, date stamped and sent for filing in the paper
medical record, whereas abnormal results (if reviewed) for patients who had been admitted
were communicated to the admitting team by telephone.
Documentation practices on actions taken for abnormal results for patients discharged from the
ED varied between physicians. These included documentation in a dedicated “Results Book”
(microbiology results only) kept in the office where administrative duties took place,
documentation in the paper medical record and/or added to the electronic discharge summary
on the patient’s electronic medical record. Other evidence of subsequent action may also have
been filed in the medical record (e.g., confirmation of a result faxed to a patient’s GP).
Abnormal results may not require action in cases where doctors ascertained through record
review that appropriate action had been taken prior to patient discharge (e.g., patient had been
commenced on appropriate antibiotics for a urinary tract infection based on clinical symptoms),
or if results were not clinically significant. Follow-up actions for abnormal results that could
not be completed were handed over to the next emergency physician on administrative duty in
the form of written notes attached to paper result reports. Result acknowledgement usually
occurred during scheduled administrative days (Monday – Friday), but occasionally took place
during clinical shifts of low activity on weekends to reduce administrative workload for the
following Monday.
3.1.2 Post eRA Workflow
All results are emailed to an electronic Message Centre, in the Cerner system which physicians
accessed during their administrative days. The Message Centre allows lists to be re-sorted by a
number of fields, including discharge status. Some physicians reported clearing admitted
patients by individually opening and endorsing all results for these patients (because the results
10
are expected to be accessed by the treating ward physician as inpatients), and others continue to
take note of critical results, while some physicians focussed only on results relating to
discharged patients. Normal results were endorsed electronically by the click of a relevant
button, with the result then being removed from the available list. Abnormal results were
endorsed (“clicked off”) after the relevant follow-up action was completed and documented in
a Reminder email which posts to the Results flow-sheet of the electronic medical record.
Handover information to other physicians in instances where follow-up action was not finalised
during a particular shift were documented in the Comment field of the electronic report. The
result was consequently unacknowledged, and remained in the list for the following physician.
While result acknowledgement usually occurred during scheduled administrative days
(Monday - Friday), physicians stated that compared to the manual system, they were more
likely to access the electronic system to endorse results outside of scheduled administrative
days due to ease of access (being able to endorse results from any computer in the clinical area)
and convenience of electronic results viewing (compared to the manual system, which relied on
manual sorting and filing, the electronic system can be easily left at any point and re-started
when time allows).
11
Figure 2 Process map detailing workflow for test result acknowledgment pre implementation on an electronic results acknowledgment system.
12
Figure 3 Process map detailing workflow for test result acknowledgment post implementation on an electronic results acknowledgment system.
13
3.2 Test Acknowledgement Process - Pre and Post Timing Observations
The number of individual acknowledgements and the total time for which each participant was
observed is shown in Table 1.
Table 1 Emergency physicians’ characteristics, total time for which they were observed and
the number of individual acknowledgements during this time. Participants were observed
during two sessions both pre and post implementation of an eRA system.
Pre Post
Physician
ID
Total Time
Observed
(min:sec)
Number of
Acknowledgements
Total Time
Observed
(min:sec)
Number of
Acknowledgements
1 10:21 60 57:02 190
2 38:53 73 77:41 163
3 83:29 169 73:42 136
4 80:13 43 101:00 60
5 60:44 113
6
87:12 120
7
89:05 116
Total 273:40 458 485:42 785
Four hundred and fifty-eight individual test acknowledgements were observed pre eRA and
785 observed post (Table 1). Four main types of test acknowledgement processes arose from
the data. Firstly, simple acknowledgements (stamp and sign pre eRA or click “OK” post)
where the physician reviewed and acknowledged reports with no reference to further
information or actions to contact the patient or GP. Secondly, more complex cases where
physicians referred to the EMR for further information before acknowledging the report.
14
Thirdly, cases where the physician referred to other information sources (information from
local intranet, internet, paper record, asking colleague) before acknowledging result reports,
and fourthly instances when contact with patient or GP was performed prior to
acknowledgement. Individual acknowledgements that required two different types of action
(e.g., other information required and GP contact) were included in both categories.
The majority of acknowledgements were radiology test results (58% post, 89% post) and
classified as simple (84% pre, 77% post) (Table 2). The percentage of acknowledgements for
which additional information was sought from the EMR was higher in the post phase (20%)
compared to the pre phase (12%).
15
Table 2 Test acknowledgement time pre and post implementation of an electronic results
acknowledgement system.
Result type Acknowledgement type Number of
Acknowledgements
Time per
Acknowledgement
Change in
Acknowledgement time
(Pre vs. Post)
Pre Post Mean
ratio
Pre (sec) Post (sec) 95% CI p-value
Microbiology Simple (Stamp and sign/Click
'OK')
158 70 1.70 5.42 9.21 1.15, 2.51 0.01*
Additional information
required from EMR
27 17 0.87 81.47 71.08 0.52, 1.49 0.62
Other additional information
required
0 1 - - - - -
Contact with patient or GP
required
12 1 1.24 21.65 26.95 0.27, 5.82 0.78
Radiology Simple (Stamp and sign/Click
'OK')
233 546 1.29 5.73 7.37 1.03, 1.61 0.03*
Additional information
required from EMR
28 140 0.84 49.45 41.64 0.62, 1.14 0.27
Other additional information
required
6 12 0.91 18.76 17.04 0.48, 1.73 0.77
Contact with patient or GP
required
2 15 0.39 18.07 7.05 0.14, 1.06 0.07
EMR= Electronic Medical Record; *p<0.05. Acknowledgments that required two actions are registered in both categories.
16
There was a borderline significant 46% increase overall in acknowledgement time for
microbiology test results (95% CI 0.99, 2.17; p=0.06) and a non-significant 10% increase for
radiology test results (95% CI 0.87, 1.39; p=0.41) post eRA. These increases were driven by
simple acknowledgements (the most frequent type) taking significantly longer on average with
the eRA. In contrast, many of the less frequent acknowledgement types showed a non-
significant decrease in acknowledgement time after introduction of the eRA (see Table 2). It is
also important to note that physician documentation of follow-up actions for abnormal test
results in the paper medical record in the pre-period may not have been captured if the record
was requested from another location in the hospital and arrived outside of observation time
frames.
A crude overall comparison indicated a significant difference in acknowledgement times
between participants (F=20.9, p<0.01). When taking into account test type, type of
acknowledgement, presence of interruption or multitasking, time of day and day of week, the
estimated mean acknowledgement times became more homogeneous between individuals, but
the differences were still significant. Acknowledgment of radiology results took significantly
longer (p<0.01) if the clinician was either interrupted or multitasking, however there was no
evidence these factors affected microbiology results. For day of the week and time of day no
significant effect was observed for either type of result.
4 DISCUSSION
This study employed both qualitative and quantitative methods to examine the impact of a new
eRA system on the work processes of senior emergency physicians involved in the
management of microbiology and radiology test results for patients discharged from the ED.
The qualitative findings revealed that the eRA system led to changes in the way that results
17
were collated, stored, documented and communicated, with important implications for patient
safety. The addition of the Comment field and Reminder email functionality centralised
documentation in the EMR and minimised the risk of results not being followed up. All
returned test results were stored in the eRA system where they remained until physician
acknowledgement took place. This ensured all test results were reviewed and signed off in a
timely manner and facilitates escalation procedures for unactioned results.
The increase in the amount of time spent using the eRA system to access additional clinical
information likely reflects the greater accessibility of this information in an integrated
electronic system. The increased contextualisation of these results provided by greater access
to clinical information during the test result endorsement process may improve the quality of
the decision-making process, particularly in cases where the senior clinician may not have been
initially involved in the clinical care of the patient. The new system also improved the
consistency of practice between physicians, particularly in the way abnormal results were
documented for discharged patients.
In terms of efficiency, although there was a non-significant increase in the average time taken
to acknowledge results in the post- period, most types of acknowledgements other than the
simple type took less time to complete. The main driver of the overall relative increase in
acknowledgement time was the increase in the time taken for simple acknowledgements.
However, in absolute terms this change was of the order of two to four seconds per
acknowledgement on average. This is a comparable duration to the screen loading time for
each result in the eRA and could potentially be mitigated by improved computer processing
speed. Within the previous paper-based system, ED physicians were required to collect and
manually identify reports for discharged patients, this could be completed with greater
efficiency with the click of a button in the new eRA Message Centre. The eRA system also
eliminated the need for administrative staff to file print outs in the paper record following each
18
acknowledgement session. Despite changes in time taken to complete acknowledgements,
physicians did not indicate a perceived loss of efficiency.
The findings from this study address a major problem related to accounting for test results that
are still pending at patient discharge [20]. Very few hospitals have managed to utilise health IT
systems for managing this problem, and physicians are generally expected to sift through
discharge summaries, rely on direct communication, or use other local strategies or individual
approaches to gather the information they need [20]. In 2005 Roy et al. reported the results of a
cross-sectional study using 2644 consecutive patients across two US tertiary care academic
hospitals, showing that 41% of patients left the hospital before all test results were reported,
and that 9.4% of these results have required further action [21].
Test result follow-up processes can vary widely between sites, even when the sites share a
common EMR [22]. For instance, Laxmisan et al.’s examination of an EMR-based intervention
across two US Veterans Health Administration sites in 2007 identified differences in local
processes and workflow features, which led them to conclude that health IT is likely to have an
important impact on the methods and procedures used to manage test results [23].
A cross-sectional survey of Veterans Affairs primary care practitioners in 2010 assessed the
perceptions of technological factors related to the notification software (e.g., ease-of-use,
content and computer interface) along with social factors (e.g., workflow, organisational
procedures). It found that 56% of respondents reported that notification procedures using the
EMR made it possible for test results to be missed [24]. The authors concluded that the follow-
up of test results can be a problem with or without an EMR [7]. The resolution of this issue lies
beyond just applying alerts and other technologies to transfer data [7], but in devoting careful
attempts aimed at optimising the efficiency and effectiveness of the test management process
[1].
19
Safety and workflow implications from our findings are generally applicable to EDs where
eRA systems are employed, however a number of limitations exist with this study. Although
there was a small number of physicians involved in the performance of the eRA process, our
study captured all participants scheduled to perform the task during our study periods pre and
post eRA system implementation and provided the requisite number of acknowledgements
needed to make a valid statistical comparison of time taken per acknowledgement pre and post
eRA module implementation. The fact that the group of physicians pre and post were not
exactly the same compromised the accuracy of estimated change in acknowledgement time.
However, in a sensitivity analysis where the models were applied only to physicians observed
in both phases, the main estimates were comparable to the results reported above. Also, the
variation in times of day observed in the pre and post periods required explicit modelling of
temporal effects where this could have been avoided by use of consistent observation times.
However, estimates of these effects in the multivariate analyses were not significant.
5 CONCLUSION
The increase in the percentage of acknowledgements where physicians sought additional
information from the EMR suggests that the eRA system was being accessed for a broader
range of activities than was practical previously. Access to relevant clinical information can
improve the quality of the decision making process during endorsement of non-normal results
by a senior clinician who was not involved in the clinical care of that patient. Health IT has the
potential to mitigate the many complex socio-technical factors which contribute to errors in the
test follow-up process, including workflow and communication factors and the lack of
appropriate organisational mechanisms to ensure that critical and clinically significant results
are not missed or lost [25].
20
Summary Table
What is already known on the topic
Electronic test result management systems can support test result management by
tracking pending test results at hospital discharge, delivering result notifications,
and electronically documenting result acknowledgement
Health IT must be designed to complement clinical workflow processes to
mitigate challenges associated with use
What is added to our knowledge
Instances where physicians sought additional clinical information during the test
result endorsement process increased significantly post system implementation
Greater accessibility to clinical information in an integrated electronic result
acknowledgement system increases contextualisation of results and improves
physician decision making during the test result endorsement process
21
Authors contributions
All authors participated in the conceptualisation and drafting of the paper. All authors
contributed to the analysis or interpretation of the data. All authors read and approved the final
manuscript.
Acknowledgements
The authors would like to thank the emergency physicians who consented to be involved in the
study.
Funding
The project was funded by an Australian Research Council Discovery Project Grant
(DP120100297) and NHMRC Program Grant (APP1054146).
Conflict of interest
The authors declare no conflict of interest.
22
REFERENCES
1 National Academies of Science Engineering and Medicine. Improving diagnosis in health
care. Washington DC: The National Acadamies Press; 2015.
2 Committee on Patient Safety and Health Information Technology; Institute of Medicine.
Health IT and Patient Safety: Building Safer Systems for Better Care. Washington DC, USA:
National Academies Press; 2011.
3 Carr S. Health IT and Diagnostic Safety: Promise and Peril. Improve Diagnosis 2015;2(5):1-
4.
4 Ingebrigtsen T, Georgiou A, Clay-Williams R, et al. The impact of clinical leadership on
health information technology adoption: Systematic review. Int J Med Inf. 2014;83(6):393-405.
5 Dalal AK, Poon EG, Karson AS, Gandhi TK, Roy CL. Lessons learned from implementation
of a computerized application for pending tests at hospital discharge. J Hosp Med.
2011;6(1):16-21.
6 Schiff GD, Bates DW. Can electronic clinical documentation help prevent diagnostic errors?
N Engl J Med. 2010;362(12):1066-9.
7 Sittig DF, Singh H. Improving Test Result Follow-up through Electronic Health Records
Requires More than Just an Alert. J Gen Intern Med. 2012:1-3.
8 Hysong SJ, Sawhney MK, Wilson L, et al. Provider management strategies of abnormal test
result alerts: a cognitive task analysis. J Am Med Inform Assoc. 2010;17(1):71-7.
9 Hysong S, Sawhney M, Wilson L, et al. Understanding the management of electronic test
result notifications in the outpatient setting. BMC Med Inform Decis Mak. 2011;11(1):22.
23
10 Georgiou A, Prgomet M, Paoloni R, et al. The Effect of Computerized Provider Order Entry
Systems on Clinical Care and Work Processes in Emergency Departments: A Systematic
Review of the Quantitative Literature. Ann Emerg Med. 2013;61(6):644-53.
11 Georgiou A, Lymer S, Forster M, et al. Lessons learned from the introduction of an
electronic safety net to enhance test result management in an Australian mothers’ hospital. J
Am Med Inform Assoc. 2014;21:1104-8.
12 Carayon P, Karsh BT, Cartmill R, et al. Incorporating Health IT Into Workflow Redesign:
Request for Information Summary Report. Rockville MD, USA: Agency for Healthcare
Research and Quality Publication No. 10-0098-EF; 2010.
13 Georgiou A, Westbrook JI, Braithwaite J. Time matters - a theoretical and empirical
examination of the temporal landscape of a hospital pathology service and the impact of e-
health. Soc Sci Med. 2011;72:1603-10.
14 Singh H, Sittig DF. Measuring and improving patient safety through health information
technology: The Health IT Safety Framework. BMJ Quality & Safety. 2016:226-32.
15 Callen J, Paoloni R, Georgiou A, Prgomet M, Westbrook J. The rate of missed test results in
an Emergency Department. Methods Inf Med. 2010;49(1):37-43.
16 Westbrook JI, Ampt A. Design, application and testing of the Work Observation Method by
Activity Timing (WOMBAT) to measure clinicians' patterns of work and communication. Int J
Med Inform. 2009 Apr;78 Suppl 1:S25-33.
17 Ballermann MA, Shaw NT, Mayes DC, Gibney RT, Westbrook JI. Validation of the Work
Observation Method By Activity Timing (WOMBAT) method of conducting time-motion
24
observations in critical care settings: an observational study. BMC Med Inform Decis Mak.
2011;11:32.
18 Westbrook JI, Creswick NJ, Duffield C, Li L, Dunsmuir WT. Changes in nurses' work
associated with computerised information systems: Opportunities for international comparative
studies using the revised Work Observation Method By Activity Timing (WOMBAT). Nurs
Inform. 2012;2012:448.
19 White SA, Bock C. New capabilities for process modeling in BPMN 2.0. In: Fischer L, ed.
BPMN 20 Handbook Second Edition: Methods, Concepts, Case Studies and Standards in
Business Process Modeling Notation (BPMN). Lighthouse Point, Florida: Future Strategies,
Incorporated; 2011: 17-30.
20 Dalal AK, Roy CL, Poon EG, et al. Impact of an automated email notification system for
results of tests pending at discharge: a cluster-randomized controlled trial. J Am Med Inform
Assoc. 2014;21(3):473-80.
21 Roy CL, Poon EG, Karson AS, et al. Patient safety concerns arising from test results that
return after hospital discharge. Ann Intern Med. 2005;143(2):121-8.
22 Metzger J, Welebob E, Bates DW, Lipsitz S, Classen DC. Mixed results in the safety
performance of computerized physician order entry. Health Aff. 2010;29(4):655-63.
23 Laxmisan A, Hakimzada F, Sayan OR, Green RA, Zhang J, Patel VL. The multitasking
clinician: Decision-making and cognitive demand during and after team handoffs in emergency
care. Int J Med Inf. 2007;76(11-12):801-11.
24 Singh H, Spitzmueller C, Petersen NJ, Sawhney MK, Sittig DF. Information overload and
missed test results in electronic health record–based settings. JAMA Intern Med. 2013:1-3.
25
25 Menon S, Smith MW, Sittig DF, et al. How context affects electronic health record-based
test result follow-up: a mixed-methods evaluation. BMJ Open. 2014;4(11):e005985.
26 Hanna D, Griswold P, Leape L, Bates D. Communicating critical test results: safe practice
recommendations. Jt Comm J Qual Patient Saf. 2005;31(2):68-80.
27 Mitchell R, Morrison S. Safety implications of missed test results for hospitalised patients:
the use of electronic discharge summary systems. BMJ Qual Saf. 2011;20(8):733.
26
APPENDIX A - INTERVIEW GUIDE
Results acknowledgement process before and after the implementation of the electronic results acknowledgement system
Brief introduction about study: We are exploring whether the electronic results acknowledgement system has had an impact on the number of test results not acknowledged. The focus is on microbiology and radiology. We ascertain acknowledgement as documentary evidence of the physician having seen the results (either documented in the eDRS, initial on the test report, written in progress notes). We are including only patients admitted and discharged from the ED.
Question 1
i. How were microbiology results acknowledged prior to the implementation of the electronic RA system? (detailed process of acknowledgement, who acknowledges, how, when, where documentation is done, includes microbiology (preliminary and final results), serology, histopathology and cytology etc.)
ii. How are microbiology results acknowledged now that you have an electronic RA system?
Question 2
i. How were radiology results acknowledged prior to the implementation of the electronic RA system? (includes x-rays, CT and US)
ii. How are radiology results acknowledged now you have an electronic RA system?
27
APPENDIX B - DEFINITIONS OF PHYSICIAN WORK TASK CATEGORIES
What
The type of report the physician is reviewing, or other activities that may take place during the results review process
Radiology
Reports from the radiology department
Microbiology
Reports from the microbiology department
Professional Communication
Any work related discussion with another staff member
Social
Any social or personal activity or discussion (phone calls, tea/personal breaks etc)
In transit
Work related movement between tasks
Sort
Sorting of reports, i.e sorting of admitted patients from ED patients both electronically and manually
Finish
When observer automatically records a new task, not knowing that the previous acknowledgement was the final one of the session
Additional Information
To record sources of additional information required before acknowledging abnormal/borderline reports. Multiple sources of information may be selected
EHR
Information from the patient Electronic Health/Medical Record
Discharge summaries Progress notes Medication charts Other pathology results PACS
Internal Information
Internal information sources on local intranet, or in the hospital
eTG MIMS AMH CIAP Reference books in office
Internet resources such as search engines and other external databases
Paper Record
Patient paper medical record
Colleague
Professional communication about a report/result with another clinician
28
Action
Actions taken by the physician after reviewing a report
Contact Pt
Any attempt at contacting the patient or relative
Contact GP
Any attempt at communicating with the GP via phone or fax
Flag Report
Decision to come back to the report at a later stage
Waiting for another report Wanting to talk to another hospital doctor Waiting for additional information such as
a paper record
Documentation
Any documentation that occurs during the acknowledgement process, including acknowledgement of results
Stamp & Sign
Act of acknowledging reports. Achieved by manually signing and date-stamping, and electronically by clicking a specific button
Red Folder (PRE only)
Details of results that require follow up are manually entered into this folder in the office
Reminder Function (POST only)
Comments section in the eRA where physicians enter follow-up actions/reasons for refusal to acknowledge
Admitted Patients
Used to identify reports for patients who were admitted
PreArrival Form
Electronic form for patients being re-admitted to the ED following an abnormal result. These patients are phoned and asked to return to the ED
Doc in Pt Notes
Any form of documentation in patient’s manual/ electronic medical record
Dictation
Verbal dictation of notes into a dictaphone/voice recognition software
