Upload
others
View
1
Download
0
Embed Size (px)
Citation preview
Today’s webinar is sponsored by
Wolters Kluwer Health Drug Information Solutions
Medi-Span® and Lexicomp®
Who we are:
Medi-Span- Embedded drug databases used in 60 EMRs, 1,000+ hospitals, and
37,000 retail pharmacies
Lexicomp- Referential drug information used in nearly 2,000 hospitals and by
80,000 mobile users
Dedicated to building innovative and trusted resources in multiple formats for use in any setting,
WKH Drug Information Solutions aims to lessen system-wide costs while helping reduce errors,
assist in improving patient safety and creating practical drug information solutions to increase
workflow efficiency throughout the continuum of care.
Combating Alert Fatigue:
New Best Practices
Confidential and Proprietary - Copyright 2013 Wolters Kluwer Health
Confidential and Proprietary - Copyright 2013 Wolters Kluwer Health
Howard Strasberg Vice President, Medical Informatics
Wolters Kluwer Health – Clinical Solutions
• Actively involved in standards development as a co-chair of the Health
Level Seven (HL7) Clinical Decision Support (CDS) Working Group, which
develops CDS standards in areas such as InfoButtons, order sets, and
decision support services.
• Prior to joining Wolters Kluwer Health in 2003, Howard was CEO of
Skolar, Inc., an online provider of clinical information and "in context"
continuing medical education (CME) for medical professionals.
• Howard received his MD degree from the University of Western Ontario
and his MS degree in Medical Information Sciences from Stanford
University. He is board certified in Family Medicine.
Our Speaker
Confidential and Proprietary - Copyright 2013 Wolters Kluwer Health
2
Important tool to help reduce adverse drug events caused by
medication errors
Types of alerts (examples):
• Drug-Drug Interactions
• Drug-Allergy Conflicts
• Dosing
• Pregnancy/Lactation/Age/Gender Precautions
• Drug-Gene Interactions
Medication Safety Screening
Confidential and Proprietary - Copyright 2013 Wolters Kluwer Health
3
2014 EHR Certification Criteria § 170.314 (a)(2)
• (i) Before a medication order is completed and acted upon during
computerized provider order entry (CPOE), interventions must
automatically and electronically indicate to a user drug-drug and drug-
allergy contraindications based on a patient’s medication list and
medication allergy list.
• (ii) Adjustments.
• (A) Enable the severity level of interventions provided for drug-drug
interaction checks to be adjusted.
• (B) Limit the ability to adjust severity levels to an identified set of
users or available as a system administrative function.
2014 EHR Certification Criteria
Confidential and Proprietary - Copyright 2013 Wolters Kluwer Health
4
Alerts occur frequently.
Dutch study[1] – 9.6% of medication orders resulted in a drug-
drug interaction alert.
Major EMR using Medi-Span data:
• 0.12 drug-drug interaction alerts per order
• 0.095 dose alert per order
• 0.05 allergy alerts per order
• …
• Overall – 0.53 alerts per order
Medication Alerts
[1] Zwart-van Rijkom JE, Uijtendaal EV, ten Berg MJ, van Solinge WW, Egberts AC. Frequency and nature of drug-drug
interactions in a Dutch university hospital. Br J Clin Pharmacol. 2009 Aug;68(2):187-93. doi: 10.1111/j.1365-
2125.2009.03443.x. PubMed PMID: 19694737; PubMed Central PMCID: PMC2767281.
Confidential and Proprietary - Copyright 2013 Wolters Kluwer Health
5
•Alerts can be conceptually divided:
• Clinically significant (signal)
• Not clinically significant (noise)
• Alert fatigue occurs when too many alerts are noise and
providers start to ignore the alerts
• A systematic review found that drug safety alerts are
ignored in 49%-96% of cases (Van Der Sijs. JAMIA 2006 [2])
• Partners study found that 53% of medication-related CDS
alerts were overridden (Nanji et al. JAMIA 2013 [3])
Alert Fatigue
[2] van der Sijs H, Aarts J, Vulto A, Berg M. Overriding of drug safety alerts in computerized physician order entry. J Am Med
Inform Assoc. 2006 Mar-Apr;13(2):138-47. Epub 2005 Dec 15. PubMed PMID: 16357358; PubMed Central PMCID: PMC1447540.
[3] Nanji KC, Slight SP, Seger DL, Cho I, Fiskio JM, Redden LM, Volk LA, Bates DW. Overrides of medication-related clinical
decision support alerts in outpatients. J Am Med Inform Assoc. 2013 Oct 28. doi: 10.1136/amiajnl-2013-001813. [Epub ahead
of print] PubMed PMID: 24166725.
Confidential and Proprietary - Copyright 2013 Wolters Kluwer Health
6
Alert Fatigue
Signal
Noise
Question 1:
Where do you draw the line?
Confidential and Proprietary - Copyright 2013 Wolters Kluwer Health
7
Alert Fatigue
Signal
Noise
Precision = 18/28 = 0.64
Recall = 18/32 = 0.56
Confidential and Proprietary - Copyright 2013 Wolters Kluwer Health
8
Alert Fatigue
Signal
Noise
Precision = 6/7 = 0.86
Recall = 6/32 = 0.19
Confidential and Proprietary - Copyright 2013 Wolters Kluwer Health
9
Alert Fatigue
Signal
Noise
Precision = 30/63 = 0.48
Recall = 30/32 = 0.94
Confidential and Proprietary - Copyright 2013 Wolters Kluwer Health
10
Alert Fatigue
Signal
Noise
Question 2:
How should we evaluate provider
responses to alerts?
Confidential and Proprietary - Copyright 2013 Wolters Kluwer Health
11
• McCoy et al provided a framework for evaluating CDS alerts
and responses (JAMIA 2012; 19:346-352 [4])
• Overriding an alert may or may not be an appropriate
provider response; cannot just look at the override rate
Evaluation Framework
Provider
Response
Appropriate
Provider
Response
Inappropriate
Alert Display
Appropriate
(Signal)
Successful alerts Provider non-
adherence
(inappropriate
overrides)
Alert Display
Inappropriate
(Noise)
Justifiable
overrides
Unintended
adverse
consequences
[4] McCoy AB, Waitman LR, Lewis JB, Wright JA, Choma DP, Miller RA, Peterson JF. A framework for evaluating the
appropriateness of clinical decision support alerts and responses. J Am Med Inform Assoc. 2012 May-Jun;19(3):346-52.
Confidential and Proprietary - Copyright 2013 Wolters Kluwer Health
12
• Nanji study in McCoy framework (assuming all alerts that
were not overridden were successful)
• Nanji: 53% of alerts were overridden AND 53% of alert
overrides were appropriate (depicted above as
28%/(28%+25%))
Evaluation Framework
Provider
Response
Appropriate
Provider
Response
Inappropriate
Alert Display
Appropriate
(Signal)
47%
Successful (not
overridden)
25%
Inappropriate
overrides
Alert Display
Inappropriate
(Noise)
28%
Justifiable
overrides
Confidential and Proprietary - Copyright 2013 Wolters Kluwer Health
13
• Evaluation of provider response shouldn’t be limited to the
ordering session; need to establish a time window
• Junior clinicians may initially override an alert, but after discussion
with senior clinicians, they may return to implement the alert’s
recommendation
• Similarly, clinicians may discuss the options with other members of the
team, with specialists and/or consult reference material before
making a management decision
• Clinicians may also override an alert but still implement its
recommendation (e.g. tell the patient not to take Drug A
within 4 hours of Drug B, but proceed with the order)
Evaluation Framework
Confidential and Proprietary - Copyright 2013 Wolters Kluwer Health
14
Alert Fatigue
Signal
Noise
Question 3:
Do people agree on which coins
(alerts) should be gold-colored?
Are some of my gold alerts your
grey alerts, and vice versa?
Confidential and Proprietary - Copyright 2013 Wolters Kluwer Health
15
• Is it possible to achieve consensus on which alerts are clinically
significant?
• ONC/RAND list identified 15 critical DDIs (Phansalkar et al; JAMIA
2012;19:735-743 [5])
• Separately, Pharmacy Quality Alliance (PQA) developed a list of 14
critical DDIs for use in evaluating health plans
• These two lists overlapped only for MAO-I interactions
• Warfarin interactions were on PQA list but not ONC/RAND list
• In our own research, we have found poor agreement among
generalist physicians on which DDIs are clinically significant
• Universal consensus on what constitutes a clinically significant
alert remains elusive
Reducing Alert Fatigue
[5] Phansalkar S, Desai AA, Bell D, Yoshida E, Doole J, Czochanski M, Middleton B, Bates DW. High-priority drug-drug
interactions for use in electronic health records. J Am Med Inform Assoc. 2012 Sep-Oct;19(5):735-43. doi: 10.1136/amiajnl-
2011-000612. Epub 2012 Apr 26. PubMed PMID: 22539083; PubMed Central PMCID: PMC3422823.
Confidential and Proprietary - Copyright 2013 Wolters Kluwer Health
16
• In the absence of a universal consensus, systems must be
tailored to the needs of individual hospitals, departments and
users
• User level
• Don’t show me this alert again
• Don’t show me this alert again for this patient
• Snooze for a period of time
• Hospital level
• Always suppress this alert
• Suppress this alert for physicians, but display it to pharmacists
Reducing Alert Fatigue
Confidential and Proprietary - Copyright 2013 Wolters Kluwer Health
17
Alert Fatigue
Signal
Noise
Question 4:
Does the color assignment depend
on the patient and other context
factors?
Is an alert gold if the patient has
kidney disease but grey otherwise?
Confidential and Proprietary - Copyright 2013 Wolters Kluwer Health
18
• Lack of granularity in CDS knowledge
• Failing to account for other EMR data (patient
demographics, other medications, lab results)
• Relevant information buried in free text without any NLP
capability to extract it (but clinician can read the free text
and conclude that the alert is inappropriate)
Causes of inappropriate alerts
Confidential and Proprietary - Copyright 2013 Wolters Kluwer Health
19
• Consider additional contextual factors
• Top contextual factors identified by subjects in Riedmann et al in an
international Delphi study (JAMIA 2011; 18:760-766 [6])
• Severity of the effect
• Clinical status of the patient
• Probability of the occurrence
• Risk factors of the patient
• Strength of evidence
• For many of these factors, data are limited
• We often don’t have data to provide the probability of occurrence for a
given set of risk factors
• Even if we did, below what level of probability of occurrence is it
considered safe to suppress an alert?
Contextual Factors
[6] Riedmann D, Jung M, Hackl WO,
Ammenwerth E. How to improve the delivery
of medication alerts within computerized
physician order entry systems: an
international Delphi study. J Am Med Inform
Assoc. 2011 Nov-Dec;18(6):760-6.
Confidential and Proprietary - Copyright 2013 Wolters Kluwer Health
20
Alert Fatigue
Signal
Noise
Question 5:
What's the best way to show you
the coins (alerts) in your workflow?
Confidential and Proprietary - Copyright 2013 Wolters Kluwer Health
21
• Seidling et al looked at factors influencing alert acceptance
[taking appropriate action] (JAMIA 2011; 18:479-484 [7])
• Display of the alert (OR 4.75)
• Setting [inpatient/outpatient] (OR 2.63)
• Level of the alert (OR 1.74)
• Display of the alert refers to such characteristics as how
alerts are grouped, where alert appears relative to
medication order, visibility, legibility, color, shape and icon
• In other words, the most important factor in alert
acceptance was the design of the alerting system
Reducing Alert Fatigue
[7] Seidling HM, Phansalkar S, Seger DL, Paterno MD, Shaykevich S, Haefeli WE, Bates DW. Factors influencing alert
acceptance: a novel approach for predicting the success of clinical decision support. J Am Med Inform Assoc. 2011 Jul-
Aug;18(4):479-84.
Confidential and Proprietary - Copyright 2013 Wolters Kluwer Health
22
Horsky et al [8] published some design recommendations:
• Color (e.g. reserve red for severe situations)
• Size (e.g. variable depending on content length)
• Layout (e.g. simple geometry)
• Font (e.g. draw attention to drug names)
• Language (clear and concise, with links to additional information)
• Interruptive vs Non-Interruptive
• Use Non-Interruptive for low severity alerts
Design of Decision Support Interventions
[8] Design of decision support interventions for medication prescribing Jan Horsky, Shobha Phansalkar, Amrita Desai, Douglas Bell, Blackford
Middleton International journal of medical informatics 14 March 2013 (Article in Press DOI: 10.1016/j.ijmedinf.2013.02.003)
Confidential and Proprietary - Copyright 2013 Wolters Kluwer Health
23
Russ et al[9] tested a redesign of medication alerts in the VA
CPRS system using a simulation study.
Changes included:
• Streamlining the information
• Reducing textual information overload
Redesigned alerts had improved usability scores for:
• Overall satisfaction
• Information quality (!)
• Interface quality
Applying Human Factors Principles
[9] Russ AL, Zillich AJ, Melton B, Spina J, Weiner M, Russell SA, McManus MS, Kobylinski A, Doebbeling BN, Hawsey J, Puleo A, Johnson E, Saleem JJ.
Applying Human Factors Principles to Improve Medication Alerts. AMIA Annual Symposium; 2012 Nov 3; Chicago, Illinois.
Confidential and Proprietary - Copyright 2013 Wolters Kluwer Health
24
Duke, Li and Dexter[10]
• Enhance alert display with patient context
• Intervention group – display most recent potassium and creatinine
along with potassium-related alerts
• Control group – standard alerts
• Hypothesis: Adherence rate would be higher in high risk patients
(elevated baseline potassium and/or creatinine)
• Results: No significant difference in adherence rates
• Possible explanations:
• Physicians ignored the alerts entirely
• Physicians did not gauge patient risk accurately
• High risk patients were carefully monitored, so less cause for concern
Context Display
[10] Duke JD, Li X, Dexter P. Adherence to drug-drug interaction alerts in high-risk patients: a trial of context-enhanced alerting. J Am Med Inform
Assoc. 2013 May 1;20(3):494-8. doi: 10.1136/amiajnl-2012-001073. Epub 2012 Nov 17. PubMed PMID: 23161895; PubMed Central PMCID: PMC3628050.
Confidential and Proprietary - Copyright 2013 Wolters Kluwer Health
25
Alert Fatigue
Signal
Noise
Question 6:
What actually happens when coins
(alerts) are shown to users? What
can we learn from the data?
Confidential and Proprietary - Copyright 2013 Wolters Kluwer Health
26
Institutions should study their alert logs to identify areas for
improvement.
Health system - 1354 pregnancy alerts in one week.
• Through simulation, identified a new filter setting that would cut down
the number of alerts by 60%.
• OR rate would still be high, but alerts per order would be reduced.
• EMR issue: precautions for Placenta Previa were triggered for all
pregnant patients, leading to unnecessary alerts.
• Solution: EMR updated its software.
• Workflow issue:
• MMR vaccine – orders were given immediately post-partum, but
nobody told the EMR that the patient was no longer pregnant.
Alert Logs
Confidential and Proprietary - Copyright 2013 Wolters Kluwer Health
27
Pharmacy - 70% of alerts were duplicate therapy alerts.
• Problem: Patient comes in with a new prescription for a drug with 0
refills remaining, but the old prescription is still active generates
duplicate therapy alert.
• Solution: Implement a days allowance (e.g. if patient comes in within
14 days of the end of the old prescription period, suppress the alert).
• Problem: Numerous alerts for patients taking three anti-hypertensive
drugs.
• Solution: Change the allowance for this class of medications (e.g. only
alert if four or more anti-hypertensive drugs).
Alert Logs
Confidential and Proprietary - Copyright 2013 Wolters Kluwer Health
28
Health system
• 89,426 Drug-Disease precaution alerts in 6 months
• 60,600 (68%) were below our recommended filter threshold
• Solution: Implement our recommended filter settings
Alert Logs
Confidential and Proprietary - Copyright 2013 Wolters Kluwer Health
29
Alert Fatigue
Signal
Noise
Question 7:
Is there an alternative to showing
you a bunch of coins (alerts) within
your workflow?
Confidential and Proprietary - Copyright 2013 Wolters Kluwer Health
30
• Don’t show an alert at all, but instead institute surveillance
monitoring for adverse effects
• Alert a provider at the first sign of an adverse effect
• Monitoring could involve vital signs, lab test and other
observations about the patient, some of which may require
NLP analysis of free text notes
Reducing Alert Fatigue
Question 1: Where do you draw the line?
• Strike a balance between precision and recall, but don't ignore
recall. Patient safety is at stake.
• Our research suggest that both precision and recall can
(simultaneously) be above 0.65.
• Don't go for 0.90 precision at the expense of a huge drop in recall
(e.g. recall of 0.20).
Recommendations
31 Confidential and Proprietary - Copyright 2013 Wolters Kluwer Health
Question 2: How should we evaluate provider responses to
alerts?
• Look at the override rate, but realize that it doesn't tell the whole
story.
• Override alert – probably appropriate
• Override alert
• May still have implemented its recommendations (e.g. implement
monitoring) [overridden literally but not functionally]
• May be inappropriate
Recommendations
32 Confidential and Proprietary - Copyright 2013 Wolters Kluwer Health
Question 3: Do people agree on which coins (alerts) should
be gold-colored?
• Make some decisions at the institutional level.
• Allow end users to make some of their own decisions.
• Consider guideposts (e.g. end user cannot suppress highest
severity alerts).
• Don't show me this alert again [for 6 months].
• Don't show me this alert again for this patient [for 6 months].
Recommendations
33 Confidential and Proprietary - Copyright 2013 Wolters Kluwer Health
Question 4: Does the color assignment depend on the patient
and other context factors?
• Use context whenever possible.
• Dose checking – include the indication, patient age, patient weight,
and patient renal function.
• Interaction checking – include the route, patient age, patient gender,
and patient's problem list.
Recommendations
34 Confidential and Proprietary - Copyright 2013 Wolters Kluwer Health
Question 5: What's the best way to show you the coins
(alerts) in your workflow?
• Pay attention to the design of the alerts (color, size, layout, font,
language)
• Distinguish between interruptive (more severe; interrupts workflow)
and informative alerts (less severe; appears on screen but does not
interrupt workflow).
• Consider displaying context information (e.g. recent lab results).
Recommendations
35 Confidential and Proprietary - Copyright 2013 Wolters Kluwer Health
Question 6: What actually happens when coins (alerts) are
shown to users? What can we learn from the data?
• Study your institution's alert data.
• Measure and improve.
• What are the most common alerts being overridden? Why?
Recommendations
36 Confidential and Proprietary - Copyright 2013 Wolters Kluwer Health
Question 7: Is there an alternative to showing you a bunch of
coins (alerts) within your workflow?
• For rare adverse drug events, consider surveillance monitoring
(e.g. Sentri7 from Pharmacy OneSource – also part of Wolters Kluwer
Health) in lieu of prospective alerting.
• Create rules such as:
• If (Drug X AND Drug Y) AND
(AST > 35 OR ALT > 40) THEN
Send Alert to Provider
Recommendations
37 Confidential and Proprietary - Copyright 2013 Wolters Kluwer Health
Q & A
38 Confidential and Proprietary - Copyright 2013 Wolters Kluwer Health
Confidential and Proprietary - Copyright 2013 Wolters Kluwer Health
Heather Ciranna
Marketing Production & Tradeshow Manager
Wolters Kluwer Health
Lexicomp | Medi-Span
For More Information
Follow @Medispan on
Twitter to learn about
upcoming webinars!
Confidential and Proprietary - Copyright 2013 Wolters Kluwer Health
40
Notice
This presentation may not be copied, reproduced, redistributed or otherwise transmitted in any
form or by any means, in whole or in part, without prior written consent from Wolters Kluwer
Health. Any unauthorized use of copyrighted material incorporated within this presentation will
be subject to legal action. The content in this presentation has been prepared strictly for
informational purposes only, and is not intended or offered as legal advice. Although the
information provided is believed to be reliable and accurate, all information is without
warranties of any kind, either express or implied, including but not limited to warranties of the
accuracy, currency, completeness of information, or the suitability of the information for any
particular purpose. Wolters Kluwer Health shall not be liable for any direct, indirect, incidental,
special, consequential, punitive or any other damages arising out of the use of any information
discussed in this presentation. In deciding your organization’s specific technology, business and
legal requirements, you must not rely on the information in this presentation as a substitute
to advice from your own legal counsel.