Implementation and evaluation of a standardized briefing checklist at shift handoff in a pediatric

emergency department

Paul C. Mullan1, MD MPHSartaj Alam2, PhD

Charles G. Macias2, MD MPHDeborah Hsu2, MD Med

Binita Patel2, MD

1Children’s National Medical Center – George Washington School of Medicine

2Texas Children’s Hospital – Baylor College of Medicine

Disclosure

• No financial disclosures or conflicts of interest from any study authors

Background

• High error rates with serious consequences occur in Emergency Departments1

• Ineffective communication was a root cause in 82% of sentinel events2

• National Patient Safety Goal 2011-2012: – Improve the effectiveness of communication

among caregivers3

1To Err is Human, IOM, 2000 2Joint Commission (2010 data) 3Joint Commission, 2011.

Background

• ED handoff risk factors for error:– Interruptions, fatigue, lack of pre-determined teams

• ED Handoff adverse outcomes:– Delays in care and disposition1

– Increased medico-legal liabilities2

• Survey of EM and PEM fellowship directors3

– 72% agreed that a standardized sign-out system would improve communication and reduce errors

1Smith D, 2011. 2Kachalia A, 2007. 3Sinha M, 2007.

ED Handoff Components

1. Patient information2. Situational awareness information– Resources, challenges, safety issues – Goal: anticipate and react more effectively– Potentially easier to standardize

Accomplishing Situational Awareness• Briefing1

– Planning event prior to another event • Explain goals• Get team input• Cover contingencies• Ensure role awareness

• Operating Room:– Surgical Safety Checklist: mortality ↓1.5 to 0.8% (p=0.003)2

– Communication failures ↓3.9 to 1.3 per surgery (p<0.01)3

• ED: large variation in briefing content and practice

1AHRQ, 2009. 2Haynes, 2009. 3Lingard, 2008.

Briefing Checklist Creation

• Best practice guidelines– Checklist, human factors, aviation literature

• Stakeholder focus groups• Staff training and feedback• Periodic modifications per user feedback– 10 versions in study period

Briefing Checklist

Objective

• To describe the implementation and user perceptions of a standardized briefing checklist for improving situational awareness during physician handoff in a pediatric ED

• Pediatric Active Shift Signout in Emergency Department: – PASSED Checklist

Briefing Checklist Setting

• 3° pediatric level-one trauma center• Academic ED• ~80,000 patients annually• Checklist used in main ED area only– 24 beds, 2 resuscitation beds

• Central desk area with computer availability

Briefing Checklist Users

• Staffing present at 8 AM & 8 PM checklist usage: – PEM providers: attendings (1-3), fellows (1-3)– Pediatric & EM Residents (4-8)– Medical students (0-4)– Charge nurses (1-2)– Respiratory therapists (0-2)

Methods

• Part 1– Quantitative observational study of checklist

process measures• Part 2– Qualitative perception survey by checklist users

Part 1: Quantitative Observational Study of Checklist Process Measures

• Usage rate (if any items checked)• Completion rate (≥80% of items checked)• Timing • Identification of potential safety events

1 2 3 4 5 6 7 8 9 10 11 120102030405060708090

100PASSED Checklists Monthly Performance

Completion RateUsage Rate

Month Post-Implementation of Checklist

Perc

enta

ge

Checklist & Handoff Performance5/6/2011 – 5/5/2012 (n=732)

Mean checkout duration of 18 minutes(Checklist + patient information)

Mean of 14 patients per handoff

Mean of 82 seconds/patient

Potential Safety EventsPotential Safety Event Number of checklists

with the eventICU-level sick patient in ED identified 48%Broken/missing equipment identified 46%Doctor or nurse staffing shortage 21%High acuity patients roomed but unseen by doctor 16%Sick patients in transport identified & discussed 16%Resident’s first shift in ED 13%Questions identified at end of handoff 10%

84% of checklists: ≥1 Event 24% of checklists: ≥3 Events

No significant change in mean time per patient handoff if 0, 1, 2, 3, or 4 potential safety events

Part 2: Qualitative Perception survey by checklist users

• 1 year post-implementation• Population: user experience in the ED pre- and post-

checklist– PEM attendings, PEM fellows, peds residents, charge nurses

• Online survey development– Face validity focus groups, pilot tested, clinical sensibility tool

testing, intra-rater reliability, Perceived contributions to 3 domains:

– Situational awareness– Institute of Medicine quality domains– Usability

• 3 reminder emails over a two-week period

Perception SurveyRole Number

RespondedNumber Invited

Percentage Responded

Pediatric residents 35 39 90%PEM fellows 10 11 91%PEM attendings 16 19 84%Charge nurses 20 25 80%ALL 81 94 86%

Test-retest intra-rater reliability: Kappa of 0.62Cronbach’s alpha of 0.70

Excluded: 2 of 94 (2%) were partial respondents

Survey Results:Situational Awareness Domain

Question:What percentage of the time are you aware of _____ by the end of team

checkout?

All users Median(n=81)

ResidentsMedian

(n=35)

FellowMedian

(n=10)

AttendingMedian

(n=16)

ChargeNurse

Median(n=20)

Pre Post Pre Post Pre Post Pre Post Pre Post

…staffing limitations? 2 5* 1 4* 1.5 5* 2 5* 3 5*

…critically ill (ICU-sick) patients in the ED 4 5* 3 5* 3 5* 4 5* 4 5*

…the names of all your colleagues 3 5* 3 5* 3 4* 3 5* 2 5*

…broken or missing equipment 3 5* 3 5* 2 4.5* 2 5* 3 4*

*p<0.05

1: 0-20% 2: 21-40% 3: 41-60% 4: 61-80% 5: 81-100%

Survey Results:Institute of Medicine Quality Domains

Question:The team checkout

promotes ___ in the ED

All users Median(n=81)

ResidentsMedian

(n=35)

FellowMedian

(n=10)

AttendingMedian

(n=16)

ChargeNurse

Median(n=20)

Pre Post Pre Post Pre Post Pre Post Pre PostSafety 3 4* 3 4* 3.5 4.5* 4 4* 3 4* Timeliness 3 4* 3 4 3 4 3 4* 3 4* Efficiency 3 4* 3 4* 3 4* 3 4* 2.5 4* Effective care 3 4* 3 4* 3.5 4.5* 4 4* 3 4*Patient-centered care 3 4* 3 4* 4 4 3 3.5* 3 4* Equitable care 3 4* 3 3* 3.5 4* 3 3 3 4*

*p<0.05

1: Strongly disagree 2: Disagree 3: Neutral 4: Agree 5: Strongly agree

Survey Results:Usability Domain

Question All users Median(n=81)

ResidentsMedian

(n=35)

FellowMedian

(n=10)

AttendingMedian

(n=16)

ChargeNurse

Median(n=20)

Pre Post Pre Post Pre Post Pre Post Pre PostTeam checkout facilitates good communication

3 4* 4 4* 4 4* 3 4* 2 4*

I am satisfied with the team checkout process 3 4* 3 4* 3 4 3 4* 3 4*The time to complete checkout is too long 3 3 3 4* 3 3 2 2 3 2*

*p<0.05

1: Strongly disagree 2: Disagree 3: Neutral 4: Agree 5: Strongly agree

Limitations

• Recording bias: – Commission & omission errors possible– No verification process

• Contamination bias: – Simultaneous debriefing program

• Unable to causally link briefing checklist to safety outcomes

Conclusions• The PASSED briefing checklist has a high usage and completion

rate• Potential safety events frequently identified• Users perceived that the briefing checklist improved team

checkout – Situational awareness– Communication– Quality of care

• Future studies– Standardized patient specific handoff information– Patient outcomes

Appreciation

• Fellows, attendings, residents, & charge nurses of Texas Children’s Hospital

• TCH section leadership– Joan Shook, Paul Sirbaugh

• Survey development helpers• Presentation review:

– Binita Patel, Jim Chamberlain

• Study implementers– Binita Patel– Sartaj Alam – Charles Macias– Deborah Hsu

References• Agency for Healthcare Research and Quality. TeamSTEPPS: Instructor guide. Accessed 2011;2012 • Agency for Healthcare Research and Quality. Briefs and Huddles Toolkit Overview. Aug 2009. • Cheung DS, Kelly JJ, Beach C, et al. Improving handoffs in the emergency department. Ann Emerg Med.

2010;55:171-80. • Haynes AB, Weiser TG, Berry WR. A surgical safety checklist to reduce morbidity and mortality in a global

population. New Engl J Med. 2009;360;5: 491-499.• Kachalia A, Gandhi TK, Puopolo AL, et al. Missed and delayed diagnoses in the emergency department: A

study of closed malpractice claims from 4 liability insurers. Ann Emerg Med. 2007;49:196-205. • Kohn, L., Corrigan, J., Donaldson, M., Committee on Quality of Health Care in America, Institute of

Medicine, eds. To err is human - building a safer health system. 2000• Lingard L, Regehr G, Orser B, et al. Evaluation of a preoperative checklist and team briefing among

surgeons, nurses, and anaesthesiologists to reduce failures in communication. Arch Surg 2008;143:12–17.• Sinha M, Shriki J, Salness R, Blackburn PA. Need for standardized sign-out in the emergency department: A

survey of emergency medicine residency and pediatric emergency medicine fellowship program directors. Acad Emerg Med. 2007;14:192-6.

• Smith D, Burris JW, Mahmoud G, Guldner G. Residents' self-perceived errors in transitions of care in the emergency department. J Grad Med Educ. 2011;3:37-40.

• The Joint Commission. Sentinel event data - root causes by event type 2004 - 1Q 2012. April 2012;2012 • The Joint Commission: National Patient Safety Goals. Accessed 4/25/2013: http://

www.jointcommission.org/assets/1/18/2011-2012_npsg_presentation_final_8-4-11.pdf

Conclusions• The PASSED briefing checklist has a high usage and

completion rate• Potential safety events frequently identified• Users perceived that the briefing checklist improved team

checkout – Situational awareness– Communication– Quality of care

• Future studies on patient outcomes needed

Questions?

End of presentation

• To improve situational awareness and quality of care delivered to ED patients, we used QI methodology to develop and implement a briefing checklist for use at ED physician handoff rounds. We hoped that our checklist used at handoff would provide an additional protective component of safety that might prevent adverse events (35). The PASSED briefing checklist was adopted by users in the ED as evidenced by its high rate of use. In the vast majority of handoffs, the checklist identified safety components that might not have been discussed otherwise by the team. As a balance measure, the identification of safety components during the briefing checklist did not add to the average time required to handoff patients. Our average patient handoff duration of 82 seconds per patient is comparable to adult ED setting studies that reported an average of 73-92 seconds (10,22). We were satisfied with the 89% completion rate especially given the multiple changes in checklist content and architecture that occurred during the study period; greater than 80% completion of a checklist has been associated with improved patient outcomes in the surgical literature (27). Our perception survey did demonstrate significant improvements with use of the checklist in both combined and individual user groups in all areas of ED situational awareness and the IOM quality domains of safety, efficiency, and effectiveness. For our primary survey outcome, the combined and individual user groups all agreed that the checklist had helped to improve communication. The only question that had disagreement among user groups related to the appropriate duration of handoff with the new checklist between two groups: pediatric residents and the charge nurses. This might have been due to a general sentiment among residents that any process that adds time to the end of their 12-hour shift is inappropriate. For the charge nurse shifts (and for the majority of PEM attending and fellow shifts), the added handoff process did not add to the length of their shift; these data argue in favor of handoff being integrated into, not added onto, an ED shift. We identified many key components to the successful creation and implementation of a checklist. One early concern from stakeholders was the need to keep the checklist short and relevant. We utilized focus groups, pilot testing before implementation, elicitation of end-user feedback post implementation, and feedback-based modifications to the checklist to keep and gain high acceptance of the checklist among users. From a human factors standpoint, we realized early on that the layout and ordering of checklist items significantly affected completion rates. By placing more checklist items at the start of the checklist, prior to starting the discussion of individual patients, we improved overall compliance. We attribute this finding to team members being conditioned to walking away from rounds at the end of discussing the last ED patient on the tracking board in the pre-checklist implementation system. Finally, in the early implementation phase, creating standard practices helped improve process adoption. The number of shifts that particular PEM attendings work in our main ED varies which can contribute to unfamiliarity with the process. However, our PEM fellows are consistently present in the main ED. By requiring fellows to lead checkout when they are present at handoff, we encouraged development of leadership skills and also helped model and provide consistency to the process. Initially compliance decreased on non-fellow days, but it improved with time as more PEM attendings participated in the new process and accepted it as standard routine. We also realized that national holidays were a high-risk time for low usage of the checklist; of the 16 checklists not used (missing or blank), 8 (50%) were from dates within 3 days of national holidays. Future rapid cycle interventions of this QI project will aim to standardize individual patient handoffs, incorporate formal teaching elements into the handoff, and implement the checklist in non-learner based areas of our ED.

• LIMITATIONS• There were some limitations in the design and results of our study. Because frequency of checklist completion and potential safety events reported were determined by the handoff leaders’ recorded values on

the checklists, it is possible that there could have been under- or over-reporting of some checklist components. One factor limiting this phenomenon was that the checklist was completed in visible proximity to multiple other providers who were often looking at the checklist simultaneously. A video study would be needed to assess for accurate rates of checklist omission and commission error. Also, this study was not able to assess objective impacts on patient outcomes or ED time processes given the retrospective nature of the study design, the lack of pre-intervention data, and multiple other confounders that would interfere in establishing any direct causality by the checklist itself. To partially account for this factor, all survey questions were given to subjects with ED experience both before and after the checklist implementation, and questions were designed to assess for the before-and-after effects attributable to the intervention itself. Whereas the results of our survey were generally very positive, these perception changes could have been influenced by other simultaneous changes that occurred during the time of this study. There were no other programs specifically targeted at the handoff process; however, a communication-oriented program on debriefing team members in situ following pediatric resuscitations was implemented during the study period (36). From a timing standpoint, we were limited in that we captured only the time from handoff start to finish; a more precise method, which was deemed not to be feasible from an end-user acceptability standpoint, would have been to record the duration of checklist completion specifically. Anecdotally, the duration of completing the checklist typically was 1-3 minutes, similar to what has been reported anecdotally elsewhere (9). Lastly, our checklist did not address the standardization of communicating actual patient information during the handoff process. Efforts are currently underway at our institution and within the American Academy of Pediatrics’ Section of Emergency Medicine to address this area.

• CONCLUSION • The handoff of patient care in the ED has the potential for identifying serious safety events. Cognitive aids such as checklists have been studied as a means to improve provider cognition that can be impaired by

internal stress, environmental factors, and human factors (37,38). We have developed and implemented a briefing checklist to be used at ED handoff to improve ED situational awareness and to identify potential safety events. We demonstrated a high rate of identifying these events, and users perceived that the checklist contributed to improved communication along with enhanced efficiency, effectiveness, and safety of care delivery. Future studies are needed to examine the effect of such a briefing checklist on ED throughput processes and patient outcomes.

• 1. Why is this topic important? Handoffs of care at physician shift change in the emergency department (ED) are at high risk for medical errors and potential safety events. A standardized method for improving situational awareness of the ED might improve patient safety.

•2. What does this study attempt to show? This study describes the creation and implementation of a standardized briefing checklist used at ED shift handoff.

•3. What are the key findings? A mean of 1.7 potential safety events were identified at each handoff. Users were surveyed and noted that the checklist promoted better communication, safety, and situational awareness.

• 4. How is patient care impacted?By identifying potential safety events at the start of a shift, checklist users perceived to have better situational awareness of their environment. Future studies are needed to examine the effect on ED throughput processes and patient outcomes.

Framework Analysis of Free-Text CommentsComment Theme % of

Total Example

IOM: Efficient 33 “Create a standardized way to checkout patients” (F)

IOM: Timely 17 “Would like to discuss holdups regarding patient flow” (C)

Positive sentiment 13 “Very effective change” (A)

Education-related 11 “Teaching needs to be integrated into checkout” (A)

IOM: Effective 10 “Signout list keeps everyone aware of EC as a whole” (C)

IOM: Safe 7 “I like that we identify the sickest patients during checkout because the charge nurses are not always aware…” (C)

IOM: Patient-centered 4 “Need a better location… to protect patient confidentiality.” (F)

None 3 “Shaking hands makes them feel welcome” (A)

Negative sentiment 1 “Spirit of checkout has changed from relaying the most pertinent information to checking a series of boxes” (F)

A:Attending. C:Charge Nurse. F: Fellow. IOM: Institute of Medicine.