Educational Video-based Scenarios and Evaluative InstrumentFor Handoff Communication

Saba Berhie1; Vineet M Arora1; Leora Horwitz2; Mark Saathoff1; Paul Staisiunas1; Jeanne M. Farnan1 1University of Chicago, Chicago, IL; 2Yale University, New Haven ,CT

Background

• Single institution and small sample size• Future work planned at partner institutions

With generous funding from: • AHRQ R03 Grant # 1R03HS018278-01• NIA T35 Grant #5T35AG029795-02

Instrument Development: Handoff CEX

Communication Professionalism Setting

Superior “Check post-transfusion HgB at 2pm. If <10, transfuse 2 U“

No inappropriate comments, not hurried

Satisfactory

“the only to do is check CBC at 2pm”

“I’ve had a rough day, lets be quick. I’ll start with the sickest first, she’s a train wreck”

Un-satisfactory

“Oh yeah, can you check labs?”

“the ER could not intubate to save their life”

Research Funding

Limitations

Workshop Objectives & Structure

Conclusions

• Utilize videos which highlight varying levels of handoff performance.

• Rate and assess each video utilizing Handoff CEX instrument

WORKSHOP OUTLINE• Two 60-minutes workshops offered • Six videos watched and rated independently by faculty participants using Handoff CEX • Post-workshop roundtable held after viewing the videos

• 3-item Handoff CEX shows promise as a reliable and valid tool to assess varying levels of videos depicting handoff performance

• Raters had difficulty distinguishing between satisfactory and superior levels• Correlation between communication and professionalism

• Future work in real-time clinical scenarios

Results• 14 faculty from 2 departments participated• 73 of a possible 90 (82%) handoff observations were captured

• Reliability•Cronbach’s alpha: 0.81•Kendall’s coefficient of concordance: 0.59

• Based on “Mini- CEX” instrument widely used in internal medicine (Norcini,2003)

Peer Evaluation Pilot Testing• Internal Medicine residents at the University of Chicago: 366 handoff observations

• Factor analysis revealed that three domains captured 82% of the variance of overall sign-out quality• Communication, Professionalism and Setting

02

46

810

Unsatisfactory Satisfactory Superior

Communication

p <0.001

•Validity • Test of trend across ordered groups: p<0.001• Two-way ANOVA

• No rater bias; results significant only for difference between “superior” & “unsatisfactory” levels of performance

• July 2011– ACGME sets new limits for resident duty hours

• Unintended consequence is increase in number of handoffs

• Few trainees receive education on effective handoff communication (Horwitz, 2006)

• Lack of validated instruments to measure handoff performance

HOW ACCOMPLISH THIS?

Specific Aims

• To create video-based examples of varying levels of handoff performance for education, adapting the approach in Arora et al MedEd Portal:

“Handoffs: A Typical Day on the Wards”

• To validate an assessment instrument: the Handoff CEX

• 6 scripts developed & reviewed• Changing handoff quality in the 3 domains of performance• De-brief instrument development

VIDEO DEVELOPMENT

MedEd Portal Submission

• Arora V, Farnan J, Paro J, Vidyarthi A, Johnson J. Teaching Video:"Handoffs: A Typical Day on the Wards” MedEdPORTAL; 2011

Figure 1: Faculty Ratings of Communication

02

46

810

Unsatisfactory Superior

Setting

p <0.001

Figure 2: Faculty Ratings of Setting

02

46

810

Unsatisfactory Satisfactory Superior

Professionalism

p <0.001

Figure 3: Faculty Ratings of Professionalism

• Ability to detect level of performance for communication impacted by level of professionalism depicted