End of Shift Milestone Evaluation Forms

• Michael S. Beeson, M.D., MBA– Program Director, Akron General

Medical Center• Amber Bradford, D.O.

– Associate Program Director, Akron General Medical Center

• Steven Warrington, M.D.• HAENOW Milestone Study Group

HAENOW?• Hennepin- Danielle Hart, M.D.• Akron General- Beeson, Bradford, Warrington• Einstein- (Beth Israel Albert Einstein College

of Medicine) Saadia Akhtar, M.D.• Nevada- Michael Epter, D.O.• Ohio State- Diane Gorgas, M.D.• Washington- Fiona Gallahue, M.D.

Objectives of Session

• Introduce concept of End of Shift Milestone Evaluation Forms

• Understand advantages/drawbacks of this method

• Introduce elements of reliability and validity of this method

Components of Session

• Milestone assessment basics• Development of End of Shift Milestone Evaluation

Forms• Description of End of Shift Milestone Evaluation

Forms• Viewing of resident-patient interaction• Scoring of 1 of 8 EM1 End of Shift Evaluation Forms• Discussion of this Methodology• Concluding remarks

We Got Milestones, Now What?• Assessment

– Objective rather than subjective

Core Competencies Introduction

• Formative vs Summative assessment

Milestone Assessment

• Focus on Reliability and Validity of methods and tools– Will different faculty rate the same?– Is the setting valid?– Is the assessment instrument valid?– How is bias controlled?

Emergency Medicine is Different

• IM/Surgery rotations with same senior residents and rounding attending(s)

• EM shifts can be with a different attending EVERY shift

• For EM, difficult to do an end of month Milestone evaluation– An attending may have worked with a specific resident

sparingly– Any assessment becomes a subjective global rating scale

General Milestone Assessment Basics

• The farther from patient care (real or simulated) an assessment is made, the more it resembles a subjective global ratings scale

• The closer to direct patient care (real or simulated) assessment is made the more objective

Turning the EM Milestones into Global Rating Scales

• Bias exists from faculty matching PGY level with expected proficiency level

End of Shift Milestone Evaluation Forms

• Struggled with how to make an efficient end of shift evaluation form

• SDOT useful, but could take 20-30 minutes to complete

• Desire to develop a brief end of shift assessment tool– Novel concept- make it useful too!

Milestones to Exclude

• Procedure-based (PC9- PC14)• Medical Knowledge (MK)• Milestones left:

– Milestone Number by Proficiency LevelLevel 1 2 3 4 Total

23 26 40 31 120

Separate Forms for EM1 and EM2-4

• EM1- Use Milestones from Proficiency levels 1-3– 89 Milestones

• EM2-4 Use Milestones from Proficiency levels 2-4– 97 Milestones

• 8 EM1 forms and 8 EM2-4 forms• Proficiency levels of each Milestone are NOT

identified

Sample End of Shift Milestone Form

Sample End of Shift Milestone Form with Anchors

Process

• Milestone evaluation training with individual faculty• A separate form is used on a weekly basis

– Program Coordinator swaps them out weekly• Weekly email of that week’s End of Shift Evaluation

Forms along with their anchors to faculty and residents• Towards end of shift a form is completed by the

attending with the resident• Form is then faxed to Residency Office by ED Unit

Clerk

Let’s Score an Intern

• View video

Additional Case Points

• Intern does not introduce himself to the patient• Intern addresses patient by the wrong name• Intern was 10 minutes late to his shift and was

told to wear his white coat to work the day before• Intern does not perform a head or neck exam but

communicates a normal exam• Intern does not communicate elevated blood

pressure to his attending

Case Points

• Intern initially prescribes Percocet to a patient with a known alcohol problem

• Intern initially wants to place the patient in an air cast splint for a distal fibular fracture

• Intern initially refers the patient to the medicine clinic for follow-up

Score the End of Shift Evaluation Form

Data Input of Each Form

• Each form is faxed to the residency office

• Program Coordinator inputs form into online database

• Demonstration of Form data input

Reports from Data• Demonstration of Resident Report

Issues to be Worked Out

• Better balancing of each form in terms of Proficiency levels

• Which Milestones are just too difficult to evaluate using this method

• Could procedural subcompetencies be evaluated this way?– Airway, Wound Care, US, Central Line Forms?

• Form completion compliance• Buy-in by residents and faculty• Unknown Inter-Rater Reliability

State of This System

• Available now!– www.cordtests.org

• Forms can be downloaded as Word document, put your logo on it, whatever…

• iPad app is nearly completed