Embed Size (px)
Citation preview
“Scoring an Oral Simulation Exam”
Elizabeth A. Witt, Ph.D.American Board of Emergency Medicine
Presented at the 2005 CLEAR Annual ConferenceSeptember 15-17 Phoenix, Arizona
Presented at the 2005 CLEAR Annual ConferenceSeptember 15-17 Phoenix, Arizona
ABEM Certification Process
• Complete residency in Emergency Medicine
• Pass written certification examination
• Pass oral certification examination, a series of simulated patient encounters
Presented at the 2005 CLEAR Annual ConferenceSeptember 15-17 Phoenix, Arizona
Purpose of ABEM’s Oral Certification Examination
• Assess clinical performance
• Test the application of knowledge of Emergency Medicine
Presented at the 2005 CLEAR Annual ConferenceSeptember 15-17 Phoenix, Arizona
Structure of ABEM’s Oral Certification Examination
• 7 simulations based on actual clinical cases– 1 field test simulation– 4 single patient encounters– 2 multiple patient encounters
• One-on-one; 7 different examiners• Examiner introduces each case and may
play role of patient, nurse, consultant, etc.
Presented at the 2005 CLEAR Annual ConferenceSeptember 15-17 Phoenix, Arizona
Presented at the 2005 CLEAR Annual ConferenceSeptember 15-17 Phoenix, Arizona
Scoring
• Done by the examiner during and immediately after the session
Presented at the 2005 CLEAR Annual ConferenceSeptember 15-17 Phoenix, Arizona
Examiner Qualifications
• ABEM diplomate at least 5 years
• Residency trained in EM (ACGME- or RCPSC-approved)
• Actively involved in the practice of clinical Emergency Medicine
• Nominated in writing by current examiner, director, or senior director
Presented at the 2005 CLEAR Annual ConferenceSeptember 15-17 Phoenix, Arizona
Examiner Qualifications, cont.
• Evaluated and recommended by ABEM’s Test Administration Committee
• Appointed by the Board of Directors
• Distinguished for high quality patient care, teaching, research, or leadership
Presented at the 2005 CLEAR Annual ConferenceSeptember 15-17 Phoenix, Arizona
Examiner Training Before Exam
• Focus on standardizing the delivery and scoring of each case
• Demonstrations
• Training video
• Scoring practice with feedback
• Case presentation practice with feedback and coaching
• The “what-ifs”
Presented at the 2005 CLEAR Annual ConferenceSeptember 15-17 Phoenix, Arizona
Examiner Training During Exam
• Observe a real simulation first
• Written materials support each case
• Observed by experienced examiner early
• End of 1st day – group discussion and individual feedback, coaching
• Scoring sheets and notes reviewed by chief examiners
• Ongoing discussions, feedback, mentoring
Presented at the 2005 CLEAR Annual ConferenceSeptember 15-17 Phoenix, Arizona
Standardized Ratings
• 8 performance criteria Scale = 1 to 8
• Critical actions Yes/No
• Dangerous action
Presented at the 2005 CLEAR Annual ConferenceSeptember 15-17 Phoenix, Arizona
Performance Criteria
• Data Acquisition• Problem Solving• Patient Management• Resource Utilization• Health Care Provided (Outcome)• Interpersonal Relations• Comprehension of Pathophysiology• Clinical Competence (Overall)
Presented at the 2005 CLEAR Annual ConferenceSeptember 15-17 Phoenix, Arizona
Presented at the 2005 CLEAR Annual ConferenceSeptember 15-17 Phoenix, Arizona
Presented at the 2005 CLEAR Annual ConferenceSeptember 15-17 Phoenix, Arizona
Final Score and Pass/Fail
• Only performance criteria ratings are used
• Two ways to pass:
1. Grand mean of all performance criteria scores > 5.75
2. Case score = mean of performance criteria for each case.
Highest and lowest case scores are averaged.
If the hi-lo average AND all of the remaining case scores > 5.0, pass
Presented at the 2005 CLEAR Annual ConferenceSeptember 15-17 Phoenix, Arizona
Example
1. Grand Mean Standard
Sum of performance criteria ratings = 375 Number of ratings (8x4) + (18x2) = 68 Grand mean = 375/68 = 5.51
5.51 > 5.75?
NO → Fail
Presented at the 2005 CLEAR Annual ConferenceSeptember 15-17 Phoenix, Arizona
Example
2. Case Score Average (High-Low) Standard
Mean of performance ratings for each case = 4.754.75, 5.235.23, 5.425.42, 5.755.75, 5.835.83, 6.086.08
High-Low Mean = (4.75+6.08)/2 = 5.425.42
Are the figures in goldgold all > 5.00?
Yes → Pass
Presented at the 2005 CLEAR Annual ConferenceSeptember 15-17 Phoenix, Arizona
Why Two Ways to Pass?
• Standard setting study + Bayesian procedures → 5.75
• BUT a 5+ is “acceptable performance”
• AND there is potential for measurement error, fluke, etc.
• So
55 6655 55 55 44 = Pass
5
Presented at the 2005 CLEAR Annual ConferenceSeptember 15-17 Phoenix, Arizona
A High Quality Exam
• Interrater Reliability 97% agreement on Critical Actions 95% agreement on “Acceptable/Unacceptable” 94% of all performance criteria ratings within 1 point
• Discriminant Validity 1. Residency trained physicians 2. Physicians not trained via residency 3. Residents beginning 2nd year of residency 4. 4th year medical students
• Correlation with written MC exam = .77• Predictive Validity
Oral exam predicts performance better than written exam does
Presented at the 2005 CLEAR Annual ConferenceSeptember 15-17 Phoenix, Arizona
Benefits of Scoring Procedure
• Stable pass rates
• High involvement of EM community
• Checks and balances
• Standardized, yet flexible assessment
Presented at the 2005 CLEAR Annual ConferenceSeptember 15-17 Phoenix, Arizona
Speaker Contact Information
Elizabeth A. Witt, Ph.D.
American Board of Emergency Medicine
East Lansing, MI
517-332-4800
www.ABEM.org
