Embed Size (px)
DESCRIPTION
Dr. March Auerbach presents data on the ability of the JIT simulation evaluation and SBME evaluation to predict success.
Citation preview
S
Marc Auerbach, Todd Chang, Daniel Fein, James Gerard, RenukaMehta, Daniel Scherzer, Jennifer Reid, Glenda Rabe, Martin Pusic,
David Kessler on behalf of the POISE Investigators
Marc Auerbach, MD, MScAssistant Professor of Clinical Pediatrics
Associate Director of Pediatric SimulationYale University School of Medicine, New Haven CT, USA
Co-director of the POISE research network
Conflict of Interest
S On behalf of myself (and any co-presenters of the papers I am responsible for), I declare to my knowledge, there are no conflicts of interest
S The POISE Network is funded by a grant by
Background
S Competence is a developmental process: for each
domain/context there is a spectrum of ability from novice to
mastery
S Gained through deliberate practice and reflection
S Each individual proceeds at a different rate
S Simulation training/assessment can facilitate skills
development to the level required for safe practice
Background
• Trainees vary in the level of supervision required when performing clinical procedures
• There are few objective methods for supervisors to assess trainees procedural skills prior to clinical performance
• Just-in-time simulation-based assessment could provide supervising physicians information on the level of supervision a trainee requires
S Performance is contextual
S What the practitioner is able to do on simulator when observed
S What she does in practice on patient when not observed
S Progression
1. Watching
2. Close supervision
3. Unsupervised
Watching
Unsupervised
Dreyfus, Maslow, Erricson
Independence
Objectives
• To explore the predictive validity of a simulation-based global skills assessment instrument for clinical infant LP success
Methods
S Design: Prospective multicenter study with historical control
S Setting: 21 academic training centers
S Population: Pediatric interns
S Assessment: Just-in-time performance, BabyStapLaerdal©
S Outcome: Success at LP on infant <365 days
Assessment Tool
Prompt = a verbal interjection to either prevent or
correct an error
• Developed via modified Delphi methods over four conference calls
• Construct validation in prior study
Validation
S 60 subjects S 20 beginner < 5 LP (medical students)
S 20 intermediate 10-20 LP (residents)
S 20 expert > 50 LP (faculty/fellows)
S ReliabilityS Overall agreement = ICC = 0.71, 95% CI 0.59 – 0.80 (p =0.000)
S Positive correlation for all paired rater comparisons (0.69 – 0.73, p = 0.000)
S Discriminant validityS GSA tool could reliably discriminate between the 3 groups
S Experts scored the highest, followed respectively by the intermediate and beginner groups (p < 0.05 for all post hoc comparisons)
Methods
SBME
Orientation
Time = 0
2009-
2010
cohort
2010-
2011
cohort
SBME
Methods
SBME
Orientation
Time = 0
2009-
2010
cohort
2010-
2011
cohort
SBME
Methods
SBME
Orientation
Time = 0
JIT
LP Clinical encounter #12009-
2010
cohort
2010-
2011
cohort
SBMELP Clinical encounter #1
Methods
SBME
Orientation
Time = 0
JIT
LP Clinical encounter #1
JIT
2009-
2010
cohort
0 to 6 months
2010-
2011
cohort
SBME*
LP#2,3…
LP#2,3…LP Clinical encounter #1
Methods
SBME
Orientation
Time = 0
JIT
LP Clinical encounter #1
JIT
2009-
2010
cohort
0 to 6 months
2010-
2011
cohort
SBME*
LP#2,3…
LP#2,3…LP Clinical encounter #1
Methods
SBME
Orientation
Time = 0
JIT
LP Clinical encounter #1
JIT
2009-
2010
cohort
0 to 6 months
2010-
2011
cohort
SBME
LP#2,3…
LP#2,3…LP Clinical encounter #1
Results
501 interns enrolled
104 interns enrolled
161 interns report 228
LPs
51 interns report 102 LPs
2010-2011
(SBME + JIT)
2009-2010
(SBME)
Results
45%(102/228)
45%(46/102)60
45
86 37
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Beginner/Competent Proficient/Expert
Simulation Assessment vs Clinical Procedural Success
Success Failure (55%)(41%)
Results
S Odds ratio for success if rated “high” on
simulator = 1.74
S (95%CI 1.01-3.00), p=0.045 (pearson chi square)
Limitations
• Reporting bias
• Assessors not blinded to clinical outcome
• Majority of trainees had minimal procedural experience
Conclusions
• Just-in-time simulation-based competency assessment offers some value in predicting intern’s clinical LP success
• Interns assessed as proficient or expert had significantly higher odds of clinical procedural success
Future directions
• Use of tool to determine level of supervision
• Further validation of tool as part of clinical pathway is needed
Acknowledgements
POISE Study Investigators:
S Akron Children’s Hospital (Holder), AI Dupont (Stryjewski), Cardinal Glennon SLU (Gerard, Scalzo), Children’s Hospital of Boston (Kothari), Children's Hospital of Los Angeles (Keeler, Mody, Ostrom), Children’s Hospital at Montefiore (Avner, Fein), Children's Hospital of New York Presbyterian (Kessler, Pusic, Tilt), Children's Hospital of Pittsburgh (Zuckerbraun, McAninch), Children’s National Medical Center (Zaveri, Chang, Birch, Agrawal, Seelbach), Cohen Children's Medical Center of New York (Rocker, Israel, Bruckner, Sherman), Emory University (Hebbar), Inova-Fairfax (Kou, D'Andrade, Hwang), University of Iowa Children's Hospital (Lindower, Rabe), Mayo Clinic (Arteaga, Matthews), Medical College of Georgia (Mehta, Sharma, Lane), Mount Sinai Medical Center (Paul, Strother), Nationwide Childrens Hospital (Scherzer), NYU Medical Center (Ching, Torch, Foltin, Cleary), Robert Wood Johnson (Pratt), Seattle Children’s Hospital (Cico, Klein, Reid), Tulane Hospital for Children (Keane, Krantz, Petrescu, Washko), University of South Florida (Haubner, Nations), University of Texas Southwestern (Srivastava), Weill Cornell (Shah, Weinberg, O'Malley), Yale University (Auerbach, Kamdar)
