Upload
beryl-clarke
View
216
Download
1
Tags:
Embed Size (px)
Citation preview
Simulation based Assessment
Simulation based Assessment
Training the Trainer SeriesOctober, 2014
Training the Trainer SeriesOctober, 2014
Assessment
• Formative
• Summative
Formative Assessment• Assessment for
learning
Summative Assessment
• Assessment of learning
Evaluation Methodology
Bould MD, et al. Assessment of procedural skills in anesthesia. Br J of Anaesth 2009; 103:472-483
Methodology:Global Rating Scale
• Overall performance by categories
• Scores (Likert)
• Drawbacks
Global Rating Scale
Blum RH, et al. Simulation-based assessment to identify critical gaps in safe anesthesia resident performance. Anesthesiology 2014; 120:129-41
Methodology:Checklists
• Delineated actions
• Weighted, timed, cumulative...
• Adverse events
• Drawbacks
Checklists
Schwid, et al. Evaluation of anesthesia residents using mannequin-based simulation, a multi institutional study. Anesthesiology 2002; 97:1434-1444
Methodology:“Time To”
• Timing of critical actions
• Pace
• Motion analysis
Methodology:Survey Data
• Self assessments
• Psychometrics: Pre and Post Testing
Survey Data
Lammers RL, et al. Simulation-based Assessment of Paramedic Pediatric Resuscitation Skills. Prehospital Emergency Care 2009; 13:345-356
Methodology:Biometrics
• “Technometrics?”
• Mannequin physiology
Validity
• The Assessment Method
• The Simulation
Validity:“Face”
• General acceptability of the tool used for the task required
• Does this make sense?
Validity:“Content”
• Does the content match what’s to be assessed?
• Task Analysis
• The Experts
The Delphi Method
• BOGSAAT
Validity:“Construct”
• Does this reflect true performance?
• Experts
• Role of experience
• Surveys
Validity:“Construct”
• Statistical Analysis:
• Discrimination: “D Statistic”
• ANOVA
• “Concurrence”
Tool Validation
Reid J, et al. The Simulation Team Assessment Tool (STAT): Development, reliability and validation. Resuscitation 2012; 83:879-886
Validity:“Criterion”
• Or “Extrapolation”
• Similar results to other performance measures
• “Predictive”
Consistency:Internal
• Requires realism
• Reflects test quality
• Pilot Testing
Consistency:Statistical Analysis
• Do testers do well or poorly on the same items?
• Generalizability: “G statistic”
Reliability:Inter-Participant
• Set-up
• Simulator
• The Subject
Reliability:The Rater
• Inter-rater reliability (external reliability)
• Statistics: IRR, ICC
Reliability:The Rater
• Rater training
• Rater refreshing
• Scoring
Reliability:The Observation
• Real time
• Recording
• Bias
Validated Tools
Cheng A, et al. Designing and Conducting Simulation-Based Research. Pediatrics 2014; 133:1090-1101.
Take Away
• Rigor in realism, rehearsing and piloting
• Mix Checklists and GRS
• Supplement with survey, psychometric, device data, etc
• More tasks with more (recorded) views
Further SourcingSchwid, et al. Evaluation of anesthesia residents using mannequin-based simulation, a multi institutional study. Anesthesiology 2002; 97:1434-1444
Devitt, et al. The validity of performance assessment using simulation. Anesthesiology 2001; 95:36-42
Devitt, et al. Testing internal consistency and construct validity during evaluation of performance in a patient simulator. Anesth Analg 1998; 86:1160-1164
Devitt JH, Kurrek MM, Cohen MM, Fish K, Fish P, Murphy PM, Szalai JP:
Testing the raters: Inter-rater reliability of standardized anaesthesia simulator
performance. Can J Anaesth 1997; 44:924–8
Gaba DM, Howard SK, Flanagan B, Smith BE, Fish KJ, Botney R: Assessment
of clinical performance during simulated crises using both technical and behavioral ratings. Anesthesiology 1998; 89:8-18
Boulet JR, et al. summative assessment in medicine: the promise of simulation for high-stakes evaluation. Acad Emerg Med 2008; 15:1017-1024
Cheng A, et al. Designing and Conducting Simulation-Based Research. Pediatrics 2014; 133:1090-1101.
Furman GE et al. Quality Assurance Best Practices for Simulation-Based Examinations. Qual Assurance Best Practices 2014; 5:226-231
Reid J, et al. The Simulation Team Assessment Tool (STAT): Development, reliability and validation. Resuscitation 2012; 83:879-886
Siassakos D, et al. The management of a simulated emergency: Better teamwork, better performance. Resuscitation 2011; 82:203-206
Irvine S and Martin J. Bridging the gap: from simulation to clinical practice. The Clinical Teacher 2014; 11:94-98
Blum RH, et al. Simulation-based assessment to identify critical gaps in safe anesthesia resident performance. Anesthesiology 2014; 120:129-41
Bould MD, et al. Assessment of procedural skills in anesthesia. Br J of Anaesth 2009; 103:472-483
McKinley RK, et al. Checklists for assessment and certification of clinical procedural skills omit essential competencies: a systematic review. Med Educ 2008; 42:338-349
Lammers RL, et al. Simulation-based Assessment of Paramedic Pediatric Resuscitation Skills. Prehospital Emergency Care 2009; 13:345-356