Upload
saad-chahine
View
47
Download
1
Embed Size (px)
DESCRIPTION
A Novel Approach for Developing Rating Scales for a Practice Ready OSCE
Citation preview
A Novel Approach for Developing Rating Scales for a Practice Ready OSCE
Bruce HolmesDivision of Medical Education, Dalhousie University
Saad ChahineMount Saint Vincent University
Clinician Assessment for Practice Program (CAPP)
• A program of the College of Physicians and Surgeons of Nova Scotia (CPSNS)
• A 3-part program assessing IMG physicians (residents in Canada, without formal Canadian training) for practice readiness.
The CAPP ProgramPart A: Initial assessment• Assessment of competence via OSCE & therapeutics
exam
Part B: 1 year mentorship with a family physician• Defined license for 1 year with performance assessment
Part C: Additional 3 years of defined license untilcertified by The College of Family Physicians
(CCFP)
Purpose of the CAPP OSCE
1. To assess the clinical competence of IMG candidates for readiness to practice.
2. To provide feedback on candidates' performance for their continuing professional development.
• The OSCE is not a “pass or fail” examination, but rather one factor in the decision to grant a defined license to practice in Nova Scotia.
Examining the Constructs of Rating Form?
• Re-design rating form
• Provide rater guidelines on how to be an examiner
• Two online practice videos for raters– Results submitted– Group data reported to compare with individual rating– Personal feedback available
OSCE Competencies
• History Taking• Physical Exam• Communication Skills• Quality of Spoken English• Counselling• Professional Behaviour• Problem Definition & Diagnosis*• Investigation & Management*
• Overall Global*
Example of Elements for Competency
Purpose of Elements“Guided Conceptualization”
• The elements act as a guide to forming a conceptualization of the competency
• Theory the elements help the examiner in self regulatory process to build an
“awareness and management of one’s own thought” (Kuhn & Dean, 2004)
• Provide “standardized” comments as feedback for candidates and Credentials Committee of College
How are Elements Related to Competencies?
• Reviewed the ratings in 2012 OSCE– 36 candidates
– Total of 432 observations
• Each set of elements correlated with competency– e.g., History; Investigation & Management
• Spearman Correlation between the element ratings and the respective competency rating was conducted one at a time
Results• History Taking; Counselling; Investigation &
Management:– High positive correlation for all element ratings
with the respective competency rating (r-values 0.74-0.82)
• Problem Definition and Diagnosis: – High positive correlation (r-values 0.70-0.81) except for
alternate diagnosis (r=0.61)
• Physical Exam – High positive correlation (r-values 0.70-0.75) except for
sensitivity (r=0.61)
Future Analysis for Competencies – CFA
History Taking
Breadth
Relevance
Focus
Sequencing
Chronology
• Include data from 2013 & 2014
• Confirmatory Factor Analysis conducted through Structural Equation Modeling
Take Home Messages
• Strong positive associations between elements and competency rating– Appears to be helpful to guide examiner ratings– Helpful feedback on competencies
• Follow-up with refinement of elements using examiner feedback
• Follow-up with data from June 2013 & 2014 OSCE