Standard Setting for a Performance-Based Examination for Medical Licensure Sydney M. Smee Medical Council of Canada Presented at the 2005 CLEAR Annual

Standard Setting for a Performance-Based Examination

for Medical LicensureSydney M. SmeeMedical Council of Canada

Presented at the 2005 CLEAR Annual ConferenceSeptember 15-17 Phoenix, Arizona


MCC Qualifying Examination Part II

OSCE format - 12 short stations● 5 or 10 minutes per patient encounter ● Physicians observe and score performance

Required for medical licensure in CanadaPrerequisites● Passed MCCQE Part I (Knowledge & Clinical reasoning)● Completed 12 months of post-graduate clinical training

Pass/Fail criterion-referenced examinationMulti-site administration - twice per yearOverall fail rate 10%-30% Implemented 1992


Why do it?

Requested by licensing authorities, largely in response to two issues:● Increase in complaints, many centered around

communication skills.● Public accountability - OSCE to serve as an

“audit” of training of all candidates seeking licensure in Canada.


Blueprint Considerations

Four domains*History-taking *Patient Interaction *Physical Examination *Management

Multi-disciplinary / multi-system content

Patient demographics

Two formats *5+5 couplets & 10 minute

Each case based on an MCC Objective


Standard for MCCQE Part II

Acceptably competent for entry to independent practiceConjunctive standard● Pass by total score AND ● Pass by minimum number of stations

High performance in a few stations does not compensate for overall poor performanceJust passing enough stations does not compensate for overall poor performance


Translating a Standard to a Pass Mark

Pilot exam: Ebel method● Items rated for relevance and importance● Pass based on most relevant and important items● Failed 40%

First two administrations: Angoff method● Estimated score for the minimally competent candidate ● Pass based on average of estimates per instrument● Pass marks varied more than the test committee liked● Test committee did not like the task

1994: Adopted borderline group method


Physicians as Scorers

Three Assumptions:

Clinicians do not require training to judge candidate behaviour according to checklists for basic clinical skills

Most clinicians can make expert judgments about candidate performance

Being judged by clinicians is vital for a high-stakes examination


Physicians as Standard Setters


Global Rating Question

Did the candidate respond satisfactorily to the needs/problems presented by this patient?

•Borderline Satisfactory•Good•Excellent

•Borderline Unsatisfactory•Unsatisfactory•Inferior


Numbers....

1,000-2,200 candidates per administration

Examiners each observe 16-32 candidates

20-60 examiners per case

Number of candidates identified as borderline per case ranges from 150-500

Collect >99% of data for global rating item


Modified Borderline Group Method

Examiners (content experts) identify borderline candidates based on the 6-point scale

Scores of borderline candidates define performance that “describes” the pass standard

Examiner judgments are translated into a pass mark by taking the mean score for the borderline candidates for each case


Pass Marks by Case Across Exams

Challenge to assess pass marks over multiple administrations● Scoring instruments are revised post-exam● Rating scale items have been revised● Rating scale items have been added to cases

As competency and difficulty of cases changes, so do cut scores


Setting Total Exam Pass Mark

Pass marks for cases are summedAdd one standard error of measure (3.2% )Pass mark falls between 1 to 1.5 SD below mean scoreStation performance is reviewed by Central Examination Committee● Then the standard for the number of stations

passed is set

Standard has been 8/12 since 2000


Outcomes

15,331 candidates became eligible in 2000 – 2005● 6,099 have yet to attempt MCCQE Part II

● 8,514 have passed

● 718 or 7.7% failed

2,243 candidates were eligible prior to 2000 and also took MCCQE Part II in 2000 – 2005● 2,166 have passed

● 77 or 3.4% failed and are likely out of the system

Fail rates do not reflect impact on repeat takers● Focused hundreds of candidates on remediation


Limitation

Current approach is easy to implement but it relies upon ● Large number of standard setters per case● Large number of test takers in borderline group

Smaller numbers would lead to more effort● Increase training of examiners● Impose stricter selection criteria on standard

setters


What’s ahead?

Increasing number of candidates to be assessed each year● Modifications to the administration are needed

• Predictive validity study currently in progress

● Use non-physician examiners?• Which type of cases, who sets standard?

● Add more administrations?• Case development / challenge of piloting content


Medical Council of Canada

Ottawa

Sydney M. Smee, M.Ed.

Manager, MCCQE Part II

www.mcc.ca

http://www.mcc.ca/

Documents

Standard Setting for a Performance-Based Examination for Medical Licensure Sydney M. Smee Medical Council of Canada Presented at the 2005 CLEAR Annual