UK Medical School Finals

Peter McCrorie

Head of the Centre for Medical and Healthcare EducationSt George’s, University of London

Summary of Survey Findings

IntroductionIntroduction

lWhy now?– GMC QABME visits– responsibility to the public– “threat” of National Examination– increased sharing of assessmentslUMAPlUniversity of LondonlHong KonglProgress tests

SurveySurvey

l 12-page questionnaire sent out to ASME contacts at each medical school

l 21 replies (plus 3 “paired” schools)Thank you

l 1 via the webl 1 other awaiting responsel 2 accidentally omitted!

Oh Dear

Content of QuestionnaireContent of Questionnaire

lWritten assessment method used:– MCQ SBA– MCQ T/F– EMQ– SAQ– Short notes– MEQ– Essay– Other

Content of QuestionnaireContent of Questionnaire

lClinical assessment method used:– OSCE– OSLER– Long cases– Mini-CEX– Short cases– DOPS– Other

Content of QuestionnaireContent of QuestionnairelOther assessment methods used:

– Portfolio/Record of Achievement– Portfolio/Reflective diary– Logbook– Attachment report– RITA style assessment– Appraisal– Video assessments– Audits– SSM/SSC reports

Content of QuestionnaireContent of Questionnaire

lTiming of Finalsl Standard setting (set/variable pass mark)lExternal examinerslExaminer training (Internal/External)lCostlNational examl Sharing of assessments

LimitationsLimitations

lNot 100% responsel Some responses less detailed than othersl Possible confusion over what is meant

by Finals e.g. penultimate year assessments – O&G, Pathology

lConfusion over terminologyl Some assessments don’t fit the patternlThe new schools haven’t had Finals yet!

What were the findings?What were the findings?

MCQsMCQs

l 13 Schools use single best answer MCQsl 3 Schools use True/False MCQslAnything between 1 and 2 minutes per QlReflection of style of question?

– shouldn’t all questions relate to clinical vignettes?

lNumber of papers taken ranges from 1-4– general, discipline-based, progress test– total time ranges from 1h to 12h

EMQsEMQs

l 17 Schools use EMQsl 3-5 vignettes per set of optionsl 1-2 minutes per questionlAgain – difference in style of question

may account for thisl 40 – 150 EMQs per paper

Other written assessmentsOther written assessments

l 5 Schools use SAQsl 4 Schools use short notesl 5 Schools use Modified Essay Questionsl 5 Schools use essays, only 3 of which are

traditional in style – others are reports rather than essays (apart from many SSM write-ups)

OSCEsOSCEs

l 20 use OSCEsl Big variation in number of stations and

length of each– from 13 stations to 45 – 5, 6, 7.5, 10, 12, 15, 30 and 45 minutes in length

l Some only use 5 minute stations, others a much wider range

l Are 5 minute stations appropriate at this level?

Comments on Comments on OSCEsOSCEs

l Spread over several dayslChange stations or not?lTell the students what the stations are?lUse only real patients?l Separately assess communication skills?l Sequential testing?lAssess procedures using DOPS?

OSLERsOSLERs, long cases, mini, long cases, mini--CEXsCEXslVariations on a themelUsed mainly for in-course assessment –

being phased out in Finals because of generalisability issues

lMini-CEXs are being gradually introduced – mainly as formative assessments or as hurdle assessments on attachments

lDOPS likewise – currently from 10-38 to be signed off – replace part of OSCE

PortfoliosPortfolios

l Here to stay?l In 1 School this is the ONLY assessment

in the final year (through RITAs)– there are NO examinations

l In a second School, there is a portfolio exemption examination and NO OTHER assessment in the final year (unless a student doesn’t get exempted!)

l Most other Schools use the portfolio formatively

PortfoliosPortfolioslThere are 2 types of portfolio – often

both types are used at the same time– record of achievement – a sort of “Mary

Poppins bag” where evidence of achievement is collected (e.g. logbook, SSM reports)

– reflective diary – a collection of personal reflections about patients seen, critical incidents, lessons learnt, good practice observed etc

lMay be in the form of an e-portfolio

Assessment of PortfolioAssessment of Portfolio

l Not itself assessed – private and personal to student, but students can draw on it for reflective essay which is assessed

l Assessed formatively – inspected, checked for completion

l Assessed summatively – inspected, checked for completion and quizzed on its contents. Sample of entries may be marked and graded

Assessment of attachmentsAssessment of attachmentslNotoriously difficult and unreliablelAssessors don’t often consult colleagues

about a student’s performance (360o)lAssess student against a checklist –

essentially knowledge, skills and attitudes/professionalism

lVariety of “formative” attachment assessments used

lRITAs and appraisals

SSMs/SSCsSSMs/SSCs

lDifferences in badgingl 0-8 in final year; all assessedlAlmost all students get an elective of

between 5 and 10 weeks – median 8wlAssessed variably by report, oral

presentation, tutor sign-off or not at all

TimingTimingl Changing with changes in MMCl Many Schools now have May/June Finals

with only one resit the following year at the same time

l Some still have November/December resitbut this is being phased out

l Some have earlier exams in January to allow a resit before F1

l Some Schools have some written, or all written, or both written and clinical Finals in the penultimate year

Standard settingStandard setting

lMost Schools now use a formal standard-setting process to decide pass marks.

lEbel and Angoff for writtenslMainly Borderline Groups for clinicalslBut does each School come up with the

same pass mark for the same station?

ExternalsExternals

lBetween 3 and 12 (up to 30 if you count penultimate year exams as part of Finals)

lMostly discipline-specificl Some include medical educationistslAbout half use an overall externall In a few Schools the external follows a

cohort through from 1st to final year

Training and costTraining and cost

lMost internal examiners are OSCE-trained; few are trained for writtens

l Few externals are trained – only briefed

lEstimated cost ranges from £8,000 to £61,000 (excluding internal examiner time)

National ExaminationNational Examination

l Very strong views on both sidesl Yes – for “fairer” F1 ranking of studentsl Yes for knowledge; less enthusiasm for

clinicalsl No – stifles curriculum and assessment

innovationsl No – assessment drives learning – students will

only concentrate in passing the licensing exam

SharingSharing

lMuch more positivelBetter alternative to national

examinationlBanks already exist - sharing

already occurs, not only nationally but internationally

SummarySummarylHuge variation in style of assessments in

the UKlAt one extreme, several written papers

and clinicals, at the other, none at alllEven the same assessments are being

used differently and at different timeslOutdated assessment methods

disappearing – essays, T/F MCQs, long cases – and new methods are appearing – mini-CEX, DOPS, portfolios

SummarySummarylAre the assessment methods being used

appropriately?lWhy are there so many variations in

OSCEs, for example?lAre standard setting methods being

applied rigorously?l Should we cut down on the number of

allowable resits – especially in Finals?l Should we bite the bullet and have a

national examination?