Report of the SJT pilots Appendix F
Improving Selection to the Foundation Programme Appendix F Report of the SJT pilots
ISFP Final Report Page 245
Report of the SJT pilots Appendix F
ISFP Final Report Page 246
i
DesignandEvaluationofaSituationalJudgmentTestforSelectiontotheFoundationProgramme
FinalReport
ImprovingSelectiontotheFoundationProgramme
WorkPsychologyGroupandtheUniversityofCambridge
ProfessorFionaPattersonVickiArcherDrMireKerrinDrDavidGoodVictoriaCarrLouiseFaulkesHelenStoker
July2011
Report of the SJT pilots Appendix F
ISFP Final Report Page 247
ii
AcknowledgementsThisprojectwasconductedbytheWorkPsychologyGroup incollaborationwiththeUniversityofCambridge on behalf of the Improving Selection to the Foundation Programme (ISFP) ProjectGroup.The ISFPProjectGroup,ChairedbyProfessorPaulONeill, includes representatives fromfifteen organisations: the Academy of Medical Royal Colleges (AoMRC), the British MedicalAssociation (BMA)MedicalStudentsCommittee, theConferenceofPostgraduateMedicalDeans(CoPMED), the fourUKDepartmentsofHealth, theGeneralMedicalCouncil (GMC), theMedicalSchools Council, the Northern Ireland Medical and Dental Training Agency (NIMDTA), NHSEducation forScotland (NES),NHSEmployers (NHSE), theScottishBoard forAcademicMedicine,theUKFoundationProgrammeOffice(UKFPO)andclinicaltutors.TheISFPProjectisfundedbytheDepartmentofHealth.
TheCodirectorsoftheprojecttodevelopaSituationalJudgementTestforISFPareProfessorFionaPattersonandDrDavidGood.ThecoreprojectteamconsistedofProfessorFionaPatterson,VickiArcher,DrMaire Kerrin,Victoria Carr, Louise Faulkes andHelen Stoker in conjunctionwith theMedicalSchoolsCouncilprojectteamincludingProfessorPaulONeill,ProfessorTonyWeetman,DrKatiePettySaphon,DrDenisShaughnessy,SiobhanFitzpatrick,AmyStringerandOliverWatson.
Report of the SJT pilots Appendix F
ISFP Final Report Page 248
iii
Contents1. ExecutiveSummary.......................................................................................................................6
2 Introduction..................................................................................................................................9
2.1 PurposeandStructureoftheReport....................................................................................9
2.2 Background...........................................................................................................................9
2.3 ExpectationsandResponsibilities.........................................................................................9
PartOne:ItemDevelopment&Review.............................................................................................11
3 InitialDesignofSJTSpecification................................................................................................11
3.1 SJTLiteratureReview..........................................................................................................11
3.2 MicropilotReviews.............................................................................................................12
3.3 SummaryofSJTDesignSpecification.................................................................................18
4 ItemDevelopment&Review......................................................................................................20
4.1 EvaluationCriteria..............................................................................................................20
4.2 ProcessOverview................................................................................................................20
4.3 ItemWritingWorkshops.....................................................................................................21
4.4 InitialItemReview..............................................................................................................29
4.5 LeadItemReview................................................................................................................29
4.6 FocusGroups......................................................................................................................30
4.7 ConcordancePanel.............................................................................................................34
4.8 TestConstruction................................................................................................................35
PartTwo:PilotAnalysis&Evaluation.................................................................................................38
5 PilotsOverview...........................................................................................................................38
5.1 PurposeandSpecificationsofthePilots.............................................................................38
5.2 DevelopmentofPilotDocumentation................................................................................39
5.3 PilotAdministration............................................................................................................39
5.4 EvaluationOverview...........................................................................................................40
5.5 ScoringConvention.............................................................................................................41
6 InitialPilot..................................................................................................................................43
6.1 PilotTestSpecification........................................................................................................43
6.2 Sample................................................................................................................................43
6.3 OverallSummaryofResults................................................................................................47
6.4 TestLevelAnalysis..............................................................................................................47
Report of the SJT pilots Appendix F
ISFP Final Report Page 249
iv
6.5 ItemLevelAnalysis..............................................................................................................50
6.6 GroupDifferences...............................................................................................................52
6.7 ConstructValidity................................................................................................................54
6.8 ParticipantFeedback..........................................................................................................54
6.9 ConclusionsandRecommendationsfromInitialPilot........................................................56
7 LargescalePilot..........................................................................................................................59
7.1 PilotTestSpecification........................................................................................................59
7.2 Sample................................................................................................................................60
7.3 OverallSummaryofResults................................................................................................69
7.4 TestLevelAnalysis..............................................................................................................70
7.5 ItemLevelAnalysis..............................................................................................................74
7.6 GroupDifferences...............................................................................................................75
7.7 ComputerVersusPaperTests.............................................................................................78
7.8 ConstructValidity................................................................................................................79
7.9 CriterionrelatedValidity....................................................................................................82
7.10 ParticipantEvaluation.........................................................................................................85
7.11 ConclusionsfromtheLargeScalePilot...............................................................................88
PartThree:Conclusions&Recommendations...................................................................................91
8 Conclusions.................................................................................................................................91
9. Recommendations......................................................................................................................91
9.1 Overview.............................................................................................................................91
9.2 TestSpecification................................................................................................................91
9.3 Analysis&Evaluation..........................................................................................................93
9.4 PrinciplesofStandardSetting.............................................................................................93
9.5 PrinciplesofTestEquating..................................................................................................94
9.6 ItemDevelopmentMethodology.....................................................................
