Improving Selection to the Foundation Programme …...design and operational (live) test specification. The pilot also provided the opportunity to continue to pilot administrative

Report of the SJT pilots Appendix F

Improving Selection to the Foundation Programme Appendix F Report of the SJT pilots

ISFP Final Report Page 245

i

Design and Evaluation of a Situational Judgment Test for Selection to the Foundation Programme

Final Report

Improving Selection to the Foundation Programme

Work Psychology Group and the University of Cambridge

Professor Fiona Patterson Vicki Archer Dr Máire Kerrin Dr David Good Victoria Carr Louise Faulkes Helen Stoker

July 2011



ii

Acknowledgements This project was conducted by the Work Psychology Group in collaboration with the University of Cambridge on behalf of the Improving Selection to the Foundation Programme (ISFP) Project Group. The ISFP Project Group, Chaired by Professor Paul O’Neill, includes representatives from fifteen organisations: the Academy of Medical Royal Colleges (AoMRC), the British Medical Association (BMA) Medical Students Committee, the Conference of Postgraduate Medical Deans (CoPMED), the four UK Departments of Health, the General Medical Council (GMC), the Medical Schools Council, the Northern Ireland Medical and Dental Training Agency (NIMDTA), NHS Education for Scotland (NES), NHS Employers (NHSE), the Scottish Board for Academic Medicine, the UK Foundation Programme Office (UKFPO) and clinical tutors. The ISFP Project is funded by the Department of Health.

The Co‐directors of the project to develop a Situational Judgement Test for ISFP are Professor Fiona Patterson and Dr David Good. The core project team consisted of Professor Fiona Patterson, Vicki Archer, Dr Maire Kerrin, Victoria Carr, Louise Faulkes and Helen Stoker in conjunction with the Medical Schools Council project team including Professor Paul O’Neill, Professor Tony Weetman, Dr Katie Petty‐Saphon, Dr Denis Shaughnessy, Siobhan Fitzpatrick, Amy Stringer and Oliver Watson.



iii

Contents 1. Executive Summary....................................................................................................................... 6

2 Introduction .................................................................................................................................. 9

2.1 Purpose and Structure of the Report.................................................................................... 9

2.2 Background ........................................................................................................................... 9

2.3 Expectations and Responsibilities ......................................................................................... 9

Part One: Item Development & Review ............................................................................................. 11

3 Initial Design of SJT Specification ................................................................................................ 11

3.1 SJT Literature Review .......................................................................................................... 11

3.2 Micropilot Reviews ............................................................................................................. 12

3.3 Summary of SJT Design Specification ................................................................................. 18

4 Item Development & Review ...................................................................................................... 20

4.1 Evaluation Criteria .............................................................................................................. 20

4.2 Process Overview ................................................................................................................ 20

4.3 Item Writing Workshops ..................................................................................................... 21

4.4 Initial Item Review .............................................................................................................. 29

4.5 Lead Item Review ................................................................................................................ 29

4.6 Focus Groups ...................................................................................................................... 30

4.7 Concordance Panel ............................................................................................................. 34

4.8 Test Construction ................................................................................................................ 35

Part Two: Pilot Analysis & Evaluation ................................................................................................. 38

5 Pilots Overview ........................................................................................................................... 38

5.1 Purpose and Specifications of the Pilots ............................................................................. 38

5.2 Development of Pilot Documentation ................................................................................ 39

5.3 Pilot Administration ............................................................................................................ 39

5.4 Evaluation Overview ........................................................................................................... 40

5.5 Scoring Convention ............................................................................................................. 41

6 Initial Pilot .................................................................................................................................. 43

6.1 Pilot Test Specification ........................................................................................................ 43

6.2 Sample ................................................................................................................................ 43

6.3 Overall Summary of Results ................................................................................................ 47

6.4 Test Level Analysis .............................................................................................................. 47



iv

6.5 Item Level Analysis .............................................................................................................. 50

6.6 Group Differences ............................................................................................................... 52

6.7 Construct Validity ................................................................................................................ 54

6.8 Participant Feedback .......................................................................................................... 54

6.9 Conclusions and Recommendations from Initial Pilot ........................................................ 56

7 Large scale Pilot .......................................................................................................................... 59

7.1 Pilot Test Specification ........................................................................................................ 59

7.2 Sample ................................................................................................................................ 60

7.3 Overall Summary of Results ................................................................................................ 69

7.4 Test Level Analysis .............................................................................................................. 70

7.5 Item Level Analysis .............................................................................................................. 74

7.6 Group Differences ............................................................................................................... 75

7.7 Computer Versus Paper Tests ............................................................................................. 78

7.8 Construct Validity ................................................................................................................ 79

7.9 Criterion‐related Validity .................................................................................................... 82

7.10 Participant Evaluation ......................................................................................................... 85

7.11 Conclusions from the Large Scale Pilot ............................................................................... 88

Part Three: Conclusions & Recommendations ................................................................................... 91

8 Conclusions ................................................................................................................................. 91

9. Recommendations ...................................................................................................................... 91

9.1 Overview ............................................................................................................................. 91

9.2 Test Specification ................................................................................................................ 91

9.3 Analysis & Evaluation .......................................................................................................... 93

9.4 Principles of Standard Setting ............................................................................................. 93

9.5 Principles of Test Equating .................................................................................................. 94

9.6 Item Development Methodology ....................................................................................... 94

9.7 Developing the Item Bank .................................................................................................. 95

9.8 Equality and Diversity ......................................................................................................... 96

10. References .............................................................................................................................. 97

11. Appendices ............................................................................................................................. 99

Appendix A: Additional Technical Detail ......................................................................................... 99

Appendix B: Example SJT Items used as Feedback Questions ...................................................... 107

Appendix C: Item Level Results for Initial Pilot ............................................................................. 108



v

Appendix D: Item Level Results for Large Scale Pilot .................................................................... 110

Appendix E: Peer Review: Professor Chris McManus ................................................................... 113

Appendix F: Peer Review: Professor Val Wass ............................................................................. 114

Appendix G: Response to Quality Assurance Peer Review Summary .......................................... 125



6

1. Executive Summary 1.1 In 2009, the Department of Health in England (DH), on behalf of the four UK health

departments, commissioned the Medical Schools Council (MSC) to lead a cross stakeholder steering group to design, develop and pilot new arrangements for the selection of trainee doctors into the Foundation Programme (FP). The FP is a two‐year generic training programme which forms the bridge between medical school and specialist/general practice training.

1.2 Following a nine month Options Appraisal, the DH accepted that piloting should take place of:

• a Situational Judgement Test (SJT) to replace white space questions as a measure of aptitude

• an Educational Performance Measure (EPM) to replace academic quartiles as a measure of academic performance

This report is concerned with the design, development and evaluation of the SJT through a series of pilot studies.

1.3 A literature review, together with a series of student micropilot reviews and a job analysis, assisted in the development of the initial design specification of the SJT. The initial design specification was used to inform the item writing process and outlined that the test would contain both ranking and multiple choice items, it would target five professional attribute domains based on the Job Analysis (commitment to professionalism, coping with pressure, effective communication, working effectively as part of a team and patient focus), and it would consist of 60 items to be taken within a two hour period.

1.4 Seven item writing workshops were run, with a total of 89 item writers attending the Continuing Professional Development (CPD) accredited training, outlining item writing principles. A total of 453 items were written by the item writers.

1.5 All items went through a thorough review process that included initial review by a psychometrician, lead item review by a clinician and review by Foundation Year 1/2 doctor focus groups, where each item was examined for fairness and realism.

1.6 Concordance panels were conducted to identify a high level of consensus between experts on the item answer keys. Six concordance panels were run, with a total of 60 subject matter experts attending. Those items that exhibited high levels of consensus (200) were included in the pilots. Those items exhibiting low levels of consensus were stored for further review.

1.7 An initial pilot was run between October and November 2010. This pilot was carried out as a proof of concept to determine whether the SJT had sufficient potential to warrant investment in a larger pilot in spring 2011.

1.8 A total of 455 applicants participated in the initial pilot from four UK medical schools. Demographic data indicated that the sample was representative in terms of sex and ethnicity.



7

1.9 The psychometric analysis of the initial pilot indicated that the SJT is a reliable measurement methodology, demonstrating internal reliability of α=0.74. Analysis also showed that the SJT is able to differentiate sufficiently between participants and item‐level analysis showed that the majority of the SJT items worked well.

1.10 Differences in performance on the SJT between males and females and between ethnic groups were found. On average males scored significantly lower than females (p

8

magnitude are typical with many selection methods and examinations 12,13,14. A richer understanding of the implications of the observed groups differences in practice (for sex and ethnicity) is needed as an impetus for future research. Without detailed systematic research in this area, causal factors cannot be reliably identified17.

1.18 In order to examine the construct validityi of the SJT, the participants at one medical school were invited to complete a NEO‐PIR personality measure to examine the associations between scores on the SJT and measures of personality. The correlations suggest that those individuals who have higher activity levels (Extraversion domain), higher openness to values (Openness to Experience domain) and higher levels of achievement striving (Conscientiousness domain) perform better on the SJT than those with lower levels of these facetsii.

1.19 To assist with establishing the criterion related validityiii of the SJT, analysis was carried out using other measures of performance from one of the medical schools. The findings from the predictive validity study indicate that the SJT correlates with all aspects of the examination data. The SJT correlates most highly with the OSCES (a behaviour based exam) and also correlates, to a lesser extent, with the written exam. In broad terms, this supports the proposition that the SJT is an aptitude test for clinical practice (i.e. high fidelity assessment that replicates what trainees will face in FY1), rather than a replication of the knowledge testing (i.e. written exams) completed throughout medical school and is good preliminary evidence of criterion‐related validity. However further longitudinal validation work will allow us to confirm this finding.

1.20 Participant feedback was sought following the test. The majority of participants felt that the content of the test seemed relevant to the Foundation Programme, that the scenario content seemed appropriate for their training level and that the content of the test appeared to be fair.

1.21 Although the number of participants in the large scale pilot was lower than planned, the datasets prove the concept, demonstrate reliability and validity of the new SJT and support the conclusion that an SJT is an appropriate method of selection in this context.

1.22 Based on the evaluation of the SJT, recommendations are provided regarding the SJT design specification as well as regarding item writing methodology. Commentary is also provided regarding the principles of standard setting and test equating.

ii This is establishing that the selection test or procedure is measuring the constructs (or attribute/skill/proficiency) that is claims to be measuring. For example, that the SJT is actually measuring the competencies that underlie successful performance in the job. ii When undertaking large numbers of correlations, it is possible that significant correlations found may be artificially inflated. As such, a conservative approach was undertaken and a Bonferroni correction was carried out. This resulted in there being no significant correlations between overall SJT score and facet levels of personality. iii This is when the outcomes of, or scores on, a selection test or procedure predict measures of performance that are already accessible.



9

2 Introduction

2.1 Purpose and Structure of the Report 2.1.1 The purpose of this report is to provide details of the design, development and evaluation

of a Situational Judgement Test (SJT) as part of the Improving Selection to the Foundation Programme (ISFP) project.

2.1.2 The report is divided into three main parts:

• Part One describes the design of the initial SJT specification and the item development and review process.

• Part Two describes the results and analysis of the pilots that were carried out.

• Part Three provides a detailed outline of the recommended SJT design and operational specification as well as recommendations going forward.

2.2 Background 2.2.1 In 2009, the Department of Health in England (DH), on behalf of the four UK health

departments, commissioned the Medical Schools Council (MSC) to lead a cross stakeholder steering group to design, develop and pilot new arrangements for the selection of trainee doctors into the Foundation Programme (FP). The FP is a two‐year generic training programme which forms the bridge between medical school and specialist/general practice training.

2.2.2 This steering group recommended the pilot of a Situational Judgement Test (SJT) and Educational Performance Measure (EPM) and that these two assessments in combination should be used for selecting applicants and allocating them to foundation schools. The SJT must therefore be developed and validated in accordance with accepted best practice, so that it provides an effective, rigorous and legally defensible method of selection.

2.2.3 The authors were commissioned to develop and validate an SJT for use in selection to the FP, informed by a detailed job analysis of the role of the Foundation Programme doctoriv.

2.3 Expectations and Responsibilities

2.3.1 The authors were responsible for aspects of the work required to produce the SJT deliverables as follows:

• defining the specification for the SJT against the criteria determined through the Job analysis (detailed in section 3.3)

• producing training materials and guidance for medical schools and applicants

• training volunteers and facilitating item writing workshops to produce the necessary numbers of items to meet the SJT specification (detailed in section 4.3)

iv The results of the Job Analysis are detailed in a separate report



10

• reviewing and quality assuring items to ensure they meet the SJT specification (detailed in section 4.4‐4.7)

• proposing the specifications for the pilots, including the number of pilots to be run, the number of medical schools and applicants to participate in each pilot, and the evaluation criteria for the pilots (detailed in section 5.1)

• analysing the data to assess the effectiveness of the tests in relation to the agreed evaluation criteria in line with best practice (detailed in sections 6 and 7)

• producing a report conforming to the agreed format and structure and making recommendations for future work, specifically covering any changes to the SJT specification

2.3.2 The Medical Schools Council was responsible for a number of aspects of the work as follows:

• scheduling of item writing workshops pilots, and the booking and provision of venues and facilities

• recruiting participants for item writing and pilots

• managing communications with participants

• managing the contribution of individual medical schools to the process

• arranging independent quality assurance activities

• gaining any necessary consent from those participating in the pilots

• providing access to other agreed data for assessment purposes

• incentives for participants



11

Part One: Item Development & Review

3 Initial Design of SJT Specification 3.1 SJT Literature Review 3.1.1 Situational Judgement Tests (SJTs) are designed to assess individuals’ judgement regarding

situations encountered in the workplace. Applicants are presented with a set of hypothetical work‐based scenarios and asked to make judgements about possible responses. Applicants’ responses are evaluated against a pre‐determined scoring key to provide a picture of their situational judgement in that particular context. SJT scenarios are based on extensive analysis of the job role, to ensure that test content reflects the most important situations in which to test applicants’ judgement and are concerned with testing non‐cognitive attributes rather than knowledge or practical skills. SJTs have become increasingly popular over the last 20 years and are used mostly in large‐scale selection processes, often at the shortlisting stage, but can also form part of workplace assessment to highlight employee development needs.

3.1.2 A literature review was carried out to assist in the development of the initial SJT design specification and to provide background information about the research evidence available on SJT development and evaluation. The literature review can be found in the Job Analysis report. A summary of the relevant evidence is provided below.

3.1.3 Item stems (questions) can vary in length and detail. Longer and more detailed questions tend to provide more work‐specific information, thus the detail may enhance the importance of work‐specific knowledge. Length may also increase the cognitive loading of a question. McDaniel et al (2001)1 suggests that tests with detailed questions have higher correlations with work performance; however some tests with less detail also had slightly higher validity.

3.1.4 Response instructions can either have a ‘knowledge’ format (what should you do) or a behavioural based format (what would you do). McDaniel et al (2007)2 noted that knowledge response instructions measure maximal performance and behavioural measures typical performance. McDaniel et al’s (2007)2 meta‐analysis showed that knowledge based response instructions tend to be correlated more positively with cognitive ability compared to behavioural instructions, and behavioural response instructions tend to be more positively correlated with personality traits compared to knowledge instructions. However, it should be noted that a correlation with either of these constructs, does not mean that the SJT measures either cognitive ability or personality, rather that there is common variance in the relationship.

3.1.5 One issue to consider with response instructions is faking. McDaniel et al (2007)2 indicate that SJTs with behavioural instructions are saturated with non‐cognitive variance and so are susceptible to self deception and impression management. Knowledge based response instructions have been found to be less prone to faking than behavioural based response instructions 3, 4 and therefore it is recommended that knowledge based response instructions be used where possible so faking becomes a non issue and does not create a



12

moral dilemma5. On a similar vein, behavioural based instructions are thought to favour ‘test wise’ applicants. ‘Low fidelity’ assessments, such as SJTs and interviews, as much as they try, can never truly tap into candidate actual behaviour, rather as to what an applicant decides to represent about themselves. Therefore ‘test wise’ applicants would still answer what they ‘should’ do, where as less ‘test wise’ applicants would answer what they ‘would’ do. As the correct or keyed answer must reflect what an applicant ‘should’ do, this would favour these ‘test wise’ applicants. In high stakes settings such as these, it would seem sensible to minimise faking issues as far as possible.

3.1.6 With regards to criterion‐related validity, knowledge based response instructions have been found to have higher criterion related validity than SJTs with behavioural instructions2 i.e. an applicant demonstrating knowledge of the correct answer has been shown to predict performance in the role. In contrast Ployhart & Erhart (2003)3 found that behavioural tendency showed increased criterion‐related validities and higher test‐retest reliabilities compared to knowledge based instructions.

3.1.7 There are a variety of different response formats that can be used in SJTs; for example, pick best, pick best/worst, pick best three, rate effectiveness and rank options. ‘Effectiveness’ response format algorithms have been found to yield higher reliability and validity than pick best/worst and rank, with ranking yielding the second highest3. However, it has been noted that pick best/worst will be at a disadvantage where reliability is concerned as only two responses are rated.

3.1.8 Weekley (2004)6 reported the findings of their study examining response types. Across four SJTs, rank‐ordered scoring showed a significant improvement in performance prediction in two of the four SJTs. The rank‐order scoring method for the leadership SJT showed validity approximately 1.5 times greater than the most commonly used method (pick the best/worst). Ranking items are also less likely to be prone to coaching, compared to pick best/worst due to the complexity of ranking responses. The choice of response format is likely to reflect the scenario content and the appropriate format to both provide and elicit the information needed.

3.1.9 There are various scoring methods for multiple choice SJTs. They are typically broken down into ‘rational’ and ‘empirical’. Rational scoring of items is based on experts’ judgements concerning the effectiveness of responses or best/worst options. When items are empirically scored, they are administered to a large pilot sample. Items are selected and/or weighted according to evidence that they differentiate between individuals who score at different levels on a criterion variable (e.g. job performance). Options that are scored relatively often by high performing individuals are scored as correct etc7. One study carried

out demonstrated a relatively high correlation between the two scoring methods (r =.72)8.

3.2 Micropilot Reviews 3.2.1 To assist in the development of the initial SJT design specification, a series of ‘micropilots’

(i.e. medical school student reviews) were carried out in several universities across the UK. The aim of the micropilot reviews was to assist in establishing the appropriateness of the



13

SJT design specification including test length, appropriateness of content, response format and response instructions.

3.2.2 The micropilots were also used as an opportunity to seek feedback on the use of SJTs for selection to the FP. For example, do participants feel that an SJT is a fair, relevant and appropriate selection method in this context? The micropilots also gave the opportunity for the researcher to provide information to students on the ISFP project.

3.2.3 A total of ten micropilot review sessions were conducted. (University of Cambridge, two sessions; University of East Anglia, two sessions; The University of Sheffield, three sessions, and The University of Manchester, three sessions). A total of 32 participants attended the micropilots. Participants’ demographic details can be found in Table 1 below.

Sex Male 12

Female 20

Not Stated 0

Age group 25 and under 25

26‐35 7

36‐45 0

46‐55 0

56‐65 0

66 and over 0

Not stated 0

Ethnicity White 15

Black 0

Asian 8

Mixed 2

Chinese 3

Other 3

Not Stated 1 Table 1: Demographic details of micropilot participants

3.2.4 Prior to attending the micropilot review, participants received a briefing document that outlined the background to the project and the aim of the micropilot. Participants were also asked to sign a confidentiality declaration to ensure security of the items reviewed.

3.2.5 The micropilots lasted for between 1.5 and 2.5 hours and each micropilot review session consisted of between three and six participants. During the micropilot reviews, participants were presented with a sample SJT test paper consisting of 16 example items. The items had been developed by an experienced item writer, using scenarios from interviews with Foundation doctors. It should be noted that these items were for illustrative purposes only. The SJT items covered a range of the possible item types and response formats that could be used in SJT design:



14

• Item Type: Eight items were outlined as ‘what would you do’ and eight items were ‘what should you do’

• Response Format: four response formats were used:

rating of effectiveness (rate effectiveness of response on scale of 1 to 5. There are no constraints as to rating each option e.g. each option could be rated 1)

ranking of five options

multiple choice (pick three answers out of eight)

choose best/worst options

3.2.6 Participants were asked to complete the items as if under examination conditions. They were provided with 30 minutes to complete the 16 example items. All participants completed the test within this time. Their responses to these items were not marked as there was no formal scoring key agreed in advance as the items had not gone through the full development and review process.

3.2.7 After completing the sample items, the researcher facilitated a short focus group discussion with the participants where feedback was requested on the SJT items that they had completed. Questions in the focus group covered the test instructions, length of time allowed for the test (e.g. is two hours and 60 questions sufficient or too long?), item format, as well as the relevance of the scenarios used in the item content. In particular, participants were asked if they felt that the content was appropriate for them at the stage in the education pathway (e.g. do you feel able to answer items that used scenarios staged in Foundation Training?). All responses were audio recorded and transcribed.

3.2.8 Table 2 presents the key questions from the micropilot reviews with example quotations.



15

Response format

Which of the four response formats did you like/feel was most appropriate?

“I like Multiple Choice the best, because in real life you end up doing more than one thing at the same time”

“I thought choose best/worst was the best, because there is always one you would do and one you would not, but then perhaps it is too easy...”

“I found ranking confusing, because you could potentially write down the wrong thing when reading across, although I appreciate that it provides you with more information about an individual”

“I think ranking is better for discriminating between people”

“I found the ranking harder. Easy to identify the best and worst but in the middle is much more difficult to decide”

“If you have a ranking then you know exactly what you’re doing. With the rating you are tempted to put the middle ground if you don’t really know, where as with the ranking you don’t have that option which makes it better. It forces you to make a decision rather than sitting on the fence”.

“Ranking is most challenging, but also think it would be the most frustrating to do.”

“Multiple Choice was probably the best one”

“I thought rating was the best in terms of differentiation and making you think”

“Rating effectiveness – if I wasn’t comfortable answering it, I would give it a 3. It’s a bit of a cop out.”

“Choosing the best/worst is so clear cut – not sure how to differentiate candidates”

“I thought the ranking was most appropriate – good way of assessing”

“Ranking and multiple choice are the best in terms of getting us to think about the questions”

“The reason why I liked multiple choice and ranking made sense was because you are seeing a thinking process; you are getting a more global understanding of why that person has thought like that.”

Response instructions

Do you prefer ‘should’ or ‘would’?

“I feel more comfortable answering questions when it asks what you should do, rather than what you would do. With ‘Would’ are they trying to get at what you would do or what the right thing to do is. Was confused when answering the question at what they were trying to get at, and felt was trying to catch me out and didn’t really like that. It is a bit sneaky. Would punishes honesty”



16

“I would feel more comfortable with should. It’s an exam so I am going automatically circle the one I think I should do but not what I might do in the real situation”

“I like the ‘would’ – it puts personality into it”

Appropriateness of context

Do you feel setting the context in FY1 is appropriate?

“I think when you get to 5th year you begin thinking like an FY1 regardless. I didn’t think I couldn’t answer the questions”

“In your final year you are treated like a FY1, quite a lot of your experiences are in your final year which would help you with this”

“Generally yes, I think the level is fine”

“I found myself going back to the experience I have had in the hospital. Having clinical experience helps and is quite important”

Length of test

Is 60 questions in two hours fair/reasonable?

“Yes. I think so”

“Anything between 90 minutes to 2 hours will be fine”

“Yes, could drag it out to 4 hours, you just have to go with your gut feelings”

Feedback on SJTs in general “I think this does seem fairer, because you can get a standardised set of questions and everyone is marked the same. White space is subjective, so in this way I think it is fairer”

“Struggle with these as can’t discuss them...I can talk to you about why I chose one or why I didn’t choose one, but don’t get that option”

“If you are looking at clinical judgement then this is great. A great way to assess by putting you in lots of scenarios – if you don’t have the judgement, then you will fall short so it’s good”

“A lot of people will feel that they demonstrate abilities quite well, but I’m not comfortable demonstrating professionalism, team work etc, on paper, but given the limitations I think it is the best alternative to an interview etc”

Table 2: Summary of micropilot feedback



17

3.2.9 All participants were asked to complete an evaluation form. The participants were asked to rate six questions about the example SJT items on a five point Likert‐type scale (where 1 = Strongly Disagree and 5 = Strongly Agree). A free text box was also provided at the end of the form for them to add any further comments not covered in the focus group.

Respondents (N=32)

Strongly Disagree

Disagree

Neither

Agree

Strongly Agree

The level of difficulty of the SJT items was appropriate for those applying to

the Foundation Programme

The content of the SJT items is relevant to those applying to the Foundation

Programme

I enjoyed taking the SJT today

The SJT is fair method of selection for those applying to the Foundation

Programme

The SJT is an appropriate method of selection for those applying to the

Foundation Programme

The SJT is able to measure the attributes necessary for the Foundation

Programme

Table 3: Micropilot evaluation form responses

3.2.10 The responses from the evaluation show an overall positive response pattern. In relation to the fourth question regarding fairness, the majority of participants agreed that the SJT is a fair method of selection for those applying to the Foundation Programme. However, 37.5% were unsure. This is not a surprising result as although information was provided about the evidence regarding the use of SJTs in selection and how this would apply to the Foundation Programme, some participants may want more information to allow them to form their opinion. Similarly, this same reasoning can be applied to the last two questions. Qualitative feedback from the workshops indicate that these views reflect apprehension that the SJT does not allow an opportunity to justify their answer, and that an interview would be a preferred way of assessing professional attributes.

% % % % %



18

3.3 Summary of SJT Design Specification

3.3.1 Based on information gathered from the literature review and the micropilot reviews, an initial SJT design specification was developed and is outlined below. In particular, evidence from the SJT used for selection to GP training is drawn upon as this is the only SJT that has been used successfully for selection into postgraduate training in medicine. One of the purposes of the pilots was to evaluate the data in relation to these areas to identify if the specification is appropriate for this context and purpose.

3.3.2 The SJT is designed to be staged in the context of the FP, with the applicants assuming the role of a Foundation Year 1 (FY1) doctor. Feedback from the micropilot reviews indicated that participants involved in the micropilots felt this was appropriate.

3.3.3 Based on the evidence from the literature reviews and the micropilot reviews, a knowledge based response instruction (‘what should you do’) was used for the SJT rather than a behavioural based response instruction (‘what would you do’). As discussed in 3.1.4, knowledge based instructions are more appropriate for high stakes selection contexts where faking and coaching are potential threats. Micropilot evaluation indicated that a knowledge based instruction ‘what should you do’ was viewed favourably. Additionally, in the context of professional behaviour and with the GMC putting a high premium on probity, it seems more appropriate to frame the response instruction as 'should'.

3.3.4 Two types of response format were used within the SJT; ranking and multiple choice. The rationale for using two response formats is based on scenario content and subject matter, rather than to enable analysis of separate traits or skills within the SJT, and is informed by experience in designing successful high stakes SJTs in other contexts. For example, the nature of some scenarios and the possible responses to them lend themselves to ranking items (requiring ability to differentiate between singular actions in response to a scenario that vary in appropriateness) whereas some scenarios lend themselves to multiple choice items (where it is necessary to do more than one thing/tackle more than one aspect in response to a scenario). Using both response formats enables a fuller range of item scenarios to be used, rather than forcing scenarios into a less appropriate item type and potentially reducing item effectiveness (e.g. asking applicants to rank three equally correct options). Evaluation of the information from the micropilot reviews indicated that the ranking and the multiple choice response instructions were viewed favourably by most participants and evidence from the literature review also supports their inclusion.

3.3.5 Evidence and previous experience in other contexts9 suggests that approximately two thirds of the items used in the test should have the ranking answer format, and one third should have the multiple choice format. Using a majority of ranking items allows for greater granularity in the data due to the more complex response/scoring format, however ranking items are typically more time‐consuming to complete and are not appropriate for all scenarios, therefore a balance with multiple response items is needed. This structure also follows the validated methodology from the SJT for use in shortlisting to General Practice training posts9 and tends to reflect the proportion of items produced of each type by item writers (see 4.3.22).



19

3.3.6 The SJT items were mapped against five professional attribute domains identified from the FY1 Job Analysis as follows; Commitment to Professionalism, Coping with Pressure, Effective Communication, Patient Focus and Working Effectively as Part of a Team. Although the items are mapped against these domains for item development and test construction purposes, due to the nature of SJTs as encompassing complex, realistic work scenarios, each item does not exclusively measure any single domain and the SJT should not be viewed as assessing five individual constructs or as a ‘blueprint’. The benefits of this are that not only does the complexity of the item increase, but the possibility of a ‘template answer’ being provided by applicants is avoided. The target domains can be seen more as facets relating to the professional attributes required for a Foundation doctor, with the test as a whole providing an overall measure of this group of relevant professional attributes.

3.3.7 Within and across target domains, the item content should be designed to cover a range of relevant topics e.g. dealing with a difficult colleague; being asked to undertake an unfamiliar procedure.

3.3.8 The test is designed to last for two hours and contain 60 items. 60 items was anticipated to be sufficient to cover the five target attribute domains in a sufficiently reliable and broad way without overloading applicants. 60 items also allows sufficient distribution of scores considering the large number (7000+) of applicants who will be taking the test. 60 items is a slightly greater number than similar tests used in other areas of medical selection, but was planned to ensure a sufficient level of test reliability. In this context, the SJT, in combination with the EPM, is the only method of selection into the FP. This is different from where SJTs are used in specialty selection (e.g. UK GP training), where the SJT is used as one of two tests for short listing and successful applicants then go on to further assessments at a Selection Centre. The score on the FP SJT is therefore proportionally more of an applicant’s overall assessment of suitability for the role compared to the GP SJT.

3.3.9 Allowing two minutes per item is consistent with previous evidence and experience which has demonstrated good reliability and validity9. Analysis of the pilot data identifies whether the number of items is sufficient to achieve reliability and whether two hours is an appropriate time, based on analysis of participant reactions.



20

4 Item Development & Review

4.1 Evaluation Criteria

4.1.1 The technical evaluation criteria defined in the project brief for the development of SJT items are outlined below:

• Items must be content valid (i.e. items assess the attributes/skills required for successful performance within the Foundation Year One (FY1) role, and seem realistic in relation to what is expected of a FY1). This will be established through using the target attribute domains to assist with item writing, using Subject Matter Experts to develop the items, and through item review and FY1/2 focus groups.

• Items must be face valid (‘looks right’ to the applicant i.e. that it would be clear that what is being assessed is relevant to the role of a FY1). This will be established through the use of Subject Matter Experts developing the items, item review and FY1/2 focus groups.

• Items must not unfairly discriminate against any one particular group. This will be achieved through careful consideration of equality and diversity issues throughout test design, including analysis of group differences at test and item level.

• Items must be acceptable to stakeholders and applicants and perceived as fair. This will be achieved through consultation with stakeholders and applicants throughout the design and development process, and obtaining applicants’ reactions, and dispelling any misconceptions.

4.2 Process Overview

4.2.1 The SJT item development and review process was held over two phases. Phase One ran between April and October 2010, and Phase Two ran between November and March 2011. The purpose of the second phase was to develop more items for the large scale pilot and undertake further psychometric analysis to be able to make recommendations as to the design and operational (live) test specification. The pilot also provided the opportunity to continue to pilot administrative and logistical arrangements, and to pilot SJT items for use in live selection.

4.2.2 The SJT item development and review process consists of a series of stages. Figure 1 summarises the stages of the SJT development and review process.



21

Item Writing Workshops & Initial Review

FY2 Focus Groups

Lead Item Reviews

Concordance Stage

SJT Design Specification

Literature Review Micropilot Reviews

Pilot

Further Review & Refinement

Figure 1: Item development and review process

4.3 Item Writing Workshops

4.3.1 A total of seven item writing workshops were held between April and December 2010. Five of the workshops were held in Phase One, between April and May and two further workshops were held in Phase Two, between November and December.

4.3.2 Invitations for the item writing workshops were sent via the UKFPO foundation school directors email list, and they were asked to cascade it to relevant colleagues. Invitations were followed up at later date with contacts in the Society for Academic Primary Care (SAPC) and a contact from Psychiatry to cascade the invitation to community clinicians. Invitations were also sent to medical school tutors via the MSC Assessment Alliance.

4.3.3 The person specification for volunteer item writers was as follows:

• familiar with the FY1 role, including responsibilities and tasks undertaken as well as the necessary skills and abilities required for the role

• to have worked, within the last two years, with FY1 doctors

• may have been involved in the process of marking the ‘white space’ questions

• motivated to produce high quality and effective SJT items

• available to commit one day to attend an item writing workshop



22

• willing to commit time to develop 6‐8 SJT items and their scoring keys within a specified time (3‐6 weeks) after the item writing workshop, working closely with another item writer

• willing to respond to feedback during the period of Item review

4.3.4 A total of 89 item writers attended the workshops. The number of item writers at each of the workshops is detailed in the table below.

London (22nd April) 22

Manchester( 4th May) 9

London (13th May) 15

Birmingham (17th May) 12

Glasgow (26th May) 12

London (11th Nov) 11

London (6th Dec) 8 Table 4: Item writing workshops

4.3.5 Previously trained item writers were also invited to attend the second phase of workshops. A total of nine previously trained item writers attended the workshops.

4.3.6 The breakdown of the demographics of the item writers, their job role and primary speciality is provided in the tables below.

Sex Male 53

Female 34

Not Stated 2

Age group 25 and under 0

26‐35 7

36‐45 38

46‐55 26

56‐65 15

66 and over 0

Not stated 3

Ethnicity White 57

Black 0

Asian 20

Mixed 0

Chinese 3

Other 5

Not Stated 4 Table 5: Demographics of item writers



23

Clinical Tutor 6

Clinical/Educational Supervisor

44

Foundation School Director 15

Lay Representative 2

Medical School Director 4

Other 16

Not stated 2

Table 6: Item writers job role

Specialty No. Speciality No.

Anaesthetics 7 Genito‐Urinary Medicine 1

Breast surgery 1 Haematology 1

Diabetes 1 Intensive Care 1

Diabetics 1 Neurology 2

Education 2 Neurosurgery 1

Emergency Medicine 2 Obstetrics & Gynaecology 3

Endocrinology 2 Orthopaedic Surgery 2

Gastroenterology 1 Paediatrics 7

Gastrointestinal Medicine 2 Psychiatry 2

General Medicine 8 Respiratory 1

General Surgery 4 Rheumatology 3

Geriatric Medicine 5 Urology 2

General Practice 10 Not stated 17

Table 7: Item writers’ primary speciality

4.3.7 Information was sent to item writers prior to their attendance of item writing workshops outlining the background to the project and the use of SJTs for selection, the purpose of the workshops, their role on the day and some example SJT items. Item writers were also provided with some initial information about item writing principles and were asked to complete some pre‐work prior to attending the workshops. They were also requested to complete follow on work. On the day, item writers were required to sign a confidentiality and code of conduct form.

4.3.8 Prior to attending the workshop, each item writer was asked to think of at least three possible scenarios or critical incidents that could form the basis of SJT items. They were advised that these should be incidents that involved some dilemma and need for judgement, therefore avoiding straightforward incidents that would elicit text book responses. Item writers were also asked to think about possible responses to the scenarios that should be actions that could plausibly be taken by the applicant in response to the scenario.



24

4.3.9 Item writers were given guidance on the SJT target domains and were advised that the scenarios should be relevant to one of these domains. This ensures that the scenario is relevant to the professional attributes that are expected of a FY1 doctor and that the content of the scenarios maps on to the person specification. However, the nature of SJT items (consisting of a scenario and a number of response options) means that a single SJT item will not exclusively measure one target domain. The content of the scenario and the correct response keys are likely to relate to the targeted domain, but a number of the other response options are likely to tap into one or more of the other SJT target attribute domains (see issues regarding construct validity above). The benefits of this are that not only does the complexity of the item increase, but the possibility of a ‘template answer’ being provided by applicants is avoided.

4.3.10 The SJT target domains which were presented in the pre‐information sent to item writers and at the item writing workshops are outlined in the table below.

Commitment to Professionalism

Displays honesty, integrity and awareness of confidentiality & ethical issues. Is trustworthy and reliable. Demonstrates commitment and enthusiasm for role. Willing to challenge unacceptable behaviour or behaviour that threatens patient safety, when appropriate. Takes responsibility for own actions.

Demonstrates desire and enthusiasm for continued learning, takes responsibility for own development. Willing to learn from others and from experience. Is open and accepting of feedback. Demonstrates a desire and willingness to teach others.

Coping with Pressure Capability to work under pressure and remain resilient. Demonstrates ability to adapt to changing circumstances and manage uncertainty. Remains calm when faced with confrontation. Develops and employs appropriate coping strategies and demonstrates judgement under pressure.

Demonstrates awareness of the boundaries of their own competence and willing to seek help when required, recognising that this is not a weakness. Exhibits appropriate level of confidence and accepts challenges to own knowledge.

Effective Communication

Actively and clearly engages patients and colleagues in equal/open dialogue. Demonstrates active listening. Communicates verbal and written information concisely and with clarity. Adapts style of communication according to individual needs and context. Able to negotiate with colleagues & patients effectively. Demonstrates courtesy, empathy and compassion.

Working Effectively as Part of a Team

Capability & willingness to work effectively in partnership with others and in multi‐disciplinary teams. Demonstrates a facilitative, collaborative approach, respecting others’ views. Offers support and advice, sharing tasks appropriately. Demonstrates an understanding of own and others’



25

roles within the team and consults with others where appropriate.

Patient Focus Ensures patient is the focus of care. Demonstrates understanding and appreciation of the needs of all patients, showing respect at all times. Takes time to build relationships with patients, demonstrating courtesy, empathy and compassion. Works in partnership with patients about their care.

Table 8: Professional Attribute SJT target domains

4.3.11 The introduction to the one day workshops covered the background to the project (presented by an ISFP representative), an overview of the test specification (including SJT target domains) and an introduction to item writing principles. Each workshop was facilitated by a minimum of two facilitators.

4.3.12 The key principles for writing the FY1 SJT items are summarised below.

• Items should not require in‐depth clinical knowledge. Measuring clinical knowledge is outside the scope of the SJT. The purpose of the SJT is to measure professional attributes of a Foundation doctor, as identified by the job analysis of the Foundation role. The SJT will be used in combination with the EPM which will cover any elements of clinical knowledge required for a Foundation doctor.

• Items should not require specific knowledge of the practices/procedures/protocols of a particular Trust/geographical area beyond that expected of all applicants.

• Items should not assume that applicants are knowledgeable about the roles and responsibilities of a FY1 doctor as applicants may come from different backgrounds (e.g. overseas training).

• The language used should not unfairly discriminate against non‐UK trained applicants.

• Scenarios should be set in the context of the Foundation Programme and the scenario content should be relevant to the work of a FY1 doctor, i.e. relate to situations they might be expected to deal with. These should be clinically based scenarios (but not require clinical knowledge). However, scenarios should be familiar to a final year medical student.

• Items should relate to a target domain as identified by the job analysis.

• Scenarios should ideally be fairly short (typically 30‐60 words and no more than 80 words) but should provide enough detail to provide dilemma and complexity.

• Scenarios and responses should be described clearly and unambiguously.

4.3.13 Item writers were requested to write two formats of items; ranking items and multiple choice items. Ranking items ask applicants to “rank in order the following actions in



26

response to this situation (1= most appropriate; 5= least appropriate)” and multiple choice items ask applicants to “choose the three most appropriate actions to take in this situation.”

4.3.14 Item writers were then asked to work in pairs creating initial scenarios. Items were then group reviewed and more time was spent developing new items and reviewing items.

4.3.15 In the second phase of workshops, small alterations were made to the content of the workshop, based on feedback and learning from the proceeding item review process.

• Item writers were specifically advised of topic areas that from previous reviews had been considered as not appropriate to cover within the scenarios due to issues with procedural knowledge or legal knowledge.

• Item writers were also advised of areas to avoid due to a large number of items already existing within this topic area.

• Greater emphasis was also placed on ensuring that the scenarios did not require clinical knowledge or knowledge of particular procedures that may vary in different hospitals. As requested from past item writers, examples of ‘poor’ SJT items were also provided as part of the training.

4.3.16 In the second phase of workshops, previously trained item writers were also invited to attend. Previously trained item writers attended a separate session to the newly trained item writers. Prior to attending the workshop, item writers were asked to re‐familiarise themselves with the SJT item writing principles. The training consisted of a brief re‐cap of SJT item writing principles and a summary of the issues outlined in 4.3.12 above. Item writers were also asked to spend some time within the workshop reviewing previously written items as a group. During this phase, item writers were asked to write between 3‐5 items. This is fewer than the number of items requested to be written in the previous five workshops. This was to encourage item writers to produce better quality items and spend more time trying to refine and develop each item. This was particularly emphasised during the workshops. Reviews of some items from previous workshops indicated that although some item writers submitted a large number of items, many were not able to be taken forward.

4.3.17 Seven of the 89 item writers did not submit any items either at or following the workshops (all from the first phase of workshops). This is a return rate of 92%. It is not clear why these seven item writers attended the workshops, but did not submit any items. For the second phase of item writing workshops it was ensured that the item writers were clear about the purpose of the workshops and the commitment that was required, as outlined in the item writer person specification.

4.3.18 In total, over the seven item writing workshops, 453 items were written, by 82 item writers. The equals an average of 5.5 items per item writer. However, there was great variation in the number of items written by each item writer which ranged from one to 19.



27

4.3.19 A greater number of items were submitted in the first phase of workshops (376 items, an average of six items per item writer) than in the second phase of workshops (77 items, an average of four items per item writer).

4.3.20 The breakdown of items relevant to each of the target domains written at or following the workshops was as follows:

• Commitment to Professionalism ‐ 180

• Coping with Pressure ‐ 91

• Effective Communication ‐ 68

• Patient Focus ‐ 46

• Working Effectively as Part of a Team ‐ 68

4.3.21 The breakdown of items regarding item format was as follows:

• Ranking ‐ 264

• Multiple Choice ‐ 189

4.3.22 This split is in line with the initial design specification of the SJT, which required that two thirds of the items used in a test should have the ranking answer format, and one third should have the multiple choice format (although item writers were not asked specifically to write more ranking items than multiple choice).

4.3.23 Item writers were asked to complete an evaluation form following the workshops. A total of 72 item writers returned the questionnaire (a response rate of 81%). Item writers were asked to respond to four questions and were also invited to provide some free text responses.

4.3.24 In total, 97.2% of item writers felt that the background information was useful reading material.

No. of Delegates Percentage of Delegates

Very Useful 44 61.1%

Useful 26 36.1%

A Little Useful 1 1.4%

Not Useful 0 ‐

Not declared 1 1.4%

Q1. The Introduction and pre‐work document was designed to give you some background to Situational Judgement Tests and item writing principles. How useful was this reading material?



28

Respondents N= 72

Extremely Dissatisfied

Dissatisfied

Neither

Satisfied

Extremely Satisfied

How satisfied were you with the pre‐workshop administration associated with the Item Writing Workshop?

How satisfied were you that the Item Writing Workshop objectives were met?

How satisfied were you with the overall content of the workshop

Table 9: Item writing workshop evaluation form responses

% % % % %



29

4.4 Initial Item Review 4.4.1 All submitted items were logged on a spreadsheet which indicated item writer, the date

the item was submitted, type of item, target domain, answer key and a short one line summary of the item. The logging spreadsheet also recorded dates the item was reviewed, when it was returned to the item writers and any comments about the item.

4.4.2 All items from the item writing workshops were reviewed by the core team of item reviewers from Work Psychology Group. The majority of items were returned within 10 working days to the item writer with comments and suggestions about the item. If necessary, item writers were asked to review their items and resubmit them in order to retain ownership of the process. A number of iterations were necessary in some cases to reach a version of an item that was ready for the next stage of review. All amendments resulting from review were highlighted clearly in the text, comments were added as annotations, and each version of the item was saved.

4.4.3 During the first phase of workshops, 42 item writers returned their items after the initial review; a total of 262 items. Twenty one item writers did not return their items after the initial review; a total of 114 items. The items that were not reviewed by item writers were finalised by the core team of item writers. Again it is not clear why these item writers did not return their items.

4.4.4 During the second phase of workshops, 17 of 19 item writers returned their items after initial review and worked in conjunction with the item of item writers to review their items; a total of 73 items. Two item writers did not return their items; a total of four items. It is encouraging that a larger proportion of item writers were willing to complete follow on work and review their items. This is likely to be the result of clearer expectations being communicated to the item writers.

4.4.5 During the first phase of workshops, following the initial review, 290 items were ready for the next stage of review and 115 items were deemed not suitable to be taken any further; an attrition rate of 31%. An item was deemed not suitable if it did not align with the item writing principles outlined in 4.3.12. If the item was deemed not suitable and not recoverable, then the item writer was informed that their item was not being taken forward to the next stage of review.

4.4.6 During the second phase of workshops, 70 of the 77 items were ready for the next stage of review. Seven items were deemed not suitable and were not taken any further in the process; an attrition rate of 9%. This is a great improvement from the first set of workshops and may be due to the improvements made to the workshop and the motivation on the part of the item writers.

4.5 Lead Item Review 4.5.1 Following the initial review, all items were reviewed by a lead item reviewer. Twelve lead

item reviewers were identified during and following the workshops as those individuals



30

who appeared to have fully understood the item writing principles and were willing to be involved further in the process.

4.5.2 Lead item reviewers were asked to consider the following points when lead item reviewing: • Target domain: do the scenario and response options reflect the target

domain?

• Scenario: is it clear and unambiguous. Are you clear about what is being asked and what is happening in the scenario?

• Scenario: is it realistic and relevant. Is the scenario something that an FY1 is likely to encounter? Would you expect them to deal with the situation?

• Options: are all options plausible (although not necessarily appropriate)? Are all options realistic options?

• Answer key: please review the key and suggest an alternative if you think it is not correct, giving your reasons for any corrections.

• Language: is the language and terminology clear? Are the terms commonly used across UK hospitals?

4.5.3 For all items written in phases one and two lead item reviewers were sent no more than 25 items each to ensure that no individual had access to too many items, although each had signed a confidentiality form. Lead item reviewers reviewed the items and then returned them by email. They were asked to use track changes to suggest any alterations to the items and make any comments in the comments box. These items were then reviewed by the core team of item reviewers and finalised ready for the next stage of review.

4.5.4 A number of items were also ‘lead item reviewed’ by previously trained item writers as part of the second phase of item writing workshops.

4.5.5 Lead item writers reviewed 360 items. At this stage, 54 items were not deemed suitable to go any further, leaving 306 items to go forward to the next stage.

4.5.6 At this stage, items were only reviewed in light of the item writing principles. Other issues such as the number of items within each format and coverage of target domains were considered as part of the test construction.

4.6 Focus Groups 4.6.1 The purpose of the FY1/FY2 focus groups was to gain feedback from job incumbents about

the relevance and fairness of the items that were due to go forward to the concordance panel. This was to help ensure that the content of the items was realistic and assisted in establishing content validityv and face validityvi.

v This is when a selection test or procedure appears to cover a representative sample of the domain being assessed e.g. the test appears to assess the competencies/skills required for successful performance within a role. vi This is when a selection test or procedure ‘looks right’ to the applicant or others i.e. the link between the test and performance on the job is clearly evident.



31

4.6.2 Focus groups took place in six locations over the two phases of development; Cambridge, Nottingham, London, Leeds, Keele and Belfast. Other foundation schools were approached for assistance in running the workshops but were unable due to time constraints. Twenty focus groups were held with between two and five individuals in each focus group, with a total of 63 individuals taking part. A breakdown of FY1s/FY2s’ demographic details can be found in the table below.

Sex Male 16

Female 46

Not Stated 1

Age Mean age 26.3

Range 23‐39

Ethnicity White 32

Black 2

Asian 19

Mixed 3

Chinese 3

Other 2

Not Stated 2 Table 10: Focus Group participants’ demographic details

4.6.3 All participants who volunteered to take part were sent briefing material outlining the purpose of the focus groups and their role on the day. All participants also completed a confidentiality and code of conduct form.

4.6.4 Participants were asked to review the test instructions and discuss whether they were clear and easy to understand. All trainees found the instructions clear and easy to understand. Trainees were then handed the SJT items and asked to consider the scenario content and the response options in their review. They were also asked to provide a possible answer key. For security reasons no individual had access to more than 20 items. Their comments and suggestions were recorded by the facilitator. Trainees were also asked about the timing of the test (60 questions in 120 minutes) and whether they felt this was appropriate. All trainees felt this would be an appropriate amount of time to complete 60 questions.

4.6.5 A total of 306 items were reviewed during the focus groups; comprising 201 ranking items and 105 multiple choice items. Below is the breakdown of the items by target domain:

• Commitment to Professionalism ‐ 117

• Coping with Pressure ‐ 56

• Effective Communication ‐ 45

• Patient Focus ‐ 41

• Working Effectively as Part of a Team ‐ 47

4.6.6 Following the focus groups, 20 of the 306 items were deemed not suitable to go forward to the next stage of review. Concerns were raised about a further 11 of the items. Their



32

comments were considered alongside the concordance comments and results (all 11 items did not go forward to the pilot).

4.6.7 All participants were asked to complete an evaluation form comprising eight questions. Participants were asked to rate each statement on a five point Likert‐type scale (where 1 = strongly disagree and 5 = strongly agree). A free text box was also provided at the end of the form for them to add any further comments not covered in the focus group.

4.6.8 Table 11 below outlines the responses to the evaluation form.

Respondents N=63

Strongly Disagree

Disagree

Neither

Agree

Strongly Agree

The level of difficulty of the SJT items reviewed today was appropriate for those applying to the Foundation

Programme

The content of the SJT items reviewed today was relevant to those applying to

the Foundation Programme

Overall, the content of the SJT items reviewed today is fair to those applying

to the Foundation Programme

The SJT is a fair method of selection for those applying to the Foundation

Programme

The SJT is an appropriate method of selection for those applying to the


The SJT is able to measure the attributes that are necessary for the


The SJT is able to discriminate between different candidates

% % % % %



33

The SJT is an enjoyable method of selection

Table 11: Focus group evaluation form responses

4.6.9 The comments from the free text box on the evaluation form were themed and the results are outlined below.

Theme No. of comments

Example comment

Ability to differentiate themselves

7 “There definitely needs to be some scope for some kind of personal statement to take into account personality and extracurricular activity”

“Slightly concerned about lack of opportunity within the form to show individual character”

Ambiguity 7 “A few questions seen today remain ambiguous in the stem”

Use of SJT in combination with other methods

6 “SJTs would be appropriate and fair in selection to Foundation Programme but as part of a wider selection process, e.g. interviews/white space questions”

Comparison to White Space Questions

5 “Much better than open questions used in the past”

Response instruction 5 “Candidates will answer according to what they should do rather than what they would do”

Relevant & fair 4 “I think that the situations are relevant and the questions appropriate”

Familiarity of scenario content/reliance on experience

4 “Questions are realistic to FY1 but may be hard given the limited clinical exposure that medical students have, especially as this varies across medical schools”

Traits assessed 2 “The SJT only tests certain traits and not others, i.e. leadership. Does not consider moral/legal aspects of medicine”

Timing/dates of test 2 “Ensure dates for this are fairly allocated”

Response options 1 “There are often a lot of appropriate answers and most can be justified”

Discriminatory 1 “I think SJTs are a great way of assigning jobs. If you score well it shows that you have had adequate hospital experience as a student and will be a



34

competent FY1”

Table 12: Themes of evaluation form free text box

4.6.10 Two of the main concerns appear to be around whether the SJT will allow them

Documents

Improving Selection to the Foundation Programme …...design and operational (live) test specification. The pilot also provided the opportunity to continue to pilot administrative