INTERNATIONAL SELECTION IN THE HEALTH PROFESSIONS … · individual books, or the Total UMAT Sum Score were not significantly correlated with any pre-clinical or clinical unit outcomes

2ND INTERNATIONAL SELECTION IN THE HEALTH PROFESSIONS CONFERENCE11–13 April 2018, Melbourne, Australia

ABSTRACT BOOKLET

PRE-CONFERENCE WORKSHOPS – 10 APRIL 2018

10:00 – 12:00

Abstract Title The Composition of the Medical Workforce

Abstract Presenters Fabiola Martin, University of Queensland Jane Hallos, University of Queensland

Theme Diversity, Undergraduate and Postgraduate

Abstract Description The medical workforce reflects the distribution of power in society. Unequal healthcare may be provided by an exclusive, involuntarily culture insensitive workforce due to a lack of patient-professional shared lived experiences. This mismatch may adversely affect patient-professional relationships, inhibiting patients from seeking help from healthcare professionals too dissimilar from themselves.

Who Should Attend Participants are welcome who wish to learn and discuss creative ways to widen the access to MD training programs as a gatekeeper to the composition of the medical workforce: health profession students, practitioners, educators and policy makers and MD training and admission administrators. (Min 10 and max 40 participants).

Structure of Workshop This interactive workshop will be conducted in form of small group discussions. A presentation will cover background information and pose specific questions (10min), participants are invited to discuss questions in small groups (20min) and present short summaries of their conclusions to the whole group (30min) for further collaborative exchange.

Intended Outcomes • Raise awareness of admission policies’ effect on social mobility and health inequalities.

• Discover & rediscover personal & system blind spots.

• Share failures & successes with widening access to underrepresented people.

• Invite the formation of a collaborative international network of a creative think-tank to influence the composition of future medical workforce.

10:00 – 12:00

Abstract Title New Insights in designing situational Judgement tests to assess non-academic attributes in selection and training.

Abstract Authors Professor Fiona Patterson, Work Psychology Group Ms Vicki Ashworth, Work Psychology Group

Theme Selection Methods, Undergraduate and Postgraduate

Abstract Description Research shows that non-academic attributes are important predictors of performance and training outcomes. How best to assess these attributes is a key challenge for selection into healthcare. Situational judgement tests (SJTs) may be a valid method. This workshop summarises evidence underpinning SJT reliability, scoring, validity, and introduces SJT item development.

Who Should Attend All those interested in selection into medicine and the healthcare professions.

Structure of Workshop Presenters will share their experience developing and evaluating SJTs for selection and training. They will illustrate, using case studies, how SJTs (text and video based) are designed and delivered with other methods across various settings. The workshop will include an SJT item writing session, with discussion and interactive group work.

Intended Outcomes By the end of the session, participants will understand the latest research evidence on the reliability and validity of SJTs, understand the features important in developing SJTs, and recognise the advantages and limitations of using SJTs for selection and training.

10:00 – 12:00

Abstract Title Developing confidence and professional identity in students from underrepresented backgrounds

Abstract Presenters Sally Curtis, University of Southampton Chloe Gilbert, University of Southampton

Theme Diversity, Undergraduate

Abstract Description Introduction Students from underrepresented backgrounds frequently do not benefit from the same social capital as traditional students, which can negatively impact progression and retention. This dynamic and interactive workshop will share best practice and engage participants in activities designed to increase students’ sense of confidence, belonging and professional identity.

Who Should Attend This interdisciplinary workshop would be ideal for educators from all health professions who are involved in teaching students from underrepresented backgrounds and also working in the broader field of diversity. It would be relevant to those involved in pastoral support of students and also to students themselves.

Structure of Workshop An initial collection of thoughts and experiences of problems associated with underrepresented students will start the session. Engagement with activities designed to support students will follow, informed by the problems initially identified. The session will end with further suggestions and appropriate support processes to address a wide range of concerns.

Intended Outcomes The aim of this workshop is to share best practice and help ensure appropriate support is considered for students from underrepresented backgrounds. This will be achieved through sharing experiences and solutions alongside useful teaching activities. This will also provide an opportunity for development of research collaborations in this area.

13:00 – 15:00

Abstract Title Resources, Research, and Reality: Developing and applying Behavioural and Scenario based questions in your Interview process from MMI to Standardized interviews

Abstract Presenters Kelly L Dore- McMaster University, Canada Lyndal Parker-Newlyn- University of Wollongong, Australia

Abstract Description This session is intended for health profession faculty and staff who are involved at all levels of learner’s trajectory in selection or the measurement of personal/professional qualities whether they use interviews or MMIs. This session will explore implications of resources and research and how to operationalise the most effective and efficient station/question format for selection interviews, screening and even professionalism assessment. Participants will learn aspects of question development which influence the quality of the information obtained regarding the applicant. Participants will develop an understanding of how the same principles can be applied to assess trainees during their training. Through this interactive session, participants will develop an understanding of the research of situational judgement assessments and behavioural descriptor questions used in selection and training. Upon completion of the workshop participants will be able to apply the frameworks to developing questions that assess personal professional qualities. Additionally, participants will develop an understanding of how to incorporate program specific missions, values and personal attributes in the process through the application of a selection blueprint. By the end of this session participants will develop a better understanding of how to develop a program specific selection model with the understanding of the potential incremental value the different measures.

Who Should Attend This session is intended for health profession faculty and staff who are involved at all levels of learner’s trajectory in selection or the measurement of personal/professional qualities whether they use interviews or MMIs.

Learning Outcomes By the end of this session participants will:

1. Understand the evidence regarding question format and various selection measures

2. Apply aspects of question format to develop more effective questions

3. Integrate the principles from selection across in-program assessments

4. Apply principles taught to develop their own selection blueprint

13:00 – 15:00

Abstract Title Scoring SJTs

Abstract Presenters Deborah O’Mara, University of Sydney, Australia Professor Margaret Hay, Monash University, Australia Fiona Patterson, Work Psychology Group, United Kingdom

Abstract Description Situational Judgement Tests (SJTs) are increasingly used in selection into the health professions at both undergraduate and post-graduate levels. Scenarios represent complex situations that mimic real-life encounters. Scoring dimensions reflect this complexity, with no single correct answer but rather deviations from a ‘most’ correct response. This multidimensionality means that traditional approaches to scoring and standard item analysis procedures do not apply to the SJT. This workshop will demonstrate the complexity of scoring SJTs, using three methods; theoretical expert approach using concordance panels, statistical cohort method and a psychometric approach. Examples will be for 2 medical schools in Australia and in UK using rating type SJTs. Data output from the three approaches will be shared and workshop participants will have an opportunity to contrast the results of different scoring on the same data and discuss the implications. How best to disseminate SJT scores and the implications to faculty and operationalise their inclusion in the selection process will be discussed.

Who Should Attend This workshop will be valuable for anyone currently using or considering using SJTs in their selection process. A knowledge of SJT item construction is useful but not essential.

13:00 – 15:00

Abstract Title Facilitating selection of Indigenous People into the health disciplines

Abstract Presenters Ms Peggy Swindle – Monash University, Australia Ms Jacqui Towns – Monash University, Australia Dr Tyson Yunkaporta – Monash University, Australia Ms Petah Atkinson – Monash University, Australia

Abstract Description Do you want to know some basic principles about how to select Indigenous people into the health disciplines? This is an interactive work-shop led by a team of Indigenous and non-Indigenous people involved in selecting Indigenous people into the health professions. In the work-shop you will participate in activities and case studies to learn leadership and advocacy skills in inclusion and equity. These skills have application across under-represented groups where equity is lacking. You will also get to use an Indigenous practice that is at least 60,000 years old to understand how to create better connection and relationship with potential applicants their families and communities. There is also a chance to celebrate and value the work you are already doing in this space and to network with others passionate in this area creating a supportive community of practice.

Who Should Attend Everyone, because everyone has a responsibility to promote equity.

Structure of Workshop Interactive activities and case studies with a demonstration.

Intended Outcomes By the end of this workshop will: understand principles of equity in selection; understand an Indigenous practice of connection and relationship strengthening; gain advocacy and leadership skills in selecting Indigenous applicants; creates new networks and learnt from others.

11 APRIL 2018 – DAY 1

11:00 – 11:15 Short Communication

Name Sophie Paynter

Organisation Monash University

Position/Role Senior Lecturer

Abstract Title The relationship of selection measures to academic and clinical course performance in an undergraduate physiotherapy course

Abstract Authors Ms Sophie Paynter, Dr. Ross Iles, Professor Margaret Hay.

Abstract Presenter Sophie Paynter

Theme Personal Characteristics

Undergraduate or Postgraduate Undergraduate

Introduction Investigations of selection tools predicting future performance in undergraduate health professions education is primarily undertaken in medicine courses, but little is known about the application to physiotherapy programs. Traditionally, selection for undergraduate physiotherapy courses only considers prior academic performance. Graduate physiotherapy competencies encompass more non-academic (behavioural and personal) characteristics than academic domains, yet selecting for these characteristics is resource intensive. This research investigated the question “What is the relationship between the interview and UMAT scores on course performance in an undergraduate physiotherapy program?”

Methods This retrospective study analysed selection data from one undergraduate physiotherapy program (Melbourne based university) from multiple selection methods (UMAT, Interview, GPA/ATAR), and demographic characteristics (gender, age). The relationships between these aspects and within course assessment scores (both academic and clinical) were determined for each year of the 4 year undergraduate program. Participants were students who began BPT / BPT(Hons) 2008–2013 (n=369). Descriptive statistics of selection data were completed and bivariate correlations explored the initial relationships between the selection variables and course performance outcomes (knowledge based written exams, simulated clinical assessments (OSCEs), overall integrated unit performance (pre-clinical) and clinical performance (as measured by the Assessment of Physiotherapy Practice (APP)).

Results Interview scores were significantly positively correlated to a moderate degree with pre-clinical OSCE performance average (r=0.297, p<0.005) and with the average pre-clinical unit marks (r=0.290, p<0.005). UMAT scores on individual books, or the Total UMAT Sum Score were not significantly correlated with any pre-clinical or clinical unit outcomes. Interview score was the only selection variable that significantly correlated positively with the average clinical placement performance grades, in the latter years of the course (r=0.306, p<0.05).

Summary of results Results suggest that interview performance is the only selection data that positively correlates with the average clinical performance of undergraduate physiotherapy students across 6 student cohorts.

Discussion Initial results support correlations between selection interview and course performance consistent with previously published work in medical education, and the only other study in an Australian physiotherapy population. ATAR and GPA held significant correlations during the pre-clinical units (overall unit marks and written examinations), however did not maintain this relationship throughout clinical placements in the final 2 years of study. Interview performance was the only selection data that positively correlated with clinical assessment ratings.

Conclusions Even though assessment of non-academic domains of applicants during selection may be more resource intensive than assessment of academic domains, a positive correlation of interview score and clinical performance in undergraduate physiotherapy students supports the use of assessing more broadly than academic performance only, at the point of course entry.

11:00 – 11:20 Research Paper Presentation

Name Aimee Gardner

Organisation Baylor College of Medicine, SurgWise Consulting

Position/Role Assistant Dean of Evaluation & Research

Abstract Title Implementing an Efficient and Effective Evidence-based Selection Program into Surgical Training

Abstract Authors Aimee K. Gardner, PhD Brian J. Dunkin, MD

Abstract Presenter Brian J. Dunkin

Theme Selection Methods

Undergraduate or Postgraduate Postgraduate

Introduction Current screening methods for selecting surgical trainees in the United States are receiving increasing scrutiny. Not only are programs spending up to $150,000 on recruitment and selection processes each year, but the outcomes of the current selection system (i.e., remediation and attrition rates) are unacceptable by any other industry standard. Applicants are experiencing significant burdens as well, missing weeks of clinical work and spending up to $10,000 attending on-site interviews. For these reasons, development of a more efficient and effective selection system for identifying surgical trainees is needed. We describe our experiences with implementing a new evidence-based selection system.

Methods This study was conducted within a fellowship program in the United States seeking to fill two one-year positions concentrating on minimally invasive and bariatric surgery. The program partnered with an expert in organizational science to conduct a job analysis, in which interviews and focus groups are conducted with key stakeholders to identify position requirements and identify requisite knowledge, skills, and abilities (KSAs) to fulfill those requirements. Based on the job analysis, assessment tools were chosen that would measure the desired and required competencies in the most efficient and effective manner. These included situational judgment tests (SJTs), personality profiles, structured interviews, and technical skills assessments. A multiple hurdle approach, in which applicants complete assessment tools in consecutive rounds, was chosen to maximize the potential of the assessments and increase administrative efficiency. The SJT and personality profiles were administered to qualified applicants on-line. Faculty were taught how to perform structured interviews.

Results Seventy-two applicants applied to the program with 56 meeting all administrative requirements. These 56 were reviewed using traditional methods (test scores, case logs, letters of recommendation). Based on this review, 44 (77%) would have been chosen for an on-site visit. All 56 were then invited to take the online SJT and personality profile with 51 (91%) completing them. Average time to complete was 42.32 ± 12.28 minutes (range 23.25 – 71.4 min.). Based on these data, 18 applicants (35%) were invited for an on-site visit (12 accepted) during which structured interviews and skills testing were performed. A final rank list was created based on all data points and their respective importance and submitted to the electronic match system. Both positions filled with a top candidate.

Summary of results The selection process did not dissuade candidates from completing the application, improved efficiency in choosing candidates for an on-site visit, and resulted in successfully matching highly competitive candidates from a relatively small pool of interviewees. In addition, it did not require a significant “extra” time commitment from the applicants.

Discussion This study demonstrates that implementing a scientific selection process into a fellowship program is feasible. Ninety-one percent of applicants invited to participate did so, allaying concerns that introducing an additional “non-standard” step to the application would prevent good candidates from engaging. It was also extremely efficient. Faculty spent less than 8 hours helping to construct and validate the selection tools and learning to conduct structured interviews. Significantly fewer applicants were invited to visit compared to traditional selection methods and they spent less than 45 minutes taking the on-line surveys; a small time investment to gain deep insight into the requirements of the job for which they are interviewing. Overall, a scientific selection program results in shared decision-making about fit and focuses on individual candidate success.

Conclusions This study demonstrates that applying selection science to the process of choosing surgical trainees is feasible, does not “turn off” competitive applicants, increases confidence and rigor in making selection decisions, and can be highly successful. The process empowered the program to make decisions about candidate selection based on objective data.


Name Dimitra Lekkas

Organisation The University of Adelaide

Position/Role Bachelor of Dental Surgery Program Coordinator

Abstract Title Moderation effects of admission tools on academic performance in dentistry

Abstract Authors Lekkas D, Liu P, Rich A, Rountree, J, Meldrum A, Winning T

Abstract Presenter Dimitra Lekkas



Introduction Relationships between admission tools and subsequent academic performance may be influenced by mediating factors. Therefore, this study evaluated the moderating effects of previous academic achievement, and Undergraduate Medical and Health Science Admission Test (UMAT) on subsequent performance and gender effects for dentistry at the University of Otago (UO), New Zealand.

Methods Admission (GPA) first-year health science, UMAT and structured interview) and dental academic performance (BDS) data were collected (2006–2013; n=533). Mediation effects of GPA on UMAT to BDS2, and UMAT on GPA to BDS relationships were investigated using Sobel Z-test for single mediator models, and bootstrap methods for multiple mediator analysis.

Results The association for GPA from first-year health science with BDS performance gradually faded over the course of four-year program. However, it was clear that the strongest association for BDS performance was performance in the previous academic year. GPA fully mediated the relationship between UMAT1 (Logical reasoning/problem solving) and BDS2.

Summary of results However, GPA partially mediated UMAT2 (Understanding people) and UMAT3 (Non-verbal reasoning), and BDS2 relationships. UMAT2 partially mediated the GPA and BDS2 relationship, while UMAT3 suppressed this relationship. For males, UMAT2 partially mediated the relationship between GPA and BDS2. In contrast, UMAT 3 suppressed this relationship for females.

Discussion As GPA fully accounted for UMAT1 and BDS relationship, its use may be redundant. However, UMAT2/UMAT3 should be retained due to GPA partial mediating effects. For males, UMAT2 mediating effects imply BDS performance required understanding people skills. For females, UMAT3 suppression implied non-verbal reasoning ability explained BDS performance variability.

Conclusions Mediation effects help explain the mechanisms through which multifaceted admission processes influence outcomes, demonstrating both causal and temporal relations. These analyses are useful for identifying admissions components for selecting students who will succeed in the programme.

Funding support: UMAT Consortium of ACER.


Name Duncan Scrimgeour

Organisation University of Aberdeen

Position/Role Intercollegiate Research Fellow and Speciality Registrar in General Surgery

Abstract Title Impact of performance in a mandatory postgraduate surgical examination on selection into specialty training

Abstract Authors Mr Duncan SG Scrimgeour, Professor Jennifer Cleland, Professor Amanda J Lee, Mr Gareth Griffiths, Ms Aileen J McKinley, Miss Clare Marx and Professor Peter A Brennan.

Abstract Presenter Mr Duncan SG Scrimgeour


I Undergraduate or Postgraduate Postgraduate

Introduction The Intercollegiate Membership of the Royal College of Surgeons (MRCS) examination is one of the most widely offered postgraduate surgical examinations in the world, with up to 6000 doctors in the United Kingdom (UK) and overseas taking the examination each year. Unlike some high-stakes medical examinations, the MRCS is yet to be validated. A quantitative study was undertaken to assess its predictive validity by investigating the relationship between MRCS (Parts A and B) and national selection interview scores for general and vascular surgery in the UK.

Methods Pearson correlation coefficients were used to examine the linear relationship between each assessment, and linear regression analyses were employed to identify potential independent predictors of the national selection score. All UK medical graduates who attempted the interview between 2011–2015 were included. Data were extracted by the lead administrator from the Intercollegiate MRCS database held by the Royal College of Surgeons (RCS) of England. Each candidate’s Part A score was merged with their Part B score to create a complete MRCS history, which included the self-declared demographics of sex, ethnicity, first language and date of birth. All scores (recorded as percentages) for candidates which had attempted the national selection interview for general and vascular surgery since its origin in May 2011 were then cross-linked with the MRCS database. National selection first attempt score was used as the main outcome variable.

Results Some 84⋅4 per cent of the candidates (1231 of 1458) were matched with MRCS data. There was a significant positive correlation between the first attempt score at Part B of the MRCS examination and the national selection score (r =0⋅38, P <0⋅001). In multivariable linear regression analysis, 17 per cent of variance in the national selection first attempt score was explained by Part B MRCS score and number of attempts (change in R2 value of 0⋅10 and 0⋅07 respectively; P <0⋅001). Candidates who required more than two attempts at Part B were predicted to score 8⋅1 per cent less than equally matched candidates who passed at their first attempt.

Summary of results There was a significant, moderately positive correlation between Part B MRCS and national selection score. Performance in Part B of the MRCS examination predicts 17 per cent of the variance in national selection first attempt score for general and vascular surgery.

Discussion Unlike the United States, examination performance is not currently used as part of the selection criteria for entering specialty training in the UK. However, given that 17 per cent of the variance in the national selection first attempt score in the present study is explained by Part B of the MRCS examination, there may be an argument for using the (currently mandatory) Part B examination as part of the selection criteria for entry into general and vascular specialty training in the UK. Given the small effect sizes and the moderate positive correlation found, it would seem sensible for other candidate factors to be considered in conjunction with Part B performance.

Conclusions Not only do these results support the validity of the MRCS exam but also the predictive validity of it. Furthermore, this could provide future surgical selection committees with additional quantifiable evidence which could be used as one of the potential selection criteria for entering higher surgical training.


Name Jacob Pearce

Organisation Australian Council for Educational Research ( ACER)

Position/Role Senior Research Fellow

Abstract Title Interrogating Gender DIF for GAMSAT Physics items

Abstract Authors Dr Jacob Pearce, Dr Luc Le

Abstract Presenter Dr Jacob Pearce



Introduction GAMSAT Section III assesses reasoning in the biological and physical sciences with items from biology, chemistry and physics contexts. Physics items are typically more difficult for females than for males. This trend may be a reflection of the different demographics of candidates from different academic backgrounds who sit the GAMSAT. Nonetheless, how is this imbalance treated technically by test-developers? This study interrogates gender DIF for physics items to determine whether the development, analysis, and ultimately selection of physics items for Section III is fair. This presentation outlines steps taken by test developers to minimise gender DIF at the item level.

Methods Differential item functioning (DIF) analysis in Item Response Theory (IRT) allows for the direct comparison of performance within items by different groups of test takers. Candidate performance on all physics trial items from 2015 and 2016 sittings of the GAMSAT were analysed by gender. Item difficulty parameter estimates for males and females were obtained separately using ACER ConQuest software. Item Characteristic Curves (ICCs) allowed for a visual check of the likelihood of answering physics items correctly by male and female candidates of matching ability. Items were flagged as displaying substantial gender DIF (by significance and magnitude) if the chi-squared DIF test was significant at the 0.05 level and the absolute DIF value was greater than 0.30 logits. Item facilities (percentage of candidates who answer correctly) were also investigated. Facilities were deemed ‘similar’ for males and females when the absolute value of the difference was <5%, and ‘different’ for ≥5%.

Results All physics trial items interrogated (n>100) could be classified according to the following taxonomy: Trial items displaying:

i. no substantial gender DIF, with similar facility for males and females (28.8%)

ii. no substantial gender DIF, with different facility for males and females (29.7%)

iii. substantial gender DIF, with similar facility data for males and females (11.0%)

iv. substantial gender DIF, with different facility data for males and females (30.5%). 41.5% of the items displayed substantial gender DIF. 28.0% of items were relatively more difficult for females and 13.5% were relatively more difficult for males.

Summary of results The trialling and review procedures ensure that the number of items with substantial gender DIF selected for use in GAMSAT Section III is minimised. However, items can still display differences in facility for males and females even if they do not display DIF. Examples of physics items will be shown.

Discussion Gender equity does not mean that we should expect exactly the same performance of males and females. Differ-ences in academic background mean that DIF may arise due to other factors. The key question is whether items in Section III accurately measure the desired proficiency of candidates. Gender equity does mean, however, that males and females of equal ability should have the same probability of achieving the same score.

Content relevance and construct validity remain crucial parameters for GAMSAT item quality. When gender DIF is detected, it is important to interrogate the type (uniform/non-uniform), and to scrutinise the items to see whether the cause of the DIF can be identified. This helps test developers remove construct irrelevant DIF from Section III and ensure high-quality items.

Conclusions The quality assurance processes for detecting gender DIF in GAMSAT Section III ensure that candidates are not being unfairly discriminated against based on gender. While gender performance may not always be equivalent on specific constructs in a selection tool, removing DIF effects as much as possible ensures fairness to candidates.


Name Priya Khanna

Organisation The Royal Australasian College of Physicians

Position/Role Researcher

Abstract Title A systematic literature review on the utility of selection methods in specialist training- Key findings

Abstract Authors Dr Priya Khanna A/Prof Chris Roberts Ms Louise Rigby

Abstract Presenter A/Prof Chris Roberts



Introduction Selecting medical graduates with appropriate competencies for further postgraduate specialty training is a challenging task involving high-stakes decisions. Given the paucity of evidence on effectiveness of selection methods in postgraduate settings, a Best Evidence Medical education (BEME) systematic literature review was undertaken by the Royal Australasian College of Physicians (RACP). The purpose of the review was to explore, analyse, and synthesise the evidence related to selection into postgraduate medical specialty training in three areas: underlying selection frameworks, effectiveness of methods of selection, and predictors of success in subsequent performance.

Methods The eleven review members came from diverse range of disciplines including health professional educators and researchers, physicians, and physicians-in-training. A systematic review protocol (study criteria, search syntax and coding sheet) was developed based on a pilot scoping search. Core bibliographic databases including PubMed; Ovid Medline; Embase, CINAHL; ERIC, and PsycINFO were used to retrieve a total of 2,640 abstracts. After removing duplicates and screening against the inclusion criteria, 202 papers remained of which 116 were included and fully coded by five pairs of reviewers. The articles were coded using a coding sheet assessing: characteristics of the study and study subjects, study endpoint, underlying frameworks, predictors of success, utility of selection methods (Utility Index), and quality, relevance and strength of the findings.

Results In terms of selection frameworks, our review highlighted gaps in underlying selection frameworks. A laissez-faire approach to selection frameworks wherein the system is defined by locally derived selection criteria heavily weighed on academic parameters seems to be giving way to the systematic introduction and evidencing of competency-based training approaches to selection in some settings.

In terms of selection tools, favourable psychometric evidence was found for multiple mini-interviews, situational judgement tests and clinical problem solving tests, although the bulk of evidence was mostly limited to the UK.

The evidence around the robustness of curriculum vitae, letters of recommendation, and personal statements was equivocal and limited. The findings on the predictors of performance were limited to academic criteria with paucity of long term evaluations.

Summary of results Locally-defined selection systems heavily weighed on academic parameters had limited psychometric evidence. Selection systems using competency-based approaches are gradually evolving, though the evidence is contextualised. Multiple selection tools in such systems were found to have favourable evidence. Predictive validity mostly limited to academic criteria with paucity of long-term evaluations.

Discussion Synthesised evidence from this review reports not only on the psychometric evidence on various selection methods, but it also illuminates a research gap in understanding the theoretical underpinnings of selection systems. Research in selection still seems to be largely based on the reductionist view of the selection process primarily focusing on past academic attainment, especially in terms of predictive value. Such studies were found to have common methodological issues.

Apart from utilising the evidence on psychometric robustness of selection methods, the broader literature from various other disciplines such as organisational psychology, organisational management, sociology, and social psychology needs to be taken into perspective when designing more standardised, centralised and holistic selection systems.

Conclusions While much has been gained in understanding the utility of individual selection methods though the evidence around many of them is equivocal, the underlying theoretical and conceptual frameworks for designing holistic, equitable, and technology-enabled selection systems suited for complex environments are yet to be investigated.


Name Deborah O’Mara

Organisation Sydney Medical School

Position/Role Assessment Lead

Abstract Title How stable is the new MCAT and what does a conversion to a GAMSAT score mean?

Abstract Authors Associate Professor Deborah O’Mara, has worked in assessment at the Sydney Medical School since 2008, being promoted to Associate Professor in 2012 and Assessment Lead in 2016. Deborah has a BA Hons I with a double major in psychology and education, a Diploma of Education and a PhD in Education. Her current research interests include the stability of rasch analysis for assessment items, methods of standard setting, situational judgement tests and the predictiveness of admissions criteria in medicine. Deborah has been the CI on two medical education grants and led assessment collaboration research on standard setting.

Abstract Presenter Deborah O’Mara



Introduction International students to most Australia graduate medical programs can use the GAMSAT or MCAT as their compulsory aptitude test, with 1 in 3 choosing the former. It is necessary to convert MCAT scores to GAMSAT equivalent scores so that international applicants can be selected and ranked on a consistent basis.

Methods The Sydney Medical School has developed a methodology for converting scores MCAT to GAMSAT score equivalents. The conversion is based on matching the percentile ranges for the weighted GAMSAT and the MCAT for a running average of 2 years. Validity checks are conducted with applicants that have both test scores.

Results The percentile ranges for the GAMSAT move very little over time. However, the percentile ranges for MCAT do alter, in part due to the change in format. An unanticipated finding from this research is the alarming pattern of variability in the new MCAT sub-scores for repeated testing.

Summary of results Significant variations in MCAT and a lesser extend GAMSAT sub-scores performance will be illustrated to highlight the questionable reliability over time of some of these aptitude scores. The Conversion from MCAT scores to GAMSAT scores will be demonstrated and illustrated over time.

Discussion Despite producing a stable methodology for converting MCAT scores to an overall GAMSAT weighted equivalent score, the meaningfulness of the conversion remains questionable. The differences found in the test-retest for the MCAT in particular do not provide support for their “aptitude” claim and the variations observed are worrying.

Conclusions Perhaps it is time for all international applicants to Australian medical schools to be selected using the same selection methods and not an “equivalent”. Many international graduates become interns in the Australian health system. Selection methods need to be robust and reliably over time.


Name Kirstie Galbraith


Position/Role Director, Experiential Development and Graduate Education

Abstract Title Development & Evaluation of a Situational Judgement Scenarios (SJS) Tool for the Faculty of Pharmacy & Pharmaceutical Sciences at Monash University

Abstract Authors Kirstie Galbraith Charlotte Flaxman Fiona Patterson Carl Kirkpatrick

Abstract Presenter Kirstie Galbraith



Introduction Situational Judgement Scenarios (SJS) represent a practicable method for assessing non-academic attributes important for success in healthcare. Using an SJS as a formative assessment may therefore help to identify students who lack these non-academic attributes. Following the success of a pilot SJS in 2016, Monash University’s Faculty of Pharmacy and Pharmaceutical Sciences have in 2017 implemented an SJS, with the objectives of providing students with feedback on their performance on the SJS, to inform their personalised learning plan, and to provide an appropriate metric to monitor and evaluate the level of progression of the non-academic attributes of pharmacy students.

Methods This study describes the design, implementation and analysis of an SJS, utilising best practice methodology, to help to identify specific development areas for pharmacy students during the 2017 academic year. The scenarios target four non-academic domains deemed important for pharmacy students: Integrity, Empathy, Team Involvement, and Critical Thinking and Problem Solving and was developed utilising a best practice methodology. Subject Matter Experts (e.g. lecturers, tutors and practicing pharmacists) were involved throughout, to develop the scenario content and answer key, ensuring that it was realistic, relevant and fair for all students. The tool was sat by all four cohorts of students across both the Australia and Malaysian campuses. Analysis was undertaken to determine impact of factors such as country of birth, language and campus, and student perceptions of the SJS, in relation to relevance, fairness and developmental support.

Results 1181 students from four year groups at two campuses completed the SJS. The tool demonstrated excellent levels of internal reliability (⋅=.88), with a close to normal distribution of total scores. The results indicate that the SJS is capable of differentiating between students, thus providing a sufficient spread of scores to support identification of students that may benefit from additional support. Results showed a significant difference in SJS scores across year levels, indicating that students further through training are more likely to achieve a higher score. Student feedback indicated they felt the SJS was relevant and fair. Native language explained some variation in students’ SJS scores, however the campus that a student was studying at appeared to contribute the most to the variation in SJS score.

Summary of results An SJS developed, validated and implemented with a cohort of undergraduate pharmacy students was able to differentiate students requiring additional support with skill development. Student characteristics explained some variation in results. Individual feedback to students on specific skills encouraged reflection and development of a personalised learning plan.

Discussion Use of Situational Judgement Scenarios as a formative tool is novel. Rather than use SJS methodology for student selection, we investigated it as a means of identifying students requiring additional support with skill development. Following a successful pilot in 2016, SJS testing was implemented in 2017 to all cohorts of pharmacy undergraduate students. Two years of data indicates that skills improve over the duration of the four year degree. Receiving feedback on individual skills, with reflective prompts to aid improvement, has been well received by students. Additional targeted interventions (skills coaching, personalised learning plans) have been implemented to a new first year program to further support the development of non-academic attributes.

Conclusions SJSs provide a useful method for assessing non-academic attributes across healthcare professions. In the context of pharmacy students, they have been used for the assessment of, and feedback about, important attributes at the domain level, with a view to support the development of pharmacy students’ skills in these attributes.


Name Ben Kumwenda


Position/Role PhD Student

Abstract Title Selecting applicants to UK postgraduate medical training programmes: a national cohort study

Abstract Authors Mr Ben Kumwenda Prof Jennifer Cleland Dr Gordon Prescott Dr Kim Walker

Abstract Presenter Ben Kumwenda



Introduction Selecting candidates from diverse backgrounds into postgraduate training programmes is crucial both in terms of workforce planning, transparency and equity issues. However, this is an area that is rarely evaluated. This study examines the relationships between a broad range of applicants’ socio-demographic characteristics and the UK Foundation Programme selection process.

Methods A longitudinal, cohort study of trainees who applied for postgraduate medical training in 2013–14. We used a linked dataset to connect data from different sources including medical school admissions, assessments, and postgraduate training. Regression analyses were used to predict the odds of applicants getting allocated to their preferred Foundation Schools.

Results Female applicants were significantly more likely (chi-square=27.48,df=2,p<0.001) than male applicants to be allocated to a foundation school of their first choice. The odds of an applicant of Asian ethnic group to be allocated to a foundation school of higher preference was 0.56 times (⋅= -0.572,p<0.001) that of a White applicant.

Summary of results Nearly 30% of the applicants were allocated to foundation schools that were not their first choice. Ethnicity and place of medical qualification were strong predictors of allocation to preferred foundation school. Except for the ethnic minorities, selection to postgraduate training does not seem to discriminate against applicants from non-traditional backgrounds.

Discussion The current algorithm for allocating trainees to their preferred foundation schools appear to be fair and transparent. Some of the observed patterns are consistent with the wider literature on postgraduate training where, for instance, those from BME backgrounds tend to do less well in many different medical examinations.

Conclusions Those involved in the selection process need to be aware of the factors associated with the allocation of trainees into their preferred choices, and need to act accordingly to ensure that the commitment to increase diversity is upheld at all levels of training.


Name Kelly Dore

Organisation McMaster University

Position/Role Senior Scientist; Associate Professor

Abstract Title Video-based Situational Judgement Tests (CASPer) Can Predict for National Licensure Scores

Abstract Authors Dr. Kelly L Dore; PhD (McMaster University) Dr. Geoffrey R Norman; PhD (McMaster University) Dr. Harold Reiter; M.D MeD, FRCPC, DABR (McMaster University) Sharyn Kreuger; Research Assistant (McMaster University)

Abstract Presenter Dr. Kelly L Dore


Undergraduate or Postgraduate Undergraduate Postgraduate

Introduction National licensure scores act as surrogates for the predictive nature of selection tools. Canada’s National licensure examination (Medical Council of Canada Qualifying Examination (MCCQE)) has outcomes which correlate to practice for both cognitive and unique non-academic outcomes. Cognitive screening measures (GPA, MCAT) correlate to outcomes on national licensure. However, the same does not hold true for screening measures of personal and professional characteristics. To address this selection gap, a video-based situational judgment test (SJT) screening for non-academic characteristics was developed, CASPer. What remains unknown is the correlation of CASPer with future performance of applicants, specifically their performance on the MCCQE.

Methods At McMaster University, applicants volunteered to complete CASPer across two admission cycles, 110 applicants in 2006 and 167 applicants in 2007. CASPer includes 8 video-based sections. All applicants’ open-ended time controlled typed responses for each section were scored independently by 2 raters. These participants’ total scores were subsequently sent to an independent third party to be combined with performance on Part I and Part II of the MCCQE where available. CASPer scores were correlated against subsequent performance, 4 to 6 years later, on Medical Council of Canada Qualifying Examination (MCCQE) Parts I and II for sub scores related to both cognitive and non-academic performance. Cognitive sub domains for cognitive outcomes included domain knowledge and clinical decision making (CDM) which correlate with practice outcomes including good use of diagnostic test and good prescribing practices. Personal/professional outcomes such as CLEO & PHELO correlate with important outcomes in practice like peer review.

Results Participants admitted to Canadian medical schools took MCCQE Parts I and II were 109 and 65 respectively. Of the national licensure outcome variables, most reflective of personal and professional characteristics, found that video-based CASPer scenarios after distattenuation correlated significantly with MCCQE Part I CLEO (r = 0.30, p = 0.038) and PHELO (r = 0.36, p = 0.014), and MCCQE Part II CLEO (r = 0.50, p = 0.025). Video-based CASPer correlated negatively and weakly with MCCQE Part I CDM, and only trended towards correlations with MCCQE Part II PI (r = 0.25, p = ns); total score (r = 0.21, p = ns). CASPer demonstrated no correlations with the medical expertise portions of MCCQE Part I—medicine, surgery, psychiatry, pediatrics, obstetrics/gynecology, and family medicine.

Summary of results CASPer consistently demonstrated positive correlations with aspects of the MCCQE related to personal/professional characteristics, specifically CLEO and PHELO, and a positive trend with aspects related to a mix of personal/professional and cognitive characteristics. Importantly, CASPer demonstrated no correlation with cognitive outcomes already predicted for by existing cognitive screens.

Discussion This study is the first to link a pre-interview screen of personal/professional characteristics to national licensure examination personal/professional subdomains three to six years later. Previous CASPer research has demonstrated strong evidence for validity including reliability, feasibility, and acceptability. However, a key component of validity evidence in selection is the ability to predict future performance. The results demonstrate moderate predictive validity of video-based CASPer for national licensure examination outcome, especially since correlations were made three to six years into training. These correlations demonstrate that CASPer is able to predict for personal/professional characteristics with similar magnitude that GPA and MCAT predict for cognitive outcomes. Importantly, CASPer is filling a void in the selection process and highlighting the value of personal/professional characteristics in the medical field.

Conclusions This research further provides evidence of the ability of a computer-based strategy to screen applicants in a feasible, reliable test, which has now demonstrated predictive validity for medical school training. With CASPer, national licensure scores are predictive of later personal/professional performance.




Position/Role Intercollegiate Research Fellow and Speciality Registrar in General Surgery

Abstract Title Which factors predict success in the mandatory UK postgraduate surgical exam: The Intercollegiate Membership of the Royal College of Surgeons (MRCS)?

Abstract Authors Mr Duncan SG Scrimgeour, Professor Jennifer Cleland, Professor Amanda J Lee and Professor Peter A Brennan.

Abstract Presenter Mr Duncan SG Scrimgeour



Introduction The Membership of the Royal College of Surgeons examination (MRCS, Parts A and B) is one of the largest postgraduate surgical exams in the world and is a prerequisite for progression to higher surgical training in the United Kingdom(UK). However, little is known about the factors that affect candidate performance.

Methods Pearson correlation coefficients were used to examine the linear relationship between MRCS Part A and B and logistic regression analysis to identify potential independent predictors of MRCS success. We included all UK medical graduates who attempted either part of MRCS between 2007 and 2016.

Results Part A – Male vs. female, OR 2.78, 95% CI 1.83–4.19, white vs. BME, OR 1.70, 95% CI 1.52–1.89, Core trainees vs. Foundation doctors, OR 0.50, 95% CI 0.32–0.77, young vs. mature graduates, OR 2.60, 95% CI 1.81–3.63. Similar results were obtained for Part B.

Summary of results A positive correlation was found between Part A and B score (r=0.41, p<0.001). Gender, ethnicity, stage of train-ing and maturity were all independent predictors of Part A success. In addition to ethnicity and stage of training, number of Part A attempts and score were identified as independent Part B predictors.

Discussion MRCS performance may become a future selection criterion for entry into higher surgical training in the UK. We have identified several predictors of MRCS success and hope that this information will help trainees to prepare for MRCS more efficiently and help to identify doctors who may require additional support.

Conclusions Candidates who do well in Part A are also more likely to perform well in Part B. Several independent predictors of MRCS outcome were identified, but only ethnicity and stage of training were found to be common predictors of both Part A and B success.


Name Deborah O’Mara

Organisation Sydney Medical School

Position/Role Assessment Lead

Abstract Title How complimentary are SJTs and MMIs?

Abstract Authors Associate Professor Deborah O’Mara, has worked in assessment at the Sydney Medical School since 2008, being promoted to Associate Professor in 2012 and Assessment Lead in 2016. Deborah has a BA Hons I with a double major in psychology and education, a Diploma of Education and a PhD in Education. Her current research interests include the stability of rasch analysis for assessment items, methods of standard setting, situational judgement tests and the predictiveness of admissions criteria in medicine. Deborah has been the CI on two medical education grants and led assessment collaboration research on standard setting.

Abstract Presenter Deborah O’Mara



Introduction A trial of a Situational Judgement Test (SJT) was conducted as part of the selection process for local applicants to the MD and/or DMD at the University of Sydney for 2017 entry. The trial was voluntary and over 1 in 3 applicants participated. As part of our normal selection criteria, all interviewees were given a 5 station Multiple Mini Interview (MMI). The correlation between the SJT and MMI was found to low if not negligible for the MD in particular. The purpose of this research was to investigate the reason for the absence of any correlation.

Methods Four hypotheses were investigated to explain the possible reasons for the low correlation between the SJT and MMI. Firstly, bias in the participation rate for the voluntary SJT trial was examined in terms of known demographic variables as well as those collected through the research Secondly, a detailed qualitative analysis was conducted on those students who scored high on the MMI and low on the SJT and students who scored high on the SJT and low on the MMI. This involved a qualitative mapping analysis of the characteristics of the particular students, their MMI interviewers. Thirdly, the correspondence between the domain scoring for the MMI and the SJT was analysed for common domains eg ethical insight. Finally, the association between the SJT and MMI and performance in the program will be examined later in 2017 to determine which measure has more predictive validity.

Results The results to-date clearly suggest that inter-personal communication as well as the gender and ethnicity of candidates interact to explain disparate patterns between the SJT and the MMI. Local applicants who scored high on the SJT but low on the MMI primarily included males from an Asian background. Local applicants who scored low on the SJT and high on the MMI primarily included those with anglo-saxon surnames and more frequently include female applicants. The association between Communication SJTs and Communication sub-scores on the MMI will be presented as well as other common domains and associations with academic and clinical assessment scores for Year 1 2017.

Summary of results The results clearly suggest that inter-personal communication as well as the gender and ethnicity of candidates interact to explain disparate patterns between the SJT and the MMI. Which is the true measure? Is the SJT less biased due to not involving interviewers or is it an incomplete assessment?

Discussion The MMI has been in use for some time but relatively little recent research has been conducted on interviewer bias. This needs to be addressed if we wish to increase the diversity of the medical student population. The ethnicity, age and gender of MMI interviewees at the SMS does not match the profile of its applicants. Other possible explanations for our findings include the low number of MMI stations (5), the reduction in MMI variability due to design control issues, and the homogeneity of candidates once selected for interview. The SJT has more discriminatory power than the MMI and can be provided to all candidates. No leakage affects were identified despite conducting the trial over a few weeks.

Conclusions Using the SJT as a tool for long listing applicants for interview is a viable and efficient alternative which would allow less MMIs with greater quality controls to be conducted. We need to invest in better training of interviewers for the MMI if we continue with this selection method.


Name Rebecca Stewart

Organisation Medical Education Experts/General Practice Training Queensland

Position/Role Medical Education Consultant

Abstract Title The Hallmarks of Education and Learning Progress and Examination Results (HELPER) Project

Abstract Authors Dr Rebecca Stewart Dr Graham Emblen Professor Parker Magin Dr Nick Cooling Dr Allison Turnock Ms Amanda Tapley

Abstract Presenter Dr Rebecca Stewart



Introduction Competence for unsupervised General Practice (GP) in Australia is determined by satisfactory performance in the Royal Australian College of General Practice (RACGP) and Australian College of Rural and Remote Medicine (ACRRM) summative examinations. To date, there has been minimal research done regarding demographic factors and in-training activities that may be predictive of assessment performance, including if selection processes could be predictive. The aim of this project was to determine to determine which demographic and/or in-training activities may be predictive of performance in General Practice summative assessments.

Methods The project was a quantitative retrospective cohort study of GP Registrars in active training with two Regional Training Organisations (RTOs) between 2010 and 2014. A multivariable analysis of demographic and in-training activities was conducted. Demographic information included age, gender, training pathway, country of graduation, leave taken in the first six months, part-time training, and education hours completed. In-training assessments comprised selection decile, Pre-Commencement Assessment (PCA) results, performance during direct observation of practice, and GP Supervisor reports. The PCA was a short (20–25 question) multiple choice assessment conducted after selection but prior to initial GP placement, focusing on the core knowledge required to commence General Practice safely. The study outcomes were remediation, pass/failure and continuous scores for RACGP and ACRRM summative assessments. The RACGP examinations include an Applied Knowled ge Test (AKT), Key Features Paper (KFP) and an Objective Structured Clinical Examination (OSCE). ACRRM assessments were not analysed due to insufficient data.

Results There were 485 GP Registrars from the two participating RTOs who entered training from 2010 to 2014 and who had attempted at least one RACGP summative examination component from 2010 – 2015. Univariate analysis revealed that poorer performance in any assessment was associated with failure of the PCA and a lower selection decile. These predictor variables were associated with poorer KFP and OSCE outcomes. With regards to multivariable outcomes, PCA performance was again predictive of KFP performance whilst selection test performance was predictive of OSCE performance. Interestingly, there was no outcome correlation with leave taken, part-time status, number of education hours attended, or performance assessments made by supervisors or in direct observation of practice.

Summary of results Younger, female registrars were more likely to complete assessment satisfactorily. Performance in a pre-commencement multiple choice exam and selection decile was predictive of performance in the KFP and OSCE. Direct observation of practice and GP Supervisor reports were informative for early intervention but did not correlate with assessment performance.

Discussion No correlation was demonstrated between assessment outcomes and performance reported by in-training activities such as supervisor reports or direct observation, both of which traditionally have been used to monitor training progression. This suggests that summative assessment performance may be independent of suboptimal training progression. Current selection processes involve a situational judgement test and mini-multiple interview, both of which have features in common with an OSCE examination, which may explain the correlation that was demonstrated. The PCA involves contextualisation of existing knowledge, which is akin to that required in the KFP and OSCE, again possibly explaining the results found in this study. In-training assessment may be measuring the ‘art’ of General Practice, whilst the ‘science is evaluated in summative assessment.

Conclusions Well-designed selection and very early assessment can predict final outcomes in GP training. Findings from this project should inform future GP selection and in-training assessment processes so that GP Registrars can be counselled as to their likely training trajectory and appropriate and timely training support provided.


Name Brad Jackel

Organisation Australian Council for Educational Research

Position/Role Senior Research Fellow, Team Leader Tertiary Humanities

Abstract Title A Diagnostic Approach to Investigating Admissions Instruments and In-course Performance.

Abstract Authors Brad Jackel

Abstract Presenter Brad Jackel



Introduction The predominant approach to researching the utility of admissions instruments and the variables they attempt to measure is “predictive validity”: the correlation of admissions variables with in-course assessment. These studies leave us with a simple correlation (or lack thereof) between assessment instruments. That correlation is not diagnostic. It may or may not suggest a problem exists but what it can’t do is define the nature of a problem or suggest a solution. This paper suggests an approach to research into admissions instruments which allows a diagnosis of the nature of any problems.

Methods A university school was undergoing substantial reform of their admissions process. Despite the school embarking on this process, the author of the paper, once involved in a consulting role, could not get an answer from any of the staff about the nature of the problem they were trying to solve: everyone agreed there was a problem, no one could define what it was, still less how the reform was going to improve matters. A short and simple questionnaire was distributed to all first year teaching staff for that school. This asked the educators to provide a list of the strengths and weaknesses of the cohort and nominate what they would most like to change about the cohort and what they were most satisfied with.

Results The identification of specific strengths and weaknesses allowed precise and targeted recommendations about what needed to be changed within the admissions process to be made.

Summary of results Factors affecting GA varied dependent context and the assessor’s demographics and experience. Although GA is often a single score, there are numerous considerations. Survey participants trust in the validity of a GA as it allows for some consideration of a checklist to be combined with clinician ‘gut feel’.

Discussion This paper presents a simple approach to researching admissions variables in real time in a manner which allows a diagnosis of the nature of any mismatch between the admissions data being collected and the qualities needed in the student cohort once accepted to a given course. It will also suggest a way in which the use of existing admissions variables can be adjusted in real time to address identifiable shortcomings without changing the existing candidate-facing admissions process. That is, it will often be possible to address identifiable problems within the student cohort by adjusting the manner in which the existing admissions variables are used with the admissions process, as opposed to embarking on a wholesale restructure of that process.

Conclusions An approach to research into admissions which is quicker, cheaper and more diagnostic than predictive validity is desperately needed. This paper gives an example of an approach that meets those needs.


Name Fiona Patterson

Position/Role Director

Abstract Title Using the In-Training Assessment Profiler (i-TAP) for early identification and intervention of postgraduate trainees likely to experience difficulties

Abstract Authors Fiona Patterson, Sathya Naidoo, Rachel Driver, Sheona MacLeod.

Abstract Presenter Fiona Patterson



Introduction Health Education East Midlands wanted to identify trainees at higher risk of experiencing difficulties in training, and to establish why this is happening. The aim was to identify those at risk of experiencing difficulty (e.g. failing examinations, requiring additional training) before they do so, and to understand their support requirements.

Methods The In-Training Assessment Profiler (i-TAP) uses existing assessment data (selection and assessment scores) to create a profile that accurately identifies trainees at significantly more risk of falling into difficulty. Identified trainees then complete an evidence-based self-assessment questionnaire to identify factors which may impede successful and timely completion of training, to inform targeted support.

Results Selection scores (Clinical Problem-Solving test, Situational Judgement Test, and Selection Centre scores), and ratings achieved in workplace based assessments, are good predictors of trainees’ likelihood of failing subsequent licensing exams (r=.14 to .47, p<.01) and requiring additional training time (ATT, accounting for 15.6% to 25.6% of variance, p<.01).

Summary of results The i-TAP successfully predicts trainee performance using selection and assessment data and can reliably inform early intervention for trainees at risk of falling into difficulty. By completing a self-assessment questionnaire, trainees and trainers gain a better understanding of the precise support requirements to assist trainees before they fall into difficulty.

Discussion Results show that existing good quality selection data allows identification of trainees that are likely to experience difficulty. The i-TAP provides an evidence-based, clear pathway which allows targeted early support to be given to trainees, to reduce the likelihood of requiring ATT, and significantly increase efficiency of training support and resources.

Conclusions Benefits of the i-TAP include: efficiency (an administrator can run the data analyses to identify trainees’ likelihood of risk); significantly earlier identification of trainees likely to struggle to optimise support interventions; reduced costs of additional training time; and reduced delays getting appropriately trained doctors into independent practice.

13:45 – 15:15 Symposium

Name David Campbell

Organisation Australian College of Rural and Remote Medicine

Position/Role Censor in Chief

Abstract Title Selecting for a rural generalist workforce – What is important?

Abstract Authors David Campbell, Lucie Walters, Tarun Sen Gupta, Anne Gately, Karen Connaughton, John Togno.

Abstract Presenter David Campbell



Introduction There is a recognised maldistribution of medical workforce in Australia. Government rural workforce policy and programs for the past two decades have been based on available evidence that identifies factors influencing choice of a rural career. Selection for Australian General Practice Training was the responsibility of a government-funded entity, General Practice Education and Training (GPET) until 2017, when College-led selection was introduced. This paper will present the process for the establishment of criteria for trainee selection adopted by the Australian College of Rural and Remote Medicine (ACRRM).

Methods A College Working Party was established to define the criteria and process for merit-based selection into the ACRRM Training Program, based on the existing process for selection into the College’s Independent Pathway, and evidence in the literature about rural career choice, including personality characteristics of applicants.

Results The working Party identified a series of criteria, including descriptors of desired characteristics, that were applied to a range of assessment methods for the purposes of trainee selection. The paper will outline the selection methodology as well as the details of the criteria identified.

Summary of results The paper will outline the selection methodology as well as the details of the merit-based criteria identified, and the subsequent analysis of the outcomes of the selection for 2018.

Discussion Existing evidence from current research about the career choices of recent ACRRM Fellows will be presented, as well as findings from the literature on personality characteristics associated with a sustainable rural career and ability to complete the training program. Targets for further research in this area will be discussed.

Conclusions A sustainable Rural Generalist Medical workforce is essential to address the current inequity of access to health care. The ACRRM selection program is designed to ensure retention of graduates in rural practice based on affirmative choice of a rural career and assessed propensity to attain a rurally-appropriate scope of practice.


Name Catherine Wills

Organisation Victorian Tertiary Admissions Centre (VTAC)

Position/Role Director

Abstract Title GEMSAS – selection for graduate-entry Medicine through a centralised admissions system.

Abstract Authors Catherine Wills

Abstract Presenter Catherine Wills



Introduction Graduate-entry medical school selection for ten medical schools in Australia utilises the Graduate Entry Medical School Admissions System (GEMSAS) as a cooperative venture under the umbrella of the GAMSAT Consortium. GEMSAS has been running since a pilot in 2009 and attracts over 5,000 applicants each year. Features of the system are advanced automation of applications, electronic retrieval of academic results, centralised assessments, and allocation of applicants to interviews and offers of places. With permission from the ACCC, medical school hopefuls are able to undergo a single interview, the results of which are standardised for use by other medical schools.

Methods This is not published research but a compilation of data and case studies from the last 8 years of selection.

Results Using a cooperative approach to a highly competitive application process has proven to be equitable to applicants, cost-effective for medical schools and provides a high-tech solution to a previously labour-intensive manual process. Using the Automated Results Transfer System (ARTS) to electronically retrieve academic results directly from Australian universities has considerably streamlined the process of obtaining and using verified results. A huge number of applications can be processed in a short period of time including calculations of GPAs according to unique rules for each medical school. Applicants can select up to six medical schools in a preference-based order and will receive only one interview offer and only one offer of a place. This makes load management a breeze for medical schools.

Summary of results GEMSAS is unique in the world as the only bespoke application system for a single field of study across a country.

Discussion • Centralised admissions versus direct applications.

• Outsourcing of selection.

• Single, transferable interview results.

• Equitable, rules-based selection practices.

• Retaining the autonomy of medical school selection practices, rules and processes.

• New model of customer service to applicants using email-only contact and leveraging almost 24/7 email responses.

Conclusions GEMSAS is a unique selection process which works by outsourcing rules-based assessments; using advanced software design; incorporating electronic from-source verified results retrieval; and at no cost to medical schools.


Name Rebecca Stewart

Organisation Medical Education Experts/GPTQ

Position/Role Medical Education Consultant

Abstract Title Global Assessment Tools in (medical) Training – The GATE project

Abstract Authors Dr Rebecca Stewart Dr Graham Emblen Dr Scott Preston Dr Marie-Louise Dick Dr Jane Smith Dr Gerard Ingham Ms Joanne Fisher

Abstract Presenter Dr Rebecca Stewart



Introduction Global Assessments (GAs) are used in Australian General Practice (GP) training including during Selection processes, during in-training reviews, and within summative assessment. Global Assessment is a common process found in many of the modalities contributing to programmatic assessment, a process which is increasingly being implemented in medical programs. Although some selection and assessment tools have been validated, the attribution of Global Assessments has yet to be clearly defined. The aim of this project was to determine factors influencing assessors in their assignment of a Global Assessment score along with gaining an understanding the behaviour of assessors themselves.

Methods Project participants were Medical Educators and GP Supervisors from the nine Australian Regional Training Organisations (RTOs) and the Remote Vocational Training Scheme. Participants were recruited via email through the General Practice Supervisors’ Association and through the relevant training organisations. A modified Delphi process was employed in the form of rounds of questionnaires with scenario-based stem questions. Participants were asked to identify the circumstances in which a GA was applied, and to self-report factors contributing to Global Assessment which were subsequently ranked using Likert scales in further Delphi rounds. Participants were requested to nominate circumstances in which there was an in-congruence between their GA score and a traditional checklist, and how this was reconciled. The questionnaires also comprised demographic and personal variables including age, gender, years of clinical experience, years of supervision provided, and extent of past inter-professional medical education experience.

Results Analysis of data from the first three rounds of the questionnaire has occurred. Up to 28 participants engaged in the three rounds from different RTOs across the country. Most respondents were female, aged over 40, and had roles as Medical Educators. GA was most commonly used in direct observation of practice, in both formative and summative contexts. Clinical knowledge, conscious incompetence, communication skills and help-seeking practices were ranked highly in considering GA. There was good agreement amongst survey participants regarding the significance of the aforementioned criteria across the training continuum and agreement regarding the robustness of GA. There was some conflicting opinion about what skills and factors can be learnt versus what should be inherent characteristics of a GP Registrar at the commencement of training.

Summary of results Factors affecting GA varied dependent context and the assessor’s demographics and experience. Although GA is often a single score, there are numerous considerations. Survey participants trust in the validity of a GA as it allows for some consideration of a checklist to be combined with clinician ‘gut feel’.

Discussion This project has identified that the factors contributing to a GA can also assess knowledge and skill domains where checklists are routine. We have demonstrated that GA extends to the non-clinical domains, including communication, professionalism and organisational skills, providing a ‘rounded’ approach to competence. As GA is conducted at multiple stages throughout GP training, it is a useful tool to measure progress and is therefore compatible with programmatic assessment. Trust in the validity of GA is strong. The strength of GA is drawn from the robustness and thoroughness often associated ‘clinical’ checklists whilst allowing for overall impressions and gut feeling. Global Assessments are viewed as more than the sum of their parts, allowing for a pragmatic assessment that provides respect and autonomy to the assessor.

Conclusions Global assessment is an integral component of selection, in-training and summative assessment in Australian General Practice. Defining the factors influencing GA has utility in improving the validity and reliability of assessment in medical education, whilst having a role in in informing bias, and/or augmenting the traditional criteria-based assessments tools.


Name Robin Ray

Position/Role Academic Lead for Selection

Abstract Title Medical selection processes and internship location: Is there a link?

Abstract Authors Dr Torres Woolley Dr Robin A Ray

Abstract Presenter Dr Robin Ray



Introduction The mission of James Cook University’s medical school is to produce graduates appropriate for northern Australia’s medical workforce. JCU’s six-year undergraduate medical program orientates its selection processes to attract applicants from rural, remote and Indigenous backgrounds across Australia. By the end of 2016, the medical school had graduated 12 cohorts of doctors. These 1286 graduates have since undertaken several thousand ‘doctor years of practice’ in regional, rural, and remote Australia.

Methods A cross-sectional survey was conducted at the JCU medical school towards the end of the 2016 academic year. Final year medical students were asked why they chose to apply to JCU to undertake their medical degree and why they chose their internship hospital. Multivariate (binary outcomes) logistic regression analysis was used to identify possible links between selection processes and internship location.

Results Predictors of accepting an internship place in a northern Australian hospital were: ‘I chose the JCU medical school because of an interest in rural medicine and/or Indigenous Health’ (p=0.013; POR=4.5); having a hometown in northern Australia at time of application to medical school (p=0.013; POR=4.4); ‘I chose the hospital because of familiarity from undergraduate training in Years 5 and 6’ (p=0.008; POR=21.5); and, ‘I believe interns will have better learning experiences in regional teaching hospitals than metropolitan hospitals’ (p=0.057; POR=3.1).

Summary of results For this cohort of final year medical students, an interest in rural medicine, a home town in northern Australia and familiarity with working in regional and rural hospitals were key factors for entry to the rural medical workforce in northern Australia.

Discussion Our study shows ‘an interest in rural medicine and/or Indigenous Health’ was a key factor in almost half of graduating doctors (41%) from the 2016 cohort who were selected to do their medical degree at JCU. Students choosing JCU for their medical degree because of an interest in rural medicine was also a significant predictor of graduates accepting an internship place in northern Australia, along with the extensive, quality learning experiences in local hospitals that had been gained during their medical degree.

Conclusions JCU’s selection processes and curriculum factors may explain why applicants from across Australia chose to do medicine at JCU, and then later choose to practise in northern Australia.


Name Scott Sypek

Organisation Women's and Children's Hospital

Position/Role Paediatrician. Director of Clinical Training

Abstract Title Can programmatic assessment be used to design and evaluate selection systems?

Abstract Authors Scott Sypek Julie Ash Ruth Sladek

Abstract Presenter Scott Sypek



Introduction The Ottawa statement (2010) assessment principles in selection. Like selection, assessment in medical education has focused on individual tools to assess competence. A programmatic assessment model (PAM) has been proposed to aid in designing programmes of assessment. Can a programmatic assessment model be used to design and evaluate selection programmes?

Methods A case study of applicants to paediatrics training was analysed using a mixed methods approach. Interviews with selection coordinators and analysis of data from the process was considered through the lens of the Programmatic Assessment model proposed by Dijkstra et al (2010).

Results A programme of assessment allows the whole picture of a student’s competence to be obtained through careful selection of assessment methods, formulation of rules and design of organisational systems. Our case study included some domains of programmatic assessment particularly around how information is collected, valued and used to make decisions.

Summary of results The advantages of using programmatic assessment with designing selection systems include creating an overview of what is being measured, which can allow for balancing content and reduce redundancy of testing tools. There is potential to use other programmatic assessment domains to include better documentation and evaluation of selection processes.

Discussion Assessment and selection have much in common. The Programmatic Assessment Framework provides a sound model to design robust, highly defensible selection processes.

Conclusions This a project in progress. We anticipate completion by the end of 2017. Dijkstra J, Van der Vleuten CP, Schuwirth LW. A new framework for designing programmes of assessment. Advances in health sciences education : theory and practice. 2010;15(3):379–93.

16:20 – 17:50 Symposium

Name Boaz Shulruf,

Organisation University of NSW, Australia

Abstract Title Let’s talk about our PATH (Purpose, Attributes, Tools & Health needs) and leave the scores behind: re-conceptual-isation of admissions.

Abstract Authors Associate Professor Boaz Shulruf, University of NSW, Australia Professor Tim Wilkinson, University of Auckland, New Zealand Philippa Poole, University of Otago, New Zealand

Abstract Presenters Associate Professor Boaz Shulruf, University of NSW, Australia Professor Tim Wilkinson, University of Auckland, New Zealand Philippa Poole, University of Otago, New Zealand



Introduction Selection processes for medical school must result in cohorts of students with attributes for the full range of careers in a health system. More diverse cohorts are more likely to achieve this. Selection debate often focuses on individual tools, including their psychometric properties and predictive validity. A programmatic approach employs a suite of tools so that the weaknesses of one tool could be countered by the strengths of another. Further, it may allow greater focus on attributes important to the practice of medicine, but less likely to be acquired by training. We discuss how such an approach might be taken.

12 APRIL 2018 – DAY 2


Name Kirstie Galbraith


Position/Role Director, Postgraduate Studies and Professional Development Unit

Abstract Title Evaluation of language and culture influence on student performance on Situational Judgement Scenarios (SJS)

Abstract Authors Kirstie Galbraith Charlotte Flaxman Fiona Patterson Carl Kirkpatrick

Abstract Presenter Kirstie Galbraith

Theme * Personal Characteristics


Introduction Situational Judgement Scenarios (SJS) represent a practicable method for assessing non-academic attributes important for success in healthcare. A formative SJS was implemented for the BPharm(Hons) program in Aus-tralia and Malaysia to identify and develop students. There is currently limited evidence evaluating the effect of language and culture on SJS performance.

Methods This study describes the design, implementation and analysis of an SJS, utilising best practice methodology, to help to identify specific non-academic development areas (e.g. empathy). The tool was completed by 1181 students across both the Australian and Malaysian campuses.

Results The SJS demonstrated excellent internal reliability (α=.88), with a close to normal distribution of total scores. When investigating differences in performance across campus, a t-test showed that Australian students performed significantly better than Malaysian students (t(1172)=4.82, p<.01). Using regression analysis this difference was due to campus rather than language.

Summary of results The results indicate that the SJS is capable of differentiating between students, thus providing a sufficient spread of scores to support identification of students that may benefit from additional support. Differences in student score due to the campus in which they study at was also found.

Discussion Psychometric analysis results and student feedback will be discussed. The impact of cultural differences on the development of non-academic attributes and exploration of how this may inform SJS development will be explored. The role of the university to develop non-academic attributes in students to change healthcare will be explored.

Conclusions Performance on SJS in this cohort was impacted by a number of factors including year of study. Native language explained some variation in students’ SJS scores, however the campus that a student was studying at appeared to contribute the most to the variation in SJS score.


Name Aimee Gardner



Abstract Title What Screening Tools Can Be Used to Predict Success in Surgical Training? An Examination of Validity Evidence for Personality, Emotional Intelligence, and Situational Judgment Tests

Abstract Authors Aimee K. Gardner, PhD Brian J. Dunkin, MD

Abstract Presenter Aimee K Gardner



Introduction The goal of this manuscript is to explore the extent to which emotional intelligence (EQ), personality profiles, and situational judgment tests (SJTs) – three screening tools that have received extensive attention in industrial selection settings – can predict success in residency.

Methods EQ, personality profiles, and SJTs were administered to residents. The relationship between these variables and residency performance were investigated through correlation and hierarchical regression analyses.

Results Fifty-one of the 61 (83.6%) eligible residents chose to participate. USMLE1, but not USMLE2, emerged as a significant predictor (t = 1.98, p < 0.05) of overall performance. Neither EQ facets nor overall EQ offered any incremental variance over USMLE1 scores. Inclusion of the personality factors was not significant and did not explain any additional portion of the variance. Inclusion of SJT scores offered 15% of incremental variance over the USMLE1 scores alone, resulting in a total of 25% of variance explained by both USMLE1 and SJT scores (F 2,57 = 7.47, p < 0.01). Both USMLE1 (t = 2.214, p < 0.05) and SJT scores (t = 2.97, p < 0.01) were significant predictors of overall residency performance.

Summary of results This study revealed little support for the use of EQ and only mild support for some distinct personality factors (i.e., agreeableness, extraversion, independence) for selection. Performance on the SJT predicted overall residency performance above and beyond what traditional cognitive measures (i.e., USMLE) contribute.

Discussion These data support further exploration of screening assessments on a larger scale across specialties and institutions. The use of situational judgment tests likely holds the most promise for being used in high-stakes assessment, but the use of personality profiles may need more examination. Despite its popularity, no support was found for emotional intelligence.

Conclusions Researchers should continue to explore the role and predictive validity of various screening methods to create comprehensive and robust selection systems.


Name Annette Burgess

Organisation University of Sydney

Position/Role Associate Professor

Abstract Title Multiple Mini Interview (MMI) for General Practice Training selection in Australia: Interviewers’ motivation

Abstract Authors Annette Burgess Chris Roberts Karyn Mossman Premala Sureshkumar

Abstract Presenter Chris Roberts



Introduction The Australian General Practice and Training uses a National Assessment Centre approach to selection into General Practice Training, which include MMIs. Setting up MMIs is a resource intensive process, requiring a large number of interviewers. In this study, we utilised self-determination theory (SDT) to consider interviewers’ motivation to take part.

Methods In 2015, 308 interviewers were recruited from 17 Regional Training Providers (RTPs) to participate in the MMI process. A convenience sample of five NAC sites was used in this study. Forty (13%) interviewers were interviewed from five NACs. Framework analysis was used to code and categorise data into themes.

Results Interviewers’ motivation to take part as interviewers were largely related to their sense of duty to their profession, a desire to contribute their expertise, provide input, and an opportunity to meet with colleagues and future trainees. Factors hindering motivation sometimes included the large number of candidates seen in one day.

Summary of results Interviewers’ motivation can be viewed through the lens of SDT. SDT proposes that for individuals to be intrinsically motivated, three key elements are needed: 1) autonomy (a sense of choice); 2) competence (desire to obtain proficiency); and 3) relatedness (sense of connectedness).

Discussion Interviewers’ motivation largely related to their interest in education, sense of duty to their profession, and desire to contribute expertise and have input to the selection process. They valued the opportunity to learn from colleagues, and develop an understanding of the needs of future trainees. Good administrative practices were important.

Conclusions Interviewers’ motivation for contributing to MMIs were largely related to their desire to contribute to their profession, and ultimately improve patient care. Interviewers recognised the importance of interviewing, and felt their individual roles made a crucial contribution to the profession of general practice. Good administration at each NAC is needed.



Organisation Centre for Healthcare Education and Research Innovation (CHERI), University of Aberdeen

Position/Role Intercollegiate Research Fellow and General Surgery Registrar

Abstract Title Does the mandatory UK surgical exam predict “on-the-job” performance during higher surgical speciality training?

Abstract Authors Peter Brennan, Amanda Lee, Gareth Griffiths and Jennifer Cleland

Abstract Presenter Duncan Scrimgeour



Introduction The Membership of the Royal College of Surgeons (MRCS) is a mandatory exam to enter higher surgical speciality training in the UK. This high-stakes examination is designed to ensure that successful candidates are competent to practice as higher surgical trainees. Annual Review of Competence Progression (ARCP) assesses trainees’ competence to progress to the next level of training and can be interpreted as a measure of “on-the-job” performance. We investigated the relationship between MRCS performance and ARCP outcomes during speciality surgical training.

Methods All UK medical graduates who passed MRCS (Parts A and B) from 2008-2016 were included. MRCS scores, number of attempts and self-declared socio-demographics were recorded for each candidate. This database was then crosslinked with all ARCP outcomes held by the Intercollegiate Surgical Curriculum Programme (ISCP). ARCP outcomes were re-categorised into: satisfactory, unsatisfactory and insufficient evidence. Multinomial logistic regression was used to identify potential independent predictors of ARCP outcomes.

Results 2570 surgical trainees underwent 11,064 ARCPs. 1589 (61.8%) trainees had only satisfactory outcomes recorded throughout the training period we studied, 510 (19.8%) had at least one unsatisfactory outcome and 471 (18.3%) supplied insufficient evidence. After adjusting for age, gender, first language and Part A performance, ethnicity (white vs. black minority ethnic, OR 1.36, 95% CI 1.08-1.71) and number of attempts at Part B (one attempt vs. two or more attempts, OR 1.50, 95% CI 1.16-1.94) were found to be independent predictors of a satisfactory ARCP outcome. Part B passing score (OR 0.98, 95% CI 0.98-0.99) was identified as an independent predictor of an unsatisfactory outcome. There were no significant independent predictors of ARCP outcome “insufficient evidence”.

Summary of results Part B MRCS performance and ethnicity are independent predictors of ARCP outcomes during higher surgical speciality training in the UK.

Conclusions This is the first study to identify predictors of “on-the-job” performance during higher surgical speciality training in the UK and provides good evidence of the predictive validity of the MRCS examination.


Name Rukhsana Ayub

Organisation National University of Medical Sciences

Position/Role Associate Professor, Head Department of Medical Education

Abstract Title Validity of multiple mini interviews for admission into an undergraduate medical education program — evidence from Pakistan

Abstract Authors Dr. Rukhsana Ayub

Abstract Presenter Dr. Rukhsana Ayub



Introduction Student selection for Undergraduate Medical Education Programs is a challenging task. Health care practice requires cultural competence, communication skills, compassion, professionalism, critical thinking and problem solving in addition to cognitive abilities. Those with such a blend of cognitive and personal attributes are more likely to select primary care thus answering the call of institutional social responsibility. Multiple mini interviews (MMI) are increasingly being used for assessment of these personal attributes. Objectives: To develop and administer MMI. To determine the descriptive and psychometric properties of the MMI station scores and to assess the construct validity of MMI stations for undergraduate students.

Methods Scenarios for assessing 9 attributes on 5 point scale for 9 OSCE like stations constructed. Faculty trained. 9 assessors assessed 365 students.

Results Mean scores, standard deviation and number of items for each station, the reliability coefficient using Cronbach’s alpha, standard error of measurement and item-total correlation of the scores on each station were determined. Mean scores ranged from 27.4% to 80.0%. Reliability of MMI stations using Cronbach’s alpha ranged from 0.64 to 0.98. Standard error of measurement ranged from 3.41% to 8.97%. Item-total correlations ranged from 0.53 to 0.96 except for one item on the station on empathy which had an item total correlation of 0.24. Exploratory Factor Analysis. using principal component analysis, with varimax rotation following Kaiser rule (i.e. eigenvalues > 1.0) was done.

Summary of results The data loaded on a total of ten factors converging after six iterations. Items of each station loaded on a separate factor except for one item on the station on empathy. Our results demonstrate factorial validity of our MMI which in turn provides important evidence of construct validity.

Discussion This study describes the development, conduction and psychometric analysis of MMI in selecting undergraduate medical students for the first time in our region. Good reliability & factorial validity of stations provides evidence that stations were assessing what they were supposed to assess. High Cronbach’s alpha shows high item cohesiveness among the sub-scales of each station and evidence of stable scores for each applicant. Our students scored well on stations testing critical thinking, problem-solving, communication skills, working in health care systems, honesty, integrity and responsibility and reliability as shown by the mean score of the stations but not on the station assessing cultural sensitivity/social awareness. This could be due to assessor’s bias or lack of training and lack of earlier exposure of students to such issues.

Conclusions MMI can be used to make reliable/ valid decisions to select students with desired personal attributes not evident on written tests/ traditional interviews. Factor analysis provides essential evidence of stations assessing attributes that were intended to be assessed. It is hoped that other institutions will adopt this student selection process.


Name Kelly Dore

Organisation McMaster University

Position/Role Senior Scientist; Associate Professor

Abstract Title Promoting Diversity: Use of the Situational Judgment Test as an Admission Screening Tool

Abstract Authors Dr. Kelly L Dore; PhD (McMaster University) Dr. Fern Juster; M.D (New York Medical College) Dr. Robin Camhi Baum MS (New York Medical College) Dr. D. Douglas Miller M.D C.M M.B.A. (New York Medical College) Dr. Harold Reiter M.D MEd, FRCPC, DABR (McMaster University)

Abstract Presenter Dr. Kelly L Dore

Theme * Diversity


Introduction Who will form the next generation of physicians? Recent decades in the United States have seen changing accountability demands across both systems and societal mandates. The resulting response can be partly addressed with changes in selection process into medical school. The AAMC (Association of American Medical Colleges) has recommended adding screening tools allowing selection of applicants with superior personal characteristics, such as situational judgment tests (SJTs) to traditional screening tools like grade point average (GPA) and Medical College Admissions Test (MCAT). However, to meet accountability mandates, the impact of these measures on demographic groups must be examined.

Methods An online video-based, open-response SJT demonstrating evidence of reliability, predictive validity and acceptability – CASPer (Computer-based Assessment for Sampling Personal characteristics) was administered monthly from July through December 2015 to the New York Medical College (NYMC) School of Medicine applicant pool. NYMC has a uniquely large applicant pool, with their applicants representing approximately 20% of all applicants to American medical schools. SJT content was developed in alignment ACGME outcomes and AAMC core inter-personal and intra-personal competencies. Pearson’s correlation coefficients and regression analyses were used to determine impact of GPA, old MCAT, new MCAT, and CASPer upon AMCAS (American Medical College Application Service) fields related to applicants’ gender, racial/ethnic, and socioeconomic backgrounds through standardized data collected in the US for every medical school applicant via the America n Medical Colleges Application Service.

Results With respect to promoting diversity as a screening tool, CASPer scores were compared to GPA, old MCAT and new MCAT, across the outcomes. CASPer scores were higher for applicants with paid employment before the age of 18 (X=5.16 vs. x=4.96, p < 0.05), which is the opposite effect of GPA and both MCATs, where scores are lower for these applicants. CASPer scores were higher for females than males (x=5.21 v x=5.03, p < 0.001), again the opposite effect of GPA and both MCATs, where males score higher. Finally, CASPer had a lower impact on those who are Underrepresented in Medicine (URM) than either the GPA or old and new MCAT tests, where URMs are more disadvantaged (F= 245.92 vs F =779.81, F= 710.41, F=922.64 respectively).

Summary of results CASPer demonstrates potential to provide additional information on applicant’s personal professional qualities, while not disadvantaging applicants as other measures of selection, GPA, old MCAT and new MCAT may. This allows CASPer to provide a fairer selection process for disadvantaged populations, such as gender, socioeconomic, and racial/ethnicity backgrounds.

Discussion Increased social accountability across the health professions must be addressed not only in program, but at the time of selection. Professional organizations (i.e. AAMC) have recommended the selection of applicants with superior personal characteristics, using Situational judgment tests (SJTs), in addition to measures of academic ability (GPA and MCAT). Psychometrically strong academic measures often disadvantage applicants from diverse populations. CASPer – an online SJT screen for personal characteristics – have evidence of reliability and predictive validity and can be applied as a screen across the entire applicant pool, such as standardize measures of cognitive ability. Of GPA, MCAT and CASPer, CASPer is the most effective in promoting diversity (paid employment before the age of 18, females and Underrepresented in Medicine) amongst applicants to professional training.

Conclusions While diversity goals are enhanced through holistic processes, screening tools with predictive validity remain a core component of student selection but at the potential cost of limiting diversity. Of GPA, MCAT and the online SJT – CASPer, the situational judgment test is the most effective in promoting diversity.

11:25 – 12:55 Symposium Presentation

Name Aimee Gardner



Abstract Title How We Select Our Surgeons across the World: Processes, Outcomes, and Opportunities for Improvement

Abstract Authors Aimee K. Gardner, PhD (USA) Brian J. Dunkin, MD (USA) Stephen Tobin, FRCS, MSurgEd (Australia)

Abstract Presenter Aimee Gardner, Brian Dunkin, Stephen Tobin



Introduction Modern surgery training requires efficiency. New technologies, evolving techniques, work hour restrictions, changes in team dynamics, and increased administrative demands for both instructors and trainees have intensified burdens placed upon training programs. These factors require that trainees rapidly adapt to their chosen training environment, quickly develop skills to work independently, and avoid delays in their training because of remediation or performance issues. This panel defines the current paradigm of selection into surgery and compares it to metrics used in industry to determine the return on investment (ROI) of a selection system: administrative efficiency, performance of those selected, and attrition rates.

Methods This panel will provide an overview of selection methods used in the United States, Australia, and New Zealand, as well as their associated outcomes: administrative efficiency, performance of those selected, and attrition rates. The panel will also discuss additional methodologies and assessment strategies innovations that have been piloted across the world that may allow programs to identify best-fit candidates more efficiently and effectively, while decreasing remediation and attrition rates and improving resident satisfaction, thus increasing the ROI of our current selection system.

Results This panel will demonstrate that surgery training programs across the world have unique challenges, but that many dedicate significant resources to the current selection process. In many systems, decisions are being made on subjective and unstandardized data points, such as letters of recommendation, personal statements, and unstructured interview scores. These unscientific and subjective methods may account for the burgeoning remediation and attrition rates seen in general surgery training today. In the United States, for example, attrition rates in surgical training are 20–30% and remediation rates consistently hover around 30%. Possible solutions include incorporation of structured interviews, personality inventories, skills testing, and situational judgment tests.

Summary of results These findings highlight that additional methodologies may allow surgery residency programs to identify best-fit candidates more efficiently and effectively, while also decreasing remediation and attrition rates.

Discussion Defined competencies, valid assessments, and efficient processes should form the foundation of the selection system for surgeons. Given the steady increase in the applicant pool for surgeon positions over the past five years in many countries, now may be the most ideal time to implement new processes so that decision makers can identify high-quality applicants that fit best into their particular training program in the most efficient manner possible. Any costs incurred by a training program investing in a more efficient selection process will be more than made up for by less money spent on remediation and attrition with the invaluable additional benefit of enhanced reputation and morale.

Conclusions The quality of this decision-making process of who will make a successful surgeon is dependent upon the quality of the data behind it. Unfortunately, selection methods adopted by many surgery training programs are not capturing critical data points for the desired competencies.

13:40 – 15:10 Symposium

Name Annette Mercer



Abstract Title Responding to challenges in conducting a longitudinal multi-centre research study

Abstract Authors Professor Jennifer Cleland, A/Professor Agnes Dodds, Professor Margaret Hay, A/Professor Annette Mercer

Abstract Presenter Annette Mercer



Introduction During the late 1990s and early 2000s it became common practice to include components other than academic scores in the selection of medical students. The most frequently-used new instruments were aptitude tests and interviews. A number of consortia were formed to oversee the construction and administration of aptitude tests such as the UMAT, GAMSAT and UKCAT. In later years these consortia have funded large-scale research projects, some at the level of one or two schools, but more recently using combined data from all the member institutions. Conducting these multi-centre longitudinal studies presents a range of challenges for the researchers.

Methods This panel will present the challenges encountered in the conduct of three different longitudinal multi-centre studies, two in Australia/New Zealand and one in the UK. Each project has a unique set of purposes, parameters and constraints, but all involve collecting selection and assessment data from a group of medical schools for the purpose of collectively investigating relationships among these data. Many of the issues for discussion are common to all three projects, as well as some which are peculiar to an individual study. The common issues and individual project issues will be identified, and retrospective solutions considered. After presenting our stories, we will use interactive presentation software to engage with the audience, to elicit and explore common experiences, and to identify strategies and solutions.

Results Some of the issues or challenges which are common to all three projects include different ways in which medical schools use their selection components; different institutional ethical approval processes and privacy policies connected to student data; considerable variety in ways in which assessment data are recorded; changes in data format over time; changes in degree structure and/or curricula; changes in personnel resulting in loss of institutional knowledge and understanding; no or limited documentation of processes, and changes in project personnel within institutions. Further are the range of understandings of university staff, both academic and professional, about the particular style of research being undertaken and the rights and responsibilities of stakeholders; and the problems of communication with busy institutional personnel. Strategies and solutions will be considered.

Summary of results The range of challenges in a multi-centre research study can be categorised into administrative and logistical issues; data format and accessibility; and the complexity of analysis in this environment. Many of these have been encountered by all the researchers involved in this presentation.

Discussion It is crucial to look beyond the level of individual institutions to address numerous questions related to the equity and value of the overall selection process and widening access. Yet systems and individual factors often act as barriers to doing so. We will discuss the data presented and the ideas gathered from the audience in relation to the dominant culture and discourses within medical education, and generate guidance on ways for centres to rise above the identified challenges to work in partnership on these important studies.

Conclusions Conducting research across institutions presents challenges in administrative procedures, in data collection and in analysis. Experience with this relatively new style of research study has led to the formulation of strategies and policies to address some of these issues, while solutions to others remain to be found.


Name Vicki Ashworth

Organisation Work Psychology Group

Position/Role Associate Director

Abstract Title Evaluating a new selection tool for innovation potential

Abstract Authors Fiona Patterson, Charlotte Flaxman, Máire Kerrin

Abstract Presenter Vicki Ashworth



Introduction As health services globally increasingly need to do more with less; innovation is an emerging attribute important for effective healthcare workers. Research suggests organisations can promote innovation by identifying innovative behaviours and characteristics in their employees (Patterson & Kerrin, 2013). Enabling identification of innovative characteristics will benefit employers and employees.

Methods The Innovation Potential Indicator (IPI) was developed to measure behaviours associated with two key innovation phases: idea generation and implementation. Examples of domains measured by this self-report tool include ‘motivation to change’ (e.g. intrinsic motivation, personal initiative) which have been identified as predictors of innovation outcomes (Burch et al., 2008).

Results The IPI has been evaluated in 12 independent studies. In a sample of 188 post-graduate physicians applying for education and training into UK general practice, the IPI demonstrated good construct and predictive validity (e.g., Motivation to Change significantly predicted creative problem-solving behaviour (β=.26, p=.006). (Patterson & Zibarras, 2016).

Summary of results 12 studies evaluating the tool were conducted, and have shown encouraging evidence to suggest its utility for identifying innovation potential. Good evidence of internal reliability (α ranging from .69 to .73) and construct validity (correlations between personality traits ranging from r= -.39 to .66, p<.05, (12 studies, N=1100)) were found.

Discussion Findings suggest the IPI is an effective reliable and valid tool for evaluating student and employees’ innovation potential. The outputs can inform selection processes, offer valuable developmental feedback, and contribute to wider heathcare innovation agendas.

Conclusions With health services under increasing pressure to do more with less, the IPI provides a unique opportunity to assess and develop innovation potential within employees. Its applications span the selection and training employee cycle and can be used as a robust way to track the innovation potential within healthcare organisations.


Name Gail Fleming

Organisation Health Education England

Position/Role Pharmacy Dean

Abstract Title Role Analysis and Development of a Professional Attributes Framework for Preregistration Pharmacists in England and Wales

Abstract Authors Fiona Patterson (Work Psychology Group & University of Cambridge, United Kingdom), Charlotte Flaxman (Work Psychology Group, United Kingdom), Vicki Ashworth (Work Psychology Group, United Kingdom), Gail Fleming (Health Education England, London).

Abstract Presenter Gail Fleming



Introduction Recently, changes internationally within education and training have impacted on the skills required within healthcare roles. In the UK, the pharmacy profession has evolved with an emphasis on responsibility for the pharmaceutical outcome of patients rather than medication supply. As part of Health Education England’s Pharmacist Education Reforms programme, a role analysis was conducted to establish a Professional Attributes Framework of the competencies required for success in the preregistration pharmacist role. Role analysis is particularly effective for identifying future role requirements and hence is a useful technique for determining the skills needed to meet future requirements (e.g. enhanced patient focus).

Methods Following a best practice methodology, a multi-method role analysis was conducted to identify the attributes associated with effective performance of a preregistration pharmacist. This consisted of 1) Literature Review 2) Stakeholder Consultation 3) Validation Questionnaire. The literature review provided background and a theoretical framework for the key attributes required for effective performance. The stakeholder consultation consisted of interviews and focus groups (n=63) and consultations (through small group discussions) during recruitment workshops (n=c.150) with a range of stakeholders across each of the sectors within pharmacy that work closely with the preregistration pharmacists. The data from the literature review and consultations were triangulated to ensure valid results. These triangulated results formed the Validation Questionnaire which asked respondents to rate the importance of the attributes previously identified (n=867).

Results Template analysis was used to analyse the data from the stakeholder consultations. The interviews were coded to identify key attributes and related behaviours specific to effective preregistration pharmacist performance. Nine attributes were identified including Person-Centred Focus, Communication and Consultation Skills and Professional Integrity and Ethics. A mapping exercise conducted independently by two researchers compared the attributes identified through the template analysis with characteristics identified during the literature review; results from the validation questionnaire found that each of the nine attributes were rated on average as ‘Important’ with Professional Integrity and Ethics being rated as ‘Very Important’. All attributes were identified as important by both hospital and community pharmacy sectors.

Summary of results Professional attributes expected of a preregistration pharmacist could be categorised into 9 domains each with more detailed descriptors. These were considered to be applicable across all areas of pharmacy practice. Professional Integrity and Ethics was considered to be the most important.

Discussion The detail that emerged during the role analysis provided the granularity essential to selecting future preregistration pharmacists, with specific behavioural indicators related to each attribute subsequently used to inform selection criteria. The validation stage confirmed the importance of the nine attributes forming the professional attributes framework, justifying the use of the framework to inform future selection of preregistration pharmacists. This framework will be used to consider innovative ways to inform decisions relating to the future development of the preregistration pharmacist selection and curricula. This brings the potential for a national process to enhance standardisation of selection decisions, utilising criteria that provide rigour, validity and reliability, alongside enhancing candidate reactions. Outcomes will be discussed in relation to the wider lessons for other changing healthcare roles globally.

Conclusions Results indicate that all attributes identified as part of the professional attributes framework are considered by stakeholders to be important, and therefore provide justification for the framework to be used to inform future selection of preregistration pharmacists and as the basis for selection criteria.


Name Andrew Pethebridge

Organisation South Eastern Sydney LHD

Position/Role Senior Staff Specialist

Abstract Title The competencies desired of applicants to enter psychiatry specialty training. A Delphi study.

Abstract Authors Dr Andrew Pethebridge, St George Hospital Sydney, University of NSW Dr Michael Ross, University of Edinburgh Ms GilL Aitken, University of Edinburgh

Abstract Presenter Dr Andrew Pethebridge



Introduction There is a growing interest in identifying key competencies associated with working in a particular medical specialty. Once identified these competencies can be used to develop assessment tools to support the selection of doctors to enter training and identify those who will offer the best ‘fit’ to work and study in that field. There is some evidence that doctors selected in this way will be more likely to succeed in specialty training programmes. The critical selection of doctors to enter speciality training is becoming of increasing importance in Australia due to the increase in the number of medical school graduates.

Methods A Delphi study was used to identify competencies desired of applicants to enter into psychiatry training and then rate the importance of these competencies in the decision to accept a doctor into training. The Delphi panel was composed of 17 Directors of Psychiatry Training (DoTs) across Australia and New Zealand.

Results Of the Delphi panel, 14 and 16 DoTs completed the first and second Delphi surveys respectively. The panel members had oversight of an average of 75.5 registrars (range 27 to 129) and were experienced in the DoT role with eight (57%) having being in that role for greater than 10 years and seven (50%) had participated in more than 20 interview panels of those seeking to enter psychiatry training. Delphi panel members identified a range of between 7 to 27 competencies considered at selection interviews to identify a total of 181 competencies. The second Delphi survey arrived at a consensus that 43 competencies were key in the decision to select an individual for entry to specialty training.

Summary of results These 43 key competencies identified through this process, clustered into twelve categories of; professionalism, working in teams, respect for others, communication skills, metacognition, personal resources, reflective, personal attributes, learning and academic skills, psychiatric knowledge and skills, medical knowledge and skills and measures of performance.

Discussion There are parallels with the the twelve competency clusters identified through this Delphi study and the domains of CanMEDS, the United Kingdom’s Person Specification for Core Training in Psychiatry and the overseas literature identifying speciality training competencies. Limitations of the study include the inability of a Delphi study to clearly identify the exact meaning of some of the competencies and the process of clustering was completed by a single individual.

Conclusions The findings of the Delphi panel and the identification of the 43 key competencies, while of value, do need to be confirmed through a process of triangulation. This can be considered the first phase of the development of specific tools for the assessment for selection into specialty training.

17:35 – 17:41 Poster Session

Name Rebecca Khan


Position/Role Research Assistant

Abstract Title Changes in SJT scores over a three-year period in applicants for an undergraduate medical course in Australia

Abstract Authors Rebecca Khan, Irene Litchwark, Sam Henry, Margaret Hay

Abstract Presenter Rebecca Khan



Introduction Due to its complexity and multidimensionality the Situational Judgement Test (SJT) is reportedly less susceptible to coaching and memorising than the MMI. These characteristics may allow a longer period of use of each item. This study explored SJT scores across three years in school leaver entry medical course applicant cohorts.

Methods A 36 scenarios (209 items) SJT for medical course selection was developed during phone interviews and work-shops with discipline representatives. Validation spanned the 2015, 2016 and 2017 admission cycles. Changes in SJT scores across the three cohorts were explored (N= 892, 2015 n = 172, 2016 n= 367, 2017 n=353).

Results The ranking median SJT total scores of the medical course applicants increased each year. A non-parametric Man-Whitney test showed a significance only between 2015 and 2017 scores (U=24439.50, z=-3.63, p=.001) (Bonferroni correction alpha <0.0167). The score difference between 2015 and 2017 was 4 points, with a small (r=0.12) effect size.

Summary of results These results indicate consistent performance on the SJT over three cohorts of school leaver medical student applicants. Where a significant difference was found, this was of a small magnitude.

Discussion It is possible that randomisation of items during test administration, and the multidimensionality of the SJT hinders item recall. The four point ranking median difference from 2015 (first implementation) to 2017 may be a cohort effect, or due to improvements in our test administration process.

Conclusions Our findings indicate that the same SJT can be successfully used across several years of selection.

10:47 – 10:53 Poster

Name Aimee Gardner



Abstract Title Applicant Perceptions of New Selection Systems for Surgical Training in the United States

Abstract Authors Aimee K. Gardner, Brian J. Dunkin

Abstract Presenter Aimee K. Gardner



Introduction Traditional screening of applicants to surgical training positions in the United States entails review of in training examinations, licensing examinations, letters of recommendation, and unstructured interviews. Unfortunately, these data are highly subjective, create substantial administrative burdens for decision makers, and have received mixed support in their ability to predict training performance. As surgical educators continue to explore innovative processes for increasing the efficiency and validity of selection systems, they must be prepared to identify how applicants will respond. The goal of this study was to explore the experiences of recent applicants who participated in a rigorous selection process.

Methods Applicants to an advanced surgical fellowship in the United States were asked to complete an online assessment containing 26 situational judgment test (SJT) items and a 108-item personality profile as part of their application package. Applicants with favorable scores on the two assessments were invited to an on-site interview in which structured interviews and skills testing were conducted. After all interviews were conducted, but before results of the match were available, an anonymous, online survey was sent to all applicants inquiring about their experiences with the new selection system. The survey asked applicants their perceptions of job relevance, communication, opportunity to perform, consistency, fairness, and ability to gain additional insight about the position requirements pertaining to each assessment phase (online, interview, skills test) in which they participated on a 1 (strongly disagree) to 5 (strongly agree) scale.

Results Twenty-one of 51 (42%) applicants completed the survey. Applicants invited for an interview (N=12) had more favorable perceptions about communication (3.50±1.38 versus 2.00± 0.82,p<0.05), opportunity to perform (3.33±1.56 versus 1.29±0.49,p<0.01), fairness (4.50±0.80 versus 3.43±1.40,p<0.05) and gaining more insight (4.25±1.22 versus 2.29±1.60,p< 0.01). Means for content (4.21±0.86) and consistency (4.79±0.42) were similar. The majority of fellows who attended the interviews (N=6) agreed it was more organized (83.4%), provided more relevant information (83.4%), had more organized faculty (83.4%), incorporated more relevant questions (66.7%), provided more information about the position (66.7%), were more polished (83.4%), and allowed applicants to better determine their “fit” compared to other programs (83.4%). The majority of fellows agreed the skills testin g was relevant (83.3%), consistent (100%), fair (100%), and a positive experience (66.7%).

Summary of results Applicants who were invited to interview had favorable perceptions of the online assessment and believed the on-site interviews were conducted better than other programs. Applicants who were not invited to interview based on their performance on the initial screening tools had less positive perceptions of the selection process.

Discussion Incorporation of a novel screening process can help obtain critical information about candidates’ knowledge and skills, while also providing them with opportunities to demonstrate their skills and learn more about the program. However, these results demonstrate that applicants’ perceptions of a new selection system are largely a function of how well they did in the selection procedure. As applicants with negative reactions to a selection experience might dissuade other potential applicants from applying and might reconsider future plans to apply to the program, programs should be mindful of providing courteous and thoughtful communication to those not selected for additional consideration.

Conclusions Ensuring a candidate positive experience will help protect an organization’s image, reputation, and ability to recruit high-qualified surgeons. These findings suggest that programs should be cognizant of how and why applicants create perceptions of their application experience.

17:47 – 17:53 Poster

Name Claudia Cunningham

Organisation University Of Aberdeen

Position/Role Specialist Trainee Restorative Dentistry/ Clinical Teaching Fellow/ PhD Student

Abstract Title Predictive Validity of Dental School Selection Criteria in Europe

Abstract Authors Cunningham C, Patterson F, Ibbeston R, Cleland JA

Abstract Presenter Claudia Cunningham



Introduction Once admitted to dental school 98% of students graduate and work as dentists. It is critical to select those with appropriate attributes for training and entry into employment at the admission stage in the education and training pipeline. Concerns have been raised that traditional measures of selection are open to bias, affected by socioeconomic status and have little or no predictive validity. To date, there has been no overview of the predictive validity of dental school selection tools. This study synthesises the dental admissions literature reporting on the relationship between performance at selection with performance throughout dental school in Europe.

Methods We performed a systematic search of SCOPUS, Pubmed and Ovid using the MeSH terms ‘Education’, ‘Dental’, ‘Criteria’ and ‘school admission’. Exclusion criteria included Non-English Language, Post-graduate, Non-European and studies that did not present empirical data (e.g., opinion pieces). The date range investigated was 1987–2017.

Results Our search yielded an initial set of 486 titles. Duplication' were removed and exclusion criteria applied, leaving 21 papers for analysis. The majority (n= 17) were single site studies; 5 studies were single-cohort. Study populations ranged from 62 to 769. No sample size calculations were stated. A variety of selection tools featured which fell into three distinct categories; Cognitive (n= 9), ‘Soft-skills’ (n=14) and, Manual dexterity (n= 5). Most studies were published within the last 10 years (n= 14). The majority were from the UK (n= 9). Most studies used a cross sectional design (n=11) and retrospective analysis (n=11)of an admission tool already in use. Only two authors reported on full cohorts through to completion of studies with the majority only investigating performance in year 1.

Summary of results Control or comparison groups, adequate sample sizes and appropriate long-term follow-up measures were rare.

Discussion Only 14 of the current 228 dental schools in Europe have published any research over the last 30 years. This research highlighted a lack of good quality studies examining the predictive validity of dental admissions selection methods and means no conclusion can be drawn as to the “best” tools for dental school selection.

Conclusions Without further research to improve the evidence base, there is a risk that dental school admissions may fail to select those candidates most likely to become competent clinicians.

17:53 – 17:59 Poster

Name Scott Sypek

Organisation Women’s and Children’s Hospital

Position/Role Paediatrician. Director of Clinical Training

Abstract Title What do (and what should) junior doctors put in their curriculum vitae?

Abstract Authors Scott Sypek

Abstract Presenter Scott Sypek



Introduction The curriculum vitae (CV) is central to every recruitment process for junior doctors entering general training positions. While considering how we should score CVs for junior doctors applying to prevocational positions, we explored what they currently include.

Methods Junior doctors applying to prevocational training positions in SA submitted CVs as part of the application process. A sample of approximately 100 CVs were analysed in detail identifying information applicants listed under the headings: work history, professional development, teaching, research, extra curricular activities, awards and achievements.

Results Study in progress.

Results will provide us with an understanding of the distribution of skills, interests and qualifications across our junior doctor cohort.

Summary of results Study in progress.

Discussion To accurately score CVs we need to be clear about what the purpose of the CV is in the application process and how this assists making selection decisions.

Conclusions Study in progress.

17:59 – 18:05 Poster

Name Kwong Chan

Organisation Griffith University

Position/Role Internationalisation Lead

Abstract Title Looking into the crystal ball – Using a 10 credit point course on Human Skills to select Pre-Med students into a MD program

Abstract Authors Dr Kwong Djee Chan Dr Cathy Wu Ms Linda Humphreys Prof Raymond Tedmon

Abstract Presenter Dr Kwong Chan



Introduction The Griffith Human Skills for Medicine Course is a 10 credit point course delivered by the School of Medicine for the Griffith Bachelor of Medical Science (MD provisional entry for school leavers) students.

The aim is to prepare the students to transform their role from being a University student into a “Physician in Training” in the Griffith Medical program. The main focus is to develop students’ general communication skills, to provide them with an understanding of how people perceive their experiences in the health environment, and how to apply human skills (eg communication skills and interpersonal skills) in such context.

Methods Participants were 57 students who were enrolled in Human Skills for Medicine and latter completed the 4 year Medical Program. 66.7% of the participants were female. Selected assessment items between the Human Skills Course and the MD program were statistically compared. Using the assessment scores (including Objective Structured non-Clinical Examination (OSCE), Workshop Preparation & Engagement Evaluation and Post Workshop Reflective journalling) the research team predicted student performance of major assessments in the Medical program.

We compared these with each of the Year 1 and 2 theme results [Doctor and Knowledge of Health and Illness (DKHI), Doctor and Patient (D&P), Doctor, Law, Ethics and Professional Practice (DLEPP), Doctor and Health in the Community (DHC)] and the Year 3 and 4 OSCE and Progress Test results. Univariate correlations and linear regression analyses were performed.

Results OSCE score was significantly correlated with each subject score in the participants performance in all 4 years of the Medical Program. Correlation coefficient r from 0.338 (p < 0.05) to 0.526 (p < 0.01). The predictive power (R2) of OSCE was highest for DHC score in Years 2, accounting for 28% of the variance . In linear regression analyses, combining OSCE score with reflective journal score significantly improved predictive power (total R2) for DLEPP score in Year 1, DHC and DLEPP scores in Years 2, accounting for 33%, 36% and 26% of the variance respectively.

Summary of results The Human Skills of Medicine OSCE score was significantly correlated with each subject score in the participants performance in all 4 years of the Medical Program.

Discussion The results prove that the human skills course is successful both as a transition course between Pre Med under-graduate course and the Post Grad MD program.

The significant correlations of the Human Skills OSCE with majority of the assessment item in the MD program shows that the skills measure in the Human Skills OSCE correlates to both content and skill assessment in the MD program.

Future analysis can look at individual human skills assessment item such as warmth, clarity and reflection and how those individual attributes correlates with students performance in the MD program.

Conclusions The non-clinical OSCE score in Human Skills for Medicine can predict medical students’ academic performance in all four years of the Medical Program. The limitation of this study is that other factors that might impact on the participants’ performance in the medical program have not been collected.

18:05 – 18:11 Poster

Name Ruth Sladek

Organisation Flinders University

Position/Role Senior Lecturer in Medical Education / Chair of Admissions Committee

Abstract Title Selection scholarship across the health professions: a bounded review

Abstract Authors Ruth M Sladek, Raechel Damarell, Svetlana King

Abstract Presenter Ruth M Sladek



Introduction Scholarship outputs are rarely compared across different professions, yet this may limit the true potential of evidence obtained in one setting to inform practice in another. In the case of selection, valuable insights may be gained by adopting a more interdisciplinary approach to identifying and understanding the relevant literature.

Methods Using a subset of 21 high quality journals that were identified as focusing on education in health professions education in a parent study, we searched for literature relating to student selection into University courses (2006 -2015) in PubMed and Scopus. Citations were analyzed.

Results This is in progress. We will describe the articles published in relation to selection in the identified medicine, dentistry, nursing, pharmaceutical, veterinary and interdisciplinary education journals. In particular, we will report citation counts and related metrics and identify themes of shared interest.

18:11 – 18 :17 Poster

Name Gemma McGrory

Organisation NHS Lanarkshire

Position/Role Clinical Teaching Fellow

Abstract Title A Junior Doctor for the Day: Using Simulation to Widen Access to Medical Careers

Abstract Authors Gemma McGrory

Abstract Presenter Gemma McGrory

Theme Diversity


Introduction Schoolchildren from deprived backgrounds struggle to access work experience. In NHS Lanarkshire, we offer a 3-day hospital work experience programme. Arguably this does not provide sufficient time for a full experience of medicine as a career, and is a passive rather than an active learning experience.

Methods ‘A Junior Doctor for the Day’ is a novel simulation widening access programme which seeks to address this problem by allowing schoolchildren to experience hands-on small group workshops using simulation of emergency and surgical specialties including assessing an unwell patient; fix a fracture in theatre; laparoscopic surgery; and critical care.

Results This proof of concept study wished to establish: (1)whether using simulation as an adjunct to work experience works in practice; (2)whether this is an acceptable additional experience in additional to traditional work experience; (3)whether a simulated programme such as this inspires deprived schoolchildren to apply to medical school.

Summary of results 34 academically able Lanarkshire schoolchildren took part in the pilot programme. The practical small group simulation sessions worked in practice and the schoolchildren found the day to be an excellent addition to work experience. Having the opportunity to have ‘hands-on’ experience was invaluable, and inspired 50% regarding medical career choice.

Discussion Work experience is often challenging to arrange and is passive observation. To promote workforce diversity and aspirations of medical careers, we should provide as realistic as possible taste of medicine to prospective applicants. Simulation makes it possible to be actively involved in emergency/surgical specialties as part of work experience.

Conclusions ‘A Junior Doctor for the Day’ is a novel application of simulation. This widening access simulation programme addresses some of the challenges associated with traditional work experience and serves as a means of inspiring schoolchildren from deprived backgrounds to aspire to a medical career.

18:05 – 18 :11 Poster

Name Kirsty Alexander


Position/Role PhD Student in Medical Education

Abstract Title A requirement, a value or a service? How widening access policy is framed on medical school webpages

Abstract Authors Kirsty Alexander Tania Fahey Palma Jennifer Cleland

Abstract Presenter Kirsty Alexander

Theme Diversity


Introduction UK governments have set medical schools ambitious ‘widening access’ (WA) targets. Institutional webpages are highly influential resources (The Student Room, 2016), however, little is known about how schools use these to present their WA initiatives, or the impact these presentations may have on potential applicants, policy makers and schools themselves.

Methods We employed a critical discourse analysis approach to examine how UK medical schools discursively frame WA policy enactment on their websites (Entman, 1993). We identified the purpose, process and outcomes of WA (Malen & Knapp, 1997); examined how these interpretations were constructed linguistically; and considered the influence of key messages.

Results Three discursive frames were identified: ‘the institutional’, ‘the value-driven’ and ‘the service oriented’. These frames utilised distinct linguistic and discursive strategies to promote WA enactment as a ‘requirement’, a ‘value’ or a ‘service’ respectively. Each frame used characteristic tenses, pronoun use and lexical choice to create its distinctive tone.

Summary of results Each frame promoted a different interpretation of WA policy to audiences: to limit the impact of WA policy to existing selection systems; to promote the integration of WA into school culture; or to position WA initiatives as an additional ‘product’ provided by schools to address a ‘need’ in the market.

Discussion The three discursive frames constructed and promoted three different interpretations of WA policy enactment. These differences may influence: how potential applicants judge the attractiveness of the institution; how current institutional members orient their attitudes towards WA; and how policy makers perceive medical schools’ implementation of policy.

Conclusions Against a backdrop of increasingly demanding WA targets, medical schools should critically consider how they currently frame their enactment of WA policy. Schools should ensure that the values promoted on their websites align with those they practice or aspire to, and that they use these highly visible resources strategically.

18:23 – 18:29 Poster

Name Clare Owen

Organisation Medical Schools Council

Position/Role Policy Adviser

Abstract Title Mapping the provision of outreach by UK schools

Abstract Authors Clare Owen Ceri Nursaw

Abstract Presenter Clare Owen

Theme Diversity


Introduction Outreach is a crucial step in widening participation to medicine, as many students as possible with the potential to become doctors should have access to it. This project looked at the geographical spread of UK medical school outreach and identified cold spots where no outreach was being delivered.

Methods Medical schools were surveyed and asked to identify the secondary schools and colleges they provided outreach to, as well as the type and intensity of the intervention. These data were mapped geographically. Performance measures and pupil demographics relating to engaged schools were then added to the map?

Results • 50% of secondary schools in the UK receive outreach from medical schools.

• There are 330 schools that show over five engagements with Medical Schools.

• There are geographical cold spots where no outreach takes place. These are in rural and urban areas.

• Better performing schools were more likely to be engaged.

Summary of results It is positive that medical schools engage with so many schools but the results show there is scope for the situation to be improved and made more efficient. 80% of applications to medical schools over three years came from just 20% of secondary schools so increasing engagement is a priority.

Discussion Medical schools have responded positively to the provision of these data and as a result are actively changing the way they deliver outreach. Change is being effected by extending the areas t they cover or developing new schemes enabling them to cover the ‘cold spots’.

Conclusions Identification of cold spots has enabled better targeting of outreach. In order to verify the fact that the provision of mapping data has changed medical school behaviour the exercise will be repeated in October 2017. Data will then be available to show the impact of this project.

18 :29 – 18:35 Poster

Name Clare Owen

Organisation Medical Schools Council

Position/Role Policy Adviser

Abstract Title Setting up a Selection Alliance to co-ordinate national activity on selection and widening participation

Abstract Authors Clare Owen Sally Curtis

Abstract Presenter Clare Owen

Theme Diversity


Introduction Medicine in the UK has been criticised for failing to attract enough students from lower socio-economic backgrounds. In addition, concern was expressed by the regulator of undergraduate medical education that medical schools used a variety of methods to select students without evidence to justify those methods.

Methods The MSC Selection Alliance brings together Admissions Deans from all 33 publicly funded UK medical schools to improve selection systems and widen participation. This is achieved by agreeing common approaches to issues such as work experience requirements and by commissioning research to form an evidence base for future approaches.

Results • Better guidance and advice on applying to medicine for applicants, teachers and careers advisers.

• Collaboration to write new MMI items and share items across institutions.

• Leading researchers commissioned to study subjects like contextual admissions and weighting and sequencing of selection processes.

• Benchmarking and monitoring of national progress in widening participation.

Summary of results Greater collaboration between medical has had tangible benefits for applicants to medicine who are now able to access advice approved by every medical admissions department in the UK. Medical schools are starting to collaborate on projects such as outreach schemes and shared MMIs, alongside sharing best practice.

Discussion Bringing UK medical schools together has created access to a large data set, providing the MSCSA with the potential to carry out wide ranging research studies that seek to improve selection methods and widening participation nationally. Applicants have benefited from a co-ordinated approach to providing advice and guidance.

Conclusions Bringing medical schools together to work on issues they all face has been beneficial in a UK context. A good degree of progress has been made in supporting the application and admissions processes and there is the potential to undertake ground-breaking work in the future utilising large data sets.

11 APRIL 2018 – DAY 3


Name Gemma McGrory

Organisation NHS Lanarkshire / University of Dundee

Position/Role Clinical Teaching Fellow / MMEd Student

Abstract Title Widening Participation in Medicine: Impact of ‘A Medical Student for the Day’ programme on perceptions of medical school in prospective applicants

Abstract Authors Gemma McGrory1,2 & Ashley Dennis2 1 Kirklands METC, NHS Lanarkshire 2 University of Dundee, Medical Education

Abstract Presenter Gemma McGrory

Theme Diversity


Introduction Schoolchildren from deprived backgrounds often struggle to obtain a medical school place. Traditional thinking cites perceived knowledge deficit and academic inability, however, contemporary thinking argues that it is related to pupils’ identity and perceptions of how they will fit in to medical school. Data from the literature suggests that schoolchildren from deprived backgrounds underestimate their chances of success when applying to medical school because they perceive that medical school is culturally outwith their reach. NHS Lanarkshire serves a deprived population and few local schoolchildren apply to medical school. ‘A Medical Student for the Day’ was developed to address this issue.

Methods In order to promote a diverse workforce, the doctors serving a local population should reflect the local population. ‘A Medical Student for the Day’ is a one-day widening access programme designed to supplement work experience for academically able schoolchildren in Lanarkshire and aims to address this issue of workforce diversity. The programme provides a ‘day in the life’ of a first year medical student through a lecture, tutorial, workshop, PBL, library and examination skills session. These activities provide first-hand experience of the teaching styles that students are exposed to at university, offers an opportunity to establish their perceptions of medical school and inspire these schoolchildren to apply to medical school. A questionnaire and modified-DREEM inventory was issued before and after taking part in the programme to 32 Lanarkshire schoolchildren to identify their perceptions of medical school before and after takin g part in the programme.

Results In contrast to the literature, the schoolchildren believe that medicine is a path open to anyone, regardless of social class, gender or ethnicity. They worried less about fitting in to the medical school environment, and much more about fitting in academically. Whilst the pre-programme questionnaires alluded to concerns of academic ability, the modified-DREEM inventories demonstrated an improvement in their academic self-perceptions at the end of the programme. The programme also positively impacted on their perceptions of medical school, taking their understanding of the medical school environment from very vague to well informed, with a corresponding increase in perceptions of medical school as a positive learning environment, and crucially, as an environment where they can see themselves.

Summary of results This study has found that Lanarkshire schoolchildren hold different views regarding who gets into medical school from that documented in the literature, but hold underlying poor perceptions of medical school as an accessible academic environment, which are positively impacted by taking part in ‘A Medical Student for the Day’.

Discussion Academically able Lanarkshire schoolchildren who come from deprived backgrounds hold the view that social class is not a barrier to medical school success (compared to academic ability). This is in contrast to other published studies where schoolchildren from similar demographics identify medical students to be culturally ‘alien’. The reasons for this are unclear but may be related to regional widening access initiatives or due to free higher education in Scotland (compared to other parts of the UK), which removes underlying financial concerns for this Scottish socioeconomic group. The data regarding teaching methods revealed that the schoolchildren had little understanding of, and held misconceptions regarding, activities at medical school and had doubts regarding their own academic ability, which were improved by taking part in this programme.

Conclusions Giving schoolchildren from deprived backgrounds a realistic experience of medical school, gives the opportunity to demonstrate that they are capable of medical school. This encourages those who had doubts about their ability to fit into the academic environment of medical school the capacity to aspire to medical school.


Name Eloise Jillings

Organisation Massey University

Position/Role Associate Veterinary Dean — Admission and Students

Abstract Title Exploring indigenous student application and selection success into veterinary training in New Zealand.

Abstract Authors Jillings, Eloise KP Gardner, Dianne Parkinson, Tim Hecker, Kent

Abstract Presenter Jillings, Eloise KP

Theme Diversity


Introduction The US/UK veterinary professions have limited ethnic diversity. In the New Zealand veterinary profession anecdotally there appears to be greater diversity, but few Maori (indigenous) in the veterinary programme and veterinary profession. The authors will evaluate these observations and if verified, assess whether they are due to low applicant recruitment or selection success.

Methods A retrospective analysis of enrolment data collected from 2003–2016, and data from the Veterinary Council of New Zealand will be performed to:

• Quantify and compare the proportion of Maori in the veterinary profession, veterinary applicants and those selected into the veterinary programme to that of New Zealand society.

• Identify whether being Maori is associated with increased or decreased likelihood of selection into the veterinary programme compared to non-Maori.

• Identify whether an admission pathway for Maori has improved the proportion of Maori selected.

Results Maori are under-represented in the veterinary profession, the veterinary applicant pool, and the students admitted to the veterinary programme when compared to the general New Zealand population. Prior to 2007 the proportion of Maori selected into the professional phase of the veterinary programme was significantly different from that of non-Maori. From 2007–2016 this difference decreased

Discussion Historical and contemporary ongoing injustices toward Maori have contributed to poorer outcomes for Maori in many areas and under-representation in many professions. This data confirms that Maori are under-represented in the veterinary profession and in the veterinary applicant pool. Whilst there was a difference in likelihood of selection for Maori and non-Maori, this has been improved through the introduction of an admission pathway for Maori. However, more appropriate support and recruitment activities must also be undertaken, particularly with intermediate and high school aged students, to attract Maori to the profession as selection policy changes alone will be insufficient

Conclusion The traditional academic only selection process showed a significantly lower proportion of Maori students selected as compared to non-Maori. Introduction of an admission pathway for Maori decreased the difference in likelihood of selection. Despite this, without greater efforts to attract students to the profession we will not achieve representation of Maori at levels approaching that of the New Zealand population


Name Jacqui Towns

Organisation Monash University, Australia

Position/Role Senior Project Coordinator – Hands on Health

Abstract Title Early exposure strategies to build career awareness in under-represented, Indigenous and refugee / asylum seeker high school student populations – the Hands on Health experience.

Abstract Authors Ms Jacqui Towns, Monash University, Australia Ms Peggy Swindle, Monash University, Australia

Abstract Presenter Ms Jacqui Towns

Theme Diversity


Introduction Hands on Health is a program developed in 2011 to increase awareness of the opportunities in health for low socioeconomic, Indigenous and refugee/ asylum seeker high school students. Year 10 students from these groups are invited to a week-long work experience in a hospital setting to create awareness of the diverse range of careers available in health. The participants are introduced to both students and professionals from similar backgrounds undertaking these educational and career pathways. This is a unique partnership between Monash University, Monash Health and Peninsula Health contributing to provide an experience with both practical and lecture based sessions. In 2018, the Hands on Health program is expanding to include in-school educational sessions for Year 9 students and On-Campus experiences for Years 11 and 12 students. Evaluation of the program will track the current Year 10 cohort for their post-high school educational or career outcomes. The long-term program objectives are to build healthcare workforces that reflect the diverse communities they serve.

Methods Recruitment of students for the Hands on Health Program from disadvantaged and underrepresented schools in South East Melbourne. Many of these schools work in collaboration with Monash University through the Schools Access Monash (SAM) partnership. Other community partnerships are being established and utilised to engage with students from Indigenous and refugee/asylum seeker backgrounds. A partnership with the Gateway Local Learning and Employment Network (LLEN) has also ensured the opportunity is available to eligible students in the local area.

The current evaluation method is by attendance numbers and feedback forms upon program completion.

2018 Program:

• Year 9 – Health Sciences Discovery Session – 1-hour incursion at schools discussing health careers

• Year 10 – Work experience Program – one week in hospital with exposure to multiple health professionals and students

• Year 11 and 12 – Vertical Enhanced Study Program Approach Sessions (VESPAS) – 2 hour - On campus interaction with health science students across year levels.

• Year 11 and 12 – Road to Medicine – 1 day program on campus discussing the process of admission to tertiary studies in Medicine.

Results Participation:

Data for participation numbers is only available for the Year 10 Work Experience program and the Road to Medicine sessions.

Year 10 Work Experience Program.

Participation numbers increasing over the years has provided a quantitative measure of the of program’s success.

It is anticipated the Work Experience program will run at capacity for 2018. The program can accommodate a maximum of 180 students over the 12 weeks of work experience sessions. Recruitment is very strong with 149 individual online applications and the Gateway LLEN and SAM schools requesting student allocations completing the available positions for the program.

Data collected from the program report in 2015/2016 indicates 123 students attended in that year, this was reported to be a five-fold increase from the first year of the program.

Road to Medicine

In 2017, 80 students participated in the Road to Medicine day at Monash University.

Feedback:

Qualitative feedback methods have been utilised throughout the program to evaluate the student engagement with the content and interactions. These methods indicate a very positive experience from both student participants, school staff and hospital staff.

Summary of results Participation in the program has been strong since its conception in 2011. The number of students applying to both work experience program and the Road to Medicine Seminar has increased.

Feedback forms have indicated a broadly positive response to the program over the years; unfortunately, career and educational outcomes for participants are not available.

Discussion The Hands on Health program is primarily an opportunity to increase awareness of health careers and educational pathways in underrepresented student populations. The early exposure to multiple health careers aims to inform aspirations, create further awareness and overcome barriers to tertiary education and employment in the health sector. Throughout the program, the students interact with both professionals and students on placement from diverse cultural backgrounds to discuss their educational pathway and better understand a career in health.

The long-term impact of the Hands on Health program is yet to be determined. Early exposure programs can influence change in both individuals and communities of high disadvantage. By increasing awareness of the educational pathways for health careers, we hope to see more students pursuing tertiary education. Increasing the cultural diversity in the healthcare workforce will potentially improve access to services, reduce communications barriers between patients and practitioners and create an environment of understanding for culture, ethnicity and beliefs. Diversification in the healthcare workforce has the potential to influence the health outcomes of disadvantaged population groups.

Conclusions The Hands on health program is a unique collaboration between Monash University, Monash Health and Peninsula Health. The program aims to increase the understanding of pathways into the study of health sciences leading to careers in health. Through a multi-layer intervention over the mid to later years of high school, we hope to demonstrate an impact to both the individual and then their wider community.


Name Ben Kumwenda


Position/Role PhD Student

Abstract Title Are efforts to attract graduate applicants to UK medical schools effective in widening access? A national cohort study

Abstract Authors Kumwenda B [1], Cleland JA [1], Greatrix R [2], Mackenzie RK [1], Prescott GJ [3]

1. Centre for Healthcare Education Research and Innovation, Institute of Education for Medical and Dental Sciences, School of Medicine, Medical Sciences & Nutrition, University of Aberdeen, UK

2. UKCAT Consortium, https://www.ukcat.ac.uk/

3. Medical Statistics Team, School of Medicine, Medical Sciences & Nutrition, University of Aberdeen, UK

Abstract Presenter Ben Kumwenda

Theme * Diversity


Introduction Attracting graduates to medicine was recommended as a means of diversifying the student population in UK medical schools. Graduates now make up nearly a quarter of the medical student population. However, the knowledge we have of graduate applicants and graduate admissions is incomplete: do graduates differ in terms of socio-demographic markers compared to school leavers? Thus, we used a contemporary dataset to compare the socioeconomic characteristics of graduates and non-graduate and how they perform in the UK clinical aptitude test (UKCAT) component of medical school selection processes.

Methods This was an observational study of 117214 applicants to medicine who took the UKCAT from 2006 to 2014, and who applied to medical school through the Universities and Colleges Admissions Services (UCAS), a UK-based organisation whose primary role is to operate the application process for British universities. Independent variables examined included applicants’ gender, ethnicity, socio-economic status, domicile, secondary school attended and the UKCAT total score. The socio-economic status (SES) of the candidates was determined by parental occupation through National Statistics Socio-Economic Classification (NS-SEC) system and Index of Multiple Deprivation (IMD), an area-based measurement of material deprivation. Applicants were assigned as graduates or non-graduates on the basis of their highest qualification. Univariate analyses were run to compare applicants’ socio-demographic variables, particularly those associated with widening partici pation (WP). Multiple logistic regression was used to predict the odds of an applicant receiving an offer to study medicine, after adjusting for confounders.

Results Irrespective of graduate or non-graduate status, most applicants were from the highest socio-economic group and were from a white ethnic background. In general, non-graduate applicants performed significantly better on the UKCAT than graduate applicants. However, Graduate applicants who received offers had significantly better mean UKCAT scores (2697.7 points, SD=244.4) compared to their non-graduate colleagues who received offers (2657.7 points, SD=235.3), p<0.001. After adjusting for UKCAT score, the odds ratio of an offer for graduates vs. non-graduates was approximately 0.5 (OR=0.48, 95% CI 0.46–0.49). The applicant: offer ratio was 3:1 for non-graduates, and 17:1 for graduate applicants. The results further showed that the association between socio-economic disadvantage and the likelihood of getting an offer for medical school affected graduates and non-graduates in a similar way.

Summary of results The odds of getting an offer for in a UK medical school are higher if the applicant is non-graduate, female, from white ethnic background and from a high socio-economic background (i.e. from most affluent neighbourhood and having a parental/guardian in the managerial or professional occupations).

Discussion The aim of diversifying the medical student population on socio-economic grounds by attracting graduates has not been successful. Indeed, it is even more difficult for graduates to get offers now than it was a few years ago. While it would be reasonable for graduates not to receive offers because their indicator of potential (i.e. the UKCAT score) is lower than their non-graduate equivalent, what is of interest is that those graduates who are offered places have significantly higher UKCAT score scores than their non-graduate equivalents. Traditional programmes need to consider if their selection processes are biased towards school-leavers and they judge graduates differently. Graduate entry programmes need to consider if their institutional practices are merely reproducing the elite norms of medicine.

Conclusions The large proportion of UK medical students who are graduates do not differ significantly from non-graduates on demographic markers, other than age.

11:00 – 11:20 Research Paper

Name Kirsty Alexander


Position/Role PhD Student in Medical Education

Abstract Title “It’s going to be hard, you know…” UK school teachers’ perceptions of medicine and the implications for widening access

Abstract Authors Kirsty Alexander Jennifer Cleland Sandra Nicholson

Abstract Presenter Kirsty Alexander

Theme Diversity


Introduction Despite UK medical schools’ ongoing efforts, progress in ‘widening access’ (WA) has been slow. 80% of UK medical applicants come from only 20% of UK secondary schools, and 50% of schools send no applicants (Medical Schools Council, 2014). Studies indicate that some qualified candidates may even be advised against applying by their teachers (e.g. McHarg et al., 2007).Although teachers influence pupils’ university-related decisions (Gorard et al., 2006), very little is known about their perceived role in fostering pupils’ aspiration and preparation for medicine, nor how they perceive suitability for the profession. This study addresses this problematic knowledge gap.

Methods Qualitative data (interviews and field notes) were collected from seven ‘disadvantaged’ (WA) secondary schools and colleges of varying size, located in three very different UK areas. The interviews were with 11 teachers who held responsibility for advising pupils on university choices and field notes from meetings with other staff (e.g. headteachers). Data collection aimed to explore teachers’ motivations, perceptions and experiences.Template analysis was used initially to distinguish key themes and to develop a coding template which was applied to all data. The concepts of Amartya Sen’s ‘capability approach’ were then used to illuminate how these emergent themes might interact and influence a teacher’s decision whether, and how, to support pupils’ aspiration, preparation and application to medicine. Understandings were developed critically within interpretivist and social constructionist theory. Approval for this study was granted by the Ethics Committee of the College of Arts and Social Sciences, University of Aberdeen.

Results Teachers perceived three key interventions within their role: providing basic information when pupils first revealed an aspiration to medicine; building committed pupils’ capability to become competitive applicants; and having ‘realistic’ discussions with pupils about their chances of success. Teachers reported that medicine was a high-risk choice. Some felt it was their role to help build pupils’ emotional resilience to help them endure the application process and likely rejections.Teachers often chose (or felt compelled) to take a more ‘hands off’ approach to advising pupils, facilitating an environment in which the self-determination and choice of the pupil was seen as crucial. However, pupils’ unwavering determination was also considered problematic, as teachers perceived that highly-motivated pupils disregarded teachers’ advice to ‘keep their options open’.

Summary of results Teachers’ reported a risk-adverse stance towards medicine. Their perceived role was often to prepare pupils for a potentially unsuccessful application or direct them to other subjects. Strong determination for medicine was considered essential for pupils to achieve their aspiration, but also perceived to reduce pupils’ openness towards more viable alternatives.

Discussion School teachers are influential in shaping pupils’ university decisions and can either be key partners aiding – or gatekeepers abetting – medical schools’ WA initiatives. This study offers an understanding of what teachers in UK ‘disadvantaged’ schools expect, prioritise and value within their role. Despite the large heterogeneity in school type, size, location and teacher role, teachers consistently reported their perception of medicine as a high-risk subject, and their concern that high aspiration for medicine blinded pupils to other, equally worthwhile, career choices.

These perceptions may inhibit application rates from pupils in underrepresented groups if pupils are encouraged to be over-cautious or adapt to ‘safer’ choices. Medical schools should strive to understand the context and causes of teachers’ perceptions, before tackling them with well-informed interventions.

Conclusions Currently WA initiatives risk being developed uninformed of the perspectives of teachers and therefore may not be optimally effective. This study offers valuable insights into both how and why teachers manage their pupils’ aspirations to medicine, and how medical schools could work to address concerns, support teachers and improve practice.


Name Anouk Wouters

Organisation VUmc School of Medical Sciences, Research in Education

Position/Role Postdoc researcher & member of selection committee

Abstract Title Does selection decrease student diversity? Results from interviews with prospective applicants

Abstract Authors Anouk Wouters, Gerda Croiset, Ulviye Isik, Rashmi A. Kusurkar

Abstract Presenter Anouk Wouters

Theme Diversity


Introduction The Dutch medical admissions system has recently moved from a lottery to a qualitative selection procedure, providing a good context to study the effects of introduction of a selection procedure on the applicants’ motivation and their preparations for selection. The framework of Self-Determination Theory (SDT) of motivation was used to answer the questions: i) What types of motivation do high school students’ have for applying to medical school? ii) What are the factors influencing the development of these types of motivation and the underlying mechanisms? iii) What are the effects of selection on high school students’ motivation for studying medicine?

Methods A qualitative study using semi-structured interviews was set up using a constructivist approach. In 2015, study counsellors and 3rd–6th year students from one predominantly White and one multi-ethnicity high school in Amsterdam were purposefully sampled. Interviews with study counsellors yielded inputs for constructing the interview guide for students. Interviews were audio-recorded and transcribed verbatim. Participants were presented a summary of their interview as a form of member-checking. Data were coded using a template based on the motivation types, autonomous (originating from within the student) and controlled motivation (originating from external reasons) described by SDT, and open coding for factors that influence motivation. The first interview was coded together with UI. Another four interviews were coded independently by AW and UI and compared. Differences were discussed until consensus was reached, resulting in a refined coding scheme. AW coded the remaining interviews. Findings were discussed and finalized within the research team.

Results Three study counsellors and 24 high school students (18 females) participated in the study. Main reasons for studying medicine pertained to autonomous motivation (e.g. scientific interest and helping people), but controlled motivation (such as parental influence, prestige) was also reported. Healthcare experience positively influenced students’ autonomous motivation and served as a reality check for expectations. Although students did apply for selection, they perceived inequality in the access to the medical study. While having medical professionals in their network sparked students’ interest and facilitated easier access to healthcare experience, lack of opportunities had the opposite effect. Students from the predominantly White school were more likely to have doctor parents. Moreover, first generation university students received less support in their study choice and selection preparations.

Summary of results Having medical professionals in their network sparked students’ interest and facilitates access to healthcare experiences. A lack of such a medical network had a negative effect on students’ motivation for studying medicine.

Discussion Findings showed a complex interplay between healthcare experience, growing up in a medical family, selection and motivation. Healthcare experience, often one of the selection criteria, helps students to form autonomous motivation for the medical study. However, such experiences, as well as support in the selection process, seem unequally accessible to students. As a result, underrepresented students’ motivation decreases, which may cause them to refrain from applying. Thus the diversity of the student population and medical profession may suffer. Medical schools should take this into account when designing their selection procedure and could create possibilities for gaining healthcare experience. Involving a diverse group of medical students in recruitment activities can serve as a form of inspiration, role modelling and stimulation of autonomous motivation.

Conclusions This study highlighted an inequality in access to resources relevant for the study choice process and selection process related to students’ background characteristics. The perceived inequality could be a demotivating factor for students without a medical network, resulting in self-selection among these students and decreased diversity among applicants.


Name Amandip Bisel

Organisation Imperial College London

Position/Role PhD Researcher

Abstract Title Identifying and addressing the challenges of widening participation in medical education

Abstract Authors Amandip Bisel, Department of Medicine, Imperial College London Professor Sue Smith, Medical Education Research Unit, Imperial College London Dr Annalisa Alexander, Outreach Department, Imperial College London Professor Kevin G. Murphy, Department of Medicine, Imperial College London

Abstract Presenter Amandip Bisel

Theme Diversity


Introduction Students from non-traditional backgrounds are underrepresented in medical schools. Despite measures from UK medical schools to support potential applicants from this cohort via outreach schemes, applications and enrolled medical students remain the lower than for any comparable professional course in the UK. Those students from non-traditional backgrounds that do obtain a place at UK medical schools are also at the greatest risk of non-completion. Current strategies aimed at tackling this issue are wide and varied, and lack evidence of their effectiveness. The sector therefore does not currently know what the best strategy is to widen access to medical education.

Methods This research uses a mixed methods approach to identify application and outcome trends including perceived pre-entry and on-course challenges. Applications to one UK Medical School over a 5 year period are being analysed to identify emerging patterns in outcomes at each stage of the application process, based on a range of demographic data. Semi-structured 1–2–1 interviews have been conducted with 11 undergraduate medical students at the same institution, representing a cross-section of students from across each year of study. Focus Groups have also been undertaken with Year 12 and 13 students from a range of selective, non-selective, independent and state schools. We are developing a thematic analysis to draw out the factors identified from the interviews and focus groups as perceived pre-entry and on-course challenges. Findings will be used to make recommendations regarding pre-entry and transitional support provided and to develop metrics to monitor the impact of any changes.

Results Results from the thematic analysis of student interviews and focus groups will be presented as will the findings from the analysis of the retrospective application data. The thematic analysis will highlight the general perceived challenges faced by all potential applicants to UK medical schools alongside the additional perceived challenges faced by students from non-traditional backgrounds. The analysis of retrospective data will identify particular attributes, aside from academic performance, that are indicators of success/failure at each stage of the application process up to and including starting as a student. These findings will be used alongside the analysis of interview/focus group data to ascertain which challenges are actual and which are perceived.

Summary of results Early findings suggest that there are disparities between the information, support and advice received between students from traditional and non-traditional backgrounds not just when applying to university but also when transitioning between key stages. On-course knowledge and expectations also appear to vary between different cohorts of students.

Discussion It is intended that delegates will be able to use the findings of the student interviews regarding the applicant journey, transition and support available during their studies to reflect on their own institutional practices and provide transferable information about student perceptions and support requirements.

Pertinent questions to address will include:

• What can universities, and medical schools in particular, do to better support the provision of information, advice and guidance for non-traditional applicants at an early stage?

• How can medical schools work with schools/students to strengthen applications from a more diverse pool of students who meet the academic criteria to study medicine?

• What changes can medical schools make to improve the transition experience and support retention of students from non-traditional backgrounds?

Conclusions A key early conclusion is that more needs to be done by medical schools to support non-traditional students in gaining accurate information, advice and guidance at an earlier stage in their education and in their transition to, and throughout, their university education.


Name Barbara Griffin

Organisation Macquarie University


Abstract Title Who tries a second time to get into Medicine and are they successful?

Abstract Authors Barbara Griffin Wendy Hu Jaime Auton Boaz Shulruf Robbert Duvivier

Abstract Presenter Barbara Griffin



Introduction It appears a significant number of unsuccessful applicants to medical school reapply the following year. We do not know exactly how many do so and whether a second application is likely to be more successful.

Re-application typically involves repeat testing. UMAT scores improve on the second attempt (Puddey et al., 2014) but research on re-sitting interviews is scarce. Furthermore, it remains unknown if increased scores trans-late into successful shortlisting for interview or student offers.

Retest improvement raises potential inequality issues in selection, potentially more so for those from low SES background.

Methods We identified 665 people who applied to at least one NSW undergraduate medical degree in 2014 having also ap-plied (unsuccessfully) in 2013. They were compared (gender, age, socioeconomic status, school type and initial selection test results) to unsuccessful applicants in 2013 who did not reapply the following year (n = 3342).

Selection test results, including UMAT 1, UMAT 2, UMAT 3, and interview scores in 2013 were compared to re-peat test results in 2014. One university conducted panel interviews and two conducted multiple mini interviews. Only 26 attended the panel interview in both years, 28 attended one university’s MMI in both years, and 65 redid the other university’s MMI. Rate of selection success was also assessed.

Results Compared to those who didn’t reapply, re-appliers: were not more/less likely to be male, were younger, had higher ATAR, UMAT 1 and UMAT 3 scores on their first application, but equivalent UMAT 2 scores. They were not different in residential SES, but were significantly more likely to have attended an academically selective high school and less likely an independent (private) school.

Percentile ranking on UMAT 1, UMAT 2, and UMAT 3 improved, although 30 to 41% actually got lower scores on the second attempt. 53% of re-applicants were not interviewed in either year but 28.4% not interviewed in 2013 were in 2014. Interview scores significantly improved at each university. 20% of re-applicants secured a place in medical school, 86.5% of whom interviewed in both years.

Summary of results All selection test scores improved on average (but with significant numbers performing more poorly). Low SES did not appear to reduce the likelihood of re-application.

Discussion Our results confirm earlier studies with regard to improvements after repeat testing on the UMAT. However, only a relatively small number were more successful in gaining an interview the second year as a result of improved UMAT scores. In contrast, repeating an interview not only improved scores but appeared to improve chances of selection.

This study indicates that those of low SES background are not less likely to reapply if unsuccessful on their first attempt.

Second attempt scores for cognitive tests have been shown to have lower validity (Lievens), raising concerns about retesting. Further research needs to assess differential validity of first vs repeat interview scores.

Conclusions The effect of repeat testing on interview scores may mean these should be varied from one year to the next.


Name Sally Curtis

Organisation University of Southampton

Position/Role Principal Teaching Fellow

Abstract Title Developing a national approach to measuring widening participation in medical schools

Abstract Authors Sally Curtis, Peter Tang, Paul Lambe, Clare Owen, Daniel Smith, Paul Garrud

Abstract Presenter Sally Curtis

Theme Diversity


Introduction The Medical Schools Council Selection Alliance (MSCSA) board was established to lead the development of work relating to selection and widening participation (WP) in the UK. The Board’s data monitoring group aims to monitor and report progress made by medical schools to widen participation. It is important that data are collected on the demographics of the students studying medicine to determine whether progress has been made in WP, especially for those from lower socio-economic backgrounds. Additionally, it is important to analyse data on other personal characteristics of students to ensure WP in its broadest sense is being maintained

Methods The Medical Schools Council liaised with UK Medical Education Database (UKMED) for access to student data collated from a variety of sources, including the Higher Education Statistics Agency (HESA). Key tasks were established to ensure an accurate and efficient monitoring and reporting process of WP.

1. Undertake mapping of course codes, types and titles to the HESA data for all medical schools

2. Establish appropriate demographic/contextual variables

3. Produce baseline entry profiles reports, nationally and for individual medical schools.

4. Access UCAS data on applicants to medicine

5. Liaise with UKMED to ensure access to all relevant data including undergraduate and postgraduate progression data

6. Explore the use of HESA module data to replace GMC Medical School Annual Returns (MSAR) data to report student profiles and progression attrition

7. Develop research projects to provide support for the best practice use of contextual variables in admissions processes and widening participation initiatives.

Results Following consultation, all UK medical schools confirmed course code data, which fed into the medical school base line entry profiles report. From the reports each medical school was able to view their students’ entry profile by: course type and year of entry and compare to the national profile and Russell Group profile. The reports contained data on a variety of demographic variables including; gender, ethnicity, parental education, POLAR, IMD, social-economic classification and school type. Additional data has become available from UKMED including UKCAT bursary and postgraduate progressions data. Research projects have been developed to look at the contextual variables to establish recommendations for best practice in contextual admissions and to determine the effectiveness of WP programmes in attracting students from low socioeconomic backgrounds

Summary of results A robust platform for monitoring and reporting the progress made by UK medical schools to widen participation has been developed. This includes national and individual medical school year by year data profiles of key demographic variables

Discussion In accordance with the national agenda for widening participation of underrepresented groups in medicine, a rigorous foundation for reporting and monitoring progress has been developed. Enabling accurate data to be reported on a national level provides a clear picture of overall progress being made towards diversifying the profession and context for analysis of individual medical schools progress. These data provide opportunities for medical schools to determine the effectiveness of admissions processes and bespoke programmes in achieving their aims. These data may also be able to reduce the workload of medical schools by negating the requirement of annual reporting of data by individual schools. Further research provides a rich opportunity to develop informed support for best practice use of contextual variable in admissions and selection processes.

Conclusions Providing clear, accurate data to medical schools facilitates accurate monitoring of progress in widening participation to medicine. Effective use of these data can establish the national WP profile and also reduce the workload of medical schools, as well as supporting best practice in contextual admissions and other widening participation initiatives

12:40 Short Communication

Name Alwyn Louw

Organisation Stellenbosch University

Position/Role Senior Lecturer

Abstract Title Towards selection for success: How should we weigh selection factors?

Abstract Authors AJ Louw

J Blitz Dr. Ruairi Connolly

F Fredericks Diversity

Abstract Presenter AJ Louw

Theme * Selection Methods


Introduction An objective for medical schools in South Africa is to widen access for students into the health sciences professions with the aim to redress inequalities of the past. To do this where there are limited places and only 10% of all permissible applications eventually access medical school, makes this challenging.

Methods The study population is all students who graduated from the Faculty of Medicine and Health Sciences (Stellenbosch University) since 2008. This is a quantitative correlation-based retrospective study analysing academic success against components which determine the student’s selection factor (SF). Academic success was defined as completing the degree on time.

Results Three different components (including various sub-components) that make up the current Selection Factor of school leavers, were identified as the research variables. The academic success of all these students will be correlated with the different selection variables (singly or in groups) to ascertain positive and negative associations.

Summary of results The possibility that specific selection components could positively correlate with academic success, is high. The possibility of two or more of these variables correlating with successful or well performing students was identified and explored. Negative correlations could also supply valuable knowledge to guide the selection process.

Discussion Faculty is interested in the prediction of student success. Components of the Selection Factor may be helpful in this regard. Information about the utility of current selection components of the SF can indicate which add value to the process, and may inform a more appropriate weighting of each.

Conclusions As a public higher education institution with an obligation to redress access, but also needing to consider throughput of students, it would be of great interest to establish which (or which combination) of the currently used SF components offer us the most useful indicators of success.


Name Lokke Gennissen

Organisation Institute of Medical Education Research Rotterdam – Erasmus MC

Position/Role PhD student

Abstract Title Organic or Organised? Decision-making process for residency selection

Abstract Authors L.M. Gennissen, A. de la Croix, C.R.M.G. Fluit, J. de Graaf, K.M. Stegers-Jager, M. de Hoog

Abstract Presenter Lokke Gennissen

Theme Diversity


Introduction Since resident selection is a high-stakes complex decision-making process, it needs to be credible, fair and publically defensible. Increased awareness of the apparent disappearance of diverse medical graduates along the medical career pipeline questions current fairness. Diversity in resident selection has been discussed in literature. However, little attention is given to authentic decision-making practices in selection research. This study aims to shed light into this black box of authentic selection decision-making practices. The availability of a wide range of perspectives in this decision-making process is assumed to foster unbiased decision-making, therefore our focus will be on participation of committee members.

Methods Seven naturally-occurring resident selection group decision-making meetings were video-recorded. These meetings were in different highly competitive specialties in two Dutch regions. We used conversation analysis to unravel the way the conversations within these meetings were structured (conversational practices). With a data-driven and iterative analytic approach, we aimed to identify the current status of practices and the existing varieties herein, the so-called lay of the land”. Once we had a sense of the practices, we focussed on practices that resulted in eliciting multiple perspectives in the discussion, and conversational situations in which every participant had the possibility to raise concerns or thoughts.

Results The observed group discussions seemed to contain the following activities, though in different combinations and sequences: Introduction, introducing the rules of the discussion, giving the context of this selection, construction of a mutual version of the different applicants, scoring the applicants, feedback to be given to the applicants and practical aspects regarding residency. We identified a spectrum of practices, ranging from organized to organic, each having different effects on possibilities for committee members to participate. Organized styles are characterized by a conversationally dominant chair, silences in the interaction, and a slow topic development. Organic styles typically have a less conversationally dominant chair, more overlapping speech, clearly voiced disagreements, and more negotiation about the organization of the discussion.

Summary of results A wide variety of practices was found in the observed discussions. Nevertheless, there were common activity types which could be recognized in all discussions. We identified a spectrum of ways to structure these meetings and revealed consequences of this structure on participation of the selection committee members.

Discussion Our results show the relevance of interactional structures in group decision-making. The identified spectrum suggests that structure, interactional norms and conventions affects participation. Participation might be especially crucial in enhancing the selection of applicants who differ from the existing workforce in background, knowledge or viewpoints. Discussing a wide variety of viewpoints on the candidate might enable a fuller picture of the candidate. A fuller picture is considered to mitigate the effects of (unconscious) subjective judgements. Although the identified spectrum does not provide one clear cut effective solution, it could help groups to become aware of these effects and to make conscious, informed decisions in constructing their desired group interactions. More empirical work is needed in this field to further unravel this complicated process.

Conclusions By disassembling the structure of resident selection group decision-making meetings, we identified a spectrum of possible ways to structure these meetings and their effects on the participation of committee members. The spectrum might help groups to make a conscious, informed choice in how they want to structure their group meetings.

13:50 – 15:20 Symposium

Name Fiona Patterson

Organisation Work Psychology Group

Position/Role Founding Director

Abstract Title Enhancing diversity and widening access in selection for medical education

Abstract Authors Professor Fiona Patterson, is founding Director of Work Psychology Group and holds a visiting researcher position at the University of Cambridge. Associate Professor Deborah O’Mara is the Assessment Lead for the Sydney MD Program. She has conducted several research studies on the integration of admissions and in-program assessment. Karen Stegers-Jager is assistant professor at the Institute of Medical Education Research Rotterdam. Her research focuses on admissions for all students, in particular non-traditional backgrounds applicants. Dr Elana Taipapaki Curtis, is Director Vision 20:20 at the University of Auckland focusing on recruitment, admission and support for support for Maori and Pacific students.

Abstract Presenter Fiona Patterson, Deborah O’Mara, Karen Stegers-Jager, Elana Taipapaki Curtis.

Theme Diversity


Introduction The need to widen access and promote diversity for selection into health professional education in general and medical education in particular is widely acknowledged internationally and is recognised as an important strategy for meeting future health workforce needs. It is a stated national goal of many countries that the health professional workforce reflects the populations served. However, given the over-reliance on academic indicators as the primary selection criterion for medical school internationally, and despite the introduction of non-academic assessments such as multi-mini interviews, access to medical education remains limited for many underrepresented communities.

Methods Participants will receive feedback on the latest research on diversity and widening access in medical education from four continents. Each presenter will focus on a different research problem which will provide evidence on methods for widening access and enhancing diversity:

• Evaluation of differential attainment on selection methods (including academic records, cognitive ability tests, situational judgement tests) for both undergraduate and postgraduate medical education in the UK.

• The effectiveness of strategies trialled in Australia to increase the diversity of the interview pool for the medical education selection process across a range of social-economic variables.

• Evidence from the Netherlands will be presented that shows that the diversity of the medical student population is maintained by assessing candidates for medical school on several independent characteristics simultaneously.

• The effect of tertiary recruitment, admission, bridging/foundation education and retention on indigenous health workforce development in New Zealand.

Results The trial of two versions of a SJT at the Sydney Medical School identified that the SJT provided complimentary information to the MMI. Analyses to-date suggest that had been used to select candidates for interview instead of only the GAMSAT, the diversity of applicants interviewed would have differed. The association with in-program performance will be available in late 2017.

Findings at Erasmus MC medical school suggest that using a compensatory selection procedure in which appli-cants can compensate for lower scores on academic criteria with higher scores on non-academic criteria and vice versa would increase diversity of the student population. Evidence suggests that the use an SJT in selection has promise in widening diversity as SJTs do not disadvantage students from low SES groups and female candidates.

Summary of results The results available so far are short term. Preliminary analyses does suggest that the association between new selection methods such as a SJT and increasing diversity is positive. However, all methods do not have the same positive impact on different aspects if diversity.

Discussion Issues to be discussed include the sustainability of introducing time intensive selection methods that may widen the diversity of applicants for interview and/or admission. This may have intended and unintended consequences on faculty, candidates selected and candidates no longer selected due to a change in policy for selection.

The pros and cons of each case study will provided a focus for discussion. For example in the Netherlands is was found that the gap in the scores between White and BME candidates was the same for the cognitive ability test and SJT scores, which is a finding consistent with other assessments within medical education. However, findings did not find any beneficial effects in terms of reducing effect sizes associated with ethnicity.

Conclusions The discussion with the audience will focus on how we can harness existing evidence to re-configure selection systems in medical and health professional education internationally to increase the diversity of students.

Documents

INTERNATIONAL SELECTION IN THE HEALTH PROFESSIONS … · individual books, or the Total UMAT Sum Score were not significantly correlated with any pre-clinical or clinical unit outcomes