A descriptive analysis of the use of workplace-based assessments in UK surgical training

Joseph Shalhoub1, Cristel Santos2, Maria Bussey2, Ian Eardley2, William Allum2

1 Department of Surgery & Cancer, Imperial College London, UK2 Joint Committee on Surgical Training, UK

For Correspondence

Mr William Allum

ISCP Surgical Director

Joint Committee on Surgical Training

35 - 43 Lincoln’s Inn Fields

London WC2A 3PE, UK

E-mail: william.allum@rmh.nhs.uk

Tel: +44 (0) 20 8661 3982

Word Count3,266 Words

1

Abstract

Background

Workplace-based assessments (WBAs) were formally introduced in the UK in 2007.

The study aim was to describe use of WBAs by UK surgical trainees and examine

variations by training region, specialty or level of training.

Methods

The database of the Intercollegiate Surgical Curriculum Programme (ISCP) was

interrogated for WBAs between August 2007 and July 2013, with in-depth analysis of

two periods: August 2011 to July 2012, and August 2012 to July 2013.

Results

Numbers of validated WBAs per trainee per year increased more than 7-fold, from

median 6 per trainee in 2007-2008, to 39 in 2011-2012, and 44 in 2012-2013. In

2011-2012, 58.4% of core trainees completed the recommended 40 WBAs; only

38.1% of specialty trainees achieving 40 validated WBAs. In 2012-2013, these

proportions increased to 67.7% and 57.0% for core and specialty trainees,

respectively. Core trainees completed more WBAs per year than specialty trainees in

the same training region. London core trainees completed the highest numbers of

WBAs in both 2011-2012 (median 67) and 2012-2013 (median 74). There was a

peak in WBAs completed by London specialty trainees in 2012-2013 (median 63).

The most validated WBAs were completed by ST1/CT1, with a gradual fall in median

WBAs to ST4, followed by a plateau; in 2012-2013 there was an increase in WBAs at

ST8. Core surgical trainees complete ~50% ‘operative’ (PBA/DOPS) and ~50% ‘non-

operative’ assessments (CBD/CEX). During specialty training, PBAs represented

~46% of WBAs, DOPS 11.2%, CBD ~23% and CEX ~15%.

Conclusions

UK surgical trainees are, on average, undertaking one WBA per week. Variation

exists in use of WBAs between training regions. Core trainees tend to use the

spectrum of WBAs more frequently than their senior colleagues. Further work is

required to examine the role of WBAs in assessment, and engagement and training

of trainers in processes and validation of WBAs.

Key Words

2

Workplace-based assessment, surgical training, Intercollegiate Surgical Curriculum

Programme

3

Introduction

Workplace-based assessments (WBAs) were formally introduced in the UK in 2007

to accompany the competency-based Modernising Medical Careers postgraduate

training programme 1. WBAs are delivered as part of the Intercollegiate Surgical

Curriculum Programme (ISCP), a joint activity by the four surgical Royal Colleges 2.

The ISCP has provided each surgical specialty with a comprehensive syllabus, a

teaching and learning framework, an assessment system, a repository for evidence

on individual trainee progress and an interactive web platform.

The use of WBAs is a mandatory component of each surgical trainee’s portfolio

through both core surgical training and specialty training. The main WBA assessment

tools include procedure-based assessments (PBA), direct observation of procedural

skills (DOPS), clinical evaluation exercise (CEX) and case-based discussion (CBD),

as well as multi-source feedback (MSF). A glossary of terms used is presented in the

Appendix.

PBA involves direct observation of an index procedure or operation with comments

given on important steps, tasks or skills at the pre-, intra- and post-operative stages

of the procedure, considered to be essential for its safe and successful completion.

The assessment form for a PBA is specific to the procedure or operation being

assessed. Similar to PBA, DOPS are more generic and relate to either less complex

procedures or parts of a larger procedure. The domains of assessment in DOPS are

generic and fixed and considered to be transferable across this spectrum of tasks.

CEX is an assessment of a trainee’s clinical skills, for example history taking, clinical

examination, and information giving. CBD refers to a formalized discussion related to

the care of a patient with a focus on knowledge and attitude. Each of these WBAs is

completed by both trainee and trainer, validated by the trainer and is recorded within

ISCP. MSF is an opportunity for members spread across the multidisciplinary health

care and administrative team to offer anonymous feedback on the trainee. The

results are then discussed at a meeting with the assigned educational supervisor and

the trainee’s annual review of competence progression (ARCP) 3.

At present, the Joint Committee on Surgical Training (JCST) recommends that each

trainee completes a minimum of 40 WBAs per year of training. This approximately

equates to one WBA per week, although trainees and trainers are encouraged to

4

complete more with a spread throughout a training post to demonstrate progression

of clinical and technical skills 5.

The purpose of this study is to report on the use by surgical trainees in the UK of the

largest platform for competence-based training worldwide. We aim to describe the

use of WBAs by surgical trainees and to examine any variations observed by training

region, surgical specialty or level of training.

5

Methods

Every assessment recorded in trainees’ portfolios is recorded with background

information about the individual trainee. These data were collated to facilitate

analysis to quantify the use of the WBAs. The ISCP database was interrogated for

WBAs linked to trainees’ portfolios between August 2007 and July 2013, with an in-

depth analysis of two periods: August 2011 to July 2012, and August 2012 to July

2013. These two time periods represent the most recent two years for which, at the

time of data analysis, complete data was available. Trainees in Core Surgery and all

but one of the surgical specialties were included from 2007. Vascular surgery only

became a separate specialty in 2012 but has been included for the second time

period. Trauma and Orthopaedics trainees were variably included in the first years of

ISCP as they were also able to use the Orthopaedic Competence Assessment

Project (OCAP) system; all had transferred to ISCP by 2012/13.

Medical graduates in the UK commence postgraduate clinical training with a 2-year

generic ‘Foundation Programme’, from which candidates apply through a competitive

national selection system for core surgical training (CST) programmes – previously

termed the senior house officer (SHO) grade. Following successful completion of

CST and the Intercollegiate Membership of the Royal College of Surgeons

examinations, trainees apply through competitive national selection for higher

surgical training (registrar grade) in one of the 10 surgical specialities 6.

The main WBA assessment tools included in the analysis were DOPS, PBA, CEX

and CBD for all trainees in both core and higher specialty training, although there

was some data available for assessment of audit, teaching and multi-source

feedback. All trainees with an ISCP appointment type of core surgery (CT), fixed-

term specialty training appointments (FTSTA), locum appointments for training (LAT),

SHO, specialist registrars (SpR), specialty registrars (StR), and StRs converted from

SpR were included in the analysis. CT1 and CT2, as well as (for ‘run-through’

specialties/programmes) ST1 and ST2 were combined for analysis for consistency of

reporting on the early years of training.

Only WBAs which had been trainer validated were included. Entries with missing

variables were excluded. Data were anonymised and allocated a unique identifier to

avoid duplicate counting, and analysed by training region (Local Education and

Training Board [LETB] in England and Deanery in Wales, Scotland and Northern

6

Ireland), surgical specialty, and level of training at the time of the assessment.

Records without appointment type, training region, specialty or training level were

excluded from further analysis. After data cleaning, data comparison and summary

statistics were performed using STATA version 11 (StataCorp LP, Texas, USA). For

the analysis, medians and percentiles were used to represent the data as these are

not affected by extreme values.

7

Results

Records without appointment type, training region, specialty or training level totalled

1,433 for 2011-2012 and 1,355 for 2012-2013; these records were excluded from

further analysis. A total of 754,165 ISCP WBAs were validated by UK surgical

trainees between August 2007 and July 2013 (Table 1). Approximately two thirds of

trainees included are male (Table 1). There has been an increase in both the

number of WBAs validated using the ISCP and the number of surgical trainees using

the ISCP for WBAs year on year between 2007 and 2013. The number of validated

WBAs per trainee has increased more than 7-fold, from a median of 6 per trainee in

2007-2008, to 39 per trainee in 2011-2012, and 44 per trainee in 2012-2013.

In 2011-2012, 55.5% of core surgical trainees had completed the JCST-

recommended 40 WBAs, with only 35% of specialty trainees in surgery achieving 40

validated WBAs. In 2012-2013, these proportions increased to 65.5% and 53.6% for

core and specialty trainees, respectively Figure 1 illustrates the number of WBAs

being completed by core and specialty trainees for these two time periods.

WBAs across training regions

The use of WBAs across training regions for 2011-2012 and 2012-2013 can be seen

in Table 2. For the year 2011-2012, the median number of WBAs undertaken by

trainees varied from 30 to 50 (Figure 2a). For 2012-2013 this variation was between

39 and 67, with the peak number undertaken by London trainees. After separating

core and specialty trainees (Figures 2b and 2c), core trainees tended to complete

more WBAs per year than specialty trainees in the same training region. London core

surgical trainees completed the highest numbers of WBAs in both 2011-2012

(median 67) and 2012-2013 (median 74). There was also a peak in WBAs completed

by London specialty trainees in 2012-2013 (median 63).

WBAs across core surgical training and surgical specialties

The use of WBAs across core surgical training and individual surgical specialties for

2011-2012 and 2012-2013 can be seen in Table 3. The median number of WBAs

completed by core surgical trainees increased from 43 in 2011-2012 to 48 in 2012-

2013. Excluding vascular surgery, which became a recognised surgical specialty in

2012, the median number of WBAs completed by specialty trainees in 2011-2012

ranged from 21 in trauma and orthopaedic surgery, to 41 in neurosurgery, oral and

maxillofacial surgery, and otolaryngology. By 2012-2013, this range increased to

8

between 39 in paediatric surgery, and 45 in otolaryngology and vascular surgery. For

trauma and orthopaedic surgery, the median number of validated WBAs almost

doubled from 21 in 2011-2013 to 41 in 2012-2013 reflecting the change from OCAP

to ISCP.

WBAs across surgical training levels

The use of WBAs across surgical training levels for 2011-2012 and 2012-2013 can

be seen in Table 4. The most validated WBAs were completed by ST1/CT1 trainees,

with a gradual fall in the median number of WBAs to ST4. This is followed by a

plateau, however in 2012-2013 there was an increase in WBA numbers at the ST8

training level.

Types of WBAs being used

Analysis of the type of WBA shows core surgical trainees complete approximately

50% ‘operative’ or interventional (PBA and DOPS) and 50% ‘non-operative’

assessments (CBD and CEX), with the remainder (namely assessment of audit,

observation of teaching and multi-source feedback) contributing a small proportion

(Figure 3a). During core training, PBAs contributed approximately 14% of validated

WBAs, whilst DOPS represented about 31%; CBDs and CEXs were equally split at

about a quarter of validated WBAs each. During specialty training, PBAs represented

approximately 46% of WBAs, DOPS 11.2%, CBD about 23% and CEX approximately

15%; the remaining WBA types again contributing a small proportion (Figure 3b).

The proportion of WBA types remained relatively stable from 2011-2012 to 2012-

2013. Variability in the type of WBAs performed across the specialties was observed.

9

Discussion

The ISCP has offered a curriculum for surgical training and a platform for monitoring

of the competency-based training that has accompanied the Modernising Medical

Careers era of postgraduate medical training 1. At present, competence in this

system is being assessed primarily through the use of WBAs. Few would disagree

that the introduction of WBAs to support competency-based training is one of the

most significant changes to educational policy in medical and surgical training in

recent years.

The ISCP is designed to allow the trainee to demonstrate progression in knowledge

and clinical and technical skills, as well as achieving professional capabilities using

WBAs as formative tools. Engagement with this approach to training has required

faculty and trainee reinforcement of its principles and, while acknowledging a specific

number should not be an absolute, the figure of 40 has been selected as it

approximates to a minimum of one WBA per week. For each training year, there are

expected levels of competence defined within the curriculum which are reviewed on

an annual basis by a team comprising a programme director, independent

educational supervisors, the postgraduate Dean and external specialty advisors. At

this review the formative assessments undertaken by clinical supervisors, together

with an overview report from an assigned educational supervisor, are considered in a

summative way to confirm the appropriateness of a trainee proceeding onto the next

year of training.

Whilst WBAs have been a part of surgical training since 2007, from the 2012-2013

academic year the American Board of Surgery has required the completion of in-

program assessments 4. These are broadly divided into operative performance

assessments and clinical performance assessments. At the time of implementation in

2012-2013, two of each of these assessments was required, increasing to six of each

from the 2015-2016 academic year 4. The Operative Performance Rating System

(OPRS) is employed for operative performance assessment. OPRS WBAs are

broadly similar to PBAs in that they are operation-specific, examine and assess

defined steps of a procedure, with each step scored against a 5-point Likert (as

compared with ‘satisfactory’ and ‘needs development’ used for PBAs). For clinical

performance assessment, similar to CEX in the UK, the American Board of Surgery

uses mini-CEX and Clinical Assessment and Management Exam – Outpatient

(CAMEO) for the direct observation of clinical assessment of patients.

10

The purpose of this study was to examine the use of WBAs across a number of

denominators in UK surgical training. This is the first time that data describing the

use of WBAs and their validation has been released by the ISCP. The data reflects

that there has been a good uptake with regards the use of WBAs since the

commencement of their use in 2007, with a steady rise in WBAs per trainee per year

from 2007 to 2013.

However, the proportions of trainees who are meeting the JCST’s set minimum for

validated WBAs of 40 per annum – although rising – remains below 70% and 60% for

core and specialty trainees, respectively, for 2012-2013. It may be that these trainees

are completing 40 WBAs, but these are not being validated. At present there is a

perceived emphasis on the numbers of WBAs completed, as compared with their

content; this may be driven, at least in part, by the minimum numbers requirement.

While this may show engagement with training it is essential that quantity does not

trump quality 7.

One of the implications of this study is to highlight that additional support is

necessary for trainees to facilitate their use of WBAs, but importantly there is

undoubtedly a need to ensure that trainers are engaged, trained and supported by

Deaneries / LETBs in the processes related to WBA completion and validation.

Mechanisms for this include through accredited courses, for example Training the

Trainers (TtT), Training and Assessment in Practice (TAiP) and Training and

Assessment in the Clinical Enviroment (TrACE), and through the Faculty of Surgical

Trainers (affiliated with the Royal College of Surgeons of Edinburgh).

Core trainees are completing more WBAs than specialty trainees but it is expected

that as trainees who have ‘grown up’ with WBAs become the majority within specialty

training, this difference will be minimised. There is no clear difference with regards

the uptake of WBAs by different specialties, in particular the numbers for trauma and

orthopaedic surgery have come up to the level of other surgical specialties in 2012-

2013 following the transition onto ISCP from OCAP.

The variation in completed WBAs by training region highlights differences in

approach across the UK by Deanery / LETB. For example in London there was a

requirement in February 2012 for trainees to complete 80 per annum 8. This

requirement has resulted in a peak in WBAs in London from both core and specialty

11

trainees in 2012-2013. Despite this peak, the medians remained below 80 per annum

at a median of 74 and 63 per annum for core and specialty trainees in surgery,

respectively. Powell and colleagues specifically evaluated trainee and trainer opinion

with regards increasing the number of WBAs per trainee per year 7. Several concerns

were highlighted including a reduction in efficacy with increasing number, an

increased burden on trainers and the development of trainees focussed on

quantitative rather than qualitative outcomes. There is however no clear guidance in

the optimum number and further evaluation is required.

The dominance of ‘operative’ or interventional WBAs, particularly PBAs, amongst

specialty trainees is perhaps unsurprising. Work examining WBAs undertaken in

Wales emphasised that usage amongst specialty trainees is skewed towards PBAs,

with the authors concluding that the annual incremental uptake of WBAs between

2007 and 2013 “is reflective of the acceptability and reliability of the PBA and the

increasing confidence of trainers and trainees with WBAs” 9. However WBAs should

also be used to evaluate non-operative skills such that the more equitable proportion

of the different WBA tools seen in Core training should be re-established in Specialty

training.

The use of WBAs in surgical training is an evolutionary process. Although the

educational value of PBAs is supported by an evidence base 3,11 there is uncertainty

about the reliability of some of the other tools 3 10. The existing evidence base for

WBAs is related to their formative rather than their summative use, and trainee and

trainer survey data underlines the desire at present that their use reflects this7. The

approach to the introduction of WBAs has somewhat inadvertently resulted in them

being used as assessments of learning. If WBAs are used as assessments for

learning additional work is required to support the educational value of the other key

WBAs, namely DOPS, CEX and CBD. Indeed the GMC has highlighted the need to

separate the use of WBAs into Supervised Learning Events (SLEs, an assessment

for learning, formative) and Assessment of Practice (AoP, an assessment of learning,

summative) 11.

Study limitations

This work represents a preliminary, purely descriptive analysis, which will form the

basis for further analyses of the rich dataset. The analysis looks at number of WBAs

completed, not how WBAs are being used nor the content or outcomes of the

assessment, i.e. whether trainees are performing appropriately for their stage in

12

training or if they are showing an improvement. Furthermore, no qualitative data has

been analysed to date. Given this analysis focuses on validated WBAs, it is

questionable that the mechanisms that trainers have for validating WBAs are

universally uniform.

Future work

Future work will aim to address the shortcomings of this present analysis, including

an examination of the qualitative elements within WBAs, how WBA numbers relate to

outcomes in terms of WBA final ratings and ARCP outcomes, and an analysis of

index surgical procedures numbers against PBAs through integration of the e-

logbook. With regards examining trainers’ input, an analysis of the quality of

feedback is planned as well as a review of individual trainers’ training activities.

There is an aim to introduce an automated report within ISCP, which may act as a

data source on training for all stakeholders.

13

Conclusions

This study confirms quantitatively the uptake of WBAs in surgical training in the UK. It

identified that surgical trainees are on average undertaking one WBA per week. The

majority of WBAs have assessed operative or interventional skills with lesser

emphasis on clinic-based interactions.

This study has shown some variation in the use of WBAs between the training

regions. Core trainees tend to use the spectrum of WBAs more frequently than their

senior colleagues. As trainees progress through training, operative or interventional

assessments predominate with limited use of non-operative assessments. Further

work is required to examine the role of WBAs in assessment, and the engagement

and training of trainers in the processes and validation of WBAs.

14

References

1. Health Do. Modernising Medical Careers, 2003.2. McKee RF. The Intercollegiate Surgical Curriculum Programme (ISCP). Surger

(Oxford) 2008;26(10):411-16.3. Shalhoub J, Vesey AT, Fitzgerald JE. What Evidence is There for the Use of

Workplace-Based Assessment in Surgical Training? J Surg Educ 2014;71(6):906-15.

4. Surgery ABo. Training & Certification - Resident Performace Assessments. Secondary Training & Certification - Resident Performace Assessments. http://www.absurgery.org/default.jsp?certgsqe_resassess.

5. Marriott J, Purdie H, Crossley J, et al. Evaluation of procedure-based assessment for assessing trainees' skills in the operating theatre. Br J Surg 2011;98(3):450-7.

6. Fitzgerald JE, Giddings CE, Khera G, et al. Improving the future of surgical training and education: Consensus recommendations from the Association of Surgeons in Training. Int J Surg 2012;10(8):389-92.

7. Powell HRF, DiMarco AN, Saeed SR. Trainee and trainer opinion on increasing numbers of workplace-based assessments. Bulletin of the Royal College of Surgeons of England 2014;96(5):160-62.

8. Khera G, Baird E. Annual Workplace-Based Assessments. In: Standfield NJ, ed. London, 2012.

9. Fishpool SJC, Stew B, Roberts C. Otolaryngology WBAs in the Wales Deanery: the first six years. Bulletin of the Royal College of Surgeons of England 2014;96(5):164-66.

10. O'Kane R. Workplace Based Assessments: The Association of Surgeons in Training, 2010.

11. Council GM. Learning and assessment in the clinical environment: the way forward: General Medical Council, 2011.

15

Tables

Table 1The adoption of workplace-based assessment (WBA) tools. Data represent trainer-validated WBAs between 2007 and 2013.

2007-2008 2008-2009 2009-2010 2010-2011 2011-2012 2012-2013 TOTAL 2007-2013

WBAs total 11,153 41,685 273.8% 82,627 98.2% 141,708 71.5% 210,249 48.4% 266,743 26.9% 754,165

Number of trainees 1,549 3,042 3,851 4,705 5,459 5,808

Female 415 26.8% 773 25.4% 1,037 26.9% 1,258 26.7% 1,497 27.4% 1,618 27.9%

Male 1,121 72.4% 2,249 73.9% 2,799 72.7% 3,437 73.0% 3,943 72.2% 4,172 71.8%

No information 13 0.8% 20 0.7% 15 0.4% 10 0.2% 19 0.3% 18 0.3%

Median WBAs per trainee 6 13 20 29 39 44

16

Table 2Workplace-based assessments by training region. HEE, Health Education England; WBA, workplace-based assessment.

Deanery / Local Education & Training Board

Mean number ofWBAs per trainee

Median number ofWBAs per trainee Inter-quartile range

2011-12 2012-13 2011-12 2012-13 Change 2011-12 2012-13East Midlands HEE 35.0 38.0 35.0 41.0 6 22-45 23 27-48 21

East of England HEE 33.4 37.9 34.0 39.0 5 17-47 30 24-51 27

KSS HEE 44.5 47.8 49.5 52.0 3 24-65 41 28-66 38

London HEEs 45.3 61.3 40.0 67.0 27 20-68 48 37-84 47

North West (Mersey Sector) HEE 33.2 41.1 34.0 43.0 9 22-44 22 30-51 21

North West (North West Sector) HEE 38.8 44.2 41.0 45.0 4 34-48 14 39-54 15

Northern East HEE 38.9 41.2 40.0 41.0 1 31-48 17 33-48 15

Northern Ireland 34.9 40.2 38.0 41.0 3 23-43 20 36-47 11

Scotland 32.9 37.2 38.0 39.0 1 16-45 29 26-46 20

South West HEE 32.2 36.9 33.0 39.0 6 17-45 28 23-48.5 26

Thames Valley HEE 36.7 38.7 40.0 42.0 2 17-50 33 27.5-49 22

Wales 30.9 37.1 30.0 40.0 10 19-42 23 19-52 33

Wessex HEE 33.5 37.4 37.0 39.0 2 17-47 30 26-47 21

West Midlands HEE 41.3 47.1 42.0 48.0 6 22-55 33 34-61 27

Yorkshire and the Humber HEE 31.0 37.5 32.0 39.0 7 17-44 27 24-48 24

17

Table 3Workplace-based assessments by surgical specialty. Core surgical training takes into account all trainees at core level (CT1/ST1 and CT2/ST2)

including neurosurgery, trauma and orthopaedic surgery, and Scottish trainees. The deadline for trauma and orthopaedic trainees to register

with ISCP was the 4th of April 2011. WBA, workplace-based assessment.

SpecialtyMean number of

WBAs per traineeMedian number ofWBAs per trainee Inter-quartile range

2011-12 2012-13 2011-12 2012-13 Change 2011-12 2012-13Core Surgical Training 43.0 47.8 43.0 48.0 5 28-56 28 32-63 31

Cardiothoracic Surgery 36.8 43.5 34.0 41.0 7 20-47 27 26-54 28

General Surgery 37.3 44.4 38.0 42.0 4 25-46 21 31-57 26

Neurosurgery 43.0 45.8 41.0 42.0 1 28-52 24 32-59 27

Oral and Maxillofacial Surgery 41.0 45.8 41.0 42.0 1 30-50 20 33-58 25

Otolaryngology 41.1 46.6 41.0 45.0 4 26-54 28 35-58 23

Paediatric Surgery 34.2 34.1 37.0 39.0 2 23-47 24 20.5-47 27

Plastic Surgery 38.0 44.4 35.0 42.0 7 21.5-48.5 27 28.5-56 28

Trauma and Orthopaedic Surgery 24.6 41.9 21.0 41.0 20 12-35 23 28-54 26Urology 33.8 38.9 36.0 39.0 3 23-43 20 27-48 21Vascular Surgery - 48.6 - 45.0 45 - - 34-59.5 26

18

Table 4Workplace-based assessments by level of training. ST8 level does not include oral and maxillo-facial surgery, and urology as trainees in these

specialties finish their training at ST7. It is also noted that the first cohort of Modernising Medial Careers trainees are finishing their training in

August 2013. CT, core surgical training; ST, specialty training in surgery; WBA, workplace-based assessment.

Training levelMean number of

WBAs per traineeMedian number ofWBAs per trainee Inter-quartile range

2011-12 2012-13 2011-12 2012-13 Change 2011-12 2012-13

ST1/CT1 41.7 47.8 44.0 49.0 5 21-57 36 26-65 39

ST2/CT2 37.5 40.8 38.0 43.0 5 17-51 34 20-57 37

ST3 26.8 33.3 23.0 32.0 9 10-41 31 12-48 36

ST4 24.9 32.0 21.0 31.0 10 9-38 29 13-45 32

ST5 25.6 30.6 23.0 29.0 6 9-39 30 10-44 34

ST6 24.6 30.4 21.0 28.0 7 9-38 29 12-44 32

ST7 27.4 30.3 25.0 28.0 3 13-39 26 12-43 31

ST8 26.7 35.5 21.5 33.0 12 5-40 35 20-45 25

19

Figure Legends

Figure 1Distribution of frequency of validated workplace-based assessments (WBAs). (a)

Core surgical trainees 2011-2012. (b) Specialty trainees in surgery 2011-2012. (c)

Core surgical trainees 2012-2013. (d) Specialty trainees in surgery 2012-2013.

Figure 2Median numbers of validated workplace-based assessments by training region. (a)

For all trainees 2011-2012 (dark blue) and 2012-2013 (light blue). (b) For core

surgical trainees 2011-2012 (blue) and 2012-2013 (red). (c) For specialty trainees in

surgery 2011-2012 (blue) and 2012-2013 (red).

Figure 3Break down of number of workplace-based assessment per type. (b) For core

surgical trainees 2011-2012 (pink) and 2012-2013 (red). (c) For specialty trainees in

surgery 2011-2012 (light blue) and 2012-2013 (dark blue). PBA, procedure-based

assessment; DOPS, direct observation of procedural skill; CBD, case-based

discussion; CEX, clinical evaluation exercise; OT, observation of teaching; MSF,

multi-source feedback.

20

Appendix

Glossary of terms presented in alphabetical order. Adapted from 3.

AoP Assessment of PracticeARCP Annual Review of Competence ProgressionCAMEO Clinic Assessment and Management Examination - OutpatientCBD Case-Based Discussion

Formalized discussion related to the care of a patient with a focus on knowledge and attitude

CEX Clinical Evaluation ExerciseAn assessment of a trainee’s clinical skills (e.g., history, examination, and information giving)

CST Core Surgical TrainingCT Core TraineeDOPS Direct Observation of Procedural Skill

Similar to PBA, but more generic and relate to either less complex procedures or parts of a larger procedure. The domains of assessment in DOPS are generic and fixed and considered to be transferable across this spectrum of tasks

FTSTA Fixed-Term Specialty Training AppointmentGMC General Medical CouncilHEE Health Education EnglandISCP Intercollegiate Surgical Curriculum ProgrammeJCST Joint Committee on Surgical TrainingLAT Locum Appointment to TrainingLETB Local Education and Training BoardMSF Multi-Source Feedback

An opportunity for members spread across the multidisciplinary health care and administrative team to offer anonymous feedback on the trainee. The results are then discussed at a meeting with the assigned educational supervisor and the ARCP

OCAP Orthopaedic Competence Assessment ProjectOPRS Operative Performance Rating SystemOT Observation of Teaching

Workplace-based assessment permitting the documentation of an observed teaching episode carried out by the trainee

PBA Procedure-Based AssessmentDirect observation of an index procedure or operation with comments given on important steps, tasks or skills at the pre-, intra- and post-operative stages of the procedure, considered to be essential for its safe and successful completion

SHO Senior House OfficerSLE Supervised Learning EventST Specialty TraineeSpR Specialist RegistrarStR Specialty RegistrarTAiP Training and Assessment in PracticeTrACE Training and Assessment in the Clinical EnvironmentTtT Training the TrainersWBA Workplace-Based Assessment

21

22