The Assessment of Observed Practice: A Literature … Assessment of Observed Practice: A Literature Review Dr Elizabeth Mills Professor Alison Blenkinsopp Professor Robert K McKinley

1

Department of Medicines ManagementSchool of Pharmacy

Keele University

The Assessment of Observed Practice: A Literature Review

Dr Elizabeth MillsProfessor Alison BlenkinsoppProfessor Robert K McKinley

Dr Patricia Black

June 2011

2

Dr Elizabeth Mills, Prof Alison Blenkinsopp, Prof Robert K McKinley, Dr Patricia Black

List of Tables......................................................................................................................................4

Acknowledgements ...........................................................................................................................4

Glossary.............................................................................................................................................6

List of Abbreviations ..........................................................................................................................8

Executive Summary ...........................................................................................................................9

Background .....................................................................................................................................29

Introduction.....................................................................................................................................31

What needs to be assessed? ........................................................................................................32

Methods of assessment ...............................................................................................................34

Principles of selection of assessment methods.............................................................................35

Methods..........................................................................................................................................37

Utility Index Ratings .....................................................................................................................40

Presentation of results.................................................................................................................41

Results.............................................................................................................................................42

Direct Observation of Clinical Skills (DOCS e.g. mini-CEX) .............................................................44

Direct Observation of Procedural/Practical Skills (DOPS) ..............................................................50

Objective Structured Clinical Examination (OSCE).........................................................................56

Incognito Standardised Patients...................................................................................................64

Multi Source Feedback.................................................................................................................67

Patient Surveys ............................................................................................................................73

Case-based Discussion .................................................................................................................76

The Long Case ..............................................................................................................................78

Portfolios of Evidence ..................................................................................................................81

Programmes of assessment .........................................................................................................85

Assessor training..........................................................................................................................92

Discussion........................................................................................................................................94

Utility of individual tools ..............................................................................................................94

Programmes of assessment .......................................................................................................100

Assessor training........................................................................................................................101


Contents

3

Strengths and limitations of this project.....................................................................................101

Utility of the current pre-registration assessments.....................................................................102

Towards the future ....................................................................................................................103

Conclusion .....................................................................................................................................105

Key issues for consideration by GPhC.............................................................................................105

References.....................................................................................................................................106

Appendices ....................................................................................................................................109

4

Table 1 Summary of practice-based assessment methods and tools (adapted from Clarke, 2007)…………………………………………………………………………………………………………………………………….34

Table 2 Summary of van der Vleuten’s Utility Index (van der Vleuten, 1996)................................36Table 3 Summary of case studies .................................................................................................39Table 4 Summary of papers included in the literature review.......................................................42Table 5 Assessment methods and tools identified in the literature review, case studies and

practice examples.............................................................................................................43Table 6 Summary of Utility of DOCS .............................................................................................49Table 7 Summary of Utility for DOPS............................................................................................55Table 8 Pharmacy Pre-registration Stakeholders Currently Using OSCEs.......................................62Table 9 Summary of Utility of OSCEs ............................................................................................63Table 10 Summary of Utility for Incognito Standardised Patients ................................................66

Table 11 Summary of Utility of MSF............................................................................................72Table 12 Summary of Utility of Patient Surveys ...........................................................................75Table 13 Summary of Utility of the Long Case .............................................................................80Table 14 Summary of Utility for Portfolios of Evidence................................................................84Table 15 Pharmacy Pre-registration Assessment Programmes in the UK, Canada, New Zealand and

Ireland ..........................................................................................................................85

Table 16 Assessment tools in the Foundation Programme ..........................................................91Table 17 Areas of the Clinical Encounter Assessed by mini-CEX ...................................................95

List of Tables

5

This project was funded by the General Pharmaceutical Council.

We are grateful to the following for their help in this work:

Dr Claire Stevens, Information Scientist, Department of Medicines Management, Keele

University, for input and support in designing and conducting the literature search and

sourcing papers

The members of RPSGB Pre-registration Liaison Group, and the NHS Pharmacy Education

and Development Committee for responding to the e-mail request for information on the

use of workplace based assessment in pre-registration training.

All those who contributed information towards the case studies:

Professor Zubin Austin, Lesley Dan Faculty of Pharmacy, University of Toronto,

Canada

Professor Graham Davies, Department of Pharmacy, King’s College London

Owain George, Competence Advisor, Pharmacy Council of New Zealand

Dr Delyth James, Senior Lecturer & Course Director (MSc in Pharmacy Clinical

Practice), Welsh School of Pharmacy, Cardiff

Jignesh Patel, School of Pharmacy, University of Brighton

Professor Paul J. Gallagher, Head of the School of Pharmacy, Royal College of

Surgeons in Ireland

All those who contributed information towards the practice examples referred to

throughout this report:

Dr Jane Portlock, School of Pharmacy, University of Portsmouth

Roisin Kavanagh, Programme Director JPB Diploma in General Pharmacy Practice,

School of Pharmacy, University of London.

Janet Gilbertson, All Wales Principal Pharmacist Education, Training and Personal

Development

Acknowledgements

•

•

•

•

o

o

o

o

o

o

o

o

o

6

In order for there to be a shared understanding of the scope of this project we have defined

‘practice-based assessments’ as:

“Assessments where performance is evaluated:

- in the workplace or in simulated practice situations.

- including the use of standardised or simulated patients

- including practice placements during undergraduate studies

- excluding written case simulations

After many years of research and debate van der Vleuten proposed a Utility Index for assessments

with ratings for the criteria reliability, validity, feasibility, acceptability and educational impact (van

der Vleuten, 1996). The Utility Index has become a widely accepted tool which acknowledges that

constraints of feasibility and acceptability have to be taken into account when selecting

assessments. The table below briefly describes each criterion of the Index.

Reliability The reproducibility of the scores obtained from an assessment.

Validity Validity is concerned with whether an instrument measures what it is purported to.

Feasibility Considers issues such as the logistics of running the assessment including trainee numbers and geographical spread; the venue for the assessment; capacity such as number ofassessors required and training of assessors; the costs involved to the HEI, regulator and trainee; and the need for any equipment or IT development.

Acceptability Acceptability is considered from the perspectives of different stakeholders including: trainees, Teachers, Higher Education Institute / Regulator, Employer / Workplace, Assessors, Patient

Educational impact

Considers the impact of the assessment on learning for example: congruence between the educational objectives and the assessment; how feedback on the assessment is provided and sustained; how formative and summative assessments are combined; how much assessment is needed; how the assessment is spread over time; in what context the assessment takes place.

Glossary

Criterion Description

Practice based assessments

Utility Index

7

Assessments that take place purely in the workplace. The Foundation Programme refers to the

assessments used in the programme as workplace based assessments.

Workplace Based Assessments

8

ACLF Advanced to Consultant Level Framework

CbD Case Based Discussion

DEPS Developing Educational & Practice Supervisors

DOCS Direct Observation of Clinical Skills

DOPS Direct Observation of Procedural Skills OR Direct Observation of Practical Skills

GLF General Level Framework

GPhC General Pharmaceutical Council

JPB Joint Programmes Board

MCQ Multiple Choice Questions

mini-CEX Mini-Clinical Evaluation Exercise

Mini-PAT Mini-Peer Assessment Tool

MRCF Medicines Related Consultation Framework

MSF Multi-Source Feedback

OSCE Objective Structured Clinical Assessment

REA Rapid Evidence Assessment

RCGP Royal College or General Practitioners

SP Standardised Patient

SPRAT Sheffield Peer Review Assessment Tool

TAB Team Assessment Behaviour

WPBA Workplace Based Assessment

List of Abbreviations

9

The General Pharmaceutical Council (GPhC) is responsible for setting and checking standards for the

initial education and training of pharmacists, including the pre-registration training period. Like

other regulators the GPhC has responsibility for ensuring, in the public interest, that the assessments

used are of appropriate quality and rigour and can withstand scrutiny and possible challenge.

Pharmacy pre-registration trainees are currently assessed through four ‘signoffs’ by their pre-

registration tutor (three progress reports and a final declaration) and also through the national

Registration Assessment (a multiple-choice examination). The purpose of the signoffs is for the tutor,

based on observation in the practice setting, to certify that the trainee has met the GPhC’s Pre-

registration Performance Standards and is fit to work as a pharmacist. The Registration Assessment

provides a national benchmark which mitigates potential variation in individual pre-registration

tutors’ practice-based assessment of trainees. Current practice-based and national assessments in

pharmacy pre-registration training have been the subject of some criticism and are recognised to

have a number of weaknesses.

Assessment of observed practice is used to evaluate performance in the workplace or in simulated

practice situations. The validity and reliability of such assessments (which we will refer to as

practice-based assessments) are dependent on both the assessment tool/s being used and the

capability of the assessor/s. Furthermore the resource implications of operationalising the different

forms of assessment may have substantial infrastructure requirements. The GPhC has indicated that

it may wish in future to review the assessments used in the pre-registration training period and

commissioned this study to consolidate existing evidence in order that any future discussions can

draw on an informed synthesis.

To provide evidence on the strengths and weaknesses of practice-based assessments in the

context of pharmacy pre-registration training for use in policy decision making by GPhC.

From a critical review of evidence, to:

1. Describe the range of practice-based assessments available, their designs and

educational basis

Executive Summary

Background

Aim -

Objectives -

10

2. Set out the evidence for their effectiveness and evaluate their utility using van der

Vleuten’s Utility Index (reliability, validity, feasibility, acceptability and educational

impact)

3. Illustrate and discuss challenges in operationalisation including scalability and sources of

variability

4. Consider and describe resource implications for different assessments

5. Critically reflect on existing and possible future assessments, focusing on utility in the

pharmacy pre-registration context

6. Take account of both the current and proposed future integrated UG/Pre-registration

programmes

7. Make preliminary recommendations for consideration by GPhC

A Rapid Evidence Assessment was undertaken using the following methods:

i) a review of peer reviewed published evidence;

ii) collection and review of information to produce a small number of case studies to illustrate

key issues and how they have been addressed in practice;

iii) identification of tools currently in use in pharmacy training as ‘practice examples’.

The analytical framework used to review and synthesise evidence was the van der Vleuten Utility

Index for educational assessment tools (reliability, validity, feasibility, acceptability and educational

impact).

“What is the utility of practice-based assessments in the context of pharmacy pre-

registration training?”

The review used a pre-defined protocol, and a search strategy designed and implemented iteratively

with support from an information specialist focusing on peer reviewed published systematic reviews,

with structured extraction of data and synthesis. Predefined inclusion criteria were used to identify

relevant papers from title and abstract. Full papers were obtained for those remaining and assessed

for the following inclusion criteria: Published in English; A systematic review or a structured non-

systematic review; Topic of paper is an assessment tool that meets the project’s definition of

Methods

Review of Peer Reviewed Published Evidence

Review question:

11

‘practice based assessments; Paper reports assessment of health care workers in practice – either

pre or post licensure; Mention of any aspect/s of the Utility Index.

Manual searches of the references of the included papers and of relevant journals and general

internet searches were undertaken to identify papers or reports not included in the electronic

databases. A search was also undertaken to identify papers in press.

Two frameworks for data extraction were developed: i) summary of papers including: details of the

reference, source type, level of evidence, assessments included, review questions and objectives,

study design, key findings, element of the Utility Index considered, conclusions and relevance to the

review question, and ii) summary of data and findings relating to elements of the Utility Index:

reliability, validity, educational impact, feasibility and acceptability.

a. Documentary analysis

Relevant policy documents were obtained and reviewed for UK medicine and nursing, and for

pharmacy licensure entrance assessments in other countries. The latter focused on countries which

use practice-based assessment tools as part of their entry to practice requirements (Canada and

New Zealand) and which have recently introduced changes to their entry to practice arrangements

(Ireland). This documentary analysis was used to identify relevant case studies.

b. Case studies

The purpose of including the case studies was to explore aspects of the Utility Index in real life use of

practice-based assessments, supplementing the literature review with exploration of issues of

feasibility, acceptability to different stakeholders, and validity in the pharmacy context.

c. Practice examples

Pre-registration stakeholders were contacted by email to inform them about the project and to

request examples of practice-based assessments currently in use in pre-registration training. The

findings are presented as ‘practice examples’ in the results section, together with other examples

submitted by stakeholders of workplace-based assessments used in postgraduate pharmacy training.

Contextual Evidence

12

The assessment methods and tools identified from the literature review were rated using the van

der Vleuten index. The table below provides a summary of the five elements of the Index.

Reliability The reproducibility of the scores obtained from an assessment.

Validity Validity is concerned with whether an instrument measures what it is purported to.

Feasibility Considers issues such as the logistics of running the assessment including trainee numbers and geographical spread; the venue for the assessment; capacity such as number of assessors required and training of assessors; the costs involved to the HEI, regulator and trainee; and the need for any equipment or IT development.

Acceptability Acceptability is considered from the perspectives of different stakeholders including: trainees, Teachers, HEI / Regulator, Employer / Workplace, Assessors, Patient

Educational impact


The evidence from the literature review, case studies and practice examples was reviewed by each

of the four study team members. Each then independently assigned a rating for each tool against

each element of the Utility Index or an ‘insufficient evidence’ statement. Consensus on these ratings

was achieved through discussion. The ratings used were low, moderate or high depending on the

amount and strength of the evidence available for the elements of the Utility Index. A further

description was applied to the ratings to provide an indication of the strength of evidence available.

These are described below:

“Potentially” – there is evidence to support the conclusion which is not strong but is congruent from two or more sources.

“Conditional” – the rating is dependent on certain conditions, particularly design and infrastructure.

Finally, an overall summary statement as to the utility of the assessment method or tool was derived

from the ratings. These statements were agreed by all four researchers.

Utility Index Ratings


13

The assessment methods and tools identified are described in table below.

Observation in the workplace

Direct Observation of Clinical Skills (DOCS)

Mini-CEX (mini-Clinical Evaluation Exercise)

MRCF (Medicines Related Consultation Framework)

Direct Observation of Procedural/Practical Skills

DOPS

Observation in a simulated practice situation

OSCE (Objective Structured Clinical Examination)

Incognito Standardised Patient

Feedback from colleagues and patients

Multi-Source Feedback (MSF)

Mini-PAT

360 degree review

SPRAT

Patient Survey

Case presentation Case Based Discussion (CbD)

The traditional long-case

Portfolio of evidence Pharmacy pre-registration trainee portfolio of evidence

The Foundation Learning Portfolio

The key findings from the literature review together with the ratings for each assessment method are presented below.

Results

Assessment method General Description of assessment methods

Examples of specific tools identified in literature review, case studies and practice examples

14

We found no evidence of use of DOCS in high stakes assessments.

There is strongest evidence about mini-CEX.

Reliability depends on the number of assessments undertaken and the number of different assessors used for an individual trainee. A minimum of 4-6

assessments need to be completed (possibly more for candidates whose performance is borderline). There is some debate about reliability of multiple

assessments by single rater vs single assessment by multiple raters (inter-rater reliability appears low).

Mini-CEX appears to be feasible and acceptable to trainees and assessors.

There is some evidence that learner performance improves after assessment.

Assessor training has been shown to influence whether action is taken following feedback.

Content validity: high for Mini-CEX, insufficient evidence other forms of DOCS.

Potentially high but dependent on number of assessments & assessors.

Insufficient evidence Moderate-high

Moderate-high

Discriminates

between poor and

good performance

Low-medium

Indications of a

positive educational

effect

Medium-high Appears to be high

Direct Observation of Clinical Skills (DOCS e.g. mini-CEX)

Key Findings

Data Source Validity Reliability Educational impact Feasibility Acceptability Summary

Literature Review Strong evidence for the utility of DOCS for formative assessment using Mini-CEX. It has potential for use in summative assessment but will require care with implementation. Insufficient evidence to support use of other DOCS tools. The MRCF looks promising but more evidence is needed on its use in different practice settings

MRCF –case study on

validation

MRCF use in practice

in community

pharmacy

15

We found no evidence of use of DOPS in high stakes assessments.

There is a lack of evidence for validity and reliability. Reliability is affected by the effect on performance of being observed – some may try harder whilst

others may feel uncomfortable resulting in impaired performance.

There is little evidence on the educational impact, beyond the perceived benefit of feedback on performance. None the less DOPS is perceived to have a

high educational impact.

Feasibility will vary from venue to venue and the procedure being assessed.

Little evidence for acceptability but on the face of it appears to be acceptable to trainees and assessors.

Insufficient evidence.

Potentially high but

same problems as

DOCS

Insufficient evidence Potentially Moderate Moderate Potentially Moderate

No evidence No evidence Potentially high for

PACE.

No evidence for DOPS

No evidence No evidence

Direct Observation of Procedural/Practical Skills (DOPS)

Key Findings


Literature Review Strong evidence for the utility DOPS for formative assessment using the RCP DOPS tool. it has potential for summative assessment but careful consideration needs to be given to how this assessment might be applied in pharmacy. There is insufficient evidence for other DOPS tools

Practice examples

16

OSCE may be considered as a component of a high stakes assessment strategy. Although labour intensive and thus costly, study authors generally seem to

evaluate their benefits as worthwhile.

There are established processes for development and testing of stations.

There is some debate around the minimum number of stations needed (10-15) and also the number of assessors per station (more than one assessor

needed) to provide sufficiently high reliability. The use of trained assessors is important.

OSCE assessments are reported in the literature to be stressful for students (but unclear whether more stressful than other forms of examination).

Moderate Potentially high Insufficient evidence Resource dependent

trainees – lowAcademic staff –mediumOverall: moderate

Medium-high High providing there are ten or more stations

N/A Depends on location and availability of standardised patients

Trainees –mediumStakeholders -medium

Objective Structured Clinical Examination (OSCE)

Key Findings


Literature Review OSCES are suitable for high stakes assessment providing the process for development and implementation uses recognised procedures.Case Study – use in

Canada for entry to practice

17

We found no evidence of use of incognito Standardised Patients (SPs) in high stakes assessments.

Incognito SPs have been defined as the validated gold standard methodology to discriminate between variations in quality of clinical practice.

Reliability can be difficult to achieve due to the difficulty obtaining sufficient numbers of tests.

Can have high educational impact if immediate feedback can be given.

Incognito SPs require training in the role they are playing and in completing the assessment check list.

Potentially high but few studies

Intrinsically low Probably moderate Low High with those practitioners who consent to participate, unknown in general population

Incognito Standardised Patients

Key Findings


Literature Review Low utility for incognito standardised patients in summative assessment.

18

We found no evidence of the use of MSF in high stakes assessments.

Consensus may be emerging that MSF is useful in formative assessment.

Although widely used in the NHS we found no data from the reviews on the validity or reliability of the Mini-PAT tool. There is some generic evidence that 360 degree feedback is valid and reliable providing there are sufficient raters.

MSF is likely to be reliable providing the trainee can access sufficient raters (some debate about numbers required; literature suggests minimum 8-10). More evidence is needed on the use of Mini-PAT and more data needed on the number of raters needed for reliability.

Evidence indicates that quality (accuracy and credibility) of feedback with MSF is critical and thus reviewer/educator training is a key issue.

There is some evidence on educational impact with studies showing up to 60% of participants intended to make changes in their practice based on feedback they had from a MSF assessment. However there is also evidence that MSF produces a strong negative response in some of those assessed – it is unclear whether or not this is linked to poor or no training of those giving feedback.Use of an electronic tool reduces administration costs.

Potentially moderateMore evidence needed

Potentially highMore evidenceneeded

Moderate with good reviewer input(Conditional)

Moderate with infrastructure support (conditional)dependent on access to sufficient assessors

Moderate-high(conditional)

Medium High in hospital

pharmacy post

registration

High in hospital

pharmacy post

registration

Multi Source Feedback

Key Findings


Literature Review Strong evidence on the utility of MSF for formative assessment using mini—PAT. More evidence is needed on its validity and reliability to recommend as a summative tool. Insufficient evidence to recommend other MSF tools

Case study – mini

PAT in hospital

pharmacy

19

Patient surveys are not widely used as a high stakes assessment, but are used as both a formative and summative assessment in continuing medical

education.

Patients are seen as appropriate people to assess various aspects of a doctor’s performance.

Very little has been written on the educational impact of patient surveys but available evidence indicates they have a positive impact.

Patient surveys appear to be feasible and low cost however there are few large scale studies addressing this.

Patient surveys are generally acceptable to doctors and patients.

Low-medium(Can be high for validated instruments)

Moderate-high if sufficient responses (and adequate response rate) and validated instruments

Insufficient evidence Low -medium

Medium

Patient Surveys

Key Findings


Literature Review Low utility for patient surveys as the major component of summative assessment

20

CbD has been used as a formative but not as a high stakes assessment.

The literature search did not identify any review on the use of CbD hence the utility of the assessment cannot be commented upon in this report.

The long case is traditionally used in high stakes assessment in undergraduate and postgraduate medical education in many parts of the world.

The long case is viewed as having high validity because it uses real patients, however this is debatable due to its inability to generalise from a single patient

interaction.

The long case is associated with poor reliability.

It is viewed as a useful assessment for providing student feedback, but it is a lengthy examination.

Moderate Low Insufficient evidence Insufficient evidence

Insufficient evidence

.

Case-based Discussion

The Long Case

Key Findings

Key Findings


Literature Review Low utility for traditional long case assessments in summative assessment

21

Structured portfolios are used as a high stakes assessment in medical Foundation Training.

Portfolios are generally deemed to have high face and content validity due to participation of staff and students in portfolio development and selection of

relevant content for inclusion.

In general the reliability of portfolio assessment is low to moderate due to the wide variability in how they are structured and assessed.

Portfolios are generally viewed as a valuable learning tool

Barriers to implementation to include lack of computer or internet facility in the clinical environment, lack of complimentary faculty development initiatives

and failure to integrate the portfolio as part of the evaluation process.

Moderate Low Mixed results, generally neutral to moderate. Can be conditional depending on structure of portfolio and quality of reviewer feedback.

Resource dependent

Potentially Moderate

Portfolios of Evidence

Key Findings


Literature Review Potential moderate utility for formative assessment but this is dependent on implementation.Their low reliability excludes them as a being suitable as the major element of summative assessment.

22

The project findings show clearly that there is now general agreement that assessment covering a

range of professional competencies needs to comprise a suite of tools based on van der Vleuten’s

principle of “a shift from individual methods to an integrated programme, intertwined with the

educational programme”. We produced case studies for examples from pharmacy and medicine

where this ‘instructional design’ approach has been put into practice. The table below summarises

pharmacy pre-registration assessment programmes in the UK, Canada, New Zealand and Ireland. In

addition a case study on assessment in the medical Foundation Year was included.

MCQ

MCQ

MCQ

The literature review findings consistently show that the reliability of practice-based assessment is

highly operator dependent and that training for assessors is crucial. Selection and training of

assessors is essential to ensure they have the skills, understand the process of the assessment, and

can address issues of equal opportunity. Two examples of available training are provided as

examples: i) e-learning for Practice Supervisors: DEPS (Developing Educational & Practice

Supervisors) programme, developed as part of work by a local group in the South East of England to

create an accreditation and QA framework for pharmacy practitioners involved in facilitating and

supervising learning in the workplace. ii) Module: Supervision Skills for the Healthcare Professional,

UCL Division of Medical Education. This module is aimed at health professionals taking on

responsibility for educational supervision and aims to enable participants to: Recognise the scope,

role and academic responsibilities of clinical and educational supervision, develop ways of making

the clinical environment an effective learning environment, and practice facilitating learners through

Programmes of assessment

Assessor training

Pharmacy Pre-registration Assessment Programmes in the UK, Canada, New Zealand and Ireland

Workplace External to / Independent of workplace

Tutor assessment in workplace

Practice based assignment

Written examination

OSCE Interview

UK

Canada

New Zealand

Ireland

v

v

v v v v v

v v v

23

developmental conversations to identify and address learning needs, improve their reflective

practice and safely develop their clinical expertise.

This report brings together evidence from the peer reviewed literature and documentary evidence,

with case studies and practice examples from the UK and more widely to inform discussion on the

utility of practice-based assessments in pharmacy pre-registration training. Van der Vleuten’s Utility

Index proved a useful ‘conceptual’ framework against which to consider the available tools. The

literature review generated data on reliability, validity and educational impact of different

assessment tools but evidence on feasibility and acceptability was sparse in published studies. Much

of the published evidence is from outside of pharmacy and we were able to supplement the

literature review findings with case studies drawn primarily from pharmacy practice to shed further

light on issues of feasibility and acceptability. In the discussion that follows we begin with the

principal findings relating to individual tools.

Published literature shows the specific Mini-CEX tool to be feasible and acceptable. Data on

reliability indicate that four to six completed Mini-CEX are required but debate remains about

reliability of multiple assessments by a single assessor versus a single assessment by a series of

assessors on different occasions. Feasibility of Mini-CEX in the community pharmacy setting is

constrained if multiple assessors are needed in order to provide a sufficient level of reliability.

Assessor training is crucial for feedback to be credible and likely to be taken on board by the trainee.

DOPS is used to refer to direct observation of both ‘Procedural’ and ‘Practical’ skills. Although some

procedural skills are relevant to pharmacy practice they may be less so in the pre-registration period.

We found examples from pre-registration stakeholders where tools are in use e.g. for dispensing

procedures.

Discussion

Utility of individual tools

)Direct Observation of Clinical Skills (DOCS

Direct Observation of Procedural Skills (DOPS)

24

There is evidence from published literature to show reliability and validity of OSCEs in formative and

summative assessment providing an appropriate blueprinting process is used in development and

testing of the assessment strategy. There appears to be consensus that a minimum of 10 stations is

needed for adequate reliability. There is also debate in the literature about the number of assessors

needed per station but as each assessment strategy is individual it is difficult to generalise. Training

of assessors is agreed to be crucial. The strengths of OSCEs are improved reliability and the ability to

test communication skills and attitudinal behaviours and weaknesses are that they are expensive,

labour intensive and that feasibility is challenging. Nevertheless OSCEs are used in high stakes

national assessments in pharmacy in Canada (the largest cohort size with around 900 candidates per

year), New Zealand and Ireland as part of pre-registration requirements and a number of pharmacy

pre-registration training providers in the UK have already incorporated OSCEs into formative

assessment.

SPs have been used in community pharmacy practice to assess the ‘does’ level of Miller’s pyramid of

clinical assessment as part of quality improvement and CPD programmes, particularly in Scotland

and Australia. A ‘mystery shopper’ type assessment could theoretically be used in pharmacy pre-

registration programmes but would be logistically challenging and of low reliability if only one

assessor was used.

The Mini-PAT assessment tool seems to be the most widely used MSF tool, is now well embedded

within medical F1 medical training and is increasingly being used in specialty training and in

revalidation pilots. This widespread use in medicine has developed without substantial published

evidence on reliability and validity although there seems to be an emerging consensus that reliability

is acceptable providing there is a minimum of 8-10 raters. Within pharmacy some evidence on Mini-

PAT has been generated from the JPB postgraduate programme, mainly in hospital pharmacy which

show that the tool is feasible and acceptable to use in hospital practice and there were some, albeit

limited, indications of educational impact. Feasibility of Mini-PAT in the community pharmacy

setting is constrained by the need for multiple raters in order to provide a sufficient level of

OSCE

Standardised patients (SPs)

MSF

25

reliability. Training of raters and those giving the feedback in MSF has been shown to be critical for

feedback to be perceived as credible and acceptable.

Patient surveys have used in medicine to obtain feedback for individual practitioners about their

consultations. Since the tools focus on individual consultations the feasibility of obtaining feedback

from patients for pre-registration pharmacy trainees would be dependent on the nature and extent

of the interactions between the trainee and patients. Sufficient responses (25+) would be needed to

give meaningful feedback.

The CbD is used in F1 where its strength is regarded as being assessment and discussion of clinical

reasoning. During the F1 year the trainee has to complete a minimum of six CbDs with at least two in

any four month period. Guidance states that different assessors should be used for each CbD

wherever possible and that assessors should have sufficient experience of the area under

consideration, typically higher specialty training, with variations between specialties. Each CbD must

represent a different clinical problem, sampling one of acute care, chronic illness, psychiatric care

etc. (categories listed in Syllabus and competencies). Whilst the use of different assessors from

different areas of practice may be feasible in a hospital pharmacy setting, for pre-registration

trainees in the community this would prove more challenging.

The traditional long case is not strictly a practice-based assessment as the trainee is not being

observed in practice but is presenting a case with their decisions and thinking. Modifications of the

long case do use direct observation of practice but the use of the long case has generally been

superseded by DOCS assessment tools. Although traditionally used as a high stakes assessment the

traditional long case has low utility as a high stakes assessment.

Patient surveys

Case Based Discussion (CbD)

The Long Case

26

There are difficulties in trying to assess the utility of the portfolio of evidence since portfolios can

vary considerably in structure and content, and consequently there is no standard tool which can be

used in assessment. This lack of structure and standardisation can, in turn, lead to problems with the

validity and reliability of portfolios. The underlying theme emerging from the literature is that any

portfolio used for assessment purposes should clearly articulate the amount, type and quality of

evidence required to establish proof of competence and the marking criteria used to evaluate the

quality of the evidence.

A clear finding of the literature review was that no single assessment method can meet the

requirements of a professional training programme involving a complex set of inter-related

competencies. A structured approach of mapping competencies against possible assessment

methods supports consideration of what combination/s of assessments might make an appropriate

programme.

The literature review showed that assessor training is an essential pre-requisite for reliable and valid

assessment using any practice-based assessment method. There are existing examples within

pharmacy which GPhC might want to consider including the JPB assessor guidance for the individual

tools used in their programme and the DEPS modules.

The findings of this project can be used by GPhC to inform discussions on future assessments in

pharmacy pre-registration programmes.

Issues of transferability of evidence from medicine and from pharmacy in other countries need some

consideration. Feasibility of large scale OSCE programmes in pharmacy has only been tested in

Canada. The pioneering work of JPB has translated and further developed WPBA tools used in



Assessor training

Towards the future

27

medicine but there are questions on scalability from the hundreds of students in postgraduate

education involved so far to the thousands that would be involved in pre-registration training.

Unsurprisingly the review of assessment tools does not support any recommendation of a specific

tool, nor would this be appropriate given current thinking among experts in assessment.

Consideration of existing and possible future ‘packages’ of assessments is therefore the way

forward.

GPhC may wish to consider the principles which will underpin future policy on assessment and may

find those adopted by Postgraduate Medical Education and Training Board (PMETB, now merged

with the GMC) and which are summarised below, a useful starting point.

Summary of PMETB principles of assessment

1. Methods must reflect the assessment’s intended purpose/content

2. Reference intended assessment content to ‘Good Medical Practice’

3. Ensure methods used to set standards are in the public domain

4. Involve lay members in the assessment process

5. Have mechanisms for giving trainees feedback on performance

6. Use appropriate criteria for examiner training

7. Use standardised documentation which is available nationally

8. Be sufficiently resourced

We found little evidence of practice-based assessment in high stakes summative assessments with

the exceptions of OSCE in Canada, New Zealand and Ireland and videotaped patient consultations in

the RCGP entry assessment. OCSEs are used as well as, not instead of, multiple choice tests. Some

would argue that the direction of travel for pharmacy, with consultations with patients becoming

more central, necessitates a national benchmarking assessment which includes communication and

consultation skills.

We included some examples of assessor training in this report and in our view assessor training is a

major challenge for GPhC regardless of whether the existing assessments are retained or changed.

28

The findings of this project can be used by GPhC in the review of pre-registration training in the

current and possible future contexts. The evidence that we have synthesised on validity, reliability,

educational impact, acceptability and feasibility of different assessment methods together with a

map against pharmacy pre-registration standards could be the basis of a blueprint for a future

programme of assessments. Our findings also have relevance to how the current national

requirements for practice-based assessment of pharmacy pre-registration trainees could be

strengthened.

1. In what ways can the current practice-based assessments conducted by pre-registration

tutors be strengthened?

2. Is a national benchmarking assessment of communication skills and attitudinal behaviours

needed or is practice-based assessment by the tutor sufficient?

3. Can the existing multiple choice examination be further developed?

4. What support needs to be offered to tutors and others involved in practice based

assessment of pharmacy pre-registration trainees?

Conclusion

Key issues for consideration by GPhC:

29

Dr Elizabeth Mills, Prof Alison Blenkinsopp, Prof Robert K McKinley, Dr Patricia Black

The General Pharmaceutical Council (GPhC) is responsible for setting and checking standards for the

initial education and training of pharmacists, including the pre-registration training period. Like

other regulators the GPhC has responsibility for ensuring, in the public interest, that the assessments

used are of appropriate quality and rigour and can withstand scrutiny and possible challenge.

Pharmacy pre-registration trainees are currently assessed through four ‘signoffs’ by their pre-

registration tutor (three progress reports and a final declaration) and also through the national

Registration Assessment (a multiple-choice examination). The purpose of the signoffs is for the tutor,

based on observation in the practice setting, to certify that the trainee has met the GPhC’s Pre-

registration Performance Standards and is fit to work as a pharmacist. The Registration Assessment

provides a national benchmark which mitigates potential variation in individual pre-registration

tutors’ practice-based assessment of trainees. Current practice-based and national assessments in

pharmacy pre-registration training have been the subject of some criticism and are recognised to

have a number of weaknesses.

Assessment of observed practice is used to evaluate performance in the workplace or in simulated

practice situations. The validity and reliability of such assessments (which we will refer to as

practice-based assessments) are dependent on both the assessment tool/s being used and the

capability of the assessor/s. Furthermore the resource implications of operationalising the different

forms of assessment may be substantial. The GPhC has indicated that it may wish in future to review

the assessments used in the pre-registration training period and has commissioned this literature

review to consolidate existing evidence in order that any future discussions can draw on an informed

synthesis.

To provide evidence on the strengths and weaknesses of practice-based assessments in the

context of pharmacy pre-registration training for use in policy decision making by GPhC.


Background

Aim -

30

From a critical review of evidence, to:

1. Describe the range of practice-based assessments available, their designs and educational

basis

2. Set out the evidence for their effectiveness and evaluate their utility using van der Vleuten’s

Utility Index (reliability, validity, feasibility, acceptability and educational impact) (van der

Vleuten, 1996)

3. Illustrate and discuss challenges in operationalisation including scalability and sources of

variability

4. Consider and describe resource implications for different assessments

5. Critically reflect on existing and possible future assessments, focusing on utility in the

pharmacy pre-registration context

6. Take account of both the current and proposed future integrated UG/Pre-registration

programmes

7. Make preliminary recommendations for consideration by GPhC

Objectives -

31

Regulators of health professions are concerned with ensuring that practitioners admitted to, and

remaining on, the professional register are competent and fit to practice. As Winslade points out,

prior to registration as a pharmacist a pre-registration trainee cannot independently perform their

future responsibilities as a pharmacist (Winslade, 2007). The assessment of observed practice prior

to registration must therefore be based on the competencies that underpin a pharmacist’s daily

performance. There is general agreement that competence is a pre-requisite to performance

(although it cannot guarantee performance). Assessment prior to registration as a pharmacist uses

surrogate measures that are considered to be predictive of performance in ‘‘real life’’ practice. The

challenge lies in designing and selecting assessments which are capable of assessing competence in

providing the sort of integrated care involved in daily practice as a pharmacist.

Miller’s pyramid of assessment (see figure 1) is commonly used in the literature on assessment of

health professionals’ competence and performance with knowledge (“knows”) at the lowest level of

the pyramid, followed by competence (“knows how”), performance (“shows how”) and action

(“does”). Assessment of observed practice takes place at the top two levels and Miller distinguished

between these depending on whether trainees were in an artificial or a real life setting (Miller,

1990).

‘The assessment of observed practice’ as described in the GPhC project specification for the current

project included assessments where performance is evaluated in the workplace or in simulated

practice situations such as OSCEs or the use of standardised patients thus corresponding to

assessments that take place at the top two levels of Miller’s pyramid of clinical assessment (see

figure 1). Observation of practice could be undertaken by the assessor directly observing the

trainee’s practice as it is taking place i.e. in real time. Alternatively the assessor could observe a

sample of the trainee’s practice which has been captured using video. The same tools are used to

assess the practice whether it is observed directly or via a video recording. Video-taped

consultations are used by the Royal College of General Practitioners (RCGP) as part of their training

programme for GPs.

In order for there to be a shared understanding of the scope of this project we have defined

‘practice-based assessments’ as:

“Assessments where performance is evaluated:

- in the workplace or in simulated practice situations.

Introduction

32

- including the use of standardised or simulated patients

- including practice placements during undergraduate studies

- excluding written case simulations

Written case simulations were not included in our project scope since they are not based on

observation of performance. Unmanned stations used in OSCEs were also excluded. These are used

to test skills such as information retrieval and interpretation. Depending on the structure of the

response framework they can also be used to test reasoning skills. However they cannot be used to

test psychomotor skills (such as communication and procedural skills), hence their exclusion from

this project.

Medical frameworks are a useful starting point to consider the areas of competence and

performance that need to be assessed. ‘Good Medical Practice’ (GMC, 2006) sets out six domains

Good Clinical care

Keeping up-to-date

Maintaining and improving performance

Teaching, training, appraising and assessing

Relationships with patients

Relationships with colleagues

Figure 1 Miller’s pyramid of clinical assessment (Miller, 1990)

What needs to be assessed?

o

o

o

o

o

o

33

The NHS Medical Assessment Compendium (NHS) built upon this to identify nine areas in which

assessment of professional competence and performance is required:

Knowledge

Clinical thinking

Consulting skills

Practical / Procedural skills

Relationships with patients

Relationships with colleagues

Written communication / medical record keeping

Patient safety / critical incidents / clinical governance data

Portfolio / reflective practice

As Munoz and colleagues point out, unlike other health professions, pharmacists are, in general,

“

”. Performance needs within pharmacy have been described as

“

” (Munoz et al, 2005). The core

skills of questioning, listening, observing and problem-solving are central to pharmacy practice.

Pharmacists also increasingly need to work collaboratively with patients and other practitioners to

ensure appropriate prescribing and use of medicines by patients. The extension of the work of

community pharmacists to include patient centred roles such as formal reviews of medicines use,

support for adherence and provision of emergency hormonal contraception have required the

development of higher level consultation skills. Thus communication techniques such as

consultation, teaching and knowledge translation to make scientific information about medicines

relevant and understandable for patients are key tools needed by pharmacists and by pharmacists

who are also non-medical prescribers

•

•

•

•

•

•

•

•

•

neither required nor expected to demonstrate a broad repertoire of psycho-motor skills related to

patient assessment and therapy

highly focused on verbal and non-verbal communication, observation, and review and management

of patient information and medical information databases, for the purpose of identifying and

resolving clients’ drug related problems or other health care needs

34

Numerous tools exist for practice-based assessment in the workplace and in simulated practice

situations and Table 1 provides a summary. Appendix 1 contains a more detailed summary of the

competency domains that can be assessed using the different assessment methods.



Observation of a consultationusing a structured checklist

Consultation skills

Professionalism

Problem identification

Problem solving

DOPS (Direct Observation of procedural skills e.g.measuring BP)

Observing practical proceduresusing a structured checklist

Practical skills Communication

Professionalism

DOCS (Direct Observation of Clinical Skills e.g. Mini-CEX)

Observation of a clinical examinationusing a structured checklist

Clinical skills Professionalism

Communication



Observation of performance in a simulated practice situation using simulated patients. Assessed using a structured checklist which could include a DOCS or DOPS tool.

Clinical skills

Problem solving

Communication

Professionalism

Incognito Standardised Patient (sometimes referred to as ‘Covert patient’ or ‘mystery

Use of standardised patient in a real life

History taking

Physical exam

Specific clinical areas

Methods of assessment

Table 1 Summary of practice-based assessment methods and tools (adapted from Clarke, 2007)

Assessment method Examples of tools in use Tested by Main Focus Also Tests

35

shopper’ practice setting

Communication


Multi-Source Feedback (MSF) e.g. Mini-PAT, 360 degree review

Assessment in all aspects of work

Professionalism Clinical care

Communication

Patient surveys Patient completes a survey after a consultation

Professionalism

Communication

Patient advocacy

Co-ordination of patient care


Review of case notes

Clinical reasoning

Professionalism



A mixture of methods

Communication

Problem solving

Decision making

Ethical and professional dilemmas

Clinical management

Clinical performance

Technical skills

Assessments used in decisions about professional registration are required to be of appropriate

quality and rigour and able to withstand scrutiny and possible legal challenge. After many years of

research and debate van der Vleuten proposed a Utility Index for assessments with ratings for the

criteria reliability, validity, feasibility, acceptability and educational impact (van der Vleuten, 1996).

The Utility Index has become a widely accepted tool which acknowledges that constraints of

feasibility and acceptability have to be taken into account when selecting assessments. Table 2

briefly describes each criterion of the Index.

Principles of selection of assessment methods

36

Reliability The reproducibility of the scores obtained from an assessment. Needs to take into account sampling across conditions of measurement, for example, a single use of an assessment tool may be highly unreliable, but with multiple use across different situations and with different assessors the tool becomes much more reliable.

Validity Validity is concerned with whether an instrument measures what it is purported to. Different types of validity are referred to in the literature. Face validity is the validity of a test at face value, and would be inferred if a number of people agree that the assessment ‘looks like’ it should assess what it sets out to. Construct validity refers to whether a scale measures or correlates with the theorised psychological construct that it purports to. In the case of assessments construct validity refers to whether the assessment checklist measures the competencies it purports to. Criterion validity compares the assessment to a well-known and previously validated assessment. Predictive validity refers to how well the assessment predicts future performance.

Feasibility Considers issues such as the logistics of running the assessment including trainee numbers and geographical spread; the venue for the assessment; capacity such as number of assessors required and training of assessors; the costs involved to the HEI, regulator and trainee; and the need for any equipment or IT development.

Acceptability Acceptability is considered from the perspectives of different stakeholders including: trainees, Teachers, Higher Education Institute / Regulator, Employer / Workplace, Assessors, Patient

Educational impact


Wass and colleagues comment that “no test can score uniformly high on all five factors and some

trade-off is inevitable”, but they also describe it as “an excellent framework for assessment design”

and (Wass et al, 2007). This Index, therefore, has been chosen as the basis for our review of

assessments of observed practice.

Table 2 Summary of van der Vleuten’s Utility Index (van der Vleuten, 1996)


37

In order to meet the project objectives the following methods were utilised:

i. a review of peer reviewed published evidence;

ii. review of information to produce a small number of real life case studies to illustrate key

issues and how they have been addressed in practice.

iii. identification of tools currently in use in pharmacy training as ‘practice examples’

“What is the utility of practice-based assessments in the context of pharmacy pre-

registration training?”

The short timescale for the project made a Rapid Evidence Assessment (REA) the most feasible and

appropriate method. REAs are used to summarise the available research evidence within the

constraints of a given timetable, typically three months or less and are widely used by policy makers

to provide support for policy decisions. REA methods recognise that in time limited review periods

the extent of material that can be included is limited.

The review was based on a pre-defined protocol, and used a search strategy focusing on peer

reviewed published systematic reviews, with structured extraction of data and synthesis to address

the review objectives.

The search strategy was designed with input from an information specialist (see Appendix 2 for

search strategy). The following databases were searched: Medline, International Pharmaceutical

Abstracts, CINAHL, British Nursing Index and PsycINFO.

The search strategy was refined iteratively, checking at each stage that known key studies were

identified.

Titles were initially reviewed by researcher EM for to identify those with potential relevance to the

review question. Titles and abstracts of these papers were then entered into a reference database

and reviewed by researcher EM and cross checked by AB using the following inclusion criteria:

Published in English

Methods

Review of Peer Reviewed Published Evidence

Review question:

•

38

Any mention of review

Any mention of work, workplace, practice based or competence assessment

Any mention of results from any group of health care workers

Mention of any aspects of the Utility Index (validity, reliability, acceptability, feasibility,

educational impact)

Papers were excluded if they did not meet all of the inclusion criteria for title and abstract. Full

papers were obtained for those remaining.

Full papers were assessed for the following inclusion criteria:

Published in English

A systematic review or a structured non-systematic review

Topic of paper is an assessment tool that meets the project’s definition of ‘practice based

assessments

Paper reports assessment of health care workers in practice – either pre or post licensure

Mention of any aspects of the Utility Index

Manual searches of the references of the included papers and of relevant journals and the internet

in general was undertaken to identify papers or reports not included in the electronic databases. A

search was also undertaken to identify papers in press.

Two frameworks for data extraction were developed. The first was to provide a summary of the

paper and included: details of the reference, source type, level of evidence (according to Melnyk and

Fineout-Overholt, 2005) assessments included, review questions and objectives, study design, key

findings, element of the Utility Index considered, conclusions and relevance to the review question.

The second data extraction framework focussed on the elements of the Utility Index: reliability,

validity, educational impact, feasibility and acceptability.

Data extraction was carried out by the researcher EM. A sample of the data extraction was cross

checked by the researcher AB.

•

•

•

•

•

•

•

•

•

39

a. Documentary analysis

Relevant policy documents were obtained for UK medicine and nursing, and for pharmacy licensure

entrance assessments in other countries. The latter focused on countries which use practice-based

assessment tools as part of their entry to practice requirements (Canada and New Zealand) and

which have recently introduced changes to their entry to practice arrangements (Ireland). This

documentary analysis was used to identify relevant case studies.

b. Case studies

The purpose of including the case studies was to explore aspects of the Utility Index in real life use of

practice-based assessments. The case studies were intended to supplement the literature review

with exploration of issues of feasibility, acceptability to different stakeholders, and validity in the

pharmacy context. Case studies were agreed with GPhC and are summarised in Table 3.

OSCEs in ‘high stakes’ assessment

National entry to practice assessment in Canada

Examine implementation and scalability including resources.

Assessment of consultation skills

Use of MRCF by Joint Programme Board in PG pharmacy programmes

Explore use of MRCF in different UK practice settings

Multi-source feedback Use of mini-PAT by Joint Programme Board in hospital pharmacy

Explore feasibility in different settings including in sole practitioner environments

Programme of practice-based assessments

Ireland NPIP & New Zealand EVOLVE

Medical Foundation Year 1 WPBA Programme

Provide information on the use of a multi-assessment programme in the workplace with national benchmarking in order to explore issues of feasibility and acceptability.

Examine experience since introduction of national programme of WPBA

Assessor training in use of practice based assessment tools

DEPS (Developing Educational & Practice Supervisors)

To explore the learning from these examples and consider the implications for pre-registration training.

Contextual Evidence

Table 3 Summary of case studies

Setting Purpose

40

Data sources for case studies comprised:

i. Published literature

ii. Public domain policy and other relevant documents

iii. Contact with key informants by email and/or phone. Key informants were sent draft text

based on desk research, asked to check it for accuracy and currency and to suggest

amendments. Questions included: What advice would you offer to a regulator considering

introducing this assessment?; With hindsight, what do you wish you had known before the

decision to introduce this assessment?

c. Practice examples

In our previous pre-registration pharmacy studies (Mills, 2008 and Mills, 2010) we developed a list of

pre-registration stakeholders with whom to consult during the projects. In the current project we

utilised this list and contacted the individuals by email to inform them about the project and to

request examples of practice-based assessments currently in use in pre-registration training. A

number of these stakeholders forwarded the request to individual pharmacy pre-registration

training providers. In addition we looked at examples of workplace based assessments used in initial

post-registration pharmacy training. The findings are presented as ‘practice examples’ in the results

section.

The van der Vleuten Utility Index is a “thinking tool”. Its strength is in providing a framework that

enables a holistic view of an assessment method. The type and nature of evidence for the individual

elements that make up the index is different. Some (reliability and validity) are amenable to

quantitative trials with statistical analysis whereas others (feasibility and acceptability) are

essentially descriptive and rely on stakeholder responses. In developing our method for rating

assessments using the index we took into account these different types of evidence.

The evidence from the literature review, case studies and practice examples was reviewed by four

independent researchers. Ratings were assigned where possible for each tool against each element

of the Utility Index. Consensus on these ratings was achieved through discussion between the four

independent researchers. The ratings used were low, moderate or high depending on the amount

and strength of the evidence available for the elements of the Utility Index. A further description

was applied to the ratings to provide an indication of the strength of evidence available. These were

Utility Index Ratings

41

“potentially” and “Conditional”. The definitions agreed by the researchers for these terms are

described below:

“Potentially” – there is evidence to support the conclusion which is not strong but is congruent from two or more sources.

“Conditional” – the rating is dependent on certain conditions, particularly design and infrastructure

Finally, an overall summary statement as to the utility of the assessment tool was derived from the ratings. These statements were agreed by all four researchers.

In the results chapter that follows a sub-section for each of the individual assessments is presented

following a structured format; key points are set-out, followed by literature review findings together

with case studies and practice examples where applicable. Each sub-section ends with a summary

table based on the van der Vleuten Utility Index and brings together evidence from the literature

review, case studies and practice examples. The chapter concludes with sections on assessor training

and on programmes of assessments with case studies and contextual information.

Presentation of results

42

The final search strategy yielded 1460 papers; 49 were identified as potentially relevant from the

titles. Full papers were obtained for 46 of these. Nine papers met the inclusion criteria for data

extraction. A further two papers were found from hand searches, taking the total number of papers

included in the review to eleven (see Table 4). Of these, seven were systematic reviews where the

review questions, review strategy and inclusion and exclusion criteria were explicit. Four were

structured non-systematic reviews.

Abdulla, 2008 A structured non-systematic reviews

Mini-PAT Not reported

Centre for Innovation in Professional Health Education and Research (CIPHER), 2007

A systematic literature review

MSF, DOPS, standardised patients, mini-CEX, patient surveys, portfolios, videotaped consultations

MSF – 29 papersDOPS – 34 papersStandardised patients – 23 papersMini-CEX – 18 papersPatient surveys – 29 papersPortfolios – 49 papersVideotaped consultations –22 papers

Hamilton et al, 2007 A systematic literature review

Direct observationMSF

40 papers

Kogan et al, 2009 A systematic literature review

DOCS 85 papers

Miller and Archer, 2010 A systemic literature review

Mini-CEXDOPSMSFCbD

16 papers

Pelgrim et al, 2010 A systematic literature review

DOCS 39 papers

Ponnamperuma et al, 2009 A systematic literature review

The long case 18 papers

Rushforth, 2007 A structured non-systematic reviews

OSCES Not reported

Shanley, 2001 A structured non-systematic reviews

OSCES Not reported

Walsh et al, 2009 A systematic literature review

OSCE 41 papers

Wood et al, 2006 A structured non-systematic reviews

MSF Not reported

Results

Table 4 Summary of papers included in the literature review

Reference Type of evidence Assessment Tool Reviewed

Number of papers reviewed

43

As Table 5 shows, the assessment tools identified were:



Mini-CEX (mini-Clinical Evaluation Exercise)


Direct Observation of Procedural/Practical Skills

DOPS



Incognito Standardised Patient


Multi-Source Feedback (MSF)

Mini-PAT

360 degree review

SPRAT

Patient Survey


The traditional long-case



Table 5 Assessment methods and tools identified in the literature review, case studies and practice examples

Assessment method General Description of assessment methods

Examples of specific tools identified in literature review, case studies and practice examples

44

We found no evidence of use of DOCS in high stakes assessments.

There is strongest evidence about mini-CEX.

Reliability depends on the number of assessments undertaken and the number of different assessors

used for an individual trainee. A minimum of 4-6 assessments need to be completed (possibly more

for candidates whose performance is borderline). There is some debate about reliability of multiple

assessments by single rater vs single assessment by multiple raters (inter-rater reliability appears

low).

Mini-CEX appears to be feasible and acceptable to trainees and assessors.

There is some evidence that learner performance improves after assessment.

Assessor training has been shown to influence whether action is taken following feedback.

The Clinical Evaluation Exercise (CEX) was developed by the American Board of Internal Medicine to

replace the traditional oral examination. The CEX traditionally involved observation of the trainee

carrying out a thorough history taking and physical examination and presenting their findings and

diagnosis, and a written report of conclusions for the supervising clinician to evaluate. The

assessment generally took 2 hours, occurred once a year with one assessor during a pre-selected

patient encounter. Research in the early 90s raised concerns about the validity, reliability and

feasibility of this tool. The mini-CEX is a response to some of the inadequacies of the original version

of the tool and capitalises on the routine interactions that supervising clinicians and trainees have on

a daily basis.

We found no evidence of use of Mini-CEX or other forms of DOCS in high stakes assessments.

Direct Observation of Clinical Skills (DOCS e.g. mini-CEX)

Key Findings

Description of DOCS (taken from CIPHER, 2007)

Use of DOCS

45

Three reviews reported on the validity of DOCs. CIPHER (2007) looked specifically at the mini-CEX

tool and reported a strong statistically significant correlation between mini-CEX and accepted

measures of clinical competence suggesting it is a valid assessment. Kogan (2009) reported low or

modest correlations for DOCS in general and other accepted forms of assessment of clinical skills.

Mini-CEX, however, had evidence of internal structure validity and validity based on relationships to

other variables. Pelgrim et al (2010) report a high criterion validity for mini-CEX evaluated through

comparisons with instruments with proven validity. A number of studies infer construct validity from

an increase in rating over time, but this remains to be proven.

Three reviews reported on the reliability of DOCs in general. CIPHER (2007) found that over 10 mini-

CEX are needed for an acceptable reliability of 0.8, but concluded that between 4 and 6 encounters

may be sufficient for most assessment purposes. More may be required for trainees obtaining

borderline scores. Pelgrim et al (2010) calculated the average reliability co-efficient for all the

instruments in their review. They found that a reliability of 0.8 could be achieved with a sample of 10

encounters. One study in the Pelgrim review showed that one assessor taking 10 assessments is

much less reliable than 10 assessors taking one encounter each. However, this was contradicted by

another study who showed that mini-CEX is reliable with one assessor and eight encounters. CIPHER

(2007) concluded in their review that reliability is enhanced through the opportunity for multiple

observations of trainees by multiple assessors in a range of clinical settings.

Kogan et al (2009) found that inter-rater reliability was often sub-optimal for DOCS in general.

CIPHER (2007) found mixed results for inter-rater reliability for mini-CEXs. There is conflicting

evidence on how inter-rater reliability is affected by training (Pelgrim et al, 2010)

Four reviews reported on the educational impact of DOCS in general. CIPHER (2007)reported a study

that showed that mini-CEX feedback is generally accompanied by at least two recommendations for

improvement, but there was no follow up to check whether recommendations were being

Validity of DOCS

Reliability of DOCS

Educational Impact of DOCS

46

implemented. Assessors need training on how to give feedback. Kogan et al (2009) reported on

studies describing improved performance after use of DOCS, but commented on the poor quality of

the studies. Miller and Archer (2010) reported four studies evaluating mini-CEX as a formative

assessment tool. In a Canadian study most medical residents agreed mini-CEX had a positive impact.

Participants highlighted that use of the mini-CEX as an assessment limited its value as an educational

tool. A second study reported by Miller and Archer found that over half of the study participants

were satisfied or very satisfied with the exercise as a tool for learning. Pelgrim et al (2010) reported

several studies that evaluated educational impact by eliciting learners’ or assessors’ attitudes

towards the instrument. Most reports were positive.

Three reviews reported on the feasibility of DOCS. The CIPHER review (2007) reported mixed

findings on feasibility. Most studies reported moderate to high feasibility and satisfaction. One study

reported by CIPHER showed that it could be used in various clinical settings with a broad range of

patient outcomes. Another reported an 89% completion rate with an average of 19 minutes spent

on observation and 8 minutes on feedback. Conversely a study in the UK reported by CIPHER has a

low uptake on mini-CEX with ‘a lack of time’ and ‘difficulty finding a suitable patient ‘cited as

barriers.

Kogan et al (2009) reported a lack of training of assessors to use the tools. Training was usually brief

(10 minutes to three hours) and occurred only once. Training often included orientating observers to

the tool and discussing feedback principles but did not include rater practice.

Pelgrim et al (2010) found that feasibility was generally considered good but without any clear

criteria. Some problems associated with feasibility were lack of time and inadequate

implementation. Pelgrim et al expressed the view that assessment instruments must be well

integrated into the curriculum and be part of routine practice. Workshops were found to be a better

way to implement the instrument than written instructions.

Acceptability was considered in two reviews. CIPHER (2007) found that evaluators and trainees rate

mini-CEX as satisfactory. Surveys by Kogan et al (2009) found that trainees generally rated

observation experience positively. Higher trainee satisfaction may be due to increased educational

Feasibility of DOCS

Acceptability of DOCS

47

interaction with staff, the opportunity for feedback and the lower levels of stress associated with

multiple short assessments.

There is strong evidence for the utility of DOCS for formative assessment using Mini-CEX.

Mini-CEX has the potential for use in summative assessment but will require care with

implementation. There is insufficient evidence to support use of other DOCS tools for summative

assessment.

The MRCF was developed as a medicines-specific consultation skills assessment tool for

The tool was intended for use in assessing

pharmacists’ skills when consulting with patients (e.g. when undertaking a medicines use review or

clinical medication review). It can also be used by other healthcare professionals undertaking a

medication-centred consultation (e.g. nurses who are non-medical prescribers). The tool has long

(46 items) and short (31 items) form versions.

The MRCF drew on a literature review which mapped existing medical consultation models to

identify the appropriate structure and content relevant to any medicines-related consultation and

further input from a number of consultation skills experts to adapt and develop the specific

medicines related content (Abdel-Tawb et al, 2011). Consultation behaviours are grouped into 5

sections: Introduction, Data collection & problem identification, Actions & solutions, Closure and

General consultation behaviours. The long form version of the tool was tested in a study in which 10

pharmacists each used MRCF to rate 15 recordings of simulated consultations. The MRCF was shown

to discriminate between good, satisfactory and poor consultations at the global and section rating

level. Test-retest reliability was good and inter-assessor reliability was moderate. Internal

consistency for the 5 sections of the tool was found to be good. Some inconsistencies were found in

assessors’ ratings at the individual consultation behaviour level (Abdel-Tawb et al, 2011).

Utility of DOCS

Case Study

Assessing consultation skills: The Medication Related Consultation Framework (MRCF)

“a

structured approach to gathering information from the patient in order to assess, identify and

resolve any potential or actual medication problems”.

48

The short form MRCF is a mandatory work based assessment in the Joint Programmes Board (JPB)

postgraduate Diploma in General Pharmacy Practice (provided by 8 Schools of Pharmacy in London

& South East) and by Cardiff University in Postgraduate Diplomas (Community & Primary care and

Hospital pharmacists). In these PG programmes the tool is used in self-assessment as part of

coursework or reflective portfolios; in observation where the tutor observes a consultation and

provides feedback; and in OSCEs where it forms the basis of formative and summative marking

schemes for the assessment of patient-centred scenarios. The short form MRCF is also used at UG

level from Year 2 to inform the teaching, learning and assessment of consultation skills in some

MPharm programmes including King’s and Cardiff.

Practice example

Use of MRCF and mini-CEX in a community pharmacy postgraduate education programme

The MRCF (observer assessed and self-assessment) and mini-CEX are used for observed

practice in community pharmacy by the University of Portsmouth as part of the MSc in

Pharmacy Practice.

A member of academic staff from the university visits the student in their workplace (usually

twice during the academic year; all students are based within 50 miles of the University) and

observes practice - usually a medicines use review (MUR) or other consultation with a patient

about a medicine. In MUR the MRCF is used and the focus is on the pharmacist asking

appropriate questions relating to medicines use. Mini-CEX would be used, for example, in a

consultation with a patient with COPD on a new inhaled medicine who needs advice and

counselling on the appropriate use of the inhalation device and other information about the

medicine. The assessments have proved popular with the course participants who report them to

be valuable for development and identification of learning needs as well as in demonstration of

skills and competence.

49

Content validity: high for Mini-CEX, insufficient

evidence other forms of DOCS.

Potentially high but dependent on number

of assessments & assessors.


Moderate-high

Moderate-high

Discriminates

between poor

and good

performance

Low-medium

Indications

of a positive

educational

effect

Medium-

high

Appears to be

high

Table 6 Summary of Utility of DOCS

Data Source Validity Reliability Educational

impact

Feasibility Acceptability Summary

Literature

Review

Strong evidence for the utility of DOCS for formative

assessment using Mini-CEX.

It has potential for use in summative

assessment but will require care with implementation.

Insufficient evidence to support use of other DOCS

tools.

The MRCF looks

promising but

more evidence

is needed on its

use in different

practice settings

MRCF –case

study on

validation

MRCF use in

practice in

community

pharmacy

50

We found no evidence of use of DOPS in high stakes assessments.

There is a lack of evidence for validity and reliability. Reliability is affected by the effect on

performance of being observed – some may try harder whilst others may feel uncomfortable

resulting in impaired performance.

There is little evidence on the educational impact, beyond the perceived benefit of feedback on

performance. None the less DOPS is perceived to have a high educational impact.

Feasibility will vary from venue to venue and the procedure being assessed.

Little evidence for acceptability but on the face of it appears to be acceptable to trainees and

assessors.

Direct observation of procedural skills (DOPS) has been defined as the observation and evaluation of

a procedural skill performed by a trainee on a real patient. Procedural skills are also known as

technical or practical skills. Evaluation by an experienced practitioner is carried out using either a

checklist of defined tasks, a global rating scale, or a combination of both. The distinguishing features

of DOPS are that it assesses procedural rather than consultation skills and involves the observation

of a single patient encounter rather than over a period of time. Finally it involves performance of the

procedure on actual patients, rather than a cadaver, simulation or animal model. DOPS is used in the

medical Foundation Programme.

No evidence was found in the literature of the use of DOPS in high stakes assessments. However,

DOPS is thought to be a useful formative assessment.

Direct Observation of Procedural/Practical Skills (DOPS)

Key Findings

Description of DOPS (Taken from CIPHER, 2007)

Use of DOPS

51

Two reviews were found that addressed the validity of DOPS. CIPHER (2007)and Hamilton et al

(2007) concluded that there was a lack of evidence for the validity of DOPS but CIPHER (2007)

suggests it has high face validity since it involves real patients and normal routine procedures.

Two reviews addressed the reliability of DOPS. CIPHER (2007) found a lack of studies assessing the

reliability of DOPS. The main issue is determining the number of procedures that needs to be

observed and determining appropriate checklists and rating scales. Hamilton et al (2007) suggest

that problems with reliability are due to the effect on performance of being observed, some may try

harder whilst others may feel uncomfortable and their performance may be impaired.

Two reviews reported on the educational impact. CIPHER (2007) found little evidence base of the

educational impact or DOPS. The immediate feedback is seen as important and DOPS is perceived to

have a high educational value. Miller and Archer (2010) reported a study in which 70% of a sample of

25 medical house officers felt that observation had helped them improve their clinical skills.

However, there was no evidence in this study or others that DOPS leads to an objective performance

improvement.

Two reviews reported on the feasibility. CIPHER (2007) concluded that DOPS has been shown to be

feasible but this may vary from venue to venue and the procedure being assessed. DOPS should be

easily integrated into routine practice, but one issue is finding time for supervisors to observe

trainees performing skills. In the Foundation Programme it is the trainee’s responsibility to select

when the assessment will take place and who will assess them. This poses difficulties for less

common medical procedures and specialities. Hamilton et al (2007) report that DOPS is feasible

since only one assessor is needed for each observation.

Validity of DOPS

Reliability of DOPS

Educational Impact of DOPS

Feasibility of DOPS

52

One review commented on acceptability (CIPHER, 2007). They found little research on acceptability

but concluded that on the face of it DOPS appears to be acceptable to trainees and assessors.

Trainees like getting immediate feedback from an experienced doctor. There are public safety issues

in assessing procedural skills such as surgical skills on real patients.

There is strong evidence for the utility DOPS for formative assessment using the RCP DOPS tool. It

has the potential for summative assessment but careful consideration needs to be given to how this

assessment might be applied in pharmacy.

There is insufficient evidence for other DOPS tools

Acceptability of DOPS

Utility of DOPS

Practice Examples

Direct Observation of Practical Skills

Practical

In medicine DOPS is used to assess a procedural skill performed on a real patient by a trainee.

In pharmacy, however, such procedural skills are rarely carried out on patients. DOPS, however

is used to assess organisational procedural skills as well as the practical skills expected of

pharmacists. Examples of how DOPS is used in pharmacy are provided below.

The Joint Programmes Board has adapted DOPS from foundation training and has developed an

assessment called Direct Observation of Skills. It is used to assess the trainee

undertaking the following tasks:

Prescription validation

Completion of relevant documentation

Error management

Final check of a prepared product

Dispensing of an unlicensed product

53

Assessment of compatibility of stability (e.g. two drugs in a syringe driver)

Preparation of products (e.g. aseptic, extemporaneous)

Risk assessment

Drug administration (e.g. calculation, correct device, route, matching prescription to drug

delivery)

The assessment is an observation of the trainee in practice, using a structured assessment

form, as in Foundation Training. The competencies being assessed are based on the General

Level Framework (GLF) (CODEG, 2007). Feedback is an essential form of the assessment. The

observed process should take no more than 15-20 minutes and the immediate feedback should

take no longer than 5 minutes.

: Joint Programmes Board Directly Observed Practical Skills Assessor Written Training

NHS Wales (Welsh Hospital Pharmacy) has developed a system to enable pre-registration

pharmacists to demonstrate their competence in checking the dispensing accuracy of

prescriptions already clinically checked by a pharmacist. The accuracy checking takes place both

on the ward and in the dispensary. The pre-registration trainee collects a portfolio of 1000

checked items. When completing the portfolio all items checked during each checking period

must be recorded. In the event of an error the trainee is required to reflect on the error and

document the reflection. Having completed the 1000 items, the trainee then undertakes an

accuracy checking test consisting of 20 items to check, that contain 6-8 errors. The trainee

must identify all errors. A panel then reviews the trainee’s portfolio and test results. If the

panel is confident that the pre-reg. trainee has demonstrated their competence, they will be

allowed to assess prescription only medicines (POMs), transcribe supply orders and accuracy

check within the scope of their training following a two week probationary period

PACE has been evaluated and shown to have a high educational impact; 91% of tutors and 89% of

trainees agreed that reflecting on an accuracy checking error helped them identify how to

change their method of checking 95% of tutors and 96% of trainees agreed that being

accredited to check increased their sense of responsibility

Source

Pre-registration Accuracy Checking Evidence (PACE)

;

.

54

A similar system has recently been introduced to enable trainees to develop their clinical

accuracy checking skills prior to registration as a pharmacist.

NHS Wales (Welsh Hospital Pharmacy)Source:

55

Insufficient

evidence.

Potentially

high but

same

problems as

DOCS

Insufficient

evidence

Potentially

Moderate

Moderate Potentially

Moderate

No evidence No evidence Potentially

high for

PACE.

No evidence

for DOPS

No evidence No evidence

Table 7 Summary of Utility for DOPS


impact


Literature

Review

Strong

evidence for the utility DOPS for formative

assessment using the RCP DOPS tool. it has potential

for summative assessment but careful

consideration needs to be given to how this

assessment might be applied in pharmacy.

There is

insufficient evidence for other DOPS tools

Practice

examples

56

OSCE may be considered as a component of a high stakes assessment strategy. Although labour

intensive and thus costly, study authors generally seem to evaluate their benefits as worthwhile.

There are established processes for development and testing of stations.

There is some debate around the minimum number of stations needed (10-15) and also the number

of assessors per station (more than one assessor needed) to provide sufficiently high reliability. The

use of trained assessors is important.

OSCE assessments are reported in the literature to be stressful for students (but unclear whether

more stressful than other forms of examination).

An OSCE is an examination where ‘students demonstrate their competence under a variety of

simulated conditions’ (Watson et al, 2002). OSCEs were first described and used in the 1970s to

assess the clinical skills and competence of final year medical trainees. They involve a series of

stations using real patients or actors to test students’ skills in real world tasks such as history taking

or clinical examination. Students move between stations at timed intervals. Examiners stay with

each station throughout the session, thus each student is examined by a number of different

examiners and all are assessed using a near identical process (Rushforth, 2007).

An OSCE may be considered as a component of a high stakes assessment strategy.

A review by Rushforth (2007) quotes a study that suggests that the validity of OSCEs is increased by

the stressful nature of the test being in congruence with the stressful real world of practice. In

Objective Structured Clinical Examination (OSCE)

Key Findings

Description of OSCE

Use of OSCEs

Validity of OSCEs

57

contrast, Shanley (2001) found that student stress about the assessment can negatively affect

validity.

OSCEs are not the real world, they are simulations. Variables such as shift patterns and day to day

pressures are not taken into account which hinders the ability to get a clear picture of the student’s

performance (Shanley, 2001). A further criticism is that patients are not treated holistically in the

OSCE, the knowledge and skills tested are compartmentalised. One of the studies in the Shanley

review found that performance in one OSCE case does not predict performance on other problems

or cases, especially in terms of problem solving and clinical reasoning.

Findings on concurrent and predictive validity has been variable in studies. However many of the

comparisons are not comparing like with like. When the OSCE is split into scores for knowledge,

problem solving, clinical skills and patient management there were good correlations between the

knowledge domain and scores from multiple choice question (MCQ) assessment (Rushforth, 2007).

Walsh (2009) reviewed nine studies which all reported correlations between OSCE scores and other

forms of evaluation methods such as MCQs, short answer tests and certifying exams. The

competencies selected for testing varied considerably from one study to the next.

OSCEs use a wide range of examiners, hence there is a reduced risk of examiner bias. It is seen as

more objective than other forms of assessment of practice. Inter-rater reliability has varied between

0.2 and 0.8. This suggests caution in relying on the judgement of single examiners. The variation in

how OSCEs are run suggests that inter-rater reliability data has limited transferability to other

situations and any new OSCEs should demonstrate their own inter-rater reliability (Rushforth, 2007).

Factors that decrease reliability include: ineffective training for simulated patients and examiners,

situations where patients are known to students therefore increasing the chance of influencing

examinees, and the risk of examiner and patient becoming fatigued. Reliability is also compromised

when examiners lack experience. The checklist can also be problematic – examiners need greater

details on the forms to guide them and greater scrutiny of items to avoid ambiguity (Shanley, 2001).

Walsh (2009) reviewed nine studies reporting findings of reliability from the medical literature.

Sample sizes ranged from 28-572. The construction of the OSCE varied considerable: 8-42 stations,

4-15 minutes per station. The overall reliability ranged from 0.40 to 0.91. In general the researchers

Reliability of OSCEs

58

suggested that the reliability scores demonstrated that the consistency of the measurement tools

was sensitive to the length of the test, station construction and heterogenicity of students being

evaluated.

Rushforth (2007) cited one study that reported that OSCEs motivate learning. Walsh (2009) suggests

that OSCEs can facilitate the identification of curriculum weaknesses.

OSCES involve complex orchestration including faculty time, cost and staffing, and ensuring

confidentiality of OSCE stations when student cohorts need to be assessed in separate subgroups

(Rushforth, 2007).

Walsh (2009) reviewed nine studies in medical education that addressed the issues of cost. The

sample size ranged from 38 to 212 students, the number of stations ranged from 6 to 17 and were 4-

15 minutes in length. The cost ranged from $6.90(US) to up to $870 (Canadian) per student.

Although not all authors reported actual cost analysis, they evaluated the relative benefit of the

OSCE and indicated that, although labour intensive and costly methods that objectively evaluated

clinical competency were essential. For all researchers the benefits associated with the OSCE process

outweighed the limitations of other forms of evaluative methods.

Rushforth (2007) reviewed a study that reported positive views of OSCEs by students. However,

‘student stress’ is a disadvantage often cited in the literature. .

OSCES are suitable for high stakes assessment providing the process for development and

implementation uses recognised procedures.

Educational Impact of OSCEs

Feasibility and Acceptability of OSCEs

Utility of OSCEs

59

Entry to practice requirements in Canada have included an objective structured clinical examination

(OSCE) for over a decade. The OSCE, together with a MCQ assessment, is organised by the Pharmacy

Examining Board of Canada (PEBC, a body independent of the regulator) for some 900 trainees each

year. Canada is internationally recognised as a centre of expertise in pharmacy OSCEs.

The OSCE was introduced after a 4-year lead in period. This was to allow Schools of Pharmacy to

change their undergraduate programmes such that all students would experience an OSCE before

their qualifying exams. The lead in time also enabled providers of courses and other support for pre-

licensure pharmacists to develop programmes. The strategy was to “set people up for success”.

OSCE test sites were initially established during 2001 in 8 cities with Schools of Pharmacy. An

important advantage of this model was the availability of standardized patient programmes

affiliated with medical schools in these cities. Chief administrators and chief examiners were

appointed in each city to oversee logistics and the training of standardized patients and assessors,

respectively. Each chief administrator was responsible for assembling a team to manage the site and

to recruit and train assessors. Central training sessions were held for site staff prior to the

examination. The 8 sites represented 6 different time zones and varied in size and character from a

19-candidate bilingual (French and English) site in Montreal, to a 200-candidate site in Toronto.

In 2001 there were 717 candidates and 647 site personnel in the 8 centres on the day of the

examination:

309 pharmacists (as station assessors, ‘quiet-station’ markers, track coordinators, and chief

examiners)

222 standardized patients (including back-up personnel)

116 administrative staff (to facilitate set-up, technical support, on-site coordination, etc).

After the examination there was follow-up to check standardization of OSCE delivery occurred: video

training, e-mail, and conference calls.

Case Study

National OSCE for entry to practice in Canada

Feasibility

Logistics and resources

60

One major cost driver for the OSCE is exam personnel. Key questions for OSCE administrators

include: (a) how many assessors are required in each station, and whether these assessors must all

be health care professionals; (b) can simulated patients act as assessors and (c) can the same person

both simulate the client role in the station and assess the candidate’s performance simultaneously?

All costs of the OSCE are met through the fee charged to candidates for the entry to practice

examination ($1500 Canada). To put this in context a typical pharmacy starting salary in community

pharmacy would be c.$75,000 and membership of the College of Pharmacists (the Canadian

regulatory body, organised at province level) is $750.00.

Educational support for the OSCE is provided through the PEBC and includes a website, paper based

materials and the opportunity for candidates to speak on a one to one basis with an adviser.

The reliability and validity of OSCEs depend on the rigour of the process of development, testing,

delivery and review. Any OSCE is only as good as its stations. Experience in Canada has shown that

although a costly and time-intensive exercise, the investment in development of stations and

assessment tools is pivotal to the success of an OSCE and essential in ensuring validity of the

outcome.

“Blueprinting” is the term used to describe OSCE development. The domains of content and process

skills to be assessed are identified and the level (complexity) and scope (range) of expected

performance are defined including.

of tasks and processes described (“do they have to be able to do this”)

of tasks and processes described (“will they be required to be able to do this”)

of tasks and processes described (“are they generally expected to be able to do this”)

Standard setters (pharmacists from different practice backgrounds and from different demographic

cohorts) evaluate each item or clinical skill/task to be performed. To ensure broad representation

from professional practice and to optimize buy-in, case writers were selected from across the

country, representing different constituencies of the profession (e.g. community practice or hospital

practice). The case development process has to be as diverse and inclusive as possible and to reflect

the pharmacy population as a whole including overseas pharmacists.

Quality assurance

Criticality

Frequency

Relevancy

61

In addition to OSCE stations involving interaction with standardised patients there are also simulated

health care professional stations to assess. The OSCE stations enable assessment of communication

skills, ‘cultural competence’ and clinical skills.

Case writers and standardized patient trainers, (SP Trainers), attended a two day workshop on

principles of case development and development of assessment instruments. Case writers with

subject matter expertise worked in pairs to develop OSCE stations with a SP Trainer. At a subsequent

two-day workshop, each case was reviewed by a different group of three pharmacists and a SP

Trainer. The involvement of practising pharmacists in the case writing and review process sent an

important signal to the profession that the content of the stations was representative of practice

and aligned with pharmacists’ real-life experiences with patients.

Extensive data collection and analysis to determine reliability showed there was no apparent

improvement in the dependability and generalizability of results when more than 15-stations were

included. Validity, reliability, generalisability and dependability may be higher with a greater number

of stations but the decision on the number of stations also needs to consider feasibility and cost-

effectiveness. Maintenance of stations requires ongoing commitment.

While the process of developing and piloting the OSCE relied heavily upon practising pharmacists, it

was also necessary to ensure acceptance of the process by the broader pharmacy community. Open,

consistent communication with pre-registrants during this process is also very important.

Acceptability

62

Cohens Use OSCEs for formative assessment.NHS East Midlands At a regional level we run two sets of OSCEs per year. These

OSCEs are formative only, but the pre-registration trainees use them to write evidences relating to various performance standards. For each task/station, there is a list of performance standards to which it may relate.

NHS North East Use OSCEs as a training tool rather than an assessment exercise.NHS East of England Use OSCE scenarios (manned and unmanned) twice during the

pre-registration training year. Run a few OSCEs as a 'formative' baseline assessment during early November and a 'summative' set in May of each training year. However, this is not strictly summative as the outcome does not prevent any trainee from registering with the GPhC. However, there is formal assessment criteria and trainees are sent their results (copied to their pre-registration tutor). The OSCEs are not mapped to the performance standards.

NHS South West OSCEs are used as a major formative assessment. Each scenario is mapped to the performance standards.

NHS South Central Use OSCE's as a formative assessment before the 39 weeks appraisal. Not been able to make this a summative assessment as it is not recognised by the GPhC. Use 4-5 OSCE stations, and concentrate on Performance standards that pertain to interpersonal skills and problem solving skills to support tutorsdecisions in signing pre-registration trainees off for the register.

NHS South East Have done a huge amount of work with OSCEs in the pre-registration year going back over 15 years. Some of this work is published. Within the OSCEs preregistration trainees are assessed using a task checklist but also against the preregistration performance standards. Preregistration trainees receive a copy of their mark against the preregistrationstandards and qualitative feedback.

Practice Examples

Our request for information from pharmacy pre-registration stakeholders about the use of

work-based assessments in pre-registration training revealed that a number of training

programmes currently incorporate OSCES. Twenty-seven stakeholders were contacted,

responses were received from thirteen stakeholders and a further two responses from

individual training sites. There were seven reports of using OSCEs. See table 8 below for

further information.

Table 8 Pharmacy Pre-registration Stakeholders Currently Using OSCEs

Stakeholder Further information

63

Moderate Potentially high


Resource dependent

trainees –low

Academic staff –medium

Overall: moderate

Medium-high

High providing there are ten or more stations

N/A Depends on location and availability of standardised patients

Trainees –medium

Stakeholders - medium

Table 9 Summary of Utility of OSCEs

Data Source Validity Reliability Educational impact


Literature Review

OSCES are suitable for high stakes assessment

providing the process for development and

implementation uses recognised procedures.

Case Study –use in Canada for entry to practice

64

We found no evidence of use of incognito Standardised Patients (SPs) in high stakes assessments.

Incognito SPs have been defined as the validated gold standard methodology to discriminate

between variations in quality of clinical practice.

Reliability can be difficult to achieve due to the difficulty obtaining sufficient numbers of tests.

Can have high educational impact if immediate feedback can be given.

Incognito SPs require training in the role they are playing and in completing the assessment check

list.

Incognito standardised patients (SPs) are individuals who have been trained to consult the doctor as

if they were a real patient. Although the doctor is informed that the SP may show up at some time

he or she does not know when to expect them. The SPs are extensively trained to portray their

clinical scenario in a highly standardised way, and also to score the doctor’s performance using a

checklist.

There is no evidence in the literature of the use of SPs in high stakes assessments.

CIPHER (2007) reported that SPs have been shown to be a highly valid assessment. It has been

defined as the validated gold standard methodology to discriminate between variations in quality of

clinical practice. An unresolved issue is whether valid checklists or rating scales can be designed for

assessing the performance of doctors. They have high face validity and doctors are unaware they are

Incognito Standardised Patients

Key Findings

Description of Standardised Patients (Taken from CIPHER, 2007)

Use of SPs

Validity of SPs

65

being observed and assessed so are unlikely to change their behaviour. Validity is threatened when

the SP is detected, and by the ability of the SP to perform in a standardised way.

CIPHER (2007) reported issues with reliability due to difficulty obtaining sufficient numbers of SPs.

To achieve a reliability of 0.8 requires 3 hours of testing time (6 patients). One suggestion put

forward to overcome this is to combine it with other methods of assessment.

CIPHER (2007) found that SPs can be a powerful learning tool since they are authentic. A study of

Canadian medical students found using incognito SPs combined with immediate feedback had a

positive effect in later performance in an examination of competence. However immediate feedback

may not always be possible.

CIPHER (2007) found some issues with the feasibility of SPs. Consent from the doctor should be

obtained before the visit. SPs need training in the role they are playing, and in completing check lists.

The amount of training depends on the circumstances but has ranged from 10 hours to two full days.

There will likely be a high number of drop outs among SPs. There are logistical difficulties in using

SPs in primary and secondary care, for example in secondary care problems included getting

simulated test results.

CIPHER (2007) found SPs to be acceptable to doctors, provided they know they will be used and

consent has been obtained.

Reliability of SPs

Educational Impact of SPs

Feasibility of SPs

Acceptability of SPs

66

Low utility for incognito standardised patients in summative assessment.

Potentially high but few studies

Intrinsically low

Probably moderate

Low High with those practitioners who consent to participate, unknown in general population

Utility of SPs

Table 10 Summary of Utility for Incognito Standardised Patients



Literature Review

Low utility for incognito standardised patients in summative assessment.

67

We found no evidence of the use of MSF in high stakes assessments.

Consensus may be emerging that MSF is useful in formative assessment.

Although widely used in the NHS we found no data from the reviews on the validity or reliability of

the Mini-PAT tool. There is some generic evidence that 360 degree feedback is valid and reliable

providing there are sufficient raters.

MSF is likely to be reliable providing the trainee can access sufficient raters (some debate about

numbers required; literature suggests minimum 8-10). More evidence is needed on the use of Mini-

PAT and more data needed on the number of raters needed for reliability.

Evidence indicates that quality (accuracy and credibility) of feedback with MSF is critical and thus

reviewer/educator training is a key issue.

There is some evidence on educational impact with studies showing up to 60% of participants

intended to make changes in their practice based on feedback they had from a MSF assessment.

However there is also evidence that MSF produces a strong negative response in some of those

assessed – it is unclear whether or not this is linked to poor or no training of those giving feedback.

Use of an electronic tool reduces administration costs.

Multi-source feedback (MSF) intends to provide a balanced and accurate assessment of performance

by evaluating an individual’s competence from multiple perspectives within their sphere of

influence. It has been used for many years in industry and is becoming more established in health

education. MSF is one of the four assessment methods currently used in Foundation Training for

medical trainees.

Feedback is objectively and systematically collected via a survey or rating scale where raters rate

trainee performance and the trainee also does a self-assessment. Feedback is subsequently

Multi Source Feedback

Key findings

Description of MSF (taken from CIPHER, 2007)

68

summarised by topic and rater category and feedback to the trainee by an educator, usually the

trainee’s t educational supervisor or tutor.

In medicine, raters may include faculty, nurses, fellow residents, medical students, allied health

professionals, administrative staff and patients in addition to self-evaluation. The purported

strength of MSF is that multiple perspectives enables a ‘gap analysis’ wherein the individual being

assessed compares how others perceive them with how they perceive themselves in order to

identify potential areas of improvement. Web-based tools have increased the ease of use and

versatility of this tool.

The tool used in the UK medical Foundation Training Programme is the mini-Peer Assessment Tool

(mini-PAT). This is a shortened version of the Sheffield Peer Review Assessment Tool (SPRAT)

No evidence was found in the literature of use of MSF in high stakes assessments, however,

consensus may be emerging that MSF is useful in formative assessment.

Four reviews were identified that addressed the validity of MSF. Abdulla (2008) found that the mini-

PAT tool used in Foundation Training has not been formally validated and there is a lack of field

evaluation. Content validity is inferred since the questionnaire conforms to Good Medical Practice.

The parent tool SPRAT has only been evaluated in one study. The criteria used to reduce SPRAT (24

questions) to mini-PAT (16 questions) is unclear.

A review by CIPHER (2007) found that the validity of MSF tools in general is low. Hamilton et al

(2007) reported on three studies and found adequate validity overall.

Wood et al (2006)reviewed a number of studies that showed that a single ‘communication’ factor

accounts for most variance in ratings. One study demonstrated that four descriptors are needed in

the tool to obtain consistent results across different raters and individual raters over time. No study

has yet demonstrated that over time, those trainees identified as best in MSF do indeed outperform

their peers.

Use of MSF

Validity of MSF

69

Four reviews were identified that addressed the reliability of MSF. Abdulla (2008) reported a lack of

studies assessing the reliability of mini-PAT. Studies have shown that for the parent tool SPRAT, four

raters are required to make a reasonably confident decision of satisfactory competence or that a

doctor is in difficulty. The CIPHER review (2007) reported a study that demonstrated reliability using

a medical colleague, co-worker, patient and self-assessment. Generally, analyses of MSF have shown

that 8-10 co-workers, 8-10 medical colleagues and 25 patients are needed to obtain reliable

estimates of physician’s competence. One of the three studies that Hamilton et al (2007) reviewed

reported adequate reliability providing that enough raters were used. The studies reviewed by

Wood et al (2006) reported the need for between 5 and 15 raters in order to provide a

representative picture for each trainee. Differences could be due to different training of rater,

inconsistency between groups of raters and too many categories to be rated.

Five reviews were identified that addressed the educational impact of MSF. The overwhelming

finding from the reviews was that the educational impact depends on the process and quality of

feedback given by the educator. The educator needs to be appropriately trained for the task. In

order to respond to the feedback trainees need to perceive it to be accurate and credible. Abdulla

(2008), CIPHER (2007) and Miller and Archer (2010) all reported studies where feedback that was

perceived to be negative and inaccurate was less likely to be accepted and used. Students can

become defensive about the feedback if it does not coincide with their self-perception.

Miller and Archer (2010) reviewed a study where participants were asked to rate the educational

impact of MSF on a 7 point Likert scale. The mean score was 4.2. CIPHER (2007) reports on one study

and Miller and Archer (2010) on two studies where between 61% and 66% of participants reported

making changes to their practice based on their MSF data.

Three reviews were identified that addressed the feasibility of MSF. The selection of raters needs

careful consideration as it can introduce selection bias. For the mini-PAT Abdulla (2008) suggests

Reliability of MSF

Educational Impact of MSF

Feasibility of MSF

70

that trainees should be given guidance on choosing raters from a variety of professional

backgrounds and different clinical environments that the trainee works in. Raters may need to

receive training in the use of the tool to provide a reliable opinion. The time taken to complete an

MSF form, provided it is kept short, is between 5-6 minutes, making it a feasible and inexpensive

approach to assessment within busy clinical environments (CIPHER, 2007).

MSF can be time consuming and administratively demanding with 20% of time spent on data

collection and 80% on designing, training and coaching (CIPHER, 2007). For the mini-PAT in the

Foundation Programme a national centre co-ordinates the distribution of forms and produces the

feedback in a standardised form. A list of raters is sent to the national centre and assessment forms

are sent directly to the nominated raters (Abdulla, 2008). The use of on-line technologies can

increase response rate, decrease response time and decrease administration time (CIPHER, 2007).

Three reviews addressed the acceptability of MSF. The CIPHER review (2007) found that factors that

influence the raters ability to make judgements include the length of time spent with the trainee,

the assessment criteria, the characteristics of the rater such as occupation, and length and context

of working relationship. Abdulla (2008) reported a lack of evidence that peer rating improved patient

outcome, but suggested that good communication skills would improve patient satisfaction. Wood

et al (2006) suggested that if the trainees like the system and find it motivating, organisational

benefits should ensue.

There is strong evidence on the utility of MSF for formative assessment using mini—PAT. More evidence is needed on its validity and reliability to recommend as a summative tool.

There is insufficient evidence to recommend other MSF tools

Acceptability of MSF

Utility of MSF

71

The mini-PAT assessment has been in use within the Joint Programme Board postgraduate Diploma

in General Pharmacy Practice Programme for several years to assess three of the General Level

Framework (GLF) (CODEG, 2007) competency clusters: ‘personal’, ‘delivery of patient care’ and

‘problem solving’. Pharmacists are required to participate in the assessment every six months.

Pharmacists have a free choice of the assessors (up to 8) whom they wish to nominate (pharmacists,

pharmacy technicians, senior medical and nursing colleagues) and must include the ‘head of section’

in which they have been working during the preceding 6 months. Assessors rate 16 attributes and

can provide verbatim feedback on areas of strength and “areas the pharmacist should particularly

focus on for development”.

The programme team evaluated the use of mini-PAT in hospital pharmacy by i) conducting an

analysis of numbers and types of assessors nominated by pharmacists (pharmacists, pharmacy

technicians, nurses, doctors and other health professionals) and the marks given by each group of

assessors across 633 mini-PAT assessments completed over a three year period and ii) a survey of

pharmacist views on the usefulness of mini-PAT (122 participants, 82% response rate) (Patel et al,

2009).

Pharmacists were the most frequently nominated group of assessors followed by pharmacy

technicians with smaller percentages of doctors and nurses. Over time pharmacists were more likely

to nominate doctors. The analysis showed that over three quarters of those nominated to complete

a mini-PAT did so (similar to reported rates in the literature for doctors).

Pharmacists were scored more highly on the ‘personal’ competency cluster than on the ‘delivery of

patient care’ or ‘problem solving’ clusters. Doctors and nurses were significantly more likely to score

pharmacists higher across all of the mini-PAT questions than did nominated pharmacists.

Pharmacists found the scores and verbatim comments received through the mini-PAT useful.

Pharmacists’ ratings of whether mini-PAT influenced their professional practice or whether the

assessments helped to reveal issues they were unaware of were lower. The programme team

Case Study

Use of mini-Peer Assessment Tool (mini-PAT) in hospital pharmacy as part of a pharmacy

postgraduate programme

72

commented that some pharmacists said that feedback was not specific enough to enable them to

change their practice.

Potentially moderate

More evidence needed

Potentially high

More evidence needed

Moderate with good reviewer input

(Conditional)

Moderate

with infrastructure support (conditional)

dependent

on access to sufficient assessors

Moderate-high

(conditional)

Medium High in

hospital

pharmacy

post

registration

High in

hospital

pharmacy

post

registration

Table 11 Summary of Utility of MSF


impact


Literature

Review

Strong evidence on the utility of MSF for

formative assessment using mini—PAT. More

evidence is needed on its validity and

reliability to recommend as a summative

tool.

Insufficient

evidence to

recommend

other MSF

tools

Case study –

mini PAT in

hospital

pharmacy

73

Patient surveys are not widely used as a high stakes assessment, but are used as both a formative

and summative assessment in continuing medical education.

Patients are seen as appropriate people to assess various aspects of a doctor’s performance.

Very little has been written on the educational impact of patient surveys but available evidence

indicates they have a positive impact.

Patient surveys appear to be feasible and low cost however there are few large scale studies

addressing this.

Patient surveys are generally acceptable to doctors and patients.

Patient surveys are a set of questions administered to a sample of a doctor’s patients asking them

about their views on various aspects of the doctor’s performance. Typically questions use either

rating categories or ask for a level of agreement with statements describing the care. Patient surveys

can be administered immediately after the consultation and completed immediately, they may be

given to the patient immediately after the consultation but returned later, or they may be sent to a

random sample of patients who complete and return them.

Patient surveys are used as a formative assessment in continuing medical education. They are not

widely used for licensure or for certification processes to establish fitness to practice.

Patient Surveys

Key Findings

Description of Patient Surveys (taken from CIPHER, 2007)

Use of the Patient Surveys

74

CIPHER (2007) report that it is increasingly accepted that patients may be the most appropriate

people to assess some aspects of a doctor’s performance. Validity is weakened as they rely on

memories and interpretations of those being surveyed.

The number of patient surveys required to achieve reliability is under debate and ultimately depends

on the format of the questionnaire, the patient being cared for, the competency domain being

assessed and the level of expertise of the student. The number of raters needed to achieve

acceptable reliability co-efficients range from 15 to 50 (CIPHER, 2007).

Very little has been written on the educational impact of patient surveys but available evidence

indicates they have a positive impact. One study showed that patient feedback on doctor’s

interpersonal skills contributed to improvements in the quality of the patient-doctor relationship

(CIPHER, 2007).

Patient surveys appear to be feasible and low cost however there are few large scale studies

addressing this. One study found the largest component of time to be staff time spent on entering

the data. Difficulties associated with patient surveys include: language and literacy, obtaining

enough surveys to provide reproducible results, the resources required to collect, aggregate and

report the results, and assessment of the student’s contribution to patient care as separate from

that of the health care team (CIPHER, 2007).

Validity of Patient Surveys

Reliability of Patient Surveys

Educational Impact of Patient Surveys

Feasibility of Patient Surveys

75

Patient surveys are generally acceptable to doctors and patients. One study found that GPs and

supervisors highly valued the role of patient feedback in interpersonal skills development. Another

large study concluded that surveys are acceptable to patients based on a response rate of 66%

(CIPHER, 2007).

Low utility for patient surveys as the major component of summative assessment.

Low-medium

(Can be high for validated instruments)

Moderate-high if sufficient responses (and adequate response rate) and validated instruments


Low -medium

Medium

Acceptability of Patient Surveys

Utility of Patient Surveys

Table 12 Summary of Utility of Patient Surveys



Literature Review

Low utility for patient surveys as the major component of summative assessment

76

Case-based discussion (CbD) in medical Foundation Training is a structured discussion with an

assessor of clinical cases managed by the foundation doctor. Its strength is assessment and

discussion of clinical reasoning. The foundation doctor selects two case records from patients they

have seen recently, and in whose notes they have made an entry. The assessor selects one of these

for the CbD session. The discussion starts from and is centred on the foundation doctor’s own record

in the notes. CbD assesses medical record keeping, clinical assessment, investigation and referral,

treatment rationale, follow up and future planning, professionalism and overall clinical care.

Feedback is provided to the trainee immediately following the discussion.

CbD has been used as a formative assessment but not a high stakes assessment.

The literature search did not identify any reviews on the use of CbD hence the utility of the

assessment cannot be commented upon in this report.

Case-based Discussion

Description of Case-based Discussion

Use of CbD

Utility of CbD

Practice Examples

CbD is used in by the Joint Programmes Board in a similar way to Foundation Training. It is used

to assess the trainee’s clinical decision making and the application of pharmaceutical knowledge

to the care of patients. It also enables the discussion of the ethical and legal frameworks of

practice and allows the trainee to explain why they acted as they did. The actual record is the

focus of the discussion and so the assessor can also assess record keeping. Feedback is provided

immediately following the discussion in which strengths, areas for improvement and agreed

action points are identified. The competencies assessed in CbD are pharmaceutical needs

assessment, treatment recommendations, patient monitoring, follow up, consideration of patient

concordance, professionalism and overall clinical judgement.

A total of 12 CbDs are carried out in each year of the diploma, each CbD should cover a

different clinical problem and a range of practice settings should also be covered e.g.

77

dispensary, ward, medicines information etc. The assessment can take place either on the ward

or in the pharmacy department. One CbD should take no longer than 20-30 minutes including

feedback and completion of the assessment form.

: Joint Programmes Board Case Based Discussion Assessor Written Training Source

78

The long case is traditionally used in high stakes assessment in undergraduate and postgraduate

medical education in many parts of the world.

The long case is viewed as having high validity because it uses real patients; however this is

debatable due to its inability to generalise from a single patient interaction.

The long case is associated with poor reliability.

It is viewed as a useful assessment for providing student feedback, but it is a lengthy examination.

The long case is a traditional clinical examination that assesses the candidate at the ‘shows how’

level of Miller’s pyramid (Miller, 1990). Classically the candidate spends some time with a patient,

taking a history and carrying out a physical examination, without examiner observation. The

candidate then presents their findings to one or more examiners and answers oral questions.

Traditionally the candidate is scored with unstructured marking criteria that are based on neither

standardised checklists nor on rating scales with descriptors related to candidate competence.

Modifications of the long case have, however, included observed encounters and the development

of structured marking criteria (Ponnamperuma et al, 2009).

The traditional long case has been modified in various ways. The Objective Structured Long

Examination Record (OSLER) is a 10-item analytical record of the traditional long case, with examiner

observed history taking and physical examination process, and a criterion referenced marking

scheme. The Direct Observation Clinical Encounter Examination (DOCEE) exposes the candidate to

multiple patient interactions in which multiple assessors from different specialities observe the

candidate carrying out a history taking and a physical examination.

The Long Case

Key Findings

Description of the Long Case

79

The long case is traditionally used in high stakes assessment in undergraduate and postgraduate

medical education in many parts of the world (Ponnamperuma et al, 2009).

Ponnamperuma et al (2009) found that the long case has greater validity because it uses real

patients. It is more authentic. It also offers direct contact between the candidate and the examiner.

However, the traditional long case had low validity due to its inability to generalise from the results

of a single patient interaction, non-observation of the candidate during patient interaction, lack of

structure and lack of patient standardisation.

Ponnamperuma et al (2009) reviewed six papers, all of which identified the poor reliability of the

long case. Improvements in reliability are achieved by increasing the number of cases. One paper

used a pair of examiners to observe the candidate taking a history. The candidate than presented

the case to another pair of examiners. Inter-rater reliability was higher for the observation part than

the presentation.

The review by Ponnamperuma et al concluded that overall the long case is an educationally valuable

test because it provides diagnostic feedback to students and teachers. It is a good method of

formative assessment.

Use of the Long Case

Validity of the Long Case

Reliability of the Long Case

Educational Impact of the Long Case

80

The long case is a lengthy examination, although in the traditional long case most of the time is not

spent on examining the patient, but represents unobserved time that the candidate spends with the

patient.

Acceptability was not addressed in the review of the long case identified in the literature review.

The traditional long-case assessment has a low utility in summative assessment.

Moderate Low Insufficient evidence



.

Feasibility of the Long Case

Acceptability of the Long Case

Utility of the Long Case

Table 13 Summary of Utility of the Long Case



Literature Review

Low utility for traditional long case assessmentsin summative assessment

81

Structured portfolios are used as a high stakes assessment in medical Foundation Training.

Portfolios are generally deemed to have high face and content validity due to participation of staff

and students in portfolio development and selection of relevant content for inclusion.

In general the reliability of portfolio assessment is low to moderate due to the wide variability in

how they are structured and assessed.

Portfolios are generally viewed as a valuable learning tool

Barriers to implementation to include lack of computer or internet facility in the clinical

environment, lack of complimentary faculty development initiatives and failure to integrate the

portfolio as part of the evaluation process.

Snadden (1998) describes a portfolio as “a collection of evidence that learning has taken place which

in practice includes documentation of learning and progression, an articulation of what has been

learned, and a reflection on these learning events/experiences.” Portfolios are used both as a

learning tool to stimulate reflective, experiential and deep learning and as an assessment method to

judge progression towards or achievement of specific learning objectives, competencies or fitness to

practice. Depending on the specialised purpose of the portfolio, its content including evidence

required, and assessment criteria vary from context to context. Any portfolio that is used for

assessment purposes should clearly articulate the amount, type and quality of evidence required to

establish proof of competence and the marking criteria used to evaluate the quality of the evidence.

Common portfolio documentation can include descriptive material and graded evidence including

written reports, critical incident reports, samples of performance evaluations, audit material, clinical

records of procedures undertaken, learning plans and written reflections about the evidence

provided and also in terms of identification of strengths, weaknesses, opportunities for

improvement and personal growth. Evidence can include workplace based assessments such as MSF,

DOPS and DOCS.


Key Findings

Description of Portfolios of Evidence (Taken from CIPHER, 2007)

82

On-line technologies are being increasingly used to support portfolio use and development and

electronic portfolios are becoming more popular due to their added flexibility.

Portfolios are used in high stakes assessment, for example a structured portfolio including examples

of workplace based assessment is used in Foundation Training. In Pharmacy portfolios are already

used to in pre-registration training to enable tutors to make a decision about the suitability of the

trainee to become a pharmacist.

Only one review addressed the validity of portfolios as a workplace-based assessment. CIPHER

(2007) found portfolios to have high face and content validity due to the participation of staff and/or

students in portfolio development and the selection of relevant content for inclusion. A study

reported in the review demonstrated the validity of portfolio assessment in dental training through

expert validation of the constructs, skills and content measured. There was significant correlation

between portfolios and traditional measures of competency.

CIPHER (2007) found that in general the reliability of portfolio assessment is low to moderate due to

the wide variability in how they are structured and assessed.

Miller and Archer (2010) reviewed three studies that looked at the impact of multiple assessment

methods on education and training. A large survey collected opinions of 539 surgical trainees on a

portfolio used to administer mini-CEX, CbD, DOPS, and MSF. Over 60% felt that the programme

aided personal development. The assessments provided a basis for feedback, but were found to be

Use of Portfolios of Evidence

Validity of Portfolios of Evidence

Reliability of Portfolios of Evidence

Educational Impact of Portfolios of Evidence

83

time consuming and a considerable administrative workload. No studies were found investigating

the impact of multiple methods on performance.

CIPHER (2007) found that portfolios are generally viewed as a valuable learning tool due to their

capacity to facilitate reflection on learning and practice, identification of student strengths and

weaknesses, presentation of evidence of clinical performance, planning for future learning and

change in practice.

CIPHER (2007) found the barriers to implementation to include lack of computer or internet facility

in the clinical environment, lack of complimentary faculty development initiatives and failure to

integrate the portfolio as part of the evaluation process.

The CIPHER review reported a study in which examiners were found to be strongly supportive of

portfolio assessment and its ability to identify strengths and weaknesses, but students were more

reserved and concerns were expressed about the amount of resources required to put the portfolio

together and variable standards of marking that were perceived to be used.

Potentially moderate utility for formative assessment but this is dependent on implementation.

Their low reliability excludes them as a being suitable as the major element of summative

assessment.

Feasibility of Portfolios of Evidence

Acceptability of Portfolios of Evidence

Utility of Portfolios of Evidence

84

Moderate Low Mixed results, generally neutral to moderate. Can be conditional depending on structure of portfolio and quality of reviewer feedback.

Resource dependent

Potentially Moderate

Table 14 Summary of Utility for Portfolios of Evidence



Literature Review

Potentiallymoderate utility for formative assessment but this is dependent on implementation.

Their low reliability excludes them as a being suitable as the major element of summative assessment.

85

There is now general agreement that assessment covering a range of professional competencies

needs to comprise a suite of tools based on van der Vleuten’s principle of “a shift from individual

methods to an integrated programme, intertwined with the educational programme” (van der

Vleuten, 2005). In this section we consider examples from pharmacy and medicine where this

‘instructional design’ approach has been put into practice.

The table below summarises pharmacy pre-registration assessment programmes in the UK, Canada,

New Zealand and Ireland.

MCQ

MCQ

MCQ

In 2009, the pharmacy regulator for Ireland (Pharmaceutical Society of Ireland, PSI) Council decided

that an academic institution would provide pre-registration training (known in Ireland as the in-

service practical training programme) on behalf of the PSI for a three-year period, starting with the

2009-2010 training year. The National Pharmacy Internship Programme (NPIP) is provided by the

Royal College of Surgeons in Ireland (RCSI) from 2009-2012 and is based on the RCSI programme for

trainees in surgery. NPIP has two components: the training programme (delivered principally online)

and the Professional Registration Examination.


Table 15 Pharmacy Pre-registration Assessment Programmes in the UK, Canada, New Zealand and Ireland

Workplace External to / Independent of workplace

Tutor assessment in workplace

Practice based assignment

Written examination

OSCE Interview

UK

Canada

New Zealand

Ireland

Case Study

The National Pharmacy Internship Programme (NPIP) in Ireland

v

v

v v v v v

v v v

86

Formative assessment includes tutor appraisals, ‘Competence Assessment and Performance

Appraisal’ (CAPA)1, ‘case moderated studies’, e-portfolio assignments2 and a dissertation based on

either an organizational development project or a clinical audit.

The Professional Registration Examination (PRE) is held twice a year and comprises:

MCQs in ‘prescribing science’: Part A 20 questions; Part B 40 questions

OSCE – 12 stations of length 5-7.5 minutes with one examiner at each (minimum 8 stations involving

interaction with a ‘simulated patient’ or demonstrating a practical skill)

The relative weightings of the PRE components (as specified in the RCSI Programme Overview) are:

MCQs in Pharmaceutical calculations (0.05)

Electronic case studies (e cases) in prescribing science (0.11)

MCQs in pharmacy law & ethics (0.17)

OSCE – 10 stations (0.67)

MCQ marking uses criterion referencing (measures intern performance against a set standard).

Criterion referencing makes no allowances for variation in test difficulty, such that standards must

be set for each item of the test. A number of methods to determine the passing standard are

available, RCSI uses the Angoff procedure. Item analysis is conducted post-assessment to ensure

quality of questions and of examiners.

OSCE marking uses the ‘Extended Criterion Referenced Marking Scheme’ developed by RCSI. The

Borderline Regression Method is used for standard setting of the interactive stations; written

stations have a pre-determined pass mark of 50%. Machine readable ‘process grids’ are used and

processing is automated using Speedwell Cliniquest software.

Royal College of Surgeons Ireland (RCSI). National Pharmacy Internship Programme: Programme

Overview 2009-10

1 CAPA was designed in RCSI to assess the progress of surgeons during training and was adapted for pharmacy2 Including: Learning Needs Assessment; Extemporaneous Preparation; Medication Usage Review; Patient Profile Templates; Medicine Information Queries; Discharge planning exercise; Aseptic preparation template

Sources:

87

Royal College of Surgeons Ireland & Pharmaceutical Society of Ireland. (2010) General Issues in the

Administration of the Professional Registration Examination 2009-10.

http://www.thepsi.ie/gns/education/becoming-a-pharmacist/professional-registration-exam.aspx

accessed 12/6/11

Royal College of Surgeons Ireland & Pharmaceutical Society of Ireland. Syllabus for the Professional

Registration Examination (June 2011)

88

All pre-registration trainees (interns) in New Zealand are required to undertake the EVOLVE training

programme of the Pharmaceutical Society of New Zealand. Assessment during the intern period in

New Zealand is based on a blueprint which maps competencies against assessment methods. These

include formative and summative written assessments, preceptor assessments and an assessment

centre (OSCE and interview). The regulator delegates responsibility to the Pharmacy Council Pre-

Registration Assessment Board (PRAB) for monitoring the assessment methodologies and tools used

for the EVOLVE training programme and for determining whether trainees meet the required

standard at the end of the year.

Accreditation standard 5 of the 2010 Standards for Pharmacy Intern Programs in New Zealand

requires that: “

” and under

5.2.1 that an ITP provider must have “

”.

Preceptors (pre-registration tutors) are required to complete a one-day workshop on Workplace

Assessor Training, (the NZQA Workplace Assessor Training Unit Standard 4098). The formative

assessments have been independently reviewed and are considered to have face validity; they have

not undergone psychometric validation.

Candidates attend an Assessment Centre after completion of requirements for intern training and

supervised practice. Initially a 5-station OSCE was used and after a recent review the New Zealand

Council:

- Is developing a pilot assessment, increasing the number of stations from 5 to 10 (and removing the

interview)

- Has issued a call to practising pharmacists to submit real life scenarios for use in the OSCE.

- Has issued a call to practising pharmacists to come forward to train as OSCE assessors

- Is considering the introduction of the MCQ developed and set by the Australian Pharmacy Council

and plans to introduce it for New Zealand

Interns pay a fee of c. $4000 New Zealand to the Pharmaceutical Society of New Zealand for the

EVOLVE training programme and assessments. They are also required to register with the Pharmacy

Case Study

Assessment programme for licensure as a pharmacist in New Zealand

An ITP (Intern Training Provider) must employ effective and validated assessment

methods within both the supervised practice and the training components of the ITP

systems in place providing interns with fair, objective reporting

on assessments with an associated robust, transparent and independent appeals process

89

Council of New Zealand as an intern pharmacist and pay an annual practising intern fee (total c. $600

New Zealand).

Pharmacy Council of New Zealand. Annual Report 2008

http://www.pharmacycouncil.org.nz/cms_show_download.php?id=83

Pharmacy Council of New Zealand. Annual Report 2009


Australian Pharmacy Council / Pharmacy Council of New Zealand (May 2010). Accreditation

Standards for Pharmacy Intern Programs in New Zealand. The prescribed qualification for the

pharmacist scope of practice in New Zealand.


The Foundation Programme was implemented in 2005 as part of the Modernising Medical Careers

reform process. Workplace based assessments were introduced into the training programme from

the start. The assessments used are described table 16, below. The ‘Developing the Clinical Teacher’

assessment was introduced recently. This is an assessment of the foundation doctor’s skills in

teaching and/or making a presentation. The assessment can be on one-to-one or group teaching.

The Foundation Training Programme Director and the Foundation School Director review the

trainee’s portfolio at the end of the year and make a judgement, based on the evidence provided in

the portfolio as to whether or not the trainee has met the requirements for satisfactory completion

of the year. Failure of any of the assessments during the year does not mean failure to complete the

year. Trainees must demonstrate a satisfactory level of competence in all assessments by the end of

the year.

In a recent review of the Foundation Programme (Collins, J, 2010) the workplace-based assessments

were seen as central to the philosophy of the programme. Regular assessment ensures progression,

provides documentary evidence of achievements and can be used to identify trainees with

problems. Despite this, workplace-based assessment tools have not gained widespread support

Sources

Case Study

Workplace Based Assessment in the Medical Foundation Programme

90

from trainers or trainees. Questions have been raised about the validity of the assessment tools and

their variable application. This is attributed to the lack of preparation of the assessors (a separate

issue that was highlighted by the report) and insufficient time in which to undertake the

assessments properly. The number of assessments required is seen by trainers and trainees as time

consuming and onerous with comments that they are no more than a tick box exercise. The

introduction of electronic forms of the assessments has reduced the burden slightly. There are also

concerns that the high scores being obtained in the assessments are not discriminating of less good

performance.

The overall recommendations from the report are that the range of assessment tools and frequency

of assessments must be urgently reviewed and modified based on data available on assessment in

the GMC surveys and feedback from trainers, otherwise the credibility of the Foundation

Programme in the eyes of teachers and trainees will be compromised. A further recommendation of

the report was the need for improved transfer of information between undergraduate and

postgraduate schools to avoid unnecessary and repetitive assessments. Finally, it was recommended

that methods must be found to support and recognise those who aspire to excellence.

The Foundation Programme Curriculum, March 2010Source:

91

MSF (using Team Assessment Behaviour)

Once at the start of the year, repeated towards the end if necessary.

Trainee’s select at least 15 raters for each MSF, a minimum of 10 returns are required. The mix of assessors should be:

2-8 doctors more senior than F2.

2-6 nurses

2-4 allied health professionals

2-4 other members of the team

DOPS 0-3 a year to demonstrate procedures not in the log book. Can replace up to 3 mini-CEX.

Different assessors should be used where possible.

Mini-CEX A minimum of nine per year. Up to 3 can be replaced by DOPS.

Up to three mini-CEX can be replaced by DOPS. Different assessors should be used wherepossible.

CBD At least six per year Two should be competed in each four month period. A different assessor should be used for each CbD.

Log book of procedural skills Trainees need to provide evidence that they can competently perform all 15 procedures listed in the log book by the end of Foundation Year 1.

Developing the clinical teacher assessment form.

At least one per year.

e-Portfolio The portfolio contains the workplace based assessments as well as audits, reflective learning reports, evidence of course attendance and probity and health declarations.

Table 16 Assessment tools in the Foundation Programme

Assessment Frequency of assessment Comments

92

The literature review findings consistently show that the reliability of practice-based assessment is

highly operator dependent and that training for assessors is crucial. Selection and training of

assessors is essential to ensure they have the skills, understand the process of the assessment, and

can address issues of equal opportunity (Wass et al, 2007). Two examples of available training are

described below, one developed specifically for pharmacy and the other a generic module intended

for a range of health professionals.

Assessor training

Practice Example

E-Learning for Practice Supervisors: DEPS (Developing Educational & Practice Supervisors)

Programme

”

An e learning programme was developed as part of work by a local group in the South East of

England to create an accreditation and QA framework for pharmacy practitioners involved in

facilitating and supervising learning in the workplace. The DEPS framework is a tool to help a

tutor become an advanced level practitioner, with extensive mapping to the Advanced and

Consultant Level Framework (ACLF) (CoDEG, 2009). Strong links between the frameworks are

supported by their similarity in structure (1 (Foundation), 2 (Advanced) and 3 (Mastery)).

The group utilised available frameworks and resources on tutoring in pharmacy and related

professions and based upon terms used in medical and other health professions three ‘tiers’ of

pharmacy tutors were identified: the Practice Supervisor (PS), Educational Supervisor (ES) and

Educational Programme Director (EPD). A “Practice supervisor in pharmacy is responsible for

overseeing a specified trainee’s work and providing developmental feedback during a period of

training. This role requires appropriate assessment skills. Practice supervisors support learners

to identify opportunities for learning in the workplace and provide supervision of trainees on a

day-today basis, identifying trainees in difficulty. Practitioners may choose to progress along

the PS to EPD continuum or to continue in a practice supervisor role where this is an important

but lesser part of their primary role.

An online three module 12 week asynchronous Practice Supervisors course

was piloted between January and June 2011 with 14 participants. Modules are facilitated by an

Pract ice S upervisors:

93

online tutor and completed in an online group setting with other practice supervisors. The

modules are: 1. Educational theory; 2. Learning relationships (planning); 3. Review and feedback

(includes “mark your trainee against standards/competencies”)

At the time of writing the pilot is ongoing. The team report that initial feedback from Module 1

participants was positive.

This module is aimed at health professionals taking on responsibility for educational supervision

and aims to enable participants to:

- Recognise the scope, role and academic responsibilities of clinical and educational supervision

- Develop ways of making the clinical environment an effective learning environment

- practise facilitating learners through developmental conversations to identify and address

learning needs, improve their reflective practice and safely develop their clinical expertise.

The module addresses other key areas of education supervisory roles including work place based

assessment, managing trainee progression including working with struggling trainees, personal

and professional development planning including career guidance.

Participants complete a written piece of work based on their independent project work to

develop a portfolio that demonstrates how their learning on this module is embedded in their

supervisory practice. They will also produce a video or transcript of an authentic supervision

session which they will analyse. The module comprises 4 days of face-to-face teaching.

Practice Example

Module: Supervision Skills for the Healthcare Professional, UCL Division of Medical

Education

94

This report brings together evidence from the peer reviewed literature and documentary evidence,

with case studies and practice examples from the UK and more widely to inform discussion on the

utility of practice-based assessments in pharmacy pre-registration training. Van der Vleuten’s Utility

Index proved a useful 'conceptual framework against which to consider the available tools. The

literature review generated data on reliability, validity and educational impact of different

assessment tools but evidence on feasibility and acceptability was sparse in published studies. Much

of the published evidence is from outside of pharmacy and we were able to supplement the

literature review findings with case studies drawn primarily from pharmacy practice to shed further

light on issues of feasibility and acceptability. In the discussion that follows we begin with the

principal findings relating to individual tools.

Published literature shows the specific Mini-CEX tool to be feasible and acceptable. Data on

reliability indicate that four to six completed Mini-CEX are required but debate remains about

reliability of multiple assessments by a single assessor versus a single assessment by a series of

assessors on different occasions. Feasibility of Mini-CEX in the community pharmacy setting is

constrained if multiple assessors are needed in order to provide a sufficient level of reliability.

Assessor training is crucial for feedback to be credible and likely to be taken on board by the trainee.

Mini-CEX is a structured assessment of an observed clinical encounter and focuses on the areas in

table 17 below (Foundation Programme, 2010).

Discussion

Utility of individual tools


95

Facilitates patient telling their story; effectively uses appropriate questions to obtain accurate, adequate information; responds appropriately to verbal and non-verbal cues.

Follows efficient, logical sequence; examination appropriate to clinical problem; explains to patient; sensitive to patient’s comfort and modesty.

Explores patient’s perspective; jargon free; open and honest; empathic; agrees management plan/therapy with patient.

Makes appropriate diagnosis and formulates a suitable management plan; selectively orders/performs appropriate diagnostic studies; considers risks and benefits.

Shows respect, compassion, empathy, establishes trust; attends to patient’s needs of comfort; respects confidentiality; behaves in an ethical manner; awareness of legal frameworks; aware of own limitations.

Prioritises; is timely and succinct; summarises.

A global judgement based on the above question areas.

The Joint Programmes Board uses mini-CEX in postgraduate pharmacy education to provide

feedback on the skills essential to providing good pharmaceutical care. It assesses the ‘Delivery of

Patient Care’ and the ‘Problem Solving’ clusters of the GLF. An example of the use of mini-CEX would

be a patient with COPD who has a new inhaled treatment and the pharmacist is explaining and

checking technique as well as discussing the new medicine.

Within pharmacy the Joint Programme Board has developed the Medication Related Consultation

Framework which we have categorised this as a variant of DOCS specific to medicines. The purpose

of a medicines related consultation is defined by JPB as “

” (Abdel-Tawab et al, 2011). The

MRCF was developed from a review of medical consultation frameworks which identified areas

relating to medicines which were not well covered by existing instruments (see Appendix 3). There

are long (46 items) and short form (31 items) versions of MRCF with some data on reliability of the

long form version. The short form MRCF is a mandatory component of assessment in the JPB

Diploma in General Pharmacy Practice and is also used in undergraduate assessment by the

Table 17 Areas of the Clinical Encounter Assessed by mini-CEX

Question area Positive indicators

History taking

Physical examination skills

Communication skills

Critical judgement

Professionalism

Organisation & efficiency

Overall clinical care

to conduct a therapeutic review with the

aim of assessing a patient’s pharmaceutical care needs. It is key to understanding how patients

relate to their medicines, including their concerns and expectations for treatment and to anticipate

or resolve medication related problems, in particular non-adherence

96

academic pharmacists involved in its development. From practice examples it appears that MRCF is

used for Medicines Use Review, Clinical Medication Review and prescribing.

A Generic Consultation Skills tool (GeCoS) for use in the assessment of medical consultations has

recently been validated in a Delphi type study (LeFroy et al, In Press) following extensive piloting.

The new tool was developed based on mapping of the competencies in Tomorrow’s Doctors, the

Calgary Cambridge guide to the medical interview and the Leicester Assessment Package. This new

tool can be reviewed for its relevance and applicability to different types of pharmacy consultations.

DOPS is used to refer to direct observation of both ‘Procedural’ and ‘Practical’ skills. ‘Procedural’

skills in UG medicine are listed in ‘Tomorrow’s Doctors’ (GMC, 2009) and include diagnostic,

therapeutic and general procedures (the latter including, for example, taking consent). DOPS is an

assessment of a procedure being conducted with a real patient. Although some procedural skills are

relevant to pharmacy practice they may be less so in the pre-registration period. We found examples

from pre-registration stakeholders where tools are in use e.g. for dispensing procedures.

There is evidence from published literature to show reliability and validity of OSCEs in formative and

summative assessment providing an appropriate blueprinting process is used in development and

testing of the assessment strategy. There appears to be consensus that a minimum of 10 stations is

needed for adequate reliability. There is also debate in the literature about the number of assessors

needed per station but as each assessment strategy is individual it is difficult to generalise. Training

of assessors is agreed to be crucial. Wass argues that strengths of OSCEs are improved reliability and

the ability to test communication skills and attitudinal behaviours and weaknesses are that they

“are expensive, labour intensive and a challenge to feasibility” (Wass, 2007). Nevertheless OSCEs

are used in high stakes national assessments in pharmacy in Canada (the largest cohort size with

around 900 candidates per year), New Zealand and Ireland as part of pre-registration requirements.

Increasing numbers of Schools of Pharmacy in the UK appear to be using OSCE assessments, thus

increasing the likelihood that trainees and academic staff will be familiar with the assessment

method by the time they reach pre-registration training. There is, as yet, no published information

Direct Observation of Procedural Skills (DOPS)

OSCE

97

on the OSCEs in use in UK Schools of Pharmacy; a US study found that around one third of Schools

used OSCEs (Sturpe, 2010). The results showed wide variation in OSCE practice and the author

concluded that “

”.

A number of pharmacy pre-registration training providers in the UK have already incorporated the

use of OSCEs for formative assessment into their training programmes. If there were to be

consideration of use of OSCE as a national summative assessment in pharmacy pre-registration

programmes a review of OSCE practice in the UK using published guidance on quality metrics (see,

for example Pell et al, 2010) would be useful in determining quality as well as capacity and

preparedness of assessors and developmental needs.

SPs have been used in community pharmacy practice to assess the ‘does’ level of Miller’s pyramid of

clinical assessment as part of quality improvement and CPD programmes, particularly in Scotland

and Australia. A ‘mystery shopper’ type assessment could theoretically be used in pharmacy pre-

registration programmes but would be logistically challenging and of low reliability if only one

assessor was used. Standardised patients are used in OSCE stations and are now used widely in

undergraduate medical schools.

The Mini-PAT assessment tool seems to be the most widely used MSF tool, is now well embedded

within medical F1 medical training and is increasingly being used in specialty training and in

revalidation pilots. This widespread use in medicine has developed without substantial published

evidence on reliability and validity although there seems to be an emerging consensus that reliability

is acceptable providing there is a minimum of 8-10 raters. Within pharmacy some evidence on Mini-

PAT has been generated from the JPB postgraduate programme, mainly in hospital pharmacy. The

findings of Patel and colleagues (Patel et al, 2009) show that the tool is feasible and acceptable to

use in hospital practice and there were some, albeit limited, indications of educational impact.

Feasibility of Mini-PAT in the community pharmacy setting is constrained by the need for multiple

raters in order to provide a sufficient level of reliability. Mini-PAT is being used by some Schools of

few colleges and schools of pharmacy conduct OSCEs in an optimal manner, and

most do not adhere to best practices in OSCE construction and administration

Standardised patients (SPs)

MSF

98

Pharmacy (including Portsmouth and Medway) in community pharmacy. Medway also uses mini-PAT

in its training programme for Pharmacists with Special Interest (PhwSI) (personal communication

Gammie S). Training of raters and those giving the feedback in MSF has been shown to be critical for

feedback to be perceived as credible and acceptable.

Scalability of MSF has been demonstrated in medicine and a recent analysis of MSF data relating to

almost 1000 doctors (McKillop et al, 2011) concluded

. Evidence on acceptability in medicine comes from a study of trainees, raters and

supervisors (Burford et al, 2010). Attitudes towards MSF were reported to be positive overall

although there differences in views about whether MSF could provide developmental feedback and

also about whether it could identify doctors in difficulty.

Patient surveys have used in medicine to obtain feedback for individual practitioners about their

consultations. Revalidation models in medicine have included the use of patient surveys, particularly

in general practice, and there are several available tools. Since the tools focus on individual

consultations the feasibility of obtaining feedback from patients for pre-registration pharmacy

trainees would be dependent on the nature and extent of the interactions between the trainee and

patients. Sufficient numbers of responses would be needed to give meaningful feedback with a

suggested minimum of 25 (Campbell et al, 2010).

CbD is not strictly a practice-based assessment in the traditional sense as here the trainee is not

being observed in practice but instead is presenting a patient case with their decisions and the

thinking behind them. The CbD is used in F1 where its strength is regarded as being assessment and

discussion of clinical reasoning. During the F1 year the trainee has to complete a minimum of six

CbDs with at least two in any four month period. Guidance states that different assessors should be

used for each CbD wherever possible and that assessors should have sufficient experience of the

area under consideration, typically higher specialty training, with variations between specialties.

Each CbD must represent a different clinical problem, sampling one of acute care, chronic illness,

psychiatric care etc (categories listed in Syllabus and competencies). Whilst the use of different

“it is feasible to electronically administer a

generic questionnaire to a large population of doctors. Generic content is appropriate for most but

not all specialties”

Patient surveys

Case Based Discussion (CbD)

99

assessors from different areas of practice may be feasible in a hospital pharmacy setting, for pre-

registration trainees in the community this would prove more challenging.

The traditional long case, like CbD is not strictly a practice-based assessment as the trainee is not

being observed in practice but is presenting a case with their decisions and thinking. Modifications of

the long case do use direct observation of practice but the use of the long case has generally been

superseded by DOCS assessment tools. Although traditionally used as a high stakes assessment the

traditional long case has low utility as a high stakes assessment.

There are difficulties in trying to assess the utility of the portfolio of evidence since portfolios can

vary considerably in structure and content, and consequently there is no standardisation in how they

are assessed. This lack of structure and standardisation can, in turn, lead to problems with the

validity and reliability of portfolios. The underlying theme emerging from the literature is that any

portfolio used for assessment purposes should clearly articulate the amount, type and quality of

evidence required to establish proof of competence and the marking criteria used to evaluate the

quality of the evidence. Portfolios should not just be a ‘suitcase’ for holding various pieces of

evidence; there needs to be some structure to the portfolio, some reflection on how the evidence

within the portfolio relates to the competencies being demonstrated, and some linking of the whole

portfolio together to identify what it shows about the individual’s practice. In this way the portfolio

becomes more than the sum of its parts and it is clear what has been achieved, how the evidence

supports this and what has been learnt (Middleton, 2011).

Concerns about inter-tutor variation in assessment of portfolio evidence in pharmacy pre-

registration training prompted a recent study in which the portfolios and tutor sign offs against

performance standards were reviewed for 22 pharmacy pre-registration trainees and their tutors in

5 NHS Trusts (Hollister & McDonald, 2011). In addition to counts of pieces and types of evidence,

performance standards claimed and those signed off, over 200 pieces of evidence submitted at sign

offs 2 and 3 were independently reviewed and the decisions compared against those of the tutor.

The results showed considerable variation in the amount and types of evidence provided in the

The Long Case


100

portfolios and also in the number of times a performance standard was demonstrated before being

signed off by a tutor. The independent review of 1178 performance standards claimed by trainees

showed that 736 (63%) were approved by the tutor but 1099 (86%) would have been approved by

the investigator.. This demonstrates the lack of reliability in assessing portfolios without clear

guidance to assessors, as highlighted in the review literature. The authors questioned the adequacy

of current guidance to tutors in ensuring that sufficient, but not too much, evidence is provided,

making reference to “ ”.

In contrast to the pharmacy pre-registration portfolio the Foundation Training portfolio has a more

defined structure and includes the workplace-based assessments as evidence that the competencies

in the Foundation Programme Curriculum have been met. Trainees are given guidance on the

amount of evidence they are required to provide in the portfolio.

Having considered the individual assessment methods and tools we now go on to consider two key

aspects of infrastructure needed for robust assessment which were consistent themes from the

literature review: programmes of assessment and the training of assessors. We then go on to

consider the strengths and limitations of the project before a discussion covering the assessments

currently used in pharmacy pre-registration training and our reflections on the project findings in

relation to the future.

A clear finding of the literature review was that no single assessment method can meet the

requirements of a professional training programme involving a complex set of inter-related

competencies. A structured approach of mapping competencies against possible assessment

methods supports consideration of what combination/s of assessments might make an appropriate

programme. The medical F1 programme has a published map of competencies and the methods

used to assess them (see Appendix 4).

We have constructed two further maps:

i) A list of generic compencies and the wpba that have been used to assess them (extracted from the

CIPHER (2007) review, see Appendix 1)

wheelbarrow syndrome


101

ii) The current pre-registration performance standards against the available assessment methods.

(Appendix 5).

We would draw the attention of GPhC to two programmed assessment approaches in current use in

pharmacy: those in the new NPIP pre-registration programme in Ireland and those in the JPB

postgraduate Diploma in General Pharmacy Practice. Both have drawn upon assessment

components used in medicine and applied them in the pharmacy setting.

As the literature review showed, assessor training is an essential pre-requisite for reliable and valid

assessment using any practice-based assessment method. There are existing examples within

pharmacy which GPhC might want to consider including the JPB assessor guidance for the individual

tools used in their programme and the DEPS modules.

This project was a Rapid Evidence Asessment (REA) using a review of reviews, a methodology

frequently used to support policy development. REAs enable an evidence-based consideration of

policy questions in a timeframe in which a traditional systematic review would not be feasible. The

search strategy for the review was developed by a specialised librarian and refined iteratively. Tests

showed the final search strategy to have the ability to find key papers.

A strength of REAs is that they draw on systematic reviews which have themselves sifted the

available studies and graded their quality. However this methodology is naturally dependent on the

quality of the source reviews including the rigour of the identification of their source papers, their

selection and exclusions, data extraction and data synthesis through to the conclusions reached. We

were inclusive in our approach and thus decided not to exclude a small number of non-systematic

reviews because they enabled us to address gaps in the evidence (for example on mini-PAT).

Currency of included studies is an issue with published reviews of literature and very recent

evidence will not have been accessed by the published reviews. We were able to address this to a

limited extent by supplementary searches for more recent studies.

Assessor training

Strengths and limitations of this project

102

The scope of our review, as agreed with GPhC, was OSCE plus assessment methods in use in WPBA in

medicine and JPB in pharmacy as the infrastructure needs and experience of using these in the UK is

now considerable. There are other assessment tools which were not included in the reviews which

could potentially make a contribution to assessment in pharmacy but where there is little or no

mainstream use in the UK.

Most of the published evidence is not from pharmacy practice so there must be some uncertainty

about the extent to which it will generalise to pharmacy. We addressed this through our use of case

studies and practice examples as supplementary evidence in the pharmacy context. This

strengthened the report, particularly in evidence on the feasibility and acceptability of particular

methods.

Having considered the individual assessment tools and key aspects of infrastructure we now move

on to discuss the current pharmacy pre-registration assessments, reflect on our findings in relation

to the future and raise issues that our analysis has identified as being important for consideration by

GPhC.

Current pre-registration assessments comprise a series of tutor sign offs plus a high stakes MCQ

examination near the end of the year. The strengths and weaknesses of the two components need

to be considered in addition to the evidence in this report about individual assessment tools and

their operation in practice.

MCQs were not part of our formal brief because they fall outside the definition of practice-based

assessments. The Modernising Pharmacy Careers Programme, in their proposals for an integrated 5-

year MPharm programme, said about the period of pre-registration training that it “

” (MPC). (Smith & Daracott, 2011). MCQs

have high levels of reliability but an important weakness of the MCQs generally is that they do not

produce active generation of knowledge (Wass et al, 2007). Developments in MCQ formats have

gone some way to addressing concerns about cueing from the offered responses by utilising clinical

scenarios and extracts of clinical and scientific data. Canada has adopted these principles. Its MCQs

are based on clinical cases and are developed using the same type of rigorous process as for their

OSCE stations.

Utility of the current pre-registration assessments

tends to be

overshadowed by the return to an academic-style assessment in the final registration exam, which is

scheduled towards the very end of the pre-registration year

103

Turning to the sign offs by the pre-registration tutor, the literature review showed clearly the critical

importance of assessor capability. It is this element of variability between tutors that GPhC has

already recognised in its decision to introduce tutor standards. Tutors sign off trainees against the

Performance Standards utilising a mix of observation of practice and the evidence the trainee has

collected and included in her/his portfolio. Responses from pre-registration stakeholders in both our

previous work on pre-registration training and the current project show that some workplace based

assessment tools are already in use, mainly in hospital pharmacy. There were also some indications

that the use of some tools may be targeted towards trainees who have difficulties meeting the

performance standards (for example, the use of 360 degree feedback where a trainee is having

problems). These developments are likely to be helpful in introducing a more systematic and

structured approach to assessment. We also found examples of tutor training and development in

assessment.

The findings of this project can be used by GPhC to inform discussions on future assessments in

pharmacy pre-registration programmes.

Issues of transferability of evidence from medicine and from pharmacy in other countries need some

consideration. Feasibility of large scale OSCE programmes in pharmacy has only been tested in

Canada. The pioneering work of JPB has translated and further developed WPBA tools used in

medicine but there are questions on scalability from the hundreds of students involved so far to the

thousands that would be involved pre-registration training.

Unsurprisingly the review of assessment tools does not support any recommendation of a specific

tool, nor would this be appropriate given current thinking among experts in assessment. Two

quotations capture the key finding from the literature review that more than one method of

assessment is needed and it is the utility of the whole that is important. van der Vleuten argues that

“

” (van der Vleuten & Schuwirth, 2005) and Wass states that “

”

Towards the future

We should not evaluate individual methods, but provide evidence of the utility of the assessment

programme as a whole to assess clinical

competence validly we are moving away from batteries of examinations to an assessment package

where performance in the workplace can be included alongside high stakes examinations such as

multiple choice tests. No single one can be valid given the complexity of clinical competency itself

104

(Wass et al, 2007). Consideration of existing and possible future ‘packages’ of assessments is

therefore the way forward.

GPhC may wish to consider the principles which will underpin future policy on assessment and may

find those adopted by Postgraduate Medical Education and Training Board (PMETB, now merged

with the GMC) and which are summarised below, a useful starting point.

Summary of PMETB principles of assessment

1. Methods must reflect the assessment’s intended purpose/content

2. Reference intended assessment content to ‘Good Medical Practice’

3. Ensure methods used to set standards are in the public domain

4. Involve lay members in the assessment process

5. Have mechanisms for giving trainees feedback on performance

6. Use appropriate criteria for examiner training

7. Use standardised documentation which is available nationally

8. Be sufficiently resourced

We found little evidence of practice-based assessment in high stakes summative assessments with

the exceptions of OSCE and videotaped patient consultations in the RCGP entry assessment. OSCEs

are now in use in entry to practice assessments in Canada, New Zealand and Ireland. However they

are used as well as, not instead of, multiple choice tests. Austin (2003), an internationally regarded

expert on OSCEs is of the view that “

”. Some

would argue that the direction of travel for pharmacy, with consultations with patients becoming

more central, necessitates a national benchmarking assessment which includes communication and

consultation skills.

The educational impact of assessment methods is important in formative assessment. Little is known

about the educational impact of the current system of tutor sign offs, or about the extent to which

feedback in discussions between tutor and trainee are effective in enabling trainees to formulate

learning plans for areas in which they need to develop. There is evidence of educational impact for

some of the individual tools reviewed in this report. However both new and existing assessment

assessment of knowledge through written tests and

performance-based assessment complement one another but do not replace one another

105

methods are dependent on the skills of the pre-registration tutor in assessment and feedback.

Educational impact is less of an issue with summative assessment.

We included some examples of assessor training in this report to illustrate the sorts of programmes

on offer through e-learning and face to face modules. In our view assessor training is a major

challenge for GPhC regardless of whether the existing assessments are retained or changed. Our

pilot survey of pre-registration trainees and tutors (Mills et al, 2010) included ratings of tutor skills

and if the survey were to be implemented in the future these data would provide an annual skills

audit providing pointers to areas where training might be needed.

The findings of this project can be used by GPhC in the review of pre-registration training in the

current and possible future contexts. The evidence that we have synthesised on validity, reliability,

educational impact, acceptability and feasibility of different assessment methods together with the

map against pharmacy pre-registration standards could be the basis of a blueprint for future

assessment. Our findings also have relevance to how the current national requirements for practice-

based assessment of pharmacy pre-registration trainees could be strengthened.

1. In what ways can the current practice-based assessments conducted by pre-registration

tutors be strengthened?

2. Is a national benchmarking assessment of communication skills and attitudinal behaviours

needed or is practice-based assessment by the tutor sufficient?

3. Can the existing multiple choice examination be further developed?

4. What support needs to be offered to tutors and others involved in practice based

assessment of pharmacy pre-registration trainees?

Conclusion

Key issues for consideration by GPhC

106

Abdel-Tawab, R., James, D.H., Fichtinger, A., Clatworthy, J., Horne, R., Davies, G. (2011) “Development and Validation of the Medication-Related Consultation Framework (MRCF)”. Patient Education and Counselling (In Press, online advance copy obtained)

Abdulla, A. (2008). A critical analysis of mini peer assessment tool (mini-Pat). 101, 22-26.

Austin, Z., O’Byrne, C., Pugsley, J., Munoz, L.Q. (2003) Development and validation processes for an Objective Structured Clinical Examination (OSCE) for Entry-to-Practice Certification in Pharmacy: the Canadian Experience. , Article 76

Burford, B., Illing, J., Kergon, C., Morrow, G., Livingston, M. (2010) User perceptions of multi-sourcefeedback tools for junior doctors. 165-76.

Campbell, J., Narayanan, A., Burford, B., Greco, M. (2010) Validation of a multi-source feedback tool for use in general practice. 165-79.

Centre for Innovation in Professional Health Education & Research (CIPHER) (2007). Review of work-based assessment methods. The University of Sydney, Australia.

Clarke, R. 2007. Preparing teachers for work-based teaching and assessing in: Assessment in Medical Education. Radcliffe Press.

Collins, J. (2010) Foundation for Excellence. An Evaluation of the Foundation Programme. Medical Education England, London.

Competency Development and Evaluation Group (CODEG). (2007). General Level Framework: A Framework for Pharmacist Development in General Pharmacy Practice. Available at http://www.codeg.org/fileadmin/codeg/pdf/glf/GLF_October_2007_Edition.pdf <accessed 23.6.11>

Competency Development and Evaluation Group (CODEG). (2009). Advanced to Consultant Level Framework. A developmental framework for pharmacists progressing to advanced levels of practice. Available at http://www.codeg.org/fileadmin/codeg/pdf/ACLF.pdf <accessed 23.6.11>

Foundation Programme (2010). Mini Clinical Evaluation Exercise (mini-CEX). Guidance for Assessors.

General Medical Council (2006). Good Medical Practice. Guidance for Doctors. GMC, London. Available at http://www.gmc-uk.org/guidance/good_medical_practice.asp <accessed 23.6.11>

General Medical Council (2009). Tomorrow’s Doctors. Outcomes and standards for undergraduate medical education. Available at http://www.gmc-uk.org/static/documents/content/TomorrowsDoctors_2009.pdf <accessed 23.6.11>

Hamilton, K., Coates, V., Kelly, B., Boore, J., Cundell, J., Gracey, J., McFetridge, B., McGonigle, M., Sinclair, M. (2007)Performance assessment in health care providers: a critical review of evidence and current practice. , 773-791.

Hollister A, McDonald J. (2011) Portfolio review and standardisation for pre-registration trainees. UKCPA.

References

67 (3),

44(2),

21(3),

15

J R Soc Med,

American Journal of Pharmaceutical Education

Med Educ,

Educ Prim Care,

Journal of Nursing Management,

107

Kogan, J., Holmboe, E., Hauer, K. (2009) Tools for direct observation and assessment of clinical skills of medical trainees. , , 1316 – 1326.

Lefroy, J., Gay, S., Gibson, S., Williams, S., McKinley, R.K. Development and face validation of an instrument to assess and improve clinical consultation skills. In press.

Mackillop, L.H., Crossley, J., Vivekananda-Schmidt, P., Wade, W., Armitage, M. (2011) A single generic multi-source feedback tool for revalidation of all UK career-grade doctors: does one size fit all? , e75-83.

Melnyk, B.N., Fineout-Overholt, E. (2005). Evidence-based practice in nursing and healthcare: a guide to best practice. Lippincott Williams and Wilkins, Philadelphia.

Middleton, H. (2011). How to build your professional portfolio (and why you should). , 119-121.

Miller, A., Archer, J.. (2010) Impact of workplace based assessment on doctors’ education and performance: a systematic review. , , c5064 DOI:10.1136/bmj.c5064

Miller, G. E. (1990) The Assessment of Clinical Skills/ Competence/ Performance. , s63-s67.

Mills, E., Black, P., Blenkinsopp, A. (2008) Future quality management strategies for pharmacy pre-registration training: A scoping study. Department of Medicines Management, University of Keele.

Mills, E., Bracegirdle, L., Blenkinsopp, A., Black, P. (2010). Future quality management strategies for pharmacy pre-registration training: Survey of trainees and tutors. Department of Medicines Management, University of Keele.

Munoz, L., O’Byrne, C., Pugsley, J,. and Austin, Z. (2005). Reliability, validity, and generalizability of anobjective structured clinical examination (OSCE) for assessment of entry-to-practice in pharmacy.

33-43

NHS. Medical Assessment Compendium. Available at http://www.ecompendium.nhs.uk/index2.asp<accessed 23.6.11>

Patel, J.P., West, D., Bates, I.P., Eggleton, A.G., Davies, G. (2009) Early experiences of the mini Peer Assessment Tool amongst hospital pharmacists in south east London.

, , 123-126.

Pelgrim, E., Kramer, A., Mokkink, H., ven den Elsen, L., Grol, R., van der Vleuten, C. (2011) In-training assessment using direct observation of single patient encounters: a literature review.

, 131-142.

Pell, G., Fuller, R., Homer, M., Roberts, T.; International Association for Medical Education. (2010) How to measure the quality of the OSCE: A review of metrics - AMEE guide no. 49. Med Teach.,

), 802-11.

Ponnamperuma, G., Karunathilake, I., McAleer, S., Davis, M. (2009) The long case and its modifications: a literature review. , 936-941.

Rushworth, H. (2007) Objective structured clinical examination (OSCE): Review of literature and implications for nursing education. , , 481-490.

JAMA

Med Teach

Clinical Pharmacist,

BMJ

Academic Medicine

Pharmacy Education

International Journal of Pharmacy Practice

Adv in Health Sci Educ,

Medical Education,

Nurse Education Today

302(12)

33(2),

3

341

65,

, 5(1),

17

16(1)

32(10

43

27

108

Shanley, E. (2001) Misplaced confidence in a profession’s ability to safeguard the public? 2001, , 136-142.

Smith, A., Darracott, R. (2011) Review of pharmacist undergraduate education and pre-registration training and proposals for reform . Discussion Paper. Modernising Pharmacy Careers Programme, Medical Education England.

Snadden, D. (1998). The use of portfolio learning in medical education. , 192-199

Sturpe, D.A. (2010) Objective structured clinical examinations in doctor of pharmacy programs in the United States. Am J Pharm Educ, 148.

Van der Vleuten, C.P.M. (1996). The assessment of professional competence: developments, research and practical implications. Advances in Health Sciences Education, , 41-67.

Van der Vleuten, C.P.M., Schuwirth, L.W.T. (2005) Assessing professional competence: from methods to programmes. Medical Education, , 309-317

Walsh, M., Hill Bailey, P., Koren, I. (2009) Objective structured clinical evaluation of clinical competence: an integrative review. , , 1584-1595 DOI: 10.1111/j.1365-2648.2009.05054.x

Wass, V., Bowden, R., Jackson, N. (2007). The principles of assessment design. In: Assessment in medical education. Radcliffe Press.

Watson, R., Stimpson, A., Topping, A., Porock, D,. (2002). Clinical competence assessment in nursing: a systematic review of the literature. , 421-431.

Winslade, N.E., Tamblyn, R.M., Taylor, L.K., Schuwirth, R.W.T., Van der Vleuten, C.P.M. (2007) Integrating performance assessment, maintenance of competence, and continuing professional development of community pharmacists. American Journal of Pharmaceutical Education, Article 15.

Wood, L., Hassell, A., Whitehouse, A., Bullock, A., Wall, D. (2006) A literature review of multi-source feedback systems within and without health services, leading to 10 tips for their successful design.

, e185-e191.

Nurse Education Today

Medical Teacher,

Journal of Advanced Nursing

Journal of Advanced Nursing,

Medical Teacher,

21

20(3)

74(8),

1

9

65(8)

39(5)

71 (1),

28 (7)

109

Appendices

Appendix 1 Generic Competencies Assessed through Workplace-Based Assessments

Generic Competency MSF DOPS Standardised Patients mini-CEX Patient surveys Portfolios

Patient care Y y

Medical Knowledge y y

Professionalism y y y

Interpersonal and communication skills y y y

Practice-based learning y

Systems-based practice y y

Team working y

Accessibility y

Trustworthiness y y

Administrative Skills y

CPD y

Collaboration y

Clinical management y y

Leadership development y

Personal health y

Procedural (technical) skills y

History taking y y

Physical Exam y y y

Clinical judgement y

Organisation/efficiency y

Counselling y y

Risk management y

110

Problem solving y

Decision making y

Response to ethical and professional dilemmas y

Information management y

Change management y

111

Limits of search: Humans; review (publication type); 01/01/1990 to 28/02/2011

Educational measurement (MeSH term) AND (clinical OR skills OR competence OR practice OR work

OR workplace OR work-based OR work based)

AND

Education, Pharmacy (MeSH term) OR (Education, Medical (MeSH term) AND (trainee OR student OR

foundation)) OR Nursing Education Research (MeSH term) OR Nurse midwives/education (MeSH

term subheading) OR Physical therapy (speciality) (Mesh term) OR occupational therapy (Mesh) OR

dentists (MeSH)

OR

Educational AND (impact OR measurement OR standard OR effects)

OR

Utility OR validity OR reliability OR feasibility OR acceptability

Educational measurement (MeSH term)

AND

OSCE OR observed structured clinical examination OR simulated patient encounter OR standardised

patient exam

OR

“direct observation”

OR

Appendix 2 Search Strategies

Medline Search Strategy(Yielded 944 papers)

SEARCH 1

SEARCH 2

112

“multi source feedback” OR MSF OR “team assessment behaviour” OR TAB OR “peer assessment

tool” OR mini PAT OR mini-PAT

OR

“clinical evaluation exercise” OR mini-CEX OR mini CEX

OR

“case based discussion” or CbD

“Medication related consultation framework”

Limits of search: 01.01.1990 to 28.2.2011, human studies

(Educational measurement) AND (clinical OR skills OR competence OR practice OR work OR

workplace OR work-based OR work based) boolian search

Limited to subjects:

Pharmacy – students

Education, pharmaceutical pharmacy students

Students – pharmacy

Interventions – education, pharmaceutical

Curriculum – education, pharmaceutical

AND

Systematic OR review


patient exam

SEARCH 3

SEARCH 1

SEARCH 2

International Pharmaceutical Abstracts (yielded 123 results)

113

AND


direct observation AND assessment

AND




AND


Clinical evaluation exercise OR mini CEX OR mini-CEX

AND


case based discussion OR CbD

AND


medication related consultation framework

Limits of search: 01.01.1990 to 28.2.2011, human studies

SEARCH 3

SEARCH 4

SEARCH 5

SEARCH 6

SEARCH 7

CINAHL, British Nursing Index and PsycINFO (yielded 393)

114

Cinalhl major heading: (clinical assessment tools OR competency assessment) AND (educational

measurement ) AND (systematic OR review (keywords) )

(All subject Terms)educational measurement and (clinical or skills or competence or practice or work

or workplace or work-based or work based) AND (MH educational measurement)

AND

systematic OR review

AND

(subject terms) Pharmacy education OR medical Education OR nurse education OR (education AND

physical therapy OR occupational therapy OR dentist)


patient exam

AND

Major heading (educational measurement)

AND


Major heading (educational measurement)

AND

Direct observation



SEARCH 1

SEARCH 2

SEARCH 3

SEARCH 4

SEARCH 5

115

OR

Clinical evaluation exercise OR mini CEX OR mini-CEX

OR

Case based discussion OR cbd AND Education

AND

systematic OR review

116

Appendix 3 Comparison of Key Activities to be undertaken in a Medication-Related Consultation against the Medical Consultation Models

Key Activities of a Medication-Related Consultation

The Model of Patient-

centred Care

The Pendleton

Consultation Model

The Inner Consultation

The Information Exchange

Model

The Three Function

Model

The DREAM

consultation

The E4 model for physician-

patient communication

The Calgary-Cambridge

Consultation Model

The SEGUE Framework

Establish FULL Medication History No Partially met No No No No No No Partially met

Explore patients’ understanding about medicines

No No Partially met Partially met No No Partially met Partially met Partially met

Explore patients’ illness understanding Yes Yes Yes Yes Yes Yes Yes Yes Yes

Elicit and address patients’ concerns about treatment

No Partially met Partially met Partially met Partially met No Partially met Partially met Partially met

Explore patients’ adherence with prescribed treatment

No No No Partially met Yes No Yes Partially met Partially met

Negotiate medicines management plan with patient

Yes Yes Yes Yes Yes Partially met Yes Yes Yes

Check patients understanding

Yes Yes Yes Partially met Yes Partially met Yes Yes Partially met

Check patient’s ‘practical’ ability to follow treatment plan

No No No No No No No No No

Refer to other healthcare professionals (if appropriate)

Yes Yes Partially met No No No No Partially met Partially met

Key: Yes =activity undertaken, No =activity not undertaken

Source: Abdel Tawab R. PhD Thesis (2005) Development and Psychometric Validation of a Framework for Medication-related Consultations.

1.

2.

3.

4.

5.

6.

7.

8.

9.

117

Appendix 4 Foundation Programme Competencies and Assessments

Competence

1 Professionalism Assessments

2 Good clinical care

3 Recognition and management of the acutely ill patient

1.1 Behaviour in the workplace TAB, CBD, probity declaration and clinical supervisor's report

1.2 Health and handling stress and fatigue TAB, clinical supervisor’s report and health declaration

1.3 Time management and continuity of care TAB, clinical supervisor’s report, feedback form and survey receipts

2.1 Eliciting a history Mini-CEX and CBD

2.2 Examination Mini-CEX

2.3 Diagnosis and clinical decision-making Min-CEX and CBD

2.4 Safe prescribing CBD

2.5 Medical record-keeping and correspondance CBD

2.6 Safe use of medical devices Min-CEX, DOPs, log book and CBD

3.1 Promptly assesses the acutely ill or collapsed patient TAB, log book, Mini-CEX & CBD

3.2 Identifies and responds to acutely abnormal physiology TAB, log book, Mini-CEX & CBD

3.3 Where appropriate, delivers a fluid challenge safely to an acutely ill patient TAB, log book, Mini-CEX & CBD

3.4 Reassesses ill patients appropriately after starting treatment TAB, log book, Mini-CEX & CBD

3.5 Undertakes a further patient review to establish a differential diagnosis TAB, log book, Mini-CEX & CBD

3.6 Obtains an arterial blood gas sample safely, interprets results correctly TAB, log book, Mini-CEX & CBD

3.7 Manages patients with impaired consciousness, including convulsions TAB, log book, Mini-CEX & CBD

3.8 Uses common analgesic drugs safely and effectively TAB, log book, Mini-CEX & CBD

3.9Understands and applies the principles of managing a patient with acute mental disorder including self harm TAB, log book, Mini-CEX & CBD

118

3.1Ensures safe continuing care of patients on handover between shifts, on call staff or with ‘hospital at night’ team by meticulous attention to detail and reflection on performance TAB, log book, Mini-CEX & CBD

4.1 Resuscitation TAB, CBD and ILS/ALS/equivalent course

4.2 Discusses Do Not Attempt Resuscitation (DNAR) orders/advance directives appropriately TAB and CBD

5.1 Discharge planning CBD and TAB

5.2 Chronic disease management CBD and TAB

6.1 Within a consultation Mini-CEX, DOPS and TAB

6.2 Breaking bad news CBD and TAB

7.1 Treats the patients as the centre of care CBD and TAB

7.2 Makes patient safety a priority in own clinical practice CBD and TAB

7.3 Promotes patient safety through good team working CBD and TAB

7.4 Understands the principles of quality and safety improvement CBD and TAB

7.5 Complaints TAB and clinical supervisor’s report

8.1 Infection Control TAB, logbook and DOPS

9.1 Nutritional care TAB, CBD and mini-CEX

4 Resuscitation

5 Discharge and planning for chronic disease management

6 Relationship with patients and communication skills

7 Patient safety within clinical governance

8 Infection Control

9 Nutritional care

119

10 Health promotion, patient education and public health

Ethical and legal issues

12 Maintaining good medical practice

13 Teaching and Training

14 Working with colleagues

10.1 Educating patients Mini-CEX, CBD and TAB

10.2 Environmental, biological and lifestyle risk factors Mini-CEX, CBD and TAB

10.3 Smoking Mini-CEX, CBD and TAB

10.4 Alcohol Mini-CEX, CBD and TAB

10.5 Epidemiology and screening Mini-CEX, CBD and TAB

11

11.1 Medical ethical principles and confidentiality CBD, TAB and clinical supervisor’s report

11.2 Valid consent Mini-CEX, CBD and DOPS

11.3 Legal framework of medical practice CBD and clinical supervisor’s report

11.4 Relevance of outside bodies Reflective reports, probity and health declarations, CBD

12.1 Lifelong learning CBD and TAB

12.2 Research, evidence, guidelines and care protocols TAB, CBD and mini-CEX

12.3 Audit Audit project review and e-portfolio

13.1 Teaching and Training Developing the clinical teacher

14.1 Communication with colleagues and teamwork for patient safety TAB and CBD

14.2 Interface with different specialties and with other professionals TAB and CBD

120

A1.1Behave in a manner consistent with membership of the profession y y

y yy

A1.2 Manage your time effectively y y yy y

A1.3Recognise your personal and professional limitations and refer appropriately y y y

y y y

A1.4Respond with willingness and flexibility to new situations and change y y y

y

A1.5Remain composed and personally effective in all situations y y y

y y y

A1.6Make decisions which demonstrate clear and logical thought y y y

y y y

A1.7Take responsibility for and accept outcomes of your own decisions y y y

y y

A1.8Amend your behaviour, when necessary, based on evaluation of your performance by yourself or others. y y

y

A2.1 Carry out tasks effectively y y yy y

A2.2

Approach tasks and situations in accordance with the law and with the Regulator's Standards of conduct, ethics and performance y y

y y

A2.3 Follow work systems correctly y yy

A2.4 Use resources effectively y y yy

Appendix 5 Assessment Tools that Could be Used to Assess the Pharmacy Pre-registration Performance Standards

Performance StandardAssessment

Number Description MSF mini-CEX DOPS CbD OSCE

MRCF Portfolio Standardised Patients Patient

Surveys

Unit A - Personal Effectiveness

A1 Manage yourself

A2 Manage work

121

y

A3.1 Recognise and define actual or potential problems y y yy y y

y

A3.2 Identify workable options to resolve the problem y y yy y y

y

A3.3Select the best solution, based on sound analysis and appropriate evidence y y y

y y y

A3.4Suggest and if appropriate implement solutions to problems y y y

y y yy

A3.5Evaluate the outcomes of the solution after implementation and if necessary redefine the problem y y

y y

A4.1Work to an acceptable standard when preparing products and delivering services y y y

y

A4.2 Check your own work effectively y y yy

A4.3 Minimise error by others through effective supervision y yy

A4.4Identify and rectify your own and others' mistakes promptly and effectively y y

y

A4.5 Minimise health and safety risks to yourself and others y yy

A4.6 Base your actions, advice and decisions on evidence y yy y

A4.7 Obtain and process the evidence you need to meet A4.6 yy

A4.8Have successfully carried out a small planned audit assignment

y

MSF mini-CEX DOPS CbD OSCE


Surveys

A3 Manage Problems

A4 Demonstrate a commitment to quality

122

A5.1Identify and prioritise your own learning and development needs y y

y

A5.2Develop your own plans to meeting identified needs using SMART learning objectives y

y

A5.3 Make full use of learning and development opportunities yy

A5.4 Evaluate whether your learning objectives have been met yy

A5.5 Identify your further learning needs yy

A5.6Record your own learning and development process and outcomes y

y

A5.7 Apply learning to practise y yy

B1.1 Communicate effectively in English y y yy y y

y

B1.2 Behave in a polite and helpful manner y y yy y y

y

B1.3Sensitively approach people who need or may need assistance y y y

y y yy

B1.4Elicit all relevant information by the use of appropriate questions y y y

y y yy

B1.5 Listen effectively to the whole message y y yy y y

y

B1.6 Respect and observe confidentiality y yy y

y

B1.7Act appropriately in response to spoken and unspoken needs of others y y

y y yy

B1.8 Behave in a manner which instils confidence y y yy y y

y

B1.9 Behave assertively y y yy

A5 Demonstrate ongoing learning and development

Unit B - Interpersonal Skills

B1 Communicate effectively



Surveys

123

B1.10 Use appropriate body language y y yy y y

y

B1.11Provide information and advice appropriate to the needs of recipients y y y

y y yy

B1.12 Handle conflict appropriately y yy

y y

B2.1Acknowledge the ideas and opinions of others and act on them when appropriate y

y

B2.2Present your own ideas and opinions appropriately when speaking and in writing y y

y

B2.3Meet commitments made to others within agreed deadlines y

y

B2.4Give constructive feedback to others based on accurate evaluation of their performance y

y

B2.5Secure help from others when necessary in an appropriate manner y

y

B2.6 Assist others when necessary y yy

B2.7 Delegate tasks appropriately yy

B2.8Supervise others in an appropriate manner to ensure that agreed outcomes are achieved y

y

B2.9Use your knowledge and skills effectively when helping others learn y y

y

C1.1 Correctly receives prescriptions into the pharmacy y yy

C1.2 Check the prescription is valid y yy y



Surveys

B2 Work effectively with others

Unit C - Medicines and Health

C1 Manage the dispensing process

124

C1.3Assess the prescription for safety and clinical appropriateness y y y

y y

C1.4 Resolve any identified problems appropriately y y yy y

C1.5 Perform calculations correctly y y yy y

C1.6 Assemble the prescription correctly y yy

C1.7Supply extemporaneously prepared products according to the correct formula y y

y

C1.8

Correctly issue dispensed items to patient or representative, with appropriate information and advice y y

y y

y

C1.9 Ensure stock is managed correctly y yy

C1.10Respond appropriately to requests to dispense prescription only items without a prescription y y y

y y

C1.11 Correctly process necessary documentation y y yy

C1.12 Effectively check prescriptions dispensed by others y y yy

C2.1Provide considered and correct answers to queries, founded on research-based evidence y y y

y y y

C2.2Pro-actively assist patients to obtain maximum benefit from their treatment y y y

y y yy

C2.3Identify and take action to minimise risk to patients from their treatment y y y

y y yy

C2.4Actively provide information and advice to healthcare professionals y y y

y

C2.5Construct medication histories using a range of sources y y y

y y y

C2.6 Use medication histories correctly y y yy y y



Surveys

C2Provide additional clinical and pharmaceutical services

125

C2.7Recognise possible adverse drug reactions, evaluate risks and take action accordingly y y y

y y y

C2.8Provide appropriate information and advice on the management of minor and common ailments y y y

y y y

y

C2.9Effectively use opportunities to promote and support healthy lifestyles and prevent disease y y y

y y yy

C2.10 Demonstrate awareness of emergency first aid y y yy

C2.11Refer or direct the person to a more suitable source of help or information when necessary y y y

y y yy



Surveys

Documents

The Assessment of Observed Practice: A Literature … Assessment of Observed Practice: A Literature Review Dr Elizabeth Mills Professor Alison Blenkinsopp Professor Robert K McKinley