Evaluating the impact of assessing doctors’

performanceDr Julian Archer

NIHR Career Development Fellow& Clinical Senior Lecturer in Medical Education,Director of CAMERA, Consultant Paediatrician

Evaluating the impact of assessing doctors’ performance

Outline Why evaluate impact of assessment? Programmatic research into medical

revalidationSome earlier work

GMC revalidation evaluation framework

Why evaluate impact?

APA Validity framework – Five headings

Content Response process Internal structure Relationship to other

variables Consequences

Medical education Predominately all

validity areas except consequences

Why? Consequences are

Complex

Socio-political

Not positivistic

EVALUATING IMPACT OR ‘CONSEQUENCES’

Programmatic approach to evaluation

What is Revalidation?

What is the meaning of Revalidation?

The Literature

Policy makers

What are the intended and unintended consequences of Revalidation in practice?

Ward/Theatre/Lab

GP

Appraisees &Appraisers

Focus groups

•To provide an understanding of Revalidation translational to policy, education and practice

•To shape and propose positive consequences while minimising negative consequences of Revalidation•To disseminate positive responses about Revalidation in order to shape future national policy

Stage 1 - Policy Stage 2 - Practice Stage 3 - Public

Private Practice

Research Questions

Setting

MethodSystematic

ReviewInterviewee

drawings

Methodology

What is the impact of Revalidation on the relationship between the profession and society?

Patients

Journalist InterviewsAnalysis of

Media

Outputs

Visual anthropology

Mental Health

Patient Focus

Groups

Video appraisals

Appraisee Interviews

Media

Activity TheoryConversation

Analysis

•Ascertain the messages that the public are getting from the media•Develop communication about Revalidation more cost effectively

Discourse analysis

STAGE 1

What is Revalidation in policy? - To explore the origins, definitions, and potential purpose of Revalidation

Methods

Policy review starting from the Merrison report (1975)

31 unique elite interviews leading members of Revalidation policy

development ranged from the leaders past and present of main stakeholder bodies such as the GMC, AoMRC, BMA, NHS Employers, 4 DHs

Discourse

PROFESSIONALISMDriven by a professional

movement for reformInternally motivated

Evolutionary DevelopmentThe Quality Agenda: for patientsRestore/maintain confidence by continuing to elevate standards

AppraisalFormative Development

Ongoing evaluation (process)Up-to-date

REGULATION Driven by medical scandal

Externally motivatedRevolutionary change requiredThe Safety Agenda: for patients

Reassurance by measuring against a fixed standard

Clinical GovernanceSummative Judgment

Point-in-time Decision (product)Fit-to-practise

PATIENTS

DEVELOPING EVALUATION FRAMEWORKS

Working with the GMC

Methods – evaluative cycle

Literature

Interviewed major stakeholders

Worked with GMC (Main stakeholder)

Develop evaluative questions

Develop possible methods for an initial pilot

Step 1: Engage stakeholders

Step 2: Describe the programme

Step 3: Focus the evaluation question

Step 4: Gather credible evidence

Hansen HF. Choosing Evaluation Models: A Discussion on Evaluation Design. Evaluation. 2005;11:447-462.

Stakeholder interview themesImportance of evaluation

“people who go through the process…without any thought of what it actually represents and how it might actually add value to the system”

What is revalidation for? Drive-up standards (performance improvement) vs.

picking out ‘bad apples’

Appraisal “to get inside the closed door of appraisal”

RO judgements “I’m not sure exactly how the RO’s decision is made”

SummaryImpact or consequences of assessment is an important part of validity evidence

Complex

Programmatic research required (drawing on theoretical perspectives)

Acknowledgements and thanks

Colleagues Sam Regan de Bere CAMERA team

Funders NIHR The Health Foundation General Medical

Council

Evaluating the impact of assessing doctors’ performance

julian.archer@plymouth.ac.ukcamera.pupsmd@plymouth.ac.uk

www.plymouth.ac.uk/peninsula/research/camera

Twitter: @CAMERAPUPSMD