View
213
Download
0
Tags:
Embed Size (px)
Citation preview
Keele University School of Medicine
Reliability of the Leicester Clinical procedure Assessment Tool (LCAT), a
tool to support holistic generic assessment of clinical procedure skills
RK McKinley, J Strand, L Schuwirth
T Gray, T Alun-Jones, H Miller
Keele University School of Medicine
Background
• Assessment– Repertoire of competence
• Micro-certification
– Changing roles• Skills extension• Skills cascade
– Volume– Implications for assessment
Keele University School of Medicine
Instrument specification• Generic• Holistic• Multi-professional • Multi-level• Multi-modal• Enable high quality assessment:
– reliable, valid, acceptable, feasible, educational impact
• Nationally accredited
Keele University School of Medicine
Methods
• Multi stage:– Systematic review– Qualitative development– National validation– Psychometrics– Implementation
Keele University School of Medicine
Stage 1: Systematic review• LCAT v1.7
Keele University School of Medicine
Stage 2: Qualitative
• Focus groups
• Observed assessments, debrief interviews
• Trial ‘hi-fi’ OSCE
• LCAT v2.5
Keele University School of Medicine
Stage 3: Delphi
• National sampling frame– Recruitment– Questionnaire
• Definition of consensus• Agreement
– Categories: All >95%– Components: All >89%
• LCAT v3.0
Keele University School of Medicine
LCAT v3.0
Category– Communication and working with
the patient– Safety – Infection Prevention– Procedural Competence– Team working
Components9
7
6
12
4
Keele University School of Medicine
Stage 4: Psychometrics
• Hi-fi OSCE– 8 stations (2X3 tracks)
• Prosthetics + simulator• Venepuncture, venous cannulation, IV drugs,
ABG, ♀&♂ urinary catheterisation, skin suture, ECG
• ‘Prep station’• 15 minutes + 5 feedback
Keele University School of Medicine
Stage 4: Psychometrics
• Analysis– Used data from 1 assessor per station– Assessors not systematically assigned to
stations– Variable group of assessors– Unbalanced design
Keele University School of Medicine
Stage 4: Psychometrics
• Analysis– P (candidate)– S (stations)– A:(PS) assessors within candidates x
stations– PS,e interaction candidates and stations +
error
Keele University School of Medicine
Stage 4: Psychometrics
• 46 candidates
• 50 assessors– 19 medical – 27 nurses– 1 midwife – 3 HCA
Keele University School of Medicine
Stage 4: Psychometrics
• Results: variance
Effect Df Variancecomponent
% of totalvariance
P (candidates) 41 2.28 24.1
S (stations) 7 0.98 10.3
PS,e (candidate x stations plus general error)
287 3.61 38.1
A:PS (assessors within candidates x stations)
149 2.59 27.3
Keele University School of Medicine
Stage 4: Psychometrics
• Results: D analysis
N stations
N Assessors
1 2 3
6 0.66 0.70 0.71
7 0.69 0.73 0.75
8 0.72 0.76 0.77
9 0.74 0.78 0.79
10 0.76 0.79 0.81
12 0.79 0.82 0.83
15 0.83 0.85 0.86
0.76
0.790.79
Keele University School of Medicine
Summary• Generic• Holistic• Multi-professional • Multi-level• Multi-modal• Enable high quality assessment:
– reliable, valid, acceptable, feasible, educational impact
• Nationally accredited
Keele University School of Medicine
Summary• Generic• Holistic• Multi-professional • Multi-level• Multi-modal• Quality:
– reliable, valid, acceptable, feasible, educational impact
• Nationally accredited
Keele University School of Medicine
Acknowledgements• Analysis:
– Ron Hoogenboom, Cees van der Vleuten and Arno Muijtjens
• Colleagues: – Linda Ward, Clinical Librarian UHL – University Hospitals of Leicester – Eastern Leicester PCT – Students and staff
• University of Leicester • De Montfort University
• Funding: – Nation Health Service University – Leicestershire, Northamptonshire and Rutland WDC
Keele University School of Medicine
Keele University School of Medicine