View
215
Download
1
Category
Tags:
Preview:
Citation preview
Innovation in a traditional world-In-training assessment in PGMELord Cohen lecture, ASME 2010
Professor Charlotte Ringsted, MD, MHPE, PhD
Director of Centre for Clinical Education
University of Copenhagen and Capital Region DK, Rigshospitalet
Innovation in a traditional world
Innovation– Outcome-based education and in-training assessment in
Postgraduate Medical Education (PGME)
Traditional world– Undergraduate Medical Education (UGME)– End-of-training assessments and exams
In-training assessment programme– Anaesthesiology in DK– Internal medicine and Child and Adolescence psychiatry
Conclusion– Lessons learned and future directions
Danish Medical Education
Undergraduate• Bachelor - 3 years• Candidate - 3 years
Postgraduate– Internship 1½ year, Foundation 1 year
– Specialist education• Introduction 1 year• Specialist Residency 4-5 years
Continuous prof. development– Expert educations 1-2 years– Various
BasicScience
ClinicalScience
Intern
Intro-year
Specialistresidency
CPD
Expert ed.
Innovation: OBE and ITA
PGME 1991 NBH rules, guidelines Goals and objectives
– Specialist societies
Speciality courses Clinical programmes Training posts CRE and supervisor Appraisal meetings (3) Trainees’ evaluation No exams
PGME reform 2001 NBH rules, guidelines Goals and objectives
– CanMEDS framework
Plus ’general courses’* Clinical programmes Training posts CRE and supervisor Appraisal meetings (3) Trainees’ evaluation In-training assessment
Challenge - The seven roles
EFPO project, 1992– Undergraduate education,
Ontario, society’s needs, eight roles
Manager
CanMEDs project, 2000– Postgraduate education,
RCPSC, entire Canada, seven roles
Medical expert
Scholar
Communicator
Collaborator
Professional
Whole person
Health advocate
Canada and Denmark – Red and white; Neighbours; Hans Island
3 personsper km2
125 personsper km2
No 3
Validity of CanMEDS roles
Survey among doctors in East DK– Responses from
3072 doctors
Roles important Increasing
confidence
Different specialitiesdifferent profiles
Ringsted et al. Med Educ 2006
Importance
Confindence*
CanMEDS roles Competence Teaching strategies Assessment
Med. exspert
Communicator
Health adv.
Collaborator
Manager.
Scholoar
Professional
Some challenge!
Intro-year Specialist Residency
What, how, when, who?
Internship
What is competence?
Competence is a habit of action
Competence = holistic overall capacity– ”The habitual and judicious use of communication,
knowledge, technical skills, clinical reasoning, emotions, values, and reflection in daily practice for the benefit of the individual and the community being served” Epstein and Hundert JAMA 2002
Competency = specific capability– ”Focuses on performance of the goal-state of instruction;
reflects expectations that are external to the programme; expressible in measurable behaviour; uses criterion standards for judging; informs learners and others about expectations” Albanese ME 2008
1. Challenge: Defining outcomes
Describing competence– Too detailed – Be able to manage - Lists of
procedural skills, diseases (+300 competencies)– Too general - The ‘other’ six roles - difficult to
define
Disintegration of the concept ‘competence’– Seven disciplines rather than an integrated,
context-based concept of competence
Expectations at various levels of training?– Different specialties, different traditions?
Various levels - AnaesthesiologyDreyfus, Epstein & Hundert
PERSONAdv. beginner
Novice
CONTEXTSmall teams
Close supervision
TASKSimpleSingle
1st yr
PERSONMasterExpert
CONTEXTComplex systems
Supervisor
TASKComplexAtypical
5th year
PERSONProficient
Competent
CONTEXTLarge teams
Distant supervision
TASKComplicated
Typical
2-4 year
Patients: ASA groups 1-5Surgery: Minor, medium, majorEM: Stay alive till assistance arrives
Internal medicine – levels?
About the difference between trainee levels:
”We pretty much do the same – patient encounters, ward rounds, ambulatory, etc. -The difference between levels is a matter of expectations - you are expected to take on a wider and deeper approach in managing the patients - and you are allowed to do more - make more decisions - simply because you are more experienced”...
Ringsted et al. Med Teach 2006
”We work in multi-professional teams – these are not stationary teams, but rather depending on the child and the situation.” – ”We learn from the team and gradually we
become more respected members of the team.– Our legitimacy is being a doctor with the right to
make certain decisions and prescribe medicine”
Child & Adolescence psychiatry
Davis et al. 2008
What is wrong
with this kid?
Conclusion - Different specialities
’Community of practice’Social task specialities
’Brick laying model’Technical task specialities
’Rings in the water’Cognitive task specialities
2. Challenge – assessment
What, when, how, who?
Beforeentry
Afterexit
Start ofrotation
End ofrotation
Duringrotation
MMI,Oral, MCQ
OSCE, Oral, MCQ
ITERABIM
Mini-CEXDOPS
Why in-training assessment?
Postgraduate education is work-based– 50% of the physician work-force are trainees– Quality of care relies on trainees’ competence
during training– ”End-of training examination is like reading
yesterday’s news”
In-training assessment, a tool for learning– Help clarify objectives according to broad aspects
of competence (CanMEDS roles)– Stimulate deep learning– Support effective and efficient education
Challenge in postgraduate education
Undergraduate education
Postgraduate education
Knows
Does
Can Can
Does
Knows
School-based
Work-based
?
• Learn how to manage cases• Learn from managing cases• Reflect in and on practice
Mastery or Development ?
Competency as capabilityrelated to specific tasks
1. 2. 3. 4. 5. 6.
Scoring
Competence as holisticcapacity related to any task•Aspects of the 7 roles
Time
987654321
No single method can measure it all – V.d .Vleuten 2010
assessment programmes are recommended
In-training-assessment programmes
Anaesthesiology
Internal Medicine
In-training assessment, Anaesthesiology Ringsted et al. Med Teach 2003
•Cusum scoring•Logbook on experience•Learning portfolio
•Communication skills (1)•Management/collaboration (2)•Academic competence (3)
1 st
year
trainingClinical skills assessments (12)
Observationin vivo / vitro
Observationin vivo / vitro
Assessment based on practice data and written reflective assignments/reports
Assessment based on practice data and written reflective assignments/reports
Longitudinalassessment
Longitudinalassessment
Assessment of written assignment
Reflection over a patient– Description of patient and operation– Theoretical and practical consideration regarding
choice of anaesthesiological approach related to patient condition, wishes, surgery, and context
– Describe potential problems and complications and discuss strategies to minimise these
– Describe actual patient course and events– Reflection related to pre-operative considerations– Use references from literature in the reflection
Why focus on theory and reflection? Klemola and Norros, ME 1997, 2001
Realistic orientation– Recognition of
uncertainty and unpredictability
– Communicative relationship: each patient is unique
Objectivistic orientation– No recognition of
uncertainty and unpredictability
– Authoritative relationship: ’a case’: coronary, asthmatic, etc
Anaesthesiology– Clinical physiology and pharmacology; Procedural skills; Monitoring
of respiratory and cardiovascular parameters; Context – patient, surgery, team
Two distinct patterns related to ‘experts’
Habit of action Klemola and Norros, ME 1997, 2001
Interpretative– Combine monitor
information with situational information and background knowledge
– Recognition of the versatility of information from several resources, oxygen SAT, End-tidal CO2, etc.
Reactive– Operate directly with
the numbers
– Contradictory use of monitors, emphasising importance regarding patient safety without understanding the mediated character and versatility of information
Knowledge and skills
Causal understanding of concepts, principles, and tool design affects retention and transfer of learning Woods et al. 2006, 2007, Schwartz 2004
Self-regulatory processes in development of expertise Zimmerman 2006
– Forethought: Task analysis, strategic planning– Performance: Contextual adaptation of strategies– Post-task: Evaluation and reflection
Bech et al. EJVS 2010
Knowledge and Anaesthsiology Klemola and Norros, ME 1997, 2001
Forethought: physiological potentials– ”He can go uphill without getting out of breath, so probably he
will tolerate anaesthesia well. Major problem might be oxygenation and ventilation.”
Adaptive strategy: physio-pharmacological experiment– ”You can’t tell how an elderly patient will react. You have to
check his responses and give drugs accordingly.”
Evaluation and reflection-in-action– ”The patient has capacity to compensate for side-effects of
anaesthesia through sympathetic activation, a kind of capacity that elderly patients do no necessarily possess. That is a safe thing to observe”
Flexner ?
Routine expert vs. Adaptive expert
”Most professionals reach a stable,average level of performanceand maintain this mediocrestatus for the rest of their careers.”
Routine expertsPerf
orm
an
ce
Experience
Ericsson 2004, Guest et al.2001, Choudhry et al. 2005, Schwartz 2004
Adaptive experts
A team and a coach
3. Challenge: Validity and reliability?
Assessment free area– Focus on evaluation of
quality of education• “To emphasize the educational
purpose of training, comprehensive formative evaluation is suggestedas alternative to specialist examinations.” Karle, Nystrup ME1995
– Strong humanist tradition in education, qualitative R&D
• Deprived of quantitative educational researchand psychometrics
– Ministry of research, innovation, and technology 2007, OECD review 2005
North America
Assessment rich area– Flooded by quantitative
data and psychometricians– National exams– Heavy focus on reliability
of tests and exams– Strong tradition of
cognitive psychology and behaviourism
• Hodges and Segouin, ME08
DK
Schuwirth & v.d. Vleuten ME 2006
A plea for new psychometric methods
In-training assessment, Anaesthesiology Ringsted et al. Med Teach 2003
•Cusum scoring•Logbook on experience•Learning portfolio
Trainees’ opinion of assessment (1-9)
15141414131315N =
10
8
6
4
2
0
610
2
143
14
115
Ringsted et al. AAS 2003
Internal medicine
Ringsted et al. Med Teach 2006; Davis et al 2005; Norgaard et al. Med Educ 2004
Intro-year trainees’ learning needs, case-mix, and quality assurance data
Structured approach to the tasks– Ward rounds– Complex patients – get an overview
Emergency care– Team-leader skills
Collaboration– Team-work and inter-personal skills
Ethics and professionalism– Difficult decisions
Usefulness of the assessment
Patient consultation (Lung) 5
Patient consultation (heart) 6
Review of primary patient record 6
Review of patient course, reflection 6.5
Audit of records (nutrition, pain, ability, temp.) 6.5
Presentation at conference 7
Ward round 7
EBM assignment 7
Presentation 7
Median (1-9)
About the written assignments
“Extremely good learning experience - to do this review of a patient’s course ”
“It was hard work” (Trainee) “This is really a valuable innovation in the
education - these assignments” (Trainee) “It was more easy than I thought - to
review these assignments” (Supervisor) “This is an advantage to the entire
department - we all learn from these..”
Kirsten Nørgaard, MHPE, 2004
Factors related to value of ITA Ringsted et al. ME 2004, Med Teach 2003, ASS 2003
The link to practice– Help in structuring teaching, training and learning
• Outcomes clear, monitoring progress, identify problems• Coupling of theory to practice
– Used as licence to practice rather than end-of-training assessment
The effect on learning– Should include a challenge to the learner– ‘We all learn more’
Assessors’ attitudes– Enthusiasm and rigour
Conclusion
Lessons learned
Future directions
Lessons learned
No 1 – Outcome-based education– ‘CanMEDs roles’ is a nice mental framework.
Need for both competency-goals (outcome/ efficiency) and competence-goals (process/ innovation)
No 2 – In-training assessment– A valuable tool for learning. Meaningful
programmes are tailored to clinical context and trainees’ level of professional development.
No 3 – The process– Useful to take a design-based research approach:
Cycle of critical review of literature, design, enactment, evaluation, and large working groups
Future direction
Some questions– Can deliberate practice, reflection in and on
practice, adaptive expertise be learned? Or is it an in-born trait in the minority of us?
– If it can be learned, how can we facilitate the development and measure the progress of it?
– How can we better align education, training, and learning with quality of practice and measure the effect?
– How can we turn education into a resource by using the capacity of our trainees to develop practice as a whole?
Time’s up
In the honour of Lord Cohen
Thank you for your attention
??
Recommended