Assessing ProfessionalismAssessing ProfessionalismCongreso Nacional de Educación Médica

National Congress on Medical Education

Puebla, Mexico

Dale Dauphinee, MD, FRCPC, FCAHS13 January 2007

Outline of TalkOutline of Talk• Principles: assessment cycle in medicine• What do we mean by professionalism?

– Two roles of the MD as a professional• Educational implications re professionalism• Need to define the attributes• Linking testing to the course of study• Where to now?• Concluding comments

Classic Assessment CycleClassic Assessment Cycle

Desired Objectives Desired Objectives or Attributesor Attributes

Educational Educational ProgramProgram

Assessment of Assessment of PerformancePerformance

Performance GapsPerformance Gaps Program RevisionsProgram Revisions

Implementation of Program: Implementation of Program: CQI* Principles in Action CQI* Principles in Action

PurposePurpose

CommitmentCommitment

CapabilityCapability

Monitoring Monitoring & Learning& Learning

ActionAction

* CQI = Continuous Quality Improvement* CQI = Continuous Quality Improvement

Professional Career Sequence:Professional Career Sequence:From Undergraduate to Practice

Admission Licensing Exam PostgraduateTraining

Medical SchoolEducation

Practice

Certifying Exam

Three areas of clinical activity where performance

assessment is needed

Change in the Hallmarks of Change in the Hallmarks of Competence ….Competence ….

Knowledgeassessment

Problem-solvingassessment

Clinical skillsassessment

Practiceassessment

Professional or clinicalProfessional or clinicalAuthenticityAuthenticity

1960 2000

(adapted from van der Vleuten 2000)

Climbing the PyramidClimbing the Pyramid

Knows

Shows how

Knows how

Does

Knows Factual tests:MCQ, essay type, oral…..

Knows how (Clinical) Context based tests:MCQ, essay type, oral…..

Shows how Performance assessment in vitro:OSCE, SP-based test…..

DoesPerformance assessment in vivo:Undercover SPs, Video, Logs…..

What Is Missing?What Is Missing?

Extending the PyramidExtending the Pyramid

Knows

Shows how

Knows how

Does

Knows

Knows how

Shows how

Does

Assessing “Meta” skillslike Professionalism

after van der Vleuten

Defining Professionalism Defining Professionalism and Its Attributesand Its Attributes

Starting Point

Assessing Professionalism Assessing Professionalism Build Upon Foundations & Principles (Pillars)Build Upon Foundations & Principles (Pillars)

• Professionalism is demonstrated through afoundation of clinical competence, communication skills, and ethical and legal understanding, upon which is built the aspiration to and wise application of the principles of professionalism: excellence, humanism, accountability and altruism

• After Stein and Arnold in Measuring Medical Professionalism (2006)

Professionalism

Ethical and Legal Understanding

Communication Skills

Clinical Competence (Knowledge of Medicine)

Exc

elle

nce

Hum

anis

m

Acc

ount

abili

ty

Altr

uism

Black = Professionalism

Red = Principles

Green = Foundation

Cannot Measure AttributesCannot Measure Attributes• Physician attributes -

some examples:– Ethical– Altruistic– Understands legal

principles– Reliable– Patient centered– Etc.

• Can measure knowledge of and/or intent to do….

• But how to measure what we do?

• Must Seek Behaviours• Consider 7 of the 21

behaviours on the McGill mini-exam with a patient…– Listened actively to patient– Recognized and met

patient’s needs– Ensure continuity of care– Demonstrated awareness of

own limitations– Solicited feedback– Maintain composure in a

difficult situation– Was available to colleagues– Demonstrated respect fro all

colleagues

Before we will return to how to Before we will return to how to measure professionalism…measure professionalism…

….. Must consider educational implications of stressing or

focusing on professionalism

Teaching Professionalism: Teaching Professionalism: Becoming a PhysicianBecoming a Physician

• Teaching as Professional, in the Service of Healing: as per Cruess*

• Framework:– Healer: as a professional– What is a professional– What’s in the literature:

• Sociology• Ethics

– Implications– Examples

• Stages:**– Setting expectations

• ‘White’ coat ceremonies• Orientation

– Providing experiences• Formal curriculum• Ethics courses• Role models• Case based learning• Community based learning

– Evaluating outcomes• Assessment prior entry to

faculty• Assessment by

– Peers– Faculty – patients

**See Stern and Papadakis (2006); NEJM 355: 1794-9

* See Cruess R and Cruess S. Academic Medicine (1997); 72: 941-52

To Begin to Assess Professionalism: To Begin to Assess Professionalism: Consider Levels of ImpactConsider Levels of Impact II

• Define: what is professionalism– differentiate from ‘disruptive’ physicians

• Define two roles of the physician: as professional & as healer– as per Drs. Richard and Sylvia Cruess

• Several layers if implement assessing professionalism– in a medical faculty, or – for a physician licensing program:

• e.g. - MCC experience in 2006

To Begin to Assess Professionalism: To Begin to Assess Professionalism: Consider Levels of ImpactConsider Levels of Impact IIII

• Consider three of many layers: – As a long-term MCC project: how to assess

• In students • In residents• In faculty who must live and demonstrate professionalism

– Current plan: revise C2LEO Objectives + Professionalism– A research and development project: how to do it

• Build on what exists internationally– e.g. CanMEDS roles (Canada) – or ACGME core competencies (USA)

• Educational plan: preparing students and trainees

Professionalism at the MCC: Professionalism at the MCC: Implementation Plan ReportImplementation Plan Report

Plan to Enhance the Assessment of Professionalism in the MCC

Qualification Process

Professionalism ObjectivesProfessionalism Objectives• Step #1: Identify behaviours (by 31 Jan 2007)• Step #2: Do gap analysis against

C2LEO objectives (March 2007)• Step #3: Initiate immediate implementation in

the MCCQE Part II for May 2007• Step #4: Revise C2LEO Objectives (2007)• Step #5: Survey Canadian medical schools on

the status of professionalism education• Step #6: Develop plan for new instrumentation

and questions based on gap analysis (2007-8)

• Step #7: Create integrated plan with faculties (start in 2007 – to complete by 2009)

SummarySummary• Plan outlined.• Two major developments: one short and one

long term– An internal immediate initiative around the MCCQE

Part II for use in May 2007 exam– More and longer term, create an integrated joint

process with the faculties and medical regulating authorities to influence the educational program

• Use existing frameworks: 7 CanMEDS roles– e.g. MD as expert or communicator or educator….

• Plus must define content: at MCC will use outcome objectives or competencies by level of training (see www.mcc.ca)

Maintaining ObjectivesMaintaining Objectives

MCC Approach

How Is Content Validated?How Is Content Validated?• Feedback from Oversight Committees• Feedback from

– Results analysis– Test Committees

• Corrections for omissions & changes– e.g. Change in policy or laws

• External review process: every three years– Oversight groups– University and regulatory communities– Practicing physicians: generalists + core groups

• Reviewed and edited by Objectives Committee and Co-editors

How Will Typical Cycle Work?How Will Typical Cycle Work?Based on a fiveBased on a five--year MCC Cycleyear MCC Cycle

• Each year feedback– from oversight groups– from Test Committees

• Every 5 years:– Complete REVISIONS

• Major revision:– External reviews

• Three sources

• Editor and committee: revise validate* revise

• *Use focus groups with practicing MDs

Feedback from Oversight Committee

Administer Assessments

e.g. Examinations

Feedback from Test Committees

MCC cycle

What Is CWhat Is C22LEO?LEO?

CCultural, CCommunication, LLegal, EEthical and OOrganizational Aspects of

Practice<www.mcc.ca>

How Does MCC Create CHow Does MCC Create C22LEO?LEO?• Test Committees for MCQs and OSCE:

– LEO and Communication material existed• Need develop new material for Culturally

sensitive issues• Pre-test all material

– First phase: externally– Second phase: on examination as pilot items

• i.e. scored but not counted data informs revisions

• Review all of above by three groups:– Test Committee; Central Examination Committee;

and Psychometric Consultant (think ‘auditor’)

Institutionalizing CInstitutionalizing C22LEOLEO• Using nationally recognized framework• Extending it to all MCC Objectives

– Means all faculties will use for graduating students• Involving MCC intra-mural expertise

– Ensures on-going quality• Sought MCC support: maintain and update

– MCC takes the fiscal risk: serves as basis of its QEs– Locate the staffing in the Evaluation Bureau at MCC

• Place C2LEO in public domain• Complementary to RCPSC & CFPC framework

Overall: Where Are We Overall: Where Are We in Canadain Canada

Scan of overall approach – independent of the plans of various national bodies to

assess professionalism

What Has Been Established What Has Been Established in Canada?in Canada?

• In-training evaluation: standardized format• Use of OSCE – at all levels!• Insertion of OSCE into licensure exit evaluation:

documented impact long term• Legal, Ethical and Organizational aspects of

practice: now into exit examinations• Key features: important - long term effects• Move to PEER review and practice based

assessment –including behaviours• Description of behaviours to be observed

What about the Bigger What about the Bigger Picture?Picture?

Impact on students and residents and educational program?

Need a plan!!!

Goal: DeGoal: De--OrthogonalizingOrthogonalizingMedical EducationMedical Education

Goal: seek integrative processes Goal: seek integrative processes -- think think obliquely (dependence on dropobliquely (dependence on drop--in sessions or in sessions or guest speakers are guest speakers are ‘‘outout’’ of style and of style and favourfavour!)!)

OrthogonalizationOrthogonalizationcomes in many comes in many

forms forms –– and implies and implies independence!independence!

After Dan After Dan FedermanFederman: 2003: 2003

Program or Program or CurriculumCurriculum

An Alternative ViewAn Alternative View

Curriculum

Teacher

Assessment

Student

After van der Vleutin - 1999

Continuum of DevelopmentContinuum of Development• Assume practice-based model of

‘curriculum’• Assume lifelong learning & accountability

Practice Demands

Learning

AssessmentPractice Demands,

Learning

& Assessment

Reality Goal: Concordance

Whose Experience Whose Experience Can Help Us?Can Help Us?

No need to re-invent the wheel!!

And Where Are the Precedents? And Where Are the Precedents? ….. Building the Case.….. Building the Case.

• Viewing medical education as a continuum– NBME Report -1973

• Focus on ‘competence’ and performance– George Miller

• Move to professionalism: identifying roles and what it is– Two roles of Dick and Sylvia Cruess

• ‘Good medical practice’– GMC in the UK: identifying behaviours to teach and assess

• Use of ‘LEO’ in Canada – now ‘C2LEO’ - with objectives– CMQ and then MCC

• Use CanMEDS roles plus competencies or objectives

Issues to Clarify: Issues to Clarify: Data from USA and Canada.Data from USA and Canada.

• Define professionalism: in terms of behaviours• More than the disruptive physician, and …

– USA: following MDs forward in time (Papadakis et al)– Canada: same story – now seeing other predictors– KEY: notion of identifying behaviours much earlier …

• McMaster’s program re admissions• Must link to expected physician behaviour• Much work to do – CQI – studies…..

– Learn and improve: seek clarification - not justification

Emerging Trends …Emerging Trends …Being Considered by MCCBeing Considered by MCC

• More assessment methods that support integration across the educational continuum– Enhance realism

• Simulation

– Support clinical education • Mini-CEX, Demonstration of procedures (DOPs), Case-

based discussion (CbD)

– Assess doctors at work• Outcomes and process

– Assess all competencies• Professionalism

MultiMulti--source Feedback: source Feedback: ABIM (USA) & CPSA (Canada)ABIM (USA) & CPSA (Canada)

• Process– Doctor

• Nominates peers and patients

• Self-rates – Assessors provide

evaluation– Doctor given self-ratings,

peer ratings, patient ratings and national mean ratings

– Quality improvement plan developed

Approaches to ConsiderApproaches to Consider• Old friends

– MCQs– Key-features or clinical decision-making– OSCEs– In-training reports – BUT – must be immediate

• Newer friends– Mini-CEX– Case Based Discussions– Multi-source feedback (360 degree)

• And then --- measure outcomes

Competency Based CertificationCompetency Based CertificationABMS Competencies vs. ABIM Approach

Medical knowledge

Professionalism

Communication & Interpersonal Skills

Patient Care (Skills)

System-based Practice

Practice-based Learning & Improvement

Secure examination

Faculty ratings

Mini-CEX

Procedure logs

Practice-improvement

Modules

Secure examination

Peer & Patient ratings of Skills

SEPS

License/Current Staff Appointments

Practice-improvement

Modules

Competencies Certification Main. of Competence(CPD)

Professional and Quality is Professional and Quality is Not New ……Not New ……

Witness: the Dutch professional drapers’ syndics in the 17th

Century ……..

What Is in a Picture?What Is in a Picture?

The syndics of the drapers’ guild – Rembrandt - Rijksmuseum

Muchas Gracias!

Thank you!Thank you!

Merci!Merci!

Basic Readings on ProfessionalismBasic Readings on Professionalism• Overviews

– Barondess JA• AMA Arch Intr. Med 2003;

163:145-151.– Brennan T et al.

• Ann Intr. Med. 2002;136: 243-246

– Cruess RL & SR.• Acad. Med. 1997;72:941-

952– Cruess RL & SR

• Teaching & Learning in Med. 2004;16:74-76

– Papadakis MA et al• NEJM: 2005;253:2673-82

– Stern DT, Papadakis M• NEJM: 2006; 355:1794-99

• Evaluating/Assessing– Arnold L.

• Acad. Med 2002;77:502-515– Epstein R, Hundert EM

• JAMA. 2002;287:226-235– Ginsburg S et al.

• Acad. Med. 2000;75:S6-11– Gauger PG et al

• Am. J Surgery. 2005; 189:479-487

– Papadakis M et al• Acad. Med. 1999;74:980-90

– Papadakis M et al.• J. Med.Licensure &

Discipline. 2006; 92:11-19

Q/A and CQIQ/A and CQI• CQI:

– Applied via same groups against pre-defined gold standards of performance

• Pre-testing and review• Piloting and review• Actual use: will discard and refer for revision

• Q/A– Test Committees: subject and CEC– In-house staff (all MScs or PhDs)– External consultant: regular

• Time to time – external consultants• Always accountable to MCC annually

EBM and Complex Real WorldEBM and Complex Real World• Applying ‘hard’ evidence to practice

setting• Other elements entering medical

decisions:Patient-MD

FactorsEvidence

Constraints

Ethics

Guidelines

Clinical Decisions

Knowledge

Point:Evidence is

only one element in a

complex set of relationships

Per Davidoff

Humanism - Humanistic• Noun*: devotion to human interests; system concerned

with human matters (not divine or supernatural), or with the human race or with man as a responsible and progressive intellectual being.– Or literary culture: Renaissance humanists– Or doctrine: emphasizing importance of common

human needs and abstention from profitless theorizing.

• Adjective*: from French – humaniste – pertaining to humanism or being humanitarian - advocates or practices humane action; seek to promote human welfare; or holding the views of same.

* Ref. Oxford Dictionary: New Edition

How ‘Standards’ Are MaintainedHow ‘Standards’ Are Maintained• Done at three levels

– Procedural standards• In-house• AERA-APA

• Quality assurance– Test Committees– Central Examination Committee– In-house expert– Consultant

• Establishing pass-marks* and scoring system*

* Note: different competency levels and thus different standard of performance for Part I (students) & Part II (residents)

Outcomes: ChallengesOutcomes: Challenges• Attribution

– Good assessment requires that the doctor be solely responsible for the patient’s outcomes

– Patient care is increasingly provided in systems by teams supported with guidelines

• Complexity– Good assessment requires that all doctors face the

same challenge– Patients with the same condition vary in severity,

comorbidities, compliance, etc.

ProcessesProcesses• For immediate action

– MCCQE Part II– Notice given in syllabus– Need criteria for aberant

behaviour• Use professionalism

behaviours• Create global judgments

– Implementation• Train examiners• CEC to validate in Dec. 2006• Utilize in May 2007 MCCQE

Part II

• For development– Committee to develop/identify

professionalism behaviours– Staff to do gap analysis

against C2LEO– Revise C2LEO against the

gap analysis by the Committee on Objectives

– Then have MCC staff assist in creating joint plan to develop new assessment methods, involving the MRAs and universities

– Implement a major stage new approach around the MCCQE process in 2009.

Traditional ViewTraditional View

Curriculum

Teacher

Assessment

Student

After van der Vleutin - 1999

Assessment Dictates LearningAssessment Dictates Learning

Practice

CPD

Assessment

Physician

After van der Vleutin - 1999

MiniMini--CEXCEX• Process

– Assessor observes a trainee with a patient

– Trainee performs a focused clinical task

– Assessor rates history taking, physical exam skills, communication, clinical judgement, professionalism, organization/effectiveness and provides feedback

• Takes 15-20 minutes• 6 assessments/year

Case Based Discussions: Case Based Discussions: CbDCbD

• Process– Trainee picks 2 case records

• Assessor selects one – Discussion centered on the

trainee’s notes – Assessor rates diagnostic

skills, clinical decision making, treatment decisions, planning, professionalism, etc.

• Takes 15-20 minutes• 6 assessments/year

Assess Doctors at Work:Assess Doctors at Work:Outcomes

• Patient outcomes– Judge physicians based on the outcomes of

their patients• Mortality and Morbidity• Plus a series of newer outcomes

– Patient satisfaction– Functional status– Cost effectiveness– Intermediate outcomes (e.g., HbA1c and lipid levels for

diabetics)

The New ParadigmThe New ParadigmTEAM

Screening of process and outcome (low power)

Diagnostic analysis of process & outcome (high power)

Screening not OK

Team OKDoctor OK

Doctor not OK

System not OK

Team not OK

After Beard, Farmer, Mann, LaDuca, Dauphinee 2001