Assessment 101:

How to determine and design effective tools for resident assessment

Anna Oswald, MD MMEd FRCPC Lara Cooke, MD, MSc (MedEd), FRCPC Clinician Educators, RCPSC 1

By the end of this, you will be able to…

• Summarize six tips for learner assessment.

• Describe at least two different tools for assessing physician competencies.

• Use blueprinting to link objectives and assessment methods.

2

Outline

• Types of Assessment • Cognitive Domains • Linking Objectives & Assessment • Blueprinting • Making ITERs more user-friendly

3

Take-Home Messages:

1. There is a tool for every CanMEDS Role 2. No single tool can do it all 3. Assessment is multimodal 4. It is unnecessary to assess all Roles all the time 5. Appropriate tools may be tailored to specialty

4

Miller GE. The assessment of clinical skills/competence/performance. Academic Medicine (Supplement) 1990; 65: S63-S7.

Knows

Shows how

Knows how

Does

Prof

essi

onal

aut

hent

icity

Written, Oral or Computer based assessment

Performance or hands on assessment

5

What do we really want to assess?

Assessment of Competency

CLINICALLY-BASED: Mini CEX DECs ITERs MSF Practice assessments Chart audits

EXTRA CLINICAL: Written tests Oral exams OSCEs & SPs Logbooks & Portfolios “Long cases” with real

patients Simulations & labs

6

7

Link Objectives and Assessment Methods

8

Objectives

Teach Assess

9

Linking Objectives & Assessment

• Look at each key competency or objective and decide which domain(s) it links to:

– Knowledge – Skills – Attitude

10

Example Core Competencies: CanMEDS Health Advocate

Physicians are able to… 1.Respond to individual patient health needs

and issues as part of patient care; 2. Respond to the health needs of the

communities that they serve; 3. Identify the determinants of health of the

populations that they serve; 4. Promote the health of individual patients,

communities and populations.

11

Example Enabling Competency #2:

Respond to the health needs of the communities that they serve

Describe the practice communities that they serve

Identify opportunities for advocacy, health promotion and disease prevention in the communities that they serve, and respond appropriately

Appreciate the possibility of competing interests between the communities served and other populations

12

Identify the core competencies of interest Specify: Knowledge, Skills or Attitudes Figure out where these are best

demonstrated in the program

13

Task 1: Linking objectives to cognitive domains

14

TASK #1

Assign one or more of Knowledge (K), Skills (S), or Attitudes (A) to each enabling Objective for the Health Advocate objectives:

Describe the practice communities that they serve

Identify opportunities for advocacy, health promotion

and disease prevention in the communities that they serve, and respond appropriately

Appreciate the possibility of competing interests between the communities served and other populations

15

Respond to the health needs of the communities that they serve

Describe the practice communities that they serve K

Identify opportunities for advocacy, health promotion and disease prevention in the communities that they serve, and respond appropriately KS

Appreciate the possibility of competing interests between the communities served and other populations A

16

Task 2: Using cognitive domains to choose the right assessment methods

17

Task 2: Assign K-S-A

CLINICALLY-BASED: Mini CEX DECs ITERs MSF Practice assessments Chart audits

EXTRA CLINICAL: Written tests Oral exams OSCEs & SPs Logbooks & Portfolios “Long cases” with real

patients Simulations & labs

18

19

Blueprinting 101

Who uses a curriculum blueprint?

20

CanMEDS ROLE**

Written Exam

Oral Exam

ITER

Expert

Collab.

Comm.

Prof.

Advocate

Scholar

Manager

Abbreviated Curriculum Blueprint

21

CanMEDS ROLE

Written Exam

Oral Exam

ITER Simula- tions

Portfolio

Expert x Collab. x Comm. x Prof. x Advocate x Scholar x Manager x

Abbreviated Curriculum Blueprint

22

CanMEDS ROLE

Written Exam

Oral Exam

ITER Simula- tions

Portfolio

Expert x x x x Collab. x x Comm. x x Prof. x x x Advocate x Scholar x x X

Manager x x x

Abbreviated Curriculum Blueprint

23

24

Tip 1: Identify the Gap

• What are the specific knowledge, skills and attitude elements (of your gap)? (eg-Uh oh, we are not assessing residents’

competence around the social determinants of health)

• Where are these elements found in your

curriculum? (e.g. in hypertension clinic, where we look at at risk

populations and primary prevention of cardiac disease)

25

Task 3: Blueprinting Exercise

26

Task #3

Choose one core competency from the CanMEDS Framework (the GAP in your

program) and complete the blueprint grid in your worksheet.

27

Moving On…

28

ITERs & Observation

29

Tip 2: Observe!

Enhance direct observation for assessments

30

Tip 3: Modify your ITER

In-training Evaluation (Report)

31

The ITER

Universal Familiar Easy Multi-dimensional Documentation Practice-Based

32

The ITER

• Halo & Millstone effect – Comprehensiveness vs. objectivity

• Leniency Bias • Generic vs. specific to the rotation • Based on direct observation???

33

Problematic ITER Descriptors:

“Acts as a health advocate” hard to rate is meaning clear?

“Advocates for individual health promotion” a bit better

“Consistently reports risk factors for disease and identifies opportunities for preventative interventions” Can evaluate Can provide feedback

34

Task #4: Enhancing ITER descriptors

35

Task #4: Change Your ITER

• Modify the following ITER items to be more useful: – “Acts as an effective collaborator.” – “Demonstrates lifelong learning.”

36

Task #4: Enhancing ITER descriptors

37

High Quality ITERS: EMPHASIZE COMMENTS!!

• Justify the Likert rating • Include trainee’s response to feedback • Have specific examples (+/-) • Provide recommendations for improving

performance • Be detailed enough for an independent

reviewer • Dudek et al, Medical Education, 2008

38

Tip 4: Tailor existing tools to serve multiple functions

39

One option is to tailor a CLASSIC tool for your purposes:

40

Example: Tailor an Existing Tool e.g. SAQ

STEM: A 70 year old female is brought to the ED

after a fall. Her vital signs are normal and she complains of right hip pain.

THE QUESTION: What SAQ would address a HEALTH ADVOCATE or other CanMEDS objective?

41

Tailor an Existing Tool The NEW standard:

List SIX preventable complications of this injury and

identify ONE potential intervention to reduce the likelihood of EACH

This is the patient’s fourth fall in two weeks. List THREE interventions that should be considered upon discharge to reduce the risk of a recurrence.

Which social determinants of health might put the patient at risk for falls?

42

43

try a new assessment tool

Portfolio Multi-source feedback

TIP 6: Don’t be a Loner

Build a Community of Practice

44

Take Home Messages:

1. There is a tool for every CanMEDS Role 2. No single tool can do it all 3. Assessment is multimodal 4. It is unnecessary to assess all Roles all the time 5. Appropriate tools may be tailored to specialty

45

Outstanding Issues?

46

Acknowledgement:

Assessment 101: How to determine and design effective tools for resident

assessment

Jonathan Sherbino MD MEd FRCPC Emergency & Trauma Physician,

Director CPD, Division EM McMaster University

Clinician Educator, RCPSC

47

48

Hyper Links

Portfolio: “Instrument of instruments”

Strengths: Representative data Promotes reflection Flexible, multifaceted Authentic Longitudinal Contrast: logbooks Good for assessing complex,

integrative competencies

Weaknesses: Only as good as its parts Reliability Adherence Standardization Time consuming to evaluate Limited utility:

Medical Expert Collaborator

49

Example: Creating a New Tool - Portfolios

Residents must submit a 500 word essay each quarter detailing: An interaction during which they have

identified an advocacy issue Identify which key or enabling competency

is reflected in the interaction What role the resident played in the

interaction What lessons were learned or changes

resulted from the interaction.

50

Outcomes

• Increases individual responsibility for learning

• Promote reflection (?)

• No data on changes in learner behaviour or organization changes

51

MSF: Definition

• Survey

• Responses sampled from sphere of influence

• Aggregate ratings – Compared to mean ratings of other individuals

assessed – May include comparison to self-assessment

52

Strengths Multiple perspectives on

behaviour Good for:

Communicator Collaborator Professional

Weaknesses Resource-intensive Poor for:

Medical Expert Scholar

53

Key Elements

• Administrative infrastructure – Collecting & aggregating data is labor intensive

• Sample Size (reliability) – 6 peers, 6 co-workers, 22 patients (PAR) – 10 peers, 25 patients (ABIM)

54

Encounter Cards (DECs)

Strengths Useful for multiple assessors Face validity Promote feedback Contribute to reliability /

validity of ITER

Weaknesses: Administration Fatigue Gaming Halo /millstone effects Limited Utility

Health Advocate Professional

55

56