Assessment instruction module

Educator Development Program School of Medicine VANDERBILT

Assessment & Instruc-onal Strategies Advanced Clinical Elec-ves (ACE)

or Ac-ng Internships (AI) at Vanderbilt

Assembled by Kim Lomis, MD & Lillian B. Nanney, Ph.D

March 2015


Learning Objectives for this Module

After viewing materials in this presentation, a course director should be able to. . . . .

1. Match learning objectives for an ACE or AI course with appropriate instructional and assessment strategies for learners. 2. Design an ACE or AI course that conforms to best practices that have been established by Vanderbilt’s School of Medicine.


Why our emphasis on assessment?

•  Assessment drives learning.

•  Assessment is a 2-way street. It allows both the learner and the instructor to determine if the learning objectives have been achieved.


Assessment Serves Dual Func-ons

Provide an Assessment OF Learning This is summa4ve = used for grading Provide an Assessment FOR Learning This is forma4ve = guides the learner

Vanderbilt’s Course Directors are expected to . . . . . .


A New Compelling Argument for Providing Medical Students with Rigorous Assessments

•  Years 1 and 2 in the School of Medicine are now foundational. They are graded as pass/fail.

•  Assessments are critical for appraising a

student’s readiness for residency.


Responsibili-es of the Assessor

•  Assessment is one of your professional du4es.

•  Feedback must be based on OBSERVATIONS of PERFORMANCE


Best Prac-ces Courtesy of the Standing Assessment CommiKee

Category: Communica4on to Learners/Course Faculty:

•  Learning objec-ves MUST be shared with students and

par-cipa-ng faculty, residents, fellows. Distribute your materials on Day 1. •  Ensure that each assessment is linked to a learning objec-ves. •  Disclose your grading procedures to students and par-cipa-ng

assessors in print and/or electronic form. Also a Day 1 ac-vity.


Category: Assessment Methods •  Grading must be based on more than one assessment

method. Collect different perspec4ves to ensure accuracy and fairness.

•  Performance data about an individual student should be

solicited from mul4ple team members (faculty, residents, other health professionals).

Category: Feedback to learners: •  Courses must provide forma4ve feedback, 4med such that

there is opportunity for the learner to improve.

Best Prac-ces Courtesy of the Standing Assessment CommiKee


•  Learners have a right to expect fairness and consistency from course to course. One purpose of this module is to norm the course directors/assessors.

•  Vanderbilt has a responsibility to ensure that learners are

progressing upward along a developmental learning curve. This determination can only be made if learners are observed and assessed.

Our Milestones Module is coming soon . . . . .

•  Learners have a right to know specific areas where they are not progressing or are deficient.

To recap - Assessment/observations must be accurate, well-documented, informative and frequent.

Assessment Does MaKer


Ideally, assessment should inform future learning:

• The Assessment should describes current performance

“You are here” • The Assessment should ar-culate behaviors necessary to aKain the next level of performance.

“Go this way”

Milestones = GPS (Global Posi-oning System)


All 3 aspects of course design must be though[ully aligned

Learning Objec-ves

Assessment Strategies

Instruc-onal Strategies


Let’s focus on Learning Objec-ves

Learning Objec-ves

Assessment Strategies

Instruc-onal Strategies

1.   How will you help students aKain them? This is your instruc-onal strategy. 2. How will you ensure they have aKained them? This is your assessment method.


A Learning Objec-ves can best be achieved by a specific type of instruc-onal tool.

Learning Objec-ves

Assessment Strategy

Instruc-onal Strategy

Each learning objec-ves must be measurable (assessable) and reasonable given the learning -me.


Examine our example below linking learning objectives with likely educational activities and then how the learner will be assessed. Our sketch would be suitable for an ACE course in Neuroradiology or Neurosurgery.

Learning Objec-ve Educa-onal Ac-vity Assessment Create a differen-al diagnosis of brain tumors based on tumor loca-on

a. par4cipate in weekly brain tumor boards b. complete a comprehensive pa4ent evalua4on (using a template) for one pa4ent of interest during the first 2 weeks c. view a prepared module on this topic d. seek evidence based literature (epidemiological, guideline ar4cles) on this topic. e. par4cipate in the radiology reading room experience when reports are being prepared. f. present assigned pa4ents to the team during daily reports.

1.  Student completes a second write-‐up by the end of course for an unknown tes4ng case.

2.  A publicized scoring rubric will be used by Resident/faculty to score two pa4ent presenta4ons during each week. The first 2 weeks are for feedback purposes. The last 2 weeks will be summa4ve.


Examine another example linking learning objectives with likely educational activities and then with learner assessment. Our sketch would be suitable for an ACE course in Endocrinology (Diabetes).

Learning Objec-ve Educa-onal Ac-vity Assessment Gain proficiency in evalua-ng and trea-ng in pa-ents with diabetes mellitus and in formula-ng or modifying a treatment regimen to control blood glucose level.

a.  Student will aWend the adult endocrinology fellows’ conference

b.  Student will aWend rounds with the diabetes consulta4ve service in the hospital

c.  Student will provide glycemic care in 4 seYngs (periopera4ve, post-‐surgical emergency and cardiac pa4ents).

d.  Students will prac4ce educa4ng pa4ents regarding how to test blood gluose, dose/administer insulin, count carbohydrate grams, use devices such as insulin pens.

1.  Assessment of the quality of pa4ent educa4on will be assessed by direct clinical observa4ons on the glucose management service once a day.

2.  Assessed by oral exam from a set of prepared cases and graded by a scoring rubric familiar with to the learner.

3.  Quality of glycemic care decisions in daily prac4ce will be scored once a day by residents/fellows on the service.


Now its your turn for prac-ce. Here is a simple table to get you started.

Write 3 plans for your course. This VSTAR module also contains a word doc with a table you can download and

use. Learning Objec-ve Educa-onal Ac-vity Assessment Strategy

1.

2.

3.


Who will be the Assessors? Consider: Other Faculty The Residents The Staff 1) Will they need some training so they can be as standardized as possible? 2) Should you furnish them with a checklist or rubric? 3) Do you need to do some “faculty” development? Do they need to see the linkage between learning objectives, instructional strategies, and assessments so they have clarity about expectations for learners at this level?

Answers: Yes, Yes, Yes


Unconscious Bias Share these reminders with your assessors

Halo Error Outstanding performance in one area, particularly early in the rotation, tends to blind observers to poorer performance in other domains. Similarly, it can be hard for a student to recover from a poor start.

Wave Effect If everyone else has felt that the learner is superior, it can be difficult to give an unsatisfactory assessment. The student is ‘”waved” through the rotation on reputation and not on performance.

Hawk or Dove Effect Take a close look at the new grading scale that Vanderbilt will be using. Are you too hard, too easy? Use the rating tools appropriately to ensure fair grading (Stay turned for the Advanced Assessment Module) .


Similarity/Dissimilarity Bias Recognize something of yourself in the Learner? - Likely a good assessment will result. Likewise, it can be hard to give a high rating to someone with a very different practice style.

Prejudices Accents, beards, gender, lip rings, tattoos? Be aware of your prejudicial tendencies (blindspots) that can interfere with fair assessment.

Unconscious Bias (con-nued) Share these reminders with your assessors


Learning Objec-ves

Assessment Strategy

Instruc-onal Strategy

Learning Objective: Interpret clinical information to formulate a prioritized differential diagnosis that guides the creation of a patient-specific management plan. Assessment Strategy: SNAPPS S - summarize the case N - narrow the differential A - analyze the differential P - probe the preceptor P - plan management S - select an issue for self directed learning


Consider these Using these Tools •  1 minute preceptor technique hWp://www.prac4caldoc.ca/teaching/prac4cal-‐prof/teaching-‐nuts-‐bolts/one-‐minute-‐preceptor/

•  SNAPPS technique (your departmental Master Clinical Teacher can host a teaching session)

hWp://www.prac4caldoc.ca/teaching/prac4cal-‐prof/teaching-‐nuts-‐bolts/snapps/

•  Mini-‐Clinical Evalua4on Exercise (Mini-‐CEX) hWp://www.abim.org/program-‐directors-‐administrators/assessment-‐tools/mini-‐cex.aspx

•  S4mulated chart review hWp://www.prac4caldoc.ca/teaching/prac4cal-‐prof/teaching-‐nuts-‐bolts/chart-‐s4mulated-‐recall/

•  Case Discussions •  Conference Presenta4ons


Use Verbal Feedback Techniques

§  Continue (You are on track)

§  Do More (of “this”)

§  Do Less (of “this”)

§  Stop (Don’t do “this” again!)

Considerations: • Keep portions small and feedback spaced out over time • Students tend to selectively “hear” good bites and miss the critique. • Best prac4ce assumes that feedback is delivered in a rela4vely private seYng.


Standing Assessment CommiKee (recapping the baseline requirements)

1)  Use Mul4-‐Modal Tools -‐ to gain many perspec-ves

2)  Distribute the plan to students -‐ remove the mystery around “What am I supposed to learn?”

3)  Distribute the plan to all the possible assessors -‐ eliminate all mystery around “What am I supposed to do with

this student.”

4)  Provide frequent feedback -‐mid and end-‐of-‐course at minimum

4)  Final grades due < 6 weeks


Upcoming Modules •  Milestones are coming to Clinical Evalua4on

We will share which competencies must be assessed.

•  Master Adap4ve Workplace Learner Cycle

We will be helping students improve their prac4ce-‐based learning skills and develop the habits necessary for lifelong learning. Courses will be asked to include assessment of the student’s progress through the Master Adap4ve Workplace Learner cycle. More informa4on to come on this soon.

Education

Assessment instruction module