Mennin Consulting, 2006 Overview of Assessment ESME 2006 Stewart Mennin, PhD

Mennin Consulting, 2006

Overview of AssessmentESME 2006

Stewart Mennin, PhD

Mennin Consulting, 2006 ESME

Objectives

Make more informed decisions about assessment

Apply information to your teaching Describe the relationship between

assessment & learning Discuss criteria for effective assessment Define & apply basic terminology


Objectives

Describe selection, supply & performance methods

Link Miller’s pyramid to assessment methods Match assessment methods to your

outcomes Draft assessment plan (blueprint)for your

teaching


GOALS OF ASSESSMENT

Enhance student learning Guide faculty teaching efforts Inform curricular development & revision Fulfill responsibilities to students, institution &

society Contribute to research


FEATURES OF ASSESSMENT

Assessment is more a curriculum design problem than a measurement problem Part of learning & can not be separate from it Assessment is about the present, past and future of learning Assessment in not a collection of instruments Assessment should be authentic, grounded in what is real


EIGHT CRITERIA: EFFECTIVE ASSESSMENT

1. Articulation of a clear conception of intended learning outcomes

2. Use variety of assessment procedures/methods

3. Close agreement between intended outcomes of instruction, learning tasks, & the assessment procedures/methods

4. Adequate sampling of student performanceGronlund, 2000


EIGHT CRITERIA: EFFECTIVE ASSESSMENT

5. Procedures that are equitable for all participants

6. Explicit, specific criteria for use in judging successful performance

7. Timely feedback that emphasizes strengths of performance & focuses attention on specific areas in need of improvement

8. A grading & reporting system consistent with the rest of the learning & assessment process

Gronlund, 2000


Key Terminology

Formative – Summative Reliability & Validity Criterion Referenced --- Norm referenced

Progress test Sequential test


Assign gradesJudge competenceCertify masteryPromotion

Feedback to student & learnerFind misconceptionsIdentify strengths &

weaknessesChange future teachingDevelop learning plan

TeacherTeacher & learner

Milestones & endFrequent & throughout

Decisions / JudgmentImprovement

Use

Evaluator

Timing

Purpose

SummativeFormative

Assessment As Learning


Reliability

Consistency of results Reliability is matter of sampling

– Across content


Domain of Interest

Test Sample

Test Sample


Assessment of Clinical Assessment of Clinical Competence is a Issue of Competence is a Issue of

SpecificitySpecificity

Competent


Reliability

Practical suggestions:– Do not rely on short tests

– Sample broadly (content, in time, examiners, patients)

– Consider efficiency in: selection of test format construction test items


Reliability

Adequate reliability requires substantial sampling (therefore: resources, testing time, examiners, patients, etc.)– Be efficient


Validity

Miller GE. The assessment of clinical skills/competence/performance. Academic Medicine (Supplement) 1990; 65: S63-S7.

Knows

Shows how

Knows how

Does

Pro

fess

iona

l aut

hent

icity

Pro

fess

iona

l aut

hent

icity


Validity

Miller GE. The assessment of clinical skills/competence/performance. Academic Medicine (Supplement) 1990; 65: S63-S7.

Knows

Shows how

Knows how

Does

Pro

fess

iona

l aut

hent

icity

Pro

fess

iona

l aut

hent

icity

Written, Oral orComputer based assessment

Performance or hands on assessment


Valid Inferences

Validity is a single concept with different forms of evidence– Content-related evidence– Criterion-related evidence– Construct-related evidence


ASSESSMENT METHODS

All assessment methods have limitations and no one method can assess all types of knowledge & skill

Multiple methods are recommended


Some Assessment Options:

Selection Item– Multiple Choice Question– One-Best Answer Questions– Extended-Matching Questions

Supply Items– Free Response Questions– Short Answer Questions– Essay Questions

Performance Examinations– Laboratory Practical Examinations (Identification and Concepts)– Clinical Skills (e.g., OSCE)– Other


Selection Methods

MCQ Matching Extended Matching True False Multiple True False

Knows

Knows how

Shows how

Does


Knows/Knows how

The stimulus format is more important than the response format

KnowsKnows

Knows how

Shows how

Does


How Well Written?Stem

a. Poor

b. Good


How Well Written?

a.

Stem

b.

c.

a. Poorb. Good

d.


How Well Written?

30-year-old man has loss of pain and temperature sensation from the neck down on the right side of the body and on the left side of the face; partial paralysis of the soft palate, larynx, and pharynx on the left; and ataxia on the left. This syndrome is most likely to result from thrombosis of which of the following arteries?A BasilarB Right posterior inferior cerebellarC Left posterior inferior cerebellarD Right superior cerebellarE Left superior cerebellar


Extended Matching

For each quotation, select the most likely author.

1. I never teach my pupils. I only attempt to provide the conditions under which they can learn.

2. All genuine knowledge originates in direct experience.

3. To know one’s ignorance is the best part of knowledge.

Authors on Education:

a. D. J. Boorsten e. Lao-Tzu

b. Camus f. S. Scott Obenshain

c. Albert Einstein g. John Pieper

d. Thomas Harris h. Mao Tse-Tung


Extended Matching Answers

Authors on Education:

a. D. J. Boorsten e. Lao-Tzu

b. Camus f. S. Scott Obenshain

c. Albert Einsteing. John Pieper

d. Thomas Harris h. Mao Tse-Tung

1. I never teach my pupils. I only attempt to provide the conditions under which they can learn. (c)

2. All genuine knowledge originates in direct experience. (h)

3. To know one’s ignorance is the best part of knowledge. (e)


Supply Methods

MEQ Essay (long & short) Practical Short Answer Written simulations

Knows

Knows how

Shows how

Does


Performance Assessment

Procedures – skills lab OSCE (with skills stations) Mini CEX Long Case Logs Portfolio Clinical Notes Observation Other Knows

Knows how

Shows how

Does


Communication Skills

Interviewing & Communication Skills


Essential Elements of Communication Skills

Open the Discussion Build a Relationship Gather Information Understand the Patient’s Perspective Share Information Reach Agreement Provide Closure

Acad Med 2001;76(4):390-393

O rg an ig ram tite l

C lin ica lS cen ario

ClinicalClinicalProblemProblem

Key Feature Approach

Testing time

Case 1

Case 2

Test usingintegral cases

Case 1

Case 2

Case 3

Case 4

Case 5

Case 6

Case 7

Test usingkey features


Shows How: OSCE

Make stations as clinically authentic as possible

Global judgments do well in OSCEs

Content specificity is the problem

Knows

Knows how

Shows how

Does


Checklist/Global Rating Scale Comparison

Checklists– Dichotomous scores

(yes/no)– Relatively objective

Record rather than interpret

– Station specific

Rating Scales– Ordinal level measures

(poor, good, excellent)– Relatively subjective– Station independent


Does: Practice Assessment

Indirect methods– Chart audit– Practice audits– Prescription data………

Direct methods– Clinical (supervisor) ratings– Undercover standardized patients– Video assessment– Patient reports– Peer reports– Clinical work samples– Mini-CEX………

Knows

Knows how

Shows how

Does


Clinical Work Samples

Repeated direct observations of clinical performance in practice using (generic) evaluation forms, completed by any significant observer (clinician, nurse, peer…..)

Illustration

Knows

Knows how

Shows how

Does


Mini Clinical Examination (Norcini, 1995)

Short observation during clinical patient contact (10-20 minutes)

Oral evaluation Generic evaluation forms completed Repeated at least 4 times by different

examiners (cf. http://www.abim.org/minicex/)

Knows

Knows how

Shows how

Does


NEW SKILLS EMPHASIZED

Learning how to learn Self-assessment Humanization Leadership Team skills

Skills of expression Professional behavior Reflection Permanent education


Knows Knows How Shows How DoesWritten Tests Essay A A

MCQ A AWritten Exercises Application Test B A

MEQ B APractical B A

Patient Exercises OSCE B ASPs B ATriple Jump B AMini CEX A BLong Case A B

Observations Direct Observation B AMed Record Review B AOral Examination B APeer Evaluation B ARating Scale B A

A = Most effective B = Less effective

Methods & Knowledge


Task

Use the table from effective teaching and match assessment methods to your outcome competencies


0

10

20

30

40

50

60

Year 1 Year 2 Year 3 Year 4 Year 5 Year 6 Ref. Group

Percentage correct

Progress TestScores on one progress test

Faculty of Medicine Maastricht University


Mean correct scores on the Progress Test from a number of Maastricht student cohorts (black lines) and reference group of family physicians in training (blue line)

Progress Test Maastricht Test Results ‘77 - ‘88

Faculty of Medicine Maastricht University


Planning: Blueprinting

Competencies Time Teaching methods In class -- out of class

Com

pete

ncie

s

Perform-2 Capable

Perform-1 Competent

SupplySelect

Methods

Clinical Skills

Critical Int

KnowComm

Human

Self Assess

ProfessProfess


Conclusions

Assessment is less of a psychometric problem, but rather an educational design problem (i.e. how to use assessment strategically for its educational effects)

Assessment requires careful planning and monitoring To effective with the entire competence pyramid

– a mix of methods that depends on the context; not a single method is best or bad

Cees van der Vleuten


Acknowledgements

Cees van der Vleuten - Maastrict Deana Richter – New Mexico Teresita McCarty - New Mexico Nancy Sinclair - New Mexico Regina Petroni Mennin – UNIFESP


www.menninconsulting.com

[email protected]

[email protected]

Presentations Available at:

Documents

Mennin Consulting, 2006 Overview of Assessment ESME 2006 Stewart Mennin, PhD