Brian Jolly Centre for Medical and Health Sciences Education Monash University Train the Trainer Workshop October 31, 2003 Hong Kong International Consortium

Brian JollyCentre for Medical and Health Sciences Education

Monash University

Train the Trainer WorkshopOctober 31, 2003

Hong Kong International Consortium

Blueprinting


Objectives

• Describe basic ideas behind blueprinting

• Develop 2 examples

• Show how this does/does not articulate with IDEAL database


Principle

• The PURPOSE of the assessment will define the content, strategy, & technique used.


Curriculum and Assessment: ‘Ideal’


Curriculum and Assessment: Real


H. C.

Curriculum and Assessment: Real


Curriculum and Assessment: Blueprinting


Face validity Content Construct Criterion Effects on learning

Aspects of Validity - Constructs








Define what is to be tested Clinical problems at some resolution

Tasks for each problem

Blueprint

Select test methods Authenticity appropriate to clinical tasks

Clinical tasks determine the method

Titrate against practical constraints

Content Steps


CompetenceCategories

CVS RS … GI ….. MSS Endo/Metab

Haem /Oncology

….

History Taking

Physical Exam

Tests and Procedures

…..

Management

Communication and Patient Education

Simple Blueprint


The Hospital Setting The Community Setting

Competence Categories

CVS RS … GI ….. CVS RS … GI

History Taking

Physical Examx x

Tests and Procedures x x

…..

Other Parameters - Hierarchical


Inpatient Outpatient/Ambulatory



History Taking

Physical Examx x

Tests and Procedures x x

…..

Hierarchical - IDEAL® Related


The Child The Elderly



History Takingx x

Physical Exam

Tests and Procedures

…..

Other Dimensions



Cognitive /Affective

Categories


Recall x xReasoning

Affect x xSkill…..

Other Dimensions - Response



Cognitive /Affective

Categories


Recall KFI KFI

Reasoning

Affect CI SP

Skill…..

Test – Response Matrix


Steps: A plausible test of Content Validity

Chart distribution of questions by level & topic

Compare this distribution with weighting of topics in curriculum or course documents

Note to establish content validity we need a demonstrated relationship

Jolly 2000, Medical Education,


Some researchers use judgments of clinicians or successful test-takers

Dimensions of blueprint not always easy to define c.f. the items

How clinical problems, tasks or other 'concrete examples' relate to content often taken for granted in blueprinting

Jolly 2000, Medical Education,

Steps: A plausible test of Content Validity


Summary

• Before selecting evaluation tools, need to create an evaluation “blueprint”

• The blueprint = matrix that outlines parameters for the assessment in 3 major areas: domains to be assessed, content & assessment instruments


• Blueprint matrix ensures balance across assessment formats - conversely

• Multiple methods usually required to achieve blueprint objectives

• Different assessment instruments allow balance of strengths and weaknesses of each method

Summary


Content

In Australia RACS has sole responsibility for goals, training delivery & standards See CANMEDS, Duties of a Doctor,

Investigate ways of addressing these issues

Get profession & public to assist

Responsibility for the ‘hard to measure’

RACS blueprint – are we a surgeon first & a specialist second?

Blueprinting Issues: e.g., RACS Assessment


Judgement (80%) Literature on judgement

Measures of judgement (as yet undeveloped)

Manual dexterity (20%) Simulation and testing in training

Reznick, Paisley vs. Beard et al

RACS Assessment - Content

Documents

Brian Jolly Centre for Medical and Health Sciences Education Monash University Train the Trainer Workshop October 31, 2003 Hong Kong International Consortium