of 58 /58
Gordon Page Emeritus Professor, Faculty of Medicine University of British Columbia Vancouver Canada

Gordon Page.pdf

Embed Size (px)

Citation preview

Page 1: Gordon Page.pdf

Gordon PageEmeritus Professor, Faculty of Medicine

University of British ColumbiaVancouver Canada

Page 2: Gordon Page.pdf
Page 3: Gordon Page.pdf

Outline

The Rationale – Why developnational examinations for thehealth professions?

How is ‘competence’ defined?

What are the desired features ofnational examinations for healthprofessionals

Page 4: Gordon Page.pdf

Outline

The Rationale – Why developnational examinations for thehealth professions?

How is ‘competence’ defined?

What are the desired features ofnational examinations for healthprofessionals

Page 5: Gordon Page.pdf

Why develop nationalexaminations in the health

professions?

Page 6: Gordon Page.pdf

Why develop nationalexaminations in the health

professions?

A Quality AssuranceStrategy

Page 7: Gordon Page.pdf

‘Assurance’ of theQuality of What?

1.The competence of the graduates ofthe Health Professions SchoolsDo the graduates possess at least the minimallevels of competence (a) to fulfill the roles thatsociety has assigned to their professions, or (b) toadvance to the next level of training in theirprofession?

Summative Formative

Page 8: Gordon Page.pdf

‘Assurance’ of theQuality of What?

1.The competence of the graduates ofthe Health Professions Schools

2. The quality of the training programsthat prepare students for these roles

Summative Formative

Page 9: Gordon Page.pdf

Why develop nationalexaminations in the healthprofessions?

The educational rationale

To convey to students what is important to learn,and to motivate them to learn it (‘Assessmentdrives learning!’)

To influence the curricula of health professionsschools by providing an operational definition ofthe competencies expected of graduatingstudents.

To provide a mechanism for evaluating the qualityof and inequities between the schools in eachhealth profession, identifying S&W in each

Page 10: Gordon Page.pdf

Why develop nationalexaminations in the healthprofessions?

The health care rationale

To ensure that the graduates of healthprofessional programs are competent – toprovide a basis for decisions on theirlicensure/registration

To define a national standard – thoseabilities that together define thecompetencies and levels of performanceexpected of all graduates

Page 11: Gordon Page.pdf

Why develop national healthprofessions examinations?

In summary …

To provide guidance and motivation tostudents and schools on what isimportant to learn

To pass judgement the competence ofgraduating students, and the educationaleffectiveness of schools

To define a national standard – thoseabilities that define the competencies andlevels of performance expected of allgraduates

Page 12: Gordon Page.pdf

Outline

The Rationale – Why developnational examinations for thehealth professions?

How is ‘competence’ defined?

What are the desired features ofnational examinations for healthprofessionals

Page 13: Gordon Page.pdf

What is ‘competence’?

Competence is the “ability to do something”

In the context of health professionseducation and practice, Epstein (2002)defined competence as, “the habitual andjudicious use of communication, knowledge,technical skills, clinical reasoning, emotions,values, and reflections in daily practice toimprove the health of the individual patientand community”

Page 14: Gordon Page.pdf

A national examination of‘competence’ must therefore testthe ability to use/apply knowledgeand skills in the context of caringfor patients.

A test of the recall of knowledge is not a testof ‘competence’. While possessing and beingable to recall knowledge is ‘the cornerstone’ ofbecoming competent, it is not sufficient.Students who memorize and can recallknowledge are often not able to effectivelyapply or use that knowledge.

Page 15: Gordon Page.pdf

Possessing Knowledgeversus

Possessing Competence

There is a bigdifference!

Page 16: Gordon Page.pdf

Non VignetteWhat is the most likely renalabnormality in children with nephroticsyndrome and normal renal function?

acute poststreptococcal glomerulonephritis

hemolytic-uremic syndrome

minimal change nephrotic syndrome

nephrotic syndrome due to focal andsegmental glomerulosclerosis

Schönlein-Henoch purpura with nephritis

(A)

(B)

*(C)

(D)

(E)

Case SM, Swanson DB. Constructing Written Test Questions for the Basicand Clinical Sciences. Page 58-9. www.nbme.com

Page 17: Gordon Page.pdf

Short Vignette

A 2-year-old boy has a 1-weekhistory of edema. Blood pressureis 100/60 mm Hg, and there isgeneralized edema and ascites.Serum concentrations are:creatinine 0.4 mg/dL, albumin 1.4g/dL, and cholesterol 569 mg/dL.Urinalysis shows 4+ protein andno blood. What is the most likelydiagnosis?

Page 18: Gordon Page.pdf

Long Vignette

A 2-year-old black child developedswelling of his eyes and ankles overthe past week. Blood pressure is100/60 mm Hg, pulse 110/min, andrespirations 28/min. In addition toswelling of his eyes and 2+ pittingedema of his ankles, he has abdominaldistension with a positive fluid wave.Serum concentrations are: creatinine0.4 mg/dL, albumin 1.4 g/dL, andcholesterol 569 mg/dL. Urinalysisshows 4+ protein and no blood.

Page 19: Gordon Page.pdf

What is the most likely renal abnormality in children withnephrotic syndrome and normal renal function?

Non Vignette

acute poststreptococcal glomerulonephritishemolytic-uremic syndrome

minimal change nephrotic syndrome

nephrotic syndrome due to focal and segmentalglomerulosclerosisSchönlein-Henoch purpura with nephritis

(A)(B)

*(C)

(D)

(E)

Short VignetteA 2-year-old boy has a 1-week history of edema. Blood pressure is

100/60 mm Hg, and there is generalized edema andascites. Serum concentrations are: creatinine 0.4 mg/dL,

albumin 1.4 g/dL, and cholesterol 569 mg/dL. Urinalysis shows4+ protein and no blood. What is the most likely diagnosis?

Long VignetteA 2-year-old black child developed swelling of his eyes andankles over the past week. Blood pressure is 100/60 mm

Hg, pulse 110/min, and respirations 28/min. In addition toswelling of his eyes and 2+ pitting edema of hisankles, he has abdominal distension with a positivefluid wave. Serum concentrations are: creatinine 0.4 mg/dL,albumin 1.4 g/dL, and cholesterol 569 mg/dL. Urinalysis shows4+ protein and no blood.

Page 20: Gordon Page.pdf

What is the most likely renal abnormality in children withnephrotic syndrome and normal renal function?

A 2-year-old boy has a 1-week history of edema. Blood pressureis 100/60 mm Hg, and there is generalized edema and ascites.Serum concentrations are: creatinine 0.4 mg/dL, albumin 1.4g/dL, and cholesterol 569 mg/dL. Urinalysis shows 4+ proteinand no blood. What is the most likely diagnosis?

A 2-year-old black child developed swelling of his eyes andankles over the past week. Blood pressure is 100/60 mm Hg,pulse 110/min, and respirations 28/min. In addition to swellingof his eyes and 2+ pitting edema of his ankles, he has abdominaldistension with a positive fluid wave. Serum concentrations are:creatinine 0.4 mg/dL, albumin 1.4 g/dL, and cholesterol 569mg/dL. Urinalysis shows 4+ protein and no blood.

A B *C* D E1 0 99 0 0

8 1 90 1 0

0 0 98 2 0

5 2 82 8 1

0 1 98 1 0

10 9 66 10 5

Overall P-Value

94

88

84

Case SM, Swanson DB. Constructing Written Test Questions for the Basic and Clinical Sciences, 1996. Page 58-9.

Does testing knowledge recall vs applicationof knowledge affect test scores?

Page 21: Gordon Page.pdf

Possessing Knowledgeversus

Possessing Competence

There is a bigdifference!

National examinations must testcompetence, not knowledge recall!

Page 22: Gordon Page.pdf

But whatcompetencies should

be tested?

Page 23: Gordon Page.pdf

Standar Kompetensi DokterKonsil Kedokteran Indonesia

(Standards of Medical Competencies)(Indonesian Medical Council)

2006

Describes the competencies expected ofstudent doctor graduating from an

Indonesian medical schools.

Page 24: Gordon Page.pdf

Areas of Competency(Standar Kompetensi Dokter 2006)

1. Effective communication

2. Clinical skills

3. Scientific basis of medical knowledge

4. Management of health problems

5. Management of information

6. Self awareness and self development

7. Ethics, morals, medico-legal aspectsand professionalism, and patient safety

Page 25: Gordon Page.pdf

Medical Council of Canada

Objectives for the QualifyingExamination

The ‘Objectives’ define thecompetencies expected of medicalgraduates entering supervised andindependent practice

http://www.mcc.ca/Objectives_Online/

Page 26: Gordon Page.pdf

Competency Frameworkfor Doctors in Canada

Page 27: Gordon Page.pdf

Summary …‘How is competence defined?’

The ability to use/applyknowledge and skills in thecontext of caring for patients.

The ‘competencies’ comprising‘competence’ should be definedby each health profession

Page 28: Gordon Page.pdf

Outline

The Rationale – Why developnational examinations for thehealth professions?

How is ‘competence’ defined?

What are the desired features ofnational examinations for healthprofessionals

Page 29: Gordon Page.pdf

Utility (U) of anAssessment Strategy

V = Validity

R = Reliability

E = Educational impact

A = Acceptability

C = Cost

U = f(V, R, E, A, C)

Cees van der Vleuten, 1990s

Page 30: Gordon Page.pdf

What are the desired featuresof national examinations forhealth professionals?

V = Validity

R = Reliability

E = Educational impact

A = Acceptability

C = Cost

Page 31: Gordon Page.pdf

What are the desired featuresof national examinations forhealth professionals?

V = ValidityR = Reliability

E = Educational impact

A = Acceptability

C = Cost

Page 32: Gordon Page.pdf

Validity – An Essential Featureof National Examinations

The question posed by validity is,“Does this examination permit usto make correct inferencesabout the competence ofexaminees?”

Page 33: Gordon Page.pdf

Validity – An Essential Featureof National Examinations

The two main factors influencingvalidity are:

Examination Content – what istested?

Examination Methods – how is thecontent tested?

Page 34: Gordon Page.pdf

Validity – An Essential Featureof National Examinations

The two factors influencingvalidity are:

Examination Content – what istested?

Examination Methods – how is thecontent tested?

Page 35: Gordon Page.pdf

International Trends in Medicine –Testing Methods used in National

Examinations in Indonesia, Canada, theUSA and Australia

An MCQ ‘written’ test Questions are ‘case-based’ Tests are computer-based 150 or more questions Use A-type MCQ questions Allow 3 or more hours of testing time

An ‘Objective Structured Clinical Examination’(OSCE) Use standardized patients (SPs) Use 3 or more hours of testing time Use 10 to 20 ‘stations’, each from 8 to 25 minutes in

length

Page 36: Gordon Page.pdf

Is there evidence supportingthe ‘validity’ of theseexamination methods?

Page 37: Gordon Page.pdf

Norcini (ASME, 2003) – Of all assessmenttechniques, written test (MCQs) scores haveshown to be the best predictors of futureclinical performance and of clinical outcomes.

Tamblyn et al (JAMA, 2002,1998) – Canadianlicensing examination (clinical decision-making) scores show sustained relationshipswith effectiveness indices of preventive care

and acute and chronic disease management

Page 38: Gordon Page.pdf

The Relationship of ExaminationScores to Concurrent ClinicalPerformance for Practice-readydoctors (Canadian study, 2007)

Correlations between:

Clinical Performance (Mini-CEX scores)

and MCQ scores .335

Clinical Performance (Mini-CEX scores)

and OSCE scores .372

Page 39: Gordon Page.pdf

Is there evidence supportingthe ‘validity’ of theseexamination methods?

Yes, it isreassuring but not

strong!

Page 40: Gordon Page.pdf

Validity – An Essential Featureof National Examinations

The two factors influencingvalidity are:

Examination Content – whatis tested?

Examination Methods – how is thecontent tested?

Page 41: Gordon Page.pdf

An Examination Blueprint

Page 42: Gordon Page.pdf

Sample Blueprint forIndonesia

Competency areas MCQ OSCE

Communication skills 10%

Clinical Skills 20% 70%

Scientific basis … 10%

Management of healthproblems

60%

Management ofinformation

10%

Self awareness &development

10%

Ethics, morals,professionalism

10%

Page 43: Gordon Page.pdf

Blueprint for MCQ Items Testing‘Management of Health Problems’

Clinical Tasks ChildHealth

Maternalhealth

Adulthealth

Mentalhealth

Populationhealth

Data gathering 5 3 13 5 4

Datainterpretationand synthesis

6 3 17 6 4

Drugmanagement

2 2 10 3 0

Non-drugmanagement

2 1 9 3 0

Page 44: Gordon Page.pdf

Georges Bordage

Page 45: Gordon Page.pdf

Clinical Decision-Making

Reduced (“empty mind”)

Compiled (“recall/recognition”)

Elaborated (“deductive thinker”)

Dispersed (“cluttered mind”)

(Bordage, G. Academic Medicine, 1994, 1999)

Page 46: Gordon Page.pdf

Clinical Decision-Making

Reduced (“empty mind”)

Compiled (“recall/recognition”)

Elaborated (“deductive thinker”)

Dispersed (“cluttered mind”)

(Bordage, G. Academic Medicine, 1994, 1999)

****************************************

Page 47: Gordon Page.pdf

Summary – ‘What to test?’

National examinations should:

Test content defined relative to anexamination blueprint

Test the application/use of knowledgeand skills (i.e., ‘competencies’)

XNot test recall of knowledge

XNot test thoroughness of datagathering

XTest clinical decision making, notclinical reasoning

Page 48: Gordon Page.pdf

What are the desired featuresof national examinations forhealth professionals?

V = Validity

R = Reliability

E = Educational impact

A = Acceptability

C = Cost

Page 49: Gordon Page.pdf

Reliability

Is a measure of the accuracy of testscores

For MCQ examinations, accuracy islargely a function of the number of itemson the test (i.e., sampling)

For OSCEs, accuracy is also a function ofthe consistency of examiners and SPs

Reliability is expressed as a reliabilitycoefficient ‘r’, where 0<r<1

‘Rule of thumb’: r > .80 for makingdecisions about individual examinees

Page 50: Gordon Page.pdf

Reliability as a function oftesting time

TestingTime inHours

1

2

4

8

MCQ

0.62

0.76

0.93

0.93

OSCE

0.43

0.60

0.76

0.86

Page 51: Gordon Page.pdf

What are the desired featuresof national examinations forhealth professionals?

V = Validity*

R = Reliability*

E = Educational Impact

A = Acceptability

C = Cost

Page 52: Gordon Page.pdf

Educational Impact

Students will learn, and schools willteach what is tested on nationalexaminations

With a view to ‘assessment drivinglearning’, is the national examination‘driving’ teaching and learning in theright direction? It will if it testscompetencies that are important tothe practice of the health profession inquestion.

Page 53: Gordon Page.pdf

Acceptability What strategies encourage acceptance of

national examinations by key stakeholders –i.e., the schools, the professional licensing andregistration bodies, the profession, and thegovernment?

The key strategy is INVOLVEMENT – indefining policies related to the role of theexamination in the profession, in defining anddeveloping examination content, in definingthe standards of acceptable performance(passing scores), in quality assurance, …

Page 54: Gordon Page.pdf

Cost

National examinations with acceptablemeasurement qualities are costly todevelop, administer, score and maintain

The major cost is peoples’ time

In the USA, Australia and Canada thecosts of the MCQ and OSCE nationalexaminations exceed $2000 per candidatefor each examination – a cost which issupported primarily by examinee fees

Page 55: Gordon Page.pdf

Cost Major cost items include:

developing test blueprints

Developing/peer review of examinationquestions/cases

developing supportive computer systems for testdevelopment, test administration, and test analysis

pilot testing and refining questions/cases

standard setting

recruiting/training/paying examiners

examination scoring

quality assurance reviews (using examinationstatistics – e.g., reliability of test scoresdiscrimination indices of test items)

maintaining test security

Page 56: Gordon Page.pdf

In summary …

The Rationale – Why developnational examinations for thehealth professions?

How is ‘competence’ defined?

What are the desired features ofnational examinations for healthprofessionals

Page 57: Gordon Page.pdf

Developing the Ideal Modelfor National HealthProfessional Examinations Is an MCQ examination and an OSCE an ideal model for a

national examination? My answer: No!

Why? Option lists in MCQs provide too much cuing to weaker

candidates, and

OSCEs lack reliability (accuracy) and hence validity.

What is the ideal model? Short answer questions in place of MCQs, and

Highly structured assessments of clinical skills in real clinicalsettings in place of OSCEs

Why not use the ideal model? Too expensive, and quality control is even more challenging!

Page 58: Gordon Page.pdf

Terima kasih