F A C U L T Y O F M E D I C I N E

PEER ASSESSMENT (PA) OF MEDICAL PEER ASSESSMENT (PA) OF MEDICAL PROFESSIONALISM BY MEDICAL PROFESSIONALISM BY MEDICAL

STUDENTSSTUDENTS

AFMC PROFESSIONALISM RESOURCE GROUP AFMC PROFESSIONALISM RESOURCE GROUP MEETINGMEETING

A Keith W Brownell MDA Keith W Brownell MD

London – April 29, 2006London – April 29, 2006

F A C U L T Y O F M E D I C I N E

OVERVIEWOVERVIEW

• ReferencesReferences

• Background to My Interest in the TopicBackground to My Interest in the Topic

• General Comments About PAGeneral Comments About PA

• Specific Comments on PA of Professionalism by Medical Specific Comments on PA of Professionalism by Medical StudentsStudents

• Some Canadian ExperiencesSome Canadian Experiences

• DiscussionDiscussion

F A C U L T Y O F M E D I C I N E

DISCLOSURE/DISCLAIMERDISCLOSURE/DISCLAIMER

None of this material is original and None of this material is original and most of it comes directly or indirectly most of it comes directly or indirectly from one of the key references which from one of the key references which

are listed on Slide 5are listed on Slide 5

F A C U L T Y O F M E D I C I N E

BACKGROUND TO INTEREST IN BACKGROUND TO INTEREST IN THIS TOPICTHIS TOPIC

F A C U L T Y O F M E D I C I N E

KEY REFERENCESKEY REFERENCES• John Norcini – Peer Assessment of Competence. Medical Education John Norcini – Peer Assessment of Competence. Medical Education

2003:37:539-543.2003:37:539-543.

• Deirdre Lynch et al – Assessing Professionalism: A Review of the Deirdre Lynch et al – Assessing Professionalism: A Review of the Literature. Medical Teacher 2004;26(4):366-373.Literature. Medical Teacher 2004;26(4):366-373.

• Carolyn Shue et al – Maximizing Participation in Peer Assessment of Carolyn Shue et al – Maximizing Participation in Peer Assessment of Professionalism: the Students Speak. Academic Medicine 2005;80(10 Professionalism: the Students Speak. Academic Medicine 2005;80(10 Suppl):S1-S5.Suppl):S1-S5.

• Elaine Daddefer et al – Peer Assessment of Professional Competence. Elaine Daddefer et al – Peer Assessment of Professional Competence. Medical Education 2005;39:713-722.Medical Education 2005;39:713-722.

• Louise Arnold & David Stern – Content and Context of Peer Assessment Louise Arnold & David Stern – Content and Context of Peer Assessment in Measuring Medical Professionalism – Edited by DT Stern – Oxford in Measuring Medical Professionalism – Edited by DT Stern – Oxford University Press 2006University Press 2006

F A C U L T Y O F M E D I C I N E

• The act of making judgments on the performance of The act of making judgments on the performance of one’s peers is ubiquitous and has formed the basis of one’s peers is ubiquitous and has formed the basis of the referral process in medicine and other professions the referral process in medicine and other professions for centuries.for centuries.

• Throughout this long history, peers have been Throughout this long history, peers have been deployed in a variety of ways to make judgments on deployed in a variety of ways to make judgments on the competence of their colleagues.the competence of their colleagues.

GENERAL COMMENTS ABOUT PA GENERAL COMMENTS ABOUT PA

F A C U L T Y O F M E D I C I N E

• Judgments about structured tasks versus global Judgments about structured tasks versus global impressions.impressions.

• Judgments about occurrence, quality or suitability.Judgments about occurrence, quality or suitability.

NATURE OF JUDGMENTSNATURE OF JUDGMENTS

F A C U L T Y O F M E D I C I N E

JUDGMENTS ABOUT JUDGMENTS ABOUT STRUCTURED STRUCTURED TASKS VERSUS GLOBAL IMPRESSIONSTASKS VERSUS GLOBAL IMPRESSIONS

• Global impressions judgments are more common than Global impressions judgments are more common than judgments about structured tasks.judgments about structured tasks.

• Each type has strengths and weaknesses – so for Each type has strengths and weaknesses – so for exampleexample

– Global impressions influenced more by halo effect. Global impressions influenced more by halo effect.

– Structured tasks are limited by the number of encounters being Structured tasks are limited by the number of encounters being assessed.assessed.

F A C U L T Y O F M E D I C I N E

JUDGMENTS ABOUT OCCURRENCE, JUDGMENTS ABOUT OCCURRENCE, QUALITY OR SUITABILITYQUALITY OR SUITABILITY

• Occurrence can be assessed by check lists.Occurrence can be assessed by check lists.

• Quality of performance is more commonly assessed Quality of performance is more commonly assessed but limited by number of observations.but limited by number of observations.

• Is the performance suitable/satisfactory – this involves Is the performance suitable/satisfactory – this involves determining if the performance was of good quality determining if the performance was of good quality and also good enough for the purpose of the and also good enough for the purpose of the evaluation. More difficult to do.evaluation. More difficult to do.

F A C U L T Y O F M E D I C I N E

ASPECTS OF COMPETENCE ASSESSEDASPECTS OF COMPETENCE ASSESSED

Technical /CognitiveTechnical /Cognitive

• Technical abilityTechnical ability

• Basic science knowledgeBasic science knowledge

• Clinical knowledgeClinical knowledge

• JudgmentJudgment

• Problem solvingProblem solving

Relationship/Non-cognitiveRelationship/Non-cognitive

• Peer relationsPeer relations

• Patient relationsPatient relations

• Reliability Reliability

• IndustryIndustry

• Personal appearancePersonal appearance

• Reaction to pressureReaction to pressure

F A C U L T Y O F M E D I C I N E

FACTORS INFLUENCING THE QUALITY FACTORS INFLUENCING THE QUALITY OF PAOF PA

• ReliabilityReliability

• RelationshipsRelationships

• StakesStakes

• EquivalenceEquivalence

F A C U L T Y O F M E D I C I N E

FACTORS INFLUENCING THE QUALITY FACTORS INFLUENCING THE QUALITY OF PA (I)OF PA (I)

ReliabilityReliability

• This is influenced by theThis is influenced by the

– Number of relevant observationsNumber of relevant observations

– Number of peers involvedNumber of peers involved

– Number of aspects of competence involvedNumber of aspects of competence involved

• Influenced little by wording of questions, number of Influenced little by wording of questions, number of points on rating scales, whether to describe all points points on rating scales, whether to describe all points on scale etc.on scale etc.

F A C U L T Y O F M E D I C I N E

FACTORS INFLUENCING THE QUALITY FACTORS INFLUENCING THE QUALITY OF PA (II)OF PA (II)

RelationshipsRelationships

• Are the peers competing with each other?Are the peers competing with each other?

• Are the rating peers friends?Are the rating peers friends?

• Are there financial relationships?Are there financial relationships?

F A C U L T Y O F M E D I C I N E

FACTORS INFLUENCING THE QUALITY FACTORS INFLUENCING THE QUALITY OF PA (III)OF PA (III)

StakesStakes

• Likely to be influenced by use to which it will be put – Likely to be influenced by use to which it will be put – with high stakes then tendency to give higher ratings. with high stakes then tendency to give higher ratings. This can be mitigated to some degree by ensuring This can be mitigated to some degree by ensuring anonymity of raters.anonymity of raters.

• Ensure that peer is being asked to rate on quality of Ensure that peer is being asked to rate on quality of performance rather than suitability for something.performance rather than suitability for something.

F A C U L T Y O F M E D I C I N E

FACTORS INFLUENCING THE QUALITY FACTORS INFLUENCING THE QUALITY OF PA (IV)OF PA (IV)

Equivalence Equivalence

This means This means

• being judged on same being judged on same activitiesactivities

• being judged by peers who being judged by peers who all have the same degree all have the same degree of stringency (need to of stringency (need to avoid hawks and doves) avoid hawks and doves)

Problems can be minimized byProblems can be minimized by

• Increasing the number of Increasing the number of peers doing the assessment peers doing the assessment

• Providing the peers with Providing the peers with clear criteria for making clear criteria for making their judgmentstheir judgments

F A C U L T Y O F M E D I C I N E

STEPS IN IMPLEMENTATIONSTEPS IN IMPLEMENTATION

• State purpose of assessment, preferably in writing.State purpose of assessment, preferably in writing.

• Assessment criteria must be developed and Assessment criteria must be developed and communicated to the participants.communicated to the participants.

• Training should be provided to all participants.Training should be provided to all participants.

• The results of the assessment should be monitored The results of the assessment should be monitored throughout the implementation process.throughout the implementation process.

• Feedback should be provided to the participants.Feedback should be provided to the participants.

F A C U L T Y O F M E D I C I N E

NOW TO MEDICAL PROFESSIONALISMNOW TO MEDICAL PROFESSIONALISM

ANDAND

PEER ASSESSMENT OF MEDICAL PEER ASSESSMENT OF MEDICAL STUDENTSSTUDENTS

F A C U L T Y O F M E D I C I N E

Why Do This?Why Do This?• Medical teachers have less time to spend in direct Medical teachers have less time to spend in direct

contact with students.contact with students.

• As a result opportunities for learners to demonstrate a As a result opportunities for learners to demonstrate a wide variety of professional and unprofessional wide variety of professional and unprofessional behaviors are more likely to occur among peers who behaviors are more likely to occur among peers who spend more time with one another and work closely spend more time with one another and work closely together as a team.together as a team.

• Professional behaviors like responsibility, effective Professional behaviors like responsibility, effective communication, interpersonal respect, thoroughness communication, interpersonal respect, thoroughness and altruism have a direct impact on peers or reflect and altruism have a direct impact on peers or reflect values that peers might be able to infer and observe in values that peers might be able to infer and observe in actions of their peers.actions of their peers.

F A C U L T Y O F M E D I C I N E

• The medical student peers are individuals who have The medical student peers are individuals who have attained the same level of training or expertise, exercise attained the same level of training or expertise, exercise no formal authority over each other, and share the same no formal authority over each other, and share the same hierarchical status in an institution.hierarchical status in an institution.

• These non-hierarchical relationships can promote both These non-hierarchical relationships can promote both authentic behavior and genuine feedback among peers authentic behavior and genuine feedback among peers while reducing the biasing influence of social desirability.while reducing the biasing influence of social desirability.

• In PA, peers are asked to judge each others’ In PA, peers are asked to judge each others’ characteristics or behaviors relevant to an evaluation characteristics or behaviors relevant to an evaluation task. task.

F A C U L T Y O F M E D I C I N E

• PA particularly useful in situations in whichPA particularly useful in situations in which

– Peers are afforded a unique view of one another’s behaviorPeers are afforded a unique view of one another’s behavior

– Peers are capable of accurately perceiving and interpreting one Peers are capable of accurately perceiving and interpreting one another’s behaviorsanother’s behaviors

– There is a need to improve the effectiveness of assessment of There is a need to improve the effectiveness of assessment of group members’ behaviorgroup members’ behavior

• Peers not only observe different behaviors but they Peers not only observe different behaviors but they also provide a different perspective on the same also provide a different perspective on the same behaviors observed by others.behaviors observed by others.

F A C U L T Y O F M E D I C I N E

Methods of PA (I)Methods of PA (I)

• Ratings – involve each member of a group rendering a Ratings – involve each member of a group rendering a judgment about other members on a specified set of judgment about other members on a specified set of behaviors, performances or characteristics using a behaviors, performances or characteristics using a scale.scale.

• Nominations – consist of group members naming a Nominations – consist of group members naming a certain number of group members as the best along a certain number of group members as the best along a particular performance dimension or quality.particular performance dimension or quality.

• Rankings – involve each group member ordering all Rankings – involve each group member ordering all other group members from best to worst on specified other group members from best to worst on specified behavioral dimensions or characteristics.behavioral dimensions or characteristics.

F A C U L T Y O F M E D I C I N E

Methods of PA (II)Methods of PA (II)

• Peer voting – assign learners a number of votes equal Peer voting – assign learners a number of votes equal to the number of group members and directs learners to the number of group members and directs learners to apportion their votes across the group members to apportion their votes across the group members according to the degree to which each member according to the degree to which each member exhibits a behavior or quality relevant to the exhibits a behavior or quality relevant to the evaluation.evaluation.

• Qualitative comments. Qualitative comments.

F A C U L T Y O F M E D I C I N E

PSYCHOMETRIC CHARACTERISTICS OF PSYCHOMETRIC CHARACTERISTICS OF DIFFERENT METHODS USED IN PADIFFERENT METHODS USED IN PA

F A C U L T Y O F M E D I C I N E

RATINGSRATINGS

• Most common method used and most supported with evidence of Most common method used and most supported with evidence of reliability and validity.reliability and validity.

• Validity includes face, content, construct, concurrent and predictive Validity includes face, content, construct, concurrent and predictive validity.validity.

• Method of choice when specific information about each group Method of choice when specific information about each group member is desired.member is desired.

• Especially useful for formative assessments which provide Especially useful for formative assessments which provide feedback.feedback.

• They are subject to problems of bias and may be less valid They are subject to problems of bias and may be less valid because they judge across the entire range of performances or because they judge across the entire range of performances or qualities.qualities.

F A C U L T Y O F M E D I C I N E

• The reaction of learners to PA and the procedures used The reaction of learners to PA and the procedures used for initiating, maintaining, and reporting peer for initiating, maintaining, and reporting peer assessments can affect the integrity of evaluations assessments can affect the integrity of evaluations reported by peers.reported by peers.

• The use to which PA results are put may influence the The use to which PA results are put may influence the peers willingness to provide genuine evaluations.peers willingness to provide genuine evaluations.

• Medical students seem to be resistant to the idea of Medical students seem to be resistant to the idea of PA.PA.

CONTEXT OF PA CONTEXT OF PA

F A C U L T Y O F M E D I C I N E

PA MOST LIKELY TO SUCCEED IF PA MOST LIKELY TO SUCCEED IF STUDENTS KNOWSTUDENTS KNOW

• How the results of it will be used.How the results of it will be used.

• That it is anonymous.That it is anonymous.

• That it is being done at an appropriate time.That it is being done at an appropriate time.

• It is mandatory.It is mandatory.

• It covers areas in which peers see themselves as It covers areas in which peers see themselves as particularly perceptive. Or in other words that the particularly perceptive. Or in other words that the content reflects the ideas and the lives of those who content reflects the ideas and the lives of those who will participate in the system.will participate in the system.

F A C U L T Y O F M E D I C I N E

PA MOST LIKELY TO SUCCEED IF IT OCCURS IN THE CONTEXT PA MOST LIKELY TO SUCCEED IF IT OCCURS IN THE CONTEXT OF A SUPPORTIVE ENVIRONMENT WHICH WOULD INCLUDE (I)OF A SUPPORTIVE ENVIRONMENT WHICH WOULD INCLUDE (I)

• School responsiveness to peer reports.School responsiveness to peer reports.

• Behavioral standards on professionalism that Faculty Behavioral standards on professionalism that Faculty consistently enforce.consistently enforce.

• Student groups whose members are amenable to teaching Student groups whose members are amenable to teaching each other and exploring professionalism issues.each other and exploring professionalism issues.

• Close relationships between students and Faculty.Close relationships between students and Faculty.

• Approachable administrators who trust students.Approachable administrators who trust students.

• Value placed on assessment aimed at learner’s Value placed on assessment aimed at learner’s improvement.improvement.

F A C U L T Y O F M E D I C I N E

• Education which explores the meaning of Education which explores the meaning of professionalism.professionalism.

• Specifies expectations for professional behavior.Specifies expectations for professional behavior.

• Provides training for feedback on professionalism and Provides training for feedback on professionalism and conflict resolution.conflict resolution.

• Highlights the importance of PA. Highlights the importance of PA.

• Provides for faculty modeling of PA. Provides for faculty modeling of PA.

PA MOST LIKELY TO SUCCEED IF IT OCCURS IN THE CONTEXT PA MOST LIKELY TO SUCCEED IF IT OCCURS IN THE CONTEXT OF A SUPPORTIVE ENVIRONMENT WHICH WOULD INCLUDE (II)OF A SUPPORTIVE ENVIRONMENT WHICH WOULD INCLUDE (II)

F A C U L T Y O F M E D I C I N E

SURVEYSURVEY

• January – sent e-mail to everyone on the AFMC January – sent e-mail to everyone on the AFMC Professional Resource Group Mailing List.Professional Resource Group Mailing List.

• No names listed from Francophone Schools, NOSM or No names listed from Francophone Schools, NOSM or Memorial on the AFMC mailing list thus no contacts Memorial on the AFMC mailing list thus no contacts were made with any of these schools.were made with any of these schools.

• Arranged telephone contacts with the people who Arranged telephone contacts with the people who responded.responded.

F A C U L T Y O F M E D I C I N E

CANADIAN EXPERIENCES WITH CANADIAN EXPERIENCES WITH MEDICAL STUDENT PAMEDICAL STUDENT PA

• University of British Columbia – Niamh Kelly – PBL.University of British Columbia – Niamh Kelly – PBL.

• University of Alberta – Lorraine Breault – Interdisciplinary HSC Course.University of Alberta – Lorraine Breault – Interdisciplinary HSC Course.

• University of Calgary – Janet Wright & Rod Crutcher – Well Physician University of Calgary – Janet Wright & Rod Crutcher – Well Physician Course.Course.

• University of Western Ontario – Peter Flanagan – PCL. University of Western Ontario – Peter Flanagan – PCL.

• University of Ottawa – Walter Hendelman – Professionalism Program.University of Ottawa – Walter Hendelman – Professionalism Program.

• Others.Others.

F A C U L T Y O F M E D I C I N E

DISCUSSION DISCUSSION

• Is peer assessment a good thing to do?Is peer assessment a good thing to do?

• If yes, why are we not doing more of it?If yes, why are we not doing more of it?

• What could this Resource Group do to move PA forward What could this Resource Group do to move PA forward in our schools?in our schools?

• Interest in making peer assessment of medical Interest in making peer assessment of medical professionalism a focus of this resource group?professionalism a focus of this resource group?

• Other issues?Other issues?