Evaluation of Peer Assessment in Teaching Patient ...nus.edu.sg/cdtl/docs/default-source/engagement-docs/...Evaluation of Peer Assessment in Teaching Patient Presentation Skills HAN

Evaluation of Peer Assessment in Teaching Patient Presentation Skills

HAN Zhe, PharmD, BCPS, BCIDPLecturer

Department of Pharmacy, Faculty of [email protected]

Higher Education Campus ConferenceCenter for Development of Teaching & Learning, Piaget Room

November 18, 2019

NUS Presentation Title 2001

Presentation Outline

Outline the use of peer assessment in teaching patient presentation skills in an undergraduate pharmacy skills module

Discuss pharmacy students’ perceptions of peer assessment in learning patient presentation skills

Describe pharmacy students’ ability to assess peers’ patient presentations

2


Patient Presentation Skills CORE Pharmacy Skill

Communicate or present patient information, therapeutic plans and recommendations to colleagues and other healthcare providers1

American College of Pharmacy Education (ACPE)2

Accreditation standards

International Pharmaceutical Federation (FIP)3

Statement on quality pharmacy and pharmaceutical sciences education

Singapore Pharmacy Council (SPC)4

Competency standards framework

1. American College of Clinical Pharmacy (ACCP). Pharmacotherapy 2014; 34(8): 794-72. ACPE. Available at: https://www.acpe-accredit.org/pdf/Standards2016FINAL.pdf. Accessed September 10, 20193. FIP. Available at: https://www.fip.org/files/fip/PharmacyEducation/Global_Conference_docs/Nanjing_Statements.pdf. Accessed September 10, 20194. SPC. Available at: https://www.healthprofessionals.gov.sg/docs/librariesprovider3/standards-exams/competency-standards-for-singapore-pharmacists-(march-2010).pdf. Accessed September 19, 2019


Peer Assessment

One of the most important facilitators of positive learning outcomes in higher education1

Offers numerous benefits1,2

- Self-reflection

- Higher level thinking

- Peer interaction

- Responsibility

- Reduces faculty workload

41. Schneider M, et al. Psychol Bull 2017; 143(6): 565-6002. Dochy F, et al. Study Higher Educ 1999; 24(3): 331-350


Peer Assessment in Pharmacy Education

5

Citation Setting, Activity, Students Key Findings

Wu K, et al. Am J Pharm Edu 2012; 76(4): Article 62

Drug information and literature evaluation course

Development of drug monograph 2nd year PharmD students (N = 158)

Most students agreed that peer assessment was a useful skill

Most students were comfortable receiving feedback from peers

Bartelme KM. Inov Pharm 2016; 7(1): Article 12

Applied patient care course series Evaluation of clinical documentation

(SOAP note) 2nd year PharmD students (N = 159)

Majority of errors accurately identified by students


Peer Assessment and Patient Presentations

61. Schneider M, et al. Psychol Bull 2017; 143(6): 565-6002. Dochy F, et al. Study Higher Educ 1999; 24(3): 331-350

Benefits of peer assessment1,2

Self-reflection Higher level thinking Peer interaction Responsibility Reduces faculty workload

Patient presentation skills: Integral to most pharmacy curriculum Opportunities for practice and feedback

may be limited by class size

Gaps in Knowledge Peer assessment in teaching patient

presentation skills in the classroom Students’ perceptions Students’ ability to conduct peer assessment


Evaluation of Peer Assessment in Teaching Patient Presentation Skills HAN Zhe, Department of Pharmacy, Faculty of Science


Study Design and Setting

Cross-sectional study

Approved by NUS Institution Review Board (S-18-059E)

Conducted at the Department of Pharmacy

- Four-year Bachelor of Science (Pharmacy) degree

8

Years 1 to 3 Year 4

Didactic coursework: anatomy, physiology, pharmaceutical sciences, pharmacology, therapeutics,

law, pharmacy professional skills

Research (final year project), experiential learning


Study Context

Pharmacy Professional Skills Development (PPSD)

- Sequence of 3 required modules, each in years 1 to 3

- Students develop skills such as dispensing, counseling and inter-professional communication

9

PPSD III: year 3 semester 2 Case-based practical sessions using 1:1 role play Apply pharmacy skills while integrating therapeutic knowledge Patient presentation skills first introduced in AY2017/18


Study Context

Conducted in AY2018/19 semester 2 (January – April 2019)

- 178 students enrolled in PPSD III

Patient presentation skills taught over 4 sessions

10

Session 4Session 1 Session 2 Session 3 Teaching and Learning Activities in AY2018/19

N = 178 students Week 5 Week 5 Week 7 Weeks 10 to 11

Didactic lecture Structure and components of patient presentations Illustrated by patient case throughout Grading rubric and SOAP note template provided


Study Context

11



Patient Presentation Peer Assessment Workshop Played a pre-recorded patient presentation based on

example patient case in session 1 Students assessed presentation using a grading

rubric

Generic Grading Rubric for Assessing Patient Presentations


Study Context

12



In-Class Discussion and Peer Assessment Group work: formulate therapeutic plan Individual work: audio record and upload patient presentation

(10-minute; MP3 or M4A file) In-class: presentation and discussion with faculty over 4 sessions Take home assignment: anonymous peer assessment of a

classmate’s patient presentation done electronically (1 week) Marks and qualitative comments from peer assessment returned

to students


Study Context

13

In-Class Patient Presentation One-on-one patient presentation with faculty or teaching

assistants (TAs) (i.e. post-graduate students with pharmacy practice experience)

In-class: 60 minutes preparation, 10 minutes presentation (graded by faculty or TAs and audio recorded), 15 minutes debriefing

Take-home assignment: anonymous peer assessment of audio-recorded presentation done electronically (1 week)

AY2018/19: 3 faculty & 6 TAs contributed a total of 128 teaching hours over 2 weeks




Study Objectives

Primary objective

- Evaluate pharmacy students’ perceptions of using peer assessment as a pedagogical strategy in learning patient presentation skills

Secondary objective

- Evaluate pharmacy students’ ability to assess peers’ patient presentations

14


Study Methodology

Anonymous self-administered survey to assess pharmacy students’ perceptions

- Administered after the last graded practical

- 4 sections, 29 items

- 5-point Likert scale, yes/no, open-ended questions

- Voluntary participation

Compared marks from faculty or TAs versus marks from peer assessment in session 4

15


Study Methodology

Survey responses

- Quantitative• Descriptive statistics

• Perceptions compared using McNemar’s test

- Qualitative survey responses• Reviewed to identify major themes

Patient presentation marks

- Compared using paired student’s t-test

STATA® version 14.0 (College Station, Texas, USA)

A priori alpha < 0.05

16


Results: Perceptions of Peer Assessment

A total of 161 students returned the survey (response rate: 90.4%)

17

93.1%

6.9%

86.3%

13.7%



18

StatementAgreea

N (%)NeutralN (%)

Disagreeb

N (%)

I am comfortable providing an honest assessment of my classmate’spatient presentation.

147 (91.3%) 13 (8.1%) 1 (0.6%)

I am comfortable receiving an assessment of my patient presentationfrom a classmate.

148 (91.9%) 13 (8.1%) 0 (0%)

I received constructive comments from my classmate on my patient presentation.

124 (77.0%) 29 (18.0%) 8 (5.0%)

My classmate’s comments helped me improve my patient presentation skills.

108 (67.1%) 49 (30.4%) 4 (2.5%)

aAgree = strongly agree + agreebDisagree = strongly disagree + disagree

Table 1. Students’ Perceptions of Peer Assessment in Learning Patient Presentation Skills (cont’d) (N = 161)



19

StatementAgreea

N (%)NeutralN (%)

Disagreeb

N (%)

I believe a classmate will provide an honest assessment of my patient presentation.

145 (90.1%) 14 (8.7%) 2 (1.2%)

I believe my classmate has the necessary skills to assess my patientpresentation.

123 (76.4%) 36 (23.4%) 2 (1.2%)

I believe I have the necessary skills to assess my classmate’s patientpresentation.

110 (68.3%) 46 (28.6%) 5 (3.1%)

Table 1. Students’ Perceptions of Peer Assessment in Learning Patient Presentation Skills (cont’d) (N = 161)

aAgree = strongly agree + agreebDisagree = strongly disagree + disagree

90.1% versus 76.4%, p < 0.01

76.4% versus 68.3%, p < 0.01



20

“Clearly pointed out the mistakes and provided some appropriate comments on how to rectify the errors.”

Figure 1. Qualitative Comments on Students’ Perceptions of Peer Assessment

“I realized that I missed out on quite a number of information for my own patient presentation while I

was going through my peer’s presentation.”

“Peer assessment I received was merely a repetition of what was stated in the rubric, hence I felt peer assessment did not value-add”

Would have been more helpful if more detailed qualitative comments”


Results: Ability to Assess Peers

21



Graded by faculty or TAs in class(simultaneous audio recording)

Anonymous peer assessment after class(Using audio recording)

Same grading rubric

StatementFaculty or TAs

(%)Peer Assessment

(%)p value

Patient Presentation Marks – Overall 72.0 61.0 < 0.01

Table 2. Comparison of Patient Presentation Marks Assigned by Faculty or TAs versus via Peer Assessment (N = 110)


Results: Ability to Assess Peers

22

Pati

ent

Pre

sen

tati

on

Mar

ks (

%)

p < 0.01 p = 0.09

Figure 2. Comparison of Patient Presentation Marks Assigned by Faculty or TAs versus via Peer Assessment (N = 110)


Discussion

Described our experiences with implementing peer assessment in teaching patient presentation skills

- Unique from other studies • Classroom setting

• Large class of undergraduate students

- Facilitated by technology• Mock electronic medical record

• Learning management system

23


Discussion: Perceptions of Peer Assessment

Peer assessment perceived positively by students

- Comfortable providing comments to (91.3%) and receiving comments from (91.9%) peers

Wu K, et al. evaluated peer assessment of drug monographs1

- Comfortable receiving comments from peers (95.7%)

- Less comfortable providing comments to peers (80.0%)• Differences in anonymity which had been identified as a positive facilitator of honest peer assessments2

241. Wu K, et al. Am J Pharm Edu 2012; 76(4): Article 622. Basheti IA, et al. Am J Pharm Edu 2010; 74(5): Article 77


Discussion: Perceptions of Peer Assessment

More confident in peers’ skills that in their own skills in assessing patient presentations

- Patient presentation was first introduced into the curriculum

- Most students with limited exposure

- Reflected generally tendency toward leniency with peers1

251. Wagner ML, et al. Am J Pharm Edu 2011; 75(7): Article 130


Discussion: Utility of Peer Assessment

Encourages critical self-evaluation of one’s own performance

Allows students’ mistakes to be identified

- Bartelme KM, et al. evaluated peer assessment of clinical documentation1

• With appropriate training, students identified 87.0% of errors in clinical documentation

Quality of feedback may be variable

261. Bartelme KM, et al. Inov Pharm 2016; 7(1). doi: https://doi.org/10.24926/iip.v7i1.422

Improvement plan: additional training, emphasize need for detailed and specific comments on how to

improve, encourage student accountability


Discussion: Ability to Assess

Patient presentation marks assigned via peer assessment were significantly lower than that assigned by faculty or TAs

Contrary to leniency with peers1 and to previous studies on peer assessment

27

Citation Peer Assessment Activity Key Finding

Wagner ML, et al. Am J Pharm Edu 2011; 75(7): Article 130

Evaluation of seminar presentations

Peer assessment grades were higher than faculty-assignedgrades

Basheti IA, et al. Am J Pharm Edu 2010; 74(5): Article 77

Evaluation of medication management reviews

Peer assessment grades were higher than faculty-assignedgrades

Storjohann T, et al.Am J Pharm Edu 2015; 79(8): Article 123

Evaluation of SOAP notes Comparable peer-assigned and

faculty-assigned scores

1. Wagner ML, et al. Am J Pharm Edu 2011; 75(7): Article 130



Peers were more stringent in assessing presentation style

- Unable to appropriately assess acceptable alternatives• “Every 24 hours” versus “once a day”

- Subjective criterion (e.g. professionalism)• Penalized for not referring to patient as madam, use of fillers (e.g. hmm…), having a cough, etc.

28

Improvement plan: provide more examples, allow clarifications through online forum


Study Limitations

All participants from the same cohort

Unable to eliminate or quantify chances that a student might recognize the voice of the peer he/she was assessing

Limited qualitative comments

- No thorough understanding of rationale for perceptions

Students with limited baseline knowledge of patient presentation skills

- Mature students may provide higher quality comments

29


Take-Home Messages

Peer assessment is well-received by pharmacy students

Peer assessment is a useful pedagogical strategy in teaching patient presentation skills

- Promotes self-reflection

- Allows individual feedback in a large class

Students may benefit from more guidance to appropriately assess presentation style and to improve quality of comments

30

<Evaluation of Peer Assessment in Teaching Patient Presentation Skills>

HAN Zhe, PharmD, BCPS, BCIDPLecturer

Department of Pharmacy, Faculty of [email protected]

Questions?


Teaching Staff

3 faculty members in the pharmacy practice domain

- Full-time or adjunct lecturers at NUS

- Education: Doctor of Pharmacy (PharmD)

- Practicing clinical pharmacists in the acute-care setting

- At least 8 years of practice experience

6 TAs

- Post-graduate pharmacy students

- At least 3 years of practice experience in acute-care or community settings

32

Session 4: Presentation content and therapeutic plan were discussed and harmonized among all teaching staff


Anonymous Peer Assessment

33

Students used unique codes to name their patient presentations and to identify that of their peers

Audio files of patient presentations uploaded and retrieved from NUS virtual learning environment



34

Grading rubric with answer key provided to students through NUS virtual learning environment



35

Comments from Peer Assessment (session 3) disseminated for student review via NUS virtual learning environment



36



In-class presentation followed by peer

assessment Non-Graded

One-on-one patient presentation

Graded

Most students perceived sessions 3 and 4 as equally useful (95.0%) but

few were comfortable with including peer assessment marks into their

overall module grade (23.6%)


Results: Facilitators of Peer Assessment

37

FactorVery Helpful

N (%)Helpful N (%)

UndecidedN (%)

Somewhat Helpful (N (%)

Not HelpfulN (%)

Making the process anonymous

64 (39.8) 83 (51.6) 11 (6.8) 3 (1.9) 0 (0)

Reviewing examples of effective peer assessments

63 (39.1) 85 (52.8) 9 (5.6) 4 (2.5) 0 (0)

Discussing patient cases in class beforehand with facultya

55 (34.4) 96 (60.0) 4 (2.5) 5 (3.1) 0 (0)

Receiving instructions from faculty

48 (29.8) 99 (61.5) 8 (5.0) 3 (1.9) 3 (1.9)

Being friends with my classmates

28 (17.4) 69 (42.9) 44 (27.3) 7 (4.3) 13 (8.1)

Reading articles on peer assessment

11 (6.8) 69 (42.9) 56 (34.8) 12 (7.5) 13 (8.1)

Table 3. Students’ Perceptions of Factor that Facilitated or Would Have Facilitated Peer Assessment

aOne missing response; analysed N = 160


Results: Students’ Perceptions of Faculty & TAs

38

StatementAgreea

N (%)NeutralN (%)

Disagreeb

N (%)

I believe faculty or TAs have the necessary skills to assess my patientpresentation.c 154 (96.3) 6 (3.8) 0 (0)

I am comfortable receiving an assessment of my patient presentationfrom faculty or TAs.c 155 (96.9) 5 (3.1) 0 (0)

I believe faculty or TAs will provide an honest assessment of my patient presentation.c 159 (99.4) 1 (0.6) 0 (0)

I received constructive comments from faculty or TAs on my patient presentation.c 154 (96.3) 3 (1.9) 3 (1.9)

Faculty or TAs’ comments helped me improve my patient presentation skills.c 147 (91.9) 13 (8.1) 0 (0)

Table 4. Students’ Perceptions of Peer Assessment in Learning Patient Presentation Skills

aAgree = strongly agree + agreebDisagree = strongly disagree + disagreecOne missing response; analysed N = 160


Results: Peer Assessment versus Faculty or TAs

39

StatementPeer

AssessmentN (%)

Faculty or TAsN (%)

p value

Assessor had necessary skills to assess my patient presentation123 (76.4%) 154 (96.3)a < 0.01

Comfortable receiving feedback from the assessor148 (91.9%) 155 (96.9)a 0.02

Received constructive feedback from the assessor124 (77.0%) 154 (96.3)a < 0.01

Assessor’s comments helped me improve my patient presentation skills 108 (67.1%) 147 (91.9)a < 0.01

Table 5. Students’ Perceptions of Peer Assessment in Learning Patient Presentation Skills

aOne missing response; analysed N = 160



Marks for therapeutic plan were comparable between faculty or TAs versus peer assessment

- Less ambiguity in assessing content, i.e. identification drug-related problems, treatment options, doses and monitoring parameters

- Answers in grading rubric

40

Documents

Evaluation of Peer Assessment in Teaching Patient ...nus.edu.sg/cdtl/docs/default-source/engagement-docs/...Evaluation of Peer Assessment in Teaching Patient Presentation Skills HAN