665.full

May 2011 Journal of Dental Education 665

Evaluation of a New Assessment Tool in Problem-Based Learning Tutorials in Dental EducationVeerasathpurush Allareddy, B.D.S., Ph.D.; Aaron M. Havens, D.M.D.; T. Howard Howell, D.D.S.; Nadeem Y. Karimbux, D.M.D., M.M.Sc.Abstract: The Harvard School of Dental Medicine (HSDM) uses a hybrid problem-based learning (PBL) approach to teach-ing in the D.M.D. curriculum. Each tutorial group at HSDM is made up of eight to ten students. The objective of this study was to examine the performance of the students in the tutorial sessions as assessed by the tutor. A total of ten tutorial blocks led by twenty-four tutorial leaders were completed at HSDM between summer 2008 and fall 2009. All of the tutors were calibrated to a new assessment system that was developed for student assessment in the tutorials. The students were assessed on three major domains: knowledge acquisition, problem-solving and analytical thinking skills, and personal and interpersonal development. The tutors evaluated the students assigned to their group after the end of each block. Students also filled out a self-assessment form. Simple descriptive statistics were used to summarize the evaluations of the tutors. A total of 290 student evaluations from ten tutorial blocks were available for analysis. Tutors reported that 64 percent of the students preferred to communicate verbally without prompting and 68 percent of students submitted written reports to the groups during tutorial sessions. Tutors reported that a majority of students brought new information to each tutorial session (86.5 percent), brought information that facilitated others learning (88.7 percent), integrated newly acquired knowledge with previous knowledge (92.7 percent), applied knowledge from self-study to explain issues during case discussions (91.7 percent), asked appropriate questions to stimulate discussions (87.3 percent), generated hypotheses to explain problems under discussion (83.8 percent), evaluated the hypotheses in light of avail-able evidence (85.5 percent), defined and took responsibility for learning goals and objectives (89.3 percent), responded well to criticism (91.7 percent), took a leadership role (74.5 percent), and demonstrated sensitivity to psychosocial issues (88.6 percent). Student communication that tended to take over the group process in a non-contributory manner was reported in only 2 percent of the evaluations. Overall, students participating in PBL tutorial sessions appear to exhibit problem-solving and analytical thinking skills and personal and interpersonal attributes.

Dr. Allareddy is Instructor in Developmental Biology, Harvard School of Dental Medicine; Dr. Havens was a D.M.D. student at Harvard School of Dental Medicine during the conduct of this study; Dr. Howell is Dean for Dental Education, Harvard School of Dental Medicine; and Dr. Karimbux is Assistant Dean for Dental Education, Harvard School of Dental Medicine. Direct cor-respondence and requests for reprints to Dr. Veerasathpurush Allareddy, Department of Developmental Biology, Harvard School of Dental Medicine, 188 Longwood Avenue, Boston, MA 02115; 216-571-1009; [email protected].

Keywords: dental education, problem-based learning, hybrid PBL, assessment

Submitted for publication 7/22/10; accepted 11/11/10

The Harvard School of Dental Medicine (HSDM) uses a hybrid problem-based learn-ing (PBL) approach to teaching in the Doc-tor of Dental Medicine (D.M.D.) program. During the first two years of the program, students study preclinical sciences along with Harvard Medical School students. Clinical training begins in the third year, and all patient care is provided in the teaching practice clinic at Harvard Dental Center. During the fourth year of the program, students also provide care at Harvard-affiliated centers.

In a typical PBL tutorial at HSDM, the dental students are divided into small groups of six to eight students with a tutor (facilitator) who moderates the group. Previous studies have demonstrated that this group size facilitates participation by all group mem-bers and also prevents segregation of students into

subsets.1 The students in each group are expected to identify problems/scenarios in a case-based format, generate hypotheses, and finally test their ideas after acquiring sufficient knowledge about the problem. At the end of each course, the tutor assesses each student on a variety of abilities along with a grade (honors/pass/fail). The assessment typically involves the students knowledge acquisition, problem-solving and critical thinking, and personal and interpersonal skills. The tutors play a major role in the success of the PBL curriculum by facilitating student discus-sions and by encouraging self-directed learning.2,3 In some PBL settings, the final grades given by the tutors can directly affect the students overall grade; therefore, the grades given by these facilitators should be as accurate and as representative as possible. All of the tutors assigned to facilitate the tutorial groups

666 Journal of Dental Education Volume 75, Number 5

of a given course should be trained and calibrated at the beginning of each block. The criteria for assign-ing grades and the measures for evaluating students performance in tutorial are discussed with the course director. The tutors also periodically update the course director on students performance.

The skills assessed in tutorials at HSDM in-clude knowledge acquisition, problem-solving and analytical skills, and personal and interpersonal development. Since 1994, HSDM has used an as-sessment system that is descriptive in nature and feedback-driven. A review showed that most tutors were subjective in their assessment and actually cut and pasted comments in several instances, undermin-ing the individual performance of students. Conse-quently, HSDM developed a new objective system for grading students in the tutorial sessions. The objec-tive of this study was to examine the performance of students in tutorial sessions as assessed by the tutors at HSDM using this new assessment method.

Materials and MethodsThe study was designed to be a prospective

longitudinal analysis of tutor evaluations of current D.M.D. students (classes of 2010 through 2013) at HSDM. The Institutional Review Board at the Harvard Medical School/Harvard School of Dental Medicine Committee on Human Studies approved the conduct of the study.

The research was performed in two stages. In the first stage, a new tutorial evaluation was devel-oped based on a review of literature describing the core skills developed in a PBL curriculum.4 This assessment form was pilot tested in one tutorial block, and slight modifications were made based on feedback to capture the performance of students in the tutorial sessions. In the second stage, the tutorial evaluations for ten blocks (across years 1, 2, and 3 of the D.M.D. program) were obtained from the Office of Dental Education, and an office assistant linked up the student evaluation sheets with the demographic data and masked personal information. The names of all students were deidentified, and any unique identifiers were blinded to all researchers to preserve confidentiality. Following this, the data were made available to the researchers in an Excel spreadsheet; they converted this data to SPSS and SAS data sets for conducting statistical analyses.

The students were evaluated in four tutorial ses-sions per ten tutorial blocks given across the first three

years of the D.M.D. program at HSDM from summer 2008 through fall 2009. A total of twenty-four tutors covered the forty tutorial sessions. Simple descriptive statistics were used to summarize the evaluations of the tutors. The evaluation form used to assess student performance appears in the Appendix.

Prior to the commencement of each course block at HSDM, all tutors meet with the course director and the assistant dean for dental education. During the meeting, the course director and assistant dean for dental education provide a brief overview of the course block, the grading system to be used to finalize the cumulative grade for students, cases to be discussed during the tutorial sessions, and grading criteria for the tutorials. Grades for the tutorials are based on a multitude of factors including participa-tion of students in terms of quantity and quality of comments, a students ability to pave the way for meaningful discussions, attendance, resources used, and critical thinking abilities. Tutors are also encour-aged to meet with the course director and assistant dean for dental education during the duration of the course block if any issues arise during the conduct of tutorial blocks. Periodic meetings are held by the course director with all tutors to discuss the progress of the tutorial sessions.

ResultsA total of 290 student evaluations from ten

tutorial blocks were available for analysis. Of these 290 evaluations, seventy-eight students (26.9 per-cent) obtained an Honors grade on the tutorials. On a scale of 0 to 100 (with 0 being no participation and 100 being fully participated) the mean score for tutorial participation was 89 (standard deviation of 13, median of 90, minimum of 20, and maximum of 100). Overall, 64 percent of the students preferred to communicate verbally without prompting, and 68 percent of the students submitted written reports to the groups during tutorial sessions.

Seven attributes were evaluated for the Knowl-edge Acquisition domain (Table 1). The tutors agreed or strongly agreed that 86.5 percent of the students brought new information to each tutorial session, 90.4 percent brought relevant information to each session, 88.7 percent brought information that facili-tated others learning, 92.7 percent integrated newly acquired knowledge with previous knowledge, and 91.7 percent applied knowledge from self-study to explain issues during case discussions. A majority


(91 percent) of the students also were assessed to have used appropriate language in the field, and 80.6 percent of the students ideas and information were assessed to be correct.

Seven attributes of students were evaluated in the Problem-Solving and Analytical Thinking Skills domain (Table 2). Tutors agreed or strongly agreed that 92.1 percent of students reported information in their own words, 95.2 percent presented information that was clear and understandable, 80.7 percent clari-fied difficult issues, 87.3 percent asked appropriate questions to stimulate discussions, 83.8 percent gen-erated hypotheses to explain problems under discus-sion, 85.5 percent evaluated the hypotheses in light of available evidence, and 89.3 percent defined and took responsibility for learning goals and objectives.

Ten attributes were evaluated in the Personal and Interpersonal Development domain (Table 3). The tutors agreed or strongly agreed that 85.9 percent of the students were always on time to each session, 91.1 percent attended every session, 91.7 percent re-sponded well to criticism by others, 90.7 percent were engaged or interested in each session, 74.5 percent took a leadership role, 95.9 percent listened care-

fully with understanding, 88.6 percent demonstrated sensitivity to psychosocial issues, and 83.1 percent assessed their own performance accurately. Student communication that tended to take over the group process in a non-contributory manner was reported in only 2 percent of the evaluations. Students inter-rupting others with comments was reported in only 1 percent of the evaluations.

DiscussionOur study provides an overview of student

performance in tutorials in a hybrid-PBL curriculum at HSDM using a newly developed objective driven assessment form. Overall, students participating in the PBL sessions appear to gain skills expected of these sessions (knowledge acquisition, problem-solving and analytical thinking skills, and positive personal and interpersonal attributes). Evidence to date has supported that PBL has a positive effect on critical thinking, problem-solving skills, lifelong self-directed learning in medical and dental students, and better preparing students to work in groups.5-13 Our study results reinforce this notion. Norman and

Table 1. Tutor evaluations of seven attributes regarding students knowledge acquisition

Strongly Strongly Agree Agree Neutral Disagree Disagree

Studentbroughtnewinformationtoeachsession. 47.2% 39.3% 12.4% 1.0% 0Studentbroughtrelevantinformationtoeachsession. 50.7% 39.7% 8.6% 1.0% 0Studentbroughtinformationthatfacilitatedotherslearning. 52.8% 35.9% 10.3% 1.0% 0Studentsideas/informationwerealwayscorrect. 20.3% 60.3% 18.6% 0.7% 0Studentusedappropriatelanguageinthefield. 44.8% 46.2% 9.0% 0 0Studentintegratednewlyacquiredknowledgewithpreviousknowledge. 54.1% 38.6% 6.9% 0.3% 0Studentappliedknowledgefromself-studytoexplainissuesincase. 56.2% 35.5% 7.9% 0.3% 0

Note: Percentagesmaynottotal100%becauseofrounding.Totalnumberofevaluationsanalyzedwas290.

Table 2. Tutor evaluations of seven attributes regarding students problem-solving and analytical thinking skills


Studentreportedinformationinhis/herownwordsratherthannotes. 49.0% 43.1% 7.2% 0.3% 0.3%Studentpresentedinformationthatwasclearandunderstandable. 52.8% 42.4% 4.1% 0.3% 0.3%Studentclarifieddifficultissues. 43.1% 37.6% 18.3% 1.0% 0Studentaskedappropriatequestionstostimulatediscussions. 50.7% 36.6% 11.7% 1.0% 0Studentgeneratedhypothesestoexplainproblemsunderdiscussion. 37.2% 46.6% 14.8% 1.4% 0Studentevaluatedhypothesesinlightofavailableevidence. 40.0% 45.5% 13.4% 1.0% 0Studentdefinedandtookresponsibilityforlearninggoalsandobjectives. 54.5% 34.8% 10.0% 0.7% 0

Note: Percentagesmaynottotal100%becauseofrounding.Totalnumberofevaluationsanalyzedwas290.


Schmidt conducted a systematic review of studies examining the psychological effects of a PBL cur-riculum.9 Their study concluded that learning in a PBL format increased retention of knowledge over several years, facilitated transfer of concepts to new problems, enhanced integration of basic science concepts into clinical problems, enhanced students interest in subject matter, and improved self-directed learning skills. Three other systematic reviews including a meta analysis by Vernon and Blake13 ex-amined the theoretically purported benefits of PBL. The data showed that students trained in a PBL cur-ricula are well prepared for self-directed independent learning, engaged in constructive and collaborative team efforts, and engaged in contextual learning.13-15 There is also evidence that PBL education is associ-ated with developing professional competencies.16 Schmidt and van der Molen conducted a study that examined the self-reported competencies of alumni of Maastricht University School of Medicine that had a problem-based curriculum.16 The study results indicate that students trained in a problem-based cur-riculum rated themselves as being better on problem-solving skills, interpersonal skills, and ability to work independently when compared to those trained in a traditional educational curriculum. The results of our study further reinforce that a majority of students not only acquire core knowledge of the subjects during the tutorial blocks at HSDM but also perform well on attributes used to evaluate current problem-solving and analytical skills.

A problem-based learning curriculum format has been shown to instill communication and critical thinking skills. In a recent report, the performance

of postdoctoral residents at HSDM who underwent a PBL curriculum were compared with non-HSDM residents who underwent a traditional curriculum.12 The results indicated HSDM residents tend to perform significantly better than their non-HSDM counterparts with regards to critical thinking abilities, communication with patients, independent learning, teamwork, and performance in small-group settings. There were also no statistically significant differences between HSDM and non-HSDM postdoctoral resi-dents regarding general dental knowledge, specialty-specific knowledge, preclinical and clinical skills, communication with staff, and patient education. This study further underscores the importance of PBL-based education in developing competencies that enable students to develop good communication skills and become lifelong thinkers and learners.

Further studies conducted to test the efficacy of a PBL curriculum compared general competencies between medical graduates of PBL and non-PBL schools.10 The investigators found that students trained in a PBL setting were better prepared in competencies such as acquiring expert knowledge, profession-specific skills, communication skills, and teamwork when compared to their non-PBL-trained counterparts. These data continue to illustrate the efficacy a PBL approach has on education. A recent study conducted compared the longitudinal effects of PBL and traditional curricula among medical students at the University of Groningen.6 Outcomes compared included self-assessed general competen-cies, clinical competencies, and indicators of career development. These data suggested that students who underwent a PBL curriculum scored higher

Table 3. Tutor evaluations of ten attributes regarding students personal and interpersonal development


Studentcommunicationstendedtotakeoverthegroupprocessin 0 2.0% 3.1% 17.0% 78.9%anon-contributorymanner.(N=289)Studentinterruptedotherswithcomments.(N=289) 0 1.0% 1.7% 16.6% 80.6%Studentwasalwaysontimetoeachsession.(N=290) 59.3% 26.6% 8.3% 4.5% 1.4%Studentattendedeverysession.(N=290) 72.8% 18.3% 3.1% 5.2% 0.7%Studentrespondedwelltocriticismbyothers.(N=287) 66.6% 25.1% 8.0% 0.3% 0Studentwasengaged/interestedineachsession.(N=290) 66.6% 24.1% 6.9% 2.4% 0Studentprovidedleadership.(N=290) 39.0% 35.5% 21.0% 3.4% 1.0%Studentlistenedcarefullywithunderstanding.(N=290) 66.9% 29.0% 3.8% 0.3% 0Studentdemonstratedsensitivitytopsychosocialissues.(N=290) 59.3% 29.3% 8.3% 0.7% 2.4%Studentassessedownperformanceaccurately.(N=267) 48.3% 34.8% 16.9% 0 0

Note: Percentagesmaynottotal100%becauseofrounding.Numberofevaluationsanalyzedvariedbystatementfrom267to290.


on general competencies when compared to those trained in a traditional curriculum. Interestingly, these studies illustrate the benefit of PBL-based education by not only improved communication skills, but by also providing a stronger foundation in competen-cies. Our study illustrates the positive impact PBL-directed learning has on fostering skills in key areas of education. There is compelling evidence from our study that PBL improves certain skills that form the theoretical basis of a PBL framework.

This study is subject to several limitations. First, it was conducted at a single institution, and consequently the results cannot be generalized to other schools that have a problem-based learning curriculum. The tutorial grades and evaluations were assigned by the tutorial leaders, and twenty tutors participated as tutorial leaders for ten blocks and forty tutorial sessions. Even though the tutors were calibrated prior to the implementation of the new grading system, there is likely to be some out of clus-tering effects of grading patterns within tutors, and this study cannot account for them. A major drawback of the study is that there is no control group and we cannot determine how the students would have fared in a traditional curriculum.

ConclusionsThis study suggests that a majority of students

participating in problem-based learning tutorial ses-sions demonstrate knowledge acquisition, problem-solving and analytical thinking skills, and personal and interpersonal attributes. In keeping with the current trend of the literature, a PBL-enhanced curriculum illustrates enhanced skill development pivotal in educating independent critical thinkers. This new assessment system is more objective in allowing the tutors to assess these skills.

REFERENCES1. Lucero S, Jackson R, Galey W, eds. Groups in problem-

based learning. New York: Springer, 1985.2. Barrows HS. The essentials of problem-based learning.

J Dent Educ 1998;62(9):6303.3. Mast TA, Coulson LR, Meyer TC, Nuenke R, Barrows HS,

Fox RD, et al. Self-directed learning: wisdom from inde-pendent study programs. Res Med Educ 1985;24:30511.

4. Kassab S, Abu-Hijleh M, Al-Shboul Q, Hamdy H. Gender-related differences in learning in student-led PBL tutorials. Educ Health (Abingdon, England) 2005;18(2):27282.

5. Beachey WD. A comparison of problem-based learning and traditional curricula in baccalaureate respiratory therapy education. Respir Care 2007;52(11):1497506.

6. Cohen-Schotanus J, Muijtjens AM, Schonrock-Adema J, Geertsma J, van der Vleuten CP. Effects of conventional and problem-based learning on clinical and general com-petencies and career development. Med Educ 2008;42(3): 25665.

7. Koh GC, Khoo HE, Wong ML, Koh D. The effects of prob-lem-based learning during medical school on physician competency: a systematic review. CMAJ 2008;178(1): 3441.

8. Moore GT, Block SD, Style CB, Mitchell R. The influ-ence of the New Pathway curriculum on Harvard medical students. Acad Med 1994;69(12):9839.

9. Norman GR, Schmidt HG. The psychological basis of problem-based learning: a review of the evidence. Acad Med 1992;67(9):55765.

10. Prince KJ, van Eijs PW, Boshuizen HP, van der Vleuten CP, Scherpbier AJ. General competencies of problem-based learning (PBL) and non-PBL graduates. Med Educ 2005; 39(4):394401.

11. Susarla SM, Medina-Martinez N, Howell TH, Karimbux NY. Problem-based learning: effects on standard out-comes. J Dent Educ 2003;67(9):100310.

12. Thammasitboon K, Sukotjo C, Howell H, Karimbux N. Problem-based learning at the Harvard School of Dental Medicine: self-assessment of performance in postdoctoral training. J Dent Educ 2007;71(8):10809.

13. Vernon DT, Blake RL. Does problem-based learning work? A meta-analysis of evaluative research. Acad Med 1993; 68(7):55063.

14. Albanese MA, Mitchell S. Problem-based learning: a review of literature on its outcomes and implementation issues. Acad Med 1993;68(1):5281.

15. Berkson L. Problem-based learning: have the expectations been met? Acad Med 1993;68(10 Suppl):S7988.

16. Schmidt HG, van der Molen HT. Self-reported compe-tency ratings of graduates of a problem-based medical curriculum. Acad Med 2001;76(5):4668.


APPENDIX

Tutorial Evaluation Report/Tutor Evaluation

Tutor Name Student Name___________________________________________________________________________ Course ________________________________________________________________________________ 1. Please rate students participation in tutorial: 2. Student prefers to communicate: A) Verbally without prompting B) Verbally when prompted by the tutor C) Verbally when prompted by students D) Other _____________________________ 3. Student non-verbal communications to tutorial are: A) Submissions of written reports to the group B) Writing/sketches on the board C) Other _____________________________ Please assess the students performance in tutorial: (SA=strongly agree; A=agree; N=neutral; D=disagree; SD=strongly disagree)

Knowledge Acquisition 1=SA 2=A 3=N 4=D 5=SD

Student brought new information to each session.

Student brought relevant information to each session. Student brought information that facilitated others learning. Students ideas/information were always correct. Student used appropriate language in the field. Student integrated newly acquired knowledge with previous knowledge. Student applied knowledge from self-study to explain issues in case.

Problem-Solving and Analytical Thinking Skills 1=SA 2=A 3=N 4=D 5=SD

Student reported information in his/her own words rather than notes.

Student presented information that was clear and understandable. Student clarified difficult issues. Student asked appropriate questions to stimulate discussions. Student generated hypotheses to explain problems under discussion. Student evaluated hypotheses in light of available evidence. Student defined and took responsibility for learning goals and objectives.


Personal and Interpersonal Development 1=SA 2=A 3=N 4=D 5=SD Student communications tended to take over the group process in a non-contributory manner.

Student interrupted others with comments. Student was always on time to each session. Student attended every session. Student responded well to criticism by others. Student was engaged/interested in each session. Student provided leadership. Student listened carefully with understanding. Student demonstrated sensitivity to psychosocial issues. Student assessed own performance accurately. Feedback for Personal Growth: Additional Comments: Tutorial Grade: _____________ Tutor Name (Printed): ___________________________________________________ Tutor Signature: ____________________________________________________________________ Note: First- and second-year students receive Pass/Fail grades only. No Honors are given until their third year.

APPENDIX (continued)

Documents

665.full