Resident Evaluation of Clinical Teachers Based on Teachers' Certification

ACAD EMERG MED d July 2003, Vol. 10, No. 7 d www.aemj.org 731

Resident Evaluation of Clinical Teachers Based onTeachers’ Certification

Ivan P. Steiner, MD, Philip W. Yoon, MD, Karen D. Kelly, BScN, PhD,Barry M. Diner, MD, Michel G. Donoff, MD, Duncan S. Mackey, MD,

Brian H. Rowe, MD, MSc

AbstractObjective: To examine the influence of emergency medicine(EM) certification of clinical teaching faculty on evaluationsprovided by residents. Methods: A prospective cohort anal-ysis was conducted of assessments between July 1994 andJuly 2000 on residents’ evaluations of EM faculty at theUniversity of Alberta, Edmonton, Canada. Resident- andfaculty-related variables were entered anonymously usingthe validated evaluation tool (ER Scale). Credentialing anddemographic information on EM faculty was supplementedby data obtained through a nine-question survey. Groupswere compared using ANOVA. Results: The 562 residentsreturned 705 (91%) valid evaluation sheets on 115 EMfaculty members. The four domains of didactic teaching,clinical teaching, approachability, and helpfulness were as-sessed. The majority of ratings were in the very good orsuperb categories for each domain. Instructors with certifi-cation in EM had higher scores in didactic, clinical teaching

compared with others, and teachers without nationalcertification scored lower in the helpful and approachablecategories (p \ 0.05). The route of obtaining EM certifica-tions either through training or practice eligibility did notaffect scores. Instructors under the age of 40 years hadhigher scores than the older age groups in three of fourcategories (p \ 0.05). Instructors working at the teachingsites on a half-time basis received higher scores than thoseworking full-time, and scores varied based on site. Overall,teaching ratings improved over the study period (p\ 0.05).Conclusions: Significant differences exist among instructorsin the EM setting that affect their teaching rating scores.National certification in EM, academic track, rotation year,and site are all correlated with better teaching performance.Key words: teachers; certification; instructors; education;evaluation; preceptors; emergency medicine; residency.ACADEMIC EMERGENCY MEDICINE 2003; 10:731–737.

The need for formal evaluation of clinical teaching hasbeen recognized as essential in medical education.1

The aims of evaluating teaching are to reward ex-cellence and identify areas for improvement amongthe teaching staff. This is a worthwhile goal, sincethere is evidence to link good teaching with improvedlearning.2–5 Through iterative feedback and correctiveaction, teaching should ultimately lead to improve-

ments in the quality of clinical practice and healthoutcomes.

Evaluation of the performance of clinical teachingfaculty by medical students and residents is a recog-nized part of this overall evaluation process in theadministration of medical education. Clinical teachershave been identified as having a key role in in-fluencing the quality of the educational process.6,7

Evaluation of teaching performance by residents andstudents has been generally found to be valid andreliable, and optimal teacher characteristics have beendescribed.7–15 Tools have been developed for theevaluation of teaching in various settings, and re-cently one specifically used for assessing teachingperformance in the emergency department (ED) wasvalidated.16 Data gathered using different instrumentsin different environments have been used in thestratification of clinical faculty based on the quality ofclinical teaching, the provision of feedback to facultymembers, the enhancing of faculty developmentstrategies, and the collection of information for facultypromotion.17–21

Despite its importance, evaluation has been poorlystudied. Evaluation is a complex, multifactorial pro-cess and in order to better understand the role offaculty assessments in the educational process, focusneeds to be placed on various teacher and learner

From the Division of Emergency Medicine (IPS, PWY, BMD, BHR),Department of Family Medicine (IPS, PWY, MGD, DSM), andDepartment of Public Health Sciences (BHR), University of Alberta,Edmonton, Alberta, Canada; and the Community Health Program(KDK), Faculty of Medicine and Dentistry, University of NorthernBritish Columbia, Prince George, British Columbia, Canada.Received August 22, 2002; revision received November 27, 2002;accepted January 24, 2003.Presented in part at the 8th Scientific Assembly of the IsraeliAssociation for Emergency Medicine, Tel Aviv, Israel, June 2001.Financial assistance for data entry and analysis for this study wasprovided by the Kingsway Emergency Physicians’ Research Fund,Edmonton, Canada.At the time of the study, Drs. Steiner, Yoon, and Mackey wereemergency medicine residency program administrators.Address for correspondence and reprints: Dr. Ivan P. Steiner,Division of Emergency Medicine & Department of Family Medicine,Faculty of Medicine, University of Alberta, Royal AlexandraHospital, 10240 Kingsway Avenue, B-309, Edmonton, Alberta,Canada, T6R1P7. Fax: 780-477-4916; e-mail: [email protected].

characteristics that may affect the evaluation. To date,a small number of studies have examined medicalstudents’ and residents’ level of training as a factor infaculty evaluation.14,22,23

Review of the literature using Medline, ERIC, andbibliographic searching failed to reveal any informa-tion on clinical teacher-related factors that may affectthe perceived quality of the teaching. Emergencymedicine (EM) is a discipline where students andresidents are exposed to a variety of clinical teacherswith diverse backgrounds.

Our program has collected faculty evaluation dataover a six-year period. The purpose of this study wasto examine teacher-related factors, in this case certi-fication level, as a potential influence on resident eval-uation of staff.

METHODS

Study Design. This study used a prospective cohortdesign. Resident participation was voluntary andresponses were anonymous and confidential. Physi-cian participation was voluntary and confidentialitywas maintained. Ethics approval was waived for thefirst component of the study; however, the HealthResearch Ethics Board of the University of Albertaapproved the physician survey.

Study Setting. Emergency medicine teaching at theUniversity of Alberta takes place in teaching and af-filiated hospitals. Learners are paired one-on-one withinstructors for their predominantly eight-hour shifts.This permits direct teaching and supervision of thelearner. At the end of each shift, the learner is givenfeedback and evaluation by this preceptor. This typeof educator–learner dyad is intended to enhance theeducational experience for the learner, in contrast tothe service role. Learners provide aggregate, end-of-the-rotation assessment on each clinical teacher theyencountered during the clinical rotation.

In general, the EM teaching faculty in Canada isheterogeneous and its members can include special-ists in EM (FRCP and ABEM), family physicians witha certificate of special competence in EM (CCFP(EM)), family physicians (CCFP), and physicianswithout any formal national accreditation. The EM-certified physicians have acquired their credentialsthrough training or practice eligibility.24,25 The struc-ture of the faculty is a reflection of the relatively recentrecognition of the specialty of EM in Canada andspecific manpower issues reflected by the low numberof EM training positions.26,27

The study was conducted in two stages. First, resi-dents were given clinical faculty evaluation forms,which included the resident’s demographic informa-tion and the evaluation tool (ER scale). After eachclinical rotation in EM, the postgraduate year (PGY) 1,2, and 3 residents of the family medicine (FM) pro-

gram returned the completed forms to the programcoordinator voluntarily and anonymously. Seventeaching sites were included in the study. In thesecond stage, a nine-item demographic questionnaire(physician survey) was distributed to all the EMclinical teachers affiliated with the Division of EM orthe Department of FM via e-mail and followed up bytelephone. Three attempts to contact all clinical teach-ers were made.

Instruments. The ER scale, which is a quantitative,psychometrically validated evaluation tool, was usedto measure the quality of instruction.16 The instru-ment consists of four separate domains that evaluatethe didactic teaching, clinical teaching, approachabil-ity, and helpfulness of the ‘‘teacher.’’ The ER scaledomains are scored using a five-point scale from un-acceptable (1) to outstanding (5).

Data Collection. Resident evaluations were collectedbetween July 1, 1994, and July 1, 2000. This periodcoincides with the early years of development of theDivision of EM and comprehensive EM training at theUniversity of Alberta (1991–2000). Any incompleteevaluation forms were excluded from the study—ifthe clinical teacher was not identified, if one of thefour domains was not completed, or if the level ofresident training was not noted. The physician surveywas conducted between December 2000 and February2001.

Statistical Considerations. Data were analyzed us-ing the SPSS-PC 9.0 statistical program (Chicago, IL).Categorical values are reported as counts and per-centages (%) and compared using chi-square tests.Continuous variables are reported as means andstandard deviations (SDs) and compared using anal-ysis of variance (ANOVA) or Student’s t-tests. Nonparametric tests (Kruskal-Wallis and Mann-WhitneyU tests) were conducted as sensitivity analyses; how-ever, since the nonparametric results were similar, theparametric results are presented. Multivariable com-parisons were made using logistic regression analysesand the adjusted odds ratios (ORs) and 95% confi-dence intervals (95% CIs) are reported. A priori, levelsof significance of \0.05 were considered significant.After the initial frequencies were reviewed, the ERscale was truncated by collapsing scores of 1 and 2 forthe analyses because of the small cell count.

RESULTS

Resident Evaluations. The 562 FM residents in theprogram during the study period voluntarily returned777/831 (94%) evaluation forms; 705 forms (91%)were properly completed. Of these, 183 (26%) formswere from FM 1, 297 (42%) from FM 2, 183 (26%) fromFM 3, and 42 (6%) from FM 1 and 2 residents. In total,

732 Steiner et al. d TEACHER CERTIFICATION STATUS

3,268 clinical teacher evaluations were received, for anaverage of 4.6 evaluations per resident.

Teaching Faculty Demographics. One hundredfifteen EM clinical teaching faculty members, 100male (87%), were evaluated in the four teachingdomains. There were 51 under the age of 40 years(44%); 40 (35%) were between 40–50 years and 24(21%) were over 50 years. The proportion of thosewithout EM certification increased with age: nine (8%)in the under 40 group, 14 (12%) in 40–49 group, and17 (15%) in those over 50. With respect to certification,23 (20%) were specialists, 53 (46%) were EM-certifiedfamily physicians, 11 (10%) were family physicians,and 28 (24%) had no specialization other than theirmedical degree. Approximately 80% of the facultyworked in one of the seven teaching sites on a full-time basis, 10% worked in one of these sites half time,and 10% worked there part time.

Overall Results (Table 1). The majority of teacherratings were in the very good to superb categories(65%). Achievement of certification by training orpractice eligibility did not affect the teaching scores.Tables 2 through 5 display the multiple regressionresults for predicting performance. Variables in eachregression model included: institution, rotation year,age, gender, teacher’s certification, method of certifi-cation, EM employment status, and residents’ leveland season of training.

Didactic Teaching (Table 2). Instructors in institu-tions B, C, and E were 50%–83% more likely and thoseat institutions F and G 49%–59% less likely to providegood didactic instruction compared with the referenceteaching site. Didactic instruction scores improvedfrom 1994 to 2000 almost fourfold. Instructors withhalf-time status were 50%–73% more likely to providegood didactic instruction compared with others.Faculty members over the age of 49 years were 38%less likely to provide good didactic instruction. Pre-ceptors without EM certification were 55%–65% less

likely to provide good didactic instruction comparedwith the specialists.

Clinical Teaching (Table 3). Instructors in institu-tion C were 73% more likely and those at institution Gwere 63% less likely to provide good clinical in-struction when compared with the reference teachingsite. Clinical instruction improved over the years;ratings from 2000 were almost three times higher thanthose from 1994. Female preceptors were 56% lesslikely to be rated as good clinical instructors. Facultymembers in the two groups older than 40 years were25–33% less likely to be rated as providing goodclinical instruction. Instructors with no EM certifica-tion were 39%–56% less likely to provide good clinicalinstruction compared with specialists.

Helpful Instructor (Table 4). Instructors at institu-tions B and C were 36%–53% more likely and those atinstitution G were 63% less likely to be helpful com-

TABLE 1. Scores by Method of Certification ofTeacher—through Training or Practice Eligibility

Scale LevelNumber ofEvaluations

MeanScore (SD) p-value

Didactic Through training 1,120 4.09 (0.75) 0.53Practice eligible 1,145 4.08 (0.74)

Approach- Through training 1,171 4.29 (0.75) 0.35able Practice eligible 1,194 4.26 (0.71)

Helpful Through training 1,173 4.28 (0.74) 0.43Practice eligible 1,194 4.26 (0.72)

Clinical Through training 1,172 4.18 (0.70) 0.50Practice eligible 1,194 4.16 (0.69)

TABLE 2. Predictors of a Good Didactic Instructor*

Factor

UnivariateOdds Ratio

[95% CI]

AdjustedOdds Ratio

[95% CI]

Institution(A)y 1.00 1.00(B) 1.29 [1.03, 1.60]z 1.50 [1.16, 1.94]z(C) 1.55 [1.19, 2.04]z 1.83 [1.32, 2.54]z(D) 0.65 [0.51, 0.83]z 1.14 [0.82, 1.57](E) 1.31 [0.94, 1.82] 1.77 [1.21, 2.59]z(F) 0.52 [0.33, 0.81]z 0.51 [0.31, 0.84]z(G) 0.65 [0.35, 1.21] 0.41 [0.21, 0.82]z

Rotation year1994 1.00 1.001995 0.95 [0.72, 1.24] 0.91 [0.68, 1.22]1996 1.51 [1.11, 2.04]z 1.38 [0.99, 1.91]1997 1.46 [1.09, 1.96]z 1.27 [0.92, 1.75]1998 2.04 [1.50, 2.77]z 1.69 [1.22, 2.36]z1999 2.13 [1.52, 2.97]z 1.80 [1.26, 2.58]z2000 4.60 [2.79, 7.59]z 3.68 [2.17, 6.24]z

EM statusFull-time 1.00 1.00Half-time 1.32 [1.04, 1.68]z 1.50 [1.12, 2.02]zPart-time 0.83 [0.55, 1.27] 0.87 [0.55, 1.39]

Age\40 years 1.00 1.0040–49 years 0.67 [0.55, 0.82]z 0.83 [0.65, 1.05]$50 years 0.56 [0.46, 0.68]z 0.62 [0.48, 0.81]z

CertificationFRCP(EM)/ABEM 1.00 1.00CCFP(EM) 0.82 [0.65, 1.04] 0.77 [0.57, 1.04]CCFP 0.46 [0.34, 0.62]z 0.45 [0.31, 0.64]zNo certification 0.33 [0.26, 0.42]z 0.35 [0.27, 0.46]z

*Variables included in the model are: institution, rotation year,age, gender, teacher’s certification, method of certification,emergency medicine employment status, and residents’ leveland season of training.yReference teaching site.zp \ 0.05.


pared with the reference institution. Ratings for help-fulness increased by more than three times from 1994to 2000. Female instructors were perceived to be 45%less helpful than male instructors, and instructors 40years of age and older were deemed 39%–45% lesshelpful compared with younger colleagues. Instruc-tors without any kind of certification were 45% lesshelpful compared with colleagues who had eithertype of EM certification.

Approachable Instructor (Table 5). Teachers atinstitutions B and C were rated 70%–41% more likelyand those at institution G were 62% less approach-able, compared with the reference teaching site. In-structors in 2000 were almost four times more likely tobe rated approachable compared with those in 1994.Emergency medicine-certified family physicians wereconsidered to be 62% more approachable than thespecialists, and instructors without any kind of certi-fication were considered less approachable (29%–91%) when compared with EM-certified physicians.

DISCUSSION

This large study of residents, using a previouslyvalidated evaluation tool, provides new evidencefor rating clinical teachers, and, to the best of ourknowledge, is the first of its kind in the emergency orother medical setting. The results indicate that, in theopinion of residents, clinical teachers with formalnational EM certification were better in all fourassessment categories comparedwith instructors with-out EM certification. The route of attaining national EMcertification, through either training or practice eligi-bility, did not appear to have an effect on the rating ofteachers. In the domain of didactic instruction, the half-time clinical group received statistically higher scores.This group includes the geographical full-time facultymemberswho are employed by the Faculty ofMedicineand Dentistry and are on an academic track. Whilereassuring, since interest in teaching is one of thecriteria for their job, these particular results for thisgroup are not surprising.

In the domains of didactic and clinical instructions,the two EM-certified groups of teachers did not differ

TABLE 3. Predictors of a Good Clinical Instructor*

Factor


[95% CI]

AdjustedOdds Ratio

[95% CI]

GenderMale 1.00 1.00Female 0.44 [0.31, 0.62]z 0.44 [0.29, 0.64]z

Institution(A)y 1.00 1.00(B) 1.02 [0.80, 1.29] 1.19 [0.92, 1.55](C) 1.21 [0.90, 1.63] 1.73 [1.22, 2.45]z(D) 0.57 [0.44, 0.74]z 0.86 [0.63, 1.17](E) 1.04 [0.73, 1.50] 1.19 [0.81, 1.75](F) 0.62 [0.37, 1.03] 0.62 [0.34, 1.12](G) 0.60 [0.31, 1.16] 0.37 [0.19, 0.75]z

Rotation year1994 1.00 1.001995 0.95 [0.71, 1.26] 0.91 [0.68, 1.23]1996 1.21 [0.88, 1.67] 1.11 [1.20, 1.55]z1997 1.29 [0.94, 1.77] 1.17 [0.84, 1.63]1998 1.66 [1.19, 2.30]z 1.52 [1.07, 2.15]z1999 1.98 [1.36, 2.88]z 1.69 [1.14, 2.52]z2000 3.56 [2.06, 6.17]z 2.79 [1.58, 4.93]z

Age\40 years 1.00 1.0040–49 years 0.66 [0.53, 0.83]z 0.75 [0.58, 0.96]z$50 years 0.58 [0.46, 0.72]z 0.67 [0.52, 0.86]z

CertificationFRCP(EM)/ABEM 1.00 1.00CCFP(EM) 0.85 [0.65, 1.10] 0.97 [0.71, 1.32]CCFP 0.47 [0.34, 0.65]z 0.61 [0.42, 0.90]zNo certification 0.36 [0.28, 0.47]z 0.44 [0.33, 0.58]z


TABLE 4. Predictors of a Helpful Instructor*

Factor


[95% CI]

AdjustedOdds Ratio

[95% CI]

GenderMale 1.00 1.00Female 0.57 [0.38, 0.84]z 0.55 [0.36, 0.84]z

Institution(A)y 1.00 1.00(B) 1.21 [0.94, 1.57] 1.36 [1.03, 1.80]z(C) 1.02 [0.76, 1.37] 1.53 [1.08, 2.17]z(D) 0.68 [0.52, 0.90]z 0.99 [0.71, 1.37](E) 1.36 [0.90, 2.04] 1.55 [1.00, 2.38](F) 0.94 [0.52, 1.69] 1.08 [0.53, 2.18](G) 0.59 [0.30, 1.16] 0.37 [0.18, 0.76]z

Rotation year1994 1.00 1.001995 1.16 [0.87, 1.56] 1.09 [0.80, 1.48]1996 1.54 [1.10, 2.15]z 1.38 [0.98, 1.95]1997 1.81 [1.29, 2.53]z 1.52 [1.07, 2.16]z1998 2.12 [1.50, 2.99]z 1.80 [1.25, 2.60]z1999 2.57 [1.72, 3.84]z 2.14 [1.40, 3.27]z2000 4.34 [2.40, 7.85]z 3.22 [1.75, 5.95]z

Age\40 years 1.00 1.0040–49 years 0.53 [0.42, 0.68]z 0.61 [0.47, 0.81]z$50 years 0.45 [0.35, 0.57]z 0.55 [0.41, 0.72]z

CertificationFRCP(EM)/ABEM 1.00 1.00CCFP(EM) 1.15 [0.88, 1.51] 1.05 [0.76, 1.45]CCFP 0.61 [0.44, 0.84]z 0.69 [0.47, 1.03]No certification 0.50 [0.39, 0.65]z 0.55 [0.41, 0.73]z



in statistically significant ways; however, the special-ists did have overall higher scores compared with theEM-certified family physicians. This appears logical tous in view of the fact that specialists have more in-depth didactic training in EM. In contrast, FM trainingincorporates understanding of human developmentand family and other social systems and the FMtrainees learn skills that qualify them to provide acomprehensive approach to the management of dis-ease and illness in patients and their families.28 TheFM-certified EM clinical teachers scored higher in thedomain of helpfulness and statistically significantlyhigher in the area of approachability.Instructors under the age of 40 years were per-

ceived as better teachers in all categories except inapproachability. These data are difficult to interpretbecause of many confounding factors. For example, itis noteworthy to mention the heterogeneity of the agegroups in terms of EM credentials, with the older agegroups having a disproportionately larger numberof non-EM-certified but clinically very experiencedindividuals. Therefore, one is left to speculate on thetrue relevance of age. Perhaps younger EM teacherswho are closer in age to the trainees may have thedepth and breadth of knowledge, the skills, and theappropriate behaviors to provide the best educationfor residents. Alternatively, it may be a reflection ofan age-related phenomenon, where residents rateyounger instructors higher since they relate better to

them on a personal level. Ultimately, the issue of com-parison of younger and older groups of individualswith equivalent EM certification levels has not beenanswered by this study.

Male gender was associated with higher scores inthe areas of clinical teaching and helpfulness. Theseresults may simply be an anomaly based on the smallproportion (13%) of female faculty and the relativelysmall number of teacher evaluations for this group.Interestingly, the small percentage of women in ourEM faculty is consistent with the findings of a largeU.S. manpower review, where the census showedfemale EM faculty at 17%.29

Reassuringly, the data show that overall evaluationscores for all categories have increased significantlyover the study period. There are several possible ex-planations for these results. First, instructors whowere teaching at the inception of the training pro-grams may be progressively more comfortable withthe teaching requirements and load. The addition oftrained graduates from the EM programs as well asrecruitment of qualified individuals from outside mayhave also improved the EM teaching faculty. Furtherevaluation is required in order to determine thevalidity of these theories.

Lastly, teachers from two institutions constantlyoutperform the others in all domains and the educa-tors from another one are consistently weak. Forobvious reasons, we cannot reveal the locations byname, but suffice it to say that such evaluations havebeen forwarded to the institutions for commendationsand remedial action, respectively. The institutions thatwere deemed strong in EM education by the residentshad a historically long and established track record inproviding EM teaching to FM residents. This experi-ence predated even the birth of the Division of EM atthe University of Alberta. Conversely, the institutionthat was deemed weaker was a relatively recent ad-dition to the EM educational arena.

Several trends emerge from this study with possiblegeneral implications for EM education. First, thesedata suggest that educational administrators shouldfocus their efforts on securing EM-certified individ-uals, regardless of age, with particular attention toindividuals with an interest in an academic track forteaching in their training programs. Second, theongoing exposure of instructors to learners andpresumably the addition of new, qualified facultymembers may need to be considered as possiblereasons for the perceived improvement in teaching.Regular review and faculty development may furtherclarify this issue and enhance teaching skills. Thecontinuous success of two institutions further sug-gests that exposure and experience are importantparameters in developing a cadre of good instructors.Third, introducing or adapting some aspects of theFM training to EM specialists’ education may enhancethe future specialists’ humanistic skills and behaviors.

TABLE 5. Predictors of an Approachable Instructor*

Factor


[95% CI]

AdjustedOdds Ratio

[95% CI]

Institution(A)y 1.00 1.00(B) 1.44 [1.11, 1.86]z 1.70 [1.28, 2.23]z(C) 1.08 [0.80, 1.45] 1.41 [1.02, 1.95]z(D) 0.75 [0.57, 1.00] 1.03 [0.75, 1.43](E) 1.37 [0.91, 2.05] 1.47 [0.96, 2.24](F) 0.91 [0.51, 1.60] 0.73 [0.41, 1.31](G) 0.61 [0.31, 1.20] 0.38 [0.19, 0.76]z

Rotation year1994 1.00 1.001995 1.25 [0.92, 1.69] 1.23 [0.90, 1.67]1996 1.43 [1.02, 2.00]z 1.31 [0.93, 1.85]1997 1.67 [1.19, 2.34]z 1.48 [1.04, 2.09]z1998 1.82 [1.29, 2.57]z 1.64 [1.16, 2.34]z1999 2.47 [1.65, 3.72]z 2.25 [1.49, 3.40]z2000 4.08 [2.25, 7.38]z 3.52 [1.91, 6.46]z

CertificationFRCP(EM)/ABEM 1.00 1.00CCFP (EM) 1.50 [1.15, 1.94]z 1.62 [1.21, 2.17]zCCFP 0.86 [0.62, 1.19] 0.95 [0.66, 1.38]No certification 0.69 [0.54, 0.89]z 0.71 [0.54, 0.94]z

*Variables included in the model are: institution, rotation year,age, gender, teacher’s certification, method of certification,and emergency medicine employment status, and residents’level and season of training.yReference teaching site.zp \ 0.05.


Currently, we know of no information on teacher-related factors in the literature. Under these circum-stances, the outcome of this study may serve as astarting point in this area for future educationalresearch.

LIMITATIONS

Each form submitted by the residents includedaggregates for each of the four evaluation categories;this may not represent an accurate reflection of theday-to-day interaction that occurs in the teaching–learning environment. Emergency department-relatedfactors such as volume of patients, acuity, and con-gestion, as well as learner- and teacher-specificindividual behavioral/attitudinal characteristics, can-not be assessed by this type of evaluation.

The residents evaluating the ED teaching facultywere off-service residents, which may produce a biasin the evaluations. Alternatively, the fact that theywere all residents from a single program may haveprovided homogeneity to the type of responses. It isconceivable that EM residents, because of their ownspecific educational focus, may have had differentopinions about the instructors. This topic could be thesubject of additional research.

In general, evaluator leniency is associated withthese types of assessments—the resident may givehigher scores. This may explain the overall higherevaluation scores. This phenomenon does not alter theresults of this study because all the faculty memberswould benefit to some extent by it and the datacollected reflect multiple residents’ evaluations. An-other limitation is the fact that there was no third-party independent observation of the quality of theencounter and because of this there is no benchmarkto compare the residents’ opinions regarding the‘‘value’’ of the encounters.

In assessing teachers of different gender and agegroups, with various certification levels, in a variety ofinstitutions, the sample sizes decrease for some sub-groups, and this may introduce some bias or weakensome of the conclusions. Finally, the actual differencesbetween groups, though statistically significant, maybe more an indication of large sample sizes than oftrue subgroup variation. Minimally clinically impor-tant differences in teaching evaluation have yet to bedetermined.

Multiple factors influence the assessment processand, because of this, only limited conclusions can bedrawn from the current study. Trends can be estab-lished, but the multiple teacher- and learner-relatedfactors that may influence the evaluation process needfurther in-depth study.

Despite these limitations, we believe that ourconclusions are valid since we have systematicallycollected data over several years in one faculty ofmedicine program and these data have allowed us to

determine trends. We invite colleagues to replicateour study using different, validated tools. Alterna-tively, a study in different settings would assist inidentifying clinical instructor-related factors that maybe associated with effective teaching.

CONCLUSIONS

Effective EM teaching is best delivered by EM-certified physicians, and EM training programsshould strive to hire these individuals, regardless oftheir age. Preference should be given those with aninterest in an academic track. In any new EM pro-gram, the constant exposure of instructors to teachingand the addition of new, qualified faculty membersimprove overall EM education throughout the system.Continued evaluation and support of all teachers,regardless of age or gender, are required. Lastly, EMspecialty programs should consider incorporatingsome of the behaviors and philosophies taught byFM programs.

The authors thank the Division of Emergency Medicine andDepartment of Family Medicine, University of Alberta, Edmonton,Canada; the British Columbia Rural and Remote Health ResearchInstitute–University of Northern British Columbia, Canada; and theCanadian Research Chairs Program, Canada Institute of HealthResearch (CIHR), Ottawa, Canada.

References

1. Snell L, Tallett S, Haist S, et al. A review of the evaluation ofclinical teaching: new perspective and challenges. Med Educ.2000; 34:862–70.

2. Jolly BC, Jones A, Dacre JE, Ezubier M, Kopelman P, Hitman G.Relationships between students’ clinical experiences inintroductory clinical courses and their performance on anOSCE. Acad Med. 1996; 71:909–16.

3. Griffith CH, Wilson JF, Haist SA, Ramsbottom-Lusier M.Relationship of how well the attending teach to theirstudents’ performance and residency choices. Acad Med.1997; 72(10, suppl):S118–S120.

4. Petzel RA, Harris IB, Masler DS. The empirical validation ofclinical teaching strategies. Eval Health Prof. 1997; 5:499–508.

5. Griffith CH, Wilson JF, Harris SA, Ramsbottom-Lusier M. Dostudents who work with better housestaff in medicine clerk-ship learn more? Acad Med. 1998; 73(10, suppl):S57–S59.

6. Stritter FH. Faculty evaluation and development. In: McGuireCH (ed). Handbook of Health Professional Education. SanFrancisco: Jossey-Bass, 1983, pp 294–318.

7. Kendrick SB, Simmons JMP, Richards BF, Roberge LP.Residents’ perception of their teachers’ facilitative behaviourand the learning value of rotations. Med Educ. 1993; 27:55–61.

8. Stritter FT, Hain JD, Grimes DA. Clinical teaching reexamined.J Med Educ. 1975; 50:876–82.

9. Irby DM. Clinical teacher effectiveness in medicine. J MedEduc. 1978; 53:808–15.

10. Irby DM, Rakestraw P. Evaluating clinical teaching inmedicine. J Med Educ. 1981; 56:181–6.

11. Downing SM, English DC, Dean BF, Roberge LP. Residents’ratings of surgical faculty: improved teaching effectivenessthrough feedback. Am Surg. 1983; 49:329–32.

12. Irby DM, Gilmoure GM, Ramset PG. Factors affecting ratingsof clinical teachers by medical students and residents.J. Med Educ. 1987; 62:1–7.


13. Risuci DA, Lutski L, Rosati RJ, Tortolani AJ. Reliability andaccuracy of resident evaluation of surgical faculty. Eval HealthProf. 1992; 15:313–24.

14. Mullan P. What are raters rating? Predicting medical students,pediatric resident and faculty ratings of clinical teachers. TeachLearn Med. 1993; 5:221–9.

15. Mayo P. Student perception of tutor effectiveness in a problembased surgery clerkship. Teach Learn Med. 1993; 5:227–33.

16. Steiner IP, Franc-Law J, Kelly KD, Rowe BH. Faculty evaluationby residents in an emergency medicine program: a newevaluation instrument. Acad Emerg Med. 2000; 7:1015–21.

17. Rothman AI, Poldre P, Cohen R. Evaluating clinical teachersfor promotion. Acad Med. 1989; 64:774–7.

18. Posluns EL. Rewarding medical teaching excellence in amajor Canadian teaching hospital. Med Teach. 1990; 12:13–22.

19. Steinert Y. Faculty development in family medicine. Areassessment. Can Fam Phys. 1993; 39:1917–22.

20. Jones FJ, Froom MA. Faculty and administration’s views ofproblems in faculty evaluation. Acad Med. 1994; 69:476–83.

21. Reiser SJ. Linking excellence in teaching to department’sbudgets. Acad Med. 1995; 70:272–5.

22. Ullian JA, Bland CJ, Simpson DE. The alternative approach todefining the roleof the clinical teacher.AcadMed. 1994; 69:832–8.

23. Paukert JL, Richards BF. How medical students and residentsdescribe the roles and characteristics of their influentialteachers. Acad Med. 2000; 75:843–5.

24. College of Family Physicians of Canada. Standards foraccreditation of residency training programs. 2001. Availableat: http://www.cfpc.ca/education/exams/requirements/emergenmed.asp.

25. Royal College of Physicians and Surgeons of Canada. 2003.Available at: http://rcpsc.medical.org/english/residency/certification/index.php3.

26. Steiner IP, Yoon PW, Holroyd BR. Manpower crisis inemergency medicine: can residency programs make animpact? Can J Emerg Med. 2001; 3:103–5.

27. Halpern P, Steiner IP. The development of EM trainingprograms in Israel and Canada—a tale of two countries.Can J Emerg Med. 2002; 4:218–23.

28. College of Family Physicians of Canada. Standards foraccreditation of residency training programs. 2001. Availableat: http://www.cfcp.ca/education/redbook/genstand.asp#PRINCIPLES.

29. Moorehead JC, Gallery ME, Hirshkorn C, et al. A study of theworkforce in emergency medicine: 1999. Ann Emerg Med.2002; 40:3–15.


Documents

Resident Evaluation of Clinical Teachers Based on Teachers' Certification