Medical Education 1987, 21, 125-129

Graduate evaluation of undergraduate teaching programmes in paediatrics

J. C . VANCE & D. McMILLAN

Department of Child Health, University of Queensland, Brisbane

Summary. In a new undergraduate teaching course in child health, medical students were encouraged to prepare and present topics to their colleagues. These presentations covered major subjects in paediatric medicine and were supervised by an experienced teacher. Students were then asked to evaluate the programme at the end of their course in child health and after graduation. Evaluation scores for these pre- sentations were high at the end of the course. Spontaneous comments suggested some dis- advantages such as poor or too-detailed student presentations. However, some noted the value of personal research. After graduation, more positive comments were made and the majority felt that the course had advantages in the area of personal research, that it was better than or as good as other teaching methods, and that it should be continued in its present format.

Key words: *Attitude of health personnel; *education, medical, undergraduate; teaching/ *methods; paediatrics/*educ; students, medi- cal; Australia

Introduction

One of the long-term aims of a medical educa- tion programme should be to prepare students for the clinical workplace and to encourage them to be responsible for their continuing education. Evaluating undergraduate teaching programmes can take many forms, including

Correspondence: Dr J . C. Vance, University De- partment of Child Health, Mater Children’s Hospi- tal, South Brisbane, Queensland 4101, Australia.

open discussion with students, completion of questionnaires at the end of their time in a particular discipline, or even their performance in the examination for that discipline. How- ever, one of the problems is that the students have not yet been in clinical practice as post- graduates and their comments lack this pers- pective. Medical students may feel that the content of the course is inappropriate and the method of teaching not useful but their opin- ions may change after graduation.

To examine these issues, the Department of Child Health at the University of Queensland undertook a small retrospective survey to de- termine how students viewed the usefulness of a particular teaching programme several years after graduation.

Materials and methods

Descriptiori of the programme

In the sixth (final) year of the medical course at the University of Queensland, students attend four 10-week terms in medicine, surgery, ob- stetrics and gynaecology and child health. Stu- dents in child health attend either the Royal Children’s Hospital or the Mater Children’s Hospital in Brisbane. While the study was in progress the proportion of students attending the Mater Children’s Hospital during each term increased from about one-third to one-half of the total, i.e. zs students now attend each term at the Matcr Children’s Hospital. Since 1979 this latter group has been encouraged to pre- pare, with the help of a supervisor, topics related to paediatric medicine and to present

I 26 J . C . Vance G D. McMillan

them to their colleagues. The programme has received enthusiastic support from the tutors who are based either at the university or at the Mater Children’s Hospital.

Description of the evaluation

At the end of each 10-week term students were asked to make anonymous comments on the course. Using a 5-point scale (ranging from excellent to no value), they gave a score for their overall impression of the programme during the ro-week term. Also, they were encouraged to make further comments which would be of value in preparing the course in future. Their participation was not compul- sory.

The second phase of the study took place several years later. The names were obtained of students who had attended the Mater Chil- dren’s Hospital for their term in child health from 1979 to 1983 inclusive. Using random sampling numbers, 20 names from each year were selected and their addresses obtained from the Register of Medical Practitioners, Queens- land (1984). A questionnaire relating to the programme was sent to these graduates. T w o mailings produced a total response rate of 69% (range 65-75%) for the 5 individual years. The following questions were asked:

( I ) What do you see as the advantages of the student presentation?

( 2 ) What do you see as the disadvantages of the student presentation?

(3) Do you feel that the student presenta- tions assisted in your personal educational de- velopment?

(4) Do you think that the student presenta- tions helped you in your subsequent practice more than/less than other forms of learning?

( 5 ) DO you feel the student presentations should be continued?

Results

Evaluation at the end qf the zo-week term

Students were asked to rate their own presenta- tions and those of their colleagues on a 5-point scale: excellent; very good; good; fair; no value. A mean score was then obtained using a score of 5 for excellent and I for no value. Table I

gives the scores allotted by the students. It shows that, in spite of changes in the number of students taught each year and the variation in the response rate to the questionnaire, the mean score for each year remained relatively constant.

Students were then given the opportunity of making written comments on the programme. This response was small; 11 (4% of students) noted advantages while 60 ( ~ 0 % of students) indicated disadvantages. The advantages were that they developed a greater knowledge of the topic (72% of response), practice in researching and presenting information and stimulating gioup discussion ( ~ 8 % ) . Disadvantages were a preference for a tutor (50% of response), poor presentation by colleagues ( I 8%), incorrect information given ( I 3 YO) and too much detail given ( I O ~ O ) .

In general discussion with students, some felt that the topic should be taught in lectures and they should not be asked to prepare it them- selves. The main reasons given were ignorance of the subject and lack of ability to present the information in a useful manner. Others reacted more positively saying that during the 6-year course the students were rarely required to prepare a topic for presentation and, while

Table I . Initial response to teaching programme

1979 1980 1981 1982 1983 Total

Total number of students 60 48 56 91 I00 3 5 5

Response (YO) 72 67 88 88 66 76

Number who answered 43 32 49 80 66 2 70 questionnaire

Mean score for student’s 3.28 3.85 3.76 4.14 3 . 8 3 3.82 presentation

Graduate evaluation of undergraduate teaching 127

some may have been daunted, many had re- sponded well to the challenge.

In general, students were outspoken about negative aspects of the programme. All their comments were spontaneous; no specific ques- tions had been posed regarding advantages and disadvantages, unlike the second part of the survey.

As a result of the criticism from final-year students, it was decided to review past gradu- ates from the programme. They were selected using random sampling numbers. It was felt that, as graduates matured, they would be more likely to appreciate the self-help teaching methods used. Positive and negative comments about the programme were sought in the questionnaire and the person completing it would remain anonymous. The responses are given in Table 2 showing year by year differ- ences. The response rate was relatively constant for all years and is similar to that obtained for the first part of the study.

Details of the responses are as follows:

Question r-Advantages of the programme. A total of 97% of the respondents acknowledged that there had been advantages. Often more than one example was given so that the total of

the following percentages exceeds IOO%.

Advantages mentioned included practice in doing research (46.3 YO), practice in presenting information ( 3 4 3 % ) , increased knowledge of the topic for the graduate (37.3%) and stimu- lated good group discussion (7.5%). These comments are similar to those made by under- graduates.

Question 2-Disadvantages o j the programme. Many of the respondents (84.1%) described disadvantages similar to those which appeared in the undergraduate appraisal. It is of interest that the more recent graduates found more disadvantages. These included: too much time being required to prepare the presentation (241 "/o); poor presentation by colleagues (22.4%); sometimes incorrect information being given, particularly if it had been obtained from foreign texts or was related to local practices which might be different (32.8%); too much detail being given (5.2%); some students just preferring to attend lectures (17.2%).

Question 3-Help with personal development. A total of 84.1% of the respondents acknow- ledged that the experience had helped them.

Quertion q-Comparisons. Graduates were asked to compare the programme with other

Table 2. Responses to postgraduate survey

1979 1980 1981 1982 1983

n (YO) n (%) n (YO) n (YO) n (YO) Total

Total number of students

Graduates approached

Size of sample Response Response rate Questions ( I ) Advantages ( 2 ) Disadvantages ( 3 ) Helped with

personal development

(4) Comparisons: helped more helped equally

( 5 ) Continue with programme

60 48 56 91 I00

20 20 20 20 20 100

33% 42 % 36% 22% 20%

I 3 I4 I 3 I4 I 5 69 65 Yo 70% 65 % 70% 75% 69%

11 (846)t 14 (100) I 3 (100) 14 (100) 1 5 (100) 97.1% 7 (53.8) 11 (78.5) 12 (92.3) I4 (100) I4 (93.3) 84.1%

11 (84.6) 14 (100) 10 (76.9) 13 (2.8) 12 (80) 86.9%

tRaw data given in table. Figures in parentheses represent the raw data expressed as a percentage of the total response.

I28 J . C. Vanre G D. MrMillan

forms of teaching and to say whether it had been of more, the same or less value in their subsequent practice. 59-4'70 of the respondents felt that the programme had been of more or equal value as other forms of teaching.

Questiori 5-Continuation of the pvogramme. A total of 88.4% of the respondents felt that the programme should be continued.

Discussion

A small descriptive study evaluated a particular type of teaching programme which placed emphasis on students doing research and pre- senting information about specific topics in paediatric medicine. The evaluation was in two parts. Firstly, all students were asked to com- plete a form, without putting their name to it, giving a score to this teaching method and adding any comments they might have. Secondly, a questionnaire was posted to gradu- ates some years later to be completed anony- mously. The results showed that, for the end-of- term evaluation, students gave favourable scores to the programme but voiced a number of criticisms such as a preference for a tutor, poor presentation and incorrect or too-detailed in- formation presented.

Such criticisms should be taken seriously but need to be reviewed in the overall context of the medical course at the University of Queensland. Students study most of the child health subject in the final (sixth) year of the course. They have had 5 years in which most of the classroom teaching has been in the form of lectures. Information has been passed on to them by lecturers and tutors and, at least initially, the students do not have to search out information for themselves. Their presentations may be poor because they have had little previous experience. However, lecturers and tutors may also present information poorly. Distorted facts with little clinical relevance may be disgorged from unsympathetic textbooks. Presenting too detailed or incorrect information may reflect inexperience in seeking out the correct facts that are relevant for local condi- tions, or it may be due to inadequate supervi- sion by the tutors assigned to help them. It can be argued that all these criticisms may arise from students not having been exposed pre-

viously to this type of teaching and thus finding it difficult, in such a short and concen- trated course, to alter their preconceptions of education. Nevertheless, the criticism could be valid.

If such criticism is justifiablc, then it is likely that negative feelings about the course will persist. Therefore, reaction to the postal survey is of interest. The response rate was fairly standard for a postal questionnaire. However, the results can only be interpreted for those who were interested enough to reply and any conclusions drawn must bear this in mind. Respondents recognized the advantages in learning to do research, increasing their know- ledge and developing their presentation skills. None of the graduates spontaneously suggested that this method should be used in other terms of sixth year, and they were not specifically asked to consider the idea.

In spite of the disadvantages, respondents gave a strong indication that the programme should be continued as it compared favourably with other teaching methods and helped them with subsequent professional development. These results appear to indicate that either graduates perceive the merits of the pro- gramme more clearly in retrospect or they have forgotten what, as students, they may have regarded as the bad components of under- graduate teaching. More favourable responses were received from students who graduated in the earlier years of the programme, which supports this suggestion. However, another interpretation of this favourable response may be that, as fewer students attended in the earlier years, classes were smaller and the effect of the presentation potentially greater. There would need to be a longer study to answer this question. Whatever the reason, graduates have given encouragement to the programme as originally developed and have indicated that it should continue.

This method of teaching has not been used a great deal in the medical course a t the Univcr- sity of Queensland as large classes of over zoo students preclude it. However, it would be possible to introduce it into the clinical compo- nents of other subjects in the medical course, such as medicine and surgery, during years 4, 5 and 6. Students might then feel less threatened

Graduate evaluation of undergraduate teaching I 2 Y

by the prospect of preparing material for pre- sentation before their colleagues and they might respond more positively to the questions asked during the survey of the course in child health. The variability in response may also reflect an occasional lack of direction by tutors who had no special training in this form of teaching. Preparing them beforehand might further improve the students’ response.

As this study is mainly descriptive and re- trospective, there may be problems in inter- preting the data. Nevertheless, some ambiva- lence was displayed by students at the time of the end-of-term evaluation. This showed up too in the postgraduate survey but it also accentuated more positive aspects of the prog- ramme, seen in the light of their current clinical practice.

The literature related to this subject is sparse. Most studies have evaluated programmes at the end of the teaching time. In a retrospective study of five cohorts of students, consumer views about parts of the undergraduate teaching course were sought (Richardson 1983). Information was obtained on the rele- vance of the content of the course but not on the actual methods of teaching for a particular subject. A similar study (Bethwaite et al. 1983) also looked at the relevance of the content rather than the method of teaching. The au- thors asked graduates to rate the encourage- ment which they received for continuing education and independent study. Nearly 40% said they had been given little incentive in this area. It could be deduced from this that too much of the teaching was in the form of lectures, giving little stimulus for personal research.

In a study similar to the present one, the same characteristics of a teaching programme were regarded as advantages and disadvantages both on graduation and several years later (Crook et a / . 1981). Because of this consistency, it was considered that postgraduate surveys

were of limited extra value. It is of interest that one of the most positive aspects was that of self-directed learning. The study emanated from McMaster University in Canada, where this approach to education is emphasized. In contrast, the relatively negative response obtained in the first survey in this study may be a reflection of the more traditional approach to medical education at the University of Quecns- land. It is therefore of real interest that this attitude changed with the second questionnaire.

This is a small, retrospective study focusing on a particular method of teaching under- graduates to determine the attitude of graduates to it. More positive responses were received as the time since graduation increased. Compari- son with other work indicates only whether the content of curricula was adequate and not the method of teaching. The retrospective survey suggests that the method of teaching described is considered valuable by graduates and should be continued, even if initially undergraduates have difficulty coping with it.

References

Bethwaite P.B., Alexander R.S., Hornblow A.R. & Handen T.W. (1983) An educational experience evaluated: the Christchurch Clinical School of Medicine. New Zealand Medical _louurral 96, 4 7 6 9 .

Crook J., Woodward C., Feldman E. & Ridge H . (1981) Long term stability of graduates’ rating of their educational programs or, what is to be gained by post program follow-up surveys? Pre- sented at the Twentieth Annual conference on Research in Medical Education, Washington, DC.

The Register of Medical Practitioners, Queensland (1984) The Medical Board of Queensland, Bris- bane, Australia.

Richardson I.M. (1983) Consumer views on the medical curriculum: a retrospective study of Aberdeen graduates. Medical Education 18, 8-10.

Received 13 M a y 1986; editorial comments to authors I July 1986; acccepted f o r publication 8 September 1986