Improving student learning through changing assessment · PDF fileImproving student learning through changing assessment – a conceptual and practical framework - 5 - students, and

EARLI Conference, Padova, 2003

Improving student learning through changing assessment – a conceptual and practical framework

Graham Gibbs, Claire Simpson [1] and Ranald Macdonald [2]

Abstract A three year research study at two universities in the UK has been examining the way the design of assessment on a wide range of science courses affects the ways students go about their studying. Its starting point has been a set eleven ‘conditions under which assessment best supports learning’, based on theory and on empirical evidence from studies of strategic changes in assessment (Gibbs,1999; Gibbs, 2002; Gibbs and Simpson, in press). These conditions concern how assessment captures students time and effort and distributes it across a course, how assessment generates productive learning activity, how assessment provides sufficient prompt feedback, the quality of feedback, and how students respond to feedback and use it to inform subsequent learning. The way these characteristics of assessment systems operate in practice has been explored through interviews with students on a series of science courses at the Open University. The ‘Assessment Experience Questionnaire’ (AEQ), which measures the extent to which the 11 conditions are met on courses, will be described. The AEQ has been administered to students on 15 Physics, Chemistry, Astronomy and Bioscience courses, each with different patterns of assessment. Assessment issues have been identified concerning the volume and distribution of student effort and in the use students make of feedback. The project will go on to implement principled changes to assessment design on these courses and to study the impact of these changes on students’ responses and study patterns using the AEQ and other evaluation evidence. [1] Student Support Research Group, Open University [2] Learning and Teaching Institute, Sheffield Hallam University Contact Prof. Graham Gibbs Student Support Research Group Open University Milton Keynes MK7 6AA UK Tel 01908 858439 Fax 01908 858438 email [email protected] Acknowledgement This research reported in this paper is part of a project entitled ‘Improving the effectiveness of formative assessment in Science’ in receipt of £250,000 from the Higher Education Funding Council for England. Their support is gratefully acknowledged.


- 2 -

Introduction The authors have long believed that assessment is one of the most powerful levers available to university teachers to change and improve students’ learning (Gibbs, 1999). At the EARLI Assessment Conference in 2002 a framework was proposed for evaluating the impact of assessment on student learning (Gibbs, 2002). This framework has been elaborated into a set of 11 ‘conditions under which assessment supports student learning’ (Gibbs and Simpson, in press) (see Table 1). ___________________________________________________________________ Table 1 Eleven conditions under which assessment supports student learning Quantity and distribution of student effort 1. Assessed tasks capture sufficient study time and effort 2. These tasks distribute student effort evenly across topics and weeks Quality and level of student effort 3. These tasks engage students in productive learning activity 4. Assessment communicates clear and high expectations to students Quantity and timing of feedback 5. Sufficient feedback is provided, both often enough and in enough detail 6. The feedback is provided quickly enough to be useful to students Quality of feedback 7. Feedback focuses on learning rather than on marks or students themselves 8. Feedback is linked to the purpose of the assignment and to criteria 9. Feedback is understandable to students, given their sophistication Student response to feedback 10. Feedback is received by students and attended to 11. Feedback is acted upon by students to improve their work or their learning ___________________________________________________________________ A three year project entitled ‘Improving the effectiveness of formative assessment in Science’ is using this conceptual framework to identify the potential for improving student learning by making principled changes to assessment. The project involves two cycles of evaluation and change in a total of 15 science courses at two universities. Further action research will be funded in Science departments in other Universities through the establishment of a national ‘Special Interest Group’ focussing on formative assessment in Science. This paper reports aspects of the first cycle of evaluation of the 15 science courses. Assessment tactics that emphasise the ‘Eleven conditions’ By reviewing literature that describes case studies of changes in assessment that have been shown to improve student learning it has been possible to identify assessment tactics that have the potential to meet each of the conditions listed in Table 1. For example it is usual for single stage summative testing not to provide feedback to students that they are likely to pay attention to or use, as they are unlikely to be tested on the same material again. In contrast two-stage classroom tests and computer marked tests, where the first stage is formative and the second stage, perhaps two weeks later, is summative, have been shown to improve student performance by increasing the use which students make of feedback from the formative stage to orient and focus their study behaviour in preparation for the summative stage (Carroll,1995; Sly, 1999). Table 2 summarises these assessment tactics.


- 3 -

___________________________________________________________________ Table 2 Assessment tactics that have the potential to meet the ‘Eleven

Conditions’ Assessment issues and related ‘conditions’

Tactics that have the potential to meet the ‘conditions’

Quantity and distribution of student effort 1 Assessed tasks capture

sufficient study time and effort 2 These tasks distribute student

effort evenly across topics and weeks

More assignments and/or assignments distributed more evenly across the course and across topics. To cope with marking load: • Completion of assignments as a course

requirement, without marking • Sampling of assignments for marking (e.g. from a

portfolio) • Mechanised and computer-based testing • Self and/or peer marking Exam demands that are unpredictable and/or sample everything, so that students have to study everything

Quality and level of student effort 3 These tasks engage students

in productive learning activity 4 Assessment communicates

clear and high expectations to students

Larger scale open-ended assignments that are challenging and induce a deep approach, especially involving interaction and collaboration with other students, in or out of class, and social pressures to deliver Clear specification of goals, criteria and standards and ‘modelling’ of products, together with student internalisation of these goals, criteria and standards (e.g. through student marking exercises, public presentation and critique of work) Avoidance of ‘multiple guess’ question tests and exams passable by memorisation, that induce a surface approach. Challenging tests and exams requiring ‘performances of understanding’

Quantity and timing of feedback 5 Sufficient feedback is

provided, both often enough and in enough detail

6 The feedback is provided quickly enough to be useful to students

Regular assignments, starting early Quality standards for volume and quality of feedback Tutor briefing/training/monitoring concerning volume and nature of feedback Mechanised feedback where mechanised tests are used Trade off of quality of feedback against speed of return (e.g. peer feedback, model answers, sampling of assignments to produce generic feedback) Development of student self-supervision that involves ongoing feedback to self as part of learning conversations (meta-cognitive awareness and skill)


- 4 -

Quality of feedback 7 Feedback focuses on learning

rather than on marks or students themselves

8 Feedback is linked to the purpose of the assignment and to criteria

9 Feedback is understandable to students, given their sophistication

No marks, only feedback. Feedback structured around goals, criteria and standards, explaining marks (e.g. using feedback sheets), not focussing on student characteristics Tutor briefing/training/monitoring concerning quality of feedback Development of students’ ability to understand feedback, and of tutor’s awareness of student difficulties in understanding feedback

Student response to feedback 10 Feedback is received by

students and attended to 11 Feedback is acted upon by

students to improve their work or their learning

Faster feedback Tutor feedback only on aspects students request Student discussion of use of feedback Two-stage assignments where feedback on stage 1 helps improve stage 2 Two stage tests where test 1 informs about areas needing revision for test 2. Integrated multi-component assignments (e.g. stages of a project, elements of a portfolio of evidence) where each assignment contributes to a larger whole. Requirement for students to demonstrate (or marks for demonstrating) response to feedback in subsequent assignments Greater emphasis on generic feedback of value to other topic areas

The project is supporting action research that will develop case studies of assessment within science courses that illustrate successful use of the kinds of tactics listed in Table 2. Courses will be provided with a conceptual and practical framework that will enable them to both diagnose aspects of assessment where there is potential for development, and select appropriate assessment tactics with which to make improvements. Measuring the extent to which the ’Eleven conditions’ are met on a course: the development of the Assessment Experience Questionnaire (AEQ) A questionnaire has been developed for the purpose of measuring the extent to which students perceive that the ’11 conditions’ have been met on a specific course on which they are studying. Each of the clusters of conditions (e.g. ‘Quantity and distribution of student effort’) has six associated questionnaire items making up a scale. ‘Quality and level of student effort’ has twelve questions: one set of six questions concerning quality and the level of effort involving coursework assignments and one set of six concerning quality and level of effort involving the examination. The questionnaire items were derived from open ended interviewing of science


- 5 -

students, and from the literature. The scales and sample items are listed in Table 3. The psychometric characteristics of the AEQ, and of its constituent scales, have been reported elsewhere (Gibbs and Simpson, 2003). ___________________________________________________________________ Table 3 Assessment Experience Questionnaire (AEQ) scales and sample items Scale 1 Time demands and distribution of student effort e.g. “In weeks when the assignments are due I put in many more hours” Scale 2 Assignments and learning e.g. “Tackling the assignments you can get away with not understanding and still get high marks” Scale 3 Quantity and timing of feedback e.g. “There is hardly any feedback on my assignments when I get them back” Scale 4 Quality of feedback e.g. “The feedback shows me how to do better next time” Scale 5 Use of feedback e.g. “The feedback prompts me to go back over material covered earlier in the course” Scale 6 The examination e.g. “In the exam you can get away with not understanding and still get good marks” ___________________________________________________________________ Sample and institutional contexts The AEQ has been administered to 150 students on each of eight science courses at University A and to between 15 and 100 students (depending on enrolment) on each of seven science courses at University B. The courses included Physics, Astronomy, Chemistry and BioScience modules of 15, 30 or 60 credits (involving 150, 300 or 600 hours of student effort). In total 776 students returned questionnaires, a response rate of 49%. The patterns of assessment in Institutions A and B, while they varied between individual courses within institutions, displayed the following broad institutional characteristics: • assignments and in-course tests in Institution B were smaller in size, occurred

closer together, and each assessed a smaller ‘chunk’ (in terms of student learning hours or course credits) than did larger and more spread out assignments or in-course tests in Institution A;

• in Institution A there is a convention of eight assignment and an exam for 60-credit courses and four assignments and an exam for 30-credit courses (both over a nine month period), with little variation from this convention. In contrast assessment patterns and assignment types were more varied in Institution B, with some (four month long) courses not using exams;

• exams in Institution B were larger (longer in duration) in relation to course size (study hours and credit weighting) than in Institution A;


- 6 -

• feedback to students in Institution A is very largely through extensive written feedback on assignments, while in Institution B written feedback is less extensive and oral and informal feedback, in and out of class, is much more frequent.

Findings Differences in student perceptions of assessment in the two universities is evident in differences in overall scale scores and in student responses to individual items. Table 4 shows marked differences in AEQ scores on the scale ‘Time demands and distribution of student effort’. Table 4 Comparison of Science courses at Universities A and B in terms of students’ volume and distribution of effort University

A University

B

Scale Scale Score t p [1] Time demands and distribution of student effort

20.3 (s.d. 3.16)

18.6 (s.d. 2.91)

7.387 (d.f. 772)

p < 0.001

Sample items % agree or strongly agree I only study things that are going to be covered in the assignments

8% 27%

% strongly agree I have to study regularly if I want to do well on the course

47% 25%

% disagree or strongly disagree

On this course it is possible to do quite well without studying much

64% 33%

[1] two-tailed t-test. There was also a marked difference between institutions in students’ responses to feedback on their courses, as seen in Table 5. Table 5 Comparison of Science courses at Universities A and B in terms of students’ perception of the volume and promptness of feedback University

A University

B

Scale Scale Score t p [1]

Quantity and timing of feedback 22.0 (s.d. 4.40)

15.6 (s.d. 4.48)

19.28 (d.f. 766)

p < 0.001

Quality of feedback 22.4 (s.d. 3.84)

19.0 (s.d. 4.00)

11.47 (d.f. 763)

p < 0.001

Sample items % agree On this course I get plenty of feedback on how I am doing

68% 26%

The feedback comes back very quickly

65% 26%

Whatever feedback I get comes too late to be useful

11% 42%


- 7 -

University A invests more resources in frequent assignments, and especially in providing extensive written feedback on assignments, while University B invests more in classroom teaching, including informal feedback during face to face contact. For the lecturers involved it is therefore more useful to examine patterns of student response on different courses within their institution. This has led to three follow-up studies. University A Two assessment related phenomena were noted by teachers of the Science courses at University A. Differences of student response to assignments and examinations First, it was noticed that there was a very strong positive correlation between assignment marks and examination marks for some courses, but not for others, raising the possibility that for some courses examinations may be testing something different. Second, there were marked differences between courses in terms of student responses to some of the AEQ questions. For example in response to the item: “I understand things better as a result of the exam” on one course 61% of students agreed while on another only 28% agreed (Average item scores 3.7 and 2.7 respectively, t(155) = 5.56 p < 0.001). This was interpreted as indicating that on some courses the exam might not only be assessing different things, but doing so in a way which might be inappropriately mis-orienting the quality of student effort. As a consequence of this preliminary diagnosis a follow-up interview and questionnaire study is examining students’ perceptions of the demands of examinations and the effects this has on the way they go about their studying, and go about their revision, on six courses. This may lead to the redesign of examinations on some courses and to efforts to re-orient student effort appropriately. Changes will be evaluated using the AEQ and also by a repeat of the follow-up study. Student use of feedback In University A very considerable resources are allocated to providing written student feedback on regular assignments. As shown above in Table 5, students’ responses to this feedback are broadly positive. However there were found to be significant differences between courses. Table 6 shows student responses to AEQ items concerning feedback between the ‘best’ and ‘worst’ Science courses. Table 6 Comparison of Science courses within University A in terms of students’ use of feedback AEQ items ‘Best’

course ‘Worst’ course

% strongly agree The feedback helps me to understand things better 36% 6% The feedback shows me how to do better next time 31% 4% The feedback prompts me to go back over material covered earlier in the course

13% 1%

% agree or strongly agree I do not use the feedback for revising 17% 44%


- 8 -

These differences were interpreted by the lecturers involved as indicating that feedback is found useful, and used by students, to very different extents on different courses. However they were not clear why. A follow-up study will explore in depth the way students respond to specific feedback on their assignments in six courses. A system for categorising types of feedback has been developed by analysing the actual feedback provided for students by their tutors. Feedback on assignments on different courses is currently being categorised, by tutors themselves, to see if feedback takes a different form on different courses or when provided by different tutors. Students will then be interviewed, by tutors, immediately after receiving their assignments back, about how they responded to specific examples of feedback of each category on their own assignments, and what if anything they did in response to this feedback. This study may lead to: • changed briefing or staff development for tutors, to change their feedback • changed assignments and patterns of assignments, that offer more scope for the

provision of feedback and use of that feedback on subsequent assignments or for revision for the exam.

Such changes will be evaluated using the AEQ and also by a repeated analysis of tutors’ feedback and the uses students make of it. University B As indicated above, the main phenomenon in University B identified by the questionnaires was the relatively poor scores on the ‘Quantity and timing of feedback ‘scale, and slightly poorer scores for the ‘Quality of feedback’ scale. There were slight differences between some of the courses, though in some cases the numbers of students were so low as to make the differences insignificant. For this reason it is more useful to look at the aggregate scores and to compare the percentages who agree and strongly agree with those who disagree and strongly disagree, as set out in Table 7. Students seem to want more feedback, more quickly, to help their learning. They are more divided as to whether the feedback they do get clears up mistakes or misunderstandings. On the whole staff were not surprised by the scores from the questionnaire and by the students’ comments; indeed, many were pleased with the results for their own courses. However, staff felt very strongly that they were giving lots of feedback in various forms and are concerned that students hold different perceptions as to what constitutes ‘feedback’. It seems to be the case that students only count something as feedback if it is written down whereas lecturers feel they are giving feedback in lectures, laboratories, workshops and more informally. This mismatch between student and lecturer perceptions is providing the focus for interviews, planning for future interventions and the way in which the nature of feedback is explained to students. However, even when special feedback sessions were provided in the past student attendance was poor, so students are complaining that they want more feedback but then do not turn up to classes where they could obtain such feedback. A feeling amongst lecturers is that verbal feedback has a high impact and can be given much closer to the time of assessment, but that this is only short lived and obviously limited to those who receive it. By contrast, written feedback, whilst potentially longer lasting as it is a permanent record, is not always


- 9 -

taken up as an explanation but rather is read as a statement of fact, i.e. the fact that something was wrong rather than why that was the case. It also takes longer to read assessments and prepare written feedback than to give oral feedback. For these reasons verbal feedback will continue to be important, though it is obviously necessary to consider how to make its impact longer lasting. Table 7 Scores for items under the ‘timing and quality of feedback’ scale for University B Items % disagree

and strongly disagree

% agree and strongly agree

Average item score

(University A) On this course I get plenty of feedback on how I am doing

46% 26% 2.72 (3.73)

The feedback comes back very quickly 64% 16% 2.3 (3.61)

There is hardly any feedback on my assignments when I get them back

40% 29% 2.89* (1.80)*

When I get things wrong or misunderstand them I don’t receive much guidance on what to do about it

33% 36% 3.11* (2.11)*

I would learn more if I received more feedback

7% 76% 3.95* (3.00)*

Whatever feedback I get comes too late to be useful

27% 42% 3.26* (2.15)*

Overall Scale score 15.6 (22.0)

* the reverse coding on these questions means that a low score indicates higher student satisfaction. The qualitative data showed some differences between courses, though there were also significant differences between students’ views on the same course. This issue is being examined further in the interviews with students as many of them are taking more than one of these courses included in the study and it is not clear to what extent they are focusing on a single course or making general comments. As with University A, the next phase of the project for University B is to consider the effects of feedback and, in particular, to devise approaches which encourage students to engage with the variety of feedback methods used. Given the greater amount of face-to-face contact compared with University A, courses will continue to build on this important aspect of course delivery, whilst trying to narrow the gap between students’ and lecturers’ perceptions of the nature and form of feedback. Course teams are also keen to make greater and more effective use of Blackboard, the University’s chosen virtual learning environment, as this becomes more embedded in students’ everyday learning experience. This will include greater use of communications facilities for peer and tutor feedback, and multiple choice questions (MCQs) with its potential for instantaneous feedback. Future development Once the follow-up studies have been complete the project will involve principled redesign of aspects of the assessment on each of the courses involved. The impact of these changes will be evaluated using the AEQ and also the more focussed


- 10 -

evaluation methods associated with each follow-up study (as in the University A examples above). The intended research outcomes are: • a development and elaboration of the conceptual framework currently involving

the ‘eleven conditions under which assessment supports student learning’ outlined in Table 1;

• the standardisation of the AEQ so that it can be administered and interpreted by university teachers to diagnose strengths and weaknesses in assessment on their courses (Gibbs & Simpson, 2003);

• the documentation of other evaluation tools (such as the category system for analysing feedback, as in University A, above);

• the documentation of a wide range of case studies of improved student learning associated with changes to assessment;

• an elaboration of Table 2, linking the conditions’ to assessment tactics. References Carroll, M. (1995) Formative assessment workshops: feedback sessions for large classes. Biomedical Education 23, 2, pp 65-67. Gibbs, G. (1999) Using assessment strategically to change the way students learn. In S. Brown & A. Glasner (Eds.) Assessment Matters in Higher Education. SRHE/Open University Press. Gibbs, G. (2002) Evaluation of the impact of formative assessment on student learning behaviour. Learning communities and assessment cultures: connecting research with practice. European Association for Research into Learning and Instruction. Newcastle: Northumbria University. August 2002. Gibbs, G. & Simpson, C. (2003) Measuring the response of students to assessment: the Assessment Experience Questionnaire. 11th International Improving Student Learning Symposium, Hinckley, England. Gibbs, G. & Simpson, C. (in press) Does your assessment support your students’ learning? Journal of Learning and Teaching in Higher Education. Sly, L. (1999) Practice tests as formative assessment improve student performance on computer managed learning assessments. Assessment and Evaluation in Higher Education, 24, 3, pp339-344.

Documents

Improving student learning through changing assessment · PDF fileImproving student learning through changing assessment – a conceptual and practical framework - 5 - students, and