Satisfied – but students want more feedback: Why did I get 37%?

1

University of ExeterLearning and Teaching Conference, May 22nd 2008

Professor Brenda SmithHigher Education Academy

Aims…

• Emphasis assessment for learning rather than of

learning

• Encourage the active engagement of students in their

learning

• Enhance assessment and feedback process for both

staff and students

Road Map of the Presentation

• Myths about assessment

• Burgess Report – a case for change

• Assessment methods – have we got it right?

• NSS tables in relation to Exeter

• Focus on feedback

John Cowan, former Director, OU, Scotland

Assessment is the engine that

drives learning

5

1.Certification

3.Learning

4.Sustainability

2.QualityAssurance

Why do we assess?

What’s wrong with assessment?

The evidence:• QAA Subject Reviews• National Student Survey• Rising concern about cheating and plagiarism• Student complaintsSome issues:• Too much summative and insufficient formative

assessment• Insufficient effective feedback that comes too late• Variations in practice• Unscholarly practices (especially with numbers)• Myths and traditions that inhibit change

Myths…

• Students benefit from jumping assessment hurdles unaided

• Experienced tutors apply consistent assessment standards

• Students don’t want feedback on exams

• That it is possible to distinguish the quality of work to a precision of one percentage point

• Moving back to exams will solve ‘cheating’ problems

• Double marking will ensure fairness and reliability

• Assessment standards are consistent across an institution

The Burgess Report

The Burgess ReportA case for change

• Present system cannot capture achievement in some key areas of interest to students and employers

• Emphasis on the top two degree classes wrongly reinforces an impression that a Lower Second or Third Class Degree is not an achievement

• Institutional methods for calculating the degree classification could be clearer

The Burgess ReportA case for change

• A summative system – is at odds with lifelong learning

• There is a need for greater emphasis on additional information that is currently contained in the European Diploma Supplement and academic transcript (an official record of a learner’s programme of study, grades achieved and credit gained)

• Institutional practices – complex and inconsistent

Complexity, variety and inconsistency of classification assessment regulations

(SACWG) surveyed 35 HEIs

25 used percentage marks; eight used grades; and two used both percentages and grades

18 institutions aggregated or averaged percentage marks to arrive at the honours degree classification; 4 used a ‘profiling’ approach; and 13 used both or either

SACWG: Student Assessment and Classification Working Group

Complexity, variety and inconsistency ofclassification assessment regulations

Variations in degree classification can be caused by:

Differing institutions’ award algorithms

‘Dropping’ a no of modules when calculating the degree classification

A shift towards course work and away from formal examinations

Institutions diverged on their approaches to borderline performances

Variations in the weightings given to the penultimate and final years of full-time study

Ref: SACWG and Mantz Yorke et al

Relative weightings given to results at Levels 2 and 3

Ratio, Level 3:level 2 Frequency

4:1 1

3:1 4

70:30 3

2:1 4

3:2 2

4:3 1

1:1 5

Ref: Mantz Yorke et al

A single index of performance, whether it is agrade-point average or an honours degree classification, tells nothing on its own aboutthe trajectory of a student’s development. It does not differentiate between the student who progresses steadily over the course of a programme and the student who starts slowly but in the final year produces work of an outstanding standard

Issues involving student trajectory

The student voice - what they would like…

More advice& support

Relevant &real-life topics

Choice of assessment

topicsVariety of assessment

methods

Feedback to help me

learn

For the assessment load to be spread out

Clear criteria

MESA Project

I never realised that assessment was

for learning

Department AYear 1

100

80

60

40

20

0

20

40

60

80

100

exam coursework

Year 2

100

80

60

40

20

0

20

40

60

80

100

exam coursework

Year 3 / Final Year

100

80

60

40

20

0

20

40

60

80

100

exam coursework

Department B

Year 1

100

80

60

40

20

0

20

40

60

80

100

exam coursework

Year 2

100

80

60

40

20

0

20

40

60

80

100

exam coursework

Year 3 / Final Year

100

80

60

40

20

0

20

40

60

80

100

exam coursework

Department C

Year 2

100

80

60

40

20

0

20

40

60

80

100

exam coursework

Year 1

100

80

60

40

20

0

20

40

60

80

100

exam coursework

Year 3 / Final Year

100

80

60

40

20

0

20

40

60

80

100

exam coursework

Department D

Year 1

100

80

60

40

20

0

20

40

60

80

100

exam coursework

Year 2

100

80

60

40

20

0

20

40

60

80

100

exam coursework

Year 3 / Final Year

100

80

60

40

20

0

20

40

60

80

100

exam coursework

Assessment Methods at Exeter?

University of Hertfordshire analysed assessment methods across the institution

Tasks• Multiple choice• Open book• IT based• Interactive• Group element• Role play

Medium• Oral• Numeric• Diagram/

Pictorial

Cognitive skills• Analytic• Evaluative• Skills focus• Primary research• Theory focus

Time-span• Reflective• Process/Periodic• Portfolio

Who assesses?

Self assessPeer assessSelf set element

Work-related

Practice focusCase Study

University of Hertfordshire

National Student Survey(Full-time students UK)

The Assessment and feedback scale 2006 2007

1. The criteria used in marking have been made clear in advance

67 68

2. Assessment arrangements and marking have been fair

73 72

3. Feedback on my work has been prompt

51 51

4. I have received detailed comments on my work

55 54

5. Feedback on my work has helped me clarify things I did not understand

51 51

Video Man Friday

• Please watch the video and ask yourself are you always clear about the criteria you set for students?

• How do you ensure that they understand what is required of them?

• Could you give students practice in setting and then marking against criteria?

Comparison of courses within ‘x’ University in terms of students’ use of feedback

Assessment Experience Questionnaire

(AEQ items)http://www.open.ac.uk/science/fdtl/overview.htm

‘Best’ course

‘Worst’ course

%

strongly agree

% strongly

agree

The feedback helps me to understand things better 36% 6%

The feedback shows me how to do better next time 31% 4%

The feedback prompts me to go back over material covered earlier in the course

13% 1%

% agree

stronglyor

agree

I do not use the feedback for revising 17% 44%

Everyone can make a difference..

What are the positive features of your module/course?

How do you:• Identify them?• Disseminate them within departments?• Disseminate across the institution?• Disseminate externally?• Draw on effective practices in learning and teaching

from external sources?• Embed that practice?

Dissemination

1. For awareness

2. For understanding

3. For action.

By ‘action’ here we mean some change of

practice inspired by, and making use of, the

materials or processes produced by your initiative

4. Evaluate the success of your dissemination

Conditions necessary for innovations to improve Learning, Teaching & Assessment

• Implement in student focused ways

• Departments that value teaching

• Workload that are not too high

• Effective leadership of learning, teaching &

assessment

• Perceived recognition and rewards

Recommendations in relation to support, adaption & implementation

1. Recognise in workload allocation formulae

2. Recognise scholarship of teaching outcomes – publications etc

3. Provide small scale grants

4. Provide sources of institutional support and advice

5. Encourage development of cross institutional networks & events

6. For academic developers to work alongside ‘project teams’

7. Ensure time release as appropriate

Ref: Carrick Institute, Australia

Managing the Assessment process

FEEDBACK TO HELP STUDENTS TO LEARN

Why feedback?

• What kind of feedback do learners need?

• How much is enough?

• How often is often enough?

• How can we make feedback efficient and effective?

• How can we ensure feedback is taken seriously?

• Should feedback be one way?

• Can students give each other feedback?

Why give feedback?

• Increase motivation and interest (Give choice)

• Improve performance

• Eliminate misconceptions

• Develop independence

• Raise confidence

• Create trust

• Individualise learning

• Develop a learning community

Paradoxes of feedback

• Teachers complain about it

• Students ignore it

• They get it too late

• When they get it it’s not useful

Questions…..

• How many hours do you spend giving feedback?

• At what time in the semester do you give most feedback?

• Could you give more feedback upfront?

• Could feedback be spread out throughout the year?

• Could you encourage peer feedback?

• Could you give feedback in different ways (MP3)?

Staff comments on feedback and marking

“They often don’t even collect their essays…..Some of them are more ready to argue about their mark than they are to collect their essay”

“They came up to me afterwards…and they said well we didn’t really understand what you meant by evaluate. And I kind of thought well isn’t it obvious, but the more I think about it I wonder if it is actually obvious to students, what the word means”

Ref:www.assessmentplus.net

45

Do we try and trip our students up?

Evaluation of tutor feedback

om

ission

Use

of E

nglish

Ask

for

clarifi

catio

n

Erro

r

Giv

e cla

rifica

tion

Pra

ise

Enco

ura

gem

ent

Feed fo

rward-iv

e

30%

20%

17%

11.5%

10.5%

3.1%

1.5%

0%0%

OU/SHU Improving the Effectiveness of Formative Assessment in Science

A study (Carless 2006) reported a substantial difference in the perceptions of tutors and their students on feedback on assessed work

A study at the University of Sydney, ran focus groups with 5,000 students to gain an in-depth understanding of student perceptions of feedback

3 dimensions emerged

Differing perceptions (most recognisable if written)

Impact of feedback – timeliness, significance & 1st year experience (difference between University and School)

Credibility of feedback – this was related to the students’ perceptions of the lecturers themselves

Do you know what your students think about feedback?

Students comments

They are not very specific when they mark. I got “some good evaluation”, what does that mean?

You put so much effort into this work that you really want feedback and to know they have read it properly. If you just get a mark you think, “I’ve worked for days and that’s it?”

I think it would be helpful as well if when we had our feedback we could relate it to the assessment criteria

Ref:www.assessmentplus.net

49

Do these comments aid learning?

A Excellent

C- Fine

C/C- Interesting

C Opening lousy - main part OK

C- Your writing is appalling

D Terrible - didn’t you learn anything at school about essay writing!

Comments difficult to understand…

This essay is not sufficiently analytical

This report is not logically structured

However we also need to focus our efforts on strengthening the skills of self-assessment

Students need to be active in this process

We need to use language that they can understand

How might we use students in this process?

Involve the studentsFeedback as a dialogue

For Student Completion

1. These are areas of my work that I think are good for the following reasons

2. Please comment on the following areas of work

3. What I want to improve or do differently next time

4. The mark I think this piece of work deserves is

For Staff Completion

Order of giving feedback

1. Good news – what was done well

2. Enhancement news – what still needs to improve

3. Options – what can be done to improve

4. Plans – what the learner intends to do

5. Commitments – what both parties agree to do, how, to what standard and by when?

Formative feedback - a challengeto course teams

• Is the feedback motivating and does it aid learning?

• Do students use feedback to improve future performance?

• Do you and the student track the

changes?

• How can you improve the need for

students to want feedback and use it as feed forward?

• Can you give credit to students

who use their feedback?

How can we engage with our students?

• Engage from day 1• Rethink induction• Get students involved in setting and marking

against the criteria• Encourage peer feedback & peer assessment• Rethink the course committee agenda• Ask their views

The one minute paper

Structured feedback session…..

Structured Feedback Session

This is an activity to give an individual and collective voice to your experiences of assessment and feedback

TASK• Individually think of all the good things about your

experiences of assessment and feedback• Write each one on a separate post-it note• In your group put all your post-its on a flip chart and say

them out loud as you post them up• Now collectively agree on the top 3 aspects of your

course in relation to assessment and feedback that rate the highest

Structured Feedback Session

TASK• Individually think of all aspects of your assessment

and feedback experiences that could be enhanced• Write each one on a separate post-it note• In your group put all your post-its on a flip chart and

say them out loud as you post them up• Now collectively agree on the top 3 enhancements

that could be improved • Taking those top 3 aspects lists some ideas that

could be taken forward

Applying the seven principles of assessment

Seven principles of good feedback practice

1. Facilitates the development of self-assessment (reflection) in learning

2. Encourages teacher and peer dialogue around learning

3. Helps clarify what good performance is (goals, criteria, standards expected)

4. Provides opportunities to close the gap between current and desired performance

(SENLEF Project)

Seven principles of good feedback practice

5. Delivers high quality information to students about their learning

6. Encourages positive motivational beliefs and self-esteem

7. Provides information to teachers that can be used to help shape the teaching

Ref: Juwah, Macfarlane-Dick, Matthew, Nicol, Ross & Smith (2004)

A vision of students

Do league tables give us the whole picture?

• Does our position on the league tables guarantee a

really positive learning experience for all our

students?

• How might we collectively work towards an

enhanced student experience?

Financial indicators Job prospects Academic indicators Ease of entry and demand Student life indicators

1 Lampeter, University of Wales2 Heythrop College, University of London3 Stirling University4 School of Pharmacy, University of London5 University of St Andrews6 St George's, University of London7 Harper Adams University College8 Bishop Grosseteste University College9 King's College, London10 University of Edinburgh

113 University of Exeter

Congratulations

EXETER is a top 20 UK University, according to all four of the main higher education league tables

BUT does that guarantee that the student experience is enhanced?

HOW can you make that student learning experience even better?

Reflections …..