Aligning student and staff expectations around

assessment: an intervention study to help undergraduate

students understand assessment criteria for examination essays

European Congress of PsychologyOslo, July 2009

Kathy Harrington, Mercedes Freedman, Savita Bakhshi, Peter O’NeillWrite Now Centre for Excellence in Teaching and Learning

London Metropolitan University

2

Study aims Improve students’ understanding of and ability to

demonstrate assessment criteria applied to their timed, unseen examinations

Leading to improved module performance In addition, we wanted to…

Build on previous research on use of assessment criteria to improve student learning

Develop the intervention through collaboration between Psychology academics and writing specialists (Writing-in-the-Disciplines model)

Incorporate peer mentoring in academic writing

3

Context of study 2nd-year cognitive psychology module Assessment 100% by examination Traditionally lower than average pass rate

Identified by university as a “killer module” Plan of response required

Write Now CETL works with discipline-based academics to improve student learning through curriculum and teaching development

Write Now CETL runs university Writing Centre staffed by trained student peer mentors in academic writing Collaborative, non-directive, supportive Enable students to take responsibility for own work

4

Pedagogical rationale Students and tutors often interpret meanings of assessment

criteria differently (Harrington et al., 2006; Lea & Street, 1998; Merry et al., 1998)

Providing clear and explicit criteria is a first step in helping students understand what tutors are looking for in written work

However, research has also shown that facilitating students' active engagement with the criteria is necessary if learning and performance are to be demonstrably enhanced (Price et al., 2001)

Structured interventions focussed on understanding and demonstrating assessment criteria have been shown to lead to improvements in student learning and performance (Norton et al., 2005; Rust et al., 2003)

Other research has found that students feel talking to peer tutors about their writing leads to better writing, and that psychology students prefer working with peer tutors from their own discipline (Bakhshi et al., 2009)

5

The intervention 4, hour-long compulsory workshops embedded as part of

module teaching across the autumn semester 2008-09

Immediately following two-hour lectures

Delivered in alternate weeks, with workshops run by subject lecturers in between (focussed more explicitly on lecture content)

Designed by team of academic writing specialists, psychology lecturer, psychology PhD student with experience of pedagogical research in area of student

writing and assessment

Delivered by academic writing specialists and peer writing mentors studying psychology (3rd-year and PhD students)

6

The intervention (continued) Exam answers posted in VLE prior to workshops for

students to read and give a grade

Set of accompanying materials developed using extracts from authentic examination answers annotated with comments in relation to assessment criteria Specific focus on cognitive psychology Use of departmental assessment criteria

In workshops, materials used to facilitate discussion about demonstrating assessment criteria at different levels of performance

Students guided in small groups to adopt role of examiner and apply criteria to whole past examination answers

Final class discussion drew out students’ insights and summarised main points

7

Data collection and analysis Attendance registers taken at workshops 2, 3 and

4 (not at first workshop) Questionnaire distributed at last workshop (n=63)

Likert scale: students’ perceptions of helpfulness of workshops in relation to

Examination writing Meeting assessment criteria Understanding subject matter of cognitive psychology

Examination grades Analysis using SPSS to produce descriptive and

inferential statistics

8

Study sample N=205 students enrolled on the module who took the

examination 40 students enrolled on module did not take the examination

Study population Total students enrolled on module

Conversion Diploma 64 (31.2%) 72 (29.4%)

Single Honours 93 (45.4%) 113 (46.1%)

Other/unidentified 5 (2.4%) 11 (4.5%)

Joint Honours 43 (21%) 49 (20%)

Total 205 (100%) 245 (100%)

9

Findings: pass rate and mean grade

Number of students

Pass rate Fail rate Mean final grade

2007-08 191 71% 29% 47%

2008-09 205 62% 38% 43.6%

10

Percentage who attended workshops

Frequency Percent Valid Percent Cumulative Percent

Valid

0 100 48.8 48.8 48.8

1 50 24.4 24.4 73.2

2 29 14.1 14.1 87.3

3 26 12.7 12.7 100.0

Total 205 100.0 100.0

• 100 (48.8%) students did not attend any workshops• 105 (51.2%) students attended at least one workshop

11

Attendance and pass/fail in examination

Pass or fail

Test Value df

Asymp. Sig. (2-

sided)Pass Fail Total

Attendanceat

at leastone

workshop

NoCount 51 49 100

ContinuityCorrection 9.047 1 .003**

% 51.0% 49.0% 100.0%

YesCount 76 29 105

% 72.4% 27.6% 100.0%

TotalCount 127 78 205

% 62.0% 38.0% 100.0%

12

Attendance and final grade category

0

10

20

30

40

50

60

0 1 2 3Number of workshops attended

%

1st

2.1

2.2

3rd

Fail

Figure 1: Percentage of students who attended workshops by grade category

13

Workshop attendance and grades: 1

Figure 2: Correlation between number of workshops attended and final grade

• Significant positive relationship (r=.314, n= 63, p<0.05) • The more workshops attended, the higher the grade

14

Workshop attendance and grades: 2

Figure 3: Total number of workshops attended and mean final grade

Mean final grade overall: 43.61%

Conversion Diploma: 60.34%Single Honours: 38.8%Other: 58.2%Joint Honours: 27.44%

15

Similar findings in other research Lusher (2007)

Small-group workshops focused on assessment criteria with 3rd-year health psychology students

Significant correlation between attendance and mean examination scores (r=0.254, N=111, p<0.01)

Multiple regression showed that performance did not independently predict attendance, so not just a matter of more able students attending workshops

16

Students’ perceptions of workshops 7-point Likert scale: strongly disagree to

strongly agree Mean scores for all items were positive

Understanding what assessment criteria are Understanding how to demonstrate them Understanding what makes a good

examination essay in Cognitive Psychology Understanding subject matter of cognitive

psychology Achieving a better grade Producing better writing

17

Conclusions Module pass rate lower this year at 62%, compared to 71% in

2007-08

However, a number of minor changes were made to content and delivery, so comparison across years problematic

In 2008-09, attendance at the workshops was significantly correlated with higher examination grades

Confounding factor is that more able students are more likely to be attending in first place

More tests needed, cf. Lusher (2007)

Students’ who attended perceived the workshops to be helpful for Understanding what assessment criteria are Understanding how to demonstrate the criteria in their own writing Understanding the subject matter of cognitive psychology Achieving a better grade Producing better writing

Difficulty of addressing needs of weaker students, even with “embedded” teaching

18

Conclusions (continued) Substantial improvement in student learning and performance

may require changing the method of assessment

However, given that this may not be possible (due to institutional constraints), other modifications will be considered based on

Student feedback on questionnaires Importance of embedding “writing to learn” activities within modules,

rather than treating writing as an add-on skill separate from learning subject matter

Benefits to students of collaborative learning environments

Planned changes to module for next year Less distinction between lectures and workshops by identifying 3-hour

“teaching blocks” instead, with varied mix of lecture and workshop activities, to try to encourage higher attendance

Time to practice writing in teaching sessions Peer review of own writing, facilitated by peer mentors

19

ReferencesBakhshi, S., Harrington, K., and O'Neill, P. (2009). Psychology students’ experiences of

academic peer mentoring at the London Metropolitan University Writing Centre, Psychology Teaching and Learning, 8(1), 6-13.

Harrington, K., Elander, J., Norton, L., Reddy, P., Aiyegbayo, O. & Pitt, E. (2006). A qualitative analysis of staff-student differences in understandings of assessment criteria, in C. Rust (Ed.), Improving Student Learning through Assessment. Oxford: Oxford Centre for Staff and Learning Development.

Lea, M. R. & Street, B. (1998). Student writing in higher education: an academic literacies approach, Studies in Higher Education, 23, pp. 157-72.

Lusher, J. (2007). How study groups can help examination performance, Health Psychology Update, 16, 1 & 2.

Norton, L., Harrington, K., Elander, J., Sinfield, S., Lusher, J., Reddy, P., Aiyegbayo, O. & Pitt, E. (2005). Supporting students to improve their essay writing through assessment criteria focused workshops, in C. Rust (Ed.), Improving Student Learning: Inclusivity and Diversity. Oxford: Oxford Centre for Staff and Learning Development.

Merry, S., Orsmond, P. & Reiling, K. (1998). Biology students’ and tutors’ understanding of a ‘good essay’, in C. Rust (Ed.), Improving Student Learning: Improving Students as Learners. Oxford: Oxford Centre for Staff and Learning Development.

Price, M. and O'Donovan, B. and Rust, C. (2001). Strategies to develop students' understanding of assessment criteria and processes, in C. Rust (Ed.), Improving Student Learning - 8: Improving Student Learning Strategically. Oxford: Oxford Centre for Staff and Learning Development.

Rust, C., Price, M. & O’Donovan, B. (2003). Improving students’ learning developing their understanding of assessment criteria and processes, Assessment and Evaluation in Higher Education, 35, pp. 453-472.

20

http://www.writenow.ac.uk