Standards, criteria and assessment of the Geography Dissertation in the UK

Dr Margaret E. Harrison

University of Gloucestershire

Dr Dawn T. Nicholson

Manchester Metropolitan University

Professor W. Brian Whalley

Queen’s University of Belfast

Assessment for Learning Conference, June 22nd 2009

Geography dissertations

'Dissertations have had a long history in geographical higher education, being widely regarded as the pinnacle of an individual's undergraduate studies and the prime source of autonomous learning' (Gold et al., 1991)

Often comprise a large proportion of the overall weighting for undergraduate degrees

Performance may be used to adjudicate degree award in borderline cases

Demonstration of independent working therefore grade awarded of interest to employers

Wider background

Marking reliability, marking to the full range Maintenance of standards in HE (grade inflation) Accountability (litigation) Increasing evidence of new assessment culture

emerging (Rust 2007) Assessment criteria (e.g. Harrison and Whalley 2008) Grade descriptors Formative assessment Feedback

But still poor practice that go unchallenged (Rust 2007)

Inconsistency, ambiguity etc

Inconsistent approaches to use of criteria (including weightings and standards) for dissertation marking (Hand and Clewes 2000)

Inconsistent use of terms in feedback (Pepper et al 2001)

Confusion among teachers over concepts of ‘criteria’ and ‘standards’ (Sadler 2005)

Considerable ambiguity concerning the use, meaning and application of assessment criteria (Webster et al 2000)

Students and criteria

Students don’t automatically understand the meaning of criteria (Penny and Grover 1996, Rust et al 2003, Woolf 2004)

Criteria need to be explicit and understood by students to effectively support learning (Gibbs and Simpson 2005)

An important principle is that students and markers are aware of and understand the assessment criteria and/or schemes that will be used to mark each assessment task. (QAA 2006, p16-17).

Some key research questions

What assessment criteria are used, how are they established and approved?

How are students assisted to interpret assessment criteria? What is the role of supervision in this?

What grade descriptors and marking schemes are used?

What are the procedures for double marking, anonymous marking, blind marking?

What happens in the event of a disagreement between first and second markers?

Identify ‘good’ practice and showcasing of innovative practice

Methods

Questionnaire survey sent to all GEES Departments – a mixture of closed and open questions

Follow-up telephone interviews with selected respondents

Number of responses: 22 This represents 20 HEIs - 2 separate responses from

2 HEIs. Responses include institutions throughout the UK with

an even split in terms of pre-1992 and post-1992 HEIs.

Credit-rating and word length

Credits (CATS)

15 20 24 30 40

No. responses

1 3 1 9 8

n=5

n=2 n=2

n=3

n=2R2 = 0.37R = 0.61df = 21

P =0.002

10

15

20

25

30

35

40

4000 6000 8000 10000 12000

Median word length

CA

TS

cre

dit

rat

ing

12 2

11

7

0

2

4

6

8

10

12

≤4000

≤6000

≤8000

≤10000

≤12000

Median word length

No

. o

f re

sp

on

ses

Form of dissertation

Responses

Printed document 21

E-document 6

Printed document & field notebook

1

Printed document & attachments (audio &/or visual)

4

Other 1

Study period and product format

Most dissertations are submitted as printed documents, but the handbook provides students with ‘the freedom to be creative about the form of the exercise.... the student can undertake a practical.... or a performance project.’

1

1110

0

2

4

6

8

10

12

8 - 9 10 - 12 > 12

Study period (months)

No

. o

f re

spo

nse

s

Elements assessed

Responses

Final document (100% of marks)

14

Final document (70-90%)

8

Written progress report (10-15%)

4

Seminar/oral presentation on progress (0-30%)

4

Final oral presentation (15%)

1

Other* 2* PowerPoint poster on progress and literature review

Elements that are assessed

2 2

4

14

0

3

6

9

12

15

70 80 90 100

% Final report weighting

No

. of

resp

on

ses

Assessment criteria

Clarity of aims, topic identification

Theoretical understanding

Awareness of literature

Research design and methodology

Quality of databaseData presentation, analysis and interpretation

Conclusions justified

Originality and perceptiveness

Sustained argument

Use of referencing conventions

Integration of illustrative materials

Standard of presentation

Use of English language

Use of complex academic terminology

Compliance with requirements

Risks and ethical considerations

Engagement with administrative processes

Cognitive, practical and personal skills

Reflective ability

Personal initiative and responsibility

Ability to work independently Conduct and competence during practical work

Use of ICT applications

Critical ability

Flair, innovation Think creatively

Assessment criteria

‘We used to have weightings for each bit but we now use a more general mark sheet. Every time we get a new external examiner they have different ideas and we tend to change our marking strategy fairly regularly’. (post 1992 HEI)

‘Established by tradition … some of our descriptors are rather woolly. … The bottom line criteria for our dissertations is that it must be original’. (pre 1992 HEI

Assessment criteria

‘Assessment scheme developed by a departmental working party … and then approved by a team meeting’ (equal weighting of 6 aspects including research skills, analysis/discussion, and presentation) (post 1992 HEI)

‘I’m not sure what you’re asking here – do you want to know the university’s quality assurance system?’ (pre 1992 HEI)

‘It was laid down at the start of the Modular Course in 1974; I’m not aware that it has ever changed. Indeed it is the only feature of the course that has never changed’. (post 1992 HEI)

Grade descriptors and marking schemes “See attached. These are departmental descriptors”

(pre-1992 HEI). “Attached – this is institutional” (post-1992 HEI). “The grade structure and broad descriptors are

university wide; the detailed interpretations were drawn up by the department” (pre-1992 HEI).

“We don’t have this by level. The University keeps promising it!” (post-1992 HEI).

Marking

“All departmental marking is anonymous, but clearly supervisors are likely to recognise the dissertations of ‘their’ supervisees” (post-1992 HEI)

95% marked by the supervisor

ALL are double marked (64% blind and 36% ‘seen’)

Six are marked anonymously…….?!

One uses viva (supervisor and second marker) to inform mark

Qualifications of the second marker – an issue?

21

6

2

8

14

0

4

8

12

16

20

Superv

isor

Anon.

Team

Double

2nd b

lind

Marking procedure

No

. of re

sp

on

ses

Disagreement generally >5% or different class Markers encouraged to discuss in first instance

“Two markers would meet and discuss the reasons for their grading, which may reveal something not noticed or unknown to one of them…..”

If no resolution, introduce third marker Use median mark from three Discuss and agree Discuss at internal exam board

If no resolution (still) Module leader decides or external examiner adjudicates

Disagreements about marks

Supervision

Supervisory contact

Responses

1-to-1 meetings

22

Small group sessions

8

Whole class sessions

11

Other 5

Examples of other activities: peer led work, presentations, field and laboratory training

9

3

10

0

2

4

6

8

10

Both Tutor Only StudentOnly

Who initiates meetings?N

o. o

f re

spo

nse

s

Feedback Only three (14%)

supervisors discuss content but don’t actually read, edit or comment on specific written material.

Some specify no. chapters or pages, % of work, specific chapters (eg lit. review) or specific date.

Feedback on complete draft rare - as work not available!

Feedback on drafts

Responses

No comment on any drafts

3

Limited number of draft chapters

8

Complete draft

11

“[Feedback] comments ‘should not compromise the independence of the project by modifying its principal organising ideas or otherwise adjusting its conceptual structure’.”

Supporting students

‘There are [also] group sessions…… and formal supervisory meetings where it is discussed’ (post-1992 HEI)

‘We top and tail the research design course with sessions on ‘What is research’ and ‘Writing up’, both of which include a discussion of expectations / features of a good dissertation / assessment criteria / grade descriptors’ (pre-1992 HEI)

‘Done as part of the supervision process’ (post-1992 HEI)

Scheme leader and tutors brief students on what is expected’ (pre-1992 HEI).

Supporting students

‘There is a lecture at the start of the final year, but this mainly comes through tutors.’ (pre-1992 HEI)

‘It is part of the tutorial support’ (post-1992 HEI)

Dissertation Guide – explanation of criteria Students peer-mark past dissertations using criteria Criteria discussed during Research design module Whole cohort lecture given to explain criteria

Examples of good practice

More good practice

“...introduction of the preliminary literature review as part of the Research Design [level 5] course”.

Students.... deliver oral presentations to a panel of supervisors and their peers”.

“Dissertation topics are not prescribed”.

“After a period in which dissertations were downgraded to an option, I think there is every chance that they might.... return to being compulsory”.

Conclusions

Requirements for dissertations vary but there is broad consensus across the sector

Consistent range of themes embedded in criteria, eg: Employability: Skills, conduct, reflection Research: Design, methods, data analysis Higher skills: Criticality, flair, innovation, originality

Lot of variation in the way that assessment criteria are established: Passive: Tradition, evolution, shared experience Reactive: Compliance with University generic criteria Proactive: Designed collaboratively and reviewed

Conclusions There is some confusion in use of the terminology Evidence of rigor and transparency in marking Disagreements are dealt with in a variety of ways and

the role of the external examiner varies considerably Supervision time is consistent but effective learning

can be achieved through other means A range of measures are in place to support students

interpret assessment criteria How do we know learning achieved?

Feedback on drafts is variable – implications for maintenance of standards and student independence

Implications

Care needed in the formulation of criteria that are explicit

HEI staff need to adhere closely to criteria to achieve equivalence and consistency

Given the confusion over terms and the widely differing practices, this has implications for staff development needs

All of this is irrelevant if students are not party to understanding criteria and expectations

Bettany-Saltikov, J., Kilinc, S. and Stow, K. (2008). Bones, boys, bombs and booze: an exploratory study of the reliability of marking dissertations across disciplines. Assessment and Evaluation in Higher Education 1, 1-19.

Gibbs, G. and Simpson, C. (2005). Conditions under which assessment supports students’ learning. Learning and Teaching in Higher Education 1, 3–31.

Gold, J. R., Jenkins, A., Lee, R., Monk, J., Riley, J., Shepard, I., and Unwin, D. (1991). Teaching Geography in Higher Education. Basil Blackwell, Oxford.

Hand, L. and Clewes, D. (2000). Marking the Difference: an investigation of the criteria used for assessing undergraduate dissertations in a business school. Assessment and Evaluation in Higher Education 25(1), 5-21.

Harrison, M. E. and Whalley, W. B. (2008). Undertaking a dissertation from start to finish: the process and product. Journal of Geography in Higher Education 32(3), 401-418.

Partington, J. (1994). Double-marking students’ work. Assessment and Evaluation in Higher Education 19(1), 57–60. Penny. A. J. and Grover, C. (1996). An analysis of student grade expectations and marker consistency. Assessment

and Evaluation in Higher Education 21(2), 173-184. Pepper, P., Webster, F. and Jenkins, A. (2001). Benchmarking in Geography: some implications for assessing

dissertations in the undergraduate curriculum. Journal of Geography in Higher Education 25(1), 23-35. Quality Assurance Agency (2006). Code of Practice for the Assurance of Academic Quality and Standards in HE:

Section 6 - Assessment of Students. Available online at: http://www.qaa.ac.uk/academicinfrastructure/codeOfPractice/section6/COP_AOS.pdf. [Accessed 4th March 2009].

Rust, C. (2007). Towards a scholarship of assessment. Assessment and Evaluation in Higher Education 32, 229-237. Rust, C., Price, M., and O'Donovan, B. (2003). Improving students’ learning by developing their understanding of

assessment criteria and processes. Assessment and Evaluation in Higher Education 28, 146-164. Sadler, D. R. (2005). Interpretations of criteria-based assessment and grading in higher education. Assessment and

Evaluation in Higher Education 30, 175-194. Webster, F., Pepper, D. and Jenkins, A. (2000). Assessing the undergraduate dissertation. Assessment and

Evaluation in Higher Education 25(1), 71-80. Woolf, H. (2004). Assessment criteria: reflections on current practices. Assessment and Evaluation in Higher Education

29(4), 479-493.

References

Thank for your attentionAny questions?

This project is being supported through the GEES Subject Centre Small-Scale Project Funding scheme

We would like to acknowledge Karen Logan, our Research Assistant from Queen’s University Belfast, for her assistance with this project.

Survey Questions 1

Do all your students undertake a Dissertation as part of their Geography Honours degree programme?

How many calendar months do your students have to work on their Dissertation?

What is the credit rating (CATS or ETCS credits) for a Dissertation in your Department?

Does any element of a Dissertation contribute to credit at less than final year?

Survey Questions 2

Does your Department specify the number of hours of supervisory contact permitted per student?

What form does the supervisory contact take and who initiates supervisory sessions?

What is your Department position concerning feedback on drafts?

What form may a Dissertation take?

Please indicate sub-elements of the Dissertation and their relative weightings.

Survey Questions 3

Please provide a copy of your Department marking schemes and/or assessment criteria.

Please indicate how this scheme / criteria is (a) established and (b) approved.

Please comment on any assistance given to students in interpreting the assessment criteria provided.

Please provide a copy of the grade descriptors or 'standards' scheme for the Dissertation.

Survey Questions 4

Which marking procedures are adopted for Dissertations (e.g. double marking; blind marking; anonymous marking).

What procedures are followed in the event of a disagreement between markers.

Please comment on whether external examiners are asked to adjudicate.

Please provide an example of good practice in Dissertation assessment.