Grading criteria and marking schemes
Liz NormanMassey University
Validity again…
• The questions must elicit the behaviour we want to evaluate
• Markers need to reward features we want to evaluate and not reward features we do not want to evaluate
• Different markers need to award similar/scores for the same candidate response
Purpose of marking schemes
• To help you during Q writing– What content is important– Whether the Q asks what you intended it to ask– Whether it is do-able in the time available
• To help you during Q marking– helps you decide how to award marks to any
particular candidate response– facilitates reliable and fair marking
Types of marking scheme
1. Model answer – “ideal” answer2. Point-based schemes3. Criteria- & level-based schemes4. Schemes with incorporated principle(s) for
discriminating levels
Model answers
• Not recommended as a sole component of a marking scheme– Usually more than would be expected to be given
by any candidate– No guidance on how to assess alternative answers
to the model provided– No guidance on how to award marks
• Can be a useful adjunct to a marking scheme
Point-based schemes
• Points for each objectively identifiable content point
• Does not indicate the relative importance of the points it awards
• Sum can be more or less than the whole• Rewards quantity not quality
Prestructural
Unistructural Able to identify, list, name, enumerate but does not describe, explain, relate or elaborate multiple aspects of a response
Multistructural
Relational
Extended abstract
Quantitative change
Qualitative change
Criteria- & level-based schemes
• Criteria – different dimensions of performance• Level – different quality/standards of
performance on a given criteria
Criteria- & level-based schemes
• Generic vs specific for the Q• Explicit vs implicit weighting
Standards Very poor Poor Fair Excellent
Criteria
Analysis and interpretation of resultsTotal 8 marks
0-2 marksInterpretation not provided or incorrect.
2-4 marksLacks one or more key elements.
4-6 marksAdequate interpretation that addresses key elements. Misses nuances of interpretation or uncommon differentials
6-8 marksThorough accurate interpretation of results. Well justified and appropriately prioritised list of differentials.
Quality of planningTotal 8 marks
0-2 marksNo plans provided or plans not appropriate or dangerous
2-4 marksPlans miss some key aspects or overly general
4-6 marksAdequate plans that address all key differentials. Some displaced in priority or not pragmatic
6-8 marksThorough detailed and well-prioritised and pragmatic plan that addresses all defined differentials.
Knowledge of current literatureTotal 5 marks
0-2 marksLittle or no literature referred to or incorrectly referred to.
3-4 marksAnswer refers to some of the key literature
4-5 marksAnswer refers to current literature including controversies and comparative work from other species.
Logical presentationTotal 4 marks
1 mark Answer is disorganised and includes a large amount of irrelevant material
2 marksAnswer is somewhat disorganized and includes some irrelevant material
3 marksThe answer is relatively well organized and contains little irrelevant material.
4 marksThe answer shows a high degree of logical thought and well-constructed argument.
Analytical vs holistic schemes
• Both are valid• Analytical (criteria scored separately)
– Better agreement between examiners– Insufficient criteria– Overlapping criteria– Really just lots of smaller holistic decisions
• Holistic (scored as a whole)– Don’t straightjacket examiners– Challenging, especially for longer answers– Less agreement between examiners
Prestructural Question may be rephrased as the answer; almost completely misses the point of the question.
Unistructural Able to identify, list, name, enumerate but does not describe, explain, relate or elaborate multiple aspects of a response
Multistructural Able to list as well as describe distinct aspects of a response (such as being able to describe aetiology, clinical features, management of thrombotic stroke) but unable to explicitly explain causes for observations; unable to present cause-effect relationships.
Relational Able to describe multiple aspects of a process and additionally explain or elaborate observations into cause-effect relationships; able to compare similarities and differences between apparently distinct phenomena. This level is taken as suggesting that the learner has understood.
Extended abstract
Highly developed; able to explain mechanisms of phenomena and apply this information to a novel context — able to develop novel hypotheses, theories, and deduce principles; creative thinking.
Prakash et al. (2010). Adv Physiol Educ, 34(3), 145-149.
SOLO levels in marking schemes
Prestructural The task may be engaged, but the student is distracted or misled by irrelevant aspects
Unistructural The student focuses on the relevant domain and works with a single aspect
Multistructural The student provides correct material with discrete, separate pieces of information that may be combined to provide a composite picture
Relational The student offers an integrated understanding of the information. The whole has a coherent structure and meaning
Extended abstract
Abstract general principles or hypotheses are provided
Scholten I, Keeves JP, Lawson MJ. High Educ 44:233–255, 2002.
Incorporating principles/rules
• Ideal is a specific criteria- & level-based schemes with incorporated principle(s) for discriminating levels
Incorporating principles
• Calais has a warmer winter and a cooler summer than Wroclaw. Explain why. (3 marks)
• Marking scheme: Looking for answers related to distance from the sea therefore latitude is not credited.– Land heats up quicker than sea (1 mark)– A clear distinction between land and sea heating
(2 marks)Ahmed & Pollitt (2011) Improving marking quality through a taxonomy of mark schemes. Assessment in Education: Principles, Policy & Practice 2011;18:259-278.
Incorporating principles
Marking scheme excerpt:Discussion should focus on strategies adopted to ensure sustainability and an evaluation of these with regard to whether or not or to what extent the Sahel can be sustainably managed. The discussion will depend on the content and whether the overall view is optimistic or pessimistic.
AQA (2013) General Certificate of Education (A-level) Geography Unit 1: Physical and Human Geography http://www.aqa.org.uk/subjects/geography/a-level/geography-2030/past-papers-and-mark-schemes
Quality vs quantity
• More complex and unstructured the Q the more assessing quality not quantity
• In very constrained tasks only judging how correct the answer is
• In very open tasks, “correctness” is less important and its quality that's judged instead
Prestructural Question may be rephrased as the answer; almost completely misses the point of the question.
Unistructural Able to identify, list, name, enumerate but does not describe, explain, relate or elaborate multiple aspects of a response
Multistructural Able to list as well as describe distinct aspects of a response (such as being able to describe aetiology, clinical features, management of thrombotic stroke) but unable to explicitly explain causes for observations; unable to present cause-effect relationships.
Relational Able to describe multiple aspects of a process and additionally explain or elaborate observations into cause-effect relationships; able to compare similarities and differences between apparently distinct phenomena. This level is taken as suggesting that the learner has understood.
Extended abstract
Highly developed; able to explain mechanisms of phenomena and apply this information to a novel context — able to develop novel hypotheses, theories, and deduce principles; creative thinking.
Prakash et al. (2010) Adv Physiol Educ, 34(3):145-149
Writing marking schemes
• Select and organise the criteria/dimensions• Develop clear descriptions for each
level/standard of each criteria• Need to think about poor answers as well as
good ones
Reconsidering question wording
• Writing marking schemes can alert you to question problems
• The question may not elicit a higher order response from the candidate.
• Candidates cannot demonstrate higher order thinking unless you ask them to
• Think about what aspects of their answer will alert you to their higher order skills
Disconnected knowledge Connected knowledge
• No particular order to aspects presented
• Inclusion of irrelevant material• Inconsistencies or incorrect
aspects• Superficial or oversimplified• Replication of material from
sources – rote learned or reproduced without significant transformation
• Aspects explained relative to one another• Logically organised answer• Analysis and or synthesis• Compares similarities and differences• Integrates multiple levels (eg: molecular, biochemical,
systemic)• Evaluates inconsistencies• Expresses reasons• Explains implications or reaches a conclusion• Expresses relative importance, value, significance of
aspects. • Selective answer that addresses the point of the
question and may be shorter than a multistructural answer
• Uses the language of the discipline - terminology and phrasing
• Relates answer to examples or experience• Relates answer to organising principles of the discipline
Biggs, J. B., & Collis, K. F. (1982). Evaluating the quality of learning: The SOLO taxonomy (structure of the observed learning outcome). New York: Academic Press.
Using table format, compare and contrast the pre-, peri- and post-anaesthetic considerations and management of each of the following species; a ferret (Mustela putorius furo), a rat (Rattus norvegicus) and a central bearded dragon (Pogona vitticeps). List the mode of action of each drug referred to.
Revising mark schemes in use
• Hopefully all types of answer are anticipated• Sometimes though it is not – can indicate
unanticipated problems with the Q• Marking schemes might need revising after
first few candidates marked• HSE should moderate marking and should
encourage team members to report marking issues early in the piece.
Key points
Marking schemes
• Give an indication of the key criteria for discriminating passing answers from failing ones
• Then add in what would be a better performance and what would be a lesser performance
• Think about both quantitative and qualitative aspects of an answer
Marking schemes
• Quantitative:– Which parts of the answer are essential– How many errors in these parts are tolerated
• Qualitative– Thinking processes you want to examine– What connections are essential/important– How the structure of an answer will help you
differentiate a structured, related, connected thought process from a multistructural list of unrelated facts
Marking schemes
• Think about how a candidate who includes all/most of the right stuff but also includes wrong stuff will be graded
• What it indicates about the knowledge structure if there are contradictions/irrelevanciesin the answer
Marking schemes
• Keep them brief• Think about them as specifying principles to
apply rather than exhaustive detail• Concentrate on specifying the differentiating
characteristics and how much of this there needs to be
• Don’t use them to educate examiners about the topic