View
2.269
Download
0
Category
Preview:
DESCRIPTION
Grading criteria and marking schemes A presentation given at the Australian and New Zealand College of Veterinary Scientists Examination Writing Workshop, November 2013 Liz Norman, Massey University, New Zealand
Citation preview
Grading criteria
and marking schemes
Liz Norman
Massey University
Australian and New Zealand College of Veterinary Scientists,
Exam Writing Workshop, November 2013
Validity again…
• The questions must elicit the behaviour we want
to evaluate
• Different markers need to award similar/scores
for the same candidate response
• Markers need to reward features we want to
evaluate and not reward features we do not
want to evaluate
Chain of inferences
1. Fellows have sufficient knowledge, skills, attitudes and judgement to be considered specialists
2. The things we can measure in examinations are things that are necessary to be a Fellow
3. A particular examination (all 4 components) is representative of all that we could measure in an examination
4. The examination score is a measure of achievement in the examination
5. The passing score of 70% correctly separates someone with sufficient knowledge, skills, attitude and judgement to be a Fellow from someone without
Purpose of marking schemes
• To help you during Q writing
– What content is important
– Whether the Q asks what you intended it to ask
– Whether it is do-able in the time available
• To help you during Q marking
– helps you decide whether an answer is good enough
to be awarded a mark
– facilitates reliable and fair marking
Types of marking scheme
1. Model answer – “ideal” answer
2. Point-based schemes
3. Generic criteria- & level-based schemes
4. Specific criteria- & level-based schemes
5. Specific criteria- & level-based schemes with
incorporated principle(s) for discriminating
levels
Model answers
• Not recommended
• Usually more than would be expected to be
given by any candidate
• No guidance on how to assess alternative
answers to the model provided
• No guidance on how to award marks
Point-based schemes
• Points for each objectively identifiable content
point
• Does not indicate the relative importance of the
points it awards
• Sum can be more or less than the whole
• Rewards quantity not quality
Criteria- & level-based schemes
• Criteria – different dimensions of performance
• Level – different quality/standards of
performance on a given criteria
Criteria- & level-based schemes
• Generic vs specific for the Q
• Explicit vs implicit weighting
Southern Cross University (2013) http://scu.edu.au/teachinglearning/download.php?doc_id=12921&site_id=301&file_ext=.pdf
Standards
Very poor Poor Fair Excellent C
rite
ria
Analysis and
interpretation
of results
Total 8 marks
0-2 marks
Interpretation not
provided or incorrect.
2-4 marks
Lacks one or more key
elements.
4-6 marks
Adequate interpretation
that addresses key
elements. Misses nuances
of interpretation or
uncommon differentials
6-8 marks
Thorough accurate
interpretation of results.
Well justified and
appropriately prioritised
list of differentials.
Quality of
planning
Total 8 marks
0-2 marks
No plans provided or
plans not appropriate
or dangerous
2-4 marks
Plans miss some key
aspects or overly
general
4-6 marks
Adequate plans that
address all key
differentials. Some
displaced in priority or not
pragmatic
6-8 marks
Thorough detailed and
well-prioritised and
pragmatic plan that
addresses all defined
differentials.
Knowledge of
current
literature
Total 5 marks
0-2 marks
Little or no literature referred to or incorrectly
referred to.
3-4 marks
Answer refers to some of
the key literature
4-5 marks
Answer refers to current
literature including
controversies and
comparative work from
other species.
Logical
presentation
Total 4 marks
1 mark
Answer is disorganised
and includes a large
amount of irrelevant
material
2 marks
Answer is somewhat
disorganized and
includes some
irrelevant material
3 marks
The answer is relatively
well organized and
contains little irrelevant
material.
4 marks
The answer shows a high
degree of logical thought
and well-constructed
argument.
Analytical vs holistic schemes
• Both are valid
• Analytical (criteria scored separately)– Better agreement between examiners
– Insufficient criteria
– Overlapping criteria
• Holistic (scored as a whole)– Challenging, especially for longer answers
– Less agreement between examiners
Whittem (2013) Guidelines for Oral Examiners, Faculty of Veterinary Science, University of Melbourne
Armstrong et al. (2008) University of Western Sydney http://www.uws.edu.au/__data/assets/pdf_file/0004/449860/Assessment_Guide.pdf
Armstrong et al. (2008) University of Western Sydney http://www.uws.edu.au/__data/assets/pdf_file/0004/449860/Assessment_Guide.pdf
Wood (2012). Macquarie University http://staff.mq.edu.au/public/download.jsp?id=56594
Incorporating principles/rules
• Ideal is a specific criteria- & level-based
schemes with incorporated principle(s) for
discriminating levels
Quality vs quantity
• More complex and unstructured the Q the more
assessing quality not quantity
• In very constrained tasks only judging how
correct the answer is
• In very open tasks, “correctness” is less
important and its quality that's judged instead
Prestructural Question may be rephrased as the answer; almost completely
misses the point of the question.
Unistructural Able to identify, list, name, enumerate but does not describe,
explain, relate or elaborate multiple aspects of a response
Multistructural Able to list as well as describe distinct aspects of a response (such
as being able to describe aetiology, clinical features, management
of thrombotic stroke) but unable to explicitly explain causes for
observations; unable to present cause-effect relationships.
Relational Able to describe multiple aspects of a process and additionally
explain or elaborate observations into cause-effect relationships;
able to compare similarities and differences between apparently
distinct phenomena. This level is taken as suggesting that the
learner has understood.
Extended
abstract
Highly developed; able to explain mechanisms of phenomena and
apply this information to a novel context — able to develop novel
hypotheses, theories, and deduce principles; creative thinking.
Prakash et al. (2010) Adv Physiol Educ, 34(3):145-149
Content vs connections
Lucander et al. (2010). European Journal of Dental Education, 14(3), 145-150.
Writing marking schemes
• Select and organise the criteria/dimensions
• Develop clear descriptions for each
level/standard of each criteria
• Need to think about poor answers as well as
good ones
Revising mark schemes in use
• Hopefully all types of answer are anticipated
• Sometimes though it is not – can indicate unanticipated problems with the Q
• Marking schemes might need revising after first few candidates marked
• HSE should moderate marking and should encourage team members to report marking issues early in the piece.
Recommended