Grading using a criterion-referenced · Web view’ by ensuring that students know in advance how you arrive at these grades (principle 3 of Assessment Policy, TLP2.1 2009). At

Grading using a criterion-referenced assessment approach

IntroductionJudgmentWhen you award a grade for a student’s response to an assessment task or for the whole unit, you use your professional judgment to make decisions. How you arrive at these final judgments must be as manageable for you as possible. These judgments also need to be valid and reliable. Your judgments are underpinned by the principle that ‘assessment practices and processes must be transparent and fair’ by ensuring that students know in advance how you arrive at these grades (principle 3 of Assessment Policy, TLP2.1 2009). At UTAS, criterion-referencing has been policy since 2007.

PrerequisitesThere are a number of prerequisites that must be in place and which underpin the advice and examples given here about grading using CRA. 1. Unit learning outcomes have been well written (refer to How to write learning outcomes –an

introduction)2. These outcomes have been mapped to the assessment tasks (see table 1 below) so that students

have at least two opportunities to demonstrate the outcomes. Otherwise if a student only has one opportunity to demonstrate an outcome in one task student and fails to do so, they have no other chance to redeem themselves. The Unit Outline proforma requires a table mapping the assessment tasks.

3. Not all learning outcomes have to be assessed in each task. This means that not all criteria (derived from these outcomes) need to be assessed in each task.

4. The tasks have been weighted in importance. See example 1 where task 3 is weighted the highest.

Example 1learning outcomes task 1 (20%) task 2 (30%) task 3 (50%)1. Demonstrate and apply knowledge of: basic pharmacological concepts the mechanism of action of drugs the therapeutic application of drugs and their

adverse affects the consequences of polypharmacy

2. Research published literature in pharmacological sciences for different topics

3. Communicate orally and in writing using professional/biomedical language and terminologies

5. Criteria for each task have been explicitly derived from the learning outcomes. Example 2 shows how the criteria derive from the learning outcomes for task 1. For advice and more examples on how to develop criteria, refer to Develop criteria (parts 1-3) and to examples of criteria sheets.

6. There is an odd number of criteria (e.g. 3 or 5) to avoid awkward and invalid trade offs (see ‘Grading a task or unit’ for an explanation with example). Refer to example 8 when using an even number of criteria.

1

http://www.teaching-learning.utas.edu.au/assessment/criteria-sheets

http://www.teaching-learning.utas.edu.au/assessment/how-do-i-write-criteria-sheets/step-3-develop-criteria-part-1

http://www.teaching-learning.utas.edu.au/planning/unit-outline-proforma

http://www.teaching-learning.utas.edu.au/planning/unit-outlines)

http://www.teaching-learning.utas.edu.au/planning/unit-outlines)

http://www.teaching-learning.utas.edu.au/assessment/moderating-teacher-judgments

Example 2learning outcomes task-specific criteria On completion of this unit, you should be able to:

To complete this task, you should:

1. Demonstrate and apply knowledge of: basic pharmacological concepts the mechanism of action of drugs the therapeutic application of drugs and their

adverse affects the consequences of polypharmacy

1. Demonstrate and apply knowledge of the following to the topic: mechanism of operation of agents aetiology therapeutic agents future means of treatment

2. Research published literature in pharmacological sciences for different topics

2. Research published literature in pharmacological sciences in relation to the topic

3. Communicate orally and in writing using professional/biomedical language and terminologies

3. Communicate in the form of an assignment about the topic academic writing (English conventions including

punctuation, sentence structure, grammar) Vancouver referencing conventions professional/biomedical language and terminologies

7. Standards have been described for the criteria for each level of achievement in the form of a criteria sheet or rubric. Refer to the Assessment website for an explanation of terminology, how to write criteria sheets and examples.

Grading a task or a unitThe key principle to grading using criterion referenced assessment (CRA) is that students must meet the standard for each criterion to be awarded a grade. This means that your judgment about the student’s response to assessment tasks is ‘referenced to standards for each criterion’. To decide on what grade the student achieves for a criterion, you match the quality of the student work to descriptors for different standards on the criteria sheet, and judge where the work MOSTLY fits. That is, you make a holistic judgment, e.g. Is it mostly a credit standard for this criterion? Is it a lower credit for this criterion? and so on. This matching exercise is not measuring students’ abilities.

You are making a professional judgment about what they have demonstrated (i.e. evidence of their achievement). This is the process of judging, regardless of whether you use marks or not. Once you have decided that a student has achieved a particular standard for each criterion, then you have to have a way of coming up with an overall grade for the task and later the unit. Grading a task or a unit therefore requires a way of combining those standards across the criteria. Students need to be informed in unit outlines of the method of grading you use.

No matter which approach you use, results must be moderated to ensure comparability of judgments.

1Approaches to combining standards to grade a task or unitThere are three common approaches which can be used individually or in combination. It is recommended that if you are having difficulty coming to an overall grade for a task or unit, confirm or change your judgments by re-examining the student’s actual responses.

1. Profiling results This involves making an on-balance judgment that requires looking at the general pattern of achievement in the criteria across the task and then the unit. This is easiest when there are

1 The structure of this section and some of the ideas presented were partially informed by the following. The examples and explanations, however, are the work of the author, Moira Cordiner, UTAS: Report and recommendations of the QUT Teaching and Learning Committee working party on CRA, chaired

by Wageeh Boles, November, 2004 A workshop on approaches to determining final grades by Jude Smith, QUT, March 2006

A workshop on approaches to determining final grades by Jude Smith, QUT, March 2006

2

http://www.teaching-learning.utas.edu.au/assessment/moderating-teacher-judgments

http://www.teaching-learning.utas.edu.au/assessment

an odd number of criteria and they are equally weighted (see examples 3, 4 and 5 which also use marks).

2. Predetermined rules This involves setting rules for how grades for each criterion are combined to reach an overall grade for a task (see example 6 and 7) or a unit (see examples 9 and 10). Rules can take account of differently weighted criteria (see example 7). These rules can be used in lookup tables in spreadsheets to make overall judgments faster especially for large cohorts.

3. Assigning marks This involves setting mark distributions either: (i) for each grade or each criterion (based on the university’s prescribed percentage distribution for awarding of HD to NN grades) as in example 3 and 4. Example 5 uses task marks allocated on the basis of these ranges to save time converting; or (ii) for each descriptor that reflects the weighting of the criteria and of the task – this is easy to do for simple criteria sheets (see example 6) but becomes tedious and time consuming for more complex sheets (and therefore not recommended).

Judging a grade for a task

1. Using profiling and marks

Examples 3, 4 and 5 use profiling and marks to come to a judgment about the grade for the task.

Example 3First year Nursing unit Task: essay weighting 30%criteria (equally weighted) HD DN CR PP NN

100-80% 79-70 69-60 59-50 49-01.Demonstrate knowledge of models of nursing care

descriptorsdescriptors descriptors



descriptorsdescriptors


2.Apply the principles of culturally safe nursing care

descriptorsdescriptors descriptorsdescriptors




descriptors

3.Communicate in academic writing descriptorsdescriptors descriptorsdescriptors





GradingOverall grade HD in lower third of the HD range (ticks indicate middle of each band) hence 84% Task weighted 30% Therefore 84% of this is 25.2/30 marks if the total of all tasks is 100.

3

Example 4Fourth year subject Task: report weighting 60%criteria (equally weighted) HD DN CR PP NN

100-80% 79-70 69-60 59-50 49-01. criterion…… …… descriptors



descriptors

descriptors descriptors

2. criterion………… descriptorsdescriptors




descriptors

3. criterion………… descriptorsdescriptorsdescriptors

descriptors

descriptorsdescriptorsdescriptors







descriptors

descriptors




descriptors

descriptors

GradingOverall grade CR in the upper third of the CR range (position of ticks indicate top of each band) hence 67%Task weighted 60% Therefore 67% of this is 40.2/60 marks if the total of all tasks is 100.

Example 5This example is another way of showing where in the range the judgment sits by dividing each range into thirds.

First year Nursing unit Task: essay weighting 30%criteria (equally weighted) HD DN CR PP NN

30-24 23-21 20-18 17-15 14-01. Demonstrate knowledge of models of

nursing caredescriptorsdescriptors





2. Apply the principles of culturally safe nursing care






3. Communicate in academic writing






GradingOverall grade HD in the lower third of the HD range (ticks indicate middle of each band) hence (27+27+22)/3 =25 marks

4

2. Using differently weighted criteria and marks

Example 6 is a simple criteria sheet that was developed for workplace supervisors to use. Marks have been distributed across the sheet based on weightings of the criteria. Note that to make it easy to allocate marks to the descriptors, the weightings for each criterion were converted to marks and doubled. Students doing this fourth year unit have five supervisors during their last placement. Each supervisor has to make judgments of students’ work readiness and knowledge based on the criteria and standards. The example shows how an overall mark can be quickly determined by the supervisor and this was the case in practice. The example is one of five assessments of the hypothetical student that resulted in the following 42/60, 39/60, 32/60, 57/60, 58/60. The total 228/300 was converted to 22.8% (the weighting of the work placement was 30%). The coordinator could easily record supervisor marks in a spreadsheet.

Example 6: A hypothetical supervisor’s ‘marked’ criteria sheet for laboratory work placement

3. Using predetermined rules, profiling and differently weighted criteriaExample 7 (for a Geography and Environmental Studies Honour thesis) shows profiling and the use of predetermined rules to assign the overall grade. Note that the rules require a set number of descriptors to be met before a particular standard such as First Class can be awarded, especially in criterion 2 (the most challenging one). In this example, no fail standard is described because this is an honours level task. Note that the rule for a fail grade in this case is: ‘failed to meet the standards for third class honours.’

5

Example 7: two page ‘marked’ Honours in Geography and Environmental Studies criteria sheetCriteria First class Second Uppers Second Lowers Third Class

In your thesis, you: In your thesis, you: In your thesis, you: In your thesis, you:

Design a research project links to theories

and literature; methods of

research and analysis

scope, limitations and challenges

drew on a judicious selection of a wide range of relevant theories and research literature to situate your project within the research literature, and inform your research focus and design

drew on a wide selection of relevant theories and research literature to inform your research focus and design

paraphrased a number of relevant theories and research literature and related these to your research focus and design

mentioned some relevant theories and research literature and how these related to your research topic

explicitly justified in detail your choice and use of research methods, methodology and analysis techniques (in terms of, e.g. relevance, rigour, replicability)

justified in some detail your choice and use of your research methods, methodology and analysis techniques, (in terms of, e.g. relevance, rigour, replicability)

described your choice and use of research methods, methodology and analysis techniques (in terms of, e.g. relevance, rigour, replicability)

listed and partially described your research methods, methodology and analysis techniques

explicitly identified and comprehensively considered the scope, limitations and challenges in conducting your research

identified and described most of the scope, limitations and challenges in conducting your research

identified and described some of the scope, limitations and challenges in conducting your research

listed a few of the scope, limitations and challenges in conducting your research

Generate data and discuss results with reference to theories and the literature analyse results draw conclusions synthesise

perspectives identify

contribution of your work

correctly and meticulously applied data analysis technique(s) to analyse results

correctly applied data analysis technique(s) to analyse your results with only minor mistakes

applied data analysis technique(s) to your results with some mistakes

partially applied basic data analysis to

your results

drew relevant and valid conclusions that contribute to the field(s) and were thoroughly substantiated with reference to the:evidence revealed in your research relationships between this evidence

and the literature

drew relevant and valid conclusions that were substantiated with reference to the:evidence revealed in your research evidence in the literature

drew some relevant conclusions that were substantiated based on:aspects of your research and aspects of the literature

drew partially substantiated conclusions loosely based on:aspects of your research and tenuous links to the literature

insightfully synthesised a new perspective by drawing out different relationships between features of your research and/or theory and the literature

synthesised a new perspective by comparing features of your research and/or theory and the literature

outlining a perspective by relating features of your research to theory and/or the literature

paraphrased the literature to describe your perspective

evaluated the contribution of your results to the research literature to pose valid questions, recommendations or directions for further research

identified and described the contribution of your results to the research literature to pose questions, recommendations or directions for further research

broadly stated the contribution of your results to the field(s) of study, posing some questions or recommendations for further research

listed some questions or recommendations for further research unrelated to the results of your research

6

Criteria First class Second Uppers Second Lowers Third Class In your thesis, you: In your thesis, you: In your thesis, you: In your thesis, you:

Communicate in the form of a thesis, a scholarly work that adheres to: structure and

format English

conventions (grammar, syntax, spelling)

referencing conventions

standards of academic integrity*

adhered to English conventions to: logically structure ideas throughout;

andclearly and concisely express ideas in

flowing and eloquent prose

adhered to English conventions to: logically structure ideas through most

of thesisclearly and concisely express ideas

throughout most of the work

mostly adhered to English conventions to, express ideas that were: logically structured in parts of the workclearly expressed in parts of the work

adhered to some English conventions to express ideas that partially conveyed meaning to the reader

employed an extensive, correct and relevant discipline specific vocabulary

employed an extensive and relevant discipline specific vocabulary with only minor mistakes and/or inconsistencies

employed discipline specific vocabulary with occasional misinterpretation of terms and/or inconsistent use of these

used some discipline-specific vocabulary and/or used this incorrectly, affecting the reader’s interpretation of the document

used referencing to explore and provide additional or analogous ideas for the reader, while strictly adhering to a referencing convention

strictly adhered to a referencing convention

followed a referencing system with some

minor errors cited some sources using an

inconsistent system of referencing

presented a scholarly work that:was free from typographical errors incorporated correctly formatted,

labelled and easily interpreted visual representations (eg. tables, graphs, plates, figures) that best suited, and complimented the data and results

presented a scholarly work that:was mostly free from typographical

errors incorporated correctly formatted,

labelled and easily interpreted visual representations (eg. tables, graphs, plates, figures) with only minor mistakes

presented a mostly scholarly work that:contained typographical errors incorporated visual representations

(eg. tables, graphs, plates, figures) that contained some formatting errors or oversights that impacted on the presentation of the thesis

presented a document that:contained a large number of

typographical errors incorporated visual representations

(eg. tables, graphs, plates, figures) that had significant flaws or oversights.

Overall grade is Second class lowers according to the rule below.

Grading rules:To achieve First Class Honours you must obtain at least 8/11 first class standard marks with at least 3/4 first class standard marks in Criteria 2.To achieve Second Class Uppers you must obtain at least 8/11 marks in the Second Class Uppers standard or above, with at least 3/4 Second class uppers standard or above marks in Criteria 2.To achieve Second Class Lowers you must obtain at least 8/11 marks in the Second Class Lowers standard or above, with at least 3/4 Second class lowers standard or above marks in Criteria 2.To achieve Third Class Honours you must obtain at least 8/11 marks in the Third Class standard or above, with at least 3/4 Third Class standard or above marks in Criteria 2.

FAIL: failed to meet the standards for a third class honours * evidence of plagiarism will override ALL other assessment criteria

7

Making judgments about a task that has an even number of criteriaExample 8 illustrates one problem you may have if you use even numbers of equally weighted criteria. Is the grade for the task below HD or DN? If you want to use an even number of criteria then you need a rule that helps you make judgments that are valid for the task. For example, a rule could be, Rule 1: ‘to achieve an HD for this task, you must have 2 HDs’. A different rule might be, Rule 2: ‘to achieve an HD for this task, you must have no less than a midrange HD and an upper range DN’. Application of these rules leads to different grades. When there are only three or five criteria, the making of an overall judgment is reasonably straightforward. When there are more criteria, judgment rules become more complicated and unwieldy. Note example 7 for honours has three criteria (and these have subcriteria), yet this is a very manageable sheet to use (it has been successfully trialled by UTAS staff).

Example 8 First year Nursing unit Task: essay weighting 30%criteria (equally weighted) HD DN CR PP NN

100-80% 79-70 69-60 59-50 49-01.Demonstrate knowledge of models of nursing care






2.Apply the principles of culturally safe nursing care





descriptors

GradingOverall grade is DN if using rule 1 OR HD if using rule 2.

Note that allocating the marks for the task would differ a little depending on whether you used rule 1 or 2.

Judging a grade for a unit

1. Using profiling In example 9, criteria A, B and C are equally weighted and explicitly derived from learning outcomes 1, 2 and 3 (for a hypothetical unit). You can use profiling by examining the pattern of achievement across the criteria, taking account of task weightings and decide that the overall grade for the unit is distinction (DN). Note that you will come to the same grade for the unit regardless of whether you judge the overall grade for each criterion first, or judge the overall grade for tasks first. Both do not involve calculations.

In this example, you could also have had a predetermined rule that said ‘to be awarded a distinction for the unit, you must achieve at least two distinctions and one credit and one of the distinctions must be for criterion A’; OR ‘to be awarded a distinction for the unit, you must achieve at least two distinctions and one credit and one of the distinctions must be for task 3’

You can of course add marks beside the grade categories. Spreadsheets can be used to come up with overall grades whether using marks or not.

Example 9Individual student profile grades for tasks

assessment task & weighting criterion A criterion B criterion C

task 1 (20%) HD DN DN DNtask 2 (30%) CR DN CR CRtask 3 (50%) DN DN CR DNoverall grade for each criterion DN DN CR

grade for unit DN

8

2. Using predetermined rules and/or marks

Example 10 uses the same tasks as example 9. It shows how marks were determined for each of three tasks based on profiling and weighting, and then how the grade for the unit was worked out. This example illustrates that you can validly arrive at the same final grade by adding marks if you have initially made valid judgments about the grade for each task, based on the criteria and standards.

The last table in example 10 is representing the last column and last row in example 9 (shaded above) but with the marks shown.

Example 10Task 1 weighting 20% (20 marks)criteria (equally weighted) HD DN CR PP NN

20-16 15-14 13-12 11-10 9-01. criterion A descriptors





2. criterion B descriptorsdescriptors





3. criterion C descriptorsdescriptors





GradingOverall grade DN in the lower third of the DN range hence 14 marks

Task 2 weighting 30% (30 marks)criteria (equally weighted) HD DN CR PP NN






descriptors

2. criterion B descriptorsdescriptors





3. criterion C descriptorsdescriptorsdescriptors




descriptors

GradingOverall grade CR in the upper third of the CR range and hence 20 marks

9

Task 3 weighting 50% (50 marks)criteria (equally weighted) HD DN CR PP NN






descriptors

2. criterion B descriptorsdescriptorsdescriptors




descriptors

3. criterion C descriptorsdescriptorsdescriptors



descriptors descriptors

GradingOverall grade DN and in the upper third of the DN range hence 38 marks

This table is a companion to the table in example 9

Unit tasks & weightingoverall grades and marks awarded for each task

HD DN CR PP NN100-80% 79-70% 69-60% 59-50% 49-0%

task 1 (20%) 14 markstask 2 (30%) 20 markstask 3 (50%) 38 marks

GradingTotal marks for the unit 72/100Overall grade for the unit DN

A variation on the previous exampleExample 11 shows how to grade a unit when each task does not assess all the criteria (as in example 1, where task 2 assesses the three criteria but tasks 1 and 2 assess two criteria) and one criterion is more heavily weighted than the others (in this case criterion A*). Thus example 11 is a variation of example 10 in which each task assesses all the criteria and the criteria were equally weighted. To grade the tasks in example 11, you need a rule because criterion A is weighted twice as heavily as each of the other criteria. For example, the rules might be:

Task 1: to achieve an HD, you must have an HD in criterion A, and no less than a DN in criterion B (and so on for the other grades).Task 2: to achieve an HD, you must have 2HDs (one must be in criterion A), plus no less than a DN in the third criterion (and so on for the other grades).Task 3: to achieve an HD, you must have an HD in criterion A, and no less than a DN in criterion C (and so on for the other grades).

To grade the unit, you need a rule such as: to be awarded an overall HD grade, you must have 2HDs for two tasks (and one must be for task 3) and no less than a DN for the other task (and so on). Marks can be used as explained in previous examples.

Example 11Unit tasks & weighting criterion A* criterion B criterion C task gradetask 1 (20%) CR PP CR

task 2 (30%) PP CR PP PP

task 3 (50%) DN CR DN

Using the above suggested rule for a unit grade, the overall unit grade is CR (student has at least 2CRs for tasks and one is for task 3, (even though it is DN) and no less than PP for the third task

10

Exams and CRATo grade an exam using a CRA approach is simple and straightforward if the exam has been designed well. This means an exam map has been used to map the questions to the criteria.

An exam map allows you to: organise the questions for each content area of the unit so that you don’t over-assess, for example,

the same content, concept, principle or skill balance the types of questions so that you assess all the criteria that are relevant to the exam; these

criteria are developed from the relevant learning outcomes. Otherwise you may, for example, inadvertently use too many recall or explain questions and not enough interpret and analyse questions (this depends on the year level of the exam of course)

check that the weightings of the criteria are reflected by the marks—this is important so that you can discriminate between students because they are given the opportunity to demonstrate their ability in the more challenging criteria.

You can use one or more of the previously explained approaches to determining the grade for an exam just as you do with any task. For example, you can profile achievement for each criterion (using marks) in a spreadsheet and if necessary, have a rule that requires a stipulated level of achievement in a particular criterion to be awarded an HD (for example). Lookup tables about these rules are easy to create so the process of grading can be automated.

Final advice re CRA and grading: For grading tasks or units, you can validly use any of the approaches explained in this document (or a combination), as long as you are consistent within a unit and inform students explicitly in unit outlines.

11

http://www.teaching-learning.utas.edu.au/assessment/authentic-assessment/designing-exams

Documents

Grading using a criterion-referenced · Web view’ by ensuring that students know in advance how you arrive at these grades (principle 3 of Assessment Policy, TLP2.1 2009). At