Upload
james-atherton
View
228
Download
1
Embed Size (px)
DESCRIPTION
Hunting assumptions about assessment; an up-dated replacement for a 2009 version.
Citation preview
Assessment Recap…
… as ever with added complications!
James AthertonUnit 3: 8 April 2014
The blog page to which some
notes refer is at:
http://pce2013.blogspot.co.uk/2014/04/8-april-back-to-assessment.html
Why?
Requirements
Approaches
Assessment
08/04/2014 - v2
Feedback(Formative)
Accreditation(Summative)
Validity
Reliability
Fairness
(Security)
Norm-referenced
Criterion-referenced
Ipsative
•“Assessment is central because it frames student learning”
(Oxford Brookes U. Position Paper on Assessment, n.d.)
•“From our students’ point of view, assessment always defines the actual curriculum”
(Ramsden P,1992)
•“Assessment is central because it frames student learning”
(Oxford Brookes U. Position Paper on Assessment, n.d.)
•“From our students’ point of view, assessment always defines the actual curriculum”
(Ramsden P,1992)
Not for nothing do students
invariably cry, “Are we being assessed on
this?”
•“Assessment is central because it frames student learning”
(Oxford Brookes U. Position Paper on Assessment, n.d.)
•“From our students’ point of view, assessment always defines the actual curriculum”
(Ramsden P,1992)
Not for nothing do students
invariably cry, “Are we being assessed on
this?”
And if the answer is “No”, then the
learning is automatically
worthless
It is assessment which turns;
• presenting
• story-telling
• demonstrating
• …
into teaching
• Assessment of learning
• (summative)
• Assessment for learning
• (formative)
Which is privileged?
• Assessment of learning
• (summative)
• Assessment for learning
• (formative)
Which is privileged?
Accreditation and
qualifications—important but not the whole
story
• Assessment of learning
• (summative)
• Assessment for learning
• (formative)
Which is privileged?
Accreditation and
qualifications—important but not the whole
storyImportance
increasingly recognised in the
rhetoric of education, if not the practice…
• Assessment of learning
• (summative)
• Assessment for learning
• (formative)
Which is privileged?
Accreditation and
qualifications—important but not the whole
storyImportance
increasingly recognised in the
rhetoric of education, if not the practice…
See link in notes
Formative
Summative
but…
Formative
Summative
but…All assessment,
including most of what is designed for summative
purposes, can be used formatively
Formative
Summative
but…All assessment,
including most of what is designed for summative
purposes, can be used
formatively…
…but only if you can get the students to
take their work back and read the comments!
PresenterPresenter AudienceLearner
Information
Feedback
TeacherTeacher/
learner
Learner/
teacher
Experiential environment
Theoretical representation
Goal-oriented behaviour
Conceptual representation
Teacher Student
Articulation/Re-articulation
Reflection/Adaptation
Reflection/Adaptation
Action/Feedback
Based on Laurillard, 2001
This is Diana Laurillard’s elaboration of the
previous graphic—link in the notes
Effect-size 2
• An effect-size of 1.0 indicates an increase of one standard deviation, typically associated with advancing [students’] achievement by one year, improving the rate of learning by 50%, or a correlation between some variable (e.g., amount of homework) and achievement of approximately .50. When
implementing a new program, an effect-size of 1.0 would mean that approximately 95% of outcomes positively enhance achievement, or average students receiving that treatment would exceed 84% of students not receiving that treatment.
Hattie, 1999:4 (my emphasis)
Just a reminder of Hattie—now familiar territory, with a link
in the notes.
Interventions and Effect Size
0
50
100
150
200
250
300
350
400
Effect Size
No. of
stu
die
s
Above average
effectsize
Influence Effect Size
•Feedback 1.13• Students' prior cognitive ability
1.04• Instructional quality 1.00• Direct instruction .82• Remediation/feedback .65• Students' disposition to learn .61• Class environment .56• Challenge of Goals .52• Peer tutoring .50• Mastery learning .50• Homework .43• Teacher Style .42• Questioning .41
•Feed-back, or
•Feed-forward ?Phil Race is the
principal advocate for feed-forward. Link in notes
Why?
Requirements
Approaches
Assessment
08/04/2014 - v2
Feedback(Formative)
Accreditation(Summative)
Validity
Reliability
Fairness
(Security)
Norm-referenced
Criterion-referenced
Ipsative
Requirements…
•Validity: the assessment measures what it is supposed to.
•What mismatches have you come across?
• (i.e. between desired outcomes and methods of assessment)
The boundaryis constantly
moving
Common observation suggests the belief is unfounded; we seldom find hard evidence of such correlations. Instead, we rely on test results for want of anything better. In any event, the skills required to perform well on school examinations may not be the same skills required to perform adequately in the situations the school trains people for.
(Becker, 1972)
We seldom argue that conventional tests measure the actual skills students are supposed to have learned. Rather, we believe that the test, … is … highly correlated with the ability to exercise those skills, though the mechanism … is seldom investigated or demonstrated.
Link to full paper in the notes
Reference points for Assessment
•Competitive (“Norm-referenced”)
See link in notes
Norm-Referenced Assessment
0
5
10
15
20
25
0 35 45 55 65 75 85 95 105
115
125
135
145
155
165
% o
f c
oh
ort
Mean
This curve is based on a real reference curve used for the standardisation of IQ tests, which is why it is not quite as graceful as a classic “bell-curve”. See
link in notes.
•When might it be appropriate to use norm-referenced assessment in a PCE setting?
Reference points for Assessment (2)
•Competitive (“Norm-referenced”)
•Competence (Criterion-referenced)[and
•Personal best (Ipsative)]
Criterion-Referencing
Threshold of competence
Object is to ensure minimum level of competence attained
One problem is whether that threshold effectively sets a
cap on the level of achievement: we briefly
discussed the question of inability to deal with
exceptional genius in the arts.
A complication of “competence”
Threshold
of Competence
Competence is as much a matter ofconsistency as of skill — what level of consistency is “competent”?
Answer
Facts Stimulus
IdeaConvergent thinking
- science and technology(?)
Divergent thinking- arts and humanities(?)
See link in notes
•How might you define achievement of intended outcomes in a divergent subject area?
Requirements (2)…
•Validity: the assessment measures what it is supposed to.
•Reliability: it produces consistent and replicable results Between judges Between occasions Between cohorts
Why?
Requirements
Approaches
Assessment
08/04/2014 - v2
Feedback(Formative)
Accreditation(Summative)
Validity
Reliability
Fairness
(Security)
Norm-referenced
Criterion-referenced
Ipsative
• "Is an indistinct photograph a picture of a person at all? Is it even always an advantage to replace a blurred picture with a sharp one? Isn't the blurred one often exactly what we need?" Wittgenstein (1953) para. 71
• “OK, you’ve got your PhD; now don’t touch anything!” (source unknown)
• “I claimed at the start that I would use the term 'ambiguity' to mean anything I liked,” (Empson, 1947; viii)16
Importance of clear criteria?
Criteriaa. Understanding and familiarity with the subjectb. Breadth of readingc. Relevance of content to essay titled. Structure of the argumente. Independence of thought and critical evaluationf. Presentation, including fluency of writing, grammar
and spellingg. Citation and referencing
How would you weight these?
(in relation to an HE1/NQF 4 task in a “soft” discipline)
20.0%20.0%13.3%13.3%20.0%
So, proportion of marks for each criterion
33223
Weighting of marks allocated to each criterion
Without necessarily being highly original, the work shows consistent critical evaluation of sources and ideas, and the author's own considered judgments
The argument is pursued clearly and in a balanced manner, supported by evidence and informed judgement
Essay addresses title and its underlying themes and assumptions, clearly explicated.
Evidence of extensive reading beyond the recommended texts, including critical evaluation of sources
Ability to contextualise subject within wider frameworks and critically explore connections
Critical evaluation and independent thought are the norm, but are either not consistent or coherent throughout.
A clear and generally structured argument let down by some unsound inferences or assertions without evidence
Essay addresses title in a focused and organised way with some awareness of any more general issues raised by it.
Recommended reading used, together with a range of other sources
Familiarity with subject demonstrated, including complex connections between elements
Evidence of occasional independent and critical thought, limited by either inconsistent or incoherent reasoning
Continuity of argument evident throughout the essay supported by evidence
Content clearly addresses set title as given.
Effective use made of reading, largely confined to reading list
Subject matter adequately understood, with some connections made
Critical evaluation limited to expressions of opinion, using mainly second-hand ideas
Argument patchy, inconsistent, or proceeding by assertion rather than evidence.
Essay generally addresses title, but with lapses, digressions or padding.
Limited use made of texts on set reading list
Evidence of knowledge, but only basic connections made between items
Uncritical and unreflective work, relying on undigested sources.
Little clear structure or argument evident.
Little or tenuous connection between title and essay content
Little or no reading evident, even within the set texts
Subject matter treated as set of discrete bits of knowledge
Independence of thought and critical evaluation
Structure of the argument
Relevance of content to essay title
Breadth of readingUnderstanding and familiarity with the subject
Criterion ECriterion DCriterion CCriterion BCriterion A
1Academic levelEducation, Individual and SocietyEduc 1002
5
4
3
2
1
More (including download) via link in
notes.
Backgroundof topic
Backgroundof topic
Knowledgeof topic
Backgroundof topic
Knowledgeof topic
Researchskills
Backgroundof topic
Knowledgeof topic
Ability toconduct argument
Researchskills
Backgroundof topic
Knowledgeof topic
Ability toconduct argument
Researchskills
“Hard” and “soft” objectives are
discussed in the notes.
Too clear and specific?Task-2
An annotated diagram which illustrates how you contribute to the Quality Assurance and Quality Improvement Processes, procedures and initiatives in the area of work you are engaged in. A short report evaluating and reflecting upon this contribution. Extended diagram(s) showing how quality framework systems work within
own teaching area making links to validity and reliability of the data collected.
Comprehensive annotations including links to areas for improvements and research.
Excellent critical analysis of the current issues relating to quality assurance and quality improvement processes in own area of teaching.
Sound application of theory to practice in order to analyse and evaluate the success of the quality procedures (both positive and negative) Applied theory to practice to identify own CPD.
Critical evaluation of the long term impact of QA and QI systems and procedures.
Detailed bibliography with substantial use of journal research sources. Quotes are presented and fully engaged.
Sound structure. Few errors of presentation.
Too clear and specific?Task-2
An annotated diagram which illustrates how you contribute to the Quality Assurance and Quality Improvement Processes, procedures and initiatives in the area of work you are engaged in. A short report evaluating and reflecting upon this contribution. Extended diagram(s) showing how quality framework systems work within
own teaching area making links to validity and reliability of the data collected.
Comprehensive annotations including links to areas for improvements and research.
Excellent critical analysis of the current issues relating to quality assurance and quality improvement processes in own area of teaching.
Sound application of theory to practice in order to analyse and evaluate the success of the quality procedures (both positive and negative) Applied theory to practice to identify own CPD.
Critical evaluation of the long term impact of QA and QI systems and procedures.
Detailed bibliography with substantial use of journal research sources. Quotes are presented and fully engaged.
Sound structure. Few errors of presentation.
This is an actual rubric from a supposedly
Master’s level course.
Too clear and specific?Task-2
An annotated diagram which illustrates how you contribute to the Quality Assurance and Quality Improvement Processes, procedures and initiatives in the area of work you are engaged in. A short report evaluating and reflecting upon this contribution. Extended diagram(s) showing how quality framework systems work within
own teaching area making links to validity and reliability of the data collected.
Comprehensive annotations including links to areas for improvements and research.
Excellent critical analysis of the current issues relating to quality assurance and quality improvement processes in own area of teaching.
Sound application of theory to practice in order to analyse and evaluate the success of the quality procedures (both positive and negative) Applied theory to practice to identify own CPD.
Critical evaluation of the long term impact of QA and QI systems and procedures.
Detailed bibliography with substantial use of journal research sources. Quotes are presented and fully engaged.
Sound structure. Few errors of presentation.
If you get this detailed and
prescriptive, can you possibly hit the level
of critical scholarship
expected of a graduate level
course?
Too clear and specific?Task-2
An annotated diagram which illustrates how you contribute to the Quality Assurance and Quality Improvement Processes, procedures and initiatives in the area of work you are engaged in. A short report evaluating and reflecting upon this contribution. Extended diagram(s) showing how quality framework systems work within
own teaching area making links to validity and reliability of the data collected.
Comprehensive annotations including links to areas for improvements and research.
Excellent critical analysis of the current issues relating to quality assurance and quality improvement processes in own area of teaching.
Sound application of theory to practice in order to analyse and evaluate the success of the quality procedures (both positive and negative) Applied theory to practice to identify own CPD.
Critical evaluation of the long term impact of QA and QI systems and procedures.
Detailed bibliography with substantial use of journal research sources. Quotes are presented and fully engaged.
Sound structure. Few errors of presentation.
If you get this detailed and
prescriptive, can you possibly hit the level
of critical scholarship
expected of a graduate level
course?
Despite fatuous usage of the
currently fashionable phrases.
Requirements (4)…
•Validity: the assessment measures what it is supposed to.
•Reliability: it produces consistent and replicable results
•Fairness
•Security
Requirements (4)…
•Validity: the assessment measures what it is supposed to.
•Reliability: it produces consistent and replicable results
•Fairness
•Security
What does “inclusivity” mean in
the context of assessment?
Why?
Requirements
Approaches
Assessment
08/04/2014 - v2
Feedback(Formative)
Accreditation(Summative)
Validity
Reliability
Fairness
(Security)
Norm-referenced
Criterion-referenced
Ipsative