If you can't read please download the document
Upload
sybil-golden
View
218
Download
2
Tags:
Embed Size (px)
Citation preview
What kinds of assessments improve learning?
Dylan Wiliam, ETS
Raising achievement matters
For individualsIncreased lifetime salaryImproved healthFor societyLower criminal justice costsLower health-care costsIncreased economic growthWheres the solution?
StructureSmall high schoolsK-8 schoolsAlignmentCurriculum reformTextbook replacementGovernanceCharter schoolsVouchersTechnologyIts the classroom
Variability at the classroom level is up to 4 times greater than at school levelIts not class sizeIts not the between-class grouping strategyIts not the within-class grouping strategyIts the teacherTeacher quality:
A labor force issue with 2 solutionsReplace existing teachers with better ones?No evidence that more pay brings in better teachersNo evidence that there are better teachers out there deterred by certification requirementsImprove the effectiveness of existing teachersThe love the one youre with strategyIt can be doneWe know how to do it, but at scale? Quickly? Sustainably?The design challenge
Key metric:Cost of buying one standard deviation of increased student achievementConstraintsSolution must be in principle scalableCost/effect comparisons
InterventionEffect (sd)Cost/yrClass-size reduction(by 30%)0.1$24kIncrease teacher content knowledge by 1 sd0.1?Formative assessment/Assessment for learning0.2Effects of formative assessment
Several major reviews of the researchNatriello (1987)Crooks (1988)Black & Wiliam (1998)Nyquist (2003)All find consistent, substantial effectsEffects of feedback
Kluger & DeNisi (1996)Review of 3000 research reportsExcluding those:without adequate controlswith poor designwith fewer than 10 participantswhere performance was not measuredwithout details of effect sizesleft 131 reports, 607 effect sizes, involving 12652 individualsAverage effect size 0.4, butEffect sizes very variable40% of effect sizes were negativeKinds of feedback (Nyquist, 2003)
Weaker feedback onlyKnowledge of results (KoR)Feedback onlyKoR + clear goals/knowledge of correct results (KCR)Weak formative assessmentKCR+ explanation (KCR+e)Moderate formative assessment(KCR+e) + specific actions for gap reductionStrong formative assessment(KCR+e) + activityEffect of formative assessment (HE)
NEffectWeaker feedback only310.16Feedback only480.23Weaker formative assessment490.30Moderate formative assessment410.33Strong formative assessment160.51Feedback and formative assessment
Feedback is information about the gap between the actual level and the reference level of a system parameter which is used to alter the gap in some way (Ramaprasad, 1983 p. 4)Formative assessment requiresdata on the actual level of some measurable attribute;data on the reference level of that attribute;a mechanism for comparing the two levels and generating information about the gap between the two levels;a mechanism by which the information can be used to alter the gap.Formative assessment
Frequent feedback is not necessarily formativeFeedback that causes improvement is not necessarily formative Assessment is formative only if the information fed back to the learner is used by the learner in making improvementsTo be formative, assessment must include a recipe for future actionAssessment for learning &
formative assessment (Black et al., 2002)
Assessment for learning is any assessment for which the first priority in its design and practice is to serve the purpose of promoting pupils learning. It thus differs from assessment designed primarily to serve the purposes of accountability, or of ranking, or of certifying competence. An assessment activity can help learning if it provides information to be used as feedback, by teachers, and by their pupils, in assessing themselves and each other, to modify the teaching and learning activities in which they are engaged.
Such assessment becomes formative assessment when the evidence is actually used to adapt the teaching work to meet learning needs.
Feedback
Feedback is therefore formative only if the information fed back is actually used in closing the gap.Three key instructional processesEstablishing where learners are in their learningEstablishing where they are goingEstablishing how to get thereAspects of formative assessment
Where the learner is goingWhere the learner isHow to get thereTeacherClarify learning intentionsEngineering effective discussionsProviding feedback that moves learners onPeerUnderstand/clarify criteria for successActivating students as instructional resources for one anotherLearnerUnderstand criteria for successActivating students as owners of their own learningFive key strategies
Clarifying and understanding learning intentions and criteria for successEngineering effective classroom discussions that elicit evidence of learningProviding feedback that moves learners forwardActivating students as instructional resources for each otherActivating students as the owners of their own learningand one big idea
Use evidence about learning to adapt instruction to meet student needsKeeping Learning on Track (KLT)
A pilot guides a plane or boat toward its destination by taking constant readings and making careful adjustments in response to wind, currents, weather, etc.Educational systems must do the same:Plan a carefully chosen route ahead of time (in essence building the track)Take readings along the way Change course as conditions dictateRegulation of learning
Teaching as engineering learning environmentsKey features:Create student engagement (pedagogies of engagement)Well-regulated (pedagogies of contingency)Long feedback cycles vs. variable feedback cyclesQuality control vs. quality assurance in learningTeaching vs. learningRegulation of activity vs. regulation of learningRegulation of learning
Proactive (upstream) regulationPlanning regulation into the learning environmentPlanning for evoking informationInteractive (downstream) regulationNegotiating the swiftly-flowing riverMoments of contingencyTightness of regulation (goals vs. horizons)Retrospective regulationStructured reflection (e.g., lesson study)Types of formative assessment
Long-cycleFocus: between unitsLength: four weeks to one yearMedium-cycleFocus: within units, between lessonsLength: one day to two weeksShort-cycleFocus: within lessonsLength: five seconds to one hourQuestioning in math: discussion
Look at the following sequence:
3, 7, 11, 15, 19, .
Which is the best rule to describe the sequence?
n + 4
3 + n
4n - 1
4n + 3
Questioning in math: diagnosis
In which of these triangles is a2 + b2 = c2 ?
A
a
c
b
C
b
c
a
E
c
b
a
B
a
b
c
D
b
a
c
F
c
a
b
Questioning in science: discussion
Ice-cubes are added to a glass of water. What happens to the level of the water as the ice-cubes melt?
The level of the water drops
The level of the water stays the same
The level of the water increases
You need more information to be sure
Questioning in science: diagnosis
The ball sitting on the table is not moving. It is not moving because:
no forces are pushing or pulling on the ball.
gravity is pulling down, but the table is in the way.
the table pushes up with the same force that gravity pulls down
gravity is holding it onto the table.
there is a force inside the ball keeping it from rolling off the table
Wilson & Draney, 2004
Questioning in English: discussion
Macbeth: mad or bad?Questioning in English: diagnosis
Where is the verb in this sentence?
The dog ran across the road
A
B
C
D
Questioning in English: diagnosis
Where does the subject end and the predicate begin in this sentence?
The dog ran across the road.
A
B
C
D
Questioning in English: diagnosis
Which of these is a good thesis statement?
The typical TV show has 9 violent incidents
There is a lot of violence on TV
The amount of violence on TV should be reduced
Some programs are more violent than others
Violence is included in programs to boost ratings
Violence on TV is interesting
I dont like the violence on TV
The essay I am going to write is about violence on TV
Questioning in history: discussion
In which year did World War II begin?
1919
1937
1938
1939
1941
Questioning in History
Why are historians concerned with bias when analyzing sources?
People can never be trusted to tell the truth
People deliberately leave out important details
People are only able to provide meaningful information if they experienced an event firsthand
People interpret the same event in different ways, according to their experience
People are unaware of the motivations for their actions
People get confused about sequences of events
Why research hasnt changed teaching
Misunderstanding nature of teacher expertiseLeaving teachers to translate into practiceFailure to attend to both content and processKlein & Klein (1981)
Six video extracts of someone delivering cardio-pulmonary resuscitation (CPR)5 of the videos are of CPR students1 of the videos is of an expert paramedicVideos shown to three groups:students, instructors and expertsSuccess rate in identifying expert:Experts: 90%Students: 50%Instructors: 30%Teacher expertise (Berliner, 1994)
Experts excel mainly in their own domainExperts often develop automaticity for the repetitive operations that are needed to accomplish their goalsExperts are more sensitive to the task demands and social situation when solving problems.Experts are more opportunistic and flexible in their teaching than novicesExperts represent problems in qualitatively different ways than novices.Experts have fast and accurate pattern recognition capabilities. Novices cannot always make sense of what they experience.Experts perceive meaningful patterns in the domain in which they are experienced.Experts begin to solve problems slower, but bring richer and more personal sources of information to bear on the problem that they are trying to solve.Knowledge creation and transmission
After Nonaka & Tageuchi, 1995
A model for teacher learning
Content (what we want teachers to change)EvidenceIdeas (strategies and techniques)Process (how to go about change)Small stepsFlexibilityChoiceAccountabilitySupportDiscussion
What are the implications of this research for publishers of formative assessment systems?Questions?
Dont forget to fill out your evaluations