ETS Confidential & Proprietary What kinds of assessment promote learning? Dylan Wiliam Director, Learning and Teaching Research Center ETS NARST Annual

ETS Confidential & Proprietary

What kinds of assessment promote learning?

Dylan Wiliam

Director, Learning and Teaching Research Center ETS

NARST Annual International Conference:

San Francisco, CA; April 2006

2 ETS Confidential & Proprietary

Overview of presentation

• Why raising achievement is important

• Why investing in teachers is the answer

• Why assessment for learning should be the focus

• How we can put this into practice


...the model that says ‘learn while you are at school the skills that you will apply during your lifetime’ is no longer tenable. These skills will be obsolete by the time you get into the workplace and need them, except for one skill – the skill of being able to learn. It is the skill of being able, not to give the right answer to questions about what you were taught in school, but to make the right response to situations that are outside the scope of what you were taught in school. We need to produce people who know how to act when they are faced with situations for which they were not specifically prepared. (Papert, 1998)

What do we need students to learn?


The four Rs of “learning power”• Resilience

– Absorption, managing distractions, noticing, perseverance

• Resourcefulness– Questioning, making links, imagining,

reasoning, capitalizing

• Reflectiveness– Planning, revising, distilling, meta-learning

• Reciprocity– Interdependence, collaboration, empathy

and listening, imitation

(Claxton 2002)


Preparation for future learning (PFL)

• Cannot be taught in isolation from other learning

• Students still need the basic skills of literacy, numeracy, concepts and facts

• Learning power is developed primarily through pedagogy, not curriculum

• We have to change the way teachers teach, not what they teach


Successful education

The test of successful education is not the amount of knowledge that a pupil takes away from school, but his [sic] appetite to know and his capacity to learn. If the school sends out children with the desire for knowledge and some idea how to acquire it, it will have done its work. Too many leave school with the appetite killed and the mind loaded with undigested lumps of information. The good schoolmaster [sic] is known by the number of valuable subjects which he declines to teach.

(Sir Richard Livingstone, President of Corpus Christi College, Oxford, 1941)


Different approaches to upper secondary schooling

• 20th century education– US: mass systems

– Europe: elite systems

• 21st century education challenge– US: increase quality

– Europe: scale up quality


Raising achievement matters

• For individuals– Increased lifetime salary

– Improved health

• For society– Lower criminal justice costs

– Lower health-care costs

– Increased economic growth


Where’s the solution?

• Structure– Small high schools– K-8 schools

• Alignment– Curriculum reform– Textbook replacement

• Governance– Charter schools– Vouchers

• Technology


It’s the classroom

• Variability at the classroom level is up to 4 times greater than at school level

• It’s not class size

• It’s not the between-class grouping strategy

• It’s not the within-class grouping strategy

• It’s the teacher


Teacher quality:

• A labor force issue with 2 solutions

• Replace existing teachers with better ones?– No evidence that more pay brings in better teachers

– No evidence that there are better teachers out there deterred by certification requirements

• Improve the effectiveness of existing teachers– The “love the one you’re with” strategy

– It can be done

– We know how to do it, but at scale? Quickly? Sustainably?


Learning power environments

• Key concept:– Teachers do not create learning

– Learners create learning

• Teaching as engineering learning environments

• Key features:– Create student engagement (pedagogies of

engagement)

– Well-regulated (pedagogies of contingency)


Why pedagogies of engagement?

• Intelligence is partly inherited– So what?

• Intelligence is partly environmental– Environment creates intelligence

– Intelligence creates environment

• Learning environments– High cognitive demand

– Inclusive

– Obligatory


Motivation: cause or effect?

competence

challenge

Flow

apathyboredom

relaxation

arousal

anxiety

worry control

high

low

low high

(Csikszentmihalyi, 1990)


Why pedagogies of contingency?

• For evaluating institutions

• For describing individuals

• For supporting learning– Monitoring learning

• Whether learning is taking place

– Diagnosing (informing) learning• What is not being learnt

– Forming learning• What to do about it


Effects of formative assessment

• Several major reviews of the research– Natriello (1987)

– Crooks (1988)

– Kluger & DeNisi (1996)

– Black & Wiliam (1998)

– Nyquist (2003)

• All find consistent, substantial effects


Cost/effect comparisons

Intervention Effect (sd)

Cost/yr/classroom

Class-size reduction (by 30%) 0.1 $30k

Increase teacher content knowledge by 1 sd

0.1 ?

Formative assessment/

Assessment for learning

0.2~0.3 $2k


Effects of feedback

• Kluger & DeNisi (1996)• Review of 3000 research reports• Excluding those:

– without adequate controls– with poor design– with fewer than 10 participants– where performance was not measured– without details of effect sizes

• left 131 reports, 607 effect sizes, involving 12652 individuals

• Average effect size 0.4, but– Effect sizes very variable– 40% of effect sizes were negative


Kinds of feedback (Nyquist, 2003)

• Weaker feedback only– Knowledge of results (KoR)

• Feedback only– KoR + clear goals or knowledge of correct results (KCR)

• Weak formative assessment– KCR+ explanation (KCR+e)

• Moderate formative assessment– (KCR+e) + specific actions for gap reduction

• Strong formative assessment– (KCR+e) + activity


Effect of formative assessment (HE)

N Effect

Weaker feedback only 31 0.16

Feedback only 48 0.23

Weaker formative assessment 49 0.30

Moderate formative assessment 41 0.33

Strong formative assessment 16 0.51


Key processes

• Establishing where the learner is in her/his learning

• Establishing where she/he is going

• Establishing how to get there.


Aspects of formative assessment

Where the learner is going

Where the learner is

How to get there

Teacher Clarify learning intentions

Engineering effective

discussions

Providing feedback that moves learners on

PeerUnderstand/

clarify criteria for success

Activating students as instructional resources for one another

Learner Understand criteria for success

Activating students as owners of their own learning


Five key strategies…

• Clarifying and understanding learning intentions and criteria for success

• Engineering effective classroom discussions that elicit evidence of learning

• Providing feedback that moves learners forward

• Activating students as instructional resources for each other

• Activating students as the owners of their own learning


…and one big idea

• Use evidence about learning to adapt instruction to meet student needs


Keeping Learning on Track (KLT)

• A pilot guides a plane or boat toward its destination by planning a route, taking constant readings and making careful adjustments in response to wind, currents, weather, etc.

• A KLT teacher does the same:– Plans a carefully chosen (possibly differentiated)

route ahead of time (in essence building the track)

– Takes readings along the way

– Changes course as conditions dictate


Regulation of learning

• Proactive (upstream) regulation– Planning regulation into the learning environment

– Planning for evoking information

• Interactive (downstream) regulation– ‘Negotiating the swiftly-flowing river’

– ‘Moments of contingency’

– Tightness of regulation (goals vs. horizons)

• Retrospective regulation– Structured reflection (e.g., lesson study)


Types of formative assessment

• Long-cycle– Focus: between units

– Length: four weeks to one year

• Medium-cycle– Focus: within units, between lessons

– Length: one day to two weeks

• Short-cycle– Focus: within lessons

– Length: five seconds to one hour

ETS Confidential & Proprietary

Putting it into practice


Why research hasn’t changed teaching

• Misunderstanding nature of teacher expertise

• Leaving teachers to “translate into practice”

• Failure to attend to both content and process


The nature of expertise in teaching

• Aristotle’s main intellectual virtues– Episteme: knowledge of universal truths– Techne: ability to make things– Phronesis: practical wisdom

• What works is not the right question– Everything works somewhere– Nothing works everywhere– What’s interesting is “under what conditions” does

this work?

• Teaching is mainly a matter of phronesis, not episteme


Kinds of knowing

• Learning theories are not nested– Constructivist theories explain misconceptions better than

associationist theories, but– Associationist theories explain acquisition of number facts

better than constructivist theories– Each new theory provides better accounts of things the

previous theory explained badly, but– Tends to explain poorly the things the previous theory

explained well.

• The best descriptions of what happens when you come to know something depends on what it is you are learning

• For some kinds of learning, telling works• Teaching is not one of them


Klein & Klein (1981)

Six video extracts of a person delivering cardio-pulmonary resuscitation (CPR)

5 of the video extracts are students

1 of the video extracts is an expert

Videos shown to three groups

Students, experts, instructors

Success rate in identifying expert:

Experts: 90%

Students: 50%

Instructors: 30%


25 3

9 4

Target number: 127

1

Countdown


Chess (Newell & Simon, 1973)

1684

33.54

Actual position

Random positioning

HighMiddleLowPositioning

Expertise


Teacher expertise

• Resembles expertise in other domains (Berliner, 1994)– Experts excel mainly in their own domain– Experts often develop automaticity for the repetitive operations that are

needed to accomplish their goals– Experts are more sensitive to the task demands and social situation

when solving problems.– Experts are more opportunistic and flexible in their teaching than

novices– Experts represent problems in qualitatively different ways than novices.– Experts have fast and accurate pattern recognition capabilities. Novices

cannot always make sense of what they experience.– Experts perceive meaningful patterns in the domain in which they are

experienced.– Experts begin to solve problems more slowly, but bring richer and more

personal sources of information to bear on the problem that they are trying to solve.


Knowledge creation and transmission

aaa

Dialogue

Learning by doing

Socializationsympathised knowledge Externalizationconceptual knowledge

Internalizationoperational knowledge Combinationsystemic knowledge

Tacit knowledge Explicit knowledgeto

from

Tacit knowledge

Explicit knowledge

Sharing experience Networking

After Nonaka & Tageuchi, 1995


A model for teacher learning

• Content (what we want teachers to change)– Evidence

– Ideas (strategies and techniques)

• Process (how to go about change)– Choice

– Flexibility

– Small steps

– Accountability

– Support


Content: strategies and techniques• Distinction between strategies and techniques

– Strategies define the territory of AfL (no brainers)– Teachers are responsible for choice of techniques

• Allows for customization/ caters for local context• Creates ownership• Shares responsibility

• Key requirements of techniques– embodiment of deep cognitive/affective principles – relevance– feasibility– acceptability


Design and intervention

Our design process

Teachers’ implementation process

cognitive/affectiveinsights

synergy/comprehensiveness

set ofcomponents

set ofcomponents

synergy/comprehensiveness

cognitive/affectiveinsights


Practical techniques questioning

• Key idea: questioning should– cause thinking– provide data that informs teaching

• Improving teacher questioning– generating questions with colleagues – closed v open– low-order v high-order– appropriate wait-time

• Getting away from I-R-E– basketball rather than serial table-tennis– ‘No hands up’ (except to ask a question)– class polls to review current attitudes towards an issue– ‘Hot Seat’ questioning

• All-student response systems– ABCD cards, Mini white-boards, Exit passes


Questioning in science: discussion

Ice-cubes are added to a glass of water. What happens to the level of the water as the ice-cubes melt?

A. The level of the water drops

B. The level of the water stays the same

C. The level of the water increases

D. You need more information to be sure

43 ETS Confidential & ProprietaryWilson & Draney, 2004

Questioning in science: diagnosis

The ball sitting on the table is not moving. It is not moving because:

A. no forces are pushing or pulling on the ball.

B. gravity is pulling down, but the table is in the way.

C. the table pushes up with the same force that gravity pulls down

D. gravity is holding it onto the table.

E. there is a force inside the ball keeping it from rolling off the table



Which of these is living?

A) A riverB) An acornC) LichenD) A Xerox machineE) A broken-off branch of a tree



A. Joule

B. Kilogram

C. Newton

D. Pascal

E. Watt

1. Energy

2. Force

3. Mass

4. Pressure

5. Weight

6. Work


Dinosaur extinction

• Why did dinosaurs become extinct?– A) Humans destroyed their habitat

– B) Humans killed them all for food

– C) There was a major change in climate


Save the ozone layer

What can we do to preserve the ozone layer?A. Reduce the amount of carbon dioxide produced

by cars and factories

B. Reduce the greenhouse effect

C. Stop cutting down the rainforests

D. Limit the numbers of cars that can be used when the level of ozone is high

E. Properly dispose of air-conditioners and fridges


Hinge Questions

• A hinge question is based on the important concept in a lesson that is critical for students to understand before you move on in the lesson.

• The question should fall about midway during the lesson.

• Every student must respond to the question within two minutes.

• You must be able to collect and interpret the responses from all students in 30 seconds


Practical techniques: feedback

• Key idea: feedback should– cause thinking

– provide guidance on how to improve

• Comment-only grading

• Focused grading

• Explicit reference to rubrics

• Suggestions on how to improve– ‘Strategy cards’ ideas for improvement

– Not giving complete solutions

• Re-timing assessment– (eg two-thirds-of-the-way-through-a-unit test)


Practical techniques: sharing learning intentions• Explaining learning intentions at start of lesson/unit

– Learning intentions

– Success criteria

• Intentions/criteria in students’ language

• Posters of key words to talk about learning– eg describe, explain, evaluate

• Planning/writing frames

• Annotated examples of different standards to ‘flesh out’ assessment rubrics (e.g. lab reports)

• Opportunities for students to design their own tests


Practical techniques:peer and self-assessment• Students assessing their own/peers’ work

– with rubrics– with exemplars– “two stars and a wish”

• Training students to pose questions/identifying group weaknesses

• Self-assessment of understanding– Traffic lights– Red/green discs

• End-of-lesson students’ review


Process: Teacher Learning Communities

• Teacher as local expert

• Sustained over time

• Supportive forum for learning

• Embedded in day-to-day reality

• Domain-specific


A four-part model

• Initial workshops

• Monthly TLC meetings

• Peer observations

• Training for leaders


Summary

• Raising achievement is important• Raising achievement requires improving teacher

quality• Improving teacher quality requires teacher

professional development• To be effective, teacher professional development

must address– What teachers do in the classroom– How teachers change what they do in the classroom

• AfL + TLCs– A point of (uniquely?) high leverage– A “Trojan Horse” into wider issues of pedagogy, psychology,

and curriculum


Questions?

Comments?

Documents

ETS Confidential & Proprietary What kinds of assessment promote learning? Dylan Wiliam Director, Learning and Teaching Research Center ETS NARST Annual