Improving Education:A triumph of hope over experience

Robert Coe

Inaugural Lecture, Durham University, 18 June 2013

∂

A triumph of hope over experience Experience

– Have educational standards really risen?– School improvement: Isn’t it time there was some?– Can we identify effective schools and teachers?– Is ‘evidence-based’ practice and policy the

answer?

Hope– So what should we do (that hasn’t failed yet)?

2

www.cem.org/publications

Have educational standards really risen?

3

∂

4

∂

5

Equivalent change in GCSE grades

∂

6

(Updated from Coe, 2007)

∂

ICCAMS (Hodgen et al)

7

School improvement: Isn’t it time there was some?

8

∂

1. Wait for a bad year or choose underperforming schools to start with. Most things self-correct or revert to expectations (you can claim the credit for this).

2. Take on any initiative, and ask everyone who put effort into it whether they feel it worked. No-one wants to feel their effort was wasted.

3. Define ‘improvement’ in terms of perceptions and ratings of teachers. DO NOT conduct any proper assessments – they may disappoint.

4. Only study schools or teachers that recognise a problem and are prepared to take on an initiative. They’ll probably improve whatever you do.

Mistaking School Improvement (1)(Coe, 2009)

∂

5. Conduct some kind of evaluation, but don’t let the design be too good – poor quality evaluations are much more likely to show positive results.

6. If any improvement occurs in any aspect of performance, focus attention on that rather than on any areas or schools that have not improved or got worse (don’t mention them!).

7. Put some effort into marketing and presentation of the school. Once you start to recruit better students, things will improve.

Mistaking School Improvement (2) (Coe, 2009)

Can we identify effective schools and teachers?

11

∂

Problems with school effectiveness research

‘Value-added’ is not effectiveness (Gorard, 2010; Dumay, Coe & Anumendem, 2013)

Characteristics of ‘effective schools’ – ‘strong leadership’, ‘high expectations’, ‘positive

climate’ and a ‘focus on teaching and learning’– Too vague– ‘Effects’ are tiny anyway (Scheerens, 2000, 2012)

Correlations, not causes (Coe & Fitz-Gibbon, 1998)

– Can ‘effective’ strategies be implemented?– If so, do they lead to improvement?

12

Is ‘evidence-based’ practice and policy the answer?

13

∂

Toolkit of Strategies to Improve Learning

The Sutton Trust-EEF Teaching and Learning Toolkit http://www.educationendowmentfoundation.org.uk/toolkit/

∂

Impact vs cost

Cost per pupil

Eff

ect

Siz

e (

mon

ths

gain

)

£00

8

£1000

Meta-cognitive

Peer tutoringEarly Years

1-1 tuitionHomework (Secondary)

Mentoring

Summer schools After

school

AspirationsPerformance pay

Teaching assistants

Smaller classes

Ability grouping

Promising May be

worth it

Notworth

it

Feedback

Phonics

Homework (Primary)

CollaborativeSmall gp

tuition Parental involvement

Individualised learning

ICT

Behaviour

Social

www.educationendowmentfoundation.org.uk/toolkit

∂

Some things that are popular or widely thought to be effective are probably not worth doing– Ability grouping (setting); After-school clubs;

Teaching assistants; Smaller classes; Performance pay; Raising aspirations

Some things look ‘promising’– Effective feedback; Meta- cognitive and self

regulation strategies; Peer tutoring/peer‐assisted learning strategies; Homework

Key messages

∂

Clear, simple advice:

Choose from the top left Go back to school and do it

17

For every complex problem there is an answer that is clear, simple, and wrong

H.L. Mencken

∂

18

Why not? We have been doing some of these things for a long

time, but have generally not seen improvement Research evidence is problematic

– Sometimes the existing evidence is thin– Research studies may not reflect real life– Context and ‘support factors’ may matter (Cartwright and Hardie,

2012)

Implementation is problematic– We may think we are doing it, but are we doing it right?– We do not know how to get large groups of teachers and

schools to implement these interventions in ways that are faithful, effective and sustainable

So what should we do (that hasn’t failed yet)?

19

∂

Four steps to improvement

Think hard about learning Invest in effective professional development Evaluate teaching quality Evaluate impact of changes

1. Think hard about learning

∂

Impact vs cost

Cost per pupil

Eff

ect

Siz

e (

mon

ths

gain

)

£00

8

£1000

Meta-cognitive

Peer tutoringEarly Years

1-1 tuitionHomework (Secondary)

Mentoring

Summer schools After

school

AspirationsPerformance pay

Teaching assistants

Smaller classes

Ability grouping

Promising May be

worth it

Notworth

it

Feedback

Phonics

Homework (Primary)

CollaborativeSmall gp

tuition Parental involvement

Individualised learning

ICT

Behaviour

Social

www.educationendowmentfoundation.org.uk/toolkit

∂

Impact vs cost

Cost per pupil

Eff

ect

Siz

e (

mon

ths

gain

)

£00

8

£1000

Meta-cognitive

Peer tutoring

Homework (Secondary)

After school

AspirationsPerformance pay

Teaching assistants

Smaller classes

Ability grouping

Feedback

Phonics

Collaborative

www.educationendowmentfoundation.org.uk/toolkit

Does your theory of learning

explain why …

These

work?

These

don’t?

∂

Poor Proxies for Learning Students are busy: lots of work is done

(especially written work) Students are engaged, interested, motivated Students are getting attention: feedback,

explanations Classroom is ordered, calm, under control Curriculum has been ‘covered’ (ie presented to

students in some form) (At least some) students have supplied correct

answers (whether or not they really understood them or could reproduce them independently)

24

∂

Learning happens when people have

to think hard

A simple theory of learning

∂

Hard questions about your school

How many minutes does an average pupil on an average day spend really thinking hard?

Do you really want pupils to be ‘stuck’ in your lessons?

If they knew the right answer but didn’t know why, how many pupils would care?

26

2. Invest in effective CPD

∂

How do we get students to learn hard things?

Eg Place value Persuasive

writing Music

composition Balancing

chemical equations

• Explain what they should do• Demonstrate it• Get them to do it (with

gradually reducing support)• Provide feedback • Get them to practise until it is

secure• Assess their skill/

understanding

∂

How do we get teachers to learn hard things?

Eg Using formative

assessment Assertive

discipline How to teach

algebra

• Explain what they should do

∂

Intense: at least 15 hours, preferably 50 Sustained: over at least two terms Content focused: on teachers’ knowledge of

subject content & how students learn it Active: opportunities to try it out & discuss Supported: external feedback and networks to

improve and sustain Evidence based: promotes strategies

supported by robust evaluation evidence

What (probably) makes CPD effective?

3. Evaluate teaching quality

∂

Every teacher needs to improve, not because they are not good enough, but because they can be even better.

Dylan Wiliam

∂

Sources of evidence:1. Colleagues (peers, SMs) observing lessons

2. Trained outsiders observing lessons

3. Pupils’ test score gains

4. Progress in NC levels (from teacher assessment)

5. Pupils’ ratings of teacher/lesson quality

6. Teacher qualifications

7. Tests of teachers’ content knowledge

8. Parents’ ratings

9. Ofsted ratings

10. Colleagues’ (including senior managers) perceptions

11. Teachers’ self-evaluation

Identifying the best teachers

∂

34

Next generation of CEM systems …

Assessments that are– Comprehensive, across the full range of curriculum areas,

levels, ages, topics and educationally relevant abilities– Diagnostic, with evidence-based follow-up– Interpretable, calibrated against norms and criteria– High psychometric quality

Feedback that is– Bespoke to individual teacher, for their students and classes– Multi-component, incorporating learning gains, pupil ratings,

peer feedback, self-evaluation, …– Diagnostic, with evidence-based follow-up

Constant experimenting

4. Evaluate impact of changes

∂

We are sure this works This is so important we need it to work Everyone is working really hard and fully committed to this Evaluating would be a lot of work We don’t have the data to be able to evaluate We don’t know how to evaluate We can’t do a really good evaluation, so what is the point of

doing it badly? We do happy sheets and ask people what they thought of it; isn’t

that enough? You can’t do randomised trials in education What works is different in different schools or contexts

Bad reasons not to evaluate

∂

Clear, well defined intervention

Good assessment of appropriate outcomes

Well-matched comparison group

Key elements of good evaluation

∂

38

A triumph of hope over experience

Experience– So far, we haven’t cracked it: don’t keep doing the

same things

Hope– Think hard about learning– Invest in effective professional development– Evaluate teaching quality– Evaluate impact of changes