ResearchEDTime for a reality check?

Robert CoeResearchED Research Leads Network Day, 13 December 2014

∂

ResearchED

Engagement with educational research– “ …the start of something wonderful”– Or just another fashion?

And what does it mean anyway? What is a reasonable hope/expectation for

research and evidence to make a difference to practice?

2

∂

Dimensions of great teaching

1. Quality of instruction

2. (Pedagogical) content knowledge

3. Classroom management / behaviour / control

4. Classroom climate / relationships / expectations

5. Beliefs (theory) about subject, learning & teaching

6. Wider professional elements: collegiality, PD, stakeholder relationships

3

RESEARCH?

∂

The problem with ‘What works?’

What you want to know:What will work for me

What research/evidence tells you:Something seems to have ‘worked’ for researchers

4

Research is always equivoca

l

Is it clear what it is?

Past tense

Does it transfer to ‘real life’ in

my context?

What outcome

s are used /

valued?

∂

A choice between two or more approaches that are– Well-defined (clear exactly what you must do – even if

one approach is ‘continue normal practice’)– Feasible (cost, time, acceptability, ethics)– Replicable (anyone who wants to can do them)

A context that is generalisable – The choice and context have to be similar enough to

those faced by others

Agreement about criteria for success– What outcome matters, and how measured?

5

Requirements for evidence-based practice

∂

Requirements for evidence-based practiceAre the same as for prediction or giving advice

Teacher: I want my pupils to learn to read. Should I use synthetic phonics or real books?

Researcher: Each context is unique, research can’t tell you that

Teacher: OK, should I give them books at all, or just do dancing?

Researcher: Each context is unique, research can’t tell you that

6

∂

Why might research help (1) Knowledge of research

Some pedagogical practices are better supported by evidence than others– Aim for what is actually effective, not just fashionable

Some interventions are better supported by evidence than others– Go with the best bets, based on evidence

Evidence can help in challenging & developing teachers’ existing theories and beliefs about learning– Effective professional learning requires this

(Timperley, et al, 2007)

7

∂

Should we challenge teacher beliefs?

Beliefs about– Why some students do not learn

• Ability, social background, effort, motivation• Their fault vs my fault

– What does it mean to learn (mathematics)• Connectionist / Transmission / Discovery

orientations (Askew et al, 1997)

– How does feedback work• Correction vs accommodation

8

∂

Impact vs cost

Cost per pupil

Eff

ect

Siz

e (

mon

ths

gain

)

£00

8

£1000

Meta-cognitive

Peer tutoringEarly Years

1-1 tuitionHomework (Secondary)

Mentoring

Summer schools After

school

AspirationsPerformance pay

Teaching assistants

Smaller classes

Ability grouping

Most promising for raising attainment

May be worth it

Small effects /

high cost

Feedback

Phonics

Homework (Primary)

CollaborativeSmall gp

tuition Parental involvement

Individualised learning

ICT

Behaviour

Social

www.educationendowmentfoundation.org.uk/toolkit

∂

Why might research help (2)

Research Mindset

Monitoring ongoing effectiveness– “Know thy impact” (Hattie)

Evaluating impact of changes– What is working for you

Critical perspective– Show me the evidence– Devil’s Advocate– “Tolerance of ambiguity” (Sherrington)

10

∂

‘Distributed Research’“Central to the philosophy is the concept of ‘distributed research’, the idea that the recipients of the feedback (i.e. teachers in schools and colleges) are themselves active researchers in the process, analysing and interpreting the data, rather than simply passive recipients. The research is seen as a collaborative process. The CEM Centre has skills in test design, the creation of monitoring systems and takes advantage of economies of scale in its management of the process. It also provides the basic structure of the analysis, using the collective dataset. After that, however, interpretation is a local matter and cannot be done from the centre, though considerable support is offered to help schools and colleges to interpret the data for themselves. The important thing is that the data are fed back to the smallest unit of management that can take responsibility for acting on the feedback (Fitz-Gibbon, 1996, p. 161).”

11

Tymms & Coe, 2003

∂

Clear, well defined, replicable intervention

Good assessment of appropriate outcomes

Well-matched comparison group

EEF DIY

Evaluatio

n Guide

Key elements of good evaluation

What could

you evaluate?

∂

Do we know a good lesson when we see one?

13

∂

Lesson Observation

1. Two teachers observe the same lesson, one rates it ‘Inadequate’. What is the probability the other will agree?

a) 10% b) 40% c) 60% d) 80%

2. An observer judges a lesson ‘Outstanding’. What is the probability that pupils are really making sustained, outstanding progress?

a) 5% b) 30% c) 50% d) 70%

14

www.cem.org/blog

∂

‘Improvement’ often isn’t School/college would have improved anyway

– Volunteers/enthusiasts improve: misattributed to intervention– Chance variation (esp. if start low)

Poor outcome measures– Perceptions of those who worked hard at it– No robust assessment of pupil learning

Poor evaluation designs– Weak evaluations more likely to show positive results – Improved intake mistaken for impact of intervention

Selective reporting– Dredging for anything positive (within a study)– Only success is publicised

(Coe, 2009, 2013)