Second Round Table: Evaluation of Education Policies: International

practice and evidences.

Professor Liz Toddliz.todd@ncl.ac.uk

Case 1: Full service extended schools: evaluation of education systems that aim to tackle

inequality

Presentation on 5th Feb 2011How can we evaluate

Effectiveness, Efficiency and Improvement of Education

Policies? International perspectives, practices and

evidences

What I am going to talk about……

Extended schools in England

• Multiple initiatives since 1997• Typically: childcare; parent support; out of hours

activities; adult learning; inter-agency working • Shift from extended schools to extended services• Emergence of area-based initiatives• Similar patterns in many other countries

What can extended schools achieve?

• Student learning: Community school students show significant and widely evident gains in academic achievement and in essential areas of nonacademic development.

• Family engagement: Families of community school students show increased stability, communication with teachers and school involvement. Parents demonstrate a greater sense of responsibility for their children’s learning success.

• School effectiveness: Community schools enjoy stronger parent-teacher relationships, increased teacher satisfaction, a more positive school environment and greater community support.

• Community vitality: Community schools promote better use of school buildings, and their neighborhoods enjoy increased security, heightened community pride, and better rapport among students and residents.

(Blank, Melaville, & Shah, 2003)

An evident need

A promising context

Every Child Matters 2003

• A response to child tragedy

• Systemic change: funding, integration of services

• ‘joined-up’ services• Getting help fast• Record keeping

A dominant rationale

Extended schools are a key vehicle for delivering the Government’s objective of lifting children out of poverty and improving outcomes for them and their families…A key priority, and challenge, for schools is to reach the most disadvantaged families within a universal framework of providing mainstream services for all families.

(HM Government, 2007)

...even if we found all the factors that make schools more or less effective, we would still not be able to affect more than 30 percent of the variance in pupils’ outcomes. It has therefore become increasingly clear that a narrow focus on the school as an institution will not be sufficient to enable work on more equitable educational outcomes to progress… Interventions will need to impact more directly on pupils’ environment and life chances.

(Muijs, 2010)

School improvement

crucial – but can only go so far

What about unemployment, social and health

problems, housing?Around 50% variation

achievement due to SES

Shouldn’t agencies work more in partnership?

Child poverty fallen in UK over 10yrs – but still 1 in 3

children in poverty

8-15% variation in achievement is as a result of

the school

How can we raise school standards?

Standards aren’t everything

Principles

Schools can’t go it alone

More holistic role for schools

http://archive.teachfind.com/ttv/www.teachers.tv/videos/extended-schools.html

Achieving clarityWithin this new model, it’s important

to ask:– What activities will the school

undertake?– How will these relate to the

work of others?– Why are these important things

to do?– What’s the evidence and who

says?– How will the activities achieve

the desired ends?

The FSES evaluation

• Detailed theory of change studies of 17 projects;

• Statistical analysis of NPD; • Cost benefit analysis of FSES provision in a

sample of 10 projects; • Brief case studies of comparator schools • Questionnaire survey of pupils, parents and

staff in FSESs and comparators; • Final questionnaire survey of all FSESs.

Case study and development of

theory of change: 17

schools.

Cost-benefit

analysis: 10 schools

Statistical analysis: outcome measures

Theory of Change

A systematic and cumulative study of the links between activities, outcomes and context of the initiative

Fullbright-Anderson, Kubisch and Connell, 1998: 16

The situationHigh deprivation, low aspirationHigh unemploymentDecline of manufacturing baseHistorically low school reputation

Main strands of action

Community re-engagement in learning and parental involvement in schooling

Services for young people

Raised school performance/profile

OutcomesRaise aspirations of communityRaise achievement and attainment in schoolRemoval of barriers to learningThriving school

The situation

• Persistent absenteeism• Area of high deprivation• Lack of value placed on education by parents

and children• Culture of non-participation in activities led by

school• Low aspirations• New school building in progress

Main strands of action

Community involvement:• Support services• Community support for pupils• Pupil support for communityPupils:• Rewards for participation• Rewards for attainment• Swift and easy referral• Transition support• Early intervention• One to one support for parents

Outcomes

For pupils and community • Increased attendance • High achievement/attainment (maintenance

of progress at least)• Increased citizenship/community cohesion• Increased social capital • Raised aspirations• Seeing school as supportive

Detailed mapping of all ES activities

Simplified mapping of main strands of activity

Evaluation plan for each school

Theory of change reflections• Schools as participants in the evaluation ‘do we have to pay?’• Double-edged sword:

– Helps inform their actions– Demands their willing contribution– A developmental process for schools– FSESs change

• Practitioner thinking– shaped by immediate demands– characterised by taken for granted assumptions

• Evaluation best built into planning stage

Findings on outcomes…• Important (transformative?) impacts on individuals &

families• Some evidence of cultural change in school• Possibility of change in communities• Benefits outweigh costs – and are redistributive• Variable association with school ‘improvement’• Weak evidence of overall attainment gains• No evidence of fundamental transformation at

societal level

Findings…• FSESs achieve less highly than majority of schools –

explained by disadvantaged intake• No evidence that being educated in an FSES enables

the majority of pupils to attain more highly than they would do if they were educated in schools that did not have this status

• Attainment gaps between pupils entitled to free school meals (FSM) and with special educational needs (SEN) on the one hand, and all other pupils on the other hand are smaller in FSESs than in other schools

• FSESs targeted children in difficulties and did so in ways which had impacts on their attainments

£144,000 …

The financial benefits if one pupil achieves 5 A*-C GCSE grades or equivalent when predicted A*-G (an estimate)

More is happening for young people, for example football and homework clubs…We’ve broken down barriers and our doors have opened…There are more adults walking the corridors…It feels less like a young person’s ghetto and more of a community.(ES co-ordinator)

It has improved the reputation of the school and it is improving all the time with the full service school, the sports hall [new build] and the healthy school. (student)

I don’t eat breakfast at home and so coming here means I get breakfast (student)

twelve parents attended the smoking cessation course and one parent stopped smoking (school nurse)

I was finding, because of the nature of the community, when I looked atmy role as a headteacher which is about leading the learning and theteaching, so much of my time was being taking up dealing with the socialwork issues… I did a review over a four week period of my time and 60%of that time was social work related and that’s not where my strengthsare. My strengths are in teaching and learning. (primary head teacher – now has time to commit to teaching and learning)

We’ve got parents in the school working as learning support assistants, two are learning support assistants, our college assistants were our dinnerladies…We’ve trained them up through NVQs and they are now our college assistants. They work full time for us. Two of them work on reception and repro-graphics having also got desk top computer skills,three of them in student support helping with issues to do with the school.(head teacher)

It’s kind of like we’ve been doing this and now we can finance it properly.In the past it’s been like on a wing and a prayer. (Assistant Headteacher,LA20)

You cannot work in an inner city and say this [extended schooling] belongs outside our curriculum. It is absolutely why we got into this work.(Deputy headteacher, LA18)

We can help other people achieve their targets. The Health Authority have targets they need to achieve and we’ve got sitting clients to help them achieve some of those really difficult targets.(ES coordinator)

Previously, you called school and spoke with the SENCO and not the teachers. Now, I get to speak with teachers and get additional information and my assessments are ten times longer. I have a much rounder picture of the children. There is lots of information I can pick up [around school] from speaking with the dinner ladies.(Social worker)

Weak management could be a problem as the extended school could takeover the school and lose the focus on schooling. (secondary head teacher)

An inhibiting factor is the capacity issue. My staff work very hard andput in long hours and lots of extra curricular activities go on and we haveto ensure that we have the capacity and energy and right personnel inplace. I regularly do a 70-80 hour week. (secondary head teacher)

The community is a very fractured place and hard to define let aloneconsult with. (ES manager)

In terms of the other agencies and regeneration issues, no, they haven’tmade contact with the school and that is really frustrating because theschool is in the heart of the community and I think we should beconsulted about central changes…but I have to hear about these things. Imean, I wasn’t consulted about Sure Start and Sure Start affects myfamilies. (head teacher)

Some process issues

• Sustainability• Disconnection of individual school model• Aims – unreasonable?• Management and co-ordination• Partnership• Evaluate• Politics of extended services

– Dominance of deficit perspectives

Some ways forward?

Promise of area-based initiatives?• Promise of participatory/assets-based

models?

Making sense of it all

• ESs are no substitute for school ‘improvement’ – but may support it

• ESs have important (if limited) supportive & redistributive effects

• ESs offer a vehicle for area change• Strategic approaches beyond the single school are important• ESs raise fundamental questions about:

– The outcomes we want from schools– The relationship between schooling and other aspects of public

services & policy– Who owns schools

Further information

FSES final report:https://www.education.gov.uk/publications/standard/publicationDetail/Page1/RR852

Cummings, C., Dyson, A, Todd, L. (2011) Beyond the school gates (London, Routledge)