The British Experience of “evidence based policy making”

1997-2004

Ben PageDirector

MORI Social Research Instituteinfo@mori.ie

What has changed Serious investment in research and performance management capacity to

gather and analyse hard data of all sorts

– mandatory local and national survey research in local government and health services, 100s of PIs……

Establishment of stronger centre to shift from policy to delivery….– Office of Public Service Reform– Strategy Unit– Prime Minister’s Delivery Unit– Among others!

Major programmes of reform and investment in learning and skills, regeneration, education, NHS, reform and modernisation in local government

Evaluation of these programmes varies dramatically

What has changed Some programmes have large scale and very costly evaluation

programmes – more around investment in local communities eg

Sure Start

New Deal for Communities

Connexxions

LSPs etc

But others have none at all – simply implementation

The Aim of Sure Start

To work with parents-to-be, parents and children to promote the physical, intellectual and social development of babies and young children - particularly those who are disadvantaged - so that they can flourish at home and when they get to school, and thereby break the cycle of disadvantage for the current generation of young children.

Sure Start - A Case Study

By 2004, 524 Sure Start programmes will be helping up to 400,000 children living in disadvantaged areas, including a 1/3 of under 4s living in poverty.

Sure Start programmes will work with parents and parents-to-be to improve children's life chances through better access to:  • Family support • Advice on nurturing • Health services • Early learning

LOCAL CONTEXT ANALYSISAIMS

Describe Sure Start Programme Areas – what and where are they?

Compare Programme Areas - with each other and nationally

Relate area characteristics to Implementation of Sure Start

Look at change over time in communities

The Digitisation of Sure Start Programme Boundaries

Step 1 - Paper maps are received from the Sure Start programmes.

Step 2 - The boundaries are captured as a series of points on the National Grid.

The shape of the Sure Start programme area can then be analysed in GIS (Geographic Information Systems) software.

A Sure Start programme area captured by GIS

The Digitisation of Sure Start Programme Boundaries

Step 4 - The postcodes become the common link to data from other sources, to profile each Sure Start programme area.

The Common Link

Child Benefit Data

Child Protection Register Data

School Exclusions Data

Birth Rates Data

The Digitisation of Sure Start Programme Boundaries

LOCAL CONTEXT ANALYSISWhat we are studying

– Demographics and Poverty– Economic Profile and Labour Market– Crime, Danger and Disorder – Adult Health and Disability– Child Health and Development– School readiness and achievement– Parenting – Neighbourhood Networks – Access to services for 0-3 year olds and their families

IMPLEMENTATION

The study incorporates quantitative and qualitative methods, and has three components:

A National Survey of 260 Sure Start Programmes

An In-Depth Study of 26 programmes A Series of Themed Evaluations

IMPACTStage 1: 2001\2002

STAGE 1

• local context analysis information

• proposed approach to Sure Start

• implementation data

100%OF SURE START COMMUNITIES

IMPACTStage 2: 2003/2004

Stage 1

STAGE 2 150 SURE START COMMUNITIES 50 CONTROL COMMUNITIES

•cross-sectional study of 9 month, and 3 year olds and their families

Control groups allows

us to identify effects

of programme

IMPACTStage 3: 2005 and beyond

Stage 1

Stage 2

STAGE 3100 SURE START COMMUNITIES

• 80 families per community • longitudinal study of 3 and 5 year olds

Interim conclusions so far!!

“Limited evidence that SSLPs may influence some aspects of parenting, i.e. acceptance”

“Stronger evidence that SOME specific SSLPs rather than SSLPs generally may be effective on a composite of child/parenting outcomes”

“The distinctive features of “effective” SSLPs may be more about community characteristics rather than programme characteristics but this needs further investigation…”

The rise and rise of performance management and inspection……..

Huge numbers of inspections – but becoming clear that peer review and self assessment are as important to improvement as inspection/external review and targets

But unlikely that improvements would have happened based solely on self assessment – inspection and measurement has become accepted over the last seven years……

There is a huge thirst for information

Increasing expectations of the “evidence base”, devolution of delivery to regional, local authority and sub-local authority level,

having data fit for purpose

How much detail can you have, and how much can you cope with?

Other challenges in England…..

– Practical limits for micro-management?– Changing agendas?

• Some evaluations were set up for regimes that have now changed so fundamentally that it is very hard to isolate the impact of a particular initiative…..

– So many initiatives, unclear what you are evaluating

More and more detailed understanding of multiple factors impacting on measurement

Similar investment and delivery standards across a service do not lead to uniform outcomes in perception or delivery

Impact of place, impact of nature of community, impact of media coverage, political belief and values are becoming more and more apparent…….

45

50

55

60

65

70

75

80

85

90

95

100

5 10 15 20 25 30 35 40 45 50 55 60

Satisfaction with Area v Deprivation

Net satisfaction (+)

Average Deprivation Score (IMD 2000)Base: MORI Norm Database

GCSE Results versus Deprivation Index

R2 = 0.61

0

10

20

30

40

50

60

70

0 10 20 30 40 50 60

GCSE Results - 5+ grade A-C

Deprivation Index

Hackney

Sutton

Camden

Portsmouth

0

10

20

30

40

50

60

70

80

5 10 15 20 25 30 35 40 45 50 55 60

Satisfaction with council versus deprivation score

Net satisfaction with Council (+%)

Deprivation Score (IMD 2000)Base: All

BrentHarlow

OldhamBirmingham

Vale of White Horse

Gateshead

Correlation -0.65

Manchester

Sunderland

Mid BedsCamden

MORI Frontiers Analysis

0

10

20

30

40

50

60

70

80

0 20 40 60 80

Deprivation and Ethnicity

% S

atisfie

d w

ith

Co

un

cil

Likely frontier of satisfaction

AverageWeak

R2 = 0.4877

65

70

75

80

85

0 20 40 60 80

Ove

rall

PC

T r

atin

g

Ethnic diversitySource: PCT patient surveys 2003/MORI

Ethnic Diversity and overall PCT ratings

Bradford CityPCT

NewhamPCT

Lambeth PCT

Haringey PCT

Greenwich PCT

South BirminghamPCT

Barking &DagenhamPCT

Preston PCT

ThurrockPCT

BasildonPCT

Airedale PCTMid Devon PCT

R2 = 0.5237

55

60

65

70

75

80

85

90

0 20 40 60 80

Rat

ings

of

over

all c

are

Ethnic diversitySource: NHS Acute Trust inpatient surveys 2001-2/MORI

Ethnic diversity and ratings of inpatient care

Newham Healthcare

NorthMiddlesexUniversity Hospital

Airedale NHS Trust

University Hospitalsof Leicester

West MiddlesexUniversity Hospital

Sandwell &West Birmingham Hospitals

MedwayNHS Trust

Taunton & SomersetNHS Trust

The Perception Gap in delivery: hard numbers

alone or “belief” in progress?

The Perception Gap in delivery: hard numbers

alone or “belief” in progress?

Optimism about future improvements to public services is

not clear despite billions of extra

investment

38%38% 36%31%28%31%30%37%39%

62%57%57%55%52%

36%46%45%54%35%

59%50%52%52%50%54%32%42%40%

Will the Government deliver on Public Services?

June 2001

Oct 2001

Mar 2002

Q Do you agree or disagree… “In the long term, this government’s policies will improve the state of England’s Public Services”

Disagree

Agree

Nov 2001

May 2002

Sept 2002

Dec 2002

Mar 2003

Jun 2003

Sept 2003

Jul 2003

Dec 2003

Mar 2004

Base: 966 British adults, 18-20 June 2004

Source: MORI Social Research Institute Delivery Index

June2004

But they have great expectations - often raised by politicians

Choose carefully what you promise....

Huge cynicism to overcome

2%

9%

25%

27%

37%

44%

48%

Base: All - Spring 2003 (1,041)

Increased numbers of doctors and nurses

Q In your view, which TWO, if any, of the following changes would best indicate to you personally that the NHS was improving?

Priorities for improvement

Waiting less than 4 hours in A&E

Reduced waiting times for inpatient admissions

Being able to see a local doctor/GP or practice nurse within 48 hours

Reduced waiting times for outpatient appointments

Being able to choose time and date and place of treatment

Don’t know/refused

Spring 2003

Number of NHS nurses in England, 1997-2003

330,620 332,200329,640

350,380

367,520

335,950

386,400

300,000

310,000

320,000

330,000

340,000

350,000

360,000

370,000

380,000

390,000

400,000

1997 1998 1999 2000 2001 2002 2003

Nu

mb

er

of

nu

rse

s

Source : Department of Health

England has lots more nurses….

… and doctorsNumber of NHS doctors (GPs & consultants) in England, 1997-2003

50,863

52,021

53,308

54,653

56,467

58,252

61,400

40,000

45,000

50,000

55,000

60,000

65,000

1997 1998 1999 2000 2001 2002 2003

Nu

mb

er o

f d

oct

ors

Source : Department of Health

Source : Department of Health

Number of inpatient waiters

0

50,000

100,000

150,000

200,000

250,000

300,000

A M J J A S O N D J F M A M J J A S O N D J F M A M J J A S O N D J F M A M J J A S O N D J F M A M J J A S O N D

2001/02 2002/03 2003/04 2004/05 2005

6+ month

9+ month

12+ month

Waiting lists ARE falling

March 04 milestone

Source : Department of Health

Number of inpatient waiters

0

50,000

100,000

150,000

200,000

250,000

300,000

A M J J A S O N D J F M A M J J A S O N D J F M A M J J A S O N D J F M A M J J A S O N D J F M A M J J A S O N D

2001/02 2002/03 2003/04 2004/05 2005

6+ month

9+ month

12+ month

Waiting lists ARE falling

“Waiting times are generally shorter than at any time in the history of the NHS.”

King’s Fund Briefing PaperMarch 2004

Respected independent organisations confirm that progress is being made

6256

52 5249

2623 24

2925

4 48 6 7

0

10

20

30

40

50

60

70

2000 (S) 2001 (W) 2002 (S) 2002 (W) 2003 (S)

But the public disagree......

% who think waiting times are getting...

Base: All -Spring 2003 (1,000) Winter 2002 (1,002), Spring 2002 (1,041) Winter 2001 (1,021) and Spring 2000 (1,046)

Same

Longer

Shorter

The Right Waiting Time Targets?

Base: All respondents (2,037)

68.692

6.125

-66.161

-8.656

-80

-60

-40

-20

0

20

40

60

80

1 month 3 months 6 months 9 months

Positive

Negative

Ratings of overallinpatient care

R2 = 0.94

Respect & dignity

Pain control

+37%

+13%

Clean wards/toilets+15%

Purpose of medicines explained +13%

Communicatedside effects

A&E organisation

Privacy to discusstreatment

+8%

+7%

+6%

Key Drivers of Perceptions of Hospital Care – things to focus on

Waiting is less important

to patients

than it is to the public

Most important drivers of satisfaction with the running of the NHS at the national level

Political belief/Support for the government

Age

Poor experiences of A&E services

Staff bad-mouthing

Media coverage

Direct NHS communication

Positive experiences of inpatient services

Greatest drivers ofoverall satisfaction

Weaker driversof overall

satisfaction

What are the issues the public see the NHS confronting?

1. Under-funded 33%

2. Bureaucracy 31%

3. Not enough doctors/nurses 27%

4. Waiting 19%

5. Cleanliness/superbugs/MRSA 16%

6. Overworked staff 12%

7. Pay 7%

8. Over-stretched/ageing population 5%

9. Staff training 5%

10. Staff retention 5%

11. Foreigners/immigrants 5%

12. Shortage of beds 4%

13. Quality of treatment/care 4%

14. Shortage of NHS dentists 3%

Source: MORI Public omnibus (12 - 16 August 04)

Crime...

The aim is to reduce domestic burglary by 25% ...

1,261,364

-

200,000

400,000

600,000

800,000

1,000,000

1,200,000

1,400,000

2000 BCS(crime in 1999)

2001/02interviews

2002/03interviews

2003/04interviews

2004/05interviews

2005/06interviews

Nu

mb

er o

f o

ffen

ces

Domestic Burglary - 25% reduction from 1998-99 to 2005British Crime Survey

Source: Home Office

946,023

… and performance is ahead of trajectory

1,261,364

965,502 973,522

-

200,000

400,000

600,000

800,000

1,000,000

1,200,000

1,400,000

2000 BCS(crime in 1999)

2001/02interviews

2002/03interviews

2003/04interviews

2004/05interviews

2005/06interviews

Nu

mb

er o

f o

ffen

ces

Domestic Burglary - 25% reduction from 1998-99 to 2005British Crime Survey

Source: Home Office

946,023

The aim is to reduce vehicle crime by 30% ...

2,941,927

-

500,000

1,000,000

1,500,000

2,000,000

2,500,000

3,000,000

3,500,000

2000 BCS(crime in 1999)

2001/02interviews

2002/03interviews

2003/04interviews

2004/05interviews

Vehicle Crime - 30% reduction from 1998-99 to 2004British Crime Survey

Nu

mb

er

of

off

en

ce

s

Source: Home Office

2,059,349

… and performance is ahead of trajectory

2,366,0702,486,314

2,941,927

-

500,000

1,000,000

1,500,000

2,000,000

2,500,000

3,000,000

3,500,000

2000 BCS(crime in 1999)

2001/02interviews

2002/03interviews

2003/04interviews

2004/05interviews

Vehicle Crime - 30% reduction from 1998-99 to 2004British Crime Survey

Nu

mb

er

of

off

en

ce

s

Source: Home Office

2,059,349

Number of Violent Crimes - British Crime Survey

2,781,0002,784,000

3,423,0003,665,000

-

500,000

1,000,000

1,500,000

2,000,000

2,500,000

3,000,000

3,500,000

4,000,000

1998 BCS 2000 BCS 2001/2 2002/3

Nu

mb

er o

f O

ffen

ces

(Crime in 1997) (Crime in 1999) Interviews InterviewsSource: Home Office

Total violent crime has fallen since 1997

But most people believe crime is rising

Beliefs about the change in the national crime rate 1996 to 2002/03

46%

30% 33%25%

30%38%

29%

29%34%

31%

34%

35%

0%

10%

20%

30%

40%

50%

60%

70%

80%

1996 1998 2000 2001 2001/02 2002/03

Pe

rcen

tag

e s

ayi

ng

cri

me

has

incr

eas

ed

Lot more Little moreSource: Home Office

Gun crime has increased - as has media coverage

-

2,000

4,000

6,000

8,000

10,000

12,000

1997/98 1998/99 1999/00 2000/01 2001/02

Nu

mb

er o

f C

rim

es

Total Recorded Crimes Involving Firearms (other than air weapons)

Source: Home Office

Not focussing on the right crime?Fear of anti social-behaviour

0%

5%

10%

15%

20%

25%

30%

35%

40%

1992 1994 1996 1998 2000 2001 2001/02 2002/03

Pe

rce

nta

ge

th

at

pe

rce

ive

dis

ord

er

to b

e a

've

ry' o

r 'f

air

ly' b

ig p

rob

lem

Vandalism and graffiti

Teenagers hanging

around on the streetsPeople using /

dealing drugs

Source: Home Office

Where there is less direct experience,

communications, media coverage and viral

marketing become more important

Waiting Lists: Media Impact

Media Impact score (exposure X slant) for

waiting lists

Rating on reducing waiting lists

4 week rolling data

-12000

-10000

-8000

-6000

-4000

-2000

0

2000

4000

6000

06-M

ar

10-A

pr

15-M

ay

19-J

un

24-J

ul

28-A

ug

02-O

ct

06-N

ov

11-D

ec

15-J

an

20-F

eb

26-M

ar

30-A

pr

04-J

un

09-J

ul

13-A

ug

17-S

ep

22-O

ct

26-N

ov

04-F

eb

11-M

ar

14-A

pr

20-M

ay

24-J

un

29-J

ul

02-S

ep

07-O

ct

11-N

ov

16-D

ec

-30

-20

-10

0

10

20

Disconnection between own

good(ish) experience and national

“political”debate?

Provision of Service Locally and Nationally

79%

65%

47%

31%

26%

-41%

-62%

17%

18%

Base: All respondents (1,031)

The government has the right policies for the NHS

% Agree

Q To what extent, if at all, do you agree or disagree with the following statements?

The NHS is providing a good service nationally

My local NHS is providing me with a good service

% Disagree

(-36)

(+16)

(+62)

I am satisfied with the way the Government is doing its job overall*

I was satisfied with my last visit to hospital

(-10)

(-47)

-31

9%

54%

26%

8%3%

Confidence in CJS - which is key?

Not very confident Fairly

confident

Very confident

In the area where you live

Q Overall, how confident are you about the way crime is dealt with…?

Not confident

at all

Don’t know

Base: 2,001 GB Feb-March 2003, aged 16+

44%

39%

12%2%

In England and Wales

Not very

confident

Fairly confident

Very confident 3%Not confident

at all

Don’t know

+29% net confident -4% net confident

Clash between local and national priorities for many

managers and local politicians

0

10

20

30

40

50

0 10 20 30 40 50 60

What are the issues affecting quality of life in local communities?

Most need improving locally

Important generally

Low level crime

Health servicesHousing

Activities for teenagers

Public transportRoad/pavement repairs

Shopping

Wages/cost of living

Low traffic

Job prospects

Facilities for young children

Sports/leisure facs

Clean streets

Education

Base: 2,031 British adults 15+, 18-22 October 2001

Culture

Community activities

Open spaces

Low pollution

Race relations Access to nature

Falling satisfaction with street scene - nearly everywhere

0

10

20

30

40

50

60

70

80

1988 1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002

Net satisfied with street cleaning

National MORI average

Leicester

BV Pilots

People’s Panel

SouthwarkCamden

PortsmouthChelmsford

Brent

Unintended consequences

of a targets culture

Performance measures: Coherence, Confusion and Unintended Consequences

Sound ‘silo’ ideas Practical collisions

Visible policing

Home Office vs CPS prosecution requirements

Visibility vs NCRS requirements

Data collection

Quality of evidence

Treatment times

Specialism resourcing

Clinical priorities vs fast treatment

Knock on effects

Inclusion vs 5 A-C gradesSocial inclusion

Excellence & enjoyment

GCSE A-C

E&E vs Key Stage SAT scores

Performance review: Continuous improvement?

*NB only the managers’ view, not necessarily the frontline or public

Real improvements seen Worthwhile despite the administrative burden*

Manager’s view

PlateauCentre’s view

Improvement

What allowance?

Something has to give

Key Points Real progress – but rarely smooth

– Cultural change takes time– Ministerial change – changing regimes, philosophies

Measurement matters – need to get in place from beginning at level you require – as Time Machines are not yet invented

Need to choose to measure and promise the right things

Need to be able to factor out exogenous factors in assessing interventions– Personality, politics, community

Multiple interventions make things much more difficult: unintended effects

Anticipate the tension between effective evaluation and the needs of delivery, and politics

Questions, Commentsemmet.o’briain@mori.ie