The British Experience of “evidence based policy making”
1997-2004
Ben PageDirector
MORI Social Research [email protected]
What has changed Serious investment in research and performance management capacity to
gather and analyse hard data of all sorts
– mandatory local and national survey research in local government and health services, 100s of PIs……
Establishment of stronger centre to shift from policy to delivery….– Office of Public Service Reform– Strategy Unit– Prime Minister’s Delivery Unit– Among others!
Major programmes of reform and investment in learning and skills, regeneration, education, NHS, reform and modernisation in local government
Evaluation of these programmes varies dramatically
What has changed Some programmes have large scale and very costly evaluation
programmes – more around investment in local communities eg
Sure Start
New Deal for Communities
Connexxions
LSPs etc
But others have none at all – simply implementation
The Aim of Sure Start
To work with parents-to-be, parents and children to promote the physical, intellectual and social development of babies and young children - particularly those who are disadvantaged - so that they can flourish at home and when they get to school, and thereby break the cycle of disadvantage for the current generation of young children.
Sure Start - A Case Study
By 2004, 524 Sure Start programmes will be helping up to 400,000 children living in disadvantaged areas, including a 1/3 of under 4s living in poverty.
Sure Start programmes will work with parents and parents-to-be to improve children's life chances through better access to: • Family support • Advice on nurturing • Health services • Early learning
LOCAL CONTEXT ANALYSISAIMS
Describe Sure Start Programme Areas – what and where are they?
Compare Programme Areas - with each other and nationally
Relate area characteristics to Implementation of Sure Start
Look at change over time in communities
The Digitisation of Sure Start Programme Boundaries
Step 1 - Paper maps are received from the Sure Start programmes.
Step 2 - The boundaries are captured as a series of points on the National Grid.
The shape of the Sure Start programme area can then be analysed in GIS (Geographic Information Systems) software.
A Sure Start programme area captured by GIS
The Digitisation of Sure Start Programme Boundaries
Step 4 - The postcodes become the common link to data from other sources, to profile each Sure Start programme area.
The Common Link
Child Benefit Data
Child Protection Register Data
School Exclusions Data
Birth Rates Data
The Digitisation of Sure Start Programme Boundaries
LOCAL CONTEXT ANALYSISWhat we are studying
– Demographics and Poverty– Economic Profile and Labour Market– Crime, Danger and Disorder – Adult Health and Disability– Child Health and Development– School readiness and achievement– Parenting – Neighbourhood Networks – Access to services for 0-3 year olds and their families
IMPLEMENTATION
The study incorporates quantitative and qualitative methods, and has three components:
A National Survey of 260 Sure Start Programmes
An In-Depth Study of 26 programmes A Series of Themed Evaluations
IMPACTStage 1: 2001\2002
STAGE 1
• local context analysis information
• proposed approach to Sure Start
• implementation data
100%OF SURE START COMMUNITIES
IMPACTStage 2: 2003/2004
Stage 1
STAGE 2 150 SURE START COMMUNITIES 50 CONTROL COMMUNITIES
•cross-sectional study of 9 month, and 3 year olds and their families
Control groups allows
us to identify effects
of programme
IMPACTStage 3: 2005 and beyond
Stage 1
Stage 2
STAGE 3100 SURE START COMMUNITIES
• 80 families per community • longitudinal study of 3 and 5 year olds
Interim conclusions so far!!
“Limited evidence that SSLPs may influence some aspects of parenting, i.e. acceptance”
“Stronger evidence that SOME specific SSLPs rather than SSLPs generally may be effective on a composite of child/parenting outcomes”
“The distinctive features of “effective” SSLPs may be more about community characteristics rather than programme characteristics but this needs further investigation…”
The rise and rise of performance management and inspection……..
Huge numbers of inspections – but becoming clear that peer review and self assessment are as important to improvement as inspection/external review and targets
But unlikely that improvements would have happened based solely on self assessment – inspection and measurement has become accepted over the last seven years……
There is a huge thirst for information
Increasing expectations of the “evidence base”, devolution of delivery to regional, local authority and sub-local authority level,
having data fit for purpose
How much detail can you have, and how much can you cope with?
Other challenges in England…..
– Practical limits for micro-management?– Changing agendas?
• Some evaluations were set up for regimes that have now changed so fundamentally that it is very hard to isolate the impact of a particular initiative…..
– So many initiatives, unclear what you are evaluating
More and more detailed understanding of multiple factors impacting on measurement
Similar investment and delivery standards across a service do not lead to uniform outcomes in perception or delivery
Impact of place, impact of nature of community, impact of media coverage, political belief and values are becoming more and more apparent…….
45
50
55
60
65
70
75
80
85
90
95
100
5 10 15 20 25 30 35 40 45 50 55 60
Satisfaction with Area v Deprivation
Net satisfaction (+)
Average Deprivation Score (IMD 2000)Base: MORI Norm Database
GCSE Results versus Deprivation Index
R2 = 0.61
0
10
20
30
40
50
60
70
0 10 20 30 40 50 60
GCSE Results - 5+ grade A-C
Deprivation Index
Hackney
Sutton
Camden
Portsmouth
0
10
20
30
40
50
60
70
80
5 10 15 20 25 30 35 40 45 50 55 60
Satisfaction with council versus deprivation score
Net satisfaction with Council (+%)
Deprivation Score (IMD 2000)Base: All
BrentHarlow
OldhamBirmingham
Vale of White Horse
Gateshead
Correlation -0.65
Manchester
Sunderland
Mid BedsCamden
MORI Frontiers Analysis
0
10
20
30
40
50
60
70
80
0 20 40 60 80
Deprivation and Ethnicity
% S
atisfie
d w
ith
Co
un
cil
Likely frontier of satisfaction
AverageWeak
R2 = 0.4877
65
70
75
80
85
0 20 40 60 80
Ove
rall
PC
T r
atin
g
Ethnic diversitySource: PCT patient surveys 2003/MORI
Ethnic Diversity and overall PCT ratings
Bradford CityPCT
NewhamPCT
Lambeth PCT
Haringey PCT
Greenwich PCT
South BirminghamPCT
Barking &DagenhamPCT
Preston PCT
ThurrockPCT
BasildonPCT
Airedale PCTMid Devon PCT
R2 = 0.5237
55
60
65
70
75
80
85
90
0 20 40 60 80
Rat
ings
of
over
all c
are
Ethnic diversitySource: NHS Acute Trust inpatient surveys 2001-2/MORI
Ethnic diversity and ratings of inpatient care
Newham Healthcare
NorthMiddlesexUniversity Hospital
Airedale NHS Trust
University Hospitalsof Leicester
West MiddlesexUniversity Hospital
Sandwell &West Birmingham Hospitals
MedwayNHS Trust
Taunton & SomersetNHS Trust
The Perception Gap in delivery: hard numbers
alone or “belief” in progress?
The Perception Gap in delivery: hard numbers
alone or “belief” in progress?
Optimism about future improvements to public services is
not clear despite billions of extra
investment
38%38% 36%31%28%31%30%37%39%
62%57%57%55%52%
36%46%45%54%35%
59%50%52%52%50%54%32%42%40%
Will the Government deliver on Public Services?
June 2001
Oct 2001
Mar 2002
Q Do you agree or disagree… “In the long term, this government’s policies will improve the state of England’s Public Services”
Disagree
Agree
Nov 2001
May 2002
Sept 2002
Dec 2002
Mar 2003
Jun 2003
Sept 2003
Jul 2003
Dec 2003
Mar 2004
Base: 966 British adults, 18-20 June 2004
Source: MORI Social Research Institute Delivery Index
June2004
But they have great expectations - often raised by politicians
Choose carefully what you promise....
Huge cynicism to overcome
2%
9%
25%
27%
37%
44%
48%
Base: All - Spring 2003 (1,041)
Increased numbers of doctors and nurses
Q In your view, which TWO, if any, of the following changes would best indicate to you personally that the NHS was improving?
Priorities for improvement
Waiting less than 4 hours in A&E
Reduced waiting times for inpatient admissions
Being able to see a local doctor/GP or practice nurse within 48 hours
Reduced waiting times for outpatient appointments
Being able to choose time and date and place of treatment
Don’t know/refused
Spring 2003
Number of NHS nurses in England, 1997-2003
330,620 332,200329,640
350,380
367,520
335,950
386,400
300,000
310,000
320,000
330,000
340,000
350,000
360,000
370,000
380,000
390,000
400,000
1997 1998 1999 2000 2001 2002 2003
Nu
mb
er
of
nu
rse
s
Source : Department of Health
England has lots more nurses….
… and doctorsNumber of NHS doctors (GPs & consultants) in England, 1997-2003
50,863
52,021
53,308
54,653
56,467
58,252
61,400
40,000
45,000
50,000
55,000
60,000
65,000
1997 1998 1999 2000 2001 2002 2003
Nu
mb
er o
f d
oct
ors
Source : Department of Health
Source : Department of Health
Number of inpatient waiters
0
50,000
100,000
150,000
200,000
250,000
300,000
A M J J A S O N D J F M A M J J A S O N D J F M A M J J A S O N D J F M A M J J A S O N D J F M A M J J A S O N D
2001/02 2002/03 2003/04 2004/05 2005
6+ month
9+ month
12+ month
Waiting lists ARE falling
March 04 milestone
Source : Department of Health
Number of inpatient waiters
0
50,000
100,000
150,000
200,000
250,000
300,000
A M J J A S O N D J F M A M J J A S O N D J F M A M J J A S O N D J F M A M J J A S O N D J F M A M J J A S O N D
2001/02 2002/03 2003/04 2004/05 2005
6+ month
9+ month
12+ month
Waiting lists ARE falling
“Waiting times are generally shorter than at any time in the history of the NHS.”
King’s Fund Briefing PaperMarch 2004
Respected independent organisations confirm that progress is being made
6256
52 5249
2623 24
2925
4 48 6 7
0
10
20
30
40
50
60
70
2000 (S) 2001 (W) 2002 (S) 2002 (W) 2003 (S)
But the public disagree......
% who think waiting times are getting...
Base: All -Spring 2003 (1,000) Winter 2002 (1,002), Spring 2002 (1,041) Winter 2001 (1,021) and Spring 2000 (1,046)
Same
Longer
Shorter
The Right Waiting Time Targets?
Base: All respondents (2,037)
68.692
6.125
-66.161
-8.656
-80
-60
-40
-20
0
20
40
60
80
1 month 3 months 6 months 9 months
Positive
Negative
Ratings of overallinpatient care
R2 = 0.94
Respect & dignity
Pain control
+37%
+13%
Clean wards/toilets+15%
Purpose of medicines explained +13%
Communicatedside effects
A&E organisation
Privacy to discusstreatment
+8%
+7%
+6%
Key Drivers of Perceptions of Hospital Care – things to focus on
Waiting is less important
to patients
than it is to the public
Most important drivers of satisfaction with the running of the NHS at the national level
Political belief/Support for the government
Age
Poor experiences of A&E services
Staff bad-mouthing
Media coverage
Direct NHS communication
Positive experiences of inpatient services
Greatest drivers ofoverall satisfaction
Weaker driversof overall
satisfaction
What are the issues the public see the NHS confronting?
1. Under-funded 33%
2. Bureaucracy 31%
3. Not enough doctors/nurses 27%
4. Waiting 19%
5. Cleanliness/superbugs/MRSA 16%
6. Overworked staff 12%
7. Pay 7%
8. Over-stretched/ageing population 5%
9. Staff training 5%
10. Staff retention 5%
11. Foreigners/immigrants 5%
12. Shortage of beds 4%
13. Quality of treatment/care 4%
14. Shortage of NHS dentists 3%
Source: MORI Public omnibus (12 - 16 August 04)
Crime...
The aim is to reduce domestic burglary by 25% ...
1,261,364
-
200,000
400,000
600,000
800,000
1,000,000
1,200,000
1,400,000
2000 BCS(crime in 1999)
2001/02interviews
2002/03interviews
2003/04interviews
2004/05interviews
2005/06interviews
Nu
mb
er o
f o
ffen
ces
Domestic Burglary - 25% reduction from 1998-99 to 2005British Crime Survey
Source: Home Office
946,023
… and performance is ahead of trajectory
1,261,364
965,502 973,522
-
200,000
400,000
600,000
800,000
1,000,000
1,200,000
1,400,000
2000 BCS(crime in 1999)
2001/02interviews
2002/03interviews
2003/04interviews
2004/05interviews
2005/06interviews
Nu
mb
er o
f o
ffen
ces
Domestic Burglary - 25% reduction from 1998-99 to 2005British Crime Survey
Source: Home Office
946,023
The aim is to reduce vehicle crime by 30% ...
2,941,927
-
500,000
1,000,000
1,500,000
2,000,000
2,500,000
3,000,000
3,500,000
2000 BCS(crime in 1999)
2001/02interviews
2002/03interviews
2003/04interviews
2004/05interviews
Vehicle Crime - 30% reduction from 1998-99 to 2004British Crime Survey
Nu
mb
er
of
off
en
ce
s
Source: Home Office
2,059,349
… and performance is ahead of trajectory
2,366,0702,486,314
2,941,927
-
500,000
1,000,000
1,500,000
2,000,000
2,500,000
3,000,000
3,500,000
2000 BCS(crime in 1999)
2001/02interviews
2002/03interviews
2003/04interviews
2004/05interviews
Vehicle Crime - 30% reduction from 1998-99 to 2004British Crime Survey
Nu
mb
er
of
off
en
ce
s
Source: Home Office
2,059,349
Number of Violent Crimes - British Crime Survey
2,781,0002,784,000
3,423,0003,665,000
-
500,000
1,000,000
1,500,000
2,000,000
2,500,000
3,000,000
3,500,000
4,000,000
1998 BCS 2000 BCS 2001/2 2002/3
Nu
mb
er o
f O
ffen
ces
(Crime in 1997) (Crime in 1999) Interviews InterviewsSource: Home Office
Total violent crime has fallen since 1997
But most people believe crime is rising
Beliefs about the change in the national crime rate 1996 to 2002/03
46%
30% 33%25%
30%38%
29%
29%34%
31%
34%
35%
0%
10%
20%
30%
40%
50%
60%
70%
80%
1996 1998 2000 2001 2001/02 2002/03
Pe
rcen
tag
e s
ayi
ng
cri
me
has
incr
eas
ed
Lot more Little moreSource: Home Office
Gun crime has increased - as has media coverage
-
2,000
4,000
6,000
8,000
10,000
12,000
1997/98 1998/99 1999/00 2000/01 2001/02
Nu
mb
er o
f C
rim
es
Total Recorded Crimes Involving Firearms (other than air weapons)
Source: Home Office
Not focussing on the right crime?Fear of anti social-behaviour
0%
5%
10%
15%
20%
25%
30%
35%
40%
1992 1994 1996 1998 2000 2001 2001/02 2002/03
Pe
rce
nta
ge
th
at
pe
rce
ive
dis
ord
er
to b
e a
've
ry' o
r 'f
air
ly' b
ig p
rob
lem
Vandalism and graffiti
Teenagers hanging
around on the streetsPeople using /
dealing drugs
Source: Home Office
Where there is less direct experience,
communications, media coverage and viral
marketing become more important
Waiting Lists: Media Impact
Media Impact score (exposure X slant) for
waiting lists
Rating on reducing waiting lists
4 week rolling data
-12000
-10000
-8000
-6000
-4000
-2000
0
2000
4000
6000
06-M
ar
10-A
pr
15-M
ay
19-J
un
24-J
ul
28-A
ug
02-O
ct
06-N
ov
11-D
ec
15-J
an
20-F
eb
26-M
ar
30-A
pr
04-J
un
09-J
ul
13-A
ug
17-S
ep
22-O
ct
26-N
ov
04-F
eb
11-M
ar
14-A
pr
20-M
ay
24-J
un
29-J
ul
02-S
ep
07-O
ct
11-N
ov
16-D
ec
-30
-20
-10
0
10
20
Disconnection between own
good(ish) experience and national
“political”debate?
Provision of Service Locally and Nationally
79%
65%
47%
31%
26%
-41%
-62%
17%
18%
Base: All respondents (1,031)
The government has the right policies for the NHS
% Agree
Q To what extent, if at all, do you agree or disagree with the following statements?
The NHS is providing a good service nationally
My local NHS is providing me with a good service
% Disagree
(-36)
(+16)
(+62)
I am satisfied with the way the Government is doing its job overall*
I was satisfied with my last visit to hospital
(-10)
(-47)
-31
9%
54%
26%
8%3%
Confidence in CJS - which is key?
Not very confident Fairly
confident
Very confident
In the area where you live
Q Overall, how confident are you about the way crime is dealt with…?
Not confident
at all
Don’t know
Base: 2,001 GB Feb-March 2003, aged 16+
44%
39%
12%2%
In England and Wales
Not very
confident
Fairly confident
Very confident 3%Not confident
at all
Don’t know
+29% net confident -4% net confident
Clash between local and national priorities for many
managers and local politicians
0
10
20
30
40
50
0 10 20 30 40 50 60
What are the issues affecting quality of life in local communities?
Most need improving locally
Important generally
Low level crime
Health servicesHousing
Activities for teenagers
Public transportRoad/pavement repairs
Shopping
Wages/cost of living
Low traffic
Job prospects
Facilities for young children
Sports/leisure facs
Clean streets
Education
Base: 2,031 British adults 15+, 18-22 October 2001
Culture
Community activities
Open spaces
Low pollution
Race relations Access to nature
Falling satisfaction with street scene - nearly everywhere
0
10
20
30
40
50
60
70
80
1988 1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002
Net satisfied with street cleaning
National MORI average
Leicester
BV Pilots
People’s Panel
SouthwarkCamden
PortsmouthChelmsford
Brent
Unintended consequences
of a targets culture
Performance measures: Coherence, Confusion and Unintended Consequences
Sound ‘silo’ ideas Practical collisions
Visible policing
Home Office vs CPS prosecution requirements
Visibility vs NCRS requirements
Data collection
Quality of evidence
Treatment times
Specialism resourcing
Clinical priorities vs fast treatment
Knock on effects
Inclusion vs 5 A-C gradesSocial inclusion
Excellence & enjoyment
GCSE A-C
E&E vs Key Stage SAT scores
Performance review: Continuous improvement?
*NB only the managers’ view, not necessarily the frontline or public
Real improvements seen Worthwhile despite the administrative burden*
Manager’s view
PlateauCentre’s view
Improvement
What allowance?
Something has to give
Key Points Real progress – but rarely smooth
– Cultural change takes time– Ministerial change – changing regimes, philosophies
Measurement matters – need to get in place from beginning at level you require – as Time Machines are not yet invented
Need to choose to measure and promise the right things
Need to be able to factor out exogenous factors in assessing interventions– Personality, politics, community
Multiple interventions make things much more difficult: unintended effects
Anticipate the tension between effective evaluation and the needs of delivery, and politics
Questions, Commentsemmet.o’[email protected]