Embed Size (px)
DESCRIPTION
PHREE 2014 Spring Conference Integration and innovation – meeting the challenges of evaluation in the new system
Citation preview
© Nuffield Trust9 April 2023
Evaluating service innovations for older people
Integration and innovation – meeting the challenges of evaluation in the new system
Martin BardsleyNuffield Trust
© Nuffield Trust
• Promote independent analysis and informed debate on healthcare policy across the UK
• Charitable organization founded in 1940
• Formerly a grant-giving organization
• Since 2008 we have been conducting in-house research and policy analysis
• Significant interest in uses of predictive risk techniques
The Nuffield Trust
William Morris1st Viscount Nuffield
(1877 -1963)
© Nuffield Trust
Predictive risk modelling
Resource allocation
Descriptive studies Evaluations
Integrated care pilots
nuffield trust
Nuffield Trust Research team – data linkage projects
Risk sharing for CCGs
nuffield trust
Combined predictive model
nuffield trust
Person based resource allocation
nuffield trust
Social care at end of life
nuffield trust
Cancer and social care
nuffield trust
Predicting social care costs
nuffield trust
Virtual Wards
nuffield trust
WSD
nuffield trust
Marie Curie Nursing Service
nuffield trust
© Nuffield Trust
Aims
Background
Exploiting routine information
2 case studies of retrospective evaluations
a. Marie Curie Nursing service
b. Partnerships for Older People
© Nuffield Trust
Ten-year trend in emergency admissions (46 million admits)
20
01
/02
Q1
20
01
/02
Q3
20
02
/03
Q1
20
02
/03
Q3
20
03
/04
Q1
20
03
/04
Q3
20
04
/05
Q1
20
04
/05
Q3
20
05
/06
Q1
20
05
/06
Q3
20
06
/07
Q1
20
06
/07
Q3
20
07
/08
Q1
20
07
/08
Q3
20
08
/09
Q1
20
08
/09
Q3
20
09
/10
Q1
20
09
/10
Q3
20
10
/11
Q1
20
10
/11
Q3
0.00
200,000.00
400,000.00
600,000.00
800,000.00
1,000,000.00
1,200,000.00
1,400,000.00
No ACS diagnosis ACS primary diagnosis ACS secondary diagnosis
Nu
mb
er
of
em
erg
en
cy
ad
mis
sio
ns
(m
il-
lio
ns
)
+35% (40%)
+34%
© Nuffield Trust
By ambulatory care sensitive conditions…
© Nuffield Trust
Interventions to reduce avoidable admissions
Primary Care ED Depts Hospital Transition
Practice features Assess/obs wards Structured Discharge
Transition care management
Medication review GPs in A&E Medication Review
Rehabilitation
Case management
Senior Clinician Review
Specialist Clinics Self management and education
Telemedicine Coordination EOL care
Hospital at home
Virtual Wards
see Purdy et al (2012) Interventions to Reduce Unplanned Hospital Admission: A series of systematic reviews. Bristol University Final Report)
© Nuffield Trust
Why the current interest in integrated care?
• Rising levels of chronic disease• Ageing population • Increasing levels of hospital admissions and readmissions,
especially among the elderly and vulnerable, and children• Economic hard times, and unsustainable health and social
care economies• And too often we still do not get it right in terms of care co-
ordination, care planning, communication with families• Interest in prevent solutions that reduce the need for hospital
admissions
© Nuffield Trust
Integration
Sara Shaw, Rebecca Rosen and Benedict Rumbold What is integrated care? An overview of
integrated care in the NHS Research report. Nuffield Trust June 2011
© Nuffield Trust
What information do we have on whether these are working?……
© Nuffield Trust
© Nuffield Trust
Data are everywhere…
GP
Local Authority
Commissioner
A&E
OP
IP
PharmacyCommunity
Health Services
Up there
HousingCouncil
Tax
Council Social
Services
Social care provider
Ambulance ControlNHS Direct
Commissioning data ...
© Nuffield Trust
Exploiting person level data
Linking data
a. over time to look at what happens to people – not just events
b. across care providers to build broader picture
Person level
Capture services provided ->costs; quality
Descriptions of health -> outcomes
© Nuffield Trust
Linkage not new
The Oxford Record Linkage Study: A Review of the Method with some Preliminary Results by E D Acheson DM MRCP and J G Evans MB MRCP (Nuffield Department of Clinical Medicine, Oxford University) Proc R Soc Med. 1964 April; 57(4): 269–274.
© Nuffield Trust
Tomb raiders?
© Nuffield Trust
Information flows
Accident and emergency 350,000 records
Outpatients1,680,000 records
Inpatients360,000 records
Social care240,000 records
Community matrons20,000 records
GPs60 practices48.5 million records
Relative size of data sets collectedFor one PCT area (WSD project)
March 2011
© Nuffield Trust
Health and social care timeline – an individual’s history
© Nuffield Trust
Data linkage Social & secondary care interface
© Nuffield Trust
Final year costs: by age
<55 55-64 65-74 75-84 85-94 >=950
2,000
4,000
6,000
8,000
10,000
12,000
14,000
Female
All costs
Hospital costs
Social care costs
Age band
Est
imate
d a
vera
ge c
ost
s p
er
deced
en
t, £
One person hospital cost profile over a year50+ year old male, total annual cost > £35,000
Outpatients DayCase Elective AE Nonelective
Time (weeks)
© Nuffield Trust
Used of linked person level data
Audit and Quality Improvement
Patient safety (e.g. monitoring drug side effects or surgical mortality rates)
Public Health programmes (immunisation; monitoring cancer rates)
Evaluate Services (are they effective and cost effective?)
Planning services (e.g. ICU bed availability; pandemic flu plans; manage changing patterns of demand)
Manage Performance (e.g. readmission targets; health outcomes indicators)
Resource allocation
Research
Why rely on using existing data for research?
Advantage Disadvantage
• Descriptors of events and health status
• Constrained by the data that are collected – and quality/consistency of coding
• Volume of cases versus costs of data collection
• Handling sensitive personal information (+/- consent)
• Comprehensive coverage • Coverage of the data – unknown unknowns
• Enables retrospective studies/ not time sensitive
• Volume of data – complex processing
© Nuffield Trust
Example (1)
Impact of Marie Curie Nursing Service on place of death &
hospital use at the end of life
http://www.nuffieldtrust.org.uk/publications/marie-curie-nursing
Chitnis, X. , Georghiou, T., Steventon, A. & Bardsley, M. J. (2013). Effect of a home-based end-of-life nursing service on hospital use at the end of life and place of death: a study using administrative data and matched controls. BMJ Supportive & Palliative Care, 1–9. doi:10.1136/bmjspcare-2012-000424
© Nuffield Trust
© Nuffield Trust
Methods
• 29,538 people who received MCNS care from January 2009 to November 2011
• Sophisticated matching techniques used to select 29,538 individually matched controls from those who died in England from January 2009 – November 2011
• Matched on demographic, clinical and prior hospital use variables
• People started receiving MCNS care on average 8 days before death
© Nuffield Trust
Evaluation: The Marie Curie Nursing Service
Intervention:
• Nursing care support to people at end of life, in their homes
Nuffield commissioned to evaluate impact:• Are recipients more likely to die at home?• Reduction in emergency hospital admissions at end of life?
Methods:• Retrospective matched control study – use of already existing
administrative data
© Nuffield Trust
Matched control studies – broad aim
>1M individuals - died Jan 2009 to Nov 2011, did not receive service
(everyone else)
Aim to find 30,000 individuals who match almost exactly on a broad range of characteristics
Use this group as study control group
30,000 individuals - died Jan 2009 to Nov 2011 & received Marie Curie nursing service before death
© Nuffield Trust
Final datasets available for analysis
Nuffield trust
ONS deaths Hospital inpatient, outpatient, AEMC data - desensitised
N = 30,000
• person details• dates of
service • type of service
Identifiers:Names, DOB,Addresses, etc
• dates & place
of death for all people in England,
• associated hospital (HES) records
Identifiers:Nuffield Trust specific HESID
© Nuffield Trust
0%
10%
20%
30%
40%
50%Comorbidities
0%
5%
10%
15%
20%
25%
30%
35%
Cancer diagnoses
Control group – how well matched? Diagnostic history
0%
10%
20%
30%
40%
50%Comorbidities
0%
5%
10%
15%
20%
25%
30%
35%
Cancer diagnoses
Marie Curie Controls
© Nuffield Trust
Results - Proportion of people dying at home
• 77% of MCNS patients died at home but only 35% of controls
• Impact of MCNS care on home deaths greater for those with no history of cancer then for those with cancer
Figure 2 – Place of death for Marie Curie Nursing Service patients & matched controls
© Nuffield Trust
Emergency admissions for cases where nursing started 3-7 days before death
© Nuffield Trust
Emergency admissions for cases where nursing started 8-14 days before death
© Nuffield Trust
Impact of MCNS care on hospital costs
Table 1 – Post index date hospital costs for Marie Curie cases and matched controls
Mean (sd) hospital costs per person
Activity Type Marie Curie cases Matched controls DifferenceEmergency admissions £463 (£1,758) £1,293 (£2,531) £830
Elective admissions £106 (£961) £350 (£1,736) £244Outpatient attendances £33 (£212) £76 (£340) £43
A&E attendances £9 (£34) £31 (£60) £22
All hospital activity £610 (£2,172) £1,750 (£3,377) £1,140
• Significantly greater reduction in costs among those with no recent history of cancer
• Also cost reduction much greater for those who started receiving MCNS care earlier (£2,200 for those >2 weeks before death)
© Nuffield Trust
Summary
• Evaluation of large-scale, existing palliative care service using well-matched controls
• Caveats – not all costs considered; unobserved differences about MCNS users
• Those who received home-based palliative care:
• Much more likely to die at home
• Lower use of hospital care (particularly unplanned)
• Lower hospital costs
• Impact of MCNS care greater for those without cancer – surprising finding, although literature limited
Example (2)
Evaluation of Community Based Interventions impact on hospital admissions
Retrospective evaluation using matched controls
Adam Steventon, Martin Bardsley, John Billings, Theo Georghiou and Geraint Lewis An evaluation of the impact of community-based interventions on hospital use. A case study of eight Partnership for Older People Projects (POPP) . Nuffield Trust March 2011
© Nuffield Trust
© Nuffield Trust
The Partnership for Older People Projects (POPPs)
“We recommend expanding the Partnerships for Older People Projects (POPPs) approach to prevention across all local authorities and PCTs.”
• £60m investment by DH with aim to:
“shift resources and culture away from institutional and hospital-
based crisis care”
• 146 interventions piloted in 29 sites.
• National evaluation of whole programme found £1.20 saving in bed days per £1 spent.
© Nuffield Trust
From the 146 interventions offered under POPP, we selected 8 for an in-depth study of hospital use
Support workers for community matrons
Intermediate care service with generic workers
Integrated health and social care teams
Out-of-hours and daytime response service
+ 4 different short term assessment and signposting services
© Nuffield Trust
Our preferred option for this evaluation: link participants to HES through a trusted third party
March 2011
Collate files and add NHS numbers
Derive HES ID
Collate patient lists
Patient identifiers (e.g. NHS number)
Trial information (e.g. start and end date)
Non-patient identifiable keys (e.g. HES ID, pseudonymised NHS number)
Participating sites
Information Centre
Nuffield Trust
© Nuffield Trust
Prevalence of health diagnoses categories in intervention and control groups
Hyperte
nsion
Inju
ryFalls
Atrial fi
brillatio
n and flu
tter
Isch
emic
heart dise
ase
Cancer
Diabete
s
Menta
l health
Congestive
heart
failu
re
COPD
Cerebro
vasc
ular d
isease
Angina
Anemia
Renal failu
re0%
10%
20%
30%
40%
50%
60%
Control Intervention
© Nuffield Trust
Overcoming regression to the mean using a control group
March 2011
-12 -11 -10 -9 -8 -7 -6 -5 -4 -3 -2 -1 1 2 3 4 5 6 7 8 9 10 11 120.0
0.1
0.2
0.3
Intervention
Month
Num
ber o
f em
erge
ncy
hosp
ital a
dmiss
ions
per
he
ad p
er m
onth
Start of intervention
© Nuffield Trust
Overcoming regression to the mean using a control group
March 2011
-12 -11 -10 -9 -8 -7 -6 -5 -4 -3 -2 -1 1 2 3 4 5 6 7 8 9 10 11 120.0
0.1
0.2
0.3
Intervention
Month
Num
ber o
f em
erge
ncy
hosp
ital a
dmiss
ions
per
he
ad p
er m
onth
Start of intervention
© Nuffield Trust
Overcoming regression to the mean using a control group
March 2011
-12 -11 -10 -9 -8 -7 -6 -5 -4 -3 -2 -1 1 2 3 4 5 6 7 8 9 10 11 120.0
0.1
0.2
0.3
Intervention
Month
Num
ber o
f em
erge
ncy
hosp
ital a
dmiss
ions
per
he
ad p
er m
onth
Start of intervention
© Nuffield Trust
Overcoming regression to the mean using a control group
March 2011
-12 -11 -10 -9 -8 -7 -6 -5 -4 -3 -2 -1 1 2 3 4 5 6 7 8 9 10 11 120.0
0.1
0.2
0.3
Control Intervention
Month
Num
ber o
f em
erge
ncy
hosp
ital a
dmiss
ions
per
he
ad p
er m
onth
Start of intervention
© Nuffield Trust
Impact of eight different interventions on hospital use
© Nuffield Trust
Conclusions
• Able to undertake a retrospective evaluation of changes in hospital use for 8 projects, over 5000 subjects
• Study took less than 3 months once permissions obtained
• Findings suggest that none of these projects were delivering the anticipated reduction in hospital use
• The approach has limitations eg there is always the risk of unmeasured confounders; end points limited by the data available.
• The ability to track individual histories using existing data sets has great strengths and wider application
© Nuffield Trust
Findings from other studies study
March 2011 © Nuffield Trust
© Nuffield Trust
And for 3 virtual wards…
© Nuffield Trust
And 11 integrated care pilots (all pilots combined n=11,296)
• Elective admissions & outpatient attendances reduced more quickly for intervention patients than matched controls.
• However, emergency admissions appeared to have increased more quickly.
Difference in difference analysis(individual patient level)
Absolute difference (per head)
Relative difference
p-value
Emergency admissions 0.02 +2 % 0.03
A&E attendance -0.01 -1% 0.26
Elective admissions -0.04 -4% 0.003
Outpatient attendance -0.20 -20% <0.001 *
* Difference also detected at practice level
© Nuffield Trust
9 observations
1. Recognise that planning and implementing large scale service changes take time
2. Define the service intervention clearly including what it is meant to achieve and how, and manage
implementation well
3. Be explicit about how the desired outcomes are supposed to arise and use interim markers of
success
4. Consider generalisability and context: they are important
5. If you want to demonstrate statistically significant change, size and time matter
6. Hospital use and costs are not the only impact measures
7. Pay attention to the process of implementation as well as outcome
8. Carefully consider the best models for evaluation
9. Work with what you have: organisation and structural change may not achieve desired outcomes
© Nuffield Trust
Summary
• Emergency admissions and urgent care seen as critical drives of need for new
services
• Many different initiatives aimed at integrating across primary/secondary care
divide – often with explicit aims to reduce emergency admissions
• Huge potential in exploiting linked data sets for retrospective evaluation of new
models of care
• Evaluation of many integrated care initiatives suggest reducing emergency
admission is very difficult – though they may have other benefits
• Some evidence that a well established programme for end of life care does reduce
need for hospital care
© Nuffield Trust9 April 2023
www.nuffieldtrust.org.uk
Sign-up for our newsletterwww.nuffieldtrust.org.uk/newsletter
Follow us on Twitter:Twitter.com/NuffieldTrust
© Nuffield Trust
