www.england.nhs.uk
Patient safety data or patient safety intelligence?
Dr Frances HealeyRGN, RMN, PhDHead of Patient Safety Insight NHS England
6 July 2015
www.england.nhs.uk
1st rule of #statisticsclub
Qualitative data are at least equally important, and probably much more important, than quantitative data…..
2
….. but it’s quantitative data that have the pitfalls & perils, so that is my focus today
www.england.nhs.uk 3
www.england.nhs.uk 4
www.england.nhs.uk 5
Dammit Jim, I’m a doctor, not a rocket
scientist
Dammit Frances, I’m a nurse, not statistician
www.england.nhs.uk
2nd rule of #statisticsclubYou are probably much less logical than you think you are
6
www.england.nhs.uk
http://blogs.bmj.com/bmj/2014/05/09/tara-lamont-on-failing-well-archie-cochranes-legacy/
@TaraJLamont
Archie Cochrane
77
www.england.nhs.uk 8www.england.nhs.uk
“The results at that stage showed a slight numerical advantage for those who had been treated at home. It was of course completely insignificant statistically.
“I rather wickedly compiled two reports, one reversing the numbers of deaths on the two sides of the trial. As we were going into committee, in the anteroom, I showed some cardiologists the results……..
www.england.nhs.uk 9
“……they were vociferous in their abuse: `Archie’, they said, `we always thought you were unethical. You must stop the trial at once…’
“I let them have their say for some time and then apologised and gave them the true results, challenging them to say, as vehemently, that coronary care units should be stopped immediately.
“There was dead silence and I felt rather sick because they were, after all, my medical colleagues.”
Professor Archibald Cochrane & Max Blythe One Man's Medicine (1989) p.211
www.england.nhs.uk
Cognitive dissonance
10
• We have a strong need for our personal beliefs and our personal actions to chime
• The discomfort we feel when they don’t is ‘cognitive dissonance’
http://britishgeriatricssociety.wordpress.com/2013/05/16/all-down-to-numbers/
• If we believe we are part of effective, motivated, caring teams, it is very hard to also simultaneously believe:o We haven’t achieved real
improvements in safety o We might be less safe than
peers
www.england.nhs.uk 11
http://www.health.org.uk/multimedia/slideshow/hard-data-soft-intelligence/
Data (mis)used for reassurance
www.england.nhs.uk
Which is not to say we shouldn’t use data for motivation
“The consistent delivery of well-executed safe care under typically difficult circumstances tends to go unrecognised"
www.england.nhs.uk 13
www.england.nhs.uk
3rd rule of #statisticsclubOne size does not fit all - there is no such thing as a good indicator, or a good data source, just one that is good in particular situation for a particular purpose
14
Oct - Dec 2003
Jan -
Mar 2004
Apr -
Jun 2004
Jul - Sep 2004
Oct - Dec 2004
Jan -
Mar 2005
Apr -
Jun 2005
Jul - Sep 2005
Oct - Dec 2005
Jan -
Mar 2006
Apr -
Jun 2006
Jul - Sep 2006
Oct - Dec 2006
Jan -
Mar 2007
Apr -
Jun 2007
Jul - Sep 2007
Oct - Dec 2007
Jan -
Mar 2008
Apr -
Jun 2008
Jul - Sep 2008
Oct - Dec 2008
Jan -
Mar 2009
Apr -
Jun 2009
Jul - Sep 2009
Oct - Dec 2009
Jan -
Mar 2010
Apr -
Jun 2010
Jul - Sep 2010
Oct - Dec 2010
Jan -
Mar 2011
Apr -
Jun 2011
Jul - Sep 2011
Oct - Dec 2011
Jan -
Mar 2012
Apr -
Jun 2012
Jul - Sep 2012
Oct - Dec 2012
Jan -
Mar 2013
Apr -
Jun 2013
Jul - Sep 2013
Oct - Dec 2013
Jan -
Mar 2014
Apr -
Jun 2014
Jul - Sep 2014
Oct - Dec 2014
0
50000
100000
150000
200000
250000
300000
350000
400000
450000
500000
Incid
ents
Sub
mitt
ed
www.england.nhs.uk 15
CULTURE indicators
STRUCTURE indicators
PROCESS indicators
planning process indicators
delivery process indicators
OUTCOME indicatorsAre we
safe today?
Types of indicator
www.england.nhs.uk
Our high tech interactive facility
16
1.2.3.4.5.
www.england.nhs.uk
What type of indicator?
17
1. CULTURE indicator
2. STRUCTURE indicator
3. PROCESS - planning process indicator
4. PROCESS - delivery process indicator
5. OUTCOME indicator
97% of patients who need a pressure reliving mattress received it within four hours
www.england.nhs.uk
What type of indicator?
18
1. CULTURE indicator
2. STRUCTURE indicator
3. PROCESS - planning process indicator
4. PROCESS - delivery process indicator
5. OUTCOME indicator
86% of nurses agree that most pressure ulcers can be prevented
www.england.nhs.uk
What type of indicator?
19
1. CULTURE indicator
2. STRUCTURE indicator
3. PROCESS - planning process indicator
4. PROCESS - delivery process indicator
5. OUTCOME indicator
We have 42 pressure relieving mattresses per 100 beds
www.england.nhs.uk
4th rule of #statisticsclubWe don’t do structural measurement nearly often enough
20
30%
9%
26%
35%
on all wards
on most wards
on one or some wards
not on any wards
“This [MH unit for older people] has no physio input. Balance and strength assessments never get done”
“We cannot put walking frames within reach as there is no room left once you have a chair beside the bed”
Royal College of Physicians 2012 Report of the 2011 inpatient falls pilot audit www.rcplondon.ac.uk
Weekend access to mobility aids for new patients
www.england.nhs.uk
Purposes of safety measurement
21
One often used model:measurement for researchmeasurement for judgement measurement for improvement
Alternative less prone to misunderstanding measurement to understand prioritiesmeasurement to see how we compare to othersmeasurement to see if we’re getting better (or worse)
www.england.nhs.uk
0-4
5-9
10-1
4
15-1
9
20-2
4
25-2
9
30-3
4
35-3
9
40-4
4
45-4
9
50-5
4
55-5
9
60-6
4
65-6
9
70-7
4
75-7
9
80-8
4
85-8
9
90-9
4
95-9
9
100+
0%
2%
4%
6%
8%
10%
12%
14%
16%
18%
Breakdown by age of falls in acute clusters
Age group
% of all repor-ted acute falls
00 (12 AM
- Midnight)
01 (1
AM)
02 (2
AM)
03 (3
AM)
04 (4
AM)
05 (5
AM)
06 (6
AM)
07 (7
AM)
08 (8
AM)
09 (9
AM)
10 (10 AM
)
11 (11 AM
)
12 (12 PM
- Midday)
13 (1
PM)
14 (2
PM)
15 (3
PM)
16 (4
PM)
17 (5
PM)
18 (6
PM)
19 (7
PM)
20 (8
PM)
21 (9
PM)
22 (10 PM
)
23 (11 PM
)
0%
1%
2%
3%
4%
5%
6%
Falls incidents by hour of occurrence, for acute clusters
Hour
% of all reported acute falls
Understanding priority areas
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Acute hospitals Community hospitals Mental health units
Location of incident
Per c
ent o
f sam
ple
Apparently unwitnessed by staff
Witnessed by staff
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Acute hospitals Community hospitals Mental health units
Location of incident
Per c
ent o
f sam
ple
Apparently unwitnessed by staff
Witnessed by staff
0%
5%
10%
15%
20%
25%
30%
35%
Acute hospitals Community hospitals Mental health units
Whilst walking
From beds
Circumstances unclear
From chairs
From toilet or commode
Other
0%
5%
10%
15%
20%
25%
30%
35%
Acute hospitals Community hospitals Mental health units
Whilst walking
From beds
Circumstances unclear
From chairs
From toilet or commode
Other
www.england.nhs.uk
5th rule of #statisticsclubMeasurement to see how we compare to others: when it comes to comparing outcomes, case mix really matters
23
(but case mix shouldn't be a problem for well-designed process measures)
www.england.nhs.uk
Safety outcomes & case mix
0%
5%
10%
15%
20%
25%
0-4 5-9 10-14 15-19 20-24 25-29 30-34 35-39 40-44 45-49 50-54 55-59 60-64 65-69 70-74 75-79 80-84 85-89 90-94 95-99 100+
Age of patient (years)
Pe
r c
en
t
Per cent of totalbed daysPer cent of totalfalls
0%
5%
10%
15%
20%
25%
0-4 5-9 10-14 15-19 20-24 25-29 30-34 35-39 40-44 45-49 50-54 55-59 60-64 65-69 70-74 75-79 80-84 85-89 90-94 95-99 100+
Age of patient (years)
Pe
r c
en
t
Per cent of totalbed daysPer cent of totalfalls
85 years +
Deandra S et al. Arch Gerontol Geriatr 56 (2013) 407–415
NPSA Slips trips and falls in hospital data update NPSA 2010
Risk factors for hospital falls Odds Ratio
History of falls 2.85 (1.14–7.15)
Cognitive impairment 1.52 (1.18–1.94)
www.england.nhs.uk
Older people are not evenly distributed
25
Therefore unadjusted higher/lower rates of falls compared to other trusts with very different age profiles are highly unlikely to be useful indicators of relative safety
All ages falls rate
0.01.02.03.04.05.06.07.08.09.0
RETIREM
ENT TO
WN B
RETIREM
ENT TO
WN D
RETIREM
ENT TO
WN A
RETIREM
ENT TO
WN C
URBAN TEACHIN
G B
URBAN TEACHIN
G C
URBAN TEACHIN
G D
URBAN TEACHIN
G A
falls
per
1,0
00 b
ed d
ays
www.england.nhs.uk
50-fold differences between wards
26Royal College of Physicians 2011 The FallSafe Quality Improvement project: report for the Health Foundation
Therefore higher/lower rates of falls compared to other wards in the same trust highly unlikely to be useful local indicators of safety
www.england.nhs.uk
Comparable and non-comparable safety indicators
www.england.nhs.uk 28
http://britishgeriatricssociety.wordpress.com/2013/12/19/fallsafe-are-culture-clashes-good-for-us/
0
5000
10000
15000
20000
25000
6th rule of #statisticsclub
Numbers don’t know if they are in research study or a QI project
www.england.nhs.uk
7th rule of #statisticsclubWhen it comes to measuring if we have got better (or worse) size matters
(and we usually have plentiful processes to measure but far fewer outcomes)
www.england.nhs.uk 30
www.england.nhs.uk
Sample safety outcome indicators: scaled to ward-level*
IF these safety outcomes were distributed evenly across acute wards, an average ward would have around:
• One case of c difficile per year
• One MRSA bloodstream infection per decade
• One new pressure ulcer per quarter
• One fall with minor injury per month
• One fall with hip fracture every five years
* Approximations based on c. 5,000 acute/rehabilitation hospital wards in England, PHE trust attributed/trust-assigned HCAI data, NRLS reported falls, assumption that acute ‘new’ p ulcer prevalence as measured by ST represents about 4 x acute p ulcer incidence
www.england.nhs.uk
What scale and time would give you a reasonable chance of being able to distinguish a 25% improvement from natural variation ?
32
For falls with injury
1. One ward two years
2. Ten wards two years
3. One medium sized hospital two years
4. Five hospitals two years
5. Fifty hospitals two years
www.england.nhs.uk
What scale and time would give you a reasonable chance of being able to distinguish a 25% improvement from natural variation ?
33
For hospital-associated MRSA?
1. One ward two years
2. Ten wards two years
3. One medium sized hospital two years
4. Five hospitals two years
5. Fifty hospitals two years
www.england.nhs.uk 34
Collaboration is not just good for learning,
it increases sample size
www.england.nhs.uk
8th rule of #statisticsclubYour data don’t have to be perfect to be good enough – but you do need to know how imperfect they are
35
www.england.nhs.uk
0
2
4
6
8
10
12
14
16
60% certain last fall was
reported
77% certain last fall was
reported
0
2
4
6
8
10
12
14
16
0
2
4
6
8
10
12
14
16
www.england.nhs.uk
9th rule of #statisticsclubIf it looks too good to be true, it probably is!
37
Jan
Feb
Mar
Apr
May Jun
Aug Oct
Nov Jan
Oct
Dec
Aug
Sep Oct
Nov
Dec Jan
Feb
Mar
Apr
May Jun
Jul
Aug
Sep Oct
Nov
Dec
2008 2009 2010 2011
0
10
20
30
40
50
60
70
80
0%
1%
2%
3%
4%
5%
6%
7%
8%
9%
10%
Falls
www.england.nhs.uk
10th rule of #statisticsclubDon’t try to measure too much
38
www.england.nhs.uk
www.england.nhs.uk
www.england.nhs.uk
www.england.nhs.uk
85%
15%
time spent improvingtime spent measuring
www.england.nhs.uk
Thank you for listening
@FrancesHealey
Please tweetfurther #statisticsclub rules of your own