Upload
others
View
5
Download
0
Embed Size (px)
Citation preview
Challenges for institutional performance measures
Responsible Data Science in health care
Nicolette de Keizer Dept Medical Informatics
Academic Medical Center, Amsterdam
2
Data science in health care
Data reuse: • Management information • Quality assurance • Research • Surveillance
3
Learning Health System
Data reuse: • Management information • Quality assurance • Research • Surveillance • Financial reimbursement
Quality registries in health care >150 registries in health care Aims: – Accountability
• Government • Insurance companies • Patients
– Quality assessment and improvement – Scientific research
Accountability
Accountability Is the data fair, accurate, transparent?
Accountability Is the data fair, accurate, transparent? Large consequences ….. – Loss of faith – Demotivation – Loss of budget
8
9
History and development
Founded in 1996 by and for intensivists
0
100000
200000
300000
400000
500000
600000
700000
800000
900000
1000000
0
10
20
30
40
50
60
70
80
90
100
1997199819992000200120022003200420052006200720082009201020112012201320142015
# admissions
# IC
Us
Quality assessment and improvement Benchmarking
Observed difference = Difference in quality of care Onverklaarde
verschillen Onverklaarde verschillen
Onverklaarde verschillen
Registratie verschillen
Patiënten kenmerken
Toeval
Kwaliteit van zorg
0,40
0,50
0,60
0,70
0,80
0,90
1,00
A B C D E F G H
indi
cato
r sco
re
ICU/institution
Variation
Observed diference
Unexplained differences
Onverklaarde verschillen
Onverklaarde verschillen
Patiënten kenmerken
Toeval
Kwaliteit van zorg
Case mix
Mortality 20% 17%
Age 68 57
Comorbidity 40% 5%
Variation
Observed difference
Unexplained differences
Onverklaarde verschillen
Case mix
Toeval
Kwaliteit van zorg
Unexplained differences
Registration, definition differences
Variation
Observed difference
Unexplained differences
Uncertainty
Quality of care
Unexplained differences
Unexplained differences
Case mix
Registration, definition differences
Mortality as quality indicator
14
Input (1st 24 hour) Output
ICU Hospital
Prognostic models: APACHE II en IV, SAPSII…
15
Standardized Mortality Ratio (SMR)
Expected mortality depends on prognostic model
Observed in-hospital mortality
Expected mortality
SMR=
Ranking institutions
A common procedure is to rank institutions by SMR, providing a league table The top/bottom 10% or 25% are sometimes labelled as excellent/poor performers
Marshall EC, Spiegelhalter DJ. BMJ 1998;316(7146):1701-4.
ICU ranking accurate?
Ranks of 40 Dutch ICUs (n=86,427) SMRs based on Apache II, SAPSII, MPM24II model Rank CIs computed by bootstrap sampling (10,000 replications) Excellent performance: with 95% certainty among top 25% institutes Very poor performance: with 95% certainty among bottom 25% institutes
Bakhshi-Raiez F et al. Crit Care Med 2007
0 10 20 30 40
215
251836223
1033408
1539141911132779
384
322
313537231729126
2016281
34243026
0 10 20 30 40
518141021258
367
3319222
39383
4030374
1527281229119
176
162431321
263413202335
0 10 20 30 40
18255
10403339363738192
2122157
1432172949
2338
1227306
3416112831351
13242620
Apache II SAPS II MPM24 II
Bak
hshi
-Rai
ez F
et a
l. C
rit C
are
Med
200
7
Results
20 ICU significantly differ in rank (2-19 positions) by 1 or more pair of models 3 ICUs rated as performance outlier by one model while others excluded this possibility with 95% certainty
215
251836223
1033408
1539141911132779
384
322
3135372317
518141021258
367
3319222
39383
4030374
1527281229119
176
18255
10403339363738192
2122157
1432172949
2338
1227306
Apache II SAPS II MPM24 II
215
251836223
1033408
1539141911132779
384
322
3135372317
518141021258
367
3319222
39383
4030374
1527281229119
176
18255
10403339363738192
2122157
1432172949
2338
1227306
Apache II SAPS II MPM24 II
215
251836223
1033408
1539141911132779
384
322
3135372317
518141021258
367
3319222
39383
4030374
1527281229119
176
18255
10403339363738192
2122157
1432172949
2338
1227306
Apache II SAPS II MPM24 II
0 10 20 30 40
81539141911132779
384
322
313537231729126
2016281
34243026
0 10 20 30 40
19222
39383
4030374
1527281229119
176
162431321
263413202335
0 10 20 30 40
192
2122157
1432172949
2338
1227306
3416112831351
13242620
Apache II SAPS II MPM24 II
Benchmark –SMR fair?
Benchmark on in-hospital mortality or long term mortality? Why choose for hospital mortality? – Sooner and easily available – Mortality not related to ICU admission
Why choose for longterm mortality? – More relevant for patients – Less influence by discharge policy
In-hospital mortality vs long term mortality
0 0,5 1 1,5 2
32
4
2
2
2
42
323
22
2
41925
2110
67
1311161824251217
32030412722281423263840153732
83519393634332942433144
SMR 3 months
0 0,5 1 1,5 2
2
3
2
34
4
2
2
3
2222
2
123456789
1011121314151617181920212223242526272829303132333435363738394041424344
SMR In-hospital
S. B
rinkm
an e
t al.
Inte
nsiv
e C
are
Med
. 201
3 N
ov;3
9(11
):192
5-31
26
Other Challenges regarding FACT
Observed difference
Unexplained differences
Uncertainty
Quality of care
Unexplained differences
Unexplained differences
Case mix
Registration, definition differences
27
Other Challenges regarding FACT The way of data collection influences SMR – Sample frequency – Coded data versus free text
Interpretation and definition of QI – Expl IGZ: Mean duration of mechanical ventilation
• Different types of ventilation • Duration in hours or calendar days • Mean based on all ICU patients or ventilated patients
28
Challenges Learning Health System
Formalisation of indicator definitions
Information models Terminological systems
29
Conclusion Performance measurement of health facilities might be biased by – Data source – Case mix correction model – Definitions (of endpoints) Need for methods to unambiguously capture health data, formalize indicators and make health data transparent for different reuse purposes