Upload
donhi
View
220
Download
0
Embed Size (px)
Citation preview
Replication analysis
of the validity and comparability
of Patient Safety Indicators (PSI):
the impact of AHRQ exclusions
by Vladimir Stevanovic and Lihan Wei
The OECD HCQI Expert Group Meeting
Paris, 3 June 2010
Background HCQI Expert Group meeting in June 2009
• Concerns were raised that the data may reflect more
coding and registration practices than actual differences
in patient safety
• Several countries expressed reservations about the
publication of PSIs in Health at a Glance 2009 due to
perceived risk of misinterpretation
Background PSI Subgroup meeting in October 2009
• The Secretariat presented a technical analysis on
the impact of the AHRQ exclusions
• The findings implied that the exclusions may have
varying impacts across all indicators apart from the
obstetric ones
• It was proposed further analysis to be undertaken
through a voluntary subsample of countries
Objectives
To improve the comparability of PSIs by:
• Assessing the impact of AHRQ exclusions on the
PSI rates, and
• Exploring whether these exclusions account for any
undesired or increased variation across countries
The scope of this analysis did NOT include exclusions
that are inherent to the concept of an indicator by
their nature (e.g. children, pregnant women)
Patient safety indicators
• PSI05 - Foreign body left in during procedure
• PSI07 - Catheter-related bloodstream infection
• PSI12 - Post-operative pulmonary embolism (PE)
or deep vein thrombosis (DVT)
• PSI13 - Postoperative sepsis
• PSI15 - Accidental puncture or laceration
Countries
• Canada*
• Denmark
• Finland
• Israel
• New Zealand+
• Norway
• Singapore
• Spain
• Sweden
• Switzerland
• United States*
* both POA and non-POA data + previous analysis
Methods
The assessment of impact of each individual
exclusion in the AHRQ algorithm
• Rate ratio of the difference between post- and
pre-exclusion rate and the pre-exclusion rate
• Allows the code criteria to be met at any dg field
• Negative value = the rate-lowering effect
Positive value = the rate-increasing effect
Example PSI05 LOS exclusion
PreExcl
88 / 865,955 = 0.0102
LOS 0 LOS 1+
Excl
27 / 321,953 = 0.0084
PostExcl
61 / 544,002 = 0.0112
Impact = (PostExcl - PreExcl) / PreExcl = (0.0112 - 0.0102) / 0.0102 * 100= +11%
Example PSI05 LOS exclusion
yes
no Excludeno
yes
Add case to
denominator
population
Excludeyes
Count denominator
population and report to
OECD secretariat
Add case to
numerator
population
yes
Case is a surgical or amedical discharge
Case is
assigned to MDC 14 or the PDx
is listed in table M3?
PDx is
identical to the numerator
definition?
no
SDx is
identical to the numerator
definition?Count numerator
population and report to
OECD secretariat
Foreign body left in during procedure
Age=18 y or>18 y?
LOS < 24hours or 0
days?
no
yes Exclude
the impact of
each individual
exclusion
Data collection
Results
• The results indicate that the exclusions within the
AHRQ algorithms have varying impact
• While clinical exclusions are considered inherent
to the construction of PSIs, length of stay and
non-elective admission type exclusions are
believed to introduce bias
PSI12 DVT/PE – length of stay <2 days
29.7
49.5
74.2
30.9
110.3 111.4
19.8
175.7
19.7
66.7
0
20
40
60
80
100
120
140
160
180
200
% im
pact
PSI12 DVT/PE – NZL
2
14 13 13
22
24
22
29
0
5
10
15
20
25
30
35
0 1 2 3 4 5 6 7
Eve
nts
(n
um
)
LOS (days)
LOS excl.
PSI12 DVT/PE – NZL
Initial rate
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0 1 2 3 4 5 6 7
LO
S e
xc
lus
ion
ra
te
LOS (days)
rate
increasing
effect
rate
lowering
effect
PSI13 Postop.sepsis – acute adm.types
-63.0
-17.7
-47.8
-69.4 -67.5 -69.7
-90.9
-54.8
123.6
-36.9 -37.8
-100
-50
0
50
100
150
% im
pact
Non-elective admission types
PSI13 Postop.sepsis – SPA & SWE
Non-elective
admission
Exclusion
rate
Impact
SPAIN 39.5% 1.91 -55%
SWEDEN 62.5% 0.28 124%
PSI13 Postop.sepsis – initial vs. final rate
0.0000
0.0020
0.0040
0.0060
0.0080
0.0100
0.0120
SPA SWE
Initial
Final
Results – cont.
• As a result of multiple exclusions, some indicators
are calculated from very small samples and are
therefore especially sensitive to variations
PSI13 Postoperative sepsis
Numerator sample sizes
Initial 2dx Final sample size
Canada 3,421 479 14.0
Denmark 2,524 78 3.1
Finland 698 92 13.2
Israel 1273 54 4.2
New Zealand 1,141 91 8.0
Norway 930 25 2.7
Singapore 2,865 10 8.0
Spain 12,794 1,038 8.1
Sweden 3,555 1,449 40.8
United States 86,892 4,894 5.6
Total 116,093 8,210 7.1
-100.0
-50.0
0.0
50.0
100.0
150.0
200.0
250.0
300.0
% im
pact
Non-elective admission types Length of stay <4 days
0.0000
1.0000
2.0000
3.0000
4.0000
5.0000
6.0000
rate
per
100
dis
ch
arg
es
Results – cont.
• Comparisons between the patient and discharge
level data show relatively consistent ratios for the
PSI05 - Foreign body left in during procedure
PSI15 - Accidental puncture or laceration
across the following rates:
- patient (qualifying event in any dg field based on patient level data),
- discharge (qualifying event in any dg field based on discharge data)
- discharge sdx (qualifying event in secondary dg field based on
discharge data as calculated from the AHRQ algorithms)
PSI05 Foreign body left in during proc
0.00000
0.00002
0.00004
0.00006
0.00008
0.00010
0.00012
0.00014
0.00016
0.00018
Canada Denmark Finland Israel New Zealand
Singapore Spain Sweden United States
patient discharge discharge sdx
PSI15 Accidental puncture or laceration
0.0000
0.0010
0.0020
0.0030
0.0040
0.0050
0.0060
0.0070
Canada Denmark Finland Israel New Zealand
Spain Sweden United States
patient discharge discharge sdx
Present on admission flag
• Data provided by Canada and the United States
show considerable differences in terms of percent
change due to POA coding for PSI12 DVT/PE and
PSI13 Postoperative sepsis
• The effect is larger in general for the US than
Canada across PSIs
Canada – POA vs. non-POA
United States – POA vs. non-POA
PSI Subgroup’s recommendations
Length of stay
• The AHRQ algorithms are built on several exclusions that are
intended to affect the bias in comparison across US states
• Patient safety indicators with short length of stay (< 24h)
exclusion have an effect of reducing bias, while longer length
of stay exclusions tend to increase bias
Recommendation 1:
Collecting data for events by day breakdowns for length
of stay may give greater understanding of the effect of this
exclusion and inform possible revisions
PSI Subgroup’s recommendations
Non-elective admission type
• The post-operative sepsis indicator shows large and varying
effects from the non-elective admission type exclusion
• This is due to varying definitions and coding practices across
countries
Recommendation 2:
Collect additional information to understand how acute and
elective admissions are defined in each country and consider
possible revision of this exclusion
PSI Subgroup’s recommendations
Patient level data
• Patient safety indicators rely heavily on the quality of principal
and secondary diagnoses data coded in hospital records
• Discharge level data does not provide the necessary information
to detect under-reporting, hence those countries with UPI may be
able to provide additional data for patient-level events
Recommendation 3:
Collect additional information on qualifying events in PDX
field only based on UPI
PSI Subgroup’s recommendations
Coding practice
• Coding may be performed by various healthcare professionals
or dedicated clinical coders
• Clinical coding practice could be also affected if the medical
records determine financial reimbursement
Recommendation 4:
Collect additional information on coding and registry
practices in each country
Members of the HCQI Expert Group are invited to:
• Comment on the findings of this analysis
• Decide on whether a limited number of PSIs
- PSI05 Foreign body left in during procedure
- PSI15 Accidental puncture or laceration
- PSI18 Obstetric trauma due to vaginal delivery with instrument
- PSI19 Obstetric trauma due to vaginal delivery without instrument
is mature enough for the publication in the OECD Quality of
Care document in preparation for the Ministerial meeting in
October 2010
• Decide on whether the collection of additional data in the year
2010/2011 is warranted to inform the future development work