Upload
edward-saunders
View
44
Download
3
Embed Size (px)
DESCRIPTION
A method for estimating the cost of reducing the false alarm rate in multi-institution performance monitoring using CUSUM charts. Alex Bottle [email protected] Imperial College London Dr Foster Unit. Overview. - PowerPoint PPT Presentation
Citation preview
© Imperial College LondonPage 1
A method for estimating the cost of reducing the false alarm rate in multi-
institution performance monitoring using CUSUM charts
Alex [email protected]
Imperial College LondonDr Foster Unit
© Imperial College LondonPage 2
Overview
• Background: cumulative sum charts and nationwide NHS mortality monitoring tool
• Extent of multiple testing
• Factors affecting false alarm rate
• Simulation for false alarm and successful detection rates
• Estimation of ‘cost’: worked example for AMI
• Summary
© Imperial College LondonPage 3
CUSUM chart essentials
• Plots one patient at a time
• Chart statistic (log-likelihood ratio) goes up if patient dies and down if patient survives
• Chart rises faster if low-risk patient dies
• If crosses preset threshold, chart ‘signals’
• Threshold choice involves consideration of type I and type II error rates
© Imperial College LondonPage 4
CUSUM charts
© Imperial College LondonPage 5
Mortality monitoring tool
• In use in ~100 acute hospitals in England
• Compares each hospital’s case-mix adjusted mortality rate with national average
• Tests for an odds ratio of at least 2
• Displayed using cumulative sum charts
• Data are updated monthly
© Imperial College LondonPage 6
Mortality monitoring tool: opening screen
© Imperial College LondonPage 7
Extent of multiple testing
Over time: threshold handles this elementBut…At each hospital trust each month:• 78 diagnosis groups• >100 procedure groups
National monitoring incurs further ‘cost’:• ~150 acute hospital trusts• Consultant-level monitoring?
© Imperial College LondonPage 8
Factors affecting the false alarm rate
• Threshold: the higher this is set, the lower the false alarm rate
• Length of monitoring: number of patients varies by hospital and diagnosis
• Expected mortality rate: e.g. 5% rates will have high FAR than 1% rates
• Size of increase (OR) to be detected (not considered here)
© Imperial College LondonPage 9
Research question
• A higher chart threshold -> lower FAR but slower detection of high mortality rates
• Compared with the conventional 5% false alarm rate, what is the ‘cost’ of having a lower false alarm rate (1% or 0.1%) to deal with all the multiple testing?
© Imperial College LondonPage 10
Simulation: FAR and SDR
• For FAR, generate 5,000 artificial hospitals with mortality rate p
• Do this for various p, p=0.1% to 30%
• Calculate FAR after t patients, t in steps of 5 from 5 to 20,000
• Do this for different thresholds h, h=0.5 to 15
• For SDR, generate hospitals with rate 2p/(1-p)
© Imperial College LondonPage 11
Using the simulation to estimate ‘cost’
• For each dx, work out the threshold h needed for FAR of 5% at average hosp
• Find number of monitored patients t needed for SDR of 80% using threshold h
• Knowing the dx’s expected death rate and OR to be detected, convert t into a number of deaths
• Repeat for FAR of 1% and 0.1%• Find the difference in number of deaths
between the pairs of FAR values
© Imperial College LondonPage 12
‘Cost’ calculation for AMI in England (1)
• National death rate=11.8%. Average number of AMIs per hospital=467
• For FAR=5%, h=5.2 -> t=185 for SDR=80%
• At rate p, this means 21.8 deaths
• At rate 2p/(1-p), this means 39.0 deaths
• ‘Excess’ deaths: 39.0 – 21.8 = 17.2
© Imperial College LondonPage 13
‘Cost’ calculation for AMI in England (2)
• For FAR=0.1%, h=8.6 -> t=305 for SDR=80%
• At rate p, this means 36.0 deaths
• At rate 2p/(1-p), this means 64.4 deaths
• ‘Excess’ deaths: 64.4 – 36.0 = 28.4
• ‘Cost’ of lowering FAR to 0.1% =
28.4 – 17.2 = 11.2 extra deaths at average hosp
© Imperial College LondonPage 14
Findings for AAA repair and CABG
• AAA repair: less common but high risk
• ‘Cost’=6.3 for FAR=0.1%, 2.4 for FAR=1%
• CABG: common but low risk
• ‘Cost’= 6.8 for FAR=0.1%, 2.8 for FAR=1%
• These are all figures for an average hospital
© Imperial College LondonPage 15
Summary
• Multiple testing can be addressed by lowering the false alarm rate: raise the threshold for CUSUM charts
• Other approaches include minimising ‘loss function’ or maximising ‘desirability function’
• The proposed measure of ‘cost’ depends on mortality rate and hospital volume
• The ‘cost’ can be derived from simulation and is intuitive to less-technical users