Upload
others
View
7
Download
0
Embed Size (px)
Citation preview
50 shades of greyhow (in)accurate are risk prediction scores
Dr Doug CampbellMeasuring, managing and minimising riskPerioperative SIG, Grand Hyatt, Melbourne
October 27th 2018
Declarations and Acknowledgements
• No relevant conflicts of interest• Funded by Precision Driven Health
Those who make many species are the 'splitters,' and those who make few are the 'lumpers.'
Myth of individualised prediction
…with apologies to XKCD
Validation 101- easy as ABCD
Validation 101- easy as ABCD
A=0B=0.96C=0.922
nzRISK – CalibrationIntercept = 0Slope = 0.97McFadden’s R2 = 0.28AUROC = 0.92
Intercept = 0Slope = 0.96McFadden’s R2 = 0.31AUROC = 0.91
Intercept = 0Slope = 0.98McFadden’s R2 = 0.37AUROC = 0.90
1 month
1 year 2 year
nzRISK – DiscriminationCentile Total Predicted Observed Predicted Observed
N (%) N % % N N
1 9481 0 0 0.9 1
2 8528 0 0 1.5 0
3 9303 0 0 2.9 1
4 8713 0.1 0.1 5.4 5
5 9026 0.1 0.1 9.5 5
6 9192 0.2 0.1 14.8 8
7 (0.2-0.3) 8784 0.2 0.3 21.6 26
8 (0.3—0.5) 9008 0.4 0.2 35.9 21
9 (0.5 – 1.5) 9004 0.9 0.9 77.5 78
10 (>1.5) 8996 5.6 5.4 508.1 489
nzRISK – DiscriminationCentile Total Predicted Observed Predicted Observed
N (%) N % % N N
1 9481 0 0 0.9 1
2 8528 0 0 1.5 0
3 9303 0 0 2.9 1
4 8713 0.1 0.1 5.4 5
5 9026 0.1 0.1 9.5 5
6 9192 0.2 0.1 14.8 8
7 (0.2-0.3) 8784 0.2 0.3 21.6 26
8 (0.3—0.5) 9008 0.4 0.2 35.9 21
9 (0.5 – 1.5) 9004 0.9 0.9 77.5 78
10 (>1.5) 8996 5.6 5.4 508.1 489
Low risk, NZRISK < 0.5%, 11% of deaths
High risk, NZRISK > 1.5%, 77% of deaths
Intermediate risk, NZRISK 0.5-1.5%, 12% of deaths
Decision curve analysis
Glance et al. Impact of the choice of model for identifying low-risk patients using the 2014 ACC/AHA perioperative guidelines. Anesthesiology 2018; 129(5): 889-900
Generalisability
SORT internal validation
AUROC = 0.91
AUROC = 0.90
SORT external validation
Generalisability
SORT internal validation
AUROC = 0.91
AUROC = 0.90
SORT external validation
Protopapa KL et al. Development and validation of the Surgical Outcome Risk Tool (SORT). Br J Surgery 2014; 101: 1774-83Campbell D et al. Development and validation of a multivariable prediction model of perioperative mortality in noncardiac surgery (NZRISK). Br J Surgery 2018; in press
What risk tool for CADENZA?
Eugene N et al. Development of the NELA risk model. BJA 2018; 121(4): 739-48
ACS-NSQIP - morbidity
DAH90 – the median is not the message
Campbell D, Boyle L, Djamali K, Cumin D, Weller J, Short T, Merry A. Unpublished data.
New Zealand 2011-6, n = 141331-month mortality = 4.2%3-month mortality = 5.7%
83 days48 days
Checklist for risk tools
• Does it model the correct outcome?– 1-month mortality, morbidity, long-term mortality
• Is it accurate?– calibration (SHARED DECISION-MAKING)– discrimination (TRIAGE)
• Is it generalizable to my patient?– validated in Australia or New Zealand?
www.nzrisk.com
www.nzrisk.com
www.nzrisk.com
Comparison - general
Calculator Covariates Country N AUROC Calibration
ACS-NSQIP 22 US 1.4 million 0.944 Brier 0.011POSPOM 17 France 5.5 million 0.929 Brier 0.004nzRISK 8 NZ 360140 0.922 Brier 0.007SORT 6 UK 16,788 0.91 H-L 12.16P-POSSUM 18 UK 10,648 total 0.83 (0.68-0.92)
H-L = Hosmer-Lemeshow statistic, McF = McFaddens pseudo rho-squared statistic, Brier = Brier score, O:E observed to expected ratio
High risk calibration