View
214
Download
0
Category
Preview:
Citation preview
MEASURES OF TEST ACCURACY AND ASSOCIATIONS
DR ODIFE, U.BSR, EDM DIVISION
OUTLINE
• INTRODUCTION• SENSITIVITY • SPECIFICITY• PREDICTIVE VALUES• RECEIVER OPERATOR CHARACTERISTICS CURVE• ODDS RATIO• SUMMARY• REFERENCES
INTRODUCTION
• Sensitivity, specificity, predictive values and receiver operator characteristics curves are measures of test accuracy
• Odds ratio is a measure of test associations.
SENSITIVITY The proportion of those people who really have the disease who are correctly identified as such
• Sensitivity = TP/(TP+FN)
TP FP
FN TN
DiseasePresent Absent
Test
Positive
Negative
SPECIFICITY
• The proportion of those people who really do not have the disease who are correctly identified as such
• Specificity = TN/(TN+FP)
TP FP
FN TN
DiseasePresent Absent
Test
Positive
Negative
SENSITIVITY AND SPECIFICITY
• Sensitivity and specificity are intrinsic characteristics of a test
• Both the sensitivity and specificity of a test need to be known in order to assess its usefulness for a diagnosis.
• A discriminating test would have sensitivity and specificity close to 100%.
• A test with high sensitivity may have low specificity and vice versa.
SENSITIVITY AND SPECIFICITY
7
0 5 10 15 20
Quantitative result of the test
TN
Non-affected:
Affected:
TP
Nu
mb
er
of
peop
le t
este
d Threshold forpositive result
Ideal situation
Sensitivity and specificity
8
0 5 10 15 20
TN TP
FN FP
Non-affected:Threshold forpositive result
Quantitative result of the test
Nu
mb
er
of
peop
le t
este
d Affected:
Realistic situation
SENSITIVITY AND SPECIFICITY
• They are not affected by the prevalence of a disease.
• Both can be altered by changing the threshold or cut-off point for diagnosing a disease.
• Lowering the threshold improves sensitivity but reduces specificity (i.e. more FP)
• Raising the threshold improves specificity but reduces sensitivity (i.e. more FN)
SENSITIVITY AND SPECIFICITY
10
TNTP
FN
FP
Non-affected:
Affected:Threshold forpositive result
Nu
mb
er
of
peop
le t
este
d
Quantitative result of the test0 5 10 15 20
Sensitivity and specificity
11
0 5 10 15 20
TN
TP
FN
FP
Non-affected:
Affected:Threshold forpositive result
Nu
mb
er
of
peop
le t
este
d
Quantitative result of the test
Positive Predictive Value
• It is the proportion of the people who test positive who truly have the disease
• Positive predictive value (PPV) = TP/(TP+FP)
TP FP
FN TN
DiseasePresent Absent
Test
Positive
Negative
Negative Predictive Value
• Is the proportion of the people who test negative who truly do not have the disease
• Negative predictive value = TN/(TN+FN)
TP FP
FN TN
DiseasePresent Absent
Test
Positive
Negative
PPV and NPV
• PPV and NPV give a direct assessment of the usefulness of the test
• They are highly dependent on the prevalence of the disease in the population
• When the prevalence is low, the PPV will be low and NPV will be high.
• When the prevalence is high, the PPV will be high and NPV will be low.
Predictive value positive of a test according to prevalence and specificity
0102030405060708090
100
0 10 20 30 40 50 60 70 80 90 100
Prevalence (%)
PVP % 70%80%90%95%
Specificity
Predictive value negative of a test according to prevalence and sensitivity
Sensitivity
0
10
20
30
40
50
60
70
80
90
100
Prevalence (%)
PVN %
70%80%90%95%
Receiver operating characteristic (ROC) curve
• Developed in the 1950's during World War II for the analysis of radar radio signals .
• It is a by-product of research into making sense of radio signals contaminated by noise.
• Characterize the trade-off between positive hits and false alarms
ROC curve
• Recognized in the 1970's as useful tool for interpreting medical test results.
• Has become very popular in biomedical applications, particularly radiology and imaging
• Can be used to compare overall performance of diagnostic tests/procedures
ROC curve
• ROC curve represents the relationship between the true-positive rate (sensitivity) and the false-positive rate (1-specificity)
• ROC curve plots sensitivity (on the y-axis) against 1-specificity (on the x-axis)
• Each point on the ROC curve represents a sensitivity/specificity pair corresponding to a particular diagnostic threshold.
ROC curve
• The performance of a diagnostic variable can be quantified by calculating the area under the ROC curve (AUROC).
• The ideal test would have an AUROC of 1, whereas a random guess would have an AUROC of 0.5.
• The ability of diagnostic variables to diagnose an outcome can be compared using ROC curves and their AUROCs.
Tru
e P
osi
tive R
ate
(s
en
siti
vit
y)
0%
100%
False Positive Rate (1-specificity)
0%
100%
ROC curve
Ideal classifier
chance
always negative
always positive
ROC Curves
True
Pos
itive
Rat
e
0%
100%
False Positive Rate0%
100%
True
Pos
itive
Rat
e
0%
100%
False Positive Rate0% 100%
A good test: A poor test:
ROC Curves
True
Pos
itive
Rat
e
0%
100%
False Positive Rate0%
100%
True
Pos
itive
Rat
e
0%
100%
False Positive Rate0%
100%
True
Pos
itive
Rat
e
0%
100%
False Positive Rate0%
100%
AUC = 50%
AUC = 90% AUC =
65%
AUC = 100%
True
Pos
itive
Rat
e
0%
100%
False Positive Rate0%
100%
ROC Curves
ROC Curve
Uses• Used in communications sectors to examine false alarm
rates• Evaluate the discriminatory ability of a test to distinguish
diseased and normal subjects • Help define optimal cut-off value of a test• Comparing diagnostic efficacy of 2 or more medical tests • Comparing 2 or more observers measuring the same test
ROC Curve
26
00 20 40 60 80 100
20
40
60
80
100
IFA Dilutions
1/101/201/40
1/80
1/160
1/320
1/640
100 - Specificity (%): Proportion of false positives
Sensitivity (%)
ROC Curves
27
0 25 50
75 100
IFAELISA
0
20
40
60
80
100
100 - Specificity (%)
Sensitivity (%)
Area under the ROC curve (AUC)
28
ROC Curves
Odds
• Odds are ratio of two probabilities • The probability of an event occurring divided
by the probability of the event not occurring.• The odds of an event = probability/(1–
probability).• Odds refer to single entity• It ranges from zero to infinity.
Odds
• A die has 6 sizes, each with a difference number• On one die, the odds of rolling a 1 is…?• 1 side has a 1 on it, 5 sides do NOT• Odds of rolling a 1:
= 1/5, or 20%
Odds ratio
• The odds of the outcome in one group divided by the odds of the outcome in the other group
• It is a ratio of two odds• E.g p1 refers to the probability of the outcome
in group 1, and p2 is the probability of the outcome in group 2.
• Odds ratio (OR)= p1/(1- p1)/ p2 /(1- p2)
Odds ratio
• As a ratio, it ranges from zero to infinity.• Interpretation of OR: –OR = 1: exposure has no association with
disease–OR > 1: exposure may be positively
associated with disease–OR < 1: exposure may be negatively
associated with disease
Odds ratio
Odds of a heart attack in men = 68/32, or 2.125Odds of a heart attack in women = 42/58, or 0.724• Odds ratio = 2.125 / 0.724 = 2.9
Cause of Death in Men and Women
Heart Attack?
Yes No
SexMen 68 32
Women 42 58
Odds Ratio
• The odds of receiving a death sentence if the defendant was Black = 28/45 = 0.6222
• The odds of receiving a death sentence if the defendant was not Black = 22/52 = 0.4231
• The impact of being black on receiving a death penalty is measured by the odds ratio which equals:
= the odds if black ÷ the odds if not black = 0.6222 ÷ 0.4231 = 1.47
Odds ratio
Uses • Appropriate measure of relative effect in case-
control studies.• Commonly used in meta-analysis• ORs are the output of logistic regression
37
Odds ratio
OR in case-control StudyProbability of case being exposed = Pcase
Probability of case being non-exposed =1-Pcase
Odds of case being exposed = Pcase/1- Pcase
Probability of control being exposed = Pcontrol
Probability of control being non-exposed =1-Pcontrol
Odds of control being exposed = Pcontrol/ 1-Pcontrol
Target population
Exposed in past
Not exposed
Exposed
Not Exposed
Odds ratio
Disease
(Cases)
No Disease
(Controls)
SUMMARY
• Sensitivity, specificity, predictive values, ROC curves and odds ratio are measures of test accuracy and associations.
• Sensitivity is the proportion of those people who really have the disease who are correctly identified as such
• Specificity is the proportion of those people who really do not have the disease who are correctly identified as such
• Sensitivity and specificity can be altered by changing the threshold or cut-off point for diagnosing a disease.
SUMMARY
• Sensitivity and specificity are intrinsic characteristics of a test and are not affected by the prevalence of a disease
• Predictive values of a test can be either positive or negative
• PPV is the proportion of the people who test positive who truly have the disease
• NPV is the proportion of the people who test negative who truly do not have the disease
SUMMARY
• ROC curve was developed in the 1950's during World War II for the analysis of radio signals .
• ROC curve is a useful tool for interpreting medical test results.
• ROC curve represents the relationship between sensitivity and specificity for a test
SUMMARY
• Predictive values are affected by the prevalence of a disease
• ROC curve can be used to compare overall performance of diagnostic tests
• The performance of a diagnostic variable can be quantified by calculating the AUROC
• An ideal test would have an AUROC of 1, whereas a random guess would have an AUROC of 0.5.
SUMMARY
• Odd of an event probability of an event occurring divided by the probability of the event not occurring.
• Odd ratio is the odds of the outcome in one group divided by the odds of the outcome in the other group
• The values of odds and OR ranges from ranges from zero to infinity.
SUMMARY
• OR of 1 means exposure has no association with disease, > 1 means exposure may be positively associated with disease and < 1 means exposure may be negatively associated with disease
• Odd ratios are useful in case-control studies, meta-analysis and logistic regression
REFERENCES
• Bland JM, Altman DG. Statistics notes. The odds ratio. BMJ 2000;320:1468.
• Holcomb WL , Chaiworapongsa T, Luke DA, Burgdorf KD. An odd measure of risk: use and misuse of the odds ratio. Obstet Gynecol 2001;98:685–8
• Katz KA. The (relative) risks of using odds ratios. Arch Dermatol 2006;142: 761–4.
REFERENCES
• Davies HT, Crombie IK, Tavakoli M. When can odds ratios mislead? BMJ 1998;316:989–91
• Schechtman E. Odds ratio, relative risk, absolute risk reduction, and the number needed to treat: which of these should we use? Value Health 2002;5:431–6.
Recommended