Methods in Epidemiology · to discuss the meaning of ... Guyatt GH. Sources of bias in diagnostic accuracy studies and ... (nonevaluable = positive) Angiography Methods in epidemiology

1

Methods in epidemiology – Medical statistics – Diagnosis 1

Diagnosis

How to evaluate diagnostic

accuracy

Methods in Epidemiology

At the end of the lecture students should be able

� to discuss the meaning of normality

� to report main biases in the evaluation of diagnostic tests

� to calculate main measures of accuracy of diagnostic

tests (sensitivity, specificity, positive and negative predictive

value)

� to interpret likelihood ratios and ROC curves when

assessing accuracy of a diagnostic test

� to measure reliability of diagnostic tests

Medical statistics

2



Context Coronary artery disease represents a leading cause of death and health care expenditure in Western countries. Establishing its anatomic diagnosis requires coronary angiography, a procedure that is costly and carries risks and discomfort.

Figure 2. Patient with severe stenosis in the first diagonal (D1) branch of the left anterior descending artery

3


Context

Multidetector computed tomography (MDCT) has been

proposed as a noninvasive method to evaluate coronary

anatomy.

Single-center studies have reported sensitivities between 30% and 95% and specificities between 86% and 98% for the detection of obstructive coronary lesions using 16-row MDCT scanners. ?Objective To determine the diagnostic accuracy of 16-row MDCT for the

detection of obstructive coronary disease in a larger

multicenter study.


Is this an observational or an experimental study?

In your opinion

4



In your opinion

Which patients are to be studied?


The CATSCAN study was designed to prospectively include

patients between the ages of 30 and 70 years, who were

referred for clinically indicated nonemergency coronary

angiography, for evaluation of chest pain, and for intermediate

or high probability of disease.

5


� Diagnostic accuracy depends on the severity spectrum of the disease

Selecting the study subjects

� The study population must be consistent with target population

� Diagnostic accuracy is usually different in outpatients and hospitalized subjects

� The main rule is the ‘Intention to diagnose’ and subjects should be selected according to clinical practice (i.e. when disease si suspected) � consecutive patients

� Characteristics of the study subjects should always be reported (sampled population)

Metodologia clinica 5.1

Actual study

Implementation

Data

Study sample

Planning

Target population

Study population

Study question

Study protocol

Study object Variables to be measured

Structure of clinical research

Inference

Parameter Estimate

iθθθθ̂

Study conclusions

Tθθθθ ( )

iSEθθθθθθθθ ˆ=

Systematic errors

Randomand

Systematic errors

6


Selection bias

� If the inclusion criteria do not include a patient spectrum

similar to the population in which the test will be used in

practice, the results of the study may have limited

applicability (e.g advanced or hospitalized subjects)

� Differences in disease severity among studies may

lead to differences in estimates of test performance


The CATSCAN study was designed to prospectively include

patients between the ages of 30 and 70 years, who were

referred for clinically indicated nonemergency coronary

angiography, for evaluation of chest pain, and for intermediate

or high probability of disease.

We estimated a sample size of 234 patients to achieve 80% power to reject the null hypotheses.

The primary hypothesis tested was whether MDCT could detect more than 50% luminal narrowing in coronary artery segments larger than 2.0 mm in diameter and have a sensitivity of more than 85% and a specificity of more than 85%.

?

7



In your opinion


Which variables are to be measured to evaluate the

diagnostic accuracy?

Main Outcome Measures Segment-based and patient-based sensitivities and

specificities for the detection of luminal stenosis of more

than 50% (of luminal diameter) based on quantitative

coronary angiography.?

Main Outcome Measures Segment-based and patient-based sensitivities and

specificities for the detection of luminal stenosis of more

than 50% (of luminal diameter) based on quantitative

coronary angiography.?


Reference standard

It is the ‘true patient status’. Rather it is the best procedure

to assess diagnosis.

Diagnostic tests

Index test

Is the test whose ability to predict the diagnosis we want

to assess

� Angiography

� MDCT

8



In your opinion


Which variables are to be measured to evaluate the

diagnostic accuracy?

How should the tests be executed?

It was anticipated that 70% of the patients enrolled in the study would complete both MDCT and conventional angiography procedures and have adequate MDCT images for interpretation.


Reference standard

It is the ‘true patient status’. Rather it is the best procedure

to assess diagnosis.

Diagnostic tests

Index test

Is the test whose ability to predict the diagnosis we want

to assess

� Angiography

� MDCT

� Execution procedures of the study tests should be clearly pre-defined

� All patients must undergo both tests � cross sectional studies

� Test should be performed in a blind way

9


Metodologia clinica 5.1

Actual study

Implementation

Data

Study sample

Planning

Target population

Study population

Study question

Study protocol

Study object Variables to be measured

Structure of clinical research

Inference

Parameter Estimate

iθθθθ̂

Study conclusions

Tθθθθ ( )

iSEθθθθθθθθ ˆ=

Systematic errors

Randomand

Systematic errors

10


Information bias

Execution

� The index test is performed long before the reference standard, so

the disease is at a more advanced stage when the last one is performed

� Treatment is started on the basis of the results of the index test,

and the reference standard is applied after treatment has started

� Only a selected sample of patients who underwent the index test

is verified by the reference standard.

Interpretation

� Interpretation of the index test or reference standard is influenced

by knowledge of the results of the other test.

� The availability of information on clinical data may affect the

estimates of the test performance

� the uninterpretable results are simply removed from the analysis

All lead to overestimation of diagnostic accuracy

Furukawa TA, Guyatt GH. Sources of bias in diagnostic accuracy studies and the diagnostic process. CMAJ 2006; 174(4):481-2.

11


Conclusions

The results of this study indicate that MDCT coronary

angiography performed with 16-row scanners is limited by a

high number of non evaluable cases and a high false-

positive rate. Thus, its routine implementation in clinical

practice is not justified. Nevertheless, given its high

sensitivity and negative predictive value, 16-row MDCT may

be useful in excluding coronary disease in selected patients

in whom a false-positive or inconclusive stress test result is

suspected.

How much is this conclusion supported by data?


Diagnosis

What is diagnosis?


12


“An effort to recognize the class or group to

which a patient’s illness belongs so that,

based on our previous experience with that

class, the subsequent clinical acts we can

afford to carry out (and the patient is willing to

follow) will maximize that patient’s health”

(Sackett DL et al, Clinical Epidemiology 1985)

Diagnosis


(Presumed) healthy

Reference

interval

Diagnosis

13


Reference

interval

Who are they?

Healthy people with too

high or too small values

� False positive

Healthy

Diagnosis


Sick

Reference

interval

Diagnosis

Healthy

14


Reference

interval

Diagnosis

SickHealthy


Reference

interval

Who are they?

Sick people with ‘normal’

values � False negative

Diagnosis

SickHealthy

15


Reference

interval

Diagnosis

SickHealthy

16

“We cannot know that which we do not already know and are unlikely to find that which we do not suspect”

(Burnum JF, Giving meaning to sign, Ann Intern Med 1993)


Diagnosis

How to evaluate diagnostic

accuracy


17

At the end of the lecture students should be able

� to discuss the meaning of normality

� to report main biases in the evaluation of diagnostic tests

� to calculate main measures of accuracy of diagnostic

tests (sensitivity, specificity, positive and negative predictive

value)

� to interpret likelihood ratios and ROC curves when

assessing accuracy of a diagnostic test

� to measure reliability of diagnostic tests

Medical statistics


How to measure the ability of the test to predict the

correct diagnosis?

� sensitivity

� specificity

� positive predictive value (PPV)

� negative predictive value (NPV)

� likelihood ratio (LR)

� area under the ROC curve

Diagnostic accuracy

18


Diagnostic accuracy

Sensitivity and Specificity



19


> 50%

> 50% ≤ 50% Total

≤ 50%

TP FP

FN TN

Please fill in the contingency table with the data of the

previous slide

MDCT

Segment-based analysis (nonevaluable = positive)

Angiography


MDCT

> 50%

> 50% ≤ 50%

79 544

10 996

89 1.540

623

1.006

1.629

≤ 50%

TP FP

FN TN

Sensitivity = TP/(TP+FN) = 79/89 = 89%

P (T+|||| D+)


Angiography

Total

20


> 50%

> 50% ≤ 50%

79 544

10 996

89 1.540

623

1.006

1.629

≤ 50%

FP

FN

Specificity = TN/(TN+FP) = 996/1.540 = 65%

P (T-|||| D-)


MDCT

Angiography

TP

TN

Total


21


> 50%

> 50% ≤ 50%

55 96

10 996

65 1.092

151

1.006

1.157

≤ 50%

FP

FN


Specificity = TN/(TN+FP) = 996/1.092 = 91%

Segment-based analysis (nonevaluable excluded)

MDCT

Angiography

TP

TN

Total


n

ppp

)1(96.1

−±

)06.089.0

89

)11.0)(89.0(96.189.0 ±=±

22



> 50%

> 50% ≤ 50%

58 58

1 70

59 128

116

71

187

≤ 50%

FP

FN


Specificity = TN/(TN+FP) = 70/128 = 54%

Patient-based analysis (nonevaluable = positive)

MDCT

Angiography

TP

TN

Total

23


> 50%

> 50% ≤ 50%

44 29

15 99

59 128

73

114

187

≤ 50%

FP

FN


Patient-based analysis (nonevaluable excluded)

Specificity = TN/(TN+FP) = 99/128 = 77%

MDCT

Angiography

TP

TN

Total


Documents

Methods in Epidemiology · to discuss the meaning of ... Guyatt GH. Sources of bias in diagnostic accuracy studies and ... (nonevaluable = positive) Angiography Methods in epidemiology