Critiquing for Evidence-based Practice: Diagnostic and Screening Tests M8120 Columbia University Fall 2001 Suzanne Bakken, RN, DNSc

Critiquing for Evidence-based Practice: Diagnostic and Screening Tests

M8120Columbia University

Fall 2001

Suzanne Bakken, RN, DNSc

Overview

Probability fundamentalsClinical role of probability revision Characterizing new information (test

performance)Probability revision methods

– Bayes’ formula– Contingency table

In class exercises

Probability Fundamentals

Strength of beliefA number between 0 and1 that expresses an

opinion about the likelihood of an eventProbability of an event that is certain to occur

is 1Probability of an event that is certain to NOT

occur is 0

Summation Principle

Probability that an event will occur plus probability that it will not occur equals 1

Probability of all possible outcomes of a chance event is always equal to 1– Blood type: What is p[AB] given p[O]=0.46,

p[A]=.40, and p[B]=.10?– Fraternal triplets: What is the probability of at

least one boy and one girl?

Diagramming Probabilities

Type O (0.46)

Type A (0.40)

Type B (0.10)

Type AB (0.04)Chance node

Sum of probabilities at chance node = 1

Conditional Probability

Probability that event A occurs given that event B is known to occur

p[AlB]p[A,B]=p[AlB] X p[B]Examples in health care

Components of Probability Estimates

Personal experiencePublished experience - evidenceAttributes of the patient

Role of Probability Revision Techniques

Abnormal Finding

Diagnosis

BeforeFinding

AfterFinding

0 1Probability of Disease

Prior Probability Posterior Probability


Negative Finding

Diagnosis

AfterFinding

BeforeFinding


Posterior Probability Prior Probability

Findings

SignsSymptomsDiagnostic testsProbabilistic relationships between

findings and disease basis of diagnostic decision support systems– Dxplain– QMR– Iliad

Definitions

Prior probability - the probability of an event before new information (finding) is acquired; pretest probability or risk

Posterior probability - the probability of an event after new information (finding) is acquired; posttest probability or risk

Probability revision - taking new information into account by converting prior probability to posterior probability

Review of Conditional Probability

p[AlB]

p[AlB]p[A and B]

p[B]=

Probability that an event is true, given that another event is true

What is the probability that someone has HIV antibody given a positive HIV test?What is the probability that someone has venothrombosis given a swollen calf?What is the probability that someone has dyspnea given the nurse says dyspnea is present?

Characterizing “Test” Performance

Compare test against gold standard (e.g., presence of disease; established test)

Ideal test - no values at which the distribution of those with the disease and without the disease overlap

Few tests ideal so …– TP– TN– FP– FN

Test Performance

True positive rate (TPR) p[+lD] = probability of an abnormal test result given that the disease is present; the number of persons WITH the disease who have an abnormal test result divided by the number of persons WITH the disease; sensitivity

Contingency Table View

Disease present = TPR + FNR = 1Disease absent = FPR + TNR = 1

TestResults

DiseasePresent

DiseaseAbsent

Total

Positive TP FP TP + FP

Negative FN TN FN + TN

TP + FN FP + TN

Test Performance

False positive rate (FPR) p[+lno D] = probability of an abnormal test result given that the disease is absent: the number of persons WITHOUT the disease who have an abnormal test result divided by the number of persons WITHOUT the disease



TestResults

DiseasePresent

DiseaseAbsent

Total



TP + FN FP + TN

Test Performance

True negative rate (TNR) p[-lno D] = probability of a normal test result given that the disease is absent; number of persons WITHOUT the disease who have a normal test result divided by number of persons WITHOUT the disease; 1 - FPR or specificity; 100% specificity = pathognomonic



TestResults

DiseasePresent

DiseaseAbsent

Total



TP + FN FP + TN

Test Performance

False negative rate (FNR) p[-l D] = probability of a normal test result given that the disease is present; number of persons WITH the disease who have a normal test result divided by number of persons WITH the disease; 1 - TPR



TestResults

DiseasePresent

DiseaseAbsent

Total



TP + FN FP + TN

Sensitivity vs. Specificity

Weighing sensitivity Vs. specificity in setting cutoff level for abnormality in a test

Consequences of FPR vs. FNR– Severity of disease– Availability of treatment– Risk of treatment

Sensitivity and specificity are characteristics of a test and a criterion for abnormality

Receiver Operating Characteristic (ROC) Curves

0 0.5 1.0

1.0

0.5

FPR (1 - specificity)

TPR

( s

en

siti

vit

y) Increased p[D]

Decreased p[D]



TestResults

DiseasePresent

DiseaseAbsent

Total



TP + FN FP + TN

Example of HIV

HIV TestResults

AntibodyPresent

AntibodyAbsent

Total

Positive 98 3 101

Negative 2 297 299

100 300 400

What is the TPR?What is the FPR?What is specificity?

Example of HIV

TPTPR =

TP + FN

TPR = sensitivity p[+lD]

Example of HIV

FPFPR =

FP + TN

FPR = p[+lno D]

Example of HIV

TNTNR =

TN + FP

TNR = specificity p[-lno D]

Nurse and Patient Rating of Symptoms

By definition, patient is gold standard for symptom rating

RN as “test” for presence or absence of symptom

Fatigue– RN yes/Pt yes = 50– RN yes/Pt no = 15– RN no/Pt yes = 20– RN no/Pt no = 15

What is sensitivity? What is specificity?

Nurse and Patient Ratings of Symptoms: Sensitivity

TPTPR =

TP + FN

Nurse and Patient Ratings of Symptoms: Specificity

TNTNR =

TN + FP

Moving from Test Characteristics to Predictive Value and Posterior Probabilities

Predictive valueForms of Bayes’

– Bayes’ formula– Contingency table view– Likelihood ratio

Prevalence

Frequency of disease in the population of interest at a given point in time

Predictive Value

Sensitivity, specificity, and their complements (FNR & FPR) focus on probability of findings given presence or absence of disease so not in a clinically useful form

Predictive value focuses on probability of disease given findings

Predictive value takes prevalence of disease in study population into account

Positive Predictive Value

number of persons with disease with abnormal findingPV+ = number of persons with abnormal finding

TPPV+ = TP + FP

The fraction of persons with an abnormal finding who have the disease

Negative Predictive Value

number of persons with normal finding WITHOUT diseasePV- = number of persons with normal finding

TNPV+ = TN + FN

The fraction of persons with an normal finding who DO NOT have the disease

Nurse and Patient Rating of Symptoms

By definition, patient is gold standard for symptom rating

RN as “test” for presence or absence of symptom

Shortness of Breath– RN yes/Pt yes = 25– RN yes/Pt no = 10– RN no/Pt yes = 15– RN no/Pt no = 50

What is PV+? What is PV-?

Nurse and Patient Ratings of Symptoms: PV+

TPPV+ =

TP + FP

Nurse and Patient Ratings of Symptoms: PV-

TNPV- =

TN + FN


Abnormal Finding

Diagnosis

BeforeFinding

AfterFinding


Prior Probability Posterior Probability

Calculating Posterior Probability with Bayes’

What is the probability that someone has HIV antibody given a positive HIV test?

Can calculate with Bayes’ if you know:– Prior probability of the disease– Probability of an abnormal test (+) result

conditional upon the presence of the disease (TPR)

– Probability of an abnormal test (+) result conditional upon the absence of the disease (FPR)

Bayes’ Theorem

p[Dl+]p[D] x p[+lD]

{p[D] x p[+lD]} + {p[no D] x p[+lno D]}

=

Not a clinically useful form!

Deriving Bayes’ Theorem

Summation principle:

p[Dl+]

p[+]

p[+,D]

=

Given definition of conditional probability:

p[+] = p[+,D] + p[+,no D]

p[Dl+]

p[+,D] + p[+, no D]

p[+,D]

=

Thus:

1

2

3

p[+lD]

p[D]

p[+,D]

=

p[+lno D]

p[no D]

p[+,no D]

=

4

and

Given principle of conditional independence, rearrange expressions above:

p[+,D] = p[D] x p[+lD] p[+,no D] = p[no D] x p[+lnoD]5

p[Dl+]

p[D] x p[+lD] +p[no D] x p[+lnoD]

p[D] x p[+lD]

=

Substitute into 1:

6

Given definition of conditional probability



Can calculate with Bayes’ if you know:– Prior probability of the disease– Probability of an abnormal (+) test result

conditional upon the presence of the disease (TPR=.98)

– Probability of an abnormal (+) test result conditional upon the absence of the disease (FPR=.01)

Bayes’ Theorem

p[Dl+]p[D] x p[+lD]

{p[D] x p[+lD]} + {p[no D] x p[+lno D]}

=

TPR

FPR1 - p[D]

When Abnormal Test Result is Present

p[Dl+]p[D] x TPR

{p[D] x TPR} + {1 - p[D] x FPR}

=

A somewhat more clinically useful form

When Normal Test Result is Present

p[Dl-]p[D] x 1 - TPR

{p[D] x 1 - TPR} + {1 - p[D] x 1 - FPR}

=

A somewhat more clinically useful form

FNR

TNR



Can calculate with Bayes’ if you know:– Prior probability of the disease (can be prevalence

or other information)– Probability of an abnormal test result conditional

upon the presence of the disease (TPR=.98)– Probability of an abnormal test result conditional

upon the absence of the disease (FPR=.01)

The Role of Prior Probability

p[Dl+]

=

Prevalence of HIV Antibody in Homosexual Men in SF in mid1980s = .5

p[Dl+] =


p[Dl+]p[.5] x .98

.49 + (.51 x .01) = .4951

=

Prevalence of HIV Antibody in Homosexual Men in SF in mid1980s = .5

p[Dl+] = .99


p[Dl+]

=

Prevalence of HIV Antibody in Female Blood Donors = .0001

p[Dl+] =


p[Dl+]p[.0001] x .98

.00098 + (.9999 x .01) = .010979

=

Prevalence of HIV Antibody in Female Blood Donors = .0001

p[Dl+] = . 089

Likelihood Ratios

Likelihood ratio =

FPR

An even more clinically useful form!

TPR

Nomogram for interpreting diagnostic test result

Diagramming Probabilities

Type O (0.46)

Type A (0.40)

Type B (0.10)

Type AB (0.04)Chance node

Sum of probabilities at chance node = 1

Path Probability

Operate

Do not operate

Disease present

Disease absent

Disease present

Disease absent

Survive

Operative death

Palliate

Operative deathOperative death

Survive

Survive

No cure

Cure

Cure

No Cure

No cure

Cure

p=.10

p=.90

p=.10

p=.90

p=.90

p=.10

p=.02

p=.98 p=.10

p=.90

p=.10

p=.90p=.90

p=.10

p=.01

p=.99

Try for the cure

Path probability of a sequence of chance events is the product of all probabilities along that sequence

Conditional Independence

Two findings are conditionally independent if TPR and FPR of one clinical finding do not depend upon the presence of the other finding

Assumption of conditional independence invoked when the same TPR (or FPR) is used in Bayes’ regardless of the prior probability of disease

Relevant in series of testsMay be invalid in some clinical situations

Interpreting Sequence of Tests

Posttest probability of first test used as pretest probability of second test

TPR and FPR of second test used in Bayes’ to calculate posttest probability following second test

Test Performance Biases Most significance source of error in measuring test performance is

due to differences between population in which test performance is measured and the population in which the test will be used

Spectrum bias - differences between populations in the spectrum of disease presentation and severity– Test population contains more sick persons than clinically relevant population– Test-referral bias - the composition of the population used to evaluate a diagnostic test

is altered when the test is a criterion for referring a patient for the definitive diagnostic procedure

TPR is usually higher in the study population than in the clinically relevant population due to few negatives (FN & TN) referred

FPR is usually higher in the study population than in the clinically relevant population due to few TN (remember FP + TN = 1)

Critically Analysis of Report of Diagnostic or Screening Test

Are the results of the study valid?

What are the results?Will the results help me in caring

for my patients?


Are the results of the study valid?– Was there an independent, blind comparison with

a reference (gold) standard?– Did the patient sample include an appropriate

spectrum of patients to whom the diagnostic test will be applied in clinical practice?

– Did the results of the test being evaluated influence the decision to perform the reference standard?

– Were the methods for performing the test described in sufficient detail to permit replication?




for my patients?


What are the results?– Are likelihood ratios for the test

result presented or data necessary for their calculation included?




for my patients?


Will the results help me in caring for my patients?– Will the reproducibility of the test

result and its interpretation be satisfactory in my setting?

– Are the results applicable to my patient?

Documents

Critiquing for Evidence-based Practice: Diagnostic and Screening Tests M8120 Columbia University Fall 2001 Suzanne Bakken, RN, DNSc