EBM for Novice

EBM for NoviceEBM for Novice

Maria Kwok, MD, MPHAssistant Clinical Professor

Section of Pediatric Emergency MedicineMorgan Stanley Children’s Hospital of New York

Presbyterian

OutlineOutline• Why is EBM important?

• What is EBM

• Understand foreground vs. background questions

• Formulate PICO questions

• Find resources, do searches

• Validity of an article

• Appraise “Therapy” articles

• Appraise “Diagnostic” articles

Why is EBM Important?Why is EBM Important?

• MEDLINE– 400,000 new entries added each year

• To keep up-to-date– Need to read 6,000 articles each day

Why is EBM Important?Why is EBM Important?

Lag time from time of “knowing” to time of “implementation”– 13 years for thrombolytic therapy– 10 years for corticosteroids for

acceleration of fetal lung maturity

(Antman EM, JAMA, 1992)

Traditional ApproachesTraditional Approaches

Self-reported reading time per week. (University setting)– Medical students 60 min.– Interns none– Senior residents 10 min.– Fellows 45 min.– Attendings graduating

• Post 1975 60 min.• Pre 1975 30 min.

There is simply no way we There is simply no way we can keep up to date in can keep up to date in

medicine using traditional medicine using traditional approaches!approaches!

OutlineOutline• Why is EBM important?

• What is EBM?

• Understand foreground vs. background questions

• Formulate PICO questions

• Find resources, do searches

• Validity of an article

• Appraise “Therapy” articles

• Appraise “Diagnostic” articles

Quiz: Define EBMQuiz: Define EBM

C) The conscientious, explicit, and judicious use of current best evidence in making decisions about the care of individual patients

A) A means of camouflaging biostatistics in medical training

B) An annoying and overused catch phrase

Research Evidence

Clinical Expertise Patient Preference

Decision MakingDecision Making

Evidence Based Medicine: A ProcessEvidence Based Medicine: A Process

Patients Ask Questions

SearchingAppraise

Validity Results Applicability

OutlineOutline• Why is EBM important?

• What is EBM

• Understand foreground vs. background questions

• Formulate PICO questions

• Find resources, do searches

• Validity of an article

• Appraise “Therapy” articles

• Appraise “Diagnostic” articles

What’s the difference between foreground and background questions?

Asking the Precise QuestionAsking the Precise Question

• Background questions• Basic aspect of a

disease• Pathophysiology• Etiology• Basic treatment

• Who, what, when, how

• Foreground questions• Specific knowledge• Have 4 parts:

•Patient/problem

•Intervention

•Comparison intervention

•Clinical outcomes

Background or Foreground?

• What is asthma?

• Is prednisone helpful in asthma?

• What are the newest medication for asthma?

• Does atrovent used acutely make you feel better?

Background or Foreground?Background or Foreground?

• What is asthma? (B)

• What are the newest medication for asthma? (B)

• Does atrovent used acutely make you feel better? (F)

• Is prednisone helpful in asthma? (F or B)• Foreground if compare to other drugs• Background if interested in how it works

OutlineOutline• Why is EBM important?

• What is EBM

• Understand foreground vs. background questions

• Formulate PICO questions

• Find resources, do searches

• Validity of an article

• Appraise “Therapy” articles

• Appraise “Diagnostic” articles

Structure of a Well-built QuestionStructure of a Well-built Question• Patient or population

• Be specific to capture the group you want• Ex: Children w/ asthma

• Intervention• Be specific

• Comparison group (if any)• Compare to standard therapy or test

• Outcome• Be precise• What are the outcome of interest

Patient Intervention Comparison (if relevant)

Outcome

In patients with asthma discharged

home from the emergency department

does the use of both inhaled and oral corticosteroids

compared with oral steroids

alone

lead to a reduction in

asthma relapse?

Formulating the Clinical Question

PICO

Can You Identify PICO?Can You Identify PICO?

• In children under 6 months, how does sleeping on back compared to sleeping on the stomach in terms of risk of SIDS?

• In children under 6 months (P), how does sleeping on back (I) compared to sleeping on the stomach (C) in terms of risk of SIDS (O)?

Can You Form a PICO Question?Can You Form a PICO Question?

Clinical scenario:

5 yo with moderate persistent asthma now in severe acute asthma exacerbation. Intern gave 2 albuterol and orapred with minimal improvement. Intern asks why how good is atrovent?

Searchable PICO QuestionSearchable PICO Question

P: Population I: Intervention/diagnostic test/risk factorC: ComparisonO: Outcome

In children with acute asthma exacerbation (P), will the addition of atrovent (I) to albuterol (C) decrease the rate of hospitalization (O)?

OutlineOutline• Why is EBM important?

• What is EBM

• Understand foreground vs. background questions

• Formulate PICO questions

• Find resources, do searches

• Validity of an article

• Appraise “Therapy” articles

• Appraise “Diagnostic” articles

Searching Superhero

Best Bets

The Evidence Pyramid

If your question is about… Look for a …

Intervention/Therapy • Randomized controlled trial

Diagnosis/Screening

To assess the accuracy of the test

To assess effect of test on health

outcome

• Cohort study

• Randomized controlled trial

Prognosis •Longitudinal cohort

Etiology/Risk factors • Randomized controlled trial• Cohort• Case-control

Pre-appraised ResourcesPre-appraised Resources• Cochrane

– http://www.cochrane.org/cochrane/revabstr/mainindex.htm– Very high quality reviews– Mostly questions of therapy

• National Guideline Clearinghouse– http://www.guideline.gov/– Guidelines of varying levels of quality– Do broad searches

• PEM database– http://researchinpem.homestead.com/homepage.html– Not pre-appraised

Pre-appraised ResourcesPre-appraised Resources

• Best Evidence Topics– http://www.bestbets.org– Developed in the ED of Manchester Royal

Infirmary in UK– Usually EM topics– Free

• Clinical Evidence (CE Concise)– http://www.clinicalevidence.com– From UK– Focus mostly on therapy– Free

Primary Search EnginesPrimary Search Engines

• Pubmed

• Ovid

Primary Search EnginesPrimary Search Engines

• Both PubMed and Ovid

• Contains MEDLINE and other bibliographic databases (Cochrane, EMBASE, etc)

• Sells access to the databases to libraries

PubMed MEDLINE vs. Ovid MEDLINEPubMed MEDLINE vs. Ovid MEDLINE

• Same DATABASE

• Different SEARCH ENGINES• Searching differently may get you

slightly different results

• If you search both databases EXACTLY the same, your results will be the same

What is MEDLINE?What is MEDLINE?

• MEDLINE is a DATABASE• MEDLINE is produced by the National

Library of Medicine

• MEDLINE is available free via Pubmed

• MEDLINE is sold to many vendors, like Ovid, who search the DATA in MEDLINE through different search engines

Database ProducerDatabase Producer

Medline

EMBASE

CINAHL

VendorsVendors

Ovid

Pubmed

Dialog

LEXIS/NEXIS

Personal PC

Library PC

PubMed: Clinical Queries

• Pre-filtered searching

• Search on questions of– Therapy– Diagnosis– Etiology– Prognosis

• Or, search for Systematic Reviews

Limit any of the categories to sensitivity (broad) or specificity (narrow)

Sensitivity vs. Specificity

SPECIFICITY• Narrower search• More relevant items• Possibility of losing

some useful citations

• LESS information to weed through

SENSITIVITY• Broader search• More irrelevant

items• Less chance of

losing useful citations

• MORE information to weed through

OVIDOVID

• Most librarians favor this search engine

• Allow you to tailor your search

Design & Execute Search StrategyDesign & Execute Search Strategy

• MeSH terms• How articles are indexed• Predefined categories from the National

Library of Medicine

• Explode• MeSH + all items underneath

• Focus• MeSH only

Design & Execute Search Strategy

• Limiters• Limit by languages, human, age, etc

• Hedges• “Clinical trials” for therapy• “Sensitivity” & “Specificity” for diagnostic tests• “Cohort studies” for prognosis

More on Searching TipsMore on Searching Tips

• “Term$”• Search engines will look for occurrences of the word

with any combination of letters following. (Ex: “Hospital$” will include hospitalization, hospitals, hospitalized)

• “Exp” for “explode”

• “Term.tw.”• Allow search engine to look in the titles and abstract

More on Searching Tips

• “Term.af”• Search all fields

• “..pg term”• Perge/delete

For More Searching TipsFor More Searching Tips

Ask your librarians!

OutlineOutline• Why is EBM important?

• What is EBM

• Understand foreground vs. background questions

• Formulate PICO questions

• Find resources, do searches

• Validity of an article

• Appraise “Therapy” articles

• Appraise “Diagnostic” articles

Appraise (Validity)Appraise (Validity)

• http://www.cebm.utoronto.ca/teach/materials/dx.htm

• http://www.cebm.utoronto.ca/teach/materials/therapy.htm

Validity of Therapy ArticlesValidity of Therapy Articles

• Randomized?

• Blinded?

• Were the subjects similar at the start of the trial?

• Were all subjects treated the same except for the treatment of interest?

• Were all patients accounted for at the end of the trial?

Validity: Validity: 1. Were the Subjects Randomized?1. Were the Subjects Randomized?

“Methods” section

Validity: 2. Were They Blinded?

“Methods” section

Validity: 3. Were the Subjects Similar at

the Start of the Trial?

“Results” section

Validity: 4. Were All Subjects Treated the Same except for the Treatment of Interest?

“Methods” section

Validity: Validity: 5. Were All Patients Accounted 5. Were All Patients Accounted

for at the End of the Trial?for at the End of the Trial?

“Results” section

Validity of the ArticleValidity of the Article

• Was the study randomized? • Were the subjects blinded?

• Were the subjects similar at the start of the trial?

• Were all subjects treated the same except for the treatment of interest?

• Were all patients accounted for at the end of the trial?

OutlineOutline• Why is EBM important?

• What is EBM

• Understand foreground vs. background questions

• Formulate PICO questions

• Find resources, do searches

• Validity of an article

• Appraise “Therapy” articles

• Appraise “Diagnostic” articles

Step 4: Appraise “Therapy” StudyStep 4: Appraise “Therapy” Study

• Therapy study– Number Needed to Treat (NNT) to prevent an

adverse outcome

• Mathematically– NNT = 1/ARR (absolute risk reduction)

• ARR = Risk of adverse outcome in the control group (CER) – those in the experimental group (EER)– ARR* = CER** – EER***

*ARR: Absolute risk reduction**CER: control event rate***EER: experimental event rate

More on NNT

• Variables must be dichotomous• Cannot calculate NNT from continuous

variables

• For NNT to be meaningful, the difference between the 2 groups must be significant

Let’s Calculate NNT

CER EER ARR NNT

50% 20%

3% 2.5%

30% 25%

CER: Control Event Rate EER: Experimental Event Rate

ARR: Absolute Risk Reduction NNT: Number Needed to Treat

NNT=1/ARR

30% 3

0.5% 200

5% 20

Step 4: Appraise “Therapy” ArticlesStep 4: Appraise “Therapy” Articles

• ARR = CER – EER• NNT = 1/ARR

• ARR = 52.6-37.5 = 15.1

• NNT = 1/15.1 x 100 = 7

Step 5: Decision MakingStep 5: Decision Making

Atrovent– Low cost medication– Minimal to no side effects– Every 7 children with severe asthma

exacerbation, 1 hospitalization will be prevented

The benefits outweigh the risks. Therefore it would be worth treating the child with Atrovent in the emergency department.

OutlineOutline• Why is EBM important?

• What is EBM

• Understand foreground vs. background questions

• Formulate PICO questions

• Find resources, do searches

• Validity of an article

• Appraise “Therapy” articles

• Appraise “Diagnostic” articles

Appraise (Validity)Appraise (Validity)

• http://www.cebm.utoronto.ca/teach/materials/dx.htm

• http://www.cebm.utoronto.ca/teach/materials/therapy.htm

Validity of Diagnostic Articles

• Was there an independent, blind comparison with a reference (“gold”) standard of diagnosis?

• Was the diagnostic test evaluated in an appropriate spectrum of patients (like those in whom it would be used in practice)?

Validity of Diagnostic Articles

• Was the reference standard applied regardless of the diagnostic test result?

• Was the test (or cluster of tests) validated in a second, independent group of patients?

Disease + Disease -

Test + A

True +

B

False +

Test - C

False -

D

True -

A + B

C + D

A + C B + D

Sensitivity & SpecificitySensitivity & Specificity

• SensitivityThe proportion of people with the target disorder (A+C) who have a positive test (A)

A/(A+C)

• SnNoutIf a test has a high sensitivity, a negative result rules out the disease

Disease +Disease + Disease -

Test +Test + AA

True +True +

B

False +

Test - C

False -

D

True -

A + B

C + D

A + CA + C B + D

Sensitivity & SpecificitySensitivity & Specificity

• Specificity

The proportion of people without the disease (B+D) who have a negative test (D)

D/(B+D)

• SpPin

When a test has a high specificity, a positive result rules IN the diagnosis

Disease + Disease -Disease -

Test + A

True +

B

False +

Test -Test - C

False -

DD

True -True -

A + B

C + D

A + C B + DB + D

Predictive ValuesPredictive Values

• Positive Predictive Value

Proportion of people with a positive test (A+B) who have the target disorder

A/(A+B)

• Negative Predictive Value

Proportion of people with a negative test (C+D) who are free of the disease (D)

D/(C+D)

Disease + Disease -

Test + AA

True +True +

B

False +

Test - C

False -

DD

True -True -

A + BA + B

C + DC + D

A + C B + D

What is Likelihood Ratio?What is Likelihood Ratio?

A likelihood that a given test result would be expected in a patient w/ the target disorder compared w/ the likelihood that the same result would be expected in a patient w/o the target disorder

Likelihood RatioLikelihood Ratio• Likelihood ratio (LR) of 1 means

The test result identifies patients w/ the disorder at the same rate as identifying patients w/o the disorders

• LR >1 The test result identifies patients w/ the

disorder at a faster rate; likelihood goes

• LR <1 The test result identifies patients w/ the

disorder at a slower rate; likelihood goes

LR of 3 means:

The result is 3 times more likely to occur in a patient w/ the target disease than in a patient without.

In general, the LR for a test result is the percentage of “disease +” patients identified by that result divided by the percentage of “disease –” patients so identified.

Dis + Dis - Rate Dis +

Rate Dis -

LR

Result A a b

Result B c d

Total X Y

What rate are Dis + patients identified by Result A?

What rate are Dis - patients identified by Result A?

What is the LR for Result A?

a/X b/Y(a/X)

(b/Y)

c/X d/Y(c/X)

(d/Y)

Disease + Disease -

Test + A

True +

B

False +

Test - C

False -

D

True -

X = A + C Y = B + D

At what rate are Dis + patients identified by a “+ test” ?

At what rate are Dis – patients identified by a “+ test”?

What is the LR for a “positive” test result?

A/XA/X

B/YB/Y

(A/X) ÷ (B/Y)

Positive Likelihood Ratio (LR +)Positive Likelihood Ratio (LR +)

LR + = sensitivity/(1 – specificity)

= True pos rate/False pos rate

Disease + Disease -

Test + A

True +

B

False +

Test - C

False -

D

True -

X = A + C Y = B + D

At what rate are Dis + patients identified by a “- test” ?

At what rate are Dis – patients identified by a “- test”?

What is the LR for a “negative” test result?

C/XC/X

D/YD/Y

(C/X) ÷ (D/Y)

Negative Likelihood Ratio (LR-)Negative Likelihood Ratio (LR-)

LR - = (1 – sensitivity)/specificity

= False neg rate/True neg rate

LR+ LR-

1 – 2 V PoorV Poor 1 – 0.5

2 – 5 PoorPoor 0.5 – 0.2

5 – 10 FairFair 0.2 – 0.1

10 – 20 GoodGood 0.1 – 0.05

> 20 V. GoodV. Good < 0.05

Guide to Likelihood ratios

Why Do We Care About LR?Why Do We Care About LR?

Allow us to think about pre-test and post-test probabilities so that we can personalize the test to our patients

Test threshold

Treatment threshold

0% 100%

No test needed

Further testing needed

Testing completed;

treatment starts

Test & Treatment Thresholds in Test & Treatment Thresholds in the Diagnostic Processthe Diagnostic Process

Step 1: Pre-test ProbabilityStep 1: Pre-test Probability

• Prevalence

The number of events (instances of a given disease or other condition) in a given population at a designated time.

It is not a rate.

It is used as pre-test probability

It is the number of “disease +” patients to the “total”

Pre-test ProbabilityPre-test Probability

5 sources to determine this: Clinical experience Regional or national prevalence statistics Practice databases Original report on accuracy of the test Studies that calculate pre-test probability

Step 3: Calculate LRStep 3: Calculate LR• Pretest probability is 30%

• LR+ = ?

Culture + Culture -

Test + 65 18

Test - 10 72

Total 75 90Rap

id S

trep

LR + = TP rate/FP rate = (65/75) (18/90) = 4.35

Am

ir, 1994

LR - = FN rate/TN rate = (10/75) (72/90) = 0.16

Fagan NomogramFagan Nomogram

Pre-test probability on the left hand column

LR in the middle

Post-test probability on the right hand column

TranslationTranslation

• You see a 4 yo child who comes in c/o sorethroat, no other complaints

• Your pre-test probability is that she has a 30% chance of having strep throat

• You wonder how good is rapid strep

• You look up an article (Amir, 1994) and find its sensitivity (86.5%) and specificity (80%)

TranslationTranslation

• Your patient rapid strep is positive

• You calculated the LR + to be 4.35

• You determined the post-test probability to be 66%

• Is that enough of a threshold for you to treat or would you rather do more test?

Test threshold

Treatment threshold

0% 100%

No test needed

Further testing needed

Testing completed;

treatment starts

Test & Treatment Thresholds in Test & Treatment Thresholds in the Diagnostic Processthe Diagnostic Process

A Different Article Showed…A Different Article Showed…

• What if the LR+ is 12?

• What is the post-test probability?

““Diagnostic” Summary…Diagnostic” Summary…

• Use the 2 x 2 table to determine likelihood ratio (LR)

• LR + = True pos rate/False pos rate

• LR - = False neg rate/True neg rate

• Use Fagan Nomogram

OutlineOutline• Why is EBM important?

• What is EBM

• Understand foreground vs. background questions

• Formulate PICO questions

• Find resources, do searches

• Validity of an article

• Appraise “Therapy” articles

• Appraise “Diagnostic” articles

Knowledge is of two kinds; we know a subject ourselves; or we know where we can find information upon it.

- Samuel Johnson