From Clinical Trial Evidence to Practice Guidelines Lost in Translation

Sanjay Kaul, MD, FACCGeorge A. Diamond MD, FACC

Division of CardiologyCedars-Sinai Medical Center and

Geffen School of Medicine at UCLALos Angeles, California

From Clinical Trial Evidence to Practice Guidelines

Lost in Translation

Complexity of American Strategy in Afghanistan

“When we understand that slide, we’ll have won the war” - General McChrystal

Complexity of Evidence-Based Medicine

Lost in a jungle of evidence, we need a compass

Evidence to Guidelines Lost in Translation

Key Issues for Discussion

• Establish the scientific evidence- Appraise and synthesize the evidence

• Elucidate the clinical context - Clinical importance vs. statistical significance

- Clinically relevant weighted outcomes

• Encourage optimal processes of care- Quality initiatives- Reimbursement initiatives

Appraisal of EvidenceDesign and Methods

Quality• Important limitations - Study design or execution (bias)

- Lack of randomization

- Lack of concealment

- ITT principle violated

- Inadequate blinding

- Loss to follow-up

- Early stopping for benefit

- Inconsistency of results - Indirectness of results - Imprecision - Publication bias

Quality• Special strengths - Randomized, controlled, prospective,

double-blind trials - Large, consistent, and precise treatment effect

- RR<0.5 or >2.0 (large)- RR<0.2 or >5.0 (very large)

- Minimal confounding & bias

Synthesis of EvidenceACC/AHA Clinical Practice Guidelines

Class I(“Useful & Effective”)

(Benefit >>> risk)(Highly

recommended)

Class II (“Conflicting Evidence”)

Class III(“Not useful/

effective, may be harmful”)

(No benefit/Harm)(Not recommended)

IIa(Benefit >>risk)

(Reasonably recommended)

IIb(Benefit ? risk)

(May be considered)

Level A(Multiple randomized

clinical trials)

Level B(Single randomized

trial or nonrandomized

studies

Level C(Consensus opinion,

case studies, or standard of care)


Self-evident Truths

• Does empirical evidence trump expert opinion?

Tricoci P et al. JAMA 2009

Scientific Evidence Underlying The ACC/AHA Clinical Practice Guidelines

Level of Evidence A

4.9

4.9

19

11

9.7

0.3 (1/320)

23.6

6.1

6.4

22.9

12

13.5

15.3

26.4

11.7

0 10 20 30

Radionuclide imaging

PacemakerCABG

PCIVA/SCD

Valvular diseaseUA/NSTEMI

SV arrhythmia

Stable anginaSecondary prevention

PerioperativeSTEMI

PADHeart failure

AF

26.3

58.2

20

47.8

58.5

70.6

29.6

56.5

54.5

8.3

32

47.2

25.1

54.3

58.6

0 10 20 30 40 50 60 70 80

Radionuclide imaging

PacemakerCABG

PCIVA/SCD

Valvular diseaseUA/NSTEMI

SV arrhythmia

Stable anginaSecondary prevention

PerioperativeSTEMI

PADHeart failure

AF



Level of Evidence C

ACC/AHA Clinical Practice Guidelines Paucity of High-Quality Evidence

Class I(Benefit >>> risk)

(Highly recommended)

Class II Class III(Risk ? Benefit)

(Not recommended)

IIa(Benefit >>risk)


IIb(Benefit ? risk)

(May be considered)

Level A(Multiple

randomized clinical trials)

19% based on high-level evidence

41% of guidelines are

based on Class II recommendations

(”uncertain evidence”)Level B

(Single randomized trial

or nonrandomized studies

Level C(Consensus

opinion, case studies, or

standard of care)

48% of guidelines are based on level C evidence (“codification of expert opinion“)


Recommendations are largely developed from lower levels of evidence or expert opinion. “Exercise caution when considering recommendations not supported by solid evidence”

“…it seems unlikely that substantial change will occur because many guideline developers seem set in their ways. If all that can be produced are biased, minimally applicable consensus statements, perhaps guidelines should be avoided completely. Unless there is evidence of appropriate changes in the guideline process, clinicians and policy makers must reject calls for adherence to guidelines. Physicians would be better off making clinical decisions based on valid primary data”

Shaneyfelt and Centor, JAMA 2009


Caveat Emptor, Caveat Lector

Guidelines that are driven by scientifically documented, high-quality evidence are more likely to be accepted by the stakeholders, thereby

reducing the variability in care and improving the quality and cost of care

2009 ACC/AHA Focused Updates for STEMI/PCI Paucity of High-Quality Evidence




(Not recommended)

IIa(Benefit >>risk)


IIb(Benefit ? risk)

(May be considered)

Level A(Multiple


12% based on high-level evidence

50% of guidelines are

based on Class II recommendations

(“conflicting evidence”)Level B

(Single randomized trial

or nonrandomized studies

Level C(Consensus

opinion, case studies, or

standard of care)

44% of guidelines are based on level C evidence (“filtered expert opinion”)

Kushner FG, Hand M et al. 2009 Focused Updates, JACC/Circulation 2009

The Laws of Diminishing Objectivity in the Interpretation of Evidence

vehemence evidence-1

Peter McCullochThe Lancet, 2004;363;9004

vehemence eminence2




• Optimal processes of care- Quality initiatives- Reimbursement initiatives



• Effect size- Absolute risk difference (NNT or NNH) - Relative risk difference

Risk ratioOdds ratioHazard ratio

• Statistical certainty/precision- Hypothesis testing (P value)- Estimation (confidence interval)

• ? Clinical importance

Little or no explicit guidance

ACC/AHA Clinical Practice GuidelinesMetrics for Assessing Strength of Evidence

1

P value Effect Size Sample Size

Statistical significance Clinical importance!

Disconnect Between Statistical Significance and Clinical Importance

Boersma et al, Lancet 2002;359:189-1198.

GUSTO IV

PRISM

PRISM-Plus

PURSUIT

PARAGON A

PARAGON B

POOLED

Trial (IIb/IIIa) Placebo 2b/3a

7800

3232

1570

9461

1513

5169

28,745

N

8.0

7.0

11.9

15.7

11.7

11.4

12.5

8.7

5.7

8.7

14.2

10.3

10.5

11.3

Better Worse0.1 1 10

Death / MI at 30 daysRisk Ratio & 95% CI (%) (%)

0.91 (0.86, 0.99)P=0.015

P=0.339Breslow-DayHomogeneity

ARR = 1.2%RRR = 9%

Statistical Significance vs. Clinical ImportanceGP IIb/IIIa Inhibitors in UA/NSTEMI

What Does a P(ee) Value of 0.05 Mean?

• ‘Fisherian’ P value of 0.05 is arbitrary and originally based on n=30!

• Always demand a P value of <0.001 for a sample size > 200 as strong evidence against the null hypothesis of zero difference

Al Feinstein

The plain fact is that in 1925 Ronald Fisher gave scientists a mathematical machine for turning “baloney into breakthroughs”, and “flukes into funding”.

Robert Matthews

1

P value Effect Size Sample Size

Lack of statistical significance lack of clinical importance!

Disconnect Between Statistical Significance and Clinical Importance

Oler A et al, JAMA 1996;276:811-15

0.1 1.0 10

Risk Ratio & 95% CI ASA+UFH ASATrial NTheroux 243

RISC 399

ATACS 214

Holdright 285

Cohen 1990 69

Gurfinkel 143

1.6% 3.3%

1.4% 3.7%

3.8% 8.3%

27.3% 30.5%

0% 3%

5.7% 9.6%

ASA+UFH Better ASA Better

Overall 1335 7.9% 10.4%0.67 (0.44-1.02)

P=0.06

Death/MI

ARR = 2.5%RRR = 33%

Statistical Significance vs. Clinical ImportanceUnfractionated Heparin in UA/NSTEMI

• MDD (minimum detectable difference, “”)- The “minimum difference” the study is powered to detect- Utilized for sample size estimation

- May or may not reflect a clinically important difference

• MCID (minimum clinically important difference)The “minimum acceptable difference” to change the behavior of the clinician, patient, payer or policy maker, given the side effects, costs and inconveniences of therapeutic interventions

Statistical Significance vs. Clinical Importance

Guideline Criteria for Clinical ImportanceImpact of Outcome, Harm, and Cost on MCID

Very low Low Moderate HighHarmVery low Low Moderate HighCost

Mortality Irreversiblemorbidity

ReversibleMorbidity

SurrogateEndpoint

Outcomeseverity

Small Large

MCID (RRR)0% 50%

Statistical Significance vs. Clinical ImportanceMCID Threshold for UA/NSTEMI ACS

“In ACS, a relative reduction of 15% in recurrent clinical events has recently been considered clinically important (GUSTO I); this level is far below the perceived threshold that drove the sample size calculations for clinical trials just a decade ago. As we develop more incrementally beneficial therapies, it is likely that the minimally important clinical difference will become even smaller.”

Califf and DeMetsCirculation. 2002;106:1015

1.0

Risk Ratio (95% CI)

0.85

MCID

Statistically significant, clinically important

Statistically not significant, may be clinically important

Statistically significant, not clinically important

Statistically not significant, clinically not important A

B

C

D

E

Statistically significant, may be clinically important

MCID = minimal clinically important difference= 15% RRD

Sackett, D

Statistical Significance vs. Clinical ImportanceStrength of Evidence

Intervention Control(%)

Rx(%)

Summary risk ratio (95% CI)

PValue

NNT(95% CI)

Interpretation ofConfidence Intervals(MCID = 15% RRR)

Aspirin(N=2,856)

12.8 5.5 0.43 (0.33-0.56)

<0.01 14 (11-19)

Statistically significant and clinically important (E)

UFH(N=1,353)

10.4 7.9 0.67 (0.44-1.02)

0.06 44 (∞-18)

Statistically not significant, maybe clinically important (B)

Enoxaparin(Early invasive)

12.8 12.1 0.96 (0.88-1.05)

0.35 171 (∞-59)

Statistically not significant, clinically not important (A)

Clopidogrel(CURE)

11.4 9.3 0.82 (0.74-0.92)

<0.01 54 (35-120)

Statistically significant, maybe clinically important (D)

GP IIb/IIIa (Early invasive)

14.5 11.8 0.81 (0.70-0.94)

0.007 37 (21-139)


Statistical Significance vs. Clinical ImportanceClass I, LOE A Recommendations for UA/NSTEMI

Impact on Death or MI

Aspirin is the only intervention listed as a performance measure!







Endpoints in Cardiovascular Clinical TrialsMACE vs MICE

DeathCardiac arrestLarge MIDisabling Stroke Emergency CABG

Major Adverse Cardiac Events

“Hard” but infrequent Silent CK/Tn Release

RestenosisReinterventionRecurrent anginaRehospitalizationGroin hematoma

Minor Inconvenient Cardiac Events

“Soft” but prevalent

Cardioprotective Effects of AntihistaminesMeans to an End or an End to Means

PlaceboAntihistamine

% p

atie

nts

Death Recurrent MI Itching

p < 0.05p < 0.05

Cardioprotective Effects of StentingClinical Outcomes at 1 Year in Stent PAMI

16.9% 24.8%0%

5%

10%

15%

20%

25%

30%D/MI/Stroke/TVR

P < 0.005

Stent(N=449)

PTCA(N=444)

2.9 2.5

P = 0.7

MI

10.7

20.9

P < 0.0005

TVR

5.63.1

0

5

10

15

20

25Stent (N=452)

PTCA (N=448)

P = 0.07

Death

%

Individual Components

0.5 0.5

P = 0.83

Stroke

Benefit driven by the “least robust” but the “most prevalent” component

Validity of the Composite Endpoint

• Components should be of comparable frequency

• Components should be of comparable clinical importance

• Components should be comparably responsive to therapy

Montori VM et al. Br Med J 2005; 330:594-596

Compositeanalysis

0.00 1.00 2.00 3.00 4.00

TVR

Stroke

MI

Death

Odds ratio

1.81 (0.93-3.53)

1.17 (0.52-2.65)

0.99 (0.14-7.05)

0.45 (0.31-0.66)

0.62 (0.45-0.86)

OR (95% CI)

5.6%

2.9%

0.5%

10.7%

16.9%

Stent

3.1%

2.5%

0.5%

20.9%

24.8%

PTCA

Cardioprotective Effects of Stenting Validity of the Composite Endpoint in Stent PAMI

Cochran’s Q = 14.64Hetero P = 0.002I2 = 80% (46-92%)

Composite: Variable gradient in clinical importance, frequency and treatment effect across components

Cardioprotective Effects of StentingWeighted Analysis of Composite Endpoint

Weight of TVR

Glo

bal P

val

ueWeightsDeath = 1

MI = 1Stroke = 1

0.000.100.200.300.400.500.600.700.800.901.00

0.10 0.20 0.30 0.40 0.50 0.60 0.70 0.80 0.90 1.00

Composite endpoint becomes significant at a TVR weight of >0.7!




(Not recommended)IIa(Benefit >>risk)


IIb(Benefit ? risk)

(May be considered)

Level A(Multiple


Level B(Single randomized

trial or nonrandomized

studies

60mg prasugrelload ASAP, 10 mg daily x 12m

Level C(Consensus

opinion, case studies, or standard

of care)

Withhold prasugrel 7 days prior to CABG or surgery

Pts with h/o TIA or stroke; active bleeding

ACC/AHA Guideline RecommendationsPrasugrel During Primary PCI for STEMI

Kushner FG, Hand M et al. 2009 Focused Updates, JACC/Circulation 2009

Benefit-Risk Balance in TRITON (All ACS Cohort)1000 Patients Treated with prasugrel instead of clopidogrel

Prasugrel vs ClopidogrelBenefit

• 24 endpoints prevented - 3 CV deaths - 0 strokes - 21 nonfatal MIs - 4 PPMIs

- 17 MI events - 13 clinically relevant MIs

Risk • 30 excess TIMI bleeds - 2 bleeding deaths - 3 TIMI Major bleeds - 5 TIMI Minor bleeds - 20 TIMI Minimal bleeds

or• 29 excess moderate/severe bleeds - 2 bleeding deaths - 9 transfusions - 6 nonfatal serious bleeds - 12 nonfatal moderate bleeds

or• 17 excess serious bleeds• ? 3-6 excess cancer (1 cancer death)

Judgments about Strength of RecommendationPrasugrel for Patients with ACS Undergoing PCI

FACTORS COMMENTSBalance between desirableand undesirable effects “The net benefits are uncertain”

Quality of the evidence “Quality of the evidence is high.”

Patient values andpreferences

“All patients and care providerswould not accept efficacy-safetytrade-off.” Alternatives available.

Costs (resource use) “The cost is high for treatmentfor long duration.”

Does the evidence favor Class I (benefit >>> risk)recommendation for prasugrel?







Quality MattersLinking Guidelines Adherence and Mortality

Peterson et al, JAMA 2006;295:1863-1912

5.955.16 4.97

4.165.06

4.634.15

6.31

012345678

<=25% 25 - 50% 50 - 75% >=75%

Hospital Composite Quality Quartiles

% In

-Hos

p M

orta

lity

Adjusted Unadjusted

Every 10% in guidelines adherence 10% in mortality (OR=0.90, 95% CI: 0.84-0.97)

• Risk-adjusted hospital deaths declined by 0.7 percentage points (95% CI,

-1.7 to 0.3) in NSTE ACS patients.

• The rate of congestive heart failure and pulmonary edema decreased by 6.5% (95% CI, -8.4 to -4.7).

2.9%

13.0%

2.2%

6.1%

0%

5%

10%

15%

Death Heart FailureJul -Dec 1999 Jul-Dec 2005

Changes in Clinical Outcomes for NSTE ACS Patients

n = 2213

p = .02 for linear trend

p <.001

n =1566 n = 2228 n = 1564

GRACE: Outcome Measures over TimeNSTE ACS

Fox et al. JAMA. 2007; 297:1892-1900

ACC Improvement InitiativesContinuous Quality Improvement

Translating Science into Practice

Improvement• D2B• H2H• FOCUS

Measurement• NCDR

Implementation - “Bridge”• Quality Practice Assessment• Clinical Decision Support• Operation Management Tools

Guidelines/Standards• Guidelines• AUC / PM

PLAN

DO

STUDY

ACT

Education and Training

FuelHigh-quality

evidence

BoostImplementation

Design, process evaluation

The Role of Evidence-Based Guidelinesin Improving Clinical Practice

Turbocharging the Guidelines

• Number of recommendations: >250• Number of pages: 157• Number of figures: 21• Number of tables: 26• Last update: 2002• Writing committee members: 15• Reviewers: 40 (6 different layers from evaluation to publication)• Conflict-of-interest disclosure

- Writing committee members: 14/15- Reviewers: 30/40

J Am Coll Cardiol 2007; DOI:10.1016/j.jacc.2007.02.028

2007 ACC/AHA Guideline Recommendations Acute Coronary Syndromes

• Quality- Rigorous and standardized methodology (GRADE) - Emphasize clinical importance over statistical significance- Transparent and explicit benefit-risk assessment

• Efficiency- User-friendly and parsimonious (avoid the 160 page report)

• Timeliness- Keep pace with advances (annual updates)

• Dissemination- Direct clinical relevance (at point of care via EMR) - Guide and inform clinical practice (performance measures)- Financial incentives (evidence-based reimbursement)

Evidence to GuidelinesFramework for Refinement

• Firewall between systematic review & guideline development

• Multidisciplinary guideline developers: methodologists,

clinical content experts, patient representatives

• Avoid LOE C recommendations (best suited as “advisories”)

• Minimize conflicts of interest (COI) for writers/reviewers

• “Zero tolerance” COI policy for chairs

• PIs of guideline-relevant trials should only serve as advisors

Evidence to GuidelinesFramework for Refinement

Evidence-Based MedicineACC Improvement Initiatives

• Turbocharging guidelines (18 currently available, 9 in

development, 6 being updated)

• Transform and transfer guidelines at the point of care

- Just in time strategies (Vivisimo, Cardio Compass)

• Appropriate use criteria (Noninvasive imaging, CABG/PCI)

• Quality initiatives (D2B, H2H, FOCUS)

• Registries - NCDR (CathPCI, ICD, CARE, ACTION-GWG,

IMPACT, PINNACLE)

• Physician incentives (PQRI, ACO)

• Patient involvement (CardioSmart)

• Treat as “guides”, not “rules”

• Patient-specific, not disease-specific

• Pragmatic/assistive, not prescriptive/directive

• Flexible and adapted to local practice

• Based on empirical high-quality evidence, not

“codified” or “filtered” expert clinical opinion

• Drive the standard of care, not be driven by them

• Inform clinical judgment, not replace it

Framework for Increased Adherence to Clinical Practice Guidelines and to EBM

ClinicalJudgment

“Evidence-Based” Not “Evidence-Bound”Three Key Dimensions

Scientificevidence

Patient preference

Complexity of Evidence-Based Guidelines

Illusion of understanding? Illusion of control?

"Yes, I have tricks in my pocket, I have things up my sleeve. But I am the opposite of a stage magician. He gives you illusion that has the appearance of truth. I give you truth in the pleasant disguise of illusion."

Tennessee Williams(The Glass Menagerie)

Caveats in Interpretation of Meta-analysis

“Although it challenges logic that one could obtain new accurate information from the quantitative integration of a number of very diverse studies, the numerous meta-analyses published speak for themselves. Used in the proper setting, I think they can make a valuable contribution. The job of the Journal will be to ensure that those published are in this setting and are methodologically sound.”

J Am Coll Cardiol, 2008; 52:237-238

Anthony N DeMaria, MDEditor-in-Chief, JACC

Has the Journal lived up to its ideals?

• Separate systematic review from guideline development

• Multidisciplinary guideline developers: methodologists,

clinical content experts, patient representatives

• Avoid LOE C recommendations (best suited as “advisories”)

• Minimize conflicts of interest (COI) for writers/reviewers

• “Zero tolerance” COI policy for chairs

• PIs of guideline-relevant trials should only serve as advisors

Turbocharging the GuidelinesACC Improvement Initiatives

Do we practice evidence-based medicine or reimbursement-based medicine?

Fee for Service or Fee for Appropriate Service?

What is the Gold Standard?

PCI for Stable CADDisconnect Between Policy and Practice

• Number of PCIs performed for stable CAD

- ~ 500,000/yr at cost of $20K per PCI ($10 billion)

• Appropriateness of PCI- Presence of ischemic symptoms- Objective evidence of ischemia by stress testing- Failed trial of optimal medical therapy and lifestyle Rx

• The real-world practice- 20% of pts referred for PCI are asymptomatic (ACC-NCDR)- 30-50% of pts have not had a stress test (Topol / Lin et al.); untold (?60-70%) number of stress tests are “negative”- 30% of pts not taking anti-ischemic meds (Samuels et al.)

• Reimbursement for PCI

- $20K per PCI

• Score based on appropriateness- Presence of ischemic symptoms (1/3)- Objective evidence of ischemia by stress testing (1/3)- Failed trial of optimal medical therapy and lifestyle Rx (1/3)

• Sliding-scale reimbursement- 20% reward for a score of 1- 20% discount for a score of 2/3- 60% discount for a score of 1/3- 100% discount for a score of 0

Evidence-Based Reimbursement for Stable CADA Financial Incentive for Health Care Reform

Diamond and Kaul, Circulation Cardiovasc Qual Outcomes 2009; Archives of Int Med 2009

Symptoms Stresstest

Treatment Score Payment Patients Reimbursement

+ + + 1 $24K 15,000 $0.36B+ + - 2/3 $16K 85,000 $1.36B+ - + 2/3 $16K 45,000 $0.72B- + + 2/3 $16K 3,750 $0.06B+ - - 1/3 $8K 255,000 $2.04B- + - 1/3 $8K 21,250 $0.17B- - + 1/3 $8K 11,250 $0.09B- - - 0 0 63,750 $0B

Total 500,000 $4.8B

Evidence-Based Reimbursement for Stable CADA Financial Incentive for Health Care Reform

• 80% of patients complain of ischemic symptoms• 50% of patients undergo stress testing; 50% of these are ischemic• 15% of patients are receiving OMT

13% reduction in caseload and 52% reduction in reimbursement

Intervention Control(%)

Rx(%)

Summary risk ratio (95% CI)

PValue

NNT(95% CI)

Interpretation ofConfidence Intervals(MCID = 15% RRR)

Aspirin(N=2,856)

12.8 5.5 0.43 (0.33-0.56)

<0.01 14 (11-19)

Statistically significant and clinically important (E)

UFH(N=1,353)

10.4 7.9 0.67 (0.44-1.02)

0.06 44 (∞-18)


Enoxaparin(Early invasive)

12.8 12.1 0.96 (0.88-1.05)

0.35 171 (∞-59)

Statistically not significant, clinically not important (A)

Clopidogrel(CURE)

11.4 9.3 0.82 (0.74-0.92)

<0.01 54 (35-120)


GP IIb/IIIa (Early invasive)

14.5 11.8 0.81 (0.70-0.94)

0.007 37 (21-139)


Early statin(A-to-Z)

12.4 11.1 0.89 (0.74-1.07)

0.21 87(∞-33)


Statistical Significance vs. Clinical ImportanceClass I, LOE A Recommendations for UA/NSTEMI

Impact on Death or MI

Documents

From Clinical Trial Evidence to Practice Guidelines Lost in Translation