Systematic Review and MetaSystematic Review and Meta--Analysis: Analysis: ApplicationsApplications

Susan D. Ross MD, FRCPCBoston MA

1

Systematic Review and Meta-Analysis: Applications

ISPOR Distance Learning Module 2

Susan D. Ross MD, FRCPC

Boston MA

2

Learning Objectives

To appreciate the pros and cons of using systematic reviews and meta-analyses to inform healthcare policy and practice decisions re: the efficacy, effectiveness, and safety of healthcare interventions.

To learn how systematic reviews and meta-analyses are and will be used for drug development and commercialization activities by industry.

3

Who uses systematic reviews?

Policy makers:

CMS (Centers for Medicare & Medicaid Services) US federal agency which administers Medicare, Medicaid, and the State Children's Health Insurance Program

NICE (National Institute for Clinical Excellence ) UK organization responsible for providing national guidance on the promotion of good health and the prevention and treatment of ill health.

4

CMS Follows General Principles of EBMAssess the Evidence re: “reasonable and necessary” care =

Published and non-published studies, expert opinion, technology assessments, professional societies, recommendations from the Medicare Coverage Advisory Committee (MCAC)

Key areas of focus include:

Methodological considerations, including study design, implementation, and analysis

Relevance of chosen outcomes; preference for patient reported outcomes

Generalizable to the Medicare population

Qualitative assessment of net risks and benefits based on individual studies

5

CMS Example: Bariatric surgery

August 2004 – November 2004: Coverage Analysis Group prepared an evidence review and summary of evidence for presentation to MCAC

May 2005: CMS opened a national coverage determination (NCD) at the request of the American Society for Bariatric Surgery, the American Obesity Association, Ethicon Endo-Surgery Inc. (a Johnson & Johnson company), INAMED Corp., U.S. Surgical, a division of Tyco Healthcare Group LP, and Transneuronix Corp.

February 2006: CMS has determined that the evidence is adequate to conclude that certain open and laparoscopic procedures are reasonable and necessary for Medicare beneficiaries who have a body-mass index (BMI) > 35, have at least one co-morbidity related to obesity, and have been previously unsuccessful with medical treatment for obesity.

6

But..Frequently Unanswered Questions in CMS Decision-making

- Cost-effectiveness?

- Adverse events?

- Off-label uses or new combinations of approved uses?

- Risks and benefits in subgroups?

- Outcomes for patient and provider types excluded from trials?

- Comparative effectiveness?

- Outcomes not measured in trials?

- Clinical utility of diagnostic tests?

- Surgery, devices and other technologies with limited regulatory review?

7

NICE

New medicines are licensed by the Medicines and Healthcare products Regulatory Agency/European Medicines Evaluation Agency.

NICE commissions independent academics to review (and Bayesian meta-analyze) published evidence on selected new & existing drugs & interventions.

NICE issues guidance on the clinical and cost effectiveness of Rx for the NHS in England and Wales.

Primary Care Trusts (local health bodies) then implement the guidance by funding Rx to patients when clinicians recommend it.

8

NICE Example: Herceptin

NICE system has led to rationing of (delayed access to) new anticancer drugs.

Herceptin (£20,000/year) was approved 1998 in the U.S. for advanced breast cancer.

EMEA approval 2000.

But not until March 2002 was NICE guidance released, for HER2+advanced breast cancer. And even then, patient access was spotty, due to variable funding decisions from UK PCTs.

9

Who uses/needs systematic reviews?

Payers/formularies, e.g.:

States

BC/BS

Kaiser-Permanente

AMCP

10

States use systematic reviews

Oregon Health & Science University (OHSU)

11

Oregon DERP

Compares efficacy of drugs within each of the 25 most expensiveclasses (the top 25 classes accounting for >50% of all pharmaexpenditures by states).

Originally a partnership between state of Oregon and Oregon EPC….now by Center for Evidence-based Policy at the OHSU…several EPCs continue to develop and update the 25 Systematic Reviews.

Project will “make available to governments, businesses, non-profit organizations, and citizens the best existing synthesis of global research on the relative effectiveness of similar medications. This research will allow these entities to determine for themselves how best to obtain the most value for their pharmaceutical expenditures”.

12

Examples from Oregon DERP

2002: Statins

all 5 statins were similar wrt Cholesterol lowering; only some had clinical outcomes; only 1 was generic – that was chosen for preferred drug lists.

2005: Triptans: Oregon and Washington used the same review of Triptans to reach different conclusions

Oregon accorded priority to the outcome of patients being pain free at 2 hours, its policymakers chose rizatriptan

Washington chose both rizatriptan and sumatriptan after considering a broader array of clinical end-points.

2007: 17 states and 2 non-profits using DERP results for drug coverage policy for their Medicaid programs

13

BC/BS uses (and produces) systematic reviews

Technology Evaluation Center (AHRQ EPC, with Kaiser Permanente collaboration)

5 Criteria to assess whether a technology improves health outcomes (e.g. length of life, quality of life, functional ability):

1. The technology must have final approval from the appropriate governmental regulatory bodies.

2. The scientific evidence must permit conclusions concerning the effect of the technology on health outcomes.

3. The technology must improve the net health outcome

4. The technology must be as beneficial as any established alternatives.

5. The improvement must be attainable outside the investigational settings.

14

Kaiser Permanente uses systematic reviews…

Kaiser Permanente is the nation's largest HMO

Nonprofit, founded in 1945

9 million members in 9 states & D.C.

Develop interregional guidelines, using evidence-based methods and teams who are expert in searching, summarizing, and critically appraising the medical literature. Kaiser Permanente invests > $80m of its $17b annual budget in research, including $12.3m for its care management institute, to "synthesize knowledge about the best clinical approaches and create, implement, and evaluate effective and efficient health care programs."

Stiefel M, et al. Kaiser Permanente's national integrated diabetes care management program. In Fox DM, Oxman AD, eds, Informing judgment: case studies of health policy and research in six countries. New York: Milbank Memorial Fund, 2001.

15

AMCP uses systematic reviews…

Academy of Managed Care Pharmacy –

In 2000, AMCP created the Format for Formulary Submissions (“AMCP” dossiers), requiring drug manufacturers to provide:

detailed product descriptions

discussion of potential problems

actual or anticipated off-label uses

data comparing the drug to other medications

reviews & summaries of key published and unpublished clinical studies

costs

spreadsheet model for predicting plan-specific outcome

16

AMCP uses systematic reviews…

By 2008:

health systems that cover approximately 150 million Americans have adopted the AMCP Format or a similar decision-making process, including national and regional managed health care systems, PBMs, the DoD, hospitals and state Medicaid agencies.

AMCP dossier Format is now viewed as an industry standard.

17

Patients use systematic reviews (see Best Buy Drugs) …

18

Providers use systematic reviews…

Witness:

PDA’s ubiquitous

EMR’s upsurge

Practice guidelines

Cochrane Collaboration (www.cochrane.org) - Library of systematic reviews

See: Sackett DL, Richardson WS, Rosenberg WMC, Haynes RB. Evidence-based Medicine: How to Practice and Teach EBM. London: Churchill Livingstone; 1997.

19

Providers use systematic reviews…

But not as much as they should…..

Witness:

Geographic variations in care, costs, and outcomes

Low level of compliance with evidence-based guidelines

Translating evidence to practice is now the biggest challenge of EBM.

20

What about Pharma use of systematic reviews?

Clinical R&D

1. To assess trial feasibility 2. To replace a 2nd RCT for sNDA

Risk Management

3. Cumulative meta-analyses for safety monitoring

Commercialization

4. To compare directly or indirectly drug efficacy or safety

Life-Cycle management

5.Efficient portfolio management (cumulative meta-analyses)

21

Case 1. Trial Feasibility:

Asthma

Question:

Will a planned multicenter phase 4 trial ($10 million estimate) succeed in proving that drug P is better than drug Z? (i.e., is efficacyof drug P superior to efficacy of drug Z?)

EBM tool:

Meta-analysis of all sponsor’s trials of drug P vs. SBA drug Z.

Answer:

NO!

22

Drug P or Z vs. Placebo: All Outcomes

Meta-Analyses of Effect Size and 95% C.I.

2 4 60Favors Treatment

-6 -4 -2Favors Placebo

DerSimonian & Laird Random Effects Model

#Study/Outcome Treatment Arms #PtsFEV 1 Z1 3 1042FEV 1 P1 4 B05

AM PEFR Z 3 1042AM PEFR P1 4 B05

PM PEFR Z1 3 1042PM PEFR P1 4 2B05

β agonist Z 3 1042

β agonist P1 4 B05

23

Drug P or Z vs. Placebo: All Outcomes, Adults

FEV1 - All 5REV1 - Adults only 3

A.M. PEFR - All 5A.M. PEFR - Adults only 3

P.M. PEFR - All 4P.M. PEFR - Adults only 2

β agonist - All 4

β agonist - Adults only 22 4 60

Favors Treatment-6 -4 -2

Favors PlaceboDerSimonian & Laird Random Effects Model

Sensitivity Analysis Adult Patients OnlyOutcome #Study/Arms

24

Drug P or Z vs. Placebo for FEV-1 By dose

P1 282P1 507P1 871P1 701P1 all dose 2361

2 40Favors Treatment

-4 --6Favors Placebo

DerSimonian & Laird Random Effects Model

Meta - Analyses of Effect Size and 95% C.I.Treatment # of Patients

Z1 88Z1 138Z1 234Z1 1182Z1 1274Z1 87Z1 81Z1 all dose 3084

25

Case 2. Replace a 2nd RCT for sNDA:

PPI in gastric ulcer

Question:

Can we avoid a clinical trial by using existing data in a sNDA?

EBM tool:

Meta-analysis of sponsor’s unpublished RCTs

Answer:

Efficacy proven in meta-analyses. Trial avoided. FDA success.

Tunis S, Sheinhait IA, Schmid C, Bishop DJ, Ross SD. The efficacy of lansoprazolecompared to histamine-2 receptor antagonists in healing gastric ulcers: a meta-analysis. Clinical Therapeutics 1997; 19: 743-757.

26

PPI in gastric ulcer

*H2RA = histamine-2 receptor antagonist

**REM = random effects model

***FEM = fixed effects model

Bayes REM** FEM*** Bayes REM** FEM*** Bayes REM** FEM***

Estimate 1.33 1.32 1.36 0.17 0.17 0.17 2.00 2.00 2.00

(95% CI) (1.19-1.49) (1.21-1.45) (0.24-1.49) (0.11-0.22) (0.12-0.22) (0.12-0.22) (1.59-2.52) (1.63-2.45) (1.63-2.45)

Estimate 1.12 1.12 1.14 0.10 0.09 0.09 4.67 4.47 5.20

(95% CI) (1.06-1.19) (1.07-1.17) (1.08-1.20) (0.05-0.14) (0.05-0.14) (0.06-0.13) (1.49-2.63) (1.54-2.57) (1.53-2.56)

PPI vs.H2RA Healing in Randomized Patients

Risk Ratio Risk Difference Odds Ratio

4 W

eeks

8 W

eeks

27

PPI vs. H2RA Healing at 4 Weeks

Intention to Treat

01020304050607080

1 2 3 4 5 6 7 8 9 10 11 12

Study ID

Hea

ling

Rat

es (%

)

PPI

H2RA

28

Risk Ratios: PPI versus H2RA Healing at 4 Weeks

Study Year #Pts1 Aoyama 1995 1072 Della Fave 1992 683 Fludas 1995 695 Takemoto 1992 1456 Okai 1995 247 Grymbowski 1990 1068 Peters 1990 4429 Facklam 1992 174

10 Sakaguchi 1995 1811 Takemoto 1990 33612 Colin 1990 15013 Tsuji 1995 16

OVERALL 1655

0.1 0.2 0.5 1 2 5 10

Z=5.95 2P<0.00001

Favors H2RAs Favors Lansoprazole

Z=5.95 2P<0.00001

29

Case 3. Cumulative meta-analyses for safety monitoring

COX-II and myocardial infarction

Question:

Could safety problems with COX-II’s have been detected earlierthan they were?

EBM tool:

Cumulative meta-analysis (real-time) of all published and unpublished RCTs

Answer:

YES (probably)!

30

Cumulative Meta-analysis (post hoc): Vioxx & RR of MI [“our findings indicate that rofecoxib should have been withdrawn several years earlier”]

Juni, et al. Lancet 2004;364:2021-29

.

31

Cumulative Meta-analysis (chronologically ordered RCTs)

• Perform a new statistical pooling every time a new

randomized controlled trial (RCT) becomes available. • Impact of each study on the pooled estimate is assessed. • Reveals (temporal) trend towards superiority of the treatment

or the control, or indifference. • Performed retrospectively, the year when a treatment could

have been found to be effective, at a given significant level (typically P < 0.05), could be identified.

• Performed prospectively, effective treatment may be

identified at the earliest possible moment.

32

Study 1

Study 2

Study 3

Study 4

Cumulative M-A 1

Cumulative M-A 2

Cumulative M-A 3

Pool Studies 1 to 2

Basic Method of Cumulative Meta-AnalysisStudies ordered

chronologically orby covariates

Pool Studies 1 to 4

Pool Studies 1 to 3

Study n-1 Cumulative M-A n-2

Study n Cumulative M-A n-1 Pool Studies 1 to n

Pool Studies 1 to n-1

33Favors Treatment Favors Control

Study Year Pts 1 Flectcher

Odds Ratio 95% CI

Intravenous Streptokinase Therapy for Acute Myocardial Infarction

0.5 1 2

P < 0.00001Favors Treatment Favors Control

0.01 1 1000.1Odds Ratio 95% CI

Fixed Effects Model (Mantel-Haenszel)

10

2 Dewar 3 European 1 4 European 2 5 Heikinheimo 6 Italian 7 Australian 8 Franfurt 9 NHLBI SMIT10 Frank11 Valerie12 Klein13 UK-Collab14 Austrain15 Australian 216 Lasierra17 N Ger Collab18 Witchitz19 European 320 ISAM21 GISSI-122 Olson23 Baroffio24 Schreiber25 Cribier26 Sainsous27 Durand28 White29 Bassand30 Vlay31 Kennedy32 ISIS-2

2365

232962

33 Wisenberg

OVERALL 36974

1388170922262432253926472738276133564084431443384821487951946935

18647186991875818796188401893819002192211932819353197213690836974

Pts1959 231963 421969 1671971 7301971 4261971 3211973 5171973 206

1974 107 1975 1081975 911976 231976 5951977 7281977 2301977 241977 4831977 581979 315 1986 17411986 117121986 52

1986 59 1986 381986 44

1986 981987 64 1987 219 1987 1071988 251988 3681988 17187

1988 66

34Favors Treatment Favors Control

Textbook / ReviewRecommendations

CumulativeYear RCTs Pts

1960

1965

1970

1975

1980

1985

1990

1 29

2 65

3 149

4 316

7 1793

10 2544 11 2651 15 3311

21

5

1 10

1 2

2 8

7

8

1 12

1 8 4

1 7 3

5 2 2 1

15 8 1

6 1

Rou

tine

Spe

cific

Rar

e / N

ever

Exp

erim

enta

l

Not

Men

tione

d

Odds Ratio (Log Scale)

17 3929 22 5452

23 5767

Thrombolytic Therapy for Acute Myocardial Infarction

33 6571 30 6346 27 6125

43 21059 54 22051 65 47185 67 47531 70 48154

0.5 1 2

P < 0.01

P < 0.001

P < 0.00001

35

Case 4: Indirect Comparisons

.Questions: Can I compare treatments (A vs. B) when no direct comparisons exist?

A ← ← ← ? → → → B

CEBM tool: meta-analysis

Answer: Yes!

36

Indirect comparisons

Example:

Systematic review & meta-analysis* of published RCTs of GPIIb/IIIainhibitors in PCI patients

Meta-analysis of OR for 30 day death or non-fatal MI for each GPIIb/IIIa inhibitor

Ross S, Klawansky S, Allen IE. Indirect comparisons of drugs using meta-analysis: validation of results. Value in Health 2001; 4: 55.

37

Results: 30 day Death/MI = OR 1.4 (advantage abciximab)(normalize OR’s to permit a simple hypothesis test on significance of Δ)

ODDS RATIO

CO

MPA

RIS

ON

GR

OU

P

38

Indirect comparison validation

TARGET*

head-to-head comparison of tirofiban vs. abciximab in PCI

OR for 30 day D/MI = 1.3 (p=0.04) (advantage abciximab)

* Topol E. TARGET (Do Tirofiban and ReoPro Give Similar Efficacy Outcomes Trial) American Heart Association 2000: 401.

39

Beyond pair-wise indirect comparisons…

Question: Can I compare treatments A, B, C…. to each other, all at once?

EBM tool: Network meta-analysis (aka, Mixed Treatment Comparisons, “umbrella” reviews – Cochrane)

Answer: Yes!

40Copyright restrictions may apply.

Psaty, B. M. et al. JAMA 2003;289:2534-2544.

Network Meta-analysis: First-Line Antihypertensive Drug Rx and Health Outcomes

41

Case 5. Efficient portfolio management

Question: Can I monitor “real-time” results of all relevant studies (RCTsand non-RCTs) as soon as they are available? (and can I make course corrections if needed?)

EBM tool: Prospective cumulative meta-analysis

Answer: Yes, you should be.

42

Planning a Prospective Cumulative Meta-Analysis

Specific hypothesis objectives

Eligibility criteria of trials and of patients within trials

Outcome definitions

Subgroups

Analysis plan

Prospective registration of trials

Collection and analysis of IPD

Publication of main results

Permit additional questions (or trials) only if data are still blinded

Simes RJ. Am J Cardiology; 1995; 76: 899-905.

43

Cumulative Meta-analyses

Examples:

Cholesterol reduction (Prospective Pravastatin Pooling (PPP) Project , Cholesterol Treatment Trialists' )

FICSIT (Frailty and Injuries: Cooperative Studies of Intervention Techniques intervention efficacy in reducing falls and frailty in elderly patients)

Atrial fibrillation antithrombotic prophylaxis (SPORTIF III and V)

44

Summary: Applications of Systematic reviews in decision-making

Pros

Really can provide best available evidence

Transparent, comprehensive, & current

Improves process of decision-making

Improves decisions

Improves outcomes (?)

45

Summary: Applications of Systematic reviews in decision-making

Cons:

Application & interpretation of results may not be consistent (or transparent) across stakeholders (since local resources, values, & preferences also differ)

Potential for misapplication of results to support cost-containment strategies

“Easy” to mislead non-sophisticated audiences (hide behind EBM shield)

Only as good as the ‘raw material’ : Most medical practice is not supported by published literature; Safety info from literature is poor; Off-label use info is variable; studies of comparative effectiveness are infrequent) etc.

Labor-intensive to do & to update

Frequently only good for “first-order” effects (i.e., not subgroups) & no costs or utilization info

46

Systematic review & Meta-analysis: Applications

Thank you!

Send Questions or Comments to: sdross720@gmail.com

47

Resources

Systematic reviews and evidence synthesis

1. Cook DJ, Mulrow CD, Haynes RB. Systematic reviews: synthesis of best evidence for clinical decisions. Ann Intern Med 1997;126:376-380

2. Users guide to the medical literature. Series in JAMA. http://pubs.ama-assn.org/misc/usersguides

3. http://www.cebm.net (Centre for Evidence-Based Medicine, Oxford, UK )

4. http://www.cochrane.org/resources/handbook/

48

Resources

Indirect treatment comparisons

1. Bucher HC, et al. The results of direct and indirect treatment comparisons in meta-analysis of randomized controlled trials. J Clin Epidemiol. 1997; 50:683-691.

2. McAlister FA, et al. JAMA 1999; 282: 1371-1377.

3. Caldwell DM et al. Simultaneous comparison of multiple treatments: combining direct and indirect evidence. BMJ 2005; 331: 897-900.

4. Eliot W & Mayer P. Incidence of diabetes in clinical trials of antihypertensive drugs: a network meta-analysis. Lancet 2007; 369: 207-.

5. Lu G & Ades AE. Combination of direct and indirect evidence in mixed treatment comparisons. Stats Med 2004; 23: 3105-3114.

6. Lumley T. Network meta-analysis for indirect treatment comparisons. Stats Med 2002; 21: 2313-2324.

49

Resources

Bayes:

1. Berger, JO (2000) “Bayesian Analysis: A Look at Today and Thoughts of Tomorrow,”Journal of the American Statistical Association. 95(452) 1269-1276.

2. Berry, D (2006) “Bayesian Statistics,” editorial in Medical Decision Making, Sept-Oct.

3. Raudenbush, SW & Bryk, AS (1985) “Empirical Bayes Meta-Analysis,” Journal of Educational Statistics. 10(2) 78-98.

4. Spiegelhalter, DJ, Myles, JP, Jones, DR, & Abrams, KR (1999) “An introduction to Bayesian methods in health technology assessment,” British Medical Journal. 319: 508-512.

5. Schmid CH. Using Bayesian Inference to Perform Meta-analysis. Evaluation & the Health Professions, Vol. 24, No. 2, 165-189 (2001)

50

Resources Individual Patient Data Meta-analysis

1. Stewart LA, Parmar MB. Meta-analysis of the literature or of individual patient data: Is there a difference? Lancet 1993; 341:418-422.

2. Steinberg K, Smith S, Stroup D, Olkin I, Lee N, Williamson G, and Thacker S. Comparison of Effect Estimates from a Meta-Analysis of Summary Data from Published Studies and from a Meta-Analysis Using Individual Patient Data for Ovarian Cancer Studies. Am J Epidemiol 1997; 145: 917–25.

Cumulative Meta-Analysis

1. Lau J, Schmid CH, Chalmers TC. Cumulative meta-analysis of clinical trials builds evidence for exemplary medical care. J Clin Epidemiol. 1995 Jan;48(1):45-57.

2. Berlin JA. Colditz GA.The Role of Meta-analysis in the Regulatory Process for Foods, Drugs, and Devices. JAMA. 1999;281:830-834.

3. Hu M, Cappelleri JC, KKG Lan. Applying the law of iterated logarithm to control type 1 error in cumulative meta-analysis of binary outcomes. Clinical Trials. 2007; 4:329-340.

4. Lan KKG, Hu M, Cappelleri JC. Applying the law of iterated logarithm to cumulative meta-analysis of a continuous endpoint. Statistica Sinica. 2003; 13:1135-1145.