EPI-820 Evidence-Based Medicine

LECTURE 10: Meta-Analysis II

Mat Reeves BVSc, PhD

2

4. Analysis

• Several considerations• Primary focus/motivation of study?

– Summary effect or explore heterogeneity?

• What kind of data are you combining?• Dichotomous (categorical) (OR/RR, RD )• Continuous (effect size [diff/SD])• Diagnostic data (sensitivity and specificity)

• Fixed vs random effects model?

3

Analysis – Primary Goal

• What should be the primary goal?• To provide a summary estimate or explore of presence and

sources of heterogeneity? It depends.

• If studies are homogeneous then generate a summary estimate with 95% CI

– Much more likely to happen in RCT’s where randomization has helped control bias and confounding

• If studies are heterogeneous then focus of study should be to investigate the sources of this variability

– More likely to happen in observational studies where differences in populations, methods and uncontrolled bias and confounding are rampant. But also occurs in RCT’s.

4

Exploring heterogeneity

• Heterogeneity is the norm rather than the exception

• Heterogeneity can result from• Methodological differences• Biological differences

• Heterogeneity Statistics (Q)• Power generally low (because study N’s are typically small)• Power also affected by size of deviations between studies

• Statistical vs clinical heterogeneity • What is the size of the statistical heterogeneity? Does it make

sense? Could it have arisen due to random error? (chance)

5

Sources of clinical heterogeneity

• Clinical heterogeneity can be due to differences in: • study design or characteristics:

– hospital vs population-based observational designs

– DBPC vs open trials

– Study population (sources), study quality

• Selection criteria for subjects, treatments or follow-up• Sub-group responses (biological interaction) – esp. in RCT’s• Bias or confounding – esp. in observation studies

• Explored using stratification/sub-group analyses• See Bernal, 1998

6

RR of Vasectomy on Prostate CA Risk Effect of study characteristics/quality (Bernal, 1998)

Study Characteristic Summary RR 95% CIDesign

Cohort 1.1 0.8 - 1.5

CCS 1.4 1.0 - 1.8

Setting

Population 1.1 1.0 - 1.3

Hospital 2.0 1.4 - 2.9

Rating of control selection

Adequate 1.1 0.9 - 1.3

Inadequate 2.2 1.4 - 3.5

Rating of presence detection bias

Adequate 1.1 1.0 - 1.3

Inadequate 1.9 1.4 - 2.6

7

Fixed and random effects models

• Homogeneity and heterogeneity • Heterogeneity depends on the degree of between-study

variability in a group of studies.

• Fixed effects models:• consider only within-study variability. • assumption is that studies use identical methods, patients,

and measurements; that they should produce identical results - any differences are only due to within-study variation only.

• Answer the question: – “Did the treatment produce benefit on average in the studies at

hand?”

8

Random Effects models

• consider both between-study and within-study variability.

• assumption is that studies are a random sample from the universe of all possible studies.

• Answer the question: – “Will the treatment produce a benefit ‘on average’?”

• Note that random effects models do not “adjust for”, “account for”, or “explain” heterogeneity

– A random effects model does not therefore solve the problem of heterogeneity!

9

Fixed and random effects models

• Can give very different answers, and you can create examples where either model gives counterintuitive results (see Petitti, page 96)

– Random effects gives non-significant summary statistic for two studies that are each significant

– Fixed effects model gives the same confidence interval when you would expect a broader and narrower CI

• Usually, though, answers are similar. • Example: Comparison of 22 meta-analyses, fixed and

random effects models gave the same answer in 19/22. In 3 cases, fixed effects models were significant while random effects models were not (Berlin, 1989).

10

Fixed and random effects models

• Differences only arise when studies are not homogenous. • When there is significant heterogeneity, the between-study

variance becomes much larger than the within, and studies of different sample size receive relatively similar weight.

• When there is homogeneity, sample size dominates, and both models give similar results.

• Random effects models are more “conservative” and generate a wider confidence interval (because they add in the between-study variance).

• Random effects models also tend to give greater weight to small studies (which maybe more biased?)

11

What to do?, what to do?

• If homogenous, use fixed effects model– random will give same results– fixed is computationally simpler

• If heterogeneous…then first ask why?! – In the face of heterogeneity, focus of analysis should be to

describe possible sources of variability - attempt to identify sources of important subgroup differences

– Example: studies using one dose showed significant effect, while lower dose did not. Then do fixed effects analysis of each sub-group and report all results.

12

Use of the Random Effects Model?….

• Many observers dispute the rationale for random-effect based analyses. For example:

– Petitti (2000) “…. in the very situations where application of the method matters (= heterogeneity), a single summary estimate of effect is inappropriate”

– Greenland (1994) – “the random effects model is the

model or summary of last resort”

13

Statistical Tests of Homogeneity (heterogeneity)

• Homogeneity calculations• Ho = studies are homogeneous

• Based on testing the sum of weighted differences between the summary effect and individual effects

• Calculate Mantel Haenszel Q, where:– Q = [weighti x (lnORmh - lnORi)2]

– To interpret, use the chi-square distribution where the degrees of freedom = S - 1 (where S is the number of studies). If p < 0.05, then there is significant heterogeneity.

• Power of such statistical tests is low (a non-significant test does not rule out clinically important heterogeneity)

14

Specific methods for dichotomous data

• Mantel-Haenszel method (fixed effects)• originally developed to handle analysis of data in multiple

strata. If you think of each study as a stratum, you can do a meta-analysis!

• data must be in form of 2 x 2 table for Mantel-Haenszel– odds ratio, rate ratio, risk ratio

• Most commonly used method for meta-analysis (has optimal statistical properties)

• Only accounts for confounding if it is incorporated into the study design (matching or randomization)

– therefore, can’t use multivariable adjusted data.

15

Mantel-Haenszel Method

Exposed Unexposed Total

Diseased ai bi gi

Non-diseased ci di hi

Total ei fi ni

16

Mantel-Haenszel Method

ORmh = (weighti x ORi) / weighti

• ORi = (ai x di) / (bi x ci)• weighti = 1 / variancei

• variancei = ni / (bi x ci)

• 95% CI = e ln(ORmh) +/- 1.96 x sqrt(var ORmh)

– var ORmh = • (F / 2 x R2) + [G / (2 x R x S)] + (H/(2 x S2)

– where:• F = [ai x di x (ai + di)]/ni

2

• G = [ai x di x (bi+ci)] + (bi x ci x (ai + di))] / ni2

• H = (bi x ci x (bi+ci)) / ni2

• R = (ai x di) / ni

• S = (bi x ci) / ni

17

(Sir Richard) Peto Method

• Fixed effects– very similar to Mantel-Haenszel method (same 2x2

requirement)– see Pettiti pages 104-107 or Hasselblad article for

formulae– computationally somewhat simpler, especially to

calculate the confidence interval– may provide biased results under some circumstances in

which Mantel-Haenszel would not– Best applied to RCT’s and not observational studies

18

General Variance Methods

• Used to summarize rate/risk differences (RD) • Fixed effects method

– RDs = (wi x RDi) / wi

• wi = 1 / variancei

– 95%CI = RDs +/- 1.96 (variances)0.5 • Variances = 1/wi

– see text page 107 for more details– formulas differ if analyzing rate ratio data (incidence-density) or risk ratio

data (cumulative risk)

• General variance-based methods also used for observational studies when study results are presented as RR with 95% CI

19

Random effects models• DerSimonian and Laird statistic

• Uses odds ratios only!

• lnORdl = (wi* x lnORi) / wi*

• wi* = 1 / [D + (1/wi)]

• wi = 1 / variancei

• D = ([Q - (S - 1)] x wi ) / [(wi)2 - wi2]

• Q = [wi x (lnORi - lnORmh)2]

• CI = exp(lnORdl + 1.96 x (variances*)^0.5

• variances* = weighti*

20

Continuous outcomes

• Two approaches:• 1. Each study used the same scale or variable (i.e. all measured SBP,

serum creatinine or Mini-Mental State score). Based on ANOVA model where studies are “groups”.

– meansummary = (weighti x meani) / weighti

– meani = meantx - meancontrol

– weighti = 1 / variancei = 1 / SDi2

• (use pooled variance)

– 95% CI = means +/- (1.96 x (variances)^0.5)• variances = 1 / weighti

– Test of homogeneity: Q = [weighti x (means - meani)2]

21

Continuous outcomes• 2. Each study used a similar but different scale (e.g., CAGE and MAST for

diagnosis of alcoholism, pulmonary function tests [PEFR, FEV1])

– dsummary = (weighti x di) / weighti

– dsummary = summary estimate of the difference in effect sizes

– di = effect size = (meantx - meancontrol) / SDpooled

– weighti = 1 / variancei = (2 x Ni) / (8 + di2)

• (use pooled variance)

– 95% CI = ds +/- (1.96 x (variances)^0.5)• variances = 1 / weighti

– Test of homogeneity: Q = [weighti x (ds - di)2]

22

Other Issues in Meta-Analysis

• Cumulative M-A• See article by Antman for example

• Pooling Studies• See article by Blettner (Type III study)

• M-A of observational studies

• M-A of diagnostic tests

• Meta-regression

23

M-A of Observation Studies

• Very controversial application with some authors rejecting the approach outright (Shapiro, 1994)

• Often applied to controversial topics where previous studies are inconclusive (due to small risks and/or small studies) • Exam: Chlorination and CA risk, EMF and CA risk.• But can never exclude bias…….

• Important to regard process as a “study of studies” and not a means of providing a summary estimate

• Very valuable process at identifying deficiencies in published literature

• See Stroup et al (JAMA 2000) – proposal for reporting

24

Meta-analysis of diagnostic tests

• See Irwig article (bibliography) for an excellent overview.

• Simply averaging sensitivity and specificity is not useful: Se Sp• Study 1 0 100• Study 2 99 99• Study 3 100 0• Mean 67 67

25

What to do?

• Can calculate a summary ROC curve, by plotting the sensitivity and specificity for each study of a diagnostic test.

• Especially useful for comparing tests• e.g. stress thallium vs stress echocardiogram for

heart disease.

• See Irwig article for details of calculations.

26

Plotting an ROC curve

1 - Sp

Se

O

O

O Each circle represents anindividual study

OO

27

Figure 3. Summary receiver-operating characteristic (SROC) curve analysis of ELISA D-dimer in the diagnosis of PE. Plotted in each of the SROC graphs are individual studies depicted as ellipses. The x- and y-dimensions of the ellipses are proportional to the square root of the number of patients available to study the sensitivity and specificity, respectively, within the analysis. Also shown is the unweighted SROC curve limited to the range where data are available. The cross (x) represents the independent random-effects pooling of sensitivity and specificity values of the studies.

28

Meta-regression

• Multivariate approach: • Use the study characteristics as independent variables

• Design, age, population source, quality score etc etc

• Use effect size or other outcome as the dependent variable• Identify significant study characteristics • Unit of observation = study• Can be useful to identify sources of heterogeneity, clarify

importance of quality scores• Exploratory only

29

Meta-regression – Example(Phillips 1991: 26 HIV studies, Dependent var = Specificity)

Variable Regression Co-efficient

T P value

Year of pub. -0.023 -0.90 > 0.05

Low HIV Prev 0.114 -2.54 < 0.05

High vs Med Quality

-0.014 -0.20 < 0.05

Low vs Med Quality

-0.087 -1.38 <0.05

30

Final comments

• Remember the “art” of meta-analysis: knowing when to use which technique, rather than “mindlessly” applying formulae to studies.

• Understanding the underlying clinical rationale for treatment, differences in populations, and differences in outcomes is critical.

• An important contribution of MA is to highlight the variability in the design, conduct, analysis and findings of a particular body of literature.

31

Bibliography

• Highly recommended reading• Hasselblad V, McCrory DC. Meta-analytic tools

for medical decision-making: a practical guide. Med Decis Mak 1997; 15: 81-96.

• Irwig L, Tosteson AN, Gatsonis C, et al. Guidelines for meta-analyses evaluating diagnostic tests. Ann Intern Med 1994; 120: 667-76.

32

Other recommended reading

• Cook DJ, Guyatt GH, Ryan G, et al. Should unpublished data be included in meta-analyses? JAMA 1993;p 269: 2749-53.

• Greenland S. A critical look at some popular meta-analytic methods. Am J Epid 1994; 140: 290-6.

• L’Abbe K, Detsky AS< O’Rourke K. Meta-analysis in clinical research. Ann Intern Med 1987; 107: 224-33.

• LeLorier J, Gregoire G, Benhaddad A, et al. Discrepancies between meta-analyses and subsequent large randomized, controlled trials. N Engl J Med 1997; 337: 536-42.

• Eddy DM, Hasselblad V, and Schachter. An introduction to a Bayesian method for meta-analysis. Med Decis Mak 1990; 10: 15-23. (REQUIRES SPECIAL SOFTWARE)

• Sacks HS, Berrier J, Reitman D, et al. Meta-analyses of randomized controlled trials. N Engl J Med 1987; 316: 450-5.

33

Other recommended reading

• Cook DJ, Sackett DL, Spitzer WO. Methodologic guidelines for systematic reviews of randomized control trials in health care from the Potsdam Consultation on Meta-Analysis. J Clin Epidemiol 1995; 48: 167-71.

• Chalmers TC, Smith H, Blackburn B, et al. A method for assessing the quality of a randomized control trial. Control Clin Trials 1981; 2: 31-49.

• Sackett DL. Applying overviews and meta-analyses at the bedside. J Clin Epidemiol 1995; 48: 61-6.

• Olkin I. Statistical and theoretical considerations in meta-analysis. J Clin Epidemiol 1995; 48: 133-46.

34

Sample meta-analyses

• Clark P, Tugwell P, Bennett K, Bombardier C. Meta-analysis of injectable gold in rheumatoid arthritis. J Rheumatol 1989; 16: 442-7.

• Rowe BH, Keller JL, Oxman AD. Effectiveness of steroid therapy in acute exacerbations of asthma: a meta-analysis. Am J Emerg Med 1992; 10: 301-10.

• Cummings P. Antibiotics to prevent infection in patients with dog bite wounds: a meta-analysis of randomized trials. Ann Emerg Med 1994; 23: 535-40.

• Callahan CM, Drake BG, Heck DA, Dittus RS. Patient outcomes following tricompartmental total knee replacement: a meta-analysis. JAMA 1994; 271: 1349-57.

• Phillips KA. The use of meta-analysis in technology assessment: a meta-analysis of the enzyme immunosorbent assay HIV antibody tests. J Clin Epidemiol 1991; 44: 925-31.