Understanding heterogeneity in systematic reviews and met-analysis meta-analysis generates a single best estimate of effectmeta-analysis generates a single

Understanding heterogeneity Understanding heterogeneity in systematic reviews and in systematic reviews and

met-analysismet-analysis• meta-analysis generates a single meta-analysis generates a single best estimate of effectbest estimate of effect– what are the underlying assumptions?what are the underlying assumptions?

• how to judge consistency of how to judge consistency of resultsresults– 4 strategies4 strategies

• what to do if inconsistencywhat to do if inconsistency

The right questionThe right question• all cancer therapy for all cancersall cancer therapy for all cancers

• all antiplatelet agents for all all antiplatelet agents for all atheroembolic events (heart, head, leg)atheroembolic events (heart, head, leg)

• all aspirin doses for strokeall aspirin doses for stroke

• 30 to 300 mg. for stroke30 to 300 mg. for stroke

• what is guide about when right to pool?what is guide about when right to pool?

What were your What were your criteria?criteria?

• what made you decide some were OK and some what made you decide some were OK and some were not?were not?

• across range of across range of – patientspatients– interventionsinterventions– comparatorscomparators– outcomesoutcomes

effect more or less sameeffect more or less same

• if notif not– big effect in severe patients, no effect in mildbig effect in severe patients, no effect in mild– big effect in high dose, no effect in lowbig effect in high dose, no effect in low– big effect in short term, none in long termbig effect in short term, none in long term

Are you happy Are you happy

pooling?pooling?

10.5

Relative Risk (95% CI)

0.73 (0.49, 1.07)

0.74 (0.59, 0.94)

0.76 (0.51, 1.12)

0.71 (0.56, 0.90)

0.73 (0.61, 0.88)

Are you happy pooling?Are you happy pooling?

10.5


0.44 (0.30, 0.65)

0.45 (0.36, 0.60)

1.25 (0.84, 1.84)

1.17 (0.92, 1.49)

0.73 (0.61, 0.88)

What criteria were you What criteria were you using?using?

• similarity of point estimatessimilarity of point estimates– less similar, less happyless similar, less happy

• overlap of confidence overlap of confidence intervalsintervals– less overlap, less happyless overlap, less happy

-40 -24 -8 8 24 40 56

RRR (95% CI)

HomogenousHomogenous

10.5


0.73 (0.49, 1.07)

0.74 (0.59, 0.94)

0.76 (0.51, 1.12)

0.71 (0.56, 0.90)

0.73 (0.61, 0.88)

test for heterogeneity what is the p-value?

what is the null hypothesis for the test for heterogeneity?

Ho: RR1 = RR2 = RR3 = RR4

p=0.99 for heterogeneity

HeterogeneousHeterogeneous

10.5


0.44 (0.30, 0.65)

0.45 (0.36, 0.60)

1.25 (0.84, 1.84)

1.17 (0.92, 1.49)

0.73 (0.61, 0.88)

p-value for heterogeneity < 0.001

test for heterogeneity what is the p-value?

HomogenousHomogenous

10.5


0.73 (0.49, 1.07)

0.74 (0.59, 0.94)

0.76 (0.51, 1.12)

0.71 (0.56, 0.90)

0.73 (0.61, 0.88)


I2=0%

What is the I2 ?


10.5


0.44 (0.30, 0.65)

0.45 (0.36, 0.60)

1.25 (0.84, 1.84)

1.17 (0.92, 1.49)

0.73 (0.61, 0.88)

p-value for heterogeneity < 0.001I2=89%

What is the I2 ?

II2 2 InterpretationInterpretation

No worries 0%

25%Only a little

concerned

50%Getting concerned

75%Very

concerned

100%Why are

we pooling?


10.5


0.44 (0.30, 0.65)

0.45 (0.36, 0.60)

1.25 (0.84, 1.84)

1.17 (0.92, 1.49)

0.73 (0.61, 0.88)

p-value for heterogeneity < 0.001I2=89%

If this result, what next?

Does Vitamin D prevent Does Vitamin D prevent non-vertebral fractures? non-vertebral fractures? • systematic review and meta-analysis

• patients: over 60

• intervention: Vitamin D (cholecalciferol or ergocalciferol)

• outcome: non-vertebral fractures– follow-up at least a year

• methods: blinded RCTS

Relative Risk with 95% CI for Vitamin D Non-vertebral Fractures

Chapuy et al, (2002) 0.85 (0.64, 1.13)Chapuy et al, (2002) 0.85 (0.64, 1.13)

Pooled Random Effect Model 0.82 (0.69 to 0.98)

p= 0.05 for heterogeneity, I2=53%

Chapuy et al, (1994) 0.79 (0.69, 0.92)Chapuy et al, (1994) 0.79 (0.69, 0.92)

Lips et al, (1996) 1.10 (0.87, 1.39)Lips et al, (1996) 1.10 (0.87, 1.39)

Dawson-Hughes et al, (1997) 0.46 (0.24, 0.88)Dawson-Hughes et al, (1997) 0.46 (0.24, 0.88)

Pfeifer et al, (2000) 0.48 (0.13, 1.78)Pfeifer et al, (2000) 0.48 (0.13, 1.78)

Meyer et al, (2002) 0.92 (0.68, 1.24)Meyer et al, (2002) 0.92 (0.68, 1.24)

Trivedi et al, (2003) 0.67 (0.46, 0.99)Trivedi et al, (2003) 0.67 (0.46, 0.99)

Favours Vitamin D Favours Control

Relative Risk 95% CI

Relative Risk with 95% CI for Vitamin D (Non-Vertebral Fractures, Dose >400)

Chapuy et al, (1994) 0.70 (0.69, 0.92)

Dawson-Hughes et al, (1997) 0.46 (0.24, 0.88)

Pfeifer et al, (2000) .48 (0.13, 1.78)

Chapuy et al, (2002) 0.85 (0.64, 1.13)

Trivedi et al, (2003) 0.67 (0.46, 0.99)

Pooled Random Effect Model 0.75 (0.63 to 0.89)

p= 0.26 for heterogeneity, I2=24%



Relative Risk with 95% CI for Vitamin D (Non-Vertebral Fractures, Dose = 400)

Lips et al (1996) 1.10 (0.87, 1.39)

Meyer et al (2002) 0.92 (0.68, 1.24)

Pooled Random Effect Mode 1.03 (0.86 to 1.24)p = 0.35 heterogeneity, I2=0%



Heterogeneity• look for explanation

• patients • interventions

• outcomes

• risk of bias

• No good explanation? What to do?

• decrease confidence in effect estimates

Beta-blockers in non-Beta-blockers in non-cardiac surgery: Strokecardiac surgery: Stroke

1 5 10 50 1000.5 0.1

Study Year Overall Event Rate Relative Risk (95% CI)

Wallace 1998 5 / 200 3.06 (0.49 to 19.02)

Pobble 2005 1 / 103 2.63 (0.11 to 62.97)

DIPOM 2006 2 / 921 4.97 (0.24 to 103.19)

MaVS 2006 6 / 496 1.83 (0.39 to 8.50)

Zaugg 2007 1 / 119 2.97 (0.12 to 72.19)

POISE 2007 60 / 8351 2.13 (1.25 to 3.64)

Random Effects Estimate 2.22 (1.39 to 3.56)

p=0.99 for heterogeneity, I²=0%


I2= 0%

Vitamin D + Calcium vs CalciumVitamin D + Calcium vs Calcium

10.5 0.1

Study Year Treatment Control Relative Risk (95% CI)

Komulainen 1998 8/116 14/116 0.57 (0.25, 1.31)

Pfeifer 2000 3/70 6/67 0.48 (0.12, 1.84)

Flicker 2005 25/313 35/312 0.71 (0.44, 1.16)

Grant 2005 179/1306 185/1311 0.97 (0.80, 1.18)

Random Effects 0.86 (0.67, 1.09)


10.5 0.1


Komulainen 1998 8/116 14/116 0.57 (0.25, 1.31)

Pfeifer 2000 3/70 6/67 0.48 (0.12, 1.84)

Flicker 2005 25/313 35/312 0.71 (0.44, 1.16)

Grant 2005 179/1306 185/1311 0.97 (0.80, 1.18)



10.5 0.1


Komulainen 1998 8/116 14/116 0.57 (0.25, 1.31)

Pfeifer 2000 3/70 6/67 0.48 (0.12, 1.84)

Flicker 2005 25/313 35/312 0.71 (0.44, 1.16)

Grant 2005 179/1306 185/1311 0.97 (0.80, 1.18)


p= 0.32 for heterogeneity


10.5 0.1


Komulainen 1998 8/116 14/116 0.57 (0.25, 1.31)

Pfeifer 2000 3/70 6/67 0.48 (0.12, 1.84)

Flicker 2005 25/313 35/312 0.71 (0.44, 1.16)

Grant 2005 179/1306 185/1311 0.97 (0.80, 1.18)



I2= 14%

Vitamin D + Calcium vs Vitamin D + Calcium vs Placebo/ControlPlacebo/Control

10.5 0.1


Chapuy 1994 255/1176 308/1127 0.79 (0.69, 0.92)

Dawson-Hughes 1997 11/187 26/202 0.46 (0.23, 0.90)

Chapuy 2002 97/393 55/190 0.85 (0.64, 1.13)

Harwood 2004 3/39 5/37 0.57 (0.15, 2.22)

Porthouse A 2005 34/714 69/1391 0.96 (0.64, 1.43)

Porthouse B 2005 24/607 22/602 1.08 (0.61, 1.91)

Jackson 2006 2102/18176 2158/18106 0.97 (0.92, 1.03)

Random Effects 0.87 (0.76, 1.004)


10.5 0.1


Chapuy 1994 255/1176 308/1127 0.79 (0.69, 0.92)

Dawson-Hughes 1997 11/187 26/202 0.46 (0.23, 0.90)

Chapuy 2002 97/393 55/190 0.85 (0.64, 1.13)

Harwood 2004 3/39 5/37 0.57 (0.15, 2.22)

Porthouse A 2005 34/714 69/1391 0.96 (0.64, 1.43)

Porthouse B 2005 24/607 22/602 1.08 (0.61, 1.91)

Jackson 2006 2102/18176 2158/18106 0.97 (0.92, 1.03)

Random Effects 0.87 (0.76, 1.004)



10.5 0.1


Chapuy 1994 255/1176 308/1127 0.79 (0.69, 0.92)

Dawson-Hughes 1997 11/187 26/202 0.46 (0.23, 0.90)

Chapuy 2002 97/393 55/190 0.85 (0.64, 1.13)

Harwood 2004 3/39 5/37 0.57 (0.15, 2.22)

Porthouse A 2005 34/714 69/1391 0.96 (0.64, 1.43)

Porthouse B 2005 24/607 22/602 1.08 (0.61, 1.91)

Jackson 2006 2102/18176 2158/18106 0.97 (0.92, 1.03)

Random Effects 0.87 (0.76, 1.004)


I2= 50%

SummarySummary• starting assumption of pooled estimate

– across pts, intervention, outcome, similar effect

• broad criteria for meta-analysis desirable– maximize precision– maximize generalizability– can check out assumption

• is there excessive heterogeneity?– point estimates too variable– confidence intervals non-overlapping– low heterogeneity p-value– high I2

• if so, look for explanation– patients, intervention, outcome, methodology

Documents

Understanding heterogeneity in systematic reviews and met-analysis meta-analysis generates a single best estimate of effectmeta-analysis generates a single