GRADE Workshop: Agenda Introduction. Guideline development ... · • Revman A free software for meta-analysis ... 36 GRADE Workshop –Ispra–11-12 December 2013 a. Introduction

1

Disc laimer: The contents of this presentation are the views of the author and do not necessarily represent an offic ial position of the European C ommission. © European Union, 2013

GRADE workshop:

Meta-analysis - the basics

Juan A Blasco Amaro, MD MPH

Public Health Policy Support Unit

Institute for Health and Consumer Protection

(JRC-IHCP)

Joint Research Centre

The European Commission’s

in-house science service

2 GRADE Workshop – Ispra– 11-12 December 2013

GRADE Workshop: Agenda

9:00 - 10:15 Introduction. Guideline development process and the GRADE approach

10:15-11:00 Types of questions. Framing a question: PICO question. Exercise

11:00-11:15 Coffee

11:15-12:00 Choosing outcomes. Relative importance of outcomes. Exercise

12:00-12:45 Study designs. Exercise

12:45-13:30 Lunch

13:30-14:30 Search of the literature. Exercise

14:30-15:00 Determinants of quality of evidence: What can lower the evidence? (I)

15:00-15:15 Coffee

15:15-17:00 Determinants of quality of evidence: What can lower the evidence? (II) Exercise

9:00-11:15 Determinants of QoE: What can lower the evidence? (III). ExerciseDeterminants of quality of evidence: What can upgrade evidence?

11:15-11:30 Coffee

11:30-13:00 Going from the evidence to the recommendation. Exercise

13:00-13:45 Lunch

13:45-16:00 Using the Guideline Development Tool (GDT) software. ExerciseFeedback and conclusions


a. Introduction

b. Representation

c. Imprecision

d. Inconsistency

e. Publication Bias

f. Exercise

Outline


¿What is a meta-analysis?

a) A combination of studies

b) Estimation of a common value

c) A graphical plot

d) A type of Systematic Review

e) Combined analyses of randomized

clinical trials

f) Research project

g) Observational study

h) Statistical method


Meta-analysis History

� -Background

� -Karl Pearson, 1904

� -First “meta-analisys” published assessing

intervention, what intervention?

- it was effective in 35%!

• -Term is introduced by Glass in 1976 and

became popular in Social Sciences

• - In 1992, a network of epidemiologists and

healthcare professionals -> Cochrane

Collaboration


Evolution - meta published in medline

20042005

20062007

20082009

20102011

20122013

0

1000

2000

3000

4000

5000

6000

7000

8000

7000-8000

6000-7000

5000-6000

4000-5000

3000-4000

2000-3000

1000-2000

0-1000

2


Meta-analysis

Define Question

Search forStudies

CollectData

HeterogeneityAnalysis

Combining effects

Interpret resultsAnd

Draw conclusions

Apply elegibility criteria

Assess studies for Risk of Bias

Sensitivity analysis Publication bias

Graphical plot

Heterogeneity measures


Review level

↓

Effect measure

Study A

Effect measure

Outcome data

Effect measure

Outcome data

Study B

Effect measure

Outcome data

Study C

Effect measure

Outcome data

Study D

Study level

↓

Source: Jo McKenzie & Miranda

Cumpston


What is a meta-analysis?

• combines the results from two or more studies

• estimates an ‘average’ or ‘common’ effect

• optional part of a systematic review

Systematic reviews

Meta-analyses


Source: Julian Higgins

Why perform a meta-analysis?

� quantify treatment effects and their uncertainty

� increase power

� increase precision

� explore differences between studies

� settle controversies from conflicting studies

� generate new hypotheses


When not to do a meta-analysis

• mixing apples with oranges• each included study must address same question

consider comparison and outcomes

requires your subjective judgement

• garbage in – garbage out• a meta-analysis is only as good as the studies in it

• if included studies are biased:

meta-analysis result will also be incorrect

will give more credibility and narrower confidence interval

• if serious reporting biases present:

unrepresentative set of studies may give misleading result



When can you do a meta-analysis?

• more than one study has measured an effect

• the studies are sufficiently similar to produce a meaningful and

useful result

• the outcome has been measured in similar ways data are

available in a format we can use

3


Software (I)

• MetaXL software page

• ClinTools (commercial)

• Comprehensive Meta-Analysis (commercial)

• MIX 2.0 Professional Excel addin with Ribbon interface for meta-

analysis (free and commercial versions).

• Meta-analysis features for Stata? (free add-ons to commercial package)

• The Meta-Analysis Calculator free on-line tool for conducting a

meta-analysis

• Metastat (Free)

• Meta-Analyst Free Windows-based tool

• ProMeta Professional software for meta-analysis Java (commercial)

• Revman A free software for meta-analysis

• Metafor-project A free software package for meta-analyses in R

• Macros in SPSS Free Macros to conduct meta-analyses in SPSS

• MAd GUI User friendly graphical user interface package for R (Free)


Software (IV)Software (III)


¿What’s behind the

“software”?


Statistical analysis


Meta-analysis is typically a two-stage process

• a summary statistic is calculated for each study, to describe the observed intervention effect.

• summary (pooled) intervention effect estimate is calculated as a weighted average of the intervention effects estimated in the individual studies


Steps in a meta-analysis

• identify comparisons to be made

• identify outcomes to be reported and statistics to be used

• collect data from each relevant study

• combine the results to obtain the summary of effect

• explore differences between the studies

• interpret the results

4


Calculating the summary result

• collect a summary statistic from each contributing study

• how do we bring them together?

• treat as one big study – add intervention & control data?

breaks randomisation, will give the wrong answer

• simple average?

weights all studies equally – some studies closer to the truth

• weighted average


Headache Caffeine Decaf Weight

Amore-Coffea 2000

2/31 10/34

Deliciozza 2004 10/40 9/40

Mama-Kaffa 1999 12/53 9/61

Morrocona 1998 3/15 1/17

Norscafe 1998 19/68 9/64

Oohlahlazza 1998 4/35 2/37

Piazza-Allerta 2003

8/35 6/37

For example


Headache Caffeine Decaf Weight

Amore-Coffea 2000

2/31 10/34 6.6%

Deliciozza 2004 10/40 9/40 21.9%

Mama-Kaffa 1999 12/53 9/61 22.2%

Morrocona 1998 3/15 1/17 2.9%

Norscafe 1998 19/68 9/64 26.4%

Oohlahlazza 1998 4/35 2/37 5.1%

Piazza-Allerta 2003

8/35 6/37 14.9%

For example


Overview

23 GRADE Workshop – Ispra– 11-12 December 2013 24 GRADE Workshop – Ispra– 11-12 December 2013

5


• for dichotomous or continuous data

• inverse-variancestraightforward, general method

• for dichotomous data only

• Mantel-Haenszel (default)good with few events – common in Cochrane reviews

weighting system depends on effect measure

• Petofor odds ratios only

good with few events and small effect sizes (OR close to 1)

Meta-analysis options


a. Introduction

b. Representation

c. Imprecision

c. Inconsistency

d. Publication Bias

e. Exercise

Outline


A forest of lines


Forest plots

Headache at 24 hours

• headings explain the comparison


Forest plots


• list of included studies



• effect estimate for each study, with CI

Forest plots

6



• scale and direction of benefit

Forest plots



• pooled effect estimate for all studies, with CI

Forest plots


a. Introduction

b. Representation

c. Imprecision

d. Inconsistency

e. Publication Bias

f. Exercise

Outline


Interpreting confidence intervals

• always present estimate with a confidence interval

• precision• point estimate is the best guess of the effect

• CI expresses uncertainty – range of values we can be reasonably

sure includes the true effect

• significance• if the CI includes the null value

rarely means evidence of no effect

effect cannot be confirmed or refuted by the available evidence

• consider what level of change is clinically important


When are results imprecise?

• For dichotomous outcomes

• Sample size, optimal information size

• Number of events

• For continuous outcomes

• minimal important difference

Assessing precision of the Results


a. Introduction

b. Representation

c. Imprecision

d. Inconsistency

e. Publication Bias

f. Exercise

Outline

7


Heterogeneity

• clinical diversity (sometimes called clinical heterogeneity)

• methodological diversity (sometimes called methodological

heterogeneity)

• statistical heterogeneity simply as heterogeneity

37 38 GRADE Workshop – Ispra– 11-12 December 2013

Clinical diversity

• participants

• e.g. condition, age, gender, location, study eligibility criteria

• interventions

• intensity/dose, duration, delivery, additional components, experience of practitioners, control (placebo, none, standard care)

• outcomes

• follow-up duration, ways of measuring, definition of an event, cut-off points


Methodological diversity

• design

• e.g. randomised vs non-randomised, crossover vs parallel,

individual vs cluster randomised

• conduct

• e.g. risk of bias (allocation concealment, blinding, etc.), approach to analysis


Statistical heterogeneity

• there will always be some random (sampling) variation between the results of different studies

• heterogeneity is variation between the effects being evaluated in the different studies

• caused by clinical and methodological diversity

• alternative to homogeneity (identical true effects underlying every study)

• study results will be more different from each other than if random variation is the only reason for the differences between the estimated intervention effects


Identifying heterogeneity

• Visual inspection of the forest plots

• Chi-squared (χχχχ2) test (Q test)

• I2 statistic to quantify heterogeneity


Visual inspection

Forest plot A Forest plot B

8


Cochran’s Q and his test

• Cochran's Q statistic, (χχχχ2) :

• Follow a distribution c2 with k-1 degrees of freedom

• (k = number of studies).

• Low power for few studies (low k).

• Statistic I2 :

• I2 = 100% x (Q – [k-1])/Q

• Percentage of variation related to heterogeinity and not to random.


Thresholds for the interpretation of I2 can be misleading

• 25% low - might not be important;

• 50% moderate - may represent moderate heterogeneity;

• 75% high - may represent substantial heterogeneity;

• 75% to 100% considerable heterogeneity.

44


Example

45 46 GRADE Workshop – Ispra– 11-12 December 2013

The I2 statistic


Fixed-effect vs random-effects

• Two models for meta-analysis available in RevMan

• Make different assumptions about heterogeneity


Fixed-effect model

• assumes all studies are measuring the same treatment effect

• estimates that one effect

• if not for random (sampling) error, all results would be identical

Common

Random (sampling)error

true effect

Studyresult


9


Random-effects model

Randomerror

Study-specificeffect

Mean of trueeffects

• assumes the treatment effect varies

between studies

• estimates the mean of the

distribution of effects

• weighted for both

within-study and between-study

variation (tau2, τ2)



No heterogeneity

Adapted from Ohlsson A, Aher SM. Early erythropoietin for preventing red blood cell transfusion in

preterm and/or low birth weight infants. Cochrane Database of Systematic Reviews 2006, Issue 3.

Fixed Random


Some heterogeneity

Fixed Random

Adapted from Adams CE, Awad G, Rathbone J, Thornley B. Chlorpromazine versus

placebo for schizophrenia. Cochrane Database of Systematic Reviews 2007, Issue 2.


Which to choose?

• Do you expect your results to be very diverse?

• Consider the underlying assumptions of the model

• fixed-effect

may be unrealistic – ignores heterogeneity

• random-effects

allows for heterogeneity

estimate of distribution of studies may not be accurate if biases are present, few studies or few events


What to do about heterogeneity

• Check that the data are correct

• Especially if the direction of effect varies if heterogeneity is very high

• Interpret fixed-effect results with cautionconsider sensitivity analysis – would random-effects have made an important

difference?

• May choose not to meta-analyseaverage result may be meaningless in practice

consider clinical & methodological comparability of studies

• Avoidchanging your effect measure or analysis model

excluding outlying studies

• explore heterogeneity


Exploring your results

• what is heterogeneity?

• assumptions about heterogeneity

• identifying heterogeneity

• exploring your results

10


Two methods available

• subgroup analysis

• Group studies by pre-specified factors

• look for differences in results and heterogeneity

• meta-regression

• examine interaction with categorical and continuous variables

• not available in RevMan


Participant subgroups

Based on Stead LF, Perera R, Bullen C, Mant D, Lancaster T. Nicotine replacement therapy for smoking cessation. Cochrane Database of Systematic Reviews 2008, Issue 1. Art. No.: CD000146. DOI: 10.1002/14651858.CD000146.pub3.


Intervention subgroups

Based o

n L

inde K

, B

ern

er

MM

, K

risto

n L

. St

John's

wort

for

majo

r depre

ssio

n. Cochra

ne D

ata

base o

f Syste

matic R

evie

ws

2008,

Issue 4

. Art

. N

o.:

CD

000448. D

OI:

10.1

002/1

4651858.C

D000448.p

ub3.


Sensitivity analysis

• not the same as subgroup analysis

• testing the impact of decisions made during the review

• inclusion of studies in the review

• definition of low risk of bias

• choice of effect measure

• assumptions about missing data

• cut-off points for dichotomised ordinal scales

• correlation coefficients

• repeat analysis using an alternative method or assumption

• don’t present multiple forest plots – just report the results

• if difference is minimal, can be more confident of conclusions

• if difference is large, interpret results with caution


Adapted from Li J, Zhang Q, Zhang M, Egger M. Intravenous magnesium for acute myocardial infarction. Cochrane Database of

Systematic Reviews 2007, Issue 2.

Sensitivity analysis


Differences in underlying treatment effect

When heterogeneity exists, but investigators fail to identify a plausible

explanation, the quality of evidence should be downgraded by one or two

levels, depending on the magnitude of the inconsistency in the results

Inconsistency may arise from differences in:

• populations (e.g. drugs may have larger relative effects in sicker

populations) • interventions (e.g. larger effects with higher drug doses)

• outcomes (e.g. diminishing treatment effect with time)

*Proposed by GRADE Working Group

Assessing inconsistency *

11


a. Introduction

b. Representation

c. Imprecision

d. Inconsistency

e. Publication Bias

f. Exercise

Outline


Funnel plots

• plot effect size against study size

• study size usually indicated by a measure like standard error

• studies will be scattered around the effect estimate

• larger studies at the top, smaller studies further down

• small studies expected to scatter more widely

• a symmetrical plot will look like an inverted funnel or triangle

• RevMan can generate funnel plots

• only appropriate with ≥ 10 studies of varying size


Symmetrical funnel plot

Sta

nd

ard

Erro

r

EffectSource: Matthias Egger & Jonathan Sterne

0.1 0.33 1 33

2

1

0

100.6


Asymmetrical funnel plot

0.1 0.33 1 33

2

1

0

100.6

Effect

Sta

nd

ard

Erro

r

Source: Matthias Egger & Jonathan Sterne

Unpublished studies


Colloids vs crystalloids for fluid resuscitation

Adapted from Perel P, Roberts I. Colloids versus crystalloids for fluid resuscitation in critically ill patients. Cochrane Database of


Death


Magnesium for myocardial infarction

Adapted from Li J, Zhang Q, Zhang M, Egger M. Intravenous magnesium for acute myocardial infarction. Cochrane Database of


12


Reasons for funnel plot asymmetry

1. chance

• artefact

• some statistics are correlated to SE, e.g. OR

• clinical diversity

• different populations different in small studies

• different implementation different in small studies

• methodological diversity

• greater risk of bias in small studies

• reporting biases (publication bias)

Source: Egger M et al. Bias in meta-analysis detected by a simple, graphical test. BMJ 1997; 315: 629


Outline

• understanding reporting biases


unavailable(unpublished)

available in principle(thesis,

conference, small journal)

easily available(Medline-indexed)

activelydisseminated(news, drug company)

Source: Matthias Egger

The dissemination of evidence


Reporting biases

• dissemination of research findings is influenced by the nature and direction of results

• statistically significant, ‘positive’ results more likely to be published…

• …therefore more likely to be included

• leads to exaggerated effects

• large studies likely to be published anyway, so small studies most likely to be affected

• non-significant results are as important to your review as significant results


Evidence for reporting bias

Source: Stern JM, Simes RJ. Publication bias: evidence of delayed publication in a cohort study of clinical research

projects BMJ 1997;315:640-645.

Proportion of

studies not

published

Years since conducted

SignificantNon-significant trendNull


Conceived

Performed

Submitted

Cited

Published

Positive studies are more likely to be

• submitted for publication...

• …and accepted (publication bias)

• …quickly (time lag bias)

• …as more than one paper(multiple publication bias)

• …in English (language bias)

• …in high-impact, indexed journals (location bias)

• … including positive outcomes (selective outcome reporting)

• …and cited by others (citation bias)


13


Publication bias is a systematic underestimate or an overestimate of the underlying beneficial or harmful effect

• Investigators fail to report studies the have undertaken

• If meta-analysis is influenced, downgrade the quality of evidence

*Proposed by GRADE Working Group

Assessing publication bias *

JRC xxxxx – © European Union, 2013Disc laimer: The contents of this presentation are the views of the author and do not necessarily represent an offic ial position of the European C ommission.

Determinants of quality of evidence: What can upgrade evidence?

Documents

GRADE Workshop: Agenda Introduction. Guideline development ... · • Revman A free software for meta-analysis ... 36 GRADE Workshop –Ispra–11-12 December 2013 a. Introduction