Graphical Representation of Meta-analysis Findings...Graphical Representation of Meta-analysis Findings Emily E. Tanner-Smith Associate Editor, Campbell Methods Coordinating Group

1

The Campbell Collaboration www.campbellcollaboration.org

Graphical Representation of Meta-analysis Findings

Emily E. Tanner-Smith Associate Editor, Campbell Methods Coordinating Group

Research Assistant Professor, Vanderbilt University

Campbell Collaboration Colloquium Chicago, IL

May 22nd, 2013


Outline •  Introduction •  Forest plots •  Funnel plots •  Bubble plots •  Other graphs •  Software resources •  Summary

2


Introduction •  Graphs are an essential tool for conveying the results of a

meta-analysis to readers •  But if poorly constructed, graphs can be misleading and/or

confuse readers •  Graphs should strive for accuracy, simplicity, clarity, and

aesthetics •  This workshop will provide an overview of expectations and

guidelines for graphical displays of meta-analysis results in Campbell Collaboration reviews


Introduction: Basic Graphing Principles •  Descriptive titles and/or captions •  Use of legends (when appropriate) •  Representative range of scale •  Properly labeled axes •  Inclusion of reference points on axes •  Graphs should reflect the statistical precision of results •  Explicit mention of any excluded data •  Data in graphs should generally be available elsewhere in the review

(except in very large reviews) •  Aesthetics (line thickness, symbol size, symbol types, parsimony)

3


FOREST PLOTS


Forest plots •  The “workhorse” graph in meta-analysis •  Display effect size estimates and confidence intervals for

each study included in the meta-analysis •  Effect size estimates typically shown with blocks

proportionate to the weight assigned to a given study –  Functions to draw the eye toward studies with larger sample

size/larger weights, and away from smaller studies with wider confidence intervals

4


Forest plots •  Estimated mean effect size with confidence interval shown at

the bottom, typically with a diamond •  In random effects meta-analyses, prediction intervals can be

used to display dispersion in the estimated effect •  Studies should be ordered in a meaningful way

–  Effect size magnitude –  Study weight (precision) –  Chronological order –  Other meaningful study characteristic


Forest plots

Source: Regehr, C., Alaggia, R., Dennis, J., Pitts, A., & Saini, M. (2013). Interventions to reduce distress in adult victims of sexual violence and rape. Campbell Systematic Reviews, 3. doi:10.4073/csr.2013.3

5


Forest plots



Forest plots


6


Forest plots



Forest plots


7


Forest plots



Forest plots


8


Forest plots

Source: Maynard, B. R., McCrea, K. T., Pigott, T. D., & Kelly, M. S. (2012). Indicated truancy interventions: Effects on school attendance among chronic truant students. Campbell Systematic Reviews, 10. doi:10.4073/csr.2012.10


Forest plots

Source: Maynard, B. R., McCrea, K. T., Pigott, T. D., & Kelly, M. S. (2012). Indicated truancy interventions: Effects on school attendance among chronic truant students. Campbell Systematic Reviews, 10. doi:10.4073/csr.2012.10

9


Forest plots with subgroups •  Display effect size estimates and confidence intervals for

each study, split by some grouping variable •  Useful for depicting results from subgroup or moderator

analyses •  May include the overall summary effect across groups, if

appropriate •  Results from statistical tests of moderation

(e.g., QB or b from a meta-regression) should be summarized on the graph or in footnotes, when appropriate


Forest plots with subgroups

Source: Fictional data

. (-0.16, 0.81)

. (0.36, 0.86)

Single Session InterventionJones, 2012Wilson, 2008Smith, 2011Walters, 2000Milton, 1999Subtotal

Multi-Session InterventionChang, 1997Liu, 1992Mapleson, 2001Steiner, 2005Lancaster, 2009Subtotal

Study

0.11 (-0.09, 0.31)0.22 (-0.12, 0.56)0.34 (0.06, 0.62)0.45 (0.21, 0.69)0.48 (0.28, 0.68)0.32 (0.17, 0.48)

0.44 (0.20, 0.68)0.49 (0.21, 0.77)0.65 (0.41, 0.89)0.71 (0.40, 1.02)0.74 (0.53, 0.95)0.61 (0.49, 0.73)

0.11 (-0.09, 0.31)0.22 (-0.12, 0.56)0.34 (0.06, 0.62)0.45 (0.21, 0.69)0.48 (0.28, 0.68)0.32 (0.17, 0.48)

0.44 (0.20, 0.68)0.49 (0.21, 0.77)0.65 (0.41, 0.89)0.71 (0.40, 1.02)0.74 (0.53, 0.95)0.61 (0.49, 0.73)

Hedges' g (95% CI)

Favors Control Favors Treatment 0-1.02 0 1.02

10




. (-0.16, 0.81)

. (0.36, 0.86)

Single Session InterventionJones, 2012Wilson, 2008Smith, 2011Walters, 2000Milton, 1999Subtotal

Multi-Session InterventionChang, 1997Liu, 1992Mapleson, 2001Steiner, 2005Lancaster, 2009Subtotal

Study

0.11 (-0.09, 0.31)0.22 (-0.12, 0.56)0.34 (0.06, 0.62)0.45 (0.21, 0.69)0.48 (0.28, 0.68)0.32 (0.17, 0.48)

0.44 (0.20, 0.68)0.49 (0.21, 0.77)0.65 (0.41, 0.89)0.71 (0.40, 1.02)0.74 (0.53, 0.95)0.61 (0.49, 0.73)

0.11 (-0.09, 0.31)0.22 (-0.12, 0.56)0.34 (0.06, 0.62)0.45 (0.21, 0.69)0.48 (0.28, 0.68)0.32 (0.17, 0.48)

0.44 (0.20, 0.68)0.49 (0.21, 0.77)0.65 (0.41, 0.89)0.71 (0.40, 1.02)0.74 (0.53, 0.95)0.61 (0.49, 0.73)

Hedges' g (95% CI)



. (-0.16, 0.81)

. (0.04, 0.89)

. (0.36, 0.86)

Overall

Subtotal

Subtotal

Multi-Session Intervention

Liu, 1992

Study

Lancaster, 2009

Milton, 1999

Chang, 1997

Wilson, 2008

Mapleson, 2001Steiner, 2005

Walters, 2000Smith, 2011

Jones, 2012Single Session Intervention

0.46 (0.33, 0.60)

0.61 (0.49, 0.73)

0.32 (0.17, 0.48)

0.49 (0.21, 0.77)

Hedges' g (95% CI)

0.74 (0.53, 0.95)

0.48 (0.28, 0.68)

0.44 (0.20, 0.68)

0.22 (-0.12, 0.56)

0.65 (0.41, 0.89)0.71 (0.40, 1.02)

0.45 (0.21, 0.69)0.34 (0.06, 0.62)

0.11 (-0.09, 0.31)




Note: Significant difference in mean effect sizes between groups (b = .28, se = .10, 95% CI [.05, .52]).

11


. (-0.16, 0.81)

. (0.04, 0.89)

. (0.36, 0.86)

Overall

Subtotal

Subtotal


Liu, 1992

Study

Lancaster, 2009

Milton, 1999

Chang, 1997

Wilson, 2008




0.46 (0.33, 0.60)

0.61 (0.49, 0.73)

0.32 (0.17, 0.48)

0.49 (0.21, 0.77)

Hedges' g (95% CI)

0.74 (0.53, 0.95)

0.48 (0.28, 0.68)

0.44 (0.20, 0.68)

0.22 (-0.12, 0.56)

0.65 (0.41, 0.89)0.71 (0.40, 1.02)

0.45 (0.21, 0.69)0.34 (0.06, 0.62)

0.11 (-0.09, 0.31)






. (-0.16, 0.81)

. (0.04, 0.89)

. (0.36, 0.86)

Overall

Subtotal

Subtotal


Liu, 1992

Study

Lancaster, 2009

Milton, 1999

Chang, 1997

Wilson, 2008




0.46 (0.33, 0.60)

0.61 (0.49, 0.73)

0.32 (0.17, 0.48)

0.49 (0.21, 0.77)

Hedges' g (95% CI)

0.74 (0.53, 0.95)

0.48 (0.28, 0.68)

0.44 (0.20, 0.68)

0.22 (-0.12, 0.56)

0.65 (0.41, 0.89)0.71 (0.40, 1.02)

0.45 (0.21, 0.69)0.34 (0.06, 0.62)

0.11 (-0.09, 0.31)





12


Summary forest plots •  Display summary (mean) effect sizes and confidence intervals for

different groups of studies •  Does not include effect size estimates from individual studies •  Useful for very large reviews where traditional forest plots may not

be feasible, but effects can be categorized into meaningful groups (e.g., across intervention, study, participant types)

•  May include the overall summary effect across groups, if appropriate

•  Results from statistical tests of moderation (e.g., QB or b from a meta-regression) should be summarized on the graph or in footnotes, when appropriate


Summary forest plots

Overall

Subtotal

Canada

Spain

China

Subtotal

South America

Subtotal

Subtotal

United KingdomSubtotal

Africa

Location

United States

Subtotal

Subtotal

Study

0.36 (0.32, 0.40)

0.08 (-0.04, 0.21)

0.61 (0.50, 0.72)

0.46 (0.35, 0.57)

0.61 (0.50, 0.72)

0.02 (-0.10, 0.13)

0.35 (0.24, 0.46)

0.32 (0.22, 0.43)

0.36 (0.32, 0.40)

0.08 (-0.04, 0.21)

0.61 (0.50, 0.72)

0.46 (0.35, 0.57)

0.61 (0.50, 0.72)

0.02 (-0.10, 0.13)

Hedges' g (95% CI)

0.35 (0.24, 0.46)

0.32 (0.22, 0.43)

Favors Control Favors Treatment 0-.723 0 .723


13


Cumulative meta-analysis forest plots •  Display results from iterative estimation of summary (mean) effect

sizes, cumulatively adding one study at a time •  Useful for showing the accumulation of evidence over time, or the

in/stability of intervention effects over time •  May also be used to explore small sample bias, cumulatively

adding studies according to sample size of primary studies •  Title should clearly specify it is a forest plot showing results from a

cumulative meta-analysis


Cumulative meta-analysis forest plots

Fletcher (1959)Dewar (1963)1st European (1969)Heikinheimo (1971)Italian (1971)2nd European (1971)2nd Frankfurt (1973)1st Australian (1973)NHLBI SMIT (1974)Valere (1975)Frank (1975)UK Collab (1976)Klein (1976)Austrian (1977)Lasierra (1977)N German (1977)Witchitz (1977)2nd Australian (1977)3rd European (1977)ISAM (1986)GISSI-1 (1986)ISIS-2 (1988)

NameStudy

-1.84 (-4.23, 0.55)-1.04 (-2.26, 0.18)-0.01 (-0.65, 0.63)0.10 (-0.36, 0.56)0.07 (-0.31, 0.46)-0.21 (-0.47, 0.05)-0.30 (-0.54, -0.06)-0.30 (-0.52, -0.07)-0.27 (-0.49, -0.05)-0.25 (-0.47, -0.04)-0.24 (-0.46, -0.03)-0.22 (-0.41, -0.03)-0.21 (-0.40, -0.02)-0.27 (-0.45, -0.10)-0.28 (-0.45, -0.11)-0.21 (-0.37, -0.05)-0.21 (-0.37, -0.05)-0.21 (-0.36, -0.06)-0.26 (-0.41, -0.11)-0.24 (-0.38, -0.11)-0.23 (-0.31, -0.14)-0.26 (-0.32, -0.19)

ratio (95% CI)Log odds

-1.84 (-4.23, 0.55)-1.04 (-2.26, 0.18)-0.01 (-0.65, 0.63)0.10 (-0.36, 0.56)0.07 (-0.31, 0.46)-0.21 (-0.47, 0.05)-0.30 (-0.54, -0.06)-0.30 (-0.52, -0.07)-0.27 (-0.49, -0.05)-0.25 (-0.47, -0.04)-0.24 (-0.46, -0.03)-0.22 (-0.41, -0.03)-0.21 (-0.40, -0.02)-0.27 (-0.45, -0.10)-0.28 (-0.45, -0.11)-0.21 (-0.37, -0.05)-0.21 (-0.37, -0.05)-0.21 (-0.36, -0.06)-0.26 (-0.41, -0.11)-0.24 (-0.38, -0.11)-0.23 (-0.31, -0.14)-0.26 (-0.32, -0.19)

ratio (95% CI)Log odds

Favors Treatment Favors Control 0-4.23 0 4.23

Source: Data accessed at http://www.stata-press.com/data/mais.html from Lau, J., Elliott, M., Antman, M. D., Jimenez-Silva, J., Kupelnick, B., Mosteller, F., & Chalmers, T. C. (1992). Cumulative meta-analysis of therapeutic trials for myocardial infarction. The New England Journal of Medicine, 327, 248-254.

14


General suggestions – forest plots •  Always include forest plots (or summary forest plots) if possible/appropriate •  Not recommended with fewer than 2 studies •  Plot ratio effect size measures on the log scale, but include axis labels on

the original anti-logged scale •  Include reference lines at the null value •  State the confidence level for confidence intervals •  Blocks for each study should be proportionate to study weight •  Sort studies in a meaningful order (e.g., effect size magnitude) •  State the direction of results •  Include prediction intervals for random effects analyses •  Include numerical data on plots (if possible)


What’s wrong with this forest plot?

Overall (I-squared = 74.6%, p = 0.000)

Study

7

2

10

8

4

ID

6

5

9

1

3

0-3.31 0 3.31

15




Study

7

2

10

8

4

ID

6

5

9

1

3

0-3.31 0 3.31

Uninformative study labels




Study

7

2

10

8

4

ID

6

5

9

1

3

0-3.31 0 3.31


Seemingly random order of effect sizes

16




Study

7

2

10

8

4

ID

6

5

9

1

3

0-3.31 0 3.31



Unclear direction of effect sizes




Study

7

2

10

8

4

ID

6

5

9

1

3

0-3.31 0 3.31




Does not include data

17




Study

7

2

10

8

4

ID

6

5

9

1

3

0-3.31 0 3.31





Unspecified confidence level




Study

7

2

10

8

4

ID

6

5

9

1

3

0-3.31 0 3.31





Unspecified confidence level

General aesthetics (white space)

18


FUNNEL PLOTS


Funnel plots •  Exploratory tool used to visually assess the possibility of

publication/small study bias in a meta-analysis •  Scatter plot of effect size (x-axis) against some measure of

study size (y-axis) –  x-axis: use log scale for ratio effect size measures, e.g., ln(OR),

ln(RR) –  y-axis: the standard error of the effect size is generally

recommended (see Sterne et al., 2005 for a review of additional y-axis options)

•  Not recommended in very small meta-analyses (e.g., n < 10)

19


Funnel plots

•  If publication bias is present, you would expect null or ‘negative’ findings from small n studies to be suppressed (i.e., missing from the plot)

•  Asymmetry in the funnel plot for small n studies may provide evidence of possible publication bias

•  Symmetry in the funnel plot provides some evidence against the possibility of publication bias


Funnel plots

Source: Wilson, S. J., Tanner-Smith, E. E., Lipsey, M. W., Steinka-Fry, K., & Morrison, J. (2011). Dropout prevention and intervention programs: Effects on school completion and dropout among school aged children and youth. Campbell Systematic Reviews, 8. doi: 10.4073/csr.2011.8

01

23

45

Sta

ndar

d E

rror

for L

OR

-10 -5 0 5 10Log Odds Ratio

Funnel plot with pseudo 95% confidence limits

20


Funnel plots

Source: Mazerolle , L., Bennett, S., Davis, J., Sargeant, E., & Manning, M. (2013). Legitimacy in policing: A systematic review. Campbell Systematic Reviews,1. doi:10.4073/csr.2013.1


Funnel plots •  Asymmetry could be due to factors other than publication

bias, e.g., –  Poor methodological quality –  Other reporting biases –  Artefactual variation –  Chance –  True heterogeneity

•  Assessing funnel plot symmetry relies entirely on subjective visual judgment

21


Contour enhanced funnel plots •  Funnel plot with additional contour lines associated with

‘milestones’ of statistical significance: p = .001, .01, .05, etc. –  If studies are missing in areas of statistical non-significance,

publication bias may be present –  If studies are missing in areas of statistical significance,

asymmetry may be due to factors other than publication bias –  If there are no studies in areas of statistical significance,

publication bias may be present •  Can help distinguish funnel plot asymmetry due to

publication bias versus other factors


0

.5

1

1.5

Sta

ndar

d er

ror

-4 -2 0 2 4Log odds ratio (lor)

Studies

p < 1%

1% < p < 5%

5% < p < 10%

p > 10%

Contour enhanced funnel plots

Source: Data accessed at http://www.stata-press.com/data/mais.html

22


General suggestions – funnel plots •  Not recommended with fewer than 10 studies •  Plot effect sizes on the horizontal axis •  Plot the standard error of the effect size on the vertical axis (generally) •  Plot ratio effect size measures on the log scale, but include axis labels on the

original anti-logged scale •  All points should be the same size (weights/precision represented in the

vertical axis) •  Include 95% pseudo-confidence limits from a fixed effect analysis •  Include contours if possible •  Data in graphs should generally be available elsewhere in the review

(except in very large reviews) •  Use different plotting symbols to distinguish subgroups, when appropriate


What’s wrong with this funnel plot?

-.2-.1

0.1

.2

Effe

ct s

ize

0 .05 .1 .15 .2Standard error

23



-.2-.1

0.1

.2

Effe

ct s

ize


Effect size on vertical axis


What’s wrong with this funnel plot? -.2 -.1 0 .1 .2

Effect size

0.05

.1.15

.2S

tandard error


24



-.2-.1

0.1

.2

Effe

ct s

ize



Points are not all the same size



-.2-.1

0.1

.2

Effe

ct s

ize




Vague labeling of axes and reference line

25



-.2-.1

0.1

.2

Effe

ct s

ize




Vague labeling of axes and reference line

No confidence bands


Bubble plots •  Scatter plot of a study covariate (x-axis) against effect size

(y-axis) •  Useful to characterize covariates that may be a source of

heterogeneity •  Provides a visual representation of results from a bivariate

meta-regression model

26


Bubble plots

Source: Data accessed at http://www.stata-press.com/data/mais.html from Thompson, S. G., & Sharp, S. G. (1999). Explaining heterogeneity in meta-analysis: A comparison of methods. Statistics in Medicine, 18, 2693-2708.

-3.0

0-2

.00

-1.0

00.

00

Sta

ndar

dize

d m

ean

diffe

renc

e ef

fect

siz

e

4 6 8 10 12Duration of follow-up (weeks)


Bubble plots


-3-2

-10

1

Sta

ndar

dize

d m

ean

diffe

renc

e

4 6 8 10 12Duration of follow-up (weeks)

Confidence intervalLinear predictionSMD effect sizePrediction including random effects

27


General suggestions – bubble plots •  Plot effect sizes on the vertical axis •  Plot the covariate on the horizontal axis •  Plot ratio effect size measures on the log scale, but include

axis labels on the original anti-logged scale •  Points should be proportionate to study weight •  Include fitted meta-regression line (if appropriate) •  Data in graphs should generally be available elsewhere in

the review (except in very large reviews)


Other graphs – Galbraith/radial plots •  Scatter plot of inverse standard error (x-axis) against a

standardized effect size (i.e., effect size divided by its standard error) (y-axis)

•  Includes an unweighted regression line constrained through the origin with slope equal to the fixed effect summary effect size estimate

•  Useful for displaying heterogeneity and aiding detection of outliers

•  Useful for displaying effect sizes in very large reviews where forest plots may be impractical

28


Other graphs – Galbraith/radial plots


g/SE

(g)

1/SE(g) 0 4.16667

-6.21956

-2

0

2 2

Favo

rs tr

eatm

ent

Fav

ors

cont

rol




g/SE

(g)

1/SE(g) 0 4.16667

-6.21956

-2

0

2 2

Favo

rs tr

eatm

ent

Fav

ors

cont

rol

More precise estimates lie farther from the origin

29




More precise estimates lie farther from the origin

Vertical scatter illustrates heterogeneity

g/SE

(g)

1/SE(g) 0 4.16667

-6.21956

-2

0

2 2

Favo

rs tr

eatm

ent

Fav

ors

cont

rol


Other graphs – Galbraith/radial plots •  Points should be the same size for study (weight/precision is

represented in the horizontal axis) •  Include confidence intervals around the fixed effect summary

effect line •  Use different plotting symbols to distinguish subgroups,

when appropriate

30


Other graphs – L’abbé plots •  Plot of control group risk (x-axis) against treatment group risk

(y-axis) •  Commonly used to depict risks, but can also be plotted on

log risk or log odds scales •  Most commonly used for binary outcome data, but can be

extended to depict means for continuous outcomes or ROC plot for diagnostic/screening test accuracy

•  Can also be used to contrast different effect size metrics (odds ratio, risk ratio, risk difference)


Other graphs – L’abbé plots

0.2

5.5

.75

1

Trea

tmen

t gro

up e

vent

rate

0 .25 .5 .75 1Control group event rate

31


Other graphs •  Density strips •  Raindrop plots •  Graphical display of study

heterogeneity (GOSH) •  CUSUM chart •  Veritas plot

•  Summary receiver-operator curve (SROC) graphs

•  Cross hairs ROC plot •  Harvest plot •  Baujat plots


Software resources •  CMA •  OpenMeta •  R •  RevMan •  SAS •  SPSS •  Stata

32


Software resources Forest plot

Summary forest plot

Cumulative forest plot

Funnel plot

Contour funnel plot

Trim and fill funnel plot

Galbraith plot

l’Abbé plot

CMA ü ü ü ü û ü û û OpenMeta ü ü ü ü û û û ü MIX ü ü ü ü ü ü ü ü R ü ü ü ü ü ü ü ü RevMan ü ü û ü ü û û û SAS ü ü ü ü û ü ü û SPSS ü û û ü û û û û Stata ü ü ü ü ü ü ü ü

Adapted from: Schild, A. H. E., & Voracek, M. (2013). Less is less: A systematic review of graph use in meta-analyses. Research Synthesis Methods, in press.


Summary •  Graphs are an important part of any meta-analysis and can

greatly facilitate interpretation, when used appropriately •  Forest plots should (almost always) be included in a Campbell

review •  Funnel plots, bubble plots, Galbraith plots, L’Abbe plots, or other

various plots may also be appropriate •  Always follow standard graphing principles, and strive for

accuracy, simplicity, clarity, aesthetic appeal, and good structure •  Making good graphs can take time/effort – using software

defaults is rarely sufficient!

33


Recommended reading Anzures-Cabrera, J., & Higgins, J. P. T. (2010). Graphical displays in meta-analysis: An overview with

suggestions for practice. Research Synthesis Methods, 1, 66-80. Borman, G. D., & Grigg, J. A. (2009). Visual and narrative interpretation. Pp. 497-519 in H. Cooper, L.

V. Hedges, & J. C. Valentine (Eds). The handbook of research synthesis and meta-analysis. New York: Russell Sage.

Galbraith, R. F. (1988). A note on graphical presentation of estimated odds ratios from several clinical trials. Statistics in Medicine, 7, 889-894.

Higgins, J. P. T. (2003). Considerations and recommendations for figures in Cochrane reviews: Graphs of statistical data. Cochrane Statistical Methods Group. Available online at: http://www.cochrane.org/training/cochrane-handbook#supplements

Lane, P. W. et al. (2012). Graphics for meta-analysis. Pp. 295-308 in A. Krause & M. O’Connell (Eds.), A picture is worth a thousand tables: Graphics in life sciences. New York: Springer.

Schild, A. H. E., & Voracek, M. (c. 2013). Less is less: A systematic review of graph use in meta-analyses. Research Synthesis Methods, in press.

Schriger, D. L., et al. (2010). Forest plots in reports of systematic reviews: A cross-sectional study reviewing current practice. International Journal of Epidemiology, 39, 421-429.


P.O. Box 7004 St. Olavs plass 0130 Oslo, Norway

E-mail: [email protected]

http://www.campbellcollaboration.org

Documents

Graphical Representation of Meta-analysis Findings...Graphical Representation of Meta-analysis Findings Emily E. Tanner-Smith Associate Editor, Campbell Methods Coordinating Group