Systematic Synthesis of the Literature: Introduction to Meta-analysis

Linda N. Meurer, MD, MPHDepartment of Family and Community Medicine

. . It is necessary, while formulating the problems of which in our advance we are to find solutions, to call into council the views of those of our predecessors who have declared an opinion on the subject, in order that we may profit by whatever is sound in their suggestions and avoid their errors.

Aristotle, De Anima

Clinical Overview Purpose:

Provide general information on a topic Good to review diagnosis and management Disseminate experience and opinions of an expert

Methods: Usually don’t include a methods secsion References chosen to illustrate points Conclusions may or may not be “evidence-based”

Critical/ Systematic Review Purpose

More focused topic; answers specific question(s) Represent a summary of systematically gathered and analyzed

primary research May lead to new conclusions/ knowledge Saves the busy clinician the work of interpreting multiple studies

on the same subject

Methods Should always include methods section with at least:

Study inclusion criteria, search strategy, analysis method References chosen through clear criteria to minimize author bias

Key characteristics of a systematic review

Clearly stated title and objectives Comprehensive strategy to search for relevant

studies (unpublished and published) Explicit and justified criteria for the inclusion or

exclusion of any study Clear presentation of characteristics of each study

included and an analysis of methodological quality Comprehensive list of all studies excluded and

justification for exclusion

Key characteristics of a systematic review (cont.)

Clear analysis of the results of the eligible studies statistical synthesis of data (meta-analysis) if

appropriate and possible; or qualitative synthesis

Structured report of the review clearly stating the aims, describing the methods and materials and reporting the results

Meta-analysis – Systematic Review with statistical synthesis

Purpose Usually answers one specific question Can generate summary estimates of effect from multiple

studies Considered primary research with included studies treated as

data

Methods Identical to other types of Systematic Reviews Explicit, systematic collection of studies Uses statistical procedures to combine data or results from

different but similar studies

Meta-analysis - advantages Increase statistical power

Resolve uncertainty when reports disagree

Improve precision of estimates of effect size

Answer questions not posed at the beginning of original studies through examination of study differences, sensitivity analyses

0.25 0.5 1.0 2.0 4.0

XXX

XX

XX

X

XX

Relative Risk

Example: Forrest Plot

Meta-analysis results often displayed graphically

Each X = results of a single study

Horizontal lines = 95% CI -X- represents weighted

summary estimate after combining all studies. Note better precision Most studies not significant

by themselves contribute to highly significant summary

Threats to validity

When considering whether the results of any study reflect ‘truth’, there are generally 4 threats:

Selection bias Study sample doesn’t represent the population of interest

Information bias Measurement errors, misclassification etc.

Confounding Association between variables due to or affected by their shared

association with another variable Chance

The probability that data reveals an association that is not real

Meta-analysis - limitations

Threat #1: Selection bias In the case of meta-analysis, reflects bias in the

selection or availability of studies included: Retrieval bias: Investigator conducting review

selects studies that support hypothesis (or are otherwise biased)

Reporting bias: Investigators of original studies only report data that supports view (e.g. drug sponsored?)

Publication bias: Only studies with statistically significant results make it to the journals

Minimizing selection bias Retrieval bias:

Systematic protocol a priori (before study starts) that includes Clear selection criteria Explicit exhaustive search for relevant articles Multiple reviewers

Reporting bias: Examine the source of support for work Conduct sub-analyses to see if source influences results

Publication bias: Seek unpublished sources of data Demonstrate through use of a funnel plot

Funnel Plot

A scatterplot of individual study results

(effect size) on the x-axis;

A measure of study size on the y-axis

As sample size goes up variance decreases a funnel shape forms

-4 -2 0 2 4 6 8

If a publication bias exists:

You might see a skewed plot

Hole in the funnel plot around the null suggests a bias

Results in an overestimate of pooled effect

-4 -2 0 2 4 6 8

Meta-analysis: potential threats to validity

Threat #2: Information biasQuality of a meta-analysis is dependent on

quality of original articles, including: Selection Measurement Confounding

The author should conduct a very careful validity assessment of each article included in the study

Meta-analysis: potential threats to validity

Threat #3: Confounding As with information bias, confounding in individual

studies will be transmitted into the meta-analysis.

Differences in populations studied, settings, specific intervention details (dose, duration), measurements used, etc. may result in differences in study results

This can increase generalizability if studies agree If studies do not agree, may need to explore confounders that

might account for disagreement (heterogeneity)

Meta-analysis: potential threats to validity

Threat #4: Chance By combining the results of smaller studies, the

increased power achieved produces a more precise estimate with greater statistical significance

Assuming the included studies are valid, inter-study variability will still occur

Statistical testing for homogeneity can determine whether this variability is greater than one would expect due to chance alone

Tests for homogeneity

Test the probability that observed differences among the results of individual studies were due to chance alone.

Reported as a Cochrane Chi Square (Q-statistic): statistical significance shows results are not

homogenous Due to outliers? What do you do?

Heterogeneity A clue that differences in the studies exist that may lead to

new discoveries: Design Population: risk factors, setting Intervention: dose, duration, preparation Measurements

Finding heterogeneity should prompt an author to explore these factors more fully

Finding heterogeneity also influences the choice of statistical model used to combine the data

You’ve found heterogeneity.So what does one do? Try to explain

Eliminate obvious outliers and retestSubgroup analysesRegression analysis on study characteristics

Incorporate between-study differencesUse a random effects model

Method/model2 statistical models used Fixed effect model

estimates treatment effect as if all studies are estimating one single true value

ignores between study variability;

Used when study results are homogeneous

Random effects model estimates treatment effect

as if each study is estimating a distinct value from a distribution of possible results

accounts for between- study variability

Should be used when heterogeneity exists

Summary

You have been introduce to the basic concepts and terminology you need to critically use a meta-analysis, including: Purpose Advantages Potential threats to validity Analysis methods

Please return to the ANGEL course page (should still be open in another window) and click proceed to move on.