Meta-analysis when the normality assumptions are violated (2008)

Meta-Analysis overviewOur bit

To sum it up

Meta-analysis when the assumptions areviolated: which method is best?

A simulation study

Evan Kontopantelis David Reeves

National Primary Care Research and Development CentreUniversity of Manchester

NPCRDC Seminar, 30th September 2008

Kontopantelis, Reeves Is MA with non-normal effects safe?


To sum it up

Outline

1 Meta-Analysis overviewGeneral informationThe heterogeneity issueMethods and problems

2 Our bitData generationMethods and performanceResults

3 To sum it upConclusionsRelevant projects



To sum it up

General informationThe heterogeneity issueMethods and problems

Meta-analysis?It’s all Greek to me.

’Meta’ is a Greek preposition meaning ’after’, someta-analysis =⇒ post-analysis.Efforts to pool results from individual studies back as far as1904.The first attempt that assessed a therapeutic interventionwas published in 1955.In 1976 Glass first used the term to describe a "statisticalanalysis of a large collection of analysis results fromindividual studies for the purpose of integrating thefindings".



To sum it up


How Meta-Analysis works.

We search for papersrelevant to the researchquestion and unsuitablepapers are filtered out.In each paper, for eachrelevant outcome, wecalculate an effect (ofintervention vs control) andits variance.Effects and their variancesare combined to calculatean overall effect and CI



To sum it up


Meta-analyses on the rise!

Meta-analysis studiesare used more and moresince they seem to be auseful ’summary’ tool.However critics arguethat one cannot combinestudies when they aretoo diverse(heterogenious).



To sum it up


Heterogeneity.The big bad wolf.

When the effect of the intervention varies significantly fromone study to another.It can be attributed to clinical and/or methodologicaldiversity.

Clinical: variability that arises from different populations,interventions, outcomes and follow-up times.Methodological: relates to differences in trial design andquality.

Detecting quantifying and dealing with heterogeneity canbe very hard.



To sum it up


Absence of heterogeneity.

Assumes that thetrue effects of thestudies are allequal anddeviations occurbecause ofimprecision ofresults.Analysed with thefixed-effectsmethod.



To sum it up


Presence of heterogeneity.

Assumes thatthere is variation inthe size of the trueeffect amongstudies (in additionto the imprecisionof results).Analysed withrandom-effectsmethods.



To sum it up


Random effect MA methods.

Estimate the between-study variance and use it inestimating the overall effect.Parametric methods:

DerSimonian Laird (1986).Maximum and Profile Likelihood (1996).

Non-parametric methods:Permutations method (1999).Non-parametric Maximum Likelihood (1999).



To sum it up


’Potential’ problems with Meta-analysis.

Heterogeneity is common and the FE model is under fire.Methods are asymptotic: accuracy improves as studiesincrease. What if we only have a handful, as is usually thecase in health services research?Parametric RE models assume that both the effects anderrors are normally distributed.Almost all RE models (except PL) do not take into accountthe uncertainty in the between-study variance estimate.DL is the most common method of analysis (easy toimplement and widely available) but is it the best?



To sum it up


Research so far...

Very few investigations of the issues.It has been shown that the number of studies in the MAand the degree of heterogeneity affect methodperformance.Performance comparisons usually focus on coverage andignore power or have not included some importantmethods (e.g. PL, PE).Method robustness has not been assessed withnon-normal data.



To sum it up

Data generationMethods and performanceResults

Computing power makes it possible.



To sum it up


Outline of the process.In a nutshell...

Simulated various distributions for the trueeffects:

Normal.Skew-Normal.Bimodal.Beta.Uniform.

Created datasets of 10,000meta-analyses for various numbers ofstudies and different degrees ofheterogeneity, for each distribution.Compared all methods in terms ofcoverage, power and accuracy of CIestimation.



To sum it up


Statistical details.Part I - Seminar room doors have been locked...

For a single study we simulated the effect size estimateand the within-study variance estimate.Two forms of outcome:

Binary (Chi-square distribution for variance).Continuous (Normal distribution for variance).

The simulated effect size is the sum of two components:The true effect size - drawn from one of the previouslymentioned distributions.A normally distributed error term (representing error inmeasurement).



To sum it up


Statistical details.Part II - Seminar room doors are still locked...

The process was repeated using four heterogeneityvalues, indicating low, moderate, large and very largeheterogeneity.The process was repeated for different meta-analysis sizes(from 2 to 35)Therefore, for each distributional assumption we get34*4=136 datasets of 10,000 MA in eachSome methods (PL, PE) are computationally expensive,hence the whole process takes time!



To sum it up


MA methods, a brief description.Part I.

T-test (T): Does not try to separate within andbetween-study variance.Fixed effects (FE): Assumes between-study variance iszero and all variance is within-study.DerSimonian-Laird (DL): The simplest random effectmethod; makes simple estimates of both within andbetween-study variance.Q method (Q): Selects FE or DL based on the outcome ofa test for heterogeneity; it closely resembles whatresearchers do in practice.



To sum it up


MA methods, a brief description.Part II.

Maximum Likelihood (ML): Improves the variance estimateusing iteration.Profile Likelihood (PL): A more advanced version of MLthat uses nested iterations for converging.Permutations method (PE): Simulates the distribution ofthe overall effect using the observed data.



To sum it up


Performance.And how it is measured.

We evaluate the methods using:Coverage: Rate of true negatives when the overall trueeffect is zero.Power: Rate of true positives when the true overall effect isnon-zero.Confidence Interval performance: a measure of how widethe (estimated around the effect) CI is, compared to its truewidth.



To sum it up


Finally, after two years of computations...i.e. blame us for the climate change.



To sum it up


No heterogeneity.Zero between study variance.



To sum it up


Coverage performance.Large heterogeneity under various distributional assumptions.



To sum it up


Power performance.Large heterogeneity under various distributional assumptions.



To sum it up


Confidence Interval performance.Large heterogeneity under various distributional assumptions.



To sum it up


Coverage by each method.Across various between-study variance distributions.



To sum it up


Power by each method.Across various between-study variance distributions.



To sum it up

ConclusionsRelevant projects

Piece of cake!



To sum it up


Summary.

Within any given method, the results were consistentacross all types of distribution shape.This gives confidence that methods are highly robustagainst even severe violations of the assumption ofnormally distributed effect sizes.If it is reasonable to assume that the effect size does notvary between studies, the FE, Q and ML methods allprovide accurate coverage coupled with good power.However, in the presence of heterogeneity the picturechanges...



To sum it up


When heterogeneity is present.Almost always that is...

Even a moderate amount of between study variance altersthe picture considerably.FE, Q and ML quickly lose coverage as heterogeneityincreases.DL rapidly goes from providing a coverage that is overlyhigh, to one that is overly low.PE, and to a lesser extend PL, now provide the bestcoverage, even with very small sample sizes.



To sum it up


Which method then?

If the priority is an accurate Type I error rate (false positive)then the simple T-test is the best method. But low powermakes it a poor choice when control of the Type II errorrate (false negative) is also important.PE gives accurate coverage in all situations and has betterpower than T, but the method is more difficult to implementand cannot be used with less than 6 studies.For very small study numbers (≤5) only PL and T givecoverage >90% and only PL gives an acccurate CI.PL has a ’reasonable’ coverage in most situations, giving itan edge over other methods.The most commonly used method, DL, is not the best.



To sum it up


Current and future work

Created a freely available MS Excel add-in thatimplements all the described MA methods and variousmeasures of heterogeneity.Working on a STATA module that will do the same.Investigate performance of heterogeneity measures undernon normally distributed data.


AppendixMA Excel add-inKey references

Meta-analysis Excel add-in, key advantages.Just in case you are interested...

Easy to use and time saving.The extracted data from each study are easily accessible,they can be quickly edited or corrected and the analysisrepeated.Includes a choice of seven Meta Analysis models includedthe advanced PL, ML and PE not currently available inother software packages.Provides a descriptive forest plot allowing multipleoutcomes per study.Reports a wide range of heterogeneity measures.Effect sizes and standard errors can be exported for use inother MA software packages.

Play video clip



References. IJust in care you are REALLY interested...

Brockwell SE, Gordon IR.A comparison of statistical methods for meta-analysis.Statistics in Medicine, 20(6):825-840, 2001.

Engels EA, Schmid CH, Terrin N, Olkin I, Lau J.Heterogeneity and statistical significance in meta-analysis:an empirical study of 125 meta-analyses.Statistics in Medicine, 19(13):1707-1728, 2000.

Follmann DA, Proschan MA.Valid inference in random effects meta-analysis.Biometrics, 55(3):732-737, 1999.



References. IIJust in care you are REALLY interested...

Hardy RJ, Thompson SG.A likelihood approach to MA with random effects.Statistics in Medicine, 15(6):619-629, 1996.

Micceri T.The Unicorn, the Normal Curve, and Other ImprobableCreatures.Psychological Bulletin, 105(1):156-166, 1989.

Ramberg JS, Dudewicz EJ, Tadikamalla PR, Mykytka EF.A Probability Distribution and Its Uses in Fitting Data.Technometrics, 21(2):201-214, 1979.


Science

Meta-analysis when the normality assumptions are violated (2008)