A simple confidence interval for meta analysis

STATISTICS IN MEDICINEStatist. Med. 2002; 21:3153–3159 (DOI: 10.1002/sim.1262)

A simple con�dence interval for meta-analysis

Kurex Sidik1 and Je�rey N. Jonkman2;∗;†

1Biometrics Research; Wyeth Research; CN 8000; Princeton; NJ 08543-8000; U.S.A.2Department of Mathematics and Statistics; Mississippi State University; Mississippi State; MS 39762; U.S.A.

SUMMARY

In the context of a random e�ects model for meta-analysis, a number of methods are available toestimate con�dence limits for the overall mean e�ect. A simple and commonly used method is theDerSimonian and Laird approach. This paper discusses an alternative simple approach for constructingthe con�dence interval, based on the t-distribution. This approach has improved coverage probabilitycompared to the DerSimonian and Laird method. Moreover, it is easy to calculate, and unlike somemethods suggested in the statistical literature, no iterative computation is required. Copyright ? 2002John Wiley & Sons, Ltd.

KEY WORDS: coverage probability; random e�ects model; simulation study; t-distribution; weightedestimation

1. INTRODUCTION

In the random e�ects model for meta-analysis, one important objective is to calculate aninterval estimate for the overall mean e�ect across studies. Recently, Brockwell and Gordon[1] presented a summary and carried out comparisons of the existing methods for the intervalestimate in terms of coverage probability. They found that the con�dence intervals from all themethods they studied generally had coverage probabilities below the speci�ed nominal level.In particular, the commonly used DerSimonian and Laird random e�ects method [2] producedcoverage probabilities below the nominal level. Although the pro�le likelihood interval ofHardy and Thompson [3] led to the highest coverage probabilities, the pro�le likelihoodapproach is not simple computationally, and involves an iterative calculation, as does thesimple likelihood method [1].In this paper we present a simple approach for constructing a 100(1−�) per cent con�dence

interval for the overall mean e�ect in the random e�ects model. The interval is de�ned usingpivotal inference based on the t-distribution. It is simple because, like the DerSimonian andLaird method, no iterative computation is needed. Moreover, we �nd that this simple approachhas a better coverage probability than the DerSimonian and Laird method.

∗ Correspondence to: Je�rey N. Jonkman, Department of Mathematics and Statistics, Mississippi State University,Mississippi State, MS 39762, U.S.A.

† E-mail: [email protected] June 2001

Copyright ? 2002 John Wiley & Sons, Ltd. Accepted December 2001

3154 K. SIDIK AND J. N. JONKMAN

Note that the aim of this paper is to present a brief discussion of the simple intervalin comparison with the commonly used DerSimonian and Laird random e�ects method. Thescope of this paper is purely in the non-Bayesian statistical framework. Therefore, we will onlycite a few papers relevant to the study. For a more complete summary, and for comparisonsof the di�erent methods of obtaining an interval estimate, we refer to the paper by Brockwelland Gordon [1]. For more detailed reviews of the various methods in meta-analysis, we referthe reader to the book by Hedges and Olkin [4], the brief note by Olkin [5], and the paperby Normand [6].

2. METHOD

For k independent studies in meta-analysis, let the random variable yi be an e�ect size esti-mate from the ith study. Some commonly used measures of e�ect size are mean di�erence,standardized mean di�erence, risk di�erence, relative risk and odds ratio. The odds ratio isfrequently of particular interest in retrospective or case control studies. The random e�ectsmodel can be de�ned as follows:

yi=�+ �i + �i; i=1; : : : ; k

where �i represents the random error across the studies, �i represents the random error within astudy, and �i and �i are assumed to be independent. Furthermore, it is assumed that �i∼N(0; �2)and �i∼N(0; �2i ). The parameter �2 is a measure of the heterogeneity between studies and issometimes called the heterogeneity variance. Therefore, the individual e�ect size yi in therandom e�ects models for meta-analysis is normally distributed with mean � and variance�2i + �

2.Consider construction of a con�dence interval for the overall mean e�ect � in the random

e�ects model. There are several methods for obtaining an interval estimate for this model [1].A simple and commonly used method is the DerSimonian and Laird approach [2]. The crucialstep of this method is to estimate the heterogeneity variance �2. DerSimonian and Laird’s [2]estimator, �2, is given by

�2= max

0;

∑ki=1 wi(yi − �)2 − (k − 1)∑ki=1wi −

∑ki=1w

2i

/∑ki=1wi

(1)

where � is the weighted estimate of the mean e�ect, given by �=∑k

i=1 wiyi=∑k

i=1 wi. Here,the weight wi=1=�2i is assumed to be known. Usually the sample estimate �

2i is substituted in

place of �2i , so that wi=1=�2i is used in practice. Using the estimate �

2, a 100(1−�) per centapproximate con�dence interval for � based on the DerSimonian and Laird random e�ectsmethod is given by

� ± z�=2 1√(∑ki=1 wi

) (2)

where wi=1=(�2i + �2) and �=

∑ki=1 wiyi=

∑ki=1 wi. Note that z�=2 is the �=2 upper quantile of

the standard normal distribution.

Copyright ? 2002 John Wiley & Sons, Ltd. Statist. Med. 2002; 21:3153–3159

SIMPLE CONFIDENCE INTERVAL FOR META-ANALYSIS 3155

In this paper, we consider an alternative simple interval estimate for � in the random e�ectsmodel. Under the assumptions that yi∼N(�; �2i +�2) and that wi is the correct weight for eachi, we have

Zw=� − �

1=√(∑k

i=1 wi) ∼ N(0; 1) and Qw=

k∑i=1wi(yi − �)2 ∼ �2(k−1)

It can be shown that Zw and Qw are independent (see the proof in the Appendix). Hence√(∑ki=1 wi

)(� − �)√{∑k

i=1 wi(yi − �)2=(k − 1)} ∼ t(k−1) (3)

Now, using this pivotal quantity, we may construct an approximate 100(1 − �) per centcon�dence interval for � as follows:

� ± tk−1; �=2

√√√√{∑ki=1 wi(yi − �)2(k − 1)∑k

i=1 wi

}(4)

where tk−1; �=2 is the �=2 upper quantile from the t-distribution with k−1 degrees of freedom.Notice that, under the assumption of known weights, Qw=[(k−1)

∑wi] is an unbiased estimator

of the variance of �.Strictly speaking, the normal distribution of Zw and the chi-square distribution of Qw

shown above require the assumption that the weights, and hence the true variance �2i + �2

of yi (i=1; : : : ; k), are known. Nevertheless, we relax this requirement by substituting theestimate �2i + �

2 in place of the true variance. Although using the estimated values may raiseproblems for the validity of the distributional assumptions, it is the common practice in statis-tical inference based on weighted estimation. To be speci�c, it is usually assumed that weightsare known, even though the actual applied weights are functions of statistics, and then oneobtains the estimate and con�dence interval based on normal or chi-square distributions, asin the case of the DerSimonian and Laird method.

3. SIMULATION

To compare the simple con�dence interval based on the t-distribution with the DerSimonianand Laird interval in terms of coverage probability, we performed a simulation study of meta-analysis for the random e�ects model. Throughout the study, the overall mean e�ect � is�xed at 0.5 and the error probability of the con�dence interval, �, is set at 0.05. We useonly one value for � because the t-distribution interval based on the pivotal quantity in (3)and the DerSimonian and Laird interval are both invariant to a location shift. Four di�erentvalues of �2 are used: 0.03; 0.05; 0.08, and 0.1. For each �2, twelve di�erent values of kare considered, that is, k=5(5)60. The number of simulation runs for the meta-analysis of kstudies is 10 000. The simulation data for each run are generated in terms of the most popularmeasure of e�ect size in meta-analysis, the log of the odds ratio. That is, the generated e�ect



size yi is interpreted as a log odds ratio. For given k, the within-study variance �2i is generatedusing the method of Brockwell and Gordon [1]. Speci�cally, a value is generated from a chi-square distribution with one degree of freedom, which is then scaled by 1=4 and restrictedto an interval between 0.009 and 0.6. This results in a bimodal distribution of �2i , with themodes at each end of the distribution. As noted by Brockwell and Gordon, values generatedin this way are consistent with a typical distribution of �2i for log odds ratios encounteredin practice. For binary outcomes, the within-study variance decreases with increasing samplesize, so large values of �2i (close to 0:6) represent small trials included in the meta-analysis,and small values of �2i represent large trials. The e�ect size yi for i=1; : : : ; k is generatedfrom a normal distribution with mean � and variance �2i + �

2.For each simulation of the meta-analysis, the con�dence intervals based on the t-distribution

and the DerSimonian and Laird method are calculated. The numbers of intervals containingthe true � are recorded for both methods. The proportion of intervals containing the true �(out of the 10 000 runs) serves as the simulation estimate of the true coverage probability.The results of the simulation study are presented in Figure 1. From the plots, it can be

seen that the coverage probabilities of the interval based on the t-distribution are larger thanthe coverage probabilities of the interval using the DerSimonian and Laird method for each�2 and all values of k. Although the coverage probabilities of the con�dence interval fromthe t-distribution, like other methods [1], are below the nominal level of 95 per cent, theyare higher than the commonly applied interval based on the DerSimonian and Laird method,particularly when k is small. This suggests that the simple con�dence interval based on thet-distribution is an improvement compared to the existing simple con�dence interval based onDerSimonian and Laird’s method.The coverage probabilities may be a�ected by the ratio of the heterogeneity between studies

(�2) to the within-study variances (�2i ). In the simulation study, the two most extreme valuesof �2 are 0:03 and 0:1. Therefore the ratio �2=�2i ranges between 0:05 and 3:33 when �

2=0:03,and between 0:167 and 11:11 when �2=0:1. That is, the between-study variance may be asmuch as 11 times the smallest within-study variance, or as little as one-twentieth of the largestwithin-study variance.We also investigated the expected lengths of the con�dence intervals from the two methods

for the four values of �2 and the twelve values of k in the simulation. We found that the sim-ulation estimate of the expected length of the interval for the t-distribution ranged between0.158 and 1.003, with an average of 0.339. The estimated expected length of the intervalusing the DerSimonian and Laird method ranged between 0.155 and 0.733, with an averageof 0.305.

4. DISCUSSION

In this paper we have discussed a simple way to construct a con�dence interval for theoverall mean e�ect in a random e�ects model for meta-analysis. The interval is based onthe t-distribution and requires no iterative computation. Moreover, we found that the con-�dence interval has higher coverage probability than the commonly used interval based onDerSimonian and Laird’s method, particularly when the number of studies is small or moder-ate. Furthermore, our interval given in (4) takes sample size into account by incorporating themeta-analysis sample size, k, in its computation, unlike the interval given in (2). Therefore,



k

Cov

erag

e P

rob.

10 20 30 40 50 60

0.80

0.85

0.90

0.95

1.00

DL

TD

(a) k

10 20 30 40 50 60

DL

TD

(b)

k

10 20 30 40 50 60

DL

TD

(c) k

10 20 30 40 50 60

DL

TD

(d)

Cov

erag

e P

rob.

0.80

0.85

0.90

0.95

1.00

Cov

erag

e P

rob.

0.80

0.85

0.90

0.95

1.00

Cov

erag

e P

rob.

0.80

0.85

0.90

0.95

1.00

Figure 1. Coverage probabilities of the intervals from t-distribution (TD) and DerSimonian and Laird(DL) methods: (a) �2=0:03; (b) �2=0:05; (c) �2=0:08; (d) �2=0:1.

we recommend the interval based on the t-distribution as an alternative approximate intervalfor the overall mean e�ect in a random e�ects model for meta-analysis.A drawback of this interval is that it results in coverage probabilities that are lower than

the nominal level. However, it has been previously shown that intervals using other methodsalso have coverage probability below the speci�ed nominal level [1]. Although Brockwelland Gordon [1] found that the coverage probability of the interval using the pro�le likelihoodmethod is quite close to its speci�ed level, they noted that it lacks the simplicity of theother intervals. The low coverage probabilities for the interval based on the t-distribution maybe attributed to the fact that estimated values of �2 and �2i are used in place of the truevalues, and therefore the pivotal quantity given in (3) does not follow an exact t-distribution.Nevertheless, we think that the simple interval based on the t-distribution is useful as a quickway to construct an approximate con�dence interval for the mean e�ect in a preliminary study.



We have illustrated our interval by using the log-odds ratio to measure e�ect size. Engelset al. [7] observed that di�erent metrics for e�ect size assign di�erent weights to the studiesin a summary meta-analysis. We cannot be certain that our results would be the same for anabsolute measure such as a risk di�erence, but we do note that Engels et al. [7] studied 125meta-analyses and concluded that the choice of metric for treatment e�ect did not appear tobe of crucial importance.Finally, it should be noted that we have only considered an interval estimate for the mean

e�ect in the context of a random e�ects model for meta-analysis. However, the simple intervaldescribed in this paper also can be applied in the �xed e�ects model. Since it is assumedthat �2=0 for the �xed e�ects model, the interval given in (4) may be applied to constructan approximate 100(1 − �) per cent interval for the overall mean e�ect by using wi=1=�2iinstead of wi=1=(�2i + �

2).

APPENDIX

To prove the independence of Zw and Qw, we use matrix notation. We assume thatY∼MN(U;V), where Y=(y1 · · ·yk)T, U=�1k×1, and V=diag{w−1

i }k×k , and we assumethat wi is the correct weight for each i. Using matrix notation, we have

(k∑i=1wi

)1=2�=AY and

k∑i=1wi(yi − �)2=YTBY

where A=(∑k

i=1 wi)−1=21TV−1 and B=V−1 − V−11(1TV−11)−11TV−1. Note that BV is an

idempotent matrix. Now we note that

AVB=(

k∑i=1wi

)−1=21TV−1V[V−1 −V−11(1TV−11)−11TV−1]

=(

k∑i=1wi

)−1=2[1TV−1 − 1TV−11(1TV−11)−11TV−1]

=(

k∑i=1wi

)−1=2[1TV−1 − 1TV−1]

= 0

Therefore the linear form (∑k

i=1 wi)1=2� and the quadratic form

∑ki=1 wi(yi−�)2 are independent

by Graybill’s theorem 4.5.2 [8], and hence Zw and Qw are independent.

ACKNOWLEDGEMENTS

We are grateful to Dr Thomas Copenhaver and Dr Sharon-Lise Normand for their support of ourresearch endeavours, and to the two referees whose comments helped improve the manuscript.



REFERENCES

1. Brockwell SE, Gordon IR. A comparison of statistical methods for meta-analysis. Statistics in Medicine 2001;20:825–840.

2. DerSimonian R, Laird N. Meta-analysis in clinical trials. Controlled Clinical Trials 1986; 7:177–188.3. Hardy RJ, Thompson SG. A likelihood approach to meta-analysis with random e�ects. Statistics in Medicine1996; 15:619–629.

4. Hedges LV, Olkin I. Statistical Methods for Meta-Analysis. Academic Press: New York, 1985.5. Olkin I. Meta-analysis: methods for combining independent studies. Editor’s introduction. Statistical Science1992; 7:226.

6. Normand S-LT. Tutorial in biostatistics, meta-analysis: formulating, evaluating, combining, and reporting.Statistics in Medicine 1999; 18:321–359.

7. Engels EA, Schmid HC, Terrin N, Olkin I, Lau J. Heterogeneity and statistical signi�cance in meta-analysis: anempirical study of 125 meta-analyses. Statistics in Medicine 2000; 19:1707–1728.

8. Graybill FA. Theory and Application of the Linear Model. Duxbury Press: Massachusetts, 1976.


Documents

A simple confidence interval for meta analysis