Lecture 5EPSY 642

Victor WillsonFall 2009

EFFECT SIZE DISTRIBUTION

Hypothesis: All effects come from the same distribution

What does this look like for studies with different sample sizes?

Funnel plot- originally used to detect bias, can show what the confidence interval around a given mean effect size looks likeNote: it is NOT smooth, since CI depends on

both sample sizes AND the effect size magnitude

EFFECT SIZE DISTRIBUTIONEach mean effect SE can be computed from

SE = 1/ (w)

For our 4 effects: 1: 0.200525 2: 0.373633 3: 0.256502 4: 0.286355

These are used to construct a 95% confidence interval around each effect

EFFECT SIZE DISTRIBUTION- SE of Overall MeanOverall mean effect SE can be computed

fromSE = 1/ (w)

For our effect mean of 0.8054, SE = 0.1297Thus, a 95% CI is approximately (.54, 1.07)The funnel plot can be constructed by

constructing a SE for each sample size pair around the overall mean- this is how the figure below was constructed in SPSS, along with each article effect mean and its CI

EFFECT SIZE DISTRIBUTION- Statistical testHypothesis: All effects come from the same

distribution: Q-testQ is a chi-square statistic based on the

variation of the effects around the mean effect

Q = wi ( g – gmean)2

Q 2 (k-1)

k

Example Computing Q Excel file

effect d w   Qi prob(Qi) sig?

1 0.58 5.43   0.7151598 0.397736175no

2 -0.05 10.24   0.7326248 0.392033721no

3 0.52 4.35   0.3957949 0.52926895no

4 0.02 9.69   0.366319 0.545017585no

5 -0.30 40.65   10.697349 0.001072891yes

6 0.14 29.94   0.1686616 0.681304025no

7 0.68 54.85   11.727452 0.000615849yes

8 -0.02 4.00   0.2125622 0.644766516no

             

  0.2154   Q= 25.015924    

      df 7    

      prob(Q)= 0.0007539    

Computational Excel fileOpen excel file: Computing QEnter the effects for the 4 studies, w for each

study (you can delete the extra lines or add new ones by inserting as needed)

from the Computing mean effect excel fileWhat Q do you get? Q = 39.57 df=3 p<.001

Interpreting QNonsignificant Q means all effects could have

come from the same distribution with a common mean

Significant Q means one or more effects or a linear combination of effects came from two different (or more) distributions

Effect component Q-statistic gives evidence for variation from the mean hypothesized effect

Interpreting Q- nonsignificantSome theorists state you should stop-

incorrect.Homogeneity of overall distribution does not

imply homogeneity with respect to hypotheses regarding mediators or moderators

Example- homogeneous means correlate perfectly with year of publication (ie. r= 1.0, p< .001)

Interpreting Q- significantSignificance means there may be

relationships with hypothesized mediators or moderators

Funnel plot and effect Q-statistics can give evidence for nonconforming effects that may or may not have characteristics you selected and coded for

MEDIATORSMediation: effect of an intervening variable

that changes the relationship between an independent and dependent variable, either removing it or (typically) reducing it.

Path model conceptualization:

Treatment

Outcome

Mediator

MEDIATORSStatistical treatment typically requires both

paths ‘a’ and ‘b’ to be significant to qualify as a mediator. Meta-analysis seems not to have investigated path ‘a’ but referred to continuous predictors as regressors

Lipsey and Wilson(2001) refer to this as “Weighted Regression Analysis”

Treatment

Outcome

Mediatora b

Weighted Regression AnalysisModel: e = b X + residualRegression analog: Q = Qregression + Qresidual

Analyze as “weighted least squares” in programs such as SPSS or SAS

In SPSS the weight function w is a variable used as the weighting

Weighted Regression AnalysisEmphasis on predictor and its standard error:

the usual regression standard error is incorrect, needs to be corrected (Hedges & Olkin, 1985):

SE’b = SEb / (MSe)½ where SEb is the standard error reported in

SPSS,and MSe is the reported regression mean

square error

Weighted Regression Q-statisticsQregression = Sum of Squaresregression

df = 1 for single predictorQresidual = Sum of Squaresresidual

df = # studies - 2

Significance tests: Each is a chi square test with appropriate degrees of freedom

9 8.99 9.05 31 19 1.7026 2 8.7813197 12.8 10.03 24 26 0.267 2 12.3699468 9.09 11.56 30 20 0.7561 2 11.229628 10.86 10.52 25 25 0.6532 1 11.8670849 7.73 7.86 22 28 1.5414 2 9.5303476 10.11 10.77 24 26 0.3507 1 12.2913387 8.57 6.91 34 16 0.4438 1 10.6517437 9.59 8.53 22 28 1.1245 2 10.6594098 7.98 10.92 30 20 0.542 1 11.5913849 12.69 8.16 28 22 0.6337 1 11.7392135 8.61 10.57 28 22 -0.5976 2 11.800795 9.34 7.43 24 26 0.3771 1 12.2623787 10.39 10.1 26 24 0.7234 2 11.7149136 8.66 9.4 21 29 0.2413 1 12.0942297 9.16 9.04 25 25 0.6637 1 11.8476438 8.18 6.43 30 20 0.9038 2 10.9287379 10.04 9.84 25 25 0.4603 1 12.1774857 12.33 11.4 22 28 0.3948 1 12.0878797 8.83 10.67 23 27 -0.1726 2 12.3742158 10.88 8.81 26 24 0.4633 1 12.1544098 9.5 8.09 28 22 0.8481 2 11.3171379 10.42 10.12 18 32 0.7114 1 10.8853669 11.82 7.13 28 22 0.5407 1 11.8916826 11.69 8.11 23 27 0.4926 1 12.05664

ANOVAb,c

Model Sum of Squares df Mean Square F Sig.

Regression 19.166 1 19.166 12.096 .002a

Residual 34.858 22 1.584Total 54.024 23a.Predictors: (Constant), AGEb.Dependent Variable: HEDGE d*c. Weighted Least Squares Regression - Weighted by w

Coefficientsa,b

Model Unstandardized Coefficients Standardized Coefficients t Sig.B Std. Error Beta

(Constant) -1.037 .465 -2.230 .036AGE .215 .062 .596 3.478 .002a. Dependent Variable: HEDGE d*b. Weighted Least Squares Regression - Weighted by w

SPSS ANALYSIS OUTPUT

ExampleSee SPSS “sample meta data set.sav” or the

excel version “sample meta data set regression”

The d effect is regressed on Ageb = 0.215, SEb = 0.062, MSe = 1.584Thus, SE’b = 0.062 / (1.584)½

= 0.0493A 95% CI around b gives (0.117, 0.313) for the

regression weight of age on outcome, p<.001

Q-statistic testsQregression = 19.166 with df=1, p < .001

Qresidual = 34.858 with df=22, p = .040

So- are the residuals homogeneous or not? Given a large number of significance tests, one might require the Type I error rate for such tests to be .001 or something small