Conducting a Meta-Analysis Using CMA: An Introduction...Mar 25, 2017  · Conducting a Meta-Analysis...

Conducting a Meta-Analysis Using CMA:

An Introduction

Alison C. Koenka, Ph.D.

March 25, 2017

2

primary studies

3

introducing…meta-analysis!

Dent & Koenka, 2016

4

• Foundation

overview

5

• Foundation

- Formal definition

- Steps in conducting a meta-analysis

- Common questions confronted

overview

6

• Foundation

• Introducing Comprehensive Meta-Analysis

(CMA)

• Inputting data into CMA

overview

7

• Foundation

• Introducing Comprehensive Meta-Analysis

(CMA)

• Inputting data into CMA

• Computing the overall effect size

• Moderator analyses

overview

8

• Foundation

• Introducing Comprehensive Meta-Analysis

(CMA)

• Inputting data into CMA

• Computing the overall effect size

• Moderator analyses

• Advanced topics and additional resources

overview

9

• Foundation

overview

10

• Research synthesis

formal definition

11

formal definition

Literature Review

Research Synthesis Theoretical Review

Meta-

Analysis

Systematic

Review

12

formal definition

Literature Review

Research Synthesis Theoretical Review

Meta-

Analysis

Systematic

Review

13

• Research synthesis

• Statistical integration of study outcomes

formal definition

14

• Research synthesis

• Statistical integration of study outcomes

• Effect sizes:

- Cohen’s d index

- Correlation coefficient (r)

- Odds ratio

formal definition

15

• Formulating the problem

• Searching the literature

• Retrieving information from studies

• Integrating study outcomes

steps in conducting a meta-analysis

16

• Formulating the problem

• Searching the literature

• Retrieving information from studies

• Integrating study outcomes

steps in conducting a meta-analysis

17

• Integrating study outcomes

• Two main components

1. Computing overall effect size

2. Conducting moderator analyses

steps in conducting a meta-analysis

18

Which effect size metric should I use?

common questions confronted

19

• d-index

- Intervention

- Experimental/quasi-experimental

common questions confronted

20

• d-index

• Correlation coefficient (r)

- Correlational data

common questions confronted

21

• d-index

• Correlation coefficient (r)

• Odds ratio

- Dichotomous outcomes

common questions confronted

22

How do I address the issue of publication bias?

common questions confronted

23

• Trim-and-fill procedure

- One of several options

common questions confronted

Duval & Tweedie (2000a; 2000b)

24

• Trim-and-fill procedure

common questions confronted

Duval & Tweedie (2000a; 2000b)

25

How do I identify independent comparisons

(and why is this important)?

common questions confronted

26

• Shifting unit of analysis approach

- One of many options

common questions confronted

Cooper (2016)

27

• Shifting unit of analysis approach

- One of many options

- Implications for importing data into CMA

common questions confronted

Cooper (2016)

28

Which model of error should I use?

common questions confronted

29

• Two options:

1. Fixed-effect model of error

common questions confronted

Bornstein et al. (2009)

30

• Two options:

1. Fixed-effect model of error:

All studies share a common effect size

common questions confronted

Bornstein et al. (2009)

31

• Two options:

1. Fixed-effect model of error:

All studies share a common effect size

Differences solely due to sampling error

common questions confronted

Bornstein et al. (2009)

32

• Two options:

1. Fixed-effect model of error

2. Random-effects model of error

common questions confronted

Bornstein et al. (2009)

33

• Two options:

1. Fixed-effect model of error

2. Random-effects model of error:

Studies do not share a common effect size

common questions confronted

Bornstein et al. (2009)

34

• Two options:

1. Fixed-effect model of error

2. Random-effects model of error:

Studies do not share a common effect size

Differences due to sampling error and true differences

common questions confronted

Bornstein et al. (2009)

35

• Foundation

• Introducing Comprehensive Meta-Analysis

(CMA)

• Inputting data into CMA

overview

36

introducing CMA

37

introducing CMA

38

• Pros:

- Extremely accessible

- Easy to interpret output

introducing CMA

39

• Pros:

- Extremely accessible

- Easy to interpret output

• Cons:

- Perhaps less flexible

- Very expensive

introducing CMA

40

• Three different versions

introducing CMA

www.meta-analysis.com

41

• Step 1: identify column names

- Identify column for

study name

effect size data

inputting data into CMA

42

• Step 1: identify column names

- Effect size data:

Click “next”

Choose default (“comparison of two groups”)

Choose “continuous (means)” for d-index,

Choose “dichotomous” for odds ratio,

OR

Choose “correlation” for r

inputting data into CMA

43

• Step 1: identify column names

- Effect size data (continued):

Choose “continuous (means)” for d-index,

Choose “Cohen’s d option”

Choose effect size data columns

Group A: treatment group

Group B: control group

inputting data into CMA

44

inputting data into CMA

45

• Major caveat of standard version:

- No “subgroup within study” column option

inputting data into CMA

46

• Open CMA and practice these steps

inputting data into CMA

47

• Five pieces of information required:

- Study name

- Effect size counter

- Effect size

- n for ‘treatment’ and ‘comparison’ group

preparing data for CMA

48

• Preparing ‘study name’ and ‘tricking’ CMA

- Necessary for proper treatment of

independent (and non-independent) effect sizes

preparing data for CMA

49

• Preparing ‘study name’ and ‘tricking’ CMA

preparing data for CMA

50

• Preparing ‘study name’ and ‘tricking’ CMA

preparing data for CMA

51

• Preparing ‘study name’ and ‘tricking’ CMA

preparing data for CMA

52

• Preparing ‘study name’ and ‘tricking’ CMA

preparing data for CMA

53

• Foundation

• Introducing Comprehensive Meta-Analysis

(CMA)

• Inputting data into CMA

• Computing the overall effect size

• Moderator analyses

overview

54

• Select “analyses” -- > run analyses

computing the overall effect size

55

• Select “analyses” -- > run analyses

- Select model(s) of error

computing the overall effect size

56

• Select “analyses” -- > run analyses

- Select model(s) of error

- Select *study* and not subgroup as unit of analysis

Select “computational options”

Select “select by…”

Click “subgroups” tab

Use study as the unit of analysis

Click “apply”

computing the overall effect size

57

• Select “analyses” -- > run analyses

- Select model(s) of error

- Select *study* and not subgroup as unit of analysis

computing the overall effect size

This output

incorrectly

uses the

subgroup as

the unit of

analysis,

treating effect

sizes as

independent

58

• Select “analyses” -- > run analyses

- Select model(s) of error

- Select *study* and not subgroup as unit of analysis

computing the overall effect size

This output

correctly

uses the study

as the unit of

analysis,

treating effect

sizes from the

same sample

as dependent

59

• Select “next table”

• Practice these steps

computing the overall effect size

60

• Select “next table”

• This is where you interpret your output

computing the overall effect size

Overall,

average

d-index

61

• Select “next table”

• This is where you interpret your output

computing the overall effect size

Indicates

whether there is

significant

variation

surrounding

average ES

62

• Identify column for….moderator variable

- Name variable (e.g., “grade level”)

- Select “categorical” for data type

conducting a moderator analysis

63

• Identify column for….moderator variable

• Paste data from Excel

conducting a moderator analysis

64

• Identify column for….moderator variable

• Paste data from Excel

• Same steps to run analyses, and

conducting a moderator analysis

65

• Identify column for….moderator variable

• Paste data from Excel

• Same steps to run analyses, and

• Computational options group by…

conducting a moderator analysis

66

• Identify column for….moderator variable

• Paste data from Excel

• Same steps to run analyses, and

• Computational options group by…

conducting a moderator analysis

67

• Follow these steps to conduct “grade level”

moderator analysis yourself

• What would we conclude?

conducting a moderator analysis

68

• Interpreting the output

conducting a moderator analysis

69

• Interpreting the output

conducting a moderator analysis

i.e., “random effects”

70

• Interpreting the output

conducting a moderator analysis

Very different

conclusions

depending on

model of error

71

• Interpreting the output

• Because of small k, will substantively interpret fixed

conducting a moderator analysis

Bornstein et al. (2009)

72

• A note on shifting unit of analysis approach

- Often need to recode study names

conducting a moderator analysis

73

• A note on shifting unit of analysis approach

conducting a moderator analysis

74

• Foundation

• Introducing Comprehensive Meta-Analysis

(CMA)

• Inputting data into CMA

• Computing the overall effect size

• Moderator analyses

• Advanced topics and additional resources

overview

75

• Option under “analysis” (but not in all versions)

publication bias

76

• Interpreting “trim and fill” output

publication bias

Duval & Tweedie (2000a; 2000b)

77

• Interpreting “trim and fill” output

publication bias

78

• Interpreting “trim and fill” output

publication bias

79

• Meta-regression

other advanced topics

Bornstein et al. (2009)

80

• Meta-regression

• Small sample size correction

other advanced topics

Bornstein et al. (2009)

81

• Meta-regression

• Small sample size correction

• Data from cluster-randomized studies

other advanced topics

Hedges (2009)

82

additional resources

Bornstein et al. (2009)

83

• Effect size calculator

additional resources

www.campbellcollaboration.org

84

• Online courses using CMA offered

www.statistics.com

additional resources

85

• CMA practice

www.meta-analysis.com

additional resources

86

• CMA troubleshooting

www.meta-analysis.com

me!

additional resources

87

koenka.1@osu.edu

thank you!