The State of Mediation Analysis in Psychological Science Issues and Solutions Rick Hoyle Duke University

The State of Mediation Analysis in Psychological ScienceIssues and SolutionsRick HoyleDuke University

York Psychology Department Colloquium, 9 November 2015 2

putativecause

explanatorymechanism

(behavioral)outcome


2006 review of articles in psychology reporting mediation analysis:

• N = 291• distribution across areas of psychology• 34% social• 24% clinical• 10% health• remainder in• developmental• I/O• cognitive• methods• program evaluation



review of social psychology articles published from 2005-2009:

• at least one mediation test• 59% of articles in JPSP• 65% of articles in PSPB


JPSP (latest issue)


JCCP (latest issue)


Psych Science (latest issue)


JEP: General (latest issue)





Origins and Early Developments


counter-attitudina

l statemen

t

cognitivedissonance

attitudechange

social psychologists, in particular, have long theorized about and drawn inferences with reference to mediators without directly observing them or accounting for them in hypothesis tests


counter-attitudina

l statemen

t

cognitivedissonance

attitudechange

self-imagethreat

self-perception

self-inconsistency


“When what a person does [is] attributed to what is going on inside him, investigation is brought to an end.” (Skinner, 1974)


“The time seems to have come when psychology must discard all reference to consciousness; when it need no longer delude itself into thinking that it is making mental states the object of observation.”

Watson (1913), “Psychology as the Behaviorist Views It”vs.

“Who can possibly care about a psychology that is silent on such topics as thinking, motivation, and volition?”

Kimble (1989), “Psychology from the Standpoint of a Generalist”


“First psychology lost its soul, then it lost its mind, then it lost consciousness; it still has behavior, of a kind.”“In order to predict the response, we must know not only the stimulus, but also the organism stimulated”“Behavior we can observe, consciousness we can observe with some difficulty, but the inner dynamics of the mental processes must be inferred rather than observed. . . . A dynamic psychology must utilize the observations of consciousness and behavior as indications of the 'workings of the mind' . . . ”Woodworth (1918), Dynamic Psychology


Tolman’s (1938) intervening variable framework

“Some of us, psychologically, just demand theories. Even if we had all the million and one concrete facts, we would still want theories to, as we would say, “explain” those facts. Theories just seem to be necessary to some of us to relieve our inner tensions.”“A theory, as I shall conceive it, is a set of “intervening variables.” These to-be-inserted variables are “constructs” which we, the theorists, evolve . . . .”


“Mentalistic concepts enter psychology as inferences from behavior. The observations that define them often suggest causes.” (Kimble, 1989)


Hyman (1955)

Step 1: establish simple relationship between two variables

Step 2: introduce additional variables to address problems of spuriousness

Step 3: engage in process of elaboration; within general schema of elaboration interpret the relationship“When the analyst interprets a relationship, he [or she] determines the process through which the assumed cause is related to what we take to be its effect. . . . What are the “links” between the two variables?”


two challenges for informative mediation analyses:

1. estimation and testing2. inference and interpretation


Statistical Tests


X

M

Yc'

a b


X

M

Yc'

a b

X Yc

difference in coefficients methodc - c' > 0


X

M

Yc'

a b

joint significance methoda > 0 and b > 0


X

M

Yc'

a b

product of coefficients methodab > 0


issue with product of coefficients method:

• problem: which standard error to use to construct the test statistic; which reference distribution for tests of significance

• test-statistic options:• Sobel approximation; normal distribution• empirical distribution of ab; tabled critical

values based on empirical distributions for different values of a and b

• distribution of product of two normal variables (zazb); critical values (or confidence interval) based on distribution of product of random variables



• problem: which standard error to use to construct the test statistic; which reference distribution to test for significance

• resampling/simulation options:• percentile bootstrap confidence interval• bias corrected bootstrap• Monte Carlo simulation



• problem: which standard error to use to construct the test statistic; which reference distribution to test for significance

• solutions:• test statistic: best option for power and

Type 1 error is empirical distribution of ab• resampling: percentile bootstrap if Type 1

error is primary consideration; bias corrected bootstrap if power is primary consideration


Inference and Interpretation


“If the mediational model is wrong (i.e., misspecified), the results from a mediational analysis are not so much meaningless, but rather they are misleading” (Kenny, 2008)Inferences about mediation are causal inferences.

1. The cause (X) precedes the mediator (M) in time, and M precedes the effect (Y) in time (i.e., temporal precedence).

2. In each case, the cause and effect are related to one another (i.e., covariation).

3. There are no rival explanations of the observed relations between the causes and effects (i.e., absence of confounds).


evaluation of approaches to explicit testing of mediation hypotheses:

findings are . . .• misleading• uninformative• informative• definitive

current

desired


X

M

Yc'

a b


necessary for informative/definitive findings:

• temporal precedence • covariation• absence of confounds

strategies and solutions• if experimental method is feasible• randomize to levels of a manipulated X• measure or observe Y at a process-

informed interval following X




strategies and solutions• if randomization to levels of a manipulated X

isn’t feasible or successful• use statistical means to isolate X by

including covariates in the model


X

M

Yc'

a b

Ci

C1

...




strategies and solutions• if X is not manipulated• longitudinal design to address temporal

order• inclusion of covariates to achieve some

measure of isolation of X


the problem of equivalent models in data from nonexperimental studies

X Y

X Y

X Y

X Y

C1.

2.

3.

4.


X

M

Yc'

a b




strategies and solutions• if experimental method is feasible• randomize to levels of a manipulated X• measure or observe M at a process-

informed interval following X




strategies and solutions• if manipulation is not possible• longitudinal design to address temporal


measure of isolation of X


X

M

Yc'

a b

Ci

C1

...


X

M

Yc'

a b




strategies and solutions• manipulation of M is not possible• longitudinal design to address temporal


measure of isolation of M


X

M

Yc'

a b

Ci

C1

...


critical issue for informative/definitive M-Y inference—and X-M inference in nonexperimental data—is temporal orderinability to draw firm directional inferences leaves open the possibility that other equivalent models provide an equally compelling account of the dataequivalent models: two or more models that, because they offer statistically equivalent accounts of the data, cannot be distinguished on statistical grounds



X Y

M

X Y

M

X Y

M

X Y

M

X Y

M

X Y

M

X Y

M

X Y

M

X Y

M

X Y

M

X Y

M

X Y

M

1.

2. 3.

4.

5. 6.

7.

8. 9.

10.

11.

12.

.23 .36

.26

.11 .42

.30

.10 .39

.34

.23 .42

.26

.23 .36

.26

.23 .36

.26

.12 .39

.30

.10 .39

.34

.10 .39

.34

.12 .30

.39

.11 .42

.30

.11 .42

.30


“. . . that fact that a study’s results are consistent with an assumed mediation model does not allow for any valid conclusion about the correctness of the model, unless the results stem from a study that allows for ruling out rival models.” (Stone-Romero & Raposa, 2008)


X1

a

b

M2

Y3


X1

a

b

M1 M2

Y2 Y3

b


X1

a

b

M1

Y1

X2

M2

Y2

X3

M3

Y3

b

a


X1

a

b

M1

Y1

X2

M2

Y2

X3

M3

Y3

a

b


X1

a

b

M1

Y1

X2

M2

Y2

X3

M3

Y3

X-M M-Y

a

b

Covariates1 Covariates2


Xa

b

M2

Y2

M3

Y3

X-M M-Y

Covariates1 Covariates2


issues with inferences about the extent of mediation:

• problem: under certain conditions the simple direct effect (c) might not be significant when the indirect effect (ab) is significant

• solution: focus testing and inference on the indirect effect in the full model; avoid extent of mediation inference



• problem: c' may be nonsignificant but mediation not full

• solution: under certain conditions, proportion of full effect that is attributable to the indirect effect can be informative• standardized c (rxy in simple mediation

model) should be at least |.20|• proportion of effect that is through the

mediator

or

• this value is descriptive (i.e., not subject to formal inference)

cab/ cc1 /'



• problem: when there is measurement error in the mediator, c' might be significant despite full mediation

• solutions: (1) use highly reliable measure of the mediator; (2) obtain multiple measures of the mediator and model it as a latent variable

brb MMobs '

')( ' cabr1c MM


proposals for tests of mediation going forward:

• they should not be done unless the results will be informative (i.e., criterion is not statistical significance)

• they should not be required for circumspect, theory-driven inferences about causal mechanisms

• when they are done, they should be done better!• randomization and manipulation when

possible• careful and thorough accounting for

confounds• consideration for time passage between

assessment of variables in the causal sequence

• use of longitudinal designs appropriate for observing the focal causal sequence


Pek, J., & Hoyle, R. H. (in press). On the (in)validity of tests of simple mediation: Threats and solutions. Social and Personality Psychology Compass.

Thank you!

[email protected]

Documents

The State of Mediation Analysis in Psychological Science Issues and Solutions Rick Hoyle Duke University