Measuring Change in Two-Wave Studies

David A. Kenny

December 15, 2013

2

The Focus Here in Measuring Change

• To understand the causes of change, e.g., intervention analysis.

• Not:– To understand the variable of

interest and how it changes.

– To measure who changes more or less in the sample.

3

A Bit of Notation• Two times

– Time 1 measure: Y1

– Time 2 measure: Y2

• X a potential cause of change in Y– easier to think of as a dichotomy (0 and

1) but not required

• X may be randomized or not

4

Change Score Analysis (CSA)

• Time 2 minus Time 1• Seen by some as a naïve

model.• Can compute the difference

score more indirectly (keep Time 1 in the model).

5

Control for Baseline (CfB)

• Adjust out for the effect of Time 1, i.e. empirically weight Time 1 and not fix its coefficient to one.

• Generally preferred method.

6

Equations

• Change Score Analysis (CSA)

Y2 – Y1 = a + bX + e

• Control for Baseline (CfB)

Y2 = a + Y1 + bX + e

(will refer to later)

7

The Coefficient for Y1

Note that we can rewrite CSA as

Y2 = a + 1Y1 + bX + e

Thus, CSA fixes the coefficient for Y1 to one, whereas CfB empirically estimates it.

For CfB usually, though not always, is less than 1. In fact, if the variances of Y1 and Y2 are equal, cannot be bigger than 1 (given that b = 0).

8

Lord’s Paradox• CSA and CfB yield different

results whenis different from 1 andthere is correlation between X and Y1.

Why Different?

9

How Different?

10

CSA: The Gap Persists

11

CfB: The Gap Narrows (< 1)

12

CSA vs. CfB

13

CfBCfB

14

Randomized Studies

• Lord’s paradox happens only when there is some association between X and Y1, i.e. a gap.

• If there is randomization, there is no gap in the population.

• By chance, there is some gap in the sample.

15

Estimate of the Treatment Effect (b)

• MD1: The mean difference between treatment groups at time 1.

• MD2: The mean difference between treatment groups at time 2.

• CSA: b = MD2 – MD1

• CfB: b = MD2 - MD1 • If , then CfB adjusts less than CSA.

16

and so if < 1…• If MD1 is positive (1s start ahead

of 0s), more gets subtracted using CSA, resulting in a smaller b than CfB.

• If MD1 is negative (1s start ahead of 0s), more gets added using CSA, resulting in a larger b than CfB.

17

Strategy

• In randomized studies and in non-randomized studies in which the gap is small, controlling for the Y1 should be done as it almost always has more power than change score analysis.

18

What Is Meant by “Small”

• Suggestion • If the effect of X on Y1 is not

statistically significant and the r between X with Y1 is less than .075 in absolute value or d of less than 0.15 in absolute value, then the gap is small.

19

But what if there is a gap at Time 1?

• The bigger the gap, the greater the divergence between the two methods.

• To choose the right analysis, we need to know why there is that gap: What causes that gap and how does it change?

20

Report the Size of the Gap!

• In non-randomized studies, we need to know the size of the gap.

• If X is a dichotomy, report d.• If X is continuous, report the

correlation of X with Y1.

21

The Assignment Variable• See Judd & Kenny book (1981)• Z is an unmeasured variable that explains

the gap at time 1.• It does so because it determines X

(selection), but it also associated with the Y1.

• If Z were measured and controlled for, there would be no gap.

22

Model for CSA

• Think of Z as a dichotomy, with levels 1 and 2.

• We measure the mean of Y1 and Y2 for different values of Z.

• We compute the gap at each time (controlling for X).

• The gap is the same at both times.

23

Continuous Z

• Z need not be a dichotomy, but the general points still hold.

• The effect of Z on Y1 or g is the same as the effect of Z on Y2, controlling for X.

24

25

Example• Z is gender.• Proportionally more men in the

treated group than the control group.

• Gender differences in Y are same at the two times (controlling for X).

26

Model for CfB• Z causes or is correlated with

causes of Y1.• However, Z has no effect on Y2

once Y1 is controlled.• Y1 serves as a complete mediator

of the Z-Y2 relationship.

27

0

28

Example• Schools try to select low performing

students into a the treatment group.• Z is child’s ability at time 1.• As Y1 is a measure of ability, it

explains any relationship between Z and Y2 (path from Z to Y2 is zero).

29

Other Presentations

• Change Score Analysis• Standardized Change Score

Analysis• Controlling for the Baseline

Analysis• Power in the Study of Change