(C) Stephen Senn 20041 Change from baseline or analysis of covariance?: Lord's Paradox and other matters. Stephen Senn

(C) Stephen Senn 2004 1

Change from baseline or analysis of covariance?: Lord's Paradox and other

matters.

Stephen Senn


Outline

• Adjustment in Randomised Clinical Trials– The argument for ANCOVA

• Lord’s Paradox– ANCOVA versus simple analysis of change scores

(SACS)

• Observational studies– The argument against ANCOVA

• Resolution?– Why ANCOVA although not perfect may be best after

all


SACS and ANCOVA

A simple randomised clinical trial in which there are two treatment groups and only two measurements per patient: a baseline measurement, X and an outcome measurement, Y. Popular choices of outcome measure are1) raw outcomes Y2) change score d = Y - X3) covariance adjusted outcomes Y - X. (where is chosen appropriately)NB As Laird (Am Stat., 37, 329-330, 1983) has shown, covariate adjusted change scores are the same as 3)


The Estimators Associated with the Measures

ˆ

ˆ

ˆ

raw t c

change t c t c

t c t c

Y Y

Y Y X X

Y Y X X

If subscript t stands for treatment and c for control we have:

1) and 2) are just special cases of 3). If is chosen to be the regression of Y on X, then 3) corresponds to analysis of covariance.

1)

2)

3)


Warning

• These three measures, measure the same thing• No question of choosing between them on the basis

of clinical relevance• Can only choose between them on the basis either of

variance, or statistical philosophy• ANCOVA may generally be expected to have the

lowest variance

• Baseline is irrelevant to the definition of the treatment effect.


0.0 0.2 0.4 0.6 0.8 1.0

correlation coefficient

0.0

0.4

0.8

1.2

1.6

2.0

varia

nce

change_scoreoutcomesANCOVA

Variances for Three Approaches to Analysis

0.5


ANCOVA and Baseline by Treatment Interaction

• It is often stated that ANCOVA relies on the parallelism assumption.

• This is not true.• If the effect of treatment varies with baseline it

varies whether or not ANCOVA is used.• ANCOVA is a first approximation and better than

either doing nothing or using change scores.


Not to use ANCOVA, because you fear parallelism may not apply, is like saying crossing the channel in a rowing boat is dangerous I prefer to swim”.


Dichotomania

• Obsessive compulsive disorder– Cochrane Collaboration has a galloping case

• Numbers Needed to Treat should have been strangled at birth

• Division of patients into sheep and goats– Ignoring existence of geep and shoats

• Use of difference from baseline– Sin number one

• Destruction of information

• Arbitrary division into responders non-responders– Sin number two

• Further destruction of information• Unjustified causal interpretation


A Red Herring

• It is sometimes claimed that measurement error invalidates ANCOVA– The reason is that if baseline is measured

with error the regression of outcome on baseline is attenuated

• However this claim is incorrect• ANCOVA is still valid

– The reason is that it is appropriate to correct for an observed imbalance using an observed regression


Counter-Claims

• There is a significant minority of papers arguing against ANCOVA as a means of dealing with bias– E.g. Liang and Zeger (2000), Sankyha,

Samuelson (1986), American Statistician

• The variance claims are accepted

• Claims are made that ANCOVA is biased unless there is balance at baseline


Justification of the Counter-Claim

1)(

)(

ctCt

ctCt

Ct

ct

cctcttt

ccc

ctt

cc

XXYYE

XXYYE

YYE

XXE

Hence

YE

YE

XE

XE


Lord’s Paradox

• Lord, F.M. (1967) “ A paradox in the interpretation of group comparisons”, Psychological Bulletin, 68, 304-305.

“A large university is interested in investigating the effects on the students of the diet provided in the university dining halls….Various types of data are gathered. In particular the weight of each student at the time of his arrival in September and his weight in the following June are recorded”


Two Statisticians

Statistician One• Calculates difference

in weight for each hall• Finds non-significant

difference in each case

• (Also no difference between halls)

Statistician Two• Adjust for initial

weight• Finds significant hall

effect• Concludes

difference between halls


A Simulated Example

• Starting and final weights for two groups of students

• Males and females

• 300 In each group

• Analysis illustrated with S Plus


m m

mmm

m

mmmm

mmmm

m

m

m

m

m

mm

mm

mmm

mm

m

m

mmmmmmm

m

mm

mm

m

mmmmmm

m

mmmmmmm

m

mm

m

m

m

mmm

mmm

mm

mmmm

m

m

mmmm

m

mm

mm

mm

mm

mmmmm

mm

m

mm

m

m

m

m

m

m

mm

mmmmm

m

mmmm

m

m

m

mm

mmm

m

mm

m

mmm

m

mm

m

m

m

m

mmmmm

mmm

m

m

m

m

mmm

m

m

m

mm

mmmm

m

m

m

m

m

m

mmm

mmm

m

m

mmm

m

m

mmmmmmmm

m

m

mmmmmmm

mmmmm

mm

m

mm

m

m

mm

m

mm

m

mm

m

m

mmm

mmmm

m

mm

m

mmm

mmm

mm

m

m

m

m

mmm

m

m

mmm

mmmmmm

m

m

mmmmm

mmm

m

m

m

m

mmm

mmm

m

mmmm

m

m

m

mmm

m

m

m

m

mmm

m

mm

Initial Weight

Fin

al W

eig

ht

40 60 80 100

40

60

80

10

0

ff

f f

ff ff

ff

f

f

ffff

f

f

f

fff

f

f

fffffff

ffffff

fffffff

f

fff

f

fffff

f

ff

fff

ff

ffffff

f

f

ff

fff

f

fffff

f

ffff

fffffff

ff

ff

ff

f

ffff

fff

f

ff

ff

f

ffff

f

f

f

ff

f

ff

f

ff

ff

f

ffff

ff

f

ff

f

ff

f

ff

ffffffff

ff

f

fff

fff

fff

ff

ff

f

ff

f

fff

f

f

ff

f

f

f

f

f

f

fff

f

fff

f

fffff

f

f

fffff

fff

ff

ff

f

ff

ff

ffff

ff

f

ff

f

f

f

ffffffffff

fffffff

f

f

f

fff

ffffff

f

f

ff

ff

f

f

ff

f f

f

f

fffff

f

fff

f

f

f

f

ff

ff

f

ffff

ff

f

f


Statistician One’s Analysis

Paired t-Test

data: Y.males and X.males t = 0.662, df = 299, p-value = 0.5085

Paired t-Test

data: Y.females and X.females t = -0.0512, df = 299, p-value = 0.9592

Standard Two-Sample t-Test

data: diff.males and diff.females t = 0.5017, df = 598, p-value = 0.6161


m m

mmm

m

mmmm

mmmm

m

m

m

m

m

mm

mm

mmm

mm

m

m

mmmmmmm

m

mm

mm

m

mmmmmm

m

mmmmmmm

m

mm

m

m

m

mmm

mmm

mm

mmmm

m

m

mmmm

m

mm

mm

mm

mm

mmmmm

mm

m

mm

m

m

m

m

m

m

mm

mmmmm

m

mmmm

m

m

m

mm

mmm

m

mm

m

mmm

m

mm

m

m

m

m

mmmmm

mmm

m

m

m

m

mmm

m

m

m

mm

mmmm

m

m

m

m

m

m

mmm

mmm

m

m

mmm

m

m

mmmmmmmm

m

m

mmmmmmm

mmmmm

mm

m

mm

m

m

mm

m

mm

m

mm

m

m

mmm

mmmm

m

mm

m

mmm

mmm

mm

m

m

m

m

mmm

m

m

mmm

mmmmmm

m

m

mmmmm

mmm

m

m

m

m

mmm

mmm

m

mmmm

m

m

m

mmm

m

m

m

m

mmm

m

mm

Initial Weight

Fin

al W

eig

ht

40 60 80 100

40

60

80

10

0

ff

f f

ff ff

ff

f

f

ffff

f

f

f

fff

f

f

fffffff

ffffff

fffffff

f

fff

f

fffff

f

ff

fff

ff

ffffff

f

f

ff

fff

f

fffff

f

ffff

fffffff

ff

ff

ff

f

ffff

fff

f

ff

ff

f

ffff

f

f

f

ff

f

ff

f

ff

ff

f

ffff

ff

f

ff

f

ff

f

ff

ffffffff

ff

f

fff

fff

fff

ff

ff

f

ff

f

fff

f

f

ff

f

f

f

f

f

f

fff

f

fff

f

fffff

f

f

fffff

fff

ff

ff

f

ff

ff

ffff

ff

f

ff

f

f

f

ffffffffff

fffffff

f

f

f

fff

ffffff

f

f

ff

ff

f

f

ff

f f

f

f

fffff

f

fff

f

f

f

f

ff

ff

f

ffff

ff

f

f


Statistician Two’s analysis

Call: lm(formula = Y ~ X + sex)

Coefficients: Value Std. Error t value Pr(>|t|) (Intercept) 12.4987 1.5135 8.2584 0.0000 X 0.8087 0.0232 34.9156 0.0000 sex 1.9816 0.2833 6.9945 0.0000


What People Usually Conclude

• Where baseline values are not expected to be equal between groups ANCOVA can mislead

• Therefore even though SACS will have a higher variance it should be preferred under such circumstances since it is obviously unbiased


A Counter Counter-Example

• Suppose we design a bizarre clinical trial

• Only person with diastolic blood pressure at baseline equal to 95mmHg or 105mmHg may enter

• In the first stratum they are randomised 3 to 1 and in the second 1 to 3

• Situation as follows


A Stupid TrialNumbers of Patients by dbp and Treatment

Treatment

A B Total

Baseline

diastolic

blood pressure

95mm Hg 300 100 400

105 mmHg 100 300 400

Total 400 400 800


Approach to Analysis

• Stratify by baseline dbp

• Produce treatment estimate for each stratum

• Overall estimate is average of the two

• Stratification deals with the imbalance


An Equivalent Approach

• Create dummy variable stratumS = -1 if baseline dbp, X = 95mmHgS = 1 if baseline dbp, X =105 mmHg

• Regress dbp at outcome, Y, on treatment indicator T and on stratum indicator S

• Estimate will be same as by stratification• If you want variance estimate to be exactly

the same you need to include interaction also


An Equivalent Equivalent Approach

• Regress Y on T and X rather than on T and S– This is called ANCOVA!

• Note that S= (X-100)/5• Hence this approach is equivalent to the

previous one, which is equivalent to stratification, which is unbiased

• On the other hand SACS is biased• Hence we have produced a counter-example


Conclusion

• Contrary to what is often claimed there are cases where ANCOVA is unbiased but SACS is biased.

• No simple statement of the form “ANCOVA is more efficient but SACS is unbiased” is possible.

• In fact it is very difficult to imagine cases where SACS is the preferred analysis


Lord’s Paradox Revisited

• Statistician one assumes that in the absence of any differential treatment effect the two groups despite different baselines would show equivalent changes

• Statistician two assumes that in the absence of any differential treatment effect the change of the groups as a whole is the same as the change within groups

• Both of these causal assumptions are untestable


However

• It is easy to design trials for which– ANCOVA is unbiased– SACS is biased– A causal interpretation

can be given

• It is rather difficult to design trials for which– SACS is unbiased– ANCOVA is biased– A causal interpretation

can be given


The Necessary Condition for ANCOVA to be Unbiased

t C t c

t C t c

t C

E Y Y X X

E Y Y X X

E Y Y

Or in everyday language that the bias in the raw comparison at outcome should be times the bias at baseline where is the individual regression effect


Cut-off Designs

Trochim and Capelleri have suggested that in many clinical trials randomisation will be unethical because some patients by the nature of their illness may be unwilling to assume the risks associated with an experimental treatment. They propose a class of designs called “cut-off” designs in which some patients are assigned to treatment in a deterministic manner on the basis of baseline values.

The position, for example, might be as given in the diagram below.


Randomise to

standard or

experimental

Experimental

treatment only

Standard

treatment only

Severe hypertensionModerate hypertensionMild hypertension

A Cut-off Design in Hypertension


Cut off Designs

• Provided that the relationship between baseline and outcome is linear ANCOVA is valid

• Cut off designs are thus a wide class of design for which ANCOVA is unbiased

• SACS will be biased

• Thus we have more counterexamples to the claims of Liang and Zeger


A Challenge

• Can you design a trial for which – SACS is unbiased – ANCOVA is biased – A causal interpretation can be given?


Some Schemes That won’t Work

• Select patient according to true baseline values– Not possible in practice since not known– Still won’t work since correlation of true values is not 1

• Select patients according to average of values at baseline and outcome– You need a crystal ball

• Select according to some other value– ANCOVA will be biased but so will SACS

• Select on binary covariate– Either this is permanent (e.g. sex), in which case causal

inference doubtful– Or it varies over time in which case there will be a regression


Conclusion

• In RCTs ANCOVA is the appropriate way to use baseline information– SACS, responder analysis, NNTs all wasteful

• A hallmark of second rate analysis

• In observational studies things are more complex– ANCOVA may not be perfect but it may be the

best you can do


Here there be tygers!

Documents

(C) Stephen Senn 20041 Change from baseline or analysis of covariance?: Lord's Paradox and other matters. Stephen Senn