Handling (and Preventing) Missing Data in RCTs

ASENTMarch 7, 2009

Janet WittesStatistics Collaborative

2

Topics

•Missing values: What, me worry?

•Methods of treatment

•Methods of prevention

•Moral: prevention is better than cure

• Or – “The moral of this tale is ‘care’.”

                              

                                

 

3

Topics turned into learning objectives

•Why are missing data important?•What is the distinction between

Ignorable and non-ignorable missing?

MAR and NMAR?

•What are general approaches to analysis?•How can we prevent (or minimize) missing?

4

Extent of missing primary outcome data

•Cardiovascular outcome trial: 1-2%

•Short-term blood pressure trial: 5-10%

•12 week pain trial: 20-40%

•12 week antipsychotic drug: 30-50%

•1 year Alzheimer’s Disease: 20-50%

Source: informal experience

5

Why people don’t care about missing data

•Many people care about missing data in outcome trials•Many do not care about missing data in symptom trials

Who cares about those who don’t take drug?

•“We know the drug won’t work if you don’t take it”

•“I am not interested in what happens after people stop.”

•No evidence that the two groups differ in Pr{missing}•We are interested in what we observe – “complete cases”•Too hard/expensive to bring back those who stop med

6

So let me convince you to care

•What many assume (explicitly or implicitly)

7

What many assume

8

What many assume

9

What statisticians fear (and assume)

10

What statisticians fear (and assume)

•Those still in study differ from those randomized

•We can’t characterize who is missing

•Who is missing differs by group

11

How papers typically report(Hard to ferret out extent and timing of missing)

ADAS-Cog Over Time

0

10

20

30

40

50

0 1 2 3 4 5 6 7 8 9 10 11 12

Month

ScoreActive

Placebo

“Missing values imputed using LOCF”(hard to tell that 48% have missing 12 month data)

12

The fundamental dilemma

•Want to know: Effect of intervention if everyone took it

•Can learn: Effect of intervention among randomized

13

So what question should we ask

What would have been the effect if people were forced to continue?

What is the effect among people who can tolerate it?

What would the effect have been if it were measured?

14

Language – Little/Rubin

•Ignorable Missing completely at random (MCAR): random number selected the missing

Missing at random (MAR)

•Given observed data, missingness mechanism does not depend on unobserved.

•Rarely definitively determined from the data at hand•Nonignorable: Not missing at random (NMAR)

15

Examples of MAR (Probability of missing depends only on values of observed)

•2 measurements of same variable made at the same time. If they differ by more than a given amount a third is taken.

3rd measurement missing for those who do not differ by the given amount.

•Subject removed from trial if condition is not sufficiently controlled (criteria pre-defined)

16

Examples of NMAR (Probability of missing depends on value of the missing data)

AD patient too demented to come for measurement

Chronic pain: VAS every two weeks

•Measured at Week 4 – pain still bad

•Patient feels better at week 5 so doesn’t come at Week 6

17

Rule of thumb: no benefit for missingness

Method of imputation shouldn’t give us stronger results than what we would have seen from the complete cases

Simple example: More uncertainty but results strengthen.

80 patients: 40/group

20 success in treated and 12 in control

P-value (Fisher’s exact) = 0.11

---------------------------------------

120 patients but 20 in each group missing

Assume missing data share results in observed

P-value now 0.040

18

Handling missing binary outcomes

•Just ignore the missing observations•Impute missing on basis of

Proportion in own group

Best case – all pbo fail; all rx succeed

Worst case – all pbo success; all rx fail

Proportion in placebo group (“not unreasonable guess”)

Proportion in opposite group (“reasonable worst case”)

•Multiple imputation

19

Problems with usual approach

•Too many degrees of freedom•Some methods overstate effect•Some methods understate effect

20

Loss of 4 points in ADAS-Cog

•Two groups – treated and control•120 per group•40% in placebo; 20% in treated•Look at relative risk (<1 is “good”)•Missing % equal in both groups

21

Example: loss of 4 points on ADAS-COG

Effect of Missing Data on Relative Risk

0.00

0.25

0.50

0.75

1.00

1.25

0 5 10 15 20

Percent Missing

WorstOppositePlaceboOwnBest

22

Continuous, longitudinal, time-to-event outcomes

•Just ignore the missing observations•Impute missing on basis of data in: Own group

Combined group

Placebo group

Opposite group (“worst reasonable case”)*

•Last Observation Carried Forward•Baseline Observation Carried Forward•Last rank carried forward#

•Carry forward trajectories•Multiple imputation

*Proschan et al (2001)., J Stat Planning 96: 155# O’Brien, Zhang, Bailey (2005). Stat Med 24:34

23

Effect size P-value

Completers (n=500/650) -0.45 0.06

LOCF -0.51 0.03

BOCF -0.28 0.25

Placebo trajectory -0.49 0.03

Effect size and p-values for pain study

24

Message

•Analyses produce very different results

•Can affect

Direction of effect

Effect size

25

Informed consent documents unclear

•Participation in this study is entirely voluntary. Your treatment and your doctor’s attitude toward you will not be affected should you decide not to participate in this study…

•You will be asked to return for follow-up visits and to provide follow-up information.

•If you agree to participate, you may withdraw from the study at any time without affecting any benefits to which you would otherwise be entitled.

26

Permissive protocols encourage missing data

“Drop-outs will not be replaced”

•Suggests that it would be ok to replace them

•Suggests that analysis will ignore them

“Expect 10% drop out, therefore increase sample size by 10%”

“The primary analysis will use the intent-to-treat pop”

•“The ITT pop is defined as all those randomized who…”

•The ITT pop is defined as the evaluable group

27

Language about withdrawal: an outcome trial

The reason that a subject discontinues from the study will be recorded in the Case Report Form.

A discontinuation occurs when an enrolled subject ceases participation in the study, regardless of the circumstances, prior to completion of the protocol.

…

The final evaluation required by the protocol will be performed at the time of study discontinuation.

28

Outcome: continuous measure at week 48

•Subjects must be withdrawn from the study (i.e., from any further study medication or study procedure) for the following reasons: At their own or their legally authorized representative’s request

If, in the investigator’s opinion, continuation in the study would be detrimental to the subject's well-being

Occurrence of an intolerable treatment-emergent adverse event as determined by the investigator and/or the subject

Failure of the subject to return to the study site for scheduled visits

Persistent noncompliance

Pregnancy

29

Prevention of missing values

•Revise informed consent forms

•Make protocols less permissive

•Define outcome measures that don’t allow “success” for missing

E.g., Define measures as success or failure and missing = failure

30

Improved informed consent document

•Participation in this study is entirely voluntary. Your treatment and your doctor’s attitude toward you will not be affected should you decided not to participate in this study…•If you agree to participate, you may withdraw from the study at any time without affecting any benefits to which you would otherwise be entitled.•You will be asked to return for follow-up visits and to provide follow-up information even if you are not taking study medication.

31

Protocols

•Be vigilant about permissive language

•Distinguish between

Stopping meds

Stopping active visits

Withdrawing consent to be followed passively

•Understand the importance of full follow-up

(even for those who stop study medication)`

32

Typical language about withdrawal in protocols

The reason that a subject discontinues from the study medication will be recorded in the Case Report Form.A discontinuation from the study occurs when an enrolled subject ceases participation a participant in the study dies, is permanently lost to follow-up, or withdraws consent, regardless of the circumstances, prior to completion of the protocol.…An final evaluation required by the protocol will be performed at the time of study discontinuation of study medication.

33

But, if there will be missing data

•Choose analytic methods that

Do not add false precision

Are reasonably conservative

Are interpretable

•Recognize need for big increase in sample size