Strategies for handling missing data in randomised trials

Strategies for handling missing data in randomised trials

Peninsula Medical School, Exeter, 14th March 2011

Ian White, MRC Biostatistics Unit, Cambridge, UK

2

Plan

1. Why do missing data matter? 2. Intention-to-treat analysis strategy for randomised trials

with missing outcomes3. Analysis methods

– regression and mixed models4. Sensitivity analysis

– methods & examples5. Missing baseline covariates

– simple methods are best

• Please ask questions!• Work with: James Carpenter & Stuart Pocock (LSHTM),

Nick Horton (USA), Simon Thompson (BSU)

3

Why do missing data matter?

1. Loss of power (cf. power with no missing data)– can’t regain lost power

2. Any analysis must make an untestable assumption about the missing data– wrong assumption biased estimates

3. Some popular analyses with missing data get biased standard errors – resulting in wrong p-values and confidence intervals

4. Some popular analyses with missing data are inefficient– confidence intervals wider than they need be

4

What to do: loss of power

Can’t solve by analysis (but can exacerbate it!).Approach by design:• Minimise amount of missing data

– Good communications with participants– Aim to follow everyone up– Make repeated attempts using different methods

• Also good to reduce the impact of missing data– Collect reasons for missing data– Collect information predictive of missing values

• Now assume that despite doing our best we still have some missing data…

5

What to do: analysis

A suitable method of analysis would:• Make the correct assumption about the missing data• Give an unbiased estimate (under that assumption)• Give an unbiased standard error (so that P-values and

confidence intervals are correct)• Be efficient (make best use of the available data)

we can never be sure what is the correct assumption sensitivity analyses are essential

6

Exposure Confounders

OutcomeLoss of power

Bias

Missing data problems: observational studies

Fail to control confounding

Potentially inefficient analysis

7

Exposure Confounders

OutcomeLoss of power

Bias

Missing data problems: randomised trials

Fail to control confounding

Potentially inefficient analysis

Baseline covariates

Randomised group

8

Plan

1. Why do missing data matter? 2. Intention-to-treat analysis strategy for

randomised trials with missing outcomes3. Analysis methods




9

• The ITT principle says that we should:– include everyone randomised – in the group to which they were assigned (whether or

not they completed the intervention given to the group)

• With missing data, this is easier said than done• I’m going to show it can also conflict with the statistical

principle that the analysis should be valid under plausible assumptions

Intention-to-treat (ITT) principle

10

Note

• I’m addressing the ITT hypothesis– “There is no difference in clinical outcome between

these two randomised groups”• & estimating the ITT difference

– difference in clinical outcome between randomised groups

11

What does intention-to-treat mean when some outcome data are missing? (1)

• A strict view is that ITT requires complete outcome data– but researchers with incomplete outcome data also

need a principle to guide them• Should all randomised individuals still be included?

– “Although those participants [who drop out] cannot be included in the analysis, it is customary still to refer to analysis of all available participants as an intention-to-treat analysis” (Altman et al, 2001)

– “Complete case analysis, which was the approach used in most trials, violates the principle of intention to treat” (Hollis & Campbell, 1999)

12

What does intention-to-treat mean when some outcome data are missing? (2)

• In new advice, the European Medicines Agency (2010) takes a more relaxed view:– “Full set analysis generally requires the imputation of

values or modelling for the unrecorded data” • The CONSORT (2010) statement says:

– “We replaced mention of ‘intention to treat’ analysis, a widely misused term, by a more explicit request for information about retaining participants in their original assigned groups”

13

Difficulties

• Including all randomised individuals in the analysis isn’t enough to make an analysis valid

• The desire to include all randomised individuals in the analysis– reduces emphasis on the appropriate assumptions– leads to uncritical use of simple imputation methods,

esp. Last Observation Carried Forward (LOCF)– leads to unnecessary use of complex methods,

esp. multiple imputation

• I’m going to say a bit about LOCF, then explain how we have re-defined ITT as an analysis strategy

14

Last observation carried forward (LOCF)0

20

40

60

80

Y

0 3 6 9 12Months

15

LOCF: why it is used

• LOCF is widely advocated• e.g. the European drugs regulator wrote

(CPMP 2001, but since revised):– “The statistical analysis of a clinical trial generally

requires the imputation of values to those data that have not been recorded …”

– “Last observation carried forward … is likely to be acceptable if measurements are expected to be relatively constant over time.”

16

LOCF: what it assumes

• Assumes last observation is representative of the missing value

• Can’t verify this assumption from the data– e.g. even if there was no observed trend over time,

drop-outs could have deteriorated• Analysts rarely give a good justification, and instead

justify LOCF (wrongly) on the grounds that– it is conservative: not true in general– it respects ITT by analysing all individuals

• LOCF widely critiqued e.g. Mallinckrodt et al (2004)• Should only be used if its assumption is plausible!• So how can we unlink ITT from methods like LOCF?

17

Strategy for intention to treat analysis with incomplete observations

(White et al, BMJ, 2011)

1. Attempt to follow up all randomised participants, even if they withdraw from allocated treatment

2. Perform a main analysis of all observed data that is valid under a plausible assumption about the missing data

3. Perform sensitivity analyses to explore the effect of departures from the assumption made in the main analysis

4. Account for all randomised participants, at least in the sensitivity analyses

1818

Example 1: QUATRO trial

• European multicentre RCT to evaluate the effectiveness of adherence therapy in improving quality of life for people with schizophrenia (Gray et al, 2006)

• Primary outcome: quality of life measured by the SF-36 MCS scale, measured at baseline and 52-week follow up

Intervention Control

Total n 204 205

Missing outcome 14% 6%

Mean of observed outcomes 40.2 41.3

SD of observed outcomes 12.0 11.5

19

QUATRO trial: analyses

• We did attempt to follow up all randomised individuals• Main assumption: no difference between missing and

observed values, once adjusted for baseline variables (MAR)

• Main analysis: analysis of covariance on complete cases– intervention effect = -0.33 (s.e. 1.11)

• Sensitivity analysis: consider possible differences between missing and observed values, allowed to be different in each arm– see later

20

Missing at random (MAR)

observed data

missing data

M

• The probability that data are missing depends on the values of the observed data, but does not depend on the values of the missing data

• p(M | Yobs, Ymis) = p(M | Yobs)

21

Plan






22

How to approach the analysis

• Principled approach to missing data:– start by identifying a plausible assumption– then choose an analysis method that’s valid under

that assumption• Some analysis methods are simple to describe but have

complex assumptions

23

The analysis toolkit

Simple methods• Last observation carried forward (LOCF)• Complete-case analysis• Mean imputation• Missing indicator method• Regression imputation

More complex methods• Multiple imputation• Likelihood-based methods• Inverse probability weighting (IPW)

24

Properties of methods

MethodFor missing covariate

For missing outcome

LOCF Not applicable Valid under LOCF assn

Complete cases InefficientSingle Y: valid under MAR

Repeated Y: inefficient

Mean imputation Fails to control confounding

Valid in RCT (see later)

SE

Missing indicator Not applicable

Regression imputation

Valid under MAR (no Y in imp. model)

SE

Multiple imputationValid under MAR Valid under MAR

Maximum likelihood

IPW Inefficient or complexValid under MAR

Simple patterns only

Valid means valid & efficient

25

A comment on MAR

• A lot of statistical literature seems to regard MAR as the correct starting point for analyses with missing data

• One argument in favour of MAR is that it tends to become more plausible as more variables are included in the model– unlike other assumptions

• I think the correct assumption depends on the clinical context

• However in this section I’m going to assume MAR

26

What method is best for missing outcomes in a RCT, assuming MAR?

With a single outcome (not repeated measures):• Regress outcome (Y) on randomised group (Z), adjusting

for baseline covariates (X)– analysis of covariance, ANCOVA– this is the likelihood-based method

• Which X?– to make MAR valid, adjust for X that predict both

outcome and missingness– to gain power, adjust for X that predict outcome

• Cases with missing Y contribute no information– complete-cases analysis is correct!

27

What method is best for missing outcomes in a RCT, assuming MAR?

With a repeated outcome:• Use a mixed model (likelihood-based)• Include all observed outcome data• Exclude any individuals with no post-baseline

observations• Include X’s as beforeSoftware:• Stata xtmixed, SAS proc mixed, R lme()Some points to note:• Don’t allow a treatment effect at baseline• Allow a different treatment effect at each follow-up time• If possible, allow for correlations between outcomes via

an unstructured variance-covariance matrix

28

What about multiple imputation?

Idea of multiple imputation:• Impute missing (single or longitudinal) outcomes from

observed outcomes and baselines repeat m times• Analyse the m completed data sets• Combine estimates by Rubin’s rules

Tutorial: White et al (2011)

29

Multiple imputation and ITT

• MI includes all randomised individuals in the analysis– appealing

• Example with single outcome (Riper et al, 2008):– “We then performed intention-to-treat analysis, using

multiple imputation to deal with loss to follow-up.” • MI is the same as fitting a [mixed] model to the observed

data if imputation model and analysis model are the same– but MI has additional random error (“Monte Carlo

error”) due to using a finite number of imputed data sets

– so why do MI?MI may be of value in a RCT•if additional information (“auxiliary variables”, e.g. compliance) can be included in the imputation model•as a way to do sensitivity analyses

30

Example: UK700 trial

• Intensive vs. Standard case management for severely mentally ill people living in the community

• 708 patients randomised in 4 UK centres• Outcome here: CPRS (psychopathology) score from 2-year

interview (missing in 11% of Intensive, 20% of Standard)

Method Patients in analysis

Treatment effect

Standard error

ANCOVA (adjusted for baseline)

595 -0.3898 1.0348

Joint model for baseline & outcome

705 -0.3898 1.0304

Multiple imputation 708 -0.3834 1.0369

0.01Monte Carlo error 0.004

31

Plan






32

67

89

10

0 1 2

Complete Drop out at time 1

Mea

n o

utc

om

e

time

How to do sensitivity analyses?

• Not LOCF for main analysis, CC for sensitivity analysis

LOCF

0 1 2

CC

0 1 2

MAR-based

33

Principled sensitivity analysis

• Not LOCF for main analysis, CC for sensitivity analysis – the alternative assumptions should contradict the

main assumption• Instead, specify the numerical value of “sensitivity

parameter(s)” governing the degree of departure from the main assumption (Kenward et al, 2001)– e.g. the degree of departure from MAR

• Want methods to exactly match the main analysis when sensitivity parameter is zero

• I’ll first describe methods based around an MAR-based main analysis, then go further

34

Mean score approach (1)

• Analysis model: E[Y|X] = h(AX)

– Y is outcome– h(.) is inverse link function (e.g. identity or invlogit)– X is a covariate vector including 1's, randomised

group Z and baseline covariates – we're interested in the component of A corresponding to Z

• Missing data occur in Y only

• MY is missingness of Y

35

Mean score approach (2)

• Analysis model: E[Y|X] = h(AX)

• Imputation model: E[Y|X,MY] = h(IX + MY)

– is a user-specified departure from MAR: could be 1 if randomised to arm 1, 0 if randomised to arm 0.

• Estimate I by regressing Y on X in complete cases (MY=0)

• If we had complete data we would estimate the analysis model using estimating equations X{Y - h(AX)} = 0

• Replace missing Y with Y* = E[Y|X,MY=1] = h(IX + )

• Solve!• Standard errors by sandwich method (based on EEs for

both I and A) or by conditional variance formula

• Simplifies in the special case of linear regression

36

Example: QUATRO data -4

-20

2E

stim

ate

d in

terv

entio

n e

ffect

-10 -8 -6 -4 -2 0Missing - Observed

Intervention only Both arms Control only

MAR

Outcome SD ≈ 12

37

Why do I allow different missing data mechanisms in different arms?

• Because data may be further from MAR after more intensive interventions

• Because results are most sensitive to this

38

Simple approach

• Assumed departures from MAR: – let = mean unobserved – mean observed outcome

– taking values 0 , 1 in arm 0, 1

• Values estimated from observed data:

– p0, p1 = fraction of missing data in arm 0, 1

– CC = intervention effect in complete cases

– = true intervention effect • Then we can show (White et al, 2007) that

– = CC + 1p1 – 0p0

– se() ≈ se(CC)

• Back-of-an-envelope calculation!

39

Are std errors affected by ? (QUATRO)1.

091.

11.

111.

121.

13

Sta

ndar

d er

ror

of

estim

ate

d in

terv

entio

n e

ffect

-10 -8 -6 -4 -2 0 = Missing – Observed


40

Alternatives to mean score approach

• Can use MI– Define imputation and analysis models as before– Fit imputation model to complete cases as before– Impute with offset

» implemented in ice

– For sensitivity analysis, best to use same seed across different values of

• Can use IPW– involves specifying in missingness model

logit P(MY=1|Y,X) = X + Y

41

Demonstration using rctmiss in Stata

Missing data are 5 units (0.5 SD) worse than observed:

. rctmiss, pmmdelta(-5): reg sf_mcs alloc sf_mcsba…in both arms

MNAR regression results------------------------------------------------------------------------------ sf_mcs | Coef. Std. Err. z P>|z| [95% Conf. Interval]-------------+---------------------------------------------------------------- alloc | -.7338669 1.096779 -0.67 0.503 -2.883514 1.415781 sf_mcsba | .4697194 .0497254 9.45 0.000 .3722594 .5671794 _cons | 22.37387 2.058745 10.87 0.000 18.33881 26.40894------------------------------------------------------------------------------

. rctmiss, pmmdelta(cond(alloc,-5,0)): reg sf_mcs alloc sf_mcsba… in intervention arm only

MNAR regression results------------------------------------------------------------------------------ sf_mcs | Coef. Std. Err. z P>|z| [95% Conf. Interval]-------------+---------------------------------------------------------------- alloc | -1.016182 1.093445 -0.93 0.353 -3.159295 1.126931 sf_mcsba | .4695661 .0496591 9.46 0.000 .372236 .5668962 _cons | 22.66207 2.052232 11.04 0.000 18.63977 26.68437------------------------------------------------------------------------------

rctmiss also produces the graph

-4-2

02

Est

ima

ted

inte

rven

tion

effe

ct

-10 -8 -6 -4 -2 0Missing - Observed


42

Corespondence with MAR when =0

. reg sf_mcs alloc sf_mcsba Standard MAR analysis

Source | SS df MS Number of obs = 349-------------+------------------------------ F( 2, 346) = 49.42 Model | 10497.5347 2 5248.76733 Prob > F = 0.0000 Residual | 36748.2816 346 106.208906 R-squared = 0.2222-------------+------------------------------ Adj R-squared = 0.2177 Total | 47245.8162 348 135.76384 Root MSE = 10.306

------------------------------------------------------------------------------ sf_mcs | Coef. Std. Err. t P>|t| [95% Conf. Interval]-------------+---------------------------------------------------------------- alloc | -.3341637 1.108198 -0.30 0.763 -2.513817 1.84549 sf_mcsba | .4703734 .0475847 9.88 0.000 .3767818 .5639651 _cons | 22.62969 2.05461 11.01 0.000 18.5886 26.67079------------------------------------------------------------------------------

. rctmiss, pmmdelta(0): reg sf_mcs alloc sf_mcsbaResults allowing for MNARUsing 349 complete cases and 37 incomplete cases------------------------------------------------------------------------------ sf_mcs | Coef. Std. Err. z P>|z| [95% Conf. Interval]-------------+---------------------------------------------------------------- alloc | -.3341637 1.108198 -0.30 0.763 -2.506193 1.837865 sf_mcsba | .4703734 .0475847 9.88 0.000 .3771091 .5636377 _cons | 22.62969 2.05461 11.01 0.000 18.60273 26.65665------------------------------------------------------------------------------

43

Main difficulty

• How to specify ?• Need to discuss with investigators

– aim to pre-specify• Alternatively report value of that changes [significance

of] conclusions

44

Example 2: smoking cessation trial

• Smokers calling a telephone helpline were randomised to tailored advice letters + standard counselling or standard counselling – Sutton & Gilbert (2007)

• Outcome: not smoking in months 4-6 (“quit”)

Intervention arm (n=599)

Control arm (n=565)

Quit 73 (12%) 51 (9%)

Didn’t quit 390 (65%) 364 (64%)

Missing 136 (23%) 150 (27%)

45

Smoking cessation trial: analyses

• Main assumption: assume missing = still smoking– Includes everyone, but not convincing

• Main analysis: impute missing values as smokers– OR 1.40 (95 CI 0.96 - 2.05)

• Sensitivity analyses: define the informatively missing odds ratio (IMOR) as the odds ratio between quitting and missing– main analysis assumes IMOR=0– sensitivity analysis varies IMOR (in one or both arms)

46

Smoking trial: sensitivity analysis1

2In

terv

en

tion o

dds

rati

o (

95%

CI)

0 0.1 0.2 0.3 0.4 0.5IMOR in specified arm(s)

intervention arm(23% missing)

both arms control arm(27% missing)

Main analysis

47

Plan






48

Missing baselines

Missing baselines in RCTs are a completely different problem from missing outcomes

• Still assume we want to estimate the effect of randomised intervention on outcome

• Baseline adjustment is used to gain precision, not avoid bias– missing baselines are not a source of bias

• Complete cases analysis is a very bad idea• White & Thompson (2005) showed that almost anything

else is OK– in particular, mean imputation or missing indicator

method– provided randomisation is respected

49

Missing indicator method

• Regress Y on covariates

– MX = 1 if X observed, else 0

– Xfill = X if X observed, else 0 (or some fixed X0)

• Can use weights (good for large X-Y correlation)• But missing indicator method is biased in observational

studies (Greenland and Finkle 1995)– get imperfect adjustment for confounders (residual

confounding)

50

What’s wrong with the missing indicator method?

05

1015

2025

Out

com

e

0 5 10 15 20 25Exposure

0

1

Confounder

51

0055

10101515

20202525

Out

com

eO

utco

me

00 55 1010 1515 2020 2525ExposureExposure

00

11

??

ConfounderConfounder

Make some confounders missing

52

05

1015

2025

Out

com

e

0 5 10 15 20 25Exposure

?

Confounder

Association is different

53

0055

10101515

20202525

Out

com

eO

utco

me

00 55 1010 1515 2020 2525ExposureExposure

00

11

??

ConfounderConfounder

Missing indicator method averages over confounder = 0, 1 and ?

Residual confounding

Residual confounding doesn’t occur in randomised trials:Covariate is independent of “exposure” so not a “confounder”

54

Missing indicator: how to do it (QUATRO)

. gen xmiss = missing(sf_mcsba)

. gen xfill = cond(xmiss, 39, sf_mcsba)

. reg sf_mcs alloc xmiss xfill

Source | SS df MS Number of obs = 367---------+------------------------------ F( 3, 363) = 32.44 Model | 10630.1786 3 3543.39287 Prob > F = 0.0000Residual | 39646.5297 363 109.21909 R-squared = 0.2114---------+------------------------------ Adj R-squared = 0.2049 Total | 50276.7084 366 137.368056 Root MSE = 10.451

-----------------------------------------------------------------------sf_mcs | Coef. Std. Err. t P>|t| [95% Conf. Interval]-------+--------------------------------------------------------------- alloc | -.2364209 1.095984 -0.22 0.829 -2.391695 1.918854 xmiss | -2.682338 2.527628 -1.06 0.289 -7.65297 2.288295 xfill | .4706946 .0482474 9.76 0.000 .3758151 .5655741 _cons | 22.57087 2.07815 10.86 0.000 18.48414 26.65759-----------------------------------------------------------------------

(or use rctmiss) n increased from 349 to 367

55

US report: “The Prevention and Treatment of Missing Data in Clinical Trials”

• Commissioned by Food & Drug Administration• Panel of top statisticians• National Research Council (Dec 2010)

1. Introduction and Background2. Trial Designs to Reduce the Frequency of Missing Data

– focus on estimands (pre-trial)3. Trial Strategies to Reduce the Frequency of Missing Data4. Drawing Inferences from Incomplete Data

– covers it all5. Principles and Methods of Sensitivity Analyses

– lots of suggestions6. Conclusions and Recommendations

56

Conclusions & discussion

• Missing baselines: use simple methods that respect randomisation

• Missing outcomes: focus on assumptions, not methods• An intention-to-treat analysis strategy should include all

individuals in sensitivity analyses– but not necessarily in main analyses

• ANCOVA and mixed models are usually the best strategy for missing outcomes in RCTs– use MI with auxiliary data (e.g. compliance) or

possibly as a way to do sensitivity analyses• Sensitivity analyses can be done in various ways

– install my software rctmiss in Stata using net from http://www.mrc-bsu.cam.ac.uk/IW_Stata/misc

57

References (1)

• Altman D et al (2001). The revised CONSORT statement for reporting randomized trials: Explanation and elaboration. Annals of Internal Medicine 134: 663–694.

• Committee for proprietary medicinal products (2001). Points to consider on missing data. http://www.emea.europa.eu/pdfs/human/ewp/177699EN.pdf

• European Medicines Agency (2010). Guideline on missing data in confirmatory clinical trials. http://www.ema.europa.eu/ema/pages/includes/document/ open_document.jsp?webContentId=WC500096793

• Greenland S, Finkle W (1995). A critical look at methods for handling missing covariates in epidemiologic regression analyses. American Journal of Epidemiology 142: 1255–1264.

• Gray R et al (2006). Adherence therapy for people with schizophrenia: European multicentre randomised controlled trial. British Journal of Psychiatry 189: 508–514.

• Hollis S, Campbell F (1999). What is meant by intention to treat analysis? Survey of published randomised controlled trials. British Medical Journal 319: 670–4.

• Kenward MG, Goetghebeur EJT, Molenberghs G (2001). Sensitivity analysis for incomplete categorical tables. Statistical Modelling 1: 31–48.

• Mallinckrodt CH et al (2004). The effect of correlation structure on treatment contrasts estimated from incomplete clinical trial data with likelihood-based repeated measures compared with last observation carried forward ANOVA. Clinical Trials 1:477–489.

58

References (2)

• Moher D, Schulz KF, Altman DA, for the CONSORT group (2001). The CONSORT statement: Revised recommendations for improving the quality of reports of parallel-group randomised trials. Lancet 357:1191–1194.

• National Research Council (2010). The Prevention and Treatment of Missing Data in Clinical Trials. Panel on Handling Missing Data in Clinical Trials. Committee on National Statistics, Division of Behavioral and Social Sciences and Education. http://www.nap.edu/catalog.php?record_id=12955.

• Sutton S, Gilbert H (2007). Effectiveness of individually tailored smoking cessation advice letters as an adjunct to telephone counselling and generic self-help materials: randomized controlled trial. Addiction 102:994–1000.

• Riper H et al (2008). Web-based self-help for problem drinkers: a pragmatic randomized trial. Addiction 103:218–227.

• White IR, Horton N, Carpenter J, Pocock SJ (2011). An intention-to-treat analysis strategy for randomised trials with missing outcome data. British Medical Journal http://www.bmj.com/content/342/bmj.d40.full.html? ijkey=08IQJ8etzBYYGLw&keytype=ref.

• White IR, Thompson SG (2005). Adjusting for partially missing baseline measurements in randomised trials. Statistics in Medicine 24:993–1007.

• White IR, Wood A, Royston P (2011). Multiple imputation using chained equations: issues and guidance for practice. Statistics in Medicine; 30: 377–399.

59

The end

Documents

Strategies for handling missing data in randomised trials