Upload
preston-ward
View
212
Download
0
Embed Size (px)
Citation preview
Strategies for handling missing data in randomised trials
Peninsula Medical School, Exeter, 14th March 2011
Ian White, MRC Biostatistics Unit, Cambridge, UK
2
Plan
1. Why do missing data matter? 2. Intention-to-treat analysis strategy for randomised trials
with missing outcomes3. Analysis methods
– regression and mixed models4. Sensitivity analysis
– methods & examples5. Missing baseline covariates
– simple methods are best
• Please ask questions!• Work with: James Carpenter & Stuart Pocock (LSHTM),
Nick Horton (USA), Simon Thompson (BSU)
3
Why do missing data matter?
1. Loss of power (cf. power with no missing data)– can’t regain lost power
2. Any analysis must make an untestable assumption about the missing data– wrong assumption biased estimates
3. Some popular analyses with missing data get biased standard errors – resulting in wrong p-values and confidence intervals
4. Some popular analyses with missing data are inefficient– confidence intervals wider than they need be
4
What to do: loss of power
Can’t solve by analysis (but can exacerbate it!).Approach by design:• Minimise amount of missing data
– Good communications with participants– Aim to follow everyone up– Make repeated attempts using different methods
• Also good to reduce the impact of missing data– Collect reasons for missing data– Collect information predictive of missing values
• Now assume that despite doing our best we still have some missing data…
5
What to do: analysis
A suitable method of analysis would:• Make the correct assumption about the missing data• Give an unbiased estimate (under that assumption)• Give an unbiased standard error (so that P-values and
confidence intervals are correct)• Be efficient (make best use of the available data)
we can never be sure what is the correct assumption sensitivity analyses are essential
6
Exposure Confounders
OutcomeLoss of power
Bias
Missing data problems: observational studies
Fail to control confounding
Potentially inefficient analysis
7
Exposure Confounders
OutcomeLoss of power
Bias
Missing data problems: randomised trials
Fail to control confounding
Potentially inefficient analysis
Baseline covariates
Randomised group
8
Plan
1. Why do missing data matter? 2. Intention-to-treat analysis strategy for
randomised trials with missing outcomes3. Analysis methods
– regression and mixed models4. Sensitivity analysis
– methods & examples5. Missing baseline covariates
– simple methods are best
9
• The ITT principle says that we should:– include everyone randomised – in the group to which they were assigned (whether or
not they completed the intervention given to the group)
• With missing data, this is easier said than done• I’m going to show it can also conflict with the statistical
principle that the analysis should be valid under plausible assumptions
Intention-to-treat (ITT) principle
10
Note
• I’m addressing the ITT hypothesis– “There is no difference in clinical outcome between
these two randomised groups”• & estimating the ITT difference
– difference in clinical outcome between randomised groups
11
What does intention-to-treat mean when some outcome data are missing? (1)
• A strict view is that ITT requires complete outcome data– but researchers with incomplete outcome data also
need a principle to guide them• Should all randomised individuals still be included?
– “Although those participants [who drop out] cannot be included in the analysis, it is customary still to refer to analysis of all available participants as an intention-to-treat analysis” (Altman et al, 2001)
– “Complete case analysis, which was the approach used in most trials, violates the principle of intention to treat” (Hollis & Campbell, 1999)
12
What does intention-to-treat mean when some outcome data are missing? (2)
• In new advice, the European Medicines Agency (2010) takes a more relaxed view:– “Full set analysis generally requires the imputation of
values or modelling for the unrecorded data” • The CONSORT (2010) statement says:
– “We replaced mention of ‘intention to treat’ analysis, a widely misused term, by a more explicit request for information about retaining participants in their original assigned groups”
13
Difficulties
• Including all randomised individuals in the analysis isn’t enough to make an analysis valid
• The desire to include all randomised individuals in the analysis– reduces emphasis on the appropriate assumptions– leads to uncritical use of simple imputation methods,
esp. Last Observation Carried Forward (LOCF)– leads to unnecessary use of complex methods,
esp. multiple imputation
• I’m going to say a bit about LOCF, then explain how we have re-defined ITT as an analysis strategy
14
Last observation carried forward (LOCF)0
20
40
60
80
Y
0 3 6 9 12Months
15
LOCF: why it is used
• LOCF is widely advocated• e.g. the European drugs regulator wrote
(CPMP 2001, but since revised):– “The statistical analysis of a clinical trial generally
requires the imputation of values to those data that have not been recorded …”
– “Last observation carried forward … is likely to be acceptable if measurements are expected to be relatively constant over time.”
16
LOCF: what it assumes
• Assumes last observation is representative of the missing value
• Can’t verify this assumption from the data– e.g. even if there was no observed trend over time,
drop-outs could have deteriorated• Analysts rarely give a good justification, and instead
justify LOCF (wrongly) on the grounds that– it is conservative: not true in general– it respects ITT by analysing all individuals
• LOCF widely critiqued e.g. Mallinckrodt et al (2004)• Should only be used if its assumption is plausible!• So how can we unlink ITT from methods like LOCF?
17
Strategy for intention to treat analysis with incomplete observations
(White et al, BMJ, 2011)
1. Attempt to follow up all randomised participants, even if they withdraw from allocated treatment
2. Perform a main analysis of all observed data that is valid under a plausible assumption about the missing data
3. Perform sensitivity analyses to explore the effect of departures from the assumption made in the main analysis
4. Account for all randomised participants, at least in the sensitivity analyses
1818
Example 1: QUATRO trial
• European multicentre RCT to evaluate the effectiveness of adherence therapy in improving quality of life for people with schizophrenia (Gray et al, 2006)
• Primary outcome: quality of life measured by the SF-36 MCS scale, measured at baseline and 52-week follow up
Intervention Control
Total n 204 205
Missing outcome 14% 6%
Mean of observed outcomes 40.2 41.3
SD of observed outcomes 12.0 11.5
19
QUATRO trial: analyses
• We did attempt to follow up all randomised individuals• Main assumption: no difference between missing and
observed values, once adjusted for baseline variables (MAR)
• Main analysis: analysis of covariance on complete cases– intervention effect = -0.33 (s.e. 1.11)
• Sensitivity analysis: consider possible differences between missing and observed values, allowed to be different in each arm– see later
20
Missing at random (MAR)
observed data
missing data
M
• The probability that data are missing depends on the values of the observed data, but does not depend on the values of the missing data
• p(M | Yobs, Ymis) = p(M | Yobs)
21
Plan
1. Why do missing data matter? 2. Intention-to-treat analysis strategy for randomised trials
with missing outcomes3. Analysis methods
– regression and mixed models4. Sensitivity analysis
– methods & examples5. Missing baseline covariates
– simple methods are best
22
How to approach the analysis
• Principled approach to missing data:– start by identifying a plausible assumption– then choose an analysis method that’s valid under
that assumption• Some analysis methods are simple to describe but have
complex assumptions
23
The analysis toolkit
Simple methods• Last observation carried forward (LOCF)• Complete-case analysis• Mean imputation• Missing indicator method• Regression imputation
More complex methods• Multiple imputation• Likelihood-based methods• Inverse probability weighting (IPW)
24
Properties of methods
MethodFor missing covariate
For missing outcome
LOCF Not applicable Valid under LOCF assn
Complete cases InefficientSingle Y: valid under MAR
Repeated Y: inefficient
Mean imputation Fails to control confounding
Valid in RCT (see later)
SE
Missing indicator Not applicable
Regression imputation
Valid under MAR (no Y in imp. model)
SE
Multiple imputationValid under MAR Valid under MAR
Maximum likelihood
IPW Inefficient or complexValid under MAR
Simple patterns only
Valid means valid & efficient
25
A comment on MAR
• A lot of statistical literature seems to regard MAR as the correct starting point for analyses with missing data
• One argument in favour of MAR is that it tends to become more plausible as more variables are included in the model– unlike other assumptions
• I think the correct assumption depends on the clinical context
• However in this section I’m going to assume MAR
26
What method is best for missing outcomes in a RCT, assuming MAR?
With a single outcome (not repeated measures):• Regress outcome (Y) on randomised group (Z), adjusting
for baseline covariates (X)– analysis of covariance, ANCOVA– this is the likelihood-based method
• Which X?– to make MAR valid, adjust for X that predict both
outcome and missingness– to gain power, adjust for X that predict outcome
• Cases with missing Y contribute no information– complete-cases analysis is correct!
27
What method is best for missing outcomes in a RCT, assuming MAR?
With a repeated outcome:• Use a mixed model (likelihood-based)• Include all observed outcome data• Exclude any individuals with no post-baseline
observations• Include X’s as beforeSoftware:• Stata xtmixed, SAS proc mixed, R lme()Some points to note:• Don’t allow a treatment effect at baseline• Allow a different treatment effect at each follow-up time• If possible, allow for correlations between outcomes via
an unstructured variance-covariance matrix
28
What about multiple imputation?
Idea of multiple imputation:• Impute missing (single or longitudinal) outcomes from
observed outcomes and baselines repeat m times• Analyse the m completed data sets• Combine estimates by Rubin’s rules
Tutorial: White et al (2011)
29
Multiple imputation and ITT
• MI includes all randomised individuals in the analysis– appealing
• Example with single outcome (Riper et al, 2008):– “We then performed intention-to-treat analysis, using
multiple imputation to deal with loss to follow-up.” • MI is the same as fitting a [mixed] model to the observed
data if imputation model and analysis model are the same– but MI has additional random error (“Monte Carlo
error”) due to using a finite number of imputed data sets
– so why do MI?MI may be of value in a RCT•if additional information (“auxiliary variables”, e.g. compliance) can be included in the imputation model•as a way to do sensitivity analyses
30
Example: UK700 trial
• Intensive vs. Standard case management for severely mentally ill people living in the community
• 708 patients randomised in 4 UK centres• Outcome here: CPRS (psychopathology) score from 2-year
interview (missing in 11% of Intensive, 20% of Standard)
Method Patients in analysis
Treatment effect
Standard error
ANCOVA (adjusted for baseline)
595 -0.3898 1.0348
Joint model for baseline & outcome
705 -0.3898 1.0304
Multiple imputation 708 -0.3834 1.0369
0.01Monte Carlo error 0.004
31
Plan
1. Why do missing data matter? 2. Intention-to-treat analysis strategy for randomised trials
with missing outcomes3. Analysis methods
– regression and mixed models4. Sensitivity analysis
– methods & examples5. Missing baseline covariates
– simple methods are best
32
67
89
10
0 1 2
Complete Drop out at time 1
Mea
n o
utc
om
e
time
How to do sensitivity analyses?
• Not LOCF for main analysis, CC for sensitivity analysis
LOCF
0 1 2
CC
0 1 2
MAR-based
33
Principled sensitivity analysis
• Not LOCF for main analysis, CC for sensitivity analysis – the alternative assumptions should contradict the
main assumption• Instead, specify the numerical value of “sensitivity
parameter(s)” governing the degree of departure from the main assumption (Kenward et al, 2001)– e.g. the degree of departure from MAR
• Want methods to exactly match the main analysis when sensitivity parameter is zero
• I’ll first describe methods based around an MAR-based main analysis, then go further
34
Mean score approach (1)
• Analysis model: E[Y|X] = h(AX)
– Y is outcome– h(.) is inverse link function (e.g. identity or invlogit)– X is a covariate vector including 1's, randomised
group Z and baseline covariates – we're interested in the component of A corresponding to Z
• Missing data occur in Y only
• MY is missingness of Y
35
Mean score approach (2)
• Analysis model: E[Y|X] = h(AX)
• Imputation model: E[Y|X,MY] = h(IX + MY)
– is a user-specified departure from MAR: could be 1 if randomised to arm 1, 0 if randomised to arm 0.
• Estimate I by regressing Y on X in complete cases (MY=0)
• If we had complete data we would estimate the analysis model using estimating equations X{Y - h(AX)} = 0
• Replace missing Y with Y* = E[Y|X,MY=1] = h(IX + )
• Solve!• Standard errors by sandwich method (based on EEs for
both I and A) or by conditional variance formula
• Simplifies in the special case of linear regression
36
Example: QUATRO data -4
-20
2E
stim
ate
d in
terv
entio
n e
ffect
-10 -8 -6 -4 -2 0Missing - Observed
Intervention only Both arms Control only
MAR
Outcome SD ≈ 12
37
Why do I allow different missing data mechanisms in different arms?
• Because data may be further from MAR after more intensive interventions
• Because results are most sensitive to this
38
Simple approach
• Assumed departures from MAR: – let = mean unobserved – mean observed outcome
– taking values 0 , 1 in arm 0, 1
• Values estimated from observed data:
– p0, p1 = fraction of missing data in arm 0, 1
– CC = intervention effect in complete cases
– = true intervention effect • Then we can show (White et al, 2007) that
– = CC + 1p1 – 0p0
– se() ≈ se(CC)
• Back-of-an-envelope calculation!
39
Are std errors affected by ? (QUATRO)1.
091.
11.
111.
121.
13
Sta
ndar
d er
ror
of
estim
ate
d in
terv
entio
n e
ffect
-10 -8 -6 -4 -2 0 = Missing – Observed
Intervention only Both arms Control only
40
Alternatives to mean score approach
• Can use MI– Define imputation and analysis models as before– Fit imputation model to complete cases as before– Impute with offset
» implemented in ice
– For sensitivity analysis, best to use same seed across different values of
• Can use IPW– involves specifying in missingness model
logit P(MY=1|Y,X) = X + Y
41
Demonstration using rctmiss in Stata
Missing data are 5 units (0.5 SD) worse than observed:
. rctmiss, pmmdelta(-5): reg sf_mcs alloc sf_mcsba…in both arms
MNAR regression results------------------------------------------------------------------------------ sf_mcs | Coef. Std. Err. z P>|z| [95% Conf. Interval]-------------+---------------------------------------------------------------- alloc | -.7338669 1.096779 -0.67 0.503 -2.883514 1.415781 sf_mcsba | .4697194 .0497254 9.45 0.000 .3722594 .5671794 _cons | 22.37387 2.058745 10.87 0.000 18.33881 26.40894------------------------------------------------------------------------------
. rctmiss, pmmdelta(cond(alloc,-5,0)): reg sf_mcs alloc sf_mcsba… in intervention arm only
MNAR regression results------------------------------------------------------------------------------ sf_mcs | Coef. Std. Err. z P>|z| [95% Conf. Interval]-------------+---------------------------------------------------------------- alloc | -1.016182 1.093445 -0.93 0.353 -3.159295 1.126931 sf_mcsba | .4695661 .0496591 9.46 0.000 .372236 .5668962 _cons | 22.66207 2.052232 11.04 0.000 18.63977 26.68437------------------------------------------------------------------------------
rctmiss also produces the graph
-4-2
02
Est
ima
ted
inte
rven
tion
effe
ct
-10 -8 -6 -4 -2 0Missing - Observed
Intervention only Both arms Control only
42
Corespondence with MAR when =0
. reg sf_mcs alloc sf_mcsba Standard MAR analysis
Source | SS df MS Number of obs = 349-------------+------------------------------ F( 2, 346) = 49.42 Model | 10497.5347 2 5248.76733 Prob > F = 0.0000 Residual | 36748.2816 346 106.208906 R-squared = 0.2222-------------+------------------------------ Adj R-squared = 0.2177 Total | 47245.8162 348 135.76384 Root MSE = 10.306
------------------------------------------------------------------------------ sf_mcs | Coef. Std. Err. t P>|t| [95% Conf. Interval]-------------+---------------------------------------------------------------- alloc | -.3341637 1.108198 -0.30 0.763 -2.513817 1.84549 sf_mcsba | .4703734 .0475847 9.88 0.000 .3767818 .5639651 _cons | 22.62969 2.05461 11.01 0.000 18.5886 26.67079------------------------------------------------------------------------------
. rctmiss, pmmdelta(0): reg sf_mcs alloc sf_mcsbaResults allowing for MNARUsing 349 complete cases and 37 incomplete cases------------------------------------------------------------------------------ sf_mcs | Coef. Std. Err. z P>|z| [95% Conf. Interval]-------------+---------------------------------------------------------------- alloc | -.3341637 1.108198 -0.30 0.763 -2.506193 1.837865 sf_mcsba | .4703734 .0475847 9.88 0.000 .3771091 .5636377 _cons | 22.62969 2.05461 11.01 0.000 18.60273 26.65665------------------------------------------------------------------------------
43
Main difficulty
• How to specify ?• Need to discuss with investigators
– aim to pre-specify• Alternatively report value of that changes [significance
of] conclusions
44
Example 2: smoking cessation trial
• Smokers calling a telephone helpline were randomised to tailored advice letters + standard counselling or standard counselling – Sutton & Gilbert (2007)
• Outcome: not smoking in months 4-6 (“quit”)
Intervention arm (n=599)
Control arm (n=565)
Quit 73 (12%) 51 (9%)
Didn’t quit 390 (65%) 364 (64%)
Missing 136 (23%) 150 (27%)
45
Smoking cessation trial: analyses
• Main assumption: assume missing = still smoking– Includes everyone, but not convincing
• Main analysis: impute missing values as smokers– OR 1.40 (95 CI 0.96 - 2.05)
• Sensitivity analyses: define the informatively missing odds ratio (IMOR) as the odds ratio between quitting and missing– main analysis assumes IMOR=0– sensitivity analysis varies IMOR (in one or both arms)
46
Smoking trial: sensitivity analysis1
2In
terv
en
tion o
dds
rati
o (
95%
CI)
0 0.1 0.2 0.3 0.4 0.5IMOR in specified arm(s)
intervention arm(23% missing)
both arms control arm(27% missing)
Main analysis
47
Plan
1. Why do missing data matter? 2. Intention-to-treat analysis strategy for randomised trials
with missing outcomes3. Analysis methods
– regression and mixed models4. Sensitivity analysis
– methods & examples5. Missing baseline covariates
– simple methods are best
48
Missing baselines
Missing baselines in RCTs are a completely different problem from missing outcomes
• Still assume we want to estimate the effect of randomised intervention on outcome
• Baseline adjustment is used to gain precision, not avoid bias– missing baselines are not a source of bias
• Complete cases analysis is a very bad idea• White & Thompson (2005) showed that almost anything
else is OK– in particular, mean imputation or missing indicator
method– provided randomisation is respected
49
Missing indicator method
• Regress Y on covariates
– MX = 1 if X observed, else 0
– Xfill = X if X observed, else 0 (or some fixed X0)
• Can use weights (good for large X-Y correlation)• But missing indicator method is biased in observational
studies (Greenland and Finkle 1995)– get imperfect adjustment for confounders (residual
confounding)
50
What’s wrong with the missing indicator method?
05
1015
2025
Out
com
e
0 5 10 15 20 25Exposure
0
1
Confounder
51
0055
10101515
20202525
Out
com
eO
utco
me
00 55 1010 1515 2020 2525ExposureExposure
00
11
??
ConfounderConfounder
Make some confounders missing
52
05
1015
2025
Out
com
e
0 5 10 15 20 25Exposure
?
Confounder
Association is different
53
0055
10101515
20202525
Out
com
eO
utco
me
00 55 1010 1515 2020 2525ExposureExposure
00
11
??
ConfounderConfounder
Missing indicator method averages over confounder = 0, 1 and ?
Residual confounding
Residual confounding doesn’t occur in randomised trials:Covariate is independent of “exposure” so not a “confounder”
54
Missing indicator: how to do it (QUATRO)
. gen xmiss = missing(sf_mcsba)
. gen xfill = cond(xmiss, 39, sf_mcsba)
. reg sf_mcs alloc xmiss xfill
Source | SS df MS Number of obs = 367---------+------------------------------ F( 3, 363) = 32.44 Model | 10630.1786 3 3543.39287 Prob > F = 0.0000Residual | 39646.5297 363 109.21909 R-squared = 0.2114---------+------------------------------ Adj R-squared = 0.2049 Total | 50276.7084 366 137.368056 Root MSE = 10.451
-----------------------------------------------------------------------sf_mcs | Coef. Std. Err. t P>|t| [95% Conf. Interval]-------+--------------------------------------------------------------- alloc | -.2364209 1.095984 -0.22 0.829 -2.391695 1.918854 xmiss | -2.682338 2.527628 -1.06 0.289 -7.65297 2.288295 xfill | .4706946 .0482474 9.76 0.000 .3758151 .5655741 _cons | 22.57087 2.07815 10.86 0.000 18.48414 26.65759-----------------------------------------------------------------------
(or use rctmiss) n increased from 349 to 367
55
US report: “The Prevention and Treatment of Missing Data in Clinical Trials”
• Commissioned by Food & Drug Administration• Panel of top statisticians• National Research Council (Dec 2010)
1. Introduction and Background2. Trial Designs to Reduce the Frequency of Missing Data
– focus on estimands (pre-trial)3. Trial Strategies to Reduce the Frequency of Missing Data4. Drawing Inferences from Incomplete Data
– covers it all5. Principles and Methods of Sensitivity Analyses
– lots of suggestions6. Conclusions and Recommendations
56
Conclusions & discussion
• Missing baselines: use simple methods that respect randomisation
• Missing outcomes: focus on assumptions, not methods• An intention-to-treat analysis strategy should include all
individuals in sensitivity analyses– but not necessarily in main analyses
• ANCOVA and mixed models are usually the best strategy for missing outcomes in RCTs– use MI with auxiliary data (e.g. compliance) or
possibly as a way to do sensitivity analyses• Sensitivity analyses can be done in various ways
– install my software rctmiss in Stata using net from http://www.mrc-bsu.cam.ac.uk/IW_Stata/misc
57
References (1)
• Altman D et al (2001). The revised CONSORT statement for reporting randomized trials: Explanation and elaboration. Annals of Internal Medicine 134: 663–694.
• Committee for proprietary medicinal products (2001). Points to consider on missing data. http://www.emea.europa.eu/pdfs/human/ewp/177699EN.pdf
• European Medicines Agency (2010). Guideline on missing data in confirmatory clinical trials. http://www.ema.europa.eu/ema/pages/includes/document/ open_document.jsp?webContentId=WC500096793
• Greenland S, Finkle W (1995). A critical look at methods for handling missing covariates in epidemiologic regression analyses. American Journal of Epidemiology 142: 1255–1264.
• Gray R et al (2006). Adherence therapy for people with schizophrenia: European multicentre randomised controlled trial. British Journal of Psychiatry 189: 508–514.
• Hollis S, Campbell F (1999). What is meant by intention to treat analysis? Survey of published randomised controlled trials. British Medical Journal 319: 670–4.
• Kenward MG, Goetghebeur EJT, Molenberghs G (2001). Sensitivity analysis for incomplete categorical tables. Statistical Modelling 1: 31–48.
• Mallinckrodt CH et al (2004). The effect of correlation structure on treatment contrasts estimated from incomplete clinical trial data with likelihood-based repeated measures compared with last observation carried forward ANOVA. Clinical Trials 1:477–489.
58
References (2)
• Moher D, Schulz KF, Altman DA, for the CONSORT group (2001). The CONSORT statement: Revised recommendations for improving the quality of reports of parallel-group randomised trials. Lancet 357:1191–1194.
• National Research Council (2010). The Prevention and Treatment of Missing Data in Clinical Trials. Panel on Handling Missing Data in Clinical Trials. Committee on National Statistics, Division of Behavioral and Social Sciences and Education. http://www.nap.edu/catalog.php?record_id=12955.
• Sutton S, Gilbert H (2007). Effectiveness of individually tailored smoking cessation advice letters as an adjunct to telephone counselling and generic self-help materials: randomized controlled trial. Addiction 102:994–1000.
• Riper H et al (2008). Web-based self-help for problem drinkers: a pragmatic randomized trial. Addiction 103:218–227.
• White IR, Horton N, Carpenter J, Pocock SJ (2011). An intention-to-treat analysis strategy for randomised trials with missing outcome data. British Medical Journal http://www.bmj.com/content/342/bmj.d40.full.html? ijkey=08IQJ8etzBYYGLw&keytype=ref.
• White IR, Thompson SG (2005). Adjusting for partially missing baseline measurements in randomised trials. Statistics in Medicine 24:993–1007.
• White IR, Wood A, Royston P (2011). Multiple imputation using chained equations: issues and guidance for practice. Statistics in Medicine; 30: 377–399.
59
The end