REVIEW ARTICLE

Marginal structural models might overcome confounding when analyzingmultiple treatment effects in observational studies

David Suareza,b,*, Josep Maria Haroa, Diego Novickc, Susana Ochoaa

aResearch & Development Unit, Sant Joan de D�eu-SSM, Fundacio Sant Joan de D�eu, RETICS RD06/0011(REM-TAP Network), Sant Boi, Barcelona, SpainbDepartment of Public Health, University of Barcelona, Barcelona, Spain

cEli Lilly and Company, Windlesham, Surrey, UK

Accepted 16 November 2007

Abstract

Objective: We review marginal structural models (MSMs) and show how they are useful when comparing the effects of multiple treat-ments on outcomes in observational studies. Until now, MSMs have not been used to compare the effects of more than two treatments.

Study Design and Setting: To illustrate the application of MSMs when patients may receive several treatments, we have reanalyzed theeffects of antipsychotic medication on achieving remission in schizophrenia using data from the SOHO study, a 3-year observational studyof health outcomes associated with the treatment of schizophrenia.

Results: The MSM results were, in general, consistent with but less statistically significant than those obtained using conventionalmethods. The MSM also showed qualitative differences in some comparisons in which the conventional analysis obtained results that werenot consistent with previous knowledge.

Conclusion: MSMs can be used to analyze multiple treatment effects. MSMs, by using inverse-probability of treatment weights, mightprovide a better control for confounding than conventional methods by improving the adjustment for treatment group differences in obser-vational studies, which may approximate their results to those of randomized controlled trials. � 2008 Elsevier Inc. All rights reserved.

Keywords: Causality; Confounding; Marginal structural models; Observational studies; Pharmacoepidemiology; Treatment effectiveness

Journal of Clinical Epidemiology 61 (2008) 525e530

1. Introduction

Randomized clinical trials (RCTs) are the ‘‘gold stan-dard’’ study designs for judging treatment benefits [1].However, the controlled circumstances in which they areconducted limit the generalizability of results to day-to-day clinical practice [2]. Observational studies are con-sidered to complement RCTs because they aim to representthe ‘‘real’’ clinical situation [3]. Their design maximizesexternal validity, but this is usually at the expense of lossof some degree of internal validity. Although several tech-niques have been developed to analyze the relationship be-tween treatments and outcomes in observational studies

Declaration of interest: David Suarez is Statistical Consultant for the

SOHO study. Josep Maria Haro has received economic compensation for

his participation in the SOHO Advisory Board. Diego Novick is a Lilly

employee.

* Corresponding author. Sant Joan de D�eu-Serveis de Salut Mental,

Fundacio Sant Joan de D�eu, Research and Development Unit, Dr. Antoni

Pujades, 42, 08830eSant Boi de Llobregat, Barcelona, Spain. Tel.: þ34-

93-600-26-82; fax: þ34-93-652-00-51.

E-mail address: david.suarez.lamas@gmail.com (D. Suarez).

0895-4356/08/$ e see front matter � 2008 Elsevier Inc. All rights reserved.

doi: 10.1016/j.jclinepi.2007.11.007

[4,5], the resultant treatment effects can only be consideredassociated effects rather than causal effects.

In RCTs, treatment influences on outcomes are usuallyconsidered as causal because the patients taking differenttreatments are supposed to be ‘‘exchangeable,’’ that is, theircharacteristics, except the intervention that is evaluated, areexpected to be the same [6]. However, in observationalstudies, the assumption of exchangeability is not validbecause patients are prescribed different medicationsprecisely because they differ in prognostic factors [7].

Marginal structural models (MSMs) are a class of causalmodels developed by Robins and colleagues [8e12]. Theyare useful for properly estimating treatment effects bycontrolling for confounding and selection bias in observa-tional studies [13,14]. A main feature of the MSM is theuse of inverse-probability of treatment (IPT) weights thatcreate an artificial population where, removing covariateimbalances, the exchangeability assumptions hold, makingpossible the estimation of causal effects [15].

To our knowledge, MSMs have not yet been used to com-pare the effects of more than two treatments. The objectivesof this paper are, first, to present a way of comparing

526 D. Suarez et al. / Journal of Clinical Epidemiology 61 (2008) 525e530

multiple treatment effects using an MSM and, second, toillustrate this by comparing the effects of antipsychotic treat-ments on remission in outpatients with schizophreniaobtained with ‘‘conventional’’ and MSM methods, usingdata from the Schizophrenia Outpatient Health Outcomes(SOHO) study, a large, 3-year, longitudinal, observationalstudy of health outcomes associated with the treatment ofschizophrenia [16].

2. Methods

2.1. The SOHO study

As the SOHO study is used to illustrate the methods pre-sented in this paper, we will briefly discuss its design. TheSOHO study is a 3-year, longitudinal, observational studyof health outcomes in the antipsychotic treatment of schizo-phrenia, in which 1,096 psychiatrists from 10 Europeancountries enrolled over 10,000 patients [17]. Outpatientswith schizophrenia, aged at least 18 years, who were initi-ating or changing antipsychotic medication within the nor-mal course of care could be enrolled. Stratified samplingwith oversampling was used to provide two patient cohortsof approximately equal size: patients who initiated therapywith or changed to olanzapine; and patients who initiatedtherapy with or changed to a nonolanzapine antipsychotic.Patients were evaluated at baseline, 3 months, 6 months,and then every 6 months up to 36 months postbaseline.The main outcomedclinical severitydwas assessed bythe investigator using a scale based on the Clinical GlobalImpression (CGI) scale [18], which evaluated positive,negative, cognitive, depressive, and overall symptoms onthe day of assessment. This physician-rated scale has valuesranging from 1 (not ill) to 7 (among the most severely illpatients). A notable feature of the SOHO study is that pa-tients could change medication at any point and still remainin the study and be evaluated.

2.2. Definition of remission

Remission was defined as a score of 3 (mild severity) orless on a scale of 1e7 on the CGI overall severity score, theCGI positive symptoms score, the CGI negative symptomsscore and the CGI cognitive symptoms score, maintainedfor a period of 6 months or more. In addition, the patientmust not have been hospitalized due to schizophreniaduring this period. The definition of remission was basedon the one proposed by Andreasen and colleagues [19],and was validated in a parallel study [20].

2.3. The analysis of remission using ‘‘conventional’’statistical methods

Here we review the analyses presented by Haro et al.[16] that compared the effects of several antipsychotictreatments on the frequency of remission. Only patients

with no or one missing visit were included in the analysis(N 5 6,516). For patients with one missing visit, valuesfrom the previous visit were carried forward for that visit.

A multiple logistic regression model was used to analyzethe factors associated with periods in remission during fol-low up. As remission was defined as having a low diseaseseverity for a period of at least 6 months, the follow-upobservations were divided into 6-month periods (6e12months, 12e18 months, 18e24 months, 24e30 months,and 30e36 months). Each 6-month period was an observa-tion in the statistical model. A Generalized EstimatingEquation (GEE) model with a logit link and an unstructuredcorrelation matrix was used to account for correlationamong observations [21]. The covariates included in themodel were the sociodemographic and clinical variables as-sessed at baseline (sociodemographic variables: country,sex; clinical variables: age at first treatment for schizophre-nia, time since first treatment, alcohol dependency, sub-stance abuse, suicide attempts, overall CGI, positive CGI,negative CGI, depressive CGI, cognitive CGI, hostility,compliance, body mass index [BMI]; social functioningvariables: marital status, living independently, having paidemployment, being socially active; extrapyramidal symp-toms; tardive dyskinesia; amenorrhoea; gynecomastia; im-potence; galactorrhoea; use of concomitant medication[anticholinergics, antidepressants, anxiolytics, and moodstabilizers]; and visit) and the medication taken at the be-ginning of each of the periods, defined as the medicationthat the patient was taking just before the first visit of theperiod occurred. Backward model reduction was conductedby dropping from the model the nonsignificant variables ata 0.05 level.

2.4. Marginal structural models (MSMs)

In observational studies, factors that influence thetime-dependent exposures or the assignment to differenttreatment groups may also influence the outcomes and,therefore, may confound the association of interest. Underthese circumstances, the observed response differencescannot be attributed directly to exposures. Proper estima-tion of causal effects must account for time-dependentconfounding.

MSMs are causal models that can eliminate the con-founding and selection bias of standard methods, whichare outlined below. The standard approach for estimatingthe effect of time-varying treatments on binary outcomes(such as the effects of antipsychotic treatment on remissionin the SOHO study) is to model the logit of the probabilityoutcome at each time point as a function of past treatment.However, the standard approach could be biased, irrespec-tive of further adjustments for past covariate and outcomehistory in the analysis. Bias may occur when (1) condi-tional on past treatment history, a time-dependent variableis a predictor of the subsequent outcome and also a predictorof subsequent treatment; and (2) past treatment history is an

D. Suarez et al. / Journal of Clinical

independent predictor of the time-dependent variable. Werefer to covariates satisfying (1) as time-dependent con-founders. For example, in the SOHO study, past CGI scores(or past side effects) are time-dependent confounders forthe effect of antipsychotic treatment on remission as theynot only predict future CGI scores, but also subsequentchanges of antipsychotic treatment [22]. Moreover, past an-tipsychotic treatment is an independent predictor of sub-sequent CGI scores [23]. Thus, the standard method forrepeated measures, such as logistic GEE regressions thatmodel the probability of remission at each time point usingprevious antipsychotic treatment, could produce biased es-timates of the effect of antipsychotic treatment regardlessof whether one adjusts for past CGI scores (or past sideeffects) in the analyses. MSMs can eliminate this type ofbias.

2.5. IPT weights

A key step in the model is the derivation of IPT weights.Different types of weights can be derived for MSMs (unsta-bilized, stabilized, augmented), but here we focus on ‘‘sta-bilized’’ weights, which have good convergence properties,are efficient, and are easy to compute [12].

To introduce the formula for ‘‘stabilized’’ IPT weightswe need some notation. Let i, an indicator of the visit, bei 5 0 for the 3-month visit, i 5 1 for the 6-month visitand so on, until i 5 5 for the 30-month visit. Let T(i) bethe prescribed antipsychotic treatment upon visit i (i.e.,the treatment that the patient was taking just before visit ioccurred). The possible treatments are olanzapine, risperi-done, quetiapine, amisulpride, clozapine, oral typicals, de-pot typicals, or a combination of antipsychotic treatments.Let C(i) be the vector of time-varying confounders atvisit i: overall CGI, positive CGI, negative CGI, depressiveCGI, cognitive CGI, extrapyramidal symptoms, tardivedyskinesia, amenorrhoea, gynecomastia, impotence, galac-torrhoea, and the BMI gain from baseline (i.e., the effec-tiveness and side effects measured in SOHO). Let B bethe baseline covariates: country, sex, age at first treatmentfor schizophrenia, time since first treatment, alcohol depen-dency, substance abuse, suicide attempts, overall CGI,positive CGI, negative CGI, depressive CGI, cognitiveCGI, hostility, compliance, BMI, marital status, livingindependently, having paid employment, being socially ac-tive, extrapyramidal symptoms, amenorrhoea, gynaecomas-tia, impotence, galactorrhoea, tardive dyskinesia, use ofconcomitant medication (anticholinergics, antidepressants,anxiolytics, and mood stabilizers), and the baseline antipsy-chotic treatment. Let TðiÞ be the set of prescribed antipsy-chotic treatments from the 3-month visit to visit i, andfinally let CðiÞ be the set of time-varying confounders fromthe 3-month visit to visit i.

Thus, the formula we propose to estimate the ‘‘stabi-lized’’ IPT weights (SW to abbreviate) at visit t, fromt 5 1 (6-month visit) to t 5 5 (36-month visit), is

SWðtÞ5Yt

k 5 1

f ðTðkÞjTðk� 1Þ;BÞf ðTðkÞjTðk� 1Þ;Cðk� 1Þ;BÞ

ðExpression1Þ

where f is the density. Expression (1) is based on the meth-odology proposed by Robins and colleagues [12]. Infor-mally, the numerator of Expression 1 is the probabilitythat a patient received his or her observed treatment condi-tioned on his or her past treatment and baseline character-istics. The denominator of Expression 1 is the probabilitythat a patient received his or her observed treatment condi-tioned on his or her past treatment and baseline character-istics, but further taking into account his or her pastconfounding history.

2.6. Estimation of ‘‘stabilized’’ IPT weights for multipletreatment comparison

To estimate the ‘‘stabilized’’ IPT weights when multipletreatment choices are possible we used a multinomial re-gression model [24] that treats each person-visit as an ob-servation in the model. For the remission analyses, weestimated the numerator of Expression 1 by fitting a multi-nomial regression model where the dependent variable istreatment (olanzapine, risperidone, quetiapine, amisulpride,clozapine, oral typicals, depot typicals, or a combination ofantipsychotic treatments) and the covariates are the set B ofbaseline variables (see Section 2.5). We also included visitas a covariate. Analogously, to estimate the denominator ofExpression 1, we fitted a multinomial regression modelwhere the dependent variable was again treatment and thecovariates were those included in the numerator plusthe set C of time-varying confounders (see Section 2.5)at the nearest evaluation to the prescribed antipsychotictreatment.

We note that previous antipsychotic treatments were notincluded as covariates in the regression models because thiswould be to some extent ‘‘over-fitting’’ as the rate of treat-ment changes between consecutive visits was small in theSOHO study [25]. In other words, if we include previousantipsychotic treatments in the multinomial regressionmodels, as almost 90% of the previous treatments are coin-cident with 6-month posterior treatments, the models willfail to converge because a quasicomplete separation ofthe data will hold. Nevertheless, we note that the baselineantipsychotic treatment was included as a covariate in themodels.

2.7. Estimation of the MSM

Once the ‘‘stabilized’’ IPT weights were derived, theMSM was fitted by using the weights in the ‘‘conventional’’model. In our analyses of remission, we used the same ap-proach as Haro et al. [16] (see Section 2.3) to fit the model,but used the ‘‘stabilized’’ IPT weights in the logistic GEEregression model. We obtained confidence intervals for

527Epidemiology 61 (2008) 525e530

528 D. Suarez et al. / Journal of Clinical Epidemiology 61 (2008) 525e530

the estimated parameters using robust standard errors.These confidence intervals have coverage probability of atleast 95% [12].

3. Results

A total of 6,516 patients with schizophrenia were in-cluded in the analyses. The majority of patients were male(58%) and the mean age of the population was 40.2 years.Although 10% of patients had never been treated with an-tipsychotics before inclusion in the study, most patientshad a long duration of illness (mean time since first treat-ment was 11.8 years). Approximately one in five patientshad paid employment at a study entry, and 29% hada spouse or partner at baseline. As expected and due tothe SOHO study design, 52% of patients started olanzapinetreatment at baseline.

Table 1 presents the estimated treatment effects of thelogistic GEE regression model (the ‘‘conventional’’estimates), already presented by Haro et al. [16], and the ef-fects estimated using the MSM. As can be seen, the MSMprovided treatment effects that are less statistically sig-nificant compared with the ‘‘conventional’’ estimates.However, in both models, patients receiving olanzapinehad a statistically significantly (P ! 0.05) greater chanceof achieving remission than patients treated with risperi-done, quetiapine, or typical antipsychotics. The moststriking finding is that the estimate comparing clozapinewith olanzapine was not statistically significant using theMSM approach (P 5 0.8209).

To understand why the estimate comparing clozapinewith olanzapine is not statistically significant in theMSM, we plotted the distribution (boxplots) of the loga-rithm of the ‘‘stabilized’’ IPT weights by overall CGI scoreprevious to treatment for both olanzapine and clozapine(Figs. 1 and 2, respectively). From Fig. 1 for olanzapine,we can see that the weights are larger for the higher CGIscores (more severe) than for the lower CGI scores (less se-vere). However, for clozapine (Fig. 2) the opposite

tendency occurs, giving larger weights to the less severepatients who, somehow, can be underrepresented in theoriginal sample. The mean (standard deviation, minimum,maximum) of the ‘‘stabilized’’ IPT weights was 1.079(0.972, 0.004, 31.867) and thus very close to the expectedvalue of 1. Nevertheless, a sensitivity analysis was con-ducted by excluding from the MSM the ‘‘stabilized’’ IPTweights lower than the percentile 1 (0.152) and greater thanthe percentile 99 (4.331). The results of the sensitivity anal-ysis were largely consistent with the presented results (datanot shown).

In the previous results, as Haro et al. [16] did, weincluded in the analyses only patients who completed the3-year follow up or at most missed only one visit. However,MSMs can also accommodate right censoring due to loss-to-follow up. Therefore, as a sensitivity analysis to take intoaccount differences between patients lost to follow up be-fore study end and patients in the study during the 3-yearfollow up, we fitted an MSM including all the availableobservations until the first missing visit (see Appendixfor details). The results were largely consistent with theprevious findings (data not shown).

4. Discussion

In this paper, we have presented an MSM application forestimating comparisons between multiple treatments, andwe have applied this strategy to analyze the effect of severalantipsychotic treatments on remission for patients withschizophrenia.

More causal models are available apart from MSMs, suchas the propensity score method [26,27] and the structuralnested models estimated by g-estimation [12]. However,the propensity score method does not have straightforwardgeneralization to studies with time-dependent treatmentsor exposures [12] (such as the SOHO study), and structuralnested models estimated by g-estimation cannot be conve-niently used to estimate the effects of treatments on dichot-omous outcomes [9] (such as remission). Therefore, MSMswere chosen for the present analysis.

Table 1

Comparison of ‘‘conventional’’ (logistic GEE model) and MSM estimates for multiple treatment effects to achieve remission (olanzapine is the reference

treatment)

Treatment

‘‘Conventional’’ estimates MSM estimates

OR Lower 95% CI Upper 95% CI OR Lower 95% CI Upper 95% CI

Amisulpride 0.726* 0.561 0.938 0.835 0.574 1.216

Clozapine 0.784* 0.649 0.946 1.038 0.753 1.430

Depot typical 0.592** 0.505 0.693 0.771* 0.622 0.956

Oral typical 0.636** 0.548 0.738 0.724* 0.527 0.995

Quetiapine 0.657** 0.563 0.766 0.729** 0.578 0.919

Risperidone 0.740** 0.661 0.829 0.764** 0.647 0.902

Combination 0.637** 0.577 0.703 1.012 0.853 1.200

Abbreviations: OR, odds ratio; CI, confidence interval; GEE, Generalized Estimating Equations; MSM, Marginal Structural Model.

An OR !1 indicates a lower likelihood of achieving remission comparing with olanzapine treatment.

*0.05 O P O 0.01.

**P ! 0.01 vs. reference category.

529Epidemiology 61 (2008) 525e530

In agreement with the findings of Haro et al. [16] usingconventional methods, our reanalyses of the effects of anti-psychotic treatments on remission for patients with schizo-phrenia found that patients receiving olanzapine hada better chance of achieving remission than patients treatedwith risperidone, quetiapine, or typical antipsychotics. Incontrast to Haro et al. [16], we found no significant differ-ence between clozapine and olanzapine treatments whenwe used the MSM. However, Haro et al. [16] did pointout that their finding that olanzapine was superior to cloza-pine was surprising because clozapine has consistently beenfound to compare favorably with other antipsychotics fortreatment-resistant patients [28,29]. These authors arguedthat a plausible explanation for their finding was that thepatients treated with clozapine were likely to be severe,treatment-resistant patients, who may improve with treat-ment, but not enough to achieve remission. As we haveseen in the Section 3, the use of the IPT weights has beenable to balance the severity differences between patients

1 2 3 4 5 6 7-4

-2

0

2

4

Lo

g S

W

Overall CGI severity

Fig. 1. Boxplots of the logarithm of ‘‘stabilized’’ inverse-probability of

treatment weights (SW) for olanzapine by previous overall CGI severity.

1-4

-2

0

2

4

Lo

g S

W

Overall CGI severity

765432

Fig. 2. Boxplots of the logarithm of ‘‘stabilized’’ inverse-probability of

treatment weights (SW) for clozapine by previous overall CGI severity.

D. Suarez et al. / Journal of Clinical

who were prescribed olanzapine and those who were pre-scribed clozapine. These results appear to be consistentwith those of previous RCTs [30].

A limitation of the present results is that we have in-cluded in the main analysis only those patients completingthe 3-year follow up (or missing at most one visit) and ina sensitivity analysis, patients’ observations until the firstmissing visit. Therefore, the impossibility of including ob-servations coming from posterior visits to the first missingvisit could bias the results [31,32]. More research is neededto apply MSMs in observational studies with intermittentmissing patterns, such as those in the SOHO study.

Another limitation of our analyses is that to assurecausal estimations with MSMs, we must not miss any rele-vant covariate that could influence the results and we can-not rule out this possibility. However, this is also true for allstandard analyses.

Lastly, a limitation of using the methods presented inthis paper with other data sets is the number of patients in-cluded in the analysis. The SOHO study is a large data setbut the estimation of the IPT weights for multiple treat-ments with small data sets could be problematic.

In conclusion, MSMs can be used to compare multipletreatment effects and, under some assumptions (accurateinformation, no unmeasured confounders, noninformativecensoring, and no model misspecification), the results mightbe considered causal, irrespective of the presence of time-varying confounders. Moreover, MSMs might enhance theresults from observational studies, providing a better controlfor confounding than conventional methods, and bringingthe results closer to those obtained with RCTs.

Acknowledgments

The SOHO Study Group: (Denmark) Karsten HaderupKristensen, Aalborg Psykiatrisk Hospital, Aalborg, Den-mark; (France) Jean-Pierre Lepine, Hopital Fernand Widal,Paris, France; (France) Isabelle Gasquet, Hopital PaulBrousse, Villejuif, France; (Germany) Dieter Naber,Universitatskrankenhaus-Eppendorf, Klinik fur Psychiatrieund Psychotherapie, Hamburg, Germany; (Greece) Venet-sanos G. Mavreas, Department of Psychiatry, Universityof Ioannina, Greece; (Ireland) Declan Murray, St. Ita’s Hos-pital, Dublin, Ireland; (Italy) Paolo Pancheri, FondazioneItaliana per lo studio della Schizofrenia (FIS), Rome, Italy;(The Netherlands) CJ Slooff, Psychosencluster GGXN-Drenthe, Kenniscentrum Scizofrenie, RA Assen, TheNetherlands; (Portugal) Joao Marques Teixeira, Universitydo Porto, Porto, Portugal; (Spain) Jordi Alonso, Health Ser-vices Research Unit, Institut Municipal d’InvestigacioMedica, Universitat Autonoma de Barcelona, Barcelona,Spain; (Spain) Josep Maria Haro, Research and Develop-ment Unit, Sant Joan de D�eu-SSM, Sant Boi, Barcelona,Spain; (Spain) Manuel Bousono, Universidad de Oviedo,Asturias, Spain; (UK) Tim Croudace, Department of Psy-chiatry, Addenbrooke’s Hospital, Cambridge, UK; (UK)

530 D. Suarez et al. / Journal of Clinical Epidemiology 61 (2008) 525e530

Peter B. Jones, University of Cambridge, Addenbrooke’sHospital, Cambridge, UK; (UK) Martin Knapp, LondonSchool of Economics, Centre for the Economics of MentalHealth, Institute of Psychiatry, London, UK.

David Suarez was supported by a research grant fromthe Instituto de Salud Carlos III, Ministerio de Sanidad yConsumo, Spain (FIS CA05/0177).

The authors thank Dr. Miguel Hernan and the twoanonymous reviewers for their helpful suggestions.

Appendix

Estimation with censored data in MSMs

To take into account right censoring it is only necessaryto modify the IPT weights to SW(t)�SW0(t) whereSW0(t) is

SW0ðtÞ5Yt

k 5 1

PrðRðkþ 1Þ5 0jRðkÞ5 0;TðkÞ;BÞPrðRðkþ 1Þ5 0jRðkÞ5 0;TðkÞ;CðkÞ;BÞ

In the above expression, R(t) 5 0 means that a patient re-mains in the study beyond time t and the other variables aredefined as in the Section 2. To estimate from the data boththe numerator and the denominator of SW0(t) multiplelogistic regressions can be fitted.

References

[1] Friedman LM, Furberg CD, DeMets DL. Fundamentals of clinical

trials. St. Louis, MO: Mosby-Year Book; 1996.

[2] Linde K, Streng A, Hoppe A, Weidenhammer W, Wagenpfeil S,

Melchart D. Randomized trial vs. observational study of acupuncture

for migraine found that patient characteristics differed but outcomes

were similar. J Clin Epidemiol 2007;60:280e7.

[3] Rubin DB. Estimating causal effects from large data sets using pro-

pensity scores. Ann Intern Med 1997;127(8 Pt 2):757e63.

[4] Haro JM, Kontodimas S, Negrin MA, Ratcliffe M, Suarez D,

Windmeijer F. Methodological aspects in the assessment of treatment

effects in observational health outcomes studies. Appl Health Econ

Health Policy 2006;5:11e25.

[5] Klungel OH, Martens EP, Psaty BM, Grobbee DE, Sullivan SD,

Stricker BH, et al. Methods to assess intended effects of drug treat-

ment in observational studies are reviewed. J Clin Epidemiol

2004;57:1223e31.

[6] Hernan MA. A definition of causal effect for epidemiological

research. J Epidemiol Commun Health 2004;58:265e71.

[7] Hernan MA, Robins JM. Estimating causal effects from epidemiolog-

ical data. J Epidemiol Commun Health 2006;60:578e86.

[8] Hernan MA, Brumback B, Robins JM. Marginal structural models to

estimate the causal effect of Zidovudine on the survival of HIV-

positive men. Epidemiology 2000;11:561e70.

[9] Hernan MA, Brumback B, Robins JM. Marginal structural models to

estimate the joint causal effect of nonrandomized treatments. J Am

Stat Assoc 2001;96:440e8.

[10] Hernan MA, Brumback B, Robins JM. Estimating the causal effect of

Zidovudine on CD4 count with a marginal structural model for

repeated measures. Stat Med 2002;21:1689e709.

[11] Robins JM. Association, causation, and marginal structural models.

Synthese 1999;121:151e79.

[12] Robins JM, Hernan MA, Brumback B. Marginal structural models and

causal inference in epidemiology. Epidemiology 2000;11:550e60.

[13] Cole SR, Hernan MA, Robins JM, Anastos K, Chmiel J, Detels R,

et al. Effect of highly active antiretroviral therapy on time to acquired

immunodeficiency syndrome or death using marginal structural

models. Am J Epidemiol 2003;158:687e94.

[14] Cole SR, Hernan MA, Margolick JB, Cohen MH, Robins JM. Mar-

ginal structural models for estimating the effect of highly active anti-

retroviral therapy initiation on CD4 cell count. Am J Epidemiol

2005;162:471e8.

[15] Joffe MM, Ten Have TR, Feldman HI, Kimmel SE. Model selection,

confounder control, and marginal structural models: review and new

applications. Am Stat 2004;58:272e9.

[16] Haro JM, Novick D, Suarez D, Alonso J, Lepine JP, Ratcliffe M.

Remission and relapse in the outpatient care of schizophrenia: three-

year results from the Schizophrenia Outpatient Health Outcomes

study. J Clin Psychopharmacol 2006;26:571e8.

[17] Haro JM, Edgell ET, Jones PB, Alonso J, Gavart S, Gregor KJ, et al.

SOHO Study Group. The European Schizophrenia Outpatient Health

Outcome (SOHO) Study: rationale, methods and recruitment. Acta

Psychiatrica Scand 2003;107:222e32.

[18] Guy W. Clinical global impression. ECDEU assessment manual for

psychopharmacology, revised. Rockville, MD: National Institute of

Mental Health; 1976.

[19] Andreasen NC, Carpenter WT Jr, Kane JM, Lasser RA, Marder SR,

Weinberger DR. Remission in schizophrenia: proposed criteria and

rationale for consensus. Am J Psychiatry 2005;162:441e9.

[20] Haro JM, Ochoa S, Gervin M, Mavreas V, Jones P. Assessment of re-

mission in schizophrenia with the CGI and CGI-SCH scales. [Letter].

Acta Psychiatr Scand 2007;115:163e4.

[21] Zeger SL, Liang KY. Longitudinal data analysis for discrete and con-

tinuous outcomes. Biometrics 1986;42:121e30.

[22] Kane JM, Leucht S, Carpenter D, Docherty JP. Expert consensus

guideline series. Optimizing pharmacologic treatment of psychotic

disorders. Introduction: methods, commentary, and summary. J Clin

Psychiatry 2003;64(Suppl 12):5e19.

[23] Haro JM, Edgell ET, Novick D, Alonso J, Kennedy L, Jones PB, et al.

SOHO advisory board. Effectiveness of antipsychotic treatment for

schizophrenia: 6-month results of the Pan-European Schizophrenia

Outpatient Health Outcomes (SOHO) study. Acta Psychiatr Scand

2005;111:220e31.

[24] Agresti A. Categorical data analysis. 2nd edition. Hoboken NJ:

Wiley; 2002.

[25] Haro JM, Suarez D, Novick D, Brown J, Usall J, Naber D. Three year

antipsychotic effectiveness in the outpatient care of schizophrenia:

observational versus randomized studies results. Eur Neuropsycho-

pharmacol 2007;17:235e44.

[26] D’Agostino RB. Propensity score methods for bias reduction in the

comparison of a treatment to a non-randomized control group. Stat

Med 1998;17:2265e81.

[27] Rosenbaum PR, Rubin DB. The central role of the propensity score in

observational studies for causal effects. Biometrika 1983;70:41e55.

[28] Mauri MC, Volonteri LS, Dell’Osso B, Regispani F, Papa P, Baldi M,

et al. Predictors of clinical outcome in schizophrenic patients re-

sponding to clozapine. J Clin Psychopharmacol 2003;23:660e4.

[29] McEvoy JP, Lieberman JA, Stroup TS, Davis SM, Meltzer HY,

Rosenheck RA, et al. CATIE Investigators. Effectiveness of clozapine

versus olanzapine, quetiapine, and risperidone in patients with

chronic schizophrenia who did not respond to prior atypical antipsy-

chotic treatment. Am J Psychiatry 2006;163:600e10.

[30] Naber D, Riedel M, Klimke A, Vorbach EU, Lambert M, Kuhn KU,

et al. Randomized double blind comparison of olanzapine vs. cloza-

pine on subjective well-being and clinical outcome in patients with

schizophrenia. Acta Psychiatr Scand 2005;111:106e15.

[31] Donders AR, van der Heijden GJ, Stijnen T, Moons KG. Review: a

gentle introduction to imputation of missing values. J Clin Epidemiol

2006;59:1087e91.

[32] Gorelick MH. Bias arising from missing data in predictive models.

J Clin Epidemiol 2006;59:1115e23.