30

Thumbnail - download.e-bookshelf.de...4.2.4 IPTW for bias removal 84 4.3 MMWS 86 4.3.1 Theoretical rationale 86 ... 8.3.4.2 Growth model in the absence of confounding 200 8.3.4.3 Weighted

  • Upload
    others

  • View
    1

  • Download
    0

Embed Size (px)

Citation preview

Page 1: Thumbnail - download.e-bookshelf.de...4.2.4 IPTW for bias removal 84 4.3 MMWS 86 4.3.1 Theoretical rationale 86 ... 8.3.4.2 Growth model in the absence of confounding 200 8.3.4.3 Weighted
pg2922
File Attachment
Thumbnailjpg

Causality in a Social World

Causality in a Social World

Moderation Meditation and Spill-over

Guanglei Hong

Department of Comparative Human DevelopmentUniversity of Chicago USA

This edition first published 2015copy 2015 John Wiley amp Sons Ltd

Registered OfficeJohn Wiley amp Sons Ltd The Atrium Southern Gate Chichester West Sussex PO19 8SQ United Kingdom

For details of our global editorial offices for customer services and for information about how to apply for permissionto reuse the copyright material in this book please see our website at wwwwileycom

The right of the author to be identified as the author of this work has been asserted in accordance with the CopyrightDesigns and Patents Act 1988

All rights reserved No part of this publication may be reproduced stored in a retrieval system or transmitted in anyform or by any means electronic mechanical photocopying recording or otherwise except as permitted by theUK Copyright Designs and Patents Act 1988 without the prior permission of the publisher

Wiley also publishes its books in a variety of electronic formats Some content that appears in print may not beavailable in electronic books

Designations used by companies to distinguish their products are often claimed as trademarks All brand names andproduct names used in this book are trade names service marks trademarks or registered trademarks of their respectiveowners The publisher is not associated with any product or vendor mentioned in this book

Limit of LiabilityDisclaimer of Warranty While the publisher and author have used their best efforts in preparing this bookthey make no representations or warranties with respect to the accuracy or completeness of the contents of this bookand specifically disclaim any implied warranties of merchantability or fitness for a particular purpose It is sold on theunderstanding that the publisher is not engaged in rendering professional services and neither the publisher nor the authorshall be liable for damages arising herefrom If professional advice or other expert assistance is required the services ofa competent professional should be sought

Library of Congress Cataloging-in-Publication Data applied for

ISBN 9781118332566

A catalogue record for this book is available from the British Library

Set in 1012pt Times by SPi Global Pondicherry India

1 2015

Contents

Preface xv

Part I Overview 1

1 Introduction 311 Concepts of moderation mediation and spill-over 3

111 Moderated treatment effects 5112 Mediated treatment effects 7113 Spill-over effects of a treatment 8

12 Weighting methods for causal inference 1013 Objectives and organization of the book 1114 How is this book situated among other publications on related topics 12

2 Review of causal inference concepts and methods 1821 Causal inference theory 18

211 Attributes versus causes 18212 Potential outcomes and individual-specific causal effects 19213 Inference about population average causal effects 22

2131 Prima facie effect 242132 Ignorability assumption 25

22 Applications to Lordrsquos paradox and Simpsonrsquos paradox 27221 Lordrsquos paradox 27222 Simpsonrsquos paradox 31

23 Identification and estimation 34231 Selection bias 35232 Sampling bias 35233 Estimation efficiency 36

Appendix 21 Potential bias in a prima facie effect 36Appendix 22 Application of the causal inference theory to Lordrsquos paradox 37

3 Review of causal inference designs and analytic methods 4031 Experimental designs 40

311 Completely randomized designs 40

312 Randomized block designs 41313 Covariance adjustment for improving efficiency 43314 Multilevel experimental designs 43

32 Quasiexperimental designs 44321 Nonequivalent comparison group designs 44322 Other quasiexperimental designs 45

33 Statistical adjustment methods 46331 ANCOVA and multiple regression 46

3311 ANCOVA for removing selection bias 463312 Potential pitfalls of ANCOVA with a vast between-group

difference 473313 Bias due to model misspecification 48

332 Matching and stratification 50333 Other statistical adjustment methods 51

3331 The IV method 513332 DID analysis 54

34 Propensity score 55341 What is a propensity score 56342 Balancing property of the propensity score 57343 Pooling conditional treatment effect estimate Matching

stratification and covariance adjustment 603431 Propensity score matching 613432 Propensity score stratification 623433 Covariance adjustment for the propensity score 663434 Sensitivity analysis 66

Appendix 3A Potential bias due to the omission of treatment-by-covariateinteraction 70

Appendix 3B Variable selection for the propensity score model 71

4 Adjustment for selection bias through weighting 7641 Weighted estimation of population parameters in survey sampling 77

411 Simple random sample 77412 Proportionate sample 78413 Disproportionate sample 79

42 Weighting adjustment for selection bias in causal inference 80421 Experimental result 81422 Quasiexperimental result 81423 Sample weight for bias removal 82424 IPTW for bias removal 84

43 MMWS 86431 Theoretical rationale 86

4311 MMWS for a discrete propensity score 874312 MMWS for a continuous propensity score 884313 MMWS for estimating the treatment effect on the treated 89

432 MMWS analytic procedure 91433 Inherent connection and major distinctions between MMWS

and IPTW 93

vi CONTENTS

Appendix 4A Proof of MMWS-adjusted mean observed outcome being unbiasedfor the population average potential outcome 95

Appendix 4B Derivation of MMWS for estimating the treatment effecton the treated 96

Appendix 4C Theoretical equivalence of MMWS and IPTW 97Appendix 4D Simulations comparing MMWS and IPTW under misspecifications

of the functional form of a propensity score model 97

5 Evaluations of multivalued treatments 10051 Defining the causal effects of multivalued treatments 10052 Existing designs and analytic methods for evaluating multivalued

treatments 102521 Experimental designs and analysis 102

5211 Randomized experiments with multipletreatment arms 102

5212 Identification under ignorability 1025213 ANOVA 103

522 Quasiexperimental designs and analysis 1055221 ANCOVA and multiple regression 1055222 Propensity score-based adjustment 1085223 Other adjustment methods 112

53 MMWS for evaluating multivalued treatments 112531 Basic rationale 113532 Analytic procedure 114

5321 MMWS for a multinomial treatment measure 1155322 MMWS for an ordinal treatment measure 120

533 Identification assumptions 12154 Summary 123Appendix 5A Multiple IV for evaluating multivalued treatments 124

Part II Moderation 127

6 Moderated treatment effects concepts and existing analytic methods 12961 What is moderation 129

611 Past discussions of moderation 1306111 Purpose of moderation research 1306112 What qualifies a variable as a moderator 132

612 Definition of moderated treatment effects 1336121 Treatment effects moderated by individual or contextual

characteristics 1336122 Joint effects of concurrent treatments 134

62 Experimental designs and analytic methods for investigatingexplicit moderators 136621 Randomized block designs 137

6211 Identification assumptions 1376212 Two-way ANOVA 1386213 Multiple regression 139

viiCONTENTS

622 Factorial designs 1406221 Identification assumptions 1406222 Analytic strategies 142

63 Existing research designs and analytic methods for investigatingimplicit moderators 142631 Multisite randomized trials 143

6311 Causal parameters and identification assumptions 1446312 Analytic strategies 145

632 Principal stratification 149Appendix 6A Derivation of bias in the fixed-effects estimator when the

treatment effect is heterogeneous in multisite randomized trials 151Appendix 6B Derivation of bias in the mixed-effects estimator when the

probability of treatment assignment varies across sites 153Appendix 6C Derivation and proof of the population weight applied to

mixed-effects models for eliminating bias in multisiterandomized trials 153

7 Marginal mean weighting through stratification for investigatingmoderated treatment effects 15971 Existing methods for moderation analyses with quasiexperimental data 159

711 Analysis of covariance and regression-based adjustment 1617111 Treatment effects moderated by subpopulation membership 1617112 Treatment effects moderated by a concurrent treatment 164

712 Propensity score-based adjustment 1657121 Propensity score matching and stratification 1667122 Inverse-probability-of-treatment weighting 167

72 MMWS estimation of treatment effects moderated by individual orcontextual characteristics 168721 Application example 170722 Analytic procedure 170

73 MMWS estimation of the joint effects of concurrent treatments 174731 Application example 174732 Analytic procedure 175733 Joint treatment effects moderated by individual or contextual

characteristics 179

8 Cumulative effects of time-varying treatments 18581 Causal effects of treatment sequences 186

811 Application example 186812 Causal parameters 187

8121 Time-varying treatments 1878122 Time-varying potential outcomes 1878123 Causal effects of 2-year treatment sequences 1878124 Causal effects of multiyear treatment sequences 190

82 Existing strategies for evaluating time-varying treatments 190821 The endogeneity problem in nonexperimental data 190822 SEM 191

viii CONTENTS

823 Fixed-effects econometric models 192824 Sequential randomization 192825 Dynamic treatment regimes 193826 Marginal structural models and structural nested models 194

83 MMWS for evaluating 2-year treatment sequences 195831 Sequential ignorability 195832 Propensity scores 196833 MMWS computation 197

8331 MMWS adjustment for year 1 treatment selection 1978332 MMWS adjustment for two-year treatment

sequence selection 198834 Two-year growth model specification 199

8341 Growth model in the absence of treatment 2008342 Growth model in the absence of confounding 2008343 Weighted 2-year growth model 202

84 MMWS for evaluating multiyear sequences of multivalued treatments 204841 Sequential ignorability of multiyear treatment sequences 204842 Propensity scores for multiyear treatment sequences 204843 MMWS computation 205844 Weighted multiyear growth model 205845 Issues of sample size 206

85 Conclusion 207Appendix 8A A saturated model for evaluating multivalued treatments

over multiple time periods 207

Part III Mediation 211

9 Concepts of mediated treatment effects and experimental designsfor investigating causal mechanisms 21391 Introduction 21492 Path coefficients 21593 Potential outcomes and potential mediators 216

931 Controlled direct effects 217932 Controlled treatment-by-mediator interaction effect 217

94 Causal effects with counterfactual mediators 219941 Natural direct effect 219942 Natural indirect effect 220943 Natural treatment-by-mediator interaction effect 220944 Unstable unit treatment value 221

95 Population causal parameters 222951 Population average natural direct effect 224952 Population average natural indirect effect 225

96 Experimental designs for studying causal mediation 225961 Sequentially randomized designs 228962 Two-phase experimental designs 228963 Three- and four-treatment arm designs 230964 Experimental causal-chain designs 231

ixCONTENTS

965 Moderation-of-process designs 231966 Augmented encouragement designs 232967 Parallel experimental designs and parallel encouragement designs 232968 Crossover experimental designs and crossover encouragement

designs 233969 Summary 234

10 Existing analytic methods for investigating causal mediation mechanisms 238101 Path analysis and SEM 239

1011 Analytic procedure for continuous outcomes 2391012 Identification assumptions 2421013 Analytic procedure for discrete outcomes 245

102 Modified regression approach 2461021 Analytic procedure for continuous outcomes 2461022 Identification assumptions 2471023 Analytic procedure for binary outcomes 248

103 Marginal structural models 2501031 Analytic procedure 2501032 Identification assumptions 252

104 Conditional structural models 2521041 Analytic procedure 2521042 Identification assumptions 253

105 Alternative weighting methods 2541051 Analytic procedure 2541052 Identification assumptions 256

106 Resampling approach 2561061 Analytic procedure 2561062 Identification assumptions 257

107 IV method 2571071 Rationale and analytic procedure 2571072 Identification assumptions 258

108 Principal stratification 2591081 Rationale and analytic procedure 2591082 Identification assumptions 260

109 Sensitivity analysis 2611091 Unadjusted confounding as a product of hypothetical

regression coefficients 2611092 Unadjusted confounding reflected in a hypothetical

correlation coefficient 2621093 Limitations when the selection mechanism differs

by treatment 2641094 Other sensitivity analyses 265

1010 Conclusion 26510101 The essentiality of sequential ignorability 26510102 Treatment-by-mediator interactions 26610103 Homogeneous versus heterogeneous causal effects 26610104 Model-based assumptions 266

x CONTENTS

Appendix 10A Bias in path analysis estimation due to the omission oftreatment-by-mediator interaction 267

11 Investigations of a simple mediation mechanism 273111 Application example national evaluation of welfare-to-work strategies 274

1111 Historical context 2741112 Research questions 2751113 Causal parameters 2751114 NEWWS Riverside data 277

112 RMPW rationale 2771121 RMPW in a sequentially randomized design 278

11211 E[Y(0 M(0))] 27911212 E[Y(1 M(1))] 28011213 E[Y(1 M(0))] 28011214 E[Y(0 M(1))] 28111215 Nonparametric outcome model 282

1122 RMPW in a sequentially randomized block design 2831123 RMPW in a standard randomized experiment 2851124 Identification assumptions 286

113 Parametric RMPW procedure 287114 Nonparametric RMPW procedure 290115 Simulation results 292

1151 Correctly specified propensity score models 2921152 Misspecified propensity score models 2941153 Comparisons with path analysis and IV results 294

116 Discussion 2951161 Advantages of the RMPW strategy 2951162 Limitations of the RMPW strategy 295

Appendix 11A Causal effect estimation through the RMPW procedure 296Appendix 11B Proof of the consistency of RMPW estimation 297

12 RMPW extensions to alternative designs and measurement 301121 RMPW extensions to mediators and outcomes of alternative distributions 301

1211 Extensions to a multicategory mediator 30212111 Parametric RMPW procedure 30212112 Nonparametric RMPW procedure 304

1212 Extensions to a continuous mediator 3041213 Extensions to a binary outcome 306

122 RMPW extensions to alternative research designs 3061221 Extensions to quasiexperimental data 3071222 Extensions to data from cluster randomized trials 308

12221 Application example and research questions 30912222 Estimation of the total effect 31012223 RMPW analysis of causal mediation mechanisms 31012224 Identification assumptions 31212225 Contrast with multilevel path analysis and SEM 31212226 Contrast with multilevel prediction models 313

xiCONTENTS

1223 Extensions to data from multisite randomized trials 31312231 Research questions and causal parameters 31412232 Estimation of the total effect and its between-site variation 31512233 RMPW analysis of causal mediation mechanisms 31612234 Identification assumptions 31912235 Contrast with multilevel path analysis and SEM 319

123 Alternative decomposition of the treatment effect 321

13 RMPW extensions to studies of complex mediation mechanisms 325131 RMPW extensions to moderated mediation 325

1311 RMPW analytic procedure for estimating and testingmoderated mediation 326

1312 Path analysisSEM approach to analyzing moderatedmediation 327

1313 Principal stratification and moderated mediation 328132 RMPW extensions to concurrent mediators 328

1321 Treatment effect decomposition 32913211 Treatment effect decomposition without

between-mediator interaction 32913212 Treatment effect decomposition with

between-mediator interaction 3311322 Identification assumptions 3331323 RMPW procedure 333

13231 Estimating E[Y(1M1(1)M2(0))] 33513232 Estimating E[Y(1M1(0)M2(0))] 33513233 Causal effect estimation with noninteracting

concurrent mediators 33613234 Estimating E[Y(1M1(0)M2(1))] 33713235 Causal effect estimation with interacting concurrent

mediators 3371324 Contrast with the linear SEM approach 3381325 Contrast with the multivariate IV approach 339

133 RMPW extensions to consecutive mediators 3401331 Treatment effect decomposition 341

13311 Natural direct effect of the treatment on the outcome 34213312 Natural indirect effect mediated by M1 only 34213313 Natural indirect effect mediated by M2 only 34213314 Natural indirect effect mediated by an M1-by-M2

interaction 34313315 Treatment-by-mediator interactions 343

1332 Identification assumptions 3451333 RMPW procedure 347

13331 Estimating E[Y(1M1(0)M2(0M1(0)))] 34713332 Estimating E[Y(1M1(1)M2(0M1(0)))] 34913333 Estimating E[Y(1M1(0)M2(1M1(1)))] 34913334 Estimating E[Y(0M1(1)M2(0M1(0)))] and

E[Y(0M1(0)M2(1M1(1)))] 351

xii CONTENTS

1334 Contrast with the linear SEM approach 3531335 Contrast with the sensitivity-based estimation of bounds for

causal effects 354134 Discussion 355Appendix 13A Derivation of RMPW for estimating population average

counterfactual outcomes of two concurrentmediators 355

Appendix 13B Derivation of RMPW for estimating population averagecounterfactual outcomes of consecutivemediators 358

Part IV Spill-over 363

14 Spill-over of treatment effects concepts and methods 365141 Spill-over A nuisance a trifle or a focus 365142 Stable versus unstable potential outcome values An example

from agriculture 367143 Consequences for causal inference when spill-over is overlooked 369144 Modified framework of causal inference 371

1441 Treatment settings 3711442 Simplified characterization of treatment settings 3731443 Causal effects of individual treatment assignment and of peer

treatment assignment 375145 Identification Challenges and solutions 376

1451 Hypothetical experiments for identifying average treatmenteffects in the presence of social interactions 376

1452 Hypothetical experiments for identifying the impact ofsocial interactions 380

1453 Application to an evaluation of kindergarten retention 382146 Analytic strategies for experimental and quasiexperimental data 384

1461 Estimation with experimental data 3841462 Propensity score stratification 3851463 MMWS 386

147 Summary 387

15 Mediation through spill-over 391151 Definition of mediated effects through spill-over in a cluster

randomized trial 3931511 Notation 3931512 Treatment effect mediated by a focal individualrsquos

compliance 3941513 Treatment effect mediated by peersrsquo compliance through

spill-over 3941514 Decomposition of the total treatment effect 395

152 Identification and estimation of the spill-over effect in a clusterrandomized design 3951521 Identification in an ideal experiment 395

xiiiCONTENTS

1522 Identification when the mediators are not randomized 3981523 Estimation of mediated effects through spill-over 400

153 Definition of mediated effects through spill-over in a multisite trial 4021531 Notation 4021532 Treatment effect mediated by a focal individualrsquos compliance 4041533 Treatment effect mediated by peersrsquo compliance

through spill-over 4041534 Direct effect of individual treatment assignment on the outcome 4051535 Direct effect of peer treatment assignment on the outcome 4051536 Decomposition of the total treatment effect 405

154 Identification and estimation of spill-over effects in a multisite trial 4061541 Identification in an ideal experiment 4071542 Identification when the mediators are not randomized 4091543 Estimation of mediated effects through spill-over 410

15431 Estimating E[Y(1pM(p)Mminus(p))] andE[Y(000Mminus(0))] 410

15432 Estimating E[Y(1p0Mminus(p))] 41115433 Estimating E[Y(1p0Mminus(0))] 41215434 Estimating E[Y(0p0Mminus(0))] 412

155 Consequences of omitting spill-over effects in causal mediation analyses 4121551 Biased inference in a cluster randomized trial 4131552 Biased inference in a multisite randomized trial 4131553 Biased inference of the local average treatment effect 415

156 Quasiexperimental application 416157 Summary 419Appendix 151 Derivation of the weight for estimating the population

average counterfactual outcome E[Y(1 p 0Mminus( p))] 419Appendix 152 Derivation of bias in the ITT effect due to the omission of

spill-over effects 420

Index 423

xiv CONTENTS

Preface

A scientific mind in training seeks comprehensive descriptions of every interesting phenom-enon In such a mind data are to be pieced together for comparisons contrasts and eventuallyinferences that are required for understanding causes and effects that may have led to thephenomenon of interest Yet without disciplined reasoning a causal inference would slip intothe rut of a ldquocasual inferencerdquo as easily as making a typographical error

As a doctoral student thirsting for rigorous methodological training at the University ofMichigan I was very fortunate to be among the first group of students on the campus attendinga causal inference seminar in 2000 jointly taught by Yu Xie from the Department of Sociol-ogy Susan Murphy from the Department of Statistics and Stephen Raudenbush from theSchool of Education The course was unique in both content and pedagogy The instructorsput together a bibliography drawn from the past and current causal inference literature origi-nated from multiple disciplines In the spirit of ldquoreciprocal teachingrdquo the students were delib-erately organized into small groups each representing a mix of disciplinary backgrounds andtook turns to teach the weekly readings under the guidance of a faculty instructor This inval-uable experience revealed to me that the pursuit of causal inference is a multidisciplinaryendeavor In the rest of my doctoral study I continued to be immersed in an extraordinaryinterdisciplinary community in the form of the Quantitative Methodology Program (QMP)seminars led by the above three instructors who were then joined by Ben Hansen(Statistics) Richard Gonzalez (Psychology) Roderick Little (Biostatistics) Jake Bowers(Political Science) and many others I have tried whenever possible to carry the same spiritinto my own teaching of causal inference courses at the University of Toronto and now at theUniversity of Chicago In these courses communicating understandings of causal problemsacross disciplinary boundaries has been particularly challenging but also stimulating and grat-ifying for myself and for students from various departments and schools

Under the supervision of Stephen Raudenbush I took a first stab at the conceptual frame-work of causal inference in my doctoral dissertation by considering peer spill-over in schoolsettings From that point on I have organized my methodological research to tackle some ofthe major obstacles to causal inferences in policy and program evaluations My workaddresses issues including (i) how to conceptualize and evaluate the causal effects of educa-tional treatments when studentsrsquo responses to alternative treatments depend on various fea-tures of the organizational settings including peer composition (ii) how to adjust forselection bias in evaluating the effects of concurrent and consecutive multivalued treatmentsand (iii) how to conceptualize and analyze causal mediation mechanisms My methodological

research has been driven by the substantive interest in understanding the role of socialinstitutionsmdashschools in particularmdashin shaping human development especially during theages of fast growth (eg childhood adolescence and early adulthood) I have shared meth-odological advances with a broad audience in social sciences through applying the causalinference methods to prominent substantive issues such as grade retention within-class group-ing services for English language learners and welfare-to-work strategies These illustrationsare used extensively in this book

Among social scientists awareness of methodological challenges in causal investigationsis higher than ever before In response to this rising demand causal inference is becoming oneof the most productive scholarly fields in the past decade I have had the benefit of followingthe work and exchanging ideas with some fellow methodologists who are constantly contri-buting new thoughts and making breakthroughs These include Daniel Almirall HowardBloom Tom Cook Michael Elliot Ken Frank Ben Hansen James Heckman JenniferHill Martin Huber Kosuke Imai Booil Jo John R Lockwood David MacKinnon DanielMcCaffrey Luke Miratrix Richard Murnane Derek Neal Lindsey Page Judea PearlStephen Raudenbush Sean Reardon Michael Seltzer Youngyun Shin Betsy SinclairMichael Sobel Peter Steiner Elizabeth Stuart Eric Thetchen Tchetchen Tyler VanderWeeleand Kazuo Yamaguchi I am grateful to Larry Hedges who offered me the opportunity of guestediting the Journal of Research in Educational Effectiveness special issue on the statisticalapproaches to studying mediator effects in education research in 2012 Many of the colleagueswhom I mentioned above contributed to the open debate in the special issue by either propos-ing or critically examining a number of innovative methods for studying causal mechanismsI anticipate that many of them are or will soon be writing their own research monographs oncausal inference if they have not already done so Therefore readers of this book are stronglyrecommended to browse past and future titles by these and other authors for alternative per-spectives and approaches My own understanding of course will continue to evolve as well inthe coming years

One has to be ambitious to stay upfront in substantive areas as well as in methodology Thebest strategy apparently is to learn from substantive experts During different phases of mytraining and professional career I have had the opportunities to work with David K CohenBrian Rowan Deborah Ball Carl Corter Janette Pelletier Takako Nomi Esther Geva DavidFrancis Stephanie Jones Joshua Brown and Heather D Hill Many of my substantive insightswere derived from conversations with these mentors and colleagues I am especially gratefulto Bob Granger former president of the William T Grant Foundation who reassured me thatldquoBy staying a bit broad you will learn a lot and many fields will benefit from your workrdquo

Like many other single-authored books this one is built on the generous contributions ofstudents and colleagues who deserve special acknowledgement Excellent research assistancefrom Jonah Deutsch Joshua Gagne Rachel Garrett Yihua Hong Xu Qin Cheng Yang andBing Yu was instrumental in the development and implementation of the new methods pre-sented in this book Richard Congdon a renowned statistical software programmer broughtrevolutionary ideas to interface designs in software development with the aim of facilitatingusersrsquo decision-making in a series of causal analyses RichardMurnane Stephen Raudenbushand Michael Sobel carefully reviewed and critiqued multiple chapters of the manuscript draftAdditional comments and suggestions on earlier drafts came fromHoward Bloom Ken FrankBen Hansen Jennifer Hill Booil Jo Ben Kelcey David MacKinnon and Fan Yang TereseSchwartzman provided valuable assistance in manuscript preparation The writing of thisbook was supported by a Scholars Award from the William T Grant Foundation and an

xvi PREFACE

Institute of Education Sciences (IES) Statistical and Research Methodology in EducationGrant from the US Department of Education All the errors in the published form are ofcourse the sole responsibility of the author

Project Editors Richard Davis Prachi Sinha Sahay and Liz Wingett and Assistant EditorHeather Kay at Wiley and project Managers P Jayapriya and R Jayavel at SPi Global havebeen wonderful to work with throughout the manuscript preparation and final production ofthe book Debbie Jupe Commissioning Editor at Wiley was remarkably effective in initiatingthe contact with me and in organizing anonymous reviews of the book proposal and of themanuscript These constructive reviews have shaped the book in important ways I cannotagree more with one of the reviewers that ldquocausality cannot be established on a pure statisticalgroundrdquo The book therefore highlights the importance of substantive knowledge and researchdesigns and places a great emphasis on clarifying and evaluating assumptions required foridentifying causal effects in the context of each application Much caution is raised againstpossible misusage of statistical techniques for analyzing causal relationships especiallywhen data are inadequate Yet I maintain that improved statistical procedures along withimproved research designs would greatly enhance our ability in attempt to empiricallyexamine well-articulated causal theories

xviiPREFACE

Part I

OVERVIEW

1

Introduction

According to an ancient Chinese fable a farmer who was eager to help his crops grow wentinto his field and pulled each seedling upward After exhausting himself with the work heannounced to his family that they were going to have a good harvest only to find the nextmorning that the plants had wilted and died Readers with minimal agricultural knowledgemay immediately point out the following the farmerrsquos intervention theory was based on acorrect observation that crops that grow taller tend to produce more yield Yet his hypothesisreflects a false understanding of the cause and the effectmdashthat seedlings pulled to be tallerwould yield as much as seedlings thriving on their own

In their classic Design and Analysis of Experiments Hinkelmann and Kempthorne (1994updated version of Kempthorne 1952) discussed two types of science descriptive science andthe development of theory These two types of science are interrelated in the following senseobservations of an event and other related events often selected and classified for descriptionby scientists naturally lead to one or more explanations that we call ldquotheoretical hypothesesrdquowhich are then screened and falsified by means of further observations experimentation andanalyses (Popper 1963) The experiment of pulling seedlings to be taller was costly but didserve the purpose of advancing this farmerrsquos knowledge of ldquowhat does not workrdquo To developa successful intervention in this case would require a series of empirical tests of explicittheories identifying potential contributors to crop growth This iterative process graduallydeepens our knowledge of the relationships between supposed causes and effectsmdashthatis causalitymdashand may eventually increase the success of agricultural medical and socialinterventions

11 Concepts of moderation mediation and spill-over

Although the story of the ancient farmer is fictitious numerous examples can be found in thereal world in which well-intended interventions fail to produce the intended benefits or inmany cases even lead to unintended consequences ldquoInterventionsrdquo and ldquotreatmentsrdquo usedinterchangeably in this book broadly refer to actions taken by agents or circumstances expe-rienced by an individual or groups of individuals Interventions are regularly seen in educa-tion physical and mental health social services business politics and law enforcement In an

Causality in a Social World Moderation Meditation and Spill-over First Edition Guanglei Hongcopy 2015 John Wiley amp Sons Ltd Published 2015 by John Wiley amp Sons Ltd

education intervention for example teachers are typically the agents who deliver a treatmentto students while the impact of the treatment on student outcomes is of ultimate causalinterest Some educational practices such as ldquoteaching to the testrdquo have been criticized tobe nearly as counterproductive as the attempt of helping seedlings grow by pulling themupward ldquoInterventionsrdquo and ldquotreatmentsrdquo under consideration do not exclude undesiredexperiences such as exposure to poverty abuse crime or bereavement A treatment plannedor unplanned becomes a focus of research if there are theoretical reasons to anticipate itsimpact positive or negative on the well-being of individuals who are embedded in socialsettings including families classrooms schools neighborhoods and workplaces

In social science research in general and in policy and program evaluations in particularquestions concerning whether an intervention works and if so which version of the interven-tion works for whom under what conditions and why are key to the advancement of scien-tific and practical knowledge Although most empirical investigations in the social sciencesconcentrate on the average effect of a treatment for a specific population as opposed to theabsence of such a treatment (ie the control condition) in-depth theoretical reasoning withregard to how the causal effect is generated substantiated by compelling empirical evidenceis crucial for advancing scientific understanding

First when there are multiple versions or different dosages of the treatment or when thereare multiple versions of the control condition a binary divide between ldquothe treatmentrdquo andldquothe controlrdquo may not be as informative as fine-grained comparisons across for exampleldquotreatment version Ardquo ldquotreatment version Brdquo ldquocontrol version Ardquo and ldquocontrol versionBrdquo For example expanding the federally funded Head Start program to poor children isexpected to generate a greater benefit when few early childhood education alternatives areavailable (call it ldquocontrol version Ardquo) than when there is an abundance of alternatives includ-ing state-sponsored preschool programs (call it ldquocontrol version Brdquo)

Second the effect of an intervention will likely vary among individuals or across socialsettings A famous example comes from medical research the well-publicized cardiovascularbenefits of initiating estrogen therapy during menopause were contradicted later by experi-mental findings that the same therapy increased postmenopausal womenrsquos risk for heartattacks The effect of an intervention may also depend on the provision of some other con-current or subsequent interventions Such heterogeneous effects are often characterized asmoderated effects in the literature

Third alternative theories may provide competing explanations for the causal mechan-isms that is the processes through which the intervention produces its effect A theoreticalconstruct characterizing the hypothesized intermediate process is called a mediator of theintervention effect The fictitious farmer never developed an elaborate theory as to whatcaused some seedlings to surpass others in growth Once scientists revealed the causal rela-tionship between access to chemical nutrients in soil and plant growth wide applications ofchemically synthesized fertilizers finally led to a major increase in crop production

Finally it is well known in agricultural experiments that a new type of fertilizer applied toone plot may spill-over to the next plot Because social connections among individuals areprevalent within organizations or through networks an individualrsquos response to the treatmentmay similarly depend on the treatment for other individuals in the same social setting whichmay lead to possible spill-overs of intervention effects among individual human beings

Answering questions with regard to moderation mediation and spill-over poses majorconceptual and analytic challenges To date psychological research often presents well-articulated theories of causal mechanisms relating stimuli to responses Yet researchers often

4 CAUSALITY IN A SOCIAL WORLD

lack rigorous analytic strategies for empirically screening competing theories explaining theobserved effect Sociologists have keen interest in the spill-over of treatment effects transmit-ted through social interactions yet have produced limited evidence quantifying such effectsAs many have pointed out in general terminological ambiguity and conceptual confusionhave been prevalent in the published applied research (Holmbeck 1997 Kraemer et al2002 2008)

A new framework for conceptualizing moderated mediated and spill-over effects hasemerged relatively recently in the statistics and econometrics literature on causal inference(eg Abbring and Heckman 2007 Frangakis and Rubin 2002 Heckman and Vytlacil2007a b Holland 1988 Hong and Raudenbush 2006 Hudgens and Halloran 2008Jo 2008 Pearl 2001 Robins and Greenland 1992 Sobel 2008) The potential for furtherconceptual and methodological development and for broad applications in the field of behav-ioral and social sciences promises to greatly advance the empirical basis of our knowledgeabout causality in the social world

This book clarifies theoretical concepts and introduces innovative statistical strategies forinvestigating the average effects of multivalued treatments moderated treatment effectsmediated treatment effects and spill-over effects in experimental or quasiexperimental dataDefining individual-specific and population average treatment effects in terms of potentialoutcomes the book relates the mathematical forms to the substantive meanings of moderatedmediated and spill-over effects in the context of application examples It also explicates andevaluates identification assumptions and contrasts innovative statistical strategies with con-ventional analytic methods

111 Moderated treatment effects

It is hard to accept the assumption that a treatment would produce the same impact for everyindividual in every possible circumstance Understanding the heterogeneity of treatmenteffects therefore is key to the development of causal theories For example some studiesreported that estrogen therapy improved cardiovascular health among women who initiatedits use during menopause According to a series of other studies however the use of estrogentherapy increased postmenopausal womenrsquos risk for heart attacks (Grodstein et al 19962000 Writing Group for the Womenrsquos Health Initiative Investigators 2002) The sharp con-trast of findings from these studies led to the hypothesis that age of initiation moderates theeffect of estrogen therapy on womenrsquos health (Manson and Bassuk 2007 Rossouw et al2007) Revelations of the moderated causal relationship greatly enrich theoretical understand-ing and in this case directly inform clinical practice

In another example dividing students by ability into small groups for reading instructionhas been controversial for decades Many believed that the practice benefits high-ability stu-dents at the expense of their low-ability peers and exacerbates educational inequality (Grantand Rothenberg 1986 Rowan and Miracle 1983 Trimble and Sinclair 1987) Recentevidence has shown that first of all the effect of ability grouping depends on a number ofconditions including whether enough time is allocated to reading instruction and how wellthe class can be managed Hence instructional time and class management are among theadditional moderators to consider for determining the merits of ability grouping (Hong andHong 2009 Hong et al 2012a b) Moreover further investigations have generated evidencethat contradicts the long-held belief that grouping undermines the interests of low-abilitystudents Quite the contrary researchers have found that grouping is beneficial for students

5INTRODUCTION

with low or medium prior abilitymdashif adequate time is provided for instruction and if the classis well managed On the other hand ability grouping appears to have a minimal impact onhigh-ability studentsrsquo literacy These results were derived from kindergarten data and maynot hold for math instruction and for higher grade levels Replications with different subpo-pulations and in different contexts enable researchers to assess the generalizability of a theory

In a third example one may hypothesize that a student is unlikely to learn algebra well inninth grade without a solid foundation in eighth-grade prealgebra One may also argue that theprogress a student made in learning eighth-grade prealgebra may not be sustained without thefurther enhancement of taking algebra in ninth grade In other words the earlier treatment(prealgebra in eighth grade) and the later treatment (algebra in ninth grade) may reinforceone other each moderates the effect of the other treatment on the final outcome (math achieve-ment at the end of ninth grade) As Hong and Raudenbush (2008) argued the cumulativeeffect of two treatments in a well-aligned sequence may exceed the sum of the benefits oftwo single-year treatments Similarly experiencing two consecutive years of inferior treat-ments such as encountering an incompetent math teacher who dampens a studentrsquos self-efficacy in math learning in eighth grade and then again in ninth grade could do more damagethan the sum of the effect of encountering an incompetent teacher only in eighth grade and theeffect of having such a teacher only in ninth grade

There has been a great amount of conceptual confusion regarding how a moderatorrelates to the treatment and the outcome On one hand many researchers have used the termsldquomoderatorsrdquo and ldquomediatorsrdquo interchangeably without understanding the crucial distinctionbetween the two An overcorrective attempt on the other hand has led to the arbitrary rec-ommendation that a moderator must occur prior to the treatment and be minimally associatedwith the treatment (James and Brett 1984 Kraemer et al 2001 2008) In Chapter 6 weclarify that in essence the causal effect of the treatment on the outcome may depend onthe moderator value A moderator can be a subpopulation identifier a contextual character-istic a concurrent treatment or a preceding or succeeding treatment In the earlier examplesage of estrogen therapy initiation and a studentrsquos prior ability are subpopulation identifiers themanageability of a class which reflects both teacher skills and peer behaviors characterizes thecontext literacy instruction time is a concurrent treatment while prealgebra is a precedingtreatment for algebra A moderator does not have to occur prior to the treatment and doesnot have to be independent of the treatment

Once a moderated causal relationship has been defined in terms of potential outcomes theresearcher then chooses an appropriate experimental design for testing the moderation theoryThe assumptions required for identifying the moderated causal relationships differ across dif-ferent designs and have implications for analyzing experimental and quasiexperimental dataRandomized block designs are suitable for examining individual or contextual characteristicsas potential moderators factorial designs enable one to determine the joint effects of two ormore concurrent treatments and sequential randomized designs are ideal for assessing thecumulative effects of consecutive treatments Multisite randomized trials constitute anadditional type of designs in which the experimental sites are often deliberately sampled torepresent a population of geographical locations or social organizations Site membershipscan be viewed as implicit moderators that summarize a host of features of a local environmentReplications of a treatment over multiple sites allow one to quantify the heterogeneity of treat-ment effects across the sites Chapters 7 and 8 are focused on statistical methods for moder-ation analyses with quasiexperimental data In particular these chapters demonstrate througha number of application examples how a nonparametric marginal mean weighting through

6 CAUSALITY IN A SOCIAL WORLD

stratification (MMWS) method can overcome some important limitations of other existingmethods

Moderation questions are not restricted to treatmentndashoutcome relationships Rather theyare prevalent in investigations of mediation and spill-over This is because heterogeneity oftreatment effects may be explained by the variation in causal mediation mechanisms acrosssubpopulations and contexts It is also because the treatment effect for a focal individualmay depend on whether there is spill-over from other individuals through social interactionsYet similar to the confusion around the concept of moderation among applied researchersthere has been a great amount of misunderstanding with regard to mediation

112 Mediated treatment effects

Questions about mediation are at the core of nearly every scientific theory In its simplestform a theory explains why treatment Z causes outcome Y by hypothesizing an intermediateprocess involving at least one mediatorM that could have been changed by the treatment andcould subsequently have an impact on the outcome A causal mediation analysis then decom-poses the total effect of the treatment on the outcome into two parts the effect transmittedthrough the hypothesized mediator called the indirect effect and the difference betweenthe total effect and the indirect effect called the direct effect The latter represents the treat-ment effect channeled through other unspecified pathways (Alwin and Hauser 1975 Baronand Kenny 1986 Duncan 1966 Shadish and Sweeney 1991) Most researchers in socialsciences have followed the convention of illustrating the direct effect and the indirect effectwith path diagrams and then specifying linear regression models postulated to represent thestructural relationships between the treatment the mediator and the outcome

In Chapter 9 we point out that the approach to defining the indirect effect and the directeffect in terms of path coefficients is often misled by oversimplistic assumptions about thestructural relationships In particular this approach typically overlooks the fact that a treat-ment may generate an impact on the outcome not only by changing the mediator value butalso by changing the mediatorndashoutcome relationship (Judd and Kenny 1981) An examplein Chapter 9 illustrates such a case An experiment randomizes students to either an experi-mental condition which provides them with study materials and encourages them to study fora test or to a control condition which provides neither study materials nor encouragement(Holland 1988 Powers and Swinton 1984) One might hypothesize that encouraging exper-imental group members to study will increase the time they spend studying a focal mediatorin this example which in turn will increase their average test scores One might furtherhypothesize that even without a change in study time providing study materials to experi-mental group members will enable them to study more effectively than they otherwise wouldConsequently the effect of time spent studying might be greater under the experimentalcondition than under the control condition Omitting the treatment-by-mediator interactioneffect will lead to bias in the estimation of the indirect effect and the direct effect

Chapter 11 describes in great detail an evaluation study contrasting a new welfare-to-workprogram emphasizing active participation in the labor force with a traditional program thatguaranteed cash assistance without a mandate to seek employment (Hong Deutsch and Hill2011) Focusing on the psychological well-being of welfare applicants who were singlemothers with preschool-aged children the researchers hypothesized that the treatment wouldincrease employment rate among the welfare recipients and that the treatment-inducedincrease in employment would likely reduce depressive symptoms under the new program

7INTRODUCTION

They also hypothesized that in contrast the same increase in employment was unlikely tohave a psychological impact under the traditional program Hence the treatment-by-mediatorinteraction effect on the outcome was an essential component of the intermediate process

We will show that by defining the indirect effect and the direct effect in terms of potentialoutcomes one can avoid invoking unwarranted assumptions about the unknown structuralrelationships (Holland 1988 Pearl 2001 Robins and Greenland 1992) The potential out-comes framework has further clarified that in a causal mediation theory the treatment and themediator are both conceivably manipulable In the aforementioned example a welfare appli-cant could be assigned at random to either the new program or the traditional one Once havingbeen assigned to one of the programs the individual might or might not be employed due tovarious structural constraints market fluctuations or other random events typically beyondher control

Because the indirect effect and the direct effect are defined in terms of potential outcomesrather than on the basis of any specific regression models it becomes possible to distinguishthe definitions of these causal effects from their identification and estimation The identifica-tion step relates a causal parameter to observable population data For example for individualsassigned at random to an experimental condition their average counterfactual outcome asso-ciated with the control condition is expected to be equal to the average observable outcome ofthose assigned to the control condition Therefore the population average causal effect can beeasily identified in a randomized experiment The estimation step then relates the sample datato the observable population quantities involved in identification while taking into account thedegree of sampling variability

A major challenge in identification is that the mediatorndashoutcome relationship tends tobe confounded by selection even if the treatment is randomized Chapter 9 reviews variousexperimental designs that have been proposed by past researchers for studying causalmediation mechanisms Chapter 10 compares the identification assumptions and the analyticprocedures across a wide range of analytic methods These discussions are followed by anintroduction of the ratio-of-mediator-probability weighting (RMPW) strategy in Chapter 11In comparison with most existing strategies for causal mediation analysis RMPW relies onrelatively fewer identification assumptions and model-based assumptions Chapters 12 and13 will show that this new strategy can be applied broadly with extensions to multilevelexperimental designs and to studies of complex mediation mechanisms involving multiplemediators

113 Spill-over effects of a treatment

It is well known in vaccination research that an unvaccinated person can benefit when mostother people in the local community are vaccinated This is viewed as a spill-over of the treat-ment effect from the vaccinated people to an unvaccinated person Similarly due to socialinteractions among individuals or groups of individuals an intervention received by somemay generate a spill-over impact on others affiliated with the same organization or connectedthrough the same network For example effective policing in one neighborhood may driveoffenders to operate in other neighborhoods In evaluating an innovative community policingprogram researchers found that a neighborhood not assigned to community policing tended tosuffer if the surrounding neighborhoods were assigned to community policing This is aninstance in which a well-intended treatment generates an unintended negative spill-over effectHowever being assigned to community policy was found particularly beneficial when thesurrounding neighborhoods also received the intervention (Verbitsky-Savitz and Raudenbush

8 CAUSALITY IN A SOCIAL WORLD

2012) In another example the impact of a school-based mentoring program targeted atstudents displaying delinquent behaviors may be enhanced for a focal student if his or herat-risk peers are assigned to the mentoring program at the same time

The previous examples challenge the ldquostable unit treatment value assumptionrdquo (SUTVA)that has been invoked in most causal inference studies This assumption states that an indi-vidualrsquos potential response to a treatment depends neither on how the treatment is assignednor on the treatment assignments of other individuals (Rubin 1980 1986) Rubin and othersbelieve that without resorting to SUTVA the causal effect of a treatment becomes hard todefine and that causal inference may become intractable However as Sobel (2006) has illus-trated with the Moving to Opportunity (MTO) experiment that offered housing vouchersenabling low-income families to move to low-poverty neighborhoods there are possible con-sequences of violating SUTVA in estimating the treatment effects on outcomes such as safetyand self-sufficiency Because social interactions among individuals may affect whether onevolunteers to participate in the study whether one moves to a low-poverty neighborhood afterreceiving the housing voucher as well as housing project residentsrsquo subjective perceptions oftheir neighborhoods the program may have a nonzero impact on the potential outcome of theuntreated As a result despite the randomization of treatment assignment the mean differencein the observed outcome between the treated units and the untreated units may be biased forthe average treatment effect Rather the observable quantity is the difference between theeffect of treating some rather than treating none for the treated and the effect of treating somerather than treating none for the untreated the latter being the pure spill-over effect for theuntreated

In making attempts to relax SUTVA researchers have proposed several alternative frame-works that incorporate possible spill-over effects (see Hong and Raudenbush 2013 for areview) Hong (2004) presented a model that involves treatments and treatment settingsA treatment setting for an individual is a local environment constituted by a set of agentsand participants along with their treatment assignments An individualrsquos potential outcomevalue under a given treatment is assumed stable when the treatment setting is fixed the poten-tial outcome may take different values when the treatment setting shifts One may investigatewhether the treatment effect depends on the treatment setting Applying this framework Hongand Raudenbush (2006) examined the effect on a childrsquos academic growth of retaining thechild in kindergarten rather than promoting the child to first grade when a relatively smallproportion of low-achieving peers in the same school are retained as opposed to when a rel-atively large proportion of the peers are retained Hudgens and Halloran (2008) presented arelated framework in which the effect on an individual of the treatment received by this indi-vidual is distinguished from the effect on the individual of the treatment received by others inthe same local community These effects can be identified if communities are assigned atrandom to different treatment assignment strategies (such as retaining a large proportion oflow-achieving students as opposed to retaining a small proportion of such students) and sub-sequently individuals within a community are randomized for treatment Chapter 14 reviewsthese frameworks and discusses identification and estimation strategies

Social contagion may also serve as an important channel through which the effect of anintervention is transmitted For example a school-wide intervention may reduce aggressivebehaviors and thereby improve studentsrsquo psychological well-being by improving the qualityof interpersonal relationships in other classes as well as in onersquos own class This is becausechildren interact not simply with their classmates but also with those from other classes in thehallways or on the playground (VanderWeele et al 2013) In this case the spill-over betweenclasses becomes a part of the mediation mechanism In a study of student mentoring whether

9INTRODUCTION

an individual actually participated in the mentoring program despite the initial treatmentassignment is typically viewed as a primary mediator The proportion of onersquos peers that par-ticipated in the program may act as a second mediator A treatment may also exert its impactpartly through regrouping individuals and thereby changing peer composition (Hong andNomi 2012) Chapter 15 discusses analytic strategies for detecting spill-over as a part ofthe causal mediation mechanism

12 Weighting methods for causal inference

In most behavioral and social science applications major methodological challenges arise dueto the selection of treatments in a quasiexperimental study and the selection of mediator valuesin experimental and quasiexperimental studies Statistical methods most familiar to research-ers in social sciences are often inadequate for causal inferences with regard to multivaluedtreatments moderation mediation and spill-over The book offers a major revision to under-standing of causality in the social world by introducing two complementary weighting stra-tegies both featuring nonparametric approaches to estimating these causal effects The newweighting methods greatly simplify model specifications while enhancing the robustness ofresults by minimizing the reliance on some key assumptions

The propensity score-basedMMWSmethod removes selection bias associated with a largenumber of covariates by equating the pretreatment composition between treatment groups(Hong 2010a 2012 Huang et al 2005) Unlike propensity score matching and stratificationthat are mostly restricted to evaluations of binary treatments the MMWS method is flexiblefor evaluating binary and multivalued treatments by approximating a completely randomizedexperiment In evaluating whether the treatment effects differ across subpopulations definedby individual characteristics or treatment settings researchers may assign weights within eachsubpopulation in order to approximate a randomized block design To investigate whetherone treatment moderates the effect of another concurrent treatment researchers may assignweights to the data to approximate a factorial randomized design The method can also beused to assess whether the effect of an initial treatment is amplified or weakened by a subse-quent treatment or to identify an optimal treatment sequence through approximating a sequen-tial randomized experiment Even though such analyses can similarly be conducted throughinverse-probability-of-treatment weighting (IPTW) that has been increasingly employed inepidemiological research (Hernaacuten Brumback and Robins 2000 Robins Hernaacuten andBrumback 2000) IPTW is known for bias and imprecision in estimation especially whenthe propensity score models are misspecified in their functional forms (Hong 2010a Kangand Schafer 2007 Schafer and Kang 2008 Waernbaum 2012) In contrast the nonparamet-ric MMWS method displays a relatively high level of robustness despite such misspecifica-tions and also gains efficiency as indicated by simulation results (Hong 2010a)

To study causal mediation mechanisms the RMPW method decomposes the total effectof a treatment into an ldquoindirect effectrdquo transmitted through a specific mediator and a ldquodirecteffectrdquo representing unspecified mechanisms In contrast with most existing methods formediation analysis the RMPW-adjusted outcome model is extremely simple and is nonpara-metric in nature It generates estimates of the causal effects along with their sampling errorswhile adjusting for pretreatment covariates that confound the mediatorndashoutcome relationshipsthrough weighting The method applies regardless of the distribution of the outcome the dis-tribution of the mediator or the functional relationship between the outcome and the mediator

10 CAUSALITY IN A SOCIAL WORLD

Page 2: Thumbnail - download.e-bookshelf.de...4.2.4 IPTW for bias removal 84 4.3 MMWS 86 4.3.1 Theoretical rationale 86 ... 8.3.4.2 Growth model in the absence of confounding 200 8.3.4.3 Weighted

Causality in a Social World

Causality in a Social World

Moderation Meditation and Spill-over

Guanglei Hong

Department of Comparative Human DevelopmentUniversity of Chicago USA

This edition first published 2015copy 2015 John Wiley amp Sons Ltd

Registered OfficeJohn Wiley amp Sons Ltd The Atrium Southern Gate Chichester West Sussex PO19 8SQ United Kingdom

For details of our global editorial offices for customer services and for information about how to apply for permissionto reuse the copyright material in this book please see our website at wwwwileycom

The right of the author to be identified as the author of this work has been asserted in accordance with the CopyrightDesigns and Patents Act 1988

All rights reserved No part of this publication may be reproduced stored in a retrieval system or transmitted in anyform or by any means electronic mechanical photocopying recording or otherwise except as permitted by theUK Copyright Designs and Patents Act 1988 without the prior permission of the publisher

Wiley also publishes its books in a variety of electronic formats Some content that appears in print may not beavailable in electronic books

Designations used by companies to distinguish their products are often claimed as trademarks All brand names andproduct names used in this book are trade names service marks trademarks or registered trademarks of their respectiveowners The publisher is not associated with any product or vendor mentioned in this book

Limit of LiabilityDisclaimer of Warranty While the publisher and author have used their best efforts in preparing this bookthey make no representations or warranties with respect to the accuracy or completeness of the contents of this bookand specifically disclaim any implied warranties of merchantability or fitness for a particular purpose It is sold on theunderstanding that the publisher is not engaged in rendering professional services and neither the publisher nor the authorshall be liable for damages arising herefrom If professional advice or other expert assistance is required the services ofa competent professional should be sought

Library of Congress Cataloging-in-Publication Data applied for

ISBN 9781118332566

A catalogue record for this book is available from the British Library

Set in 1012pt Times by SPi Global Pondicherry India

1 2015

Contents

Preface xv

Part I Overview 1

1 Introduction 311 Concepts of moderation mediation and spill-over 3

111 Moderated treatment effects 5112 Mediated treatment effects 7113 Spill-over effects of a treatment 8

12 Weighting methods for causal inference 1013 Objectives and organization of the book 1114 How is this book situated among other publications on related topics 12

2 Review of causal inference concepts and methods 1821 Causal inference theory 18

211 Attributes versus causes 18212 Potential outcomes and individual-specific causal effects 19213 Inference about population average causal effects 22

2131 Prima facie effect 242132 Ignorability assumption 25

22 Applications to Lordrsquos paradox and Simpsonrsquos paradox 27221 Lordrsquos paradox 27222 Simpsonrsquos paradox 31

23 Identification and estimation 34231 Selection bias 35232 Sampling bias 35233 Estimation efficiency 36

Appendix 21 Potential bias in a prima facie effect 36Appendix 22 Application of the causal inference theory to Lordrsquos paradox 37

3 Review of causal inference designs and analytic methods 4031 Experimental designs 40

311 Completely randomized designs 40

312 Randomized block designs 41313 Covariance adjustment for improving efficiency 43314 Multilevel experimental designs 43

32 Quasiexperimental designs 44321 Nonequivalent comparison group designs 44322 Other quasiexperimental designs 45

33 Statistical adjustment methods 46331 ANCOVA and multiple regression 46

3311 ANCOVA for removing selection bias 463312 Potential pitfalls of ANCOVA with a vast between-group

difference 473313 Bias due to model misspecification 48

332 Matching and stratification 50333 Other statistical adjustment methods 51

3331 The IV method 513332 DID analysis 54

34 Propensity score 55341 What is a propensity score 56342 Balancing property of the propensity score 57343 Pooling conditional treatment effect estimate Matching

stratification and covariance adjustment 603431 Propensity score matching 613432 Propensity score stratification 623433 Covariance adjustment for the propensity score 663434 Sensitivity analysis 66

Appendix 3A Potential bias due to the omission of treatment-by-covariateinteraction 70

Appendix 3B Variable selection for the propensity score model 71

4 Adjustment for selection bias through weighting 7641 Weighted estimation of population parameters in survey sampling 77

411 Simple random sample 77412 Proportionate sample 78413 Disproportionate sample 79

42 Weighting adjustment for selection bias in causal inference 80421 Experimental result 81422 Quasiexperimental result 81423 Sample weight for bias removal 82424 IPTW for bias removal 84

43 MMWS 86431 Theoretical rationale 86

4311 MMWS for a discrete propensity score 874312 MMWS for a continuous propensity score 884313 MMWS for estimating the treatment effect on the treated 89

432 MMWS analytic procedure 91433 Inherent connection and major distinctions between MMWS

and IPTW 93

vi CONTENTS

Appendix 4A Proof of MMWS-adjusted mean observed outcome being unbiasedfor the population average potential outcome 95

Appendix 4B Derivation of MMWS for estimating the treatment effecton the treated 96

Appendix 4C Theoretical equivalence of MMWS and IPTW 97Appendix 4D Simulations comparing MMWS and IPTW under misspecifications

of the functional form of a propensity score model 97

5 Evaluations of multivalued treatments 10051 Defining the causal effects of multivalued treatments 10052 Existing designs and analytic methods for evaluating multivalued

treatments 102521 Experimental designs and analysis 102

5211 Randomized experiments with multipletreatment arms 102

5212 Identification under ignorability 1025213 ANOVA 103

522 Quasiexperimental designs and analysis 1055221 ANCOVA and multiple regression 1055222 Propensity score-based adjustment 1085223 Other adjustment methods 112

53 MMWS for evaluating multivalued treatments 112531 Basic rationale 113532 Analytic procedure 114

5321 MMWS for a multinomial treatment measure 1155322 MMWS for an ordinal treatment measure 120

533 Identification assumptions 12154 Summary 123Appendix 5A Multiple IV for evaluating multivalued treatments 124

Part II Moderation 127

6 Moderated treatment effects concepts and existing analytic methods 12961 What is moderation 129

611 Past discussions of moderation 1306111 Purpose of moderation research 1306112 What qualifies a variable as a moderator 132

612 Definition of moderated treatment effects 1336121 Treatment effects moderated by individual or contextual

characteristics 1336122 Joint effects of concurrent treatments 134

62 Experimental designs and analytic methods for investigatingexplicit moderators 136621 Randomized block designs 137

6211 Identification assumptions 1376212 Two-way ANOVA 1386213 Multiple regression 139

viiCONTENTS

622 Factorial designs 1406221 Identification assumptions 1406222 Analytic strategies 142

63 Existing research designs and analytic methods for investigatingimplicit moderators 142631 Multisite randomized trials 143

6311 Causal parameters and identification assumptions 1446312 Analytic strategies 145

632 Principal stratification 149Appendix 6A Derivation of bias in the fixed-effects estimator when the

treatment effect is heterogeneous in multisite randomized trials 151Appendix 6B Derivation of bias in the mixed-effects estimator when the

probability of treatment assignment varies across sites 153Appendix 6C Derivation and proof of the population weight applied to

mixed-effects models for eliminating bias in multisiterandomized trials 153

7 Marginal mean weighting through stratification for investigatingmoderated treatment effects 15971 Existing methods for moderation analyses with quasiexperimental data 159

711 Analysis of covariance and regression-based adjustment 1617111 Treatment effects moderated by subpopulation membership 1617112 Treatment effects moderated by a concurrent treatment 164

712 Propensity score-based adjustment 1657121 Propensity score matching and stratification 1667122 Inverse-probability-of-treatment weighting 167

72 MMWS estimation of treatment effects moderated by individual orcontextual characteristics 168721 Application example 170722 Analytic procedure 170

73 MMWS estimation of the joint effects of concurrent treatments 174731 Application example 174732 Analytic procedure 175733 Joint treatment effects moderated by individual or contextual

characteristics 179

8 Cumulative effects of time-varying treatments 18581 Causal effects of treatment sequences 186

811 Application example 186812 Causal parameters 187

8121 Time-varying treatments 1878122 Time-varying potential outcomes 1878123 Causal effects of 2-year treatment sequences 1878124 Causal effects of multiyear treatment sequences 190

82 Existing strategies for evaluating time-varying treatments 190821 The endogeneity problem in nonexperimental data 190822 SEM 191

viii CONTENTS

823 Fixed-effects econometric models 192824 Sequential randomization 192825 Dynamic treatment regimes 193826 Marginal structural models and structural nested models 194

83 MMWS for evaluating 2-year treatment sequences 195831 Sequential ignorability 195832 Propensity scores 196833 MMWS computation 197

8331 MMWS adjustment for year 1 treatment selection 1978332 MMWS adjustment for two-year treatment

sequence selection 198834 Two-year growth model specification 199

8341 Growth model in the absence of treatment 2008342 Growth model in the absence of confounding 2008343 Weighted 2-year growth model 202

84 MMWS for evaluating multiyear sequences of multivalued treatments 204841 Sequential ignorability of multiyear treatment sequences 204842 Propensity scores for multiyear treatment sequences 204843 MMWS computation 205844 Weighted multiyear growth model 205845 Issues of sample size 206

85 Conclusion 207Appendix 8A A saturated model for evaluating multivalued treatments

over multiple time periods 207

Part III Mediation 211

9 Concepts of mediated treatment effects and experimental designsfor investigating causal mechanisms 21391 Introduction 21492 Path coefficients 21593 Potential outcomes and potential mediators 216

931 Controlled direct effects 217932 Controlled treatment-by-mediator interaction effect 217

94 Causal effects with counterfactual mediators 219941 Natural direct effect 219942 Natural indirect effect 220943 Natural treatment-by-mediator interaction effect 220944 Unstable unit treatment value 221

95 Population causal parameters 222951 Population average natural direct effect 224952 Population average natural indirect effect 225

96 Experimental designs for studying causal mediation 225961 Sequentially randomized designs 228962 Two-phase experimental designs 228963 Three- and four-treatment arm designs 230964 Experimental causal-chain designs 231

ixCONTENTS

965 Moderation-of-process designs 231966 Augmented encouragement designs 232967 Parallel experimental designs and parallel encouragement designs 232968 Crossover experimental designs and crossover encouragement

designs 233969 Summary 234

10 Existing analytic methods for investigating causal mediation mechanisms 238101 Path analysis and SEM 239

1011 Analytic procedure for continuous outcomes 2391012 Identification assumptions 2421013 Analytic procedure for discrete outcomes 245

102 Modified regression approach 2461021 Analytic procedure for continuous outcomes 2461022 Identification assumptions 2471023 Analytic procedure for binary outcomes 248

103 Marginal structural models 2501031 Analytic procedure 2501032 Identification assumptions 252

104 Conditional structural models 2521041 Analytic procedure 2521042 Identification assumptions 253

105 Alternative weighting methods 2541051 Analytic procedure 2541052 Identification assumptions 256

106 Resampling approach 2561061 Analytic procedure 2561062 Identification assumptions 257

107 IV method 2571071 Rationale and analytic procedure 2571072 Identification assumptions 258

108 Principal stratification 2591081 Rationale and analytic procedure 2591082 Identification assumptions 260

109 Sensitivity analysis 2611091 Unadjusted confounding as a product of hypothetical

regression coefficients 2611092 Unadjusted confounding reflected in a hypothetical

correlation coefficient 2621093 Limitations when the selection mechanism differs

by treatment 2641094 Other sensitivity analyses 265

1010 Conclusion 26510101 The essentiality of sequential ignorability 26510102 Treatment-by-mediator interactions 26610103 Homogeneous versus heterogeneous causal effects 26610104 Model-based assumptions 266

x CONTENTS

Appendix 10A Bias in path analysis estimation due to the omission oftreatment-by-mediator interaction 267

11 Investigations of a simple mediation mechanism 273111 Application example national evaluation of welfare-to-work strategies 274

1111 Historical context 2741112 Research questions 2751113 Causal parameters 2751114 NEWWS Riverside data 277

112 RMPW rationale 2771121 RMPW in a sequentially randomized design 278

11211 E[Y(0 M(0))] 27911212 E[Y(1 M(1))] 28011213 E[Y(1 M(0))] 28011214 E[Y(0 M(1))] 28111215 Nonparametric outcome model 282

1122 RMPW in a sequentially randomized block design 2831123 RMPW in a standard randomized experiment 2851124 Identification assumptions 286

113 Parametric RMPW procedure 287114 Nonparametric RMPW procedure 290115 Simulation results 292

1151 Correctly specified propensity score models 2921152 Misspecified propensity score models 2941153 Comparisons with path analysis and IV results 294

116 Discussion 2951161 Advantages of the RMPW strategy 2951162 Limitations of the RMPW strategy 295

Appendix 11A Causal effect estimation through the RMPW procedure 296Appendix 11B Proof of the consistency of RMPW estimation 297

12 RMPW extensions to alternative designs and measurement 301121 RMPW extensions to mediators and outcomes of alternative distributions 301

1211 Extensions to a multicategory mediator 30212111 Parametric RMPW procedure 30212112 Nonparametric RMPW procedure 304

1212 Extensions to a continuous mediator 3041213 Extensions to a binary outcome 306

122 RMPW extensions to alternative research designs 3061221 Extensions to quasiexperimental data 3071222 Extensions to data from cluster randomized trials 308

12221 Application example and research questions 30912222 Estimation of the total effect 31012223 RMPW analysis of causal mediation mechanisms 31012224 Identification assumptions 31212225 Contrast with multilevel path analysis and SEM 31212226 Contrast with multilevel prediction models 313

xiCONTENTS

1223 Extensions to data from multisite randomized trials 31312231 Research questions and causal parameters 31412232 Estimation of the total effect and its between-site variation 31512233 RMPW analysis of causal mediation mechanisms 31612234 Identification assumptions 31912235 Contrast with multilevel path analysis and SEM 319

123 Alternative decomposition of the treatment effect 321

13 RMPW extensions to studies of complex mediation mechanisms 325131 RMPW extensions to moderated mediation 325

1311 RMPW analytic procedure for estimating and testingmoderated mediation 326

1312 Path analysisSEM approach to analyzing moderatedmediation 327

1313 Principal stratification and moderated mediation 328132 RMPW extensions to concurrent mediators 328

1321 Treatment effect decomposition 32913211 Treatment effect decomposition without

between-mediator interaction 32913212 Treatment effect decomposition with

between-mediator interaction 3311322 Identification assumptions 3331323 RMPW procedure 333

13231 Estimating E[Y(1M1(1)M2(0))] 33513232 Estimating E[Y(1M1(0)M2(0))] 33513233 Causal effect estimation with noninteracting

concurrent mediators 33613234 Estimating E[Y(1M1(0)M2(1))] 33713235 Causal effect estimation with interacting concurrent

mediators 3371324 Contrast with the linear SEM approach 3381325 Contrast with the multivariate IV approach 339

133 RMPW extensions to consecutive mediators 3401331 Treatment effect decomposition 341

13311 Natural direct effect of the treatment on the outcome 34213312 Natural indirect effect mediated by M1 only 34213313 Natural indirect effect mediated by M2 only 34213314 Natural indirect effect mediated by an M1-by-M2

interaction 34313315 Treatment-by-mediator interactions 343

1332 Identification assumptions 3451333 RMPW procedure 347

13331 Estimating E[Y(1M1(0)M2(0M1(0)))] 34713332 Estimating E[Y(1M1(1)M2(0M1(0)))] 34913333 Estimating E[Y(1M1(0)M2(1M1(1)))] 34913334 Estimating E[Y(0M1(1)M2(0M1(0)))] and

E[Y(0M1(0)M2(1M1(1)))] 351

xii CONTENTS

1334 Contrast with the linear SEM approach 3531335 Contrast with the sensitivity-based estimation of bounds for

causal effects 354134 Discussion 355Appendix 13A Derivation of RMPW for estimating population average

counterfactual outcomes of two concurrentmediators 355

Appendix 13B Derivation of RMPW for estimating population averagecounterfactual outcomes of consecutivemediators 358

Part IV Spill-over 363

14 Spill-over of treatment effects concepts and methods 365141 Spill-over A nuisance a trifle or a focus 365142 Stable versus unstable potential outcome values An example

from agriculture 367143 Consequences for causal inference when spill-over is overlooked 369144 Modified framework of causal inference 371

1441 Treatment settings 3711442 Simplified characterization of treatment settings 3731443 Causal effects of individual treatment assignment and of peer

treatment assignment 375145 Identification Challenges and solutions 376

1451 Hypothetical experiments for identifying average treatmenteffects in the presence of social interactions 376

1452 Hypothetical experiments for identifying the impact ofsocial interactions 380

1453 Application to an evaluation of kindergarten retention 382146 Analytic strategies for experimental and quasiexperimental data 384

1461 Estimation with experimental data 3841462 Propensity score stratification 3851463 MMWS 386

147 Summary 387

15 Mediation through spill-over 391151 Definition of mediated effects through spill-over in a cluster

randomized trial 3931511 Notation 3931512 Treatment effect mediated by a focal individualrsquos

compliance 3941513 Treatment effect mediated by peersrsquo compliance through

spill-over 3941514 Decomposition of the total treatment effect 395

152 Identification and estimation of the spill-over effect in a clusterrandomized design 3951521 Identification in an ideal experiment 395

xiiiCONTENTS

1522 Identification when the mediators are not randomized 3981523 Estimation of mediated effects through spill-over 400

153 Definition of mediated effects through spill-over in a multisite trial 4021531 Notation 4021532 Treatment effect mediated by a focal individualrsquos compliance 4041533 Treatment effect mediated by peersrsquo compliance

through spill-over 4041534 Direct effect of individual treatment assignment on the outcome 4051535 Direct effect of peer treatment assignment on the outcome 4051536 Decomposition of the total treatment effect 405

154 Identification and estimation of spill-over effects in a multisite trial 4061541 Identification in an ideal experiment 4071542 Identification when the mediators are not randomized 4091543 Estimation of mediated effects through spill-over 410

15431 Estimating E[Y(1pM(p)Mminus(p))] andE[Y(000Mminus(0))] 410

15432 Estimating E[Y(1p0Mminus(p))] 41115433 Estimating E[Y(1p0Mminus(0))] 41215434 Estimating E[Y(0p0Mminus(0))] 412

155 Consequences of omitting spill-over effects in causal mediation analyses 4121551 Biased inference in a cluster randomized trial 4131552 Biased inference in a multisite randomized trial 4131553 Biased inference of the local average treatment effect 415

156 Quasiexperimental application 416157 Summary 419Appendix 151 Derivation of the weight for estimating the population

average counterfactual outcome E[Y(1 p 0Mminus( p))] 419Appendix 152 Derivation of bias in the ITT effect due to the omission of

spill-over effects 420

Index 423

xiv CONTENTS

Preface

A scientific mind in training seeks comprehensive descriptions of every interesting phenom-enon In such a mind data are to be pieced together for comparisons contrasts and eventuallyinferences that are required for understanding causes and effects that may have led to thephenomenon of interest Yet without disciplined reasoning a causal inference would slip intothe rut of a ldquocasual inferencerdquo as easily as making a typographical error

As a doctoral student thirsting for rigorous methodological training at the University ofMichigan I was very fortunate to be among the first group of students on the campus attendinga causal inference seminar in 2000 jointly taught by Yu Xie from the Department of Sociol-ogy Susan Murphy from the Department of Statistics and Stephen Raudenbush from theSchool of Education The course was unique in both content and pedagogy The instructorsput together a bibliography drawn from the past and current causal inference literature origi-nated from multiple disciplines In the spirit of ldquoreciprocal teachingrdquo the students were delib-erately organized into small groups each representing a mix of disciplinary backgrounds andtook turns to teach the weekly readings under the guidance of a faculty instructor This inval-uable experience revealed to me that the pursuit of causal inference is a multidisciplinaryendeavor In the rest of my doctoral study I continued to be immersed in an extraordinaryinterdisciplinary community in the form of the Quantitative Methodology Program (QMP)seminars led by the above three instructors who were then joined by Ben Hansen(Statistics) Richard Gonzalez (Psychology) Roderick Little (Biostatistics) Jake Bowers(Political Science) and many others I have tried whenever possible to carry the same spiritinto my own teaching of causal inference courses at the University of Toronto and now at theUniversity of Chicago In these courses communicating understandings of causal problemsacross disciplinary boundaries has been particularly challenging but also stimulating and grat-ifying for myself and for students from various departments and schools

Under the supervision of Stephen Raudenbush I took a first stab at the conceptual frame-work of causal inference in my doctoral dissertation by considering peer spill-over in schoolsettings From that point on I have organized my methodological research to tackle some ofthe major obstacles to causal inferences in policy and program evaluations My workaddresses issues including (i) how to conceptualize and evaluate the causal effects of educa-tional treatments when studentsrsquo responses to alternative treatments depend on various fea-tures of the organizational settings including peer composition (ii) how to adjust forselection bias in evaluating the effects of concurrent and consecutive multivalued treatmentsand (iii) how to conceptualize and analyze causal mediation mechanisms My methodological

research has been driven by the substantive interest in understanding the role of socialinstitutionsmdashschools in particularmdashin shaping human development especially during theages of fast growth (eg childhood adolescence and early adulthood) I have shared meth-odological advances with a broad audience in social sciences through applying the causalinference methods to prominent substantive issues such as grade retention within-class group-ing services for English language learners and welfare-to-work strategies These illustrationsare used extensively in this book

Among social scientists awareness of methodological challenges in causal investigationsis higher than ever before In response to this rising demand causal inference is becoming oneof the most productive scholarly fields in the past decade I have had the benefit of followingthe work and exchanging ideas with some fellow methodologists who are constantly contri-buting new thoughts and making breakthroughs These include Daniel Almirall HowardBloom Tom Cook Michael Elliot Ken Frank Ben Hansen James Heckman JenniferHill Martin Huber Kosuke Imai Booil Jo John R Lockwood David MacKinnon DanielMcCaffrey Luke Miratrix Richard Murnane Derek Neal Lindsey Page Judea PearlStephen Raudenbush Sean Reardon Michael Seltzer Youngyun Shin Betsy SinclairMichael Sobel Peter Steiner Elizabeth Stuart Eric Thetchen Tchetchen Tyler VanderWeeleand Kazuo Yamaguchi I am grateful to Larry Hedges who offered me the opportunity of guestediting the Journal of Research in Educational Effectiveness special issue on the statisticalapproaches to studying mediator effects in education research in 2012 Many of the colleagueswhom I mentioned above contributed to the open debate in the special issue by either propos-ing or critically examining a number of innovative methods for studying causal mechanismsI anticipate that many of them are or will soon be writing their own research monographs oncausal inference if they have not already done so Therefore readers of this book are stronglyrecommended to browse past and future titles by these and other authors for alternative per-spectives and approaches My own understanding of course will continue to evolve as well inthe coming years

One has to be ambitious to stay upfront in substantive areas as well as in methodology Thebest strategy apparently is to learn from substantive experts During different phases of mytraining and professional career I have had the opportunities to work with David K CohenBrian Rowan Deborah Ball Carl Corter Janette Pelletier Takako Nomi Esther Geva DavidFrancis Stephanie Jones Joshua Brown and Heather D Hill Many of my substantive insightswere derived from conversations with these mentors and colleagues I am especially gratefulto Bob Granger former president of the William T Grant Foundation who reassured me thatldquoBy staying a bit broad you will learn a lot and many fields will benefit from your workrdquo

Like many other single-authored books this one is built on the generous contributions ofstudents and colleagues who deserve special acknowledgement Excellent research assistancefrom Jonah Deutsch Joshua Gagne Rachel Garrett Yihua Hong Xu Qin Cheng Yang andBing Yu was instrumental in the development and implementation of the new methods pre-sented in this book Richard Congdon a renowned statistical software programmer broughtrevolutionary ideas to interface designs in software development with the aim of facilitatingusersrsquo decision-making in a series of causal analyses RichardMurnane Stephen Raudenbushand Michael Sobel carefully reviewed and critiqued multiple chapters of the manuscript draftAdditional comments and suggestions on earlier drafts came fromHoward Bloom Ken FrankBen Hansen Jennifer Hill Booil Jo Ben Kelcey David MacKinnon and Fan Yang TereseSchwartzman provided valuable assistance in manuscript preparation The writing of thisbook was supported by a Scholars Award from the William T Grant Foundation and an

xvi PREFACE

Institute of Education Sciences (IES) Statistical and Research Methodology in EducationGrant from the US Department of Education All the errors in the published form are ofcourse the sole responsibility of the author

Project Editors Richard Davis Prachi Sinha Sahay and Liz Wingett and Assistant EditorHeather Kay at Wiley and project Managers P Jayapriya and R Jayavel at SPi Global havebeen wonderful to work with throughout the manuscript preparation and final production ofthe book Debbie Jupe Commissioning Editor at Wiley was remarkably effective in initiatingthe contact with me and in organizing anonymous reviews of the book proposal and of themanuscript These constructive reviews have shaped the book in important ways I cannotagree more with one of the reviewers that ldquocausality cannot be established on a pure statisticalgroundrdquo The book therefore highlights the importance of substantive knowledge and researchdesigns and places a great emphasis on clarifying and evaluating assumptions required foridentifying causal effects in the context of each application Much caution is raised againstpossible misusage of statistical techniques for analyzing causal relationships especiallywhen data are inadequate Yet I maintain that improved statistical procedures along withimproved research designs would greatly enhance our ability in attempt to empiricallyexamine well-articulated causal theories

xviiPREFACE

Part I

OVERVIEW

1

Introduction

According to an ancient Chinese fable a farmer who was eager to help his crops grow wentinto his field and pulled each seedling upward After exhausting himself with the work heannounced to his family that they were going to have a good harvest only to find the nextmorning that the plants had wilted and died Readers with minimal agricultural knowledgemay immediately point out the following the farmerrsquos intervention theory was based on acorrect observation that crops that grow taller tend to produce more yield Yet his hypothesisreflects a false understanding of the cause and the effectmdashthat seedlings pulled to be tallerwould yield as much as seedlings thriving on their own

In their classic Design and Analysis of Experiments Hinkelmann and Kempthorne (1994updated version of Kempthorne 1952) discussed two types of science descriptive science andthe development of theory These two types of science are interrelated in the following senseobservations of an event and other related events often selected and classified for descriptionby scientists naturally lead to one or more explanations that we call ldquotheoretical hypothesesrdquowhich are then screened and falsified by means of further observations experimentation andanalyses (Popper 1963) The experiment of pulling seedlings to be taller was costly but didserve the purpose of advancing this farmerrsquos knowledge of ldquowhat does not workrdquo To developa successful intervention in this case would require a series of empirical tests of explicittheories identifying potential contributors to crop growth This iterative process graduallydeepens our knowledge of the relationships between supposed causes and effectsmdashthatis causalitymdashand may eventually increase the success of agricultural medical and socialinterventions

11 Concepts of moderation mediation and spill-over

Although the story of the ancient farmer is fictitious numerous examples can be found in thereal world in which well-intended interventions fail to produce the intended benefits or inmany cases even lead to unintended consequences ldquoInterventionsrdquo and ldquotreatmentsrdquo usedinterchangeably in this book broadly refer to actions taken by agents or circumstances expe-rienced by an individual or groups of individuals Interventions are regularly seen in educa-tion physical and mental health social services business politics and law enforcement In an

Causality in a Social World Moderation Meditation and Spill-over First Edition Guanglei Hongcopy 2015 John Wiley amp Sons Ltd Published 2015 by John Wiley amp Sons Ltd

education intervention for example teachers are typically the agents who deliver a treatmentto students while the impact of the treatment on student outcomes is of ultimate causalinterest Some educational practices such as ldquoteaching to the testrdquo have been criticized tobe nearly as counterproductive as the attempt of helping seedlings grow by pulling themupward ldquoInterventionsrdquo and ldquotreatmentsrdquo under consideration do not exclude undesiredexperiences such as exposure to poverty abuse crime or bereavement A treatment plannedor unplanned becomes a focus of research if there are theoretical reasons to anticipate itsimpact positive or negative on the well-being of individuals who are embedded in socialsettings including families classrooms schools neighborhoods and workplaces

In social science research in general and in policy and program evaluations in particularquestions concerning whether an intervention works and if so which version of the interven-tion works for whom under what conditions and why are key to the advancement of scien-tific and practical knowledge Although most empirical investigations in the social sciencesconcentrate on the average effect of a treatment for a specific population as opposed to theabsence of such a treatment (ie the control condition) in-depth theoretical reasoning withregard to how the causal effect is generated substantiated by compelling empirical evidenceis crucial for advancing scientific understanding

First when there are multiple versions or different dosages of the treatment or when thereare multiple versions of the control condition a binary divide between ldquothe treatmentrdquo andldquothe controlrdquo may not be as informative as fine-grained comparisons across for exampleldquotreatment version Ardquo ldquotreatment version Brdquo ldquocontrol version Ardquo and ldquocontrol versionBrdquo For example expanding the federally funded Head Start program to poor children isexpected to generate a greater benefit when few early childhood education alternatives areavailable (call it ldquocontrol version Ardquo) than when there is an abundance of alternatives includ-ing state-sponsored preschool programs (call it ldquocontrol version Brdquo)

Second the effect of an intervention will likely vary among individuals or across socialsettings A famous example comes from medical research the well-publicized cardiovascularbenefits of initiating estrogen therapy during menopause were contradicted later by experi-mental findings that the same therapy increased postmenopausal womenrsquos risk for heartattacks The effect of an intervention may also depend on the provision of some other con-current or subsequent interventions Such heterogeneous effects are often characterized asmoderated effects in the literature

Third alternative theories may provide competing explanations for the causal mechan-isms that is the processes through which the intervention produces its effect A theoreticalconstruct characterizing the hypothesized intermediate process is called a mediator of theintervention effect The fictitious farmer never developed an elaborate theory as to whatcaused some seedlings to surpass others in growth Once scientists revealed the causal rela-tionship between access to chemical nutrients in soil and plant growth wide applications ofchemically synthesized fertilizers finally led to a major increase in crop production

Finally it is well known in agricultural experiments that a new type of fertilizer applied toone plot may spill-over to the next plot Because social connections among individuals areprevalent within organizations or through networks an individualrsquos response to the treatmentmay similarly depend on the treatment for other individuals in the same social setting whichmay lead to possible spill-overs of intervention effects among individual human beings

Answering questions with regard to moderation mediation and spill-over poses majorconceptual and analytic challenges To date psychological research often presents well-articulated theories of causal mechanisms relating stimuli to responses Yet researchers often

4 CAUSALITY IN A SOCIAL WORLD

lack rigorous analytic strategies for empirically screening competing theories explaining theobserved effect Sociologists have keen interest in the spill-over of treatment effects transmit-ted through social interactions yet have produced limited evidence quantifying such effectsAs many have pointed out in general terminological ambiguity and conceptual confusionhave been prevalent in the published applied research (Holmbeck 1997 Kraemer et al2002 2008)

A new framework for conceptualizing moderated mediated and spill-over effects hasemerged relatively recently in the statistics and econometrics literature on causal inference(eg Abbring and Heckman 2007 Frangakis and Rubin 2002 Heckman and Vytlacil2007a b Holland 1988 Hong and Raudenbush 2006 Hudgens and Halloran 2008Jo 2008 Pearl 2001 Robins and Greenland 1992 Sobel 2008) The potential for furtherconceptual and methodological development and for broad applications in the field of behav-ioral and social sciences promises to greatly advance the empirical basis of our knowledgeabout causality in the social world

This book clarifies theoretical concepts and introduces innovative statistical strategies forinvestigating the average effects of multivalued treatments moderated treatment effectsmediated treatment effects and spill-over effects in experimental or quasiexperimental dataDefining individual-specific and population average treatment effects in terms of potentialoutcomes the book relates the mathematical forms to the substantive meanings of moderatedmediated and spill-over effects in the context of application examples It also explicates andevaluates identification assumptions and contrasts innovative statistical strategies with con-ventional analytic methods

111 Moderated treatment effects

It is hard to accept the assumption that a treatment would produce the same impact for everyindividual in every possible circumstance Understanding the heterogeneity of treatmenteffects therefore is key to the development of causal theories For example some studiesreported that estrogen therapy improved cardiovascular health among women who initiatedits use during menopause According to a series of other studies however the use of estrogentherapy increased postmenopausal womenrsquos risk for heart attacks (Grodstein et al 19962000 Writing Group for the Womenrsquos Health Initiative Investigators 2002) The sharp con-trast of findings from these studies led to the hypothesis that age of initiation moderates theeffect of estrogen therapy on womenrsquos health (Manson and Bassuk 2007 Rossouw et al2007) Revelations of the moderated causal relationship greatly enrich theoretical understand-ing and in this case directly inform clinical practice

In another example dividing students by ability into small groups for reading instructionhas been controversial for decades Many believed that the practice benefits high-ability stu-dents at the expense of their low-ability peers and exacerbates educational inequality (Grantand Rothenberg 1986 Rowan and Miracle 1983 Trimble and Sinclair 1987) Recentevidence has shown that first of all the effect of ability grouping depends on a number ofconditions including whether enough time is allocated to reading instruction and how wellthe class can be managed Hence instructional time and class management are among theadditional moderators to consider for determining the merits of ability grouping (Hong andHong 2009 Hong et al 2012a b) Moreover further investigations have generated evidencethat contradicts the long-held belief that grouping undermines the interests of low-abilitystudents Quite the contrary researchers have found that grouping is beneficial for students

5INTRODUCTION

with low or medium prior abilitymdashif adequate time is provided for instruction and if the classis well managed On the other hand ability grouping appears to have a minimal impact onhigh-ability studentsrsquo literacy These results were derived from kindergarten data and maynot hold for math instruction and for higher grade levels Replications with different subpo-pulations and in different contexts enable researchers to assess the generalizability of a theory

In a third example one may hypothesize that a student is unlikely to learn algebra well inninth grade without a solid foundation in eighth-grade prealgebra One may also argue that theprogress a student made in learning eighth-grade prealgebra may not be sustained without thefurther enhancement of taking algebra in ninth grade In other words the earlier treatment(prealgebra in eighth grade) and the later treatment (algebra in ninth grade) may reinforceone other each moderates the effect of the other treatment on the final outcome (math achieve-ment at the end of ninth grade) As Hong and Raudenbush (2008) argued the cumulativeeffect of two treatments in a well-aligned sequence may exceed the sum of the benefits oftwo single-year treatments Similarly experiencing two consecutive years of inferior treat-ments such as encountering an incompetent math teacher who dampens a studentrsquos self-efficacy in math learning in eighth grade and then again in ninth grade could do more damagethan the sum of the effect of encountering an incompetent teacher only in eighth grade and theeffect of having such a teacher only in ninth grade

There has been a great amount of conceptual confusion regarding how a moderatorrelates to the treatment and the outcome On one hand many researchers have used the termsldquomoderatorsrdquo and ldquomediatorsrdquo interchangeably without understanding the crucial distinctionbetween the two An overcorrective attempt on the other hand has led to the arbitrary rec-ommendation that a moderator must occur prior to the treatment and be minimally associatedwith the treatment (James and Brett 1984 Kraemer et al 2001 2008) In Chapter 6 weclarify that in essence the causal effect of the treatment on the outcome may depend onthe moderator value A moderator can be a subpopulation identifier a contextual character-istic a concurrent treatment or a preceding or succeeding treatment In the earlier examplesage of estrogen therapy initiation and a studentrsquos prior ability are subpopulation identifiers themanageability of a class which reflects both teacher skills and peer behaviors characterizes thecontext literacy instruction time is a concurrent treatment while prealgebra is a precedingtreatment for algebra A moderator does not have to occur prior to the treatment and doesnot have to be independent of the treatment

Once a moderated causal relationship has been defined in terms of potential outcomes theresearcher then chooses an appropriate experimental design for testing the moderation theoryThe assumptions required for identifying the moderated causal relationships differ across dif-ferent designs and have implications for analyzing experimental and quasiexperimental dataRandomized block designs are suitable for examining individual or contextual characteristicsas potential moderators factorial designs enable one to determine the joint effects of two ormore concurrent treatments and sequential randomized designs are ideal for assessing thecumulative effects of consecutive treatments Multisite randomized trials constitute anadditional type of designs in which the experimental sites are often deliberately sampled torepresent a population of geographical locations or social organizations Site membershipscan be viewed as implicit moderators that summarize a host of features of a local environmentReplications of a treatment over multiple sites allow one to quantify the heterogeneity of treat-ment effects across the sites Chapters 7 and 8 are focused on statistical methods for moder-ation analyses with quasiexperimental data In particular these chapters demonstrate througha number of application examples how a nonparametric marginal mean weighting through

6 CAUSALITY IN A SOCIAL WORLD

stratification (MMWS) method can overcome some important limitations of other existingmethods

Moderation questions are not restricted to treatmentndashoutcome relationships Rather theyare prevalent in investigations of mediation and spill-over This is because heterogeneity oftreatment effects may be explained by the variation in causal mediation mechanisms acrosssubpopulations and contexts It is also because the treatment effect for a focal individualmay depend on whether there is spill-over from other individuals through social interactionsYet similar to the confusion around the concept of moderation among applied researchersthere has been a great amount of misunderstanding with regard to mediation

112 Mediated treatment effects

Questions about mediation are at the core of nearly every scientific theory In its simplestform a theory explains why treatment Z causes outcome Y by hypothesizing an intermediateprocess involving at least one mediatorM that could have been changed by the treatment andcould subsequently have an impact on the outcome A causal mediation analysis then decom-poses the total effect of the treatment on the outcome into two parts the effect transmittedthrough the hypothesized mediator called the indirect effect and the difference betweenthe total effect and the indirect effect called the direct effect The latter represents the treat-ment effect channeled through other unspecified pathways (Alwin and Hauser 1975 Baronand Kenny 1986 Duncan 1966 Shadish and Sweeney 1991) Most researchers in socialsciences have followed the convention of illustrating the direct effect and the indirect effectwith path diagrams and then specifying linear regression models postulated to represent thestructural relationships between the treatment the mediator and the outcome

In Chapter 9 we point out that the approach to defining the indirect effect and the directeffect in terms of path coefficients is often misled by oversimplistic assumptions about thestructural relationships In particular this approach typically overlooks the fact that a treat-ment may generate an impact on the outcome not only by changing the mediator value butalso by changing the mediatorndashoutcome relationship (Judd and Kenny 1981) An examplein Chapter 9 illustrates such a case An experiment randomizes students to either an experi-mental condition which provides them with study materials and encourages them to study fora test or to a control condition which provides neither study materials nor encouragement(Holland 1988 Powers and Swinton 1984) One might hypothesize that encouraging exper-imental group members to study will increase the time they spend studying a focal mediatorin this example which in turn will increase their average test scores One might furtherhypothesize that even without a change in study time providing study materials to experi-mental group members will enable them to study more effectively than they otherwise wouldConsequently the effect of time spent studying might be greater under the experimentalcondition than under the control condition Omitting the treatment-by-mediator interactioneffect will lead to bias in the estimation of the indirect effect and the direct effect

Chapter 11 describes in great detail an evaluation study contrasting a new welfare-to-workprogram emphasizing active participation in the labor force with a traditional program thatguaranteed cash assistance without a mandate to seek employment (Hong Deutsch and Hill2011) Focusing on the psychological well-being of welfare applicants who were singlemothers with preschool-aged children the researchers hypothesized that the treatment wouldincrease employment rate among the welfare recipients and that the treatment-inducedincrease in employment would likely reduce depressive symptoms under the new program

7INTRODUCTION

They also hypothesized that in contrast the same increase in employment was unlikely tohave a psychological impact under the traditional program Hence the treatment-by-mediatorinteraction effect on the outcome was an essential component of the intermediate process

We will show that by defining the indirect effect and the direct effect in terms of potentialoutcomes one can avoid invoking unwarranted assumptions about the unknown structuralrelationships (Holland 1988 Pearl 2001 Robins and Greenland 1992) The potential out-comes framework has further clarified that in a causal mediation theory the treatment and themediator are both conceivably manipulable In the aforementioned example a welfare appli-cant could be assigned at random to either the new program or the traditional one Once havingbeen assigned to one of the programs the individual might or might not be employed due tovarious structural constraints market fluctuations or other random events typically beyondher control

Because the indirect effect and the direct effect are defined in terms of potential outcomesrather than on the basis of any specific regression models it becomes possible to distinguishthe definitions of these causal effects from their identification and estimation The identifica-tion step relates a causal parameter to observable population data For example for individualsassigned at random to an experimental condition their average counterfactual outcome asso-ciated with the control condition is expected to be equal to the average observable outcome ofthose assigned to the control condition Therefore the population average causal effect can beeasily identified in a randomized experiment The estimation step then relates the sample datato the observable population quantities involved in identification while taking into account thedegree of sampling variability

A major challenge in identification is that the mediatorndashoutcome relationship tends tobe confounded by selection even if the treatment is randomized Chapter 9 reviews variousexperimental designs that have been proposed by past researchers for studying causalmediation mechanisms Chapter 10 compares the identification assumptions and the analyticprocedures across a wide range of analytic methods These discussions are followed by anintroduction of the ratio-of-mediator-probability weighting (RMPW) strategy in Chapter 11In comparison with most existing strategies for causal mediation analysis RMPW relies onrelatively fewer identification assumptions and model-based assumptions Chapters 12 and13 will show that this new strategy can be applied broadly with extensions to multilevelexperimental designs and to studies of complex mediation mechanisms involving multiplemediators

113 Spill-over effects of a treatment

It is well known in vaccination research that an unvaccinated person can benefit when mostother people in the local community are vaccinated This is viewed as a spill-over of the treat-ment effect from the vaccinated people to an unvaccinated person Similarly due to socialinteractions among individuals or groups of individuals an intervention received by somemay generate a spill-over impact on others affiliated with the same organization or connectedthrough the same network For example effective policing in one neighborhood may driveoffenders to operate in other neighborhoods In evaluating an innovative community policingprogram researchers found that a neighborhood not assigned to community policing tended tosuffer if the surrounding neighborhoods were assigned to community policing This is aninstance in which a well-intended treatment generates an unintended negative spill-over effectHowever being assigned to community policy was found particularly beneficial when thesurrounding neighborhoods also received the intervention (Verbitsky-Savitz and Raudenbush

8 CAUSALITY IN A SOCIAL WORLD

2012) In another example the impact of a school-based mentoring program targeted atstudents displaying delinquent behaviors may be enhanced for a focal student if his or herat-risk peers are assigned to the mentoring program at the same time

The previous examples challenge the ldquostable unit treatment value assumptionrdquo (SUTVA)that has been invoked in most causal inference studies This assumption states that an indi-vidualrsquos potential response to a treatment depends neither on how the treatment is assignednor on the treatment assignments of other individuals (Rubin 1980 1986) Rubin and othersbelieve that without resorting to SUTVA the causal effect of a treatment becomes hard todefine and that causal inference may become intractable However as Sobel (2006) has illus-trated with the Moving to Opportunity (MTO) experiment that offered housing vouchersenabling low-income families to move to low-poverty neighborhoods there are possible con-sequences of violating SUTVA in estimating the treatment effects on outcomes such as safetyand self-sufficiency Because social interactions among individuals may affect whether onevolunteers to participate in the study whether one moves to a low-poverty neighborhood afterreceiving the housing voucher as well as housing project residentsrsquo subjective perceptions oftheir neighborhoods the program may have a nonzero impact on the potential outcome of theuntreated As a result despite the randomization of treatment assignment the mean differencein the observed outcome between the treated units and the untreated units may be biased forthe average treatment effect Rather the observable quantity is the difference between theeffect of treating some rather than treating none for the treated and the effect of treating somerather than treating none for the untreated the latter being the pure spill-over effect for theuntreated

In making attempts to relax SUTVA researchers have proposed several alternative frame-works that incorporate possible spill-over effects (see Hong and Raudenbush 2013 for areview) Hong (2004) presented a model that involves treatments and treatment settingsA treatment setting for an individual is a local environment constituted by a set of agentsand participants along with their treatment assignments An individualrsquos potential outcomevalue under a given treatment is assumed stable when the treatment setting is fixed the poten-tial outcome may take different values when the treatment setting shifts One may investigatewhether the treatment effect depends on the treatment setting Applying this framework Hongand Raudenbush (2006) examined the effect on a childrsquos academic growth of retaining thechild in kindergarten rather than promoting the child to first grade when a relatively smallproportion of low-achieving peers in the same school are retained as opposed to when a rel-atively large proportion of the peers are retained Hudgens and Halloran (2008) presented arelated framework in which the effect on an individual of the treatment received by this indi-vidual is distinguished from the effect on the individual of the treatment received by others inthe same local community These effects can be identified if communities are assigned atrandom to different treatment assignment strategies (such as retaining a large proportion oflow-achieving students as opposed to retaining a small proportion of such students) and sub-sequently individuals within a community are randomized for treatment Chapter 14 reviewsthese frameworks and discusses identification and estimation strategies

Social contagion may also serve as an important channel through which the effect of anintervention is transmitted For example a school-wide intervention may reduce aggressivebehaviors and thereby improve studentsrsquo psychological well-being by improving the qualityof interpersonal relationships in other classes as well as in onersquos own class This is becausechildren interact not simply with their classmates but also with those from other classes in thehallways or on the playground (VanderWeele et al 2013) In this case the spill-over betweenclasses becomes a part of the mediation mechanism In a study of student mentoring whether

9INTRODUCTION

an individual actually participated in the mentoring program despite the initial treatmentassignment is typically viewed as a primary mediator The proportion of onersquos peers that par-ticipated in the program may act as a second mediator A treatment may also exert its impactpartly through regrouping individuals and thereby changing peer composition (Hong andNomi 2012) Chapter 15 discusses analytic strategies for detecting spill-over as a part ofthe causal mediation mechanism

12 Weighting methods for causal inference

In most behavioral and social science applications major methodological challenges arise dueto the selection of treatments in a quasiexperimental study and the selection of mediator valuesin experimental and quasiexperimental studies Statistical methods most familiar to research-ers in social sciences are often inadequate for causal inferences with regard to multivaluedtreatments moderation mediation and spill-over The book offers a major revision to under-standing of causality in the social world by introducing two complementary weighting stra-tegies both featuring nonparametric approaches to estimating these causal effects The newweighting methods greatly simplify model specifications while enhancing the robustness ofresults by minimizing the reliance on some key assumptions

The propensity score-basedMMWSmethod removes selection bias associated with a largenumber of covariates by equating the pretreatment composition between treatment groups(Hong 2010a 2012 Huang et al 2005) Unlike propensity score matching and stratificationthat are mostly restricted to evaluations of binary treatments the MMWS method is flexiblefor evaluating binary and multivalued treatments by approximating a completely randomizedexperiment In evaluating whether the treatment effects differ across subpopulations definedby individual characteristics or treatment settings researchers may assign weights within eachsubpopulation in order to approximate a randomized block design To investigate whetherone treatment moderates the effect of another concurrent treatment researchers may assignweights to the data to approximate a factorial randomized design The method can also beused to assess whether the effect of an initial treatment is amplified or weakened by a subse-quent treatment or to identify an optimal treatment sequence through approximating a sequen-tial randomized experiment Even though such analyses can similarly be conducted throughinverse-probability-of-treatment weighting (IPTW) that has been increasingly employed inepidemiological research (Hernaacuten Brumback and Robins 2000 Robins Hernaacuten andBrumback 2000) IPTW is known for bias and imprecision in estimation especially whenthe propensity score models are misspecified in their functional forms (Hong 2010a Kangand Schafer 2007 Schafer and Kang 2008 Waernbaum 2012) In contrast the nonparamet-ric MMWS method displays a relatively high level of robustness despite such misspecifica-tions and also gains efficiency as indicated by simulation results (Hong 2010a)

To study causal mediation mechanisms the RMPW method decomposes the total effectof a treatment into an ldquoindirect effectrdquo transmitted through a specific mediator and a ldquodirecteffectrdquo representing unspecified mechanisms In contrast with most existing methods formediation analysis the RMPW-adjusted outcome model is extremely simple and is nonpara-metric in nature It generates estimates of the causal effects along with their sampling errorswhile adjusting for pretreatment covariates that confound the mediatorndashoutcome relationshipsthrough weighting The method applies regardless of the distribution of the outcome the dis-tribution of the mediator or the functional relationship between the outcome and the mediator

10 CAUSALITY IN A SOCIAL WORLD

Page 3: Thumbnail - download.e-bookshelf.de...4.2.4 IPTW for bias removal 84 4.3 MMWS 86 4.3.1 Theoretical rationale 86 ... 8.3.4.2 Growth model in the absence of confounding 200 8.3.4.3 Weighted

Causality in a Social World

Moderation Meditation and Spill-over

Guanglei Hong

Department of Comparative Human DevelopmentUniversity of Chicago USA

This edition first published 2015copy 2015 John Wiley amp Sons Ltd

Registered OfficeJohn Wiley amp Sons Ltd The Atrium Southern Gate Chichester West Sussex PO19 8SQ United Kingdom

For details of our global editorial offices for customer services and for information about how to apply for permissionto reuse the copyright material in this book please see our website at wwwwileycom

The right of the author to be identified as the author of this work has been asserted in accordance with the CopyrightDesigns and Patents Act 1988

All rights reserved No part of this publication may be reproduced stored in a retrieval system or transmitted in anyform or by any means electronic mechanical photocopying recording or otherwise except as permitted by theUK Copyright Designs and Patents Act 1988 without the prior permission of the publisher

Wiley also publishes its books in a variety of electronic formats Some content that appears in print may not beavailable in electronic books

Designations used by companies to distinguish their products are often claimed as trademarks All brand names andproduct names used in this book are trade names service marks trademarks or registered trademarks of their respectiveowners The publisher is not associated with any product or vendor mentioned in this book

Limit of LiabilityDisclaimer of Warranty While the publisher and author have used their best efforts in preparing this bookthey make no representations or warranties with respect to the accuracy or completeness of the contents of this bookand specifically disclaim any implied warranties of merchantability or fitness for a particular purpose It is sold on theunderstanding that the publisher is not engaged in rendering professional services and neither the publisher nor the authorshall be liable for damages arising herefrom If professional advice or other expert assistance is required the services ofa competent professional should be sought

Library of Congress Cataloging-in-Publication Data applied for

ISBN 9781118332566

A catalogue record for this book is available from the British Library

Set in 1012pt Times by SPi Global Pondicherry India

1 2015

Contents

Preface xv

Part I Overview 1

1 Introduction 311 Concepts of moderation mediation and spill-over 3

111 Moderated treatment effects 5112 Mediated treatment effects 7113 Spill-over effects of a treatment 8

12 Weighting methods for causal inference 1013 Objectives and organization of the book 1114 How is this book situated among other publications on related topics 12

2 Review of causal inference concepts and methods 1821 Causal inference theory 18

211 Attributes versus causes 18212 Potential outcomes and individual-specific causal effects 19213 Inference about population average causal effects 22

2131 Prima facie effect 242132 Ignorability assumption 25

22 Applications to Lordrsquos paradox and Simpsonrsquos paradox 27221 Lordrsquos paradox 27222 Simpsonrsquos paradox 31

23 Identification and estimation 34231 Selection bias 35232 Sampling bias 35233 Estimation efficiency 36

Appendix 21 Potential bias in a prima facie effect 36Appendix 22 Application of the causal inference theory to Lordrsquos paradox 37

3 Review of causal inference designs and analytic methods 4031 Experimental designs 40

311 Completely randomized designs 40

312 Randomized block designs 41313 Covariance adjustment for improving efficiency 43314 Multilevel experimental designs 43

32 Quasiexperimental designs 44321 Nonequivalent comparison group designs 44322 Other quasiexperimental designs 45

33 Statistical adjustment methods 46331 ANCOVA and multiple regression 46

3311 ANCOVA for removing selection bias 463312 Potential pitfalls of ANCOVA with a vast between-group

difference 473313 Bias due to model misspecification 48

332 Matching and stratification 50333 Other statistical adjustment methods 51

3331 The IV method 513332 DID analysis 54

34 Propensity score 55341 What is a propensity score 56342 Balancing property of the propensity score 57343 Pooling conditional treatment effect estimate Matching

stratification and covariance adjustment 603431 Propensity score matching 613432 Propensity score stratification 623433 Covariance adjustment for the propensity score 663434 Sensitivity analysis 66

Appendix 3A Potential bias due to the omission of treatment-by-covariateinteraction 70

Appendix 3B Variable selection for the propensity score model 71

4 Adjustment for selection bias through weighting 7641 Weighted estimation of population parameters in survey sampling 77

411 Simple random sample 77412 Proportionate sample 78413 Disproportionate sample 79

42 Weighting adjustment for selection bias in causal inference 80421 Experimental result 81422 Quasiexperimental result 81423 Sample weight for bias removal 82424 IPTW for bias removal 84

43 MMWS 86431 Theoretical rationale 86

4311 MMWS for a discrete propensity score 874312 MMWS for a continuous propensity score 884313 MMWS for estimating the treatment effect on the treated 89

432 MMWS analytic procedure 91433 Inherent connection and major distinctions between MMWS

and IPTW 93

vi CONTENTS

Appendix 4A Proof of MMWS-adjusted mean observed outcome being unbiasedfor the population average potential outcome 95

Appendix 4B Derivation of MMWS for estimating the treatment effecton the treated 96

Appendix 4C Theoretical equivalence of MMWS and IPTW 97Appendix 4D Simulations comparing MMWS and IPTW under misspecifications

of the functional form of a propensity score model 97

5 Evaluations of multivalued treatments 10051 Defining the causal effects of multivalued treatments 10052 Existing designs and analytic methods for evaluating multivalued

treatments 102521 Experimental designs and analysis 102

5211 Randomized experiments with multipletreatment arms 102

5212 Identification under ignorability 1025213 ANOVA 103

522 Quasiexperimental designs and analysis 1055221 ANCOVA and multiple regression 1055222 Propensity score-based adjustment 1085223 Other adjustment methods 112

53 MMWS for evaluating multivalued treatments 112531 Basic rationale 113532 Analytic procedure 114

5321 MMWS for a multinomial treatment measure 1155322 MMWS for an ordinal treatment measure 120

533 Identification assumptions 12154 Summary 123Appendix 5A Multiple IV for evaluating multivalued treatments 124

Part II Moderation 127

6 Moderated treatment effects concepts and existing analytic methods 12961 What is moderation 129

611 Past discussions of moderation 1306111 Purpose of moderation research 1306112 What qualifies a variable as a moderator 132

612 Definition of moderated treatment effects 1336121 Treatment effects moderated by individual or contextual

characteristics 1336122 Joint effects of concurrent treatments 134

62 Experimental designs and analytic methods for investigatingexplicit moderators 136621 Randomized block designs 137

6211 Identification assumptions 1376212 Two-way ANOVA 1386213 Multiple regression 139

viiCONTENTS

622 Factorial designs 1406221 Identification assumptions 1406222 Analytic strategies 142

63 Existing research designs and analytic methods for investigatingimplicit moderators 142631 Multisite randomized trials 143

6311 Causal parameters and identification assumptions 1446312 Analytic strategies 145

632 Principal stratification 149Appendix 6A Derivation of bias in the fixed-effects estimator when the

treatment effect is heterogeneous in multisite randomized trials 151Appendix 6B Derivation of bias in the mixed-effects estimator when the

probability of treatment assignment varies across sites 153Appendix 6C Derivation and proof of the population weight applied to

mixed-effects models for eliminating bias in multisiterandomized trials 153

7 Marginal mean weighting through stratification for investigatingmoderated treatment effects 15971 Existing methods for moderation analyses with quasiexperimental data 159

711 Analysis of covariance and regression-based adjustment 1617111 Treatment effects moderated by subpopulation membership 1617112 Treatment effects moderated by a concurrent treatment 164

712 Propensity score-based adjustment 1657121 Propensity score matching and stratification 1667122 Inverse-probability-of-treatment weighting 167

72 MMWS estimation of treatment effects moderated by individual orcontextual characteristics 168721 Application example 170722 Analytic procedure 170

73 MMWS estimation of the joint effects of concurrent treatments 174731 Application example 174732 Analytic procedure 175733 Joint treatment effects moderated by individual or contextual

characteristics 179

8 Cumulative effects of time-varying treatments 18581 Causal effects of treatment sequences 186

811 Application example 186812 Causal parameters 187

8121 Time-varying treatments 1878122 Time-varying potential outcomes 1878123 Causal effects of 2-year treatment sequences 1878124 Causal effects of multiyear treatment sequences 190

82 Existing strategies for evaluating time-varying treatments 190821 The endogeneity problem in nonexperimental data 190822 SEM 191

viii CONTENTS

823 Fixed-effects econometric models 192824 Sequential randomization 192825 Dynamic treatment regimes 193826 Marginal structural models and structural nested models 194

83 MMWS for evaluating 2-year treatment sequences 195831 Sequential ignorability 195832 Propensity scores 196833 MMWS computation 197

8331 MMWS adjustment for year 1 treatment selection 1978332 MMWS adjustment for two-year treatment

sequence selection 198834 Two-year growth model specification 199

8341 Growth model in the absence of treatment 2008342 Growth model in the absence of confounding 2008343 Weighted 2-year growth model 202

84 MMWS for evaluating multiyear sequences of multivalued treatments 204841 Sequential ignorability of multiyear treatment sequences 204842 Propensity scores for multiyear treatment sequences 204843 MMWS computation 205844 Weighted multiyear growth model 205845 Issues of sample size 206

85 Conclusion 207Appendix 8A A saturated model for evaluating multivalued treatments

over multiple time periods 207

Part III Mediation 211

9 Concepts of mediated treatment effects and experimental designsfor investigating causal mechanisms 21391 Introduction 21492 Path coefficients 21593 Potential outcomes and potential mediators 216

931 Controlled direct effects 217932 Controlled treatment-by-mediator interaction effect 217

94 Causal effects with counterfactual mediators 219941 Natural direct effect 219942 Natural indirect effect 220943 Natural treatment-by-mediator interaction effect 220944 Unstable unit treatment value 221

95 Population causal parameters 222951 Population average natural direct effect 224952 Population average natural indirect effect 225

96 Experimental designs for studying causal mediation 225961 Sequentially randomized designs 228962 Two-phase experimental designs 228963 Three- and four-treatment arm designs 230964 Experimental causal-chain designs 231

ixCONTENTS

965 Moderation-of-process designs 231966 Augmented encouragement designs 232967 Parallel experimental designs and parallel encouragement designs 232968 Crossover experimental designs and crossover encouragement

designs 233969 Summary 234

10 Existing analytic methods for investigating causal mediation mechanisms 238101 Path analysis and SEM 239

1011 Analytic procedure for continuous outcomes 2391012 Identification assumptions 2421013 Analytic procedure for discrete outcomes 245

102 Modified regression approach 2461021 Analytic procedure for continuous outcomes 2461022 Identification assumptions 2471023 Analytic procedure for binary outcomes 248

103 Marginal structural models 2501031 Analytic procedure 2501032 Identification assumptions 252

104 Conditional structural models 2521041 Analytic procedure 2521042 Identification assumptions 253

105 Alternative weighting methods 2541051 Analytic procedure 2541052 Identification assumptions 256

106 Resampling approach 2561061 Analytic procedure 2561062 Identification assumptions 257

107 IV method 2571071 Rationale and analytic procedure 2571072 Identification assumptions 258

108 Principal stratification 2591081 Rationale and analytic procedure 2591082 Identification assumptions 260

109 Sensitivity analysis 2611091 Unadjusted confounding as a product of hypothetical

regression coefficients 2611092 Unadjusted confounding reflected in a hypothetical

correlation coefficient 2621093 Limitations when the selection mechanism differs

by treatment 2641094 Other sensitivity analyses 265

1010 Conclusion 26510101 The essentiality of sequential ignorability 26510102 Treatment-by-mediator interactions 26610103 Homogeneous versus heterogeneous causal effects 26610104 Model-based assumptions 266

x CONTENTS

Appendix 10A Bias in path analysis estimation due to the omission oftreatment-by-mediator interaction 267

11 Investigations of a simple mediation mechanism 273111 Application example national evaluation of welfare-to-work strategies 274

1111 Historical context 2741112 Research questions 2751113 Causal parameters 2751114 NEWWS Riverside data 277

112 RMPW rationale 2771121 RMPW in a sequentially randomized design 278

11211 E[Y(0 M(0))] 27911212 E[Y(1 M(1))] 28011213 E[Y(1 M(0))] 28011214 E[Y(0 M(1))] 28111215 Nonparametric outcome model 282

1122 RMPW in a sequentially randomized block design 2831123 RMPW in a standard randomized experiment 2851124 Identification assumptions 286

113 Parametric RMPW procedure 287114 Nonparametric RMPW procedure 290115 Simulation results 292

1151 Correctly specified propensity score models 2921152 Misspecified propensity score models 2941153 Comparisons with path analysis and IV results 294

116 Discussion 2951161 Advantages of the RMPW strategy 2951162 Limitations of the RMPW strategy 295

Appendix 11A Causal effect estimation through the RMPW procedure 296Appendix 11B Proof of the consistency of RMPW estimation 297

12 RMPW extensions to alternative designs and measurement 301121 RMPW extensions to mediators and outcomes of alternative distributions 301

1211 Extensions to a multicategory mediator 30212111 Parametric RMPW procedure 30212112 Nonparametric RMPW procedure 304

1212 Extensions to a continuous mediator 3041213 Extensions to a binary outcome 306

122 RMPW extensions to alternative research designs 3061221 Extensions to quasiexperimental data 3071222 Extensions to data from cluster randomized trials 308

12221 Application example and research questions 30912222 Estimation of the total effect 31012223 RMPW analysis of causal mediation mechanisms 31012224 Identification assumptions 31212225 Contrast with multilevel path analysis and SEM 31212226 Contrast with multilevel prediction models 313

xiCONTENTS

1223 Extensions to data from multisite randomized trials 31312231 Research questions and causal parameters 31412232 Estimation of the total effect and its between-site variation 31512233 RMPW analysis of causal mediation mechanisms 31612234 Identification assumptions 31912235 Contrast with multilevel path analysis and SEM 319

123 Alternative decomposition of the treatment effect 321

13 RMPW extensions to studies of complex mediation mechanisms 325131 RMPW extensions to moderated mediation 325

1311 RMPW analytic procedure for estimating and testingmoderated mediation 326

1312 Path analysisSEM approach to analyzing moderatedmediation 327

1313 Principal stratification and moderated mediation 328132 RMPW extensions to concurrent mediators 328

1321 Treatment effect decomposition 32913211 Treatment effect decomposition without

between-mediator interaction 32913212 Treatment effect decomposition with

between-mediator interaction 3311322 Identification assumptions 3331323 RMPW procedure 333

13231 Estimating E[Y(1M1(1)M2(0))] 33513232 Estimating E[Y(1M1(0)M2(0))] 33513233 Causal effect estimation with noninteracting

concurrent mediators 33613234 Estimating E[Y(1M1(0)M2(1))] 33713235 Causal effect estimation with interacting concurrent

mediators 3371324 Contrast with the linear SEM approach 3381325 Contrast with the multivariate IV approach 339

133 RMPW extensions to consecutive mediators 3401331 Treatment effect decomposition 341

13311 Natural direct effect of the treatment on the outcome 34213312 Natural indirect effect mediated by M1 only 34213313 Natural indirect effect mediated by M2 only 34213314 Natural indirect effect mediated by an M1-by-M2

interaction 34313315 Treatment-by-mediator interactions 343

1332 Identification assumptions 3451333 RMPW procedure 347

13331 Estimating E[Y(1M1(0)M2(0M1(0)))] 34713332 Estimating E[Y(1M1(1)M2(0M1(0)))] 34913333 Estimating E[Y(1M1(0)M2(1M1(1)))] 34913334 Estimating E[Y(0M1(1)M2(0M1(0)))] and

E[Y(0M1(0)M2(1M1(1)))] 351

xii CONTENTS

1334 Contrast with the linear SEM approach 3531335 Contrast with the sensitivity-based estimation of bounds for

causal effects 354134 Discussion 355Appendix 13A Derivation of RMPW for estimating population average

counterfactual outcomes of two concurrentmediators 355

Appendix 13B Derivation of RMPW for estimating population averagecounterfactual outcomes of consecutivemediators 358

Part IV Spill-over 363

14 Spill-over of treatment effects concepts and methods 365141 Spill-over A nuisance a trifle or a focus 365142 Stable versus unstable potential outcome values An example

from agriculture 367143 Consequences for causal inference when spill-over is overlooked 369144 Modified framework of causal inference 371

1441 Treatment settings 3711442 Simplified characterization of treatment settings 3731443 Causal effects of individual treatment assignment and of peer

treatment assignment 375145 Identification Challenges and solutions 376

1451 Hypothetical experiments for identifying average treatmenteffects in the presence of social interactions 376

1452 Hypothetical experiments for identifying the impact ofsocial interactions 380

1453 Application to an evaluation of kindergarten retention 382146 Analytic strategies for experimental and quasiexperimental data 384

1461 Estimation with experimental data 3841462 Propensity score stratification 3851463 MMWS 386

147 Summary 387

15 Mediation through spill-over 391151 Definition of mediated effects through spill-over in a cluster

randomized trial 3931511 Notation 3931512 Treatment effect mediated by a focal individualrsquos

compliance 3941513 Treatment effect mediated by peersrsquo compliance through

spill-over 3941514 Decomposition of the total treatment effect 395

152 Identification and estimation of the spill-over effect in a clusterrandomized design 3951521 Identification in an ideal experiment 395

xiiiCONTENTS

1522 Identification when the mediators are not randomized 3981523 Estimation of mediated effects through spill-over 400

153 Definition of mediated effects through spill-over in a multisite trial 4021531 Notation 4021532 Treatment effect mediated by a focal individualrsquos compliance 4041533 Treatment effect mediated by peersrsquo compliance

through spill-over 4041534 Direct effect of individual treatment assignment on the outcome 4051535 Direct effect of peer treatment assignment on the outcome 4051536 Decomposition of the total treatment effect 405

154 Identification and estimation of spill-over effects in a multisite trial 4061541 Identification in an ideal experiment 4071542 Identification when the mediators are not randomized 4091543 Estimation of mediated effects through spill-over 410

15431 Estimating E[Y(1pM(p)Mminus(p))] andE[Y(000Mminus(0))] 410

15432 Estimating E[Y(1p0Mminus(p))] 41115433 Estimating E[Y(1p0Mminus(0))] 41215434 Estimating E[Y(0p0Mminus(0))] 412

155 Consequences of omitting spill-over effects in causal mediation analyses 4121551 Biased inference in a cluster randomized trial 4131552 Biased inference in a multisite randomized trial 4131553 Biased inference of the local average treatment effect 415

156 Quasiexperimental application 416157 Summary 419Appendix 151 Derivation of the weight for estimating the population

average counterfactual outcome E[Y(1 p 0Mminus( p))] 419Appendix 152 Derivation of bias in the ITT effect due to the omission of

spill-over effects 420

Index 423

xiv CONTENTS

Preface

A scientific mind in training seeks comprehensive descriptions of every interesting phenom-enon In such a mind data are to be pieced together for comparisons contrasts and eventuallyinferences that are required for understanding causes and effects that may have led to thephenomenon of interest Yet without disciplined reasoning a causal inference would slip intothe rut of a ldquocasual inferencerdquo as easily as making a typographical error

As a doctoral student thirsting for rigorous methodological training at the University ofMichigan I was very fortunate to be among the first group of students on the campus attendinga causal inference seminar in 2000 jointly taught by Yu Xie from the Department of Sociol-ogy Susan Murphy from the Department of Statistics and Stephen Raudenbush from theSchool of Education The course was unique in both content and pedagogy The instructorsput together a bibliography drawn from the past and current causal inference literature origi-nated from multiple disciplines In the spirit of ldquoreciprocal teachingrdquo the students were delib-erately organized into small groups each representing a mix of disciplinary backgrounds andtook turns to teach the weekly readings under the guidance of a faculty instructor This inval-uable experience revealed to me that the pursuit of causal inference is a multidisciplinaryendeavor In the rest of my doctoral study I continued to be immersed in an extraordinaryinterdisciplinary community in the form of the Quantitative Methodology Program (QMP)seminars led by the above three instructors who were then joined by Ben Hansen(Statistics) Richard Gonzalez (Psychology) Roderick Little (Biostatistics) Jake Bowers(Political Science) and many others I have tried whenever possible to carry the same spiritinto my own teaching of causal inference courses at the University of Toronto and now at theUniversity of Chicago In these courses communicating understandings of causal problemsacross disciplinary boundaries has been particularly challenging but also stimulating and grat-ifying for myself and for students from various departments and schools

Under the supervision of Stephen Raudenbush I took a first stab at the conceptual frame-work of causal inference in my doctoral dissertation by considering peer spill-over in schoolsettings From that point on I have organized my methodological research to tackle some ofthe major obstacles to causal inferences in policy and program evaluations My workaddresses issues including (i) how to conceptualize and evaluate the causal effects of educa-tional treatments when studentsrsquo responses to alternative treatments depend on various fea-tures of the organizational settings including peer composition (ii) how to adjust forselection bias in evaluating the effects of concurrent and consecutive multivalued treatmentsand (iii) how to conceptualize and analyze causal mediation mechanisms My methodological

research has been driven by the substantive interest in understanding the role of socialinstitutionsmdashschools in particularmdashin shaping human development especially during theages of fast growth (eg childhood adolescence and early adulthood) I have shared meth-odological advances with a broad audience in social sciences through applying the causalinference methods to prominent substantive issues such as grade retention within-class group-ing services for English language learners and welfare-to-work strategies These illustrationsare used extensively in this book

Among social scientists awareness of methodological challenges in causal investigationsis higher than ever before In response to this rising demand causal inference is becoming oneof the most productive scholarly fields in the past decade I have had the benefit of followingthe work and exchanging ideas with some fellow methodologists who are constantly contri-buting new thoughts and making breakthroughs These include Daniel Almirall HowardBloom Tom Cook Michael Elliot Ken Frank Ben Hansen James Heckman JenniferHill Martin Huber Kosuke Imai Booil Jo John R Lockwood David MacKinnon DanielMcCaffrey Luke Miratrix Richard Murnane Derek Neal Lindsey Page Judea PearlStephen Raudenbush Sean Reardon Michael Seltzer Youngyun Shin Betsy SinclairMichael Sobel Peter Steiner Elizabeth Stuart Eric Thetchen Tchetchen Tyler VanderWeeleand Kazuo Yamaguchi I am grateful to Larry Hedges who offered me the opportunity of guestediting the Journal of Research in Educational Effectiveness special issue on the statisticalapproaches to studying mediator effects in education research in 2012 Many of the colleagueswhom I mentioned above contributed to the open debate in the special issue by either propos-ing or critically examining a number of innovative methods for studying causal mechanismsI anticipate that many of them are or will soon be writing their own research monographs oncausal inference if they have not already done so Therefore readers of this book are stronglyrecommended to browse past and future titles by these and other authors for alternative per-spectives and approaches My own understanding of course will continue to evolve as well inthe coming years

One has to be ambitious to stay upfront in substantive areas as well as in methodology Thebest strategy apparently is to learn from substantive experts During different phases of mytraining and professional career I have had the opportunities to work with David K CohenBrian Rowan Deborah Ball Carl Corter Janette Pelletier Takako Nomi Esther Geva DavidFrancis Stephanie Jones Joshua Brown and Heather D Hill Many of my substantive insightswere derived from conversations with these mentors and colleagues I am especially gratefulto Bob Granger former president of the William T Grant Foundation who reassured me thatldquoBy staying a bit broad you will learn a lot and many fields will benefit from your workrdquo

Like many other single-authored books this one is built on the generous contributions ofstudents and colleagues who deserve special acknowledgement Excellent research assistancefrom Jonah Deutsch Joshua Gagne Rachel Garrett Yihua Hong Xu Qin Cheng Yang andBing Yu was instrumental in the development and implementation of the new methods pre-sented in this book Richard Congdon a renowned statistical software programmer broughtrevolutionary ideas to interface designs in software development with the aim of facilitatingusersrsquo decision-making in a series of causal analyses RichardMurnane Stephen Raudenbushand Michael Sobel carefully reviewed and critiqued multiple chapters of the manuscript draftAdditional comments and suggestions on earlier drafts came fromHoward Bloom Ken FrankBen Hansen Jennifer Hill Booil Jo Ben Kelcey David MacKinnon and Fan Yang TereseSchwartzman provided valuable assistance in manuscript preparation The writing of thisbook was supported by a Scholars Award from the William T Grant Foundation and an

xvi PREFACE

Institute of Education Sciences (IES) Statistical and Research Methodology in EducationGrant from the US Department of Education All the errors in the published form are ofcourse the sole responsibility of the author

Project Editors Richard Davis Prachi Sinha Sahay and Liz Wingett and Assistant EditorHeather Kay at Wiley and project Managers P Jayapriya and R Jayavel at SPi Global havebeen wonderful to work with throughout the manuscript preparation and final production ofthe book Debbie Jupe Commissioning Editor at Wiley was remarkably effective in initiatingthe contact with me and in organizing anonymous reviews of the book proposal and of themanuscript These constructive reviews have shaped the book in important ways I cannotagree more with one of the reviewers that ldquocausality cannot be established on a pure statisticalgroundrdquo The book therefore highlights the importance of substantive knowledge and researchdesigns and places a great emphasis on clarifying and evaluating assumptions required foridentifying causal effects in the context of each application Much caution is raised againstpossible misusage of statistical techniques for analyzing causal relationships especiallywhen data are inadequate Yet I maintain that improved statistical procedures along withimproved research designs would greatly enhance our ability in attempt to empiricallyexamine well-articulated causal theories

xviiPREFACE

Part I

OVERVIEW

1

Introduction

According to an ancient Chinese fable a farmer who was eager to help his crops grow wentinto his field and pulled each seedling upward After exhausting himself with the work heannounced to his family that they were going to have a good harvest only to find the nextmorning that the plants had wilted and died Readers with minimal agricultural knowledgemay immediately point out the following the farmerrsquos intervention theory was based on acorrect observation that crops that grow taller tend to produce more yield Yet his hypothesisreflects a false understanding of the cause and the effectmdashthat seedlings pulled to be tallerwould yield as much as seedlings thriving on their own

In their classic Design and Analysis of Experiments Hinkelmann and Kempthorne (1994updated version of Kempthorne 1952) discussed two types of science descriptive science andthe development of theory These two types of science are interrelated in the following senseobservations of an event and other related events often selected and classified for descriptionby scientists naturally lead to one or more explanations that we call ldquotheoretical hypothesesrdquowhich are then screened and falsified by means of further observations experimentation andanalyses (Popper 1963) The experiment of pulling seedlings to be taller was costly but didserve the purpose of advancing this farmerrsquos knowledge of ldquowhat does not workrdquo To developa successful intervention in this case would require a series of empirical tests of explicittheories identifying potential contributors to crop growth This iterative process graduallydeepens our knowledge of the relationships between supposed causes and effectsmdashthatis causalitymdashand may eventually increase the success of agricultural medical and socialinterventions

11 Concepts of moderation mediation and spill-over

Although the story of the ancient farmer is fictitious numerous examples can be found in thereal world in which well-intended interventions fail to produce the intended benefits or inmany cases even lead to unintended consequences ldquoInterventionsrdquo and ldquotreatmentsrdquo usedinterchangeably in this book broadly refer to actions taken by agents or circumstances expe-rienced by an individual or groups of individuals Interventions are regularly seen in educa-tion physical and mental health social services business politics and law enforcement In an

Causality in a Social World Moderation Meditation and Spill-over First Edition Guanglei Hongcopy 2015 John Wiley amp Sons Ltd Published 2015 by John Wiley amp Sons Ltd

education intervention for example teachers are typically the agents who deliver a treatmentto students while the impact of the treatment on student outcomes is of ultimate causalinterest Some educational practices such as ldquoteaching to the testrdquo have been criticized tobe nearly as counterproductive as the attempt of helping seedlings grow by pulling themupward ldquoInterventionsrdquo and ldquotreatmentsrdquo under consideration do not exclude undesiredexperiences such as exposure to poverty abuse crime or bereavement A treatment plannedor unplanned becomes a focus of research if there are theoretical reasons to anticipate itsimpact positive or negative on the well-being of individuals who are embedded in socialsettings including families classrooms schools neighborhoods and workplaces

In social science research in general and in policy and program evaluations in particularquestions concerning whether an intervention works and if so which version of the interven-tion works for whom under what conditions and why are key to the advancement of scien-tific and practical knowledge Although most empirical investigations in the social sciencesconcentrate on the average effect of a treatment for a specific population as opposed to theabsence of such a treatment (ie the control condition) in-depth theoretical reasoning withregard to how the causal effect is generated substantiated by compelling empirical evidenceis crucial for advancing scientific understanding

First when there are multiple versions or different dosages of the treatment or when thereare multiple versions of the control condition a binary divide between ldquothe treatmentrdquo andldquothe controlrdquo may not be as informative as fine-grained comparisons across for exampleldquotreatment version Ardquo ldquotreatment version Brdquo ldquocontrol version Ardquo and ldquocontrol versionBrdquo For example expanding the federally funded Head Start program to poor children isexpected to generate a greater benefit when few early childhood education alternatives areavailable (call it ldquocontrol version Ardquo) than when there is an abundance of alternatives includ-ing state-sponsored preschool programs (call it ldquocontrol version Brdquo)

Second the effect of an intervention will likely vary among individuals or across socialsettings A famous example comes from medical research the well-publicized cardiovascularbenefits of initiating estrogen therapy during menopause were contradicted later by experi-mental findings that the same therapy increased postmenopausal womenrsquos risk for heartattacks The effect of an intervention may also depend on the provision of some other con-current or subsequent interventions Such heterogeneous effects are often characterized asmoderated effects in the literature

Third alternative theories may provide competing explanations for the causal mechan-isms that is the processes through which the intervention produces its effect A theoreticalconstruct characterizing the hypothesized intermediate process is called a mediator of theintervention effect The fictitious farmer never developed an elaborate theory as to whatcaused some seedlings to surpass others in growth Once scientists revealed the causal rela-tionship between access to chemical nutrients in soil and plant growth wide applications ofchemically synthesized fertilizers finally led to a major increase in crop production

Finally it is well known in agricultural experiments that a new type of fertilizer applied toone plot may spill-over to the next plot Because social connections among individuals areprevalent within organizations or through networks an individualrsquos response to the treatmentmay similarly depend on the treatment for other individuals in the same social setting whichmay lead to possible spill-overs of intervention effects among individual human beings

Answering questions with regard to moderation mediation and spill-over poses majorconceptual and analytic challenges To date psychological research often presents well-articulated theories of causal mechanisms relating stimuli to responses Yet researchers often

4 CAUSALITY IN A SOCIAL WORLD

lack rigorous analytic strategies for empirically screening competing theories explaining theobserved effect Sociologists have keen interest in the spill-over of treatment effects transmit-ted through social interactions yet have produced limited evidence quantifying such effectsAs many have pointed out in general terminological ambiguity and conceptual confusionhave been prevalent in the published applied research (Holmbeck 1997 Kraemer et al2002 2008)

A new framework for conceptualizing moderated mediated and spill-over effects hasemerged relatively recently in the statistics and econometrics literature on causal inference(eg Abbring and Heckman 2007 Frangakis and Rubin 2002 Heckman and Vytlacil2007a b Holland 1988 Hong and Raudenbush 2006 Hudgens and Halloran 2008Jo 2008 Pearl 2001 Robins and Greenland 1992 Sobel 2008) The potential for furtherconceptual and methodological development and for broad applications in the field of behav-ioral and social sciences promises to greatly advance the empirical basis of our knowledgeabout causality in the social world

This book clarifies theoretical concepts and introduces innovative statistical strategies forinvestigating the average effects of multivalued treatments moderated treatment effectsmediated treatment effects and spill-over effects in experimental or quasiexperimental dataDefining individual-specific and population average treatment effects in terms of potentialoutcomes the book relates the mathematical forms to the substantive meanings of moderatedmediated and spill-over effects in the context of application examples It also explicates andevaluates identification assumptions and contrasts innovative statistical strategies with con-ventional analytic methods

111 Moderated treatment effects

It is hard to accept the assumption that a treatment would produce the same impact for everyindividual in every possible circumstance Understanding the heterogeneity of treatmenteffects therefore is key to the development of causal theories For example some studiesreported that estrogen therapy improved cardiovascular health among women who initiatedits use during menopause According to a series of other studies however the use of estrogentherapy increased postmenopausal womenrsquos risk for heart attacks (Grodstein et al 19962000 Writing Group for the Womenrsquos Health Initiative Investigators 2002) The sharp con-trast of findings from these studies led to the hypothesis that age of initiation moderates theeffect of estrogen therapy on womenrsquos health (Manson and Bassuk 2007 Rossouw et al2007) Revelations of the moderated causal relationship greatly enrich theoretical understand-ing and in this case directly inform clinical practice

In another example dividing students by ability into small groups for reading instructionhas been controversial for decades Many believed that the practice benefits high-ability stu-dents at the expense of their low-ability peers and exacerbates educational inequality (Grantand Rothenberg 1986 Rowan and Miracle 1983 Trimble and Sinclair 1987) Recentevidence has shown that first of all the effect of ability grouping depends on a number ofconditions including whether enough time is allocated to reading instruction and how wellthe class can be managed Hence instructional time and class management are among theadditional moderators to consider for determining the merits of ability grouping (Hong andHong 2009 Hong et al 2012a b) Moreover further investigations have generated evidencethat contradicts the long-held belief that grouping undermines the interests of low-abilitystudents Quite the contrary researchers have found that grouping is beneficial for students

5INTRODUCTION

with low or medium prior abilitymdashif adequate time is provided for instruction and if the classis well managed On the other hand ability grouping appears to have a minimal impact onhigh-ability studentsrsquo literacy These results were derived from kindergarten data and maynot hold for math instruction and for higher grade levels Replications with different subpo-pulations and in different contexts enable researchers to assess the generalizability of a theory

In a third example one may hypothesize that a student is unlikely to learn algebra well inninth grade without a solid foundation in eighth-grade prealgebra One may also argue that theprogress a student made in learning eighth-grade prealgebra may not be sustained without thefurther enhancement of taking algebra in ninth grade In other words the earlier treatment(prealgebra in eighth grade) and the later treatment (algebra in ninth grade) may reinforceone other each moderates the effect of the other treatment on the final outcome (math achieve-ment at the end of ninth grade) As Hong and Raudenbush (2008) argued the cumulativeeffect of two treatments in a well-aligned sequence may exceed the sum of the benefits oftwo single-year treatments Similarly experiencing two consecutive years of inferior treat-ments such as encountering an incompetent math teacher who dampens a studentrsquos self-efficacy in math learning in eighth grade and then again in ninth grade could do more damagethan the sum of the effect of encountering an incompetent teacher only in eighth grade and theeffect of having such a teacher only in ninth grade

There has been a great amount of conceptual confusion regarding how a moderatorrelates to the treatment and the outcome On one hand many researchers have used the termsldquomoderatorsrdquo and ldquomediatorsrdquo interchangeably without understanding the crucial distinctionbetween the two An overcorrective attempt on the other hand has led to the arbitrary rec-ommendation that a moderator must occur prior to the treatment and be minimally associatedwith the treatment (James and Brett 1984 Kraemer et al 2001 2008) In Chapter 6 weclarify that in essence the causal effect of the treatment on the outcome may depend onthe moderator value A moderator can be a subpopulation identifier a contextual character-istic a concurrent treatment or a preceding or succeeding treatment In the earlier examplesage of estrogen therapy initiation and a studentrsquos prior ability are subpopulation identifiers themanageability of a class which reflects both teacher skills and peer behaviors characterizes thecontext literacy instruction time is a concurrent treatment while prealgebra is a precedingtreatment for algebra A moderator does not have to occur prior to the treatment and doesnot have to be independent of the treatment

Once a moderated causal relationship has been defined in terms of potential outcomes theresearcher then chooses an appropriate experimental design for testing the moderation theoryThe assumptions required for identifying the moderated causal relationships differ across dif-ferent designs and have implications for analyzing experimental and quasiexperimental dataRandomized block designs are suitable for examining individual or contextual characteristicsas potential moderators factorial designs enable one to determine the joint effects of two ormore concurrent treatments and sequential randomized designs are ideal for assessing thecumulative effects of consecutive treatments Multisite randomized trials constitute anadditional type of designs in which the experimental sites are often deliberately sampled torepresent a population of geographical locations or social organizations Site membershipscan be viewed as implicit moderators that summarize a host of features of a local environmentReplications of a treatment over multiple sites allow one to quantify the heterogeneity of treat-ment effects across the sites Chapters 7 and 8 are focused on statistical methods for moder-ation analyses with quasiexperimental data In particular these chapters demonstrate througha number of application examples how a nonparametric marginal mean weighting through

6 CAUSALITY IN A SOCIAL WORLD

stratification (MMWS) method can overcome some important limitations of other existingmethods

Moderation questions are not restricted to treatmentndashoutcome relationships Rather theyare prevalent in investigations of mediation and spill-over This is because heterogeneity oftreatment effects may be explained by the variation in causal mediation mechanisms acrosssubpopulations and contexts It is also because the treatment effect for a focal individualmay depend on whether there is spill-over from other individuals through social interactionsYet similar to the confusion around the concept of moderation among applied researchersthere has been a great amount of misunderstanding with regard to mediation

112 Mediated treatment effects

Questions about mediation are at the core of nearly every scientific theory In its simplestform a theory explains why treatment Z causes outcome Y by hypothesizing an intermediateprocess involving at least one mediatorM that could have been changed by the treatment andcould subsequently have an impact on the outcome A causal mediation analysis then decom-poses the total effect of the treatment on the outcome into two parts the effect transmittedthrough the hypothesized mediator called the indirect effect and the difference betweenthe total effect and the indirect effect called the direct effect The latter represents the treat-ment effect channeled through other unspecified pathways (Alwin and Hauser 1975 Baronand Kenny 1986 Duncan 1966 Shadish and Sweeney 1991) Most researchers in socialsciences have followed the convention of illustrating the direct effect and the indirect effectwith path diagrams and then specifying linear regression models postulated to represent thestructural relationships between the treatment the mediator and the outcome

In Chapter 9 we point out that the approach to defining the indirect effect and the directeffect in terms of path coefficients is often misled by oversimplistic assumptions about thestructural relationships In particular this approach typically overlooks the fact that a treat-ment may generate an impact on the outcome not only by changing the mediator value butalso by changing the mediatorndashoutcome relationship (Judd and Kenny 1981) An examplein Chapter 9 illustrates such a case An experiment randomizes students to either an experi-mental condition which provides them with study materials and encourages them to study fora test or to a control condition which provides neither study materials nor encouragement(Holland 1988 Powers and Swinton 1984) One might hypothesize that encouraging exper-imental group members to study will increase the time they spend studying a focal mediatorin this example which in turn will increase their average test scores One might furtherhypothesize that even without a change in study time providing study materials to experi-mental group members will enable them to study more effectively than they otherwise wouldConsequently the effect of time spent studying might be greater under the experimentalcondition than under the control condition Omitting the treatment-by-mediator interactioneffect will lead to bias in the estimation of the indirect effect and the direct effect

Chapter 11 describes in great detail an evaluation study contrasting a new welfare-to-workprogram emphasizing active participation in the labor force with a traditional program thatguaranteed cash assistance without a mandate to seek employment (Hong Deutsch and Hill2011) Focusing on the psychological well-being of welfare applicants who were singlemothers with preschool-aged children the researchers hypothesized that the treatment wouldincrease employment rate among the welfare recipients and that the treatment-inducedincrease in employment would likely reduce depressive symptoms under the new program

7INTRODUCTION

They also hypothesized that in contrast the same increase in employment was unlikely tohave a psychological impact under the traditional program Hence the treatment-by-mediatorinteraction effect on the outcome was an essential component of the intermediate process

We will show that by defining the indirect effect and the direct effect in terms of potentialoutcomes one can avoid invoking unwarranted assumptions about the unknown structuralrelationships (Holland 1988 Pearl 2001 Robins and Greenland 1992) The potential out-comes framework has further clarified that in a causal mediation theory the treatment and themediator are both conceivably manipulable In the aforementioned example a welfare appli-cant could be assigned at random to either the new program or the traditional one Once havingbeen assigned to one of the programs the individual might or might not be employed due tovarious structural constraints market fluctuations or other random events typically beyondher control

Because the indirect effect and the direct effect are defined in terms of potential outcomesrather than on the basis of any specific regression models it becomes possible to distinguishthe definitions of these causal effects from their identification and estimation The identifica-tion step relates a causal parameter to observable population data For example for individualsassigned at random to an experimental condition their average counterfactual outcome asso-ciated with the control condition is expected to be equal to the average observable outcome ofthose assigned to the control condition Therefore the population average causal effect can beeasily identified in a randomized experiment The estimation step then relates the sample datato the observable population quantities involved in identification while taking into account thedegree of sampling variability

A major challenge in identification is that the mediatorndashoutcome relationship tends tobe confounded by selection even if the treatment is randomized Chapter 9 reviews variousexperimental designs that have been proposed by past researchers for studying causalmediation mechanisms Chapter 10 compares the identification assumptions and the analyticprocedures across a wide range of analytic methods These discussions are followed by anintroduction of the ratio-of-mediator-probability weighting (RMPW) strategy in Chapter 11In comparison with most existing strategies for causal mediation analysis RMPW relies onrelatively fewer identification assumptions and model-based assumptions Chapters 12 and13 will show that this new strategy can be applied broadly with extensions to multilevelexperimental designs and to studies of complex mediation mechanisms involving multiplemediators

113 Spill-over effects of a treatment

It is well known in vaccination research that an unvaccinated person can benefit when mostother people in the local community are vaccinated This is viewed as a spill-over of the treat-ment effect from the vaccinated people to an unvaccinated person Similarly due to socialinteractions among individuals or groups of individuals an intervention received by somemay generate a spill-over impact on others affiliated with the same organization or connectedthrough the same network For example effective policing in one neighborhood may driveoffenders to operate in other neighborhoods In evaluating an innovative community policingprogram researchers found that a neighborhood not assigned to community policing tended tosuffer if the surrounding neighborhoods were assigned to community policing This is aninstance in which a well-intended treatment generates an unintended negative spill-over effectHowever being assigned to community policy was found particularly beneficial when thesurrounding neighborhoods also received the intervention (Verbitsky-Savitz and Raudenbush

8 CAUSALITY IN A SOCIAL WORLD

2012) In another example the impact of a school-based mentoring program targeted atstudents displaying delinquent behaviors may be enhanced for a focal student if his or herat-risk peers are assigned to the mentoring program at the same time

The previous examples challenge the ldquostable unit treatment value assumptionrdquo (SUTVA)that has been invoked in most causal inference studies This assumption states that an indi-vidualrsquos potential response to a treatment depends neither on how the treatment is assignednor on the treatment assignments of other individuals (Rubin 1980 1986) Rubin and othersbelieve that without resorting to SUTVA the causal effect of a treatment becomes hard todefine and that causal inference may become intractable However as Sobel (2006) has illus-trated with the Moving to Opportunity (MTO) experiment that offered housing vouchersenabling low-income families to move to low-poverty neighborhoods there are possible con-sequences of violating SUTVA in estimating the treatment effects on outcomes such as safetyand self-sufficiency Because social interactions among individuals may affect whether onevolunteers to participate in the study whether one moves to a low-poverty neighborhood afterreceiving the housing voucher as well as housing project residentsrsquo subjective perceptions oftheir neighborhoods the program may have a nonzero impact on the potential outcome of theuntreated As a result despite the randomization of treatment assignment the mean differencein the observed outcome between the treated units and the untreated units may be biased forthe average treatment effect Rather the observable quantity is the difference between theeffect of treating some rather than treating none for the treated and the effect of treating somerather than treating none for the untreated the latter being the pure spill-over effect for theuntreated

In making attempts to relax SUTVA researchers have proposed several alternative frame-works that incorporate possible spill-over effects (see Hong and Raudenbush 2013 for areview) Hong (2004) presented a model that involves treatments and treatment settingsA treatment setting for an individual is a local environment constituted by a set of agentsand participants along with their treatment assignments An individualrsquos potential outcomevalue under a given treatment is assumed stable when the treatment setting is fixed the poten-tial outcome may take different values when the treatment setting shifts One may investigatewhether the treatment effect depends on the treatment setting Applying this framework Hongand Raudenbush (2006) examined the effect on a childrsquos academic growth of retaining thechild in kindergarten rather than promoting the child to first grade when a relatively smallproportion of low-achieving peers in the same school are retained as opposed to when a rel-atively large proportion of the peers are retained Hudgens and Halloran (2008) presented arelated framework in which the effect on an individual of the treatment received by this indi-vidual is distinguished from the effect on the individual of the treatment received by others inthe same local community These effects can be identified if communities are assigned atrandom to different treatment assignment strategies (such as retaining a large proportion oflow-achieving students as opposed to retaining a small proportion of such students) and sub-sequently individuals within a community are randomized for treatment Chapter 14 reviewsthese frameworks and discusses identification and estimation strategies

Social contagion may also serve as an important channel through which the effect of anintervention is transmitted For example a school-wide intervention may reduce aggressivebehaviors and thereby improve studentsrsquo psychological well-being by improving the qualityof interpersonal relationships in other classes as well as in onersquos own class This is becausechildren interact not simply with their classmates but also with those from other classes in thehallways or on the playground (VanderWeele et al 2013) In this case the spill-over betweenclasses becomes a part of the mediation mechanism In a study of student mentoring whether

9INTRODUCTION

an individual actually participated in the mentoring program despite the initial treatmentassignment is typically viewed as a primary mediator The proportion of onersquos peers that par-ticipated in the program may act as a second mediator A treatment may also exert its impactpartly through regrouping individuals and thereby changing peer composition (Hong andNomi 2012) Chapter 15 discusses analytic strategies for detecting spill-over as a part ofthe causal mediation mechanism

12 Weighting methods for causal inference

In most behavioral and social science applications major methodological challenges arise dueto the selection of treatments in a quasiexperimental study and the selection of mediator valuesin experimental and quasiexperimental studies Statistical methods most familiar to research-ers in social sciences are often inadequate for causal inferences with regard to multivaluedtreatments moderation mediation and spill-over The book offers a major revision to under-standing of causality in the social world by introducing two complementary weighting stra-tegies both featuring nonparametric approaches to estimating these causal effects The newweighting methods greatly simplify model specifications while enhancing the robustness ofresults by minimizing the reliance on some key assumptions

The propensity score-basedMMWSmethod removes selection bias associated with a largenumber of covariates by equating the pretreatment composition between treatment groups(Hong 2010a 2012 Huang et al 2005) Unlike propensity score matching and stratificationthat are mostly restricted to evaluations of binary treatments the MMWS method is flexiblefor evaluating binary and multivalued treatments by approximating a completely randomizedexperiment In evaluating whether the treatment effects differ across subpopulations definedby individual characteristics or treatment settings researchers may assign weights within eachsubpopulation in order to approximate a randomized block design To investigate whetherone treatment moderates the effect of another concurrent treatment researchers may assignweights to the data to approximate a factorial randomized design The method can also beused to assess whether the effect of an initial treatment is amplified or weakened by a subse-quent treatment or to identify an optimal treatment sequence through approximating a sequen-tial randomized experiment Even though such analyses can similarly be conducted throughinverse-probability-of-treatment weighting (IPTW) that has been increasingly employed inepidemiological research (Hernaacuten Brumback and Robins 2000 Robins Hernaacuten andBrumback 2000) IPTW is known for bias and imprecision in estimation especially whenthe propensity score models are misspecified in their functional forms (Hong 2010a Kangand Schafer 2007 Schafer and Kang 2008 Waernbaum 2012) In contrast the nonparamet-ric MMWS method displays a relatively high level of robustness despite such misspecifica-tions and also gains efficiency as indicated by simulation results (Hong 2010a)

To study causal mediation mechanisms the RMPW method decomposes the total effectof a treatment into an ldquoindirect effectrdquo transmitted through a specific mediator and a ldquodirecteffectrdquo representing unspecified mechanisms In contrast with most existing methods formediation analysis the RMPW-adjusted outcome model is extremely simple and is nonpara-metric in nature It generates estimates of the causal effects along with their sampling errorswhile adjusting for pretreatment covariates that confound the mediatorndashoutcome relationshipsthrough weighting The method applies regardless of the distribution of the outcome the dis-tribution of the mediator or the functional relationship between the outcome and the mediator

10 CAUSALITY IN A SOCIAL WORLD

Page 4: Thumbnail - download.e-bookshelf.de...4.2.4 IPTW for bias removal 84 4.3 MMWS 86 4.3.1 Theoretical rationale 86 ... 8.3.4.2 Growth model in the absence of confounding 200 8.3.4.3 Weighted

This edition first published 2015copy 2015 John Wiley amp Sons Ltd

Registered OfficeJohn Wiley amp Sons Ltd The Atrium Southern Gate Chichester West Sussex PO19 8SQ United Kingdom

For details of our global editorial offices for customer services and for information about how to apply for permissionto reuse the copyright material in this book please see our website at wwwwileycom

The right of the author to be identified as the author of this work has been asserted in accordance with the CopyrightDesigns and Patents Act 1988

All rights reserved No part of this publication may be reproduced stored in a retrieval system or transmitted in anyform or by any means electronic mechanical photocopying recording or otherwise except as permitted by theUK Copyright Designs and Patents Act 1988 without the prior permission of the publisher

Wiley also publishes its books in a variety of electronic formats Some content that appears in print may not beavailable in electronic books

Designations used by companies to distinguish their products are often claimed as trademarks All brand names andproduct names used in this book are trade names service marks trademarks or registered trademarks of their respectiveowners The publisher is not associated with any product or vendor mentioned in this book

Limit of LiabilityDisclaimer of Warranty While the publisher and author have used their best efforts in preparing this bookthey make no representations or warranties with respect to the accuracy or completeness of the contents of this bookand specifically disclaim any implied warranties of merchantability or fitness for a particular purpose It is sold on theunderstanding that the publisher is not engaged in rendering professional services and neither the publisher nor the authorshall be liable for damages arising herefrom If professional advice or other expert assistance is required the services ofa competent professional should be sought

Library of Congress Cataloging-in-Publication Data applied for

ISBN 9781118332566

A catalogue record for this book is available from the British Library

Set in 1012pt Times by SPi Global Pondicherry India

1 2015

Contents

Preface xv

Part I Overview 1

1 Introduction 311 Concepts of moderation mediation and spill-over 3

111 Moderated treatment effects 5112 Mediated treatment effects 7113 Spill-over effects of a treatment 8

12 Weighting methods for causal inference 1013 Objectives and organization of the book 1114 How is this book situated among other publications on related topics 12

2 Review of causal inference concepts and methods 1821 Causal inference theory 18

211 Attributes versus causes 18212 Potential outcomes and individual-specific causal effects 19213 Inference about population average causal effects 22

2131 Prima facie effect 242132 Ignorability assumption 25

22 Applications to Lordrsquos paradox and Simpsonrsquos paradox 27221 Lordrsquos paradox 27222 Simpsonrsquos paradox 31

23 Identification and estimation 34231 Selection bias 35232 Sampling bias 35233 Estimation efficiency 36

Appendix 21 Potential bias in a prima facie effect 36Appendix 22 Application of the causal inference theory to Lordrsquos paradox 37

3 Review of causal inference designs and analytic methods 4031 Experimental designs 40

311 Completely randomized designs 40

312 Randomized block designs 41313 Covariance adjustment for improving efficiency 43314 Multilevel experimental designs 43

32 Quasiexperimental designs 44321 Nonequivalent comparison group designs 44322 Other quasiexperimental designs 45

33 Statistical adjustment methods 46331 ANCOVA and multiple regression 46

3311 ANCOVA for removing selection bias 463312 Potential pitfalls of ANCOVA with a vast between-group

difference 473313 Bias due to model misspecification 48

332 Matching and stratification 50333 Other statistical adjustment methods 51

3331 The IV method 513332 DID analysis 54

34 Propensity score 55341 What is a propensity score 56342 Balancing property of the propensity score 57343 Pooling conditional treatment effect estimate Matching

stratification and covariance adjustment 603431 Propensity score matching 613432 Propensity score stratification 623433 Covariance adjustment for the propensity score 663434 Sensitivity analysis 66

Appendix 3A Potential bias due to the omission of treatment-by-covariateinteraction 70

Appendix 3B Variable selection for the propensity score model 71

4 Adjustment for selection bias through weighting 7641 Weighted estimation of population parameters in survey sampling 77

411 Simple random sample 77412 Proportionate sample 78413 Disproportionate sample 79

42 Weighting adjustment for selection bias in causal inference 80421 Experimental result 81422 Quasiexperimental result 81423 Sample weight for bias removal 82424 IPTW for bias removal 84

43 MMWS 86431 Theoretical rationale 86

4311 MMWS for a discrete propensity score 874312 MMWS for a continuous propensity score 884313 MMWS for estimating the treatment effect on the treated 89

432 MMWS analytic procedure 91433 Inherent connection and major distinctions between MMWS

and IPTW 93

vi CONTENTS

Appendix 4A Proof of MMWS-adjusted mean observed outcome being unbiasedfor the population average potential outcome 95

Appendix 4B Derivation of MMWS for estimating the treatment effecton the treated 96

Appendix 4C Theoretical equivalence of MMWS and IPTW 97Appendix 4D Simulations comparing MMWS and IPTW under misspecifications

of the functional form of a propensity score model 97

5 Evaluations of multivalued treatments 10051 Defining the causal effects of multivalued treatments 10052 Existing designs and analytic methods for evaluating multivalued

treatments 102521 Experimental designs and analysis 102

5211 Randomized experiments with multipletreatment arms 102

5212 Identification under ignorability 1025213 ANOVA 103

522 Quasiexperimental designs and analysis 1055221 ANCOVA and multiple regression 1055222 Propensity score-based adjustment 1085223 Other adjustment methods 112

53 MMWS for evaluating multivalued treatments 112531 Basic rationale 113532 Analytic procedure 114

5321 MMWS for a multinomial treatment measure 1155322 MMWS for an ordinal treatment measure 120

533 Identification assumptions 12154 Summary 123Appendix 5A Multiple IV for evaluating multivalued treatments 124

Part II Moderation 127

6 Moderated treatment effects concepts and existing analytic methods 12961 What is moderation 129

611 Past discussions of moderation 1306111 Purpose of moderation research 1306112 What qualifies a variable as a moderator 132

612 Definition of moderated treatment effects 1336121 Treatment effects moderated by individual or contextual

characteristics 1336122 Joint effects of concurrent treatments 134

62 Experimental designs and analytic methods for investigatingexplicit moderators 136621 Randomized block designs 137

6211 Identification assumptions 1376212 Two-way ANOVA 1386213 Multiple regression 139

viiCONTENTS

622 Factorial designs 1406221 Identification assumptions 1406222 Analytic strategies 142

63 Existing research designs and analytic methods for investigatingimplicit moderators 142631 Multisite randomized trials 143

6311 Causal parameters and identification assumptions 1446312 Analytic strategies 145

632 Principal stratification 149Appendix 6A Derivation of bias in the fixed-effects estimator when the

treatment effect is heterogeneous in multisite randomized trials 151Appendix 6B Derivation of bias in the mixed-effects estimator when the

probability of treatment assignment varies across sites 153Appendix 6C Derivation and proof of the population weight applied to

mixed-effects models for eliminating bias in multisiterandomized trials 153

7 Marginal mean weighting through stratification for investigatingmoderated treatment effects 15971 Existing methods for moderation analyses with quasiexperimental data 159

711 Analysis of covariance and regression-based adjustment 1617111 Treatment effects moderated by subpopulation membership 1617112 Treatment effects moderated by a concurrent treatment 164

712 Propensity score-based adjustment 1657121 Propensity score matching and stratification 1667122 Inverse-probability-of-treatment weighting 167

72 MMWS estimation of treatment effects moderated by individual orcontextual characteristics 168721 Application example 170722 Analytic procedure 170

73 MMWS estimation of the joint effects of concurrent treatments 174731 Application example 174732 Analytic procedure 175733 Joint treatment effects moderated by individual or contextual

characteristics 179

8 Cumulative effects of time-varying treatments 18581 Causal effects of treatment sequences 186

811 Application example 186812 Causal parameters 187

8121 Time-varying treatments 1878122 Time-varying potential outcomes 1878123 Causal effects of 2-year treatment sequences 1878124 Causal effects of multiyear treatment sequences 190

82 Existing strategies for evaluating time-varying treatments 190821 The endogeneity problem in nonexperimental data 190822 SEM 191

viii CONTENTS

823 Fixed-effects econometric models 192824 Sequential randomization 192825 Dynamic treatment regimes 193826 Marginal structural models and structural nested models 194

83 MMWS for evaluating 2-year treatment sequences 195831 Sequential ignorability 195832 Propensity scores 196833 MMWS computation 197

8331 MMWS adjustment for year 1 treatment selection 1978332 MMWS adjustment for two-year treatment

sequence selection 198834 Two-year growth model specification 199

8341 Growth model in the absence of treatment 2008342 Growth model in the absence of confounding 2008343 Weighted 2-year growth model 202

84 MMWS for evaluating multiyear sequences of multivalued treatments 204841 Sequential ignorability of multiyear treatment sequences 204842 Propensity scores for multiyear treatment sequences 204843 MMWS computation 205844 Weighted multiyear growth model 205845 Issues of sample size 206

85 Conclusion 207Appendix 8A A saturated model for evaluating multivalued treatments

over multiple time periods 207

Part III Mediation 211

9 Concepts of mediated treatment effects and experimental designsfor investigating causal mechanisms 21391 Introduction 21492 Path coefficients 21593 Potential outcomes and potential mediators 216

931 Controlled direct effects 217932 Controlled treatment-by-mediator interaction effect 217

94 Causal effects with counterfactual mediators 219941 Natural direct effect 219942 Natural indirect effect 220943 Natural treatment-by-mediator interaction effect 220944 Unstable unit treatment value 221

95 Population causal parameters 222951 Population average natural direct effect 224952 Population average natural indirect effect 225

96 Experimental designs for studying causal mediation 225961 Sequentially randomized designs 228962 Two-phase experimental designs 228963 Three- and four-treatment arm designs 230964 Experimental causal-chain designs 231

ixCONTENTS

965 Moderation-of-process designs 231966 Augmented encouragement designs 232967 Parallel experimental designs and parallel encouragement designs 232968 Crossover experimental designs and crossover encouragement

designs 233969 Summary 234

10 Existing analytic methods for investigating causal mediation mechanisms 238101 Path analysis and SEM 239

1011 Analytic procedure for continuous outcomes 2391012 Identification assumptions 2421013 Analytic procedure for discrete outcomes 245

102 Modified regression approach 2461021 Analytic procedure for continuous outcomes 2461022 Identification assumptions 2471023 Analytic procedure for binary outcomes 248

103 Marginal structural models 2501031 Analytic procedure 2501032 Identification assumptions 252

104 Conditional structural models 2521041 Analytic procedure 2521042 Identification assumptions 253

105 Alternative weighting methods 2541051 Analytic procedure 2541052 Identification assumptions 256

106 Resampling approach 2561061 Analytic procedure 2561062 Identification assumptions 257

107 IV method 2571071 Rationale and analytic procedure 2571072 Identification assumptions 258

108 Principal stratification 2591081 Rationale and analytic procedure 2591082 Identification assumptions 260

109 Sensitivity analysis 2611091 Unadjusted confounding as a product of hypothetical

regression coefficients 2611092 Unadjusted confounding reflected in a hypothetical

correlation coefficient 2621093 Limitations when the selection mechanism differs

by treatment 2641094 Other sensitivity analyses 265

1010 Conclusion 26510101 The essentiality of sequential ignorability 26510102 Treatment-by-mediator interactions 26610103 Homogeneous versus heterogeneous causal effects 26610104 Model-based assumptions 266

x CONTENTS

Appendix 10A Bias in path analysis estimation due to the omission oftreatment-by-mediator interaction 267

11 Investigations of a simple mediation mechanism 273111 Application example national evaluation of welfare-to-work strategies 274

1111 Historical context 2741112 Research questions 2751113 Causal parameters 2751114 NEWWS Riverside data 277

112 RMPW rationale 2771121 RMPW in a sequentially randomized design 278

11211 E[Y(0 M(0))] 27911212 E[Y(1 M(1))] 28011213 E[Y(1 M(0))] 28011214 E[Y(0 M(1))] 28111215 Nonparametric outcome model 282

1122 RMPW in a sequentially randomized block design 2831123 RMPW in a standard randomized experiment 2851124 Identification assumptions 286

113 Parametric RMPW procedure 287114 Nonparametric RMPW procedure 290115 Simulation results 292

1151 Correctly specified propensity score models 2921152 Misspecified propensity score models 2941153 Comparisons with path analysis and IV results 294

116 Discussion 2951161 Advantages of the RMPW strategy 2951162 Limitations of the RMPW strategy 295

Appendix 11A Causal effect estimation through the RMPW procedure 296Appendix 11B Proof of the consistency of RMPW estimation 297

12 RMPW extensions to alternative designs and measurement 301121 RMPW extensions to mediators and outcomes of alternative distributions 301

1211 Extensions to a multicategory mediator 30212111 Parametric RMPW procedure 30212112 Nonparametric RMPW procedure 304

1212 Extensions to a continuous mediator 3041213 Extensions to a binary outcome 306

122 RMPW extensions to alternative research designs 3061221 Extensions to quasiexperimental data 3071222 Extensions to data from cluster randomized trials 308

12221 Application example and research questions 30912222 Estimation of the total effect 31012223 RMPW analysis of causal mediation mechanisms 31012224 Identification assumptions 31212225 Contrast with multilevel path analysis and SEM 31212226 Contrast with multilevel prediction models 313

xiCONTENTS

1223 Extensions to data from multisite randomized trials 31312231 Research questions and causal parameters 31412232 Estimation of the total effect and its between-site variation 31512233 RMPW analysis of causal mediation mechanisms 31612234 Identification assumptions 31912235 Contrast with multilevel path analysis and SEM 319

123 Alternative decomposition of the treatment effect 321

13 RMPW extensions to studies of complex mediation mechanisms 325131 RMPW extensions to moderated mediation 325

1311 RMPW analytic procedure for estimating and testingmoderated mediation 326

1312 Path analysisSEM approach to analyzing moderatedmediation 327

1313 Principal stratification and moderated mediation 328132 RMPW extensions to concurrent mediators 328

1321 Treatment effect decomposition 32913211 Treatment effect decomposition without

between-mediator interaction 32913212 Treatment effect decomposition with

between-mediator interaction 3311322 Identification assumptions 3331323 RMPW procedure 333

13231 Estimating E[Y(1M1(1)M2(0))] 33513232 Estimating E[Y(1M1(0)M2(0))] 33513233 Causal effect estimation with noninteracting

concurrent mediators 33613234 Estimating E[Y(1M1(0)M2(1))] 33713235 Causal effect estimation with interacting concurrent

mediators 3371324 Contrast with the linear SEM approach 3381325 Contrast with the multivariate IV approach 339

133 RMPW extensions to consecutive mediators 3401331 Treatment effect decomposition 341

13311 Natural direct effect of the treatment on the outcome 34213312 Natural indirect effect mediated by M1 only 34213313 Natural indirect effect mediated by M2 only 34213314 Natural indirect effect mediated by an M1-by-M2

interaction 34313315 Treatment-by-mediator interactions 343

1332 Identification assumptions 3451333 RMPW procedure 347

13331 Estimating E[Y(1M1(0)M2(0M1(0)))] 34713332 Estimating E[Y(1M1(1)M2(0M1(0)))] 34913333 Estimating E[Y(1M1(0)M2(1M1(1)))] 34913334 Estimating E[Y(0M1(1)M2(0M1(0)))] and

E[Y(0M1(0)M2(1M1(1)))] 351

xii CONTENTS

1334 Contrast with the linear SEM approach 3531335 Contrast with the sensitivity-based estimation of bounds for

causal effects 354134 Discussion 355Appendix 13A Derivation of RMPW for estimating population average

counterfactual outcomes of two concurrentmediators 355

Appendix 13B Derivation of RMPW for estimating population averagecounterfactual outcomes of consecutivemediators 358

Part IV Spill-over 363

14 Spill-over of treatment effects concepts and methods 365141 Spill-over A nuisance a trifle or a focus 365142 Stable versus unstable potential outcome values An example

from agriculture 367143 Consequences for causal inference when spill-over is overlooked 369144 Modified framework of causal inference 371

1441 Treatment settings 3711442 Simplified characterization of treatment settings 3731443 Causal effects of individual treatment assignment and of peer

treatment assignment 375145 Identification Challenges and solutions 376

1451 Hypothetical experiments for identifying average treatmenteffects in the presence of social interactions 376

1452 Hypothetical experiments for identifying the impact ofsocial interactions 380

1453 Application to an evaluation of kindergarten retention 382146 Analytic strategies for experimental and quasiexperimental data 384

1461 Estimation with experimental data 3841462 Propensity score stratification 3851463 MMWS 386

147 Summary 387

15 Mediation through spill-over 391151 Definition of mediated effects through spill-over in a cluster

randomized trial 3931511 Notation 3931512 Treatment effect mediated by a focal individualrsquos

compliance 3941513 Treatment effect mediated by peersrsquo compliance through

spill-over 3941514 Decomposition of the total treatment effect 395

152 Identification and estimation of the spill-over effect in a clusterrandomized design 3951521 Identification in an ideal experiment 395

xiiiCONTENTS

1522 Identification when the mediators are not randomized 3981523 Estimation of mediated effects through spill-over 400

153 Definition of mediated effects through spill-over in a multisite trial 4021531 Notation 4021532 Treatment effect mediated by a focal individualrsquos compliance 4041533 Treatment effect mediated by peersrsquo compliance

through spill-over 4041534 Direct effect of individual treatment assignment on the outcome 4051535 Direct effect of peer treatment assignment on the outcome 4051536 Decomposition of the total treatment effect 405

154 Identification and estimation of spill-over effects in a multisite trial 4061541 Identification in an ideal experiment 4071542 Identification when the mediators are not randomized 4091543 Estimation of mediated effects through spill-over 410

15431 Estimating E[Y(1pM(p)Mminus(p))] andE[Y(000Mminus(0))] 410

15432 Estimating E[Y(1p0Mminus(p))] 41115433 Estimating E[Y(1p0Mminus(0))] 41215434 Estimating E[Y(0p0Mminus(0))] 412

155 Consequences of omitting spill-over effects in causal mediation analyses 4121551 Biased inference in a cluster randomized trial 4131552 Biased inference in a multisite randomized trial 4131553 Biased inference of the local average treatment effect 415

156 Quasiexperimental application 416157 Summary 419Appendix 151 Derivation of the weight for estimating the population

average counterfactual outcome E[Y(1 p 0Mminus( p))] 419Appendix 152 Derivation of bias in the ITT effect due to the omission of

spill-over effects 420

Index 423

xiv CONTENTS

Preface

A scientific mind in training seeks comprehensive descriptions of every interesting phenom-enon In such a mind data are to be pieced together for comparisons contrasts and eventuallyinferences that are required for understanding causes and effects that may have led to thephenomenon of interest Yet without disciplined reasoning a causal inference would slip intothe rut of a ldquocasual inferencerdquo as easily as making a typographical error

As a doctoral student thirsting for rigorous methodological training at the University ofMichigan I was very fortunate to be among the first group of students on the campus attendinga causal inference seminar in 2000 jointly taught by Yu Xie from the Department of Sociol-ogy Susan Murphy from the Department of Statistics and Stephen Raudenbush from theSchool of Education The course was unique in both content and pedagogy The instructorsput together a bibliography drawn from the past and current causal inference literature origi-nated from multiple disciplines In the spirit of ldquoreciprocal teachingrdquo the students were delib-erately organized into small groups each representing a mix of disciplinary backgrounds andtook turns to teach the weekly readings under the guidance of a faculty instructor This inval-uable experience revealed to me that the pursuit of causal inference is a multidisciplinaryendeavor In the rest of my doctoral study I continued to be immersed in an extraordinaryinterdisciplinary community in the form of the Quantitative Methodology Program (QMP)seminars led by the above three instructors who were then joined by Ben Hansen(Statistics) Richard Gonzalez (Psychology) Roderick Little (Biostatistics) Jake Bowers(Political Science) and many others I have tried whenever possible to carry the same spiritinto my own teaching of causal inference courses at the University of Toronto and now at theUniversity of Chicago In these courses communicating understandings of causal problemsacross disciplinary boundaries has been particularly challenging but also stimulating and grat-ifying for myself and for students from various departments and schools

Under the supervision of Stephen Raudenbush I took a first stab at the conceptual frame-work of causal inference in my doctoral dissertation by considering peer spill-over in schoolsettings From that point on I have organized my methodological research to tackle some ofthe major obstacles to causal inferences in policy and program evaluations My workaddresses issues including (i) how to conceptualize and evaluate the causal effects of educa-tional treatments when studentsrsquo responses to alternative treatments depend on various fea-tures of the organizational settings including peer composition (ii) how to adjust forselection bias in evaluating the effects of concurrent and consecutive multivalued treatmentsand (iii) how to conceptualize and analyze causal mediation mechanisms My methodological

research has been driven by the substantive interest in understanding the role of socialinstitutionsmdashschools in particularmdashin shaping human development especially during theages of fast growth (eg childhood adolescence and early adulthood) I have shared meth-odological advances with a broad audience in social sciences through applying the causalinference methods to prominent substantive issues such as grade retention within-class group-ing services for English language learners and welfare-to-work strategies These illustrationsare used extensively in this book

Among social scientists awareness of methodological challenges in causal investigationsis higher than ever before In response to this rising demand causal inference is becoming oneof the most productive scholarly fields in the past decade I have had the benefit of followingthe work and exchanging ideas with some fellow methodologists who are constantly contri-buting new thoughts and making breakthroughs These include Daniel Almirall HowardBloom Tom Cook Michael Elliot Ken Frank Ben Hansen James Heckman JenniferHill Martin Huber Kosuke Imai Booil Jo John R Lockwood David MacKinnon DanielMcCaffrey Luke Miratrix Richard Murnane Derek Neal Lindsey Page Judea PearlStephen Raudenbush Sean Reardon Michael Seltzer Youngyun Shin Betsy SinclairMichael Sobel Peter Steiner Elizabeth Stuart Eric Thetchen Tchetchen Tyler VanderWeeleand Kazuo Yamaguchi I am grateful to Larry Hedges who offered me the opportunity of guestediting the Journal of Research in Educational Effectiveness special issue on the statisticalapproaches to studying mediator effects in education research in 2012 Many of the colleagueswhom I mentioned above contributed to the open debate in the special issue by either propos-ing or critically examining a number of innovative methods for studying causal mechanismsI anticipate that many of them are or will soon be writing their own research monographs oncausal inference if they have not already done so Therefore readers of this book are stronglyrecommended to browse past and future titles by these and other authors for alternative per-spectives and approaches My own understanding of course will continue to evolve as well inthe coming years

One has to be ambitious to stay upfront in substantive areas as well as in methodology Thebest strategy apparently is to learn from substantive experts During different phases of mytraining and professional career I have had the opportunities to work with David K CohenBrian Rowan Deborah Ball Carl Corter Janette Pelletier Takako Nomi Esther Geva DavidFrancis Stephanie Jones Joshua Brown and Heather D Hill Many of my substantive insightswere derived from conversations with these mentors and colleagues I am especially gratefulto Bob Granger former president of the William T Grant Foundation who reassured me thatldquoBy staying a bit broad you will learn a lot and many fields will benefit from your workrdquo

Like many other single-authored books this one is built on the generous contributions ofstudents and colleagues who deserve special acknowledgement Excellent research assistancefrom Jonah Deutsch Joshua Gagne Rachel Garrett Yihua Hong Xu Qin Cheng Yang andBing Yu was instrumental in the development and implementation of the new methods pre-sented in this book Richard Congdon a renowned statistical software programmer broughtrevolutionary ideas to interface designs in software development with the aim of facilitatingusersrsquo decision-making in a series of causal analyses RichardMurnane Stephen Raudenbushand Michael Sobel carefully reviewed and critiqued multiple chapters of the manuscript draftAdditional comments and suggestions on earlier drafts came fromHoward Bloom Ken FrankBen Hansen Jennifer Hill Booil Jo Ben Kelcey David MacKinnon and Fan Yang TereseSchwartzman provided valuable assistance in manuscript preparation The writing of thisbook was supported by a Scholars Award from the William T Grant Foundation and an

xvi PREFACE

Institute of Education Sciences (IES) Statistical and Research Methodology in EducationGrant from the US Department of Education All the errors in the published form are ofcourse the sole responsibility of the author

Project Editors Richard Davis Prachi Sinha Sahay and Liz Wingett and Assistant EditorHeather Kay at Wiley and project Managers P Jayapriya and R Jayavel at SPi Global havebeen wonderful to work with throughout the manuscript preparation and final production ofthe book Debbie Jupe Commissioning Editor at Wiley was remarkably effective in initiatingthe contact with me and in organizing anonymous reviews of the book proposal and of themanuscript These constructive reviews have shaped the book in important ways I cannotagree more with one of the reviewers that ldquocausality cannot be established on a pure statisticalgroundrdquo The book therefore highlights the importance of substantive knowledge and researchdesigns and places a great emphasis on clarifying and evaluating assumptions required foridentifying causal effects in the context of each application Much caution is raised againstpossible misusage of statistical techniques for analyzing causal relationships especiallywhen data are inadequate Yet I maintain that improved statistical procedures along withimproved research designs would greatly enhance our ability in attempt to empiricallyexamine well-articulated causal theories

xviiPREFACE

Part I

OVERVIEW

1

Introduction

According to an ancient Chinese fable a farmer who was eager to help his crops grow wentinto his field and pulled each seedling upward After exhausting himself with the work heannounced to his family that they were going to have a good harvest only to find the nextmorning that the plants had wilted and died Readers with minimal agricultural knowledgemay immediately point out the following the farmerrsquos intervention theory was based on acorrect observation that crops that grow taller tend to produce more yield Yet his hypothesisreflects a false understanding of the cause and the effectmdashthat seedlings pulled to be tallerwould yield as much as seedlings thriving on their own

In their classic Design and Analysis of Experiments Hinkelmann and Kempthorne (1994updated version of Kempthorne 1952) discussed two types of science descriptive science andthe development of theory These two types of science are interrelated in the following senseobservations of an event and other related events often selected and classified for descriptionby scientists naturally lead to one or more explanations that we call ldquotheoretical hypothesesrdquowhich are then screened and falsified by means of further observations experimentation andanalyses (Popper 1963) The experiment of pulling seedlings to be taller was costly but didserve the purpose of advancing this farmerrsquos knowledge of ldquowhat does not workrdquo To developa successful intervention in this case would require a series of empirical tests of explicittheories identifying potential contributors to crop growth This iterative process graduallydeepens our knowledge of the relationships between supposed causes and effectsmdashthatis causalitymdashand may eventually increase the success of agricultural medical and socialinterventions

11 Concepts of moderation mediation and spill-over

Although the story of the ancient farmer is fictitious numerous examples can be found in thereal world in which well-intended interventions fail to produce the intended benefits or inmany cases even lead to unintended consequences ldquoInterventionsrdquo and ldquotreatmentsrdquo usedinterchangeably in this book broadly refer to actions taken by agents or circumstances expe-rienced by an individual or groups of individuals Interventions are regularly seen in educa-tion physical and mental health social services business politics and law enforcement In an

Causality in a Social World Moderation Meditation and Spill-over First Edition Guanglei Hongcopy 2015 John Wiley amp Sons Ltd Published 2015 by John Wiley amp Sons Ltd

education intervention for example teachers are typically the agents who deliver a treatmentto students while the impact of the treatment on student outcomes is of ultimate causalinterest Some educational practices such as ldquoteaching to the testrdquo have been criticized tobe nearly as counterproductive as the attempt of helping seedlings grow by pulling themupward ldquoInterventionsrdquo and ldquotreatmentsrdquo under consideration do not exclude undesiredexperiences such as exposure to poverty abuse crime or bereavement A treatment plannedor unplanned becomes a focus of research if there are theoretical reasons to anticipate itsimpact positive or negative on the well-being of individuals who are embedded in socialsettings including families classrooms schools neighborhoods and workplaces

In social science research in general and in policy and program evaluations in particularquestions concerning whether an intervention works and if so which version of the interven-tion works for whom under what conditions and why are key to the advancement of scien-tific and practical knowledge Although most empirical investigations in the social sciencesconcentrate on the average effect of a treatment for a specific population as opposed to theabsence of such a treatment (ie the control condition) in-depth theoretical reasoning withregard to how the causal effect is generated substantiated by compelling empirical evidenceis crucial for advancing scientific understanding

First when there are multiple versions or different dosages of the treatment or when thereare multiple versions of the control condition a binary divide between ldquothe treatmentrdquo andldquothe controlrdquo may not be as informative as fine-grained comparisons across for exampleldquotreatment version Ardquo ldquotreatment version Brdquo ldquocontrol version Ardquo and ldquocontrol versionBrdquo For example expanding the federally funded Head Start program to poor children isexpected to generate a greater benefit when few early childhood education alternatives areavailable (call it ldquocontrol version Ardquo) than when there is an abundance of alternatives includ-ing state-sponsored preschool programs (call it ldquocontrol version Brdquo)

Second the effect of an intervention will likely vary among individuals or across socialsettings A famous example comes from medical research the well-publicized cardiovascularbenefits of initiating estrogen therapy during menopause were contradicted later by experi-mental findings that the same therapy increased postmenopausal womenrsquos risk for heartattacks The effect of an intervention may also depend on the provision of some other con-current or subsequent interventions Such heterogeneous effects are often characterized asmoderated effects in the literature

Third alternative theories may provide competing explanations for the causal mechan-isms that is the processes through which the intervention produces its effect A theoreticalconstruct characterizing the hypothesized intermediate process is called a mediator of theintervention effect The fictitious farmer never developed an elaborate theory as to whatcaused some seedlings to surpass others in growth Once scientists revealed the causal rela-tionship between access to chemical nutrients in soil and plant growth wide applications ofchemically synthesized fertilizers finally led to a major increase in crop production

Finally it is well known in agricultural experiments that a new type of fertilizer applied toone plot may spill-over to the next plot Because social connections among individuals areprevalent within organizations or through networks an individualrsquos response to the treatmentmay similarly depend on the treatment for other individuals in the same social setting whichmay lead to possible spill-overs of intervention effects among individual human beings

Answering questions with regard to moderation mediation and spill-over poses majorconceptual and analytic challenges To date psychological research often presents well-articulated theories of causal mechanisms relating stimuli to responses Yet researchers often

4 CAUSALITY IN A SOCIAL WORLD

lack rigorous analytic strategies for empirically screening competing theories explaining theobserved effect Sociologists have keen interest in the spill-over of treatment effects transmit-ted through social interactions yet have produced limited evidence quantifying such effectsAs many have pointed out in general terminological ambiguity and conceptual confusionhave been prevalent in the published applied research (Holmbeck 1997 Kraemer et al2002 2008)

A new framework for conceptualizing moderated mediated and spill-over effects hasemerged relatively recently in the statistics and econometrics literature on causal inference(eg Abbring and Heckman 2007 Frangakis and Rubin 2002 Heckman and Vytlacil2007a b Holland 1988 Hong and Raudenbush 2006 Hudgens and Halloran 2008Jo 2008 Pearl 2001 Robins and Greenland 1992 Sobel 2008) The potential for furtherconceptual and methodological development and for broad applications in the field of behav-ioral and social sciences promises to greatly advance the empirical basis of our knowledgeabout causality in the social world

This book clarifies theoretical concepts and introduces innovative statistical strategies forinvestigating the average effects of multivalued treatments moderated treatment effectsmediated treatment effects and spill-over effects in experimental or quasiexperimental dataDefining individual-specific and population average treatment effects in terms of potentialoutcomes the book relates the mathematical forms to the substantive meanings of moderatedmediated and spill-over effects in the context of application examples It also explicates andevaluates identification assumptions and contrasts innovative statistical strategies with con-ventional analytic methods

111 Moderated treatment effects

It is hard to accept the assumption that a treatment would produce the same impact for everyindividual in every possible circumstance Understanding the heterogeneity of treatmenteffects therefore is key to the development of causal theories For example some studiesreported that estrogen therapy improved cardiovascular health among women who initiatedits use during menopause According to a series of other studies however the use of estrogentherapy increased postmenopausal womenrsquos risk for heart attacks (Grodstein et al 19962000 Writing Group for the Womenrsquos Health Initiative Investigators 2002) The sharp con-trast of findings from these studies led to the hypothesis that age of initiation moderates theeffect of estrogen therapy on womenrsquos health (Manson and Bassuk 2007 Rossouw et al2007) Revelations of the moderated causal relationship greatly enrich theoretical understand-ing and in this case directly inform clinical practice

In another example dividing students by ability into small groups for reading instructionhas been controversial for decades Many believed that the practice benefits high-ability stu-dents at the expense of their low-ability peers and exacerbates educational inequality (Grantand Rothenberg 1986 Rowan and Miracle 1983 Trimble and Sinclair 1987) Recentevidence has shown that first of all the effect of ability grouping depends on a number ofconditions including whether enough time is allocated to reading instruction and how wellthe class can be managed Hence instructional time and class management are among theadditional moderators to consider for determining the merits of ability grouping (Hong andHong 2009 Hong et al 2012a b) Moreover further investigations have generated evidencethat contradicts the long-held belief that grouping undermines the interests of low-abilitystudents Quite the contrary researchers have found that grouping is beneficial for students

5INTRODUCTION

with low or medium prior abilitymdashif adequate time is provided for instruction and if the classis well managed On the other hand ability grouping appears to have a minimal impact onhigh-ability studentsrsquo literacy These results were derived from kindergarten data and maynot hold for math instruction and for higher grade levels Replications with different subpo-pulations and in different contexts enable researchers to assess the generalizability of a theory

In a third example one may hypothesize that a student is unlikely to learn algebra well inninth grade without a solid foundation in eighth-grade prealgebra One may also argue that theprogress a student made in learning eighth-grade prealgebra may not be sustained without thefurther enhancement of taking algebra in ninth grade In other words the earlier treatment(prealgebra in eighth grade) and the later treatment (algebra in ninth grade) may reinforceone other each moderates the effect of the other treatment on the final outcome (math achieve-ment at the end of ninth grade) As Hong and Raudenbush (2008) argued the cumulativeeffect of two treatments in a well-aligned sequence may exceed the sum of the benefits oftwo single-year treatments Similarly experiencing two consecutive years of inferior treat-ments such as encountering an incompetent math teacher who dampens a studentrsquos self-efficacy in math learning in eighth grade and then again in ninth grade could do more damagethan the sum of the effect of encountering an incompetent teacher only in eighth grade and theeffect of having such a teacher only in ninth grade

There has been a great amount of conceptual confusion regarding how a moderatorrelates to the treatment and the outcome On one hand many researchers have used the termsldquomoderatorsrdquo and ldquomediatorsrdquo interchangeably without understanding the crucial distinctionbetween the two An overcorrective attempt on the other hand has led to the arbitrary rec-ommendation that a moderator must occur prior to the treatment and be minimally associatedwith the treatment (James and Brett 1984 Kraemer et al 2001 2008) In Chapter 6 weclarify that in essence the causal effect of the treatment on the outcome may depend onthe moderator value A moderator can be a subpopulation identifier a contextual character-istic a concurrent treatment or a preceding or succeeding treatment In the earlier examplesage of estrogen therapy initiation and a studentrsquos prior ability are subpopulation identifiers themanageability of a class which reflects both teacher skills and peer behaviors characterizes thecontext literacy instruction time is a concurrent treatment while prealgebra is a precedingtreatment for algebra A moderator does not have to occur prior to the treatment and doesnot have to be independent of the treatment

Once a moderated causal relationship has been defined in terms of potential outcomes theresearcher then chooses an appropriate experimental design for testing the moderation theoryThe assumptions required for identifying the moderated causal relationships differ across dif-ferent designs and have implications for analyzing experimental and quasiexperimental dataRandomized block designs are suitable for examining individual or contextual characteristicsas potential moderators factorial designs enable one to determine the joint effects of two ormore concurrent treatments and sequential randomized designs are ideal for assessing thecumulative effects of consecutive treatments Multisite randomized trials constitute anadditional type of designs in which the experimental sites are often deliberately sampled torepresent a population of geographical locations or social organizations Site membershipscan be viewed as implicit moderators that summarize a host of features of a local environmentReplications of a treatment over multiple sites allow one to quantify the heterogeneity of treat-ment effects across the sites Chapters 7 and 8 are focused on statistical methods for moder-ation analyses with quasiexperimental data In particular these chapters demonstrate througha number of application examples how a nonparametric marginal mean weighting through

6 CAUSALITY IN A SOCIAL WORLD

stratification (MMWS) method can overcome some important limitations of other existingmethods

Moderation questions are not restricted to treatmentndashoutcome relationships Rather theyare prevalent in investigations of mediation and spill-over This is because heterogeneity oftreatment effects may be explained by the variation in causal mediation mechanisms acrosssubpopulations and contexts It is also because the treatment effect for a focal individualmay depend on whether there is spill-over from other individuals through social interactionsYet similar to the confusion around the concept of moderation among applied researchersthere has been a great amount of misunderstanding with regard to mediation

112 Mediated treatment effects

Questions about mediation are at the core of nearly every scientific theory In its simplestform a theory explains why treatment Z causes outcome Y by hypothesizing an intermediateprocess involving at least one mediatorM that could have been changed by the treatment andcould subsequently have an impact on the outcome A causal mediation analysis then decom-poses the total effect of the treatment on the outcome into two parts the effect transmittedthrough the hypothesized mediator called the indirect effect and the difference betweenthe total effect and the indirect effect called the direct effect The latter represents the treat-ment effect channeled through other unspecified pathways (Alwin and Hauser 1975 Baronand Kenny 1986 Duncan 1966 Shadish and Sweeney 1991) Most researchers in socialsciences have followed the convention of illustrating the direct effect and the indirect effectwith path diagrams and then specifying linear regression models postulated to represent thestructural relationships between the treatment the mediator and the outcome

In Chapter 9 we point out that the approach to defining the indirect effect and the directeffect in terms of path coefficients is often misled by oversimplistic assumptions about thestructural relationships In particular this approach typically overlooks the fact that a treat-ment may generate an impact on the outcome not only by changing the mediator value butalso by changing the mediatorndashoutcome relationship (Judd and Kenny 1981) An examplein Chapter 9 illustrates such a case An experiment randomizes students to either an experi-mental condition which provides them with study materials and encourages them to study fora test or to a control condition which provides neither study materials nor encouragement(Holland 1988 Powers and Swinton 1984) One might hypothesize that encouraging exper-imental group members to study will increase the time they spend studying a focal mediatorin this example which in turn will increase their average test scores One might furtherhypothesize that even without a change in study time providing study materials to experi-mental group members will enable them to study more effectively than they otherwise wouldConsequently the effect of time spent studying might be greater under the experimentalcondition than under the control condition Omitting the treatment-by-mediator interactioneffect will lead to bias in the estimation of the indirect effect and the direct effect

Chapter 11 describes in great detail an evaluation study contrasting a new welfare-to-workprogram emphasizing active participation in the labor force with a traditional program thatguaranteed cash assistance without a mandate to seek employment (Hong Deutsch and Hill2011) Focusing on the psychological well-being of welfare applicants who were singlemothers with preschool-aged children the researchers hypothesized that the treatment wouldincrease employment rate among the welfare recipients and that the treatment-inducedincrease in employment would likely reduce depressive symptoms under the new program

7INTRODUCTION

They also hypothesized that in contrast the same increase in employment was unlikely tohave a psychological impact under the traditional program Hence the treatment-by-mediatorinteraction effect on the outcome was an essential component of the intermediate process

We will show that by defining the indirect effect and the direct effect in terms of potentialoutcomes one can avoid invoking unwarranted assumptions about the unknown structuralrelationships (Holland 1988 Pearl 2001 Robins and Greenland 1992) The potential out-comes framework has further clarified that in a causal mediation theory the treatment and themediator are both conceivably manipulable In the aforementioned example a welfare appli-cant could be assigned at random to either the new program or the traditional one Once havingbeen assigned to one of the programs the individual might or might not be employed due tovarious structural constraints market fluctuations or other random events typically beyondher control

Because the indirect effect and the direct effect are defined in terms of potential outcomesrather than on the basis of any specific regression models it becomes possible to distinguishthe definitions of these causal effects from their identification and estimation The identifica-tion step relates a causal parameter to observable population data For example for individualsassigned at random to an experimental condition their average counterfactual outcome asso-ciated with the control condition is expected to be equal to the average observable outcome ofthose assigned to the control condition Therefore the population average causal effect can beeasily identified in a randomized experiment The estimation step then relates the sample datato the observable population quantities involved in identification while taking into account thedegree of sampling variability

A major challenge in identification is that the mediatorndashoutcome relationship tends tobe confounded by selection even if the treatment is randomized Chapter 9 reviews variousexperimental designs that have been proposed by past researchers for studying causalmediation mechanisms Chapter 10 compares the identification assumptions and the analyticprocedures across a wide range of analytic methods These discussions are followed by anintroduction of the ratio-of-mediator-probability weighting (RMPW) strategy in Chapter 11In comparison with most existing strategies for causal mediation analysis RMPW relies onrelatively fewer identification assumptions and model-based assumptions Chapters 12 and13 will show that this new strategy can be applied broadly with extensions to multilevelexperimental designs and to studies of complex mediation mechanisms involving multiplemediators

113 Spill-over effects of a treatment

It is well known in vaccination research that an unvaccinated person can benefit when mostother people in the local community are vaccinated This is viewed as a spill-over of the treat-ment effect from the vaccinated people to an unvaccinated person Similarly due to socialinteractions among individuals or groups of individuals an intervention received by somemay generate a spill-over impact on others affiliated with the same organization or connectedthrough the same network For example effective policing in one neighborhood may driveoffenders to operate in other neighborhoods In evaluating an innovative community policingprogram researchers found that a neighborhood not assigned to community policing tended tosuffer if the surrounding neighborhoods were assigned to community policing This is aninstance in which a well-intended treatment generates an unintended negative spill-over effectHowever being assigned to community policy was found particularly beneficial when thesurrounding neighborhoods also received the intervention (Verbitsky-Savitz and Raudenbush

8 CAUSALITY IN A SOCIAL WORLD

2012) In another example the impact of a school-based mentoring program targeted atstudents displaying delinquent behaviors may be enhanced for a focal student if his or herat-risk peers are assigned to the mentoring program at the same time

The previous examples challenge the ldquostable unit treatment value assumptionrdquo (SUTVA)that has been invoked in most causal inference studies This assumption states that an indi-vidualrsquos potential response to a treatment depends neither on how the treatment is assignednor on the treatment assignments of other individuals (Rubin 1980 1986) Rubin and othersbelieve that without resorting to SUTVA the causal effect of a treatment becomes hard todefine and that causal inference may become intractable However as Sobel (2006) has illus-trated with the Moving to Opportunity (MTO) experiment that offered housing vouchersenabling low-income families to move to low-poverty neighborhoods there are possible con-sequences of violating SUTVA in estimating the treatment effects on outcomes such as safetyand self-sufficiency Because social interactions among individuals may affect whether onevolunteers to participate in the study whether one moves to a low-poverty neighborhood afterreceiving the housing voucher as well as housing project residentsrsquo subjective perceptions oftheir neighborhoods the program may have a nonzero impact on the potential outcome of theuntreated As a result despite the randomization of treatment assignment the mean differencein the observed outcome between the treated units and the untreated units may be biased forthe average treatment effect Rather the observable quantity is the difference between theeffect of treating some rather than treating none for the treated and the effect of treating somerather than treating none for the untreated the latter being the pure spill-over effect for theuntreated

In making attempts to relax SUTVA researchers have proposed several alternative frame-works that incorporate possible spill-over effects (see Hong and Raudenbush 2013 for areview) Hong (2004) presented a model that involves treatments and treatment settingsA treatment setting for an individual is a local environment constituted by a set of agentsand participants along with their treatment assignments An individualrsquos potential outcomevalue under a given treatment is assumed stable when the treatment setting is fixed the poten-tial outcome may take different values when the treatment setting shifts One may investigatewhether the treatment effect depends on the treatment setting Applying this framework Hongand Raudenbush (2006) examined the effect on a childrsquos academic growth of retaining thechild in kindergarten rather than promoting the child to first grade when a relatively smallproportion of low-achieving peers in the same school are retained as opposed to when a rel-atively large proportion of the peers are retained Hudgens and Halloran (2008) presented arelated framework in which the effect on an individual of the treatment received by this indi-vidual is distinguished from the effect on the individual of the treatment received by others inthe same local community These effects can be identified if communities are assigned atrandom to different treatment assignment strategies (such as retaining a large proportion oflow-achieving students as opposed to retaining a small proportion of such students) and sub-sequently individuals within a community are randomized for treatment Chapter 14 reviewsthese frameworks and discusses identification and estimation strategies

Social contagion may also serve as an important channel through which the effect of anintervention is transmitted For example a school-wide intervention may reduce aggressivebehaviors and thereby improve studentsrsquo psychological well-being by improving the qualityof interpersonal relationships in other classes as well as in onersquos own class This is becausechildren interact not simply with their classmates but also with those from other classes in thehallways or on the playground (VanderWeele et al 2013) In this case the spill-over betweenclasses becomes a part of the mediation mechanism In a study of student mentoring whether

9INTRODUCTION

an individual actually participated in the mentoring program despite the initial treatmentassignment is typically viewed as a primary mediator The proportion of onersquos peers that par-ticipated in the program may act as a second mediator A treatment may also exert its impactpartly through regrouping individuals and thereby changing peer composition (Hong andNomi 2012) Chapter 15 discusses analytic strategies for detecting spill-over as a part ofthe causal mediation mechanism

12 Weighting methods for causal inference

In most behavioral and social science applications major methodological challenges arise dueto the selection of treatments in a quasiexperimental study and the selection of mediator valuesin experimental and quasiexperimental studies Statistical methods most familiar to research-ers in social sciences are often inadequate for causal inferences with regard to multivaluedtreatments moderation mediation and spill-over The book offers a major revision to under-standing of causality in the social world by introducing two complementary weighting stra-tegies both featuring nonparametric approaches to estimating these causal effects The newweighting methods greatly simplify model specifications while enhancing the robustness ofresults by minimizing the reliance on some key assumptions

The propensity score-basedMMWSmethod removes selection bias associated with a largenumber of covariates by equating the pretreatment composition between treatment groups(Hong 2010a 2012 Huang et al 2005) Unlike propensity score matching and stratificationthat are mostly restricted to evaluations of binary treatments the MMWS method is flexiblefor evaluating binary and multivalued treatments by approximating a completely randomizedexperiment In evaluating whether the treatment effects differ across subpopulations definedby individual characteristics or treatment settings researchers may assign weights within eachsubpopulation in order to approximate a randomized block design To investigate whetherone treatment moderates the effect of another concurrent treatment researchers may assignweights to the data to approximate a factorial randomized design The method can also beused to assess whether the effect of an initial treatment is amplified or weakened by a subse-quent treatment or to identify an optimal treatment sequence through approximating a sequen-tial randomized experiment Even though such analyses can similarly be conducted throughinverse-probability-of-treatment weighting (IPTW) that has been increasingly employed inepidemiological research (Hernaacuten Brumback and Robins 2000 Robins Hernaacuten andBrumback 2000) IPTW is known for bias and imprecision in estimation especially whenthe propensity score models are misspecified in their functional forms (Hong 2010a Kangand Schafer 2007 Schafer and Kang 2008 Waernbaum 2012) In contrast the nonparamet-ric MMWS method displays a relatively high level of robustness despite such misspecifica-tions and also gains efficiency as indicated by simulation results (Hong 2010a)

To study causal mediation mechanisms the RMPW method decomposes the total effectof a treatment into an ldquoindirect effectrdquo transmitted through a specific mediator and a ldquodirecteffectrdquo representing unspecified mechanisms In contrast with most existing methods formediation analysis the RMPW-adjusted outcome model is extremely simple and is nonpara-metric in nature It generates estimates of the causal effects along with their sampling errorswhile adjusting for pretreatment covariates that confound the mediatorndashoutcome relationshipsthrough weighting The method applies regardless of the distribution of the outcome the dis-tribution of the mediator or the functional relationship between the outcome and the mediator

10 CAUSALITY IN A SOCIAL WORLD

Page 5: Thumbnail - download.e-bookshelf.de...4.2.4 IPTW for bias removal 84 4.3 MMWS 86 4.3.1 Theoretical rationale 86 ... 8.3.4.2 Growth model in the absence of confounding 200 8.3.4.3 Weighted

Contents

Preface xv

Part I Overview 1

1 Introduction 311 Concepts of moderation mediation and spill-over 3

111 Moderated treatment effects 5112 Mediated treatment effects 7113 Spill-over effects of a treatment 8

12 Weighting methods for causal inference 1013 Objectives and organization of the book 1114 How is this book situated among other publications on related topics 12

2 Review of causal inference concepts and methods 1821 Causal inference theory 18

211 Attributes versus causes 18212 Potential outcomes and individual-specific causal effects 19213 Inference about population average causal effects 22

2131 Prima facie effect 242132 Ignorability assumption 25

22 Applications to Lordrsquos paradox and Simpsonrsquos paradox 27221 Lordrsquos paradox 27222 Simpsonrsquos paradox 31

23 Identification and estimation 34231 Selection bias 35232 Sampling bias 35233 Estimation efficiency 36

Appendix 21 Potential bias in a prima facie effect 36Appendix 22 Application of the causal inference theory to Lordrsquos paradox 37

3 Review of causal inference designs and analytic methods 4031 Experimental designs 40

311 Completely randomized designs 40

312 Randomized block designs 41313 Covariance adjustment for improving efficiency 43314 Multilevel experimental designs 43

32 Quasiexperimental designs 44321 Nonequivalent comparison group designs 44322 Other quasiexperimental designs 45

33 Statistical adjustment methods 46331 ANCOVA and multiple regression 46

3311 ANCOVA for removing selection bias 463312 Potential pitfalls of ANCOVA with a vast between-group

difference 473313 Bias due to model misspecification 48

332 Matching and stratification 50333 Other statistical adjustment methods 51

3331 The IV method 513332 DID analysis 54

34 Propensity score 55341 What is a propensity score 56342 Balancing property of the propensity score 57343 Pooling conditional treatment effect estimate Matching

stratification and covariance adjustment 603431 Propensity score matching 613432 Propensity score stratification 623433 Covariance adjustment for the propensity score 663434 Sensitivity analysis 66

Appendix 3A Potential bias due to the omission of treatment-by-covariateinteraction 70

Appendix 3B Variable selection for the propensity score model 71

4 Adjustment for selection bias through weighting 7641 Weighted estimation of population parameters in survey sampling 77

411 Simple random sample 77412 Proportionate sample 78413 Disproportionate sample 79

42 Weighting adjustment for selection bias in causal inference 80421 Experimental result 81422 Quasiexperimental result 81423 Sample weight for bias removal 82424 IPTW for bias removal 84

43 MMWS 86431 Theoretical rationale 86

4311 MMWS for a discrete propensity score 874312 MMWS for a continuous propensity score 884313 MMWS for estimating the treatment effect on the treated 89

432 MMWS analytic procedure 91433 Inherent connection and major distinctions between MMWS

and IPTW 93

vi CONTENTS

Appendix 4A Proof of MMWS-adjusted mean observed outcome being unbiasedfor the population average potential outcome 95

Appendix 4B Derivation of MMWS for estimating the treatment effecton the treated 96

Appendix 4C Theoretical equivalence of MMWS and IPTW 97Appendix 4D Simulations comparing MMWS and IPTW under misspecifications

of the functional form of a propensity score model 97

5 Evaluations of multivalued treatments 10051 Defining the causal effects of multivalued treatments 10052 Existing designs and analytic methods for evaluating multivalued

treatments 102521 Experimental designs and analysis 102

5211 Randomized experiments with multipletreatment arms 102

5212 Identification under ignorability 1025213 ANOVA 103

522 Quasiexperimental designs and analysis 1055221 ANCOVA and multiple regression 1055222 Propensity score-based adjustment 1085223 Other adjustment methods 112

53 MMWS for evaluating multivalued treatments 112531 Basic rationale 113532 Analytic procedure 114

5321 MMWS for a multinomial treatment measure 1155322 MMWS for an ordinal treatment measure 120

533 Identification assumptions 12154 Summary 123Appendix 5A Multiple IV for evaluating multivalued treatments 124

Part II Moderation 127

6 Moderated treatment effects concepts and existing analytic methods 12961 What is moderation 129

611 Past discussions of moderation 1306111 Purpose of moderation research 1306112 What qualifies a variable as a moderator 132

612 Definition of moderated treatment effects 1336121 Treatment effects moderated by individual or contextual

characteristics 1336122 Joint effects of concurrent treatments 134

62 Experimental designs and analytic methods for investigatingexplicit moderators 136621 Randomized block designs 137

6211 Identification assumptions 1376212 Two-way ANOVA 1386213 Multiple regression 139

viiCONTENTS

622 Factorial designs 1406221 Identification assumptions 1406222 Analytic strategies 142

63 Existing research designs and analytic methods for investigatingimplicit moderators 142631 Multisite randomized trials 143

6311 Causal parameters and identification assumptions 1446312 Analytic strategies 145

632 Principal stratification 149Appendix 6A Derivation of bias in the fixed-effects estimator when the

treatment effect is heterogeneous in multisite randomized trials 151Appendix 6B Derivation of bias in the mixed-effects estimator when the

probability of treatment assignment varies across sites 153Appendix 6C Derivation and proof of the population weight applied to

mixed-effects models for eliminating bias in multisiterandomized trials 153

7 Marginal mean weighting through stratification for investigatingmoderated treatment effects 15971 Existing methods for moderation analyses with quasiexperimental data 159

711 Analysis of covariance and regression-based adjustment 1617111 Treatment effects moderated by subpopulation membership 1617112 Treatment effects moderated by a concurrent treatment 164

712 Propensity score-based adjustment 1657121 Propensity score matching and stratification 1667122 Inverse-probability-of-treatment weighting 167

72 MMWS estimation of treatment effects moderated by individual orcontextual characteristics 168721 Application example 170722 Analytic procedure 170

73 MMWS estimation of the joint effects of concurrent treatments 174731 Application example 174732 Analytic procedure 175733 Joint treatment effects moderated by individual or contextual

characteristics 179

8 Cumulative effects of time-varying treatments 18581 Causal effects of treatment sequences 186

811 Application example 186812 Causal parameters 187

8121 Time-varying treatments 1878122 Time-varying potential outcomes 1878123 Causal effects of 2-year treatment sequences 1878124 Causal effects of multiyear treatment sequences 190

82 Existing strategies for evaluating time-varying treatments 190821 The endogeneity problem in nonexperimental data 190822 SEM 191

viii CONTENTS

823 Fixed-effects econometric models 192824 Sequential randomization 192825 Dynamic treatment regimes 193826 Marginal structural models and structural nested models 194

83 MMWS for evaluating 2-year treatment sequences 195831 Sequential ignorability 195832 Propensity scores 196833 MMWS computation 197

8331 MMWS adjustment for year 1 treatment selection 1978332 MMWS adjustment for two-year treatment

sequence selection 198834 Two-year growth model specification 199

8341 Growth model in the absence of treatment 2008342 Growth model in the absence of confounding 2008343 Weighted 2-year growth model 202

84 MMWS for evaluating multiyear sequences of multivalued treatments 204841 Sequential ignorability of multiyear treatment sequences 204842 Propensity scores for multiyear treatment sequences 204843 MMWS computation 205844 Weighted multiyear growth model 205845 Issues of sample size 206

85 Conclusion 207Appendix 8A A saturated model for evaluating multivalued treatments

over multiple time periods 207

Part III Mediation 211

9 Concepts of mediated treatment effects and experimental designsfor investigating causal mechanisms 21391 Introduction 21492 Path coefficients 21593 Potential outcomes and potential mediators 216

931 Controlled direct effects 217932 Controlled treatment-by-mediator interaction effect 217

94 Causal effects with counterfactual mediators 219941 Natural direct effect 219942 Natural indirect effect 220943 Natural treatment-by-mediator interaction effect 220944 Unstable unit treatment value 221

95 Population causal parameters 222951 Population average natural direct effect 224952 Population average natural indirect effect 225

96 Experimental designs for studying causal mediation 225961 Sequentially randomized designs 228962 Two-phase experimental designs 228963 Three- and four-treatment arm designs 230964 Experimental causal-chain designs 231

ixCONTENTS

965 Moderation-of-process designs 231966 Augmented encouragement designs 232967 Parallel experimental designs and parallel encouragement designs 232968 Crossover experimental designs and crossover encouragement

designs 233969 Summary 234

10 Existing analytic methods for investigating causal mediation mechanisms 238101 Path analysis and SEM 239

1011 Analytic procedure for continuous outcomes 2391012 Identification assumptions 2421013 Analytic procedure for discrete outcomes 245

102 Modified regression approach 2461021 Analytic procedure for continuous outcomes 2461022 Identification assumptions 2471023 Analytic procedure for binary outcomes 248

103 Marginal structural models 2501031 Analytic procedure 2501032 Identification assumptions 252

104 Conditional structural models 2521041 Analytic procedure 2521042 Identification assumptions 253

105 Alternative weighting methods 2541051 Analytic procedure 2541052 Identification assumptions 256

106 Resampling approach 2561061 Analytic procedure 2561062 Identification assumptions 257

107 IV method 2571071 Rationale and analytic procedure 2571072 Identification assumptions 258

108 Principal stratification 2591081 Rationale and analytic procedure 2591082 Identification assumptions 260

109 Sensitivity analysis 2611091 Unadjusted confounding as a product of hypothetical

regression coefficients 2611092 Unadjusted confounding reflected in a hypothetical

correlation coefficient 2621093 Limitations when the selection mechanism differs

by treatment 2641094 Other sensitivity analyses 265

1010 Conclusion 26510101 The essentiality of sequential ignorability 26510102 Treatment-by-mediator interactions 26610103 Homogeneous versus heterogeneous causal effects 26610104 Model-based assumptions 266

x CONTENTS

Appendix 10A Bias in path analysis estimation due to the omission oftreatment-by-mediator interaction 267

11 Investigations of a simple mediation mechanism 273111 Application example national evaluation of welfare-to-work strategies 274

1111 Historical context 2741112 Research questions 2751113 Causal parameters 2751114 NEWWS Riverside data 277

112 RMPW rationale 2771121 RMPW in a sequentially randomized design 278

11211 E[Y(0 M(0))] 27911212 E[Y(1 M(1))] 28011213 E[Y(1 M(0))] 28011214 E[Y(0 M(1))] 28111215 Nonparametric outcome model 282

1122 RMPW in a sequentially randomized block design 2831123 RMPW in a standard randomized experiment 2851124 Identification assumptions 286

113 Parametric RMPW procedure 287114 Nonparametric RMPW procedure 290115 Simulation results 292

1151 Correctly specified propensity score models 2921152 Misspecified propensity score models 2941153 Comparisons with path analysis and IV results 294

116 Discussion 2951161 Advantages of the RMPW strategy 2951162 Limitations of the RMPW strategy 295

Appendix 11A Causal effect estimation through the RMPW procedure 296Appendix 11B Proof of the consistency of RMPW estimation 297

12 RMPW extensions to alternative designs and measurement 301121 RMPW extensions to mediators and outcomes of alternative distributions 301

1211 Extensions to a multicategory mediator 30212111 Parametric RMPW procedure 30212112 Nonparametric RMPW procedure 304

1212 Extensions to a continuous mediator 3041213 Extensions to a binary outcome 306

122 RMPW extensions to alternative research designs 3061221 Extensions to quasiexperimental data 3071222 Extensions to data from cluster randomized trials 308

12221 Application example and research questions 30912222 Estimation of the total effect 31012223 RMPW analysis of causal mediation mechanisms 31012224 Identification assumptions 31212225 Contrast with multilevel path analysis and SEM 31212226 Contrast with multilevel prediction models 313

xiCONTENTS

1223 Extensions to data from multisite randomized trials 31312231 Research questions and causal parameters 31412232 Estimation of the total effect and its between-site variation 31512233 RMPW analysis of causal mediation mechanisms 31612234 Identification assumptions 31912235 Contrast with multilevel path analysis and SEM 319

123 Alternative decomposition of the treatment effect 321

13 RMPW extensions to studies of complex mediation mechanisms 325131 RMPW extensions to moderated mediation 325

1311 RMPW analytic procedure for estimating and testingmoderated mediation 326

1312 Path analysisSEM approach to analyzing moderatedmediation 327

1313 Principal stratification and moderated mediation 328132 RMPW extensions to concurrent mediators 328

1321 Treatment effect decomposition 32913211 Treatment effect decomposition without

between-mediator interaction 32913212 Treatment effect decomposition with

between-mediator interaction 3311322 Identification assumptions 3331323 RMPW procedure 333

13231 Estimating E[Y(1M1(1)M2(0))] 33513232 Estimating E[Y(1M1(0)M2(0))] 33513233 Causal effect estimation with noninteracting

concurrent mediators 33613234 Estimating E[Y(1M1(0)M2(1))] 33713235 Causal effect estimation with interacting concurrent

mediators 3371324 Contrast with the linear SEM approach 3381325 Contrast with the multivariate IV approach 339

133 RMPW extensions to consecutive mediators 3401331 Treatment effect decomposition 341

13311 Natural direct effect of the treatment on the outcome 34213312 Natural indirect effect mediated by M1 only 34213313 Natural indirect effect mediated by M2 only 34213314 Natural indirect effect mediated by an M1-by-M2

interaction 34313315 Treatment-by-mediator interactions 343

1332 Identification assumptions 3451333 RMPW procedure 347

13331 Estimating E[Y(1M1(0)M2(0M1(0)))] 34713332 Estimating E[Y(1M1(1)M2(0M1(0)))] 34913333 Estimating E[Y(1M1(0)M2(1M1(1)))] 34913334 Estimating E[Y(0M1(1)M2(0M1(0)))] and

E[Y(0M1(0)M2(1M1(1)))] 351

xii CONTENTS

1334 Contrast with the linear SEM approach 3531335 Contrast with the sensitivity-based estimation of bounds for

causal effects 354134 Discussion 355Appendix 13A Derivation of RMPW for estimating population average

counterfactual outcomes of two concurrentmediators 355

Appendix 13B Derivation of RMPW for estimating population averagecounterfactual outcomes of consecutivemediators 358

Part IV Spill-over 363

14 Spill-over of treatment effects concepts and methods 365141 Spill-over A nuisance a trifle or a focus 365142 Stable versus unstable potential outcome values An example

from agriculture 367143 Consequences for causal inference when spill-over is overlooked 369144 Modified framework of causal inference 371

1441 Treatment settings 3711442 Simplified characterization of treatment settings 3731443 Causal effects of individual treatment assignment and of peer

treatment assignment 375145 Identification Challenges and solutions 376

1451 Hypothetical experiments for identifying average treatmenteffects in the presence of social interactions 376

1452 Hypothetical experiments for identifying the impact ofsocial interactions 380

1453 Application to an evaluation of kindergarten retention 382146 Analytic strategies for experimental and quasiexperimental data 384

1461 Estimation with experimental data 3841462 Propensity score stratification 3851463 MMWS 386

147 Summary 387

15 Mediation through spill-over 391151 Definition of mediated effects through spill-over in a cluster

randomized trial 3931511 Notation 3931512 Treatment effect mediated by a focal individualrsquos

compliance 3941513 Treatment effect mediated by peersrsquo compliance through

spill-over 3941514 Decomposition of the total treatment effect 395

152 Identification and estimation of the spill-over effect in a clusterrandomized design 3951521 Identification in an ideal experiment 395

xiiiCONTENTS

1522 Identification when the mediators are not randomized 3981523 Estimation of mediated effects through spill-over 400

153 Definition of mediated effects through spill-over in a multisite trial 4021531 Notation 4021532 Treatment effect mediated by a focal individualrsquos compliance 4041533 Treatment effect mediated by peersrsquo compliance

through spill-over 4041534 Direct effect of individual treatment assignment on the outcome 4051535 Direct effect of peer treatment assignment on the outcome 4051536 Decomposition of the total treatment effect 405

154 Identification and estimation of spill-over effects in a multisite trial 4061541 Identification in an ideal experiment 4071542 Identification when the mediators are not randomized 4091543 Estimation of mediated effects through spill-over 410

15431 Estimating E[Y(1pM(p)Mminus(p))] andE[Y(000Mminus(0))] 410

15432 Estimating E[Y(1p0Mminus(p))] 41115433 Estimating E[Y(1p0Mminus(0))] 41215434 Estimating E[Y(0p0Mminus(0))] 412

155 Consequences of omitting spill-over effects in causal mediation analyses 4121551 Biased inference in a cluster randomized trial 4131552 Biased inference in a multisite randomized trial 4131553 Biased inference of the local average treatment effect 415

156 Quasiexperimental application 416157 Summary 419Appendix 151 Derivation of the weight for estimating the population

average counterfactual outcome E[Y(1 p 0Mminus( p))] 419Appendix 152 Derivation of bias in the ITT effect due to the omission of

spill-over effects 420

Index 423

xiv CONTENTS

Preface

A scientific mind in training seeks comprehensive descriptions of every interesting phenom-enon In such a mind data are to be pieced together for comparisons contrasts and eventuallyinferences that are required for understanding causes and effects that may have led to thephenomenon of interest Yet without disciplined reasoning a causal inference would slip intothe rut of a ldquocasual inferencerdquo as easily as making a typographical error

As a doctoral student thirsting for rigorous methodological training at the University ofMichigan I was very fortunate to be among the first group of students on the campus attendinga causal inference seminar in 2000 jointly taught by Yu Xie from the Department of Sociol-ogy Susan Murphy from the Department of Statistics and Stephen Raudenbush from theSchool of Education The course was unique in both content and pedagogy The instructorsput together a bibliography drawn from the past and current causal inference literature origi-nated from multiple disciplines In the spirit of ldquoreciprocal teachingrdquo the students were delib-erately organized into small groups each representing a mix of disciplinary backgrounds andtook turns to teach the weekly readings under the guidance of a faculty instructor This inval-uable experience revealed to me that the pursuit of causal inference is a multidisciplinaryendeavor In the rest of my doctoral study I continued to be immersed in an extraordinaryinterdisciplinary community in the form of the Quantitative Methodology Program (QMP)seminars led by the above three instructors who were then joined by Ben Hansen(Statistics) Richard Gonzalez (Psychology) Roderick Little (Biostatistics) Jake Bowers(Political Science) and many others I have tried whenever possible to carry the same spiritinto my own teaching of causal inference courses at the University of Toronto and now at theUniversity of Chicago In these courses communicating understandings of causal problemsacross disciplinary boundaries has been particularly challenging but also stimulating and grat-ifying for myself and for students from various departments and schools

Under the supervision of Stephen Raudenbush I took a first stab at the conceptual frame-work of causal inference in my doctoral dissertation by considering peer spill-over in schoolsettings From that point on I have organized my methodological research to tackle some ofthe major obstacles to causal inferences in policy and program evaluations My workaddresses issues including (i) how to conceptualize and evaluate the causal effects of educa-tional treatments when studentsrsquo responses to alternative treatments depend on various fea-tures of the organizational settings including peer composition (ii) how to adjust forselection bias in evaluating the effects of concurrent and consecutive multivalued treatmentsand (iii) how to conceptualize and analyze causal mediation mechanisms My methodological

research has been driven by the substantive interest in understanding the role of socialinstitutionsmdashschools in particularmdashin shaping human development especially during theages of fast growth (eg childhood adolescence and early adulthood) I have shared meth-odological advances with a broad audience in social sciences through applying the causalinference methods to prominent substantive issues such as grade retention within-class group-ing services for English language learners and welfare-to-work strategies These illustrationsare used extensively in this book

Among social scientists awareness of methodological challenges in causal investigationsis higher than ever before In response to this rising demand causal inference is becoming oneof the most productive scholarly fields in the past decade I have had the benefit of followingthe work and exchanging ideas with some fellow methodologists who are constantly contri-buting new thoughts and making breakthroughs These include Daniel Almirall HowardBloom Tom Cook Michael Elliot Ken Frank Ben Hansen James Heckman JenniferHill Martin Huber Kosuke Imai Booil Jo John R Lockwood David MacKinnon DanielMcCaffrey Luke Miratrix Richard Murnane Derek Neal Lindsey Page Judea PearlStephen Raudenbush Sean Reardon Michael Seltzer Youngyun Shin Betsy SinclairMichael Sobel Peter Steiner Elizabeth Stuart Eric Thetchen Tchetchen Tyler VanderWeeleand Kazuo Yamaguchi I am grateful to Larry Hedges who offered me the opportunity of guestediting the Journal of Research in Educational Effectiveness special issue on the statisticalapproaches to studying mediator effects in education research in 2012 Many of the colleagueswhom I mentioned above contributed to the open debate in the special issue by either propos-ing or critically examining a number of innovative methods for studying causal mechanismsI anticipate that many of them are or will soon be writing their own research monographs oncausal inference if they have not already done so Therefore readers of this book are stronglyrecommended to browse past and future titles by these and other authors for alternative per-spectives and approaches My own understanding of course will continue to evolve as well inthe coming years

One has to be ambitious to stay upfront in substantive areas as well as in methodology Thebest strategy apparently is to learn from substantive experts During different phases of mytraining and professional career I have had the opportunities to work with David K CohenBrian Rowan Deborah Ball Carl Corter Janette Pelletier Takako Nomi Esther Geva DavidFrancis Stephanie Jones Joshua Brown and Heather D Hill Many of my substantive insightswere derived from conversations with these mentors and colleagues I am especially gratefulto Bob Granger former president of the William T Grant Foundation who reassured me thatldquoBy staying a bit broad you will learn a lot and many fields will benefit from your workrdquo

Like many other single-authored books this one is built on the generous contributions ofstudents and colleagues who deserve special acknowledgement Excellent research assistancefrom Jonah Deutsch Joshua Gagne Rachel Garrett Yihua Hong Xu Qin Cheng Yang andBing Yu was instrumental in the development and implementation of the new methods pre-sented in this book Richard Congdon a renowned statistical software programmer broughtrevolutionary ideas to interface designs in software development with the aim of facilitatingusersrsquo decision-making in a series of causal analyses RichardMurnane Stephen Raudenbushand Michael Sobel carefully reviewed and critiqued multiple chapters of the manuscript draftAdditional comments and suggestions on earlier drafts came fromHoward Bloom Ken FrankBen Hansen Jennifer Hill Booil Jo Ben Kelcey David MacKinnon and Fan Yang TereseSchwartzman provided valuable assistance in manuscript preparation The writing of thisbook was supported by a Scholars Award from the William T Grant Foundation and an

xvi PREFACE

Institute of Education Sciences (IES) Statistical and Research Methodology in EducationGrant from the US Department of Education All the errors in the published form are ofcourse the sole responsibility of the author

Project Editors Richard Davis Prachi Sinha Sahay and Liz Wingett and Assistant EditorHeather Kay at Wiley and project Managers P Jayapriya and R Jayavel at SPi Global havebeen wonderful to work with throughout the manuscript preparation and final production ofthe book Debbie Jupe Commissioning Editor at Wiley was remarkably effective in initiatingthe contact with me and in organizing anonymous reviews of the book proposal and of themanuscript These constructive reviews have shaped the book in important ways I cannotagree more with one of the reviewers that ldquocausality cannot be established on a pure statisticalgroundrdquo The book therefore highlights the importance of substantive knowledge and researchdesigns and places a great emphasis on clarifying and evaluating assumptions required foridentifying causal effects in the context of each application Much caution is raised againstpossible misusage of statistical techniques for analyzing causal relationships especiallywhen data are inadequate Yet I maintain that improved statistical procedures along withimproved research designs would greatly enhance our ability in attempt to empiricallyexamine well-articulated causal theories

xviiPREFACE

Part I

OVERVIEW

1

Introduction

According to an ancient Chinese fable a farmer who was eager to help his crops grow wentinto his field and pulled each seedling upward After exhausting himself with the work heannounced to his family that they were going to have a good harvest only to find the nextmorning that the plants had wilted and died Readers with minimal agricultural knowledgemay immediately point out the following the farmerrsquos intervention theory was based on acorrect observation that crops that grow taller tend to produce more yield Yet his hypothesisreflects a false understanding of the cause and the effectmdashthat seedlings pulled to be tallerwould yield as much as seedlings thriving on their own

In their classic Design and Analysis of Experiments Hinkelmann and Kempthorne (1994updated version of Kempthorne 1952) discussed two types of science descriptive science andthe development of theory These two types of science are interrelated in the following senseobservations of an event and other related events often selected and classified for descriptionby scientists naturally lead to one or more explanations that we call ldquotheoretical hypothesesrdquowhich are then screened and falsified by means of further observations experimentation andanalyses (Popper 1963) The experiment of pulling seedlings to be taller was costly but didserve the purpose of advancing this farmerrsquos knowledge of ldquowhat does not workrdquo To developa successful intervention in this case would require a series of empirical tests of explicittheories identifying potential contributors to crop growth This iterative process graduallydeepens our knowledge of the relationships between supposed causes and effectsmdashthatis causalitymdashand may eventually increase the success of agricultural medical and socialinterventions

11 Concepts of moderation mediation and spill-over

Although the story of the ancient farmer is fictitious numerous examples can be found in thereal world in which well-intended interventions fail to produce the intended benefits or inmany cases even lead to unintended consequences ldquoInterventionsrdquo and ldquotreatmentsrdquo usedinterchangeably in this book broadly refer to actions taken by agents or circumstances expe-rienced by an individual or groups of individuals Interventions are regularly seen in educa-tion physical and mental health social services business politics and law enforcement In an

Causality in a Social World Moderation Meditation and Spill-over First Edition Guanglei Hongcopy 2015 John Wiley amp Sons Ltd Published 2015 by John Wiley amp Sons Ltd

education intervention for example teachers are typically the agents who deliver a treatmentto students while the impact of the treatment on student outcomes is of ultimate causalinterest Some educational practices such as ldquoteaching to the testrdquo have been criticized tobe nearly as counterproductive as the attempt of helping seedlings grow by pulling themupward ldquoInterventionsrdquo and ldquotreatmentsrdquo under consideration do not exclude undesiredexperiences such as exposure to poverty abuse crime or bereavement A treatment plannedor unplanned becomes a focus of research if there are theoretical reasons to anticipate itsimpact positive or negative on the well-being of individuals who are embedded in socialsettings including families classrooms schools neighborhoods and workplaces

In social science research in general and in policy and program evaluations in particularquestions concerning whether an intervention works and if so which version of the interven-tion works for whom under what conditions and why are key to the advancement of scien-tific and practical knowledge Although most empirical investigations in the social sciencesconcentrate on the average effect of a treatment for a specific population as opposed to theabsence of such a treatment (ie the control condition) in-depth theoretical reasoning withregard to how the causal effect is generated substantiated by compelling empirical evidenceis crucial for advancing scientific understanding

First when there are multiple versions or different dosages of the treatment or when thereare multiple versions of the control condition a binary divide between ldquothe treatmentrdquo andldquothe controlrdquo may not be as informative as fine-grained comparisons across for exampleldquotreatment version Ardquo ldquotreatment version Brdquo ldquocontrol version Ardquo and ldquocontrol versionBrdquo For example expanding the federally funded Head Start program to poor children isexpected to generate a greater benefit when few early childhood education alternatives areavailable (call it ldquocontrol version Ardquo) than when there is an abundance of alternatives includ-ing state-sponsored preschool programs (call it ldquocontrol version Brdquo)

Second the effect of an intervention will likely vary among individuals or across socialsettings A famous example comes from medical research the well-publicized cardiovascularbenefits of initiating estrogen therapy during menopause were contradicted later by experi-mental findings that the same therapy increased postmenopausal womenrsquos risk for heartattacks The effect of an intervention may also depend on the provision of some other con-current or subsequent interventions Such heterogeneous effects are often characterized asmoderated effects in the literature

Third alternative theories may provide competing explanations for the causal mechan-isms that is the processes through which the intervention produces its effect A theoreticalconstruct characterizing the hypothesized intermediate process is called a mediator of theintervention effect The fictitious farmer never developed an elaborate theory as to whatcaused some seedlings to surpass others in growth Once scientists revealed the causal rela-tionship between access to chemical nutrients in soil and plant growth wide applications ofchemically synthesized fertilizers finally led to a major increase in crop production

Finally it is well known in agricultural experiments that a new type of fertilizer applied toone plot may spill-over to the next plot Because social connections among individuals areprevalent within organizations or through networks an individualrsquos response to the treatmentmay similarly depend on the treatment for other individuals in the same social setting whichmay lead to possible spill-overs of intervention effects among individual human beings

Answering questions with regard to moderation mediation and spill-over poses majorconceptual and analytic challenges To date psychological research often presents well-articulated theories of causal mechanisms relating stimuli to responses Yet researchers often

4 CAUSALITY IN A SOCIAL WORLD

lack rigorous analytic strategies for empirically screening competing theories explaining theobserved effect Sociologists have keen interest in the spill-over of treatment effects transmit-ted through social interactions yet have produced limited evidence quantifying such effectsAs many have pointed out in general terminological ambiguity and conceptual confusionhave been prevalent in the published applied research (Holmbeck 1997 Kraemer et al2002 2008)

A new framework for conceptualizing moderated mediated and spill-over effects hasemerged relatively recently in the statistics and econometrics literature on causal inference(eg Abbring and Heckman 2007 Frangakis and Rubin 2002 Heckman and Vytlacil2007a b Holland 1988 Hong and Raudenbush 2006 Hudgens and Halloran 2008Jo 2008 Pearl 2001 Robins and Greenland 1992 Sobel 2008) The potential for furtherconceptual and methodological development and for broad applications in the field of behav-ioral and social sciences promises to greatly advance the empirical basis of our knowledgeabout causality in the social world

This book clarifies theoretical concepts and introduces innovative statistical strategies forinvestigating the average effects of multivalued treatments moderated treatment effectsmediated treatment effects and spill-over effects in experimental or quasiexperimental dataDefining individual-specific and population average treatment effects in terms of potentialoutcomes the book relates the mathematical forms to the substantive meanings of moderatedmediated and spill-over effects in the context of application examples It also explicates andevaluates identification assumptions and contrasts innovative statistical strategies with con-ventional analytic methods

111 Moderated treatment effects

It is hard to accept the assumption that a treatment would produce the same impact for everyindividual in every possible circumstance Understanding the heterogeneity of treatmenteffects therefore is key to the development of causal theories For example some studiesreported that estrogen therapy improved cardiovascular health among women who initiatedits use during menopause According to a series of other studies however the use of estrogentherapy increased postmenopausal womenrsquos risk for heart attacks (Grodstein et al 19962000 Writing Group for the Womenrsquos Health Initiative Investigators 2002) The sharp con-trast of findings from these studies led to the hypothesis that age of initiation moderates theeffect of estrogen therapy on womenrsquos health (Manson and Bassuk 2007 Rossouw et al2007) Revelations of the moderated causal relationship greatly enrich theoretical understand-ing and in this case directly inform clinical practice

In another example dividing students by ability into small groups for reading instructionhas been controversial for decades Many believed that the practice benefits high-ability stu-dents at the expense of their low-ability peers and exacerbates educational inequality (Grantand Rothenberg 1986 Rowan and Miracle 1983 Trimble and Sinclair 1987) Recentevidence has shown that first of all the effect of ability grouping depends on a number ofconditions including whether enough time is allocated to reading instruction and how wellthe class can be managed Hence instructional time and class management are among theadditional moderators to consider for determining the merits of ability grouping (Hong andHong 2009 Hong et al 2012a b) Moreover further investigations have generated evidencethat contradicts the long-held belief that grouping undermines the interests of low-abilitystudents Quite the contrary researchers have found that grouping is beneficial for students

5INTRODUCTION

with low or medium prior abilitymdashif adequate time is provided for instruction and if the classis well managed On the other hand ability grouping appears to have a minimal impact onhigh-ability studentsrsquo literacy These results were derived from kindergarten data and maynot hold for math instruction and for higher grade levels Replications with different subpo-pulations and in different contexts enable researchers to assess the generalizability of a theory

In a third example one may hypothesize that a student is unlikely to learn algebra well inninth grade without a solid foundation in eighth-grade prealgebra One may also argue that theprogress a student made in learning eighth-grade prealgebra may not be sustained without thefurther enhancement of taking algebra in ninth grade In other words the earlier treatment(prealgebra in eighth grade) and the later treatment (algebra in ninth grade) may reinforceone other each moderates the effect of the other treatment on the final outcome (math achieve-ment at the end of ninth grade) As Hong and Raudenbush (2008) argued the cumulativeeffect of two treatments in a well-aligned sequence may exceed the sum of the benefits oftwo single-year treatments Similarly experiencing two consecutive years of inferior treat-ments such as encountering an incompetent math teacher who dampens a studentrsquos self-efficacy in math learning in eighth grade and then again in ninth grade could do more damagethan the sum of the effect of encountering an incompetent teacher only in eighth grade and theeffect of having such a teacher only in ninth grade

There has been a great amount of conceptual confusion regarding how a moderatorrelates to the treatment and the outcome On one hand many researchers have used the termsldquomoderatorsrdquo and ldquomediatorsrdquo interchangeably without understanding the crucial distinctionbetween the two An overcorrective attempt on the other hand has led to the arbitrary rec-ommendation that a moderator must occur prior to the treatment and be minimally associatedwith the treatment (James and Brett 1984 Kraemer et al 2001 2008) In Chapter 6 weclarify that in essence the causal effect of the treatment on the outcome may depend onthe moderator value A moderator can be a subpopulation identifier a contextual character-istic a concurrent treatment or a preceding or succeeding treatment In the earlier examplesage of estrogen therapy initiation and a studentrsquos prior ability are subpopulation identifiers themanageability of a class which reflects both teacher skills and peer behaviors characterizes thecontext literacy instruction time is a concurrent treatment while prealgebra is a precedingtreatment for algebra A moderator does not have to occur prior to the treatment and doesnot have to be independent of the treatment

Once a moderated causal relationship has been defined in terms of potential outcomes theresearcher then chooses an appropriate experimental design for testing the moderation theoryThe assumptions required for identifying the moderated causal relationships differ across dif-ferent designs and have implications for analyzing experimental and quasiexperimental dataRandomized block designs are suitable for examining individual or contextual characteristicsas potential moderators factorial designs enable one to determine the joint effects of two ormore concurrent treatments and sequential randomized designs are ideal for assessing thecumulative effects of consecutive treatments Multisite randomized trials constitute anadditional type of designs in which the experimental sites are often deliberately sampled torepresent a population of geographical locations or social organizations Site membershipscan be viewed as implicit moderators that summarize a host of features of a local environmentReplications of a treatment over multiple sites allow one to quantify the heterogeneity of treat-ment effects across the sites Chapters 7 and 8 are focused on statistical methods for moder-ation analyses with quasiexperimental data In particular these chapters demonstrate througha number of application examples how a nonparametric marginal mean weighting through

6 CAUSALITY IN A SOCIAL WORLD

stratification (MMWS) method can overcome some important limitations of other existingmethods

Moderation questions are not restricted to treatmentndashoutcome relationships Rather theyare prevalent in investigations of mediation and spill-over This is because heterogeneity oftreatment effects may be explained by the variation in causal mediation mechanisms acrosssubpopulations and contexts It is also because the treatment effect for a focal individualmay depend on whether there is spill-over from other individuals through social interactionsYet similar to the confusion around the concept of moderation among applied researchersthere has been a great amount of misunderstanding with regard to mediation

112 Mediated treatment effects

Questions about mediation are at the core of nearly every scientific theory In its simplestform a theory explains why treatment Z causes outcome Y by hypothesizing an intermediateprocess involving at least one mediatorM that could have been changed by the treatment andcould subsequently have an impact on the outcome A causal mediation analysis then decom-poses the total effect of the treatment on the outcome into two parts the effect transmittedthrough the hypothesized mediator called the indirect effect and the difference betweenthe total effect and the indirect effect called the direct effect The latter represents the treat-ment effect channeled through other unspecified pathways (Alwin and Hauser 1975 Baronand Kenny 1986 Duncan 1966 Shadish and Sweeney 1991) Most researchers in socialsciences have followed the convention of illustrating the direct effect and the indirect effectwith path diagrams and then specifying linear regression models postulated to represent thestructural relationships between the treatment the mediator and the outcome

In Chapter 9 we point out that the approach to defining the indirect effect and the directeffect in terms of path coefficients is often misled by oversimplistic assumptions about thestructural relationships In particular this approach typically overlooks the fact that a treat-ment may generate an impact on the outcome not only by changing the mediator value butalso by changing the mediatorndashoutcome relationship (Judd and Kenny 1981) An examplein Chapter 9 illustrates such a case An experiment randomizes students to either an experi-mental condition which provides them with study materials and encourages them to study fora test or to a control condition which provides neither study materials nor encouragement(Holland 1988 Powers and Swinton 1984) One might hypothesize that encouraging exper-imental group members to study will increase the time they spend studying a focal mediatorin this example which in turn will increase their average test scores One might furtherhypothesize that even without a change in study time providing study materials to experi-mental group members will enable them to study more effectively than they otherwise wouldConsequently the effect of time spent studying might be greater under the experimentalcondition than under the control condition Omitting the treatment-by-mediator interactioneffect will lead to bias in the estimation of the indirect effect and the direct effect

Chapter 11 describes in great detail an evaluation study contrasting a new welfare-to-workprogram emphasizing active participation in the labor force with a traditional program thatguaranteed cash assistance without a mandate to seek employment (Hong Deutsch and Hill2011) Focusing on the psychological well-being of welfare applicants who were singlemothers with preschool-aged children the researchers hypothesized that the treatment wouldincrease employment rate among the welfare recipients and that the treatment-inducedincrease in employment would likely reduce depressive symptoms under the new program

7INTRODUCTION

They also hypothesized that in contrast the same increase in employment was unlikely tohave a psychological impact under the traditional program Hence the treatment-by-mediatorinteraction effect on the outcome was an essential component of the intermediate process

We will show that by defining the indirect effect and the direct effect in terms of potentialoutcomes one can avoid invoking unwarranted assumptions about the unknown structuralrelationships (Holland 1988 Pearl 2001 Robins and Greenland 1992) The potential out-comes framework has further clarified that in a causal mediation theory the treatment and themediator are both conceivably manipulable In the aforementioned example a welfare appli-cant could be assigned at random to either the new program or the traditional one Once havingbeen assigned to one of the programs the individual might or might not be employed due tovarious structural constraints market fluctuations or other random events typically beyondher control

Because the indirect effect and the direct effect are defined in terms of potential outcomesrather than on the basis of any specific regression models it becomes possible to distinguishthe definitions of these causal effects from their identification and estimation The identifica-tion step relates a causal parameter to observable population data For example for individualsassigned at random to an experimental condition their average counterfactual outcome asso-ciated with the control condition is expected to be equal to the average observable outcome ofthose assigned to the control condition Therefore the population average causal effect can beeasily identified in a randomized experiment The estimation step then relates the sample datato the observable population quantities involved in identification while taking into account thedegree of sampling variability

A major challenge in identification is that the mediatorndashoutcome relationship tends tobe confounded by selection even if the treatment is randomized Chapter 9 reviews variousexperimental designs that have been proposed by past researchers for studying causalmediation mechanisms Chapter 10 compares the identification assumptions and the analyticprocedures across a wide range of analytic methods These discussions are followed by anintroduction of the ratio-of-mediator-probability weighting (RMPW) strategy in Chapter 11In comparison with most existing strategies for causal mediation analysis RMPW relies onrelatively fewer identification assumptions and model-based assumptions Chapters 12 and13 will show that this new strategy can be applied broadly with extensions to multilevelexperimental designs and to studies of complex mediation mechanisms involving multiplemediators

113 Spill-over effects of a treatment

It is well known in vaccination research that an unvaccinated person can benefit when mostother people in the local community are vaccinated This is viewed as a spill-over of the treat-ment effect from the vaccinated people to an unvaccinated person Similarly due to socialinteractions among individuals or groups of individuals an intervention received by somemay generate a spill-over impact on others affiliated with the same organization or connectedthrough the same network For example effective policing in one neighborhood may driveoffenders to operate in other neighborhoods In evaluating an innovative community policingprogram researchers found that a neighborhood not assigned to community policing tended tosuffer if the surrounding neighborhoods were assigned to community policing This is aninstance in which a well-intended treatment generates an unintended negative spill-over effectHowever being assigned to community policy was found particularly beneficial when thesurrounding neighborhoods also received the intervention (Verbitsky-Savitz and Raudenbush

8 CAUSALITY IN A SOCIAL WORLD

2012) In another example the impact of a school-based mentoring program targeted atstudents displaying delinquent behaviors may be enhanced for a focal student if his or herat-risk peers are assigned to the mentoring program at the same time

The previous examples challenge the ldquostable unit treatment value assumptionrdquo (SUTVA)that has been invoked in most causal inference studies This assumption states that an indi-vidualrsquos potential response to a treatment depends neither on how the treatment is assignednor on the treatment assignments of other individuals (Rubin 1980 1986) Rubin and othersbelieve that without resorting to SUTVA the causal effect of a treatment becomes hard todefine and that causal inference may become intractable However as Sobel (2006) has illus-trated with the Moving to Opportunity (MTO) experiment that offered housing vouchersenabling low-income families to move to low-poverty neighborhoods there are possible con-sequences of violating SUTVA in estimating the treatment effects on outcomes such as safetyand self-sufficiency Because social interactions among individuals may affect whether onevolunteers to participate in the study whether one moves to a low-poverty neighborhood afterreceiving the housing voucher as well as housing project residentsrsquo subjective perceptions oftheir neighborhoods the program may have a nonzero impact on the potential outcome of theuntreated As a result despite the randomization of treatment assignment the mean differencein the observed outcome between the treated units and the untreated units may be biased forthe average treatment effect Rather the observable quantity is the difference between theeffect of treating some rather than treating none for the treated and the effect of treating somerather than treating none for the untreated the latter being the pure spill-over effect for theuntreated

In making attempts to relax SUTVA researchers have proposed several alternative frame-works that incorporate possible spill-over effects (see Hong and Raudenbush 2013 for areview) Hong (2004) presented a model that involves treatments and treatment settingsA treatment setting for an individual is a local environment constituted by a set of agentsand participants along with their treatment assignments An individualrsquos potential outcomevalue under a given treatment is assumed stable when the treatment setting is fixed the poten-tial outcome may take different values when the treatment setting shifts One may investigatewhether the treatment effect depends on the treatment setting Applying this framework Hongand Raudenbush (2006) examined the effect on a childrsquos academic growth of retaining thechild in kindergarten rather than promoting the child to first grade when a relatively smallproportion of low-achieving peers in the same school are retained as opposed to when a rel-atively large proportion of the peers are retained Hudgens and Halloran (2008) presented arelated framework in which the effect on an individual of the treatment received by this indi-vidual is distinguished from the effect on the individual of the treatment received by others inthe same local community These effects can be identified if communities are assigned atrandom to different treatment assignment strategies (such as retaining a large proportion oflow-achieving students as opposed to retaining a small proportion of such students) and sub-sequently individuals within a community are randomized for treatment Chapter 14 reviewsthese frameworks and discusses identification and estimation strategies

Social contagion may also serve as an important channel through which the effect of anintervention is transmitted For example a school-wide intervention may reduce aggressivebehaviors and thereby improve studentsrsquo psychological well-being by improving the qualityof interpersonal relationships in other classes as well as in onersquos own class This is becausechildren interact not simply with their classmates but also with those from other classes in thehallways or on the playground (VanderWeele et al 2013) In this case the spill-over betweenclasses becomes a part of the mediation mechanism In a study of student mentoring whether

9INTRODUCTION

an individual actually participated in the mentoring program despite the initial treatmentassignment is typically viewed as a primary mediator The proportion of onersquos peers that par-ticipated in the program may act as a second mediator A treatment may also exert its impactpartly through regrouping individuals and thereby changing peer composition (Hong andNomi 2012) Chapter 15 discusses analytic strategies for detecting spill-over as a part ofthe causal mediation mechanism

12 Weighting methods for causal inference

In most behavioral and social science applications major methodological challenges arise dueto the selection of treatments in a quasiexperimental study and the selection of mediator valuesin experimental and quasiexperimental studies Statistical methods most familiar to research-ers in social sciences are often inadequate for causal inferences with regard to multivaluedtreatments moderation mediation and spill-over The book offers a major revision to under-standing of causality in the social world by introducing two complementary weighting stra-tegies both featuring nonparametric approaches to estimating these causal effects The newweighting methods greatly simplify model specifications while enhancing the robustness ofresults by minimizing the reliance on some key assumptions

The propensity score-basedMMWSmethod removes selection bias associated with a largenumber of covariates by equating the pretreatment composition between treatment groups(Hong 2010a 2012 Huang et al 2005) Unlike propensity score matching and stratificationthat are mostly restricted to evaluations of binary treatments the MMWS method is flexiblefor evaluating binary and multivalued treatments by approximating a completely randomizedexperiment In evaluating whether the treatment effects differ across subpopulations definedby individual characteristics or treatment settings researchers may assign weights within eachsubpopulation in order to approximate a randomized block design To investigate whetherone treatment moderates the effect of another concurrent treatment researchers may assignweights to the data to approximate a factorial randomized design The method can also beused to assess whether the effect of an initial treatment is amplified or weakened by a subse-quent treatment or to identify an optimal treatment sequence through approximating a sequen-tial randomized experiment Even though such analyses can similarly be conducted throughinverse-probability-of-treatment weighting (IPTW) that has been increasingly employed inepidemiological research (Hernaacuten Brumback and Robins 2000 Robins Hernaacuten andBrumback 2000) IPTW is known for bias and imprecision in estimation especially whenthe propensity score models are misspecified in their functional forms (Hong 2010a Kangand Schafer 2007 Schafer and Kang 2008 Waernbaum 2012) In contrast the nonparamet-ric MMWS method displays a relatively high level of robustness despite such misspecifica-tions and also gains efficiency as indicated by simulation results (Hong 2010a)

To study causal mediation mechanisms the RMPW method decomposes the total effectof a treatment into an ldquoindirect effectrdquo transmitted through a specific mediator and a ldquodirecteffectrdquo representing unspecified mechanisms In contrast with most existing methods formediation analysis the RMPW-adjusted outcome model is extremely simple and is nonpara-metric in nature It generates estimates of the causal effects along with their sampling errorswhile adjusting for pretreatment covariates that confound the mediatorndashoutcome relationshipsthrough weighting The method applies regardless of the distribution of the outcome the dis-tribution of the mediator or the functional relationship between the outcome and the mediator

10 CAUSALITY IN A SOCIAL WORLD

Page 6: Thumbnail - download.e-bookshelf.de...4.2.4 IPTW for bias removal 84 4.3 MMWS 86 4.3.1 Theoretical rationale 86 ... 8.3.4.2 Growth model in the absence of confounding 200 8.3.4.3 Weighted

312 Randomized block designs 41313 Covariance adjustment for improving efficiency 43314 Multilevel experimental designs 43

32 Quasiexperimental designs 44321 Nonequivalent comparison group designs 44322 Other quasiexperimental designs 45

33 Statistical adjustment methods 46331 ANCOVA and multiple regression 46

3311 ANCOVA for removing selection bias 463312 Potential pitfalls of ANCOVA with a vast between-group

difference 473313 Bias due to model misspecification 48

332 Matching and stratification 50333 Other statistical adjustment methods 51

3331 The IV method 513332 DID analysis 54

34 Propensity score 55341 What is a propensity score 56342 Balancing property of the propensity score 57343 Pooling conditional treatment effect estimate Matching

stratification and covariance adjustment 603431 Propensity score matching 613432 Propensity score stratification 623433 Covariance adjustment for the propensity score 663434 Sensitivity analysis 66

Appendix 3A Potential bias due to the omission of treatment-by-covariateinteraction 70

Appendix 3B Variable selection for the propensity score model 71

4 Adjustment for selection bias through weighting 7641 Weighted estimation of population parameters in survey sampling 77

411 Simple random sample 77412 Proportionate sample 78413 Disproportionate sample 79

42 Weighting adjustment for selection bias in causal inference 80421 Experimental result 81422 Quasiexperimental result 81423 Sample weight for bias removal 82424 IPTW for bias removal 84

43 MMWS 86431 Theoretical rationale 86

4311 MMWS for a discrete propensity score 874312 MMWS for a continuous propensity score 884313 MMWS for estimating the treatment effect on the treated 89

432 MMWS analytic procedure 91433 Inherent connection and major distinctions between MMWS

and IPTW 93

vi CONTENTS

Appendix 4A Proof of MMWS-adjusted mean observed outcome being unbiasedfor the population average potential outcome 95

Appendix 4B Derivation of MMWS for estimating the treatment effecton the treated 96

Appendix 4C Theoretical equivalence of MMWS and IPTW 97Appendix 4D Simulations comparing MMWS and IPTW under misspecifications

of the functional form of a propensity score model 97

5 Evaluations of multivalued treatments 10051 Defining the causal effects of multivalued treatments 10052 Existing designs and analytic methods for evaluating multivalued

treatments 102521 Experimental designs and analysis 102

5211 Randomized experiments with multipletreatment arms 102

5212 Identification under ignorability 1025213 ANOVA 103

522 Quasiexperimental designs and analysis 1055221 ANCOVA and multiple regression 1055222 Propensity score-based adjustment 1085223 Other adjustment methods 112

53 MMWS for evaluating multivalued treatments 112531 Basic rationale 113532 Analytic procedure 114

5321 MMWS for a multinomial treatment measure 1155322 MMWS for an ordinal treatment measure 120

533 Identification assumptions 12154 Summary 123Appendix 5A Multiple IV for evaluating multivalued treatments 124

Part II Moderation 127

6 Moderated treatment effects concepts and existing analytic methods 12961 What is moderation 129

611 Past discussions of moderation 1306111 Purpose of moderation research 1306112 What qualifies a variable as a moderator 132

612 Definition of moderated treatment effects 1336121 Treatment effects moderated by individual or contextual

characteristics 1336122 Joint effects of concurrent treatments 134

62 Experimental designs and analytic methods for investigatingexplicit moderators 136621 Randomized block designs 137

6211 Identification assumptions 1376212 Two-way ANOVA 1386213 Multiple regression 139

viiCONTENTS

622 Factorial designs 1406221 Identification assumptions 1406222 Analytic strategies 142

63 Existing research designs and analytic methods for investigatingimplicit moderators 142631 Multisite randomized trials 143

6311 Causal parameters and identification assumptions 1446312 Analytic strategies 145

632 Principal stratification 149Appendix 6A Derivation of bias in the fixed-effects estimator when the

treatment effect is heterogeneous in multisite randomized trials 151Appendix 6B Derivation of bias in the mixed-effects estimator when the

probability of treatment assignment varies across sites 153Appendix 6C Derivation and proof of the population weight applied to

mixed-effects models for eliminating bias in multisiterandomized trials 153

7 Marginal mean weighting through stratification for investigatingmoderated treatment effects 15971 Existing methods for moderation analyses with quasiexperimental data 159

711 Analysis of covariance and regression-based adjustment 1617111 Treatment effects moderated by subpopulation membership 1617112 Treatment effects moderated by a concurrent treatment 164

712 Propensity score-based adjustment 1657121 Propensity score matching and stratification 1667122 Inverse-probability-of-treatment weighting 167

72 MMWS estimation of treatment effects moderated by individual orcontextual characteristics 168721 Application example 170722 Analytic procedure 170

73 MMWS estimation of the joint effects of concurrent treatments 174731 Application example 174732 Analytic procedure 175733 Joint treatment effects moderated by individual or contextual

characteristics 179

8 Cumulative effects of time-varying treatments 18581 Causal effects of treatment sequences 186

811 Application example 186812 Causal parameters 187

8121 Time-varying treatments 1878122 Time-varying potential outcomes 1878123 Causal effects of 2-year treatment sequences 1878124 Causal effects of multiyear treatment sequences 190

82 Existing strategies for evaluating time-varying treatments 190821 The endogeneity problem in nonexperimental data 190822 SEM 191

viii CONTENTS

823 Fixed-effects econometric models 192824 Sequential randomization 192825 Dynamic treatment regimes 193826 Marginal structural models and structural nested models 194

83 MMWS for evaluating 2-year treatment sequences 195831 Sequential ignorability 195832 Propensity scores 196833 MMWS computation 197

8331 MMWS adjustment for year 1 treatment selection 1978332 MMWS adjustment for two-year treatment

sequence selection 198834 Two-year growth model specification 199

8341 Growth model in the absence of treatment 2008342 Growth model in the absence of confounding 2008343 Weighted 2-year growth model 202

84 MMWS for evaluating multiyear sequences of multivalued treatments 204841 Sequential ignorability of multiyear treatment sequences 204842 Propensity scores for multiyear treatment sequences 204843 MMWS computation 205844 Weighted multiyear growth model 205845 Issues of sample size 206

85 Conclusion 207Appendix 8A A saturated model for evaluating multivalued treatments

over multiple time periods 207

Part III Mediation 211

9 Concepts of mediated treatment effects and experimental designsfor investigating causal mechanisms 21391 Introduction 21492 Path coefficients 21593 Potential outcomes and potential mediators 216

931 Controlled direct effects 217932 Controlled treatment-by-mediator interaction effect 217

94 Causal effects with counterfactual mediators 219941 Natural direct effect 219942 Natural indirect effect 220943 Natural treatment-by-mediator interaction effect 220944 Unstable unit treatment value 221

95 Population causal parameters 222951 Population average natural direct effect 224952 Population average natural indirect effect 225

96 Experimental designs for studying causal mediation 225961 Sequentially randomized designs 228962 Two-phase experimental designs 228963 Three- and four-treatment arm designs 230964 Experimental causal-chain designs 231

ixCONTENTS

965 Moderation-of-process designs 231966 Augmented encouragement designs 232967 Parallel experimental designs and parallel encouragement designs 232968 Crossover experimental designs and crossover encouragement

designs 233969 Summary 234

10 Existing analytic methods for investigating causal mediation mechanisms 238101 Path analysis and SEM 239

1011 Analytic procedure for continuous outcomes 2391012 Identification assumptions 2421013 Analytic procedure for discrete outcomes 245

102 Modified regression approach 2461021 Analytic procedure for continuous outcomes 2461022 Identification assumptions 2471023 Analytic procedure for binary outcomes 248

103 Marginal structural models 2501031 Analytic procedure 2501032 Identification assumptions 252

104 Conditional structural models 2521041 Analytic procedure 2521042 Identification assumptions 253

105 Alternative weighting methods 2541051 Analytic procedure 2541052 Identification assumptions 256

106 Resampling approach 2561061 Analytic procedure 2561062 Identification assumptions 257

107 IV method 2571071 Rationale and analytic procedure 2571072 Identification assumptions 258

108 Principal stratification 2591081 Rationale and analytic procedure 2591082 Identification assumptions 260

109 Sensitivity analysis 2611091 Unadjusted confounding as a product of hypothetical

regression coefficients 2611092 Unadjusted confounding reflected in a hypothetical

correlation coefficient 2621093 Limitations when the selection mechanism differs

by treatment 2641094 Other sensitivity analyses 265

1010 Conclusion 26510101 The essentiality of sequential ignorability 26510102 Treatment-by-mediator interactions 26610103 Homogeneous versus heterogeneous causal effects 26610104 Model-based assumptions 266

x CONTENTS

Appendix 10A Bias in path analysis estimation due to the omission oftreatment-by-mediator interaction 267

11 Investigations of a simple mediation mechanism 273111 Application example national evaluation of welfare-to-work strategies 274

1111 Historical context 2741112 Research questions 2751113 Causal parameters 2751114 NEWWS Riverside data 277

112 RMPW rationale 2771121 RMPW in a sequentially randomized design 278

11211 E[Y(0 M(0))] 27911212 E[Y(1 M(1))] 28011213 E[Y(1 M(0))] 28011214 E[Y(0 M(1))] 28111215 Nonparametric outcome model 282

1122 RMPW in a sequentially randomized block design 2831123 RMPW in a standard randomized experiment 2851124 Identification assumptions 286

113 Parametric RMPW procedure 287114 Nonparametric RMPW procedure 290115 Simulation results 292

1151 Correctly specified propensity score models 2921152 Misspecified propensity score models 2941153 Comparisons with path analysis and IV results 294

116 Discussion 2951161 Advantages of the RMPW strategy 2951162 Limitations of the RMPW strategy 295

Appendix 11A Causal effect estimation through the RMPW procedure 296Appendix 11B Proof of the consistency of RMPW estimation 297

12 RMPW extensions to alternative designs and measurement 301121 RMPW extensions to mediators and outcomes of alternative distributions 301

1211 Extensions to a multicategory mediator 30212111 Parametric RMPW procedure 30212112 Nonparametric RMPW procedure 304

1212 Extensions to a continuous mediator 3041213 Extensions to a binary outcome 306

122 RMPW extensions to alternative research designs 3061221 Extensions to quasiexperimental data 3071222 Extensions to data from cluster randomized trials 308

12221 Application example and research questions 30912222 Estimation of the total effect 31012223 RMPW analysis of causal mediation mechanisms 31012224 Identification assumptions 31212225 Contrast with multilevel path analysis and SEM 31212226 Contrast with multilevel prediction models 313

xiCONTENTS

1223 Extensions to data from multisite randomized trials 31312231 Research questions and causal parameters 31412232 Estimation of the total effect and its between-site variation 31512233 RMPW analysis of causal mediation mechanisms 31612234 Identification assumptions 31912235 Contrast with multilevel path analysis and SEM 319

123 Alternative decomposition of the treatment effect 321

13 RMPW extensions to studies of complex mediation mechanisms 325131 RMPW extensions to moderated mediation 325

1311 RMPW analytic procedure for estimating and testingmoderated mediation 326

1312 Path analysisSEM approach to analyzing moderatedmediation 327

1313 Principal stratification and moderated mediation 328132 RMPW extensions to concurrent mediators 328

1321 Treatment effect decomposition 32913211 Treatment effect decomposition without

between-mediator interaction 32913212 Treatment effect decomposition with

between-mediator interaction 3311322 Identification assumptions 3331323 RMPW procedure 333

13231 Estimating E[Y(1M1(1)M2(0))] 33513232 Estimating E[Y(1M1(0)M2(0))] 33513233 Causal effect estimation with noninteracting

concurrent mediators 33613234 Estimating E[Y(1M1(0)M2(1))] 33713235 Causal effect estimation with interacting concurrent

mediators 3371324 Contrast with the linear SEM approach 3381325 Contrast with the multivariate IV approach 339

133 RMPW extensions to consecutive mediators 3401331 Treatment effect decomposition 341

13311 Natural direct effect of the treatment on the outcome 34213312 Natural indirect effect mediated by M1 only 34213313 Natural indirect effect mediated by M2 only 34213314 Natural indirect effect mediated by an M1-by-M2

interaction 34313315 Treatment-by-mediator interactions 343

1332 Identification assumptions 3451333 RMPW procedure 347

13331 Estimating E[Y(1M1(0)M2(0M1(0)))] 34713332 Estimating E[Y(1M1(1)M2(0M1(0)))] 34913333 Estimating E[Y(1M1(0)M2(1M1(1)))] 34913334 Estimating E[Y(0M1(1)M2(0M1(0)))] and

E[Y(0M1(0)M2(1M1(1)))] 351

xii CONTENTS

1334 Contrast with the linear SEM approach 3531335 Contrast with the sensitivity-based estimation of bounds for

causal effects 354134 Discussion 355Appendix 13A Derivation of RMPW for estimating population average

counterfactual outcomes of two concurrentmediators 355

Appendix 13B Derivation of RMPW for estimating population averagecounterfactual outcomes of consecutivemediators 358

Part IV Spill-over 363

14 Spill-over of treatment effects concepts and methods 365141 Spill-over A nuisance a trifle or a focus 365142 Stable versus unstable potential outcome values An example

from agriculture 367143 Consequences for causal inference when spill-over is overlooked 369144 Modified framework of causal inference 371

1441 Treatment settings 3711442 Simplified characterization of treatment settings 3731443 Causal effects of individual treatment assignment and of peer

treatment assignment 375145 Identification Challenges and solutions 376

1451 Hypothetical experiments for identifying average treatmenteffects in the presence of social interactions 376

1452 Hypothetical experiments for identifying the impact ofsocial interactions 380

1453 Application to an evaluation of kindergarten retention 382146 Analytic strategies for experimental and quasiexperimental data 384

1461 Estimation with experimental data 3841462 Propensity score stratification 3851463 MMWS 386

147 Summary 387

15 Mediation through spill-over 391151 Definition of mediated effects through spill-over in a cluster

randomized trial 3931511 Notation 3931512 Treatment effect mediated by a focal individualrsquos

compliance 3941513 Treatment effect mediated by peersrsquo compliance through

spill-over 3941514 Decomposition of the total treatment effect 395

152 Identification and estimation of the spill-over effect in a clusterrandomized design 3951521 Identification in an ideal experiment 395

xiiiCONTENTS

1522 Identification when the mediators are not randomized 3981523 Estimation of mediated effects through spill-over 400

153 Definition of mediated effects through spill-over in a multisite trial 4021531 Notation 4021532 Treatment effect mediated by a focal individualrsquos compliance 4041533 Treatment effect mediated by peersrsquo compliance

through spill-over 4041534 Direct effect of individual treatment assignment on the outcome 4051535 Direct effect of peer treatment assignment on the outcome 4051536 Decomposition of the total treatment effect 405

154 Identification and estimation of spill-over effects in a multisite trial 4061541 Identification in an ideal experiment 4071542 Identification when the mediators are not randomized 4091543 Estimation of mediated effects through spill-over 410

15431 Estimating E[Y(1pM(p)Mminus(p))] andE[Y(000Mminus(0))] 410

15432 Estimating E[Y(1p0Mminus(p))] 41115433 Estimating E[Y(1p0Mminus(0))] 41215434 Estimating E[Y(0p0Mminus(0))] 412

155 Consequences of omitting spill-over effects in causal mediation analyses 4121551 Biased inference in a cluster randomized trial 4131552 Biased inference in a multisite randomized trial 4131553 Biased inference of the local average treatment effect 415

156 Quasiexperimental application 416157 Summary 419Appendix 151 Derivation of the weight for estimating the population

average counterfactual outcome E[Y(1 p 0Mminus( p))] 419Appendix 152 Derivation of bias in the ITT effect due to the omission of

spill-over effects 420

Index 423

xiv CONTENTS

Preface

A scientific mind in training seeks comprehensive descriptions of every interesting phenom-enon In such a mind data are to be pieced together for comparisons contrasts and eventuallyinferences that are required for understanding causes and effects that may have led to thephenomenon of interest Yet without disciplined reasoning a causal inference would slip intothe rut of a ldquocasual inferencerdquo as easily as making a typographical error

As a doctoral student thirsting for rigorous methodological training at the University ofMichigan I was very fortunate to be among the first group of students on the campus attendinga causal inference seminar in 2000 jointly taught by Yu Xie from the Department of Sociol-ogy Susan Murphy from the Department of Statistics and Stephen Raudenbush from theSchool of Education The course was unique in both content and pedagogy The instructorsput together a bibliography drawn from the past and current causal inference literature origi-nated from multiple disciplines In the spirit of ldquoreciprocal teachingrdquo the students were delib-erately organized into small groups each representing a mix of disciplinary backgrounds andtook turns to teach the weekly readings under the guidance of a faculty instructor This inval-uable experience revealed to me that the pursuit of causal inference is a multidisciplinaryendeavor In the rest of my doctoral study I continued to be immersed in an extraordinaryinterdisciplinary community in the form of the Quantitative Methodology Program (QMP)seminars led by the above three instructors who were then joined by Ben Hansen(Statistics) Richard Gonzalez (Psychology) Roderick Little (Biostatistics) Jake Bowers(Political Science) and many others I have tried whenever possible to carry the same spiritinto my own teaching of causal inference courses at the University of Toronto and now at theUniversity of Chicago In these courses communicating understandings of causal problemsacross disciplinary boundaries has been particularly challenging but also stimulating and grat-ifying for myself and for students from various departments and schools

Under the supervision of Stephen Raudenbush I took a first stab at the conceptual frame-work of causal inference in my doctoral dissertation by considering peer spill-over in schoolsettings From that point on I have organized my methodological research to tackle some ofthe major obstacles to causal inferences in policy and program evaluations My workaddresses issues including (i) how to conceptualize and evaluate the causal effects of educa-tional treatments when studentsrsquo responses to alternative treatments depend on various fea-tures of the organizational settings including peer composition (ii) how to adjust forselection bias in evaluating the effects of concurrent and consecutive multivalued treatmentsand (iii) how to conceptualize and analyze causal mediation mechanisms My methodological

research has been driven by the substantive interest in understanding the role of socialinstitutionsmdashschools in particularmdashin shaping human development especially during theages of fast growth (eg childhood adolescence and early adulthood) I have shared meth-odological advances with a broad audience in social sciences through applying the causalinference methods to prominent substantive issues such as grade retention within-class group-ing services for English language learners and welfare-to-work strategies These illustrationsare used extensively in this book

Among social scientists awareness of methodological challenges in causal investigationsis higher than ever before In response to this rising demand causal inference is becoming oneof the most productive scholarly fields in the past decade I have had the benefit of followingthe work and exchanging ideas with some fellow methodologists who are constantly contri-buting new thoughts and making breakthroughs These include Daniel Almirall HowardBloom Tom Cook Michael Elliot Ken Frank Ben Hansen James Heckman JenniferHill Martin Huber Kosuke Imai Booil Jo John R Lockwood David MacKinnon DanielMcCaffrey Luke Miratrix Richard Murnane Derek Neal Lindsey Page Judea PearlStephen Raudenbush Sean Reardon Michael Seltzer Youngyun Shin Betsy SinclairMichael Sobel Peter Steiner Elizabeth Stuart Eric Thetchen Tchetchen Tyler VanderWeeleand Kazuo Yamaguchi I am grateful to Larry Hedges who offered me the opportunity of guestediting the Journal of Research in Educational Effectiveness special issue on the statisticalapproaches to studying mediator effects in education research in 2012 Many of the colleagueswhom I mentioned above contributed to the open debate in the special issue by either propos-ing or critically examining a number of innovative methods for studying causal mechanismsI anticipate that many of them are or will soon be writing their own research monographs oncausal inference if they have not already done so Therefore readers of this book are stronglyrecommended to browse past and future titles by these and other authors for alternative per-spectives and approaches My own understanding of course will continue to evolve as well inthe coming years

One has to be ambitious to stay upfront in substantive areas as well as in methodology Thebest strategy apparently is to learn from substantive experts During different phases of mytraining and professional career I have had the opportunities to work with David K CohenBrian Rowan Deborah Ball Carl Corter Janette Pelletier Takako Nomi Esther Geva DavidFrancis Stephanie Jones Joshua Brown and Heather D Hill Many of my substantive insightswere derived from conversations with these mentors and colleagues I am especially gratefulto Bob Granger former president of the William T Grant Foundation who reassured me thatldquoBy staying a bit broad you will learn a lot and many fields will benefit from your workrdquo

Like many other single-authored books this one is built on the generous contributions ofstudents and colleagues who deserve special acknowledgement Excellent research assistancefrom Jonah Deutsch Joshua Gagne Rachel Garrett Yihua Hong Xu Qin Cheng Yang andBing Yu was instrumental in the development and implementation of the new methods pre-sented in this book Richard Congdon a renowned statistical software programmer broughtrevolutionary ideas to interface designs in software development with the aim of facilitatingusersrsquo decision-making in a series of causal analyses RichardMurnane Stephen Raudenbushand Michael Sobel carefully reviewed and critiqued multiple chapters of the manuscript draftAdditional comments and suggestions on earlier drafts came fromHoward Bloom Ken FrankBen Hansen Jennifer Hill Booil Jo Ben Kelcey David MacKinnon and Fan Yang TereseSchwartzman provided valuable assistance in manuscript preparation The writing of thisbook was supported by a Scholars Award from the William T Grant Foundation and an

xvi PREFACE

Institute of Education Sciences (IES) Statistical and Research Methodology in EducationGrant from the US Department of Education All the errors in the published form are ofcourse the sole responsibility of the author

Project Editors Richard Davis Prachi Sinha Sahay and Liz Wingett and Assistant EditorHeather Kay at Wiley and project Managers P Jayapriya and R Jayavel at SPi Global havebeen wonderful to work with throughout the manuscript preparation and final production ofthe book Debbie Jupe Commissioning Editor at Wiley was remarkably effective in initiatingthe contact with me and in organizing anonymous reviews of the book proposal and of themanuscript These constructive reviews have shaped the book in important ways I cannotagree more with one of the reviewers that ldquocausality cannot be established on a pure statisticalgroundrdquo The book therefore highlights the importance of substantive knowledge and researchdesigns and places a great emphasis on clarifying and evaluating assumptions required foridentifying causal effects in the context of each application Much caution is raised againstpossible misusage of statistical techniques for analyzing causal relationships especiallywhen data are inadequate Yet I maintain that improved statistical procedures along withimproved research designs would greatly enhance our ability in attempt to empiricallyexamine well-articulated causal theories

xviiPREFACE

Part I

OVERVIEW

1

Introduction

According to an ancient Chinese fable a farmer who was eager to help his crops grow wentinto his field and pulled each seedling upward After exhausting himself with the work heannounced to his family that they were going to have a good harvest only to find the nextmorning that the plants had wilted and died Readers with minimal agricultural knowledgemay immediately point out the following the farmerrsquos intervention theory was based on acorrect observation that crops that grow taller tend to produce more yield Yet his hypothesisreflects a false understanding of the cause and the effectmdashthat seedlings pulled to be tallerwould yield as much as seedlings thriving on their own

In their classic Design and Analysis of Experiments Hinkelmann and Kempthorne (1994updated version of Kempthorne 1952) discussed two types of science descriptive science andthe development of theory These two types of science are interrelated in the following senseobservations of an event and other related events often selected and classified for descriptionby scientists naturally lead to one or more explanations that we call ldquotheoretical hypothesesrdquowhich are then screened and falsified by means of further observations experimentation andanalyses (Popper 1963) The experiment of pulling seedlings to be taller was costly but didserve the purpose of advancing this farmerrsquos knowledge of ldquowhat does not workrdquo To developa successful intervention in this case would require a series of empirical tests of explicittheories identifying potential contributors to crop growth This iterative process graduallydeepens our knowledge of the relationships between supposed causes and effectsmdashthatis causalitymdashand may eventually increase the success of agricultural medical and socialinterventions

11 Concepts of moderation mediation and spill-over

Although the story of the ancient farmer is fictitious numerous examples can be found in thereal world in which well-intended interventions fail to produce the intended benefits or inmany cases even lead to unintended consequences ldquoInterventionsrdquo and ldquotreatmentsrdquo usedinterchangeably in this book broadly refer to actions taken by agents or circumstances expe-rienced by an individual or groups of individuals Interventions are regularly seen in educa-tion physical and mental health social services business politics and law enforcement In an

Causality in a Social World Moderation Meditation and Spill-over First Edition Guanglei Hongcopy 2015 John Wiley amp Sons Ltd Published 2015 by John Wiley amp Sons Ltd

education intervention for example teachers are typically the agents who deliver a treatmentto students while the impact of the treatment on student outcomes is of ultimate causalinterest Some educational practices such as ldquoteaching to the testrdquo have been criticized tobe nearly as counterproductive as the attempt of helping seedlings grow by pulling themupward ldquoInterventionsrdquo and ldquotreatmentsrdquo under consideration do not exclude undesiredexperiences such as exposure to poverty abuse crime or bereavement A treatment plannedor unplanned becomes a focus of research if there are theoretical reasons to anticipate itsimpact positive or negative on the well-being of individuals who are embedded in socialsettings including families classrooms schools neighborhoods and workplaces

In social science research in general and in policy and program evaluations in particularquestions concerning whether an intervention works and if so which version of the interven-tion works for whom under what conditions and why are key to the advancement of scien-tific and practical knowledge Although most empirical investigations in the social sciencesconcentrate on the average effect of a treatment for a specific population as opposed to theabsence of such a treatment (ie the control condition) in-depth theoretical reasoning withregard to how the causal effect is generated substantiated by compelling empirical evidenceis crucial for advancing scientific understanding

First when there are multiple versions or different dosages of the treatment or when thereare multiple versions of the control condition a binary divide between ldquothe treatmentrdquo andldquothe controlrdquo may not be as informative as fine-grained comparisons across for exampleldquotreatment version Ardquo ldquotreatment version Brdquo ldquocontrol version Ardquo and ldquocontrol versionBrdquo For example expanding the federally funded Head Start program to poor children isexpected to generate a greater benefit when few early childhood education alternatives areavailable (call it ldquocontrol version Ardquo) than when there is an abundance of alternatives includ-ing state-sponsored preschool programs (call it ldquocontrol version Brdquo)

Second the effect of an intervention will likely vary among individuals or across socialsettings A famous example comes from medical research the well-publicized cardiovascularbenefits of initiating estrogen therapy during menopause were contradicted later by experi-mental findings that the same therapy increased postmenopausal womenrsquos risk for heartattacks The effect of an intervention may also depend on the provision of some other con-current or subsequent interventions Such heterogeneous effects are often characterized asmoderated effects in the literature

Third alternative theories may provide competing explanations for the causal mechan-isms that is the processes through which the intervention produces its effect A theoreticalconstruct characterizing the hypothesized intermediate process is called a mediator of theintervention effect The fictitious farmer never developed an elaborate theory as to whatcaused some seedlings to surpass others in growth Once scientists revealed the causal rela-tionship between access to chemical nutrients in soil and plant growth wide applications ofchemically synthesized fertilizers finally led to a major increase in crop production

Finally it is well known in agricultural experiments that a new type of fertilizer applied toone plot may spill-over to the next plot Because social connections among individuals areprevalent within organizations or through networks an individualrsquos response to the treatmentmay similarly depend on the treatment for other individuals in the same social setting whichmay lead to possible spill-overs of intervention effects among individual human beings

Answering questions with regard to moderation mediation and spill-over poses majorconceptual and analytic challenges To date psychological research often presents well-articulated theories of causal mechanisms relating stimuli to responses Yet researchers often

4 CAUSALITY IN A SOCIAL WORLD

lack rigorous analytic strategies for empirically screening competing theories explaining theobserved effect Sociologists have keen interest in the spill-over of treatment effects transmit-ted through social interactions yet have produced limited evidence quantifying such effectsAs many have pointed out in general terminological ambiguity and conceptual confusionhave been prevalent in the published applied research (Holmbeck 1997 Kraemer et al2002 2008)

A new framework for conceptualizing moderated mediated and spill-over effects hasemerged relatively recently in the statistics and econometrics literature on causal inference(eg Abbring and Heckman 2007 Frangakis and Rubin 2002 Heckman and Vytlacil2007a b Holland 1988 Hong and Raudenbush 2006 Hudgens and Halloran 2008Jo 2008 Pearl 2001 Robins and Greenland 1992 Sobel 2008) The potential for furtherconceptual and methodological development and for broad applications in the field of behav-ioral and social sciences promises to greatly advance the empirical basis of our knowledgeabout causality in the social world

This book clarifies theoretical concepts and introduces innovative statistical strategies forinvestigating the average effects of multivalued treatments moderated treatment effectsmediated treatment effects and spill-over effects in experimental or quasiexperimental dataDefining individual-specific and population average treatment effects in terms of potentialoutcomes the book relates the mathematical forms to the substantive meanings of moderatedmediated and spill-over effects in the context of application examples It also explicates andevaluates identification assumptions and contrasts innovative statistical strategies with con-ventional analytic methods

111 Moderated treatment effects

It is hard to accept the assumption that a treatment would produce the same impact for everyindividual in every possible circumstance Understanding the heterogeneity of treatmenteffects therefore is key to the development of causal theories For example some studiesreported that estrogen therapy improved cardiovascular health among women who initiatedits use during menopause According to a series of other studies however the use of estrogentherapy increased postmenopausal womenrsquos risk for heart attacks (Grodstein et al 19962000 Writing Group for the Womenrsquos Health Initiative Investigators 2002) The sharp con-trast of findings from these studies led to the hypothesis that age of initiation moderates theeffect of estrogen therapy on womenrsquos health (Manson and Bassuk 2007 Rossouw et al2007) Revelations of the moderated causal relationship greatly enrich theoretical understand-ing and in this case directly inform clinical practice

In another example dividing students by ability into small groups for reading instructionhas been controversial for decades Many believed that the practice benefits high-ability stu-dents at the expense of their low-ability peers and exacerbates educational inequality (Grantand Rothenberg 1986 Rowan and Miracle 1983 Trimble and Sinclair 1987) Recentevidence has shown that first of all the effect of ability grouping depends on a number ofconditions including whether enough time is allocated to reading instruction and how wellthe class can be managed Hence instructional time and class management are among theadditional moderators to consider for determining the merits of ability grouping (Hong andHong 2009 Hong et al 2012a b) Moreover further investigations have generated evidencethat contradicts the long-held belief that grouping undermines the interests of low-abilitystudents Quite the contrary researchers have found that grouping is beneficial for students

5INTRODUCTION

with low or medium prior abilitymdashif adequate time is provided for instruction and if the classis well managed On the other hand ability grouping appears to have a minimal impact onhigh-ability studentsrsquo literacy These results were derived from kindergarten data and maynot hold for math instruction and for higher grade levels Replications with different subpo-pulations and in different contexts enable researchers to assess the generalizability of a theory

In a third example one may hypothesize that a student is unlikely to learn algebra well inninth grade without a solid foundation in eighth-grade prealgebra One may also argue that theprogress a student made in learning eighth-grade prealgebra may not be sustained without thefurther enhancement of taking algebra in ninth grade In other words the earlier treatment(prealgebra in eighth grade) and the later treatment (algebra in ninth grade) may reinforceone other each moderates the effect of the other treatment on the final outcome (math achieve-ment at the end of ninth grade) As Hong and Raudenbush (2008) argued the cumulativeeffect of two treatments in a well-aligned sequence may exceed the sum of the benefits oftwo single-year treatments Similarly experiencing two consecutive years of inferior treat-ments such as encountering an incompetent math teacher who dampens a studentrsquos self-efficacy in math learning in eighth grade and then again in ninth grade could do more damagethan the sum of the effect of encountering an incompetent teacher only in eighth grade and theeffect of having such a teacher only in ninth grade

There has been a great amount of conceptual confusion regarding how a moderatorrelates to the treatment and the outcome On one hand many researchers have used the termsldquomoderatorsrdquo and ldquomediatorsrdquo interchangeably without understanding the crucial distinctionbetween the two An overcorrective attempt on the other hand has led to the arbitrary rec-ommendation that a moderator must occur prior to the treatment and be minimally associatedwith the treatment (James and Brett 1984 Kraemer et al 2001 2008) In Chapter 6 weclarify that in essence the causal effect of the treatment on the outcome may depend onthe moderator value A moderator can be a subpopulation identifier a contextual character-istic a concurrent treatment or a preceding or succeeding treatment In the earlier examplesage of estrogen therapy initiation and a studentrsquos prior ability are subpopulation identifiers themanageability of a class which reflects both teacher skills and peer behaviors characterizes thecontext literacy instruction time is a concurrent treatment while prealgebra is a precedingtreatment for algebra A moderator does not have to occur prior to the treatment and doesnot have to be independent of the treatment

Once a moderated causal relationship has been defined in terms of potential outcomes theresearcher then chooses an appropriate experimental design for testing the moderation theoryThe assumptions required for identifying the moderated causal relationships differ across dif-ferent designs and have implications for analyzing experimental and quasiexperimental dataRandomized block designs are suitable for examining individual or contextual characteristicsas potential moderators factorial designs enable one to determine the joint effects of two ormore concurrent treatments and sequential randomized designs are ideal for assessing thecumulative effects of consecutive treatments Multisite randomized trials constitute anadditional type of designs in which the experimental sites are often deliberately sampled torepresent a population of geographical locations or social organizations Site membershipscan be viewed as implicit moderators that summarize a host of features of a local environmentReplications of a treatment over multiple sites allow one to quantify the heterogeneity of treat-ment effects across the sites Chapters 7 and 8 are focused on statistical methods for moder-ation analyses with quasiexperimental data In particular these chapters demonstrate througha number of application examples how a nonparametric marginal mean weighting through

6 CAUSALITY IN A SOCIAL WORLD

stratification (MMWS) method can overcome some important limitations of other existingmethods

Moderation questions are not restricted to treatmentndashoutcome relationships Rather theyare prevalent in investigations of mediation and spill-over This is because heterogeneity oftreatment effects may be explained by the variation in causal mediation mechanisms acrosssubpopulations and contexts It is also because the treatment effect for a focal individualmay depend on whether there is spill-over from other individuals through social interactionsYet similar to the confusion around the concept of moderation among applied researchersthere has been a great amount of misunderstanding with regard to mediation

112 Mediated treatment effects

Questions about mediation are at the core of nearly every scientific theory In its simplestform a theory explains why treatment Z causes outcome Y by hypothesizing an intermediateprocess involving at least one mediatorM that could have been changed by the treatment andcould subsequently have an impact on the outcome A causal mediation analysis then decom-poses the total effect of the treatment on the outcome into two parts the effect transmittedthrough the hypothesized mediator called the indirect effect and the difference betweenthe total effect and the indirect effect called the direct effect The latter represents the treat-ment effect channeled through other unspecified pathways (Alwin and Hauser 1975 Baronand Kenny 1986 Duncan 1966 Shadish and Sweeney 1991) Most researchers in socialsciences have followed the convention of illustrating the direct effect and the indirect effectwith path diagrams and then specifying linear regression models postulated to represent thestructural relationships between the treatment the mediator and the outcome

In Chapter 9 we point out that the approach to defining the indirect effect and the directeffect in terms of path coefficients is often misled by oversimplistic assumptions about thestructural relationships In particular this approach typically overlooks the fact that a treat-ment may generate an impact on the outcome not only by changing the mediator value butalso by changing the mediatorndashoutcome relationship (Judd and Kenny 1981) An examplein Chapter 9 illustrates such a case An experiment randomizes students to either an experi-mental condition which provides them with study materials and encourages them to study fora test or to a control condition which provides neither study materials nor encouragement(Holland 1988 Powers and Swinton 1984) One might hypothesize that encouraging exper-imental group members to study will increase the time they spend studying a focal mediatorin this example which in turn will increase their average test scores One might furtherhypothesize that even without a change in study time providing study materials to experi-mental group members will enable them to study more effectively than they otherwise wouldConsequently the effect of time spent studying might be greater under the experimentalcondition than under the control condition Omitting the treatment-by-mediator interactioneffect will lead to bias in the estimation of the indirect effect and the direct effect

Chapter 11 describes in great detail an evaluation study contrasting a new welfare-to-workprogram emphasizing active participation in the labor force with a traditional program thatguaranteed cash assistance without a mandate to seek employment (Hong Deutsch and Hill2011) Focusing on the psychological well-being of welfare applicants who were singlemothers with preschool-aged children the researchers hypothesized that the treatment wouldincrease employment rate among the welfare recipients and that the treatment-inducedincrease in employment would likely reduce depressive symptoms under the new program

7INTRODUCTION

They also hypothesized that in contrast the same increase in employment was unlikely tohave a psychological impact under the traditional program Hence the treatment-by-mediatorinteraction effect on the outcome was an essential component of the intermediate process

We will show that by defining the indirect effect and the direct effect in terms of potentialoutcomes one can avoid invoking unwarranted assumptions about the unknown structuralrelationships (Holland 1988 Pearl 2001 Robins and Greenland 1992) The potential out-comes framework has further clarified that in a causal mediation theory the treatment and themediator are both conceivably manipulable In the aforementioned example a welfare appli-cant could be assigned at random to either the new program or the traditional one Once havingbeen assigned to one of the programs the individual might or might not be employed due tovarious structural constraints market fluctuations or other random events typically beyondher control

Because the indirect effect and the direct effect are defined in terms of potential outcomesrather than on the basis of any specific regression models it becomes possible to distinguishthe definitions of these causal effects from their identification and estimation The identifica-tion step relates a causal parameter to observable population data For example for individualsassigned at random to an experimental condition their average counterfactual outcome asso-ciated with the control condition is expected to be equal to the average observable outcome ofthose assigned to the control condition Therefore the population average causal effect can beeasily identified in a randomized experiment The estimation step then relates the sample datato the observable population quantities involved in identification while taking into account thedegree of sampling variability

A major challenge in identification is that the mediatorndashoutcome relationship tends tobe confounded by selection even if the treatment is randomized Chapter 9 reviews variousexperimental designs that have been proposed by past researchers for studying causalmediation mechanisms Chapter 10 compares the identification assumptions and the analyticprocedures across a wide range of analytic methods These discussions are followed by anintroduction of the ratio-of-mediator-probability weighting (RMPW) strategy in Chapter 11In comparison with most existing strategies for causal mediation analysis RMPW relies onrelatively fewer identification assumptions and model-based assumptions Chapters 12 and13 will show that this new strategy can be applied broadly with extensions to multilevelexperimental designs and to studies of complex mediation mechanisms involving multiplemediators

113 Spill-over effects of a treatment

It is well known in vaccination research that an unvaccinated person can benefit when mostother people in the local community are vaccinated This is viewed as a spill-over of the treat-ment effect from the vaccinated people to an unvaccinated person Similarly due to socialinteractions among individuals or groups of individuals an intervention received by somemay generate a spill-over impact on others affiliated with the same organization or connectedthrough the same network For example effective policing in one neighborhood may driveoffenders to operate in other neighborhoods In evaluating an innovative community policingprogram researchers found that a neighborhood not assigned to community policing tended tosuffer if the surrounding neighborhoods were assigned to community policing This is aninstance in which a well-intended treatment generates an unintended negative spill-over effectHowever being assigned to community policy was found particularly beneficial when thesurrounding neighborhoods also received the intervention (Verbitsky-Savitz and Raudenbush

8 CAUSALITY IN A SOCIAL WORLD

2012) In another example the impact of a school-based mentoring program targeted atstudents displaying delinquent behaviors may be enhanced for a focal student if his or herat-risk peers are assigned to the mentoring program at the same time

The previous examples challenge the ldquostable unit treatment value assumptionrdquo (SUTVA)that has been invoked in most causal inference studies This assumption states that an indi-vidualrsquos potential response to a treatment depends neither on how the treatment is assignednor on the treatment assignments of other individuals (Rubin 1980 1986) Rubin and othersbelieve that without resorting to SUTVA the causal effect of a treatment becomes hard todefine and that causal inference may become intractable However as Sobel (2006) has illus-trated with the Moving to Opportunity (MTO) experiment that offered housing vouchersenabling low-income families to move to low-poverty neighborhoods there are possible con-sequences of violating SUTVA in estimating the treatment effects on outcomes such as safetyand self-sufficiency Because social interactions among individuals may affect whether onevolunteers to participate in the study whether one moves to a low-poverty neighborhood afterreceiving the housing voucher as well as housing project residentsrsquo subjective perceptions oftheir neighborhoods the program may have a nonzero impact on the potential outcome of theuntreated As a result despite the randomization of treatment assignment the mean differencein the observed outcome between the treated units and the untreated units may be biased forthe average treatment effect Rather the observable quantity is the difference between theeffect of treating some rather than treating none for the treated and the effect of treating somerather than treating none for the untreated the latter being the pure spill-over effect for theuntreated

In making attempts to relax SUTVA researchers have proposed several alternative frame-works that incorporate possible spill-over effects (see Hong and Raudenbush 2013 for areview) Hong (2004) presented a model that involves treatments and treatment settingsA treatment setting for an individual is a local environment constituted by a set of agentsand participants along with their treatment assignments An individualrsquos potential outcomevalue under a given treatment is assumed stable when the treatment setting is fixed the poten-tial outcome may take different values when the treatment setting shifts One may investigatewhether the treatment effect depends on the treatment setting Applying this framework Hongand Raudenbush (2006) examined the effect on a childrsquos academic growth of retaining thechild in kindergarten rather than promoting the child to first grade when a relatively smallproportion of low-achieving peers in the same school are retained as opposed to when a rel-atively large proportion of the peers are retained Hudgens and Halloran (2008) presented arelated framework in which the effect on an individual of the treatment received by this indi-vidual is distinguished from the effect on the individual of the treatment received by others inthe same local community These effects can be identified if communities are assigned atrandom to different treatment assignment strategies (such as retaining a large proportion oflow-achieving students as opposed to retaining a small proportion of such students) and sub-sequently individuals within a community are randomized for treatment Chapter 14 reviewsthese frameworks and discusses identification and estimation strategies

Social contagion may also serve as an important channel through which the effect of anintervention is transmitted For example a school-wide intervention may reduce aggressivebehaviors and thereby improve studentsrsquo psychological well-being by improving the qualityof interpersonal relationships in other classes as well as in onersquos own class This is becausechildren interact not simply with their classmates but also with those from other classes in thehallways or on the playground (VanderWeele et al 2013) In this case the spill-over betweenclasses becomes a part of the mediation mechanism In a study of student mentoring whether

9INTRODUCTION

an individual actually participated in the mentoring program despite the initial treatmentassignment is typically viewed as a primary mediator The proportion of onersquos peers that par-ticipated in the program may act as a second mediator A treatment may also exert its impactpartly through regrouping individuals and thereby changing peer composition (Hong andNomi 2012) Chapter 15 discusses analytic strategies for detecting spill-over as a part ofthe causal mediation mechanism

12 Weighting methods for causal inference

In most behavioral and social science applications major methodological challenges arise dueto the selection of treatments in a quasiexperimental study and the selection of mediator valuesin experimental and quasiexperimental studies Statistical methods most familiar to research-ers in social sciences are often inadequate for causal inferences with regard to multivaluedtreatments moderation mediation and spill-over The book offers a major revision to under-standing of causality in the social world by introducing two complementary weighting stra-tegies both featuring nonparametric approaches to estimating these causal effects The newweighting methods greatly simplify model specifications while enhancing the robustness ofresults by minimizing the reliance on some key assumptions

The propensity score-basedMMWSmethod removes selection bias associated with a largenumber of covariates by equating the pretreatment composition between treatment groups(Hong 2010a 2012 Huang et al 2005) Unlike propensity score matching and stratificationthat are mostly restricted to evaluations of binary treatments the MMWS method is flexiblefor evaluating binary and multivalued treatments by approximating a completely randomizedexperiment In evaluating whether the treatment effects differ across subpopulations definedby individual characteristics or treatment settings researchers may assign weights within eachsubpopulation in order to approximate a randomized block design To investigate whetherone treatment moderates the effect of another concurrent treatment researchers may assignweights to the data to approximate a factorial randomized design The method can also beused to assess whether the effect of an initial treatment is amplified or weakened by a subse-quent treatment or to identify an optimal treatment sequence through approximating a sequen-tial randomized experiment Even though such analyses can similarly be conducted throughinverse-probability-of-treatment weighting (IPTW) that has been increasingly employed inepidemiological research (Hernaacuten Brumback and Robins 2000 Robins Hernaacuten andBrumback 2000) IPTW is known for bias and imprecision in estimation especially whenthe propensity score models are misspecified in their functional forms (Hong 2010a Kangand Schafer 2007 Schafer and Kang 2008 Waernbaum 2012) In contrast the nonparamet-ric MMWS method displays a relatively high level of robustness despite such misspecifica-tions and also gains efficiency as indicated by simulation results (Hong 2010a)

To study causal mediation mechanisms the RMPW method decomposes the total effectof a treatment into an ldquoindirect effectrdquo transmitted through a specific mediator and a ldquodirecteffectrdquo representing unspecified mechanisms In contrast with most existing methods formediation analysis the RMPW-adjusted outcome model is extremely simple and is nonpara-metric in nature It generates estimates of the causal effects along with their sampling errorswhile adjusting for pretreatment covariates that confound the mediatorndashoutcome relationshipsthrough weighting The method applies regardless of the distribution of the outcome the dis-tribution of the mediator or the functional relationship between the outcome and the mediator

10 CAUSALITY IN A SOCIAL WORLD

Page 7: Thumbnail - download.e-bookshelf.de...4.2.4 IPTW for bias removal 84 4.3 MMWS 86 4.3.1 Theoretical rationale 86 ... 8.3.4.2 Growth model in the absence of confounding 200 8.3.4.3 Weighted

Appendix 4A Proof of MMWS-adjusted mean observed outcome being unbiasedfor the population average potential outcome 95

Appendix 4B Derivation of MMWS for estimating the treatment effecton the treated 96

Appendix 4C Theoretical equivalence of MMWS and IPTW 97Appendix 4D Simulations comparing MMWS and IPTW under misspecifications

of the functional form of a propensity score model 97

5 Evaluations of multivalued treatments 10051 Defining the causal effects of multivalued treatments 10052 Existing designs and analytic methods for evaluating multivalued

treatments 102521 Experimental designs and analysis 102

5211 Randomized experiments with multipletreatment arms 102

5212 Identification under ignorability 1025213 ANOVA 103

522 Quasiexperimental designs and analysis 1055221 ANCOVA and multiple regression 1055222 Propensity score-based adjustment 1085223 Other adjustment methods 112

53 MMWS for evaluating multivalued treatments 112531 Basic rationale 113532 Analytic procedure 114

5321 MMWS for a multinomial treatment measure 1155322 MMWS for an ordinal treatment measure 120

533 Identification assumptions 12154 Summary 123Appendix 5A Multiple IV for evaluating multivalued treatments 124

Part II Moderation 127

6 Moderated treatment effects concepts and existing analytic methods 12961 What is moderation 129

611 Past discussions of moderation 1306111 Purpose of moderation research 1306112 What qualifies a variable as a moderator 132

612 Definition of moderated treatment effects 1336121 Treatment effects moderated by individual or contextual

characteristics 1336122 Joint effects of concurrent treatments 134

62 Experimental designs and analytic methods for investigatingexplicit moderators 136621 Randomized block designs 137

6211 Identification assumptions 1376212 Two-way ANOVA 1386213 Multiple regression 139

viiCONTENTS

622 Factorial designs 1406221 Identification assumptions 1406222 Analytic strategies 142

63 Existing research designs and analytic methods for investigatingimplicit moderators 142631 Multisite randomized trials 143

6311 Causal parameters and identification assumptions 1446312 Analytic strategies 145

632 Principal stratification 149Appendix 6A Derivation of bias in the fixed-effects estimator when the

treatment effect is heterogeneous in multisite randomized trials 151Appendix 6B Derivation of bias in the mixed-effects estimator when the

probability of treatment assignment varies across sites 153Appendix 6C Derivation and proof of the population weight applied to

mixed-effects models for eliminating bias in multisiterandomized trials 153

7 Marginal mean weighting through stratification for investigatingmoderated treatment effects 15971 Existing methods for moderation analyses with quasiexperimental data 159

711 Analysis of covariance and regression-based adjustment 1617111 Treatment effects moderated by subpopulation membership 1617112 Treatment effects moderated by a concurrent treatment 164

712 Propensity score-based adjustment 1657121 Propensity score matching and stratification 1667122 Inverse-probability-of-treatment weighting 167

72 MMWS estimation of treatment effects moderated by individual orcontextual characteristics 168721 Application example 170722 Analytic procedure 170

73 MMWS estimation of the joint effects of concurrent treatments 174731 Application example 174732 Analytic procedure 175733 Joint treatment effects moderated by individual or contextual

characteristics 179

8 Cumulative effects of time-varying treatments 18581 Causal effects of treatment sequences 186

811 Application example 186812 Causal parameters 187

8121 Time-varying treatments 1878122 Time-varying potential outcomes 1878123 Causal effects of 2-year treatment sequences 1878124 Causal effects of multiyear treatment sequences 190

82 Existing strategies for evaluating time-varying treatments 190821 The endogeneity problem in nonexperimental data 190822 SEM 191

viii CONTENTS

823 Fixed-effects econometric models 192824 Sequential randomization 192825 Dynamic treatment regimes 193826 Marginal structural models and structural nested models 194

83 MMWS for evaluating 2-year treatment sequences 195831 Sequential ignorability 195832 Propensity scores 196833 MMWS computation 197

8331 MMWS adjustment for year 1 treatment selection 1978332 MMWS adjustment for two-year treatment

sequence selection 198834 Two-year growth model specification 199

8341 Growth model in the absence of treatment 2008342 Growth model in the absence of confounding 2008343 Weighted 2-year growth model 202

84 MMWS for evaluating multiyear sequences of multivalued treatments 204841 Sequential ignorability of multiyear treatment sequences 204842 Propensity scores for multiyear treatment sequences 204843 MMWS computation 205844 Weighted multiyear growth model 205845 Issues of sample size 206

85 Conclusion 207Appendix 8A A saturated model for evaluating multivalued treatments

over multiple time periods 207

Part III Mediation 211

9 Concepts of mediated treatment effects and experimental designsfor investigating causal mechanisms 21391 Introduction 21492 Path coefficients 21593 Potential outcomes and potential mediators 216

931 Controlled direct effects 217932 Controlled treatment-by-mediator interaction effect 217

94 Causal effects with counterfactual mediators 219941 Natural direct effect 219942 Natural indirect effect 220943 Natural treatment-by-mediator interaction effect 220944 Unstable unit treatment value 221

95 Population causal parameters 222951 Population average natural direct effect 224952 Population average natural indirect effect 225

96 Experimental designs for studying causal mediation 225961 Sequentially randomized designs 228962 Two-phase experimental designs 228963 Three- and four-treatment arm designs 230964 Experimental causal-chain designs 231

ixCONTENTS

965 Moderation-of-process designs 231966 Augmented encouragement designs 232967 Parallel experimental designs and parallel encouragement designs 232968 Crossover experimental designs and crossover encouragement

designs 233969 Summary 234

10 Existing analytic methods for investigating causal mediation mechanisms 238101 Path analysis and SEM 239

1011 Analytic procedure for continuous outcomes 2391012 Identification assumptions 2421013 Analytic procedure for discrete outcomes 245

102 Modified regression approach 2461021 Analytic procedure for continuous outcomes 2461022 Identification assumptions 2471023 Analytic procedure for binary outcomes 248

103 Marginal structural models 2501031 Analytic procedure 2501032 Identification assumptions 252

104 Conditional structural models 2521041 Analytic procedure 2521042 Identification assumptions 253

105 Alternative weighting methods 2541051 Analytic procedure 2541052 Identification assumptions 256

106 Resampling approach 2561061 Analytic procedure 2561062 Identification assumptions 257

107 IV method 2571071 Rationale and analytic procedure 2571072 Identification assumptions 258

108 Principal stratification 2591081 Rationale and analytic procedure 2591082 Identification assumptions 260

109 Sensitivity analysis 2611091 Unadjusted confounding as a product of hypothetical

regression coefficients 2611092 Unadjusted confounding reflected in a hypothetical

correlation coefficient 2621093 Limitations when the selection mechanism differs

by treatment 2641094 Other sensitivity analyses 265

1010 Conclusion 26510101 The essentiality of sequential ignorability 26510102 Treatment-by-mediator interactions 26610103 Homogeneous versus heterogeneous causal effects 26610104 Model-based assumptions 266

x CONTENTS

Appendix 10A Bias in path analysis estimation due to the omission oftreatment-by-mediator interaction 267

11 Investigations of a simple mediation mechanism 273111 Application example national evaluation of welfare-to-work strategies 274

1111 Historical context 2741112 Research questions 2751113 Causal parameters 2751114 NEWWS Riverside data 277

112 RMPW rationale 2771121 RMPW in a sequentially randomized design 278

11211 E[Y(0 M(0))] 27911212 E[Y(1 M(1))] 28011213 E[Y(1 M(0))] 28011214 E[Y(0 M(1))] 28111215 Nonparametric outcome model 282

1122 RMPW in a sequentially randomized block design 2831123 RMPW in a standard randomized experiment 2851124 Identification assumptions 286

113 Parametric RMPW procedure 287114 Nonparametric RMPW procedure 290115 Simulation results 292

1151 Correctly specified propensity score models 2921152 Misspecified propensity score models 2941153 Comparisons with path analysis and IV results 294

116 Discussion 2951161 Advantages of the RMPW strategy 2951162 Limitations of the RMPW strategy 295

Appendix 11A Causal effect estimation through the RMPW procedure 296Appendix 11B Proof of the consistency of RMPW estimation 297

12 RMPW extensions to alternative designs and measurement 301121 RMPW extensions to mediators and outcomes of alternative distributions 301

1211 Extensions to a multicategory mediator 30212111 Parametric RMPW procedure 30212112 Nonparametric RMPW procedure 304

1212 Extensions to a continuous mediator 3041213 Extensions to a binary outcome 306

122 RMPW extensions to alternative research designs 3061221 Extensions to quasiexperimental data 3071222 Extensions to data from cluster randomized trials 308

12221 Application example and research questions 30912222 Estimation of the total effect 31012223 RMPW analysis of causal mediation mechanisms 31012224 Identification assumptions 31212225 Contrast with multilevel path analysis and SEM 31212226 Contrast with multilevel prediction models 313

xiCONTENTS

1223 Extensions to data from multisite randomized trials 31312231 Research questions and causal parameters 31412232 Estimation of the total effect and its between-site variation 31512233 RMPW analysis of causal mediation mechanisms 31612234 Identification assumptions 31912235 Contrast with multilevel path analysis and SEM 319

123 Alternative decomposition of the treatment effect 321

13 RMPW extensions to studies of complex mediation mechanisms 325131 RMPW extensions to moderated mediation 325

1311 RMPW analytic procedure for estimating and testingmoderated mediation 326

1312 Path analysisSEM approach to analyzing moderatedmediation 327

1313 Principal stratification and moderated mediation 328132 RMPW extensions to concurrent mediators 328

1321 Treatment effect decomposition 32913211 Treatment effect decomposition without

between-mediator interaction 32913212 Treatment effect decomposition with

between-mediator interaction 3311322 Identification assumptions 3331323 RMPW procedure 333

13231 Estimating E[Y(1M1(1)M2(0))] 33513232 Estimating E[Y(1M1(0)M2(0))] 33513233 Causal effect estimation with noninteracting

concurrent mediators 33613234 Estimating E[Y(1M1(0)M2(1))] 33713235 Causal effect estimation with interacting concurrent

mediators 3371324 Contrast with the linear SEM approach 3381325 Contrast with the multivariate IV approach 339

133 RMPW extensions to consecutive mediators 3401331 Treatment effect decomposition 341

13311 Natural direct effect of the treatment on the outcome 34213312 Natural indirect effect mediated by M1 only 34213313 Natural indirect effect mediated by M2 only 34213314 Natural indirect effect mediated by an M1-by-M2

interaction 34313315 Treatment-by-mediator interactions 343

1332 Identification assumptions 3451333 RMPW procedure 347

13331 Estimating E[Y(1M1(0)M2(0M1(0)))] 34713332 Estimating E[Y(1M1(1)M2(0M1(0)))] 34913333 Estimating E[Y(1M1(0)M2(1M1(1)))] 34913334 Estimating E[Y(0M1(1)M2(0M1(0)))] and

E[Y(0M1(0)M2(1M1(1)))] 351

xii CONTENTS

1334 Contrast with the linear SEM approach 3531335 Contrast with the sensitivity-based estimation of bounds for

causal effects 354134 Discussion 355Appendix 13A Derivation of RMPW for estimating population average

counterfactual outcomes of two concurrentmediators 355

Appendix 13B Derivation of RMPW for estimating population averagecounterfactual outcomes of consecutivemediators 358

Part IV Spill-over 363

14 Spill-over of treatment effects concepts and methods 365141 Spill-over A nuisance a trifle or a focus 365142 Stable versus unstable potential outcome values An example

from agriculture 367143 Consequences for causal inference when spill-over is overlooked 369144 Modified framework of causal inference 371

1441 Treatment settings 3711442 Simplified characterization of treatment settings 3731443 Causal effects of individual treatment assignment and of peer

treatment assignment 375145 Identification Challenges and solutions 376

1451 Hypothetical experiments for identifying average treatmenteffects in the presence of social interactions 376

1452 Hypothetical experiments for identifying the impact ofsocial interactions 380

1453 Application to an evaluation of kindergarten retention 382146 Analytic strategies for experimental and quasiexperimental data 384

1461 Estimation with experimental data 3841462 Propensity score stratification 3851463 MMWS 386

147 Summary 387

15 Mediation through spill-over 391151 Definition of mediated effects through spill-over in a cluster

randomized trial 3931511 Notation 3931512 Treatment effect mediated by a focal individualrsquos

compliance 3941513 Treatment effect mediated by peersrsquo compliance through

spill-over 3941514 Decomposition of the total treatment effect 395

152 Identification and estimation of the spill-over effect in a clusterrandomized design 3951521 Identification in an ideal experiment 395

xiiiCONTENTS

1522 Identification when the mediators are not randomized 3981523 Estimation of mediated effects through spill-over 400

153 Definition of mediated effects through spill-over in a multisite trial 4021531 Notation 4021532 Treatment effect mediated by a focal individualrsquos compliance 4041533 Treatment effect mediated by peersrsquo compliance

through spill-over 4041534 Direct effect of individual treatment assignment on the outcome 4051535 Direct effect of peer treatment assignment on the outcome 4051536 Decomposition of the total treatment effect 405

154 Identification and estimation of spill-over effects in a multisite trial 4061541 Identification in an ideal experiment 4071542 Identification when the mediators are not randomized 4091543 Estimation of mediated effects through spill-over 410

15431 Estimating E[Y(1pM(p)Mminus(p))] andE[Y(000Mminus(0))] 410

15432 Estimating E[Y(1p0Mminus(p))] 41115433 Estimating E[Y(1p0Mminus(0))] 41215434 Estimating E[Y(0p0Mminus(0))] 412

155 Consequences of omitting spill-over effects in causal mediation analyses 4121551 Biased inference in a cluster randomized trial 4131552 Biased inference in a multisite randomized trial 4131553 Biased inference of the local average treatment effect 415

156 Quasiexperimental application 416157 Summary 419Appendix 151 Derivation of the weight for estimating the population

average counterfactual outcome E[Y(1 p 0Mminus( p))] 419Appendix 152 Derivation of bias in the ITT effect due to the omission of

spill-over effects 420

Index 423

xiv CONTENTS

Preface

A scientific mind in training seeks comprehensive descriptions of every interesting phenom-enon In such a mind data are to be pieced together for comparisons contrasts and eventuallyinferences that are required for understanding causes and effects that may have led to thephenomenon of interest Yet without disciplined reasoning a causal inference would slip intothe rut of a ldquocasual inferencerdquo as easily as making a typographical error

As a doctoral student thirsting for rigorous methodological training at the University ofMichigan I was very fortunate to be among the first group of students on the campus attendinga causal inference seminar in 2000 jointly taught by Yu Xie from the Department of Sociol-ogy Susan Murphy from the Department of Statistics and Stephen Raudenbush from theSchool of Education The course was unique in both content and pedagogy The instructorsput together a bibliography drawn from the past and current causal inference literature origi-nated from multiple disciplines In the spirit of ldquoreciprocal teachingrdquo the students were delib-erately organized into small groups each representing a mix of disciplinary backgrounds andtook turns to teach the weekly readings under the guidance of a faculty instructor This inval-uable experience revealed to me that the pursuit of causal inference is a multidisciplinaryendeavor In the rest of my doctoral study I continued to be immersed in an extraordinaryinterdisciplinary community in the form of the Quantitative Methodology Program (QMP)seminars led by the above three instructors who were then joined by Ben Hansen(Statistics) Richard Gonzalez (Psychology) Roderick Little (Biostatistics) Jake Bowers(Political Science) and many others I have tried whenever possible to carry the same spiritinto my own teaching of causal inference courses at the University of Toronto and now at theUniversity of Chicago In these courses communicating understandings of causal problemsacross disciplinary boundaries has been particularly challenging but also stimulating and grat-ifying for myself and for students from various departments and schools

Under the supervision of Stephen Raudenbush I took a first stab at the conceptual frame-work of causal inference in my doctoral dissertation by considering peer spill-over in schoolsettings From that point on I have organized my methodological research to tackle some ofthe major obstacles to causal inferences in policy and program evaluations My workaddresses issues including (i) how to conceptualize and evaluate the causal effects of educa-tional treatments when studentsrsquo responses to alternative treatments depend on various fea-tures of the organizational settings including peer composition (ii) how to adjust forselection bias in evaluating the effects of concurrent and consecutive multivalued treatmentsand (iii) how to conceptualize and analyze causal mediation mechanisms My methodological

research has been driven by the substantive interest in understanding the role of socialinstitutionsmdashschools in particularmdashin shaping human development especially during theages of fast growth (eg childhood adolescence and early adulthood) I have shared meth-odological advances with a broad audience in social sciences through applying the causalinference methods to prominent substantive issues such as grade retention within-class group-ing services for English language learners and welfare-to-work strategies These illustrationsare used extensively in this book

Among social scientists awareness of methodological challenges in causal investigationsis higher than ever before In response to this rising demand causal inference is becoming oneof the most productive scholarly fields in the past decade I have had the benefit of followingthe work and exchanging ideas with some fellow methodologists who are constantly contri-buting new thoughts and making breakthroughs These include Daniel Almirall HowardBloom Tom Cook Michael Elliot Ken Frank Ben Hansen James Heckman JenniferHill Martin Huber Kosuke Imai Booil Jo John R Lockwood David MacKinnon DanielMcCaffrey Luke Miratrix Richard Murnane Derek Neal Lindsey Page Judea PearlStephen Raudenbush Sean Reardon Michael Seltzer Youngyun Shin Betsy SinclairMichael Sobel Peter Steiner Elizabeth Stuart Eric Thetchen Tchetchen Tyler VanderWeeleand Kazuo Yamaguchi I am grateful to Larry Hedges who offered me the opportunity of guestediting the Journal of Research in Educational Effectiveness special issue on the statisticalapproaches to studying mediator effects in education research in 2012 Many of the colleagueswhom I mentioned above contributed to the open debate in the special issue by either propos-ing or critically examining a number of innovative methods for studying causal mechanismsI anticipate that many of them are or will soon be writing their own research monographs oncausal inference if they have not already done so Therefore readers of this book are stronglyrecommended to browse past and future titles by these and other authors for alternative per-spectives and approaches My own understanding of course will continue to evolve as well inthe coming years

One has to be ambitious to stay upfront in substantive areas as well as in methodology Thebest strategy apparently is to learn from substantive experts During different phases of mytraining and professional career I have had the opportunities to work with David K CohenBrian Rowan Deborah Ball Carl Corter Janette Pelletier Takako Nomi Esther Geva DavidFrancis Stephanie Jones Joshua Brown and Heather D Hill Many of my substantive insightswere derived from conversations with these mentors and colleagues I am especially gratefulto Bob Granger former president of the William T Grant Foundation who reassured me thatldquoBy staying a bit broad you will learn a lot and many fields will benefit from your workrdquo

Like many other single-authored books this one is built on the generous contributions ofstudents and colleagues who deserve special acknowledgement Excellent research assistancefrom Jonah Deutsch Joshua Gagne Rachel Garrett Yihua Hong Xu Qin Cheng Yang andBing Yu was instrumental in the development and implementation of the new methods pre-sented in this book Richard Congdon a renowned statistical software programmer broughtrevolutionary ideas to interface designs in software development with the aim of facilitatingusersrsquo decision-making in a series of causal analyses RichardMurnane Stephen Raudenbushand Michael Sobel carefully reviewed and critiqued multiple chapters of the manuscript draftAdditional comments and suggestions on earlier drafts came fromHoward Bloom Ken FrankBen Hansen Jennifer Hill Booil Jo Ben Kelcey David MacKinnon and Fan Yang TereseSchwartzman provided valuable assistance in manuscript preparation The writing of thisbook was supported by a Scholars Award from the William T Grant Foundation and an

xvi PREFACE

Institute of Education Sciences (IES) Statistical and Research Methodology in EducationGrant from the US Department of Education All the errors in the published form are ofcourse the sole responsibility of the author

Project Editors Richard Davis Prachi Sinha Sahay and Liz Wingett and Assistant EditorHeather Kay at Wiley and project Managers P Jayapriya and R Jayavel at SPi Global havebeen wonderful to work with throughout the manuscript preparation and final production ofthe book Debbie Jupe Commissioning Editor at Wiley was remarkably effective in initiatingthe contact with me and in organizing anonymous reviews of the book proposal and of themanuscript These constructive reviews have shaped the book in important ways I cannotagree more with one of the reviewers that ldquocausality cannot be established on a pure statisticalgroundrdquo The book therefore highlights the importance of substantive knowledge and researchdesigns and places a great emphasis on clarifying and evaluating assumptions required foridentifying causal effects in the context of each application Much caution is raised againstpossible misusage of statistical techniques for analyzing causal relationships especiallywhen data are inadequate Yet I maintain that improved statistical procedures along withimproved research designs would greatly enhance our ability in attempt to empiricallyexamine well-articulated causal theories

xviiPREFACE

Part I

OVERVIEW

1

Introduction

According to an ancient Chinese fable a farmer who was eager to help his crops grow wentinto his field and pulled each seedling upward After exhausting himself with the work heannounced to his family that they were going to have a good harvest only to find the nextmorning that the plants had wilted and died Readers with minimal agricultural knowledgemay immediately point out the following the farmerrsquos intervention theory was based on acorrect observation that crops that grow taller tend to produce more yield Yet his hypothesisreflects a false understanding of the cause and the effectmdashthat seedlings pulled to be tallerwould yield as much as seedlings thriving on their own

In their classic Design and Analysis of Experiments Hinkelmann and Kempthorne (1994updated version of Kempthorne 1952) discussed two types of science descriptive science andthe development of theory These two types of science are interrelated in the following senseobservations of an event and other related events often selected and classified for descriptionby scientists naturally lead to one or more explanations that we call ldquotheoretical hypothesesrdquowhich are then screened and falsified by means of further observations experimentation andanalyses (Popper 1963) The experiment of pulling seedlings to be taller was costly but didserve the purpose of advancing this farmerrsquos knowledge of ldquowhat does not workrdquo To developa successful intervention in this case would require a series of empirical tests of explicittheories identifying potential contributors to crop growth This iterative process graduallydeepens our knowledge of the relationships between supposed causes and effectsmdashthatis causalitymdashand may eventually increase the success of agricultural medical and socialinterventions

11 Concepts of moderation mediation and spill-over

Although the story of the ancient farmer is fictitious numerous examples can be found in thereal world in which well-intended interventions fail to produce the intended benefits or inmany cases even lead to unintended consequences ldquoInterventionsrdquo and ldquotreatmentsrdquo usedinterchangeably in this book broadly refer to actions taken by agents or circumstances expe-rienced by an individual or groups of individuals Interventions are regularly seen in educa-tion physical and mental health social services business politics and law enforcement In an

Causality in a Social World Moderation Meditation and Spill-over First Edition Guanglei Hongcopy 2015 John Wiley amp Sons Ltd Published 2015 by John Wiley amp Sons Ltd

education intervention for example teachers are typically the agents who deliver a treatmentto students while the impact of the treatment on student outcomes is of ultimate causalinterest Some educational practices such as ldquoteaching to the testrdquo have been criticized tobe nearly as counterproductive as the attempt of helping seedlings grow by pulling themupward ldquoInterventionsrdquo and ldquotreatmentsrdquo under consideration do not exclude undesiredexperiences such as exposure to poverty abuse crime or bereavement A treatment plannedor unplanned becomes a focus of research if there are theoretical reasons to anticipate itsimpact positive or negative on the well-being of individuals who are embedded in socialsettings including families classrooms schools neighborhoods and workplaces

In social science research in general and in policy and program evaluations in particularquestions concerning whether an intervention works and if so which version of the interven-tion works for whom under what conditions and why are key to the advancement of scien-tific and practical knowledge Although most empirical investigations in the social sciencesconcentrate on the average effect of a treatment for a specific population as opposed to theabsence of such a treatment (ie the control condition) in-depth theoretical reasoning withregard to how the causal effect is generated substantiated by compelling empirical evidenceis crucial for advancing scientific understanding

First when there are multiple versions or different dosages of the treatment or when thereare multiple versions of the control condition a binary divide between ldquothe treatmentrdquo andldquothe controlrdquo may not be as informative as fine-grained comparisons across for exampleldquotreatment version Ardquo ldquotreatment version Brdquo ldquocontrol version Ardquo and ldquocontrol versionBrdquo For example expanding the federally funded Head Start program to poor children isexpected to generate a greater benefit when few early childhood education alternatives areavailable (call it ldquocontrol version Ardquo) than when there is an abundance of alternatives includ-ing state-sponsored preschool programs (call it ldquocontrol version Brdquo)

Second the effect of an intervention will likely vary among individuals or across socialsettings A famous example comes from medical research the well-publicized cardiovascularbenefits of initiating estrogen therapy during menopause were contradicted later by experi-mental findings that the same therapy increased postmenopausal womenrsquos risk for heartattacks The effect of an intervention may also depend on the provision of some other con-current or subsequent interventions Such heterogeneous effects are often characterized asmoderated effects in the literature

Third alternative theories may provide competing explanations for the causal mechan-isms that is the processes through which the intervention produces its effect A theoreticalconstruct characterizing the hypothesized intermediate process is called a mediator of theintervention effect The fictitious farmer never developed an elaborate theory as to whatcaused some seedlings to surpass others in growth Once scientists revealed the causal rela-tionship between access to chemical nutrients in soil and plant growth wide applications ofchemically synthesized fertilizers finally led to a major increase in crop production

Finally it is well known in agricultural experiments that a new type of fertilizer applied toone plot may spill-over to the next plot Because social connections among individuals areprevalent within organizations or through networks an individualrsquos response to the treatmentmay similarly depend on the treatment for other individuals in the same social setting whichmay lead to possible spill-overs of intervention effects among individual human beings

Answering questions with regard to moderation mediation and spill-over poses majorconceptual and analytic challenges To date psychological research often presents well-articulated theories of causal mechanisms relating stimuli to responses Yet researchers often

4 CAUSALITY IN A SOCIAL WORLD

lack rigorous analytic strategies for empirically screening competing theories explaining theobserved effect Sociologists have keen interest in the spill-over of treatment effects transmit-ted through social interactions yet have produced limited evidence quantifying such effectsAs many have pointed out in general terminological ambiguity and conceptual confusionhave been prevalent in the published applied research (Holmbeck 1997 Kraemer et al2002 2008)

A new framework for conceptualizing moderated mediated and spill-over effects hasemerged relatively recently in the statistics and econometrics literature on causal inference(eg Abbring and Heckman 2007 Frangakis and Rubin 2002 Heckman and Vytlacil2007a b Holland 1988 Hong and Raudenbush 2006 Hudgens and Halloran 2008Jo 2008 Pearl 2001 Robins and Greenland 1992 Sobel 2008) The potential for furtherconceptual and methodological development and for broad applications in the field of behav-ioral and social sciences promises to greatly advance the empirical basis of our knowledgeabout causality in the social world

This book clarifies theoretical concepts and introduces innovative statistical strategies forinvestigating the average effects of multivalued treatments moderated treatment effectsmediated treatment effects and spill-over effects in experimental or quasiexperimental dataDefining individual-specific and population average treatment effects in terms of potentialoutcomes the book relates the mathematical forms to the substantive meanings of moderatedmediated and spill-over effects in the context of application examples It also explicates andevaluates identification assumptions and contrasts innovative statistical strategies with con-ventional analytic methods

111 Moderated treatment effects

It is hard to accept the assumption that a treatment would produce the same impact for everyindividual in every possible circumstance Understanding the heterogeneity of treatmenteffects therefore is key to the development of causal theories For example some studiesreported that estrogen therapy improved cardiovascular health among women who initiatedits use during menopause According to a series of other studies however the use of estrogentherapy increased postmenopausal womenrsquos risk for heart attacks (Grodstein et al 19962000 Writing Group for the Womenrsquos Health Initiative Investigators 2002) The sharp con-trast of findings from these studies led to the hypothesis that age of initiation moderates theeffect of estrogen therapy on womenrsquos health (Manson and Bassuk 2007 Rossouw et al2007) Revelations of the moderated causal relationship greatly enrich theoretical understand-ing and in this case directly inform clinical practice

In another example dividing students by ability into small groups for reading instructionhas been controversial for decades Many believed that the practice benefits high-ability stu-dents at the expense of their low-ability peers and exacerbates educational inequality (Grantand Rothenberg 1986 Rowan and Miracle 1983 Trimble and Sinclair 1987) Recentevidence has shown that first of all the effect of ability grouping depends on a number ofconditions including whether enough time is allocated to reading instruction and how wellthe class can be managed Hence instructional time and class management are among theadditional moderators to consider for determining the merits of ability grouping (Hong andHong 2009 Hong et al 2012a b) Moreover further investigations have generated evidencethat contradicts the long-held belief that grouping undermines the interests of low-abilitystudents Quite the contrary researchers have found that grouping is beneficial for students

5INTRODUCTION

with low or medium prior abilitymdashif adequate time is provided for instruction and if the classis well managed On the other hand ability grouping appears to have a minimal impact onhigh-ability studentsrsquo literacy These results were derived from kindergarten data and maynot hold for math instruction and for higher grade levels Replications with different subpo-pulations and in different contexts enable researchers to assess the generalizability of a theory

In a third example one may hypothesize that a student is unlikely to learn algebra well inninth grade without a solid foundation in eighth-grade prealgebra One may also argue that theprogress a student made in learning eighth-grade prealgebra may not be sustained without thefurther enhancement of taking algebra in ninth grade In other words the earlier treatment(prealgebra in eighth grade) and the later treatment (algebra in ninth grade) may reinforceone other each moderates the effect of the other treatment on the final outcome (math achieve-ment at the end of ninth grade) As Hong and Raudenbush (2008) argued the cumulativeeffect of two treatments in a well-aligned sequence may exceed the sum of the benefits oftwo single-year treatments Similarly experiencing two consecutive years of inferior treat-ments such as encountering an incompetent math teacher who dampens a studentrsquos self-efficacy in math learning in eighth grade and then again in ninth grade could do more damagethan the sum of the effect of encountering an incompetent teacher only in eighth grade and theeffect of having such a teacher only in ninth grade

There has been a great amount of conceptual confusion regarding how a moderatorrelates to the treatment and the outcome On one hand many researchers have used the termsldquomoderatorsrdquo and ldquomediatorsrdquo interchangeably without understanding the crucial distinctionbetween the two An overcorrective attempt on the other hand has led to the arbitrary rec-ommendation that a moderator must occur prior to the treatment and be minimally associatedwith the treatment (James and Brett 1984 Kraemer et al 2001 2008) In Chapter 6 weclarify that in essence the causal effect of the treatment on the outcome may depend onthe moderator value A moderator can be a subpopulation identifier a contextual character-istic a concurrent treatment or a preceding or succeeding treatment In the earlier examplesage of estrogen therapy initiation and a studentrsquos prior ability are subpopulation identifiers themanageability of a class which reflects both teacher skills and peer behaviors characterizes thecontext literacy instruction time is a concurrent treatment while prealgebra is a precedingtreatment for algebra A moderator does not have to occur prior to the treatment and doesnot have to be independent of the treatment

Once a moderated causal relationship has been defined in terms of potential outcomes theresearcher then chooses an appropriate experimental design for testing the moderation theoryThe assumptions required for identifying the moderated causal relationships differ across dif-ferent designs and have implications for analyzing experimental and quasiexperimental dataRandomized block designs are suitable for examining individual or contextual characteristicsas potential moderators factorial designs enable one to determine the joint effects of two ormore concurrent treatments and sequential randomized designs are ideal for assessing thecumulative effects of consecutive treatments Multisite randomized trials constitute anadditional type of designs in which the experimental sites are often deliberately sampled torepresent a population of geographical locations or social organizations Site membershipscan be viewed as implicit moderators that summarize a host of features of a local environmentReplications of a treatment over multiple sites allow one to quantify the heterogeneity of treat-ment effects across the sites Chapters 7 and 8 are focused on statistical methods for moder-ation analyses with quasiexperimental data In particular these chapters demonstrate througha number of application examples how a nonparametric marginal mean weighting through

6 CAUSALITY IN A SOCIAL WORLD

stratification (MMWS) method can overcome some important limitations of other existingmethods

Moderation questions are not restricted to treatmentndashoutcome relationships Rather theyare prevalent in investigations of mediation and spill-over This is because heterogeneity oftreatment effects may be explained by the variation in causal mediation mechanisms acrosssubpopulations and contexts It is also because the treatment effect for a focal individualmay depend on whether there is spill-over from other individuals through social interactionsYet similar to the confusion around the concept of moderation among applied researchersthere has been a great amount of misunderstanding with regard to mediation

112 Mediated treatment effects

Questions about mediation are at the core of nearly every scientific theory In its simplestform a theory explains why treatment Z causes outcome Y by hypothesizing an intermediateprocess involving at least one mediatorM that could have been changed by the treatment andcould subsequently have an impact on the outcome A causal mediation analysis then decom-poses the total effect of the treatment on the outcome into two parts the effect transmittedthrough the hypothesized mediator called the indirect effect and the difference betweenthe total effect and the indirect effect called the direct effect The latter represents the treat-ment effect channeled through other unspecified pathways (Alwin and Hauser 1975 Baronand Kenny 1986 Duncan 1966 Shadish and Sweeney 1991) Most researchers in socialsciences have followed the convention of illustrating the direct effect and the indirect effectwith path diagrams and then specifying linear regression models postulated to represent thestructural relationships between the treatment the mediator and the outcome

In Chapter 9 we point out that the approach to defining the indirect effect and the directeffect in terms of path coefficients is often misled by oversimplistic assumptions about thestructural relationships In particular this approach typically overlooks the fact that a treat-ment may generate an impact on the outcome not only by changing the mediator value butalso by changing the mediatorndashoutcome relationship (Judd and Kenny 1981) An examplein Chapter 9 illustrates such a case An experiment randomizes students to either an experi-mental condition which provides them with study materials and encourages them to study fora test or to a control condition which provides neither study materials nor encouragement(Holland 1988 Powers and Swinton 1984) One might hypothesize that encouraging exper-imental group members to study will increase the time they spend studying a focal mediatorin this example which in turn will increase their average test scores One might furtherhypothesize that even without a change in study time providing study materials to experi-mental group members will enable them to study more effectively than they otherwise wouldConsequently the effect of time spent studying might be greater under the experimentalcondition than under the control condition Omitting the treatment-by-mediator interactioneffect will lead to bias in the estimation of the indirect effect and the direct effect

Chapter 11 describes in great detail an evaluation study contrasting a new welfare-to-workprogram emphasizing active participation in the labor force with a traditional program thatguaranteed cash assistance without a mandate to seek employment (Hong Deutsch and Hill2011) Focusing on the psychological well-being of welfare applicants who were singlemothers with preschool-aged children the researchers hypothesized that the treatment wouldincrease employment rate among the welfare recipients and that the treatment-inducedincrease in employment would likely reduce depressive symptoms under the new program

7INTRODUCTION

They also hypothesized that in contrast the same increase in employment was unlikely tohave a psychological impact under the traditional program Hence the treatment-by-mediatorinteraction effect on the outcome was an essential component of the intermediate process

We will show that by defining the indirect effect and the direct effect in terms of potentialoutcomes one can avoid invoking unwarranted assumptions about the unknown structuralrelationships (Holland 1988 Pearl 2001 Robins and Greenland 1992) The potential out-comes framework has further clarified that in a causal mediation theory the treatment and themediator are both conceivably manipulable In the aforementioned example a welfare appli-cant could be assigned at random to either the new program or the traditional one Once havingbeen assigned to one of the programs the individual might or might not be employed due tovarious structural constraints market fluctuations or other random events typically beyondher control

Because the indirect effect and the direct effect are defined in terms of potential outcomesrather than on the basis of any specific regression models it becomes possible to distinguishthe definitions of these causal effects from their identification and estimation The identifica-tion step relates a causal parameter to observable population data For example for individualsassigned at random to an experimental condition their average counterfactual outcome asso-ciated with the control condition is expected to be equal to the average observable outcome ofthose assigned to the control condition Therefore the population average causal effect can beeasily identified in a randomized experiment The estimation step then relates the sample datato the observable population quantities involved in identification while taking into account thedegree of sampling variability

A major challenge in identification is that the mediatorndashoutcome relationship tends tobe confounded by selection even if the treatment is randomized Chapter 9 reviews variousexperimental designs that have been proposed by past researchers for studying causalmediation mechanisms Chapter 10 compares the identification assumptions and the analyticprocedures across a wide range of analytic methods These discussions are followed by anintroduction of the ratio-of-mediator-probability weighting (RMPW) strategy in Chapter 11In comparison with most existing strategies for causal mediation analysis RMPW relies onrelatively fewer identification assumptions and model-based assumptions Chapters 12 and13 will show that this new strategy can be applied broadly with extensions to multilevelexperimental designs and to studies of complex mediation mechanisms involving multiplemediators

113 Spill-over effects of a treatment

It is well known in vaccination research that an unvaccinated person can benefit when mostother people in the local community are vaccinated This is viewed as a spill-over of the treat-ment effect from the vaccinated people to an unvaccinated person Similarly due to socialinteractions among individuals or groups of individuals an intervention received by somemay generate a spill-over impact on others affiliated with the same organization or connectedthrough the same network For example effective policing in one neighborhood may driveoffenders to operate in other neighborhoods In evaluating an innovative community policingprogram researchers found that a neighborhood not assigned to community policing tended tosuffer if the surrounding neighborhoods were assigned to community policing This is aninstance in which a well-intended treatment generates an unintended negative spill-over effectHowever being assigned to community policy was found particularly beneficial when thesurrounding neighborhoods also received the intervention (Verbitsky-Savitz and Raudenbush

8 CAUSALITY IN A SOCIAL WORLD

2012) In another example the impact of a school-based mentoring program targeted atstudents displaying delinquent behaviors may be enhanced for a focal student if his or herat-risk peers are assigned to the mentoring program at the same time

The previous examples challenge the ldquostable unit treatment value assumptionrdquo (SUTVA)that has been invoked in most causal inference studies This assumption states that an indi-vidualrsquos potential response to a treatment depends neither on how the treatment is assignednor on the treatment assignments of other individuals (Rubin 1980 1986) Rubin and othersbelieve that without resorting to SUTVA the causal effect of a treatment becomes hard todefine and that causal inference may become intractable However as Sobel (2006) has illus-trated with the Moving to Opportunity (MTO) experiment that offered housing vouchersenabling low-income families to move to low-poverty neighborhoods there are possible con-sequences of violating SUTVA in estimating the treatment effects on outcomes such as safetyand self-sufficiency Because social interactions among individuals may affect whether onevolunteers to participate in the study whether one moves to a low-poverty neighborhood afterreceiving the housing voucher as well as housing project residentsrsquo subjective perceptions oftheir neighborhoods the program may have a nonzero impact on the potential outcome of theuntreated As a result despite the randomization of treatment assignment the mean differencein the observed outcome between the treated units and the untreated units may be biased forthe average treatment effect Rather the observable quantity is the difference between theeffect of treating some rather than treating none for the treated and the effect of treating somerather than treating none for the untreated the latter being the pure spill-over effect for theuntreated

In making attempts to relax SUTVA researchers have proposed several alternative frame-works that incorporate possible spill-over effects (see Hong and Raudenbush 2013 for areview) Hong (2004) presented a model that involves treatments and treatment settingsA treatment setting for an individual is a local environment constituted by a set of agentsand participants along with their treatment assignments An individualrsquos potential outcomevalue under a given treatment is assumed stable when the treatment setting is fixed the poten-tial outcome may take different values when the treatment setting shifts One may investigatewhether the treatment effect depends on the treatment setting Applying this framework Hongand Raudenbush (2006) examined the effect on a childrsquos academic growth of retaining thechild in kindergarten rather than promoting the child to first grade when a relatively smallproportion of low-achieving peers in the same school are retained as opposed to when a rel-atively large proportion of the peers are retained Hudgens and Halloran (2008) presented arelated framework in which the effect on an individual of the treatment received by this indi-vidual is distinguished from the effect on the individual of the treatment received by others inthe same local community These effects can be identified if communities are assigned atrandom to different treatment assignment strategies (such as retaining a large proportion oflow-achieving students as opposed to retaining a small proportion of such students) and sub-sequently individuals within a community are randomized for treatment Chapter 14 reviewsthese frameworks and discusses identification and estimation strategies

Social contagion may also serve as an important channel through which the effect of anintervention is transmitted For example a school-wide intervention may reduce aggressivebehaviors and thereby improve studentsrsquo psychological well-being by improving the qualityof interpersonal relationships in other classes as well as in onersquos own class This is becausechildren interact not simply with their classmates but also with those from other classes in thehallways or on the playground (VanderWeele et al 2013) In this case the spill-over betweenclasses becomes a part of the mediation mechanism In a study of student mentoring whether

9INTRODUCTION

an individual actually participated in the mentoring program despite the initial treatmentassignment is typically viewed as a primary mediator The proportion of onersquos peers that par-ticipated in the program may act as a second mediator A treatment may also exert its impactpartly through regrouping individuals and thereby changing peer composition (Hong andNomi 2012) Chapter 15 discusses analytic strategies for detecting spill-over as a part ofthe causal mediation mechanism

12 Weighting methods for causal inference

In most behavioral and social science applications major methodological challenges arise dueto the selection of treatments in a quasiexperimental study and the selection of mediator valuesin experimental and quasiexperimental studies Statistical methods most familiar to research-ers in social sciences are often inadequate for causal inferences with regard to multivaluedtreatments moderation mediation and spill-over The book offers a major revision to under-standing of causality in the social world by introducing two complementary weighting stra-tegies both featuring nonparametric approaches to estimating these causal effects The newweighting methods greatly simplify model specifications while enhancing the robustness ofresults by minimizing the reliance on some key assumptions

The propensity score-basedMMWSmethod removes selection bias associated with a largenumber of covariates by equating the pretreatment composition between treatment groups(Hong 2010a 2012 Huang et al 2005) Unlike propensity score matching and stratificationthat are mostly restricted to evaluations of binary treatments the MMWS method is flexiblefor evaluating binary and multivalued treatments by approximating a completely randomizedexperiment In evaluating whether the treatment effects differ across subpopulations definedby individual characteristics or treatment settings researchers may assign weights within eachsubpopulation in order to approximate a randomized block design To investigate whetherone treatment moderates the effect of another concurrent treatment researchers may assignweights to the data to approximate a factorial randomized design The method can also beused to assess whether the effect of an initial treatment is amplified or weakened by a subse-quent treatment or to identify an optimal treatment sequence through approximating a sequen-tial randomized experiment Even though such analyses can similarly be conducted throughinverse-probability-of-treatment weighting (IPTW) that has been increasingly employed inepidemiological research (Hernaacuten Brumback and Robins 2000 Robins Hernaacuten andBrumback 2000) IPTW is known for bias and imprecision in estimation especially whenthe propensity score models are misspecified in their functional forms (Hong 2010a Kangand Schafer 2007 Schafer and Kang 2008 Waernbaum 2012) In contrast the nonparamet-ric MMWS method displays a relatively high level of robustness despite such misspecifica-tions and also gains efficiency as indicated by simulation results (Hong 2010a)

To study causal mediation mechanisms the RMPW method decomposes the total effectof a treatment into an ldquoindirect effectrdquo transmitted through a specific mediator and a ldquodirecteffectrdquo representing unspecified mechanisms In contrast with most existing methods formediation analysis the RMPW-adjusted outcome model is extremely simple and is nonpara-metric in nature It generates estimates of the causal effects along with their sampling errorswhile adjusting for pretreatment covariates that confound the mediatorndashoutcome relationshipsthrough weighting The method applies regardless of the distribution of the outcome the dis-tribution of the mediator or the functional relationship between the outcome and the mediator

10 CAUSALITY IN A SOCIAL WORLD

Page 8: Thumbnail - download.e-bookshelf.de...4.2.4 IPTW for bias removal 84 4.3 MMWS 86 4.3.1 Theoretical rationale 86 ... 8.3.4.2 Growth model in the absence of confounding 200 8.3.4.3 Weighted

622 Factorial designs 1406221 Identification assumptions 1406222 Analytic strategies 142

63 Existing research designs and analytic methods for investigatingimplicit moderators 142631 Multisite randomized trials 143

6311 Causal parameters and identification assumptions 1446312 Analytic strategies 145

632 Principal stratification 149Appendix 6A Derivation of bias in the fixed-effects estimator when the

treatment effect is heterogeneous in multisite randomized trials 151Appendix 6B Derivation of bias in the mixed-effects estimator when the

probability of treatment assignment varies across sites 153Appendix 6C Derivation and proof of the population weight applied to

mixed-effects models for eliminating bias in multisiterandomized trials 153

7 Marginal mean weighting through stratification for investigatingmoderated treatment effects 15971 Existing methods for moderation analyses with quasiexperimental data 159

711 Analysis of covariance and regression-based adjustment 1617111 Treatment effects moderated by subpopulation membership 1617112 Treatment effects moderated by a concurrent treatment 164

712 Propensity score-based adjustment 1657121 Propensity score matching and stratification 1667122 Inverse-probability-of-treatment weighting 167

72 MMWS estimation of treatment effects moderated by individual orcontextual characteristics 168721 Application example 170722 Analytic procedure 170

73 MMWS estimation of the joint effects of concurrent treatments 174731 Application example 174732 Analytic procedure 175733 Joint treatment effects moderated by individual or contextual

characteristics 179

8 Cumulative effects of time-varying treatments 18581 Causal effects of treatment sequences 186

811 Application example 186812 Causal parameters 187

8121 Time-varying treatments 1878122 Time-varying potential outcomes 1878123 Causal effects of 2-year treatment sequences 1878124 Causal effects of multiyear treatment sequences 190

82 Existing strategies for evaluating time-varying treatments 190821 The endogeneity problem in nonexperimental data 190822 SEM 191

viii CONTENTS

823 Fixed-effects econometric models 192824 Sequential randomization 192825 Dynamic treatment regimes 193826 Marginal structural models and structural nested models 194

83 MMWS for evaluating 2-year treatment sequences 195831 Sequential ignorability 195832 Propensity scores 196833 MMWS computation 197

8331 MMWS adjustment for year 1 treatment selection 1978332 MMWS adjustment for two-year treatment

sequence selection 198834 Two-year growth model specification 199

8341 Growth model in the absence of treatment 2008342 Growth model in the absence of confounding 2008343 Weighted 2-year growth model 202

84 MMWS for evaluating multiyear sequences of multivalued treatments 204841 Sequential ignorability of multiyear treatment sequences 204842 Propensity scores for multiyear treatment sequences 204843 MMWS computation 205844 Weighted multiyear growth model 205845 Issues of sample size 206

85 Conclusion 207Appendix 8A A saturated model for evaluating multivalued treatments

over multiple time periods 207

Part III Mediation 211

9 Concepts of mediated treatment effects and experimental designsfor investigating causal mechanisms 21391 Introduction 21492 Path coefficients 21593 Potential outcomes and potential mediators 216

931 Controlled direct effects 217932 Controlled treatment-by-mediator interaction effect 217

94 Causal effects with counterfactual mediators 219941 Natural direct effect 219942 Natural indirect effect 220943 Natural treatment-by-mediator interaction effect 220944 Unstable unit treatment value 221

95 Population causal parameters 222951 Population average natural direct effect 224952 Population average natural indirect effect 225

96 Experimental designs for studying causal mediation 225961 Sequentially randomized designs 228962 Two-phase experimental designs 228963 Three- and four-treatment arm designs 230964 Experimental causal-chain designs 231

ixCONTENTS

965 Moderation-of-process designs 231966 Augmented encouragement designs 232967 Parallel experimental designs and parallel encouragement designs 232968 Crossover experimental designs and crossover encouragement

designs 233969 Summary 234

10 Existing analytic methods for investigating causal mediation mechanisms 238101 Path analysis and SEM 239

1011 Analytic procedure for continuous outcomes 2391012 Identification assumptions 2421013 Analytic procedure for discrete outcomes 245

102 Modified regression approach 2461021 Analytic procedure for continuous outcomes 2461022 Identification assumptions 2471023 Analytic procedure for binary outcomes 248

103 Marginal structural models 2501031 Analytic procedure 2501032 Identification assumptions 252

104 Conditional structural models 2521041 Analytic procedure 2521042 Identification assumptions 253

105 Alternative weighting methods 2541051 Analytic procedure 2541052 Identification assumptions 256

106 Resampling approach 2561061 Analytic procedure 2561062 Identification assumptions 257

107 IV method 2571071 Rationale and analytic procedure 2571072 Identification assumptions 258

108 Principal stratification 2591081 Rationale and analytic procedure 2591082 Identification assumptions 260

109 Sensitivity analysis 2611091 Unadjusted confounding as a product of hypothetical

regression coefficients 2611092 Unadjusted confounding reflected in a hypothetical

correlation coefficient 2621093 Limitations when the selection mechanism differs

by treatment 2641094 Other sensitivity analyses 265

1010 Conclusion 26510101 The essentiality of sequential ignorability 26510102 Treatment-by-mediator interactions 26610103 Homogeneous versus heterogeneous causal effects 26610104 Model-based assumptions 266

x CONTENTS

Appendix 10A Bias in path analysis estimation due to the omission oftreatment-by-mediator interaction 267

11 Investigations of a simple mediation mechanism 273111 Application example national evaluation of welfare-to-work strategies 274

1111 Historical context 2741112 Research questions 2751113 Causal parameters 2751114 NEWWS Riverside data 277

112 RMPW rationale 2771121 RMPW in a sequentially randomized design 278

11211 E[Y(0 M(0))] 27911212 E[Y(1 M(1))] 28011213 E[Y(1 M(0))] 28011214 E[Y(0 M(1))] 28111215 Nonparametric outcome model 282

1122 RMPW in a sequentially randomized block design 2831123 RMPW in a standard randomized experiment 2851124 Identification assumptions 286

113 Parametric RMPW procedure 287114 Nonparametric RMPW procedure 290115 Simulation results 292

1151 Correctly specified propensity score models 2921152 Misspecified propensity score models 2941153 Comparisons with path analysis and IV results 294

116 Discussion 2951161 Advantages of the RMPW strategy 2951162 Limitations of the RMPW strategy 295

Appendix 11A Causal effect estimation through the RMPW procedure 296Appendix 11B Proof of the consistency of RMPW estimation 297

12 RMPW extensions to alternative designs and measurement 301121 RMPW extensions to mediators and outcomes of alternative distributions 301

1211 Extensions to a multicategory mediator 30212111 Parametric RMPW procedure 30212112 Nonparametric RMPW procedure 304

1212 Extensions to a continuous mediator 3041213 Extensions to a binary outcome 306

122 RMPW extensions to alternative research designs 3061221 Extensions to quasiexperimental data 3071222 Extensions to data from cluster randomized trials 308

12221 Application example and research questions 30912222 Estimation of the total effect 31012223 RMPW analysis of causal mediation mechanisms 31012224 Identification assumptions 31212225 Contrast with multilevel path analysis and SEM 31212226 Contrast with multilevel prediction models 313

xiCONTENTS

1223 Extensions to data from multisite randomized trials 31312231 Research questions and causal parameters 31412232 Estimation of the total effect and its between-site variation 31512233 RMPW analysis of causal mediation mechanisms 31612234 Identification assumptions 31912235 Contrast with multilevel path analysis and SEM 319

123 Alternative decomposition of the treatment effect 321

13 RMPW extensions to studies of complex mediation mechanisms 325131 RMPW extensions to moderated mediation 325

1311 RMPW analytic procedure for estimating and testingmoderated mediation 326

1312 Path analysisSEM approach to analyzing moderatedmediation 327

1313 Principal stratification and moderated mediation 328132 RMPW extensions to concurrent mediators 328

1321 Treatment effect decomposition 32913211 Treatment effect decomposition without

between-mediator interaction 32913212 Treatment effect decomposition with

between-mediator interaction 3311322 Identification assumptions 3331323 RMPW procedure 333

13231 Estimating E[Y(1M1(1)M2(0))] 33513232 Estimating E[Y(1M1(0)M2(0))] 33513233 Causal effect estimation with noninteracting

concurrent mediators 33613234 Estimating E[Y(1M1(0)M2(1))] 33713235 Causal effect estimation with interacting concurrent

mediators 3371324 Contrast with the linear SEM approach 3381325 Contrast with the multivariate IV approach 339

133 RMPW extensions to consecutive mediators 3401331 Treatment effect decomposition 341

13311 Natural direct effect of the treatment on the outcome 34213312 Natural indirect effect mediated by M1 only 34213313 Natural indirect effect mediated by M2 only 34213314 Natural indirect effect mediated by an M1-by-M2

interaction 34313315 Treatment-by-mediator interactions 343

1332 Identification assumptions 3451333 RMPW procedure 347

13331 Estimating E[Y(1M1(0)M2(0M1(0)))] 34713332 Estimating E[Y(1M1(1)M2(0M1(0)))] 34913333 Estimating E[Y(1M1(0)M2(1M1(1)))] 34913334 Estimating E[Y(0M1(1)M2(0M1(0)))] and

E[Y(0M1(0)M2(1M1(1)))] 351

xii CONTENTS

1334 Contrast with the linear SEM approach 3531335 Contrast with the sensitivity-based estimation of bounds for

causal effects 354134 Discussion 355Appendix 13A Derivation of RMPW for estimating population average

counterfactual outcomes of two concurrentmediators 355

Appendix 13B Derivation of RMPW for estimating population averagecounterfactual outcomes of consecutivemediators 358

Part IV Spill-over 363

14 Spill-over of treatment effects concepts and methods 365141 Spill-over A nuisance a trifle or a focus 365142 Stable versus unstable potential outcome values An example

from agriculture 367143 Consequences for causal inference when spill-over is overlooked 369144 Modified framework of causal inference 371

1441 Treatment settings 3711442 Simplified characterization of treatment settings 3731443 Causal effects of individual treatment assignment and of peer

treatment assignment 375145 Identification Challenges and solutions 376

1451 Hypothetical experiments for identifying average treatmenteffects in the presence of social interactions 376

1452 Hypothetical experiments for identifying the impact ofsocial interactions 380

1453 Application to an evaluation of kindergarten retention 382146 Analytic strategies for experimental and quasiexperimental data 384

1461 Estimation with experimental data 3841462 Propensity score stratification 3851463 MMWS 386

147 Summary 387

15 Mediation through spill-over 391151 Definition of mediated effects through spill-over in a cluster

randomized trial 3931511 Notation 3931512 Treatment effect mediated by a focal individualrsquos

compliance 3941513 Treatment effect mediated by peersrsquo compliance through

spill-over 3941514 Decomposition of the total treatment effect 395

152 Identification and estimation of the spill-over effect in a clusterrandomized design 3951521 Identification in an ideal experiment 395

xiiiCONTENTS

1522 Identification when the mediators are not randomized 3981523 Estimation of mediated effects through spill-over 400

153 Definition of mediated effects through spill-over in a multisite trial 4021531 Notation 4021532 Treatment effect mediated by a focal individualrsquos compliance 4041533 Treatment effect mediated by peersrsquo compliance

through spill-over 4041534 Direct effect of individual treatment assignment on the outcome 4051535 Direct effect of peer treatment assignment on the outcome 4051536 Decomposition of the total treatment effect 405

154 Identification and estimation of spill-over effects in a multisite trial 4061541 Identification in an ideal experiment 4071542 Identification when the mediators are not randomized 4091543 Estimation of mediated effects through spill-over 410

15431 Estimating E[Y(1pM(p)Mminus(p))] andE[Y(000Mminus(0))] 410

15432 Estimating E[Y(1p0Mminus(p))] 41115433 Estimating E[Y(1p0Mminus(0))] 41215434 Estimating E[Y(0p0Mminus(0))] 412

155 Consequences of omitting spill-over effects in causal mediation analyses 4121551 Biased inference in a cluster randomized trial 4131552 Biased inference in a multisite randomized trial 4131553 Biased inference of the local average treatment effect 415

156 Quasiexperimental application 416157 Summary 419Appendix 151 Derivation of the weight for estimating the population

average counterfactual outcome E[Y(1 p 0Mminus( p))] 419Appendix 152 Derivation of bias in the ITT effect due to the omission of

spill-over effects 420

Index 423

xiv CONTENTS

Preface

A scientific mind in training seeks comprehensive descriptions of every interesting phenom-enon In such a mind data are to be pieced together for comparisons contrasts and eventuallyinferences that are required for understanding causes and effects that may have led to thephenomenon of interest Yet without disciplined reasoning a causal inference would slip intothe rut of a ldquocasual inferencerdquo as easily as making a typographical error

As a doctoral student thirsting for rigorous methodological training at the University ofMichigan I was very fortunate to be among the first group of students on the campus attendinga causal inference seminar in 2000 jointly taught by Yu Xie from the Department of Sociol-ogy Susan Murphy from the Department of Statistics and Stephen Raudenbush from theSchool of Education The course was unique in both content and pedagogy The instructorsput together a bibliography drawn from the past and current causal inference literature origi-nated from multiple disciplines In the spirit of ldquoreciprocal teachingrdquo the students were delib-erately organized into small groups each representing a mix of disciplinary backgrounds andtook turns to teach the weekly readings under the guidance of a faculty instructor This inval-uable experience revealed to me that the pursuit of causal inference is a multidisciplinaryendeavor In the rest of my doctoral study I continued to be immersed in an extraordinaryinterdisciplinary community in the form of the Quantitative Methodology Program (QMP)seminars led by the above three instructors who were then joined by Ben Hansen(Statistics) Richard Gonzalez (Psychology) Roderick Little (Biostatistics) Jake Bowers(Political Science) and many others I have tried whenever possible to carry the same spiritinto my own teaching of causal inference courses at the University of Toronto and now at theUniversity of Chicago In these courses communicating understandings of causal problemsacross disciplinary boundaries has been particularly challenging but also stimulating and grat-ifying for myself and for students from various departments and schools

Under the supervision of Stephen Raudenbush I took a first stab at the conceptual frame-work of causal inference in my doctoral dissertation by considering peer spill-over in schoolsettings From that point on I have organized my methodological research to tackle some ofthe major obstacles to causal inferences in policy and program evaluations My workaddresses issues including (i) how to conceptualize and evaluate the causal effects of educa-tional treatments when studentsrsquo responses to alternative treatments depend on various fea-tures of the organizational settings including peer composition (ii) how to adjust forselection bias in evaluating the effects of concurrent and consecutive multivalued treatmentsand (iii) how to conceptualize and analyze causal mediation mechanisms My methodological

research has been driven by the substantive interest in understanding the role of socialinstitutionsmdashschools in particularmdashin shaping human development especially during theages of fast growth (eg childhood adolescence and early adulthood) I have shared meth-odological advances with a broad audience in social sciences through applying the causalinference methods to prominent substantive issues such as grade retention within-class group-ing services for English language learners and welfare-to-work strategies These illustrationsare used extensively in this book

Among social scientists awareness of methodological challenges in causal investigationsis higher than ever before In response to this rising demand causal inference is becoming oneof the most productive scholarly fields in the past decade I have had the benefit of followingthe work and exchanging ideas with some fellow methodologists who are constantly contri-buting new thoughts and making breakthroughs These include Daniel Almirall HowardBloom Tom Cook Michael Elliot Ken Frank Ben Hansen James Heckman JenniferHill Martin Huber Kosuke Imai Booil Jo John R Lockwood David MacKinnon DanielMcCaffrey Luke Miratrix Richard Murnane Derek Neal Lindsey Page Judea PearlStephen Raudenbush Sean Reardon Michael Seltzer Youngyun Shin Betsy SinclairMichael Sobel Peter Steiner Elizabeth Stuart Eric Thetchen Tchetchen Tyler VanderWeeleand Kazuo Yamaguchi I am grateful to Larry Hedges who offered me the opportunity of guestediting the Journal of Research in Educational Effectiveness special issue on the statisticalapproaches to studying mediator effects in education research in 2012 Many of the colleagueswhom I mentioned above contributed to the open debate in the special issue by either propos-ing or critically examining a number of innovative methods for studying causal mechanismsI anticipate that many of them are or will soon be writing their own research monographs oncausal inference if they have not already done so Therefore readers of this book are stronglyrecommended to browse past and future titles by these and other authors for alternative per-spectives and approaches My own understanding of course will continue to evolve as well inthe coming years

One has to be ambitious to stay upfront in substantive areas as well as in methodology Thebest strategy apparently is to learn from substantive experts During different phases of mytraining and professional career I have had the opportunities to work with David K CohenBrian Rowan Deborah Ball Carl Corter Janette Pelletier Takako Nomi Esther Geva DavidFrancis Stephanie Jones Joshua Brown and Heather D Hill Many of my substantive insightswere derived from conversations with these mentors and colleagues I am especially gratefulto Bob Granger former president of the William T Grant Foundation who reassured me thatldquoBy staying a bit broad you will learn a lot and many fields will benefit from your workrdquo

Like many other single-authored books this one is built on the generous contributions ofstudents and colleagues who deserve special acknowledgement Excellent research assistancefrom Jonah Deutsch Joshua Gagne Rachel Garrett Yihua Hong Xu Qin Cheng Yang andBing Yu was instrumental in the development and implementation of the new methods pre-sented in this book Richard Congdon a renowned statistical software programmer broughtrevolutionary ideas to interface designs in software development with the aim of facilitatingusersrsquo decision-making in a series of causal analyses RichardMurnane Stephen Raudenbushand Michael Sobel carefully reviewed and critiqued multiple chapters of the manuscript draftAdditional comments and suggestions on earlier drafts came fromHoward Bloom Ken FrankBen Hansen Jennifer Hill Booil Jo Ben Kelcey David MacKinnon and Fan Yang TereseSchwartzman provided valuable assistance in manuscript preparation The writing of thisbook was supported by a Scholars Award from the William T Grant Foundation and an

xvi PREFACE

Institute of Education Sciences (IES) Statistical and Research Methodology in EducationGrant from the US Department of Education All the errors in the published form are ofcourse the sole responsibility of the author

Project Editors Richard Davis Prachi Sinha Sahay and Liz Wingett and Assistant EditorHeather Kay at Wiley and project Managers P Jayapriya and R Jayavel at SPi Global havebeen wonderful to work with throughout the manuscript preparation and final production ofthe book Debbie Jupe Commissioning Editor at Wiley was remarkably effective in initiatingthe contact with me and in organizing anonymous reviews of the book proposal and of themanuscript These constructive reviews have shaped the book in important ways I cannotagree more with one of the reviewers that ldquocausality cannot be established on a pure statisticalgroundrdquo The book therefore highlights the importance of substantive knowledge and researchdesigns and places a great emphasis on clarifying and evaluating assumptions required foridentifying causal effects in the context of each application Much caution is raised againstpossible misusage of statistical techniques for analyzing causal relationships especiallywhen data are inadequate Yet I maintain that improved statistical procedures along withimproved research designs would greatly enhance our ability in attempt to empiricallyexamine well-articulated causal theories

xviiPREFACE

Part I

OVERVIEW

1

Introduction

According to an ancient Chinese fable a farmer who was eager to help his crops grow wentinto his field and pulled each seedling upward After exhausting himself with the work heannounced to his family that they were going to have a good harvest only to find the nextmorning that the plants had wilted and died Readers with minimal agricultural knowledgemay immediately point out the following the farmerrsquos intervention theory was based on acorrect observation that crops that grow taller tend to produce more yield Yet his hypothesisreflects a false understanding of the cause and the effectmdashthat seedlings pulled to be tallerwould yield as much as seedlings thriving on their own

In their classic Design and Analysis of Experiments Hinkelmann and Kempthorne (1994updated version of Kempthorne 1952) discussed two types of science descriptive science andthe development of theory These two types of science are interrelated in the following senseobservations of an event and other related events often selected and classified for descriptionby scientists naturally lead to one or more explanations that we call ldquotheoretical hypothesesrdquowhich are then screened and falsified by means of further observations experimentation andanalyses (Popper 1963) The experiment of pulling seedlings to be taller was costly but didserve the purpose of advancing this farmerrsquos knowledge of ldquowhat does not workrdquo To developa successful intervention in this case would require a series of empirical tests of explicittheories identifying potential contributors to crop growth This iterative process graduallydeepens our knowledge of the relationships between supposed causes and effectsmdashthatis causalitymdashand may eventually increase the success of agricultural medical and socialinterventions

11 Concepts of moderation mediation and spill-over

Although the story of the ancient farmer is fictitious numerous examples can be found in thereal world in which well-intended interventions fail to produce the intended benefits or inmany cases even lead to unintended consequences ldquoInterventionsrdquo and ldquotreatmentsrdquo usedinterchangeably in this book broadly refer to actions taken by agents or circumstances expe-rienced by an individual or groups of individuals Interventions are regularly seen in educa-tion physical and mental health social services business politics and law enforcement In an

Causality in a Social World Moderation Meditation and Spill-over First Edition Guanglei Hongcopy 2015 John Wiley amp Sons Ltd Published 2015 by John Wiley amp Sons Ltd

education intervention for example teachers are typically the agents who deliver a treatmentto students while the impact of the treatment on student outcomes is of ultimate causalinterest Some educational practices such as ldquoteaching to the testrdquo have been criticized tobe nearly as counterproductive as the attempt of helping seedlings grow by pulling themupward ldquoInterventionsrdquo and ldquotreatmentsrdquo under consideration do not exclude undesiredexperiences such as exposure to poverty abuse crime or bereavement A treatment plannedor unplanned becomes a focus of research if there are theoretical reasons to anticipate itsimpact positive or negative on the well-being of individuals who are embedded in socialsettings including families classrooms schools neighborhoods and workplaces

In social science research in general and in policy and program evaluations in particularquestions concerning whether an intervention works and if so which version of the interven-tion works for whom under what conditions and why are key to the advancement of scien-tific and practical knowledge Although most empirical investigations in the social sciencesconcentrate on the average effect of a treatment for a specific population as opposed to theabsence of such a treatment (ie the control condition) in-depth theoretical reasoning withregard to how the causal effect is generated substantiated by compelling empirical evidenceis crucial for advancing scientific understanding

First when there are multiple versions or different dosages of the treatment or when thereare multiple versions of the control condition a binary divide between ldquothe treatmentrdquo andldquothe controlrdquo may not be as informative as fine-grained comparisons across for exampleldquotreatment version Ardquo ldquotreatment version Brdquo ldquocontrol version Ardquo and ldquocontrol versionBrdquo For example expanding the federally funded Head Start program to poor children isexpected to generate a greater benefit when few early childhood education alternatives areavailable (call it ldquocontrol version Ardquo) than when there is an abundance of alternatives includ-ing state-sponsored preschool programs (call it ldquocontrol version Brdquo)

Second the effect of an intervention will likely vary among individuals or across socialsettings A famous example comes from medical research the well-publicized cardiovascularbenefits of initiating estrogen therapy during menopause were contradicted later by experi-mental findings that the same therapy increased postmenopausal womenrsquos risk for heartattacks The effect of an intervention may also depend on the provision of some other con-current or subsequent interventions Such heterogeneous effects are often characterized asmoderated effects in the literature

Third alternative theories may provide competing explanations for the causal mechan-isms that is the processes through which the intervention produces its effect A theoreticalconstruct characterizing the hypothesized intermediate process is called a mediator of theintervention effect The fictitious farmer never developed an elaborate theory as to whatcaused some seedlings to surpass others in growth Once scientists revealed the causal rela-tionship between access to chemical nutrients in soil and plant growth wide applications ofchemically synthesized fertilizers finally led to a major increase in crop production

Finally it is well known in agricultural experiments that a new type of fertilizer applied toone plot may spill-over to the next plot Because social connections among individuals areprevalent within organizations or through networks an individualrsquos response to the treatmentmay similarly depend on the treatment for other individuals in the same social setting whichmay lead to possible spill-overs of intervention effects among individual human beings

Answering questions with regard to moderation mediation and spill-over poses majorconceptual and analytic challenges To date psychological research often presents well-articulated theories of causal mechanisms relating stimuli to responses Yet researchers often

4 CAUSALITY IN A SOCIAL WORLD

lack rigorous analytic strategies for empirically screening competing theories explaining theobserved effect Sociologists have keen interest in the spill-over of treatment effects transmit-ted through social interactions yet have produced limited evidence quantifying such effectsAs many have pointed out in general terminological ambiguity and conceptual confusionhave been prevalent in the published applied research (Holmbeck 1997 Kraemer et al2002 2008)

A new framework for conceptualizing moderated mediated and spill-over effects hasemerged relatively recently in the statistics and econometrics literature on causal inference(eg Abbring and Heckman 2007 Frangakis and Rubin 2002 Heckman and Vytlacil2007a b Holland 1988 Hong and Raudenbush 2006 Hudgens and Halloran 2008Jo 2008 Pearl 2001 Robins and Greenland 1992 Sobel 2008) The potential for furtherconceptual and methodological development and for broad applications in the field of behav-ioral and social sciences promises to greatly advance the empirical basis of our knowledgeabout causality in the social world

This book clarifies theoretical concepts and introduces innovative statistical strategies forinvestigating the average effects of multivalued treatments moderated treatment effectsmediated treatment effects and spill-over effects in experimental or quasiexperimental dataDefining individual-specific and population average treatment effects in terms of potentialoutcomes the book relates the mathematical forms to the substantive meanings of moderatedmediated and spill-over effects in the context of application examples It also explicates andevaluates identification assumptions and contrasts innovative statistical strategies with con-ventional analytic methods

111 Moderated treatment effects

It is hard to accept the assumption that a treatment would produce the same impact for everyindividual in every possible circumstance Understanding the heterogeneity of treatmenteffects therefore is key to the development of causal theories For example some studiesreported that estrogen therapy improved cardiovascular health among women who initiatedits use during menopause According to a series of other studies however the use of estrogentherapy increased postmenopausal womenrsquos risk for heart attacks (Grodstein et al 19962000 Writing Group for the Womenrsquos Health Initiative Investigators 2002) The sharp con-trast of findings from these studies led to the hypothesis that age of initiation moderates theeffect of estrogen therapy on womenrsquos health (Manson and Bassuk 2007 Rossouw et al2007) Revelations of the moderated causal relationship greatly enrich theoretical understand-ing and in this case directly inform clinical practice

In another example dividing students by ability into small groups for reading instructionhas been controversial for decades Many believed that the practice benefits high-ability stu-dents at the expense of their low-ability peers and exacerbates educational inequality (Grantand Rothenberg 1986 Rowan and Miracle 1983 Trimble and Sinclair 1987) Recentevidence has shown that first of all the effect of ability grouping depends on a number ofconditions including whether enough time is allocated to reading instruction and how wellthe class can be managed Hence instructional time and class management are among theadditional moderators to consider for determining the merits of ability grouping (Hong andHong 2009 Hong et al 2012a b) Moreover further investigations have generated evidencethat contradicts the long-held belief that grouping undermines the interests of low-abilitystudents Quite the contrary researchers have found that grouping is beneficial for students

5INTRODUCTION

with low or medium prior abilitymdashif adequate time is provided for instruction and if the classis well managed On the other hand ability grouping appears to have a minimal impact onhigh-ability studentsrsquo literacy These results were derived from kindergarten data and maynot hold for math instruction and for higher grade levels Replications with different subpo-pulations and in different contexts enable researchers to assess the generalizability of a theory

In a third example one may hypothesize that a student is unlikely to learn algebra well inninth grade without a solid foundation in eighth-grade prealgebra One may also argue that theprogress a student made in learning eighth-grade prealgebra may not be sustained without thefurther enhancement of taking algebra in ninth grade In other words the earlier treatment(prealgebra in eighth grade) and the later treatment (algebra in ninth grade) may reinforceone other each moderates the effect of the other treatment on the final outcome (math achieve-ment at the end of ninth grade) As Hong and Raudenbush (2008) argued the cumulativeeffect of two treatments in a well-aligned sequence may exceed the sum of the benefits oftwo single-year treatments Similarly experiencing two consecutive years of inferior treat-ments such as encountering an incompetent math teacher who dampens a studentrsquos self-efficacy in math learning in eighth grade and then again in ninth grade could do more damagethan the sum of the effect of encountering an incompetent teacher only in eighth grade and theeffect of having such a teacher only in ninth grade

There has been a great amount of conceptual confusion regarding how a moderatorrelates to the treatment and the outcome On one hand many researchers have used the termsldquomoderatorsrdquo and ldquomediatorsrdquo interchangeably without understanding the crucial distinctionbetween the two An overcorrective attempt on the other hand has led to the arbitrary rec-ommendation that a moderator must occur prior to the treatment and be minimally associatedwith the treatment (James and Brett 1984 Kraemer et al 2001 2008) In Chapter 6 weclarify that in essence the causal effect of the treatment on the outcome may depend onthe moderator value A moderator can be a subpopulation identifier a contextual character-istic a concurrent treatment or a preceding or succeeding treatment In the earlier examplesage of estrogen therapy initiation and a studentrsquos prior ability are subpopulation identifiers themanageability of a class which reflects both teacher skills and peer behaviors characterizes thecontext literacy instruction time is a concurrent treatment while prealgebra is a precedingtreatment for algebra A moderator does not have to occur prior to the treatment and doesnot have to be independent of the treatment

Once a moderated causal relationship has been defined in terms of potential outcomes theresearcher then chooses an appropriate experimental design for testing the moderation theoryThe assumptions required for identifying the moderated causal relationships differ across dif-ferent designs and have implications for analyzing experimental and quasiexperimental dataRandomized block designs are suitable for examining individual or contextual characteristicsas potential moderators factorial designs enable one to determine the joint effects of two ormore concurrent treatments and sequential randomized designs are ideal for assessing thecumulative effects of consecutive treatments Multisite randomized trials constitute anadditional type of designs in which the experimental sites are often deliberately sampled torepresent a population of geographical locations or social organizations Site membershipscan be viewed as implicit moderators that summarize a host of features of a local environmentReplications of a treatment over multiple sites allow one to quantify the heterogeneity of treat-ment effects across the sites Chapters 7 and 8 are focused on statistical methods for moder-ation analyses with quasiexperimental data In particular these chapters demonstrate througha number of application examples how a nonparametric marginal mean weighting through

6 CAUSALITY IN A SOCIAL WORLD

stratification (MMWS) method can overcome some important limitations of other existingmethods

Moderation questions are not restricted to treatmentndashoutcome relationships Rather theyare prevalent in investigations of mediation and spill-over This is because heterogeneity oftreatment effects may be explained by the variation in causal mediation mechanisms acrosssubpopulations and contexts It is also because the treatment effect for a focal individualmay depend on whether there is spill-over from other individuals through social interactionsYet similar to the confusion around the concept of moderation among applied researchersthere has been a great amount of misunderstanding with regard to mediation

112 Mediated treatment effects

Questions about mediation are at the core of nearly every scientific theory In its simplestform a theory explains why treatment Z causes outcome Y by hypothesizing an intermediateprocess involving at least one mediatorM that could have been changed by the treatment andcould subsequently have an impact on the outcome A causal mediation analysis then decom-poses the total effect of the treatment on the outcome into two parts the effect transmittedthrough the hypothesized mediator called the indirect effect and the difference betweenthe total effect and the indirect effect called the direct effect The latter represents the treat-ment effect channeled through other unspecified pathways (Alwin and Hauser 1975 Baronand Kenny 1986 Duncan 1966 Shadish and Sweeney 1991) Most researchers in socialsciences have followed the convention of illustrating the direct effect and the indirect effectwith path diagrams and then specifying linear regression models postulated to represent thestructural relationships between the treatment the mediator and the outcome

In Chapter 9 we point out that the approach to defining the indirect effect and the directeffect in terms of path coefficients is often misled by oversimplistic assumptions about thestructural relationships In particular this approach typically overlooks the fact that a treat-ment may generate an impact on the outcome not only by changing the mediator value butalso by changing the mediatorndashoutcome relationship (Judd and Kenny 1981) An examplein Chapter 9 illustrates such a case An experiment randomizes students to either an experi-mental condition which provides them with study materials and encourages them to study fora test or to a control condition which provides neither study materials nor encouragement(Holland 1988 Powers and Swinton 1984) One might hypothesize that encouraging exper-imental group members to study will increase the time they spend studying a focal mediatorin this example which in turn will increase their average test scores One might furtherhypothesize that even without a change in study time providing study materials to experi-mental group members will enable them to study more effectively than they otherwise wouldConsequently the effect of time spent studying might be greater under the experimentalcondition than under the control condition Omitting the treatment-by-mediator interactioneffect will lead to bias in the estimation of the indirect effect and the direct effect

Chapter 11 describes in great detail an evaluation study contrasting a new welfare-to-workprogram emphasizing active participation in the labor force with a traditional program thatguaranteed cash assistance without a mandate to seek employment (Hong Deutsch and Hill2011) Focusing on the psychological well-being of welfare applicants who were singlemothers with preschool-aged children the researchers hypothesized that the treatment wouldincrease employment rate among the welfare recipients and that the treatment-inducedincrease in employment would likely reduce depressive symptoms under the new program

7INTRODUCTION

They also hypothesized that in contrast the same increase in employment was unlikely tohave a psychological impact under the traditional program Hence the treatment-by-mediatorinteraction effect on the outcome was an essential component of the intermediate process

We will show that by defining the indirect effect and the direct effect in terms of potentialoutcomes one can avoid invoking unwarranted assumptions about the unknown structuralrelationships (Holland 1988 Pearl 2001 Robins and Greenland 1992) The potential out-comes framework has further clarified that in a causal mediation theory the treatment and themediator are both conceivably manipulable In the aforementioned example a welfare appli-cant could be assigned at random to either the new program or the traditional one Once havingbeen assigned to one of the programs the individual might or might not be employed due tovarious structural constraints market fluctuations or other random events typically beyondher control

Because the indirect effect and the direct effect are defined in terms of potential outcomesrather than on the basis of any specific regression models it becomes possible to distinguishthe definitions of these causal effects from their identification and estimation The identifica-tion step relates a causal parameter to observable population data For example for individualsassigned at random to an experimental condition their average counterfactual outcome asso-ciated with the control condition is expected to be equal to the average observable outcome ofthose assigned to the control condition Therefore the population average causal effect can beeasily identified in a randomized experiment The estimation step then relates the sample datato the observable population quantities involved in identification while taking into account thedegree of sampling variability

A major challenge in identification is that the mediatorndashoutcome relationship tends tobe confounded by selection even if the treatment is randomized Chapter 9 reviews variousexperimental designs that have been proposed by past researchers for studying causalmediation mechanisms Chapter 10 compares the identification assumptions and the analyticprocedures across a wide range of analytic methods These discussions are followed by anintroduction of the ratio-of-mediator-probability weighting (RMPW) strategy in Chapter 11In comparison with most existing strategies for causal mediation analysis RMPW relies onrelatively fewer identification assumptions and model-based assumptions Chapters 12 and13 will show that this new strategy can be applied broadly with extensions to multilevelexperimental designs and to studies of complex mediation mechanisms involving multiplemediators

113 Spill-over effects of a treatment

It is well known in vaccination research that an unvaccinated person can benefit when mostother people in the local community are vaccinated This is viewed as a spill-over of the treat-ment effect from the vaccinated people to an unvaccinated person Similarly due to socialinteractions among individuals or groups of individuals an intervention received by somemay generate a spill-over impact on others affiliated with the same organization or connectedthrough the same network For example effective policing in one neighborhood may driveoffenders to operate in other neighborhoods In evaluating an innovative community policingprogram researchers found that a neighborhood not assigned to community policing tended tosuffer if the surrounding neighborhoods were assigned to community policing This is aninstance in which a well-intended treatment generates an unintended negative spill-over effectHowever being assigned to community policy was found particularly beneficial when thesurrounding neighborhoods also received the intervention (Verbitsky-Savitz and Raudenbush

8 CAUSALITY IN A SOCIAL WORLD

2012) In another example the impact of a school-based mentoring program targeted atstudents displaying delinquent behaviors may be enhanced for a focal student if his or herat-risk peers are assigned to the mentoring program at the same time

The previous examples challenge the ldquostable unit treatment value assumptionrdquo (SUTVA)that has been invoked in most causal inference studies This assumption states that an indi-vidualrsquos potential response to a treatment depends neither on how the treatment is assignednor on the treatment assignments of other individuals (Rubin 1980 1986) Rubin and othersbelieve that without resorting to SUTVA the causal effect of a treatment becomes hard todefine and that causal inference may become intractable However as Sobel (2006) has illus-trated with the Moving to Opportunity (MTO) experiment that offered housing vouchersenabling low-income families to move to low-poverty neighborhoods there are possible con-sequences of violating SUTVA in estimating the treatment effects on outcomes such as safetyand self-sufficiency Because social interactions among individuals may affect whether onevolunteers to participate in the study whether one moves to a low-poverty neighborhood afterreceiving the housing voucher as well as housing project residentsrsquo subjective perceptions oftheir neighborhoods the program may have a nonzero impact on the potential outcome of theuntreated As a result despite the randomization of treatment assignment the mean differencein the observed outcome between the treated units and the untreated units may be biased forthe average treatment effect Rather the observable quantity is the difference between theeffect of treating some rather than treating none for the treated and the effect of treating somerather than treating none for the untreated the latter being the pure spill-over effect for theuntreated

In making attempts to relax SUTVA researchers have proposed several alternative frame-works that incorporate possible spill-over effects (see Hong and Raudenbush 2013 for areview) Hong (2004) presented a model that involves treatments and treatment settingsA treatment setting for an individual is a local environment constituted by a set of agentsand participants along with their treatment assignments An individualrsquos potential outcomevalue under a given treatment is assumed stable when the treatment setting is fixed the poten-tial outcome may take different values when the treatment setting shifts One may investigatewhether the treatment effect depends on the treatment setting Applying this framework Hongand Raudenbush (2006) examined the effect on a childrsquos academic growth of retaining thechild in kindergarten rather than promoting the child to first grade when a relatively smallproportion of low-achieving peers in the same school are retained as opposed to when a rel-atively large proportion of the peers are retained Hudgens and Halloran (2008) presented arelated framework in which the effect on an individual of the treatment received by this indi-vidual is distinguished from the effect on the individual of the treatment received by others inthe same local community These effects can be identified if communities are assigned atrandom to different treatment assignment strategies (such as retaining a large proportion oflow-achieving students as opposed to retaining a small proportion of such students) and sub-sequently individuals within a community are randomized for treatment Chapter 14 reviewsthese frameworks and discusses identification and estimation strategies

Social contagion may also serve as an important channel through which the effect of anintervention is transmitted For example a school-wide intervention may reduce aggressivebehaviors and thereby improve studentsrsquo psychological well-being by improving the qualityof interpersonal relationships in other classes as well as in onersquos own class This is becausechildren interact not simply with their classmates but also with those from other classes in thehallways or on the playground (VanderWeele et al 2013) In this case the spill-over betweenclasses becomes a part of the mediation mechanism In a study of student mentoring whether

9INTRODUCTION

an individual actually participated in the mentoring program despite the initial treatmentassignment is typically viewed as a primary mediator The proportion of onersquos peers that par-ticipated in the program may act as a second mediator A treatment may also exert its impactpartly through regrouping individuals and thereby changing peer composition (Hong andNomi 2012) Chapter 15 discusses analytic strategies for detecting spill-over as a part ofthe causal mediation mechanism

12 Weighting methods for causal inference

In most behavioral and social science applications major methodological challenges arise dueto the selection of treatments in a quasiexperimental study and the selection of mediator valuesin experimental and quasiexperimental studies Statistical methods most familiar to research-ers in social sciences are often inadequate for causal inferences with regard to multivaluedtreatments moderation mediation and spill-over The book offers a major revision to under-standing of causality in the social world by introducing two complementary weighting stra-tegies both featuring nonparametric approaches to estimating these causal effects The newweighting methods greatly simplify model specifications while enhancing the robustness ofresults by minimizing the reliance on some key assumptions

The propensity score-basedMMWSmethod removes selection bias associated with a largenumber of covariates by equating the pretreatment composition between treatment groups(Hong 2010a 2012 Huang et al 2005) Unlike propensity score matching and stratificationthat are mostly restricted to evaluations of binary treatments the MMWS method is flexiblefor evaluating binary and multivalued treatments by approximating a completely randomizedexperiment In evaluating whether the treatment effects differ across subpopulations definedby individual characteristics or treatment settings researchers may assign weights within eachsubpopulation in order to approximate a randomized block design To investigate whetherone treatment moderates the effect of another concurrent treatment researchers may assignweights to the data to approximate a factorial randomized design The method can also beused to assess whether the effect of an initial treatment is amplified or weakened by a subse-quent treatment or to identify an optimal treatment sequence through approximating a sequen-tial randomized experiment Even though such analyses can similarly be conducted throughinverse-probability-of-treatment weighting (IPTW) that has been increasingly employed inepidemiological research (Hernaacuten Brumback and Robins 2000 Robins Hernaacuten andBrumback 2000) IPTW is known for bias and imprecision in estimation especially whenthe propensity score models are misspecified in their functional forms (Hong 2010a Kangand Schafer 2007 Schafer and Kang 2008 Waernbaum 2012) In contrast the nonparamet-ric MMWS method displays a relatively high level of robustness despite such misspecifica-tions and also gains efficiency as indicated by simulation results (Hong 2010a)

To study causal mediation mechanisms the RMPW method decomposes the total effectof a treatment into an ldquoindirect effectrdquo transmitted through a specific mediator and a ldquodirecteffectrdquo representing unspecified mechanisms In contrast with most existing methods formediation analysis the RMPW-adjusted outcome model is extremely simple and is nonpara-metric in nature It generates estimates of the causal effects along with their sampling errorswhile adjusting for pretreatment covariates that confound the mediatorndashoutcome relationshipsthrough weighting The method applies regardless of the distribution of the outcome the dis-tribution of the mediator or the functional relationship between the outcome and the mediator

10 CAUSALITY IN A SOCIAL WORLD

Page 9: Thumbnail - download.e-bookshelf.de...4.2.4 IPTW for bias removal 84 4.3 MMWS 86 4.3.1 Theoretical rationale 86 ... 8.3.4.2 Growth model in the absence of confounding 200 8.3.4.3 Weighted

823 Fixed-effects econometric models 192824 Sequential randomization 192825 Dynamic treatment regimes 193826 Marginal structural models and structural nested models 194

83 MMWS for evaluating 2-year treatment sequences 195831 Sequential ignorability 195832 Propensity scores 196833 MMWS computation 197

8331 MMWS adjustment for year 1 treatment selection 1978332 MMWS adjustment for two-year treatment

sequence selection 198834 Two-year growth model specification 199

8341 Growth model in the absence of treatment 2008342 Growth model in the absence of confounding 2008343 Weighted 2-year growth model 202

84 MMWS for evaluating multiyear sequences of multivalued treatments 204841 Sequential ignorability of multiyear treatment sequences 204842 Propensity scores for multiyear treatment sequences 204843 MMWS computation 205844 Weighted multiyear growth model 205845 Issues of sample size 206

85 Conclusion 207Appendix 8A A saturated model for evaluating multivalued treatments

over multiple time periods 207

Part III Mediation 211

9 Concepts of mediated treatment effects and experimental designsfor investigating causal mechanisms 21391 Introduction 21492 Path coefficients 21593 Potential outcomes and potential mediators 216

931 Controlled direct effects 217932 Controlled treatment-by-mediator interaction effect 217

94 Causal effects with counterfactual mediators 219941 Natural direct effect 219942 Natural indirect effect 220943 Natural treatment-by-mediator interaction effect 220944 Unstable unit treatment value 221

95 Population causal parameters 222951 Population average natural direct effect 224952 Population average natural indirect effect 225

96 Experimental designs for studying causal mediation 225961 Sequentially randomized designs 228962 Two-phase experimental designs 228963 Three- and four-treatment arm designs 230964 Experimental causal-chain designs 231

ixCONTENTS

965 Moderation-of-process designs 231966 Augmented encouragement designs 232967 Parallel experimental designs and parallel encouragement designs 232968 Crossover experimental designs and crossover encouragement

designs 233969 Summary 234

10 Existing analytic methods for investigating causal mediation mechanisms 238101 Path analysis and SEM 239

1011 Analytic procedure for continuous outcomes 2391012 Identification assumptions 2421013 Analytic procedure for discrete outcomes 245

102 Modified regression approach 2461021 Analytic procedure for continuous outcomes 2461022 Identification assumptions 2471023 Analytic procedure for binary outcomes 248

103 Marginal structural models 2501031 Analytic procedure 2501032 Identification assumptions 252

104 Conditional structural models 2521041 Analytic procedure 2521042 Identification assumptions 253

105 Alternative weighting methods 2541051 Analytic procedure 2541052 Identification assumptions 256

106 Resampling approach 2561061 Analytic procedure 2561062 Identification assumptions 257

107 IV method 2571071 Rationale and analytic procedure 2571072 Identification assumptions 258

108 Principal stratification 2591081 Rationale and analytic procedure 2591082 Identification assumptions 260

109 Sensitivity analysis 2611091 Unadjusted confounding as a product of hypothetical

regression coefficients 2611092 Unadjusted confounding reflected in a hypothetical

correlation coefficient 2621093 Limitations when the selection mechanism differs

by treatment 2641094 Other sensitivity analyses 265

1010 Conclusion 26510101 The essentiality of sequential ignorability 26510102 Treatment-by-mediator interactions 26610103 Homogeneous versus heterogeneous causal effects 26610104 Model-based assumptions 266

x CONTENTS

Appendix 10A Bias in path analysis estimation due to the omission oftreatment-by-mediator interaction 267

11 Investigations of a simple mediation mechanism 273111 Application example national evaluation of welfare-to-work strategies 274

1111 Historical context 2741112 Research questions 2751113 Causal parameters 2751114 NEWWS Riverside data 277

112 RMPW rationale 2771121 RMPW in a sequentially randomized design 278

11211 E[Y(0 M(0))] 27911212 E[Y(1 M(1))] 28011213 E[Y(1 M(0))] 28011214 E[Y(0 M(1))] 28111215 Nonparametric outcome model 282

1122 RMPW in a sequentially randomized block design 2831123 RMPW in a standard randomized experiment 2851124 Identification assumptions 286

113 Parametric RMPW procedure 287114 Nonparametric RMPW procedure 290115 Simulation results 292

1151 Correctly specified propensity score models 2921152 Misspecified propensity score models 2941153 Comparisons with path analysis and IV results 294

116 Discussion 2951161 Advantages of the RMPW strategy 2951162 Limitations of the RMPW strategy 295

Appendix 11A Causal effect estimation through the RMPW procedure 296Appendix 11B Proof of the consistency of RMPW estimation 297

12 RMPW extensions to alternative designs and measurement 301121 RMPW extensions to mediators and outcomes of alternative distributions 301

1211 Extensions to a multicategory mediator 30212111 Parametric RMPW procedure 30212112 Nonparametric RMPW procedure 304

1212 Extensions to a continuous mediator 3041213 Extensions to a binary outcome 306

122 RMPW extensions to alternative research designs 3061221 Extensions to quasiexperimental data 3071222 Extensions to data from cluster randomized trials 308

12221 Application example and research questions 30912222 Estimation of the total effect 31012223 RMPW analysis of causal mediation mechanisms 31012224 Identification assumptions 31212225 Contrast with multilevel path analysis and SEM 31212226 Contrast with multilevel prediction models 313

xiCONTENTS

1223 Extensions to data from multisite randomized trials 31312231 Research questions and causal parameters 31412232 Estimation of the total effect and its between-site variation 31512233 RMPW analysis of causal mediation mechanisms 31612234 Identification assumptions 31912235 Contrast with multilevel path analysis and SEM 319

123 Alternative decomposition of the treatment effect 321

13 RMPW extensions to studies of complex mediation mechanisms 325131 RMPW extensions to moderated mediation 325

1311 RMPW analytic procedure for estimating and testingmoderated mediation 326

1312 Path analysisSEM approach to analyzing moderatedmediation 327

1313 Principal stratification and moderated mediation 328132 RMPW extensions to concurrent mediators 328

1321 Treatment effect decomposition 32913211 Treatment effect decomposition without

between-mediator interaction 32913212 Treatment effect decomposition with

between-mediator interaction 3311322 Identification assumptions 3331323 RMPW procedure 333

13231 Estimating E[Y(1M1(1)M2(0))] 33513232 Estimating E[Y(1M1(0)M2(0))] 33513233 Causal effect estimation with noninteracting

concurrent mediators 33613234 Estimating E[Y(1M1(0)M2(1))] 33713235 Causal effect estimation with interacting concurrent

mediators 3371324 Contrast with the linear SEM approach 3381325 Contrast with the multivariate IV approach 339

133 RMPW extensions to consecutive mediators 3401331 Treatment effect decomposition 341

13311 Natural direct effect of the treatment on the outcome 34213312 Natural indirect effect mediated by M1 only 34213313 Natural indirect effect mediated by M2 only 34213314 Natural indirect effect mediated by an M1-by-M2

interaction 34313315 Treatment-by-mediator interactions 343

1332 Identification assumptions 3451333 RMPW procedure 347

13331 Estimating E[Y(1M1(0)M2(0M1(0)))] 34713332 Estimating E[Y(1M1(1)M2(0M1(0)))] 34913333 Estimating E[Y(1M1(0)M2(1M1(1)))] 34913334 Estimating E[Y(0M1(1)M2(0M1(0)))] and

E[Y(0M1(0)M2(1M1(1)))] 351

xii CONTENTS

1334 Contrast with the linear SEM approach 3531335 Contrast with the sensitivity-based estimation of bounds for

causal effects 354134 Discussion 355Appendix 13A Derivation of RMPW for estimating population average

counterfactual outcomes of two concurrentmediators 355

Appendix 13B Derivation of RMPW for estimating population averagecounterfactual outcomes of consecutivemediators 358

Part IV Spill-over 363

14 Spill-over of treatment effects concepts and methods 365141 Spill-over A nuisance a trifle or a focus 365142 Stable versus unstable potential outcome values An example

from agriculture 367143 Consequences for causal inference when spill-over is overlooked 369144 Modified framework of causal inference 371

1441 Treatment settings 3711442 Simplified characterization of treatment settings 3731443 Causal effects of individual treatment assignment and of peer

treatment assignment 375145 Identification Challenges and solutions 376

1451 Hypothetical experiments for identifying average treatmenteffects in the presence of social interactions 376

1452 Hypothetical experiments for identifying the impact ofsocial interactions 380

1453 Application to an evaluation of kindergarten retention 382146 Analytic strategies for experimental and quasiexperimental data 384

1461 Estimation with experimental data 3841462 Propensity score stratification 3851463 MMWS 386

147 Summary 387

15 Mediation through spill-over 391151 Definition of mediated effects through spill-over in a cluster

randomized trial 3931511 Notation 3931512 Treatment effect mediated by a focal individualrsquos

compliance 3941513 Treatment effect mediated by peersrsquo compliance through

spill-over 3941514 Decomposition of the total treatment effect 395

152 Identification and estimation of the spill-over effect in a clusterrandomized design 3951521 Identification in an ideal experiment 395

xiiiCONTENTS

1522 Identification when the mediators are not randomized 3981523 Estimation of mediated effects through spill-over 400

153 Definition of mediated effects through spill-over in a multisite trial 4021531 Notation 4021532 Treatment effect mediated by a focal individualrsquos compliance 4041533 Treatment effect mediated by peersrsquo compliance

through spill-over 4041534 Direct effect of individual treatment assignment on the outcome 4051535 Direct effect of peer treatment assignment on the outcome 4051536 Decomposition of the total treatment effect 405

154 Identification and estimation of spill-over effects in a multisite trial 4061541 Identification in an ideal experiment 4071542 Identification when the mediators are not randomized 4091543 Estimation of mediated effects through spill-over 410

15431 Estimating E[Y(1pM(p)Mminus(p))] andE[Y(000Mminus(0))] 410

15432 Estimating E[Y(1p0Mminus(p))] 41115433 Estimating E[Y(1p0Mminus(0))] 41215434 Estimating E[Y(0p0Mminus(0))] 412

155 Consequences of omitting spill-over effects in causal mediation analyses 4121551 Biased inference in a cluster randomized trial 4131552 Biased inference in a multisite randomized trial 4131553 Biased inference of the local average treatment effect 415

156 Quasiexperimental application 416157 Summary 419Appendix 151 Derivation of the weight for estimating the population

average counterfactual outcome E[Y(1 p 0Mminus( p))] 419Appendix 152 Derivation of bias in the ITT effect due to the omission of

spill-over effects 420

Index 423

xiv CONTENTS

Preface

A scientific mind in training seeks comprehensive descriptions of every interesting phenom-enon In such a mind data are to be pieced together for comparisons contrasts and eventuallyinferences that are required for understanding causes and effects that may have led to thephenomenon of interest Yet without disciplined reasoning a causal inference would slip intothe rut of a ldquocasual inferencerdquo as easily as making a typographical error

As a doctoral student thirsting for rigorous methodological training at the University ofMichigan I was very fortunate to be among the first group of students on the campus attendinga causal inference seminar in 2000 jointly taught by Yu Xie from the Department of Sociol-ogy Susan Murphy from the Department of Statistics and Stephen Raudenbush from theSchool of Education The course was unique in both content and pedagogy The instructorsput together a bibliography drawn from the past and current causal inference literature origi-nated from multiple disciplines In the spirit of ldquoreciprocal teachingrdquo the students were delib-erately organized into small groups each representing a mix of disciplinary backgrounds andtook turns to teach the weekly readings under the guidance of a faculty instructor This inval-uable experience revealed to me that the pursuit of causal inference is a multidisciplinaryendeavor In the rest of my doctoral study I continued to be immersed in an extraordinaryinterdisciplinary community in the form of the Quantitative Methodology Program (QMP)seminars led by the above three instructors who were then joined by Ben Hansen(Statistics) Richard Gonzalez (Psychology) Roderick Little (Biostatistics) Jake Bowers(Political Science) and many others I have tried whenever possible to carry the same spiritinto my own teaching of causal inference courses at the University of Toronto and now at theUniversity of Chicago In these courses communicating understandings of causal problemsacross disciplinary boundaries has been particularly challenging but also stimulating and grat-ifying for myself and for students from various departments and schools

Under the supervision of Stephen Raudenbush I took a first stab at the conceptual frame-work of causal inference in my doctoral dissertation by considering peer spill-over in schoolsettings From that point on I have organized my methodological research to tackle some ofthe major obstacles to causal inferences in policy and program evaluations My workaddresses issues including (i) how to conceptualize and evaluate the causal effects of educa-tional treatments when studentsrsquo responses to alternative treatments depend on various fea-tures of the organizational settings including peer composition (ii) how to adjust forselection bias in evaluating the effects of concurrent and consecutive multivalued treatmentsand (iii) how to conceptualize and analyze causal mediation mechanisms My methodological

research has been driven by the substantive interest in understanding the role of socialinstitutionsmdashschools in particularmdashin shaping human development especially during theages of fast growth (eg childhood adolescence and early adulthood) I have shared meth-odological advances with a broad audience in social sciences through applying the causalinference methods to prominent substantive issues such as grade retention within-class group-ing services for English language learners and welfare-to-work strategies These illustrationsare used extensively in this book

Among social scientists awareness of methodological challenges in causal investigationsis higher than ever before In response to this rising demand causal inference is becoming oneof the most productive scholarly fields in the past decade I have had the benefit of followingthe work and exchanging ideas with some fellow methodologists who are constantly contri-buting new thoughts and making breakthroughs These include Daniel Almirall HowardBloom Tom Cook Michael Elliot Ken Frank Ben Hansen James Heckman JenniferHill Martin Huber Kosuke Imai Booil Jo John R Lockwood David MacKinnon DanielMcCaffrey Luke Miratrix Richard Murnane Derek Neal Lindsey Page Judea PearlStephen Raudenbush Sean Reardon Michael Seltzer Youngyun Shin Betsy SinclairMichael Sobel Peter Steiner Elizabeth Stuart Eric Thetchen Tchetchen Tyler VanderWeeleand Kazuo Yamaguchi I am grateful to Larry Hedges who offered me the opportunity of guestediting the Journal of Research in Educational Effectiveness special issue on the statisticalapproaches to studying mediator effects in education research in 2012 Many of the colleagueswhom I mentioned above contributed to the open debate in the special issue by either propos-ing or critically examining a number of innovative methods for studying causal mechanismsI anticipate that many of them are or will soon be writing their own research monographs oncausal inference if they have not already done so Therefore readers of this book are stronglyrecommended to browse past and future titles by these and other authors for alternative per-spectives and approaches My own understanding of course will continue to evolve as well inthe coming years

One has to be ambitious to stay upfront in substantive areas as well as in methodology Thebest strategy apparently is to learn from substantive experts During different phases of mytraining and professional career I have had the opportunities to work with David K CohenBrian Rowan Deborah Ball Carl Corter Janette Pelletier Takako Nomi Esther Geva DavidFrancis Stephanie Jones Joshua Brown and Heather D Hill Many of my substantive insightswere derived from conversations with these mentors and colleagues I am especially gratefulto Bob Granger former president of the William T Grant Foundation who reassured me thatldquoBy staying a bit broad you will learn a lot and many fields will benefit from your workrdquo

Like many other single-authored books this one is built on the generous contributions ofstudents and colleagues who deserve special acknowledgement Excellent research assistancefrom Jonah Deutsch Joshua Gagne Rachel Garrett Yihua Hong Xu Qin Cheng Yang andBing Yu was instrumental in the development and implementation of the new methods pre-sented in this book Richard Congdon a renowned statistical software programmer broughtrevolutionary ideas to interface designs in software development with the aim of facilitatingusersrsquo decision-making in a series of causal analyses RichardMurnane Stephen Raudenbushand Michael Sobel carefully reviewed and critiqued multiple chapters of the manuscript draftAdditional comments and suggestions on earlier drafts came fromHoward Bloom Ken FrankBen Hansen Jennifer Hill Booil Jo Ben Kelcey David MacKinnon and Fan Yang TereseSchwartzman provided valuable assistance in manuscript preparation The writing of thisbook was supported by a Scholars Award from the William T Grant Foundation and an

xvi PREFACE

Institute of Education Sciences (IES) Statistical and Research Methodology in EducationGrant from the US Department of Education All the errors in the published form are ofcourse the sole responsibility of the author

Project Editors Richard Davis Prachi Sinha Sahay and Liz Wingett and Assistant EditorHeather Kay at Wiley and project Managers P Jayapriya and R Jayavel at SPi Global havebeen wonderful to work with throughout the manuscript preparation and final production ofthe book Debbie Jupe Commissioning Editor at Wiley was remarkably effective in initiatingthe contact with me and in organizing anonymous reviews of the book proposal and of themanuscript These constructive reviews have shaped the book in important ways I cannotagree more with one of the reviewers that ldquocausality cannot be established on a pure statisticalgroundrdquo The book therefore highlights the importance of substantive knowledge and researchdesigns and places a great emphasis on clarifying and evaluating assumptions required foridentifying causal effects in the context of each application Much caution is raised againstpossible misusage of statistical techniques for analyzing causal relationships especiallywhen data are inadequate Yet I maintain that improved statistical procedures along withimproved research designs would greatly enhance our ability in attempt to empiricallyexamine well-articulated causal theories

xviiPREFACE

Part I

OVERVIEW

1

Introduction

According to an ancient Chinese fable a farmer who was eager to help his crops grow wentinto his field and pulled each seedling upward After exhausting himself with the work heannounced to his family that they were going to have a good harvest only to find the nextmorning that the plants had wilted and died Readers with minimal agricultural knowledgemay immediately point out the following the farmerrsquos intervention theory was based on acorrect observation that crops that grow taller tend to produce more yield Yet his hypothesisreflects a false understanding of the cause and the effectmdashthat seedlings pulled to be tallerwould yield as much as seedlings thriving on their own

In their classic Design and Analysis of Experiments Hinkelmann and Kempthorne (1994updated version of Kempthorne 1952) discussed two types of science descriptive science andthe development of theory These two types of science are interrelated in the following senseobservations of an event and other related events often selected and classified for descriptionby scientists naturally lead to one or more explanations that we call ldquotheoretical hypothesesrdquowhich are then screened and falsified by means of further observations experimentation andanalyses (Popper 1963) The experiment of pulling seedlings to be taller was costly but didserve the purpose of advancing this farmerrsquos knowledge of ldquowhat does not workrdquo To developa successful intervention in this case would require a series of empirical tests of explicittheories identifying potential contributors to crop growth This iterative process graduallydeepens our knowledge of the relationships between supposed causes and effectsmdashthatis causalitymdashand may eventually increase the success of agricultural medical and socialinterventions

11 Concepts of moderation mediation and spill-over

Although the story of the ancient farmer is fictitious numerous examples can be found in thereal world in which well-intended interventions fail to produce the intended benefits or inmany cases even lead to unintended consequences ldquoInterventionsrdquo and ldquotreatmentsrdquo usedinterchangeably in this book broadly refer to actions taken by agents or circumstances expe-rienced by an individual or groups of individuals Interventions are regularly seen in educa-tion physical and mental health social services business politics and law enforcement In an

Causality in a Social World Moderation Meditation and Spill-over First Edition Guanglei Hongcopy 2015 John Wiley amp Sons Ltd Published 2015 by John Wiley amp Sons Ltd

education intervention for example teachers are typically the agents who deliver a treatmentto students while the impact of the treatment on student outcomes is of ultimate causalinterest Some educational practices such as ldquoteaching to the testrdquo have been criticized tobe nearly as counterproductive as the attempt of helping seedlings grow by pulling themupward ldquoInterventionsrdquo and ldquotreatmentsrdquo under consideration do not exclude undesiredexperiences such as exposure to poverty abuse crime or bereavement A treatment plannedor unplanned becomes a focus of research if there are theoretical reasons to anticipate itsimpact positive or negative on the well-being of individuals who are embedded in socialsettings including families classrooms schools neighborhoods and workplaces

In social science research in general and in policy and program evaluations in particularquestions concerning whether an intervention works and if so which version of the interven-tion works for whom under what conditions and why are key to the advancement of scien-tific and practical knowledge Although most empirical investigations in the social sciencesconcentrate on the average effect of a treatment for a specific population as opposed to theabsence of such a treatment (ie the control condition) in-depth theoretical reasoning withregard to how the causal effect is generated substantiated by compelling empirical evidenceis crucial for advancing scientific understanding

First when there are multiple versions or different dosages of the treatment or when thereare multiple versions of the control condition a binary divide between ldquothe treatmentrdquo andldquothe controlrdquo may not be as informative as fine-grained comparisons across for exampleldquotreatment version Ardquo ldquotreatment version Brdquo ldquocontrol version Ardquo and ldquocontrol versionBrdquo For example expanding the federally funded Head Start program to poor children isexpected to generate a greater benefit when few early childhood education alternatives areavailable (call it ldquocontrol version Ardquo) than when there is an abundance of alternatives includ-ing state-sponsored preschool programs (call it ldquocontrol version Brdquo)

Second the effect of an intervention will likely vary among individuals or across socialsettings A famous example comes from medical research the well-publicized cardiovascularbenefits of initiating estrogen therapy during menopause were contradicted later by experi-mental findings that the same therapy increased postmenopausal womenrsquos risk for heartattacks The effect of an intervention may also depend on the provision of some other con-current or subsequent interventions Such heterogeneous effects are often characterized asmoderated effects in the literature

Third alternative theories may provide competing explanations for the causal mechan-isms that is the processes through which the intervention produces its effect A theoreticalconstruct characterizing the hypothesized intermediate process is called a mediator of theintervention effect The fictitious farmer never developed an elaborate theory as to whatcaused some seedlings to surpass others in growth Once scientists revealed the causal rela-tionship between access to chemical nutrients in soil and plant growth wide applications ofchemically synthesized fertilizers finally led to a major increase in crop production

Finally it is well known in agricultural experiments that a new type of fertilizer applied toone plot may spill-over to the next plot Because social connections among individuals areprevalent within organizations or through networks an individualrsquos response to the treatmentmay similarly depend on the treatment for other individuals in the same social setting whichmay lead to possible spill-overs of intervention effects among individual human beings

Answering questions with regard to moderation mediation and spill-over poses majorconceptual and analytic challenges To date psychological research often presents well-articulated theories of causal mechanisms relating stimuli to responses Yet researchers often

4 CAUSALITY IN A SOCIAL WORLD

lack rigorous analytic strategies for empirically screening competing theories explaining theobserved effect Sociologists have keen interest in the spill-over of treatment effects transmit-ted through social interactions yet have produced limited evidence quantifying such effectsAs many have pointed out in general terminological ambiguity and conceptual confusionhave been prevalent in the published applied research (Holmbeck 1997 Kraemer et al2002 2008)

A new framework for conceptualizing moderated mediated and spill-over effects hasemerged relatively recently in the statistics and econometrics literature on causal inference(eg Abbring and Heckman 2007 Frangakis and Rubin 2002 Heckman and Vytlacil2007a b Holland 1988 Hong and Raudenbush 2006 Hudgens and Halloran 2008Jo 2008 Pearl 2001 Robins and Greenland 1992 Sobel 2008) The potential for furtherconceptual and methodological development and for broad applications in the field of behav-ioral and social sciences promises to greatly advance the empirical basis of our knowledgeabout causality in the social world

This book clarifies theoretical concepts and introduces innovative statistical strategies forinvestigating the average effects of multivalued treatments moderated treatment effectsmediated treatment effects and spill-over effects in experimental or quasiexperimental dataDefining individual-specific and population average treatment effects in terms of potentialoutcomes the book relates the mathematical forms to the substantive meanings of moderatedmediated and spill-over effects in the context of application examples It also explicates andevaluates identification assumptions and contrasts innovative statistical strategies with con-ventional analytic methods

111 Moderated treatment effects

It is hard to accept the assumption that a treatment would produce the same impact for everyindividual in every possible circumstance Understanding the heterogeneity of treatmenteffects therefore is key to the development of causal theories For example some studiesreported that estrogen therapy improved cardiovascular health among women who initiatedits use during menopause According to a series of other studies however the use of estrogentherapy increased postmenopausal womenrsquos risk for heart attacks (Grodstein et al 19962000 Writing Group for the Womenrsquos Health Initiative Investigators 2002) The sharp con-trast of findings from these studies led to the hypothesis that age of initiation moderates theeffect of estrogen therapy on womenrsquos health (Manson and Bassuk 2007 Rossouw et al2007) Revelations of the moderated causal relationship greatly enrich theoretical understand-ing and in this case directly inform clinical practice

In another example dividing students by ability into small groups for reading instructionhas been controversial for decades Many believed that the practice benefits high-ability stu-dents at the expense of their low-ability peers and exacerbates educational inequality (Grantand Rothenberg 1986 Rowan and Miracle 1983 Trimble and Sinclair 1987) Recentevidence has shown that first of all the effect of ability grouping depends on a number ofconditions including whether enough time is allocated to reading instruction and how wellthe class can be managed Hence instructional time and class management are among theadditional moderators to consider for determining the merits of ability grouping (Hong andHong 2009 Hong et al 2012a b) Moreover further investigations have generated evidencethat contradicts the long-held belief that grouping undermines the interests of low-abilitystudents Quite the contrary researchers have found that grouping is beneficial for students

5INTRODUCTION

with low or medium prior abilitymdashif adequate time is provided for instruction and if the classis well managed On the other hand ability grouping appears to have a minimal impact onhigh-ability studentsrsquo literacy These results were derived from kindergarten data and maynot hold for math instruction and for higher grade levels Replications with different subpo-pulations and in different contexts enable researchers to assess the generalizability of a theory

In a third example one may hypothesize that a student is unlikely to learn algebra well inninth grade without a solid foundation in eighth-grade prealgebra One may also argue that theprogress a student made in learning eighth-grade prealgebra may not be sustained without thefurther enhancement of taking algebra in ninth grade In other words the earlier treatment(prealgebra in eighth grade) and the later treatment (algebra in ninth grade) may reinforceone other each moderates the effect of the other treatment on the final outcome (math achieve-ment at the end of ninth grade) As Hong and Raudenbush (2008) argued the cumulativeeffect of two treatments in a well-aligned sequence may exceed the sum of the benefits oftwo single-year treatments Similarly experiencing two consecutive years of inferior treat-ments such as encountering an incompetent math teacher who dampens a studentrsquos self-efficacy in math learning in eighth grade and then again in ninth grade could do more damagethan the sum of the effect of encountering an incompetent teacher only in eighth grade and theeffect of having such a teacher only in ninth grade

There has been a great amount of conceptual confusion regarding how a moderatorrelates to the treatment and the outcome On one hand many researchers have used the termsldquomoderatorsrdquo and ldquomediatorsrdquo interchangeably without understanding the crucial distinctionbetween the two An overcorrective attempt on the other hand has led to the arbitrary rec-ommendation that a moderator must occur prior to the treatment and be minimally associatedwith the treatment (James and Brett 1984 Kraemer et al 2001 2008) In Chapter 6 weclarify that in essence the causal effect of the treatment on the outcome may depend onthe moderator value A moderator can be a subpopulation identifier a contextual character-istic a concurrent treatment or a preceding or succeeding treatment In the earlier examplesage of estrogen therapy initiation and a studentrsquos prior ability are subpopulation identifiers themanageability of a class which reflects both teacher skills and peer behaviors characterizes thecontext literacy instruction time is a concurrent treatment while prealgebra is a precedingtreatment for algebra A moderator does not have to occur prior to the treatment and doesnot have to be independent of the treatment

Once a moderated causal relationship has been defined in terms of potential outcomes theresearcher then chooses an appropriate experimental design for testing the moderation theoryThe assumptions required for identifying the moderated causal relationships differ across dif-ferent designs and have implications for analyzing experimental and quasiexperimental dataRandomized block designs are suitable for examining individual or contextual characteristicsas potential moderators factorial designs enable one to determine the joint effects of two ormore concurrent treatments and sequential randomized designs are ideal for assessing thecumulative effects of consecutive treatments Multisite randomized trials constitute anadditional type of designs in which the experimental sites are often deliberately sampled torepresent a population of geographical locations or social organizations Site membershipscan be viewed as implicit moderators that summarize a host of features of a local environmentReplications of a treatment over multiple sites allow one to quantify the heterogeneity of treat-ment effects across the sites Chapters 7 and 8 are focused on statistical methods for moder-ation analyses with quasiexperimental data In particular these chapters demonstrate througha number of application examples how a nonparametric marginal mean weighting through

6 CAUSALITY IN A SOCIAL WORLD

stratification (MMWS) method can overcome some important limitations of other existingmethods

Moderation questions are not restricted to treatmentndashoutcome relationships Rather theyare prevalent in investigations of mediation and spill-over This is because heterogeneity oftreatment effects may be explained by the variation in causal mediation mechanisms acrosssubpopulations and contexts It is also because the treatment effect for a focal individualmay depend on whether there is spill-over from other individuals through social interactionsYet similar to the confusion around the concept of moderation among applied researchersthere has been a great amount of misunderstanding with regard to mediation

112 Mediated treatment effects

Questions about mediation are at the core of nearly every scientific theory In its simplestform a theory explains why treatment Z causes outcome Y by hypothesizing an intermediateprocess involving at least one mediatorM that could have been changed by the treatment andcould subsequently have an impact on the outcome A causal mediation analysis then decom-poses the total effect of the treatment on the outcome into two parts the effect transmittedthrough the hypothesized mediator called the indirect effect and the difference betweenthe total effect and the indirect effect called the direct effect The latter represents the treat-ment effect channeled through other unspecified pathways (Alwin and Hauser 1975 Baronand Kenny 1986 Duncan 1966 Shadish and Sweeney 1991) Most researchers in socialsciences have followed the convention of illustrating the direct effect and the indirect effectwith path diagrams and then specifying linear regression models postulated to represent thestructural relationships between the treatment the mediator and the outcome

In Chapter 9 we point out that the approach to defining the indirect effect and the directeffect in terms of path coefficients is often misled by oversimplistic assumptions about thestructural relationships In particular this approach typically overlooks the fact that a treat-ment may generate an impact on the outcome not only by changing the mediator value butalso by changing the mediatorndashoutcome relationship (Judd and Kenny 1981) An examplein Chapter 9 illustrates such a case An experiment randomizes students to either an experi-mental condition which provides them with study materials and encourages them to study fora test or to a control condition which provides neither study materials nor encouragement(Holland 1988 Powers and Swinton 1984) One might hypothesize that encouraging exper-imental group members to study will increase the time they spend studying a focal mediatorin this example which in turn will increase their average test scores One might furtherhypothesize that even without a change in study time providing study materials to experi-mental group members will enable them to study more effectively than they otherwise wouldConsequently the effect of time spent studying might be greater under the experimentalcondition than under the control condition Omitting the treatment-by-mediator interactioneffect will lead to bias in the estimation of the indirect effect and the direct effect

Chapter 11 describes in great detail an evaluation study contrasting a new welfare-to-workprogram emphasizing active participation in the labor force with a traditional program thatguaranteed cash assistance without a mandate to seek employment (Hong Deutsch and Hill2011) Focusing on the psychological well-being of welfare applicants who were singlemothers with preschool-aged children the researchers hypothesized that the treatment wouldincrease employment rate among the welfare recipients and that the treatment-inducedincrease in employment would likely reduce depressive symptoms under the new program

7INTRODUCTION

They also hypothesized that in contrast the same increase in employment was unlikely tohave a psychological impact under the traditional program Hence the treatment-by-mediatorinteraction effect on the outcome was an essential component of the intermediate process

We will show that by defining the indirect effect and the direct effect in terms of potentialoutcomes one can avoid invoking unwarranted assumptions about the unknown structuralrelationships (Holland 1988 Pearl 2001 Robins and Greenland 1992) The potential out-comes framework has further clarified that in a causal mediation theory the treatment and themediator are both conceivably manipulable In the aforementioned example a welfare appli-cant could be assigned at random to either the new program or the traditional one Once havingbeen assigned to one of the programs the individual might or might not be employed due tovarious structural constraints market fluctuations or other random events typically beyondher control

Because the indirect effect and the direct effect are defined in terms of potential outcomesrather than on the basis of any specific regression models it becomes possible to distinguishthe definitions of these causal effects from their identification and estimation The identifica-tion step relates a causal parameter to observable population data For example for individualsassigned at random to an experimental condition their average counterfactual outcome asso-ciated with the control condition is expected to be equal to the average observable outcome ofthose assigned to the control condition Therefore the population average causal effect can beeasily identified in a randomized experiment The estimation step then relates the sample datato the observable population quantities involved in identification while taking into account thedegree of sampling variability

A major challenge in identification is that the mediatorndashoutcome relationship tends tobe confounded by selection even if the treatment is randomized Chapter 9 reviews variousexperimental designs that have been proposed by past researchers for studying causalmediation mechanisms Chapter 10 compares the identification assumptions and the analyticprocedures across a wide range of analytic methods These discussions are followed by anintroduction of the ratio-of-mediator-probability weighting (RMPW) strategy in Chapter 11In comparison with most existing strategies for causal mediation analysis RMPW relies onrelatively fewer identification assumptions and model-based assumptions Chapters 12 and13 will show that this new strategy can be applied broadly with extensions to multilevelexperimental designs and to studies of complex mediation mechanisms involving multiplemediators

113 Spill-over effects of a treatment

It is well known in vaccination research that an unvaccinated person can benefit when mostother people in the local community are vaccinated This is viewed as a spill-over of the treat-ment effect from the vaccinated people to an unvaccinated person Similarly due to socialinteractions among individuals or groups of individuals an intervention received by somemay generate a spill-over impact on others affiliated with the same organization or connectedthrough the same network For example effective policing in one neighborhood may driveoffenders to operate in other neighborhoods In evaluating an innovative community policingprogram researchers found that a neighborhood not assigned to community policing tended tosuffer if the surrounding neighborhoods were assigned to community policing This is aninstance in which a well-intended treatment generates an unintended negative spill-over effectHowever being assigned to community policy was found particularly beneficial when thesurrounding neighborhoods also received the intervention (Verbitsky-Savitz and Raudenbush

8 CAUSALITY IN A SOCIAL WORLD

2012) In another example the impact of a school-based mentoring program targeted atstudents displaying delinquent behaviors may be enhanced for a focal student if his or herat-risk peers are assigned to the mentoring program at the same time

The previous examples challenge the ldquostable unit treatment value assumptionrdquo (SUTVA)that has been invoked in most causal inference studies This assumption states that an indi-vidualrsquos potential response to a treatment depends neither on how the treatment is assignednor on the treatment assignments of other individuals (Rubin 1980 1986) Rubin and othersbelieve that without resorting to SUTVA the causal effect of a treatment becomes hard todefine and that causal inference may become intractable However as Sobel (2006) has illus-trated with the Moving to Opportunity (MTO) experiment that offered housing vouchersenabling low-income families to move to low-poverty neighborhoods there are possible con-sequences of violating SUTVA in estimating the treatment effects on outcomes such as safetyand self-sufficiency Because social interactions among individuals may affect whether onevolunteers to participate in the study whether one moves to a low-poverty neighborhood afterreceiving the housing voucher as well as housing project residentsrsquo subjective perceptions oftheir neighborhoods the program may have a nonzero impact on the potential outcome of theuntreated As a result despite the randomization of treatment assignment the mean differencein the observed outcome between the treated units and the untreated units may be biased forthe average treatment effect Rather the observable quantity is the difference between theeffect of treating some rather than treating none for the treated and the effect of treating somerather than treating none for the untreated the latter being the pure spill-over effect for theuntreated

In making attempts to relax SUTVA researchers have proposed several alternative frame-works that incorporate possible spill-over effects (see Hong and Raudenbush 2013 for areview) Hong (2004) presented a model that involves treatments and treatment settingsA treatment setting for an individual is a local environment constituted by a set of agentsand participants along with their treatment assignments An individualrsquos potential outcomevalue under a given treatment is assumed stable when the treatment setting is fixed the poten-tial outcome may take different values when the treatment setting shifts One may investigatewhether the treatment effect depends on the treatment setting Applying this framework Hongand Raudenbush (2006) examined the effect on a childrsquos academic growth of retaining thechild in kindergarten rather than promoting the child to first grade when a relatively smallproportion of low-achieving peers in the same school are retained as opposed to when a rel-atively large proportion of the peers are retained Hudgens and Halloran (2008) presented arelated framework in which the effect on an individual of the treatment received by this indi-vidual is distinguished from the effect on the individual of the treatment received by others inthe same local community These effects can be identified if communities are assigned atrandom to different treatment assignment strategies (such as retaining a large proportion oflow-achieving students as opposed to retaining a small proportion of such students) and sub-sequently individuals within a community are randomized for treatment Chapter 14 reviewsthese frameworks and discusses identification and estimation strategies

Social contagion may also serve as an important channel through which the effect of anintervention is transmitted For example a school-wide intervention may reduce aggressivebehaviors and thereby improve studentsrsquo psychological well-being by improving the qualityof interpersonal relationships in other classes as well as in onersquos own class This is becausechildren interact not simply with their classmates but also with those from other classes in thehallways or on the playground (VanderWeele et al 2013) In this case the spill-over betweenclasses becomes a part of the mediation mechanism In a study of student mentoring whether

9INTRODUCTION

an individual actually participated in the mentoring program despite the initial treatmentassignment is typically viewed as a primary mediator The proportion of onersquos peers that par-ticipated in the program may act as a second mediator A treatment may also exert its impactpartly through regrouping individuals and thereby changing peer composition (Hong andNomi 2012) Chapter 15 discusses analytic strategies for detecting spill-over as a part ofthe causal mediation mechanism

12 Weighting methods for causal inference

In most behavioral and social science applications major methodological challenges arise dueto the selection of treatments in a quasiexperimental study and the selection of mediator valuesin experimental and quasiexperimental studies Statistical methods most familiar to research-ers in social sciences are often inadequate for causal inferences with regard to multivaluedtreatments moderation mediation and spill-over The book offers a major revision to under-standing of causality in the social world by introducing two complementary weighting stra-tegies both featuring nonparametric approaches to estimating these causal effects The newweighting methods greatly simplify model specifications while enhancing the robustness ofresults by minimizing the reliance on some key assumptions

The propensity score-basedMMWSmethod removes selection bias associated with a largenumber of covariates by equating the pretreatment composition between treatment groups(Hong 2010a 2012 Huang et al 2005) Unlike propensity score matching and stratificationthat are mostly restricted to evaluations of binary treatments the MMWS method is flexiblefor evaluating binary and multivalued treatments by approximating a completely randomizedexperiment In evaluating whether the treatment effects differ across subpopulations definedby individual characteristics or treatment settings researchers may assign weights within eachsubpopulation in order to approximate a randomized block design To investigate whetherone treatment moderates the effect of another concurrent treatment researchers may assignweights to the data to approximate a factorial randomized design The method can also beused to assess whether the effect of an initial treatment is amplified or weakened by a subse-quent treatment or to identify an optimal treatment sequence through approximating a sequen-tial randomized experiment Even though such analyses can similarly be conducted throughinverse-probability-of-treatment weighting (IPTW) that has been increasingly employed inepidemiological research (Hernaacuten Brumback and Robins 2000 Robins Hernaacuten andBrumback 2000) IPTW is known for bias and imprecision in estimation especially whenthe propensity score models are misspecified in their functional forms (Hong 2010a Kangand Schafer 2007 Schafer and Kang 2008 Waernbaum 2012) In contrast the nonparamet-ric MMWS method displays a relatively high level of robustness despite such misspecifica-tions and also gains efficiency as indicated by simulation results (Hong 2010a)

To study causal mediation mechanisms the RMPW method decomposes the total effectof a treatment into an ldquoindirect effectrdquo transmitted through a specific mediator and a ldquodirecteffectrdquo representing unspecified mechanisms In contrast with most existing methods formediation analysis the RMPW-adjusted outcome model is extremely simple and is nonpara-metric in nature It generates estimates of the causal effects along with their sampling errorswhile adjusting for pretreatment covariates that confound the mediatorndashoutcome relationshipsthrough weighting The method applies regardless of the distribution of the outcome the dis-tribution of the mediator or the functional relationship between the outcome and the mediator

10 CAUSALITY IN A SOCIAL WORLD

Page 10: Thumbnail - download.e-bookshelf.de...4.2.4 IPTW for bias removal 84 4.3 MMWS 86 4.3.1 Theoretical rationale 86 ... 8.3.4.2 Growth model in the absence of confounding 200 8.3.4.3 Weighted

965 Moderation-of-process designs 231966 Augmented encouragement designs 232967 Parallel experimental designs and parallel encouragement designs 232968 Crossover experimental designs and crossover encouragement

designs 233969 Summary 234

10 Existing analytic methods for investigating causal mediation mechanisms 238101 Path analysis and SEM 239

1011 Analytic procedure for continuous outcomes 2391012 Identification assumptions 2421013 Analytic procedure for discrete outcomes 245

102 Modified regression approach 2461021 Analytic procedure for continuous outcomes 2461022 Identification assumptions 2471023 Analytic procedure for binary outcomes 248

103 Marginal structural models 2501031 Analytic procedure 2501032 Identification assumptions 252

104 Conditional structural models 2521041 Analytic procedure 2521042 Identification assumptions 253

105 Alternative weighting methods 2541051 Analytic procedure 2541052 Identification assumptions 256

106 Resampling approach 2561061 Analytic procedure 2561062 Identification assumptions 257

107 IV method 2571071 Rationale and analytic procedure 2571072 Identification assumptions 258

108 Principal stratification 2591081 Rationale and analytic procedure 2591082 Identification assumptions 260

109 Sensitivity analysis 2611091 Unadjusted confounding as a product of hypothetical

regression coefficients 2611092 Unadjusted confounding reflected in a hypothetical

correlation coefficient 2621093 Limitations when the selection mechanism differs

by treatment 2641094 Other sensitivity analyses 265

1010 Conclusion 26510101 The essentiality of sequential ignorability 26510102 Treatment-by-mediator interactions 26610103 Homogeneous versus heterogeneous causal effects 26610104 Model-based assumptions 266

x CONTENTS

Appendix 10A Bias in path analysis estimation due to the omission oftreatment-by-mediator interaction 267

11 Investigations of a simple mediation mechanism 273111 Application example national evaluation of welfare-to-work strategies 274

1111 Historical context 2741112 Research questions 2751113 Causal parameters 2751114 NEWWS Riverside data 277

112 RMPW rationale 2771121 RMPW in a sequentially randomized design 278

11211 E[Y(0 M(0))] 27911212 E[Y(1 M(1))] 28011213 E[Y(1 M(0))] 28011214 E[Y(0 M(1))] 28111215 Nonparametric outcome model 282

1122 RMPW in a sequentially randomized block design 2831123 RMPW in a standard randomized experiment 2851124 Identification assumptions 286

113 Parametric RMPW procedure 287114 Nonparametric RMPW procedure 290115 Simulation results 292

1151 Correctly specified propensity score models 2921152 Misspecified propensity score models 2941153 Comparisons with path analysis and IV results 294

116 Discussion 2951161 Advantages of the RMPW strategy 2951162 Limitations of the RMPW strategy 295

Appendix 11A Causal effect estimation through the RMPW procedure 296Appendix 11B Proof of the consistency of RMPW estimation 297

12 RMPW extensions to alternative designs and measurement 301121 RMPW extensions to mediators and outcomes of alternative distributions 301

1211 Extensions to a multicategory mediator 30212111 Parametric RMPW procedure 30212112 Nonparametric RMPW procedure 304

1212 Extensions to a continuous mediator 3041213 Extensions to a binary outcome 306

122 RMPW extensions to alternative research designs 3061221 Extensions to quasiexperimental data 3071222 Extensions to data from cluster randomized trials 308

12221 Application example and research questions 30912222 Estimation of the total effect 31012223 RMPW analysis of causal mediation mechanisms 31012224 Identification assumptions 31212225 Contrast with multilevel path analysis and SEM 31212226 Contrast with multilevel prediction models 313

xiCONTENTS

1223 Extensions to data from multisite randomized trials 31312231 Research questions and causal parameters 31412232 Estimation of the total effect and its between-site variation 31512233 RMPW analysis of causal mediation mechanisms 31612234 Identification assumptions 31912235 Contrast with multilevel path analysis and SEM 319

123 Alternative decomposition of the treatment effect 321

13 RMPW extensions to studies of complex mediation mechanisms 325131 RMPW extensions to moderated mediation 325

1311 RMPW analytic procedure for estimating and testingmoderated mediation 326

1312 Path analysisSEM approach to analyzing moderatedmediation 327

1313 Principal stratification and moderated mediation 328132 RMPW extensions to concurrent mediators 328

1321 Treatment effect decomposition 32913211 Treatment effect decomposition without

between-mediator interaction 32913212 Treatment effect decomposition with

between-mediator interaction 3311322 Identification assumptions 3331323 RMPW procedure 333

13231 Estimating E[Y(1M1(1)M2(0))] 33513232 Estimating E[Y(1M1(0)M2(0))] 33513233 Causal effect estimation with noninteracting

concurrent mediators 33613234 Estimating E[Y(1M1(0)M2(1))] 33713235 Causal effect estimation with interacting concurrent

mediators 3371324 Contrast with the linear SEM approach 3381325 Contrast with the multivariate IV approach 339

133 RMPW extensions to consecutive mediators 3401331 Treatment effect decomposition 341

13311 Natural direct effect of the treatment on the outcome 34213312 Natural indirect effect mediated by M1 only 34213313 Natural indirect effect mediated by M2 only 34213314 Natural indirect effect mediated by an M1-by-M2

interaction 34313315 Treatment-by-mediator interactions 343

1332 Identification assumptions 3451333 RMPW procedure 347

13331 Estimating E[Y(1M1(0)M2(0M1(0)))] 34713332 Estimating E[Y(1M1(1)M2(0M1(0)))] 34913333 Estimating E[Y(1M1(0)M2(1M1(1)))] 34913334 Estimating E[Y(0M1(1)M2(0M1(0)))] and

E[Y(0M1(0)M2(1M1(1)))] 351

xii CONTENTS

1334 Contrast with the linear SEM approach 3531335 Contrast with the sensitivity-based estimation of bounds for

causal effects 354134 Discussion 355Appendix 13A Derivation of RMPW for estimating population average

counterfactual outcomes of two concurrentmediators 355

Appendix 13B Derivation of RMPW for estimating population averagecounterfactual outcomes of consecutivemediators 358

Part IV Spill-over 363

14 Spill-over of treatment effects concepts and methods 365141 Spill-over A nuisance a trifle or a focus 365142 Stable versus unstable potential outcome values An example

from agriculture 367143 Consequences for causal inference when spill-over is overlooked 369144 Modified framework of causal inference 371

1441 Treatment settings 3711442 Simplified characterization of treatment settings 3731443 Causal effects of individual treatment assignment and of peer

treatment assignment 375145 Identification Challenges and solutions 376

1451 Hypothetical experiments for identifying average treatmenteffects in the presence of social interactions 376

1452 Hypothetical experiments for identifying the impact ofsocial interactions 380

1453 Application to an evaluation of kindergarten retention 382146 Analytic strategies for experimental and quasiexperimental data 384

1461 Estimation with experimental data 3841462 Propensity score stratification 3851463 MMWS 386

147 Summary 387

15 Mediation through spill-over 391151 Definition of mediated effects through spill-over in a cluster

randomized trial 3931511 Notation 3931512 Treatment effect mediated by a focal individualrsquos

compliance 3941513 Treatment effect mediated by peersrsquo compliance through

spill-over 3941514 Decomposition of the total treatment effect 395

152 Identification and estimation of the spill-over effect in a clusterrandomized design 3951521 Identification in an ideal experiment 395

xiiiCONTENTS

1522 Identification when the mediators are not randomized 3981523 Estimation of mediated effects through spill-over 400

153 Definition of mediated effects through spill-over in a multisite trial 4021531 Notation 4021532 Treatment effect mediated by a focal individualrsquos compliance 4041533 Treatment effect mediated by peersrsquo compliance

through spill-over 4041534 Direct effect of individual treatment assignment on the outcome 4051535 Direct effect of peer treatment assignment on the outcome 4051536 Decomposition of the total treatment effect 405

154 Identification and estimation of spill-over effects in a multisite trial 4061541 Identification in an ideal experiment 4071542 Identification when the mediators are not randomized 4091543 Estimation of mediated effects through spill-over 410

15431 Estimating E[Y(1pM(p)Mminus(p))] andE[Y(000Mminus(0))] 410

15432 Estimating E[Y(1p0Mminus(p))] 41115433 Estimating E[Y(1p0Mminus(0))] 41215434 Estimating E[Y(0p0Mminus(0))] 412

155 Consequences of omitting spill-over effects in causal mediation analyses 4121551 Biased inference in a cluster randomized trial 4131552 Biased inference in a multisite randomized trial 4131553 Biased inference of the local average treatment effect 415

156 Quasiexperimental application 416157 Summary 419Appendix 151 Derivation of the weight for estimating the population

average counterfactual outcome E[Y(1 p 0Mminus( p))] 419Appendix 152 Derivation of bias in the ITT effect due to the omission of

spill-over effects 420

Index 423

xiv CONTENTS

Preface

A scientific mind in training seeks comprehensive descriptions of every interesting phenom-enon In such a mind data are to be pieced together for comparisons contrasts and eventuallyinferences that are required for understanding causes and effects that may have led to thephenomenon of interest Yet without disciplined reasoning a causal inference would slip intothe rut of a ldquocasual inferencerdquo as easily as making a typographical error

As a doctoral student thirsting for rigorous methodological training at the University ofMichigan I was very fortunate to be among the first group of students on the campus attendinga causal inference seminar in 2000 jointly taught by Yu Xie from the Department of Sociol-ogy Susan Murphy from the Department of Statistics and Stephen Raudenbush from theSchool of Education The course was unique in both content and pedagogy The instructorsput together a bibliography drawn from the past and current causal inference literature origi-nated from multiple disciplines In the spirit of ldquoreciprocal teachingrdquo the students were delib-erately organized into small groups each representing a mix of disciplinary backgrounds andtook turns to teach the weekly readings under the guidance of a faculty instructor This inval-uable experience revealed to me that the pursuit of causal inference is a multidisciplinaryendeavor In the rest of my doctoral study I continued to be immersed in an extraordinaryinterdisciplinary community in the form of the Quantitative Methodology Program (QMP)seminars led by the above three instructors who were then joined by Ben Hansen(Statistics) Richard Gonzalez (Psychology) Roderick Little (Biostatistics) Jake Bowers(Political Science) and many others I have tried whenever possible to carry the same spiritinto my own teaching of causal inference courses at the University of Toronto and now at theUniversity of Chicago In these courses communicating understandings of causal problemsacross disciplinary boundaries has been particularly challenging but also stimulating and grat-ifying for myself and for students from various departments and schools

Under the supervision of Stephen Raudenbush I took a first stab at the conceptual frame-work of causal inference in my doctoral dissertation by considering peer spill-over in schoolsettings From that point on I have organized my methodological research to tackle some ofthe major obstacles to causal inferences in policy and program evaluations My workaddresses issues including (i) how to conceptualize and evaluate the causal effects of educa-tional treatments when studentsrsquo responses to alternative treatments depend on various fea-tures of the organizational settings including peer composition (ii) how to adjust forselection bias in evaluating the effects of concurrent and consecutive multivalued treatmentsand (iii) how to conceptualize and analyze causal mediation mechanisms My methodological

research has been driven by the substantive interest in understanding the role of socialinstitutionsmdashschools in particularmdashin shaping human development especially during theages of fast growth (eg childhood adolescence and early adulthood) I have shared meth-odological advances with a broad audience in social sciences through applying the causalinference methods to prominent substantive issues such as grade retention within-class group-ing services for English language learners and welfare-to-work strategies These illustrationsare used extensively in this book

Among social scientists awareness of methodological challenges in causal investigationsis higher than ever before In response to this rising demand causal inference is becoming oneof the most productive scholarly fields in the past decade I have had the benefit of followingthe work and exchanging ideas with some fellow methodologists who are constantly contri-buting new thoughts and making breakthroughs These include Daniel Almirall HowardBloom Tom Cook Michael Elliot Ken Frank Ben Hansen James Heckman JenniferHill Martin Huber Kosuke Imai Booil Jo John R Lockwood David MacKinnon DanielMcCaffrey Luke Miratrix Richard Murnane Derek Neal Lindsey Page Judea PearlStephen Raudenbush Sean Reardon Michael Seltzer Youngyun Shin Betsy SinclairMichael Sobel Peter Steiner Elizabeth Stuart Eric Thetchen Tchetchen Tyler VanderWeeleand Kazuo Yamaguchi I am grateful to Larry Hedges who offered me the opportunity of guestediting the Journal of Research in Educational Effectiveness special issue on the statisticalapproaches to studying mediator effects in education research in 2012 Many of the colleagueswhom I mentioned above contributed to the open debate in the special issue by either propos-ing or critically examining a number of innovative methods for studying causal mechanismsI anticipate that many of them are or will soon be writing their own research monographs oncausal inference if they have not already done so Therefore readers of this book are stronglyrecommended to browse past and future titles by these and other authors for alternative per-spectives and approaches My own understanding of course will continue to evolve as well inthe coming years

One has to be ambitious to stay upfront in substantive areas as well as in methodology Thebest strategy apparently is to learn from substantive experts During different phases of mytraining and professional career I have had the opportunities to work with David K CohenBrian Rowan Deborah Ball Carl Corter Janette Pelletier Takako Nomi Esther Geva DavidFrancis Stephanie Jones Joshua Brown and Heather D Hill Many of my substantive insightswere derived from conversations with these mentors and colleagues I am especially gratefulto Bob Granger former president of the William T Grant Foundation who reassured me thatldquoBy staying a bit broad you will learn a lot and many fields will benefit from your workrdquo

Like many other single-authored books this one is built on the generous contributions ofstudents and colleagues who deserve special acknowledgement Excellent research assistancefrom Jonah Deutsch Joshua Gagne Rachel Garrett Yihua Hong Xu Qin Cheng Yang andBing Yu was instrumental in the development and implementation of the new methods pre-sented in this book Richard Congdon a renowned statistical software programmer broughtrevolutionary ideas to interface designs in software development with the aim of facilitatingusersrsquo decision-making in a series of causal analyses RichardMurnane Stephen Raudenbushand Michael Sobel carefully reviewed and critiqued multiple chapters of the manuscript draftAdditional comments and suggestions on earlier drafts came fromHoward Bloom Ken FrankBen Hansen Jennifer Hill Booil Jo Ben Kelcey David MacKinnon and Fan Yang TereseSchwartzman provided valuable assistance in manuscript preparation The writing of thisbook was supported by a Scholars Award from the William T Grant Foundation and an

xvi PREFACE

Institute of Education Sciences (IES) Statistical and Research Methodology in EducationGrant from the US Department of Education All the errors in the published form are ofcourse the sole responsibility of the author

Project Editors Richard Davis Prachi Sinha Sahay and Liz Wingett and Assistant EditorHeather Kay at Wiley and project Managers P Jayapriya and R Jayavel at SPi Global havebeen wonderful to work with throughout the manuscript preparation and final production ofthe book Debbie Jupe Commissioning Editor at Wiley was remarkably effective in initiatingthe contact with me and in organizing anonymous reviews of the book proposal and of themanuscript These constructive reviews have shaped the book in important ways I cannotagree more with one of the reviewers that ldquocausality cannot be established on a pure statisticalgroundrdquo The book therefore highlights the importance of substantive knowledge and researchdesigns and places a great emphasis on clarifying and evaluating assumptions required foridentifying causal effects in the context of each application Much caution is raised againstpossible misusage of statistical techniques for analyzing causal relationships especiallywhen data are inadequate Yet I maintain that improved statistical procedures along withimproved research designs would greatly enhance our ability in attempt to empiricallyexamine well-articulated causal theories

xviiPREFACE

Part I

OVERVIEW

1

Introduction

According to an ancient Chinese fable a farmer who was eager to help his crops grow wentinto his field and pulled each seedling upward After exhausting himself with the work heannounced to his family that they were going to have a good harvest only to find the nextmorning that the plants had wilted and died Readers with minimal agricultural knowledgemay immediately point out the following the farmerrsquos intervention theory was based on acorrect observation that crops that grow taller tend to produce more yield Yet his hypothesisreflects a false understanding of the cause and the effectmdashthat seedlings pulled to be tallerwould yield as much as seedlings thriving on their own

In their classic Design and Analysis of Experiments Hinkelmann and Kempthorne (1994updated version of Kempthorne 1952) discussed two types of science descriptive science andthe development of theory These two types of science are interrelated in the following senseobservations of an event and other related events often selected and classified for descriptionby scientists naturally lead to one or more explanations that we call ldquotheoretical hypothesesrdquowhich are then screened and falsified by means of further observations experimentation andanalyses (Popper 1963) The experiment of pulling seedlings to be taller was costly but didserve the purpose of advancing this farmerrsquos knowledge of ldquowhat does not workrdquo To developa successful intervention in this case would require a series of empirical tests of explicittheories identifying potential contributors to crop growth This iterative process graduallydeepens our knowledge of the relationships between supposed causes and effectsmdashthatis causalitymdashand may eventually increase the success of agricultural medical and socialinterventions

11 Concepts of moderation mediation and spill-over

Although the story of the ancient farmer is fictitious numerous examples can be found in thereal world in which well-intended interventions fail to produce the intended benefits or inmany cases even lead to unintended consequences ldquoInterventionsrdquo and ldquotreatmentsrdquo usedinterchangeably in this book broadly refer to actions taken by agents or circumstances expe-rienced by an individual or groups of individuals Interventions are regularly seen in educa-tion physical and mental health social services business politics and law enforcement In an

Causality in a Social World Moderation Meditation and Spill-over First Edition Guanglei Hongcopy 2015 John Wiley amp Sons Ltd Published 2015 by John Wiley amp Sons Ltd

education intervention for example teachers are typically the agents who deliver a treatmentto students while the impact of the treatment on student outcomes is of ultimate causalinterest Some educational practices such as ldquoteaching to the testrdquo have been criticized tobe nearly as counterproductive as the attempt of helping seedlings grow by pulling themupward ldquoInterventionsrdquo and ldquotreatmentsrdquo under consideration do not exclude undesiredexperiences such as exposure to poverty abuse crime or bereavement A treatment plannedor unplanned becomes a focus of research if there are theoretical reasons to anticipate itsimpact positive or negative on the well-being of individuals who are embedded in socialsettings including families classrooms schools neighborhoods and workplaces

In social science research in general and in policy and program evaluations in particularquestions concerning whether an intervention works and if so which version of the interven-tion works for whom under what conditions and why are key to the advancement of scien-tific and practical knowledge Although most empirical investigations in the social sciencesconcentrate on the average effect of a treatment for a specific population as opposed to theabsence of such a treatment (ie the control condition) in-depth theoretical reasoning withregard to how the causal effect is generated substantiated by compelling empirical evidenceis crucial for advancing scientific understanding

First when there are multiple versions or different dosages of the treatment or when thereare multiple versions of the control condition a binary divide between ldquothe treatmentrdquo andldquothe controlrdquo may not be as informative as fine-grained comparisons across for exampleldquotreatment version Ardquo ldquotreatment version Brdquo ldquocontrol version Ardquo and ldquocontrol versionBrdquo For example expanding the federally funded Head Start program to poor children isexpected to generate a greater benefit when few early childhood education alternatives areavailable (call it ldquocontrol version Ardquo) than when there is an abundance of alternatives includ-ing state-sponsored preschool programs (call it ldquocontrol version Brdquo)

Second the effect of an intervention will likely vary among individuals or across socialsettings A famous example comes from medical research the well-publicized cardiovascularbenefits of initiating estrogen therapy during menopause were contradicted later by experi-mental findings that the same therapy increased postmenopausal womenrsquos risk for heartattacks The effect of an intervention may also depend on the provision of some other con-current or subsequent interventions Such heterogeneous effects are often characterized asmoderated effects in the literature

Third alternative theories may provide competing explanations for the causal mechan-isms that is the processes through which the intervention produces its effect A theoreticalconstruct characterizing the hypothesized intermediate process is called a mediator of theintervention effect The fictitious farmer never developed an elaborate theory as to whatcaused some seedlings to surpass others in growth Once scientists revealed the causal rela-tionship between access to chemical nutrients in soil and plant growth wide applications ofchemically synthesized fertilizers finally led to a major increase in crop production

Finally it is well known in agricultural experiments that a new type of fertilizer applied toone plot may spill-over to the next plot Because social connections among individuals areprevalent within organizations or through networks an individualrsquos response to the treatmentmay similarly depend on the treatment for other individuals in the same social setting whichmay lead to possible spill-overs of intervention effects among individual human beings

Answering questions with regard to moderation mediation and spill-over poses majorconceptual and analytic challenges To date psychological research often presents well-articulated theories of causal mechanisms relating stimuli to responses Yet researchers often

4 CAUSALITY IN A SOCIAL WORLD

lack rigorous analytic strategies for empirically screening competing theories explaining theobserved effect Sociologists have keen interest in the spill-over of treatment effects transmit-ted through social interactions yet have produced limited evidence quantifying such effectsAs many have pointed out in general terminological ambiguity and conceptual confusionhave been prevalent in the published applied research (Holmbeck 1997 Kraemer et al2002 2008)

A new framework for conceptualizing moderated mediated and spill-over effects hasemerged relatively recently in the statistics and econometrics literature on causal inference(eg Abbring and Heckman 2007 Frangakis and Rubin 2002 Heckman and Vytlacil2007a b Holland 1988 Hong and Raudenbush 2006 Hudgens and Halloran 2008Jo 2008 Pearl 2001 Robins and Greenland 1992 Sobel 2008) The potential for furtherconceptual and methodological development and for broad applications in the field of behav-ioral and social sciences promises to greatly advance the empirical basis of our knowledgeabout causality in the social world

This book clarifies theoretical concepts and introduces innovative statistical strategies forinvestigating the average effects of multivalued treatments moderated treatment effectsmediated treatment effects and spill-over effects in experimental or quasiexperimental dataDefining individual-specific and population average treatment effects in terms of potentialoutcomes the book relates the mathematical forms to the substantive meanings of moderatedmediated and spill-over effects in the context of application examples It also explicates andevaluates identification assumptions and contrasts innovative statistical strategies with con-ventional analytic methods

111 Moderated treatment effects

It is hard to accept the assumption that a treatment would produce the same impact for everyindividual in every possible circumstance Understanding the heterogeneity of treatmenteffects therefore is key to the development of causal theories For example some studiesreported that estrogen therapy improved cardiovascular health among women who initiatedits use during menopause According to a series of other studies however the use of estrogentherapy increased postmenopausal womenrsquos risk for heart attacks (Grodstein et al 19962000 Writing Group for the Womenrsquos Health Initiative Investigators 2002) The sharp con-trast of findings from these studies led to the hypothesis that age of initiation moderates theeffect of estrogen therapy on womenrsquos health (Manson and Bassuk 2007 Rossouw et al2007) Revelations of the moderated causal relationship greatly enrich theoretical understand-ing and in this case directly inform clinical practice

In another example dividing students by ability into small groups for reading instructionhas been controversial for decades Many believed that the practice benefits high-ability stu-dents at the expense of their low-ability peers and exacerbates educational inequality (Grantand Rothenberg 1986 Rowan and Miracle 1983 Trimble and Sinclair 1987) Recentevidence has shown that first of all the effect of ability grouping depends on a number ofconditions including whether enough time is allocated to reading instruction and how wellthe class can be managed Hence instructional time and class management are among theadditional moderators to consider for determining the merits of ability grouping (Hong andHong 2009 Hong et al 2012a b) Moreover further investigations have generated evidencethat contradicts the long-held belief that grouping undermines the interests of low-abilitystudents Quite the contrary researchers have found that grouping is beneficial for students

5INTRODUCTION

with low or medium prior abilitymdashif adequate time is provided for instruction and if the classis well managed On the other hand ability grouping appears to have a minimal impact onhigh-ability studentsrsquo literacy These results were derived from kindergarten data and maynot hold for math instruction and for higher grade levels Replications with different subpo-pulations and in different contexts enable researchers to assess the generalizability of a theory

In a third example one may hypothesize that a student is unlikely to learn algebra well inninth grade without a solid foundation in eighth-grade prealgebra One may also argue that theprogress a student made in learning eighth-grade prealgebra may not be sustained without thefurther enhancement of taking algebra in ninth grade In other words the earlier treatment(prealgebra in eighth grade) and the later treatment (algebra in ninth grade) may reinforceone other each moderates the effect of the other treatment on the final outcome (math achieve-ment at the end of ninth grade) As Hong and Raudenbush (2008) argued the cumulativeeffect of two treatments in a well-aligned sequence may exceed the sum of the benefits oftwo single-year treatments Similarly experiencing two consecutive years of inferior treat-ments such as encountering an incompetent math teacher who dampens a studentrsquos self-efficacy in math learning in eighth grade and then again in ninth grade could do more damagethan the sum of the effect of encountering an incompetent teacher only in eighth grade and theeffect of having such a teacher only in ninth grade

There has been a great amount of conceptual confusion regarding how a moderatorrelates to the treatment and the outcome On one hand many researchers have used the termsldquomoderatorsrdquo and ldquomediatorsrdquo interchangeably without understanding the crucial distinctionbetween the two An overcorrective attempt on the other hand has led to the arbitrary rec-ommendation that a moderator must occur prior to the treatment and be minimally associatedwith the treatment (James and Brett 1984 Kraemer et al 2001 2008) In Chapter 6 weclarify that in essence the causal effect of the treatment on the outcome may depend onthe moderator value A moderator can be a subpopulation identifier a contextual character-istic a concurrent treatment or a preceding or succeeding treatment In the earlier examplesage of estrogen therapy initiation and a studentrsquos prior ability are subpopulation identifiers themanageability of a class which reflects both teacher skills and peer behaviors characterizes thecontext literacy instruction time is a concurrent treatment while prealgebra is a precedingtreatment for algebra A moderator does not have to occur prior to the treatment and doesnot have to be independent of the treatment

Once a moderated causal relationship has been defined in terms of potential outcomes theresearcher then chooses an appropriate experimental design for testing the moderation theoryThe assumptions required for identifying the moderated causal relationships differ across dif-ferent designs and have implications for analyzing experimental and quasiexperimental dataRandomized block designs are suitable for examining individual or contextual characteristicsas potential moderators factorial designs enable one to determine the joint effects of two ormore concurrent treatments and sequential randomized designs are ideal for assessing thecumulative effects of consecutive treatments Multisite randomized trials constitute anadditional type of designs in which the experimental sites are often deliberately sampled torepresent a population of geographical locations or social organizations Site membershipscan be viewed as implicit moderators that summarize a host of features of a local environmentReplications of a treatment over multiple sites allow one to quantify the heterogeneity of treat-ment effects across the sites Chapters 7 and 8 are focused on statistical methods for moder-ation analyses with quasiexperimental data In particular these chapters demonstrate througha number of application examples how a nonparametric marginal mean weighting through

6 CAUSALITY IN A SOCIAL WORLD

stratification (MMWS) method can overcome some important limitations of other existingmethods

Moderation questions are not restricted to treatmentndashoutcome relationships Rather theyare prevalent in investigations of mediation and spill-over This is because heterogeneity oftreatment effects may be explained by the variation in causal mediation mechanisms acrosssubpopulations and contexts It is also because the treatment effect for a focal individualmay depend on whether there is spill-over from other individuals through social interactionsYet similar to the confusion around the concept of moderation among applied researchersthere has been a great amount of misunderstanding with regard to mediation

112 Mediated treatment effects

Questions about mediation are at the core of nearly every scientific theory In its simplestform a theory explains why treatment Z causes outcome Y by hypothesizing an intermediateprocess involving at least one mediatorM that could have been changed by the treatment andcould subsequently have an impact on the outcome A causal mediation analysis then decom-poses the total effect of the treatment on the outcome into two parts the effect transmittedthrough the hypothesized mediator called the indirect effect and the difference betweenthe total effect and the indirect effect called the direct effect The latter represents the treat-ment effect channeled through other unspecified pathways (Alwin and Hauser 1975 Baronand Kenny 1986 Duncan 1966 Shadish and Sweeney 1991) Most researchers in socialsciences have followed the convention of illustrating the direct effect and the indirect effectwith path diagrams and then specifying linear regression models postulated to represent thestructural relationships between the treatment the mediator and the outcome

In Chapter 9 we point out that the approach to defining the indirect effect and the directeffect in terms of path coefficients is often misled by oversimplistic assumptions about thestructural relationships In particular this approach typically overlooks the fact that a treat-ment may generate an impact on the outcome not only by changing the mediator value butalso by changing the mediatorndashoutcome relationship (Judd and Kenny 1981) An examplein Chapter 9 illustrates such a case An experiment randomizes students to either an experi-mental condition which provides them with study materials and encourages them to study fora test or to a control condition which provides neither study materials nor encouragement(Holland 1988 Powers and Swinton 1984) One might hypothesize that encouraging exper-imental group members to study will increase the time they spend studying a focal mediatorin this example which in turn will increase their average test scores One might furtherhypothesize that even without a change in study time providing study materials to experi-mental group members will enable them to study more effectively than they otherwise wouldConsequently the effect of time spent studying might be greater under the experimentalcondition than under the control condition Omitting the treatment-by-mediator interactioneffect will lead to bias in the estimation of the indirect effect and the direct effect

Chapter 11 describes in great detail an evaluation study contrasting a new welfare-to-workprogram emphasizing active participation in the labor force with a traditional program thatguaranteed cash assistance without a mandate to seek employment (Hong Deutsch and Hill2011) Focusing on the psychological well-being of welfare applicants who were singlemothers with preschool-aged children the researchers hypothesized that the treatment wouldincrease employment rate among the welfare recipients and that the treatment-inducedincrease in employment would likely reduce depressive symptoms under the new program

7INTRODUCTION

They also hypothesized that in contrast the same increase in employment was unlikely tohave a psychological impact under the traditional program Hence the treatment-by-mediatorinteraction effect on the outcome was an essential component of the intermediate process

We will show that by defining the indirect effect and the direct effect in terms of potentialoutcomes one can avoid invoking unwarranted assumptions about the unknown structuralrelationships (Holland 1988 Pearl 2001 Robins and Greenland 1992) The potential out-comes framework has further clarified that in a causal mediation theory the treatment and themediator are both conceivably manipulable In the aforementioned example a welfare appli-cant could be assigned at random to either the new program or the traditional one Once havingbeen assigned to one of the programs the individual might or might not be employed due tovarious structural constraints market fluctuations or other random events typically beyondher control

Because the indirect effect and the direct effect are defined in terms of potential outcomesrather than on the basis of any specific regression models it becomes possible to distinguishthe definitions of these causal effects from their identification and estimation The identifica-tion step relates a causal parameter to observable population data For example for individualsassigned at random to an experimental condition their average counterfactual outcome asso-ciated with the control condition is expected to be equal to the average observable outcome ofthose assigned to the control condition Therefore the population average causal effect can beeasily identified in a randomized experiment The estimation step then relates the sample datato the observable population quantities involved in identification while taking into account thedegree of sampling variability

A major challenge in identification is that the mediatorndashoutcome relationship tends tobe confounded by selection even if the treatment is randomized Chapter 9 reviews variousexperimental designs that have been proposed by past researchers for studying causalmediation mechanisms Chapter 10 compares the identification assumptions and the analyticprocedures across a wide range of analytic methods These discussions are followed by anintroduction of the ratio-of-mediator-probability weighting (RMPW) strategy in Chapter 11In comparison with most existing strategies for causal mediation analysis RMPW relies onrelatively fewer identification assumptions and model-based assumptions Chapters 12 and13 will show that this new strategy can be applied broadly with extensions to multilevelexperimental designs and to studies of complex mediation mechanisms involving multiplemediators

113 Spill-over effects of a treatment

It is well known in vaccination research that an unvaccinated person can benefit when mostother people in the local community are vaccinated This is viewed as a spill-over of the treat-ment effect from the vaccinated people to an unvaccinated person Similarly due to socialinteractions among individuals or groups of individuals an intervention received by somemay generate a spill-over impact on others affiliated with the same organization or connectedthrough the same network For example effective policing in one neighborhood may driveoffenders to operate in other neighborhoods In evaluating an innovative community policingprogram researchers found that a neighborhood not assigned to community policing tended tosuffer if the surrounding neighborhoods were assigned to community policing This is aninstance in which a well-intended treatment generates an unintended negative spill-over effectHowever being assigned to community policy was found particularly beneficial when thesurrounding neighborhoods also received the intervention (Verbitsky-Savitz and Raudenbush

8 CAUSALITY IN A SOCIAL WORLD

2012) In another example the impact of a school-based mentoring program targeted atstudents displaying delinquent behaviors may be enhanced for a focal student if his or herat-risk peers are assigned to the mentoring program at the same time

The previous examples challenge the ldquostable unit treatment value assumptionrdquo (SUTVA)that has been invoked in most causal inference studies This assumption states that an indi-vidualrsquos potential response to a treatment depends neither on how the treatment is assignednor on the treatment assignments of other individuals (Rubin 1980 1986) Rubin and othersbelieve that without resorting to SUTVA the causal effect of a treatment becomes hard todefine and that causal inference may become intractable However as Sobel (2006) has illus-trated with the Moving to Opportunity (MTO) experiment that offered housing vouchersenabling low-income families to move to low-poverty neighborhoods there are possible con-sequences of violating SUTVA in estimating the treatment effects on outcomes such as safetyand self-sufficiency Because social interactions among individuals may affect whether onevolunteers to participate in the study whether one moves to a low-poverty neighborhood afterreceiving the housing voucher as well as housing project residentsrsquo subjective perceptions oftheir neighborhoods the program may have a nonzero impact on the potential outcome of theuntreated As a result despite the randomization of treatment assignment the mean differencein the observed outcome between the treated units and the untreated units may be biased forthe average treatment effect Rather the observable quantity is the difference between theeffect of treating some rather than treating none for the treated and the effect of treating somerather than treating none for the untreated the latter being the pure spill-over effect for theuntreated

In making attempts to relax SUTVA researchers have proposed several alternative frame-works that incorporate possible spill-over effects (see Hong and Raudenbush 2013 for areview) Hong (2004) presented a model that involves treatments and treatment settingsA treatment setting for an individual is a local environment constituted by a set of agentsand participants along with their treatment assignments An individualrsquos potential outcomevalue under a given treatment is assumed stable when the treatment setting is fixed the poten-tial outcome may take different values when the treatment setting shifts One may investigatewhether the treatment effect depends on the treatment setting Applying this framework Hongand Raudenbush (2006) examined the effect on a childrsquos academic growth of retaining thechild in kindergarten rather than promoting the child to first grade when a relatively smallproportion of low-achieving peers in the same school are retained as opposed to when a rel-atively large proportion of the peers are retained Hudgens and Halloran (2008) presented arelated framework in which the effect on an individual of the treatment received by this indi-vidual is distinguished from the effect on the individual of the treatment received by others inthe same local community These effects can be identified if communities are assigned atrandom to different treatment assignment strategies (such as retaining a large proportion oflow-achieving students as opposed to retaining a small proportion of such students) and sub-sequently individuals within a community are randomized for treatment Chapter 14 reviewsthese frameworks and discusses identification and estimation strategies

Social contagion may also serve as an important channel through which the effect of anintervention is transmitted For example a school-wide intervention may reduce aggressivebehaviors and thereby improve studentsrsquo psychological well-being by improving the qualityof interpersonal relationships in other classes as well as in onersquos own class This is becausechildren interact not simply with their classmates but also with those from other classes in thehallways or on the playground (VanderWeele et al 2013) In this case the spill-over betweenclasses becomes a part of the mediation mechanism In a study of student mentoring whether

9INTRODUCTION

an individual actually participated in the mentoring program despite the initial treatmentassignment is typically viewed as a primary mediator The proportion of onersquos peers that par-ticipated in the program may act as a second mediator A treatment may also exert its impactpartly through regrouping individuals and thereby changing peer composition (Hong andNomi 2012) Chapter 15 discusses analytic strategies for detecting spill-over as a part ofthe causal mediation mechanism

12 Weighting methods for causal inference

In most behavioral and social science applications major methodological challenges arise dueto the selection of treatments in a quasiexperimental study and the selection of mediator valuesin experimental and quasiexperimental studies Statistical methods most familiar to research-ers in social sciences are often inadequate for causal inferences with regard to multivaluedtreatments moderation mediation and spill-over The book offers a major revision to under-standing of causality in the social world by introducing two complementary weighting stra-tegies both featuring nonparametric approaches to estimating these causal effects The newweighting methods greatly simplify model specifications while enhancing the robustness ofresults by minimizing the reliance on some key assumptions

The propensity score-basedMMWSmethod removes selection bias associated with a largenumber of covariates by equating the pretreatment composition between treatment groups(Hong 2010a 2012 Huang et al 2005) Unlike propensity score matching and stratificationthat are mostly restricted to evaluations of binary treatments the MMWS method is flexiblefor evaluating binary and multivalued treatments by approximating a completely randomizedexperiment In evaluating whether the treatment effects differ across subpopulations definedby individual characteristics or treatment settings researchers may assign weights within eachsubpopulation in order to approximate a randomized block design To investigate whetherone treatment moderates the effect of another concurrent treatment researchers may assignweights to the data to approximate a factorial randomized design The method can also beused to assess whether the effect of an initial treatment is amplified or weakened by a subse-quent treatment or to identify an optimal treatment sequence through approximating a sequen-tial randomized experiment Even though such analyses can similarly be conducted throughinverse-probability-of-treatment weighting (IPTW) that has been increasingly employed inepidemiological research (Hernaacuten Brumback and Robins 2000 Robins Hernaacuten andBrumback 2000) IPTW is known for bias and imprecision in estimation especially whenthe propensity score models are misspecified in their functional forms (Hong 2010a Kangand Schafer 2007 Schafer and Kang 2008 Waernbaum 2012) In contrast the nonparamet-ric MMWS method displays a relatively high level of robustness despite such misspecifica-tions and also gains efficiency as indicated by simulation results (Hong 2010a)

To study causal mediation mechanisms the RMPW method decomposes the total effectof a treatment into an ldquoindirect effectrdquo transmitted through a specific mediator and a ldquodirecteffectrdquo representing unspecified mechanisms In contrast with most existing methods formediation analysis the RMPW-adjusted outcome model is extremely simple and is nonpara-metric in nature It generates estimates of the causal effects along with their sampling errorswhile adjusting for pretreatment covariates that confound the mediatorndashoutcome relationshipsthrough weighting The method applies regardless of the distribution of the outcome the dis-tribution of the mediator or the functional relationship between the outcome and the mediator

10 CAUSALITY IN A SOCIAL WORLD

Page 11: Thumbnail - download.e-bookshelf.de...4.2.4 IPTW for bias removal 84 4.3 MMWS 86 4.3.1 Theoretical rationale 86 ... 8.3.4.2 Growth model in the absence of confounding 200 8.3.4.3 Weighted

Appendix 10A Bias in path analysis estimation due to the omission oftreatment-by-mediator interaction 267

11 Investigations of a simple mediation mechanism 273111 Application example national evaluation of welfare-to-work strategies 274

1111 Historical context 2741112 Research questions 2751113 Causal parameters 2751114 NEWWS Riverside data 277

112 RMPW rationale 2771121 RMPW in a sequentially randomized design 278

11211 E[Y(0 M(0))] 27911212 E[Y(1 M(1))] 28011213 E[Y(1 M(0))] 28011214 E[Y(0 M(1))] 28111215 Nonparametric outcome model 282

1122 RMPW in a sequentially randomized block design 2831123 RMPW in a standard randomized experiment 2851124 Identification assumptions 286

113 Parametric RMPW procedure 287114 Nonparametric RMPW procedure 290115 Simulation results 292

1151 Correctly specified propensity score models 2921152 Misspecified propensity score models 2941153 Comparisons with path analysis and IV results 294

116 Discussion 2951161 Advantages of the RMPW strategy 2951162 Limitations of the RMPW strategy 295

Appendix 11A Causal effect estimation through the RMPW procedure 296Appendix 11B Proof of the consistency of RMPW estimation 297

12 RMPW extensions to alternative designs and measurement 301121 RMPW extensions to mediators and outcomes of alternative distributions 301

1211 Extensions to a multicategory mediator 30212111 Parametric RMPW procedure 30212112 Nonparametric RMPW procedure 304

1212 Extensions to a continuous mediator 3041213 Extensions to a binary outcome 306

122 RMPW extensions to alternative research designs 3061221 Extensions to quasiexperimental data 3071222 Extensions to data from cluster randomized trials 308

12221 Application example and research questions 30912222 Estimation of the total effect 31012223 RMPW analysis of causal mediation mechanisms 31012224 Identification assumptions 31212225 Contrast with multilevel path analysis and SEM 31212226 Contrast with multilevel prediction models 313

xiCONTENTS

1223 Extensions to data from multisite randomized trials 31312231 Research questions and causal parameters 31412232 Estimation of the total effect and its between-site variation 31512233 RMPW analysis of causal mediation mechanisms 31612234 Identification assumptions 31912235 Contrast with multilevel path analysis and SEM 319

123 Alternative decomposition of the treatment effect 321

13 RMPW extensions to studies of complex mediation mechanisms 325131 RMPW extensions to moderated mediation 325

1311 RMPW analytic procedure for estimating and testingmoderated mediation 326

1312 Path analysisSEM approach to analyzing moderatedmediation 327

1313 Principal stratification and moderated mediation 328132 RMPW extensions to concurrent mediators 328

1321 Treatment effect decomposition 32913211 Treatment effect decomposition without

between-mediator interaction 32913212 Treatment effect decomposition with

between-mediator interaction 3311322 Identification assumptions 3331323 RMPW procedure 333

13231 Estimating E[Y(1M1(1)M2(0))] 33513232 Estimating E[Y(1M1(0)M2(0))] 33513233 Causal effect estimation with noninteracting

concurrent mediators 33613234 Estimating E[Y(1M1(0)M2(1))] 33713235 Causal effect estimation with interacting concurrent

mediators 3371324 Contrast with the linear SEM approach 3381325 Contrast with the multivariate IV approach 339

133 RMPW extensions to consecutive mediators 3401331 Treatment effect decomposition 341

13311 Natural direct effect of the treatment on the outcome 34213312 Natural indirect effect mediated by M1 only 34213313 Natural indirect effect mediated by M2 only 34213314 Natural indirect effect mediated by an M1-by-M2

interaction 34313315 Treatment-by-mediator interactions 343

1332 Identification assumptions 3451333 RMPW procedure 347

13331 Estimating E[Y(1M1(0)M2(0M1(0)))] 34713332 Estimating E[Y(1M1(1)M2(0M1(0)))] 34913333 Estimating E[Y(1M1(0)M2(1M1(1)))] 34913334 Estimating E[Y(0M1(1)M2(0M1(0)))] and

E[Y(0M1(0)M2(1M1(1)))] 351

xii CONTENTS

1334 Contrast with the linear SEM approach 3531335 Contrast with the sensitivity-based estimation of bounds for

causal effects 354134 Discussion 355Appendix 13A Derivation of RMPW for estimating population average

counterfactual outcomes of two concurrentmediators 355

Appendix 13B Derivation of RMPW for estimating population averagecounterfactual outcomes of consecutivemediators 358

Part IV Spill-over 363

14 Spill-over of treatment effects concepts and methods 365141 Spill-over A nuisance a trifle or a focus 365142 Stable versus unstable potential outcome values An example

from agriculture 367143 Consequences for causal inference when spill-over is overlooked 369144 Modified framework of causal inference 371

1441 Treatment settings 3711442 Simplified characterization of treatment settings 3731443 Causal effects of individual treatment assignment and of peer

treatment assignment 375145 Identification Challenges and solutions 376

1451 Hypothetical experiments for identifying average treatmenteffects in the presence of social interactions 376

1452 Hypothetical experiments for identifying the impact ofsocial interactions 380

1453 Application to an evaluation of kindergarten retention 382146 Analytic strategies for experimental and quasiexperimental data 384

1461 Estimation with experimental data 3841462 Propensity score stratification 3851463 MMWS 386

147 Summary 387

15 Mediation through spill-over 391151 Definition of mediated effects through spill-over in a cluster

randomized trial 3931511 Notation 3931512 Treatment effect mediated by a focal individualrsquos

compliance 3941513 Treatment effect mediated by peersrsquo compliance through

spill-over 3941514 Decomposition of the total treatment effect 395

152 Identification and estimation of the spill-over effect in a clusterrandomized design 3951521 Identification in an ideal experiment 395

xiiiCONTENTS

1522 Identification when the mediators are not randomized 3981523 Estimation of mediated effects through spill-over 400

153 Definition of mediated effects through spill-over in a multisite trial 4021531 Notation 4021532 Treatment effect mediated by a focal individualrsquos compliance 4041533 Treatment effect mediated by peersrsquo compliance

through spill-over 4041534 Direct effect of individual treatment assignment on the outcome 4051535 Direct effect of peer treatment assignment on the outcome 4051536 Decomposition of the total treatment effect 405

154 Identification and estimation of spill-over effects in a multisite trial 4061541 Identification in an ideal experiment 4071542 Identification when the mediators are not randomized 4091543 Estimation of mediated effects through spill-over 410

15431 Estimating E[Y(1pM(p)Mminus(p))] andE[Y(000Mminus(0))] 410

15432 Estimating E[Y(1p0Mminus(p))] 41115433 Estimating E[Y(1p0Mminus(0))] 41215434 Estimating E[Y(0p0Mminus(0))] 412

155 Consequences of omitting spill-over effects in causal mediation analyses 4121551 Biased inference in a cluster randomized trial 4131552 Biased inference in a multisite randomized trial 4131553 Biased inference of the local average treatment effect 415

156 Quasiexperimental application 416157 Summary 419Appendix 151 Derivation of the weight for estimating the population

average counterfactual outcome E[Y(1 p 0Mminus( p))] 419Appendix 152 Derivation of bias in the ITT effect due to the omission of

spill-over effects 420

Index 423

xiv CONTENTS

Preface

A scientific mind in training seeks comprehensive descriptions of every interesting phenom-enon In such a mind data are to be pieced together for comparisons contrasts and eventuallyinferences that are required for understanding causes and effects that may have led to thephenomenon of interest Yet without disciplined reasoning a causal inference would slip intothe rut of a ldquocasual inferencerdquo as easily as making a typographical error

As a doctoral student thirsting for rigorous methodological training at the University ofMichigan I was very fortunate to be among the first group of students on the campus attendinga causal inference seminar in 2000 jointly taught by Yu Xie from the Department of Sociol-ogy Susan Murphy from the Department of Statistics and Stephen Raudenbush from theSchool of Education The course was unique in both content and pedagogy The instructorsput together a bibliography drawn from the past and current causal inference literature origi-nated from multiple disciplines In the spirit of ldquoreciprocal teachingrdquo the students were delib-erately organized into small groups each representing a mix of disciplinary backgrounds andtook turns to teach the weekly readings under the guidance of a faculty instructor This inval-uable experience revealed to me that the pursuit of causal inference is a multidisciplinaryendeavor In the rest of my doctoral study I continued to be immersed in an extraordinaryinterdisciplinary community in the form of the Quantitative Methodology Program (QMP)seminars led by the above three instructors who were then joined by Ben Hansen(Statistics) Richard Gonzalez (Psychology) Roderick Little (Biostatistics) Jake Bowers(Political Science) and many others I have tried whenever possible to carry the same spiritinto my own teaching of causal inference courses at the University of Toronto and now at theUniversity of Chicago In these courses communicating understandings of causal problemsacross disciplinary boundaries has been particularly challenging but also stimulating and grat-ifying for myself and for students from various departments and schools

Under the supervision of Stephen Raudenbush I took a first stab at the conceptual frame-work of causal inference in my doctoral dissertation by considering peer spill-over in schoolsettings From that point on I have organized my methodological research to tackle some ofthe major obstacles to causal inferences in policy and program evaluations My workaddresses issues including (i) how to conceptualize and evaluate the causal effects of educa-tional treatments when studentsrsquo responses to alternative treatments depend on various fea-tures of the organizational settings including peer composition (ii) how to adjust forselection bias in evaluating the effects of concurrent and consecutive multivalued treatmentsand (iii) how to conceptualize and analyze causal mediation mechanisms My methodological

research has been driven by the substantive interest in understanding the role of socialinstitutionsmdashschools in particularmdashin shaping human development especially during theages of fast growth (eg childhood adolescence and early adulthood) I have shared meth-odological advances with a broad audience in social sciences through applying the causalinference methods to prominent substantive issues such as grade retention within-class group-ing services for English language learners and welfare-to-work strategies These illustrationsare used extensively in this book

Among social scientists awareness of methodological challenges in causal investigationsis higher than ever before In response to this rising demand causal inference is becoming oneof the most productive scholarly fields in the past decade I have had the benefit of followingthe work and exchanging ideas with some fellow methodologists who are constantly contri-buting new thoughts and making breakthroughs These include Daniel Almirall HowardBloom Tom Cook Michael Elliot Ken Frank Ben Hansen James Heckman JenniferHill Martin Huber Kosuke Imai Booil Jo John R Lockwood David MacKinnon DanielMcCaffrey Luke Miratrix Richard Murnane Derek Neal Lindsey Page Judea PearlStephen Raudenbush Sean Reardon Michael Seltzer Youngyun Shin Betsy SinclairMichael Sobel Peter Steiner Elizabeth Stuart Eric Thetchen Tchetchen Tyler VanderWeeleand Kazuo Yamaguchi I am grateful to Larry Hedges who offered me the opportunity of guestediting the Journal of Research in Educational Effectiveness special issue on the statisticalapproaches to studying mediator effects in education research in 2012 Many of the colleagueswhom I mentioned above contributed to the open debate in the special issue by either propos-ing or critically examining a number of innovative methods for studying causal mechanismsI anticipate that many of them are or will soon be writing their own research monographs oncausal inference if they have not already done so Therefore readers of this book are stronglyrecommended to browse past and future titles by these and other authors for alternative per-spectives and approaches My own understanding of course will continue to evolve as well inthe coming years

One has to be ambitious to stay upfront in substantive areas as well as in methodology Thebest strategy apparently is to learn from substantive experts During different phases of mytraining and professional career I have had the opportunities to work with David K CohenBrian Rowan Deborah Ball Carl Corter Janette Pelletier Takako Nomi Esther Geva DavidFrancis Stephanie Jones Joshua Brown and Heather D Hill Many of my substantive insightswere derived from conversations with these mentors and colleagues I am especially gratefulto Bob Granger former president of the William T Grant Foundation who reassured me thatldquoBy staying a bit broad you will learn a lot and many fields will benefit from your workrdquo

Like many other single-authored books this one is built on the generous contributions ofstudents and colleagues who deserve special acknowledgement Excellent research assistancefrom Jonah Deutsch Joshua Gagne Rachel Garrett Yihua Hong Xu Qin Cheng Yang andBing Yu was instrumental in the development and implementation of the new methods pre-sented in this book Richard Congdon a renowned statistical software programmer broughtrevolutionary ideas to interface designs in software development with the aim of facilitatingusersrsquo decision-making in a series of causal analyses RichardMurnane Stephen Raudenbushand Michael Sobel carefully reviewed and critiqued multiple chapters of the manuscript draftAdditional comments and suggestions on earlier drafts came fromHoward Bloom Ken FrankBen Hansen Jennifer Hill Booil Jo Ben Kelcey David MacKinnon and Fan Yang TereseSchwartzman provided valuable assistance in manuscript preparation The writing of thisbook was supported by a Scholars Award from the William T Grant Foundation and an

xvi PREFACE

Institute of Education Sciences (IES) Statistical and Research Methodology in EducationGrant from the US Department of Education All the errors in the published form are ofcourse the sole responsibility of the author

Project Editors Richard Davis Prachi Sinha Sahay and Liz Wingett and Assistant EditorHeather Kay at Wiley and project Managers P Jayapriya and R Jayavel at SPi Global havebeen wonderful to work with throughout the manuscript preparation and final production ofthe book Debbie Jupe Commissioning Editor at Wiley was remarkably effective in initiatingthe contact with me and in organizing anonymous reviews of the book proposal and of themanuscript These constructive reviews have shaped the book in important ways I cannotagree more with one of the reviewers that ldquocausality cannot be established on a pure statisticalgroundrdquo The book therefore highlights the importance of substantive knowledge and researchdesigns and places a great emphasis on clarifying and evaluating assumptions required foridentifying causal effects in the context of each application Much caution is raised againstpossible misusage of statistical techniques for analyzing causal relationships especiallywhen data are inadequate Yet I maintain that improved statistical procedures along withimproved research designs would greatly enhance our ability in attempt to empiricallyexamine well-articulated causal theories

xviiPREFACE

Part I

OVERVIEW

1

Introduction

According to an ancient Chinese fable a farmer who was eager to help his crops grow wentinto his field and pulled each seedling upward After exhausting himself with the work heannounced to his family that they were going to have a good harvest only to find the nextmorning that the plants had wilted and died Readers with minimal agricultural knowledgemay immediately point out the following the farmerrsquos intervention theory was based on acorrect observation that crops that grow taller tend to produce more yield Yet his hypothesisreflects a false understanding of the cause and the effectmdashthat seedlings pulled to be tallerwould yield as much as seedlings thriving on their own

In their classic Design and Analysis of Experiments Hinkelmann and Kempthorne (1994updated version of Kempthorne 1952) discussed two types of science descriptive science andthe development of theory These two types of science are interrelated in the following senseobservations of an event and other related events often selected and classified for descriptionby scientists naturally lead to one or more explanations that we call ldquotheoretical hypothesesrdquowhich are then screened and falsified by means of further observations experimentation andanalyses (Popper 1963) The experiment of pulling seedlings to be taller was costly but didserve the purpose of advancing this farmerrsquos knowledge of ldquowhat does not workrdquo To developa successful intervention in this case would require a series of empirical tests of explicittheories identifying potential contributors to crop growth This iterative process graduallydeepens our knowledge of the relationships between supposed causes and effectsmdashthatis causalitymdashand may eventually increase the success of agricultural medical and socialinterventions

11 Concepts of moderation mediation and spill-over

Although the story of the ancient farmer is fictitious numerous examples can be found in thereal world in which well-intended interventions fail to produce the intended benefits or inmany cases even lead to unintended consequences ldquoInterventionsrdquo and ldquotreatmentsrdquo usedinterchangeably in this book broadly refer to actions taken by agents or circumstances expe-rienced by an individual or groups of individuals Interventions are regularly seen in educa-tion physical and mental health social services business politics and law enforcement In an

Causality in a Social World Moderation Meditation and Spill-over First Edition Guanglei Hongcopy 2015 John Wiley amp Sons Ltd Published 2015 by John Wiley amp Sons Ltd

education intervention for example teachers are typically the agents who deliver a treatmentto students while the impact of the treatment on student outcomes is of ultimate causalinterest Some educational practices such as ldquoteaching to the testrdquo have been criticized tobe nearly as counterproductive as the attempt of helping seedlings grow by pulling themupward ldquoInterventionsrdquo and ldquotreatmentsrdquo under consideration do not exclude undesiredexperiences such as exposure to poverty abuse crime or bereavement A treatment plannedor unplanned becomes a focus of research if there are theoretical reasons to anticipate itsimpact positive or negative on the well-being of individuals who are embedded in socialsettings including families classrooms schools neighborhoods and workplaces

In social science research in general and in policy and program evaluations in particularquestions concerning whether an intervention works and if so which version of the interven-tion works for whom under what conditions and why are key to the advancement of scien-tific and practical knowledge Although most empirical investigations in the social sciencesconcentrate on the average effect of a treatment for a specific population as opposed to theabsence of such a treatment (ie the control condition) in-depth theoretical reasoning withregard to how the causal effect is generated substantiated by compelling empirical evidenceis crucial for advancing scientific understanding

First when there are multiple versions or different dosages of the treatment or when thereare multiple versions of the control condition a binary divide between ldquothe treatmentrdquo andldquothe controlrdquo may not be as informative as fine-grained comparisons across for exampleldquotreatment version Ardquo ldquotreatment version Brdquo ldquocontrol version Ardquo and ldquocontrol versionBrdquo For example expanding the federally funded Head Start program to poor children isexpected to generate a greater benefit when few early childhood education alternatives areavailable (call it ldquocontrol version Ardquo) than when there is an abundance of alternatives includ-ing state-sponsored preschool programs (call it ldquocontrol version Brdquo)

Second the effect of an intervention will likely vary among individuals or across socialsettings A famous example comes from medical research the well-publicized cardiovascularbenefits of initiating estrogen therapy during menopause were contradicted later by experi-mental findings that the same therapy increased postmenopausal womenrsquos risk for heartattacks The effect of an intervention may also depend on the provision of some other con-current or subsequent interventions Such heterogeneous effects are often characterized asmoderated effects in the literature

Third alternative theories may provide competing explanations for the causal mechan-isms that is the processes through which the intervention produces its effect A theoreticalconstruct characterizing the hypothesized intermediate process is called a mediator of theintervention effect The fictitious farmer never developed an elaborate theory as to whatcaused some seedlings to surpass others in growth Once scientists revealed the causal rela-tionship between access to chemical nutrients in soil and plant growth wide applications ofchemically synthesized fertilizers finally led to a major increase in crop production

Finally it is well known in agricultural experiments that a new type of fertilizer applied toone plot may spill-over to the next plot Because social connections among individuals areprevalent within organizations or through networks an individualrsquos response to the treatmentmay similarly depend on the treatment for other individuals in the same social setting whichmay lead to possible spill-overs of intervention effects among individual human beings

Answering questions with regard to moderation mediation and spill-over poses majorconceptual and analytic challenges To date psychological research often presents well-articulated theories of causal mechanisms relating stimuli to responses Yet researchers often

4 CAUSALITY IN A SOCIAL WORLD

lack rigorous analytic strategies for empirically screening competing theories explaining theobserved effect Sociologists have keen interest in the spill-over of treatment effects transmit-ted through social interactions yet have produced limited evidence quantifying such effectsAs many have pointed out in general terminological ambiguity and conceptual confusionhave been prevalent in the published applied research (Holmbeck 1997 Kraemer et al2002 2008)

A new framework for conceptualizing moderated mediated and spill-over effects hasemerged relatively recently in the statistics and econometrics literature on causal inference(eg Abbring and Heckman 2007 Frangakis and Rubin 2002 Heckman and Vytlacil2007a b Holland 1988 Hong and Raudenbush 2006 Hudgens and Halloran 2008Jo 2008 Pearl 2001 Robins and Greenland 1992 Sobel 2008) The potential for furtherconceptual and methodological development and for broad applications in the field of behav-ioral and social sciences promises to greatly advance the empirical basis of our knowledgeabout causality in the social world

This book clarifies theoretical concepts and introduces innovative statistical strategies forinvestigating the average effects of multivalued treatments moderated treatment effectsmediated treatment effects and spill-over effects in experimental or quasiexperimental dataDefining individual-specific and population average treatment effects in terms of potentialoutcomes the book relates the mathematical forms to the substantive meanings of moderatedmediated and spill-over effects in the context of application examples It also explicates andevaluates identification assumptions and contrasts innovative statistical strategies with con-ventional analytic methods

111 Moderated treatment effects

It is hard to accept the assumption that a treatment would produce the same impact for everyindividual in every possible circumstance Understanding the heterogeneity of treatmenteffects therefore is key to the development of causal theories For example some studiesreported that estrogen therapy improved cardiovascular health among women who initiatedits use during menopause According to a series of other studies however the use of estrogentherapy increased postmenopausal womenrsquos risk for heart attacks (Grodstein et al 19962000 Writing Group for the Womenrsquos Health Initiative Investigators 2002) The sharp con-trast of findings from these studies led to the hypothesis that age of initiation moderates theeffect of estrogen therapy on womenrsquos health (Manson and Bassuk 2007 Rossouw et al2007) Revelations of the moderated causal relationship greatly enrich theoretical understand-ing and in this case directly inform clinical practice

In another example dividing students by ability into small groups for reading instructionhas been controversial for decades Many believed that the practice benefits high-ability stu-dents at the expense of their low-ability peers and exacerbates educational inequality (Grantand Rothenberg 1986 Rowan and Miracle 1983 Trimble and Sinclair 1987) Recentevidence has shown that first of all the effect of ability grouping depends on a number ofconditions including whether enough time is allocated to reading instruction and how wellthe class can be managed Hence instructional time and class management are among theadditional moderators to consider for determining the merits of ability grouping (Hong andHong 2009 Hong et al 2012a b) Moreover further investigations have generated evidencethat contradicts the long-held belief that grouping undermines the interests of low-abilitystudents Quite the contrary researchers have found that grouping is beneficial for students

5INTRODUCTION

with low or medium prior abilitymdashif adequate time is provided for instruction and if the classis well managed On the other hand ability grouping appears to have a minimal impact onhigh-ability studentsrsquo literacy These results were derived from kindergarten data and maynot hold for math instruction and for higher grade levels Replications with different subpo-pulations and in different contexts enable researchers to assess the generalizability of a theory

In a third example one may hypothesize that a student is unlikely to learn algebra well inninth grade without a solid foundation in eighth-grade prealgebra One may also argue that theprogress a student made in learning eighth-grade prealgebra may not be sustained without thefurther enhancement of taking algebra in ninth grade In other words the earlier treatment(prealgebra in eighth grade) and the later treatment (algebra in ninth grade) may reinforceone other each moderates the effect of the other treatment on the final outcome (math achieve-ment at the end of ninth grade) As Hong and Raudenbush (2008) argued the cumulativeeffect of two treatments in a well-aligned sequence may exceed the sum of the benefits oftwo single-year treatments Similarly experiencing two consecutive years of inferior treat-ments such as encountering an incompetent math teacher who dampens a studentrsquos self-efficacy in math learning in eighth grade and then again in ninth grade could do more damagethan the sum of the effect of encountering an incompetent teacher only in eighth grade and theeffect of having such a teacher only in ninth grade

There has been a great amount of conceptual confusion regarding how a moderatorrelates to the treatment and the outcome On one hand many researchers have used the termsldquomoderatorsrdquo and ldquomediatorsrdquo interchangeably without understanding the crucial distinctionbetween the two An overcorrective attempt on the other hand has led to the arbitrary rec-ommendation that a moderator must occur prior to the treatment and be minimally associatedwith the treatment (James and Brett 1984 Kraemer et al 2001 2008) In Chapter 6 weclarify that in essence the causal effect of the treatment on the outcome may depend onthe moderator value A moderator can be a subpopulation identifier a contextual character-istic a concurrent treatment or a preceding or succeeding treatment In the earlier examplesage of estrogen therapy initiation and a studentrsquos prior ability are subpopulation identifiers themanageability of a class which reflects both teacher skills and peer behaviors characterizes thecontext literacy instruction time is a concurrent treatment while prealgebra is a precedingtreatment for algebra A moderator does not have to occur prior to the treatment and doesnot have to be independent of the treatment

Once a moderated causal relationship has been defined in terms of potential outcomes theresearcher then chooses an appropriate experimental design for testing the moderation theoryThe assumptions required for identifying the moderated causal relationships differ across dif-ferent designs and have implications for analyzing experimental and quasiexperimental dataRandomized block designs are suitable for examining individual or contextual characteristicsas potential moderators factorial designs enable one to determine the joint effects of two ormore concurrent treatments and sequential randomized designs are ideal for assessing thecumulative effects of consecutive treatments Multisite randomized trials constitute anadditional type of designs in which the experimental sites are often deliberately sampled torepresent a population of geographical locations or social organizations Site membershipscan be viewed as implicit moderators that summarize a host of features of a local environmentReplications of a treatment over multiple sites allow one to quantify the heterogeneity of treat-ment effects across the sites Chapters 7 and 8 are focused on statistical methods for moder-ation analyses with quasiexperimental data In particular these chapters demonstrate througha number of application examples how a nonparametric marginal mean weighting through

6 CAUSALITY IN A SOCIAL WORLD

stratification (MMWS) method can overcome some important limitations of other existingmethods

Moderation questions are not restricted to treatmentndashoutcome relationships Rather theyare prevalent in investigations of mediation and spill-over This is because heterogeneity oftreatment effects may be explained by the variation in causal mediation mechanisms acrosssubpopulations and contexts It is also because the treatment effect for a focal individualmay depend on whether there is spill-over from other individuals through social interactionsYet similar to the confusion around the concept of moderation among applied researchersthere has been a great amount of misunderstanding with regard to mediation

112 Mediated treatment effects

Questions about mediation are at the core of nearly every scientific theory In its simplestform a theory explains why treatment Z causes outcome Y by hypothesizing an intermediateprocess involving at least one mediatorM that could have been changed by the treatment andcould subsequently have an impact on the outcome A causal mediation analysis then decom-poses the total effect of the treatment on the outcome into two parts the effect transmittedthrough the hypothesized mediator called the indirect effect and the difference betweenthe total effect and the indirect effect called the direct effect The latter represents the treat-ment effect channeled through other unspecified pathways (Alwin and Hauser 1975 Baronand Kenny 1986 Duncan 1966 Shadish and Sweeney 1991) Most researchers in socialsciences have followed the convention of illustrating the direct effect and the indirect effectwith path diagrams and then specifying linear regression models postulated to represent thestructural relationships between the treatment the mediator and the outcome

In Chapter 9 we point out that the approach to defining the indirect effect and the directeffect in terms of path coefficients is often misled by oversimplistic assumptions about thestructural relationships In particular this approach typically overlooks the fact that a treat-ment may generate an impact on the outcome not only by changing the mediator value butalso by changing the mediatorndashoutcome relationship (Judd and Kenny 1981) An examplein Chapter 9 illustrates such a case An experiment randomizes students to either an experi-mental condition which provides them with study materials and encourages them to study fora test or to a control condition which provides neither study materials nor encouragement(Holland 1988 Powers and Swinton 1984) One might hypothesize that encouraging exper-imental group members to study will increase the time they spend studying a focal mediatorin this example which in turn will increase their average test scores One might furtherhypothesize that even without a change in study time providing study materials to experi-mental group members will enable them to study more effectively than they otherwise wouldConsequently the effect of time spent studying might be greater under the experimentalcondition than under the control condition Omitting the treatment-by-mediator interactioneffect will lead to bias in the estimation of the indirect effect and the direct effect

Chapter 11 describes in great detail an evaluation study contrasting a new welfare-to-workprogram emphasizing active participation in the labor force with a traditional program thatguaranteed cash assistance without a mandate to seek employment (Hong Deutsch and Hill2011) Focusing on the psychological well-being of welfare applicants who were singlemothers with preschool-aged children the researchers hypothesized that the treatment wouldincrease employment rate among the welfare recipients and that the treatment-inducedincrease in employment would likely reduce depressive symptoms under the new program

7INTRODUCTION

They also hypothesized that in contrast the same increase in employment was unlikely tohave a psychological impact under the traditional program Hence the treatment-by-mediatorinteraction effect on the outcome was an essential component of the intermediate process

We will show that by defining the indirect effect and the direct effect in terms of potentialoutcomes one can avoid invoking unwarranted assumptions about the unknown structuralrelationships (Holland 1988 Pearl 2001 Robins and Greenland 1992) The potential out-comes framework has further clarified that in a causal mediation theory the treatment and themediator are both conceivably manipulable In the aforementioned example a welfare appli-cant could be assigned at random to either the new program or the traditional one Once havingbeen assigned to one of the programs the individual might or might not be employed due tovarious structural constraints market fluctuations or other random events typically beyondher control

Because the indirect effect and the direct effect are defined in terms of potential outcomesrather than on the basis of any specific regression models it becomes possible to distinguishthe definitions of these causal effects from their identification and estimation The identifica-tion step relates a causal parameter to observable population data For example for individualsassigned at random to an experimental condition their average counterfactual outcome asso-ciated with the control condition is expected to be equal to the average observable outcome ofthose assigned to the control condition Therefore the population average causal effect can beeasily identified in a randomized experiment The estimation step then relates the sample datato the observable population quantities involved in identification while taking into account thedegree of sampling variability

A major challenge in identification is that the mediatorndashoutcome relationship tends tobe confounded by selection even if the treatment is randomized Chapter 9 reviews variousexperimental designs that have been proposed by past researchers for studying causalmediation mechanisms Chapter 10 compares the identification assumptions and the analyticprocedures across a wide range of analytic methods These discussions are followed by anintroduction of the ratio-of-mediator-probability weighting (RMPW) strategy in Chapter 11In comparison with most existing strategies for causal mediation analysis RMPW relies onrelatively fewer identification assumptions and model-based assumptions Chapters 12 and13 will show that this new strategy can be applied broadly with extensions to multilevelexperimental designs and to studies of complex mediation mechanisms involving multiplemediators

113 Spill-over effects of a treatment

It is well known in vaccination research that an unvaccinated person can benefit when mostother people in the local community are vaccinated This is viewed as a spill-over of the treat-ment effect from the vaccinated people to an unvaccinated person Similarly due to socialinteractions among individuals or groups of individuals an intervention received by somemay generate a spill-over impact on others affiliated with the same organization or connectedthrough the same network For example effective policing in one neighborhood may driveoffenders to operate in other neighborhoods In evaluating an innovative community policingprogram researchers found that a neighborhood not assigned to community policing tended tosuffer if the surrounding neighborhoods were assigned to community policing This is aninstance in which a well-intended treatment generates an unintended negative spill-over effectHowever being assigned to community policy was found particularly beneficial when thesurrounding neighborhoods also received the intervention (Verbitsky-Savitz and Raudenbush

8 CAUSALITY IN A SOCIAL WORLD

2012) In another example the impact of a school-based mentoring program targeted atstudents displaying delinquent behaviors may be enhanced for a focal student if his or herat-risk peers are assigned to the mentoring program at the same time

The previous examples challenge the ldquostable unit treatment value assumptionrdquo (SUTVA)that has been invoked in most causal inference studies This assumption states that an indi-vidualrsquos potential response to a treatment depends neither on how the treatment is assignednor on the treatment assignments of other individuals (Rubin 1980 1986) Rubin and othersbelieve that without resorting to SUTVA the causal effect of a treatment becomes hard todefine and that causal inference may become intractable However as Sobel (2006) has illus-trated with the Moving to Opportunity (MTO) experiment that offered housing vouchersenabling low-income families to move to low-poverty neighborhoods there are possible con-sequences of violating SUTVA in estimating the treatment effects on outcomes such as safetyand self-sufficiency Because social interactions among individuals may affect whether onevolunteers to participate in the study whether one moves to a low-poverty neighborhood afterreceiving the housing voucher as well as housing project residentsrsquo subjective perceptions oftheir neighborhoods the program may have a nonzero impact on the potential outcome of theuntreated As a result despite the randomization of treatment assignment the mean differencein the observed outcome between the treated units and the untreated units may be biased forthe average treatment effect Rather the observable quantity is the difference between theeffect of treating some rather than treating none for the treated and the effect of treating somerather than treating none for the untreated the latter being the pure spill-over effect for theuntreated

In making attempts to relax SUTVA researchers have proposed several alternative frame-works that incorporate possible spill-over effects (see Hong and Raudenbush 2013 for areview) Hong (2004) presented a model that involves treatments and treatment settingsA treatment setting for an individual is a local environment constituted by a set of agentsand participants along with their treatment assignments An individualrsquos potential outcomevalue under a given treatment is assumed stable when the treatment setting is fixed the poten-tial outcome may take different values when the treatment setting shifts One may investigatewhether the treatment effect depends on the treatment setting Applying this framework Hongand Raudenbush (2006) examined the effect on a childrsquos academic growth of retaining thechild in kindergarten rather than promoting the child to first grade when a relatively smallproportion of low-achieving peers in the same school are retained as opposed to when a rel-atively large proportion of the peers are retained Hudgens and Halloran (2008) presented arelated framework in which the effect on an individual of the treatment received by this indi-vidual is distinguished from the effect on the individual of the treatment received by others inthe same local community These effects can be identified if communities are assigned atrandom to different treatment assignment strategies (such as retaining a large proportion oflow-achieving students as opposed to retaining a small proportion of such students) and sub-sequently individuals within a community are randomized for treatment Chapter 14 reviewsthese frameworks and discusses identification and estimation strategies

Social contagion may also serve as an important channel through which the effect of anintervention is transmitted For example a school-wide intervention may reduce aggressivebehaviors and thereby improve studentsrsquo psychological well-being by improving the qualityof interpersonal relationships in other classes as well as in onersquos own class This is becausechildren interact not simply with their classmates but also with those from other classes in thehallways or on the playground (VanderWeele et al 2013) In this case the spill-over betweenclasses becomes a part of the mediation mechanism In a study of student mentoring whether

9INTRODUCTION

an individual actually participated in the mentoring program despite the initial treatmentassignment is typically viewed as a primary mediator The proportion of onersquos peers that par-ticipated in the program may act as a second mediator A treatment may also exert its impactpartly through regrouping individuals and thereby changing peer composition (Hong andNomi 2012) Chapter 15 discusses analytic strategies for detecting spill-over as a part ofthe causal mediation mechanism

12 Weighting methods for causal inference

In most behavioral and social science applications major methodological challenges arise dueto the selection of treatments in a quasiexperimental study and the selection of mediator valuesin experimental and quasiexperimental studies Statistical methods most familiar to research-ers in social sciences are often inadequate for causal inferences with regard to multivaluedtreatments moderation mediation and spill-over The book offers a major revision to under-standing of causality in the social world by introducing two complementary weighting stra-tegies both featuring nonparametric approaches to estimating these causal effects The newweighting methods greatly simplify model specifications while enhancing the robustness ofresults by minimizing the reliance on some key assumptions

The propensity score-basedMMWSmethod removes selection bias associated with a largenumber of covariates by equating the pretreatment composition between treatment groups(Hong 2010a 2012 Huang et al 2005) Unlike propensity score matching and stratificationthat are mostly restricted to evaluations of binary treatments the MMWS method is flexiblefor evaluating binary and multivalued treatments by approximating a completely randomizedexperiment In evaluating whether the treatment effects differ across subpopulations definedby individual characteristics or treatment settings researchers may assign weights within eachsubpopulation in order to approximate a randomized block design To investigate whetherone treatment moderates the effect of another concurrent treatment researchers may assignweights to the data to approximate a factorial randomized design The method can also beused to assess whether the effect of an initial treatment is amplified or weakened by a subse-quent treatment or to identify an optimal treatment sequence through approximating a sequen-tial randomized experiment Even though such analyses can similarly be conducted throughinverse-probability-of-treatment weighting (IPTW) that has been increasingly employed inepidemiological research (Hernaacuten Brumback and Robins 2000 Robins Hernaacuten andBrumback 2000) IPTW is known for bias and imprecision in estimation especially whenthe propensity score models are misspecified in their functional forms (Hong 2010a Kangand Schafer 2007 Schafer and Kang 2008 Waernbaum 2012) In contrast the nonparamet-ric MMWS method displays a relatively high level of robustness despite such misspecifica-tions and also gains efficiency as indicated by simulation results (Hong 2010a)

To study causal mediation mechanisms the RMPW method decomposes the total effectof a treatment into an ldquoindirect effectrdquo transmitted through a specific mediator and a ldquodirecteffectrdquo representing unspecified mechanisms In contrast with most existing methods formediation analysis the RMPW-adjusted outcome model is extremely simple and is nonpara-metric in nature It generates estimates of the causal effects along with their sampling errorswhile adjusting for pretreatment covariates that confound the mediatorndashoutcome relationshipsthrough weighting The method applies regardless of the distribution of the outcome the dis-tribution of the mediator or the functional relationship between the outcome and the mediator

10 CAUSALITY IN A SOCIAL WORLD

Page 12: Thumbnail - download.e-bookshelf.de...4.2.4 IPTW for bias removal 84 4.3 MMWS 86 4.3.1 Theoretical rationale 86 ... 8.3.4.2 Growth model in the absence of confounding 200 8.3.4.3 Weighted

1223 Extensions to data from multisite randomized trials 31312231 Research questions and causal parameters 31412232 Estimation of the total effect and its between-site variation 31512233 RMPW analysis of causal mediation mechanisms 31612234 Identification assumptions 31912235 Contrast with multilevel path analysis and SEM 319

123 Alternative decomposition of the treatment effect 321

13 RMPW extensions to studies of complex mediation mechanisms 325131 RMPW extensions to moderated mediation 325

1311 RMPW analytic procedure for estimating and testingmoderated mediation 326

1312 Path analysisSEM approach to analyzing moderatedmediation 327

1313 Principal stratification and moderated mediation 328132 RMPW extensions to concurrent mediators 328

1321 Treatment effect decomposition 32913211 Treatment effect decomposition without

between-mediator interaction 32913212 Treatment effect decomposition with

between-mediator interaction 3311322 Identification assumptions 3331323 RMPW procedure 333

13231 Estimating E[Y(1M1(1)M2(0))] 33513232 Estimating E[Y(1M1(0)M2(0))] 33513233 Causal effect estimation with noninteracting

concurrent mediators 33613234 Estimating E[Y(1M1(0)M2(1))] 33713235 Causal effect estimation with interacting concurrent

mediators 3371324 Contrast with the linear SEM approach 3381325 Contrast with the multivariate IV approach 339

133 RMPW extensions to consecutive mediators 3401331 Treatment effect decomposition 341

13311 Natural direct effect of the treatment on the outcome 34213312 Natural indirect effect mediated by M1 only 34213313 Natural indirect effect mediated by M2 only 34213314 Natural indirect effect mediated by an M1-by-M2

interaction 34313315 Treatment-by-mediator interactions 343

1332 Identification assumptions 3451333 RMPW procedure 347

13331 Estimating E[Y(1M1(0)M2(0M1(0)))] 34713332 Estimating E[Y(1M1(1)M2(0M1(0)))] 34913333 Estimating E[Y(1M1(0)M2(1M1(1)))] 34913334 Estimating E[Y(0M1(1)M2(0M1(0)))] and

E[Y(0M1(0)M2(1M1(1)))] 351

xii CONTENTS

1334 Contrast with the linear SEM approach 3531335 Contrast with the sensitivity-based estimation of bounds for

causal effects 354134 Discussion 355Appendix 13A Derivation of RMPW for estimating population average

counterfactual outcomes of two concurrentmediators 355

Appendix 13B Derivation of RMPW for estimating population averagecounterfactual outcomes of consecutivemediators 358

Part IV Spill-over 363

14 Spill-over of treatment effects concepts and methods 365141 Spill-over A nuisance a trifle or a focus 365142 Stable versus unstable potential outcome values An example

from agriculture 367143 Consequences for causal inference when spill-over is overlooked 369144 Modified framework of causal inference 371

1441 Treatment settings 3711442 Simplified characterization of treatment settings 3731443 Causal effects of individual treatment assignment and of peer

treatment assignment 375145 Identification Challenges and solutions 376

1451 Hypothetical experiments for identifying average treatmenteffects in the presence of social interactions 376

1452 Hypothetical experiments for identifying the impact ofsocial interactions 380

1453 Application to an evaluation of kindergarten retention 382146 Analytic strategies for experimental and quasiexperimental data 384

1461 Estimation with experimental data 3841462 Propensity score stratification 3851463 MMWS 386

147 Summary 387

15 Mediation through spill-over 391151 Definition of mediated effects through spill-over in a cluster

randomized trial 3931511 Notation 3931512 Treatment effect mediated by a focal individualrsquos

compliance 3941513 Treatment effect mediated by peersrsquo compliance through

spill-over 3941514 Decomposition of the total treatment effect 395

152 Identification and estimation of the spill-over effect in a clusterrandomized design 3951521 Identification in an ideal experiment 395

xiiiCONTENTS

1522 Identification when the mediators are not randomized 3981523 Estimation of mediated effects through spill-over 400

153 Definition of mediated effects through spill-over in a multisite trial 4021531 Notation 4021532 Treatment effect mediated by a focal individualrsquos compliance 4041533 Treatment effect mediated by peersrsquo compliance

through spill-over 4041534 Direct effect of individual treatment assignment on the outcome 4051535 Direct effect of peer treatment assignment on the outcome 4051536 Decomposition of the total treatment effect 405

154 Identification and estimation of spill-over effects in a multisite trial 4061541 Identification in an ideal experiment 4071542 Identification when the mediators are not randomized 4091543 Estimation of mediated effects through spill-over 410

15431 Estimating E[Y(1pM(p)Mminus(p))] andE[Y(000Mminus(0))] 410

15432 Estimating E[Y(1p0Mminus(p))] 41115433 Estimating E[Y(1p0Mminus(0))] 41215434 Estimating E[Y(0p0Mminus(0))] 412

155 Consequences of omitting spill-over effects in causal mediation analyses 4121551 Biased inference in a cluster randomized trial 4131552 Biased inference in a multisite randomized trial 4131553 Biased inference of the local average treatment effect 415

156 Quasiexperimental application 416157 Summary 419Appendix 151 Derivation of the weight for estimating the population

average counterfactual outcome E[Y(1 p 0Mminus( p))] 419Appendix 152 Derivation of bias in the ITT effect due to the omission of

spill-over effects 420

Index 423

xiv CONTENTS

Preface

A scientific mind in training seeks comprehensive descriptions of every interesting phenom-enon In such a mind data are to be pieced together for comparisons contrasts and eventuallyinferences that are required for understanding causes and effects that may have led to thephenomenon of interest Yet without disciplined reasoning a causal inference would slip intothe rut of a ldquocasual inferencerdquo as easily as making a typographical error

As a doctoral student thirsting for rigorous methodological training at the University ofMichigan I was very fortunate to be among the first group of students on the campus attendinga causal inference seminar in 2000 jointly taught by Yu Xie from the Department of Sociol-ogy Susan Murphy from the Department of Statistics and Stephen Raudenbush from theSchool of Education The course was unique in both content and pedagogy The instructorsput together a bibliography drawn from the past and current causal inference literature origi-nated from multiple disciplines In the spirit of ldquoreciprocal teachingrdquo the students were delib-erately organized into small groups each representing a mix of disciplinary backgrounds andtook turns to teach the weekly readings under the guidance of a faculty instructor This inval-uable experience revealed to me that the pursuit of causal inference is a multidisciplinaryendeavor In the rest of my doctoral study I continued to be immersed in an extraordinaryinterdisciplinary community in the form of the Quantitative Methodology Program (QMP)seminars led by the above three instructors who were then joined by Ben Hansen(Statistics) Richard Gonzalez (Psychology) Roderick Little (Biostatistics) Jake Bowers(Political Science) and many others I have tried whenever possible to carry the same spiritinto my own teaching of causal inference courses at the University of Toronto and now at theUniversity of Chicago In these courses communicating understandings of causal problemsacross disciplinary boundaries has been particularly challenging but also stimulating and grat-ifying for myself and for students from various departments and schools

Under the supervision of Stephen Raudenbush I took a first stab at the conceptual frame-work of causal inference in my doctoral dissertation by considering peer spill-over in schoolsettings From that point on I have organized my methodological research to tackle some ofthe major obstacles to causal inferences in policy and program evaluations My workaddresses issues including (i) how to conceptualize and evaluate the causal effects of educa-tional treatments when studentsrsquo responses to alternative treatments depend on various fea-tures of the organizational settings including peer composition (ii) how to adjust forselection bias in evaluating the effects of concurrent and consecutive multivalued treatmentsand (iii) how to conceptualize and analyze causal mediation mechanisms My methodological

research has been driven by the substantive interest in understanding the role of socialinstitutionsmdashschools in particularmdashin shaping human development especially during theages of fast growth (eg childhood adolescence and early adulthood) I have shared meth-odological advances with a broad audience in social sciences through applying the causalinference methods to prominent substantive issues such as grade retention within-class group-ing services for English language learners and welfare-to-work strategies These illustrationsare used extensively in this book

Among social scientists awareness of methodological challenges in causal investigationsis higher than ever before In response to this rising demand causal inference is becoming oneof the most productive scholarly fields in the past decade I have had the benefit of followingthe work and exchanging ideas with some fellow methodologists who are constantly contri-buting new thoughts and making breakthroughs These include Daniel Almirall HowardBloom Tom Cook Michael Elliot Ken Frank Ben Hansen James Heckman JenniferHill Martin Huber Kosuke Imai Booil Jo John R Lockwood David MacKinnon DanielMcCaffrey Luke Miratrix Richard Murnane Derek Neal Lindsey Page Judea PearlStephen Raudenbush Sean Reardon Michael Seltzer Youngyun Shin Betsy SinclairMichael Sobel Peter Steiner Elizabeth Stuart Eric Thetchen Tchetchen Tyler VanderWeeleand Kazuo Yamaguchi I am grateful to Larry Hedges who offered me the opportunity of guestediting the Journal of Research in Educational Effectiveness special issue on the statisticalapproaches to studying mediator effects in education research in 2012 Many of the colleagueswhom I mentioned above contributed to the open debate in the special issue by either propos-ing or critically examining a number of innovative methods for studying causal mechanismsI anticipate that many of them are or will soon be writing their own research monographs oncausal inference if they have not already done so Therefore readers of this book are stronglyrecommended to browse past and future titles by these and other authors for alternative per-spectives and approaches My own understanding of course will continue to evolve as well inthe coming years

One has to be ambitious to stay upfront in substantive areas as well as in methodology Thebest strategy apparently is to learn from substantive experts During different phases of mytraining and professional career I have had the opportunities to work with David K CohenBrian Rowan Deborah Ball Carl Corter Janette Pelletier Takako Nomi Esther Geva DavidFrancis Stephanie Jones Joshua Brown and Heather D Hill Many of my substantive insightswere derived from conversations with these mentors and colleagues I am especially gratefulto Bob Granger former president of the William T Grant Foundation who reassured me thatldquoBy staying a bit broad you will learn a lot and many fields will benefit from your workrdquo

Like many other single-authored books this one is built on the generous contributions ofstudents and colleagues who deserve special acknowledgement Excellent research assistancefrom Jonah Deutsch Joshua Gagne Rachel Garrett Yihua Hong Xu Qin Cheng Yang andBing Yu was instrumental in the development and implementation of the new methods pre-sented in this book Richard Congdon a renowned statistical software programmer broughtrevolutionary ideas to interface designs in software development with the aim of facilitatingusersrsquo decision-making in a series of causal analyses RichardMurnane Stephen Raudenbushand Michael Sobel carefully reviewed and critiqued multiple chapters of the manuscript draftAdditional comments and suggestions on earlier drafts came fromHoward Bloom Ken FrankBen Hansen Jennifer Hill Booil Jo Ben Kelcey David MacKinnon and Fan Yang TereseSchwartzman provided valuable assistance in manuscript preparation The writing of thisbook was supported by a Scholars Award from the William T Grant Foundation and an

xvi PREFACE

Institute of Education Sciences (IES) Statistical and Research Methodology in EducationGrant from the US Department of Education All the errors in the published form are ofcourse the sole responsibility of the author

Project Editors Richard Davis Prachi Sinha Sahay and Liz Wingett and Assistant EditorHeather Kay at Wiley and project Managers P Jayapriya and R Jayavel at SPi Global havebeen wonderful to work with throughout the manuscript preparation and final production ofthe book Debbie Jupe Commissioning Editor at Wiley was remarkably effective in initiatingthe contact with me and in organizing anonymous reviews of the book proposal and of themanuscript These constructive reviews have shaped the book in important ways I cannotagree more with one of the reviewers that ldquocausality cannot be established on a pure statisticalgroundrdquo The book therefore highlights the importance of substantive knowledge and researchdesigns and places a great emphasis on clarifying and evaluating assumptions required foridentifying causal effects in the context of each application Much caution is raised againstpossible misusage of statistical techniques for analyzing causal relationships especiallywhen data are inadequate Yet I maintain that improved statistical procedures along withimproved research designs would greatly enhance our ability in attempt to empiricallyexamine well-articulated causal theories

xviiPREFACE

Part I

OVERVIEW

1

Introduction

According to an ancient Chinese fable a farmer who was eager to help his crops grow wentinto his field and pulled each seedling upward After exhausting himself with the work heannounced to his family that they were going to have a good harvest only to find the nextmorning that the plants had wilted and died Readers with minimal agricultural knowledgemay immediately point out the following the farmerrsquos intervention theory was based on acorrect observation that crops that grow taller tend to produce more yield Yet his hypothesisreflects a false understanding of the cause and the effectmdashthat seedlings pulled to be tallerwould yield as much as seedlings thriving on their own

In their classic Design and Analysis of Experiments Hinkelmann and Kempthorne (1994updated version of Kempthorne 1952) discussed two types of science descriptive science andthe development of theory These two types of science are interrelated in the following senseobservations of an event and other related events often selected and classified for descriptionby scientists naturally lead to one or more explanations that we call ldquotheoretical hypothesesrdquowhich are then screened and falsified by means of further observations experimentation andanalyses (Popper 1963) The experiment of pulling seedlings to be taller was costly but didserve the purpose of advancing this farmerrsquos knowledge of ldquowhat does not workrdquo To developa successful intervention in this case would require a series of empirical tests of explicittheories identifying potential contributors to crop growth This iterative process graduallydeepens our knowledge of the relationships between supposed causes and effectsmdashthatis causalitymdashand may eventually increase the success of agricultural medical and socialinterventions

11 Concepts of moderation mediation and spill-over

Although the story of the ancient farmer is fictitious numerous examples can be found in thereal world in which well-intended interventions fail to produce the intended benefits or inmany cases even lead to unintended consequences ldquoInterventionsrdquo and ldquotreatmentsrdquo usedinterchangeably in this book broadly refer to actions taken by agents or circumstances expe-rienced by an individual or groups of individuals Interventions are regularly seen in educa-tion physical and mental health social services business politics and law enforcement In an

Causality in a Social World Moderation Meditation and Spill-over First Edition Guanglei Hongcopy 2015 John Wiley amp Sons Ltd Published 2015 by John Wiley amp Sons Ltd

education intervention for example teachers are typically the agents who deliver a treatmentto students while the impact of the treatment on student outcomes is of ultimate causalinterest Some educational practices such as ldquoteaching to the testrdquo have been criticized tobe nearly as counterproductive as the attempt of helping seedlings grow by pulling themupward ldquoInterventionsrdquo and ldquotreatmentsrdquo under consideration do not exclude undesiredexperiences such as exposure to poverty abuse crime or bereavement A treatment plannedor unplanned becomes a focus of research if there are theoretical reasons to anticipate itsimpact positive or negative on the well-being of individuals who are embedded in socialsettings including families classrooms schools neighborhoods and workplaces

In social science research in general and in policy and program evaluations in particularquestions concerning whether an intervention works and if so which version of the interven-tion works for whom under what conditions and why are key to the advancement of scien-tific and practical knowledge Although most empirical investigations in the social sciencesconcentrate on the average effect of a treatment for a specific population as opposed to theabsence of such a treatment (ie the control condition) in-depth theoretical reasoning withregard to how the causal effect is generated substantiated by compelling empirical evidenceis crucial for advancing scientific understanding

First when there are multiple versions or different dosages of the treatment or when thereare multiple versions of the control condition a binary divide between ldquothe treatmentrdquo andldquothe controlrdquo may not be as informative as fine-grained comparisons across for exampleldquotreatment version Ardquo ldquotreatment version Brdquo ldquocontrol version Ardquo and ldquocontrol versionBrdquo For example expanding the federally funded Head Start program to poor children isexpected to generate a greater benefit when few early childhood education alternatives areavailable (call it ldquocontrol version Ardquo) than when there is an abundance of alternatives includ-ing state-sponsored preschool programs (call it ldquocontrol version Brdquo)

Second the effect of an intervention will likely vary among individuals or across socialsettings A famous example comes from medical research the well-publicized cardiovascularbenefits of initiating estrogen therapy during menopause were contradicted later by experi-mental findings that the same therapy increased postmenopausal womenrsquos risk for heartattacks The effect of an intervention may also depend on the provision of some other con-current or subsequent interventions Such heterogeneous effects are often characterized asmoderated effects in the literature

Third alternative theories may provide competing explanations for the causal mechan-isms that is the processes through which the intervention produces its effect A theoreticalconstruct characterizing the hypothesized intermediate process is called a mediator of theintervention effect The fictitious farmer never developed an elaborate theory as to whatcaused some seedlings to surpass others in growth Once scientists revealed the causal rela-tionship between access to chemical nutrients in soil and plant growth wide applications ofchemically synthesized fertilizers finally led to a major increase in crop production

Finally it is well known in agricultural experiments that a new type of fertilizer applied toone plot may spill-over to the next plot Because social connections among individuals areprevalent within organizations or through networks an individualrsquos response to the treatmentmay similarly depend on the treatment for other individuals in the same social setting whichmay lead to possible spill-overs of intervention effects among individual human beings

Answering questions with regard to moderation mediation and spill-over poses majorconceptual and analytic challenges To date psychological research often presents well-articulated theories of causal mechanisms relating stimuli to responses Yet researchers often

4 CAUSALITY IN A SOCIAL WORLD

lack rigorous analytic strategies for empirically screening competing theories explaining theobserved effect Sociologists have keen interest in the spill-over of treatment effects transmit-ted through social interactions yet have produced limited evidence quantifying such effectsAs many have pointed out in general terminological ambiguity and conceptual confusionhave been prevalent in the published applied research (Holmbeck 1997 Kraemer et al2002 2008)

A new framework for conceptualizing moderated mediated and spill-over effects hasemerged relatively recently in the statistics and econometrics literature on causal inference(eg Abbring and Heckman 2007 Frangakis and Rubin 2002 Heckman and Vytlacil2007a b Holland 1988 Hong and Raudenbush 2006 Hudgens and Halloran 2008Jo 2008 Pearl 2001 Robins and Greenland 1992 Sobel 2008) The potential for furtherconceptual and methodological development and for broad applications in the field of behav-ioral and social sciences promises to greatly advance the empirical basis of our knowledgeabout causality in the social world

This book clarifies theoretical concepts and introduces innovative statistical strategies forinvestigating the average effects of multivalued treatments moderated treatment effectsmediated treatment effects and spill-over effects in experimental or quasiexperimental dataDefining individual-specific and population average treatment effects in terms of potentialoutcomes the book relates the mathematical forms to the substantive meanings of moderatedmediated and spill-over effects in the context of application examples It also explicates andevaluates identification assumptions and contrasts innovative statistical strategies with con-ventional analytic methods

111 Moderated treatment effects

It is hard to accept the assumption that a treatment would produce the same impact for everyindividual in every possible circumstance Understanding the heterogeneity of treatmenteffects therefore is key to the development of causal theories For example some studiesreported that estrogen therapy improved cardiovascular health among women who initiatedits use during menopause According to a series of other studies however the use of estrogentherapy increased postmenopausal womenrsquos risk for heart attacks (Grodstein et al 19962000 Writing Group for the Womenrsquos Health Initiative Investigators 2002) The sharp con-trast of findings from these studies led to the hypothesis that age of initiation moderates theeffect of estrogen therapy on womenrsquos health (Manson and Bassuk 2007 Rossouw et al2007) Revelations of the moderated causal relationship greatly enrich theoretical understand-ing and in this case directly inform clinical practice

In another example dividing students by ability into small groups for reading instructionhas been controversial for decades Many believed that the practice benefits high-ability stu-dents at the expense of their low-ability peers and exacerbates educational inequality (Grantand Rothenberg 1986 Rowan and Miracle 1983 Trimble and Sinclair 1987) Recentevidence has shown that first of all the effect of ability grouping depends on a number ofconditions including whether enough time is allocated to reading instruction and how wellthe class can be managed Hence instructional time and class management are among theadditional moderators to consider for determining the merits of ability grouping (Hong andHong 2009 Hong et al 2012a b) Moreover further investigations have generated evidencethat contradicts the long-held belief that grouping undermines the interests of low-abilitystudents Quite the contrary researchers have found that grouping is beneficial for students

5INTRODUCTION

with low or medium prior abilitymdashif adequate time is provided for instruction and if the classis well managed On the other hand ability grouping appears to have a minimal impact onhigh-ability studentsrsquo literacy These results were derived from kindergarten data and maynot hold for math instruction and for higher grade levels Replications with different subpo-pulations and in different contexts enable researchers to assess the generalizability of a theory

In a third example one may hypothesize that a student is unlikely to learn algebra well inninth grade without a solid foundation in eighth-grade prealgebra One may also argue that theprogress a student made in learning eighth-grade prealgebra may not be sustained without thefurther enhancement of taking algebra in ninth grade In other words the earlier treatment(prealgebra in eighth grade) and the later treatment (algebra in ninth grade) may reinforceone other each moderates the effect of the other treatment on the final outcome (math achieve-ment at the end of ninth grade) As Hong and Raudenbush (2008) argued the cumulativeeffect of two treatments in a well-aligned sequence may exceed the sum of the benefits oftwo single-year treatments Similarly experiencing two consecutive years of inferior treat-ments such as encountering an incompetent math teacher who dampens a studentrsquos self-efficacy in math learning in eighth grade and then again in ninth grade could do more damagethan the sum of the effect of encountering an incompetent teacher only in eighth grade and theeffect of having such a teacher only in ninth grade

There has been a great amount of conceptual confusion regarding how a moderatorrelates to the treatment and the outcome On one hand many researchers have used the termsldquomoderatorsrdquo and ldquomediatorsrdquo interchangeably without understanding the crucial distinctionbetween the two An overcorrective attempt on the other hand has led to the arbitrary rec-ommendation that a moderator must occur prior to the treatment and be minimally associatedwith the treatment (James and Brett 1984 Kraemer et al 2001 2008) In Chapter 6 weclarify that in essence the causal effect of the treatment on the outcome may depend onthe moderator value A moderator can be a subpopulation identifier a contextual character-istic a concurrent treatment or a preceding or succeeding treatment In the earlier examplesage of estrogen therapy initiation and a studentrsquos prior ability are subpopulation identifiers themanageability of a class which reflects both teacher skills and peer behaviors characterizes thecontext literacy instruction time is a concurrent treatment while prealgebra is a precedingtreatment for algebra A moderator does not have to occur prior to the treatment and doesnot have to be independent of the treatment

Once a moderated causal relationship has been defined in terms of potential outcomes theresearcher then chooses an appropriate experimental design for testing the moderation theoryThe assumptions required for identifying the moderated causal relationships differ across dif-ferent designs and have implications for analyzing experimental and quasiexperimental dataRandomized block designs are suitable for examining individual or contextual characteristicsas potential moderators factorial designs enable one to determine the joint effects of two ormore concurrent treatments and sequential randomized designs are ideal for assessing thecumulative effects of consecutive treatments Multisite randomized trials constitute anadditional type of designs in which the experimental sites are often deliberately sampled torepresent a population of geographical locations or social organizations Site membershipscan be viewed as implicit moderators that summarize a host of features of a local environmentReplications of a treatment over multiple sites allow one to quantify the heterogeneity of treat-ment effects across the sites Chapters 7 and 8 are focused on statistical methods for moder-ation analyses with quasiexperimental data In particular these chapters demonstrate througha number of application examples how a nonparametric marginal mean weighting through

6 CAUSALITY IN A SOCIAL WORLD

stratification (MMWS) method can overcome some important limitations of other existingmethods

Moderation questions are not restricted to treatmentndashoutcome relationships Rather theyare prevalent in investigations of mediation and spill-over This is because heterogeneity oftreatment effects may be explained by the variation in causal mediation mechanisms acrosssubpopulations and contexts It is also because the treatment effect for a focal individualmay depend on whether there is spill-over from other individuals through social interactionsYet similar to the confusion around the concept of moderation among applied researchersthere has been a great amount of misunderstanding with regard to mediation

112 Mediated treatment effects

Questions about mediation are at the core of nearly every scientific theory In its simplestform a theory explains why treatment Z causes outcome Y by hypothesizing an intermediateprocess involving at least one mediatorM that could have been changed by the treatment andcould subsequently have an impact on the outcome A causal mediation analysis then decom-poses the total effect of the treatment on the outcome into two parts the effect transmittedthrough the hypothesized mediator called the indirect effect and the difference betweenthe total effect and the indirect effect called the direct effect The latter represents the treat-ment effect channeled through other unspecified pathways (Alwin and Hauser 1975 Baronand Kenny 1986 Duncan 1966 Shadish and Sweeney 1991) Most researchers in socialsciences have followed the convention of illustrating the direct effect and the indirect effectwith path diagrams and then specifying linear regression models postulated to represent thestructural relationships between the treatment the mediator and the outcome

In Chapter 9 we point out that the approach to defining the indirect effect and the directeffect in terms of path coefficients is often misled by oversimplistic assumptions about thestructural relationships In particular this approach typically overlooks the fact that a treat-ment may generate an impact on the outcome not only by changing the mediator value butalso by changing the mediatorndashoutcome relationship (Judd and Kenny 1981) An examplein Chapter 9 illustrates such a case An experiment randomizes students to either an experi-mental condition which provides them with study materials and encourages them to study fora test or to a control condition which provides neither study materials nor encouragement(Holland 1988 Powers and Swinton 1984) One might hypothesize that encouraging exper-imental group members to study will increase the time they spend studying a focal mediatorin this example which in turn will increase their average test scores One might furtherhypothesize that even without a change in study time providing study materials to experi-mental group members will enable them to study more effectively than they otherwise wouldConsequently the effect of time spent studying might be greater under the experimentalcondition than under the control condition Omitting the treatment-by-mediator interactioneffect will lead to bias in the estimation of the indirect effect and the direct effect

Chapter 11 describes in great detail an evaluation study contrasting a new welfare-to-workprogram emphasizing active participation in the labor force with a traditional program thatguaranteed cash assistance without a mandate to seek employment (Hong Deutsch and Hill2011) Focusing on the psychological well-being of welfare applicants who were singlemothers with preschool-aged children the researchers hypothesized that the treatment wouldincrease employment rate among the welfare recipients and that the treatment-inducedincrease in employment would likely reduce depressive symptoms under the new program

7INTRODUCTION

They also hypothesized that in contrast the same increase in employment was unlikely tohave a psychological impact under the traditional program Hence the treatment-by-mediatorinteraction effect on the outcome was an essential component of the intermediate process

We will show that by defining the indirect effect and the direct effect in terms of potentialoutcomes one can avoid invoking unwarranted assumptions about the unknown structuralrelationships (Holland 1988 Pearl 2001 Robins and Greenland 1992) The potential out-comes framework has further clarified that in a causal mediation theory the treatment and themediator are both conceivably manipulable In the aforementioned example a welfare appli-cant could be assigned at random to either the new program or the traditional one Once havingbeen assigned to one of the programs the individual might or might not be employed due tovarious structural constraints market fluctuations or other random events typically beyondher control

Because the indirect effect and the direct effect are defined in terms of potential outcomesrather than on the basis of any specific regression models it becomes possible to distinguishthe definitions of these causal effects from their identification and estimation The identifica-tion step relates a causal parameter to observable population data For example for individualsassigned at random to an experimental condition their average counterfactual outcome asso-ciated with the control condition is expected to be equal to the average observable outcome ofthose assigned to the control condition Therefore the population average causal effect can beeasily identified in a randomized experiment The estimation step then relates the sample datato the observable population quantities involved in identification while taking into account thedegree of sampling variability

A major challenge in identification is that the mediatorndashoutcome relationship tends tobe confounded by selection even if the treatment is randomized Chapter 9 reviews variousexperimental designs that have been proposed by past researchers for studying causalmediation mechanisms Chapter 10 compares the identification assumptions and the analyticprocedures across a wide range of analytic methods These discussions are followed by anintroduction of the ratio-of-mediator-probability weighting (RMPW) strategy in Chapter 11In comparison with most existing strategies for causal mediation analysis RMPW relies onrelatively fewer identification assumptions and model-based assumptions Chapters 12 and13 will show that this new strategy can be applied broadly with extensions to multilevelexperimental designs and to studies of complex mediation mechanisms involving multiplemediators

113 Spill-over effects of a treatment

It is well known in vaccination research that an unvaccinated person can benefit when mostother people in the local community are vaccinated This is viewed as a spill-over of the treat-ment effect from the vaccinated people to an unvaccinated person Similarly due to socialinteractions among individuals or groups of individuals an intervention received by somemay generate a spill-over impact on others affiliated with the same organization or connectedthrough the same network For example effective policing in one neighborhood may driveoffenders to operate in other neighborhoods In evaluating an innovative community policingprogram researchers found that a neighborhood not assigned to community policing tended tosuffer if the surrounding neighborhoods were assigned to community policing This is aninstance in which a well-intended treatment generates an unintended negative spill-over effectHowever being assigned to community policy was found particularly beneficial when thesurrounding neighborhoods also received the intervention (Verbitsky-Savitz and Raudenbush

8 CAUSALITY IN A SOCIAL WORLD

2012) In another example the impact of a school-based mentoring program targeted atstudents displaying delinquent behaviors may be enhanced for a focal student if his or herat-risk peers are assigned to the mentoring program at the same time

The previous examples challenge the ldquostable unit treatment value assumptionrdquo (SUTVA)that has been invoked in most causal inference studies This assumption states that an indi-vidualrsquos potential response to a treatment depends neither on how the treatment is assignednor on the treatment assignments of other individuals (Rubin 1980 1986) Rubin and othersbelieve that without resorting to SUTVA the causal effect of a treatment becomes hard todefine and that causal inference may become intractable However as Sobel (2006) has illus-trated with the Moving to Opportunity (MTO) experiment that offered housing vouchersenabling low-income families to move to low-poverty neighborhoods there are possible con-sequences of violating SUTVA in estimating the treatment effects on outcomes such as safetyand self-sufficiency Because social interactions among individuals may affect whether onevolunteers to participate in the study whether one moves to a low-poverty neighborhood afterreceiving the housing voucher as well as housing project residentsrsquo subjective perceptions oftheir neighborhoods the program may have a nonzero impact on the potential outcome of theuntreated As a result despite the randomization of treatment assignment the mean differencein the observed outcome between the treated units and the untreated units may be biased forthe average treatment effect Rather the observable quantity is the difference between theeffect of treating some rather than treating none for the treated and the effect of treating somerather than treating none for the untreated the latter being the pure spill-over effect for theuntreated

In making attempts to relax SUTVA researchers have proposed several alternative frame-works that incorporate possible spill-over effects (see Hong and Raudenbush 2013 for areview) Hong (2004) presented a model that involves treatments and treatment settingsA treatment setting for an individual is a local environment constituted by a set of agentsand participants along with their treatment assignments An individualrsquos potential outcomevalue under a given treatment is assumed stable when the treatment setting is fixed the poten-tial outcome may take different values when the treatment setting shifts One may investigatewhether the treatment effect depends on the treatment setting Applying this framework Hongand Raudenbush (2006) examined the effect on a childrsquos academic growth of retaining thechild in kindergarten rather than promoting the child to first grade when a relatively smallproportion of low-achieving peers in the same school are retained as opposed to when a rel-atively large proportion of the peers are retained Hudgens and Halloran (2008) presented arelated framework in which the effect on an individual of the treatment received by this indi-vidual is distinguished from the effect on the individual of the treatment received by others inthe same local community These effects can be identified if communities are assigned atrandom to different treatment assignment strategies (such as retaining a large proportion oflow-achieving students as opposed to retaining a small proportion of such students) and sub-sequently individuals within a community are randomized for treatment Chapter 14 reviewsthese frameworks and discusses identification and estimation strategies

Social contagion may also serve as an important channel through which the effect of anintervention is transmitted For example a school-wide intervention may reduce aggressivebehaviors and thereby improve studentsrsquo psychological well-being by improving the qualityof interpersonal relationships in other classes as well as in onersquos own class This is becausechildren interact not simply with their classmates but also with those from other classes in thehallways or on the playground (VanderWeele et al 2013) In this case the spill-over betweenclasses becomes a part of the mediation mechanism In a study of student mentoring whether

9INTRODUCTION

an individual actually participated in the mentoring program despite the initial treatmentassignment is typically viewed as a primary mediator The proportion of onersquos peers that par-ticipated in the program may act as a second mediator A treatment may also exert its impactpartly through regrouping individuals and thereby changing peer composition (Hong andNomi 2012) Chapter 15 discusses analytic strategies for detecting spill-over as a part ofthe causal mediation mechanism

12 Weighting methods for causal inference

In most behavioral and social science applications major methodological challenges arise dueto the selection of treatments in a quasiexperimental study and the selection of mediator valuesin experimental and quasiexperimental studies Statistical methods most familiar to research-ers in social sciences are often inadequate for causal inferences with regard to multivaluedtreatments moderation mediation and spill-over The book offers a major revision to under-standing of causality in the social world by introducing two complementary weighting stra-tegies both featuring nonparametric approaches to estimating these causal effects The newweighting methods greatly simplify model specifications while enhancing the robustness ofresults by minimizing the reliance on some key assumptions

The propensity score-basedMMWSmethod removes selection bias associated with a largenumber of covariates by equating the pretreatment composition between treatment groups(Hong 2010a 2012 Huang et al 2005) Unlike propensity score matching and stratificationthat are mostly restricted to evaluations of binary treatments the MMWS method is flexiblefor evaluating binary and multivalued treatments by approximating a completely randomizedexperiment In evaluating whether the treatment effects differ across subpopulations definedby individual characteristics or treatment settings researchers may assign weights within eachsubpopulation in order to approximate a randomized block design To investigate whetherone treatment moderates the effect of another concurrent treatment researchers may assignweights to the data to approximate a factorial randomized design The method can also beused to assess whether the effect of an initial treatment is amplified or weakened by a subse-quent treatment or to identify an optimal treatment sequence through approximating a sequen-tial randomized experiment Even though such analyses can similarly be conducted throughinverse-probability-of-treatment weighting (IPTW) that has been increasingly employed inepidemiological research (Hernaacuten Brumback and Robins 2000 Robins Hernaacuten andBrumback 2000) IPTW is known for bias and imprecision in estimation especially whenthe propensity score models are misspecified in their functional forms (Hong 2010a Kangand Schafer 2007 Schafer and Kang 2008 Waernbaum 2012) In contrast the nonparamet-ric MMWS method displays a relatively high level of robustness despite such misspecifica-tions and also gains efficiency as indicated by simulation results (Hong 2010a)

To study causal mediation mechanisms the RMPW method decomposes the total effectof a treatment into an ldquoindirect effectrdquo transmitted through a specific mediator and a ldquodirecteffectrdquo representing unspecified mechanisms In contrast with most existing methods formediation analysis the RMPW-adjusted outcome model is extremely simple and is nonpara-metric in nature It generates estimates of the causal effects along with their sampling errorswhile adjusting for pretreatment covariates that confound the mediatorndashoutcome relationshipsthrough weighting The method applies regardless of the distribution of the outcome the dis-tribution of the mediator or the functional relationship between the outcome and the mediator

10 CAUSALITY IN A SOCIAL WORLD

Page 13: Thumbnail - download.e-bookshelf.de...4.2.4 IPTW for bias removal 84 4.3 MMWS 86 4.3.1 Theoretical rationale 86 ... 8.3.4.2 Growth model in the absence of confounding 200 8.3.4.3 Weighted

1334 Contrast with the linear SEM approach 3531335 Contrast with the sensitivity-based estimation of bounds for

causal effects 354134 Discussion 355Appendix 13A Derivation of RMPW for estimating population average

counterfactual outcomes of two concurrentmediators 355

Appendix 13B Derivation of RMPW for estimating population averagecounterfactual outcomes of consecutivemediators 358

Part IV Spill-over 363

14 Spill-over of treatment effects concepts and methods 365141 Spill-over A nuisance a trifle or a focus 365142 Stable versus unstable potential outcome values An example

from agriculture 367143 Consequences for causal inference when spill-over is overlooked 369144 Modified framework of causal inference 371

1441 Treatment settings 3711442 Simplified characterization of treatment settings 3731443 Causal effects of individual treatment assignment and of peer

treatment assignment 375145 Identification Challenges and solutions 376

1451 Hypothetical experiments for identifying average treatmenteffects in the presence of social interactions 376

1452 Hypothetical experiments for identifying the impact ofsocial interactions 380

1453 Application to an evaluation of kindergarten retention 382146 Analytic strategies for experimental and quasiexperimental data 384

1461 Estimation with experimental data 3841462 Propensity score stratification 3851463 MMWS 386

147 Summary 387

15 Mediation through spill-over 391151 Definition of mediated effects through spill-over in a cluster

randomized trial 3931511 Notation 3931512 Treatment effect mediated by a focal individualrsquos

compliance 3941513 Treatment effect mediated by peersrsquo compliance through

spill-over 3941514 Decomposition of the total treatment effect 395

152 Identification and estimation of the spill-over effect in a clusterrandomized design 3951521 Identification in an ideal experiment 395

xiiiCONTENTS

1522 Identification when the mediators are not randomized 3981523 Estimation of mediated effects through spill-over 400

153 Definition of mediated effects through spill-over in a multisite trial 4021531 Notation 4021532 Treatment effect mediated by a focal individualrsquos compliance 4041533 Treatment effect mediated by peersrsquo compliance

through spill-over 4041534 Direct effect of individual treatment assignment on the outcome 4051535 Direct effect of peer treatment assignment on the outcome 4051536 Decomposition of the total treatment effect 405

154 Identification and estimation of spill-over effects in a multisite trial 4061541 Identification in an ideal experiment 4071542 Identification when the mediators are not randomized 4091543 Estimation of mediated effects through spill-over 410

15431 Estimating E[Y(1pM(p)Mminus(p))] andE[Y(000Mminus(0))] 410

15432 Estimating E[Y(1p0Mminus(p))] 41115433 Estimating E[Y(1p0Mminus(0))] 41215434 Estimating E[Y(0p0Mminus(0))] 412

155 Consequences of omitting spill-over effects in causal mediation analyses 4121551 Biased inference in a cluster randomized trial 4131552 Biased inference in a multisite randomized trial 4131553 Biased inference of the local average treatment effect 415

156 Quasiexperimental application 416157 Summary 419Appendix 151 Derivation of the weight for estimating the population

average counterfactual outcome E[Y(1 p 0Mminus( p))] 419Appendix 152 Derivation of bias in the ITT effect due to the omission of

spill-over effects 420

Index 423

xiv CONTENTS

Preface

A scientific mind in training seeks comprehensive descriptions of every interesting phenom-enon In such a mind data are to be pieced together for comparisons contrasts and eventuallyinferences that are required for understanding causes and effects that may have led to thephenomenon of interest Yet without disciplined reasoning a causal inference would slip intothe rut of a ldquocasual inferencerdquo as easily as making a typographical error

As a doctoral student thirsting for rigorous methodological training at the University ofMichigan I was very fortunate to be among the first group of students on the campus attendinga causal inference seminar in 2000 jointly taught by Yu Xie from the Department of Sociol-ogy Susan Murphy from the Department of Statistics and Stephen Raudenbush from theSchool of Education The course was unique in both content and pedagogy The instructorsput together a bibliography drawn from the past and current causal inference literature origi-nated from multiple disciplines In the spirit of ldquoreciprocal teachingrdquo the students were delib-erately organized into small groups each representing a mix of disciplinary backgrounds andtook turns to teach the weekly readings under the guidance of a faculty instructor This inval-uable experience revealed to me that the pursuit of causal inference is a multidisciplinaryendeavor In the rest of my doctoral study I continued to be immersed in an extraordinaryinterdisciplinary community in the form of the Quantitative Methodology Program (QMP)seminars led by the above three instructors who were then joined by Ben Hansen(Statistics) Richard Gonzalez (Psychology) Roderick Little (Biostatistics) Jake Bowers(Political Science) and many others I have tried whenever possible to carry the same spiritinto my own teaching of causal inference courses at the University of Toronto and now at theUniversity of Chicago In these courses communicating understandings of causal problemsacross disciplinary boundaries has been particularly challenging but also stimulating and grat-ifying for myself and for students from various departments and schools

Under the supervision of Stephen Raudenbush I took a first stab at the conceptual frame-work of causal inference in my doctoral dissertation by considering peer spill-over in schoolsettings From that point on I have organized my methodological research to tackle some ofthe major obstacles to causal inferences in policy and program evaluations My workaddresses issues including (i) how to conceptualize and evaluate the causal effects of educa-tional treatments when studentsrsquo responses to alternative treatments depend on various fea-tures of the organizational settings including peer composition (ii) how to adjust forselection bias in evaluating the effects of concurrent and consecutive multivalued treatmentsand (iii) how to conceptualize and analyze causal mediation mechanisms My methodological

research has been driven by the substantive interest in understanding the role of socialinstitutionsmdashschools in particularmdashin shaping human development especially during theages of fast growth (eg childhood adolescence and early adulthood) I have shared meth-odological advances with a broad audience in social sciences through applying the causalinference methods to prominent substantive issues such as grade retention within-class group-ing services for English language learners and welfare-to-work strategies These illustrationsare used extensively in this book

Among social scientists awareness of methodological challenges in causal investigationsis higher than ever before In response to this rising demand causal inference is becoming oneof the most productive scholarly fields in the past decade I have had the benefit of followingthe work and exchanging ideas with some fellow methodologists who are constantly contri-buting new thoughts and making breakthroughs These include Daniel Almirall HowardBloom Tom Cook Michael Elliot Ken Frank Ben Hansen James Heckman JenniferHill Martin Huber Kosuke Imai Booil Jo John R Lockwood David MacKinnon DanielMcCaffrey Luke Miratrix Richard Murnane Derek Neal Lindsey Page Judea PearlStephen Raudenbush Sean Reardon Michael Seltzer Youngyun Shin Betsy SinclairMichael Sobel Peter Steiner Elizabeth Stuart Eric Thetchen Tchetchen Tyler VanderWeeleand Kazuo Yamaguchi I am grateful to Larry Hedges who offered me the opportunity of guestediting the Journal of Research in Educational Effectiveness special issue on the statisticalapproaches to studying mediator effects in education research in 2012 Many of the colleagueswhom I mentioned above contributed to the open debate in the special issue by either propos-ing or critically examining a number of innovative methods for studying causal mechanismsI anticipate that many of them are or will soon be writing their own research monographs oncausal inference if they have not already done so Therefore readers of this book are stronglyrecommended to browse past and future titles by these and other authors for alternative per-spectives and approaches My own understanding of course will continue to evolve as well inthe coming years

One has to be ambitious to stay upfront in substantive areas as well as in methodology Thebest strategy apparently is to learn from substantive experts During different phases of mytraining and professional career I have had the opportunities to work with David K CohenBrian Rowan Deborah Ball Carl Corter Janette Pelletier Takako Nomi Esther Geva DavidFrancis Stephanie Jones Joshua Brown and Heather D Hill Many of my substantive insightswere derived from conversations with these mentors and colleagues I am especially gratefulto Bob Granger former president of the William T Grant Foundation who reassured me thatldquoBy staying a bit broad you will learn a lot and many fields will benefit from your workrdquo

Like many other single-authored books this one is built on the generous contributions ofstudents and colleagues who deserve special acknowledgement Excellent research assistancefrom Jonah Deutsch Joshua Gagne Rachel Garrett Yihua Hong Xu Qin Cheng Yang andBing Yu was instrumental in the development and implementation of the new methods pre-sented in this book Richard Congdon a renowned statistical software programmer broughtrevolutionary ideas to interface designs in software development with the aim of facilitatingusersrsquo decision-making in a series of causal analyses RichardMurnane Stephen Raudenbushand Michael Sobel carefully reviewed and critiqued multiple chapters of the manuscript draftAdditional comments and suggestions on earlier drafts came fromHoward Bloom Ken FrankBen Hansen Jennifer Hill Booil Jo Ben Kelcey David MacKinnon and Fan Yang TereseSchwartzman provided valuable assistance in manuscript preparation The writing of thisbook was supported by a Scholars Award from the William T Grant Foundation and an

xvi PREFACE

Institute of Education Sciences (IES) Statistical and Research Methodology in EducationGrant from the US Department of Education All the errors in the published form are ofcourse the sole responsibility of the author

Project Editors Richard Davis Prachi Sinha Sahay and Liz Wingett and Assistant EditorHeather Kay at Wiley and project Managers P Jayapriya and R Jayavel at SPi Global havebeen wonderful to work with throughout the manuscript preparation and final production ofthe book Debbie Jupe Commissioning Editor at Wiley was remarkably effective in initiatingthe contact with me and in organizing anonymous reviews of the book proposal and of themanuscript These constructive reviews have shaped the book in important ways I cannotagree more with one of the reviewers that ldquocausality cannot be established on a pure statisticalgroundrdquo The book therefore highlights the importance of substantive knowledge and researchdesigns and places a great emphasis on clarifying and evaluating assumptions required foridentifying causal effects in the context of each application Much caution is raised againstpossible misusage of statistical techniques for analyzing causal relationships especiallywhen data are inadequate Yet I maintain that improved statistical procedures along withimproved research designs would greatly enhance our ability in attempt to empiricallyexamine well-articulated causal theories

xviiPREFACE

Part I

OVERVIEW

1

Introduction

According to an ancient Chinese fable a farmer who was eager to help his crops grow wentinto his field and pulled each seedling upward After exhausting himself with the work heannounced to his family that they were going to have a good harvest only to find the nextmorning that the plants had wilted and died Readers with minimal agricultural knowledgemay immediately point out the following the farmerrsquos intervention theory was based on acorrect observation that crops that grow taller tend to produce more yield Yet his hypothesisreflects a false understanding of the cause and the effectmdashthat seedlings pulled to be tallerwould yield as much as seedlings thriving on their own

In their classic Design and Analysis of Experiments Hinkelmann and Kempthorne (1994updated version of Kempthorne 1952) discussed two types of science descriptive science andthe development of theory These two types of science are interrelated in the following senseobservations of an event and other related events often selected and classified for descriptionby scientists naturally lead to one or more explanations that we call ldquotheoretical hypothesesrdquowhich are then screened and falsified by means of further observations experimentation andanalyses (Popper 1963) The experiment of pulling seedlings to be taller was costly but didserve the purpose of advancing this farmerrsquos knowledge of ldquowhat does not workrdquo To developa successful intervention in this case would require a series of empirical tests of explicittheories identifying potential contributors to crop growth This iterative process graduallydeepens our knowledge of the relationships between supposed causes and effectsmdashthatis causalitymdashand may eventually increase the success of agricultural medical and socialinterventions

11 Concepts of moderation mediation and spill-over

Although the story of the ancient farmer is fictitious numerous examples can be found in thereal world in which well-intended interventions fail to produce the intended benefits or inmany cases even lead to unintended consequences ldquoInterventionsrdquo and ldquotreatmentsrdquo usedinterchangeably in this book broadly refer to actions taken by agents or circumstances expe-rienced by an individual or groups of individuals Interventions are regularly seen in educa-tion physical and mental health social services business politics and law enforcement In an

Causality in a Social World Moderation Meditation and Spill-over First Edition Guanglei Hongcopy 2015 John Wiley amp Sons Ltd Published 2015 by John Wiley amp Sons Ltd

education intervention for example teachers are typically the agents who deliver a treatmentto students while the impact of the treatment on student outcomes is of ultimate causalinterest Some educational practices such as ldquoteaching to the testrdquo have been criticized tobe nearly as counterproductive as the attempt of helping seedlings grow by pulling themupward ldquoInterventionsrdquo and ldquotreatmentsrdquo under consideration do not exclude undesiredexperiences such as exposure to poverty abuse crime or bereavement A treatment plannedor unplanned becomes a focus of research if there are theoretical reasons to anticipate itsimpact positive or negative on the well-being of individuals who are embedded in socialsettings including families classrooms schools neighborhoods and workplaces

In social science research in general and in policy and program evaluations in particularquestions concerning whether an intervention works and if so which version of the interven-tion works for whom under what conditions and why are key to the advancement of scien-tific and practical knowledge Although most empirical investigations in the social sciencesconcentrate on the average effect of a treatment for a specific population as opposed to theabsence of such a treatment (ie the control condition) in-depth theoretical reasoning withregard to how the causal effect is generated substantiated by compelling empirical evidenceis crucial for advancing scientific understanding

First when there are multiple versions or different dosages of the treatment or when thereare multiple versions of the control condition a binary divide between ldquothe treatmentrdquo andldquothe controlrdquo may not be as informative as fine-grained comparisons across for exampleldquotreatment version Ardquo ldquotreatment version Brdquo ldquocontrol version Ardquo and ldquocontrol versionBrdquo For example expanding the federally funded Head Start program to poor children isexpected to generate a greater benefit when few early childhood education alternatives areavailable (call it ldquocontrol version Ardquo) than when there is an abundance of alternatives includ-ing state-sponsored preschool programs (call it ldquocontrol version Brdquo)

Second the effect of an intervention will likely vary among individuals or across socialsettings A famous example comes from medical research the well-publicized cardiovascularbenefits of initiating estrogen therapy during menopause were contradicted later by experi-mental findings that the same therapy increased postmenopausal womenrsquos risk for heartattacks The effect of an intervention may also depend on the provision of some other con-current or subsequent interventions Such heterogeneous effects are often characterized asmoderated effects in the literature

Third alternative theories may provide competing explanations for the causal mechan-isms that is the processes through which the intervention produces its effect A theoreticalconstruct characterizing the hypothesized intermediate process is called a mediator of theintervention effect The fictitious farmer never developed an elaborate theory as to whatcaused some seedlings to surpass others in growth Once scientists revealed the causal rela-tionship between access to chemical nutrients in soil and plant growth wide applications ofchemically synthesized fertilizers finally led to a major increase in crop production

Finally it is well known in agricultural experiments that a new type of fertilizer applied toone plot may spill-over to the next plot Because social connections among individuals areprevalent within organizations or through networks an individualrsquos response to the treatmentmay similarly depend on the treatment for other individuals in the same social setting whichmay lead to possible spill-overs of intervention effects among individual human beings

Answering questions with regard to moderation mediation and spill-over poses majorconceptual and analytic challenges To date psychological research often presents well-articulated theories of causal mechanisms relating stimuli to responses Yet researchers often

4 CAUSALITY IN A SOCIAL WORLD

lack rigorous analytic strategies for empirically screening competing theories explaining theobserved effect Sociologists have keen interest in the spill-over of treatment effects transmit-ted through social interactions yet have produced limited evidence quantifying such effectsAs many have pointed out in general terminological ambiguity and conceptual confusionhave been prevalent in the published applied research (Holmbeck 1997 Kraemer et al2002 2008)

A new framework for conceptualizing moderated mediated and spill-over effects hasemerged relatively recently in the statistics and econometrics literature on causal inference(eg Abbring and Heckman 2007 Frangakis and Rubin 2002 Heckman and Vytlacil2007a b Holland 1988 Hong and Raudenbush 2006 Hudgens and Halloran 2008Jo 2008 Pearl 2001 Robins and Greenland 1992 Sobel 2008) The potential for furtherconceptual and methodological development and for broad applications in the field of behav-ioral and social sciences promises to greatly advance the empirical basis of our knowledgeabout causality in the social world

This book clarifies theoretical concepts and introduces innovative statistical strategies forinvestigating the average effects of multivalued treatments moderated treatment effectsmediated treatment effects and spill-over effects in experimental or quasiexperimental dataDefining individual-specific and population average treatment effects in terms of potentialoutcomes the book relates the mathematical forms to the substantive meanings of moderatedmediated and spill-over effects in the context of application examples It also explicates andevaluates identification assumptions and contrasts innovative statistical strategies with con-ventional analytic methods

111 Moderated treatment effects

It is hard to accept the assumption that a treatment would produce the same impact for everyindividual in every possible circumstance Understanding the heterogeneity of treatmenteffects therefore is key to the development of causal theories For example some studiesreported that estrogen therapy improved cardiovascular health among women who initiatedits use during menopause According to a series of other studies however the use of estrogentherapy increased postmenopausal womenrsquos risk for heart attacks (Grodstein et al 19962000 Writing Group for the Womenrsquos Health Initiative Investigators 2002) The sharp con-trast of findings from these studies led to the hypothesis that age of initiation moderates theeffect of estrogen therapy on womenrsquos health (Manson and Bassuk 2007 Rossouw et al2007) Revelations of the moderated causal relationship greatly enrich theoretical understand-ing and in this case directly inform clinical practice

In another example dividing students by ability into small groups for reading instructionhas been controversial for decades Many believed that the practice benefits high-ability stu-dents at the expense of their low-ability peers and exacerbates educational inequality (Grantand Rothenberg 1986 Rowan and Miracle 1983 Trimble and Sinclair 1987) Recentevidence has shown that first of all the effect of ability grouping depends on a number ofconditions including whether enough time is allocated to reading instruction and how wellthe class can be managed Hence instructional time and class management are among theadditional moderators to consider for determining the merits of ability grouping (Hong andHong 2009 Hong et al 2012a b) Moreover further investigations have generated evidencethat contradicts the long-held belief that grouping undermines the interests of low-abilitystudents Quite the contrary researchers have found that grouping is beneficial for students

5INTRODUCTION

with low or medium prior abilitymdashif adequate time is provided for instruction and if the classis well managed On the other hand ability grouping appears to have a minimal impact onhigh-ability studentsrsquo literacy These results were derived from kindergarten data and maynot hold for math instruction and for higher grade levels Replications with different subpo-pulations and in different contexts enable researchers to assess the generalizability of a theory

In a third example one may hypothesize that a student is unlikely to learn algebra well inninth grade without a solid foundation in eighth-grade prealgebra One may also argue that theprogress a student made in learning eighth-grade prealgebra may not be sustained without thefurther enhancement of taking algebra in ninth grade In other words the earlier treatment(prealgebra in eighth grade) and the later treatment (algebra in ninth grade) may reinforceone other each moderates the effect of the other treatment on the final outcome (math achieve-ment at the end of ninth grade) As Hong and Raudenbush (2008) argued the cumulativeeffect of two treatments in a well-aligned sequence may exceed the sum of the benefits oftwo single-year treatments Similarly experiencing two consecutive years of inferior treat-ments such as encountering an incompetent math teacher who dampens a studentrsquos self-efficacy in math learning in eighth grade and then again in ninth grade could do more damagethan the sum of the effect of encountering an incompetent teacher only in eighth grade and theeffect of having such a teacher only in ninth grade

There has been a great amount of conceptual confusion regarding how a moderatorrelates to the treatment and the outcome On one hand many researchers have used the termsldquomoderatorsrdquo and ldquomediatorsrdquo interchangeably without understanding the crucial distinctionbetween the two An overcorrective attempt on the other hand has led to the arbitrary rec-ommendation that a moderator must occur prior to the treatment and be minimally associatedwith the treatment (James and Brett 1984 Kraemer et al 2001 2008) In Chapter 6 weclarify that in essence the causal effect of the treatment on the outcome may depend onthe moderator value A moderator can be a subpopulation identifier a contextual character-istic a concurrent treatment or a preceding or succeeding treatment In the earlier examplesage of estrogen therapy initiation and a studentrsquos prior ability are subpopulation identifiers themanageability of a class which reflects both teacher skills and peer behaviors characterizes thecontext literacy instruction time is a concurrent treatment while prealgebra is a precedingtreatment for algebra A moderator does not have to occur prior to the treatment and doesnot have to be independent of the treatment

Once a moderated causal relationship has been defined in terms of potential outcomes theresearcher then chooses an appropriate experimental design for testing the moderation theoryThe assumptions required for identifying the moderated causal relationships differ across dif-ferent designs and have implications for analyzing experimental and quasiexperimental dataRandomized block designs are suitable for examining individual or contextual characteristicsas potential moderators factorial designs enable one to determine the joint effects of two ormore concurrent treatments and sequential randomized designs are ideal for assessing thecumulative effects of consecutive treatments Multisite randomized trials constitute anadditional type of designs in which the experimental sites are often deliberately sampled torepresent a population of geographical locations or social organizations Site membershipscan be viewed as implicit moderators that summarize a host of features of a local environmentReplications of a treatment over multiple sites allow one to quantify the heterogeneity of treat-ment effects across the sites Chapters 7 and 8 are focused on statistical methods for moder-ation analyses with quasiexperimental data In particular these chapters demonstrate througha number of application examples how a nonparametric marginal mean weighting through

6 CAUSALITY IN A SOCIAL WORLD

stratification (MMWS) method can overcome some important limitations of other existingmethods

Moderation questions are not restricted to treatmentndashoutcome relationships Rather theyare prevalent in investigations of mediation and spill-over This is because heterogeneity oftreatment effects may be explained by the variation in causal mediation mechanisms acrosssubpopulations and contexts It is also because the treatment effect for a focal individualmay depend on whether there is spill-over from other individuals through social interactionsYet similar to the confusion around the concept of moderation among applied researchersthere has been a great amount of misunderstanding with regard to mediation

112 Mediated treatment effects

Questions about mediation are at the core of nearly every scientific theory In its simplestform a theory explains why treatment Z causes outcome Y by hypothesizing an intermediateprocess involving at least one mediatorM that could have been changed by the treatment andcould subsequently have an impact on the outcome A causal mediation analysis then decom-poses the total effect of the treatment on the outcome into two parts the effect transmittedthrough the hypothesized mediator called the indirect effect and the difference betweenthe total effect and the indirect effect called the direct effect The latter represents the treat-ment effect channeled through other unspecified pathways (Alwin and Hauser 1975 Baronand Kenny 1986 Duncan 1966 Shadish and Sweeney 1991) Most researchers in socialsciences have followed the convention of illustrating the direct effect and the indirect effectwith path diagrams and then specifying linear regression models postulated to represent thestructural relationships between the treatment the mediator and the outcome

In Chapter 9 we point out that the approach to defining the indirect effect and the directeffect in terms of path coefficients is often misled by oversimplistic assumptions about thestructural relationships In particular this approach typically overlooks the fact that a treat-ment may generate an impact on the outcome not only by changing the mediator value butalso by changing the mediatorndashoutcome relationship (Judd and Kenny 1981) An examplein Chapter 9 illustrates such a case An experiment randomizes students to either an experi-mental condition which provides them with study materials and encourages them to study fora test or to a control condition which provides neither study materials nor encouragement(Holland 1988 Powers and Swinton 1984) One might hypothesize that encouraging exper-imental group members to study will increase the time they spend studying a focal mediatorin this example which in turn will increase their average test scores One might furtherhypothesize that even without a change in study time providing study materials to experi-mental group members will enable them to study more effectively than they otherwise wouldConsequently the effect of time spent studying might be greater under the experimentalcondition than under the control condition Omitting the treatment-by-mediator interactioneffect will lead to bias in the estimation of the indirect effect and the direct effect

Chapter 11 describes in great detail an evaluation study contrasting a new welfare-to-workprogram emphasizing active participation in the labor force with a traditional program thatguaranteed cash assistance without a mandate to seek employment (Hong Deutsch and Hill2011) Focusing on the psychological well-being of welfare applicants who were singlemothers with preschool-aged children the researchers hypothesized that the treatment wouldincrease employment rate among the welfare recipients and that the treatment-inducedincrease in employment would likely reduce depressive symptoms under the new program

7INTRODUCTION

They also hypothesized that in contrast the same increase in employment was unlikely tohave a psychological impact under the traditional program Hence the treatment-by-mediatorinteraction effect on the outcome was an essential component of the intermediate process

We will show that by defining the indirect effect and the direct effect in terms of potentialoutcomes one can avoid invoking unwarranted assumptions about the unknown structuralrelationships (Holland 1988 Pearl 2001 Robins and Greenland 1992) The potential out-comes framework has further clarified that in a causal mediation theory the treatment and themediator are both conceivably manipulable In the aforementioned example a welfare appli-cant could be assigned at random to either the new program or the traditional one Once havingbeen assigned to one of the programs the individual might or might not be employed due tovarious structural constraints market fluctuations or other random events typically beyondher control

Because the indirect effect and the direct effect are defined in terms of potential outcomesrather than on the basis of any specific regression models it becomes possible to distinguishthe definitions of these causal effects from their identification and estimation The identifica-tion step relates a causal parameter to observable population data For example for individualsassigned at random to an experimental condition their average counterfactual outcome asso-ciated with the control condition is expected to be equal to the average observable outcome ofthose assigned to the control condition Therefore the population average causal effect can beeasily identified in a randomized experiment The estimation step then relates the sample datato the observable population quantities involved in identification while taking into account thedegree of sampling variability

A major challenge in identification is that the mediatorndashoutcome relationship tends tobe confounded by selection even if the treatment is randomized Chapter 9 reviews variousexperimental designs that have been proposed by past researchers for studying causalmediation mechanisms Chapter 10 compares the identification assumptions and the analyticprocedures across a wide range of analytic methods These discussions are followed by anintroduction of the ratio-of-mediator-probability weighting (RMPW) strategy in Chapter 11In comparison with most existing strategies for causal mediation analysis RMPW relies onrelatively fewer identification assumptions and model-based assumptions Chapters 12 and13 will show that this new strategy can be applied broadly with extensions to multilevelexperimental designs and to studies of complex mediation mechanisms involving multiplemediators

113 Spill-over effects of a treatment

It is well known in vaccination research that an unvaccinated person can benefit when mostother people in the local community are vaccinated This is viewed as a spill-over of the treat-ment effect from the vaccinated people to an unvaccinated person Similarly due to socialinteractions among individuals or groups of individuals an intervention received by somemay generate a spill-over impact on others affiliated with the same organization or connectedthrough the same network For example effective policing in one neighborhood may driveoffenders to operate in other neighborhoods In evaluating an innovative community policingprogram researchers found that a neighborhood not assigned to community policing tended tosuffer if the surrounding neighborhoods were assigned to community policing This is aninstance in which a well-intended treatment generates an unintended negative spill-over effectHowever being assigned to community policy was found particularly beneficial when thesurrounding neighborhoods also received the intervention (Verbitsky-Savitz and Raudenbush

8 CAUSALITY IN A SOCIAL WORLD

2012) In another example the impact of a school-based mentoring program targeted atstudents displaying delinquent behaviors may be enhanced for a focal student if his or herat-risk peers are assigned to the mentoring program at the same time

The previous examples challenge the ldquostable unit treatment value assumptionrdquo (SUTVA)that has been invoked in most causal inference studies This assumption states that an indi-vidualrsquos potential response to a treatment depends neither on how the treatment is assignednor on the treatment assignments of other individuals (Rubin 1980 1986) Rubin and othersbelieve that without resorting to SUTVA the causal effect of a treatment becomes hard todefine and that causal inference may become intractable However as Sobel (2006) has illus-trated with the Moving to Opportunity (MTO) experiment that offered housing vouchersenabling low-income families to move to low-poverty neighborhoods there are possible con-sequences of violating SUTVA in estimating the treatment effects on outcomes such as safetyand self-sufficiency Because social interactions among individuals may affect whether onevolunteers to participate in the study whether one moves to a low-poverty neighborhood afterreceiving the housing voucher as well as housing project residentsrsquo subjective perceptions oftheir neighborhoods the program may have a nonzero impact on the potential outcome of theuntreated As a result despite the randomization of treatment assignment the mean differencein the observed outcome between the treated units and the untreated units may be biased forthe average treatment effect Rather the observable quantity is the difference between theeffect of treating some rather than treating none for the treated and the effect of treating somerather than treating none for the untreated the latter being the pure spill-over effect for theuntreated

In making attempts to relax SUTVA researchers have proposed several alternative frame-works that incorporate possible spill-over effects (see Hong and Raudenbush 2013 for areview) Hong (2004) presented a model that involves treatments and treatment settingsA treatment setting for an individual is a local environment constituted by a set of agentsand participants along with their treatment assignments An individualrsquos potential outcomevalue under a given treatment is assumed stable when the treatment setting is fixed the poten-tial outcome may take different values when the treatment setting shifts One may investigatewhether the treatment effect depends on the treatment setting Applying this framework Hongand Raudenbush (2006) examined the effect on a childrsquos academic growth of retaining thechild in kindergarten rather than promoting the child to first grade when a relatively smallproportion of low-achieving peers in the same school are retained as opposed to when a rel-atively large proportion of the peers are retained Hudgens and Halloran (2008) presented arelated framework in which the effect on an individual of the treatment received by this indi-vidual is distinguished from the effect on the individual of the treatment received by others inthe same local community These effects can be identified if communities are assigned atrandom to different treatment assignment strategies (such as retaining a large proportion oflow-achieving students as opposed to retaining a small proportion of such students) and sub-sequently individuals within a community are randomized for treatment Chapter 14 reviewsthese frameworks and discusses identification and estimation strategies

Social contagion may also serve as an important channel through which the effect of anintervention is transmitted For example a school-wide intervention may reduce aggressivebehaviors and thereby improve studentsrsquo psychological well-being by improving the qualityof interpersonal relationships in other classes as well as in onersquos own class This is becausechildren interact not simply with their classmates but also with those from other classes in thehallways or on the playground (VanderWeele et al 2013) In this case the spill-over betweenclasses becomes a part of the mediation mechanism In a study of student mentoring whether

9INTRODUCTION

an individual actually participated in the mentoring program despite the initial treatmentassignment is typically viewed as a primary mediator The proportion of onersquos peers that par-ticipated in the program may act as a second mediator A treatment may also exert its impactpartly through regrouping individuals and thereby changing peer composition (Hong andNomi 2012) Chapter 15 discusses analytic strategies for detecting spill-over as a part ofthe causal mediation mechanism

12 Weighting methods for causal inference

In most behavioral and social science applications major methodological challenges arise dueto the selection of treatments in a quasiexperimental study and the selection of mediator valuesin experimental and quasiexperimental studies Statistical methods most familiar to research-ers in social sciences are often inadequate for causal inferences with regard to multivaluedtreatments moderation mediation and spill-over The book offers a major revision to under-standing of causality in the social world by introducing two complementary weighting stra-tegies both featuring nonparametric approaches to estimating these causal effects The newweighting methods greatly simplify model specifications while enhancing the robustness ofresults by minimizing the reliance on some key assumptions

The propensity score-basedMMWSmethod removes selection bias associated with a largenumber of covariates by equating the pretreatment composition between treatment groups(Hong 2010a 2012 Huang et al 2005) Unlike propensity score matching and stratificationthat are mostly restricted to evaluations of binary treatments the MMWS method is flexiblefor evaluating binary and multivalued treatments by approximating a completely randomizedexperiment In evaluating whether the treatment effects differ across subpopulations definedby individual characteristics or treatment settings researchers may assign weights within eachsubpopulation in order to approximate a randomized block design To investigate whetherone treatment moderates the effect of another concurrent treatment researchers may assignweights to the data to approximate a factorial randomized design The method can also beused to assess whether the effect of an initial treatment is amplified or weakened by a subse-quent treatment or to identify an optimal treatment sequence through approximating a sequen-tial randomized experiment Even though such analyses can similarly be conducted throughinverse-probability-of-treatment weighting (IPTW) that has been increasingly employed inepidemiological research (Hernaacuten Brumback and Robins 2000 Robins Hernaacuten andBrumback 2000) IPTW is known for bias and imprecision in estimation especially whenthe propensity score models are misspecified in their functional forms (Hong 2010a Kangand Schafer 2007 Schafer and Kang 2008 Waernbaum 2012) In contrast the nonparamet-ric MMWS method displays a relatively high level of robustness despite such misspecifica-tions and also gains efficiency as indicated by simulation results (Hong 2010a)

To study causal mediation mechanisms the RMPW method decomposes the total effectof a treatment into an ldquoindirect effectrdquo transmitted through a specific mediator and a ldquodirecteffectrdquo representing unspecified mechanisms In contrast with most existing methods formediation analysis the RMPW-adjusted outcome model is extremely simple and is nonpara-metric in nature It generates estimates of the causal effects along with their sampling errorswhile adjusting for pretreatment covariates that confound the mediatorndashoutcome relationshipsthrough weighting The method applies regardless of the distribution of the outcome the dis-tribution of the mediator or the functional relationship between the outcome and the mediator

10 CAUSALITY IN A SOCIAL WORLD

Page 14: Thumbnail - download.e-bookshelf.de...4.2.4 IPTW for bias removal 84 4.3 MMWS 86 4.3.1 Theoretical rationale 86 ... 8.3.4.2 Growth model in the absence of confounding 200 8.3.4.3 Weighted

1522 Identification when the mediators are not randomized 3981523 Estimation of mediated effects through spill-over 400

153 Definition of mediated effects through spill-over in a multisite trial 4021531 Notation 4021532 Treatment effect mediated by a focal individualrsquos compliance 4041533 Treatment effect mediated by peersrsquo compliance

through spill-over 4041534 Direct effect of individual treatment assignment on the outcome 4051535 Direct effect of peer treatment assignment on the outcome 4051536 Decomposition of the total treatment effect 405

154 Identification and estimation of spill-over effects in a multisite trial 4061541 Identification in an ideal experiment 4071542 Identification when the mediators are not randomized 4091543 Estimation of mediated effects through spill-over 410

15431 Estimating E[Y(1pM(p)Mminus(p))] andE[Y(000Mminus(0))] 410

15432 Estimating E[Y(1p0Mminus(p))] 41115433 Estimating E[Y(1p0Mminus(0))] 41215434 Estimating E[Y(0p0Mminus(0))] 412

155 Consequences of omitting spill-over effects in causal mediation analyses 4121551 Biased inference in a cluster randomized trial 4131552 Biased inference in a multisite randomized trial 4131553 Biased inference of the local average treatment effect 415

156 Quasiexperimental application 416157 Summary 419Appendix 151 Derivation of the weight for estimating the population

average counterfactual outcome E[Y(1 p 0Mminus( p))] 419Appendix 152 Derivation of bias in the ITT effect due to the omission of

spill-over effects 420

Index 423

xiv CONTENTS

Preface

A scientific mind in training seeks comprehensive descriptions of every interesting phenom-enon In such a mind data are to be pieced together for comparisons contrasts and eventuallyinferences that are required for understanding causes and effects that may have led to thephenomenon of interest Yet without disciplined reasoning a causal inference would slip intothe rut of a ldquocasual inferencerdquo as easily as making a typographical error

As a doctoral student thirsting for rigorous methodological training at the University ofMichigan I was very fortunate to be among the first group of students on the campus attendinga causal inference seminar in 2000 jointly taught by Yu Xie from the Department of Sociol-ogy Susan Murphy from the Department of Statistics and Stephen Raudenbush from theSchool of Education The course was unique in both content and pedagogy The instructorsput together a bibliography drawn from the past and current causal inference literature origi-nated from multiple disciplines In the spirit of ldquoreciprocal teachingrdquo the students were delib-erately organized into small groups each representing a mix of disciplinary backgrounds andtook turns to teach the weekly readings under the guidance of a faculty instructor This inval-uable experience revealed to me that the pursuit of causal inference is a multidisciplinaryendeavor In the rest of my doctoral study I continued to be immersed in an extraordinaryinterdisciplinary community in the form of the Quantitative Methodology Program (QMP)seminars led by the above three instructors who were then joined by Ben Hansen(Statistics) Richard Gonzalez (Psychology) Roderick Little (Biostatistics) Jake Bowers(Political Science) and many others I have tried whenever possible to carry the same spiritinto my own teaching of causal inference courses at the University of Toronto and now at theUniversity of Chicago In these courses communicating understandings of causal problemsacross disciplinary boundaries has been particularly challenging but also stimulating and grat-ifying for myself and for students from various departments and schools

Under the supervision of Stephen Raudenbush I took a first stab at the conceptual frame-work of causal inference in my doctoral dissertation by considering peer spill-over in schoolsettings From that point on I have organized my methodological research to tackle some ofthe major obstacles to causal inferences in policy and program evaluations My workaddresses issues including (i) how to conceptualize and evaluate the causal effects of educa-tional treatments when studentsrsquo responses to alternative treatments depend on various fea-tures of the organizational settings including peer composition (ii) how to adjust forselection bias in evaluating the effects of concurrent and consecutive multivalued treatmentsand (iii) how to conceptualize and analyze causal mediation mechanisms My methodological

research has been driven by the substantive interest in understanding the role of socialinstitutionsmdashschools in particularmdashin shaping human development especially during theages of fast growth (eg childhood adolescence and early adulthood) I have shared meth-odological advances with a broad audience in social sciences through applying the causalinference methods to prominent substantive issues such as grade retention within-class group-ing services for English language learners and welfare-to-work strategies These illustrationsare used extensively in this book

Among social scientists awareness of methodological challenges in causal investigationsis higher than ever before In response to this rising demand causal inference is becoming oneof the most productive scholarly fields in the past decade I have had the benefit of followingthe work and exchanging ideas with some fellow methodologists who are constantly contri-buting new thoughts and making breakthroughs These include Daniel Almirall HowardBloom Tom Cook Michael Elliot Ken Frank Ben Hansen James Heckman JenniferHill Martin Huber Kosuke Imai Booil Jo John R Lockwood David MacKinnon DanielMcCaffrey Luke Miratrix Richard Murnane Derek Neal Lindsey Page Judea PearlStephen Raudenbush Sean Reardon Michael Seltzer Youngyun Shin Betsy SinclairMichael Sobel Peter Steiner Elizabeth Stuart Eric Thetchen Tchetchen Tyler VanderWeeleand Kazuo Yamaguchi I am grateful to Larry Hedges who offered me the opportunity of guestediting the Journal of Research in Educational Effectiveness special issue on the statisticalapproaches to studying mediator effects in education research in 2012 Many of the colleagueswhom I mentioned above contributed to the open debate in the special issue by either propos-ing or critically examining a number of innovative methods for studying causal mechanismsI anticipate that many of them are or will soon be writing their own research monographs oncausal inference if they have not already done so Therefore readers of this book are stronglyrecommended to browse past and future titles by these and other authors for alternative per-spectives and approaches My own understanding of course will continue to evolve as well inthe coming years

One has to be ambitious to stay upfront in substantive areas as well as in methodology Thebest strategy apparently is to learn from substantive experts During different phases of mytraining and professional career I have had the opportunities to work with David K CohenBrian Rowan Deborah Ball Carl Corter Janette Pelletier Takako Nomi Esther Geva DavidFrancis Stephanie Jones Joshua Brown and Heather D Hill Many of my substantive insightswere derived from conversations with these mentors and colleagues I am especially gratefulto Bob Granger former president of the William T Grant Foundation who reassured me thatldquoBy staying a bit broad you will learn a lot and many fields will benefit from your workrdquo

Like many other single-authored books this one is built on the generous contributions ofstudents and colleagues who deserve special acknowledgement Excellent research assistancefrom Jonah Deutsch Joshua Gagne Rachel Garrett Yihua Hong Xu Qin Cheng Yang andBing Yu was instrumental in the development and implementation of the new methods pre-sented in this book Richard Congdon a renowned statistical software programmer broughtrevolutionary ideas to interface designs in software development with the aim of facilitatingusersrsquo decision-making in a series of causal analyses RichardMurnane Stephen Raudenbushand Michael Sobel carefully reviewed and critiqued multiple chapters of the manuscript draftAdditional comments and suggestions on earlier drafts came fromHoward Bloom Ken FrankBen Hansen Jennifer Hill Booil Jo Ben Kelcey David MacKinnon and Fan Yang TereseSchwartzman provided valuable assistance in manuscript preparation The writing of thisbook was supported by a Scholars Award from the William T Grant Foundation and an

xvi PREFACE

Institute of Education Sciences (IES) Statistical and Research Methodology in EducationGrant from the US Department of Education All the errors in the published form are ofcourse the sole responsibility of the author

Project Editors Richard Davis Prachi Sinha Sahay and Liz Wingett and Assistant EditorHeather Kay at Wiley and project Managers P Jayapriya and R Jayavel at SPi Global havebeen wonderful to work with throughout the manuscript preparation and final production ofthe book Debbie Jupe Commissioning Editor at Wiley was remarkably effective in initiatingthe contact with me and in organizing anonymous reviews of the book proposal and of themanuscript These constructive reviews have shaped the book in important ways I cannotagree more with one of the reviewers that ldquocausality cannot be established on a pure statisticalgroundrdquo The book therefore highlights the importance of substantive knowledge and researchdesigns and places a great emphasis on clarifying and evaluating assumptions required foridentifying causal effects in the context of each application Much caution is raised againstpossible misusage of statistical techniques for analyzing causal relationships especiallywhen data are inadequate Yet I maintain that improved statistical procedures along withimproved research designs would greatly enhance our ability in attempt to empiricallyexamine well-articulated causal theories

xviiPREFACE

Part I

OVERVIEW

1

Introduction

According to an ancient Chinese fable a farmer who was eager to help his crops grow wentinto his field and pulled each seedling upward After exhausting himself with the work heannounced to his family that they were going to have a good harvest only to find the nextmorning that the plants had wilted and died Readers with minimal agricultural knowledgemay immediately point out the following the farmerrsquos intervention theory was based on acorrect observation that crops that grow taller tend to produce more yield Yet his hypothesisreflects a false understanding of the cause and the effectmdashthat seedlings pulled to be tallerwould yield as much as seedlings thriving on their own

In their classic Design and Analysis of Experiments Hinkelmann and Kempthorne (1994updated version of Kempthorne 1952) discussed two types of science descriptive science andthe development of theory These two types of science are interrelated in the following senseobservations of an event and other related events often selected and classified for descriptionby scientists naturally lead to one or more explanations that we call ldquotheoretical hypothesesrdquowhich are then screened and falsified by means of further observations experimentation andanalyses (Popper 1963) The experiment of pulling seedlings to be taller was costly but didserve the purpose of advancing this farmerrsquos knowledge of ldquowhat does not workrdquo To developa successful intervention in this case would require a series of empirical tests of explicittheories identifying potential contributors to crop growth This iterative process graduallydeepens our knowledge of the relationships between supposed causes and effectsmdashthatis causalitymdashand may eventually increase the success of agricultural medical and socialinterventions

11 Concepts of moderation mediation and spill-over

Although the story of the ancient farmer is fictitious numerous examples can be found in thereal world in which well-intended interventions fail to produce the intended benefits or inmany cases even lead to unintended consequences ldquoInterventionsrdquo and ldquotreatmentsrdquo usedinterchangeably in this book broadly refer to actions taken by agents or circumstances expe-rienced by an individual or groups of individuals Interventions are regularly seen in educa-tion physical and mental health social services business politics and law enforcement In an

Causality in a Social World Moderation Meditation and Spill-over First Edition Guanglei Hongcopy 2015 John Wiley amp Sons Ltd Published 2015 by John Wiley amp Sons Ltd

education intervention for example teachers are typically the agents who deliver a treatmentto students while the impact of the treatment on student outcomes is of ultimate causalinterest Some educational practices such as ldquoteaching to the testrdquo have been criticized tobe nearly as counterproductive as the attempt of helping seedlings grow by pulling themupward ldquoInterventionsrdquo and ldquotreatmentsrdquo under consideration do not exclude undesiredexperiences such as exposure to poverty abuse crime or bereavement A treatment plannedor unplanned becomes a focus of research if there are theoretical reasons to anticipate itsimpact positive or negative on the well-being of individuals who are embedded in socialsettings including families classrooms schools neighborhoods and workplaces

In social science research in general and in policy and program evaluations in particularquestions concerning whether an intervention works and if so which version of the interven-tion works for whom under what conditions and why are key to the advancement of scien-tific and practical knowledge Although most empirical investigations in the social sciencesconcentrate on the average effect of a treatment for a specific population as opposed to theabsence of such a treatment (ie the control condition) in-depth theoretical reasoning withregard to how the causal effect is generated substantiated by compelling empirical evidenceis crucial for advancing scientific understanding

First when there are multiple versions or different dosages of the treatment or when thereare multiple versions of the control condition a binary divide between ldquothe treatmentrdquo andldquothe controlrdquo may not be as informative as fine-grained comparisons across for exampleldquotreatment version Ardquo ldquotreatment version Brdquo ldquocontrol version Ardquo and ldquocontrol versionBrdquo For example expanding the federally funded Head Start program to poor children isexpected to generate a greater benefit when few early childhood education alternatives areavailable (call it ldquocontrol version Ardquo) than when there is an abundance of alternatives includ-ing state-sponsored preschool programs (call it ldquocontrol version Brdquo)

Second the effect of an intervention will likely vary among individuals or across socialsettings A famous example comes from medical research the well-publicized cardiovascularbenefits of initiating estrogen therapy during menopause were contradicted later by experi-mental findings that the same therapy increased postmenopausal womenrsquos risk for heartattacks The effect of an intervention may also depend on the provision of some other con-current or subsequent interventions Such heterogeneous effects are often characterized asmoderated effects in the literature

Third alternative theories may provide competing explanations for the causal mechan-isms that is the processes through which the intervention produces its effect A theoreticalconstruct characterizing the hypothesized intermediate process is called a mediator of theintervention effect The fictitious farmer never developed an elaborate theory as to whatcaused some seedlings to surpass others in growth Once scientists revealed the causal rela-tionship between access to chemical nutrients in soil and plant growth wide applications ofchemically synthesized fertilizers finally led to a major increase in crop production

Finally it is well known in agricultural experiments that a new type of fertilizer applied toone plot may spill-over to the next plot Because social connections among individuals areprevalent within organizations or through networks an individualrsquos response to the treatmentmay similarly depend on the treatment for other individuals in the same social setting whichmay lead to possible spill-overs of intervention effects among individual human beings

Answering questions with regard to moderation mediation and spill-over poses majorconceptual and analytic challenges To date psychological research often presents well-articulated theories of causal mechanisms relating stimuli to responses Yet researchers often

4 CAUSALITY IN A SOCIAL WORLD

lack rigorous analytic strategies for empirically screening competing theories explaining theobserved effect Sociologists have keen interest in the spill-over of treatment effects transmit-ted through social interactions yet have produced limited evidence quantifying such effectsAs many have pointed out in general terminological ambiguity and conceptual confusionhave been prevalent in the published applied research (Holmbeck 1997 Kraemer et al2002 2008)

A new framework for conceptualizing moderated mediated and spill-over effects hasemerged relatively recently in the statistics and econometrics literature on causal inference(eg Abbring and Heckman 2007 Frangakis and Rubin 2002 Heckman and Vytlacil2007a b Holland 1988 Hong and Raudenbush 2006 Hudgens and Halloran 2008Jo 2008 Pearl 2001 Robins and Greenland 1992 Sobel 2008) The potential for furtherconceptual and methodological development and for broad applications in the field of behav-ioral and social sciences promises to greatly advance the empirical basis of our knowledgeabout causality in the social world

This book clarifies theoretical concepts and introduces innovative statistical strategies forinvestigating the average effects of multivalued treatments moderated treatment effectsmediated treatment effects and spill-over effects in experimental or quasiexperimental dataDefining individual-specific and population average treatment effects in terms of potentialoutcomes the book relates the mathematical forms to the substantive meanings of moderatedmediated and spill-over effects in the context of application examples It also explicates andevaluates identification assumptions and contrasts innovative statistical strategies with con-ventional analytic methods

111 Moderated treatment effects

It is hard to accept the assumption that a treatment would produce the same impact for everyindividual in every possible circumstance Understanding the heterogeneity of treatmenteffects therefore is key to the development of causal theories For example some studiesreported that estrogen therapy improved cardiovascular health among women who initiatedits use during menopause According to a series of other studies however the use of estrogentherapy increased postmenopausal womenrsquos risk for heart attacks (Grodstein et al 19962000 Writing Group for the Womenrsquos Health Initiative Investigators 2002) The sharp con-trast of findings from these studies led to the hypothesis that age of initiation moderates theeffect of estrogen therapy on womenrsquos health (Manson and Bassuk 2007 Rossouw et al2007) Revelations of the moderated causal relationship greatly enrich theoretical understand-ing and in this case directly inform clinical practice

In another example dividing students by ability into small groups for reading instructionhas been controversial for decades Many believed that the practice benefits high-ability stu-dents at the expense of their low-ability peers and exacerbates educational inequality (Grantand Rothenberg 1986 Rowan and Miracle 1983 Trimble and Sinclair 1987) Recentevidence has shown that first of all the effect of ability grouping depends on a number ofconditions including whether enough time is allocated to reading instruction and how wellthe class can be managed Hence instructional time and class management are among theadditional moderators to consider for determining the merits of ability grouping (Hong andHong 2009 Hong et al 2012a b) Moreover further investigations have generated evidencethat contradicts the long-held belief that grouping undermines the interests of low-abilitystudents Quite the contrary researchers have found that grouping is beneficial for students

5INTRODUCTION

with low or medium prior abilitymdashif adequate time is provided for instruction and if the classis well managed On the other hand ability grouping appears to have a minimal impact onhigh-ability studentsrsquo literacy These results were derived from kindergarten data and maynot hold for math instruction and for higher grade levels Replications with different subpo-pulations and in different contexts enable researchers to assess the generalizability of a theory

In a third example one may hypothesize that a student is unlikely to learn algebra well inninth grade without a solid foundation in eighth-grade prealgebra One may also argue that theprogress a student made in learning eighth-grade prealgebra may not be sustained without thefurther enhancement of taking algebra in ninth grade In other words the earlier treatment(prealgebra in eighth grade) and the later treatment (algebra in ninth grade) may reinforceone other each moderates the effect of the other treatment on the final outcome (math achieve-ment at the end of ninth grade) As Hong and Raudenbush (2008) argued the cumulativeeffect of two treatments in a well-aligned sequence may exceed the sum of the benefits oftwo single-year treatments Similarly experiencing two consecutive years of inferior treat-ments such as encountering an incompetent math teacher who dampens a studentrsquos self-efficacy in math learning in eighth grade and then again in ninth grade could do more damagethan the sum of the effect of encountering an incompetent teacher only in eighth grade and theeffect of having such a teacher only in ninth grade

There has been a great amount of conceptual confusion regarding how a moderatorrelates to the treatment and the outcome On one hand many researchers have used the termsldquomoderatorsrdquo and ldquomediatorsrdquo interchangeably without understanding the crucial distinctionbetween the two An overcorrective attempt on the other hand has led to the arbitrary rec-ommendation that a moderator must occur prior to the treatment and be minimally associatedwith the treatment (James and Brett 1984 Kraemer et al 2001 2008) In Chapter 6 weclarify that in essence the causal effect of the treatment on the outcome may depend onthe moderator value A moderator can be a subpopulation identifier a contextual character-istic a concurrent treatment or a preceding or succeeding treatment In the earlier examplesage of estrogen therapy initiation and a studentrsquos prior ability are subpopulation identifiers themanageability of a class which reflects both teacher skills and peer behaviors characterizes thecontext literacy instruction time is a concurrent treatment while prealgebra is a precedingtreatment for algebra A moderator does not have to occur prior to the treatment and doesnot have to be independent of the treatment

Once a moderated causal relationship has been defined in terms of potential outcomes theresearcher then chooses an appropriate experimental design for testing the moderation theoryThe assumptions required for identifying the moderated causal relationships differ across dif-ferent designs and have implications for analyzing experimental and quasiexperimental dataRandomized block designs are suitable for examining individual or contextual characteristicsas potential moderators factorial designs enable one to determine the joint effects of two ormore concurrent treatments and sequential randomized designs are ideal for assessing thecumulative effects of consecutive treatments Multisite randomized trials constitute anadditional type of designs in which the experimental sites are often deliberately sampled torepresent a population of geographical locations or social organizations Site membershipscan be viewed as implicit moderators that summarize a host of features of a local environmentReplications of a treatment over multiple sites allow one to quantify the heterogeneity of treat-ment effects across the sites Chapters 7 and 8 are focused on statistical methods for moder-ation analyses with quasiexperimental data In particular these chapters demonstrate througha number of application examples how a nonparametric marginal mean weighting through

6 CAUSALITY IN A SOCIAL WORLD

stratification (MMWS) method can overcome some important limitations of other existingmethods

Moderation questions are not restricted to treatmentndashoutcome relationships Rather theyare prevalent in investigations of mediation and spill-over This is because heterogeneity oftreatment effects may be explained by the variation in causal mediation mechanisms acrosssubpopulations and contexts It is also because the treatment effect for a focal individualmay depend on whether there is spill-over from other individuals through social interactionsYet similar to the confusion around the concept of moderation among applied researchersthere has been a great amount of misunderstanding with regard to mediation

112 Mediated treatment effects

Questions about mediation are at the core of nearly every scientific theory In its simplestform a theory explains why treatment Z causes outcome Y by hypothesizing an intermediateprocess involving at least one mediatorM that could have been changed by the treatment andcould subsequently have an impact on the outcome A causal mediation analysis then decom-poses the total effect of the treatment on the outcome into two parts the effect transmittedthrough the hypothesized mediator called the indirect effect and the difference betweenthe total effect and the indirect effect called the direct effect The latter represents the treat-ment effect channeled through other unspecified pathways (Alwin and Hauser 1975 Baronand Kenny 1986 Duncan 1966 Shadish and Sweeney 1991) Most researchers in socialsciences have followed the convention of illustrating the direct effect and the indirect effectwith path diagrams and then specifying linear regression models postulated to represent thestructural relationships between the treatment the mediator and the outcome

In Chapter 9 we point out that the approach to defining the indirect effect and the directeffect in terms of path coefficients is often misled by oversimplistic assumptions about thestructural relationships In particular this approach typically overlooks the fact that a treat-ment may generate an impact on the outcome not only by changing the mediator value butalso by changing the mediatorndashoutcome relationship (Judd and Kenny 1981) An examplein Chapter 9 illustrates such a case An experiment randomizes students to either an experi-mental condition which provides them with study materials and encourages them to study fora test or to a control condition which provides neither study materials nor encouragement(Holland 1988 Powers and Swinton 1984) One might hypothesize that encouraging exper-imental group members to study will increase the time they spend studying a focal mediatorin this example which in turn will increase their average test scores One might furtherhypothesize that even without a change in study time providing study materials to experi-mental group members will enable them to study more effectively than they otherwise wouldConsequently the effect of time spent studying might be greater under the experimentalcondition than under the control condition Omitting the treatment-by-mediator interactioneffect will lead to bias in the estimation of the indirect effect and the direct effect

Chapter 11 describes in great detail an evaluation study contrasting a new welfare-to-workprogram emphasizing active participation in the labor force with a traditional program thatguaranteed cash assistance without a mandate to seek employment (Hong Deutsch and Hill2011) Focusing on the psychological well-being of welfare applicants who were singlemothers with preschool-aged children the researchers hypothesized that the treatment wouldincrease employment rate among the welfare recipients and that the treatment-inducedincrease in employment would likely reduce depressive symptoms under the new program

7INTRODUCTION

They also hypothesized that in contrast the same increase in employment was unlikely tohave a psychological impact under the traditional program Hence the treatment-by-mediatorinteraction effect on the outcome was an essential component of the intermediate process

We will show that by defining the indirect effect and the direct effect in terms of potentialoutcomes one can avoid invoking unwarranted assumptions about the unknown structuralrelationships (Holland 1988 Pearl 2001 Robins and Greenland 1992) The potential out-comes framework has further clarified that in a causal mediation theory the treatment and themediator are both conceivably manipulable In the aforementioned example a welfare appli-cant could be assigned at random to either the new program or the traditional one Once havingbeen assigned to one of the programs the individual might or might not be employed due tovarious structural constraints market fluctuations or other random events typically beyondher control

Because the indirect effect and the direct effect are defined in terms of potential outcomesrather than on the basis of any specific regression models it becomes possible to distinguishthe definitions of these causal effects from their identification and estimation The identifica-tion step relates a causal parameter to observable population data For example for individualsassigned at random to an experimental condition their average counterfactual outcome asso-ciated with the control condition is expected to be equal to the average observable outcome ofthose assigned to the control condition Therefore the population average causal effect can beeasily identified in a randomized experiment The estimation step then relates the sample datato the observable population quantities involved in identification while taking into account thedegree of sampling variability

A major challenge in identification is that the mediatorndashoutcome relationship tends tobe confounded by selection even if the treatment is randomized Chapter 9 reviews variousexperimental designs that have been proposed by past researchers for studying causalmediation mechanisms Chapter 10 compares the identification assumptions and the analyticprocedures across a wide range of analytic methods These discussions are followed by anintroduction of the ratio-of-mediator-probability weighting (RMPW) strategy in Chapter 11In comparison with most existing strategies for causal mediation analysis RMPW relies onrelatively fewer identification assumptions and model-based assumptions Chapters 12 and13 will show that this new strategy can be applied broadly with extensions to multilevelexperimental designs and to studies of complex mediation mechanisms involving multiplemediators

113 Spill-over effects of a treatment

It is well known in vaccination research that an unvaccinated person can benefit when mostother people in the local community are vaccinated This is viewed as a spill-over of the treat-ment effect from the vaccinated people to an unvaccinated person Similarly due to socialinteractions among individuals or groups of individuals an intervention received by somemay generate a spill-over impact on others affiliated with the same organization or connectedthrough the same network For example effective policing in one neighborhood may driveoffenders to operate in other neighborhoods In evaluating an innovative community policingprogram researchers found that a neighborhood not assigned to community policing tended tosuffer if the surrounding neighborhoods were assigned to community policing This is aninstance in which a well-intended treatment generates an unintended negative spill-over effectHowever being assigned to community policy was found particularly beneficial when thesurrounding neighborhoods also received the intervention (Verbitsky-Savitz and Raudenbush

8 CAUSALITY IN A SOCIAL WORLD

2012) In another example the impact of a school-based mentoring program targeted atstudents displaying delinquent behaviors may be enhanced for a focal student if his or herat-risk peers are assigned to the mentoring program at the same time

The previous examples challenge the ldquostable unit treatment value assumptionrdquo (SUTVA)that has been invoked in most causal inference studies This assumption states that an indi-vidualrsquos potential response to a treatment depends neither on how the treatment is assignednor on the treatment assignments of other individuals (Rubin 1980 1986) Rubin and othersbelieve that without resorting to SUTVA the causal effect of a treatment becomes hard todefine and that causal inference may become intractable However as Sobel (2006) has illus-trated with the Moving to Opportunity (MTO) experiment that offered housing vouchersenabling low-income families to move to low-poverty neighborhoods there are possible con-sequences of violating SUTVA in estimating the treatment effects on outcomes such as safetyand self-sufficiency Because social interactions among individuals may affect whether onevolunteers to participate in the study whether one moves to a low-poverty neighborhood afterreceiving the housing voucher as well as housing project residentsrsquo subjective perceptions oftheir neighborhoods the program may have a nonzero impact on the potential outcome of theuntreated As a result despite the randomization of treatment assignment the mean differencein the observed outcome between the treated units and the untreated units may be biased forthe average treatment effect Rather the observable quantity is the difference between theeffect of treating some rather than treating none for the treated and the effect of treating somerather than treating none for the untreated the latter being the pure spill-over effect for theuntreated

In making attempts to relax SUTVA researchers have proposed several alternative frame-works that incorporate possible spill-over effects (see Hong and Raudenbush 2013 for areview) Hong (2004) presented a model that involves treatments and treatment settingsA treatment setting for an individual is a local environment constituted by a set of agentsand participants along with their treatment assignments An individualrsquos potential outcomevalue under a given treatment is assumed stable when the treatment setting is fixed the poten-tial outcome may take different values when the treatment setting shifts One may investigatewhether the treatment effect depends on the treatment setting Applying this framework Hongand Raudenbush (2006) examined the effect on a childrsquos academic growth of retaining thechild in kindergarten rather than promoting the child to first grade when a relatively smallproportion of low-achieving peers in the same school are retained as opposed to when a rel-atively large proportion of the peers are retained Hudgens and Halloran (2008) presented arelated framework in which the effect on an individual of the treatment received by this indi-vidual is distinguished from the effect on the individual of the treatment received by others inthe same local community These effects can be identified if communities are assigned atrandom to different treatment assignment strategies (such as retaining a large proportion oflow-achieving students as opposed to retaining a small proportion of such students) and sub-sequently individuals within a community are randomized for treatment Chapter 14 reviewsthese frameworks and discusses identification and estimation strategies

Social contagion may also serve as an important channel through which the effect of anintervention is transmitted For example a school-wide intervention may reduce aggressivebehaviors and thereby improve studentsrsquo psychological well-being by improving the qualityof interpersonal relationships in other classes as well as in onersquos own class This is becausechildren interact not simply with their classmates but also with those from other classes in thehallways or on the playground (VanderWeele et al 2013) In this case the spill-over betweenclasses becomes a part of the mediation mechanism In a study of student mentoring whether

9INTRODUCTION

an individual actually participated in the mentoring program despite the initial treatmentassignment is typically viewed as a primary mediator The proportion of onersquos peers that par-ticipated in the program may act as a second mediator A treatment may also exert its impactpartly through regrouping individuals and thereby changing peer composition (Hong andNomi 2012) Chapter 15 discusses analytic strategies for detecting spill-over as a part ofthe causal mediation mechanism

12 Weighting methods for causal inference

In most behavioral and social science applications major methodological challenges arise dueto the selection of treatments in a quasiexperimental study and the selection of mediator valuesin experimental and quasiexperimental studies Statistical methods most familiar to research-ers in social sciences are often inadequate for causal inferences with regard to multivaluedtreatments moderation mediation and spill-over The book offers a major revision to under-standing of causality in the social world by introducing two complementary weighting stra-tegies both featuring nonparametric approaches to estimating these causal effects The newweighting methods greatly simplify model specifications while enhancing the robustness ofresults by minimizing the reliance on some key assumptions

The propensity score-basedMMWSmethod removes selection bias associated with a largenumber of covariates by equating the pretreatment composition between treatment groups(Hong 2010a 2012 Huang et al 2005) Unlike propensity score matching and stratificationthat are mostly restricted to evaluations of binary treatments the MMWS method is flexiblefor evaluating binary and multivalued treatments by approximating a completely randomizedexperiment In evaluating whether the treatment effects differ across subpopulations definedby individual characteristics or treatment settings researchers may assign weights within eachsubpopulation in order to approximate a randomized block design To investigate whetherone treatment moderates the effect of another concurrent treatment researchers may assignweights to the data to approximate a factorial randomized design The method can also beused to assess whether the effect of an initial treatment is amplified or weakened by a subse-quent treatment or to identify an optimal treatment sequence through approximating a sequen-tial randomized experiment Even though such analyses can similarly be conducted throughinverse-probability-of-treatment weighting (IPTW) that has been increasingly employed inepidemiological research (Hernaacuten Brumback and Robins 2000 Robins Hernaacuten andBrumback 2000) IPTW is known for bias and imprecision in estimation especially whenthe propensity score models are misspecified in their functional forms (Hong 2010a Kangand Schafer 2007 Schafer and Kang 2008 Waernbaum 2012) In contrast the nonparamet-ric MMWS method displays a relatively high level of robustness despite such misspecifica-tions and also gains efficiency as indicated by simulation results (Hong 2010a)

To study causal mediation mechanisms the RMPW method decomposes the total effectof a treatment into an ldquoindirect effectrdquo transmitted through a specific mediator and a ldquodirecteffectrdquo representing unspecified mechanisms In contrast with most existing methods formediation analysis the RMPW-adjusted outcome model is extremely simple and is nonpara-metric in nature It generates estimates of the causal effects along with their sampling errorswhile adjusting for pretreatment covariates that confound the mediatorndashoutcome relationshipsthrough weighting The method applies regardless of the distribution of the outcome the dis-tribution of the mediator or the functional relationship between the outcome and the mediator

10 CAUSALITY IN A SOCIAL WORLD

Page 15: Thumbnail - download.e-bookshelf.de...4.2.4 IPTW for bias removal 84 4.3 MMWS 86 4.3.1 Theoretical rationale 86 ... 8.3.4.2 Growth model in the absence of confounding 200 8.3.4.3 Weighted

Preface

A scientific mind in training seeks comprehensive descriptions of every interesting phenom-enon In such a mind data are to be pieced together for comparisons contrasts and eventuallyinferences that are required for understanding causes and effects that may have led to thephenomenon of interest Yet without disciplined reasoning a causal inference would slip intothe rut of a ldquocasual inferencerdquo as easily as making a typographical error

As a doctoral student thirsting for rigorous methodological training at the University ofMichigan I was very fortunate to be among the first group of students on the campus attendinga causal inference seminar in 2000 jointly taught by Yu Xie from the Department of Sociol-ogy Susan Murphy from the Department of Statistics and Stephen Raudenbush from theSchool of Education The course was unique in both content and pedagogy The instructorsput together a bibliography drawn from the past and current causal inference literature origi-nated from multiple disciplines In the spirit of ldquoreciprocal teachingrdquo the students were delib-erately organized into small groups each representing a mix of disciplinary backgrounds andtook turns to teach the weekly readings under the guidance of a faculty instructor This inval-uable experience revealed to me that the pursuit of causal inference is a multidisciplinaryendeavor In the rest of my doctoral study I continued to be immersed in an extraordinaryinterdisciplinary community in the form of the Quantitative Methodology Program (QMP)seminars led by the above three instructors who were then joined by Ben Hansen(Statistics) Richard Gonzalez (Psychology) Roderick Little (Biostatistics) Jake Bowers(Political Science) and many others I have tried whenever possible to carry the same spiritinto my own teaching of causal inference courses at the University of Toronto and now at theUniversity of Chicago In these courses communicating understandings of causal problemsacross disciplinary boundaries has been particularly challenging but also stimulating and grat-ifying for myself and for students from various departments and schools

Under the supervision of Stephen Raudenbush I took a first stab at the conceptual frame-work of causal inference in my doctoral dissertation by considering peer spill-over in schoolsettings From that point on I have organized my methodological research to tackle some ofthe major obstacles to causal inferences in policy and program evaluations My workaddresses issues including (i) how to conceptualize and evaluate the causal effects of educa-tional treatments when studentsrsquo responses to alternative treatments depend on various fea-tures of the organizational settings including peer composition (ii) how to adjust forselection bias in evaluating the effects of concurrent and consecutive multivalued treatmentsand (iii) how to conceptualize and analyze causal mediation mechanisms My methodological

research has been driven by the substantive interest in understanding the role of socialinstitutionsmdashschools in particularmdashin shaping human development especially during theages of fast growth (eg childhood adolescence and early adulthood) I have shared meth-odological advances with a broad audience in social sciences through applying the causalinference methods to prominent substantive issues such as grade retention within-class group-ing services for English language learners and welfare-to-work strategies These illustrationsare used extensively in this book

Among social scientists awareness of methodological challenges in causal investigationsis higher than ever before In response to this rising demand causal inference is becoming oneof the most productive scholarly fields in the past decade I have had the benefit of followingthe work and exchanging ideas with some fellow methodologists who are constantly contri-buting new thoughts and making breakthroughs These include Daniel Almirall HowardBloom Tom Cook Michael Elliot Ken Frank Ben Hansen James Heckman JenniferHill Martin Huber Kosuke Imai Booil Jo John R Lockwood David MacKinnon DanielMcCaffrey Luke Miratrix Richard Murnane Derek Neal Lindsey Page Judea PearlStephen Raudenbush Sean Reardon Michael Seltzer Youngyun Shin Betsy SinclairMichael Sobel Peter Steiner Elizabeth Stuart Eric Thetchen Tchetchen Tyler VanderWeeleand Kazuo Yamaguchi I am grateful to Larry Hedges who offered me the opportunity of guestediting the Journal of Research in Educational Effectiveness special issue on the statisticalapproaches to studying mediator effects in education research in 2012 Many of the colleagueswhom I mentioned above contributed to the open debate in the special issue by either propos-ing or critically examining a number of innovative methods for studying causal mechanismsI anticipate that many of them are or will soon be writing their own research monographs oncausal inference if they have not already done so Therefore readers of this book are stronglyrecommended to browse past and future titles by these and other authors for alternative per-spectives and approaches My own understanding of course will continue to evolve as well inthe coming years

One has to be ambitious to stay upfront in substantive areas as well as in methodology Thebest strategy apparently is to learn from substantive experts During different phases of mytraining and professional career I have had the opportunities to work with David K CohenBrian Rowan Deborah Ball Carl Corter Janette Pelletier Takako Nomi Esther Geva DavidFrancis Stephanie Jones Joshua Brown and Heather D Hill Many of my substantive insightswere derived from conversations with these mentors and colleagues I am especially gratefulto Bob Granger former president of the William T Grant Foundation who reassured me thatldquoBy staying a bit broad you will learn a lot and many fields will benefit from your workrdquo

Like many other single-authored books this one is built on the generous contributions ofstudents and colleagues who deserve special acknowledgement Excellent research assistancefrom Jonah Deutsch Joshua Gagne Rachel Garrett Yihua Hong Xu Qin Cheng Yang andBing Yu was instrumental in the development and implementation of the new methods pre-sented in this book Richard Congdon a renowned statistical software programmer broughtrevolutionary ideas to interface designs in software development with the aim of facilitatingusersrsquo decision-making in a series of causal analyses RichardMurnane Stephen Raudenbushand Michael Sobel carefully reviewed and critiqued multiple chapters of the manuscript draftAdditional comments and suggestions on earlier drafts came fromHoward Bloom Ken FrankBen Hansen Jennifer Hill Booil Jo Ben Kelcey David MacKinnon and Fan Yang TereseSchwartzman provided valuable assistance in manuscript preparation The writing of thisbook was supported by a Scholars Award from the William T Grant Foundation and an

xvi PREFACE

Institute of Education Sciences (IES) Statistical and Research Methodology in EducationGrant from the US Department of Education All the errors in the published form are ofcourse the sole responsibility of the author

Project Editors Richard Davis Prachi Sinha Sahay and Liz Wingett and Assistant EditorHeather Kay at Wiley and project Managers P Jayapriya and R Jayavel at SPi Global havebeen wonderful to work with throughout the manuscript preparation and final production ofthe book Debbie Jupe Commissioning Editor at Wiley was remarkably effective in initiatingthe contact with me and in organizing anonymous reviews of the book proposal and of themanuscript These constructive reviews have shaped the book in important ways I cannotagree more with one of the reviewers that ldquocausality cannot be established on a pure statisticalgroundrdquo The book therefore highlights the importance of substantive knowledge and researchdesigns and places a great emphasis on clarifying and evaluating assumptions required foridentifying causal effects in the context of each application Much caution is raised againstpossible misusage of statistical techniques for analyzing causal relationships especiallywhen data are inadequate Yet I maintain that improved statistical procedures along withimproved research designs would greatly enhance our ability in attempt to empiricallyexamine well-articulated causal theories

xviiPREFACE

Part I

OVERVIEW

1

Introduction

According to an ancient Chinese fable a farmer who was eager to help his crops grow wentinto his field and pulled each seedling upward After exhausting himself with the work heannounced to his family that they were going to have a good harvest only to find the nextmorning that the plants had wilted and died Readers with minimal agricultural knowledgemay immediately point out the following the farmerrsquos intervention theory was based on acorrect observation that crops that grow taller tend to produce more yield Yet his hypothesisreflects a false understanding of the cause and the effectmdashthat seedlings pulled to be tallerwould yield as much as seedlings thriving on their own

In their classic Design and Analysis of Experiments Hinkelmann and Kempthorne (1994updated version of Kempthorne 1952) discussed two types of science descriptive science andthe development of theory These two types of science are interrelated in the following senseobservations of an event and other related events often selected and classified for descriptionby scientists naturally lead to one or more explanations that we call ldquotheoretical hypothesesrdquowhich are then screened and falsified by means of further observations experimentation andanalyses (Popper 1963) The experiment of pulling seedlings to be taller was costly but didserve the purpose of advancing this farmerrsquos knowledge of ldquowhat does not workrdquo To developa successful intervention in this case would require a series of empirical tests of explicittheories identifying potential contributors to crop growth This iterative process graduallydeepens our knowledge of the relationships between supposed causes and effectsmdashthatis causalitymdashand may eventually increase the success of agricultural medical and socialinterventions

11 Concepts of moderation mediation and spill-over

Although the story of the ancient farmer is fictitious numerous examples can be found in thereal world in which well-intended interventions fail to produce the intended benefits or inmany cases even lead to unintended consequences ldquoInterventionsrdquo and ldquotreatmentsrdquo usedinterchangeably in this book broadly refer to actions taken by agents or circumstances expe-rienced by an individual or groups of individuals Interventions are regularly seen in educa-tion physical and mental health social services business politics and law enforcement In an

Causality in a Social World Moderation Meditation and Spill-over First Edition Guanglei Hongcopy 2015 John Wiley amp Sons Ltd Published 2015 by John Wiley amp Sons Ltd

education intervention for example teachers are typically the agents who deliver a treatmentto students while the impact of the treatment on student outcomes is of ultimate causalinterest Some educational practices such as ldquoteaching to the testrdquo have been criticized tobe nearly as counterproductive as the attempt of helping seedlings grow by pulling themupward ldquoInterventionsrdquo and ldquotreatmentsrdquo under consideration do not exclude undesiredexperiences such as exposure to poverty abuse crime or bereavement A treatment plannedor unplanned becomes a focus of research if there are theoretical reasons to anticipate itsimpact positive or negative on the well-being of individuals who are embedded in socialsettings including families classrooms schools neighborhoods and workplaces

In social science research in general and in policy and program evaluations in particularquestions concerning whether an intervention works and if so which version of the interven-tion works for whom under what conditions and why are key to the advancement of scien-tific and practical knowledge Although most empirical investigations in the social sciencesconcentrate on the average effect of a treatment for a specific population as opposed to theabsence of such a treatment (ie the control condition) in-depth theoretical reasoning withregard to how the causal effect is generated substantiated by compelling empirical evidenceis crucial for advancing scientific understanding

First when there are multiple versions or different dosages of the treatment or when thereare multiple versions of the control condition a binary divide between ldquothe treatmentrdquo andldquothe controlrdquo may not be as informative as fine-grained comparisons across for exampleldquotreatment version Ardquo ldquotreatment version Brdquo ldquocontrol version Ardquo and ldquocontrol versionBrdquo For example expanding the federally funded Head Start program to poor children isexpected to generate a greater benefit when few early childhood education alternatives areavailable (call it ldquocontrol version Ardquo) than when there is an abundance of alternatives includ-ing state-sponsored preschool programs (call it ldquocontrol version Brdquo)

Second the effect of an intervention will likely vary among individuals or across socialsettings A famous example comes from medical research the well-publicized cardiovascularbenefits of initiating estrogen therapy during menopause were contradicted later by experi-mental findings that the same therapy increased postmenopausal womenrsquos risk for heartattacks The effect of an intervention may also depend on the provision of some other con-current or subsequent interventions Such heterogeneous effects are often characterized asmoderated effects in the literature

Third alternative theories may provide competing explanations for the causal mechan-isms that is the processes through which the intervention produces its effect A theoreticalconstruct characterizing the hypothesized intermediate process is called a mediator of theintervention effect The fictitious farmer never developed an elaborate theory as to whatcaused some seedlings to surpass others in growth Once scientists revealed the causal rela-tionship between access to chemical nutrients in soil and plant growth wide applications ofchemically synthesized fertilizers finally led to a major increase in crop production

Finally it is well known in agricultural experiments that a new type of fertilizer applied toone plot may spill-over to the next plot Because social connections among individuals areprevalent within organizations or through networks an individualrsquos response to the treatmentmay similarly depend on the treatment for other individuals in the same social setting whichmay lead to possible spill-overs of intervention effects among individual human beings

Answering questions with regard to moderation mediation and spill-over poses majorconceptual and analytic challenges To date psychological research often presents well-articulated theories of causal mechanisms relating stimuli to responses Yet researchers often

4 CAUSALITY IN A SOCIAL WORLD

lack rigorous analytic strategies for empirically screening competing theories explaining theobserved effect Sociologists have keen interest in the spill-over of treatment effects transmit-ted through social interactions yet have produced limited evidence quantifying such effectsAs many have pointed out in general terminological ambiguity and conceptual confusionhave been prevalent in the published applied research (Holmbeck 1997 Kraemer et al2002 2008)

A new framework for conceptualizing moderated mediated and spill-over effects hasemerged relatively recently in the statistics and econometrics literature on causal inference(eg Abbring and Heckman 2007 Frangakis and Rubin 2002 Heckman and Vytlacil2007a b Holland 1988 Hong and Raudenbush 2006 Hudgens and Halloran 2008Jo 2008 Pearl 2001 Robins and Greenland 1992 Sobel 2008) The potential for furtherconceptual and methodological development and for broad applications in the field of behav-ioral and social sciences promises to greatly advance the empirical basis of our knowledgeabout causality in the social world

This book clarifies theoretical concepts and introduces innovative statistical strategies forinvestigating the average effects of multivalued treatments moderated treatment effectsmediated treatment effects and spill-over effects in experimental or quasiexperimental dataDefining individual-specific and population average treatment effects in terms of potentialoutcomes the book relates the mathematical forms to the substantive meanings of moderatedmediated and spill-over effects in the context of application examples It also explicates andevaluates identification assumptions and contrasts innovative statistical strategies with con-ventional analytic methods

111 Moderated treatment effects

It is hard to accept the assumption that a treatment would produce the same impact for everyindividual in every possible circumstance Understanding the heterogeneity of treatmenteffects therefore is key to the development of causal theories For example some studiesreported that estrogen therapy improved cardiovascular health among women who initiatedits use during menopause According to a series of other studies however the use of estrogentherapy increased postmenopausal womenrsquos risk for heart attacks (Grodstein et al 19962000 Writing Group for the Womenrsquos Health Initiative Investigators 2002) The sharp con-trast of findings from these studies led to the hypothesis that age of initiation moderates theeffect of estrogen therapy on womenrsquos health (Manson and Bassuk 2007 Rossouw et al2007) Revelations of the moderated causal relationship greatly enrich theoretical understand-ing and in this case directly inform clinical practice

In another example dividing students by ability into small groups for reading instructionhas been controversial for decades Many believed that the practice benefits high-ability stu-dents at the expense of their low-ability peers and exacerbates educational inequality (Grantand Rothenberg 1986 Rowan and Miracle 1983 Trimble and Sinclair 1987) Recentevidence has shown that first of all the effect of ability grouping depends on a number ofconditions including whether enough time is allocated to reading instruction and how wellthe class can be managed Hence instructional time and class management are among theadditional moderators to consider for determining the merits of ability grouping (Hong andHong 2009 Hong et al 2012a b) Moreover further investigations have generated evidencethat contradicts the long-held belief that grouping undermines the interests of low-abilitystudents Quite the contrary researchers have found that grouping is beneficial for students

5INTRODUCTION

with low or medium prior abilitymdashif adequate time is provided for instruction and if the classis well managed On the other hand ability grouping appears to have a minimal impact onhigh-ability studentsrsquo literacy These results were derived from kindergarten data and maynot hold for math instruction and for higher grade levels Replications with different subpo-pulations and in different contexts enable researchers to assess the generalizability of a theory

In a third example one may hypothesize that a student is unlikely to learn algebra well inninth grade without a solid foundation in eighth-grade prealgebra One may also argue that theprogress a student made in learning eighth-grade prealgebra may not be sustained without thefurther enhancement of taking algebra in ninth grade In other words the earlier treatment(prealgebra in eighth grade) and the later treatment (algebra in ninth grade) may reinforceone other each moderates the effect of the other treatment on the final outcome (math achieve-ment at the end of ninth grade) As Hong and Raudenbush (2008) argued the cumulativeeffect of two treatments in a well-aligned sequence may exceed the sum of the benefits oftwo single-year treatments Similarly experiencing two consecutive years of inferior treat-ments such as encountering an incompetent math teacher who dampens a studentrsquos self-efficacy in math learning in eighth grade and then again in ninth grade could do more damagethan the sum of the effect of encountering an incompetent teacher only in eighth grade and theeffect of having such a teacher only in ninth grade

There has been a great amount of conceptual confusion regarding how a moderatorrelates to the treatment and the outcome On one hand many researchers have used the termsldquomoderatorsrdquo and ldquomediatorsrdquo interchangeably without understanding the crucial distinctionbetween the two An overcorrective attempt on the other hand has led to the arbitrary rec-ommendation that a moderator must occur prior to the treatment and be minimally associatedwith the treatment (James and Brett 1984 Kraemer et al 2001 2008) In Chapter 6 weclarify that in essence the causal effect of the treatment on the outcome may depend onthe moderator value A moderator can be a subpopulation identifier a contextual character-istic a concurrent treatment or a preceding or succeeding treatment In the earlier examplesage of estrogen therapy initiation and a studentrsquos prior ability are subpopulation identifiers themanageability of a class which reflects both teacher skills and peer behaviors characterizes thecontext literacy instruction time is a concurrent treatment while prealgebra is a precedingtreatment for algebra A moderator does not have to occur prior to the treatment and doesnot have to be independent of the treatment

Once a moderated causal relationship has been defined in terms of potential outcomes theresearcher then chooses an appropriate experimental design for testing the moderation theoryThe assumptions required for identifying the moderated causal relationships differ across dif-ferent designs and have implications for analyzing experimental and quasiexperimental dataRandomized block designs are suitable for examining individual or contextual characteristicsas potential moderators factorial designs enable one to determine the joint effects of two ormore concurrent treatments and sequential randomized designs are ideal for assessing thecumulative effects of consecutive treatments Multisite randomized trials constitute anadditional type of designs in which the experimental sites are often deliberately sampled torepresent a population of geographical locations or social organizations Site membershipscan be viewed as implicit moderators that summarize a host of features of a local environmentReplications of a treatment over multiple sites allow one to quantify the heterogeneity of treat-ment effects across the sites Chapters 7 and 8 are focused on statistical methods for moder-ation analyses with quasiexperimental data In particular these chapters demonstrate througha number of application examples how a nonparametric marginal mean weighting through

6 CAUSALITY IN A SOCIAL WORLD

stratification (MMWS) method can overcome some important limitations of other existingmethods

Moderation questions are not restricted to treatmentndashoutcome relationships Rather theyare prevalent in investigations of mediation and spill-over This is because heterogeneity oftreatment effects may be explained by the variation in causal mediation mechanisms acrosssubpopulations and contexts It is also because the treatment effect for a focal individualmay depend on whether there is spill-over from other individuals through social interactionsYet similar to the confusion around the concept of moderation among applied researchersthere has been a great amount of misunderstanding with regard to mediation

112 Mediated treatment effects

Questions about mediation are at the core of nearly every scientific theory In its simplestform a theory explains why treatment Z causes outcome Y by hypothesizing an intermediateprocess involving at least one mediatorM that could have been changed by the treatment andcould subsequently have an impact on the outcome A causal mediation analysis then decom-poses the total effect of the treatment on the outcome into two parts the effect transmittedthrough the hypothesized mediator called the indirect effect and the difference betweenthe total effect and the indirect effect called the direct effect The latter represents the treat-ment effect channeled through other unspecified pathways (Alwin and Hauser 1975 Baronand Kenny 1986 Duncan 1966 Shadish and Sweeney 1991) Most researchers in socialsciences have followed the convention of illustrating the direct effect and the indirect effectwith path diagrams and then specifying linear regression models postulated to represent thestructural relationships between the treatment the mediator and the outcome

In Chapter 9 we point out that the approach to defining the indirect effect and the directeffect in terms of path coefficients is often misled by oversimplistic assumptions about thestructural relationships In particular this approach typically overlooks the fact that a treat-ment may generate an impact on the outcome not only by changing the mediator value butalso by changing the mediatorndashoutcome relationship (Judd and Kenny 1981) An examplein Chapter 9 illustrates such a case An experiment randomizes students to either an experi-mental condition which provides them with study materials and encourages them to study fora test or to a control condition which provides neither study materials nor encouragement(Holland 1988 Powers and Swinton 1984) One might hypothesize that encouraging exper-imental group members to study will increase the time they spend studying a focal mediatorin this example which in turn will increase their average test scores One might furtherhypothesize that even without a change in study time providing study materials to experi-mental group members will enable them to study more effectively than they otherwise wouldConsequently the effect of time spent studying might be greater under the experimentalcondition than under the control condition Omitting the treatment-by-mediator interactioneffect will lead to bias in the estimation of the indirect effect and the direct effect

Chapter 11 describes in great detail an evaluation study contrasting a new welfare-to-workprogram emphasizing active participation in the labor force with a traditional program thatguaranteed cash assistance without a mandate to seek employment (Hong Deutsch and Hill2011) Focusing on the psychological well-being of welfare applicants who were singlemothers with preschool-aged children the researchers hypothesized that the treatment wouldincrease employment rate among the welfare recipients and that the treatment-inducedincrease in employment would likely reduce depressive symptoms under the new program

7INTRODUCTION

They also hypothesized that in contrast the same increase in employment was unlikely tohave a psychological impact under the traditional program Hence the treatment-by-mediatorinteraction effect on the outcome was an essential component of the intermediate process

We will show that by defining the indirect effect and the direct effect in terms of potentialoutcomes one can avoid invoking unwarranted assumptions about the unknown structuralrelationships (Holland 1988 Pearl 2001 Robins and Greenland 1992) The potential out-comes framework has further clarified that in a causal mediation theory the treatment and themediator are both conceivably manipulable In the aforementioned example a welfare appli-cant could be assigned at random to either the new program or the traditional one Once havingbeen assigned to one of the programs the individual might or might not be employed due tovarious structural constraints market fluctuations or other random events typically beyondher control

Because the indirect effect and the direct effect are defined in terms of potential outcomesrather than on the basis of any specific regression models it becomes possible to distinguishthe definitions of these causal effects from their identification and estimation The identifica-tion step relates a causal parameter to observable population data For example for individualsassigned at random to an experimental condition their average counterfactual outcome asso-ciated with the control condition is expected to be equal to the average observable outcome ofthose assigned to the control condition Therefore the population average causal effect can beeasily identified in a randomized experiment The estimation step then relates the sample datato the observable population quantities involved in identification while taking into account thedegree of sampling variability

A major challenge in identification is that the mediatorndashoutcome relationship tends tobe confounded by selection even if the treatment is randomized Chapter 9 reviews variousexperimental designs that have been proposed by past researchers for studying causalmediation mechanisms Chapter 10 compares the identification assumptions and the analyticprocedures across a wide range of analytic methods These discussions are followed by anintroduction of the ratio-of-mediator-probability weighting (RMPW) strategy in Chapter 11In comparison with most existing strategies for causal mediation analysis RMPW relies onrelatively fewer identification assumptions and model-based assumptions Chapters 12 and13 will show that this new strategy can be applied broadly with extensions to multilevelexperimental designs and to studies of complex mediation mechanisms involving multiplemediators

113 Spill-over effects of a treatment

It is well known in vaccination research that an unvaccinated person can benefit when mostother people in the local community are vaccinated This is viewed as a spill-over of the treat-ment effect from the vaccinated people to an unvaccinated person Similarly due to socialinteractions among individuals or groups of individuals an intervention received by somemay generate a spill-over impact on others affiliated with the same organization or connectedthrough the same network For example effective policing in one neighborhood may driveoffenders to operate in other neighborhoods In evaluating an innovative community policingprogram researchers found that a neighborhood not assigned to community policing tended tosuffer if the surrounding neighborhoods were assigned to community policing This is aninstance in which a well-intended treatment generates an unintended negative spill-over effectHowever being assigned to community policy was found particularly beneficial when thesurrounding neighborhoods also received the intervention (Verbitsky-Savitz and Raudenbush

8 CAUSALITY IN A SOCIAL WORLD

2012) In another example the impact of a school-based mentoring program targeted atstudents displaying delinquent behaviors may be enhanced for a focal student if his or herat-risk peers are assigned to the mentoring program at the same time

The previous examples challenge the ldquostable unit treatment value assumptionrdquo (SUTVA)that has been invoked in most causal inference studies This assumption states that an indi-vidualrsquos potential response to a treatment depends neither on how the treatment is assignednor on the treatment assignments of other individuals (Rubin 1980 1986) Rubin and othersbelieve that without resorting to SUTVA the causal effect of a treatment becomes hard todefine and that causal inference may become intractable However as Sobel (2006) has illus-trated with the Moving to Opportunity (MTO) experiment that offered housing vouchersenabling low-income families to move to low-poverty neighborhoods there are possible con-sequences of violating SUTVA in estimating the treatment effects on outcomes such as safetyand self-sufficiency Because social interactions among individuals may affect whether onevolunteers to participate in the study whether one moves to a low-poverty neighborhood afterreceiving the housing voucher as well as housing project residentsrsquo subjective perceptions oftheir neighborhoods the program may have a nonzero impact on the potential outcome of theuntreated As a result despite the randomization of treatment assignment the mean differencein the observed outcome between the treated units and the untreated units may be biased forthe average treatment effect Rather the observable quantity is the difference between theeffect of treating some rather than treating none for the treated and the effect of treating somerather than treating none for the untreated the latter being the pure spill-over effect for theuntreated

In making attempts to relax SUTVA researchers have proposed several alternative frame-works that incorporate possible spill-over effects (see Hong and Raudenbush 2013 for areview) Hong (2004) presented a model that involves treatments and treatment settingsA treatment setting for an individual is a local environment constituted by a set of agentsand participants along with their treatment assignments An individualrsquos potential outcomevalue under a given treatment is assumed stable when the treatment setting is fixed the poten-tial outcome may take different values when the treatment setting shifts One may investigatewhether the treatment effect depends on the treatment setting Applying this framework Hongand Raudenbush (2006) examined the effect on a childrsquos academic growth of retaining thechild in kindergarten rather than promoting the child to first grade when a relatively smallproportion of low-achieving peers in the same school are retained as opposed to when a rel-atively large proportion of the peers are retained Hudgens and Halloran (2008) presented arelated framework in which the effect on an individual of the treatment received by this indi-vidual is distinguished from the effect on the individual of the treatment received by others inthe same local community These effects can be identified if communities are assigned atrandom to different treatment assignment strategies (such as retaining a large proportion oflow-achieving students as opposed to retaining a small proportion of such students) and sub-sequently individuals within a community are randomized for treatment Chapter 14 reviewsthese frameworks and discusses identification and estimation strategies

Social contagion may also serve as an important channel through which the effect of anintervention is transmitted For example a school-wide intervention may reduce aggressivebehaviors and thereby improve studentsrsquo psychological well-being by improving the qualityof interpersonal relationships in other classes as well as in onersquos own class This is becausechildren interact not simply with their classmates but also with those from other classes in thehallways or on the playground (VanderWeele et al 2013) In this case the spill-over betweenclasses becomes a part of the mediation mechanism In a study of student mentoring whether

9INTRODUCTION

an individual actually participated in the mentoring program despite the initial treatmentassignment is typically viewed as a primary mediator The proportion of onersquos peers that par-ticipated in the program may act as a second mediator A treatment may also exert its impactpartly through regrouping individuals and thereby changing peer composition (Hong andNomi 2012) Chapter 15 discusses analytic strategies for detecting spill-over as a part ofthe causal mediation mechanism

12 Weighting methods for causal inference

In most behavioral and social science applications major methodological challenges arise dueto the selection of treatments in a quasiexperimental study and the selection of mediator valuesin experimental and quasiexperimental studies Statistical methods most familiar to research-ers in social sciences are often inadequate for causal inferences with regard to multivaluedtreatments moderation mediation and spill-over The book offers a major revision to under-standing of causality in the social world by introducing two complementary weighting stra-tegies both featuring nonparametric approaches to estimating these causal effects The newweighting methods greatly simplify model specifications while enhancing the robustness ofresults by minimizing the reliance on some key assumptions

The propensity score-basedMMWSmethod removes selection bias associated with a largenumber of covariates by equating the pretreatment composition between treatment groups(Hong 2010a 2012 Huang et al 2005) Unlike propensity score matching and stratificationthat are mostly restricted to evaluations of binary treatments the MMWS method is flexiblefor evaluating binary and multivalued treatments by approximating a completely randomizedexperiment In evaluating whether the treatment effects differ across subpopulations definedby individual characteristics or treatment settings researchers may assign weights within eachsubpopulation in order to approximate a randomized block design To investigate whetherone treatment moderates the effect of another concurrent treatment researchers may assignweights to the data to approximate a factorial randomized design The method can also beused to assess whether the effect of an initial treatment is amplified or weakened by a subse-quent treatment or to identify an optimal treatment sequence through approximating a sequen-tial randomized experiment Even though such analyses can similarly be conducted throughinverse-probability-of-treatment weighting (IPTW) that has been increasingly employed inepidemiological research (Hernaacuten Brumback and Robins 2000 Robins Hernaacuten andBrumback 2000) IPTW is known for bias and imprecision in estimation especially whenthe propensity score models are misspecified in their functional forms (Hong 2010a Kangand Schafer 2007 Schafer and Kang 2008 Waernbaum 2012) In contrast the nonparamet-ric MMWS method displays a relatively high level of robustness despite such misspecifica-tions and also gains efficiency as indicated by simulation results (Hong 2010a)

To study causal mediation mechanisms the RMPW method decomposes the total effectof a treatment into an ldquoindirect effectrdquo transmitted through a specific mediator and a ldquodirecteffectrdquo representing unspecified mechanisms In contrast with most existing methods formediation analysis the RMPW-adjusted outcome model is extremely simple and is nonpara-metric in nature It generates estimates of the causal effects along with their sampling errorswhile adjusting for pretreatment covariates that confound the mediatorndashoutcome relationshipsthrough weighting The method applies regardless of the distribution of the outcome the dis-tribution of the mediator or the functional relationship between the outcome and the mediator

10 CAUSALITY IN A SOCIAL WORLD

Page 16: Thumbnail - download.e-bookshelf.de...4.2.4 IPTW for bias removal 84 4.3 MMWS 86 4.3.1 Theoretical rationale 86 ... 8.3.4.2 Growth model in the absence of confounding 200 8.3.4.3 Weighted

research has been driven by the substantive interest in understanding the role of socialinstitutionsmdashschools in particularmdashin shaping human development especially during theages of fast growth (eg childhood adolescence and early adulthood) I have shared meth-odological advances with a broad audience in social sciences through applying the causalinference methods to prominent substantive issues such as grade retention within-class group-ing services for English language learners and welfare-to-work strategies These illustrationsare used extensively in this book

Among social scientists awareness of methodological challenges in causal investigationsis higher than ever before In response to this rising demand causal inference is becoming oneof the most productive scholarly fields in the past decade I have had the benefit of followingthe work and exchanging ideas with some fellow methodologists who are constantly contri-buting new thoughts and making breakthroughs These include Daniel Almirall HowardBloom Tom Cook Michael Elliot Ken Frank Ben Hansen James Heckman JenniferHill Martin Huber Kosuke Imai Booil Jo John R Lockwood David MacKinnon DanielMcCaffrey Luke Miratrix Richard Murnane Derek Neal Lindsey Page Judea PearlStephen Raudenbush Sean Reardon Michael Seltzer Youngyun Shin Betsy SinclairMichael Sobel Peter Steiner Elizabeth Stuart Eric Thetchen Tchetchen Tyler VanderWeeleand Kazuo Yamaguchi I am grateful to Larry Hedges who offered me the opportunity of guestediting the Journal of Research in Educational Effectiveness special issue on the statisticalapproaches to studying mediator effects in education research in 2012 Many of the colleagueswhom I mentioned above contributed to the open debate in the special issue by either propos-ing or critically examining a number of innovative methods for studying causal mechanismsI anticipate that many of them are or will soon be writing their own research monographs oncausal inference if they have not already done so Therefore readers of this book are stronglyrecommended to browse past and future titles by these and other authors for alternative per-spectives and approaches My own understanding of course will continue to evolve as well inthe coming years

One has to be ambitious to stay upfront in substantive areas as well as in methodology Thebest strategy apparently is to learn from substantive experts During different phases of mytraining and professional career I have had the opportunities to work with David K CohenBrian Rowan Deborah Ball Carl Corter Janette Pelletier Takako Nomi Esther Geva DavidFrancis Stephanie Jones Joshua Brown and Heather D Hill Many of my substantive insightswere derived from conversations with these mentors and colleagues I am especially gratefulto Bob Granger former president of the William T Grant Foundation who reassured me thatldquoBy staying a bit broad you will learn a lot and many fields will benefit from your workrdquo

Like many other single-authored books this one is built on the generous contributions ofstudents and colleagues who deserve special acknowledgement Excellent research assistancefrom Jonah Deutsch Joshua Gagne Rachel Garrett Yihua Hong Xu Qin Cheng Yang andBing Yu was instrumental in the development and implementation of the new methods pre-sented in this book Richard Congdon a renowned statistical software programmer broughtrevolutionary ideas to interface designs in software development with the aim of facilitatingusersrsquo decision-making in a series of causal analyses RichardMurnane Stephen Raudenbushand Michael Sobel carefully reviewed and critiqued multiple chapters of the manuscript draftAdditional comments and suggestions on earlier drafts came fromHoward Bloom Ken FrankBen Hansen Jennifer Hill Booil Jo Ben Kelcey David MacKinnon and Fan Yang TereseSchwartzman provided valuable assistance in manuscript preparation The writing of thisbook was supported by a Scholars Award from the William T Grant Foundation and an

xvi PREFACE

Institute of Education Sciences (IES) Statistical and Research Methodology in EducationGrant from the US Department of Education All the errors in the published form are ofcourse the sole responsibility of the author

Project Editors Richard Davis Prachi Sinha Sahay and Liz Wingett and Assistant EditorHeather Kay at Wiley and project Managers P Jayapriya and R Jayavel at SPi Global havebeen wonderful to work with throughout the manuscript preparation and final production ofthe book Debbie Jupe Commissioning Editor at Wiley was remarkably effective in initiatingthe contact with me and in organizing anonymous reviews of the book proposal and of themanuscript These constructive reviews have shaped the book in important ways I cannotagree more with one of the reviewers that ldquocausality cannot be established on a pure statisticalgroundrdquo The book therefore highlights the importance of substantive knowledge and researchdesigns and places a great emphasis on clarifying and evaluating assumptions required foridentifying causal effects in the context of each application Much caution is raised againstpossible misusage of statistical techniques for analyzing causal relationships especiallywhen data are inadequate Yet I maintain that improved statistical procedures along withimproved research designs would greatly enhance our ability in attempt to empiricallyexamine well-articulated causal theories

xviiPREFACE

Part I

OVERVIEW

1

Introduction

According to an ancient Chinese fable a farmer who was eager to help his crops grow wentinto his field and pulled each seedling upward After exhausting himself with the work heannounced to his family that they were going to have a good harvest only to find the nextmorning that the plants had wilted and died Readers with minimal agricultural knowledgemay immediately point out the following the farmerrsquos intervention theory was based on acorrect observation that crops that grow taller tend to produce more yield Yet his hypothesisreflects a false understanding of the cause and the effectmdashthat seedlings pulled to be tallerwould yield as much as seedlings thriving on their own

In their classic Design and Analysis of Experiments Hinkelmann and Kempthorne (1994updated version of Kempthorne 1952) discussed two types of science descriptive science andthe development of theory These two types of science are interrelated in the following senseobservations of an event and other related events often selected and classified for descriptionby scientists naturally lead to one or more explanations that we call ldquotheoretical hypothesesrdquowhich are then screened and falsified by means of further observations experimentation andanalyses (Popper 1963) The experiment of pulling seedlings to be taller was costly but didserve the purpose of advancing this farmerrsquos knowledge of ldquowhat does not workrdquo To developa successful intervention in this case would require a series of empirical tests of explicittheories identifying potential contributors to crop growth This iterative process graduallydeepens our knowledge of the relationships between supposed causes and effectsmdashthatis causalitymdashand may eventually increase the success of agricultural medical and socialinterventions

11 Concepts of moderation mediation and spill-over

Although the story of the ancient farmer is fictitious numerous examples can be found in thereal world in which well-intended interventions fail to produce the intended benefits or inmany cases even lead to unintended consequences ldquoInterventionsrdquo and ldquotreatmentsrdquo usedinterchangeably in this book broadly refer to actions taken by agents or circumstances expe-rienced by an individual or groups of individuals Interventions are regularly seen in educa-tion physical and mental health social services business politics and law enforcement In an

Causality in a Social World Moderation Meditation and Spill-over First Edition Guanglei Hongcopy 2015 John Wiley amp Sons Ltd Published 2015 by John Wiley amp Sons Ltd

education intervention for example teachers are typically the agents who deliver a treatmentto students while the impact of the treatment on student outcomes is of ultimate causalinterest Some educational practices such as ldquoteaching to the testrdquo have been criticized tobe nearly as counterproductive as the attempt of helping seedlings grow by pulling themupward ldquoInterventionsrdquo and ldquotreatmentsrdquo under consideration do not exclude undesiredexperiences such as exposure to poverty abuse crime or bereavement A treatment plannedor unplanned becomes a focus of research if there are theoretical reasons to anticipate itsimpact positive or negative on the well-being of individuals who are embedded in socialsettings including families classrooms schools neighborhoods and workplaces

In social science research in general and in policy and program evaluations in particularquestions concerning whether an intervention works and if so which version of the interven-tion works for whom under what conditions and why are key to the advancement of scien-tific and practical knowledge Although most empirical investigations in the social sciencesconcentrate on the average effect of a treatment for a specific population as opposed to theabsence of such a treatment (ie the control condition) in-depth theoretical reasoning withregard to how the causal effect is generated substantiated by compelling empirical evidenceis crucial for advancing scientific understanding

First when there are multiple versions or different dosages of the treatment or when thereare multiple versions of the control condition a binary divide between ldquothe treatmentrdquo andldquothe controlrdquo may not be as informative as fine-grained comparisons across for exampleldquotreatment version Ardquo ldquotreatment version Brdquo ldquocontrol version Ardquo and ldquocontrol versionBrdquo For example expanding the federally funded Head Start program to poor children isexpected to generate a greater benefit when few early childhood education alternatives areavailable (call it ldquocontrol version Ardquo) than when there is an abundance of alternatives includ-ing state-sponsored preschool programs (call it ldquocontrol version Brdquo)

Second the effect of an intervention will likely vary among individuals or across socialsettings A famous example comes from medical research the well-publicized cardiovascularbenefits of initiating estrogen therapy during menopause were contradicted later by experi-mental findings that the same therapy increased postmenopausal womenrsquos risk for heartattacks The effect of an intervention may also depend on the provision of some other con-current or subsequent interventions Such heterogeneous effects are often characterized asmoderated effects in the literature

Third alternative theories may provide competing explanations for the causal mechan-isms that is the processes through which the intervention produces its effect A theoreticalconstruct characterizing the hypothesized intermediate process is called a mediator of theintervention effect The fictitious farmer never developed an elaborate theory as to whatcaused some seedlings to surpass others in growth Once scientists revealed the causal rela-tionship between access to chemical nutrients in soil and plant growth wide applications ofchemically synthesized fertilizers finally led to a major increase in crop production

Finally it is well known in agricultural experiments that a new type of fertilizer applied toone plot may spill-over to the next plot Because social connections among individuals areprevalent within organizations or through networks an individualrsquos response to the treatmentmay similarly depend on the treatment for other individuals in the same social setting whichmay lead to possible spill-overs of intervention effects among individual human beings

Answering questions with regard to moderation mediation and spill-over poses majorconceptual and analytic challenges To date psychological research often presents well-articulated theories of causal mechanisms relating stimuli to responses Yet researchers often

4 CAUSALITY IN A SOCIAL WORLD

lack rigorous analytic strategies for empirically screening competing theories explaining theobserved effect Sociologists have keen interest in the spill-over of treatment effects transmit-ted through social interactions yet have produced limited evidence quantifying such effectsAs many have pointed out in general terminological ambiguity and conceptual confusionhave been prevalent in the published applied research (Holmbeck 1997 Kraemer et al2002 2008)

A new framework for conceptualizing moderated mediated and spill-over effects hasemerged relatively recently in the statistics and econometrics literature on causal inference(eg Abbring and Heckman 2007 Frangakis and Rubin 2002 Heckman and Vytlacil2007a b Holland 1988 Hong and Raudenbush 2006 Hudgens and Halloran 2008Jo 2008 Pearl 2001 Robins and Greenland 1992 Sobel 2008) The potential for furtherconceptual and methodological development and for broad applications in the field of behav-ioral and social sciences promises to greatly advance the empirical basis of our knowledgeabout causality in the social world

This book clarifies theoretical concepts and introduces innovative statistical strategies forinvestigating the average effects of multivalued treatments moderated treatment effectsmediated treatment effects and spill-over effects in experimental or quasiexperimental dataDefining individual-specific and population average treatment effects in terms of potentialoutcomes the book relates the mathematical forms to the substantive meanings of moderatedmediated and spill-over effects in the context of application examples It also explicates andevaluates identification assumptions and contrasts innovative statistical strategies with con-ventional analytic methods

111 Moderated treatment effects

It is hard to accept the assumption that a treatment would produce the same impact for everyindividual in every possible circumstance Understanding the heterogeneity of treatmenteffects therefore is key to the development of causal theories For example some studiesreported that estrogen therapy improved cardiovascular health among women who initiatedits use during menopause According to a series of other studies however the use of estrogentherapy increased postmenopausal womenrsquos risk for heart attacks (Grodstein et al 19962000 Writing Group for the Womenrsquos Health Initiative Investigators 2002) The sharp con-trast of findings from these studies led to the hypothesis that age of initiation moderates theeffect of estrogen therapy on womenrsquos health (Manson and Bassuk 2007 Rossouw et al2007) Revelations of the moderated causal relationship greatly enrich theoretical understand-ing and in this case directly inform clinical practice

In another example dividing students by ability into small groups for reading instructionhas been controversial for decades Many believed that the practice benefits high-ability stu-dents at the expense of their low-ability peers and exacerbates educational inequality (Grantand Rothenberg 1986 Rowan and Miracle 1983 Trimble and Sinclair 1987) Recentevidence has shown that first of all the effect of ability grouping depends on a number ofconditions including whether enough time is allocated to reading instruction and how wellthe class can be managed Hence instructional time and class management are among theadditional moderators to consider for determining the merits of ability grouping (Hong andHong 2009 Hong et al 2012a b) Moreover further investigations have generated evidencethat contradicts the long-held belief that grouping undermines the interests of low-abilitystudents Quite the contrary researchers have found that grouping is beneficial for students

5INTRODUCTION

with low or medium prior abilitymdashif adequate time is provided for instruction and if the classis well managed On the other hand ability grouping appears to have a minimal impact onhigh-ability studentsrsquo literacy These results were derived from kindergarten data and maynot hold for math instruction and for higher grade levels Replications with different subpo-pulations and in different contexts enable researchers to assess the generalizability of a theory

In a third example one may hypothesize that a student is unlikely to learn algebra well inninth grade without a solid foundation in eighth-grade prealgebra One may also argue that theprogress a student made in learning eighth-grade prealgebra may not be sustained without thefurther enhancement of taking algebra in ninth grade In other words the earlier treatment(prealgebra in eighth grade) and the later treatment (algebra in ninth grade) may reinforceone other each moderates the effect of the other treatment on the final outcome (math achieve-ment at the end of ninth grade) As Hong and Raudenbush (2008) argued the cumulativeeffect of two treatments in a well-aligned sequence may exceed the sum of the benefits oftwo single-year treatments Similarly experiencing two consecutive years of inferior treat-ments such as encountering an incompetent math teacher who dampens a studentrsquos self-efficacy in math learning in eighth grade and then again in ninth grade could do more damagethan the sum of the effect of encountering an incompetent teacher only in eighth grade and theeffect of having such a teacher only in ninth grade

There has been a great amount of conceptual confusion regarding how a moderatorrelates to the treatment and the outcome On one hand many researchers have used the termsldquomoderatorsrdquo and ldquomediatorsrdquo interchangeably without understanding the crucial distinctionbetween the two An overcorrective attempt on the other hand has led to the arbitrary rec-ommendation that a moderator must occur prior to the treatment and be minimally associatedwith the treatment (James and Brett 1984 Kraemer et al 2001 2008) In Chapter 6 weclarify that in essence the causal effect of the treatment on the outcome may depend onthe moderator value A moderator can be a subpopulation identifier a contextual character-istic a concurrent treatment or a preceding or succeeding treatment In the earlier examplesage of estrogen therapy initiation and a studentrsquos prior ability are subpopulation identifiers themanageability of a class which reflects both teacher skills and peer behaviors characterizes thecontext literacy instruction time is a concurrent treatment while prealgebra is a precedingtreatment for algebra A moderator does not have to occur prior to the treatment and doesnot have to be independent of the treatment

Once a moderated causal relationship has been defined in terms of potential outcomes theresearcher then chooses an appropriate experimental design for testing the moderation theoryThe assumptions required for identifying the moderated causal relationships differ across dif-ferent designs and have implications for analyzing experimental and quasiexperimental dataRandomized block designs are suitable for examining individual or contextual characteristicsas potential moderators factorial designs enable one to determine the joint effects of two ormore concurrent treatments and sequential randomized designs are ideal for assessing thecumulative effects of consecutive treatments Multisite randomized trials constitute anadditional type of designs in which the experimental sites are often deliberately sampled torepresent a population of geographical locations or social organizations Site membershipscan be viewed as implicit moderators that summarize a host of features of a local environmentReplications of a treatment over multiple sites allow one to quantify the heterogeneity of treat-ment effects across the sites Chapters 7 and 8 are focused on statistical methods for moder-ation analyses with quasiexperimental data In particular these chapters demonstrate througha number of application examples how a nonparametric marginal mean weighting through

6 CAUSALITY IN A SOCIAL WORLD

stratification (MMWS) method can overcome some important limitations of other existingmethods

Moderation questions are not restricted to treatmentndashoutcome relationships Rather theyare prevalent in investigations of mediation and spill-over This is because heterogeneity oftreatment effects may be explained by the variation in causal mediation mechanisms acrosssubpopulations and contexts It is also because the treatment effect for a focal individualmay depend on whether there is spill-over from other individuals through social interactionsYet similar to the confusion around the concept of moderation among applied researchersthere has been a great amount of misunderstanding with regard to mediation

112 Mediated treatment effects

Questions about mediation are at the core of nearly every scientific theory In its simplestform a theory explains why treatment Z causes outcome Y by hypothesizing an intermediateprocess involving at least one mediatorM that could have been changed by the treatment andcould subsequently have an impact on the outcome A causal mediation analysis then decom-poses the total effect of the treatment on the outcome into two parts the effect transmittedthrough the hypothesized mediator called the indirect effect and the difference betweenthe total effect and the indirect effect called the direct effect The latter represents the treat-ment effect channeled through other unspecified pathways (Alwin and Hauser 1975 Baronand Kenny 1986 Duncan 1966 Shadish and Sweeney 1991) Most researchers in socialsciences have followed the convention of illustrating the direct effect and the indirect effectwith path diagrams and then specifying linear regression models postulated to represent thestructural relationships between the treatment the mediator and the outcome

In Chapter 9 we point out that the approach to defining the indirect effect and the directeffect in terms of path coefficients is often misled by oversimplistic assumptions about thestructural relationships In particular this approach typically overlooks the fact that a treat-ment may generate an impact on the outcome not only by changing the mediator value butalso by changing the mediatorndashoutcome relationship (Judd and Kenny 1981) An examplein Chapter 9 illustrates such a case An experiment randomizes students to either an experi-mental condition which provides them with study materials and encourages them to study fora test or to a control condition which provides neither study materials nor encouragement(Holland 1988 Powers and Swinton 1984) One might hypothesize that encouraging exper-imental group members to study will increase the time they spend studying a focal mediatorin this example which in turn will increase their average test scores One might furtherhypothesize that even without a change in study time providing study materials to experi-mental group members will enable them to study more effectively than they otherwise wouldConsequently the effect of time spent studying might be greater under the experimentalcondition than under the control condition Omitting the treatment-by-mediator interactioneffect will lead to bias in the estimation of the indirect effect and the direct effect

Chapter 11 describes in great detail an evaluation study contrasting a new welfare-to-workprogram emphasizing active participation in the labor force with a traditional program thatguaranteed cash assistance without a mandate to seek employment (Hong Deutsch and Hill2011) Focusing on the psychological well-being of welfare applicants who were singlemothers with preschool-aged children the researchers hypothesized that the treatment wouldincrease employment rate among the welfare recipients and that the treatment-inducedincrease in employment would likely reduce depressive symptoms under the new program

7INTRODUCTION

They also hypothesized that in contrast the same increase in employment was unlikely tohave a psychological impact under the traditional program Hence the treatment-by-mediatorinteraction effect on the outcome was an essential component of the intermediate process

We will show that by defining the indirect effect and the direct effect in terms of potentialoutcomes one can avoid invoking unwarranted assumptions about the unknown structuralrelationships (Holland 1988 Pearl 2001 Robins and Greenland 1992) The potential out-comes framework has further clarified that in a causal mediation theory the treatment and themediator are both conceivably manipulable In the aforementioned example a welfare appli-cant could be assigned at random to either the new program or the traditional one Once havingbeen assigned to one of the programs the individual might or might not be employed due tovarious structural constraints market fluctuations or other random events typically beyondher control

Because the indirect effect and the direct effect are defined in terms of potential outcomesrather than on the basis of any specific regression models it becomes possible to distinguishthe definitions of these causal effects from their identification and estimation The identifica-tion step relates a causal parameter to observable population data For example for individualsassigned at random to an experimental condition their average counterfactual outcome asso-ciated with the control condition is expected to be equal to the average observable outcome ofthose assigned to the control condition Therefore the population average causal effect can beeasily identified in a randomized experiment The estimation step then relates the sample datato the observable population quantities involved in identification while taking into account thedegree of sampling variability

A major challenge in identification is that the mediatorndashoutcome relationship tends tobe confounded by selection even if the treatment is randomized Chapter 9 reviews variousexperimental designs that have been proposed by past researchers for studying causalmediation mechanisms Chapter 10 compares the identification assumptions and the analyticprocedures across a wide range of analytic methods These discussions are followed by anintroduction of the ratio-of-mediator-probability weighting (RMPW) strategy in Chapter 11In comparison with most existing strategies for causal mediation analysis RMPW relies onrelatively fewer identification assumptions and model-based assumptions Chapters 12 and13 will show that this new strategy can be applied broadly with extensions to multilevelexperimental designs and to studies of complex mediation mechanisms involving multiplemediators

113 Spill-over effects of a treatment

It is well known in vaccination research that an unvaccinated person can benefit when mostother people in the local community are vaccinated This is viewed as a spill-over of the treat-ment effect from the vaccinated people to an unvaccinated person Similarly due to socialinteractions among individuals or groups of individuals an intervention received by somemay generate a spill-over impact on others affiliated with the same organization or connectedthrough the same network For example effective policing in one neighborhood may driveoffenders to operate in other neighborhoods In evaluating an innovative community policingprogram researchers found that a neighborhood not assigned to community policing tended tosuffer if the surrounding neighborhoods were assigned to community policing This is aninstance in which a well-intended treatment generates an unintended negative spill-over effectHowever being assigned to community policy was found particularly beneficial when thesurrounding neighborhoods also received the intervention (Verbitsky-Savitz and Raudenbush

8 CAUSALITY IN A SOCIAL WORLD

2012) In another example the impact of a school-based mentoring program targeted atstudents displaying delinquent behaviors may be enhanced for a focal student if his or herat-risk peers are assigned to the mentoring program at the same time

The previous examples challenge the ldquostable unit treatment value assumptionrdquo (SUTVA)that has been invoked in most causal inference studies This assumption states that an indi-vidualrsquos potential response to a treatment depends neither on how the treatment is assignednor on the treatment assignments of other individuals (Rubin 1980 1986) Rubin and othersbelieve that without resorting to SUTVA the causal effect of a treatment becomes hard todefine and that causal inference may become intractable However as Sobel (2006) has illus-trated with the Moving to Opportunity (MTO) experiment that offered housing vouchersenabling low-income families to move to low-poverty neighborhoods there are possible con-sequences of violating SUTVA in estimating the treatment effects on outcomes such as safetyand self-sufficiency Because social interactions among individuals may affect whether onevolunteers to participate in the study whether one moves to a low-poverty neighborhood afterreceiving the housing voucher as well as housing project residentsrsquo subjective perceptions oftheir neighborhoods the program may have a nonzero impact on the potential outcome of theuntreated As a result despite the randomization of treatment assignment the mean differencein the observed outcome between the treated units and the untreated units may be biased forthe average treatment effect Rather the observable quantity is the difference between theeffect of treating some rather than treating none for the treated and the effect of treating somerather than treating none for the untreated the latter being the pure spill-over effect for theuntreated

In making attempts to relax SUTVA researchers have proposed several alternative frame-works that incorporate possible spill-over effects (see Hong and Raudenbush 2013 for areview) Hong (2004) presented a model that involves treatments and treatment settingsA treatment setting for an individual is a local environment constituted by a set of agentsand participants along with their treatment assignments An individualrsquos potential outcomevalue under a given treatment is assumed stable when the treatment setting is fixed the poten-tial outcome may take different values when the treatment setting shifts One may investigatewhether the treatment effect depends on the treatment setting Applying this framework Hongand Raudenbush (2006) examined the effect on a childrsquos academic growth of retaining thechild in kindergarten rather than promoting the child to first grade when a relatively smallproportion of low-achieving peers in the same school are retained as opposed to when a rel-atively large proportion of the peers are retained Hudgens and Halloran (2008) presented arelated framework in which the effect on an individual of the treatment received by this indi-vidual is distinguished from the effect on the individual of the treatment received by others inthe same local community These effects can be identified if communities are assigned atrandom to different treatment assignment strategies (such as retaining a large proportion oflow-achieving students as opposed to retaining a small proportion of such students) and sub-sequently individuals within a community are randomized for treatment Chapter 14 reviewsthese frameworks and discusses identification and estimation strategies

Social contagion may also serve as an important channel through which the effect of anintervention is transmitted For example a school-wide intervention may reduce aggressivebehaviors and thereby improve studentsrsquo psychological well-being by improving the qualityof interpersonal relationships in other classes as well as in onersquos own class This is becausechildren interact not simply with their classmates but also with those from other classes in thehallways or on the playground (VanderWeele et al 2013) In this case the spill-over betweenclasses becomes a part of the mediation mechanism In a study of student mentoring whether

9INTRODUCTION

an individual actually participated in the mentoring program despite the initial treatmentassignment is typically viewed as a primary mediator The proportion of onersquos peers that par-ticipated in the program may act as a second mediator A treatment may also exert its impactpartly through regrouping individuals and thereby changing peer composition (Hong andNomi 2012) Chapter 15 discusses analytic strategies for detecting spill-over as a part ofthe causal mediation mechanism

12 Weighting methods for causal inference

In most behavioral and social science applications major methodological challenges arise dueto the selection of treatments in a quasiexperimental study and the selection of mediator valuesin experimental and quasiexperimental studies Statistical methods most familiar to research-ers in social sciences are often inadequate for causal inferences with regard to multivaluedtreatments moderation mediation and spill-over The book offers a major revision to under-standing of causality in the social world by introducing two complementary weighting stra-tegies both featuring nonparametric approaches to estimating these causal effects The newweighting methods greatly simplify model specifications while enhancing the robustness ofresults by minimizing the reliance on some key assumptions

The propensity score-basedMMWSmethod removes selection bias associated with a largenumber of covariates by equating the pretreatment composition between treatment groups(Hong 2010a 2012 Huang et al 2005) Unlike propensity score matching and stratificationthat are mostly restricted to evaluations of binary treatments the MMWS method is flexiblefor evaluating binary and multivalued treatments by approximating a completely randomizedexperiment In evaluating whether the treatment effects differ across subpopulations definedby individual characteristics or treatment settings researchers may assign weights within eachsubpopulation in order to approximate a randomized block design To investigate whetherone treatment moderates the effect of another concurrent treatment researchers may assignweights to the data to approximate a factorial randomized design The method can also beused to assess whether the effect of an initial treatment is amplified or weakened by a subse-quent treatment or to identify an optimal treatment sequence through approximating a sequen-tial randomized experiment Even though such analyses can similarly be conducted throughinverse-probability-of-treatment weighting (IPTW) that has been increasingly employed inepidemiological research (Hernaacuten Brumback and Robins 2000 Robins Hernaacuten andBrumback 2000) IPTW is known for bias and imprecision in estimation especially whenthe propensity score models are misspecified in their functional forms (Hong 2010a Kangand Schafer 2007 Schafer and Kang 2008 Waernbaum 2012) In contrast the nonparamet-ric MMWS method displays a relatively high level of robustness despite such misspecifica-tions and also gains efficiency as indicated by simulation results (Hong 2010a)

To study causal mediation mechanisms the RMPW method decomposes the total effectof a treatment into an ldquoindirect effectrdquo transmitted through a specific mediator and a ldquodirecteffectrdquo representing unspecified mechanisms In contrast with most existing methods formediation analysis the RMPW-adjusted outcome model is extremely simple and is nonpara-metric in nature It generates estimates of the causal effects along with their sampling errorswhile adjusting for pretreatment covariates that confound the mediatorndashoutcome relationshipsthrough weighting The method applies regardless of the distribution of the outcome the dis-tribution of the mediator or the functional relationship between the outcome and the mediator

10 CAUSALITY IN A SOCIAL WORLD

Page 17: Thumbnail - download.e-bookshelf.de...4.2.4 IPTW for bias removal 84 4.3 MMWS 86 4.3.1 Theoretical rationale 86 ... 8.3.4.2 Growth model in the absence of confounding 200 8.3.4.3 Weighted

Institute of Education Sciences (IES) Statistical and Research Methodology in EducationGrant from the US Department of Education All the errors in the published form are ofcourse the sole responsibility of the author

Project Editors Richard Davis Prachi Sinha Sahay and Liz Wingett and Assistant EditorHeather Kay at Wiley and project Managers P Jayapriya and R Jayavel at SPi Global havebeen wonderful to work with throughout the manuscript preparation and final production ofthe book Debbie Jupe Commissioning Editor at Wiley was remarkably effective in initiatingthe contact with me and in organizing anonymous reviews of the book proposal and of themanuscript These constructive reviews have shaped the book in important ways I cannotagree more with one of the reviewers that ldquocausality cannot be established on a pure statisticalgroundrdquo The book therefore highlights the importance of substantive knowledge and researchdesigns and places a great emphasis on clarifying and evaluating assumptions required foridentifying causal effects in the context of each application Much caution is raised againstpossible misusage of statistical techniques for analyzing causal relationships especiallywhen data are inadequate Yet I maintain that improved statistical procedures along withimproved research designs would greatly enhance our ability in attempt to empiricallyexamine well-articulated causal theories

xviiPREFACE

Part I

OVERVIEW

1

Introduction

According to an ancient Chinese fable a farmer who was eager to help his crops grow wentinto his field and pulled each seedling upward After exhausting himself with the work heannounced to his family that they were going to have a good harvest only to find the nextmorning that the plants had wilted and died Readers with minimal agricultural knowledgemay immediately point out the following the farmerrsquos intervention theory was based on acorrect observation that crops that grow taller tend to produce more yield Yet his hypothesisreflects a false understanding of the cause and the effectmdashthat seedlings pulled to be tallerwould yield as much as seedlings thriving on their own

In their classic Design and Analysis of Experiments Hinkelmann and Kempthorne (1994updated version of Kempthorne 1952) discussed two types of science descriptive science andthe development of theory These two types of science are interrelated in the following senseobservations of an event and other related events often selected and classified for descriptionby scientists naturally lead to one or more explanations that we call ldquotheoretical hypothesesrdquowhich are then screened and falsified by means of further observations experimentation andanalyses (Popper 1963) The experiment of pulling seedlings to be taller was costly but didserve the purpose of advancing this farmerrsquos knowledge of ldquowhat does not workrdquo To developa successful intervention in this case would require a series of empirical tests of explicittheories identifying potential contributors to crop growth This iterative process graduallydeepens our knowledge of the relationships between supposed causes and effectsmdashthatis causalitymdashand may eventually increase the success of agricultural medical and socialinterventions

11 Concepts of moderation mediation and spill-over

Although the story of the ancient farmer is fictitious numerous examples can be found in thereal world in which well-intended interventions fail to produce the intended benefits or inmany cases even lead to unintended consequences ldquoInterventionsrdquo and ldquotreatmentsrdquo usedinterchangeably in this book broadly refer to actions taken by agents or circumstances expe-rienced by an individual or groups of individuals Interventions are regularly seen in educa-tion physical and mental health social services business politics and law enforcement In an

Causality in a Social World Moderation Meditation and Spill-over First Edition Guanglei Hongcopy 2015 John Wiley amp Sons Ltd Published 2015 by John Wiley amp Sons Ltd

education intervention for example teachers are typically the agents who deliver a treatmentto students while the impact of the treatment on student outcomes is of ultimate causalinterest Some educational practices such as ldquoteaching to the testrdquo have been criticized tobe nearly as counterproductive as the attempt of helping seedlings grow by pulling themupward ldquoInterventionsrdquo and ldquotreatmentsrdquo under consideration do not exclude undesiredexperiences such as exposure to poverty abuse crime or bereavement A treatment plannedor unplanned becomes a focus of research if there are theoretical reasons to anticipate itsimpact positive or negative on the well-being of individuals who are embedded in socialsettings including families classrooms schools neighborhoods and workplaces

In social science research in general and in policy and program evaluations in particularquestions concerning whether an intervention works and if so which version of the interven-tion works for whom under what conditions and why are key to the advancement of scien-tific and practical knowledge Although most empirical investigations in the social sciencesconcentrate on the average effect of a treatment for a specific population as opposed to theabsence of such a treatment (ie the control condition) in-depth theoretical reasoning withregard to how the causal effect is generated substantiated by compelling empirical evidenceis crucial for advancing scientific understanding

First when there are multiple versions or different dosages of the treatment or when thereare multiple versions of the control condition a binary divide between ldquothe treatmentrdquo andldquothe controlrdquo may not be as informative as fine-grained comparisons across for exampleldquotreatment version Ardquo ldquotreatment version Brdquo ldquocontrol version Ardquo and ldquocontrol versionBrdquo For example expanding the federally funded Head Start program to poor children isexpected to generate a greater benefit when few early childhood education alternatives areavailable (call it ldquocontrol version Ardquo) than when there is an abundance of alternatives includ-ing state-sponsored preschool programs (call it ldquocontrol version Brdquo)

Second the effect of an intervention will likely vary among individuals or across socialsettings A famous example comes from medical research the well-publicized cardiovascularbenefits of initiating estrogen therapy during menopause were contradicted later by experi-mental findings that the same therapy increased postmenopausal womenrsquos risk for heartattacks The effect of an intervention may also depend on the provision of some other con-current or subsequent interventions Such heterogeneous effects are often characterized asmoderated effects in the literature

Third alternative theories may provide competing explanations for the causal mechan-isms that is the processes through which the intervention produces its effect A theoreticalconstruct characterizing the hypothesized intermediate process is called a mediator of theintervention effect The fictitious farmer never developed an elaborate theory as to whatcaused some seedlings to surpass others in growth Once scientists revealed the causal rela-tionship between access to chemical nutrients in soil and plant growth wide applications ofchemically synthesized fertilizers finally led to a major increase in crop production

Finally it is well known in agricultural experiments that a new type of fertilizer applied toone plot may spill-over to the next plot Because social connections among individuals areprevalent within organizations or through networks an individualrsquos response to the treatmentmay similarly depend on the treatment for other individuals in the same social setting whichmay lead to possible spill-overs of intervention effects among individual human beings

Answering questions with regard to moderation mediation and spill-over poses majorconceptual and analytic challenges To date psychological research often presents well-articulated theories of causal mechanisms relating stimuli to responses Yet researchers often

4 CAUSALITY IN A SOCIAL WORLD

lack rigorous analytic strategies for empirically screening competing theories explaining theobserved effect Sociologists have keen interest in the spill-over of treatment effects transmit-ted through social interactions yet have produced limited evidence quantifying such effectsAs many have pointed out in general terminological ambiguity and conceptual confusionhave been prevalent in the published applied research (Holmbeck 1997 Kraemer et al2002 2008)

A new framework for conceptualizing moderated mediated and spill-over effects hasemerged relatively recently in the statistics and econometrics literature on causal inference(eg Abbring and Heckman 2007 Frangakis and Rubin 2002 Heckman and Vytlacil2007a b Holland 1988 Hong and Raudenbush 2006 Hudgens and Halloran 2008Jo 2008 Pearl 2001 Robins and Greenland 1992 Sobel 2008) The potential for furtherconceptual and methodological development and for broad applications in the field of behav-ioral and social sciences promises to greatly advance the empirical basis of our knowledgeabout causality in the social world

This book clarifies theoretical concepts and introduces innovative statistical strategies forinvestigating the average effects of multivalued treatments moderated treatment effectsmediated treatment effects and spill-over effects in experimental or quasiexperimental dataDefining individual-specific and population average treatment effects in terms of potentialoutcomes the book relates the mathematical forms to the substantive meanings of moderatedmediated and spill-over effects in the context of application examples It also explicates andevaluates identification assumptions and contrasts innovative statistical strategies with con-ventional analytic methods

111 Moderated treatment effects

It is hard to accept the assumption that a treatment would produce the same impact for everyindividual in every possible circumstance Understanding the heterogeneity of treatmenteffects therefore is key to the development of causal theories For example some studiesreported that estrogen therapy improved cardiovascular health among women who initiatedits use during menopause According to a series of other studies however the use of estrogentherapy increased postmenopausal womenrsquos risk for heart attacks (Grodstein et al 19962000 Writing Group for the Womenrsquos Health Initiative Investigators 2002) The sharp con-trast of findings from these studies led to the hypothesis that age of initiation moderates theeffect of estrogen therapy on womenrsquos health (Manson and Bassuk 2007 Rossouw et al2007) Revelations of the moderated causal relationship greatly enrich theoretical understand-ing and in this case directly inform clinical practice

In another example dividing students by ability into small groups for reading instructionhas been controversial for decades Many believed that the practice benefits high-ability stu-dents at the expense of their low-ability peers and exacerbates educational inequality (Grantand Rothenberg 1986 Rowan and Miracle 1983 Trimble and Sinclair 1987) Recentevidence has shown that first of all the effect of ability grouping depends on a number ofconditions including whether enough time is allocated to reading instruction and how wellthe class can be managed Hence instructional time and class management are among theadditional moderators to consider for determining the merits of ability grouping (Hong andHong 2009 Hong et al 2012a b) Moreover further investigations have generated evidencethat contradicts the long-held belief that grouping undermines the interests of low-abilitystudents Quite the contrary researchers have found that grouping is beneficial for students

5INTRODUCTION

with low or medium prior abilitymdashif adequate time is provided for instruction and if the classis well managed On the other hand ability grouping appears to have a minimal impact onhigh-ability studentsrsquo literacy These results were derived from kindergarten data and maynot hold for math instruction and for higher grade levels Replications with different subpo-pulations and in different contexts enable researchers to assess the generalizability of a theory

In a third example one may hypothesize that a student is unlikely to learn algebra well inninth grade without a solid foundation in eighth-grade prealgebra One may also argue that theprogress a student made in learning eighth-grade prealgebra may not be sustained without thefurther enhancement of taking algebra in ninth grade In other words the earlier treatment(prealgebra in eighth grade) and the later treatment (algebra in ninth grade) may reinforceone other each moderates the effect of the other treatment on the final outcome (math achieve-ment at the end of ninth grade) As Hong and Raudenbush (2008) argued the cumulativeeffect of two treatments in a well-aligned sequence may exceed the sum of the benefits oftwo single-year treatments Similarly experiencing two consecutive years of inferior treat-ments such as encountering an incompetent math teacher who dampens a studentrsquos self-efficacy in math learning in eighth grade and then again in ninth grade could do more damagethan the sum of the effect of encountering an incompetent teacher only in eighth grade and theeffect of having such a teacher only in ninth grade

There has been a great amount of conceptual confusion regarding how a moderatorrelates to the treatment and the outcome On one hand many researchers have used the termsldquomoderatorsrdquo and ldquomediatorsrdquo interchangeably without understanding the crucial distinctionbetween the two An overcorrective attempt on the other hand has led to the arbitrary rec-ommendation that a moderator must occur prior to the treatment and be minimally associatedwith the treatment (James and Brett 1984 Kraemer et al 2001 2008) In Chapter 6 weclarify that in essence the causal effect of the treatment on the outcome may depend onthe moderator value A moderator can be a subpopulation identifier a contextual character-istic a concurrent treatment or a preceding or succeeding treatment In the earlier examplesage of estrogen therapy initiation and a studentrsquos prior ability are subpopulation identifiers themanageability of a class which reflects both teacher skills and peer behaviors characterizes thecontext literacy instruction time is a concurrent treatment while prealgebra is a precedingtreatment for algebra A moderator does not have to occur prior to the treatment and doesnot have to be independent of the treatment

Once a moderated causal relationship has been defined in terms of potential outcomes theresearcher then chooses an appropriate experimental design for testing the moderation theoryThe assumptions required for identifying the moderated causal relationships differ across dif-ferent designs and have implications for analyzing experimental and quasiexperimental dataRandomized block designs are suitable for examining individual or contextual characteristicsas potential moderators factorial designs enable one to determine the joint effects of two ormore concurrent treatments and sequential randomized designs are ideal for assessing thecumulative effects of consecutive treatments Multisite randomized trials constitute anadditional type of designs in which the experimental sites are often deliberately sampled torepresent a population of geographical locations or social organizations Site membershipscan be viewed as implicit moderators that summarize a host of features of a local environmentReplications of a treatment over multiple sites allow one to quantify the heterogeneity of treat-ment effects across the sites Chapters 7 and 8 are focused on statistical methods for moder-ation analyses with quasiexperimental data In particular these chapters demonstrate througha number of application examples how a nonparametric marginal mean weighting through

6 CAUSALITY IN A SOCIAL WORLD

stratification (MMWS) method can overcome some important limitations of other existingmethods

Moderation questions are not restricted to treatmentndashoutcome relationships Rather theyare prevalent in investigations of mediation and spill-over This is because heterogeneity oftreatment effects may be explained by the variation in causal mediation mechanisms acrosssubpopulations and contexts It is also because the treatment effect for a focal individualmay depend on whether there is spill-over from other individuals through social interactionsYet similar to the confusion around the concept of moderation among applied researchersthere has been a great amount of misunderstanding with regard to mediation

112 Mediated treatment effects

Questions about mediation are at the core of nearly every scientific theory In its simplestform a theory explains why treatment Z causes outcome Y by hypothesizing an intermediateprocess involving at least one mediatorM that could have been changed by the treatment andcould subsequently have an impact on the outcome A causal mediation analysis then decom-poses the total effect of the treatment on the outcome into two parts the effect transmittedthrough the hypothesized mediator called the indirect effect and the difference betweenthe total effect and the indirect effect called the direct effect The latter represents the treat-ment effect channeled through other unspecified pathways (Alwin and Hauser 1975 Baronand Kenny 1986 Duncan 1966 Shadish and Sweeney 1991) Most researchers in socialsciences have followed the convention of illustrating the direct effect and the indirect effectwith path diagrams and then specifying linear regression models postulated to represent thestructural relationships between the treatment the mediator and the outcome

In Chapter 9 we point out that the approach to defining the indirect effect and the directeffect in terms of path coefficients is often misled by oversimplistic assumptions about thestructural relationships In particular this approach typically overlooks the fact that a treat-ment may generate an impact on the outcome not only by changing the mediator value butalso by changing the mediatorndashoutcome relationship (Judd and Kenny 1981) An examplein Chapter 9 illustrates such a case An experiment randomizes students to either an experi-mental condition which provides them with study materials and encourages them to study fora test or to a control condition which provides neither study materials nor encouragement(Holland 1988 Powers and Swinton 1984) One might hypothesize that encouraging exper-imental group members to study will increase the time they spend studying a focal mediatorin this example which in turn will increase their average test scores One might furtherhypothesize that even without a change in study time providing study materials to experi-mental group members will enable them to study more effectively than they otherwise wouldConsequently the effect of time spent studying might be greater under the experimentalcondition than under the control condition Omitting the treatment-by-mediator interactioneffect will lead to bias in the estimation of the indirect effect and the direct effect

Chapter 11 describes in great detail an evaluation study contrasting a new welfare-to-workprogram emphasizing active participation in the labor force with a traditional program thatguaranteed cash assistance without a mandate to seek employment (Hong Deutsch and Hill2011) Focusing on the psychological well-being of welfare applicants who were singlemothers with preschool-aged children the researchers hypothesized that the treatment wouldincrease employment rate among the welfare recipients and that the treatment-inducedincrease in employment would likely reduce depressive symptoms under the new program

7INTRODUCTION

They also hypothesized that in contrast the same increase in employment was unlikely tohave a psychological impact under the traditional program Hence the treatment-by-mediatorinteraction effect on the outcome was an essential component of the intermediate process

We will show that by defining the indirect effect and the direct effect in terms of potentialoutcomes one can avoid invoking unwarranted assumptions about the unknown structuralrelationships (Holland 1988 Pearl 2001 Robins and Greenland 1992) The potential out-comes framework has further clarified that in a causal mediation theory the treatment and themediator are both conceivably manipulable In the aforementioned example a welfare appli-cant could be assigned at random to either the new program or the traditional one Once havingbeen assigned to one of the programs the individual might or might not be employed due tovarious structural constraints market fluctuations or other random events typically beyondher control

Because the indirect effect and the direct effect are defined in terms of potential outcomesrather than on the basis of any specific regression models it becomes possible to distinguishthe definitions of these causal effects from their identification and estimation The identifica-tion step relates a causal parameter to observable population data For example for individualsassigned at random to an experimental condition their average counterfactual outcome asso-ciated with the control condition is expected to be equal to the average observable outcome ofthose assigned to the control condition Therefore the population average causal effect can beeasily identified in a randomized experiment The estimation step then relates the sample datato the observable population quantities involved in identification while taking into account thedegree of sampling variability

A major challenge in identification is that the mediatorndashoutcome relationship tends tobe confounded by selection even if the treatment is randomized Chapter 9 reviews variousexperimental designs that have been proposed by past researchers for studying causalmediation mechanisms Chapter 10 compares the identification assumptions and the analyticprocedures across a wide range of analytic methods These discussions are followed by anintroduction of the ratio-of-mediator-probability weighting (RMPW) strategy in Chapter 11In comparison with most existing strategies for causal mediation analysis RMPW relies onrelatively fewer identification assumptions and model-based assumptions Chapters 12 and13 will show that this new strategy can be applied broadly with extensions to multilevelexperimental designs and to studies of complex mediation mechanisms involving multiplemediators

113 Spill-over effects of a treatment

It is well known in vaccination research that an unvaccinated person can benefit when mostother people in the local community are vaccinated This is viewed as a spill-over of the treat-ment effect from the vaccinated people to an unvaccinated person Similarly due to socialinteractions among individuals or groups of individuals an intervention received by somemay generate a spill-over impact on others affiliated with the same organization or connectedthrough the same network For example effective policing in one neighborhood may driveoffenders to operate in other neighborhoods In evaluating an innovative community policingprogram researchers found that a neighborhood not assigned to community policing tended tosuffer if the surrounding neighborhoods were assigned to community policing This is aninstance in which a well-intended treatment generates an unintended negative spill-over effectHowever being assigned to community policy was found particularly beneficial when thesurrounding neighborhoods also received the intervention (Verbitsky-Savitz and Raudenbush

8 CAUSALITY IN A SOCIAL WORLD

2012) In another example the impact of a school-based mentoring program targeted atstudents displaying delinquent behaviors may be enhanced for a focal student if his or herat-risk peers are assigned to the mentoring program at the same time

The previous examples challenge the ldquostable unit treatment value assumptionrdquo (SUTVA)that has been invoked in most causal inference studies This assumption states that an indi-vidualrsquos potential response to a treatment depends neither on how the treatment is assignednor on the treatment assignments of other individuals (Rubin 1980 1986) Rubin and othersbelieve that without resorting to SUTVA the causal effect of a treatment becomes hard todefine and that causal inference may become intractable However as Sobel (2006) has illus-trated with the Moving to Opportunity (MTO) experiment that offered housing vouchersenabling low-income families to move to low-poverty neighborhoods there are possible con-sequences of violating SUTVA in estimating the treatment effects on outcomes such as safetyand self-sufficiency Because social interactions among individuals may affect whether onevolunteers to participate in the study whether one moves to a low-poverty neighborhood afterreceiving the housing voucher as well as housing project residentsrsquo subjective perceptions oftheir neighborhoods the program may have a nonzero impact on the potential outcome of theuntreated As a result despite the randomization of treatment assignment the mean differencein the observed outcome between the treated units and the untreated units may be biased forthe average treatment effect Rather the observable quantity is the difference between theeffect of treating some rather than treating none for the treated and the effect of treating somerather than treating none for the untreated the latter being the pure spill-over effect for theuntreated

In making attempts to relax SUTVA researchers have proposed several alternative frame-works that incorporate possible spill-over effects (see Hong and Raudenbush 2013 for areview) Hong (2004) presented a model that involves treatments and treatment settingsA treatment setting for an individual is a local environment constituted by a set of agentsand participants along with their treatment assignments An individualrsquos potential outcomevalue under a given treatment is assumed stable when the treatment setting is fixed the poten-tial outcome may take different values when the treatment setting shifts One may investigatewhether the treatment effect depends on the treatment setting Applying this framework Hongand Raudenbush (2006) examined the effect on a childrsquos academic growth of retaining thechild in kindergarten rather than promoting the child to first grade when a relatively smallproportion of low-achieving peers in the same school are retained as opposed to when a rel-atively large proportion of the peers are retained Hudgens and Halloran (2008) presented arelated framework in which the effect on an individual of the treatment received by this indi-vidual is distinguished from the effect on the individual of the treatment received by others inthe same local community These effects can be identified if communities are assigned atrandom to different treatment assignment strategies (such as retaining a large proportion oflow-achieving students as opposed to retaining a small proportion of such students) and sub-sequently individuals within a community are randomized for treatment Chapter 14 reviewsthese frameworks and discusses identification and estimation strategies

Social contagion may also serve as an important channel through which the effect of anintervention is transmitted For example a school-wide intervention may reduce aggressivebehaviors and thereby improve studentsrsquo psychological well-being by improving the qualityof interpersonal relationships in other classes as well as in onersquos own class This is becausechildren interact not simply with their classmates but also with those from other classes in thehallways or on the playground (VanderWeele et al 2013) In this case the spill-over betweenclasses becomes a part of the mediation mechanism In a study of student mentoring whether

9INTRODUCTION

an individual actually participated in the mentoring program despite the initial treatmentassignment is typically viewed as a primary mediator The proportion of onersquos peers that par-ticipated in the program may act as a second mediator A treatment may also exert its impactpartly through regrouping individuals and thereby changing peer composition (Hong andNomi 2012) Chapter 15 discusses analytic strategies for detecting spill-over as a part ofthe causal mediation mechanism

12 Weighting methods for causal inference

In most behavioral and social science applications major methodological challenges arise dueto the selection of treatments in a quasiexperimental study and the selection of mediator valuesin experimental and quasiexperimental studies Statistical methods most familiar to research-ers in social sciences are often inadequate for causal inferences with regard to multivaluedtreatments moderation mediation and spill-over The book offers a major revision to under-standing of causality in the social world by introducing two complementary weighting stra-tegies both featuring nonparametric approaches to estimating these causal effects The newweighting methods greatly simplify model specifications while enhancing the robustness ofresults by minimizing the reliance on some key assumptions

The propensity score-basedMMWSmethod removes selection bias associated with a largenumber of covariates by equating the pretreatment composition between treatment groups(Hong 2010a 2012 Huang et al 2005) Unlike propensity score matching and stratificationthat are mostly restricted to evaluations of binary treatments the MMWS method is flexiblefor evaluating binary and multivalued treatments by approximating a completely randomizedexperiment In evaluating whether the treatment effects differ across subpopulations definedby individual characteristics or treatment settings researchers may assign weights within eachsubpopulation in order to approximate a randomized block design To investigate whetherone treatment moderates the effect of another concurrent treatment researchers may assignweights to the data to approximate a factorial randomized design The method can also beused to assess whether the effect of an initial treatment is amplified or weakened by a subse-quent treatment or to identify an optimal treatment sequence through approximating a sequen-tial randomized experiment Even though such analyses can similarly be conducted throughinverse-probability-of-treatment weighting (IPTW) that has been increasingly employed inepidemiological research (Hernaacuten Brumback and Robins 2000 Robins Hernaacuten andBrumback 2000) IPTW is known for bias and imprecision in estimation especially whenthe propensity score models are misspecified in their functional forms (Hong 2010a Kangand Schafer 2007 Schafer and Kang 2008 Waernbaum 2012) In contrast the nonparamet-ric MMWS method displays a relatively high level of robustness despite such misspecifica-tions and also gains efficiency as indicated by simulation results (Hong 2010a)

To study causal mediation mechanisms the RMPW method decomposes the total effectof a treatment into an ldquoindirect effectrdquo transmitted through a specific mediator and a ldquodirecteffectrdquo representing unspecified mechanisms In contrast with most existing methods formediation analysis the RMPW-adjusted outcome model is extremely simple and is nonpara-metric in nature It generates estimates of the causal effects along with their sampling errorswhile adjusting for pretreatment covariates that confound the mediatorndashoutcome relationshipsthrough weighting The method applies regardless of the distribution of the outcome the dis-tribution of the mediator or the functional relationship between the outcome and the mediator

10 CAUSALITY IN A SOCIAL WORLD

Page 18: Thumbnail - download.e-bookshelf.de...4.2.4 IPTW for bias removal 84 4.3 MMWS 86 4.3.1 Theoretical rationale 86 ... 8.3.4.2 Growth model in the absence of confounding 200 8.3.4.3 Weighted

Part I

OVERVIEW

1

Introduction

According to an ancient Chinese fable a farmer who was eager to help his crops grow wentinto his field and pulled each seedling upward After exhausting himself with the work heannounced to his family that they were going to have a good harvest only to find the nextmorning that the plants had wilted and died Readers with minimal agricultural knowledgemay immediately point out the following the farmerrsquos intervention theory was based on acorrect observation that crops that grow taller tend to produce more yield Yet his hypothesisreflects a false understanding of the cause and the effectmdashthat seedlings pulled to be tallerwould yield as much as seedlings thriving on their own

In their classic Design and Analysis of Experiments Hinkelmann and Kempthorne (1994updated version of Kempthorne 1952) discussed two types of science descriptive science andthe development of theory These two types of science are interrelated in the following senseobservations of an event and other related events often selected and classified for descriptionby scientists naturally lead to one or more explanations that we call ldquotheoretical hypothesesrdquowhich are then screened and falsified by means of further observations experimentation andanalyses (Popper 1963) The experiment of pulling seedlings to be taller was costly but didserve the purpose of advancing this farmerrsquos knowledge of ldquowhat does not workrdquo To developa successful intervention in this case would require a series of empirical tests of explicittheories identifying potential contributors to crop growth This iterative process graduallydeepens our knowledge of the relationships between supposed causes and effectsmdashthatis causalitymdashand may eventually increase the success of agricultural medical and socialinterventions

11 Concepts of moderation mediation and spill-over

Although the story of the ancient farmer is fictitious numerous examples can be found in thereal world in which well-intended interventions fail to produce the intended benefits or inmany cases even lead to unintended consequences ldquoInterventionsrdquo and ldquotreatmentsrdquo usedinterchangeably in this book broadly refer to actions taken by agents or circumstances expe-rienced by an individual or groups of individuals Interventions are regularly seen in educa-tion physical and mental health social services business politics and law enforcement In an

Causality in a Social World Moderation Meditation and Spill-over First Edition Guanglei Hongcopy 2015 John Wiley amp Sons Ltd Published 2015 by John Wiley amp Sons Ltd

education intervention for example teachers are typically the agents who deliver a treatmentto students while the impact of the treatment on student outcomes is of ultimate causalinterest Some educational practices such as ldquoteaching to the testrdquo have been criticized tobe nearly as counterproductive as the attempt of helping seedlings grow by pulling themupward ldquoInterventionsrdquo and ldquotreatmentsrdquo under consideration do not exclude undesiredexperiences such as exposure to poverty abuse crime or bereavement A treatment plannedor unplanned becomes a focus of research if there are theoretical reasons to anticipate itsimpact positive or negative on the well-being of individuals who are embedded in socialsettings including families classrooms schools neighborhoods and workplaces

In social science research in general and in policy and program evaluations in particularquestions concerning whether an intervention works and if so which version of the interven-tion works for whom under what conditions and why are key to the advancement of scien-tific and practical knowledge Although most empirical investigations in the social sciencesconcentrate on the average effect of a treatment for a specific population as opposed to theabsence of such a treatment (ie the control condition) in-depth theoretical reasoning withregard to how the causal effect is generated substantiated by compelling empirical evidenceis crucial for advancing scientific understanding

First when there are multiple versions or different dosages of the treatment or when thereare multiple versions of the control condition a binary divide between ldquothe treatmentrdquo andldquothe controlrdquo may not be as informative as fine-grained comparisons across for exampleldquotreatment version Ardquo ldquotreatment version Brdquo ldquocontrol version Ardquo and ldquocontrol versionBrdquo For example expanding the federally funded Head Start program to poor children isexpected to generate a greater benefit when few early childhood education alternatives areavailable (call it ldquocontrol version Ardquo) than when there is an abundance of alternatives includ-ing state-sponsored preschool programs (call it ldquocontrol version Brdquo)

Second the effect of an intervention will likely vary among individuals or across socialsettings A famous example comes from medical research the well-publicized cardiovascularbenefits of initiating estrogen therapy during menopause were contradicted later by experi-mental findings that the same therapy increased postmenopausal womenrsquos risk for heartattacks The effect of an intervention may also depend on the provision of some other con-current or subsequent interventions Such heterogeneous effects are often characterized asmoderated effects in the literature

Third alternative theories may provide competing explanations for the causal mechan-isms that is the processes through which the intervention produces its effect A theoreticalconstruct characterizing the hypothesized intermediate process is called a mediator of theintervention effect The fictitious farmer never developed an elaborate theory as to whatcaused some seedlings to surpass others in growth Once scientists revealed the causal rela-tionship between access to chemical nutrients in soil and plant growth wide applications ofchemically synthesized fertilizers finally led to a major increase in crop production

Finally it is well known in agricultural experiments that a new type of fertilizer applied toone plot may spill-over to the next plot Because social connections among individuals areprevalent within organizations or through networks an individualrsquos response to the treatmentmay similarly depend on the treatment for other individuals in the same social setting whichmay lead to possible spill-overs of intervention effects among individual human beings

Answering questions with regard to moderation mediation and spill-over poses majorconceptual and analytic challenges To date psychological research often presents well-articulated theories of causal mechanisms relating stimuli to responses Yet researchers often

4 CAUSALITY IN A SOCIAL WORLD

lack rigorous analytic strategies for empirically screening competing theories explaining theobserved effect Sociologists have keen interest in the spill-over of treatment effects transmit-ted through social interactions yet have produced limited evidence quantifying such effectsAs many have pointed out in general terminological ambiguity and conceptual confusionhave been prevalent in the published applied research (Holmbeck 1997 Kraemer et al2002 2008)

A new framework for conceptualizing moderated mediated and spill-over effects hasemerged relatively recently in the statistics and econometrics literature on causal inference(eg Abbring and Heckman 2007 Frangakis and Rubin 2002 Heckman and Vytlacil2007a b Holland 1988 Hong and Raudenbush 2006 Hudgens and Halloran 2008Jo 2008 Pearl 2001 Robins and Greenland 1992 Sobel 2008) The potential for furtherconceptual and methodological development and for broad applications in the field of behav-ioral and social sciences promises to greatly advance the empirical basis of our knowledgeabout causality in the social world

This book clarifies theoretical concepts and introduces innovative statistical strategies forinvestigating the average effects of multivalued treatments moderated treatment effectsmediated treatment effects and spill-over effects in experimental or quasiexperimental dataDefining individual-specific and population average treatment effects in terms of potentialoutcomes the book relates the mathematical forms to the substantive meanings of moderatedmediated and spill-over effects in the context of application examples It also explicates andevaluates identification assumptions and contrasts innovative statistical strategies with con-ventional analytic methods

111 Moderated treatment effects

It is hard to accept the assumption that a treatment would produce the same impact for everyindividual in every possible circumstance Understanding the heterogeneity of treatmenteffects therefore is key to the development of causal theories For example some studiesreported that estrogen therapy improved cardiovascular health among women who initiatedits use during menopause According to a series of other studies however the use of estrogentherapy increased postmenopausal womenrsquos risk for heart attacks (Grodstein et al 19962000 Writing Group for the Womenrsquos Health Initiative Investigators 2002) The sharp con-trast of findings from these studies led to the hypothesis that age of initiation moderates theeffect of estrogen therapy on womenrsquos health (Manson and Bassuk 2007 Rossouw et al2007) Revelations of the moderated causal relationship greatly enrich theoretical understand-ing and in this case directly inform clinical practice

In another example dividing students by ability into small groups for reading instructionhas been controversial for decades Many believed that the practice benefits high-ability stu-dents at the expense of their low-ability peers and exacerbates educational inequality (Grantand Rothenberg 1986 Rowan and Miracle 1983 Trimble and Sinclair 1987) Recentevidence has shown that first of all the effect of ability grouping depends on a number ofconditions including whether enough time is allocated to reading instruction and how wellthe class can be managed Hence instructional time and class management are among theadditional moderators to consider for determining the merits of ability grouping (Hong andHong 2009 Hong et al 2012a b) Moreover further investigations have generated evidencethat contradicts the long-held belief that grouping undermines the interests of low-abilitystudents Quite the contrary researchers have found that grouping is beneficial for students

5INTRODUCTION

with low or medium prior abilitymdashif adequate time is provided for instruction and if the classis well managed On the other hand ability grouping appears to have a minimal impact onhigh-ability studentsrsquo literacy These results were derived from kindergarten data and maynot hold for math instruction and for higher grade levels Replications with different subpo-pulations and in different contexts enable researchers to assess the generalizability of a theory

In a third example one may hypothesize that a student is unlikely to learn algebra well inninth grade without a solid foundation in eighth-grade prealgebra One may also argue that theprogress a student made in learning eighth-grade prealgebra may not be sustained without thefurther enhancement of taking algebra in ninth grade In other words the earlier treatment(prealgebra in eighth grade) and the later treatment (algebra in ninth grade) may reinforceone other each moderates the effect of the other treatment on the final outcome (math achieve-ment at the end of ninth grade) As Hong and Raudenbush (2008) argued the cumulativeeffect of two treatments in a well-aligned sequence may exceed the sum of the benefits oftwo single-year treatments Similarly experiencing two consecutive years of inferior treat-ments such as encountering an incompetent math teacher who dampens a studentrsquos self-efficacy in math learning in eighth grade and then again in ninth grade could do more damagethan the sum of the effect of encountering an incompetent teacher only in eighth grade and theeffect of having such a teacher only in ninth grade

There has been a great amount of conceptual confusion regarding how a moderatorrelates to the treatment and the outcome On one hand many researchers have used the termsldquomoderatorsrdquo and ldquomediatorsrdquo interchangeably without understanding the crucial distinctionbetween the two An overcorrective attempt on the other hand has led to the arbitrary rec-ommendation that a moderator must occur prior to the treatment and be minimally associatedwith the treatment (James and Brett 1984 Kraemer et al 2001 2008) In Chapter 6 weclarify that in essence the causal effect of the treatment on the outcome may depend onthe moderator value A moderator can be a subpopulation identifier a contextual character-istic a concurrent treatment or a preceding or succeeding treatment In the earlier examplesage of estrogen therapy initiation and a studentrsquos prior ability are subpopulation identifiers themanageability of a class which reflects both teacher skills and peer behaviors characterizes thecontext literacy instruction time is a concurrent treatment while prealgebra is a precedingtreatment for algebra A moderator does not have to occur prior to the treatment and doesnot have to be independent of the treatment

Once a moderated causal relationship has been defined in terms of potential outcomes theresearcher then chooses an appropriate experimental design for testing the moderation theoryThe assumptions required for identifying the moderated causal relationships differ across dif-ferent designs and have implications for analyzing experimental and quasiexperimental dataRandomized block designs are suitable for examining individual or contextual characteristicsas potential moderators factorial designs enable one to determine the joint effects of two ormore concurrent treatments and sequential randomized designs are ideal for assessing thecumulative effects of consecutive treatments Multisite randomized trials constitute anadditional type of designs in which the experimental sites are often deliberately sampled torepresent a population of geographical locations or social organizations Site membershipscan be viewed as implicit moderators that summarize a host of features of a local environmentReplications of a treatment over multiple sites allow one to quantify the heterogeneity of treat-ment effects across the sites Chapters 7 and 8 are focused on statistical methods for moder-ation analyses with quasiexperimental data In particular these chapters demonstrate througha number of application examples how a nonparametric marginal mean weighting through

6 CAUSALITY IN A SOCIAL WORLD

stratification (MMWS) method can overcome some important limitations of other existingmethods

Moderation questions are not restricted to treatmentndashoutcome relationships Rather theyare prevalent in investigations of mediation and spill-over This is because heterogeneity oftreatment effects may be explained by the variation in causal mediation mechanisms acrosssubpopulations and contexts It is also because the treatment effect for a focal individualmay depend on whether there is spill-over from other individuals through social interactionsYet similar to the confusion around the concept of moderation among applied researchersthere has been a great amount of misunderstanding with regard to mediation

112 Mediated treatment effects

Questions about mediation are at the core of nearly every scientific theory In its simplestform a theory explains why treatment Z causes outcome Y by hypothesizing an intermediateprocess involving at least one mediatorM that could have been changed by the treatment andcould subsequently have an impact on the outcome A causal mediation analysis then decom-poses the total effect of the treatment on the outcome into two parts the effect transmittedthrough the hypothesized mediator called the indirect effect and the difference betweenthe total effect and the indirect effect called the direct effect The latter represents the treat-ment effect channeled through other unspecified pathways (Alwin and Hauser 1975 Baronand Kenny 1986 Duncan 1966 Shadish and Sweeney 1991) Most researchers in socialsciences have followed the convention of illustrating the direct effect and the indirect effectwith path diagrams and then specifying linear regression models postulated to represent thestructural relationships between the treatment the mediator and the outcome

In Chapter 9 we point out that the approach to defining the indirect effect and the directeffect in terms of path coefficients is often misled by oversimplistic assumptions about thestructural relationships In particular this approach typically overlooks the fact that a treat-ment may generate an impact on the outcome not only by changing the mediator value butalso by changing the mediatorndashoutcome relationship (Judd and Kenny 1981) An examplein Chapter 9 illustrates such a case An experiment randomizes students to either an experi-mental condition which provides them with study materials and encourages them to study fora test or to a control condition which provides neither study materials nor encouragement(Holland 1988 Powers and Swinton 1984) One might hypothesize that encouraging exper-imental group members to study will increase the time they spend studying a focal mediatorin this example which in turn will increase their average test scores One might furtherhypothesize that even without a change in study time providing study materials to experi-mental group members will enable them to study more effectively than they otherwise wouldConsequently the effect of time spent studying might be greater under the experimentalcondition than under the control condition Omitting the treatment-by-mediator interactioneffect will lead to bias in the estimation of the indirect effect and the direct effect

Chapter 11 describes in great detail an evaluation study contrasting a new welfare-to-workprogram emphasizing active participation in the labor force with a traditional program thatguaranteed cash assistance without a mandate to seek employment (Hong Deutsch and Hill2011) Focusing on the psychological well-being of welfare applicants who were singlemothers with preschool-aged children the researchers hypothesized that the treatment wouldincrease employment rate among the welfare recipients and that the treatment-inducedincrease in employment would likely reduce depressive symptoms under the new program

7INTRODUCTION

They also hypothesized that in contrast the same increase in employment was unlikely tohave a psychological impact under the traditional program Hence the treatment-by-mediatorinteraction effect on the outcome was an essential component of the intermediate process

We will show that by defining the indirect effect and the direct effect in terms of potentialoutcomes one can avoid invoking unwarranted assumptions about the unknown structuralrelationships (Holland 1988 Pearl 2001 Robins and Greenland 1992) The potential out-comes framework has further clarified that in a causal mediation theory the treatment and themediator are both conceivably manipulable In the aforementioned example a welfare appli-cant could be assigned at random to either the new program or the traditional one Once havingbeen assigned to one of the programs the individual might or might not be employed due tovarious structural constraints market fluctuations or other random events typically beyondher control

Because the indirect effect and the direct effect are defined in terms of potential outcomesrather than on the basis of any specific regression models it becomes possible to distinguishthe definitions of these causal effects from their identification and estimation The identifica-tion step relates a causal parameter to observable population data For example for individualsassigned at random to an experimental condition their average counterfactual outcome asso-ciated with the control condition is expected to be equal to the average observable outcome ofthose assigned to the control condition Therefore the population average causal effect can beeasily identified in a randomized experiment The estimation step then relates the sample datato the observable population quantities involved in identification while taking into account thedegree of sampling variability

A major challenge in identification is that the mediatorndashoutcome relationship tends tobe confounded by selection even if the treatment is randomized Chapter 9 reviews variousexperimental designs that have been proposed by past researchers for studying causalmediation mechanisms Chapter 10 compares the identification assumptions and the analyticprocedures across a wide range of analytic methods These discussions are followed by anintroduction of the ratio-of-mediator-probability weighting (RMPW) strategy in Chapter 11In comparison with most existing strategies for causal mediation analysis RMPW relies onrelatively fewer identification assumptions and model-based assumptions Chapters 12 and13 will show that this new strategy can be applied broadly with extensions to multilevelexperimental designs and to studies of complex mediation mechanisms involving multiplemediators

113 Spill-over effects of a treatment

It is well known in vaccination research that an unvaccinated person can benefit when mostother people in the local community are vaccinated This is viewed as a spill-over of the treat-ment effect from the vaccinated people to an unvaccinated person Similarly due to socialinteractions among individuals or groups of individuals an intervention received by somemay generate a spill-over impact on others affiliated with the same organization or connectedthrough the same network For example effective policing in one neighborhood may driveoffenders to operate in other neighborhoods In evaluating an innovative community policingprogram researchers found that a neighborhood not assigned to community policing tended tosuffer if the surrounding neighborhoods were assigned to community policing This is aninstance in which a well-intended treatment generates an unintended negative spill-over effectHowever being assigned to community policy was found particularly beneficial when thesurrounding neighborhoods also received the intervention (Verbitsky-Savitz and Raudenbush

8 CAUSALITY IN A SOCIAL WORLD

2012) In another example the impact of a school-based mentoring program targeted atstudents displaying delinquent behaviors may be enhanced for a focal student if his or herat-risk peers are assigned to the mentoring program at the same time

The previous examples challenge the ldquostable unit treatment value assumptionrdquo (SUTVA)that has been invoked in most causal inference studies This assumption states that an indi-vidualrsquos potential response to a treatment depends neither on how the treatment is assignednor on the treatment assignments of other individuals (Rubin 1980 1986) Rubin and othersbelieve that without resorting to SUTVA the causal effect of a treatment becomes hard todefine and that causal inference may become intractable However as Sobel (2006) has illus-trated with the Moving to Opportunity (MTO) experiment that offered housing vouchersenabling low-income families to move to low-poverty neighborhoods there are possible con-sequences of violating SUTVA in estimating the treatment effects on outcomes such as safetyand self-sufficiency Because social interactions among individuals may affect whether onevolunteers to participate in the study whether one moves to a low-poverty neighborhood afterreceiving the housing voucher as well as housing project residentsrsquo subjective perceptions oftheir neighborhoods the program may have a nonzero impact on the potential outcome of theuntreated As a result despite the randomization of treatment assignment the mean differencein the observed outcome between the treated units and the untreated units may be biased forthe average treatment effect Rather the observable quantity is the difference between theeffect of treating some rather than treating none for the treated and the effect of treating somerather than treating none for the untreated the latter being the pure spill-over effect for theuntreated

In making attempts to relax SUTVA researchers have proposed several alternative frame-works that incorporate possible spill-over effects (see Hong and Raudenbush 2013 for areview) Hong (2004) presented a model that involves treatments and treatment settingsA treatment setting for an individual is a local environment constituted by a set of agentsand participants along with their treatment assignments An individualrsquos potential outcomevalue under a given treatment is assumed stable when the treatment setting is fixed the poten-tial outcome may take different values when the treatment setting shifts One may investigatewhether the treatment effect depends on the treatment setting Applying this framework Hongand Raudenbush (2006) examined the effect on a childrsquos academic growth of retaining thechild in kindergarten rather than promoting the child to first grade when a relatively smallproportion of low-achieving peers in the same school are retained as opposed to when a rel-atively large proportion of the peers are retained Hudgens and Halloran (2008) presented arelated framework in which the effect on an individual of the treatment received by this indi-vidual is distinguished from the effect on the individual of the treatment received by others inthe same local community These effects can be identified if communities are assigned atrandom to different treatment assignment strategies (such as retaining a large proportion oflow-achieving students as opposed to retaining a small proportion of such students) and sub-sequently individuals within a community are randomized for treatment Chapter 14 reviewsthese frameworks and discusses identification and estimation strategies

Social contagion may also serve as an important channel through which the effect of anintervention is transmitted For example a school-wide intervention may reduce aggressivebehaviors and thereby improve studentsrsquo psychological well-being by improving the qualityof interpersonal relationships in other classes as well as in onersquos own class This is becausechildren interact not simply with their classmates but also with those from other classes in thehallways or on the playground (VanderWeele et al 2013) In this case the spill-over betweenclasses becomes a part of the mediation mechanism In a study of student mentoring whether

9INTRODUCTION

an individual actually participated in the mentoring program despite the initial treatmentassignment is typically viewed as a primary mediator The proportion of onersquos peers that par-ticipated in the program may act as a second mediator A treatment may also exert its impactpartly through regrouping individuals and thereby changing peer composition (Hong andNomi 2012) Chapter 15 discusses analytic strategies for detecting spill-over as a part ofthe causal mediation mechanism

12 Weighting methods for causal inference

In most behavioral and social science applications major methodological challenges arise dueto the selection of treatments in a quasiexperimental study and the selection of mediator valuesin experimental and quasiexperimental studies Statistical methods most familiar to research-ers in social sciences are often inadequate for causal inferences with regard to multivaluedtreatments moderation mediation and spill-over The book offers a major revision to under-standing of causality in the social world by introducing two complementary weighting stra-tegies both featuring nonparametric approaches to estimating these causal effects The newweighting methods greatly simplify model specifications while enhancing the robustness ofresults by minimizing the reliance on some key assumptions

The propensity score-basedMMWSmethod removes selection bias associated with a largenumber of covariates by equating the pretreatment composition between treatment groups(Hong 2010a 2012 Huang et al 2005) Unlike propensity score matching and stratificationthat are mostly restricted to evaluations of binary treatments the MMWS method is flexiblefor evaluating binary and multivalued treatments by approximating a completely randomizedexperiment In evaluating whether the treatment effects differ across subpopulations definedby individual characteristics or treatment settings researchers may assign weights within eachsubpopulation in order to approximate a randomized block design To investigate whetherone treatment moderates the effect of another concurrent treatment researchers may assignweights to the data to approximate a factorial randomized design The method can also beused to assess whether the effect of an initial treatment is amplified or weakened by a subse-quent treatment or to identify an optimal treatment sequence through approximating a sequen-tial randomized experiment Even though such analyses can similarly be conducted throughinverse-probability-of-treatment weighting (IPTW) that has been increasingly employed inepidemiological research (Hernaacuten Brumback and Robins 2000 Robins Hernaacuten andBrumback 2000) IPTW is known for bias and imprecision in estimation especially whenthe propensity score models are misspecified in their functional forms (Hong 2010a Kangand Schafer 2007 Schafer and Kang 2008 Waernbaum 2012) In contrast the nonparamet-ric MMWS method displays a relatively high level of robustness despite such misspecifica-tions and also gains efficiency as indicated by simulation results (Hong 2010a)

To study causal mediation mechanisms the RMPW method decomposes the total effectof a treatment into an ldquoindirect effectrdquo transmitted through a specific mediator and a ldquodirecteffectrdquo representing unspecified mechanisms In contrast with most existing methods formediation analysis the RMPW-adjusted outcome model is extremely simple and is nonpara-metric in nature It generates estimates of the causal effects along with their sampling errorswhile adjusting for pretreatment covariates that confound the mediatorndashoutcome relationshipsthrough weighting The method applies regardless of the distribution of the outcome the dis-tribution of the mediator or the functional relationship between the outcome and the mediator

10 CAUSALITY IN A SOCIAL WORLD

Page 19: Thumbnail - download.e-bookshelf.de...4.2.4 IPTW for bias removal 84 4.3 MMWS 86 4.3.1 Theoretical rationale 86 ... 8.3.4.2 Growth model in the absence of confounding 200 8.3.4.3 Weighted

1

Introduction

According to an ancient Chinese fable a farmer who was eager to help his crops grow wentinto his field and pulled each seedling upward After exhausting himself with the work heannounced to his family that they were going to have a good harvest only to find the nextmorning that the plants had wilted and died Readers with minimal agricultural knowledgemay immediately point out the following the farmerrsquos intervention theory was based on acorrect observation that crops that grow taller tend to produce more yield Yet his hypothesisreflects a false understanding of the cause and the effectmdashthat seedlings pulled to be tallerwould yield as much as seedlings thriving on their own

In their classic Design and Analysis of Experiments Hinkelmann and Kempthorne (1994updated version of Kempthorne 1952) discussed two types of science descriptive science andthe development of theory These two types of science are interrelated in the following senseobservations of an event and other related events often selected and classified for descriptionby scientists naturally lead to one or more explanations that we call ldquotheoretical hypothesesrdquowhich are then screened and falsified by means of further observations experimentation andanalyses (Popper 1963) The experiment of pulling seedlings to be taller was costly but didserve the purpose of advancing this farmerrsquos knowledge of ldquowhat does not workrdquo To developa successful intervention in this case would require a series of empirical tests of explicittheories identifying potential contributors to crop growth This iterative process graduallydeepens our knowledge of the relationships between supposed causes and effectsmdashthatis causalitymdashand may eventually increase the success of agricultural medical and socialinterventions

11 Concepts of moderation mediation and spill-over

Although the story of the ancient farmer is fictitious numerous examples can be found in thereal world in which well-intended interventions fail to produce the intended benefits or inmany cases even lead to unintended consequences ldquoInterventionsrdquo and ldquotreatmentsrdquo usedinterchangeably in this book broadly refer to actions taken by agents or circumstances expe-rienced by an individual or groups of individuals Interventions are regularly seen in educa-tion physical and mental health social services business politics and law enforcement In an

Causality in a Social World Moderation Meditation and Spill-over First Edition Guanglei Hongcopy 2015 John Wiley amp Sons Ltd Published 2015 by John Wiley amp Sons Ltd

education intervention for example teachers are typically the agents who deliver a treatmentto students while the impact of the treatment on student outcomes is of ultimate causalinterest Some educational practices such as ldquoteaching to the testrdquo have been criticized tobe nearly as counterproductive as the attempt of helping seedlings grow by pulling themupward ldquoInterventionsrdquo and ldquotreatmentsrdquo under consideration do not exclude undesiredexperiences such as exposure to poverty abuse crime or bereavement A treatment plannedor unplanned becomes a focus of research if there are theoretical reasons to anticipate itsimpact positive or negative on the well-being of individuals who are embedded in socialsettings including families classrooms schools neighborhoods and workplaces

In social science research in general and in policy and program evaluations in particularquestions concerning whether an intervention works and if so which version of the interven-tion works for whom under what conditions and why are key to the advancement of scien-tific and practical knowledge Although most empirical investigations in the social sciencesconcentrate on the average effect of a treatment for a specific population as opposed to theabsence of such a treatment (ie the control condition) in-depth theoretical reasoning withregard to how the causal effect is generated substantiated by compelling empirical evidenceis crucial for advancing scientific understanding

First when there are multiple versions or different dosages of the treatment or when thereare multiple versions of the control condition a binary divide between ldquothe treatmentrdquo andldquothe controlrdquo may not be as informative as fine-grained comparisons across for exampleldquotreatment version Ardquo ldquotreatment version Brdquo ldquocontrol version Ardquo and ldquocontrol versionBrdquo For example expanding the federally funded Head Start program to poor children isexpected to generate a greater benefit when few early childhood education alternatives areavailable (call it ldquocontrol version Ardquo) than when there is an abundance of alternatives includ-ing state-sponsored preschool programs (call it ldquocontrol version Brdquo)

Second the effect of an intervention will likely vary among individuals or across socialsettings A famous example comes from medical research the well-publicized cardiovascularbenefits of initiating estrogen therapy during menopause were contradicted later by experi-mental findings that the same therapy increased postmenopausal womenrsquos risk for heartattacks The effect of an intervention may also depend on the provision of some other con-current or subsequent interventions Such heterogeneous effects are often characterized asmoderated effects in the literature

Third alternative theories may provide competing explanations for the causal mechan-isms that is the processes through which the intervention produces its effect A theoreticalconstruct characterizing the hypothesized intermediate process is called a mediator of theintervention effect The fictitious farmer never developed an elaborate theory as to whatcaused some seedlings to surpass others in growth Once scientists revealed the causal rela-tionship between access to chemical nutrients in soil and plant growth wide applications ofchemically synthesized fertilizers finally led to a major increase in crop production

Finally it is well known in agricultural experiments that a new type of fertilizer applied toone plot may spill-over to the next plot Because social connections among individuals areprevalent within organizations or through networks an individualrsquos response to the treatmentmay similarly depend on the treatment for other individuals in the same social setting whichmay lead to possible spill-overs of intervention effects among individual human beings

Answering questions with regard to moderation mediation and spill-over poses majorconceptual and analytic challenges To date psychological research often presents well-articulated theories of causal mechanisms relating stimuli to responses Yet researchers often

4 CAUSALITY IN A SOCIAL WORLD

lack rigorous analytic strategies for empirically screening competing theories explaining theobserved effect Sociologists have keen interest in the spill-over of treatment effects transmit-ted through social interactions yet have produced limited evidence quantifying such effectsAs many have pointed out in general terminological ambiguity and conceptual confusionhave been prevalent in the published applied research (Holmbeck 1997 Kraemer et al2002 2008)

A new framework for conceptualizing moderated mediated and spill-over effects hasemerged relatively recently in the statistics and econometrics literature on causal inference(eg Abbring and Heckman 2007 Frangakis and Rubin 2002 Heckman and Vytlacil2007a b Holland 1988 Hong and Raudenbush 2006 Hudgens and Halloran 2008Jo 2008 Pearl 2001 Robins and Greenland 1992 Sobel 2008) The potential for furtherconceptual and methodological development and for broad applications in the field of behav-ioral and social sciences promises to greatly advance the empirical basis of our knowledgeabout causality in the social world

This book clarifies theoretical concepts and introduces innovative statistical strategies forinvestigating the average effects of multivalued treatments moderated treatment effectsmediated treatment effects and spill-over effects in experimental or quasiexperimental dataDefining individual-specific and population average treatment effects in terms of potentialoutcomes the book relates the mathematical forms to the substantive meanings of moderatedmediated and spill-over effects in the context of application examples It also explicates andevaluates identification assumptions and contrasts innovative statistical strategies with con-ventional analytic methods

111 Moderated treatment effects

It is hard to accept the assumption that a treatment would produce the same impact for everyindividual in every possible circumstance Understanding the heterogeneity of treatmenteffects therefore is key to the development of causal theories For example some studiesreported that estrogen therapy improved cardiovascular health among women who initiatedits use during menopause According to a series of other studies however the use of estrogentherapy increased postmenopausal womenrsquos risk for heart attacks (Grodstein et al 19962000 Writing Group for the Womenrsquos Health Initiative Investigators 2002) The sharp con-trast of findings from these studies led to the hypothesis that age of initiation moderates theeffect of estrogen therapy on womenrsquos health (Manson and Bassuk 2007 Rossouw et al2007) Revelations of the moderated causal relationship greatly enrich theoretical understand-ing and in this case directly inform clinical practice

In another example dividing students by ability into small groups for reading instructionhas been controversial for decades Many believed that the practice benefits high-ability stu-dents at the expense of their low-ability peers and exacerbates educational inequality (Grantand Rothenberg 1986 Rowan and Miracle 1983 Trimble and Sinclair 1987) Recentevidence has shown that first of all the effect of ability grouping depends on a number ofconditions including whether enough time is allocated to reading instruction and how wellthe class can be managed Hence instructional time and class management are among theadditional moderators to consider for determining the merits of ability grouping (Hong andHong 2009 Hong et al 2012a b) Moreover further investigations have generated evidencethat contradicts the long-held belief that grouping undermines the interests of low-abilitystudents Quite the contrary researchers have found that grouping is beneficial for students

5INTRODUCTION

with low or medium prior abilitymdashif adequate time is provided for instruction and if the classis well managed On the other hand ability grouping appears to have a minimal impact onhigh-ability studentsrsquo literacy These results were derived from kindergarten data and maynot hold for math instruction and for higher grade levels Replications with different subpo-pulations and in different contexts enable researchers to assess the generalizability of a theory

In a third example one may hypothesize that a student is unlikely to learn algebra well inninth grade without a solid foundation in eighth-grade prealgebra One may also argue that theprogress a student made in learning eighth-grade prealgebra may not be sustained without thefurther enhancement of taking algebra in ninth grade In other words the earlier treatment(prealgebra in eighth grade) and the later treatment (algebra in ninth grade) may reinforceone other each moderates the effect of the other treatment on the final outcome (math achieve-ment at the end of ninth grade) As Hong and Raudenbush (2008) argued the cumulativeeffect of two treatments in a well-aligned sequence may exceed the sum of the benefits oftwo single-year treatments Similarly experiencing two consecutive years of inferior treat-ments such as encountering an incompetent math teacher who dampens a studentrsquos self-efficacy in math learning in eighth grade and then again in ninth grade could do more damagethan the sum of the effect of encountering an incompetent teacher only in eighth grade and theeffect of having such a teacher only in ninth grade

There has been a great amount of conceptual confusion regarding how a moderatorrelates to the treatment and the outcome On one hand many researchers have used the termsldquomoderatorsrdquo and ldquomediatorsrdquo interchangeably without understanding the crucial distinctionbetween the two An overcorrective attempt on the other hand has led to the arbitrary rec-ommendation that a moderator must occur prior to the treatment and be minimally associatedwith the treatment (James and Brett 1984 Kraemer et al 2001 2008) In Chapter 6 weclarify that in essence the causal effect of the treatment on the outcome may depend onthe moderator value A moderator can be a subpopulation identifier a contextual character-istic a concurrent treatment or a preceding or succeeding treatment In the earlier examplesage of estrogen therapy initiation and a studentrsquos prior ability are subpopulation identifiers themanageability of a class which reflects both teacher skills and peer behaviors characterizes thecontext literacy instruction time is a concurrent treatment while prealgebra is a precedingtreatment for algebra A moderator does not have to occur prior to the treatment and doesnot have to be independent of the treatment

Once a moderated causal relationship has been defined in terms of potential outcomes theresearcher then chooses an appropriate experimental design for testing the moderation theoryThe assumptions required for identifying the moderated causal relationships differ across dif-ferent designs and have implications for analyzing experimental and quasiexperimental dataRandomized block designs are suitable for examining individual or contextual characteristicsas potential moderators factorial designs enable one to determine the joint effects of two ormore concurrent treatments and sequential randomized designs are ideal for assessing thecumulative effects of consecutive treatments Multisite randomized trials constitute anadditional type of designs in which the experimental sites are often deliberately sampled torepresent a population of geographical locations or social organizations Site membershipscan be viewed as implicit moderators that summarize a host of features of a local environmentReplications of a treatment over multiple sites allow one to quantify the heterogeneity of treat-ment effects across the sites Chapters 7 and 8 are focused on statistical methods for moder-ation analyses with quasiexperimental data In particular these chapters demonstrate througha number of application examples how a nonparametric marginal mean weighting through

6 CAUSALITY IN A SOCIAL WORLD

stratification (MMWS) method can overcome some important limitations of other existingmethods

Moderation questions are not restricted to treatmentndashoutcome relationships Rather theyare prevalent in investigations of mediation and spill-over This is because heterogeneity oftreatment effects may be explained by the variation in causal mediation mechanisms acrosssubpopulations and contexts It is also because the treatment effect for a focal individualmay depend on whether there is spill-over from other individuals through social interactionsYet similar to the confusion around the concept of moderation among applied researchersthere has been a great amount of misunderstanding with regard to mediation

112 Mediated treatment effects

Questions about mediation are at the core of nearly every scientific theory In its simplestform a theory explains why treatment Z causes outcome Y by hypothesizing an intermediateprocess involving at least one mediatorM that could have been changed by the treatment andcould subsequently have an impact on the outcome A causal mediation analysis then decom-poses the total effect of the treatment on the outcome into two parts the effect transmittedthrough the hypothesized mediator called the indirect effect and the difference betweenthe total effect and the indirect effect called the direct effect The latter represents the treat-ment effect channeled through other unspecified pathways (Alwin and Hauser 1975 Baronand Kenny 1986 Duncan 1966 Shadish and Sweeney 1991) Most researchers in socialsciences have followed the convention of illustrating the direct effect and the indirect effectwith path diagrams and then specifying linear regression models postulated to represent thestructural relationships between the treatment the mediator and the outcome

In Chapter 9 we point out that the approach to defining the indirect effect and the directeffect in terms of path coefficients is often misled by oversimplistic assumptions about thestructural relationships In particular this approach typically overlooks the fact that a treat-ment may generate an impact on the outcome not only by changing the mediator value butalso by changing the mediatorndashoutcome relationship (Judd and Kenny 1981) An examplein Chapter 9 illustrates such a case An experiment randomizes students to either an experi-mental condition which provides them with study materials and encourages them to study fora test or to a control condition which provides neither study materials nor encouragement(Holland 1988 Powers and Swinton 1984) One might hypothesize that encouraging exper-imental group members to study will increase the time they spend studying a focal mediatorin this example which in turn will increase their average test scores One might furtherhypothesize that even without a change in study time providing study materials to experi-mental group members will enable them to study more effectively than they otherwise wouldConsequently the effect of time spent studying might be greater under the experimentalcondition than under the control condition Omitting the treatment-by-mediator interactioneffect will lead to bias in the estimation of the indirect effect and the direct effect

Chapter 11 describes in great detail an evaluation study contrasting a new welfare-to-workprogram emphasizing active participation in the labor force with a traditional program thatguaranteed cash assistance without a mandate to seek employment (Hong Deutsch and Hill2011) Focusing on the psychological well-being of welfare applicants who were singlemothers with preschool-aged children the researchers hypothesized that the treatment wouldincrease employment rate among the welfare recipients and that the treatment-inducedincrease in employment would likely reduce depressive symptoms under the new program

7INTRODUCTION

They also hypothesized that in contrast the same increase in employment was unlikely tohave a psychological impact under the traditional program Hence the treatment-by-mediatorinteraction effect on the outcome was an essential component of the intermediate process

We will show that by defining the indirect effect and the direct effect in terms of potentialoutcomes one can avoid invoking unwarranted assumptions about the unknown structuralrelationships (Holland 1988 Pearl 2001 Robins and Greenland 1992) The potential out-comes framework has further clarified that in a causal mediation theory the treatment and themediator are both conceivably manipulable In the aforementioned example a welfare appli-cant could be assigned at random to either the new program or the traditional one Once havingbeen assigned to one of the programs the individual might or might not be employed due tovarious structural constraints market fluctuations or other random events typically beyondher control

Because the indirect effect and the direct effect are defined in terms of potential outcomesrather than on the basis of any specific regression models it becomes possible to distinguishthe definitions of these causal effects from their identification and estimation The identifica-tion step relates a causal parameter to observable population data For example for individualsassigned at random to an experimental condition their average counterfactual outcome asso-ciated with the control condition is expected to be equal to the average observable outcome ofthose assigned to the control condition Therefore the population average causal effect can beeasily identified in a randomized experiment The estimation step then relates the sample datato the observable population quantities involved in identification while taking into account thedegree of sampling variability

A major challenge in identification is that the mediatorndashoutcome relationship tends tobe confounded by selection even if the treatment is randomized Chapter 9 reviews variousexperimental designs that have been proposed by past researchers for studying causalmediation mechanisms Chapter 10 compares the identification assumptions and the analyticprocedures across a wide range of analytic methods These discussions are followed by anintroduction of the ratio-of-mediator-probability weighting (RMPW) strategy in Chapter 11In comparison with most existing strategies for causal mediation analysis RMPW relies onrelatively fewer identification assumptions and model-based assumptions Chapters 12 and13 will show that this new strategy can be applied broadly with extensions to multilevelexperimental designs and to studies of complex mediation mechanisms involving multiplemediators

113 Spill-over effects of a treatment

It is well known in vaccination research that an unvaccinated person can benefit when mostother people in the local community are vaccinated This is viewed as a spill-over of the treat-ment effect from the vaccinated people to an unvaccinated person Similarly due to socialinteractions among individuals or groups of individuals an intervention received by somemay generate a spill-over impact on others affiliated with the same organization or connectedthrough the same network For example effective policing in one neighborhood may driveoffenders to operate in other neighborhoods In evaluating an innovative community policingprogram researchers found that a neighborhood not assigned to community policing tended tosuffer if the surrounding neighborhoods were assigned to community policing This is aninstance in which a well-intended treatment generates an unintended negative spill-over effectHowever being assigned to community policy was found particularly beneficial when thesurrounding neighborhoods also received the intervention (Verbitsky-Savitz and Raudenbush

8 CAUSALITY IN A SOCIAL WORLD

2012) In another example the impact of a school-based mentoring program targeted atstudents displaying delinquent behaviors may be enhanced for a focal student if his or herat-risk peers are assigned to the mentoring program at the same time

The previous examples challenge the ldquostable unit treatment value assumptionrdquo (SUTVA)that has been invoked in most causal inference studies This assumption states that an indi-vidualrsquos potential response to a treatment depends neither on how the treatment is assignednor on the treatment assignments of other individuals (Rubin 1980 1986) Rubin and othersbelieve that without resorting to SUTVA the causal effect of a treatment becomes hard todefine and that causal inference may become intractable However as Sobel (2006) has illus-trated with the Moving to Opportunity (MTO) experiment that offered housing vouchersenabling low-income families to move to low-poverty neighborhoods there are possible con-sequences of violating SUTVA in estimating the treatment effects on outcomes such as safetyand self-sufficiency Because social interactions among individuals may affect whether onevolunteers to participate in the study whether one moves to a low-poverty neighborhood afterreceiving the housing voucher as well as housing project residentsrsquo subjective perceptions oftheir neighborhoods the program may have a nonzero impact on the potential outcome of theuntreated As a result despite the randomization of treatment assignment the mean differencein the observed outcome between the treated units and the untreated units may be biased forthe average treatment effect Rather the observable quantity is the difference between theeffect of treating some rather than treating none for the treated and the effect of treating somerather than treating none for the untreated the latter being the pure spill-over effect for theuntreated

In making attempts to relax SUTVA researchers have proposed several alternative frame-works that incorporate possible spill-over effects (see Hong and Raudenbush 2013 for areview) Hong (2004) presented a model that involves treatments and treatment settingsA treatment setting for an individual is a local environment constituted by a set of agentsand participants along with their treatment assignments An individualrsquos potential outcomevalue under a given treatment is assumed stable when the treatment setting is fixed the poten-tial outcome may take different values when the treatment setting shifts One may investigatewhether the treatment effect depends on the treatment setting Applying this framework Hongand Raudenbush (2006) examined the effect on a childrsquos academic growth of retaining thechild in kindergarten rather than promoting the child to first grade when a relatively smallproportion of low-achieving peers in the same school are retained as opposed to when a rel-atively large proportion of the peers are retained Hudgens and Halloran (2008) presented arelated framework in which the effect on an individual of the treatment received by this indi-vidual is distinguished from the effect on the individual of the treatment received by others inthe same local community These effects can be identified if communities are assigned atrandom to different treatment assignment strategies (such as retaining a large proportion oflow-achieving students as opposed to retaining a small proportion of such students) and sub-sequently individuals within a community are randomized for treatment Chapter 14 reviewsthese frameworks and discusses identification and estimation strategies

Social contagion may also serve as an important channel through which the effect of anintervention is transmitted For example a school-wide intervention may reduce aggressivebehaviors and thereby improve studentsrsquo psychological well-being by improving the qualityof interpersonal relationships in other classes as well as in onersquos own class This is becausechildren interact not simply with their classmates but also with those from other classes in thehallways or on the playground (VanderWeele et al 2013) In this case the spill-over betweenclasses becomes a part of the mediation mechanism In a study of student mentoring whether

9INTRODUCTION

an individual actually participated in the mentoring program despite the initial treatmentassignment is typically viewed as a primary mediator The proportion of onersquos peers that par-ticipated in the program may act as a second mediator A treatment may also exert its impactpartly through regrouping individuals and thereby changing peer composition (Hong andNomi 2012) Chapter 15 discusses analytic strategies for detecting spill-over as a part ofthe causal mediation mechanism

12 Weighting methods for causal inference

In most behavioral and social science applications major methodological challenges arise dueto the selection of treatments in a quasiexperimental study and the selection of mediator valuesin experimental and quasiexperimental studies Statistical methods most familiar to research-ers in social sciences are often inadequate for causal inferences with regard to multivaluedtreatments moderation mediation and spill-over The book offers a major revision to under-standing of causality in the social world by introducing two complementary weighting stra-tegies both featuring nonparametric approaches to estimating these causal effects The newweighting methods greatly simplify model specifications while enhancing the robustness ofresults by minimizing the reliance on some key assumptions

The propensity score-basedMMWSmethod removes selection bias associated with a largenumber of covariates by equating the pretreatment composition between treatment groups(Hong 2010a 2012 Huang et al 2005) Unlike propensity score matching and stratificationthat are mostly restricted to evaluations of binary treatments the MMWS method is flexiblefor evaluating binary and multivalued treatments by approximating a completely randomizedexperiment In evaluating whether the treatment effects differ across subpopulations definedby individual characteristics or treatment settings researchers may assign weights within eachsubpopulation in order to approximate a randomized block design To investigate whetherone treatment moderates the effect of another concurrent treatment researchers may assignweights to the data to approximate a factorial randomized design The method can also beused to assess whether the effect of an initial treatment is amplified or weakened by a subse-quent treatment or to identify an optimal treatment sequence through approximating a sequen-tial randomized experiment Even though such analyses can similarly be conducted throughinverse-probability-of-treatment weighting (IPTW) that has been increasingly employed inepidemiological research (Hernaacuten Brumback and Robins 2000 Robins Hernaacuten andBrumback 2000) IPTW is known for bias and imprecision in estimation especially whenthe propensity score models are misspecified in their functional forms (Hong 2010a Kangand Schafer 2007 Schafer and Kang 2008 Waernbaum 2012) In contrast the nonparamet-ric MMWS method displays a relatively high level of robustness despite such misspecifica-tions and also gains efficiency as indicated by simulation results (Hong 2010a)

To study causal mediation mechanisms the RMPW method decomposes the total effectof a treatment into an ldquoindirect effectrdquo transmitted through a specific mediator and a ldquodirecteffectrdquo representing unspecified mechanisms In contrast with most existing methods formediation analysis the RMPW-adjusted outcome model is extremely simple and is nonpara-metric in nature It generates estimates of the causal effects along with their sampling errorswhile adjusting for pretreatment covariates that confound the mediatorndashoutcome relationshipsthrough weighting The method applies regardless of the distribution of the outcome the dis-tribution of the mediator or the functional relationship between the outcome and the mediator

10 CAUSALITY IN A SOCIAL WORLD

Page 20: Thumbnail - download.e-bookshelf.de...4.2.4 IPTW for bias removal 84 4.3 MMWS 86 4.3.1 Theoretical rationale 86 ... 8.3.4.2 Growth model in the absence of confounding 200 8.3.4.3 Weighted

education intervention for example teachers are typically the agents who deliver a treatmentto students while the impact of the treatment on student outcomes is of ultimate causalinterest Some educational practices such as ldquoteaching to the testrdquo have been criticized tobe nearly as counterproductive as the attempt of helping seedlings grow by pulling themupward ldquoInterventionsrdquo and ldquotreatmentsrdquo under consideration do not exclude undesiredexperiences such as exposure to poverty abuse crime or bereavement A treatment plannedor unplanned becomes a focus of research if there are theoretical reasons to anticipate itsimpact positive or negative on the well-being of individuals who are embedded in socialsettings including families classrooms schools neighborhoods and workplaces

In social science research in general and in policy and program evaluations in particularquestions concerning whether an intervention works and if so which version of the interven-tion works for whom under what conditions and why are key to the advancement of scien-tific and practical knowledge Although most empirical investigations in the social sciencesconcentrate on the average effect of a treatment for a specific population as opposed to theabsence of such a treatment (ie the control condition) in-depth theoretical reasoning withregard to how the causal effect is generated substantiated by compelling empirical evidenceis crucial for advancing scientific understanding

First when there are multiple versions or different dosages of the treatment or when thereare multiple versions of the control condition a binary divide between ldquothe treatmentrdquo andldquothe controlrdquo may not be as informative as fine-grained comparisons across for exampleldquotreatment version Ardquo ldquotreatment version Brdquo ldquocontrol version Ardquo and ldquocontrol versionBrdquo For example expanding the federally funded Head Start program to poor children isexpected to generate a greater benefit when few early childhood education alternatives areavailable (call it ldquocontrol version Ardquo) than when there is an abundance of alternatives includ-ing state-sponsored preschool programs (call it ldquocontrol version Brdquo)

Second the effect of an intervention will likely vary among individuals or across socialsettings A famous example comes from medical research the well-publicized cardiovascularbenefits of initiating estrogen therapy during menopause were contradicted later by experi-mental findings that the same therapy increased postmenopausal womenrsquos risk for heartattacks The effect of an intervention may also depend on the provision of some other con-current or subsequent interventions Such heterogeneous effects are often characterized asmoderated effects in the literature

Third alternative theories may provide competing explanations for the causal mechan-isms that is the processes through which the intervention produces its effect A theoreticalconstruct characterizing the hypothesized intermediate process is called a mediator of theintervention effect The fictitious farmer never developed an elaborate theory as to whatcaused some seedlings to surpass others in growth Once scientists revealed the causal rela-tionship between access to chemical nutrients in soil and plant growth wide applications ofchemically synthesized fertilizers finally led to a major increase in crop production

Finally it is well known in agricultural experiments that a new type of fertilizer applied toone plot may spill-over to the next plot Because social connections among individuals areprevalent within organizations or through networks an individualrsquos response to the treatmentmay similarly depend on the treatment for other individuals in the same social setting whichmay lead to possible spill-overs of intervention effects among individual human beings

Answering questions with regard to moderation mediation and spill-over poses majorconceptual and analytic challenges To date psychological research often presents well-articulated theories of causal mechanisms relating stimuli to responses Yet researchers often

4 CAUSALITY IN A SOCIAL WORLD

lack rigorous analytic strategies for empirically screening competing theories explaining theobserved effect Sociologists have keen interest in the spill-over of treatment effects transmit-ted through social interactions yet have produced limited evidence quantifying such effectsAs many have pointed out in general terminological ambiguity and conceptual confusionhave been prevalent in the published applied research (Holmbeck 1997 Kraemer et al2002 2008)

A new framework for conceptualizing moderated mediated and spill-over effects hasemerged relatively recently in the statistics and econometrics literature on causal inference(eg Abbring and Heckman 2007 Frangakis and Rubin 2002 Heckman and Vytlacil2007a b Holland 1988 Hong and Raudenbush 2006 Hudgens and Halloran 2008Jo 2008 Pearl 2001 Robins and Greenland 1992 Sobel 2008) The potential for furtherconceptual and methodological development and for broad applications in the field of behav-ioral and social sciences promises to greatly advance the empirical basis of our knowledgeabout causality in the social world

This book clarifies theoretical concepts and introduces innovative statistical strategies forinvestigating the average effects of multivalued treatments moderated treatment effectsmediated treatment effects and spill-over effects in experimental or quasiexperimental dataDefining individual-specific and population average treatment effects in terms of potentialoutcomes the book relates the mathematical forms to the substantive meanings of moderatedmediated and spill-over effects in the context of application examples It also explicates andevaluates identification assumptions and contrasts innovative statistical strategies with con-ventional analytic methods

111 Moderated treatment effects

It is hard to accept the assumption that a treatment would produce the same impact for everyindividual in every possible circumstance Understanding the heterogeneity of treatmenteffects therefore is key to the development of causal theories For example some studiesreported that estrogen therapy improved cardiovascular health among women who initiatedits use during menopause According to a series of other studies however the use of estrogentherapy increased postmenopausal womenrsquos risk for heart attacks (Grodstein et al 19962000 Writing Group for the Womenrsquos Health Initiative Investigators 2002) The sharp con-trast of findings from these studies led to the hypothesis that age of initiation moderates theeffect of estrogen therapy on womenrsquos health (Manson and Bassuk 2007 Rossouw et al2007) Revelations of the moderated causal relationship greatly enrich theoretical understand-ing and in this case directly inform clinical practice

In another example dividing students by ability into small groups for reading instructionhas been controversial for decades Many believed that the practice benefits high-ability stu-dents at the expense of their low-ability peers and exacerbates educational inequality (Grantand Rothenberg 1986 Rowan and Miracle 1983 Trimble and Sinclair 1987) Recentevidence has shown that first of all the effect of ability grouping depends on a number ofconditions including whether enough time is allocated to reading instruction and how wellthe class can be managed Hence instructional time and class management are among theadditional moderators to consider for determining the merits of ability grouping (Hong andHong 2009 Hong et al 2012a b) Moreover further investigations have generated evidencethat contradicts the long-held belief that grouping undermines the interests of low-abilitystudents Quite the contrary researchers have found that grouping is beneficial for students

5INTRODUCTION

with low or medium prior abilitymdashif adequate time is provided for instruction and if the classis well managed On the other hand ability grouping appears to have a minimal impact onhigh-ability studentsrsquo literacy These results were derived from kindergarten data and maynot hold for math instruction and for higher grade levels Replications with different subpo-pulations and in different contexts enable researchers to assess the generalizability of a theory

In a third example one may hypothesize that a student is unlikely to learn algebra well inninth grade without a solid foundation in eighth-grade prealgebra One may also argue that theprogress a student made in learning eighth-grade prealgebra may not be sustained without thefurther enhancement of taking algebra in ninth grade In other words the earlier treatment(prealgebra in eighth grade) and the later treatment (algebra in ninth grade) may reinforceone other each moderates the effect of the other treatment on the final outcome (math achieve-ment at the end of ninth grade) As Hong and Raudenbush (2008) argued the cumulativeeffect of two treatments in a well-aligned sequence may exceed the sum of the benefits oftwo single-year treatments Similarly experiencing two consecutive years of inferior treat-ments such as encountering an incompetent math teacher who dampens a studentrsquos self-efficacy in math learning in eighth grade and then again in ninth grade could do more damagethan the sum of the effect of encountering an incompetent teacher only in eighth grade and theeffect of having such a teacher only in ninth grade

There has been a great amount of conceptual confusion regarding how a moderatorrelates to the treatment and the outcome On one hand many researchers have used the termsldquomoderatorsrdquo and ldquomediatorsrdquo interchangeably without understanding the crucial distinctionbetween the two An overcorrective attempt on the other hand has led to the arbitrary rec-ommendation that a moderator must occur prior to the treatment and be minimally associatedwith the treatment (James and Brett 1984 Kraemer et al 2001 2008) In Chapter 6 weclarify that in essence the causal effect of the treatment on the outcome may depend onthe moderator value A moderator can be a subpopulation identifier a contextual character-istic a concurrent treatment or a preceding or succeeding treatment In the earlier examplesage of estrogen therapy initiation and a studentrsquos prior ability are subpopulation identifiers themanageability of a class which reflects both teacher skills and peer behaviors characterizes thecontext literacy instruction time is a concurrent treatment while prealgebra is a precedingtreatment for algebra A moderator does not have to occur prior to the treatment and doesnot have to be independent of the treatment

Once a moderated causal relationship has been defined in terms of potential outcomes theresearcher then chooses an appropriate experimental design for testing the moderation theoryThe assumptions required for identifying the moderated causal relationships differ across dif-ferent designs and have implications for analyzing experimental and quasiexperimental dataRandomized block designs are suitable for examining individual or contextual characteristicsas potential moderators factorial designs enable one to determine the joint effects of two ormore concurrent treatments and sequential randomized designs are ideal for assessing thecumulative effects of consecutive treatments Multisite randomized trials constitute anadditional type of designs in which the experimental sites are often deliberately sampled torepresent a population of geographical locations or social organizations Site membershipscan be viewed as implicit moderators that summarize a host of features of a local environmentReplications of a treatment over multiple sites allow one to quantify the heterogeneity of treat-ment effects across the sites Chapters 7 and 8 are focused on statistical methods for moder-ation analyses with quasiexperimental data In particular these chapters demonstrate througha number of application examples how a nonparametric marginal mean weighting through

6 CAUSALITY IN A SOCIAL WORLD

stratification (MMWS) method can overcome some important limitations of other existingmethods

Moderation questions are not restricted to treatmentndashoutcome relationships Rather theyare prevalent in investigations of mediation and spill-over This is because heterogeneity oftreatment effects may be explained by the variation in causal mediation mechanisms acrosssubpopulations and contexts It is also because the treatment effect for a focal individualmay depend on whether there is spill-over from other individuals through social interactionsYet similar to the confusion around the concept of moderation among applied researchersthere has been a great amount of misunderstanding with regard to mediation

112 Mediated treatment effects

Questions about mediation are at the core of nearly every scientific theory In its simplestform a theory explains why treatment Z causes outcome Y by hypothesizing an intermediateprocess involving at least one mediatorM that could have been changed by the treatment andcould subsequently have an impact on the outcome A causal mediation analysis then decom-poses the total effect of the treatment on the outcome into two parts the effect transmittedthrough the hypothesized mediator called the indirect effect and the difference betweenthe total effect and the indirect effect called the direct effect The latter represents the treat-ment effect channeled through other unspecified pathways (Alwin and Hauser 1975 Baronand Kenny 1986 Duncan 1966 Shadish and Sweeney 1991) Most researchers in socialsciences have followed the convention of illustrating the direct effect and the indirect effectwith path diagrams and then specifying linear regression models postulated to represent thestructural relationships between the treatment the mediator and the outcome

In Chapter 9 we point out that the approach to defining the indirect effect and the directeffect in terms of path coefficients is often misled by oversimplistic assumptions about thestructural relationships In particular this approach typically overlooks the fact that a treat-ment may generate an impact on the outcome not only by changing the mediator value butalso by changing the mediatorndashoutcome relationship (Judd and Kenny 1981) An examplein Chapter 9 illustrates such a case An experiment randomizes students to either an experi-mental condition which provides them with study materials and encourages them to study fora test or to a control condition which provides neither study materials nor encouragement(Holland 1988 Powers and Swinton 1984) One might hypothesize that encouraging exper-imental group members to study will increase the time they spend studying a focal mediatorin this example which in turn will increase their average test scores One might furtherhypothesize that even without a change in study time providing study materials to experi-mental group members will enable them to study more effectively than they otherwise wouldConsequently the effect of time spent studying might be greater under the experimentalcondition than under the control condition Omitting the treatment-by-mediator interactioneffect will lead to bias in the estimation of the indirect effect and the direct effect

Chapter 11 describes in great detail an evaluation study contrasting a new welfare-to-workprogram emphasizing active participation in the labor force with a traditional program thatguaranteed cash assistance without a mandate to seek employment (Hong Deutsch and Hill2011) Focusing on the psychological well-being of welfare applicants who were singlemothers with preschool-aged children the researchers hypothesized that the treatment wouldincrease employment rate among the welfare recipients and that the treatment-inducedincrease in employment would likely reduce depressive symptoms under the new program

7INTRODUCTION

They also hypothesized that in contrast the same increase in employment was unlikely tohave a psychological impact under the traditional program Hence the treatment-by-mediatorinteraction effect on the outcome was an essential component of the intermediate process

We will show that by defining the indirect effect and the direct effect in terms of potentialoutcomes one can avoid invoking unwarranted assumptions about the unknown structuralrelationships (Holland 1988 Pearl 2001 Robins and Greenland 1992) The potential out-comes framework has further clarified that in a causal mediation theory the treatment and themediator are both conceivably manipulable In the aforementioned example a welfare appli-cant could be assigned at random to either the new program or the traditional one Once havingbeen assigned to one of the programs the individual might or might not be employed due tovarious structural constraints market fluctuations or other random events typically beyondher control

Because the indirect effect and the direct effect are defined in terms of potential outcomesrather than on the basis of any specific regression models it becomes possible to distinguishthe definitions of these causal effects from their identification and estimation The identifica-tion step relates a causal parameter to observable population data For example for individualsassigned at random to an experimental condition their average counterfactual outcome asso-ciated with the control condition is expected to be equal to the average observable outcome ofthose assigned to the control condition Therefore the population average causal effect can beeasily identified in a randomized experiment The estimation step then relates the sample datato the observable population quantities involved in identification while taking into account thedegree of sampling variability

A major challenge in identification is that the mediatorndashoutcome relationship tends tobe confounded by selection even if the treatment is randomized Chapter 9 reviews variousexperimental designs that have been proposed by past researchers for studying causalmediation mechanisms Chapter 10 compares the identification assumptions and the analyticprocedures across a wide range of analytic methods These discussions are followed by anintroduction of the ratio-of-mediator-probability weighting (RMPW) strategy in Chapter 11In comparison with most existing strategies for causal mediation analysis RMPW relies onrelatively fewer identification assumptions and model-based assumptions Chapters 12 and13 will show that this new strategy can be applied broadly with extensions to multilevelexperimental designs and to studies of complex mediation mechanisms involving multiplemediators

113 Spill-over effects of a treatment

It is well known in vaccination research that an unvaccinated person can benefit when mostother people in the local community are vaccinated This is viewed as a spill-over of the treat-ment effect from the vaccinated people to an unvaccinated person Similarly due to socialinteractions among individuals or groups of individuals an intervention received by somemay generate a spill-over impact on others affiliated with the same organization or connectedthrough the same network For example effective policing in one neighborhood may driveoffenders to operate in other neighborhoods In evaluating an innovative community policingprogram researchers found that a neighborhood not assigned to community policing tended tosuffer if the surrounding neighborhoods were assigned to community policing This is aninstance in which a well-intended treatment generates an unintended negative spill-over effectHowever being assigned to community policy was found particularly beneficial when thesurrounding neighborhoods also received the intervention (Verbitsky-Savitz and Raudenbush

8 CAUSALITY IN A SOCIAL WORLD

2012) In another example the impact of a school-based mentoring program targeted atstudents displaying delinquent behaviors may be enhanced for a focal student if his or herat-risk peers are assigned to the mentoring program at the same time

The previous examples challenge the ldquostable unit treatment value assumptionrdquo (SUTVA)that has been invoked in most causal inference studies This assumption states that an indi-vidualrsquos potential response to a treatment depends neither on how the treatment is assignednor on the treatment assignments of other individuals (Rubin 1980 1986) Rubin and othersbelieve that without resorting to SUTVA the causal effect of a treatment becomes hard todefine and that causal inference may become intractable However as Sobel (2006) has illus-trated with the Moving to Opportunity (MTO) experiment that offered housing vouchersenabling low-income families to move to low-poverty neighborhoods there are possible con-sequences of violating SUTVA in estimating the treatment effects on outcomes such as safetyand self-sufficiency Because social interactions among individuals may affect whether onevolunteers to participate in the study whether one moves to a low-poverty neighborhood afterreceiving the housing voucher as well as housing project residentsrsquo subjective perceptions oftheir neighborhoods the program may have a nonzero impact on the potential outcome of theuntreated As a result despite the randomization of treatment assignment the mean differencein the observed outcome between the treated units and the untreated units may be biased forthe average treatment effect Rather the observable quantity is the difference between theeffect of treating some rather than treating none for the treated and the effect of treating somerather than treating none for the untreated the latter being the pure spill-over effect for theuntreated

In making attempts to relax SUTVA researchers have proposed several alternative frame-works that incorporate possible spill-over effects (see Hong and Raudenbush 2013 for areview) Hong (2004) presented a model that involves treatments and treatment settingsA treatment setting for an individual is a local environment constituted by a set of agentsand participants along with their treatment assignments An individualrsquos potential outcomevalue under a given treatment is assumed stable when the treatment setting is fixed the poten-tial outcome may take different values when the treatment setting shifts One may investigatewhether the treatment effect depends on the treatment setting Applying this framework Hongand Raudenbush (2006) examined the effect on a childrsquos academic growth of retaining thechild in kindergarten rather than promoting the child to first grade when a relatively smallproportion of low-achieving peers in the same school are retained as opposed to when a rel-atively large proportion of the peers are retained Hudgens and Halloran (2008) presented arelated framework in which the effect on an individual of the treatment received by this indi-vidual is distinguished from the effect on the individual of the treatment received by others inthe same local community These effects can be identified if communities are assigned atrandom to different treatment assignment strategies (such as retaining a large proportion oflow-achieving students as opposed to retaining a small proportion of such students) and sub-sequently individuals within a community are randomized for treatment Chapter 14 reviewsthese frameworks and discusses identification and estimation strategies

Social contagion may also serve as an important channel through which the effect of anintervention is transmitted For example a school-wide intervention may reduce aggressivebehaviors and thereby improve studentsrsquo psychological well-being by improving the qualityof interpersonal relationships in other classes as well as in onersquos own class This is becausechildren interact not simply with their classmates but also with those from other classes in thehallways or on the playground (VanderWeele et al 2013) In this case the spill-over betweenclasses becomes a part of the mediation mechanism In a study of student mentoring whether

9INTRODUCTION

an individual actually participated in the mentoring program despite the initial treatmentassignment is typically viewed as a primary mediator The proportion of onersquos peers that par-ticipated in the program may act as a second mediator A treatment may also exert its impactpartly through regrouping individuals and thereby changing peer composition (Hong andNomi 2012) Chapter 15 discusses analytic strategies for detecting spill-over as a part ofthe causal mediation mechanism

12 Weighting methods for causal inference

In most behavioral and social science applications major methodological challenges arise dueto the selection of treatments in a quasiexperimental study and the selection of mediator valuesin experimental and quasiexperimental studies Statistical methods most familiar to research-ers in social sciences are often inadequate for causal inferences with regard to multivaluedtreatments moderation mediation and spill-over The book offers a major revision to under-standing of causality in the social world by introducing two complementary weighting stra-tegies both featuring nonparametric approaches to estimating these causal effects The newweighting methods greatly simplify model specifications while enhancing the robustness ofresults by minimizing the reliance on some key assumptions

The propensity score-basedMMWSmethod removes selection bias associated with a largenumber of covariates by equating the pretreatment composition between treatment groups(Hong 2010a 2012 Huang et al 2005) Unlike propensity score matching and stratificationthat are mostly restricted to evaluations of binary treatments the MMWS method is flexiblefor evaluating binary and multivalued treatments by approximating a completely randomizedexperiment In evaluating whether the treatment effects differ across subpopulations definedby individual characteristics or treatment settings researchers may assign weights within eachsubpopulation in order to approximate a randomized block design To investigate whetherone treatment moderates the effect of another concurrent treatment researchers may assignweights to the data to approximate a factorial randomized design The method can also beused to assess whether the effect of an initial treatment is amplified or weakened by a subse-quent treatment or to identify an optimal treatment sequence through approximating a sequen-tial randomized experiment Even though such analyses can similarly be conducted throughinverse-probability-of-treatment weighting (IPTW) that has been increasingly employed inepidemiological research (Hernaacuten Brumback and Robins 2000 Robins Hernaacuten andBrumback 2000) IPTW is known for bias and imprecision in estimation especially whenthe propensity score models are misspecified in their functional forms (Hong 2010a Kangand Schafer 2007 Schafer and Kang 2008 Waernbaum 2012) In contrast the nonparamet-ric MMWS method displays a relatively high level of robustness despite such misspecifica-tions and also gains efficiency as indicated by simulation results (Hong 2010a)

To study causal mediation mechanisms the RMPW method decomposes the total effectof a treatment into an ldquoindirect effectrdquo transmitted through a specific mediator and a ldquodirecteffectrdquo representing unspecified mechanisms In contrast with most existing methods formediation analysis the RMPW-adjusted outcome model is extremely simple and is nonpara-metric in nature It generates estimates of the causal effects along with their sampling errorswhile adjusting for pretreatment covariates that confound the mediatorndashoutcome relationshipsthrough weighting The method applies regardless of the distribution of the outcome the dis-tribution of the mediator or the functional relationship between the outcome and the mediator

10 CAUSALITY IN A SOCIAL WORLD

Page 21: Thumbnail - download.e-bookshelf.de...4.2.4 IPTW for bias removal 84 4.3 MMWS 86 4.3.1 Theoretical rationale 86 ... 8.3.4.2 Growth model in the absence of confounding 200 8.3.4.3 Weighted

lack rigorous analytic strategies for empirically screening competing theories explaining theobserved effect Sociologists have keen interest in the spill-over of treatment effects transmit-ted through social interactions yet have produced limited evidence quantifying such effectsAs many have pointed out in general terminological ambiguity and conceptual confusionhave been prevalent in the published applied research (Holmbeck 1997 Kraemer et al2002 2008)

A new framework for conceptualizing moderated mediated and spill-over effects hasemerged relatively recently in the statistics and econometrics literature on causal inference(eg Abbring and Heckman 2007 Frangakis and Rubin 2002 Heckman and Vytlacil2007a b Holland 1988 Hong and Raudenbush 2006 Hudgens and Halloran 2008Jo 2008 Pearl 2001 Robins and Greenland 1992 Sobel 2008) The potential for furtherconceptual and methodological development and for broad applications in the field of behav-ioral and social sciences promises to greatly advance the empirical basis of our knowledgeabout causality in the social world

This book clarifies theoretical concepts and introduces innovative statistical strategies forinvestigating the average effects of multivalued treatments moderated treatment effectsmediated treatment effects and spill-over effects in experimental or quasiexperimental dataDefining individual-specific and population average treatment effects in terms of potentialoutcomes the book relates the mathematical forms to the substantive meanings of moderatedmediated and spill-over effects in the context of application examples It also explicates andevaluates identification assumptions and contrasts innovative statistical strategies with con-ventional analytic methods

111 Moderated treatment effects

It is hard to accept the assumption that a treatment would produce the same impact for everyindividual in every possible circumstance Understanding the heterogeneity of treatmenteffects therefore is key to the development of causal theories For example some studiesreported that estrogen therapy improved cardiovascular health among women who initiatedits use during menopause According to a series of other studies however the use of estrogentherapy increased postmenopausal womenrsquos risk for heart attacks (Grodstein et al 19962000 Writing Group for the Womenrsquos Health Initiative Investigators 2002) The sharp con-trast of findings from these studies led to the hypothesis that age of initiation moderates theeffect of estrogen therapy on womenrsquos health (Manson and Bassuk 2007 Rossouw et al2007) Revelations of the moderated causal relationship greatly enrich theoretical understand-ing and in this case directly inform clinical practice

In another example dividing students by ability into small groups for reading instructionhas been controversial for decades Many believed that the practice benefits high-ability stu-dents at the expense of their low-ability peers and exacerbates educational inequality (Grantand Rothenberg 1986 Rowan and Miracle 1983 Trimble and Sinclair 1987) Recentevidence has shown that first of all the effect of ability grouping depends on a number ofconditions including whether enough time is allocated to reading instruction and how wellthe class can be managed Hence instructional time and class management are among theadditional moderators to consider for determining the merits of ability grouping (Hong andHong 2009 Hong et al 2012a b) Moreover further investigations have generated evidencethat contradicts the long-held belief that grouping undermines the interests of low-abilitystudents Quite the contrary researchers have found that grouping is beneficial for students

5INTRODUCTION

with low or medium prior abilitymdashif adequate time is provided for instruction and if the classis well managed On the other hand ability grouping appears to have a minimal impact onhigh-ability studentsrsquo literacy These results were derived from kindergarten data and maynot hold for math instruction and for higher grade levels Replications with different subpo-pulations and in different contexts enable researchers to assess the generalizability of a theory

In a third example one may hypothesize that a student is unlikely to learn algebra well inninth grade without a solid foundation in eighth-grade prealgebra One may also argue that theprogress a student made in learning eighth-grade prealgebra may not be sustained without thefurther enhancement of taking algebra in ninth grade In other words the earlier treatment(prealgebra in eighth grade) and the later treatment (algebra in ninth grade) may reinforceone other each moderates the effect of the other treatment on the final outcome (math achieve-ment at the end of ninth grade) As Hong and Raudenbush (2008) argued the cumulativeeffect of two treatments in a well-aligned sequence may exceed the sum of the benefits oftwo single-year treatments Similarly experiencing two consecutive years of inferior treat-ments such as encountering an incompetent math teacher who dampens a studentrsquos self-efficacy in math learning in eighth grade and then again in ninth grade could do more damagethan the sum of the effect of encountering an incompetent teacher only in eighth grade and theeffect of having such a teacher only in ninth grade

There has been a great amount of conceptual confusion regarding how a moderatorrelates to the treatment and the outcome On one hand many researchers have used the termsldquomoderatorsrdquo and ldquomediatorsrdquo interchangeably without understanding the crucial distinctionbetween the two An overcorrective attempt on the other hand has led to the arbitrary rec-ommendation that a moderator must occur prior to the treatment and be minimally associatedwith the treatment (James and Brett 1984 Kraemer et al 2001 2008) In Chapter 6 weclarify that in essence the causal effect of the treatment on the outcome may depend onthe moderator value A moderator can be a subpopulation identifier a contextual character-istic a concurrent treatment or a preceding or succeeding treatment In the earlier examplesage of estrogen therapy initiation and a studentrsquos prior ability are subpopulation identifiers themanageability of a class which reflects both teacher skills and peer behaviors characterizes thecontext literacy instruction time is a concurrent treatment while prealgebra is a precedingtreatment for algebra A moderator does not have to occur prior to the treatment and doesnot have to be independent of the treatment

Once a moderated causal relationship has been defined in terms of potential outcomes theresearcher then chooses an appropriate experimental design for testing the moderation theoryThe assumptions required for identifying the moderated causal relationships differ across dif-ferent designs and have implications for analyzing experimental and quasiexperimental dataRandomized block designs are suitable for examining individual or contextual characteristicsas potential moderators factorial designs enable one to determine the joint effects of two ormore concurrent treatments and sequential randomized designs are ideal for assessing thecumulative effects of consecutive treatments Multisite randomized trials constitute anadditional type of designs in which the experimental sites are often deliberately sampled torepresent a population of geographical locations or social organizations Site membershipscan be viewed as implicit moderators that summarize a host of features of a local environmentReplications of a treatment over multiple sites allow one to quantify the heterogeneity of treat-ment effects across the sites Chapters 7 and 8 are focused on statistical methods for moder-ation analyses with quasiexperimental data In particular these chapters demonstrate througha number of application examples how a nonparametric marginal mean weighting through

6 CAUSALITY IN A SOCIAL WORLD

stratification (MMWS) method can overcome some important limitations of other existingmethods

Moderation questions are not restricted to treatmentndashoutcome relationships Rather theyare prevalent in investigations of mediation and spill-over This is because heterogeneity oftreatment effects may be explained by the variation in causal mediation mechanisms acrosssubpopulations and contexts It is also because the treatment effect for a focal individualmay depend on whether there is spill-over from other individuals through social interactionsYet similar to the confusion around the concept of moderation among applied researchersthere has been a great amount of misunderstanding with regard to mediation

112 Mediated treatment effects

Questions about mediation are at the core of nearly every scientific theory In its simplestform a theory explains why treatment Z causes outcome Y by hypothesizing an intermediateprocess involving at least one mediatorM that could have been changed by the treatment andcould subsequently have an impact on the outcome A causal mediation analysis then decom-poses the total effect of the treatment on the outcome into two parts the effect transmittedthrough the hypothesized mediator called the indirect effect and the difference betweenthe total effect and the indirect effect called the direct effect The latter represents the treat-ment effect channeled through other unspecified pathways (Alwin and Hauser 1975 Baronand Kenny 1986 Duncan 1966 Shadish and Sweeney 1991) Most researchers in socialsciences have followed the convention of illustrating the direct effect and the indirect effectwith path diagrams and then specifying linear regression models postulated to represent thestructural relationships between the treatment the mediator and the outcome

In Chapter 9 we point out that the approach to defining the indirect effect and the directeffect in terms of path coefficients is often misled by oversimplistic assumptions about thestructural relationships In particular this approach typically overlooks the fact that a treat-ment may generate an impact on the outcome not only by changing the mediator value butalso by changing the mediatorndashoutcome relationship (Judd and Kenny 1981) An examplein Chapter 9 illustrates such a case An experiment randomizes students to either an experi-mental condition which provides them with study materials and encourages them to study fora test or to a control condition which provides neither study materials nor encouragement(Holland 1988 Powers and Swinton 1984) One might hypothesize that encouraging exper-imental group members to study will increase the time they spend studying a focal mediatorin this example which in turn will increase their average test scores One might furtherhypothesize that even without a change in study time providing study materials to experi-mental group members will enable them to study more effectively than they otherwise wouldConsequently the effect of time spent studying might be greater under the experimentalcondition than under the control condition Omitting the treatment-by-mediator interactioneffect will lead to bias in the estimation of the indirect effect and the direct effect

Chapter 11 describes in great detail an evaluation study contrasting a new welfare-to-workprogram emphasizing active participation in the labor force with a traditional program thatguaranteed cash assistance without a mandate to seek employment (Hong Deutsch and Hill2011) Focusing on the psychological well-being of welfare applicants who were singlemothers with preschool-aged children the researchers hypothesized that the treatment wouldincrease employment rate among the welfare recipients and that the treatment-inducedincrease in employment would likely reduce depressive symptoms under the new program

7INTRODUCTION

They also hypothesized that in contrast the same increase in employment was unlikely tohave a psychological impact under the traditional program Hence the treatment-by-mediatorinteraction effect on the outcome was an essential component of the intermediate process

We will show that by defining the indirect effect and the direct effect in terms of potentialoutcomes one can avoid invoking unwarranted assumptions about the unknown structuralrelationships (Holland 1988 Pearl 2001 Robins and Greenland 1992) The potential out-comes framework has further clarified that in a causal mediation theory the treatment and themediator are both conceivably manipulable In the aforementioned example a welfare appli-cant could be assigned at random to either the new program or the traditional one Once havingbeen assigned to one of the programs the individual might or might not be employed due tovarious structural constraints market fluctuations or other random events typically beyondher control

Because the indirect effect and the direct effect are defined in terms of potential outcomesrather than on the basis of any specific regression models it becomes possible to distinguishthe definitions of these causal effects from their identification and estimation The identifica-tion step relates a causal parameter to observable population data For example for individualsassigned at random to an experimental condition their average counterfactual outcome asso-ciated with the control condition is expected to be equal to the average observable outcome ofthose assigned to the control condition Therefore the population average causal effect can beeasily identified in a randomized experiment The estimation step then relates the sample datato the observable population quantities involved in identification while taking into account thedegree of sampling variability

A major challenge in identification is that the mediatorndashoutcome relationship tends tobe confounded by selection even if the treatment is randomized Chapter 9 reviews variousexperimental designs that have been proposed by past researchers for studying causalmediation mechanisms Chapter 10 compares the identification assumptions and the analyticprocedures across a wide range of analytic methods These discussions are followed by anintroduction of the ratio-of-mediator-probability weighting (RMPW) strategy in Chapter 11In comparison with most existing strategies for causal mediation analysis RMPW relies onrelatively fewer identification assumptions and model-based assumptions Chapters 12 and13 will show that this new strategy can be applied broadly with extensions to multilevelexperimental designs and to studies of complex mediation mechanisms involving multiplemediators

113 Spill-over effects of a treatment

It is well known in vaccination research that an unvaccinated person can benefit when mostother people in the local community are vaccinated This is viewed as a spill-over of the treat-ment effect from the vaccinated people to an unvaccinated person Similarly due to socialinteractions among individuals or groups of individuals an intervention received by somemay generate a spill-over impact on others affiliated with the same organization or connectedthrough the same network For example effective policing in one neighborhood may driveoffenders to operate in other neighborhoods In evaluating an innovative community policingprogram researchers found that a neighborhood not assigned to community policing tended tosuffer if the surrounding neighborhoods were assigned to community policing This is aninstance in which a well-intended treatment generates an unintended negative spill-over effectHowever being assigned to community policy was found particularly beneficial when thesurrounding neighborhoods also received the intervention (Verbitsky-Savitz and Raudenbush

8 CAUSALITY IN A SOCIAL WORLD

2012) In another example the impact of a school-based mentoring program targeted atstudents displaying delinquent behaviors may be enhanced for a focal student if his or herat-risk peers are assigned to the mentoring program at the same time

The previous examples challenge the ldquostable unit treatment value assumptionrdquo (SUTVA)that has been invoked in most causal inference studies This assumption states that an indi-vidualrsquos potential response to a treatment depends neither on how the treatment is assignednor on the treatment assignments of other individuals (Rubin 1980 1986) Rubin and othersbelieve that without resorting to SUTVA the causal effect of a treatment becomes hard todefine and that causal inference may become intractable However as Sobel (2006) has illus-trated with the Moving to Opportunity (MTO) experiment that offered housing vouchersenabling low-income families to move to low-poverty neighborhoods there are possible con-sequences of violating SUTVA in estimating the treatment effects on outcomes such as safetyand self-sufficiency Because social interactions among individuals may affect whether onevolunteers to participate in the study whether one moves to a low-poverty neighborhood afterreceiving the housing voucher as well as housing project residentsrsquo subjective perceptions oftheir neighborhoods the program may have a nonzero impact on the potential outcome of theuntreated As a result despite the randomization of treatment assignment the mean differencein the observed outcome between the treated units and the untreated units may be biased forthe average treatment effect Rather the observable quantity is the difference between theeffect of treating some rather than treating none for the treated and the effect of treating somerather than treating none for the untreated the latter being the pure spill-over effect for theuntreated

In making attempts to relax SUTVA researchers have proposed several alternative frame-works that incorporate possible spill-over effects (see Hong and Raudenbush 2013 for areview) Hong (2004) presented a model that involves treatments and treatment settingsA treatment setting for an individual is a local environment constituted by a set of agentsand participants along with their treatment assignments An individualrsquos potential outcomevalue under a given treatment is assumed stable when the treatment setting is fixed the poten-tial outcome may take different values when the treatment setting shifts One may investigatewhether the treatment effect depends on the treatment setting Applying this framework Hongand Raudenbush (2006) examined the effect on a childrsquos academic growth of retaining thechild in kindergarten rather than promoting the child to first grade when a relatively smallproportion of low-achieving peers in the same school are retained as opposed to when a rel-atively large proportion of the peers are retained Hudgens and Halloran (2008) presented arelated framework in which the effect on an individual of the treatment received by this indi-vidual is distinguished from the effect on the individual of the treatment received by others inthe same local community These effects can be identified if communities are assigned atrandom to different treatment assignment strategies (such as retaining a large proportion oflow-achieving students as opposed to retaining a small proportion of such students) and sub-sequently individuals within a community are randomized for treatment Chapter 14 reviewsthese frameworks and discusses identification and estimation strategies

Social contagion may also serve as an important channel through which the effect of anintervention is transmitted For example a school-wide intervention may reduce aggressivebehaviors and thereby improve studentsrsquo psychological well-being by improving the qualityof interpersonal relationships in other classes as well as in onersquos own class This is becausechildren interact not simply with their classmates but also with those from other classes in thehallways or on the playground (VanderWeele et al 2013) In this case the spill-over betweenclasses becomes a part of the mediation mechanism In a study of student mentoring whether

9INTRODUCTION

an individual actually participated in the mentoring program despite the initial treatmentassignment is typically viewed as a primary mediator The proportion of onersquos peers that par-ticipated in the program may act as a second mediator A treatment may also exert its impactpartly through regrouping individuals and thereby changing peer composition (Hong andNomi 2012) Chapter 15 discusses analytic strategies for detecting spill-over as a part ofthe causal mediation mechanism

12 Weighting methods for causal inference

In most behavioral and social science applications major methodological challenges arise dueto the selection of treatments in a quasiexperimental study and the selection of mediator valuesin experimental and quasiexperimental studies Statistical methods most familiar to research-ers in social sciences are often inadequate for causal inferences with regard to multivaluedtreatments moderation mediation and spill-over The book offers a major revision to under-standing of causality in the social world by introducing two complementary weighting stra-tegies both featuring nonparametric approaches to estimating these causal effects The newweighting methods greatly simplify model specifications while enhancing the robustness ofresults by minimizing the reliance on some key assumptions

The propensity score-basedMMWSmethod removes selection bias associated with a largenumber of covariates by equating the pretreatment composition between treatment groups(Hong 2010a 2012 Huang et al 2005) Unlike propensity score matching and stratificationthat are mostly restricted to evaluations of binary treatments the MMWS method is flexiblefor evaluating binary and multivalued treatments by approximating a completely randomizedexperiment In evaluating whether the treatment effects differ across subpopulations definedby individual characteristics or treatment settings researchers may assign weights within eachsubpopulation in order to approximate a randomized block design To investigate whetherone treatment moderates the effect of another concurrent treatment researchers may assignweights to the data to approximate a factorial randomized design The method can also beused to assess whether the effect of an initial treatment is amplified or weakened by a subse-quent treatment or to identify an optimal treatment sequence through approximating a sequen-tial randomized experiment Even though such analyses can similarly be conducted throughinverse-probability-of-treatment weighting (IPTW) that has been increasingly employed inepidemiological research (Hernaacuten Brumback and Robins 2000 Robins Hernaacuten andBrumback 2000) IPTW is known for bias and imprecision in estimation especially whenthe propensity score models are misspecified in their functional forms (Hong 2010a Kangand Schafer 2007 Schafer and Kang 2008 Waernbaum 2012) In contrast the nonparamet-ric MMWS method displays a relatively high level of robustness despite such misspecifica-tions and also gains efficiency as indicated by simulation results (Hong 2010a)

To study causal mediation mechanisms the RMPW method decomposes the total effectof a treatment into an ldquoindirect effectrdquo transmitted through a specific mediator and a ldquodirecteffectrdquo representing unspecified mechanisms In contrast with most existing methods formediation analysis the RMPW-adjusted outcome model is extremely simple and is nonpara-metric in nature It generates estimates of the causal effects along with their sampling errorswhile adjusting for pretreatment covariates that confound the mediatorndashoutcome relationshipsthrough weighting The method applies regardless of the distribution of the outcome the dis-tribution of the mediator or the functional relationship between the outcome and the mediator

10 CAUSALITY IN A SOCIAL WORLD

Page 22: Thumbnail - download.e-bookshelf.de...4.2.4 IPTW for bias removal 84 4.3 MMWS 86 4.3.1 Theoretical rationale 86 ... 8.3.4.2 Growth model in the absence of confounding 200 8.3.4.3 Weighted

with low or medium prior abilitymdashif adequate time is provided for instruction and if the classis well managed On the other hand ability grouping appears to have a minimal impact onhigh-ability studentsrsquo literacy These results were derived from kindergarten data and maynot hold for math instruction and for higher grade levels Replications with different subpo-pulations and in different contexts enable researchers to assess the generalizability of a theory

In a third example one may hypothesize that a student is unlikely to learn algebra well inninth grade without a solid foundation in eighth-grade prealgebra One may also argue that theprogress a student made in learning eighth-grade prealgebra may not be sustained without thefurther enhancement of taking algebra in ninth grade In other words the earlier treatment(prealgebra in eighth grade) and the later treatment (algebra in ninth grade) may reinforceone other each moderates the effect of the other treatment on the final outcome (math achieve-ment at the end of ninth grade) As Hong and Raudenbush (2008) argued the cumulativeeffect of two treatments in a well-aligned sequence may exceed the sum of the benefits oftwo single-year treatments Similarly experiencing two consecutive years of inferior treat-ments such as encountering an incompetent math teacher who dampens a studentrsquos self-efficacy in math learning in eighth grade and then again in ninth grade could do more damagethan the sum of the effect of encountering an incompetent teacher only in eighth grade and theeffect of having such a teacher only in ninth grade

There has been a great amount of conceptual confusion regarding how a moderatorrelates to the treatment and the outcome On one hand many researchers have used the termsldquomoderatorsrdquo and ldquomediatorsrdquo interchangeably without understanding the crucial distinctionbetween the two An overcorrective attempt on the other hand has led to the arbitrary rec-ommendation that a moderator must occur prior to the treatment and be minimally associatedwith the treatment (James and Brett 1984 Kraemer et al 2001 2008) In Chapter 6 weclarify that in essence the causal effect of the treatment on the outcome may depend onthe moderator value A moderator can be a subpopulation identifier a contextual character-istic a concurrent treatment or a preceding or succeeding treatment In the earlier examplesage of estrogen therapy initiation and a studentrsquos prior ability are subpopulation identifiers themanageability of a class which reflects both teacher skills and peer behaviors characterizes thecontext literacy instruction time is a concurrent treatment while prealgebra is a precedingtreatment for algebra A moderator does not have to occur prior to the treatment and doesnot have to be independent of the treatment

Once a moderated causal relationship has been defined in terms of potential outcomes theresearcher then chooses an appropriate experimental design for testing the moderation theoryThe assumptions required for identifying the moderated causal relationships differ across dif-ferent designs and have implications for analyzing experimental and quasiexperimental dataRandomized block designs are suitable for examining individual or contextual characteristicsas potential moderators factorial designs enable one to determine the joint effects of two ormore concurrent treatments and sequential randomized designs are ideal for assessing thecumulative effects of consecutive treatments Multisite randomized trials constitute anadditional type of designs in which the experimental sites are often deliberately sampled torepresent a population of geographical locations or social organizations Site membershipscan be viewed as implicit moderators that summarize a host of features of a local environmentReplications of a treatment over multiple sites allow one to quantify the heterogeneity of treat-ment effects across the sites Chapters 7 and 8 are focused on statistical methods for moder-ation analyses with quasiexperimental data In particular these chapters demonstrate througha number of application examples how a nonparametric marginal mean weighting through

6 CAUSALITY IN A SOCIAL WORLD

stratification (MMWS) method can overcome some important limitations of other existingmethods

Moderation questions are not restricted to treatmentndashoutcome relationships Rather theyare prevalent in investigations of mediation and spill-over This is because heterogeneity oftreatment effects may be explained by the variation in causal mediation mechanisms acrosssubpopulations and contexts It is also because the treatment effect for a focal individualmay depend on whether there is spill-over from other individuals through social interactionsYet similar to the confusion around the concept of moderation among applied researchersthere has been a great amount of misunderstanding with regard to mediation

112 Mediated treatment effects

Questions about mediation are at the core of nearly every scientific theory In its simplestform a theory explains why treatment Z causes outcome Y by hypothesizing an intermediateprocess involving at least one mediatorM that could have been changed by the treatment andcould subsequently have an impact on the outcome A causal mediation analysis then decom-poses the total effect of the treatment on the outcome into two parts the effect transmittedthrough the hypothesized mediator called the indirect effect and the difference betweenthe total effect and the indirect effect called the direct effect The latter represents the treat-ment effect channeled through other unspecified pathways (Alwin and Hauser 1975 Baronand Kenny 1986 Duncan 1966 Shadish and Sweeney 1991) Most researchers in socialsciences have followed the convention of illustrating the direct effect and the indirect effectwith path diagrams and then specifying linear regression models postulated to represent thestructural relationships between the treatment the mediator and the outcome

In Chapter 9 we point out that the approach to defining the indirect effect and the directeffect in terms of path coefficients is often misled by oversimplistic assumptions about thestructural relationships In particular this approach typically overlooks the fact that a treat-ment may generate an impact on the outcome not only by changing the mediator value butalso by changing the mediatorndashoutcome relationship (Judd and Kenny 1981) An examplein Chapter 9 illustrates such a case An experiment randomizes students to either an experi-mental condition which provides them with study materials and encourages them to study fora test or to a control condition which provides neither study materials nor encouragement(Holland 1988 Powers and Swinton 1984) One might hypothesize that encouraging exper-imental group members to study will increase the time they spend studying a focal mediatorin this example which in turn will increase their average test scores One might furtherhypothesize that even without a change in study time providing study materials to experi-mental group members will enable them to study more effectively than they otherwise wouldConsequently the effect of time spent studying might be greater under the experimentalcondition than under the control condition Omitting the treatment-by-mediator interactioneffect will lead to bias in the estimation of the indirect effect and the direct effect

Chapter 11 describes in great detail an evaluation study contrasting a new welfare-to-workprogram emphasizing active participation in the labor force with a traditional program thatguaranteed cash assistance without a mandate to seek employment (Hong Deutsch and Hill2011) Focusing on the psychological well-being of welfare applicants who were singlemothers with preschool-aged children the researchers hypothesized that the treatment wouldincrease employment rate among the welfare recipients and that the treatment-inducedincrease in employment would likely reduce depressive symptoms under the new program

7INTRODUCTION

They also hypothesized that in contrast the same increase in employment was unlikely tohave a psychological impact under the traditional program Hence the treatment-by-mediatorinteraction effect on the outcome was an essential component of the intermediate process

We will show that by defining the indirect effect and the direct effect in terms of potentialoutcomes one can avoid invoking unwarranted assumptions about the unknown structuralrelationships (Holland 1988 Pearl 2001 Robins and Greenland 1992) The potential out-comes framework has further clarified that in a causal mediation theory the treatment and themediator are both conceivably manipulable In the aforementioned example a welfare appli-cant could be assigned at random to either the new program or the traditional one Once havingbeen assigned to one of the programs the individual might or might not be employed due tovarious structural constraints market fluctuations or other random events typically beyondher control

Because the indirect effect and the direct effect are defined in terms of potential outcomesrather than on the basis of any specific regression models it becomes possible to distinguishthe definitions of these causal effects from their identification and estimation The identifica-tion step relates a causal parameter to observable population data For example for individualsassigned at random to an experimental condition their average counterfactual outcome asso-ciated with the control condition is expected to be equal to the average observable outcome ofthose assigned to the control condition Therefore the population average causal effect can beeasily identified in a randomized experiment The estimation step then relates the sample datato the observable population quantities involved in identification while taking into account thedegree of sampling variability

A major challenge in identification is that the mediatorndashoutcome relationship tends tobe confounded by selection even if the treatment is randomized Chapter 9 reviews variousexperimental designs that have been proposed by past researchers for studying causalmediation mechanisms Chapter 10 compares the identification assumptions and the analyticprocedures across a wide range of analytic methods These discussions are followed by anintroduction of the ratio-of-mediator-probability weighting (RMPW) strategy in Chapter 11In comparison with most existing strategies for causal mediation analysis RMPW relies onrelatively fewer identification assumptions and model-based assumptions Chapters 12 and13 will show that this new strategy can be applied broadly with extensions to multilevelexperimental designs and to studies of complex mediation mechanisms involving multiplemediators

113 Spill-over effects of a treatment

It is well known in vaccination research that an unvaccinated person can benefit when mostother people in the local community are vaccinated This is viewed as a spill-over of the treat-ment effect from the vaccinated people to an unvaccinated person Similarly due to socialinteractions among individuals or groups of individuals an intervention received by somemay generate a spill-over impact on others affiliated with the same organization or connectedthrough the same network For example effective policing in one neighborhood may driveoffenders to operate in other neighborhoods In evaluating an innovative community policingprogram researchers found that a neighborhood not assigned to community policing tended tosuffer if the surrounding neighborhoods were assigned to community policing This is aninstance in which a well-intended treatment generates an unintended negative spill-over effectHowever being assigned to community policy was found particularly beneficial when thesurrounding neighborhoods also received the intervention (Verbitsky-Savitz and Raudenbush

8 CAUSALITY IN A SOCIAL WORLD

2012) In another example the impact of a school-based mentoring program targeted atstudents displaying delinquent behaviors may be enhanced for a focal student if his or herat-risk peers are assigned to the mentoring program at the same time

The previous examples challenge the ldquostable unit treatment value assumptionrdquo (SUTVA)that has been invoked in most causal inference studies This assumption states that an indi-vidualrsquos potential response to a treatment depends neither on how the treatment is assignednor on the treatment assignments of other individuals (Rubin 1980 1986) Rubin and othersbelieve that without resorting to SUTVA the causal effect of a treatment becomes hard todefine and that causal inference may become intractable However as Sobel (2006) has illus-trated with the Moving to Opportunity (MTO) experiment that offered housing vouchersenabling low-income families to move to low-poverty neighborhoods there are possible con-sequences of violating SUTVA in estimating the treatment effects on outcomes such as safetyand self-sufficiency Because social interactions among individuals may affect whether onevolunteers to participate in the study whether one moves to a low-poverty neighborhood afterreceiving the housing voucher as well as housing project residentsrsquo subjective perceptions oftheir neighborhoods the program may have a nonzero impact on the potential outcome of theuntreated As a result despite the randomization of treatment assignment the mean differencein the observed outcome between the treated units and the untreated units may be biased forthe average treatment effect Rather the observable quantity is the difference between theeffect of treating some rather than treating none for the treated and the effect of treating somerather than treating none for the untreated the latter being the pure spill-over effect for theuntreated

In making attempts to relax SUTVA researchers have proposed several alternative frame-works that incorporate possible spill-over effects (see Hong and Raudenbush 2013 for areview) Hong (2004) presented a model that involves treatments and treatment settingsA treatment setting for an individual is a local environment constituted by a set of agentsand participants along with their treatment assignments An individualrsquos potential outcomevalue under a given treatment is assumed stable when the treatment setting is fixed the poten-tial outcome may take different values when the treatment setting shifts One may investigatewhether the treatment effect depends on the treatment setting Applying this framework Hongand Raudenbush (2006) examined the effect on a childrsquos academic growth of retaining thechild in kindergarten rather than promoting the child to first grade when a relatively smallproportion of low-achieving peers in the same school are retained as opposed to when a rel-atively large proportion of the peers are retained Hudgens and Halloran (2008) presented arelated framework in which the effect on an individual of the treatment received by this indi-vidual is distinguished from the effect on the individual of the treatment received by others inthe same local community These effects can be identified if communities are assigned atrandom to different treatment assignment strategies (such as retaining a large proportion oflow-achieving students as opposed to retaining a small proportion of such students) and sub-sequently individuals within a community are randomized for treatment Chapter 14 reviewsthese frameworks and discusses identification and estimation strategies

Social contagion may also serve as an important channel through which the effect of anintervention is transmitted For example a school-wide intervention may reduce aggressivebehaviors and thereby improve studentsrsquo psychological well-being by improving the qualityof interpersonal relationships in other classes as well as in onersquos own class This is becausechildren interact not simply with their classmates but also with those from other classes in thehallways or on the playground (VanderWeele et al 2013) In this case the spill-over betweenclasses becomes a part of the mediation mechanism In a study of student mentoring whether

9INTRODUCTION

an individual actually participated in the mentoring program despite the initial treatmentassignment is typically viewed as a primary mediator The proportion of onersquos peers that par-ticipated in the program may act as a second mediator A treatment may also exert its impactpartly through regrouping individuals and thereby changing peer composition (Hong andNomi 2012) Chapter 15 discusses analytic strategies for detecting spill-over as a part ofthe causal mediation mechanism

12 Weighting methods for causal inference

In most behavioral and social science applications major methodological challenges arise dueto the selection of treatments in a quasiexperimental study and the selection of mediator valuesin experimental and quasiexperimental studies Statistical methods most familiar to research-ers in social sciences are often inadequate for causal inferences with regard to multivaluedtreatments moderation mediation and spill-over The book offers a major revision to under-standing of causality in the social world by introducing two complementary weighting stra-tegies both featuring nonparametric approaches to estimating these causal effects The newweighting methods greatly simplify model specifications while enhancing the robustness ofresults by minimizing the reliance on some key assumptions

The propensity score-basedMMWSmethod removes selection bias associated with a largenumber of covariates by equating the pretreatment composition between treatment groups(Hong 2010a 2012 Huang et al 2005) Unlike propensity score matching and stratificationthat are mostly restricted to evaluations of binary treatments the MMWS method is flexiblefor evaluating binary and multivalued treatments by approximating a completely randomizedexperiment In evaluating whether the treatment effects differ across subpopulations definedby individual characteristics or treatment settings researchers may assign weights within eachsubpopulation in order to approximate a randomized block design To investigate whetherone treatment moderates the effect of another concurrent treatment researchers may assignweights to the data to approximate a factorial randomized design The method can also beused to assess whether the effect of an initial treatment is amplified or weakened by a subse-quent treatment or to identify an optimal treatment sequence through approximating a sequen-tial randomized experiment Even though such analyses can similarly be conducted throughinverse-probability-of-treatment weighting (IPTW) that has been increasingly employed inepidemiological research (Hernaacuten Brumback and Robins 2000 Robins Hernaacuten andBrumback 2000) IPTW is known for bias and imprecision in estimation especially whenthe propensity score models are misspecified in their functional forms (Hong 2010a Kangand Schafer 2007 Schafer and Kang 2008 Waernbaum 2012) In contrast the nonparamet-ric MMWS method displays a relatively high level of robustness despite such misspecifica-tions and also gains efficiency as indicated by simulation results (Hong 2010a)

To study causal mediation mechanisms the RMPW method decomposes the total effectof a treatment into an ldquoindirect effectrdquo transmitted through a specific mediator and a ldquodirecteffectrdquo representing unspecified mechanisms In contrast with most existing methods formediation analysis the RMPW-adjusted outcome model is extremely simple and is nonpara-metric in nature It generates estimates of the causal effects along with their sampling errorswhile adjusting for pretreatment covariates that confound the mediatorndashoutcome relationshipsthrough weighting The method applies regardless of the distribution of the outcome the dis-tribution of the mediator or the functional relationship between the outcome and the mediator

10 CAUSALITY IN A SOCIAL WORLD

Page 23: Thumbnail - download.e-bookshelf.de...4.2.4 IPTW for bias removal 84 4.3 MMWS 86 4.3.1 Theoretical rationale 86 ... 8.3.4.2 Growth model in the absence of confounding 200 8.3.4.3 Weighted

stratification (MMWS) method can overcome some important limitations of other existingmethods

Moderation questions are not restricted to treatmentndashoutcome relationships Rather theyare prevalent in investigations of mediation and spill-over This is because heterogeneity oftreatment effects may be explained by the variation in causal mediation mechanisms acrosssubpopulations and contexts It is also because the treatment effect for a focal individualmay depend on whether there is spill-over from other individuals through social interactionsYet similar to the confusion around the concept of moderation among applied researchersthere has been a great amount of misunderstanding with regard to mediation

112 Mediated treatment effects

Questions about mediation are at the core of nearly every scientific theory In its simplestform a theory explains why treatment Z causes outcome Y by hypothesizing an intermediateprocess involving at least one mediatorM that could have been changed by the treatment andcould subsequently have an impact on the outcome A causal mediation analysis then decom-poses the total effect of the treatment on the outcome into two parts the effect transmittedthrough the hypothesized mediator called the indirect effect and the difference betweenthe total effect and the indirect effect called the direct effect The latter represents the treat-ment effect channeled through other unspecified pathways (Alwin and Hauser 1975 Baronand Kenny 1986 Duncan 1966 Shadish and Sweeney 1991) Most researchers in socialsciences have followed the convention of illustrating the direct effect and the indirect effectwith path diagrams and then specifying linear regression models postulated to represent thestructural relationships between the treatment the mediator and the outcome

In Chapter 9 we point out that the approach to defining the indirect effect and the directeffect in terms of path coefficients is often misled by oversimplistic assumptions about thestructural relationships In particular this approach typically overlooks the fact that a treat-ment may generate an impact on the outcome not only by changing the mediator value butalso by changing the mediatorndashoutcome relationship (Judd and Kenny 1981) An examplein Chapter 9 illustrates such a case An experiment randomizes students to either an experi-mental condition which provides them with study materials and encourages them to study fora test or to a control condition which provides neither study materials nor encouragement(Holland 1988 Powers and Swinton 1984) One might hypothesize that encouraging exper-imental group members to study will increase the time they spend studying a focal mediatorin this example which in turn will increase their average test scores One might furtherhypothesize that even without a change in study time providing study materials to experi-mental group members will enable them to study more effectively than they otherwise wouldConsequently the effect of time spent studying might be greater under the experimentalcondition than under the control condition Omitting the treatment-by-mediator interactioneffect will lead to bias in the estimation of the indirect effect and the direct effect

Chapter 11 describes in great detail an evaluation study contrasting a new welfare-to-workprogram emphasizing active participation in the labor force with a traditional program thatguaranteed cash assistance without a mandate to seek employment (Hong Deutsch and Hill2011) Focusing on the psychological well-being of welfare applicants who were singlemothers with preschool-aged children the researchers hypothesized that the treatment wouldincrease employment rate among the welfare recipients and that the treatment-inducedincrease in employment would likely reduce depressive symptoms under the new program

7INTRODUCTION

They also hypothesized that in contrast the same increase in employment was unlikely tohave a psychological impact under the traditional program Hence the treatment-by-mediatorinteraction effect on the outcome was an essential component of the intermediate process

We will show that by defining the indirect effect and the direct effect in terms of potentialoutcomes one can avoid invoking unwarranted assumptions about the unknown structuralrelationships (Holland 1988 Pearl 2001 Robins and Greenland 1992) The potential out-comes framework has further clarified that in a causal mediation theory the treatment and themediator are both conceivably manipulable In the aforementioned example a welfare appli-cant could be assigned at random to either the new program or the traditional one Once havingbeen assigned to one of the programs the individual might or might not be employed due tovarious structural constraints market fluctuations or other random events typically beyondher control

Because the indirect effect and the direct effect are defined in terms of potential outcomesrather than on the basis of any specific regression models it becomes possible to distinguishthe definitions of these causal effects from their identification and estimation The identifica-tion step relates a causal parameter to observable population data For example for individualsassigned at random to an experimental condition their average counterfactual outcome asso-ciated with the control condition is expected to be equal to the average observable outcome ofthose assigned to the control condition Therefore the population average causal effect can beeasily identified in a randomized experiment The estimation step then relates the sample datato the observable population quantities involved in identification while taking into account thedegree of sampling variability

A major challenge in identification is that the mediatorndashoutcome relationship tends tobe confounded by selection even if the treatment is randomized Chapter 9 reviews variousexperimental designs that have been proposed by past researchers for studying causalmediation mechanisms Chapter 10 compares the identification assumptions and the analyticprocedures across a wide range of analytic methods These discussions are followed by anintroduction of the ratio-of-mediator-probability weighting (RMPW) strategy in Chapter 11In comparison with most existing strategies for causal mediation analysis RMPW relies onrelatively fewer identification assumptions and model-based assumptions Chapters 12 and13 will show that this new strategy can be applied broadly with extensions to multilevelexperimental designs and to studies of complex mediation mechanisms involving multiplemediators

113 Spill-over effects of a treatment

It is well known in vaccination research that an unvaccinated person can benefit when mostother people in the local community are vaccinated This is viewed as a spill-over of the treat-ment effect from the vaccinated people to an unvaccinated person Similarly due to socialinteractions among individuals or groups of individuals an intervention received by somemay generate a spill-over impact on others affiliated with the same organization or connectedthrough the same network For example effective policing in one neighborhood may driveoffenders to operate in other neighborhoods In evaluating an innovative community policingprogram researchers found that a neighborhood not assigned to community policing tended tosuffer if the surrounding neighborhoods were assigned to community policing This is aninstance in which a well-intended treatment generates an unintended negative spill-over effectHowever being assigned to community policy was found particularly beneficial when thesurrounding neighborhoods also received the intervention (Verbitsky-Savitz and Raudenbush

8 CAUSALITY IN A SOCIAL WORLD

2012) In another example the impact of a school-based mentoring program targeted atstudents displaying delinquent behaviors may be enhanced for a focal student if his or herat-risk peers are assigned to the mentoring program at the same time

The previous examples challenge the ldquostable unit treatment value assumptionrdquo (SUTVA)that has been invoked in most causal inference studies This assumption states that an indi-vidualrsquos potential response to a treatment depends neither on how the treatment is assignednor on the treatment assignments of other individuals (Rubin 1980 1986) Rubin and othersbelieve that without resorting to SUTVA the causal effect of a treatment becomes hard todefine and that causal inference may become intractable However as Sobel (2006) has illus-trated with the Moving to Opportunity (MTO) experiment that offered housing vouchersenabling low-income families to move to low-poverty neighborhoods there are possible con-sequences of violating SUTVA in estimating the treatment effects on outcomes such as safetyand self-sufficiency Because social interactions among individuals may affect whether onevolunteers to participate in the study whether one moves to a low-poverty neighborhood afterreceiving the housing voucher as well as housing project residentsrsquo subjective perceptions oftheir neighborhoods the program may have a nonzero impact on the potential outcome of theuntreated As a result despite the randomization of treatment assignment the mean differencein the observed outcome between the treated units and the untreated units may be biased forthe average treatment effect Rather the observable quantity is the difference between theeffect of treating some rather than treating none for the treated and the effect of treating somerather than treating none for the untreated the latter being the pure spill-over effect for theuntreated

In making attempts to relax SUTVA researchers have proposed several alternative frame-works that incorporate possible spill-over effects (see Hong and Raudenbush 2013 for areview) Hong (2004) presented a model that involves treatments and treatment settingsA treatment setting for an individual is a local environment constituted by a set of agentsand participants along with their treatment assignments An individualrsquos potential outcomevalue under a given treatment is assumed stable when the treatment setting is fixed the poten-tial outcome may take different values when the treatment setting shifts One may investigatewhether the treatment effect depends on the treatment setting Applying this framework Hongand Raudenbush (2006) examined the effect on a childrsquos academic growth of retaining thechild in kindergarten rather than promoting the child to first grade when a relatively smallproportion of low-achieving peers in the same school are retained as opposed to when a rel-atively large proportion of the peers are retained Hudgens and Halloran (2008) presented arelated framework in which the effect on an individual of the treatment received by this indi-vidual is distinguished from the effect on the individual of the treatment received by others inthe same local community These effects can be identified if communities are assigned atrandom to different treatment assignment strategies (such as retaining a large proportion oflow-achieving students as opposed to retaining a small proportion of such students) and sub-sequently individuals within a community are randomized for treatment Chapter 14 reviewsthese frameworks and discusses identification and estimation strategies

Social contagion may also serve as an important channel through which the effect of anintervention is transmitted For example a school-wide intervention may reduce aggressivebehaviors and thereby improve studentsrsquo psychological well-being by improving the qualityof interpersonal relationships in other classes as well as in onersquos own class This is becausechildren interact not simply with their classmates but also with those from other classes in thehallways or on the playground (VanderWeele et al 2013) In this case the spill-over betweenclasses becomes a part of the mediation mechanism In a study of student mentoring whether

9INTRODUCTION

an individual actually participated in the mentoring program despite the initial treatmentassignment is typically viewed as a primary mediator The proportion of onersquos peers that par-ticipated in the program may act as a second mediator A treatment may also exert its impactpartly through regrouping individuals and thereby changing peer composition (Hong andNomi 2012) Chapter 15 discusses analytic strategies for detecting spill-over as a part ofthe causal mediation mechanism

12 Weighting methods for causal inference

In most behavioral and social science applications major methodological challenges arise dueto the selection of treatments in a quasiexperimental study and the selection of mediator valuesin experimental and quasiexperimental studies Statistical methods most familiar to research-ers in social sciences are often inadequate for causal inferences with regard to multivaluedtreatments moderation mediation and spill-over The book offers a major revision to under-standing of causality in the social world by introducing two complementary weighting stra-tegies both featuring nonparametric approaches to estimating these causal effects The newweighting methods greatly simplify model specifications while enhancing the robustness ofresults by minimizing the reliance on some key assumptions

The propensity score-basedMMWSmethod removes selection bias associated with a largenumber of covariates by equating the pretreatment composition between treatment groups(Hong 2010a 2012 Huang et al 2005) Unlike propensity score matching and stratificationthat are mostly restricted to evaluations of binary treatments the MMWS method is flexiblefor evaluating binary and multivalued treatments by approximating a completely randomizedexperiment In evaluating whether the treatment effects differ across subpopulations definedby individual characteristics or treatment settings researchers may assign weights within eachsubpopulation in order to approximate a randomized block design To investigate whetherone treatment moderates the effect of another concurrent treatment researchers may assignweights to the data to approximate a factorial randomized design The method can also beused to assess whether the effect of an initial treatment is amplified or weakened by a subse-quent treatment or to identify an optimal treatment sequence through approximating a sequen-tial randomized experiment Even though such analyses can similarly be conducted throughinverse-probability-of-treatment weighting (IPTW) that has been increasingly employed inepidemiological research (Hernaacuten Brumback and Robins 2000 Robins Hernaacuten andBrumback 2000) IPTW is known for bias and imprecision in estimation especially whenthe propensity score models are misspecified in their functional forms (Hong 2010a Kangand Schafer 2007 Schafer and Kang 2008 Waernbaum 2012) In contrast the nonparamet-ric MMWS method displays a relatively high level of robustness despite such misspecifica-tions and also gains efficiency as indicated by simulation results (Hong 2010a)

To study causal mediation mechanisms the RMPW method decomposes the total effectof a treatment into an ldquoindirect effectrdquo transmitted through a specific mediator and a ldquodirecteffectrdquo representing unspecified mechanisms In contrast with most existing methods formediation analysis the RMPW-adjusted outcome model is extremely simple and is nonpara-metric in nature It generates estimates of the causal effects along with their sampling errorswhile adjusting for pretreatment covariates that confound the mediatorndashoutcome relationshipsthrough weighting The method applies regardless of the distribution of the outcome the dis-tribution of the mediator or the functional relationship between the outcome and the mediator

10 CAUSALITY IN A SOCIAL WORLD

Page 24: Thumbnail - download.e-bookshelf.de...4.2.4 IPTW for bias removal 84 4.3 MMWS 86 4.3.1 Theoretical rationale 86 ... 8.3.4.2 Growth model in the absence of confounding 200 8.3.4.3 Weighted

They also hypothesized that in contrast the same increase in employment was unlikely tohave a psychological impact under the traditional program Hence the treatment-by-mediatorinteraction effect on the outcome was an essential component of the intermediate process

We will show that by defining the indirect effect and the direct effect in terms of potentialoutcomes one can avoid invoking unwarranted assumptions about the unknown structuralrelationships (Holland 1988 Pearl 2001 Robins and Greenland 1992) The potential out-comes framework has further clarified that in a causal mediation theory the treatment and themediator are both conceivably manipulable In the aforementioned example a welfare appli-cant could be assigned at random to either the new program or the traditional one Once havingbeen assigned to one of the programs the individual might or might not be employed due tovarious structural constraints market fluctuations or other random events typically beyondher control

Because the indirect effect and the direct effect are defined in terms of potential outcomesrather than on the basis of any specific regression models it becomes possible to distinguishthe definitions of these causal effects from their identification and estimation The identifica-tion step relates a causal parameter to observable population data For example for individualsassigned at random to an experimental condition their average counterfactual outcome asso-ciated with the control condition is expected to be equal to the average observable outcome ofthose assigned to the control condition Therefore the population average causal effect can beeasily identified in a randomized experiment The estimation step then relates the sample datato the observable population quantities involved in identification while taking into account thedegree of sampling variability

A major challenge in identification is that the mediatorndashoutcome relationship tends tobe confounded by selection even if the treatment is randomized Chapter 9 reviews variousexperimental designs that have been proposed by past researchers for studying causalmediation mechanisms Chapter 10 compares the identification assumptions and the analyticprocedures across a wide range of analytic methods These discussions are followed by anintroduction of the ratio-of-mediator-probability weighting (RMPW) strategy in Chapter 11In comparison with most existing strategies for causal mediation analysis RMPW relies onrelatively fewer identification assumptions and model-based assumptions Chapters 12 and13 will show that this new strategy can be applied broadly with extensions to multilevelexperimental designs and to studies of complex mediation mechanisms involving multiplemediators

113 Spill-over effects of a treatment

It is well known in vaccination research that an unvaccinated person can benefit when mostother people in the local community are vaccinated This is viewed as a spill-over of the treat-ment effect from the vaccinated people to an unvaccinated person Similarly due to socialinteractions among individuals or groups of individuals an intervention received by somemay generate a spill-over impact on others affiliated with the same organization or connectedthrough the same network For example effective policing in one neighborhood may driveoffenders to operate in other neighborhoods In evaluating an innovative community policingprogram researchers found that a neighborhood not assigned to community policing tended tosuffer if the surrounding neighborhoods were assigned to community policing This is aninstance in which a well-intended treatment generates an unintended negative spill-over effectHowever being assigned to community policy was found particularly beneficial when thesurrounding neighborhoods also received the intervention (Verbitsky-Savitz and Raudenbush

8 CAUSALITY IN A SOCIAL WORLD

2012) In another example the impact of a school-based mentoring program targeted atstudents displaying delinquent behaviors may be enhanced for a focal student if his or herat-risk peers are assigned to the mentoring program at the same time

The previous examples challenge the ldquostable unit treatment value assumptionrdquo (SUTVA)that has been invoked in most causal inference studies This assumption states that an indi-vidualrsquos potential response to a treatment depends neither on how the treatment is assignednor on the treatment assignments of other individuals (Rubin 1980 1986) Rubin and othersbelieve that without resorting to SUTVA the causal effect of a treatment becomes hard todefine and that causal inference may become intractable However as Sobel (2006) has illus-trated with the Moving to Opportunity (MTO) experiment that offered housing vouchersenabling low-income families to move to low-poverty neighborhoods there are possible con-sequences of violating SUTVA in estimating the treatment effects on outcomes such as safetyand self-sufficiency Because social interactions among individuals may affect whether onevolunteers to participate in the study whether one moves to a low-poverty neighborhood afterreceiving the housing voucher as well as housing project residentsrsquo subjective perceptions oftheir neighborhoods the program may have a nonzero impact on the potential outcome of theuntreated As a result despite the randomization of treatment assignment the mean differencein the observed outcome between the treated units and the untreated units may be biased forthe average treatment effect Rather the observable quantity is the difference between theeffect of treating some rather than treating none for the treated and the effect of treating somerather than treating none for the untreated the latter being the pure spill-over effect for theuntreated

In making attempts to relax SUTVA researchers have proposed several alternative frame-works that incorporate possible spill-over effects (see Hong and Raudenbush 2013 for areview) Hong (2004) presented a model that involves treatments and treatment settingsA treatment setting for an individual is a local environment constituted by a set of agentsand participants along with their treatment assignments An individualrsquos potential outcomevalue under a given treatment is assumed stable when the treatment setting is fixed the poten-tial outcome may take different values when the treatment setting shifts One may investigatewhether the treatment effect depends on the treatment setting Applying this framework Hongand Raudenbush (2006) examined the effect on a childrsquos academic growth of retaining thechild in kindergarten rather than promoting the child to first grade when a relatively smallproportion of low-achieving peers in the same school are retained as opposed to when a rel-atively large proportion of the peers are retained Hudgens and Halloran (2008) presented arelated framework in which the effect on an individual of the treatment received by this indi-vidual is distinguished from the effect on the individual of the treatment received by others inthe same local community These effects can be identified if communities are assigned atrandom to different treatment assignment strategies (such as retaining a large proportion oflow-achieving students as opposed to retaining a small proportion of such students) and sub-sequently individuals within a community are randomized for treatment Chapter 14 reviewsthese frameworks and discusses identification and estimation strategies

Social contagion may also serve as an important channel through which the effect of anintervention is transmitted For example a school-wide intervention may reduce aggressivebehaviors and thereby improve studentsrsquo psychological well-being by improving the qualityof interpersonal relationships in other classes as well as in onersquos own class This is becausechildren interact not simply with their classmates but also with those from other classes in thehallways or on the playground (VanderWeele et al 2013) In this case the spill-over betweenclasses becomes a part of the mediation mechanism In a study of student mentoring whether

9INTRODUCTION

an individual actually participated in the mentoring program despite the initial treatmentassignment is typically viewed as a primary mediator The proportion of onersquos peers that par-ticipated in the program may act as a second mediator A treatment may also exert its impactpartly through regrouping individuals and thereby changing peer composition (Hong andNomi 2012) Chapter 15 discusses analytic strategies for detecting spill-over as a part ofthe causal mediation mechanism

12 Weighting methods for causal inference

In most behavioral and social science applications major methodological challenges arise dueto the selection of treatments in a quasiexperimental study and the selection of mediator valuesin experimental and quasiexperimental studies Statistical methods most familiar to research-ers in social sciences are often inadequate for causal inferences with regard to multivaluedtreatments moderation mediation and spill-over The book offers a major revision to under-standing of causality in the social world by introducing two complementary weighting stra-tegies both featuring nonparametric approaches to estimating these causal effects The newweighting methods greatly simplify model specifications while enhancing the robustness ofresults by minimizing the reliance on some key assumptions

The propensity score-basedMMWSmethod removes selection bias associated with a largenumber of covariates by equating the pretreatment composition between treatment groups(Hong 2010a 2012 Huang et al 2005) Unlike propensity score matching and stratificationthat are mostly restricted to evaluations of binary treatments the MMWS method is flexiblefor evaluating binary and multivalued treatments by approximating a completely randomizedexperiment In evaluating whether the treatment effects differ across subpopulations definedby individual characteristics or treatment settings researchers may assign weights within eachsubpopulation in order to approximate a randomized block design To investigate whetherone treatment moderates the effect of another concurrent treatment researchers may assignweights to the data to approximate a factorial randomized design The method can also beused to assess whether the effect of an initial treatment is amplified or weakened by a subse-quent treatment or to identify an optimal treatment sequence through approximating a sequen-tial randomized experiment Even though such analyses can similarly be conducted throughinverse-probability-of-treatment weighting (IPTW) that has been increasingly employed inepidemiological research (Hernaacuten Brumback and Robins 2000 Robins Hernaacuten andBrumback 2000) IPTW is known for bias and imprecision in estimation especially whenthe propensity score models are misspecified in their functional forms (Hong 2010a Kangand Schafer 2007 Schafer and Kang 2008 Waernbaum 2012) In contrast the nonparamet-ric MMWS method displays a relatively high level of robustness despite such misspecifica-tions and also gains efficiency as indicated by simulation results (Hong 2010a)

To study causal mediation mechanisms the RMPW method decomposes the total effectof a treatment into an ldquoindirect effectrdquo transmitted through a specific mediator and a ldquodirecteffectrdquo representing unspecified mechanisms In contrast with most existing methods formediation analysis the RMPW-adjusted outcome model is extremely simple and is nonpara-metric in nature It generates estimates of the causal effects along with their sampling errorswhile adjusting for pretreatment covariates that confound the mediatorndashoutcome relationshipsthrough weighting The method applies regardless of the distribution of the outcome the dis-tribution of the mediator or the functional relationship between the outcome and the mediator

10 CAUSALITY IN A SOCIAL WORLD

Page 25: Thumbnail - download.e-bookshelf.de...4.2.4 IPTW for bias removal 84 4.3 MMWS 86 4.3.1 Theoretical rationale 86 ... 8.3.4.2 Growth model in the absence of confounding 200 8.3.4.3 Weighted

2012) In another example the impact of a school-based mentoring program targeted atstudents displaying delinquent behaviors may be enhanced for a focal student if his or herat-risk peers are assigned to the mentoring program at the same time

The previous examples challenge the ldquostable unit treatment value assumptionrdquo (SUTVA)that has been invoked in most causal inference studies This assumption states that an indi-vidualrsquos potential response to a treatment depends neither on how the treatment is assignednor on the treatment assignments of other individuals (Rubin 1980 1986) Rubin and othersbelieve that without resorting to SUTVA the causal effect of a treatment becomes hard todefine and that causal inference may become intractable However as Sobel (2006) has illus-trated with the Moving to Opportunity (MTO) experiment that offered housing vouchersenabling low-income families to move to low-poverty neighborhoods there are possible con-sequences of violating SUTVA in estimating the treatment effects on outcomes such as safetyand self-sufficiency Because social interactions among individuals may affect whether onevolunteers to participate in the study whether one moves to a low-poverty neighborhood afterreceiving the housing voucher as well as housing project residentsrsquo subjective perceptions oftheir neighborhoods the program may have a nonzero impact on the potential outcome of theuntreated As a result despite the randomization of treatment assignment the mean differencein the observed outcome between the treated units and the untreated units may be biased forthe average treatment effect Rather the observable quantity is the difference between theeffect of treating some rather than treating none for the treated and the effect of treating somerather than treating none for the untreated the latter being the pure spill-over effect for theuntreated

In making attempts to relax SUTVA researchers have proposed several alternative frame-works that incorporate possible spill-over effects (see Hong and Raudenbush 2013 for areview) Hong (2004) presented a model that involves treatments and treatment settingsA treatment setting for an individual is a local environment constituted by a set of agentsand participants along with their treatment assignments An individualrsquos potential outcomevalue under a given treatment is assumed stable when the treatment setting is fixed the poten-tial outcome may take different values when the treatment setting shifts One may investigatewhether the treatment effect depends on the treatment setting Applying this framework Hongand Raudenbush (2006) examined the effect on a childrsquos academic growth of retaining thechild in kindergarten rather than promoting the child to first grade when a relatively smallproportion of low-achieving peers in the same school are retained as opposed to when a rel-atively large proportion of the peers are retained Hudgens and Halloran (2008) presented arelated framework in which the effect on an individual of the treatment received by this indi-vidual is distinguished from the effect on the individual of the treatment received by others inthe same local community These effects can be identified if communities are assigned atrandom to different treatment assignment strategies (such as retaining a large proportion oflow-achieving students as opposed to retaining a small proportion of such students) and sub-sequently individuals within a community are randomized for treatment Chapter 14 reviewsthese frameworks and discusses identification and estimation strategies

Social contagion may also serve as an important channel through which the effect of anintervention is transmitted For example a school-wide intervention may reduce aggressivebehaviors and thereby improve studentsrsquo psychological well-being by improving the qualityof interpersonal relationships in other classes as well as in onersquos own class This is becausechildren interact not simply with their classmates but also with those from other classes in thehallways or on the playground (VanderWeele et al 2013) In this case the spill-over betweenclasses becomes a part of the mediation mechanism In a study of student mentoring whether

9INTRODUCTION

an individual actually participated in the mentoring program despite the initial treatmentassignment is typically viewed as a primary mediator The proportion of onersquos peers that par-ticipated in the program may act as a second mediator A treatment may also exert its impactpartly through regrouping individuals and thereby changing peer composition (Hong andNomi 2012) Chapter 15 discusses analytic strategies for detecting spill-over as a part ofthe causal mediation mechanism

12 Weighting methods for causal inference

In most behavioral and social science applications major methodological challenges arise dueto the selection of treatments in a quasiexperimental study and the selection of mediator valuesin experimental and quasiexperimental studies Statistical methods most familiar to research-ers in social sciences are often inadequate for causal inferences with regard to multivaluedtreatments moderation mediation and spill-over The book offers a major revision to under-standing of causality in the social world by introducing two complementary weighting stra-tegies both featuring nonparametric approaches to estimating these causal effects The newweighting methods greatly simplify model specifications while enhancing the robustness ofresults by minimizing the reliance on some key assumptions

The propensity score-basedMMWSmethod removes selection bias associated with a largenumber of covariates by equating the pretreatment composition between treatment groups(Hong 2010a 2012 Huang et al 2005) Unlike propensity score matching and stratificationthat are mostly restricted to evaluations of binary treatments the MMWS method is flexiblefor evaluating binary and multivalued treatments by approximating a completely randomizedexperiment In evaluating whether the treatment effects differ across subpopulations definedby individual characteristics or treatment settings researchers may assign weights within eachsubpopulation in order to approximate a randomized block design To investigate whetherone treatment moderates the effect of another concurrent treatment researchers may assignweights to the data to approximate a factorial randomized design The method can also beused to assess whether the effect of an initial treatment is amplified or weakened by a subse-quent treatment or to identify an optimal treatment sequence through approximating a sequen-tial randomized experiment Even though such analyses can similarly be conducted throughinverse-probability-of-treatment weighting (IPTW) that has been increasingly employed inepidemiological research (Hernaacuten Brumback and Robins 2000 Robins Hernaacuten andBrumback 2000) IPTW is known for bias and imprecision in estimation especially whenthe propensity score models are misspecified in their functional forms (Hong 2010a Kangand Schafer 2007 Schafer and Kang 2008 Waernbaum 2012) In contrast the nonparamet-ric MMWS method displays a relatively high level of robustness despite such misspecifica-tions and also gains efficiency as indicated by simulation results (Hong 2010a)

To study causal mediation mechanisms the RMPW method decomposes the total effectof a treatment into an ldquoindirect effectrdquo transmitted through a specific mediator and a ldquodirecteffectrdquo representing unspecified mechanisms In contrast with most existing methods formediation analysis the RMPW-adjusted outcome model is extremely simple and is nonpara-metric in nature It generates estimates of the causal effects along with their sampling errorswhile adjusting for pretreatment covariates that confound the mediatorndashoutcome relationshipsthrough weighting The method applies regardless of the distribution of the outcome the dis-tribution of the mediator or the functional relationship between the outcome and the mediator

10 CAUSALITY IN A SOCIAL WORLD

Page 26: Thumbnail - download.e-bookshelf.de...4.2.4 IPTW for bias removal 84 4.3 MMWS 86 4.3.1 Theoretical rationale 86 ... 8.3.4.2 Growth model in the absence of confounding 200 8.3.4.3 Weighted

an individual actually participated in the mentoring program despite the initial treatmentassignment is typically viewed as a primary mediator The proportion of onersquos peers that par-ticipated in the program may act as a second mediator A treatment may also exert its impactpartly through regrouping individuals and thereby changing peer composition (Hong andNomi 2012) Chapter 15 discusses analytic strategies for detecting spill-over as a part ofthe causal mediation mechanism

12 Weighting methods for causal inference

In most behavioral and social science applications major methodological challenges arise dueto the selection of treatments in a quasiexperimental study and the selection of mediator valuesin experimental and quasiexperimental studies Statistical methods most familiar to research-ers in social sciences are often inadequate for causal inferences with regard to multivaluedtreatments moderation mediation and spill-over The book offers a major revision to under-standing of causality in the social world by introducing two complementary weighting stra-tegies both featuring nonparametric approaches to estimating these causal effects The newweighting methods greatly simplify model specifications while enhancing the robustness ofresults by minimizing the reliance on some key assumptions

The propensity score-basedMMWSmethod removes selection bias associated with a largenumber of covariates by equating the pretreatment composition between treatment groups(Hong 2010a 2012 Huang et al 2005) Unlike propensity score matching and stratificationthat are mostly restricted to evaluations of binary treatments the MMWS method is flexiblefor evaluating binary and multivalued treatments by approximating a completely randomizedexperiment In evaluating whether the treatment effects differ across subpopulations definedby individual characteristics or treatment settings researchers may assign weights within eachsubpopulation in order to approximate a randomized block design To investigate whetherone treatment moderates the effect of another concurrent treatment researchers may assignweights to the data to approximate a factorial randomized design The method can also beused to assess whether the effect of an initial treatment is amplified or weakened by a subse-quent treatment or to identify an optimal treatment sequence through approximating a sequen-tial randomized experiment Even though such analyses can similarly be conducted throughinverse-probability-of-treatment weighting (IPTW) that has been increasingly employed inepidemiological research (Hernaacuten Brumback and Robins 2000 Robins Hernaacuten andBrumback 2000) IPTW is known for bias and imprecision in estimation especially whenthe propensity score models are misspecified in their functional forms (Hong 2010a Kangand Schafer 2007 Schafer and Kang 2008 Waernbaum 2012) In contrast the nonparamet-ric MMWS method displays a relatively high level of robustness despite such misspecifica-tions and also gains efficiency as indicated by simulation results (Hong 2010a)

To study causal mediation mechanisms the RMPW method decomposes the total effectof a treatment into an ldquoindirect effectrdquo transmitted through a specific mediator and a ldquodirecteffectrdquo representing unspecified mechanisms In contrast with most existing methods formediation analysis the RMPW-adjusted outcome model is extremely simple and is nonpara-metric in nature It generates estimates of the causal effects along with their sampling errorswhile adjusting for pretreatment covariates that confound the mediatorndashoutcome relationshipsthrough weighting The method applies regardless of the distribution of the outcome the dis-tribution of the mediator or the functional relationship between the outcome and the mediator

10 CAUSALITY IN A SOCIAL WORLD