Identifying When E ect Restoration Will Improve Estimates ... · Huseyin Oktay Akanksha Atrey David Jensen Abstract Several methods have been developed that combine mul-tiple models

Identifying When Effect RestorationWill Improve Estimates of Causal Effect

Huseyin Oktay∗ Akanksha Atrey∗ David Jensen∗

Abstract

Several methods have been developed that combine mul-

tiple models learned on different data sets and then use

that combination to reach conclusions that would not

have been possible with any one of the models alone. We

examine one such method—effect restoration—which was

originally developed to mitigate the effects of poorly mea-

sured confounding variables in a causal model. We show

how effect restoration can be used to combine results from

different machine learning models and how the combined

model can be used to estimate causal effects that are not

identifiable from either of the original studies alone. We

characterize the performance of effect restoration by using

both theoretical analysis and simulation studies. Specif-

ically, we show how conditional independence tests and

common assumptions can help distinguish when effect

restoration should and should not be applied, and we use

empirical analysis to show the limited range of conditions

under which effect restoration should be applied in prac-

tical situations.

1 Introduction

Growing use of machine learning has led to an interestin combining models learned on different data setsand using those models to make inferences that wouldnot have been possible with any one model. This isparticularly valuable when the goal is causal inference[17,20], one of the most challenging tasks in machinelearning. For example, researchers in statistics havelong used meta-analysis to produce causal estimateswith statistical power that exceeds any individualstudy [9]. Researchers in causal graphical modelshave taken a very different approach and studied howto learn a single model from multiple data sets withoverlapping sets of variables [23].

In this paper, we use models learned on multi-ple data sets to correct for confounding variables,perhaps the most common threat to the validity ofcausal inferences. Ignoring confounding variables in-troduces bias in estimates of treatment effect [17,20],and conditioning on confounding variables is a com-

∗University of Massachusetts Amherst, Amherst, MA, USA

{hoktay, aatrey, jensen}@cs.umass.edu

mon approach to correct this bias. We study how toaccurately condition on confounding variables evenwhen those variables are not accurately measured ina data set.

Specifically, we examine how models learned froma second data set can be used to recover or restorea confounding variable and thus reduce the biasof the first study’s estimate of causal effect. Todo this, we apply effect restoration, a techniqueoriginally proposed by Kuroki & Pearl [8] to adjustfor measurement error in confounding variables. Weshow that the models necessary to implement thismethod can be drawn from different data sets andthat, under specific conditions, such models cangreatly reduce the bias of causal estimates.

For example, consider the graphical model inFig. 1b. One data set contains a treatment X, anoutcome Y , and another variable W . Both X andY are caused by an unmeasured confounder U thatalso causes W . A second data set contains both Uand W . Effect restoration can be used to combinethe studies, restoring the effect of U on X and Y byusing knowledge of P (U,W ) from the second study.When the underlying graphical structure in Fig. 1b isassumed and the two studies draw data from similardistributions, effect restoration will reduce the biasdue to the unmeasured confounder in the first study.

This application of effect restoration is relativelystraightforward, and the theoretical properties of ef-fect restoration established by Kuroki & Pearl alsohold in this particular application. However, muchremains unknown about the practical utility of theproposed methods. First, it is far from straightfor-ward to conclude that the assumed structure holdsfor a specific combination of treatment, outcome, andpotential confounders. Second, it is unclear whetherthe adjustment made by effect restoration still pro-vides a bias-free estimate when the underlying gener-ative process does not correspond to the structure inFig. 1b, a question Kuroki & Pearl considered out-of-the scope in their study. Finally, the range of poten-tial benefits and the conditions under those benefitsoccur have not been previously examined.

In this paper, we extend the use cases of effect

Copyright c© 2019 by SIAMUnauthorized reproduction of this article is prohibited

(a) (b)

Figure 1: Graphical models with confounding. In (a),U is a confounder for treatment (X) and outcome(Y ). In (b), one study (shown in black) containsmeasured variablesX, Y , and W , and an unmeasuredconfounder U , and a second study (shown in gray)contains measured variables U and W .

restoration and characterize its behaviour under var-ious realistic scenarios. Specifically:

RQ1. How does the actual causal structure affect theaccuracy of effect restoration?

RQ2. Practically, what conditions are necessary foreffect restoration to substantially improve es-timates of causal effect?

RQ3. For a given set of observations, X, Y , and W ,what are the sufficient conditions to identifythe underlying graphical structure?

Our answers to these questions make importantcontributions to the understanding of effect restora-tion. First, we show that most variants of the struc-ture assumed in the original work on effect restora-tion lead the method to either have no effect or toharm estimates of the causal effect of treatment onoutcome. Second, we use empirical analysis to showthat the relative benefit of effect restoration is high-est for estimating small treatment effects with largeconfounding bias. By using empirical data from areal-world randomized experiment, we show how ef-fect restoration removes bias more than its naturalalternatives. Third, by leveraging graphical modelsand d -separation, we show for the first time that sim-ple rules and typical temporal ordering assumptionsare sufficient to identify whether an empirical systemhas the causal structure necessary for effect restora-tion to improve causal estimates.

2 Related Work

A wide variety of methods for estimating causaleffects assume that all confounders are observed.This assumption, sometimes referred to as causalsufficiency [20], implies that all variables that arecauses of two or more observed variables in a givendata set are also observed. Causal sufficiency is

typically required to guarantee consistent and/orunbiased causal estimates [5, 18,22].

We explore the use of effect restoration to accountfor bias from unobserved confounder variables. Apartfrom randomized experiments, several methods havebeen proposed to account for unobserved confoundersin non-experimental contexts. These include instru-mental variable designs [1], which can produce unbi-ased estimates of causal effect under some very re-strictive assumptions, as well as sensitivity analysistechniques, which are used to assess the likelihood ofthe existence of a potential unobserved confounderby estimating the required confounding bias to re-verse the estimated effect [10]. In contrast, our workaccounts for unobserved confounders by using modelsof confounders derived from a separate data set.

Our approach resembles transfer learning ap-proaches in machine learning where a target learningtask is performed by using knowledge obtained fromprevious related tasks [11, 12, 15, 25]. Our study de-parts from the standard transfer learning paradigmbecause we first obtain knowledge from a predictivelearning task and then use such knowledge for a sub-sequent causal estimation task.

3 Background

To formalize the problem of effect restoration, we usegraphical models and Pearl’s do-calculus [17]. For ex-ample, in the graphical model shown in Fig. 1a, wedenote the direct effect of X on Y as P (Y |do(X)). Inobservational studies, this quantity is different thansimply conditioning on X (i.e., P (Y |X)). Condition-ing denotes the probability distribution of Y in eachpossible world defined by a specific X value. How-ever, the do operator denotes the probability distri-bution of Y after actively setting the value of X (i.e.,intervention). Hence P (Y | do(X)) represents the ef-fect of manipulating the values of X and P (Y |X) rep-resents passive observation. Readers should consultPearl [17] for additional discussion of the do-calculus.

For the graphical model in Fig. 1a, when all vari-ables are observed and under identifiability condi-tions (again, see Pearl [17] for details), the probabilityP (Y | do(X)) can be estimated [8]:

(3.1) P (Y | do(X = x)) =∑U

P (X,Y, U)

P (X | U)

Given this interventional distribution, for a binaryX, the treatment effect (TE) of X on Y can becalculated (note that Y1 is equal to Y = 1 and theformer format is chosen for all variables for brevity):

TE = log

(P (Y1 | do(X1))

P (Y0 | do(X1))

)− log

(P (Y1 | do(X0))

P (Y0 | do(X0))

)


Figure 2: Bias and Variance Results for Different Underlying Structures.

Table 1: Conditional (In)dependence Relationships for All Simple Graphical Structures.

U is temporally prior to X and Y U is temporally posterior to X or Y CycleA B C D E F G H I

X ⊥⊥ Y 6 6 6 6 6 6 6 6 N/AX ⊥⊥ Y |W 6 6 6 6 6 6 6 6 N/AX ⊥⊥W 6 4 6 4 6 6 6 6 N/AX ⊥⊥W |Y 6 6 6 4 6 4 6 6 N/AY ⊥⊥W 6 6 6 4 6 6 6 6 N/AY ⊥⊥W |X 6 6 4 4 6 6 4 6 N/A

Table 2: Effect of Activity Level on Weight Gain

Weight GainActivity Level 0 1

0 0.20 0.801 0.60 0.40

From perfect observations of X, Y , and U , un-biased estimates of TE can be obtained using vari-ous modeling methods. However, these methods failto provide unbiased estimates if U is measured witherror. Kuroki & Pearl used the do-calculus to ad-just for confounding variables with measurement er-

ror [8]. Specifically, the adjustment uses knowledgeof the error distribution to correct the causal esti-mate made using the observed values. Fig. 1b showsthe graphical model assumed in their extended frame-work. They propose that, under certain conditions,the TE of X on Y can be restored bias-free given anobserved surrogate variable for the confounder U (i.e.,W ) and a known error distribution (i.e., P (W | U)).

Our experiments use a specific formulation ofthe Kuroki & Pearl TE estimator with the do-calculus framework. For example, when estimatingthe effect of activity level (X) on weight gain (Y ),age (U) is a potential confounding variable because


it affects both activity level and weight gain. Ourgoal is to estimate the interventional distribution ofP (WeightGain | do(ActivityLevel)).

Assume the distribution shown in Table 2. Ac-cording to this distribution, when a person has lowactivity level, the odds of weight gain is 0.80

0.20 = 4.Whereas, when a person has high activity level, theodds of weight gain is 0.40

0.60 = 0.66. We define the dif-ference in the log-odds ratio for different treatmentvalues as the causal effect of treatment (e.g., activitylevel) on outcome (e.g., weight gain).

4 Effects of Underlying Structure on Biasand Variance

The effect restoration method proposed by Kuroki &Pearl assumes U is a confounder as shown in Fig. 1b.Here, we relax this assumption and characterize theperformance of effect restoration under all possiblemodifications of the simple graphical structure sug-gested in the original paper.

Fig. 2 shows all possible modified structuresbetween X, Y , W , and U with the assumptionsthat are typical in many social science and medicalstudies [5, 17, 22]: (1) X is temporally prior toY ; (2) W is a noisy measurement of U ; and (3)U is temporally prior to X and Y . We employsimulation studies to characterize the performance ofeffect restoration for each graphical model structure.

4.1 Data Generation We generated continuousand discrete synthetic data consistent with the graph-ical structures in Fig. 2. We explain the discrete datageneration process for the graphical structure shownin column A of the figure. Other graphical struc-tures follow the same steps except that the added, re-moved, or reversed dependencies in the structure en-tail adding, removing, or changing the order of stepsin the data generation process, respectively.

In the generative processes below, italic lettersdenote scalar values (e.g., N ); upper-case bold char-acters denote vectors (e.g., W); each element of avector is accessed by an index subscript (e.g., wi);correlations between variables are referred with sub-scripts such as ρuw denoting the correlation betweenU and W; marginal and conditional probabilities aredenoted by the upper-case letter P. Following is thepseudocode to generate binary data for the structurein column A in Fig. 2.

We vary ρuw, ρux, ρuy and ρxy between (0,1).We model P (Y | U,X) as a noisy-OR conditionalprobability distribution [6] where:

P (Y = 0 |U,X) = (1− λ0) ∗ (1− ρuy)U ∗ (1− ρxy)X

P (Y = 1 |U,X) = 1− (1− λ0) ∗ (1− ρuy)U ∗ (1− ρxy)X

Algorithm 1 Binary Data Generator

Initialize correlation values ρuw, ρuy, ρux, ρxyDraw prior Pu ∼ Uniform(0, 1)Draw N values for U from Bernoulli(Pu)for ui in U do

Draw p′ ∼ Uniform(0, 1)if p′ < ρuw thenwi = ui

elseDraw a value for wi ∼ Bernoulli(p = 0.5)

for ui in U doDraw p′ ∼ Uniform(0, 1)if p′ < ρux thenxi = ui

elseDraw a value for xi ∼ Bernoulli(p = 0.5)

Draw Y ∼ Binomial(N,P (Y = 1 |U,X))

We set the value of λ0 = 0.01 as the noise pa-rameter of the noisy-OR model. The noisy-OR modelis a popular choice to represent discrete conditionalprobability distributions [16] due to its compact rep-resentation with few parameters and its ability to ap-proximate many learned distributions [26].

For each parameter setting {ρuw, ρuy, ρux, ρxy},we generate 50 data sets, each containing 10,000 in-stances. As noted, for other graphical structures werevise the data generation process for the correspond-ing removed dependence. For example, in Fig. 2 col-umn B, the dependence between U and X is removed.To account for this, we sample a prior for PX froma uniform distribution; we sample values for X usingthe prior instead of using P (X | U).

In our experiments, we calculate the true treat-ment effect (i.e., TE) as our ground truth by using thevalues of U . We estimate the treatment effect (i.e.,TE’) with the following three approaches: (1) IgnoreW: Simply ignoring the measurements of W , (2) Ig-nore measurement error: Using W and ignoringthe measurement error, and (3) Effect restoration:Using W and adjusting for the measurement error.Note that these three methods do not use values ofU , only use values of X, Y , and W . We measure thestandard deviation of error values across experimentsand the lines correspond to the mean standard devi-ation for varying strength of effect values betweenU and W . We measure the bias for each approachin each experiment by normalized error, as shownin equation 4.2 and the lines correspond to the localweighted regression of bias values for varying strengthof effect between U and W .

(4.2) ε =TE − TE′

TE


Figure 3: Bias as Treatment and Confounding EffectChange.

4.2 Bias and Variance for Different Underly-ing Structures We perform simulation analysis foreach graphical structure in the first row of Fig. 2.We plot both the variance and the bias in estimatingthe treatment effect in the second and third row, re-spectively. In each plot, we show the behaviour foreach of the three different approaches as the mea-surement error changes along the x-axis. We calcu-late the measurement error by the strength of de-pendence between U and W using the Cramer’s Vcoefficient. The stronger the dependence, the weakerthe measurement error. On the y-axis we plot locallysmoothed normalized absolute error for bias and lo-cally smoothed standard deviation for the variancefor the three different approaches. Fig. 2 shows theresults for a fixed treatment and confounding effectfor all graphical structures considered.

When the confounding variable is simply ignoredfor the graphical model in the first column in Fig. 2,unsurprisingly, we see a constant bias in our estimateof the treatment effect. However, when values ofW are used as if they are the perfect observationsof U (i.e., when the measurement error is ignored),the bias in treatment effect estimate is reduced.This reduction is particularly significant when W ishighly correlated with U (i.e., measurement error issmall). Finally, when values of W are used with theeffect restoration adjustment, the bias is consistentlyreduced more than the other approaches.

Furthermore, the smaller the measurement error

between W and U , the smaller the bias in estimation.Also, the larger the measurement error, the more therelative benefit of explicitly adjusting for it using ef-fect restoration versus simply ignoring it. However,when U and W are poorly correlated (i.e., measure-ment error is high), applying effect restoration comeswith the cost of increased variance, as shown in thesecond row in Fig. 2.

For the other graphical models in Fig. 2 columnsB, C, D, we observe that ignoring or using W directlyprovide consistent estimates for the treatment effect.However, simply applying effect restoration might in-crease bias as well as variance. Hence, applying effectrestoration regardless of the underlying structure canresult in an incorrect estimate.

We perform additional experiments for thegraphical model in column A by changing the valuesof treatment and confounding effect via modifyingthe correlation coefficient. Fig. 3 shows the results ofour experiments. In these plots, the treatment effectincreases along the big y-axis and confounding effectincreases along the big x-axis. Along the x-axis ineach plot, the strength of effect between U and W in-creases. These results suggest that effect restorationis most effective when the treatment effect is smalland the confounding effect is high.

5 Detecting the Underlying Structure

In the previous section, we presented empirical ev-idence that when the underlying structure deviatesfrom the confounding variable case, the adjustmentprovided by effect restoration can increase bias andvariance. This raises a natural question: Can we de-tect when to apply effect restoration? Instead of as-suming that the confounding bias exists, we proposeto verify if it exists by using d-separation and typicaltemporal ordering constraints on the variables.

Note that U may or may not be a confoundingvariable for X and Y . Our goal is to identify sufficientconditions to determine if U is a confounding variableand only apply effect restoration when it is.

In Table 1, we list all possible underlying struc-tures with variables X, Y , W , and U that satisfy thestated assumptions 1 and 2 in Section 4. There arenine possible graphical structures. We individuallyaccount for dependence and independence relation-ships in each of them. One of the possible structurescontains a cycle (i.e., the structure in the last columnof Table 1 is not a DAG) and hence out of scope forour discussion. In four of these structures, U is tem-porally prior to X and Y (i.e., columns A throughD). In the remaining four structures, U is temporallyposterior to X and Y (i.e., columns E to H.) In therows of Table 1, for each graphical model, we list


(a) Relationships Between

Indexing, Disk Reads, and

Cache Hits. System.

(b) Bias in the Estimated

Effect of Cache Hits on

Disk Reads.

Figure 4: Effect Restoration on PostgreSQL Data.

all the marginal and conditional independence rela-tions between X, Y , and W . Columns of the tablecorrespond to possible underlying graphical models.Each column vector corresponds to the expected con-ditional dependence and independence relations forthe corresponding graphical model.

First, we note that many of these graphical struc-tures are identifiable based on empirically testableconditional independence relations (i.e., structures incolumns B, D, and F in Table 1) assuming accurateconditional independence tests. However, some struc-tures are indistinguishable with the given set of con-ditional independence relations (e.g., structures in A,E, and H share the same set of conditional indepen-dence relations).

Second, if we also make a common assumptionthat covariates are measured pre-treatment [24], thenonly the structures in columns of A, B, C, and D willbe possible. Furthermore, conditional independencerelations would be sufficient to distinguish amongeach of these graphical models.

Thus, a combination of common temporal as-sumptions and conditional independence relations aresufficient to determine the underlying graphical struc-ture from X, Y , and W .

Finally, we note that two of the structures pos-sible when those common temporal assumptions arenot made introduce significant bias to causal esti-mates if effect restoration is applied. Applying effectrestoration to structure E will remove one pathwayfor causal effect and thus (typically) underestimatethe total effect of do(X). Applying effect restora-tion to structure H will induce dependence betweenX and Y even if there exists no direct causal depen-dence between these variables. This classic exampleof conditioning on the descendant of a collider would(typically) overestimate the total effect of do(X).

6 Empirical Evaluation of Effect Restoration

In this section, we empirically evaluate the perfor-mance of effect restoration in real-world settings aswell as demonstrate how effect restoration can beused to combine results from different machine learn-ing models to estimate causal effects.

6.1 Effect Restoration in Real Data We firstevaluate the performance of effect restoration onexperimental data obtained from real-world settings.We use the experimental data compiled by Garantand Jensen [4] about the effects of interventions onlarge-scale software systems.

Specifically, we use their experimental data setabout PostgreSQL, a large open-source relationaldatabase management system. The authors ranover 11, 000 queries under each of eight differentsystem configurations. This creates a nearly idealdata set for causal estimation because each subject(i.e., query) is observed in each treatment condition(i.e., configuration setting). This approach allowsdirect interventional estimates of the effect of atreatment on outcome variables in the context ofother variables representing characteristics of queriesand intermediate states of the database server.

The authors then use GES, a score-based algo-rithm for structure learning [2], to recover the un-derlying structure for the PostgreSQL domain. Fromtheir partially learned graphical structure, we focuson specific regions in which the sub-structures satisfythe effect restoration conditions as shown in Table 1.

We identify two cases in which we can apply effectrestoration. One is shown in Fig. 4a and representsthe relationships between indexing (whether indexesare available), disk reads (the level of disk reads),and cache hits (the level of cache hits). Generallyspeaking, indexing affects both disk reads and cachehits. Indexing reduces the disk reads (i.e., a result ofa query can be retrieved with fewer block reads) andincreases the cache hits. Fig. 4a shows the cache hitsas a cause of disk reads. The authors note that thedirection of this edge between disk reads and cachehits might also be in the opposite direction. In ouranalysis, we separately analyzed graphical structuresin both directions and the results for effect restorationwere similar in both.

For simplicity, we converted each variable to abinary variable by using the median value of eachvariable as a threshold. Our goal is to estimate theeffect of cache hits on disk reads under noisy mea-surements of indexing. To obtain such noisy measure-ments, we manually added noise to the values of in-dexing to obtain indexing’ by sometimes ignoring theconditional dependence between its true values and


Figure 5: Effect Restoration for Unobserved Variables in Synthetic Data.

randomly assigning values as controlled by a noise pa-rameter. Then we use these noisy measurements toadjust for the confounding bias of indexing. As in ourprior experiments, we compare three approaches: (1)Ignoring the values of noisy measurements for index-ing, (2) Using the values of indexing, ignoring thatthey are noisy, (3) Using the values by correcting forthe noise in their measurements. We calculate thetrue effect by using the true values of indexing.

In Fig. 4b, we plot the normalized error in theestimated treatment effect with respect to measure-ment error in indexing. As with our results on simu-lated data, estimation with effect restoration providessignificantly smaller bias than alternatives. In addi-tion, as the measurement error of the confoundingvariable increases (i.e., the strength of effect betweenU and W decreases), the relative benefit of applyingeffect restoration increases over simply ignoring themeasurement error.

6.2 Effect Restoration with Predictive Mod-els Conditioning on a confounding variable by usingnoisy measurements can also be thought of condition-ing on an unobserved variable when both its predic-tions and the error distribution of estimations canbe inferred using an independent estimation process.For example, assume we want to estimate the effectof activity level on weight gain. Assume age is a con-founding variable (i.e., age causes both activity leveland weight gain) and that it is unobserved for thepopulation of interest. Clearly, in this example, weneed to adjust for the effects of age to get an unbi-ased estimate. One idea is to estimate age by usingother observables for the subjects under study. Forexample, such observables can be based on users’ so-cial media content [13], links on social networks [27],first names [14], or names with personal images [3].

We can use the predictions of the model as noisymeasurements of age and, given its error distribution,adjust for its confounding bias. This idea assumes

that the population under study is similar to thepopulation from which the predictive model wasestimated so that we can transfer the knowledge ofthat model to obtain the corresponding predictionsand error distribution. This assumption, referred toas external validity or transportability, is common tonearly all statistical modeling.

6.2.1 Empirical Results on Synthetic DataWe evaluate the idea of using predictive modelsfor effect restoration by constructing a scenario inwhich we observe activity levels, and weight gainrecordings of a population along with their firstnames. Although one can use more complicatedmodels for predicting age, here we use the modeldescribed by Oktay et al. [14] because of its simplicity.

In our experiments, we generate synthetic datafor each subject with activity levels, weight gain, anda first name. We then use the first names to inferan age value for each subject. Finally, we use theinferred age values along with the corresponding errordistribution as reported by Oktay et al. [14] to adjustfor the confounding effect of age.

One might suggest that instead of using a modelto estimate age, we could simply condition on thevalues of first names. There are three reasons toavoid this approach. First, the number of possiblefirst names is large and using first names directlycan lead to a high-variance estimator of causal effectfor most data sets. Second, conditioning on firstnames leaves the back-door path unblocked betweenactivity levels and weight gain, as shown in Fig. 5(see Pearl [17] for a detailed discussion of back-door paths). This implies that the confounding biaswould still exist. Third, our proposal is to plug inany predictive model for the unobserved confounder,and such models can use many independent variablesrather than just one. Again, conditioning on manyindependent variables can lead to a high-varianceestimator. In fact, from the perspective of effect


Figure 6: Effects of Unobserved Variables on StackOverflow Data.

restoration, it may be desirable to use high-capacitymodels in causal estimation process to drive downprediction error and subsequently reduce bias.

In this way, our proposed approach—using apredictive model for effect restoration in pursuit of alow-variance estimator of causal effect—is similar inspirit to the use of predictive models for propensityscore matching [19]. Both approaches use a predictivemodel to summarize the effect of a potentially largenumber of variables.

Fig. 5a shows the graphical model representationof the experiment. As before, we compare estimat-ing the effect when a confounding variable is ignored,when the error in the confounding variable is ignored,and finally when effect restoration is used. Our re-sults with an independent estimation process are sim-ilar to those obtained earlier from both the simula-tion and real data experiments. We observe the mostreduction in bias when correction based on measure-ment error is used. We also show that as the influenceof confounding variable increases, the relative benefitof effect restoration increases.

6.2.2 Empirical Results on Stack OverflowData We demonstrate an application of effectrestoration using predictive models on real data froma popular programming questions and answers site,Stack Overflow. Our data consists of questions, an-swers and users in Stack Overflow from 2008 to early2018. Users interact by posting questions, answer-ing existing questions, and voting. For each questionand answer, an associated score is accumulated asusers vote on them. Furthermore, user reputationsare built based on the scores of their posts.

Recent studies suggest that different demo-graphic groups behave differently on Stack Overflow[7, 21]. Hence, practitioners might want to condition

on demographic variables while estimating causal ef-fects, which are often unobserved. For example, agecan be an important variable to consider. However,despite the users’ having the ability to self-reporttheir age, the data is often missing and potentiallyincorrect. One way to adjust for the effect of age isto apply predictive models to infer users’ ages andthen use those estimated values and error distribu-tion to perform effect restoration. Again, we employthe age model described by Oktay et al. [14].

We use 51 treatment-outcome pairs obtainedfrom question and answer variables. Using condi-tional independence tests on the variable pairs, asdescribed in Section 5, we identify causal structureswhere age can be a confounding variable. Specifi-cally, we use chi-squared independence tests to getthe marginal and conditional independence betweenall pairs of variables. Due to a large sample size(> 400k), the p-values are significant even for low ef-fect sizes. Thus, we focus on marginal independenceeffect sizes that are greater than 0.1. This narrowsdown the set of possible variable pairs to 14, out ofwhich we consider three structures with the highestconfounding effects. We use continuous treatmentand outcome variables, and binary confounder vari-ables for the three structures.

Fig. 6 contains the graphical model representa-tions of the top three causal structures and the ob-served effects using the three causal estimation meth-ods. Similar to our results on the simulated data,conditioning on estimated values reduces effect sizeestimates more than simply ignoring the confound-ing variables. Furthermore, applying effect restora-tion reduces the effect size estimates even more. Thissuggests that we are able to remove the effect of ageon our treatment and outcome variables.


7 Conclusions and Future Work

In this paper, we demonstrate the effectiveness of us-ing effect restoration to combine results from differentmachine learning methods. First, we characterize thebehaviour of effect restoration with measurement er-ror under several plausible graphical structures. Weshow that it is desirable to use effect restoration onlyin one of the four possible graphical models. Inour simulation analysis, effect restoration adjustmentis most effective for small treatment and large con-founding effects. Next, we show that the combinationof common temporal assumptions and d-separationrules can identify if the underlying structure matchesthe conditions under which effect restoration is effec-tive. We also provide empirical evidence that effectrestoration can reduce bias on causal estimation tasksin real data. Finally, we show that this approach canbe used to adjust for unobserved confounding whenused with independent predictive models and theircorresponding error distributions.

Several future research directions appear promis-ing. First, our empirical study of effect restorationcould be generalized for mixed-type data sets. Sec-ond, the use of high-capacity predictive models andtheir limitations could be studied. Third, the impli-cations of effect restoration for estimating joint causalstructures could be further explored.

References

[1] J. D. Angrist, G. W. Imbens, and D. B. Ru-bin, Identification of causal effects using instrumen-tal variables, Journal of the American Statistical As-sociation, 91 (1996), pp. 444–455.

[2] D. M. Chickering and C. Meek, Finding optimalBayesian networks, in UAI, 2002, pp. 94–102.

[3] A. Gallagher and T. Chen, Estimating age,gender and identity using first name priors, inCVPR, 2008.

[4] D. Garant and D. Jensen, Evaluating causalmodels by comparing interventional distributions, inSIGKDD Workshop on Causal Discovery, 2016.

[5] K. Hirano, G. Imbens, and G. Ridder, Efficientestimation of average treatment effects using theestimated propensity score, Econometrica, 71 (2003),pp. 1161–1189.

[6] D. Koller and N. Friedman, Probabilistic Graph-ical Models: Principles and Techniques, The MITPress, 2009.

[7] G. Kowalik and R. Nielek, Senior programmers:Characteristics of elderly users from Stack Overflow,in SOCINFO, 2016, pp. 87–96.

[8] M. Kuroki and J. Pearl, Measurement bias andeffect restoration in causal inference, Biometrika,101 (2014), pp. 423–437.

[9] M. W. Lipsey and D. B. Wilson, Practical Meta-Analysis., Sage Publications, Inc, 2001.

[10] W. Liu, S. J. Kuramoto, and E. A. Stuart,An introduction to sensitivity analysis for unob-served confounding in non-experimental preventionresearch, Prevention Science, 14 (2013).

[11] M. Long, J. Wang, G. Ding, D. Shen,and Q. Yang, Transfer learning with graph co-regularization, in AAAI, 2012.

[12] S. Magliacane, T. van Ommen, T. Claassen,S. Bongers, P. Versteeg, and J. M. Mooij,Causal transfer learning, CoRR, (2017).

[13] D. Nguyen, R. Gravel, D. Trieschnigg, andT. Meder, “How old do you think I am?”: A studyof language and age in Twitter, in ICWSM, 2013.

[14] H. Oktay, Z. Ertem, and A. Firat, Demographicbreakdown of Twitter users: An analysis based onnames, in ASE SocialCom, 2014.

[15] S. J. Pan and Q. Yang, A survey on transferlearning, IEEE Transactions on Knowledge andData Engineering, 22 (2010), pp. 1345–1359.

[16] J. Pearl, Probabilistic Reasoning in IntelligentSystems: Networks of Plausible Inference, MorganKaufmann Publishers Inc., 1988.

[17] , Causality: Models, Reasoning and Inference,Cambridge University Press, 2nd ed., 2009.

[18] M. A. Pourhoseingholi, A. R. Baghestani, andM. Vahedi, How to control confounding effects bystatistical analysis, Gastroenterology and Hepatol-ogy From Bed to Bench, 5 (2012), pp. 79–83.

[19] P. R. Rosenbaum and D. B. Rubin, The centralrole of the propensity score in observational studiesfor causal effects, Biometrika, 70 (1983), pp. 41–55.

[20] P. Spirtes, C. Glymour, and R. Scheines, Cau-sation, Prediction, and Search, The MIT Press,2nd ed., 2000.

[21] Stack Overflow, Developer survey results,2017. https://insights.stackoverflow.com/

survey/2017, Last accessed on 2019-01-23.[22] E. A. Stuart, Matching methods for causal infer-

ence: A review and a look forward, Statistical Sci-ence, 25 (2010), p. 1.

[23] S. Triantafillou, I. Tsamardinos, and I. Tol-lis, Learning causal structure from overlapping vari-able sets, in AISTATS, 2010, pp. 860–867.

[24] C. Winship and M. Sobel, Causal Inference inSociological Studies, Sage Publications, 2004.

[25] Q. Yang, V. W. Zheng, B. Li, and H. H. Zhuo,Transfer learning by reusing structured knowledge.,AI Magazine, 32 (2011), pp. 95–106.

[26] A. Zagorecki and M. J. Druzdzel, Knowl-edge engineering for Bayesian networks: How com-mon are noisy-max distributions in practice?, IEEETransactions on Systems, Man, and Cybernetics:Systems, 43 (2013), pp. 186–195.

[27] F. A. Zamal, W. Liu, and D. Ruths, Homophilyand latent attribute inference: Inferring latent at-tributes of Twitter users from neighbors, in ICWSM,2012.


Documents

Identifying When E ect Restoration Will Improve Estimates ... · Huseyin Oktay Akanksha Atrey David Jensen Abstract Several methods have been developed that combine mul-tiple models