Fitting and Understanding Multilevel Models Andrew Gelman

Eectiveness Ubiquity Way of life

Fitting and understanding multilevel (hierarchical) modelsAndrew Gelman Department of Statistics and Department of Political Science Columbia University 8 December 2004

Andrew Gelman

Fitting and understanding multilevel models


Making more use of existing information

The problem: not enough data to estimate eects with condence The solution: make your studies broader and deeperBroader: extend to other countries, other years, other outcomes, . . . Deeper: inferences for individual states, demographic subgroups, components of outcomes, . . .

The solution: multilevel modelingRegression with coecients grouped into batches No such thing as too many predictors

Andrew Gelman






Andrew Gelman






Andrew Gelman






Andrew Gelman






Andrew Gelman






Andrew Gelman






Andrew Gelman






Andrew Gelman



Fitting and understanding multilevel modelsThe eectiveness of multilevel models Multilevel models in unexpected places Multilevel models as a way of life collaborators:Iain Pardoe, Dept of Decision Sciences, University of Oregon David Park, Dept of Political Science, Washington University Joe Bafumi, Dept of Political Science, Columbia University Boris Shor, Dept of Political Science, Columbia University Noah Kaplan, Dept of Political Science, University of Houston Shouhao Zhao, Dept of Statistics, Columbia University Zaiying Huang, Circulation, New York Times

Andrew Gelman




Andrew Gelman




Andrew Gelman




Andrew Gelman




Andrew Gelman



Outline of talk

The eectiveness of multilevel modelsState-level opinions from national polls (crossed multilevel modeling and poststratication)

Multilevel models in unexpected placesEstimating incumbency advantage and its variation Before-after studies

Multilevel models as a way of lifeBuilding and tting models Displaying and summarizing inferences

Andrew Gelman



Outline of talk




Andrew Gelman



Outline of talk




Andrew Gelman



Outline of talk




Andrew Gelman



Outline of talk




Andrew Gelman



Outline of talk




Andrew Gelman



Outline of talk




Andrew Gelman



Outline of talk




Andrew Gelman



Outline of talk




Andrew Gelman



State-level opinions from national polls Poststratication Validation

National opinion trendsPercentage support for the death penalty 50 60 70 80 1940

1950

1960

1970 Year

1980

1990

2000

Andrew Gelman




State-level opinion trends

Goal: estimating time series within each state One poll at a time: small-area estimation It works! Validated for pre-election polls Combining surveys: model for parallel time series Multilevel modeling + poststratication Poststratication cells: sex ethnicity age education state

Andrew Gelman






Andrew Gelman






Andrew Gelman






Andrew Gelman






Andrew Gelman






Andrew Gelman




Multilevel modeling of opinionsLogistic regression: Pr(yi = 1) = logit1 ((X )i ) X includes demographic and geographic predictors Group-level model for the 16 age education predictors Group-level model for the 50 state predictors Bayesian inference, summarize by posterior simulations of : Simulation 1 75 1 ** ** . . . .. . . . . . . . 1000 ** **

Andrew Gelman





Andrew Gelman





Andrew Gelman





Andrew Gelman





Andrew Gelman





Andrew Gelman




Interlude: why multilevel = hierarchical

Logistic regression: Pr(yi = 1) = logit1 ((X )i ) X includes demographic and geographic predictors Group-level model for the 16 age education predictors Group-level model for the 50 state predictors Crossed (nonnested) structure of age, education, state Several overlapping hierarchies

Andrew Gelman






Andrew Gelman






Andrew Gelman






Andrew Gelman






Andrew Gelman






Andrew Gelman






Andrew Gelman




Poststratication to estimate state opinions

Implied inference for j = logit1 (X ) in each of 3264 cells j (e.g., black female, age 1829, college graduate, Massachusetts) PoststraticationWithin each state s, average over 64 cells: js Nj j js Nj Nj = population in cell j (from Census) 1000 simulation draws propagate to uncertainty for each j

Andrew Gelman






Andrew Gelman






Andrew Gelman






Andrew Gelman






Andrew Gelman






Andrew Gelman




CBS/New York Times pre-election polls from 1988

Validation study: t model on poll data and compare to election results Competing estimates:No pooling: separate estimate within each state Complete pooling: no state predictors Hierarchical model and poststratify

Mean absolute state errors:No pooling: 10.4% Complete pooling: 5.4% Hierarchical model with poststratication: 4.5%

Andrew Gelman







Andrew Gelman







Andrew Gelman







Andrew Gelman







Andrew Gelman







Andrew Gelman







Andrew Gelman







Andrew Gelman







Andrew Gelman







Andrew Gelman




Validation study: comparison of state errors1988 election outcome vs. poll estimateno pooling of state effects Actual election outcome 0.2 0.4 0.6 0.8 1.0 complete pooling (no state effects) Actual election outcome 0.2 0.4 0.6 0.8 1.0 Actual election outcome 0.2 0.4 0.6 0.8 1.0 multilevel model

0.0

0.0

0.0

0.2 0.4 0.6 0.8 1.0 Estimated Bush support

0.0


0.0 0.0


Andrew Gelman



General framework Estimating incumbency advantage and its variation Interactions in before-after studies

Multilevel models always

Anything worth doing is worth doing repeatedly A method is any procedure applied more than once City planningOutward expansion: tting a model to other countries, other years, other outcomes, . . . Inlling: inferences for individual states, demographic subgroups, components of data, . . .

Frequentist statistical theory of repeated inferences

Andrew Gelman







Andrew Gelman







Andrew Gelman







Andrew Gelman







Andrew Gelman







Andrew Gelman




Incumbency advantage in U.S. House elections

Regression approach (Gelman and King, 1990):For any year, compare districts with and without incs running Control for vote in previous election Control for incumbent party vit = 0 + 1 vi,t1 + 2 Pit + Iit + it

Other estimates (sophomore surge, etc.) have selection bias

Andrew Gelman







Andrew Gelman







Andrew Gelman







Andrew Gelman







Andrew Gelman







Andrew Gelman




Estimated incumbency advantage from lagged regressionsEst inc adv from lagged regression 0.0 0.05 0.10 0.15 1900

1920

1940 1960 Year

1980

2000

Andrew Gelman




Can we do better?

Regression estimate: vit = 0 + 1 vi,t1 + 2 Pit + Iit +

it

Political science problem: is assumed to be same in all districts Statistics problem: the model doesnt t the data Well show pictures of the model not tting Well set up a model allowing inc advantage to vary

Andrew Gelman




Can we do better?


it


Andrew Gelman




Can we do better?


it


Andrew Gelman




Can we do better?


it


Andrew Gelman




Can we do better?


it


Andrew Gelman




Model mistUnder the model, parallel lines are tted to the circles (open seats) and dots (incs running for reelection)Democratic vote in 1988 0.2 0.4 0.6 0.8 1.0

o o o o oo oo o o o o oo o o

o oo o o

o

Coefficients for lagged vote 0.2 0.4 0.6 0.8 1.0 1.2

0.0

incumbents running open seats

0.0

0.2 0.4 0.6 0.8 1.0 Democratic vote in 1986

1900

1920

1940 1960 Year

1980

2000

Andrew Gelman




Multilevel model

for t = 1, 2: vit = 0.5 + t + i + it Iit +

it

2 1 is the national vote swing i is the normal vote for district i: mean 0, sd . it is the inc advantage in district i at time t: mean , sd it s are independent errors: mean 0 and sd .

Candidate-level incumbency eects:If the same incumbent is running in years 1 and 2, then i2 i1 Otherwise, i1 and i2 are independent

Andrew Gelman




Multilevel model

for t = 1, 2: vit = 0.5 + t + i + it Iit +

it



Andrew Gelman




Multilevel model

for t = 1, 2: vit = 0.5 + t + i + it Iit +

it



Andrew Gelman




Multilevel model

for t = 1, 2: vit = 0.5 + t + i + it Iit +

it



Andrew Gelman




Multilevel model

for t = 1, 2: vit = 0.5 + t + i + it Iit +

it



Andrew Gelman




Multilevel model

for t = 1, 2: vit = 0.5 + t + i + it Iit +

it



Andrew Gelman




Multilevel model

for t = 1, 2: vit = 0.5 + t + i + it Iit +

it



Andrew Gelman




Multilevel model

for t = 1, 2: vit = 0.5 + t + i + it Iit +

it



Andrew Gelman




Fitting the multilevel modelBayesian inference Linear parameters: national vote swings, district eects, incumbency eects 3 variance parameters: district eects, incumbency eects, residual errors Need to model a selection eect: information provided by the incumbent party at time 1 Solve analytically for Pr(inclusion), include factor in the likelihood Gibbs-Metropolis sampling, program in Splus

Andrew Gelman





Andrew Gelman





Andrew Gelman





Andrew Gelman





Andrew Gelman





Andrew Gelman




Estimated incumbency advantage and its variationAverage Incumbency Advantage 0.12 SD of District Effects 1900 1920 1940 1960 1980 2000 0.08 0.15 0.0 1900 0.05 0.10

0.0

0.04

1920

1940

1960

1980

2000

Year

Year

Residual SD of Election Results 1900 1920 1940 1960 1980 2000

SD of Incumbency Advantage

0.06

0.04

0.02

0.0

0.0 1900

0.02

0.04

0.06

1920

1940

1960

1980

2000

Year

Year

Andrew Gelman




Est inc adv from lagged regression 0.0 0.05 0.10 0.15 1900

Compare old and new estimates

1920

1940 1960 Year

1980

2000

Average Incumbency Advantage

0.0 1900

0.04

0.08

0.12

1920

1940

1960

1980

2000

Year

Andrew Gelman


ntage

0.06



No-interaction modelBefore-after data with treatment and control groups Default model: constant treatment eectsFishers classical null hyp: eect is zero for all cases Regression model: yi = Ti + Xi + i"after" measurement, ytreatment

control

"before" measurement, x

Andrew Gelman





control


Andrew Gelman





control


Andrew Gelman





control


Andrew Gelman




Actual data show interactions

Treatment interacts with before measurement Before-after correlation is higher for controls than for treated units ExamplesAn observational study of legislative redistricting An experiment with pre-test, post-test data Congressional elections with incumbents and open seats

Andrew Gelman






Andrew Gelman






Andrew Gelman






Andrew Gelman






Andrew Gelman






Andrew Gelman




Observational study of legislative redistricting before-after data(favors Democrats)

.. . . .. . ... . . . .. . . .. . .. . . .. .. .. .. . .. . ... . o .. . o . . .. . . . .x .o... .. .. .o . . o . . .x. . o x ... x . .... . . . . . . . .o . x . x . x . . . x .. . ... x . . . . . . . . . .x . .

Estimated partisan bias (adjusted for state) -0.05 0.0 0.05

no redistricting

Dem. redistrict bipartisan redistrict Rep. redistrict

(favors Republicans)

-0.05

0.0

0.05

Estimated partisan bias in previous election

Andrew Gelman




Experiment: correlation between pre-test and post-test data for controls and for treated units1.0 controls

0.8

correlation 0.9

treated 1 2 grade 3 4

Andrew Gelman




Correlation between two successive Congressional elections for incumbents running (controls) and open seats (treated)0.8 incumbents

correlation 0.4 0.6

0.0

0.2

open seats

1900

1920

1940 year

1960

1980

2000

Andrew Gelman




Interactions as variance components

Unit-level error term i For control units, i persists from time 1 to time 2 For treatment units, i changes:Subtractive treatment error (i only at time 1) Additive treatment error (i only at time 2) Replacement treatment error

Under all these models, the before-after correlation is higher for controls than treated units

Andrew Gelman







Andrew Gelman







Andrew Gelman







Andrew Gelman







Andrew Gelman







Andrew Gelman







Andrew Gelman



Building and tting models Displaying and summarizing inferences Conclusions

Some new tools

Building and tting multilevel models Displaying and summarizing inferences

Andrew Gelman




Some new tools


Andrew Gelman




Some new tools


Andrew Gelman




Building and tting multilevel models

A reparameterization can change a model (even if it leaves the likelihood unchanged) Redundant additive parameterization Redundant multiplicative parameterization Weakly-informative prior distribution for group-level variance parameters

Andrew Gelman






Andrew Gelman






Andrew Gelman






Andrew Gelman






Andrew Gelman




Redundant parameterizationage state Data model: Pr(yi = 1) = logit1 0 + age(i) + state(i)

Usual model for the coecients:2 jage N(0, age ), for j = 1, . . . , 4 2 jstate N(0, state ), for j = 1, . . . , 50

Additively redundant model:2 jage N(age , age ), for j = 1, . . . , 4 2 jstate N(state , state ), for j = 1, . . . , 50

Why add the redundant age , state ?Iterative algorithm moves more smoothlyAndrew Gelman Fitting and understanding multilevel models

































Redundant additive parameterizationModelage state Pr(yi = 1) = logit1 0 + age(i) + state(i) 2 jage N(age , age ), for j = 1, . . . , 4 2 jstate N(state , state ), for j = 1, . . . , 50

Identify using centered parameters: jage = jage age , for j = 1, . . . , 4 jstate = jstate state , for j = 1, . . . , 50 Redene the constant term: 0 = 0 + age + age

Andrew Gelman






Andrew Gelman






Andrew Gelman




Redundant multiplicative parameterizationNew modelage state Pr(yi = 1) = logit1 0 + age age(i) + state state(i) 2 jage N(age , age ), for j = 1, . . . , 4 2 jstate N(state , state ), for j = 1, . . . , 50

Identify using centered and scaled parameters: jage = age (jage age ), for j = 1, . . . , 4 jstate = state jstate state , for j = 1, . . . , 50 Faster convergence More general model, connections to factor analysisAndrew Gelman Fitting and understanding multilevel models















Weakly informative prior distribution for the multilevel variance parameterRedundant multiplicative parameterization:age state Pr(yi = 1) = logit1 0 + age age(i) + state state(i) 2 jage N(age , age ), for j = 1, . . . , 4 2 jstate N(state , state ), for j = 1, . . . , 50

Separate prior distributions on the and parameters:Normal on Inverse-gamma on 2

Generalizes and xes problems with the standard choices of prior distributionsAndrew Gelman Fitting and understanding multilevel models























Displaying and summarizing inferences

Displaying parameters in groups rather than as a long list Average predictive eects R 2 and partial pooling factors Analysis of variance

Andrew Gelman






Andrew Gelman






Andrew Gelman






Andrew Gelman






Andrew Gelman




Raw display of inferenceB.0 b.female b.black b.female.black B.age[1] B.age[2] B.age[3] B.age[4] B.edu[1] B.edu[2] B.edu[3] B.edu[4] B.age.edu[1,1] B.age.edu[1,2] B.age.edu[1,3] B.age.edu[1,4] B.age.edu[2,1] B.age.edu[2,2] B.age.edu[2,3] B.age.edu[2,4] B.age.edu[3,1] B.age.edu[3,2] B.age.edu[3,3] B.age.edu[3,4] B.age.edu[4,1] B.age.edu[4,2] B.age.edu[4,3] B.age.edu[4,4] B.state[1] B.state[2] mean 0.402 -0.094 -1.701 -0.143 0.084 -0.072 0.044 -0.057 -0.148 -0.022 0.148 0.023 -0.044 0.059 0.049 0.001 0.066 -0.081 -0.004 -0.042 0.034 0.007 0.033 -0.009 -0.141 -0.014 0.046 0.042 0.201 0.466 sd 2.5% 25% 0.147 0.044 0.326 0.102 -0.283 -0.162 0.305 -2.323 -1.910 0.393 -0.834 -0.383 0.088 -0.053 0.012 0.087 -0.260 -0.121 0.077 -0.105 -0.007 0.096 -0.265 -0.115 0.131 -0.436 -0.241 0.082 -0.182 -0.069 0.112 -0.032 0.065 0.090 -0.170 -0.030 0.133 -0.363 -0.104 0.123 -0.153 -0.011 0.124 -0.146 -0.023 0.116 -0.237 -0.061 0.152 -0.208 -0.008 0.127 -0.407 -0.137 0.102 -0.226 -0.048 0.108 -0.282 -0.100 0.135 -0.215 -0.030 0.102 -0.213 -0.039 0.130 -0.215 -0.029 0.109 -0.236 -0.064 0.190 -0.672 -0.224 0.119 -0.280 -0.059 0.132 -0.192 -0.024 0.142 -0.193 -0.022 0.211 -0.131 0.047 0.252 Andrew Gelman 0.008 0.310 50% 75% 97.5% Rhat n.eff 0.413 0.499 0.652 1.024 110 -0.095 -0.034 0.107 1.001 1000 -1.691 -1.486 -1.152 1.014 500 -0.155 0.104 0.620 1.007 1000 0.075 0.140 0.277 1.062 45 -0.054 -0.006 0.052 1.017 190 0.038 0.095 0.203 1.029 130 -0.052 0.001 0.133 1.076 32 -0.137 -0.044 0.053 1.074 31 -0.021 0.025 0.152 1.028 160 0.142 0.228 0.370 1.049 45 0.015 0.074 0.224 1.061 37 -0.019 0.025 0.170 1.018 1000 0.032 0.118 0.353 1.016 580 0.022 0.104 0.349 1.015 280 0.000 0.052 0.280 1.010 1000 0.032 0.124 0.449 1.022 140 -0.055 0.001 0.094 1.057 120 0.000 0.041 0.215 1.008 940 -0.026 0.014 0.157 1.017 170 0.009 0.091 0.361 1.021 230 0.003 0.052 0.220 1.019 610 0.009 0.076 0.410 1.080 61 -0.005 0.043 0.214 1.024 150 -0.086 -0.003 0.100 1.036 270 -0.008 0.033 0.239 1.017 240 0.019 0.108 0.332 1.010 210 0.016 0.095 0.377 1.015 160 0.172 0.326 0.646 1.003 960 Fitting and 0.603 understanding multilevel models 0.440 1.047 1.001 1000



Raw graphical displayBugs model at "C:/books/multilevel/election88/model4.bug", 3 chains, each with 2001 iterations 80% interval for each chain 4 B.0 b.female b.black b.female.black B.age[1] [2] [3] [4] B.edu[1] [2] [3] [4] * B.age.edu[1,1] [1,2] [1,3] [1,4] [2,1] [2,2] [2,3] [2,4] [3,1] [3,2] * B.state[1] [2] [3] [4] [5] [6] [7] [8] [9] [10] B.region[1] [2] [3] [4] [5] 2 0 2 1q

Rhat 1.5 2+ B.00.6 0.4 0.2 0 0.1 0 0.1 0.2 0.3 1 1.5 2 2.5 0.5 1

medians and 80% intervalsq q

q

q

b.female

q q

q

q q q q

b.black

q q

q q q q

0 b.female.black0.5 0.4 0.2 0 0.2 0.4 0.5 0 0.5 0.5 0 0.5 1 2 0 2 4 1 0.5 0 0.5 1 0.6 0.4 0.2 0 1 0.5 0

q

B.age

q q

q q q q q

q q q q q q q q q q

1 2 3 4q q q

B.edu

q q q

1 2 3 4q q q q q q q q q q q q q q q q

B.age.edu

3 1 2 4 1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q

* B.state

q q q q q q q q q q

1 2 3 4 5 6 7 8 910 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40

B.region

q q

q

q q q

1 2 3 4 5

Sigma.age

q q q

q q q q

Sigma.edu

q q q

Andrew Gelmanq

0.3 Sigma.age.edu0.2 0.1 Fitting and 0

understanding multilevel models

q q q



Better graphical display 1: demographics2.5female black female x black 1829 3044 4564 65+ no h.s. high school some college college grad 1829 x no h.s. 1829 x high school 1829 x some college 1829 x college grad 3044 x no h.s. 3044 x high school 3044 x some college 3044 x college grad 4564 x no h.s. 4564 x high school 4564 x some college 4564 x college grad 65+ x no h.s. 65+ x high school 65+ x some college 65+ x college grad

2

1.5

1

0.5

0

0.5

1

2.5

2

1.5

1

0.5

0

0.5

1

Andrew Gelman




Better graphical display 2: within statesAlaska Pr (support Bush) 0.0 0.4 0.8 Pr (support Bush) 0.0 0.4 0.8 Arizona Pr (support Bush) 0.0 0.4 0.8 Arkansas Pr (support Bush) 0.0 0.4 0.8 California

2.0 1.0 0.0 linear predictor Colorado

2.0 1.0 0.0 linear predictor Connecticut

2.0 1.0 0.0 linear predictor Delaware

2.0 1.0 0.0 linear predictor District of Columbia

Pr (support Bush) 0.0 0.4 0.8



2.0 1.0 0.0 linear predictor





Andrew Gelman




Better graphical display 3: between statesNortheast0.5 0.5 regression intercept regression intercept

Midwest0.5 regression intercept

Southregression intercept TN AL VA MS NC SC OK LA GA KY TXFL AR 0.5

West

UT AZ WY NV AK ID WA CO HI CA MT NM OR 0.0 0.5 0.5

0.0

0.0

NE

0.5

0.5

0.5

0.6

0.7

0.5

0.6

0.7

0.5 0.5

NJ CT VT WV PAME DE RI MA NY MD

NH

MO MI WI IAIL MN

0.0

OH

KS IN ND SD

0.6

0.7

0.6

0.7

R vote in prev elections




Andrew Gelman




Average predictive eectsWhat is E (y | x1 = high) E(y | x1 = low), with all other xs held constant? y

u

vAndrew Gelman Fitting and understanding multilevel models



Average predictive eects

What is E (y | x1 = high) E(y | x1 = low), with all other xs held constant? In general, dierence can depend on x Average over distribution of x in the dataYou cant just use a central value of x

Compute APE for each input variable x Multilevel factors are categorical input variables

Andrew Gelman







Andrew Gelman







Andrew Gelman







Andrew Gelman







Andrew Gelman





What is E (y | x1 = high) E(y | x1 = low), with all other xs held constant? In general, dierence can depend on x Average o

Documents

Fitting and Understanding Multilevel Models Andrew Gelman