Posterior Predictive Checking and Generalized Graphical Modelsgelman/presentations/ggr.pdf · 2009-11-26 · I Simulateyrep fromthepredictivedistribution,p(yrepj ;y) I Comparey tothereplicateddatasetsyrep

1/21

Posterior Predictive Checking and GeneralizedGraphical Models

Andrew Gelman

Dept of Statistics and Dept of Political Science, Columbia University, New York(Visiting Sciences Po, Paris, for 2009–2010)

24 novembre 2009

Andrew Gelman Predictive Checking and Graphical Models

2/21

Generalized graphical models

I Expand graphical modeling to include:

I Predictive model checkingI Fake-data simulationI Scaffolding

I Common features:

I Small changes to an existing fitted modelI Comparisons of nodes between models


2/21


I Expand graphical modeling to include:

I Predictive model checkingI Fake-data simulationI Scaffolding

I Common features:



2/21


I Expand graphical modeling to include:I Predictive model checking

I Fake-data simulationI Scaffolding

I Common features:



2/21


I Expand graphical modeling to include:I Predictive model checkingI Fake-data simulation

I ScaffoldingI Common features:



2/21


I Expand graphical modeling to include:I Predictive model checkingI Fake-data simulationI Scaffolding

I Common features:



2/21



I Common features:



2/21



I Common features:I Small changes to an existing fitted model

I Comparisons of nodes between models


2/21



I Common features:I Small changes to an existing fitted modelI Comparisons of nodes between models


3/21

Goals

I (applied) Building confidence in our computations and ourmodels

I (methodological) Being able to do this routinelyI (theoretical): A unified framework for model building, model

fitting, and model checkingI (computational): Implementing in a Bugs-like language


3/21

Goals





3/21

Goals


I (methodological) Being able to do this routinely

I (theoretical): A unified framework for model building, modelfitting, and model checking

I (computational): Implementing in a Bugs-like language


3/21

Goals



fitting, and model checking

I (computational): Implementing in a Bugs-like language


3/21

Goals





4/21

6 challenges in statistical modeling

I Setting up a realistic (i.e., complicated) modelI Regularization or partial poolingI Fitting the modelI Checking the fit to dataI Confidence buildingI Understanding the fitted model


4/21


I Setting up a realistic (i.e., complicated) model

I Regularization or partial poolingI Fitting the modelI Checking the fit to dataI Confidence buildingI Understanding the fitted model


4/21


I Setting up a realistic (i.e., complicated) modelI Regularization or partial pooling

I Fitting the modelI Checking the fit to dataI Confidence buildingI Understanding the fitted model


4/21


I Setting up a realistic (i.e., complicated) modelI Regularization or partial poolingI Fitting the model

I Checking the fit to dataI Confidence buildingI Understanding the fitted model


4/21


I Setting up a realistic (i.e., complicated) modelI Regularization or partial poolingI Fitting the modelI Checking the fit to data

I Confidence buildingI Understanding the fitted model


4/21


I Setting up a realistic (i.e., complicated) modelI Regularization or partial poolingI Fitting the modelI Checking the fit to dataI Confidence building

I Understanding the fitted model


4/21


I Setting up a realistic (i.e., complicated) modelI Regularization or partial poolingI Fitting the modelI Checking the fit to dataI Confidence buildingI Understanding the fitted model


5/21

The models I’m fitting

I Hierarchical generalized linear models

I yi = α+ βxi + εiI yi = αj[i ] + βj[i ]xi + εi (separate regression in each group)

I

(αjβj

)∼ N

((µα

µβ

),

(σ2α ρσασβ

ρσασβ σ2β

)), for j = 1, . . . , J

I Also can have group-level predictors and nonnested groupingfactors


5/21


I Hierarchical generalized linear modelsI yi = α+ βxi + εi

I yi = αj[i ] + βj[i ]xi + εi (separate regression in each group)

I

(αjβj

)∼ N

((µα

µβ

),

(σ2α ρσασβ

ρσασβ σ2β

)), for j = 1, . . . , J



5/21


I Hierarchical generalized linear modelsI yi = α+ βxi + εiI yi = αj[i ] + βj[i ]xi + εi (separate regression in each group)

I

(αjβj

)∼ N

((µα

µβ

),

(σ2α ρσασβ

ρσασβ σ2β

)), for j = 1, . . . , J



5/21



I

(αjβj

)∼ N

((µα

µβ

),

(σ2α ρσασβ

ρσασβ σ2β

)), for j = 1, . . . , J



5/21



I

(αjβj

)∼ N

((µα

µβ

),

(σ2α ρσασβ

ρσασβ σ2β

)), for j = 1, . . . , J



6/21

Application: public opinion in population subgroups


7/21

Models for deep interactions

I Main effects, 2-way, 3-way, etc.I Example: predicting public opinion given 4 age categories, 5

income categories, 50 statesI 4 + 5 + 50 + 4× 5 + 4× 50 + 5× 50 + 4× 5× 50 parameters

(“effects”)I Also, group-level predictors (linear trends for age and income,

previous voting patterns for states)I Need a richer modeling language

I glmer (y ~ z.age*z.inc*rvote.st + (z.age*z.inc | st) +(z.age*rvote.st | inc) + (z.inc*rvote.st | age) +(z.age | inc*st) + (z.inc | age*st) + (z.st |age*inc) +(1 | age*inc*st), family=binomial(link="logistic"))

I No easy way to write this in Bugs or to program it oneself!


7/21


I Main effects, 2-way, 3-way, etc.

I Example: predicting public opinion given 4 age categories, 5income categories, 50 states

I 4 + 5 + 50 + 4× 5 + 4× 50 + 5× 50 + 4× 5× 50 parameters(“effects”)

I Also, group-level predictors (linear trends for age and income,previous voting patterns for states)

I Need a richer modeling language




7/21



income categories, 50 states

I 4 + 5 + 50 + 4× 5 + 4× 50 + 5× 50 + 4× 5× 50 parameters(“effects”)






7/21




(“effects”)






7/21





previous voting patterns for states)





7/21









7/21









7/21









8/21

Posterior predictive checking: 3 examples

Example 1: a normal distribution is fit to the following data:


9/21

Example 1 of 3: checking a fit to a univariate dataset

20 replicated datasets under the model:


10/21

Example 2: checking a model fit to data with time ordering

> plot (y, type="l")> lines (y.rep)


11/21

Example 3: checking a model with three-way structure

Data and 7 replications:


12/21

Theoretical framework for predictive checking

I Data model: θ → yI Data and replicated data: θ → y , y rep

I Posterior predictive distribution,p(y rep|y) =

∫p(y rep|θ, y)p(θ|y)dθ

I Computation:

I Simulate θ from the posterior distribution, p(θ|y)I Simulate y rep from the predictive distribution, p(y rep|θ, y)I Compare y to the replicated datasets y rep

I The generalized graphical model:M --> theta --> y

\\y.rep


12/21


I Data model: θ → y

I Data and replicated data: θ → y , y rep



I Computation:



\\y.rep


12/21





I Computation:



\\y.rep


12/21





I Computation:



\\y.rep


12/21





I Computation:



\\y.rep


12/21





I Computation:I Simulate θ from the posterior distribution, p(θ|y)

I Simulate y rep from the predictive distribution, p(y rep|θ, y)I Compare y to the replicated datasets y rep


\\y.rep


12/21





I Computation:I Simulate θ from the posterior distribution, p(θ|y)I Simulate y rep from the predictive distribution, p(y rep|θ, y)

I Compare y to the replicated datasets y rep


\\y.rep


12/21





I Computation:I Simulate θ from the posterior distribution, p(θ|y)I Simulate y rep from the predictive distribution, p(y rep|θ, y)I Compare y to the replicated datasets y rep


\\y.rep


12/21





I Computation:I Simulate θ from the posterior distribution, p(θ|y)I Simulate y rep from the predictive distribution, p(y rep|θ, y)I Compare y to the replicated datasets y rep


\\y.rep


13/21

Quelques pensées sur la vérification posterior predictive

I Tous les modèles sont fauxI Nous voudrons trouver les aspects des modèles que ne sont

pas en forme des donnéesI L’objectif, c’est apprendre les lacunes de notre histoire

substantiveI Par example, problèmes du modèle des erreurs, ou des

interactions importantes que nous n’avons encore inclusesI Voilà la connection d’analyse exploratoire des données (EDA)

et la presentation visuelleI Les “p-values” sont les moins importants choses dans la

vérification posterior predictive!


13/21


I Tous les modèles sont faux

I Nous voudrons trouver les aspects des modèles que ne sontpas en forme des données

I L’objectif, c’est apprendre les lacunes de notre histoiresubstantive

I Par example, problèmes du modèle des erreurs, ou desinteractions importantes que nous n’avons encore incluses

I Voilà la connection d’analyse exploratoire des données (EDA)et la presentation visuelle

I Les “p-values” sont les moins importants choses dans lavérification posterior predictive!


13/21



pas en forme des données

I L’objectif, c’est apprendre les lacunes de notre histoiresubstantive





13/21




substantive





13/21





interactions importantes que nous n’avons encore incluses




13/21






et la presentation visuelle



13/21






et la presentation visuelleI Les “p-values” sont les moins importants choses dans la

vérification posterior predictive!


14/21

Checking graphical models through predictive replications

I Quick summary of posterior predictive checking

I Data y , inference from p(θ|y)I Predictive replications from p(y rep|θ)I Compare y to y rep using (visual) test variablesI Graphical structure: y ← θ → y rep

I More general formulation

I Data y , inference from p(θ|X , y)I Predictive replications from p(y rep|X , θ)I Connection to graphical models!


14/21


I Quick summary of posterior predictive checking

I Data y , inference from p(θ|y)I Predictive replications from p(y rep|θ)I Compare y to y rep using (visual) test variablesI Graphical structure: y ← θ → y rep




14/21


I Quick summary of posterior predictive checkingI Data y , inference from p(θ|y)

I Predictive replications from p(y rep|θ)I Compare y to y rep using (visual) test variablesI Graphical structure: y ← θ → y rep




14/21


I Quick summary of posterior predictive checkingI Data y , inference from p(θ|y)I Predictive replications from p(y rep|θ)

I Compare y to y rep using (visual) test variablesI Graphical structure: y ← θ → y rep




14/21


I Quick summary of posterior predictive checkingI Data y , inference from p(θ|y)I Predictive replications from p(y rep|θ)I Compare y to y rep using (visual) test variables

I Graphical structure: y ← θ → y rep




14/21


I Quick summary of posterior predictive checkingI Data y , inference from p(θ|y)I Predictive replications from p(y rep|θ)I Compare y to y rep using (visual) test variablesI Graphical structure: y ← θ → y rep




14/21






14/21



I More general formulationI Data y , inference from p(θ|X , y)

I Predictive replications from p(y rep|X , θ)I Connection to graphical models!


14/21



I More general formulationI Data y , inference from p(θ|X , y)I Predictive replications from p(y rep|X , θ)

I Connection to graphical models!


14/21



I More general formulationI Data y , inference from p(θ|X , y)I Predictive replications from p(y rep|X , θ)I Connection to graphical models!


15/21

Predictive checking and graphical models

I A posterior predictive check requires:

I Set of conditioning variables θI Set of fixed design variables X (e.g., sample size)I Test variable T (y) (more generally, T (X , y , θ)

I Simulating posterior predictive replications is a fundamentaloperation in graphical models

I Requires a new node, y rep, whose distribution is implied by theexisting model


15/21


I A posterior predictive check requires:

I Set of conditioning variables θI Set of fixed design variables X (e.g., sample size)I Test variable T (y) (more generally, T (X , y , θ)




15/21


I A posterior predictive check requires:I Set of conditioning variables θ

I Set of fixed design variables X (e.g., sample size)I Test variable T (y) (more generally, T (X , y , θ)




15/21


I A posterior predictive check requires:I Set of conditioning variables θI Set of fixed design variables X (e.g., sample size)

I Test variable T (y) (more generally, T (X , y , θ)




15/21


I A posterior predictive check requires:I Set of conditioning variables θI Set of fixed design variables X (e.g., sample size)I Test variable T (y) (more generally, T (X , y , θ)




15/21






15/21






16/21

Fake-data debugging

I Models can be debugged by simulating fake data:

I Sample θtrue from the prior distribution p(θ)I Sample y from the model p(y |θ)I Perform Bayesian inference, simulations from p(θ|y)I Check calibration of posterior means, predictive intervals, etc.

compared to θtrue

I General procedure in Cook, Gelman, and Rubin (2007)

I Fake-data simulation is a fundamental operation in graphicalmodels

I θtrue is a new nodeM --> theta.true --> y

\ /\ /

\ /theta


16/21

Fake-data debugging

I Models can be debugged by simulating fake data:

I Sample θtrue from the prior distribution p(θ)I Sample y from the model p(y |θ)I Perform Bayesian inference, simulations from p(θ|y)I Check calibration of posterior means, predictive intervals, etc.

compared to θtrue




\ /\ /

\ /theta


16/21

Fake-data debugging

I Models can be debugged by simulating fake data:I Sample θtrue from the prior distribution p(θ)

I Sample y from the model p(y |θ)I Perform Bayesian inference, simulations from p(θ|y)I Check calibration of posterior means, predictive intervals, etc.

compared to θtrue




\ /\ /

\ /theta


16/21

Fake-data debugging

I Models can be debugged by simulating fake data:I Sample θtrue from the prior distribution p(θ)I Sample y from the model p(y |θ)

I Perform Bayesian inference, simulations from p(θ|y)I Check calibration of posterior means, predictive intervals, etc.

compared to θtrue




\ /\ /

\ /theta


16/21

Fake-data debugging

I Models can be debugged by simulating fake data:I Sample θtrue from the prior distribution p(θ)I Sample y from the model p(y |θ)I Perform Bayesian inference, simulations from p(θ|y)

I Check calibration of posterior means, predictive intervals, etc.compared to θtrue




\ /\ /

\ /theta


16/21

Fake-data debugging

I Models can be debugged by simulating fake data:I Sample θtrue from the prior distribution p(θ)I Sample y from the model p(y |θ)I Perform Bayesian inference, simulations from p(θ|y)I Check calibration of posterior means, predictive intervals, etc.

compared to θtrue




\ /\ /

\ /theta


16/21

Fake-data debugging


compared to θtrue




\ /\ /

\ /theta


16/21

Fake-data debugging


compared to θtrue




\ /\ /

\ /theta


16/21

Fake-data debugging


compared to θtrue




\ /\ /

\ /theta


17/21


I Step 0 (already done): Expressing a statistical model as agraph; Bayesian computation on the graph

I Step 1: Graph of models

I Each model is a node of this super-graphI Two models are connected if they differ by only one feature

(adding/removing a variable, allowing a parameter to vary bygroup, adding/removing a grouping factor, changing aprobability distribution or link function, . . . )

I Step 2: Integrated graph

I Nodes within models are linked within a larger graphI All models coexistI Analogy to computational method of parallel tempering


17/21









17/21









17/21



I Step 1: Graph of modelsI Each model is a node of this super-graph

I Two models are connected if they differ by only one feature(adding/removing a variable, allowing a parameter to vary bygroup, adding/removing a grouping factor, changing aprobability distribution or link function, . . . )




17/21



I Step 1: Graph of modelsI Each model is a node of this super-graphI Two models are connected if they differ by only one feature





17/21








17/21





I Step 2: Integrated graphI Nodes within models are linked within a larger graph

I All models coexistI Analogy to computational method of parallel tempering


17/21





I Step 2: Integrated graphI Nodes within models are linked within a larger graphI All models coexist

I Analogy to computational method of parallel tempering


17/21





I Step 2: Integrated graphI Nodes within models are linked within a larger graphI All models coexistI Analogy to computational method of parallel tempering


18/21

Imagine a cleaned-up Bugs language

I Example:for (i in 1:n){

y[i] ~ dnorm (y.hat[i], tau.y)y.hat[i] <- a[state[i]] + b[state[i]]*x[i]e.y[i] <- y[i] - y.hat[i]

}tau.y <- pow(sigma.y, -2)sigma.y ~ dunif (0, 100)

I We would prefer:y ~ norm (a[state] + b[state]*x, sigma.y)

I Also, instead of y.hat, sigma.y, e.y, we want a more general“operator” notation, for example E(y), sd(y), error(y)


18/21








18/21








18/21








19/21

Automatic posterior predictive checking

I Example in Bugs:for (i in 1:n){

y[i] ~ dnorm (y.hat[i], tau.y)y.rep[i] <- dnorm (y.hat[i], tau.y). . .

I But y rep should be included automaticallyI Implicit graphical structure for model checking: y ← θ → y rep


19/21






19/21




I But y rep should be included automatically

I Implicit graphical structure for model checking: y ← θ → y rep


19/21






20/21

Predictive checking and fake-data debugging

I Model checking or debugging in ideal graphical model software(“DreamBUGS”):

I Set an on/off switch for each node: is it conditioned on oraveraged over?

I Specify a test summary (numerical or graphical) of data andparameters

I Various off-the-shelf test summaries will be available

I Design of data collection is integrated with graphical modeling


20/21








20/21








20/21








20/21








20/21








21/21

Résumé et les directions vers l’avenir

I On peut généraliser les modéles graphiques:M --> theta --> y M --> theta --> y

\ \\ ou \y.rep theta.rep --> y.rep

I Tous les quantitées—θ, y , y rep—existent ensembleI La vérification, c’est possible être plus formal dans la théorie et

dans la computationI Tout est complètement Bayesien—il n’y a jamais le utilisation

double des données!I On peut faire un réseau des modèles pour augmenter notre

comprensionI C’est la unification thèoreticale (de Bayes et de l’enquête)I Et la unification de l’inference et la utilization des modéles

dans les applications


21/21










21/21




I Tous les quantitées—θ, y , y rep—existent ensemble

I La vérification, c’est possible être plus formal dans la théorie etdans la computation

I Tout est complètement Bayesien—il n’y a jamais le utilisationdouble des données!

I On peut faire un réseau des modèles pour augmenter notrecomprension

I C’est la unification thèoreticale (de Bayes et de l’enquête)I Et la unification de l’inference et la utilization des modéles



21/21





dans la computation

I Tout est complètement Bayesien—il n’y a jamais le utilisationdouble des données!





21/21






double des données!





21/21







comprension




21/21







comprensionI C’est la unification thèoreticale (de Bayes et de l’enquête)

I Et la unification de l’inference et la utilization des modélesdans les applications


21/21










Documents

Posterior Predictive Checking and Generalized Graphical Modelsgelman/presentations/ggr.pdf · 2009-11-26 · I Simulateyrep fromthepredictivedistribution,p(yrepj ;y) I Comparey tothereplicateddatasetsyrep