Wrong models and the empirical law of epistemology

Daniel Silk

Theoretical Systems Biology GroupImperial College

27th of March 2013

Outline

Wrong models

Motivation

How to cope with wrong models

The empirical law of epistemology

Group talk - March 2012 Daniel Silk 1 of 13

Sources of error in models

Model inputs

1. Parameter uncertainty

2. Initial condition uncertainty

• Likelihood p(y |X )

• Posterior and predictivedistributions

• Sensitivity analyses• Confidence intervals• Lyapunov exponents

Model structure

1. Incorrect wiring

2. Simplification - leaving the wrongstuff out

• Likelihood p(y |X , M)

• Posterior and predictivedistributions

• ...• ...

Group talk - March 2012 Daniel Silk Wrong models 2 of 13

Sources of error in models

Model inputs

1. Parameter uncertainty

2. Initial condition uncertainty

• Likelihood p(y |X )

• Posterior and predictivedistributions

• Sensitivity analyses• Confidence intervals• Lyapunov exponents

Model structure

1. Incorrect wiring

2. Simplification - leaving the wrongstuff out

• Likelihood p(y |X , M)

• Posterior and predictivedistributions

• ...• ...

Group talk - March 2012 Daniel Silk Wrong models 2 of 13

Experimental design for model selection

T1 T2

y

time , t

T1

T2

P(yT2|M))

P(yT1|M)

yT2

yT1

Space of experimental conditions,

Most informative experiment,

a b

c

dyn-1 yn yn+1

xn-1 xn xn+1f( . |θ ) f( . |θ ) f( . |θ ) f( . |θ )

g( . |θ ) g( . |θ ) g( . |θ )

vn-1 vn vn+1

un-1 un un+1

Group talk - March 2012 Daniel Silk Motivation 3 of 13

Application: crosstalk

Simple regulatory cascades

stim1(t) X1

X3

X2

X4

X5

X6

X7

X8

stim2(t)

?

out(t)

Pathway 1 Pathway 2

Post

erio

r pro

babi

lity

of m

odel

1

Tim

e de

lay

betw

een

stim

uli

Time between 2nd stimulus and measurement1 5 9 13 17

17

13

9

5

1

Group talk - March 2012 Daniel Silk Motivation 4 of 13

Application: JAK STAT

R

IFN_R_JAK

IFN_R_JAK

IFN_R_JAK2

IFN_R_JAKPhos_2 IFN_R_JAKPhos_2_SHP_2

R_JAK

IFN_R_JAKPhos_2_STAT1cPhos IFN_R_JAKPhos_2_STAT1c

JAK

IFN

SHP_2

SHP_2

STAT1cPhos

STAT1c

STAT1cPhos

PPX PPX_STAT1cPhos

STAT1cPhos

STAT1c

STAT1cPhos

STAT1c

STAT1c_STAT1cPhos

PPX PPX_STAT1cPhos_2

STAT1cPhos_2

STAT1c_STAT1cPhos

STAT1cPhos STAT1cPhos_2

STAT1cPhos

STAT1cSTAT1cPhos

Group talk - March 2012 Daniel Silk Motivation 5 of 13

Application: JAK STAT

R_JAK

IFN_R_JAK

IFN_R_JAK2

IFN_R_JAKPhos_2

STAT1cPhos

STAT1cPhos_2

STAT1c

IFN_R_JAKPhos_2_SHP_2

SHP_2

T20 60

Post

erio

r pro

babi

lity

of m

odel

1

0.76

0.51

0.26

0.1

1.0 2.0 3.0 4.0 5.0 6.0 7.0 8.0Measurement time

IFN

stim

ulat

ion

stre

ngth

Post

erio

r pro

babi

lity

of m

odel

1

a b

Group talk - March 2012 Daniel Silk Motivation 5 of 13

Model extensions

Linear ODE models

Base model

Extended model

J =

J =

Perturbation to base model entry (1,1)

Pert

urba

tion

to b

ase

mod

el e

ntry

(2,1

)

True extention chosen

False extension chosen

Inconclusive

Group talk - March 2012 Daniel Silk Motivation 6 of 13

Model averaging

Mi = {fi(X ), pi(X )} is a set of plausible models for predicting Z .

Form a weighted average prediction,

p(Z |D) =∑

i

ωip(Z |Mi , D),

where the weights ωi = p(Mi |D) are calculated as the posteriorprobability for each model Mi .

• Predictions take into account model uncertainty.• Have to do some tweaking to include the possibility that the true

model is not in {Mi }

• Does not give structural insights.

Group talk - March 2012 Daniel Silk How to cope with wrong models 7 of 13

Model discrepancy

Group talk - March 2012 Daniel Silk How to cope with wrong models 8 of 13

Model discrepancy

• Model: Z = f (X ) + δ.• Z is what they want to predict. X is the uncertain inputs to the

model (parameters and initial conditions).• δ is the called the discrepancy.• Idea: Decompose model into a series of sub-functions. Investigate

discrepancies (λi ) at the sub-function level and compare theirrelative importance.

• Do this via a variance based sensitivity analysis -◦ Define p(X , δ) (X and δ assumed independant)◦ Estimate ratios such as

varλi [E(Z |λi)]

var[Z ] to compare importance of sub

discrepancies, or varδ[EX (Z |δ)]varX [Eδ(Z |X)] to compare parameter and structural

uncertainties.

Group talk - March 2012 Daniel Silk How to cope with wrong models 8 of 13

Model discrepancy

Technical report

Group talk - March 2012 Daniel Silk How to cope with wrong models 8 of 13

Model discrepancy

• Accepted practice – evaluate f (X1) − f (X2) to reduce systematicmodel biases.

• Instead consider the discrepancy δ := Z − f (X̂ ), where X̂ give thebest fit.

• Learn p(δ) from a Multi Model Ensemble (MME).• Use p(δ) in evaluating X and Z .• Very strong assumption – second order exchangeability of the

MME• Hand wavy ideas about when this is met.• Need an MME...

Group talk - March 2012 Daniel Silk How to cope with wrong models 8 of 13

Model invalidation

“Model validation is a misnomer”• Prove that a give model cannot explain a set of observations. (e.g.

Anderson & Papachristodoulou, ACC, 2009)• Control theory - can the model predict the stimuli needed to drive

the system through a given trajectory. (Apgar et. al, PLOS CB,2008)

• ...

Group talk - March 2012 Daniel Silk How to cope with wrong models 9 of 13

Model invalidation

Model 1 Model 2

Model 3 Model 4

Parameter 1 Parameter 1

Parameter 1 Parameter 1

Para

met

er 2

Para

met

er 2

Para

met

er 2

Para

met

er 2

Group talk - March 2012 Daniel Silk How to cope with wrong models 10 of 13

Parameter invariability based model invalidation

An idea...• Incorrect models often absorb structural inaccuracies into the

parameter values.• A good model’s parameters should be invariant to experimental

choices.• Can we discard a model whose parameter posterior distribution

changes too much for different data sets?• Is the way it changes informative?• Can we cleverly choose experiments to efficiently invalidate a

model?

Group talk - March 2012 Daniel Silk How to cope with wrong models 11 of 13

The unreasonable effectiveness of mathematics

Group talk - March 2012 Daniel Silk The empirical law of epistemology 12 of 13

Mathematics

“The concepts of mathematics are ... chosen for their amenability to clevermanipulations and to striking, brilliant arguments.”

Group talk - March 2012 Daniel Silk The empirical law of epistemology 12 of 13

Physics

“The physicist is interested in discovering the... ‘laws of nature’. ”

Group talk - March 2012 Daniel Silk The empirical law of epistemology 12 of 13

The empirical law of epistemology

Concepts developed for their mathematical elegance can alsorepresent the laws of nature in a way that give fantasticallyaccurate predictions.• Quantum mechanics

◦ Matrix mechanics◦ Hilbert spaces

• Mathematically simple rule for the second derivative of a particle’sposition under gravity.

• Almost entirely mathematical theories that agree withmeasurements to extreme accuracy (Lamb shift).

“If the empirical law of epistemology were not correct, we would lack theencouragement and reassurance which are emotional necessities withoutwhich the “laws of nature” could not have been successfully explored”

Group talk - March 2012 Daniel Silk The empirical law of epistemology 12 of 13

Summary

• The consequences of structural errors are rarely considered...• ...but can be severe.• It is hard to move through model space. What is a ‘small’ change

to a model?• Model averaging, discrepancies and invalidation may be useful.• We can hope that the empirical law of epistemology holds for

biological systems.• ‘Essentially, all models are wrong, but some are useful...’

Group talk - March 2012 Daniel Silk The empirical law of epistemology 13 of 13