DATA-DRIVEN COMPUTATIONAL STATISTICS AND STOCHASTIC TECHNIQUES FOR THE ROBUST DESIGN OF CONTINUUM SYSTEMS Materials Process Design and Control Laboratory

DATA-DRIVEN COMPUTATIONAL DATA-DRIVEN COMPUTATIONAL STATISTICS AND STOCHASTIC STATISTICS AND STOCHASTIC

TECHNIQUES FOR THE ROBUST DESIGN TECHNIQUES FOR THE ROBUST DESIGN OF CONTINUUM SYSTEMSOF CONTINUUM SYSTEMS

Materials Process Design and Control LaboratoryMaterials Process Design and Control Laboratory

CCOORRNNEELLLL U N I V E R S I T Y


Nicholas Zabaras

Materials Process Design and Control LaboratorySibley School of Mechanical and Aerospace Engineering

188 Frank H. T. Rhodes HallCornell University

Ithaca, NY 14853-3801

Email: [email protected]: http://www.mae.cornell.edu/zabaras/

Keynote Lecture: Keynote Lecture: Inverse Problems, Design and Optimization SymposiumInverse Problems, Design and Optimization Symposium , Rio de Janeiro, Brazil, March 17-19, 2004, Rio de Janeiro, Brazil, March 17-19, 2004

Event –

Titanic, the worlds largest luxury liner sank on April 14, 1912

Cause – Lack of knowledge about the damaging properties of extra sulphur in steel

Phase transition in steel to a more brittle beta-phase

Inference –

Uncertainty in material properties can lead to catastrophic results




UNCERTAINTY IN MATERIALS – TITANIC PLIGHTUNCERTAINTY IN MATERIALS – TITANIC PLIGHT

COLUMBIA SPACE-SHUTTLE DISASTERCOLUMBIA SPACE-SHUTTLE DISASTER




• Uncertainty in modeling Uncertainty in modeling

• Oblique impact of debris and subsequent collision with Oblique impact of debris and subsequent collision with the shuttlethe shuttle

• Material properties at the operating conditions not well Material properties at the operating conditions not well investigatedinvestigated

RESULT –RESULT –

• CatastropheCatastrophe

UNCERTAINTY IN MATERIALS PROCESSESUNCERTAINTY IN MATERIALS PROCESSES




Modeled as flow in media with variable porosity.

Only a statistical description is possible.

Macroscopically, thus need to have a stochastic

framework for analysis

Meso-scale (dendritic structures seen)

Structure of dendrites affect macroscopic quantity like porosity

Dendritic structure is a strong function of initial process conditions

Small perturbation in initial material concentrations, temperature, flow profile can significantly alter the dendritic profiles

“ Can we employ a multiscale stochastic formulation to model initial uncertainty and provide a statistical characterization for porosity”?

Typical dendritic structures obtained due to small perturbations in

initial conditions

Approaches:

Probabilistic approaches

--- e.g. Spectral stochastic methods

Statistical approaches

I) Frequentist approach

II) Bayesian approach




Stochastic optimization and uncertainty quantification --- importance and requirements

process uncertainty

Driving force

DAQ uncertainty

initial uncertainty

Result (Y) – to - cause (θ) ?

or performancerequirements

Mathematical representation

Y = F(θ, ωi, ωs) + ωm

process uncertainty

Why stochastic optimization & uncertainty quantification ?• need for sensitivity of inverse solutions to the system uncertainties• robustness, reliability and system performance requirements• uncertainties are unavoidable and may change the dynamics of the system• need complete probabilistic description of uncertainties and design solutions

What needs to be done?• probabilistic modeling of uncertainties• uncertainty propagation (forward sensitivity)• techniques for robust design• stochastic optimization techniques• probabilistic description of inverse solution• higher order statistics computation• prediction under uncertainties

ASPECTS OF DESIGN UNDER UNCERTAINTYASPECTS OF DESIGN UNDER UNCERTAINTY




Are PDFs of other unspecified

process conditions (due to interactions)

feasible?

Required product with desired material properties and shape with specified confidence (output PDFs)

Can we obtain the

PDFs by existent testing?

Update model PDFs and database (digital library)

Reference material data and

process conditions PDFs

Are PDFs of design variables

technically feasible?

Digital library for accessing previous simulation, experimental data

Information about uncertainly levels in materials processes

High performance computing

environment for “robust design”

PDFs of design variables and other process conditions

Yes

Yes

Yes

Interface with digital library and expert advice to modify design objectives, material models, process models

No

No

No

Digital library




PRESENTATION ORGANIZATION

• A spectral stochastic approach to the A spectral stochastic approach to the analysis and robust design of continuum analysis and robust design of continuum systems (work with Velamur A. Badri systems (work with Velamur A. Badri Narayanan)Narayanan)

• A Bayesian inference approach to A Bayesian inference approach to inverse/design problems in continuum inverse/design problems in continuum systems (work with Jingbo Wang)systems (work with Jingbo Wang)

• Conclusions and discussionsConclusions and discussions

Part APart A

A Spectral Stochastic Approach to the A Spectral Stochastic Approach to the Analysis and Robust Design of Analysis and Robust Design of

Continuum SystemsContinuum Systems




UNCERTAINTY QUANTIFICATION - PRIMERUNCERTAINTY QUANTIFICATION - PRIMER




• Probabilistic characterization of uncertaintyProbabilistic characterization of uncertainty

Models: Black box or Models: Black box or PDE-basedPDE-based

Model uncertaintyModel uncertainty

• Uncertainty in constitutive Uncertainty in constitutive modelsmodels

• Uncertainty in governing Uncertainty in governing equation parameters, equation parameters, process conditionsprocess conditions

Characterization by joint Characterization by joint probability distribution functionsprobability distribution functions

• For each uncertainty input, a For each uncertainty input, a probability distribution is probability distribution is attributedattributed

• Statistical description of Statistical description of uncertaintyuncertainty

• Suitable for most engineering Suitable for most engineering systemssystems

Uncertain quantities modeled as Uncertain quantities modeled as random variablesrandom variables and and stochastic stochastic processesprocesses

RANDOM VARIABLES - MOTIVATIONRANDOM VARIABLES - MOTIVATION




Motivating example – Tossing of a pair of dice (experiment)Motivating example – Tossing of a pair of dice (experiment)

• Sample space Sample space = set of all possible outcomes = all integers i and j such that = set of all possible outcomes = all integers i and j such that

• Define a function X = i + jDefine a function X = i + j

• X takes values inside the set X takes values inside the set BB = {2,…,12} = {2,…,12}

• Further for each value X takes we can assign a probabilityFurther for each value X takes we can assign a probability

}6,1:),{( jiji

Definition : X in the above example is an example of a real valued random variable. It is a Definition : X in the above example is an example of a real valued random variable. It is a function that maps elements in sample space to the real line with an associated probabilityfunction that maps elements in sample space to the real line with an associated probability

• If sample space is continuousIf sample space is continuous

• And the values X takes are also continuous And the values X takes are also continuous

• Then X is a continuous random variable with the following relationThen X is a continuous random variable with the following relation

)(1

)(][AX A

X dyyfdPAXP

X taking X taking values inside values inside an interval Aan interval A

Probability Probability measuremeasure

PDF of XPDF of X

STOCHASTIC PROCESSES - MOTIVATIONSTOCHASTIC PROCESSES - MOTIVATION




• Definition – Probability spaceDefinition – Probability space

The sample space The sample space , the collection of all possible events in a sample space , the collection of all possible events in a sample space FF and the and the probability law probability law P P that assigns some probability to all such combinations constitute a that assigns some probability to all such combinations constitute a probability space (probability space (, , FF, , P P ))

• Some factsSome facts

A real value random variable maps the probability space to the real lineA real value random variable maps the probability space to the real line

• Stochastic processStochastic process

What if we have a system property with spatial uncertainty?What if we have a system property with spatial uncertainty?

Porous mediumPorous medium

• Random porosityRandom porosity

• Each point in space Each point in space corresponds to a random corresponds to a random porosity valueporosity value

• Porosity of a point is very close Porosity of a point is very close to the porosity of nearby point – to the porosity of nearby point – notion of correlationnotion of correlation

RANDOM VARIABLES – EXPECTATION, NORMSRANDOM VARIABLES – EXPECTATION, NORMS




• The statistical average of a function of a random variable is defined asThe statistical average of a function of a random variable is defined as

A

X dyyfygXgE )()()]([

Provided the integral exists. ‘Provided the integral exists. ‘AA’ is the subset of real line where the probability density function ’ is the subset of real line where the probability density function ffXX(y) (y) is is

positive. positive.

• Few important statistical definitionsFew important statistical definitions

• For a stochastic process For a stochastic process W(x,t,W(x,t,) )

Covariance kernelCovariance kernel

LL22 norm norm

)],','(),,([)',',,( txWtxWtxtx

Convergence in LConvergence in L22 (convergence in mean square sense) forms the backbone of spectral stochastic methods (convergence in mean square sense) forms the backbone of spectral stochastic methods

),()],,([),,(2

1

)(2txydytxWEtxW

TDTDL

DIMENSION REDUCTION - ISSUESDIMENSION REDUCTION - ISSUES




• A majority of uncertain inputs (material data, process conditions) exhibit spatial uncertaintyA majority of uncertain inputs (material data, process conditions) exhibit spatial uncertainty

• Modeling spatial uncertainty with stochastic processes involves representing each point as Modeling spatial uncertainty with stochastic processes involves representing each point as a random variable – Computationally impossiblea random variable – Computationally impossible

• Techniques to reduce the dimensionality (represent stochastic processes with as few Techniques to reduce the dimensionality (represent stochastic processes with as few random variables as possible) ?random variables as possible) ?

Karhunen-Loeve expansionsKarhunen-Loeve expansions

Covariance kernel of the Covariance kernel of the uncertain inputsuncertain inputs

Eigen- decompositionEigen- decomposition

Use first few eigen modes Use first few eigen modes to represent the to represent the

stochastic processstochastic process

These eigen modes are These eigen modes are enough to capture around enough to capture around

90% of the covariance 90% of the covariance kernelkernel

(Visualize as 90% of (Visualize as 90% of energy in the system energy in the system since covariance is a since covariance is a

measure of kinetic measure of kinetic energy)energy)

KARHUNEN-LOEVE – BRIEF EQUATIONSKARHUNEN-LOEVE – BRIEF EQUATIONS




n

iiii txtxWEtxW(n)

1

)(),()],,([),,(

Random Random variablesvariables

Stochastic Stochastic processprocess

MeanMean

Approximate the stochastic process as a summation involving independent random variablesApproximate the stochastic process as a summation involving independent random variables

Eigen-decomposition of covariance kernelEigen-decomposition of covariance kernel

)','(');,(')','()',',,( txytxdytxtxtx ii

TD

i

Convergent in mean square senseConvergent in mean square sense

)(),,(),,()(2

NntxWtx(n)WTDL

Most optimal of all possible lower dimensional representations of a stochastic processMost optimal of all possible lower dimensional representations of a stochastic process

KARHUNEN-LOEVE – SHORTCOMINGSKARHUNEN-LOEVE – SHORTCOMINGS




• Needs apriori knowledge of Covariance KernelNeeds apriori knowledge of Covariance Kernel

- Available for uncertain inputs- Available for uncertain inputs

- Available for certain model for uncertainties (constitutive relations)- Available for certain model for uncertainties (constitutive relations)

• Covariance kernels of output response of a physical system not availableCovariance kernels of output response of a physical system not available

• Covariance kernels of certain non-Gaussian inputs highly complicatedCovariance kernels of certain non-Gaussian inputs highly complicated

Can we construct uncertainty representation Can we construct uncertainty representation schemes for output and other non-Gaussian inputs schemes for output and other non-Gaussian inputs

similar to the Karhunen-Loeve expansion?similar to the Karhunen-Loeve expansion?

n

iii

n txWtxW0

)( )(),(~

),,(

Generalized Polynomial Chaos expansionGeneralized Polynomial Chaos expansion

Approximation Approximation of stochastic of stochastic

processprocess

Chaos Chaos polynomialspolynomials

(random (random variables)variables)

POLYNOMIAL CHAOS – GAUSSIAN INPUTSPOLYNOMIAL CHAOS – GAUSSIAN INPUTS




Inputs are Gaussian random variablesInputs are Gaussian random variables

• Best representation of outputs is obtained by choosing the chaos polynomials from an Best representation of outputs is obtained by choosing the chaos polynomials from an

Hermite-chaos familyHermite-chaos family

Hermite chaosHermite chaos

First few polynomialsFirst few polynomials

n

ii

n

nn

nnH1

2

11

2

1exp)1(),,(

Related one-to-one with Related one-to-one with ii(())

1

1

212

11

0

One uncertain inputOne uncertain input

POLYNOMIAL CHAOS – NON-GAUSSIAN INPUTSPOLYNOMIAL CHAOS – NON-GAUSSIAN INPUTS




• For non-Gaussian inputs, the chaos polynomials are chosen from the Askey series of hypergeometric For non-Gaussian inputs, the chaos polynomials are chosen from the Askey series of hypergeometric

orthogonal polynomialsorthogonal polynomials

• This choice leads to optimal mean square convergenceThis choice leads to optimal mean square convergence

Chaos polynomialChaos polynomial Support spaceSupport space Random variableRandom variable

LegendreLegendre [[]] UniformUniform

JacobiJacobi [[]] BetaBeta

HermiteHermite [-[-∞,∞,∞∞]] Normal, LogNormalNormal, LogNormal

LaguerreLaguerre [0, [0, ∞]∞] GammaGamma

• Combinations of uncertain inputs can use combination of these polynomials for uncertainty Combinations of uncertain inputs can use combination of these polynomials for uncertainty representationrepresentation

• Number of chaos polynomials used to represent output uncertainty depends on Number of chaos polynomials used to represent output uncertainty depends on

- Type of uncertainty in input- Type of uncertainty in input- Distribution of input uncertainty- Distribution of input uncertainty- Number of terms in KLE of input- Number of terms in KLE of input- Degree of uncertainty propagation desired (first order, second order …)- Degree of uncertainty propagation desired (first order, second order …)

USING SPECTRAL STOCHASTIC METHODS IN FEM USING SPECTRAL STOCHASTIC METHODS IN FEM




g

h

DNeumann conditions specified

Temperature specified

TRANSIENT STOCHASTIC HEAT CONDUCTIONTRANSIENT STOCHASTIC HEAT CONDUCTION

)},0{,(),,(),(),0,(

),,(),,(),,(

),,(),,(),,(),,(

),,(),,()(

0

DtxxTxT

TtxtxqnT

k

TtxtxTtxT

DtxTktT

C

h

gg

Diffusion equationDiffusion equation

Dirichlet BCDirichlet BC

Neumann BCNeumann BC

Initial conditionsInitial conditions

ABSTRACT WEAK FORM IN STOCHASTIC SPACES ABSTRACT WEAK FORM IN STOCHASTIC SPACES




• Recall - Basic assumption in any spectral stochastic representation of uncertainty is that of finite varianceRecall - Basic assumption in any spectral stochastic representation of uncertainty is that of finite variance

• Heat equation requires twice differentiable temperature solutions – Strong conditionHeat equation requires twice differentiable temperature solutions – Strong condition

• Strong condition translates to once differentiable temperature field – Weak conditionStrong condition translates to once differentiable temperature field – Weak condition

Definition: Definition: The stochastic function space HThe stochastic function space H11(D) x L(D) x L22((TT) x L) x L22(() is used to denote ) is used to denote

all stochastic processes that have square integrable spatial derivatives, are all stochastic processes that have square integrable spatial derivatives, are square integrable in time and have finite variancesquare integrable in time and have finite variance

D T

dtdPxdtxu

LTLDHtxu

),,(

)()()(),,(

2

22

1

WEAK FORMULATION WEAK FORMULATION




dtdPxdqwwb

dtdPxdwTkwt

TCwTa

where

Ttxtxw

TtxtxTtxT

wbwTa

LTLDHtxw

thatsuchLTLDHtxTFind

h T

D T

g

gg

)(

),(

)(),,(0),,(

)(),,(),,(),,(

)(),(

)()()(),,(

)()()(),,(

221

221

• Trial functions are stochastic and have a finite varianceTrial functions are stochastic and have a finite variance

• QuestionsQuestions- How to choose finite element weighting functions that are stochastic ?- How to choose finite element weighting functions that are stochastic ?- Will the resulting FE representation satisfy the basic assumptions (finite variance) ?- Will the resulting FE representation satisfy the basic assumptions (finite variance) ?

FE FORMULATION DETAILS - INTERPOLATIONFE FORMULATION DETAILS - INTERPOLATION




Spatial interpolationSpatial interpolation

Divide the domain Divide the domain into element into element

(regions)(regions)

TT11

TT22

TT33

TT44

4

1

)(),(),,(i

ii xNtTtxT

Stochastic representationStochastic representation

P

rriri tTtT

0

)()(),(

Nodal unknownsNodal unknowns Chaos polynomialsChaos polynomials

Tensor product of basisTensor product of basis

4

1 0

)()()(),,(i

P

rriir xNtTtxT

• Consider each node to be comprised of (P) unknowns in stochastic caseConsider each node to be comprised of (P) unknowns in stochastic case

FE FORMULATION DETAILS – MATVEC SYSTEMSFE FORMULATION DETAILS – MATVEC SYSTEMS




)(

)(

)(

)()(),,(

)()()()(),,(][

)()()()(),,(][

][][

1

1

1

e

e

e

dPxdxNtxqAf

dPxdxNxNtxkAK

dPxdxNxNtxCAM

fTKTM

r

Nel

em

D

sr

Nel

emn

D

sr

Nel

emn

• Deterministic case – Each node has 1 DOFDeterministic case – Each node has 1 DOF

• Here – Each node has P+1 DOFsHere – Each node has P+1 DOFs

• m = (m = (-1)(P+1)+r, n=(-1)(P+1)+r, n=(-1)(P+1)+s-1)(P+1)+s

• The matrix system is thus (P+1) times larger than a deterministic FEM systemThe matrix system is thus (P+1) times larger than a deterministic FEM system

EXAMPLE – 2D STOCHATIC HEAT CONDUCTIONEXAMPLE – 2D STOCHATIC HEAT CONDUCTION




T0

T0

L

(0,0)

(1,1)

)1.0,1(0 ~)1.0,1(~)( NeLogNormalT Non-Gaussian BCNon-Gaussian BC

Boundary temperature Boundary temperature specified as a random specified as a random variable distributed as variable distributed as LogNormalLogNormal

Thermal conductivityThermal conductivity )1,0(~,5.01~)( 22 Nk

Output representationOutput representation • Two uncertain inputs modeled as random variablesTwo uncertain inputs modeled as random variables

- Two-dimensional KLE for inputs- Two-dimensional KLE for inputs- Two-dimensional third order Hermite-Chaos for output- Two-dimensional third order Hermite-Chaos for output

TEMPERATURE STATISTICSTEMPERATURE STATISTICS




MeanMean Standard deviationStandard deviation

• Mean solution reaches steady state faster than standard deviationMean solution reaches steady state faster than standard deviation

• Uncertainty has a direct effect on the arrival to steady stateUncertainty has a direct effect on the arrival to steady state

• Though input uncertainty variation in temperature is small (10% of mean) output variation in Though input uncertainty variation in temperature is small (10% of mean) output variation in temperature temperature

is about 50% of mean !!is about 50% of mean !!

Known flux specification 0

hD

Unknown flux ?

I

Temperature readings from sensor available with complete statistics along the internal boundary I

STOCHASTIC INVERSE HEAT CONDUCTIONSTOCHASTIC INVERSE HEAT CONDUCTION




),,(),,(),,();,,(

),,(),,(),,(

),,(),,(),,(

)},0{,(),(),(),0,(

),,(),,()(

I0

00

h

in

TtxtxYqtxT

TtxtxqnT

k

TtxtxfnT

k

DxxTxT

TDtxTktT

C

Diffusion equationDiffusion equation

Initial conditionInitial condition

Known flux BCKnown flux BC

Unknown flux BCUnknown flux BC

MeasurementsMeasurements

Definition of the objective function as a norm in an appropriate function space

Define gradient of objective from directional derivative

Sensitivity of temperature with

respect to perturbation in flux

ddd)],,();,,([2

1

),,();,,(2

1)(

)T()()ˆ(

T

20

2

)T(00

02000

2

ttxYqtxT

txYqtxTqJ

LqqJqJ

I

IL

DEFINITION OF STOCHASTIC OBJECTIVE FUNCTIONDEFINITION OF STOCHASTIC OBJECTIVE FUNCTION




)T(00'

)T(0000

02

20

)),((

))],,,();,,(([)()(

L

Lq

qqJ

TDtxYqtxTqJqqJI

)();,,();,,(2

)T(2000000

Lq qTDqtxTqqtxT

Definition of temperature sensitivity

GRADIENT OF OBJECTIVE – ADJOINT FORMULATIONGRADIENT OF OBJECTIVE – ADJOINT FORMULATION




Gradient of the objective function can be evaluated only in an distributional sense

Simplify above using integration by parts to get adjoint equations

• Solution of adjoint problem at the boundary with unknown flux specification is defined as the gradient of the objective function

• Adjoint problem is driven by the temperature residual = difference between estimated temperature and sensor measurements

0 2

2 0

0 0 0 ( T )

'0 0 ( T )

( ) ([ ( , , ; ) ( , , )], )

( ( ), )

Iq L

L

J q q T x t q Y x t D T

J q q

2 2

0

(D T ) (D T )( *( ), ) ( , ( )) 0L L

qD T

L L

Definition of the adjoint variable for evaluating gradient of the objective function

Adjoint definitionAdjoint definition

Temperature SensitivityTemperature Sensitivity

0 0 0( ) ( , , ; ), ( , , ) TJ q x t q x t

CONTINUUM STOCHASTIC SENSITIVITY METHOD- CSSMCONTINUUM STOCHASTIC SENSITIVITY METHOD- CSSM




Definition of stochastic parameter sensitivity –Gateaux differential of the parametric stochastic field w.r.t perturbations in parameters

Observations

• Mathematically rigorous definition of a continuum stochastic sensitivity field

• Deterministic sensitivity information is contained in the direct simulation. Stochastic sensitivity is concerned with perturbations in the PDF

0 0 00 0

0

( , , ; ) ( , , ; )( , , ; , ) lim

T x t q q T x t qx t q q

Temperature Temperature sensitivity sensitivity parameterparameter

Perturbation Perturbation in PDF of in PDF of

fluxflux

CSSM – PHYSICAL INTERPRETATIONCSSM – PHYSICAL INTERPRETATION




Original PDF

Output

PDF obtained after perturbing the

design parameters

How does perturbation in the Joint probability distribution of design parameters affect the solution?

How to characterize perturbations in PDFs?

CSSM – a summary

•An accurate differentiate-then-discretize approach is used here

• The temperature field is design differentiated to obtain the sensitivity field. Since we assume continuous, finite-variance stochastic processes, this approach is valid

• Sensitivities are essentially linear, since we are interested only in infinitesimal perturbations in PDF of design variables

Key questionsKey questions

INVERSE STOCHASTIC HEAT CONDUCTIONINVERSE STOCHASTIC HEAT CONDUCTION




g

0

DBoundary condition unknown


Body under thermal loading

K,C

Unknown material properties – modeled as

stochastic fields

Temperature measurements taken in certain locations in the

body

• Measurement errors

• Environment fluctuations

Given: temperature measurements polluted with uncertainty

Estimate: The unknown boundary condition with uncertainty limits

Constraints: thermal properties of the solid are random

ROBUST STOCHASTIC HEAT CONDUCTIONROBUST STOCHASTIC HEAT CONDUCTION




g

0

DBoundary condition unknown


Body under thermal loading

K,C

Unknown material properties – modeled as

stochastic fields

Desired temperature at specific locations

with tolerable uncertainty limits

Given: desired temperature in parts of the body

Estimate: How well we should know the unknown boundary condition

Constraints: Tolerable uncertainty limits in desired temperature

• Uncertain material properties

Points to measurement and

experimental techniques to obtain

the boundary condition within

desired uncertainty limits

DEFINITION OF SUBPROBLEMSDEFINITION OF SUBPROBLEMS




in

h

0 0

0 0

h

( ) ( , , ) (D T )

( ,0, ) ( , ) ( , ) (D {0} )

( , , ) ( , , ) ( T )

( , , ) ( , , ) ( T )

( ) ( , , ) (D T )

( ,0, ; , ) 0 ( , ) (D {0} )

0 ( , , ) (

TC k T x t

tT x T x x

Tk f x t x t

nT

k q x t x tn

C k x tt

x q q x

k x tn

I

0 0

max max

0 I

T )

( , , ) ( , , ) ( T )

( ) ( , , ) (D T )

( , , ) 0 ( , ) (D { } )

0 ( , , ) ( T )

( , , ; ) ( , , ) ( , , ) ( T )

k q x t x tn

C k x tt

x t x t

k x tnT

k T x t q Y x t x tn

h

0

DGuess flux q0

applied

Known flux applied

Direct Direct problemproblem

h

0

DFlux

perturbation

Insulated Sensitivity Sensitivity problemproblem

h

0

DInsulated

Insulated Adjoint Adjoint problemproblem

I

TRIANGULAR FLUX PROBLEMTRIANGULAR FLUX PROBLEM




Non-Gaussian thermal conductivity

• A chi-square distribution is chosen for thermal conductivity, this ensures that k > 0

Triangular heat flux profile is used as a basic template on which unknown flux probability distributions are built

Non-dimensional time

flux

0 0.25 0.5 0.75 10

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

• FEM solution at x = 0.3 is used to model sensor mean temperature readings

• Measurement error (Gaussian noise) is used to pollute the solution at x=0.3

• The unknown flux is probabilistically reconstructed

• Using the estimated flux predictions are made at x=0.5

tri

2.5 , 0 0.4

2 2.5 , 0.4 0.8

0, 0.8 1

t t

q t t

t

Unknown flux

X = 0

X = L

InsulatedTemperature

sensor readings

2( ) 1 ( ( ) 1), ( ) (0,1)k N Shifted chi-squareShifted chi-square

SENSOR DATA – SOLUTION OF DIRECT SSFEM PROBLEMSENSOR DATA – SOLUTION OF DIRECT SSFEM PROBLEM




• Sensor data at x = 0.3 obtained as a solution of a direct stochastic heat conduction problem using SSFEMSensor data at x = 0.3 obtained as a solution of a direct stochastic heat conduction problem using SSFEM

• No measurement error is added to the solutionNo measurement error is added to the solution

+ + + + + + + + + + + + + ++

++

++

++

++

++

++

++

++

++


Mea

no

ptim

alflu

x

0 0.25 0.5 0.75 10

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

11 iter5 iter20 iter40 iter60 iter80 iterExact

+

+ + + + + + + + + + + + + ++

++

++

++

++

++

++

++

++

++


Fir

sto

rder

term

inP

CE

ofo

ptim

alflu

x

0 0.25 0.5 0.75 10

0.01

0.02

0.03

0.04

0.05

0.06

0.07

0.08

0.09

0.11 iter5 iter20 iter40 iter60 iter80 iterExact

+

+ + + + + + + + + + + + + + + + + ++

++

++

++

++

++

++ +

+


Sec

on

do

rder

term

inP

CE

ofo

ptim

alflu

x

0 0.25 0.5 0.75 1-0.01

-0.008

-0.006

-0.004

-0.002

0

0.002

0.004

0.006

0.008

0.01

0.012

0.014 1 iter5 iter20 iter40 iter60 iter80 iterExact

+

Mean optimal fluxMean optimal flux

• Converges in 10 Converges in 10 iterationsiterations

• Exact mean is Exact mean is reconstructedreconstructed

First order PCE termFirst order PCE term


• captures most of captures most of standard deviation in standard deviation in fluxflux

Second order PCE termSecond order PCE term


• Nearly zero – optimal Nearly zero – optimal flux is also nearly flux is also nearly GaussianGaussian

SENSOR DATA – DETERMINISTIC SOLUTION + NOISESENSOR DATA – DETERMINISTIC SOLUTION + NOISE




• Sensor data at x = 0.3 obtained as a solution of a deterministic heat conduction problemSensor data at x = 0.3 obtained as a solution of a deterministic heat conduction problem

• Gaussian measurement noise is added to the temperature solution to obtain simulated temperature Gaussian measurement noise is added to the temperature solution to obtain simulated temperature readingsreadings

0 0.25 0.5 0.75 10

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

0 0.1 0.2-0.05

-0.025

0

0.025

0.05

0 0.1 0.2-0.05

-0.025

0

0.025

0.05

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Non-dimensional timeM

ean

flux

and

first

PC

Ete

rm0 0.25 0.5 0.75 1

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1 MeanFirst PCE term+

+

+

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++


Mea

nflu

xan

dfir

stP

CE

term

0 0.25 0.5 0.75 1

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1 MeanFirst PCE term+

Mean Mean temperature temperature

readingsreadings

Large noise levelLarge noise level Small noise levelSmall noise level

• Estimation is closely related to the accuracy of temperature readings

• Unlike deterministic inverse problems, large error in measured data does not affect estimation of mean flux

TEMPORALLY CORRELATED MEASUREMENTSTEMPORALLY CORRELATED MEASUREMENTS




• Sensor data at x = 0.3 obtained as a solution of a direct stochastic heat conduction problem using SSFEMSensor data at x = 0.3 obtained as a solution of a direct stochastic heat conduction problem using SSFEM

• Measurement is considered to be a Gaussian process with mean equal to deterministic solutionMeasurement is considered to be a Gaussian process with mean equal to deterministic solution

• The covariance function of the Gaussian process is defined asThe covariance function of the Gaussian process is defined as

1 2 1 2Cov( , ) max 1 / ,0 , 0t t t t b b

• Covariance function is by definition positiveCovariance function is by definition positive

• Individual sensor readings are generated as realizations of the Gaussian processIndividual sensor readings are generated as realizations of the Gaussian process

•The realizations constitute the available dataThe realizations constitute the available data

• The sensor readings are now used to reconstruct the Karhunen-Loeve expansion The sensor readings are now used to reconstruct the Karhunen-Loeve expansion

mean1

( , , ) ( ) ( ) ( )N

i ii

T d t T t T t

Sensor Sensor locationlocation

Mean Mean sensor sensor

readingsreadings

Instrument Instrument standard standard deviationdeviation

I.I.D I.I.D GaussiansGaussians

TEMPORALLY CORRELATED MEASUREMENTS - RESULTSTEMPORALLY CORRELATED MEASUREMENTS - RESULTS




t

q+ tp

0 0.25 0.5 0.75 1-0.1

-0.05

0

0.05

0.1

Order 1Order 2Order 3Order 4

t

q+ tp

0 0.25 0.5 0.75 1

0

0.25

0.5

0.75

1

t

q+ tr

i

0 0.25 0.5 0.75 1

-0.01

0

0.01

0.02

0.03

0.04 Order 1Order 2Order 3Order 4

t

q+ tr

i

0 0.25 0.5 0.75 1

0

0.25

0.5

0.75

1

Higher order PCE termsHigher order PCE termsMean Mean fluxflux

Mean fluxMean flux Higher order PCE termsHigher order PCE terms

tri

0, 0 0.1

2.5 0.25, 0.1 0.5

2.25 2.5 , 0.5 0.9

0, 0.9 1

t

t tq

t t

t

pulse

0, 0 0.1

1.0, 0.1 0.4

0, 0.4 0.6

1.0, 0.6 0.9

0, 0.9 1

t

t

q t

t

t

STOCHASTIC DESIGN – DIFFERENT PROBLEMSSTOCHASTIC DESIGN – DIFFERENT PROBLEMS




High density regions

Low density regions - tails

Reliability type optimization

• Optimization to prevent rare probability events

• Failure implies a catastrophe e.g.- Titanic, Columbia shuttle.

• Requires definition of a failure surface – highly nonlinear

• Optimization in bounded possibly non-convex surfaces

Robust optimization

• Optimization to prevent performance deterioration due to fluctuations in inputs

• Failure implies loss of performance

•Requires definition of a robust objective based on physics of the problem

• Completely stochastic optimization

FINITE Vs INFINITE DIMENSIONAL OPTIMIZATIONFINITE Vs INFINITE DIMENSIONAL OPTIMIZATION




Finite dimensional

• Introduce parametric representations for design variables

• Robust design statement now is a parametric optimization problem in finite dimensional space

• Extent of parameterization governs accuracy

Infinite dimensional

• Design variables are considered as functions and are not parameterized priory

• Leads to a function space optimization problem

• Corresponds to an infinite dimensional parameterization, thus require regularizationApproaches for

robust design of continuum systems

Parameter spaceFinite-dimensional optimization framework can be derived as a

special case.

Part BPart B

A Bayesian Computational Statistics A Bayesian Computational Statistics Approach to Stochastic Optimization and Approach to Stochastic Optimization and Uncertainty Quantification in Continuum Uncertainty Quantification in Continuum

System Design/ControlSystem Design/Control




Fundamentals of Bayesian statistical inferenceFundamentals of Bayesian statistical inference

• Classical statistics and Bayesian statistics Classical: study the probability of a hypothesis in restricted circumstances Bayesian: study the probability of a hypothesis both unconditionally (its prior probability) and given some “evidence” (its posterior probability)

)|( YθP)(

)()|(

YP

θPθYP

• Essence of Bayesian inference - interested in values of unknown quantities θ=[θ1, θ2, … θk]T

- suppose have some ‘a priori belief’ about P(θ) - obtained some data Y=[Y1, Y2, … Yn]T relevant to θ Find out an expression to incorporate both prior beliefs and the data

)|()(1

)|( YppC

Yp

Priori pdf

Likelihood Posterior pdf




• Bayes’ formula

• An example: Y ~ N(θ, σ2), θ ~ N(θo, σo2)

θ|Y ~ N(θ1, σ12)

22

2

22

2

1o

o

oo y

22

2221

o

o




The LikelihoodThe Likelihood

The role of likelihood is to incorporate distribution information from measurementdata

Y = F(θ) + ω

For a typical system as,

the likelihood is determined by the distribution of ω, e.g. whenω ~ N(0, σ2) :

FYθp T

2))((

2

1exp{)|(

Y FY ))}((

Symbolically, the likelihood tries to “filter” the prior belief of θ.

Likelihood is the conditional probability of observation Y given the parameter θ.

It is important to realize likelihood as an interface to data (Bayesian inference is a data driven model)

little data some data more data




The prior distributionThe prior distribution

• role of a prior pdf --- incorporate known to a priori information --- regularize the likelihood• a prior can be a normal pdf or an “improper” pdf• techniques of prior distribution modeling --- accumulated distribution information --- conjugate prior distributions --- physical constraints --- local uniforms --- spatial statistics models

A prior distribution is the unconditional belief of certain unknowns (hypothesis)before the related observations (evidence) are achieved.

A decision should be based on what is known and the evidence --- the priordistribution makes Bayesian the most rational inference approach

An example of conjugate prior:if Y|θ,σ ~ N(θ, σ), then θ ~ N(θo, σo), σ ~ inv-Gamma (a,b).

A class π of prior distributions is said to form a conjugate family if the posterior density p(θ|X) is in the class for all X whenever the prior density is in π.

spatial models are of special importance for stochastic continuum systemse.g. Gauss random fields (GRF), Markov random fields (MRF), …




Various Bayesian formulationsVarious Bayesian formulations

* A hierarchical structure

)()(),|()|,( ppYpYp

)()|(),|()|,( ppYpYp

)()|( pYp )|( p Ym Yo, )|( Yop

usually no knowledge of hyper-parameters, this formulation can diminish theeffect of poor knowledge on hyper-parameters

* An augmented formulation

provides a complete probabilistic description to system uncertainties

* An Expectation-Maximization formulation

more robust formulation (iterative regularization) when there are missing data

Bayesian is an adaptive model --- posteriors can be treated as priors for new data




Spatial statisticsSpatial statistics

A popular model: Markov Random Field (MRF)

• has an exponential form• explores the spatial and temporal dependence• closely related to Tikhonov regularization

}))((exp{)(~

ji jiijWp

Some basic facts:• analyzing spatially indexed data• exploiting the spatial locations of data• studying spatial phenomena existing throughout continuous spatial regions but are only observed at a finite sample of locations (originates from statistical physics)

Advantages for prior distribution in continuum system• regularize the inverse problem by exploiting the spatial dependence --- a physically more rational approach• data are usually available at a few locations• efficient multi-scale modeling

Updating of spatial distribution or exploiting of the posterior distribution requires efficient and accurate numerical tools

All joint distributions can be represented as an MRF, so do MCMC algorithms




Markov Chain Monte Carlo simulation (MCMC)Markov Chain Monte Carlo simulation (MCMC)

What’s attractive: The Metropolis algorithm, an instance of MCMC, has been ranked among the ten algorithms that have had the greatest influence on the development and practice of science and engineering in the 20th century (Beichl &Sullivan, 2000)

MCMC motivationsMCMC motivations::Integration and optimization problems in large dimensional spaces, which Integration and optimization problems in large dimensional spaces, which play a fundamental role in machine learning, physics, statistics…play a fundamental role in machine learning, physics, statistics…* Bayesian inference and learnin* Bayesian inference and learningg (a) (a) NormalizatioNormalization: to obtain the posterior p(n: to obtain the posterior p(x x |y) given the prior p(x) and likelihood |y) given the prior p(x) and likelihood

p(p(y y |x)|x) (b) (b) MarginalizatioMarginalization: given the joint posterior of x and z, to obtain marginal posteriorn: given the joint posterior of x and z, to obtain marginal posterior (c) (c) Conditional expectationConditional expectation* Statistical mechanics* Statistical mechanics* Optimization* Optimization* Penalized likelihood model selection* Penalized likelihood model selection




Markov Chain Monte Carlo simulation (MCMC)Markov Chain Monte Carlo simulation (MCMC)

Monte Carlo PrincipleMonte Carlo Principle1. draw an i.i.d. set of samples {1. draw an i.i.d. set of samples {x x (i)} I=1:(i)} I=1:NN from a target density p(x) from a target density p(x)

2. approximate the target density with the following empirical point-mass function2. approximate the target density with the following empirical point-mass function

3. approximate the integral (expectation) I(3. approximate the integral (expectation) I(ff) with tractable sums I) with tractable sums INN( ( f f ))

N

ixN x

Nxp

i1

)(1

)(

N

iX

NiN dxxpxffIxf

NfI

1

)()()()(1

)(

Sampling strategiesSampling strategies1. Rejection sampling:1. Rejection sampling:

sample from a distribution p(x) by sampling from

another easy-to-sample proposal distribution q(x) when

p(x) <=Mq(x), M is a constant (Robert &Casella, 1999)

2. Importance sampling:2. Importance sampling:

3. MCMC3. MCMC

)()()(ˆ1

iN

i

iN xxwxp

N

i

ii xwxffI1

)()()(ˆ




Markov Chain Monte Carlo simulation (MCMC)Markov Chain Monte Carlo simulation (MCMC)MCMC is a strategy for generating samples MCMC is a strategy for generating samples xxii while exploring the state space X using a Markov while exploring the state space X using a Markov chain mechanism. This mechanism is constructed so that the chain spends more time in chain mechanism. This mechanism is constructed so that the chain spends more time in the most important regions. the most important regions.

Concept of Markov ChainConcept of Markov Chainr.v. r.v. xx є є X ={xX ={x11 x x22 ..., ...,xxss }. The stochastic process }. The stochastic process

xxii is called a Markov chain if is called a Markov chain if p(xp(xii| | xxi-1i-1 ,..., ,..., xx11) = ) = T(xT(xii| | xxi-1i-1). ).

The chain is homogeneous if T remains invariant for The chain is homogeneous if T remains invariant for

all i, with sum of all i, with sum of T(xT(xii| | xxi-1i-1) for all i is one. A chain is ) for all i is one. A chain is

stable if p(x) converges to a invariant distribution in stable if p(x) converges to a invariant distribution in

spite of initial state. A chain is stable as long as the spite of initial state. A chain is stable as long as the

transition (T) satisfies transition (T) satisfies irreducibilitirreducibility and y and aperiodicitaperiodicity.y.

MCMC samplers are irreducible and aperiodic Markov chains that have the

target distribution as the invariant distribution. To design these samplers, the

detailed balance has to be satisfied.

)|()()|()( 111 iiiiii xxTxpxxTxp




Metropolis – Hastings (MH) algorithm --- Metropolis – Hastings (MH) algorithm --- Basic from of MCMCBasic from of MCMC

An MH step of invariant distribution p(x) and proposal distribution q(An MH step of invariant distribution p(x) and proposal distribution q(xx**|x) involves|x) involves

sampling a candidate value sampling a candidate value xx** given the current value given the current value x x according to q(according to q(xx**|x). |x).

Initialize xInitialize x00

For i=0:N-1For i=0:N-1

sample u~U(0,1)sample u~U(0,1)

sample sample xx** ~ ~ q(q(xx**|x|xii) )

if u < A(xif u < A(xii, , xx**)=min)=min{1, p(x{1, p(x**)q(x)q(xii|x|x**)/(p(x)/(p(xii)q(x)q(x**|x|xii))}))}

xxi+1i+1=x=x**

else xelse xi+1i+1=x=xii

Some properties of MH

(a) The normalizing constant of the target distribution is not required.

(b) It is easy to simulate several independent chains in parallel.

(c) The success or failure of the algorithm often hinges on the choice of proposal distribution.




Extensions of MH algorithmExtensions of MH algorithm(a) Independent sampler: q(x*|xi) =q(x*). (b) Metropolis algorithm: q(x*|xi) =q(xi|x*).

(c) Simulated annealing

Initialize x0 and set T0=1For i=0:N-1 sample u~U(0,1) sample x* ~ q(x*|xi)

if u < A(xi, x*)=min{1, p1/Ti(x*)q(xi|x*)/(p1/Ti(xi)q(x*|xi))}

xi+1=x*

else xi+1=xi

set Ti according to cooling design

(d) Mixtures and cycles of MCMC kernels

Initialize x0

For i=0:N-1 sample u~U(0,1) if u<a apply MH with proposal K1 (global proposal) else apply MH with proposal K2 (local proposal)

(e) Cycles of kernels

Initialize x0

For i=0:N-1 - sample the block xi+1

b1 according to proposal distribution q1(xi+1

b1|xi+1-b1, xi

b1) and target distribution p(xi+1

b1|xi+1-b1)

- sample the block xi+1b2 according to proposal

distribution q1(xi+1b2|xi+1

-b2, xib2) and target

distribution p(xi+1b2|xi+1

-b2) . . - sample the block xi+1

bs according to proposal distribution q1(xi+1

bs|xi+1-bs, xi

bs) and target distribution p(xi+1

bs|xi+1-bs)

(f) Gibbs sampler

)|(~ 11

ijj

ij xxpx

Initialize x0

For i = 0:N-1 For j = 1:m sample




Stochastic inverse problemStochastic inverse problem

• Parameterization of unknown quantity q

m

iii txwq

1

),( discrete unknown θ

• System input and output relation

Input qInput q Direct numericalSolver F

Direct numericalSolver F Measurement YMeasurement Y

Y = F(θ,ωs) + ωm

)()|()(

)()|()|( pY pY p

pY pY p

)()(),|()|,( ppYpYp

)()|(),|()|,( ppYpYp

simple form

augmented model

hierarchical model

)()|( pYp )|( p Ym Yo , )|( YopEM model

(missing data)

• Bayesian formulation

• Prior distribution modeling

• provides estimate with associated probability bounds• quantifies uncertainty in noise• provides solution at various accuracy levels• captures modes of nonlinear problems• collects more samples from higher density regions• works for implicit likelihood

• allows for various prior models• accumulated knowledge• exploits spatial relation of unknowns

Advantages of Bayesian approachAdvantages of Bayesian approach




Bayesian formulation and regularization theory

A simple example:

--- Deterministic estimator with zeroth order Tikhonov regularization

--- Bayesian interpretation

MAP estimator

})(21{minˆ 22

2 pLLS YFaug

TWFYYp T2 )

21

exp(}))((2

1exp{)|(

FY ))(( PPDF

}2

])([])([21{minˆ

2 WYFYFaug TT

MAP

LS and MAP estimates are identical taking α=λσ2/2 and W identity matrix

Some key literatures on Bayesian inference of continuum systems



Materials Process Design and Control Laboratory

• C. Ferrero, K. Gallagher, Stochastic thermal history modeling. 1. Constraining heat flow histories

and their uncertainty, Marine and Petroleum Geology 19 (2002) 633-648.

• B. DeVolder, J. Glimm, J.W. Grove, Y. Kang, Y. Lee, K. Pao, D.H. Sharp, K. Ye, Uncertainty

quantification for multiscale simulations, Journal of Fluids Engineering 124 (2002) 29-41.

• D. Higdon, H. Lee, Z. Bi, A Bayesian approach to characterizing uncertainty in inverse problems

using coarse and fine-scale information, IEEE Transactions on Signal Processing 50(2) (2002)

389-399.

• H. K. H. Lee, D. M. Higdon, Z. Bi, M. A.R. Ferreira and M. West, Markov random field models for

high-dimensional parameters in simulations of fluid flow in porous media, Technometrics 44 (3)

(2002).

• A.F. Emery, Stochastic regularization for thermal problems with uncertain parameters, Inverse

Problems in Engineering 9 (2001) 109-125.

• T. D. Fadale, A. V. Nenarokomov, A. F. Emery, Uncertainties in parameter estimation: the inverse

problem, Int. J. Heat Mass Transfer 38(3) (1995) 511-518.

• Anna M. Michalak, Peter K. Kitanidis, A Method for Enforcing Parameter Nonnegativity in

Bayesian Inverse Problems with An Applicaiton to Contaminant Source Identification, {\em Water

Resour. Res.}, 39(2), 1033, doi:10.1029/2002WR001480, 2003.

Γo

Γg

Γh

* ***

****

** unknown heat flux

known temperature

known heat flux

thermocouples

),( TktTC P

in ,

,),( gTtxT ,g

],0[ maxtt

,),(

hqn

txTk

on

,h),()0,( 0 xTxT

,),(

0qn

txTk

)(unknown ,0

],0[ maxtt

on ],0[ maxtt

in

on ],0[ maxtt

Inverse Heat Conduction Problem (IHCP)




Bayesian formulation for IHCP --- A parametric approach




• Parameterization of unknown heat flux q0

m

iii txwq

10 ),( Unknown vector θ

Input θInput θ direct numericalsolver F

direct numericalsolver F

Measurement YMeasurement Y

simulationnoise

numerical error

Y = F(θ) + ω

• System input and output relation

random

• Likelihood function

FYθp T2

))((2

1exp{)|(

Y FY ))}(( --- known σ

--- unknown σ FYθ,p T2

))((2

1(σ2)-n/2exp{σ2)|(

Y FY ))}((

Assumptions• numerical error much less then measurement noise• ω iid ~ N(0, σ2)

Bayesian formulation for IHCP --- A parametric approach




Markov Random Field (MRF)

}))((exp{)(~

jijiijWp

}2

exp{)( 2/ Wp Tm

2

21

)( uu

else

ji

jin

Wi

ij ~

,0

,1

,

Prior distribution modeling

--- Single layer posterior:

--- Augmented posterior:

TWFYYp T2 )

2

1exp(}))((

2

1exp{)|(

FY ))((

)exp(

(σ2)-n/2exp{)|,(

Yp

TW

FY T2

)2

1m/2exp(

}))((2

1

FY ))((

• Gibbs sampler

Y = Hθ + Yi + ω

• Modified Gibbs sampler --- use Gibbs sampler for θ --- take full conditional maximum of σ2

Gibbs sampler and modified Gibbs sampler




1

2i

i ii i

b

a a

2

2 21 1

2N N

si s sii ii i p

s s

H Ha W b

s s st t p ji j ik kt i j i k i

Y H W W

),(~| 2iiii N

)|(~ 11

ijj

ij xxpx

Initialize x0

For i = 0:N-1 For j = 1:m sample

xq

dL

Y (d,iΔt)

--- True q in simulationq

0 0.4 0.8

1.0

--- Normalized governing equation

2

2

xT

tT

1t 0 ,0 1x

0),0( xT 1x 0

0

LxxT

)(0

tqxT

x

,

,

, 1t 0

1t 0,

t1.0

--- Discretization of q(t)

θiθi-1 θi+1

dt

1D IHCP example




--- Case 1, 2, 3 d = 0.3 Δt = 0.02 (n=50) dt = 0.04 (m=26) σ = 0.001, 0.005, 0.010 (2.5% Tmax)

MLE estimate Posterior mean

1D IHCP example …




1D IHCP example …




--- Effect of --- Effect of σσ

True σ 12

0.21 0.31 0.41 0.51 0.610

4

8

q (t = 0.16)

0.80 0.90 1.000

4

8

12

1.10

q (t = 0.40)

0.25 0.35 0.45 0.550

4

8

12

q (t = 0.64)

Marginal PDFs

True 95% probability bounds

1D IHCP example …




Guess of 2σ

Marginal PDFs

0.1 0.2 0.3 0.40.5 0.6 0.70

2

4

6

q (t = 0.16)

0.6 0.7 0.8 0.9 1.0 1.1 1.20

2

4

6

q (t = 0.40)

0.10.20.3 0.40.50.6 0.70

2

4

6

q (t = 0.64)

Gets wider

1D IHCP example …




--- Temperature prediction at d=0.5

Unknown σ

95% probability bounds using augmented model

• Normalized governing equations

,2

2

2

2

fy

T

x

T

t

T

,10 x ,10 y ,0t

,1 xx

TT ,

1 yyTT

,0t

,0

xx

qx

T

,

0

y

y

qy

T

,0t

,0TT t = 0.

• What is the problem

Know f, α, T0, Tx and Ty, reconstruction qx and qy through temperature measurements

2D IHCP example




T(t=0)=T0

T=

0

T=0

• True quantities in simulation

f = 0, α = 1, Tx = 0, Ty = 0, T0 = 2•sin(πx)•sin(πy),

qx = 2•π•sin(πy)•exp(-2π2t), qy = 2•π•sin(πx)•exp(-2π2t), This problem has an analytical solution in the form of T = 2•sin(πx)•sin(πy)•exp(-2π2t)

• Temperature measurements arecollected at 13 sites *

**

*

**

* * * * ** *

2D IHCP example …




true qx

--- Case I ! 13 thermocouples ! 25 measurement steps ! σ = 5.0e-3 (1% ΔTmax) ! α = 5.0e-5

MLE estimate of qx Posterior mean estimate of qx

true qyMLE estimate of qyPosterior mean estimate of qy

Relative error is 28.76%

2D IHCP example …




Relative erroris 4.63%

--- Case I ! 13 thermocouples ! 25 measurement steps ! σ = 5.0e-3 (1% ΔTmax) ! α = 5.0e-5

-1.0 0.0 1.00.0

1.0

2.0

q (0, 0, 0.005) -2.5 -1.5 -0.5

0.0

0.6

1.2

q (0, 0.0625, 0.005) -4.0 -3.0 -2.0

0.0

0.8

1.6

q (0, 0.1875, 0.005)

-5.5 -4.5 -3.50.0

0.8

1.6

q (0, 0.25, 0.005) -6.5 -5.5 -4.5

0.0

0.8

1.6

q (0, 0.375, 0.005)

q (0, 0.125, 0.005) -3.5 -2.5 -1.5

0.0

0.4

0.8

q (0, 0.3125, 0.005) -5.5 -3.5 -1.5

0.0

0.4

0.8

q (0, 0.4375, 0.005) -7.2 -5.7 -3.2

0.0

0.4

0.8

q (0, 0.5, 0.005) -7.5 -5.5 -3.5

0.0

0.4

0.8

q (0, 0.5625, 0.005) -7.5 -5.5 -3.5

0.0

0.4

0.8

-6.5 -5.5 -4.00.0

0.8

1.6

q (0, 0.625, 0.005)

-6.5 -4.5 -2.50.0

0.4

0.8

q (0, 0.6875, 0.005) -5.1 -4.1 -3.1

0.0

0.8

1.6

q (0, 0.75, 0.005) -4.2 -3.2 -2.20.0

0.8

1.6

q (0, 0.8125, 0.005)

-3.2 -2.2 -1.20.0

0.8

1.6

q (0, 0.875, 0.005) -2.5 -1.5 -0.5 0.5

0.0

0.8

1.6

q (0, 0.9375, 0.005) -1.0 0.0 1.0

0.5

1.5

2.5

q (0, 1.0, 0.005)

Marginal PDFs of

qx at t=0.005

2D IHCP example …




true qxMLE estimate of qx Posterior mean estimate of qx




--- Case II ! 13 thermocouples ! 25 measurement steps ! σ = 1.0e-2 (2% ΔTmax) ! α = 2.5e-4

2D IHCP example …




true qxMLE estimate of qx Posterior mean estimate of qx




--- Case III ! 13 thermocouples ! 25 measurement steps ! σ = 2.0e-2 (4% ΔTmax) ! α = 5.0e-4

2D IHCP example …




Inverse Heat Radiation Problem




S(gray

boundary)thermocouple

participatingmedia

Vheat source

*

**

*

*

What g(t) causes measured T?

2 ( ) ( )p r

TC k T g t G x x y y z zq

t

4( ) ( )

4 bs I I I r d Is

4b

b

TI

4

14 ( ( ) )

4br I I r s dq

0

1( ) ( ) 0b n s

I r s I n I r d n ss s

wT T

Bayesian inverse formulation




θi-1

dt

neighbors of θi

t

ghi

θi θi+1

• discretization of g(t)

• Likelihood

• Prior

• Posterior

1

ˆ ( ) ( )m

i ii

g t h t

2 2

1 ( ( )) ( ( ))( ) exp{ }

(2 ) 2

T

n nT T

Y F Y Fp Y

2 1( ) exp( )

2m Tp W

TWFYYp T2 )

2

1exp(}))((

2

1exp{)|(

FY ))((

Direct simulation




A finite element (FE) + S4 method framework

• Ordinate discretization

• Weak formulation of FEM

• algorithm

0

10

j

j ii b j jn s

I I n w I ns s

,

*)*,*,()(

)1(

)()(

WdvTC

Wdvzzyyxxtgqt

dvWTktWdvTC

i

p

rV

i

V

i

pV

24

1

.~

4

~

~~

ijjVbV

iViiV

dvWwIdvWI

dvWIdvWIs

24

1

( ) ( )4

i i i j j bj

I I I r w Is

1. Set T(i)guess = T(I-1);

2. Substitute T(i)guess to compute Ib ;

3. Solve intensity eq for I(i);4. Compute ;5. Solve temperature eq to update T(i)

guess;6. If the solution converged, set T(i)

guess

as T(i) and save I(i); otherwise, go to 2;7. Go to the next time step.

rq

Reduced order modeling --- A POD based approach




• homogeneous part of the direct problem

• inhomogeneous part of the direct problem

• eigenfunction problem

• reduced order models

2 0Ik T

4( ) ( )

4I I I I

bs I I I r d Is

4( )II bb

TI

0

1( ) 0I I

b n sI I n I r d n ss s

IwT T atS

2 ( ) ( )h

hp r

TC k T g t G x x y y z zq

t

4( ) ( )

4h h h I

b bs I I I r d I Is

0

1( ) 0h h

n sI n I r d n ss s

0hT atS

( ) ( )

1

1 eNi i

Vie

U U dvN

1

( ) ( ) ( )TK

h Ti i

i

T t r a t r

1

( ) ( ) ( )IK

h Ii i

i

I t r s b t r s

1

( ) 1TK

jj ji i j j T

i

daM H a S Q g t j K

dt

1 1

1I IK K

ji i ji i j Ii i

A b B b D j K

2( )T

j p jVM C dv

T Tji j iV

H k dv

Tj jrV

S dvq

( )Tj jV

Q G x x y y z z dv 4

{ ( ) }I I I Iji i j i jV

A s d dv

4 4{( ) }I I

ji i jVB d d dv

4

( )I Ij b b jV

D I I d dv

MCMC algorithm --- a cycle design of single component update




• implicit likelihood MH sampler• increasing acceptance probability single component update

Algorithm:

mimic the structure of Gibbs sampler

A cycle of symmetric MH samplers

symmetric MH sampler( ) ( 1) ( ) ( ) ( )

2

1 1( ) exp{ ( )}

22i i i

j j j j j jqjqj

q

},...,,,...,,{ )()(1

)1(1

)1(2

)1(1

)1( im

ij

ij

iiij

Initialize Initialize θθ00

For i=0:N-1For i=0:N-1

For j=1:mFor j=1:m

sample u~U(0,1)sample u~U(0,1)

sample sample θθ**jj ~ ~ qqjj((θθ **

jj | | θθ ii

-j-j , , θθ ii

jj ) )

if u < A(if u < A(θθ iijj , , θθ **

jj ))

θθi+1i+1==θθ**

else else θθi+1i+1==θθii

A testing example




x

z y

1m

1m

1m

g(t)(0.5m, 0.5m, 0.5m)

800K

800K

800K

800K

800K

O

800K

***

12

3

o t

g(t)

400kW/m3

0.05s0.01s 0.04s

80kW/m3

o t

g(t)

0.02s 0.04s 0.05s

160kW/m3

80kW/m3

g1(t)

g2(t)

Schematic of the example

Profile of testing heat sources

Frame 001 09 Mar 2004 Frame 001 09 Mar 2004

Basis fields




Frame 001 09 Mar 2004 Frame 001 09 Mar 2004 Frame 001 09 Mar 2004 Frame 001 09 Mar 2004

Frame 001 09 Mar 2004 Frame 001 09 Mar 2004 Frame 001 09 Mar 2004 Frame 001 09 Mar 2004 Frame 001 09 Mar 2004 Frame 001 09 Mar 2004

Frame 001 09 Mar 2004 Frame 001 09 Mar 2004

Frame 001 09 Mar 2004 Frame 001 09 Mar 2004 Frame 001 09 Mar 2004 Frame 001 09 Mar 2004

1st, 3rd and 6th basis of Th

1st, 3rd and 6th basis of Ih

along direction s =[0.9082483 0.2958759 0.2958759]

1st, 3rd and 6th basis of Ih

along direction s =[-0.9082483 0.2958759 0.2958759]

Homogeneous temperature solution




Th computed by full model Th computed by reduced order model

Comparison of reduced order solutions atthermocouple locations

Heat source reconstruction




MAP estimates of g1 at different magnitude of noise

MAP estimates of g2 at different magnitude of noise

Posterior mean of g1 when σT =0.005

Posterior mean of g2 when σT =0.005




Computational (Bayesian & spatial) statistics approach --- advances and obstacles

Design/control variablesor unknown parameters (g) • thermal conditions (heat flux)• mechanical force• external field force• chemical reactions• processing speed•other mechanisms

Outliers

Get rid of polluted data

Data

min

ing

Uncertainty modeling

Continuum system:

• thermal transfer• material deformation• fluid flow …

Observed data orstate variable requirements

distribution estimate

Statistical formulationof objective:• direct simulation• likelihood modeling• statistical approach• loss function • stochastic regularization• computational considerations

point estimate

Bayesian & spatial

• estimates with probabilities• quantify uncertainty in noise• solution at various accuracy level• complete probabilistic description

Advantages:• data driven in nature• simulation in deterministic space• stochastic regularization (spatial models)• global optimization

Obstacles:• lack of accurate distribution information• computation cost




probabilistic modeling• uncertainty nature• physical constraints• statistical learning• model selection/validation

Prior Learning

Bayesian formulation

Posterior exploration

prior distribution modeling• accumulated information• conjugate family• non-informative priors• physical constraints• spatial statistical models MRF, kringing …

)()|()(

)()|()|( pY pY p

pY pY p

)()(),|()|,( ppYpYp

)()|(),|()|,( ppYpYp

simple form

augmented model

hierarchical model

likelihood computation• system equations• boundary conditions• numerical simulation (FEM, FD, FV, SN, …)• multi-length scale simulation• parallel implementation

asymptotic study• simulated annealing• genetic algorithms

gradient optimization• conjugate gradient• steepest descent gradient sampling strategies

• importance sampling & rejection sampling• MCMC - Metropolis-Hasting - symmetric sampler - hybrid & cycle - reverse jump - sequential MCMC

reduced-order modeling(POD)• Galerkin formulation• eigenvalue problem

)()|( pYp )|( p Ym Yo , )|( YopEM model

(missing data)

system uncertainty propagation study

key uncertainties measurement data mining• outlier detection• instrument illustration• probabilistic modeling•hypothesis testing

• perfect samplingdigital library

Machine learning(SVM)

model selection

A typical Bayesian framework on stochastic inverse problems

Conclusions and discussionConclusions and discussion




ASPECTS OF ANALYSIS UNDER UNCERTAINTYASPECTS OF ANALYSIS UNDER UNCERTAINTY




Statistical information

available based on prior experiments

No prior statistical

information

Assume prior distributions for

input PDFs

Bayesian Analysis

Assume PDFs of input based on

analysis of physics of the problem and

by experience

Input PDFs

Update prior PDFs based on

experimental data

Karhunen-Loeve Karhunen-Loeve decompositiondecomposition

High High performance performance computingcomputing

Digital Digital databasedatabase

User User interfaceinterface

Analysis Analysis toolboxtoolbox

Refine the input Refine the input stochastic modelstochastic model

Stochastic continuum models governed by partial

differential equations

Discrete data based “Black-Box” type input-output

models

SSFEM based analysis Monte Carlo analysis

accelerated by spectral stochastic

expansions

Input resolution enough

to attain output convergence?

ExitExit

ROBUST DESIGN – DATA TO DESIGN MODELSROBUST DESIGN – DATA TO DESIGN MODELS




Stochastic design toolbox

Data from tests and experiments

Filtering data in two-pass Bayesian approach, Spectral stochastic input modeling

Stochastic forward

uncertainty propagation

Highly correlated data

Bayesian, MCMC, Markovian models

Loosely correlated data

Spectral stochastic, VMS, Support space models

Completely data driven model

Bayesian inverse statistical inference,

MCMC, MH algorithms

Analytical design model, robust design objectives

Spectral stochastic design optimization, Continuum

stochastic sensitivity method

Meta models

Partially data driven models with spectral

stochastic design methodologies

Robust design solutions

Optimal material data, process conditions with

PDFs

Testing design under expected operating

conditions

Duty cycle feedback

Bayesian post design inference

Duty cycles, operational feedback

Post design

concurrent updation

Point to optimal testing and data collection

Point to model inadequacies, correct input stochastic PDFs

Design feedback

Documents

DATA-DRIVEN COMPUTATIONAL STATISTICS AND STOCHASTIC TECHNIQUES FOR THE ROBUST DESIGN OF CONTINUUM SYSTEMS Materials Process Design and Control Laboratory