A Definition of Simulation Uncertainty and A View of Total ...Dynamic Experimentation Division UNCERTAINTY QUANTIFICATION WORKING GROUPO-6/28/01-9 Measure theoretic methods Probability

UNCERTAINTY QUANTIFICATIONWORKING GROUP

Mark AndersonExperiment & Diagnostic Design GroupDynamic Experimentation DivisionLos Alamos National Laboratory

June 28, 2001

A Definition ofSimulation Uncertainty

&A View of Total Uncertainty

DXDynamic Experimentation Division UNCERTAINTY QUANTIFICATION WORKING GROUPO-6/28/01-2

PART 1—A DEF’N OF SIMULATION UNCERTAINTY

■ Why do we care?■ What is it?■ How does it apply to our models?■ What technologies are available?■ What technologies are being developed?■ What is the path forward?

Overview


WHY DO WE CARE ABOUT UNCERTAINTY?

■ Science-based stockpile stewardship requires data and models● Test measurements● High-fidelity physics-based models (FEM, etc.)● Low-fidelity physics-based models (SDOF, etc.)● Surrogate models

■ Decisions will be based on our model predictions● Safety● Security● Economic● Military

■ Accuracy and robustness is crucial to acceptance■ Accuracy/robustness ⇒ quantified uncertainties


WHAT IS UNCERTAINTY?

■ Aleatoric uncertainty (also called Variability)● Inherent variation● Irreducible

■ Epistemic uncertainty (also called simply Uncertainty)● Potential deficiency● Lack of knowledge● Reducible?

■ Prejudicial uncertainty (also called Error)● Recognizable deficiency● Bias● Reducible


HOW DOES IT APPLY TO OUR MODELS?

■ Sources of uncertainty● Measurements

– Noise– Resolution,– Quantization– Processing

● Mathematical models– Equations– Geometry– BCs/ICs– Inputs– Deterministic chaos

● Numerical models– Weak formulations– Discretizations– Approximate solution algorithms– Truncation and roundoff

● Surrogate models– Approximation error– Interpolation error– Extrapolation error

● Model parameters

■ Phases of the modeling process

Observation of Nature

Conceptual Modeling

Mathematical Modeling

Numerical Modeling

Numerical Implementation

Numerical Evaluation

Surrogate Modeling

Surrogate Implementation

Surrogate Evaluation

Valid

atio

n &

Unc

erta

inty

Qua

ntifi

catio

n


A SIMPLE EXAMPLE■ Measurement uncertainty

● Forcing function & ICs● Response

■ Math model uncertainty

● Equation form● Forcing function & Ics● Sensitive dependence on ICs

■ Numerical solution uncertainty● Integration algorithm● Time step (discretization)

■ Parameters

“Truth” Model

m

x( )tF

( ) 3xkxf ε+=

( )( ) ( )

00

3

0,0 xxxxtFxxkxm&&

&&

===++ ε

( )( ) ( ) 00

ˆ0,ˆ0

ˆ

xxxxtFkxxm&&

&&

==

=+

km ˆ,ˆ


Total Uncertainty

Parameters

DataModel

Solution

WHAT TECHNOLOGIES ARE AVAILABLE?■ Data

● Calibration w.r.t. conventional standards● Noise characterization● “Similar” or “inverse” signal processing

■ Mathematical models (Not much!)■ Numerical models

● Bounds for discretization errors● Bounds for approximate solution techniques● Bounds for truncation/roundoff errors

■ Surrogate models● DOE● Residual analysis● ANOVA

■ Parameters● Sensitivity analysis● Monte Carlo● Reliability methods (FORM, SORM, AMV, AMV+, FPI)● Fuzzy set & interval propagation methods● Stochastic FEM

■ Total uncertainty


GENERIC VIEW OF TOTAL UNCERTAINTY

■ Generic class of model-test pairs

■ Normalized comparisons

■ Uncertainty propagation

0 0.01 0.02 0.03 0.04−1

0

1

2

3

4

5

6

7

Time (sec. full scale)

Dis

plac

emen

t (in

. ful

l sca

le)

Model Model +/− 1 σTest

101

102

100

101

102

103

104

Frequency (Hz)

Acc

eler

atio

n/F

orce

Node 4 Z−Acceleration / 4 X−Force

ModelModel +/− 2 σTest


■ Measure theoretic methods● Probability theories—frequentist, Bayesian, Koopman-Carnap● Dempster-Schafer theory● Possibility theory

■ Set theoretic methods● Fuzzy set theories—classical, grey, intuitionistic, rough● Interval arithmetic● Convex sets & convex modeling

■ Dynamical systems methods● Strange attractor theory● Liapunov exponents● Complexity theory

WHAT TECHNOLOGIES ARE BEING DEVELOPED?


PROBABILITY THEORY V. DST

■ Probability theory—Based on classical measure theory (additivity)

■ Dempster-Schafer theory—Based on fuzzy measure theory (monotonicity & semicontinuity)

[ ]( )( )

( ) ( ) ( )( ) ( )

( ) ( ) ( )( ) ( )UL

UI

IL

IU

ii

n

kjkj

ii

ii

ii

n

kjkj

ii

ii

X

A

AAAA

A

AAAA

X

Pr1

PrPrPr

Pr1

PrPrPr

1Pr0Pr

1,02:Pr

1

1

+

<

+

<

−++

∑−∑=

−++

∑−∑=

==∅→ [ ] [ ]

( ) ( )( ) ( )

( ) ( ) ( )( ) ( )

( ) ( ) ( )( ) ( )UL

UI

IL

IU

ii

n

kjkj

ii

ii

ii

n

kjkj

ii

ii

XX

A

AAAA

A

AAAA

XX

Pl1

PlPlPl

Bel1

BelBelBel

1Pl1Bel0Pl0Bel

1,02:Pl1,02:Bel

1

1

+

<

+

<

−++

∑−∑≤

−++

∑−∑≥

===∅=∅→→


PROBABILITY THEORY V. POSSIBILITY THEORY

■ Probability theory—Based on classical measure theory (additivity)

■ Possibility theory—Based on fuzzy measure theory (semicontinuity)

[ ]( )( )

( ) ( ) ( )( ) ( )

( ) ( ) ( )( ) ( )UL

UI

IL

IU

ii

n

kjkj

ii

ii

ii

n

kjkj

ii

ii

X

A

AAAA

A

AAAA

X

Pr1

PrPrPr

Pr1

PrPrPr

1Pr0Pr

1,02:Pr

1

1

+

<

+

<

−++

∑−∑=

−++

∑−∑=

==∅→ [ ] [ ]

( ) ( )( ) ( )

( ) ( )

( ) ( )iii

i

iii

XX

AA

A

XX

NecinfNec

APossupPos

1Nec1Pos0Nec0Pos

1,02:Nec1,02:Pos

i

=

=

===∅=∅→→

I

U


POTENTIAL UNCERTAINTY METRICS■ Hartley measure for nonspecificity

■ Generalized Hartley measure for nonspecificity in DST

■ U-uncertainty measure for nonspecificity in possibility theory

■ Shannon entropy for total uncertainty in probability theory

■ Generalized Shannon entropy for total uncertainty in DST

■ Hamming distance for fuzzy sets

( ) AAAAH ofy cardinalit is ,log2=

( ) ( ) [ ] ( ) ( ) 1,0,1,02:,log22

2 =∑=∅→∑=∈∈ XX A

X

AAmmmAAmmN

( ) ( ) ( )∑−=∈Xx

xpxppS 2log

( ) ( ) ( ) { }( ) irrxxrirrrU ii

n

iii ∀≥=∑ −=

+=

+ 12

21 ,Pos,log

( ) ( ) ( ) X

Axx

Xxxxp

ApABelppAUx

2,logmaxBel 2 ∈∀∑≤∑−=∈∈

( ) ( )[ ] ( ) function membership is ,121 xAxAAfXx∑ −−=∈


WHAT IS THE PATH FORWARD?

■ Some type of uncertainty quantification is required■ Salient points

● Measure predictive capability ⇒ Compare data & predictions● Experiments should be designed to facilitate comparisons● “Adequate” quantification of predictive capability ⇒ lots of data● Interpolation/extrapolation beyond observations ⇒ inference

■ No comprehensive framework/toolbox exists● Application dependent● Different types of uncertainty require different tools

■ Hypothetical approach● Characterize measurement uncertainty● Characterize/propagate parametric uncertainty● Bound/propagate solution uncertainty● Estimate (generically) total uncertainty● “Subtract” to estimate model uncertainty


PART 2—A VIEW OF TOTAL UNCERTAINTY

■ What is total uncertainty?■ Prototypical application: linear structural dynamics

● Methodology● Example: Space truss structure

■ Generalization to arbitrary applications● Methodology● Example: Nose cone crushing● Example: Blast response of R/C wall

■ Conclusions

Overview


WHAT IS TOTAL UNCERTAINTY?

■ Total uncertainty is simply a measure of the difference between experimental data and model predictions

■ Practical considerations● There rarely exists enough samples for a given simulation scenario● Simple differencing leads to “small differences of large numbers”

■ A candidate approach● Consider “generic classes” of test-analysis comparisons● Normalize information so that differences are “perturbations”


PROTOTYPICAL APPLICATION: LINEAR DYNAMICS

■ Classical normal modes

■ Modal mass and stiffness matrices

■ Assumed “true” modal mass and stiffness matrices

■ Normalization of test modes

■ Cross-orthogonality of analysis and test modes

■ Differences between analysis and test modes

( )( ) test0

analysis00000

K

K

=−=−

φλφλ

MKMK

λφφφφ

00000

0000

==

==

KkIMm

T

T

kkkkmImmm

∆+=∆+=∆+=∆+=

λ00

0

IMT =φφ 0

ψφψφφφφψφ

==

=00000

0 assumedMM TT

K

( ) ψφψφφφφλλλ

∆=−=−=∆−=∆

000

0

I


PROTOTYPICAL APPLICATION (CONT’D)

■ Structure-specific covariance matrix of uncertainty (biased)

■ Generic covariance matrix of total uncertainty (unbiased)

■ Propagate thru model● Linear covariance propagation● Interval propagation● Monte Carlo simulation

■ Normalized modal metrics for total uncertainty quantification

■ Vectorization of normalized differences

( )( )ζζζ

λλψψλλλ

ψψ

0

21000210~

−=∆∆−∆−∆=∆

∆+∆−=∆−− T

T

k

m

( )( )( )

∆∆∆

=∆ζvec

~vecvec

~ km

r

[ ]( )( )[ ]T

rrrr

r

rrESrE

~~~~

~

~~~

∆∆

∆

−∆−∆=

∆=

µµµ

[ ]Trr

r

rrES ~~assumed0

~~

~

∆∆==

∆Kµ


EXAMPLE: SPACE TRUSS STRUCTURE

● Force input at Nodes 4 and 7● Acceleration response measured at Nodes 1 through 32

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

X

Y

Z

Fixed end

Free end

NASA/LaRC 8-bay truss structure


STRUCTURE-SPECIFIC VS. GENERIC VARIATIONStructure-specific variability

5 4 32

1 12

34

50

0.1

0.2

0.3

0.4

0.5

RMS

Erro

r

Mode No.Mode No.

Modal Mass

5 43

21 1

23

45

0

0.1

0.2

0.3

0.4

0.5

RM

S Er

ror

Mode No.Mode No.

Normalized Modal Siffness

54

32

1 12

34

50

0.1

0.2

0.3

0.4

0.5

RM

S Er

ror

Mode No.Mode No.

Modal Mass

54

32

1 12

34

50

0.1

0.2

0.3

0.4

0.5

RMS

Erro

r

Mode No. Mode No.

Normalized Modal Stiffness

Generic class variability


PREDICTIVE ACCURACY FOR SPACE TRUSS

101

102

100

101

102

103

104

Frequency (Hz)

Acc

eler

atio

n/F

orce


ModelModel +/− 1 σ SpecificModel +/− 1 σ Generic

101

102

100

101

102

103

104

Frequency (Hz)

Acc

eler

atio

n/F

orce

Node 20 X−Acceleration / 4 X−Force


101

102

100

101

102

103

104

Frequency (Hz)

Acc

eler

atio

n/F

orce

Node 4 X−Acceleration / 4 X−Force


101

102

100

101

102

103

104

Frequency (Hz)

Acc

eler

atio

n/F

orce




GENERALIZATION TO ARBITRARY APPLICATIONS

■ Response matrix

■ Singular value decomposition

( ) ( ) ( )( ) ( ) ( )

( ) ( ) ( )

=

nmmm

n

n

txtxtx

txtxtxtxtxtx

X

θθθ

θθθθθθ

;;;

;;;;;;

21

22212

12111

L

MOMM

L

L

} ( )}} ( )}

( )} ( ) ( )}

}

( )}

0

0 0

,

T

p np n pp pm m pm p T

T Tn p nm p p m p n p

T

m mm n n n

T T T Tp

T

X U V

D

D I

X X D

φη η

φ

η φ ηη φ φ

φ η

×× −×× −×

⊥ − ×− × − × −

⊥×

× ×

= Σ

=

= = =

= =

144244314442444314243

[Note: Papers use " " ]


GENERALIZATION (CONT’D)

■ Differences between analysis and test

■ Cross-orthogonality matrices

■ Normalized differences

■ Vectorization of differences

■ Structure-specific covariance matrix of uncertainty (biased)

■ Generic covariance matrix of total uncertainty (unbiased)

0

0

0

D D Dφ φ φ

η η η

∆ = −

∆ = −

∆ = −

( ) ( )00

1Trace

p

p

I

I

D D DD

ψ ψ

ν ν

∆ = −

∆ = −

∆ = −%

0

0

T

T

ψ φ φ

ν ηη

=

=

( )( )( )

∆∆∆

=∆ζvec

~vecvec

~ km

r

[ ]( )( )[ ]T

rrRR

r

rrESrE

~~~~

~

~~~

∆∆

∆

−∆−∆=

∆=

µµµ

[ ]TRR

r

rrES ~~assumed0

~~

~

∆∆==

∆Kµ


GENERALIZATION (CONT’D)

■ Consider data as function of normalized parameters

■ First order Taylor series approximation of data

■ “Propagate” uncertainty

■ Generate uncertainty bands

( ) ( ) ( )0

0, , iur ur ij

j r r

uu T r u u r u r Tr

=

∂∆ ≈ ∆ ∆ = − =

∂% %

% %

% % %%

( ) ( )vecu r X r = % %

( )( )

TUU

T Tur ur

Tur urRR

S E u u

E T r r T

T S T

= ∆ ∆

≈ ∆ ∆

≈

% %

% %% %

% %

( )

( )

110

diag

0u u

UU

U

UU n n

S

S

σ

=

O


EXAMPLE: NOSE CONE CRUSHINGNosecone aeroshell Test setup

��

��

��

��

��

Leading Edge

Impactor

DeformedShape

Aeroshell

Blivet

Simplified, axisymmetric DYNA3D model Actual buckling pattern


PREDICTIVE ACCURACY FOR NOSE CONE

0 0.005 0.01 0.015 0.02 0.025 0.03 0.035−20

−10

0

10

20

30

40

50

Time (s)

Acc

eler

atio

n (g

)

Pre−Update Predictive Accuracy for Unit 0

Nominal Analysis

+/−σ Uncertainty Bounds

Mean Test

0 0.005 0.01 0.015 0.02 0.025 0.03 0.035−20

−10

0

10

20

30

40

50

Time (s)

Acc

eler

atio

n (g

)

Post−Update Predictive Accuracy for Unit 0

Revised Analysis

+/−σ Uncertainty Bounds

Mean Test

Pre-update predictive accuracy Post-update predictive accuracy


EXAMPLE: BLAST RESPONSE OF R/C WALL

■ Scenario● 3-room buried bunker● Explosion in center room

■ Measurements● Displacements (12 sensors, 4 locations)● Pressures (10 sensors, 9 distinct locations)

■ Structure model● DYNA3D (customized LLNL version)● ~80,000 continuum elements● ~20,000 beam elements

■ Load model● CFD code (SHARC?)● Uncoupled from structure● Validated w.r.t. pressure measurements


GENERIC CLASS VARIATIONTypical Singular Values

01020304050607080

1 2 3 4

Mode Number

Sing

ular

Val

ue TestAnalysis

Normalized Singular Values

0

0.5

1

1.5

2

2.5

1 2 3 4

Mode Number

RM

S E

rror

12

34

12

34

0

0.2

0.4

0.6

0.8

1

RMS Error

Mode Number

Mode Number

Right Eigenvector Cross-Orthogonality

12

34

12

34

00.20.40.60.8

11.2

1.4

RMS Error

Mode Number

Mode Number

Left Eigenvector Cross-Orthogonality


PRE-UPDATE PREDICTIVE ACCURACY FOR R/C WALL

0 0.01 0.02 0.03 0.04−1

0

1

2

3

4

5

6

7


Dis

plac

emen

t (in

. ful

l sca

le)


0 0.01 0.02 0.03 0.04−1

0

1

2

3

4

5

6

7


Dis

plac

emen

t (in

. ful

l sca

le)


0 0.01 0.02 0.03 0.04−1

0

1

2

3

4

5

6

7


Dis

plac

emen

t (in

. ful

l sca

le)


0 0.01 0.02 0.03 0.04−1

0

1

2

3

4

5

6

7


Dis

plac

emen

t (in

. ful

l sca

le)


Left Horizontal Quarter Point Upper Vertical Quarter Point

Center Point Lower Vertical Quarter Point


0 0.01 0.02 0.03 0.04−1

0

1

2

3

4

5

6

7


Dis

plac

emen

t (in

. ful

l sca

le)


0 0.01 0.02 0.03 0.04−1

0

1

2

3

4

5

6

7


Dis

plac

emen

t (in

. ful

l sca

le)


0 0.01 0.02 0.03 0.04−1

0

1

2

3

4

5

6

7


Dis

plac

emen

t (in

. ful

l sca

le)


0 0.01 0.02 0.03 0.04−1

0

1

2

3

4

5

6

7


Dis

plac

emen

t (in

. ful

l sca

le)


POST-UPDATE PREDICTIVE ACCURACY FOR R/C WALLLeft Horizontal Quarter Point Upper Vertical Quarter Point

Center Point Lower Vertical Quarter Point


CONCLUSIONS

■ Total uncertainty quantification requires● A generic class of test-analysis pairs● A means of normalizing the differences● A method for propagating uncertainty thru the model

■ Total uncertainty is more realistic than parametric uncertainty

■ Some unresolved questions● How are generic classes defined/interpreted?● What are the statistical issues involved?● Can this approach be made rigorous?● Can total uncertainty be used in conjunction with other types of

uncertainty quantification to deal with the issue of model uncertainty?

Documents

A Definition of Simulation Uncertainty and A View of Total ...Dynamic Experimentation Division UNCERTAINTY QUANTIFICATION WORKING GROUPO-6/28/01-9 Measure theoretic methods Probability