Ralph C. SmithDepartment of Mathematics

North Carolina State University

Support: DOE Consortium for Advanced Simulation of LWR (CASL)

National Science Foundation (NSF)NNSA Consortium for Nonproliferation Enabling Capabilities (CNEC)

Air Force Office of Scientific Research (AFOSR)

A Mutual Information-Based Framework to Use High-Fidelity Codes to Calibrate Low-Fidelity Codes (Hi2Lo)

97 97.5 98 98.5 990

1

2

3

4

5

6

7

8 yIteration 3Iteration 7Iteration 15

Example 1: Pressurized Water Reactors (PWR) -- CASL

Models:• Involve neutron transport, thermal-hydraulics, chemistry.

• Inherently multi-scale, multi-physics.

CRUD Measurements: Consist of low resolution images at limited number of locations.

Example: Pressurized Water Reactors (PWR)Thermo-Hydraulic Equations: Mass, momentum and energy balance for fluid

Hi2Lo Goals: • Use experimental design to guide use of high-fidelity codes and experiments to

“optimally” calibrate low-fidelity models.

• Use high-fidelity codes to improve closure relations for low-fidelity codes

Example: Shearon Harris outside Raleigh

@

@t(↵f⇢f ) +r · (↵f⇢f vf ) = -�

↵f⇢f@vf

@t+ ↵f⇢f vf ·rvf +r · �R

f + ↵fr · �+ ↵frpf

= -F R - F + �(vf - vg)/2 + ↵f⇢f g

@

@t(↵f⇢f ef ) +r · (↵f⇢f ef vf + Th) = (Tg - Tf )H + Tf�f

-Tg(H - ↵gr · h) + h ·rT - �[ef + Tf (s⇤ - sf )]

-pf

✓@↵f

@t+r · (↵f vf ) +

�

⇢f

◆

Example: Pressurized Water Reactors (PWR)Thermo-Hydraulic Equations: Mass, momentum and energy balance for fluid

• Brian Williams (LANL)• Brian Adams, Vince Mousseau, Chris Jones, Natalie Gordon, Lindsay Gilkey,

Laura Swiler, Kathryn Maupin (Sandia) • Bob Salko, Kevin Clarno (ORNL)• Yixing Sung, Emre Tatli (Westinghouse)

Note: Large collaborative effort

Example: Shearon Harris outside Raleigh

@

@t(↵f⇢f ) +r · (↵f⇢f vf ) = -�

↵f⇢f@vf

@t+ ↵f⇢f vf ·rvf +r · �R

f + ↵fr · �+ ↵frpf

= -F R - F + �(vf - vg)/2 + ↵f⇢f g

@

@t(↵f⇢f ef ) +r · (↵f⇢f ef vf + Th) = (Tg - Tf )H + Tf�f

-Tg(H - ↵gr · h) + h ·rT - �[ef + Tf (s⇤ - sf )]

-pf

✓@↵f

@t+r · (↵f vf ) +

�

⇢f

◆

Example 2: Radiation Source Localization in Urban Setting CNEC: Consortium for Nonproliferation Enabling Capabilities

Photofrom[Ştefănescu,etal.,2016],courtesyofJ.Hite.

Kathleen Schmidt, Razvan Stefanescu, Jared Cook: NCSU MathematicsIsaac Michaud: NCSU StatisticsJason Hite, John Mattingly: NCSU Nuclear Engineering

Example 2: Radiation Source Localization in Urban Setting Model: i th detector

Notes: • Simplified Boltzmann transport model.

• Represent source and detectors as points.

• Consider only uncollided flux.

• Ray tracing used to quantify detector responses.

�i = E[Ri(I0, r0)] =�ti✏iAi I0

4⇡|ri - r0|2exp

-NiX

ni=1

�ni sni

!

+ E[Bi�ti ]

10 Detectors

Goal: Use experimental design to guide fixed and moving sensor strategies. Parameters: ✓ = [x0, y0, I0]

Bayesian Calibration: Nuclear Power Plant ApplicationExample: Dittus—Boelter Relation

i.e., [0, 0.046], [0, 1.6], [0,0.8]

Industry Standard: Conservative, uniform, bounds

Bayesian Analysis: Employ conservative bounds as priors

Note: • Substantial reduction in parameter uncertainty

• Quantifies correlation between parameters

Nu = 0.023Re0.8Pr 0.4

2� ⇡ 0.0035 2� ⇡ 0.06 2� ⇡ 0.03

✓1

✓2

✓2

✓3

Prediction Intervals: Nuclear Power Plant Application Strategy: Propagate parameter uncertainties through COBRA-TF to determine

uncertainty in maximum fuel temperature.

Ramifications: • Temperature uncertainty reduced from 40 degrees to 5 degrees.

• Can run plant 20 degrees hotter, which significantly improves efficiency.

• Warrants continued calibration of closure relations.

• Accommodates disparate data sets.

Prediction Intervals: Nuclear Power Plant Application Strategy: Propagate parameter uncertainties through COBRA-TF to determine

uncertainty in maximum fuel temperature.

Ramifications: • Temperature uncertainty reduced from 40 degrees to 5 degrees.

• Can run plant 20 degrees hotter, which significantly improves efficiency.

• Warrants continued calibration of closure relations.

• Accommodates disparate data sets.

High-to-Low Framework:• Objective 1: Use synthetic data generated from validated high-fidelity

codes to calibrate lower-fidelity codes.• Objective 2: Employ high-fidelity codes to improve closure relations and

reduce model discrepancy in low-fidelity codes.• Goal: Using information-theoretic framework, calibrate parameters in

the low-fidelity code using as few high-fidelity simulations as possible.

Scope of UQ: Nuclear Power Plant Analysis

Code Requirements:• CFD simulations must reflect same physics

and design configurations as CTF.

• CFD errors/uncertainties should be quantified through mesh analysis and statistical validation.

• CFD simulations must be performed for designs where code has been verified and validated.

• One must formulate statistical models for QoI with probabilistic or bound-delineated errors.

STAR−CCM+i

+ Calibration ExperimentValidation Experiment

Inputs: θ

Inputs: θ

χ2

χ3

ε iErrorsMeasurement

χ1

δ(χ i ,θ̂ )

S i

Inputs: θ or g(θ)

Low−Fidelity

VerificationHigh−Fidelity

COBRA−TF; e.g., DB

Simulation Codes

ExperimentsPhysical

Thermal−Hydraulic

n

Gaussian Process

Response Surface

Simulation−Based Models

ni S i ε iyi = f(χ i ,θ)+ + + +δ(χ i ,θ̂ )

+

ValidationRegime

+ + +

+

++ +

+

ValidationCalibration

Design of ExperimentsPrediction Intervals

Low−Fidelity Statistical ModelHigh−Fidelity

Physical Models

Scope of UQ: Nuclear Power Plant Analysis

STAR−CCM+i

+ Calibration ExperimentValidation Experiment

Inputs: θ

Inputs: θ

χ2

χ3

ε iErrorsMeasurement

χ1

δ(χ i ,θ̂ )

S i

Inputs: θ or g(θ)

Low−Fidelity

VerificationHigh−Fidelity

COBRA−TF; e.g., DB

Simulation Codes

ExperimentsPhysical

Thermal−Hydraulic

n

Gaussian Process

Response Surface

Simulation−Based Models

ni S i ε iyi = f(χ i ,θ)+ + + +δ(χ i ,θ̂ )

+

ValidationRegime

+ + +

+

++ +

+

ValidationCalibration

Design of ExperimentsPrediction Intervals

Low−Fidelity Statistical ModelHigh−Fidelity

Physical Models

Code Requirements:• Computational budgets dictate that a

limited number of STAR simulations will be available to generate synthetic data to inform or calibrate CTF.

– Necessitates efficient experimental design

Example 1: Motivation and Strategy

Statistical Models:

Notation:

Low-Fidelity Model:

Friction Factor: f = 64/Re

dn = d`(✓, ⇠n) + �(⇠n) + "n(⇠n)

edn = dh(⇠n) + e"n(⇠n)

• ⇠n: Design conditions; e.g., Reynolds numbers

• �(⇠n): Model discrepancy

• ✏n(⇠n): Random observation or discretization errors

f (✓) = ✓1Re✓2

• d`(✓, ⇠n): Low-fidelity model; e.g., friction factor

• dh(⇠n): High-fidelity model; e.g., Hydra

• ✓: Low-fidelity model parameters

High-Fidelity Model: Hydra computing Poiseulle flow

Design Algorithm

Existing data:

Calibrate parameters of low-fidelity model:

Choose new design to reduce uncertainty in

Evaluate high-fidelity model at

Delayed Rejection Adaptive Metropolis (DRAM)

kNN or ANN Estimate of Mutual Information

dn = d`(✓, ⇠n) + �(⇠n) + "n(⇠n)

edn = dh(⇠n) + e"n(⇠n) Dn-1 = {ed1, ed2, · · · , edn-1}

[(⇠1, ed1), (⇠2, ed2), · · · , (⇠n-1, edn-1)]

d`(✓, ⇠n)

⇠n ✓

⇠n : edn = dh(⇠n) + e"n(⇠n)

Delayed Rejection Adaptive Metropolis (DRAM)

1. Determine q0 = arg min

q

NX

i=1

[�i - f (ti , q)]2

2. For k = 1, · · · , M(a) Construct candidate q⇤ ⇠ N(qk-1

, V )

(b) Compute likelihood

SSq⇤ =NX

i=1

[�i - f (ti , q⇤)]2

⇡(�|q) =1

(2⇡�2)n/2

e-SSq/2�2

(c) Accept q⇤with probability dictated by likelihood

Algorithm: [Haario et al., 2006] – MATLAB, Python, QUESO-Dakota

Mutual Information

Note:

Bayesian Framework: Quantifies change in knowledge due to new data

Goal:

Mutual Information: Two random variables I(X;Y)• Measure of variables’ mutual dependence• I(X,Y) quantifies reduction in uncertainty in X that knowing Y provides

Marginal Entropies

Set Analogy

I(X ; Y ) = H(X ) + H(Y )- H(X , Y )

= H(X )- H(X |Y )

= H(Y )- H(Y |X )

p(✓|Dn) =p(Dn|✓)p(✓)

p(Dn)=

p(edn, Dn-1|✓)p(✓)p(edn, Dn-1)

Provide framework to optimize information in

edn based on design ⇠n

Mutual Information

Physical Entropy:

Shannon Entropy: Quantifies unpredictability of information content

Mutual Information:

Note:

Utility Function

S = -kB

X

i

pi log pi • pi : Probability that system is in i thmicrostate

I(X ; Y ) =

Z

X,Y

p(x , y) log

p(x , y)

p(x)p(y)dxdy

H(X ) = -

Z

X

p(x) log p(x)dx

I(X ; Y ) =

Z

X,Y

p(x , y) log

p(x , y)p(x)p(y)

dxdy

=

Z

XY

p(x , y) log

p(x , y)p(x)

dxdy -

Z

XY

p(x , y) log p(y)dxdy

=

Z

XY

p(x)p(y |x) log p(y |x)dxdy -

Z

XY

p(x , y) log p(y)dxdy

= -

Z

X

p(x)H(Y |X )dx -

Z

Y

log p(y)p(y)dy

= -H(Y |X ) + H(Y )

Mutual Information

Utility Function:

Strategy:

• Marginalize over set of unknown future observations to compute average amount of information provided by design :

• Choose design condition that yields largest mutual information

• Experimental Design: [Terejanu et al., 2012]

• Implementation Issue: Efficient evaluation of mutual information

• Solution: Employ kth nearest neighbor (kNN) algorithm [Kraskov et al., 2004]

U(dn, ⇠n) =

Z

⌦p(✓|dn, Dn-1

) log p(✓|dn, Dn-1

)d✓-

Z

⌦p(✓|Dn-1

log p(✓|Dn-1

d✓

⇠n

I(✓; dn|Dn-1, ⇠n) =

Z

D

U(dn, ⇠n)p(dn|Dn-1, ⇠n)ddn

⇠⇤n = arg max

⇠n2⌅I(✓; dn|Dn-1

, ⇠n)

Example 1: Hydra

Example: Laminar Poiseuille flow to verify Hydra and low-fidelity model

Design Variable:

Friction Factor: f = 64/Re

Low-Fidelity Model:True Parameters:

0 500 1000 1500 20000

0.1

0.2

0.3

0.4

0.5

0.6

0.7

1

234 5 6 7 8 9

Reynolds Number

Fric

tion

Fact

or

Low−FidelityHigh−FidelityInitial Points

60 70 800

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

Parameter a

−1.1 −1 −0.90

100

200

300

400

500

600

Parameter b

Stage 9Stage 5Stage 1

Stage 9Stage 5Stage 1

f (✓) = ✓1Re✓2

✓ = [64,-1]

⇠ = Re

Example 1: Hydra

0 500 1000 1500 20000

0.1

0.2

0.3

0.4

0.5

0.6

0.7

1

234 5 6 7 8 9

Reynolds Number

Fric

tion

Fact

or

Low−FidelityHigh−FidelityInitial Points

60 70 800

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

Parameter a

−1.1 −1 −0.90

100

200

300

400

500

600

Parameter b

Stage 9Stage 5Stage 1

Stage 9Stage 5Stage 1

Model ComparisonDensity Evolution

Hydra Verification

Hydra-TH Values Analytic ValuesRe dp/dz Vavg f dp/dz Vavg f100 0.1851 0.0038 0.6782 0.1870 0.0039 0.6400200 0.3752 0.0075 0.3392 0.3740 0.0077 0.3200300 0.5623 0.0112 0.2269 0.5611 0.0116 0.2133400 0.7494 0.0150 0.1705 0.7481 0.0154 0.1600500 0.9366 0.0187 0.1365 0.9351 0.0193 0.1280

Example 2: Turbulent Mixing in COBRA-TF (CTF)

0.0 0.2 0.4 0.6 0.8 1.0

240

260

280

300

320

Scaled Input

Ts18

BETAExPRESTINGINAFLUX

Problem Setup:• Configuration (Design) Variables in STAR-

CCM+

– ExPRES: Initial pressure of fluid domain

– TIN: Initial temperature in fluid domain

– GIN: Inlet mass flow rate

– AFLUX: Average linear heat rate per rod

• Calibration Variable in CTF

– BETA: Turbulent mixing factor

• Experimental Data from WEC

– 22 mixing tests each of which produce 36 outlet temperatures

Dakota Workflow: Kathryn Maupin, Laura Swiler, Brian Adams, Brian Williams

Surrogate Required for COBRA-TF (CTF)

Ts1 Ts2 Ts3 Ts4 Ts5 Ts6

Range 114.2526 113.2762 112.8341 112.8384 113.2961 114.2797

PRESS 0.737077 0.639302 0.67627 0.679073 0.621807 0.759441

PRESS/Range 0.006451 0.005644 0.005993 0.006018 0.005488 0.006645

RMSE 0.457771 0.485425 0.518064 0.517842 0.484804 0.465511

RMSE/Range 0.004007 0.004285 0.004591 0.004589 0.004279 0.004073

Mutual Information Estimation in Dakota:• Requires 5000 independent samples

• MCMC with 20% burn-in removed and subsampling rate of 3 requires minimum of 18,750 iterations

• This necessitates construction and verification of fast surrogate for CTF

• Gaussian process (GP) surrogate trained and verified for all 36 subchannels

Surrogate Verification: PRESS (1-fold cross validation) and RMSE (50 test runs)

STAR-CCM+ Calibration of BETA in CTF

Hi2Lo Workflow and Results:• Calibrate BETA to initial simulations and/or

experiments.

• Generate predictions of HiFi code output using LoFi code – or its surrogate – at each candidate design.

• Estimate MI between calibrated BETA samples and HiFi predictions at each candidate.

– Select candidate with largest MI.

• Run HiFi code at optimal candidate, add calibration dataset, and recalibrate BETA.

– Repeat process until MI is sufficiently small or design budget is exhausted.

0.00 0.02 0.04 0.06 0.08 0.10

020

4060

80100

120

140

BETA Posteriors

BETA

PDF

1234567891011

BETA Posteriors

5 10 15 20

1.4

1.6

1.8

2.0

2.2

2.4

2.6

Current Design Size

Mut

ual I

nfor

mat

ion

●

●

●

●

●

●

●

● ●

●

● ● ●

●

●●

● ●

●

●

●

Mut

ual I

nfor

mat

ion

Design Step

Order of STAR-CCM+ Evaluations Selected by MI

ExPRES TIN GIN AFLUX MI ExPRES TIN GIN AFLUX MI1475 406.5 16.24 4.231 2390 465.5 18.60 4.584 2.17

1500 398.8 16.56 4.804 2.71 2320 551.1 14.87 3.702 2.19

2320 521.0 18.49 4.434 2.53 1475 406.6 16.24 4.214 2.182390 590.5 18.51 4.535 2.48 2390 556.9 22.19 5.645 2.092390 556.5 18.44 4.462 2.66 2375 553.0 14.85 3.735 2.162320 556.3 18.48 4.544 2.60 1465 518.5 22.26 3.780 2.192320 518.0 22.14 5.681 2.47 1465 514.5 18.58 3.202 2.232320 520.0 14.76 3.710 2.62 2320 593.7 18.38 4.621 2.232390 518.5 22.13 5.637 2.40 2320 553.8 22.23 5.661 1.992390 518.5 14.77 3.719 2.39 2320 591.5 22.11 5.564 1.482390 520.0 18.42 4.560 2.30 2390 591.4 22.26 5.641 1.32

Range of TIN essentiallycovered in 3 iterations

Example 3: Radiation Source Localization in Urban Setting Model: i th detector

Notes: • Simplified Boltzmann transport model.

• Represent source and detectors as points.

• Consider only uncollided flux.

• Ray tracing used to quantify detector responses.

�i = E[Ri(I0, r0)] =�ti✏iAi I0

4⇡|ri - r0|2exp

-NiX

ni=1

�ni sni

!

+ E[Bi�ti ]

10 Detectors

Goal: Use experimental design to guide fixed and moving sensor strategies. Parameters: ✓ = [x0, y0, I0]

Example 3: Radiation Source Localization in Urban Setting Algorithm:1.Set N equal to the number of samples to be used in kNN algorithm

2.Define the set of M possible discrete sensor locations

3.Obtain initial estimates for (x,y,I) using sensors placed at 3 locations from set of discrete sensor locations.

4.This leaves M-3 possible sensor locations.

10 Sensor Locations

5. For j = 4:M-1

a) Employ MCMC algorithm DRAM to construct 3xN matrix of parameter chains that are sent to kNN algorithm.

b) Employ kNN algorithm to determine design conditions that optimizes MI and hence indicates where 1 of 3 sensors should be moved.

c) Append location and response to data set.

d) Remove that location from design conditions.

0 50 100 150 20020

40

60

80

100

120

140

160

1

2 3

4

5

6

7

8

9

10

1112 13

14

15

16

17

18

1920

21

22

23

24

25 26

Example 3: Radiation Source Localization in Urban Setting Results: Order of chosen detector locations

Example 3: Radiation Source Localization in Urban Setting Results: Convergence of Posterior Distributions:

157.6 157.8 158 158.2 158.40

2

4

6

8

10 xIteration 3Iteration 7Iteration 15

3.2137 3.2138 3.2139 3.214 3.2141 3.2142 3.2143#109

0

1

2

3

4

5

6

7

8 #10-6 IIteration 3Iteration 7Iteration 15

97 97.5 98 98.5 990

1

2

3

4

5

6

7

8 yIteration 3Iteration 7Iteration 15

Concluding RemarksConclusions: • Mutual information (MI) framework can guide where in the design space to generate data using validated high-fidelity codes or experiments.

• MI framework can also be employed to guide sensor strategies for radiation detection.

• Computation of MI is natural in a Bayeisan framework. 0 500 1000 1500 20000

0.1

0.2

0.3

0.4

0.5

0.6

0.7

1

234 5 6 7 8 9

Reynolds Number

Fric

tion

Fact

or

Low−FidelityHigh−FidelityInitial Points

Future Work: • Extension to continuous design variables.

• Extend theory to accommodate highly correlated responses.

• Complete implementation of Hi2Lo framework and workflow in Dakota.

• Complete the CASL STAR-CCM+/CTF Hi2Lo analysis and implementation via Dakota.

• Experimentally test sensor placement strategy. 97 97.5 98 98.5 990

1

2

3

4

5

6

7

8 yIteration 3Iteration 7Iteration 15