New Directions in the Application

of Model Order Reduction

D.C. Sorensen

I Collaborators: M. Heinkenschloss, K. WillcoxS. Chaturantabut, R. Nong

I Support: AFOSR and NSF

MTNS 08 Blacksburg, VA July 2008

Projection Methods for MOR

Impossible Calculations Made Possible withROM

Experiments with many instances of same reduced model

Brief Intro to Gramian Based Model Reduction

Proper Orthogonal Decomposition (POD)

Balanced Reduction

Neural Modeling: Local Reduction ⇒ Many Interactions

Nonlinear MOR: Application of Empirical Interpolation (EIM)

Process/Design Variation: Monte-Carlo via ROM

LTI Model Reduction by Projection

x = Ax + Bu

y = Cx

Approximate x ∈ SV = Range(V), a k-diml. subspacei.e. Put x = Vx, and then force

WT [V ˙x − (AVx + Bu)] = 0

y = CVx

If WTV = Ik , then the k dimensional reduced model is

˙x = Ax + Bu

y = Cx

where A = WTAV, B = WTB, C = CV.

Moment Matching ↔ Krylov Subspace Projection

Based on Lanczos, Arnoldi, Rational Krylov methods

Pade via Lanczos (PVL)

Freund, Feldmann

Bai

Multipoint Rational Interpolation

Grimme

Gallivan, Grimme, Van Dooren

Recent: Optimal H2 approximation via interpolationGugercin, Antoulas, Beattie

Gramian Based Model Reduction

Proper Orthogonal Decomposition (POD)Principal Component Analysis (PCA)

x(t) = f(x(t),u(t)), y = g(x(t),u(t))

The Gramian

P =

∫ ∞

ox(τ)x(τ)Tdτ

Eigenvectors of P

P = VS2VT

Orthogonal Basisx(t) = VSw(t)

PCA or POD Reduced Basis

Low Rank Approximation

x ≈ Vk xk(t)

Galerkin condition – Global Basis

˙xk = VTk f(Vk xk(t),u(t))

Global Approximation Error ? (H2 bound for LTI)

‖x− Vk xk‖2 ≈ σk+1

Snapshot Approximation to P

P ≈ 1

m

m∑j=1

x(tj)x(tj)T = XXT

Truncate SVD : X = VSUT ≈ VkSkUTk

SVD Compression

m

k ( m + n)

m x n

v.s.

Storage

Advantage of SVD Compression

k

n

0 20 40 60 80 100 120 140 160 180 2000

10

20

30

40

50

60

70

80

90

100SVD of Clown

Image Compression - Feature Detection

original rank = 10

rank = 30 rank = 50

POD in CFD

Extensive Literature

Karhunen-Loeve, L. Sirovich

Burns, King

Kunisch and Volkwein

Gunzburger

Many, many others

Incorporating Observations – Balancing

Lall, Marsden and Glavaski

K. Willcox and J. Peraire

POD vs. FEM

I Both are Galerkin Projection

I POD - Global Basis fns. vs FEM - Local Basis fns.

I FEM - Complex Behavior via Mesh Refinement/ Higher OrderHigh Dimension - Sparse Matrices

I POD - Complex Behavior contained in Global BasisLow Dimension - Dense Matrices

I Caveat: POD is ad hoc: Must sample rich set of inputs andparameter settings

I Qx: How to automate parameter/input sampling for POD

POD for LTI systems

Impulse Response: H(s) = C(sI− A)−1B, s ≥ 0

Input to State Map: x(t) = eAtB

Controllability Gramian:

P =

∫ ∞

ox(τ)x(τ)Tdτ =

∫ ∞

oeAτBBT eAT τdτ

State to Output Map: y(t) = CeAtx(0)

Observability Gramian:

Q =

∫ ∞

oeAT τCTCeAτdτ

Balanced Reduction (Moore 81)

Lyapunov Equations for system Gramians

AP + PAT + BBT = 0 ATQ+QA + CTC = 0

With P = Q = S : Want Gramians Diagonal and Equal

States Difficult to Reach are also Difficult to Observe

Reduced Model Ak = WTk AVk , Bk = WT

k B , Ck = CkVk

I PVk = WkSk QWk = VkSk

I Reduced Model Gramians Pk = Sk and Qk = Sk .

Hankel Norm Error estimate (Glover 84)

Why Balanced Truncation?

I Hankel singular values =√

λ(PQ)

I Model reduction H∞ error (Glover)

‖y − y‖2 ≤ 2× (sum neglected singular values)‖u‖2I Extends to MIMO

I Preserves Stability

Key Challenge

I Approximately solve large scale Lyapunov Equationsin Low Rank Factored Form

CD Player Frequency Response

‖y − y‖2 ≤ 2× (sum neglected singular values)‖u‖2

100

101

102

103

104

105

106

107

10−10

10−8

10−6

10−4

10−2

100

102

Freq−Response CD−Player : τ = 0.001, n = 120 , k = 12

|G(jω

)|

Frequency ω

OriginalReduced

CD Player Frequency Response

‖y − y‖2 ≤ 2× (sum neglected singular values)‖u‖2

100

101

102

103

104

105

106

107

10−10

10−8

10−6

10−4

10−2

100

102

Freq−Response CD−Player : τ = 1e−005, n = 120 , k = 37

|G(jω

)|

Frequency ω

OriginalReduced

CD Player - Hankel Singular Values√

λ(PQ)

‖y − y‖2 ≤ 2× (sum neglected singular values)‖u‖2

0 20 40 60 80 100 12010

−14

10−12

10−10

10−8

10−6

10−4

10−2

100

102

Hankel Singular Values

Approximate Balancing

AP + PAT + BBT = 0 ATQ+QA + CTC = 0

• Sparse Case: Iteratively Solve in Low Rank Factored Form,

P ≈ UkUTk , Q ≈ LkL

Tk

[X,S,Y] = svd(UTk Lk)

Wk = LYkS−1/2k and Vk = UXkS

−1/2k .

Now: PWk ≈ VkSk and QVk ≈WkSk

Recent Progress LTI MOR

I Low Rank Approximate Solutions to Lyapunov Eqnsn = 1M Now Possible – Large Scale BTMOR Possible

I Optimal H2 reduction – IRKAPromising for Large no. Inputs

I Descriptor Systemse.g. Stykel (LAA 06) – general theory and approach

Balanced Truncation MOR of Oseen Eqn.

Semi-Discrete Oseen Equations: A Descriptor System

Ed

dtv(t) = Av(t) + Bu(t), y(t) = Cv(t) + Du(t)

0 1 2 3 4 5 6 7−150

−100

−50

0

50

100

150

Time

fullreduced

10−4

10−2

100

102

104

75.5

76

76.5

77

77.5

78

78.5

79

79.5

80

Frequency

fullreduced

Figure: Time response (left) and frequency response (right) for the full

order model (circles) and for the reduced order model (solid line).

Velocity Profile

full 22K dof reduced 15 dof

Figure: Velocities generated with the full order model (left col-

umn) and with the reduced order model (right column) at t =

1.0996, 2.9845, 3.7699, 4.86965, 6.2832 (top to bottom).

Heinkenschloss, Sun, S., SISSC 08

NOTICEState Variables will be y

for remainder

Reduced Order Neural Modeling

Steve Cox Tony Kellems

LINEAR MODELSBalanced Truncation Optimal H2

Nan Xiao and Derrick Roos Ryan Nong

Complex Model (Dim 160 K) → 20 variable ROM

NONLINEAR MODELSEmpirical Interpolation (EIM) - Patera

Saifon Chaturantabut T. Kellems

Nonlinear H-H Neuron Model (Dim 1198) → 30 variable ROM

Complex nonlinear behavior well approximated

Neuron Image AR-1-20-04-A

Image from NeuromartJ.O. Martinez, Rice-Baylor Archive of Neuronal Morphology,

http://www.caam.rice.edu/ cox/neuromart/, accessed 29 July 08

“Developed” using Neurolucida and NeuroExplorer

Neuron Cell

90µm

Hodgkin-Huxley Neuron Model

Full Non-Linear Model

Ij ,syn is the synaptic input into branch j

aj

2Ri∂xxvj = Cm∂tvj + GNam

3j hj(vj − ENa)

+ GKn4j (vj − EK ) + Gl(vj − El) + Ij ,syn(x , t)

Kinetics of the potassium (n) and sodium (h,m) channels

∂tmj = αm(vj)(1−mj)− βm(vj)mj

∂thj = αh(vj)(1− hj)− βh(vj)hj

∂tnj = αn(vj)(1− nj)− βn(vj)nj .

Ultimate Goal for Neural Modeling

I Ultimate goal is to simulate a few-Million neuron systemover a minute of brain-time: Feasibility Demonstrated

I Currently limited to a 10K neuron systemover a few brain-seconds without new technology

I Single Cell Simulation Time Reduction: 100 - 1000 times

I ROM computation time: seconds - few minutes

I 20 - 30 neuron types sufficient for Cortex

I Parallel computing required - under development(Kellems)

Cell Response - Lin and Non-Lin

0 5 10 15 20 25 30−0.2

−0.1

0

0.1

0.2

0.3

0.4

0.5

0.6v

Morphology: RC−3−04−04−B.asc

t (ms)

0 5 10 15 20 25 300.052

0.0525

0.053

0.0535

0.054

0.0545

0.055

0.0555

0.056

0.0565

0.057

m

0 5 10 15 20 25 300.589

0.59

0.591

0.592

0.593

0.594

0.595

0.596

0.597

0.598

h

0 5 10 15 20 25 300.3175

0.318

0.3185

0.319

0.3195

0.32

0.3205

0.321

0.3215

0.322

n

Lin HHBTFull HH

Cell Response - Linear

0 5 10 15 20 25 30−0.2

−0.1

0

0.1

0.2

0.3

0.4

0.5

0.6v

Morphology: RC−3−04−04−B.asc

t (ms)

0 5 10 15 20 25 300.052

0.0525

0.053

0.0535

0.054

0.0545

0.055

0.0555

0.056

0.0565

0.057

m

0 5 10 15 20 25 300.589

0.59

0.591

0.592

0.593

0.594

0.595

0.596

0.597

0.598

h

0 5 10 15 20 25 300.3175

0.318

0.3185

0.319

0.3195

0.32

0.3205

0.321

0.3215

0.322

n

Lin HHBTFull HH

Cell Response - Near Threshold

0 5 10 15 20 25 30−1

−0.5

0

0.5

1

1.5

2v

Morphology: RC−3−04−04−B.asc

t (ms)

0 5 10 15 20 25 300.045

0.05

0.055

0.06

0.065

0.07

m

0 5 10 15 20 25 30

0.565

0.57

0.575

0.58

0.585

0.59

0.595

0.6

0.605

h

0 5 10 15 20 25 300.315

0.32

0.325

0.33

0.335

0.34

n

Lin HHBTFull HH

Optimal H2 Methods: IRKA

PROBLEM:

Many inputs ⇒ Controllability Gramian Not Low Rank

Kellems and Nong

Using Variant of IRKA

Gugercin, Antoulas, Beattie (2008)

Reduced 160K Neuron Model to a ROM of order 20.

Solve times to construct ROM are under 2 minsHigh End Workstation(Sun Ultra 20) in MATLAB

Parameter Study Experiment 158 hrs (full) → 3.4 hrs (ROM)

ROM Results on Realistic Neuron

AR-1-20-04-A (Rosenkranz lab)Full system size 6726 Reduced system size 25

60µm

B

0 20 40 60 80 100−1

0

1

2

3

4Comparison of Soma Voltages

Time (ms)

Som

a P

oten

tial w

.r.t.

res

t (m

V)

NonlinearLinearizedReduced (k = 25)

0 20 40 60 80 100 12010

−12

10−10

10−8

10−6

10−4

10−2

Errors in Soma Voltage (Lin. vs. Reduced)

Time (ms)

Som

a V

olta

ge E

rror

(m

V)

Abs. ErrorRel. Error

Error vs HS-Values: AR-1-20-04-A

0 10 20 30 40 50 60 70 80 90−20

−15

−10

−5

0

Number of Singular Values used (10 runs/value)

log 10

of E

rror (

v lin v

s. v BT

)

Error follows Hankel singular value decay: AR−1−20−04−A, 35 Stimuli

AMax. Abs. Error25−75% Error BarsNormalized HSV’s

Highly Branched Neuron

n408 (Pyapali et al., 1998)

Full system size 41, 364 165, 330Reduced system size 20 20

110µm

C

0 10 20 30 400

200

400

600

800

1000

k (reduced system size)

Time to compute reduced model using IRKA

Sec

onds

A

0 10 20 30 40−8

−6

−4

−2

0

2

log 10

of M

ax. A

bs. E

rror

k (reduced system size)

Output error vs. full systemB

h = 2 µm (N = 41364)

h = 0.5 µ m (N = 165330)

Model Reduction of Nonlinear Terms

Saifon Chaturantabut

Implemenation of EIM method of Patera et.al. (2004)

Test Case: FitzHugh-Nagumo equations

εvt(x , t) = ε2vxx(x , t) + f (v(x , t))− w(x , t)

wt(x , t) = βv(x , t)− γw(x , t),

f (v) = v(v − 0.1)(1− v)

IC’s and BC’s

v(x , 0) = 0, w(x , 0) = 0 x ∈ [0, L]

vx(0, t) = −i0(t), vx(L, t) = 0 t ≥ 0

After FEM discretization,

Eyt = Ay + g(t) +N (y), y(0) = 0

Problem with Direct POD

If we apply the POD basis directly to construct a discretizedsystem, the original system of order N:

E ddt y(t) = Ay(t) + g(t) +N (y(t))

become a system of order k N:

E ddt y(t) = Ay(t) + g(t) + N (y(t)),

where the nonlinear term :

N (y(t)) = UT︸︷︷︸k×N

N (Uy(t))︸ ︷︷ ︸N×1

⇒ Computational Complexity still depends on N!!

Nonlinear Approximation via EIM

WANT:

N (y(t))← C︸︷︷︸k×nm

N (y(t))︸ ︷︷ ︸nm×1

99K k , nm N

Complexity k Independent of N

EIM Steps

Eyt = Ay + g(t) +N (y), y(0) = 0

1) Run trajectory and collect snapshots Y = [y1, y2, . . . ym]perhaps with several different inputs and parameter values.

2) Truncate SVD of snapshots to get a POD basis for Trajectory

3) Collect nonlinear snapshots sj = f (yj) = [f (y1,j), f (y2,j), . . . , f (yN,j)]T

4) Truncate SVD of nonlinear snapshots to get another POD basisfor the nonlinear term

5) Select EIM interpolation points and approximate nonlinear termvia collocation in the non-linear POD basis

6) Construct the nonlinear ROM from the reduced linear andnonlinear terms

Nonlinear Approximation via EIM Contd.

HOW:

CN (y(t)) =

(UT

∫Ω[Ψ(x)]TQ(x)dx

)︸ ︷︷ ︸

C

(Q−1

z fz(t))︸ ︷︷ ︸

N (y(t))

where

I fz(t) = [f (Φ(z1)y(t)), f (Φ(z2)y(t)), . . . f (Φ(znm)y(t))]T

I zj Empirical Interpolation Points

and where

I Ψ(x) = [ψ1(x), ψ2(x), . . . ψN(x)] - FEM basis

I Φ(x) = [φ1(x), φ2(x), . . . φk(x)] - POD basis- From Snapshots Ψ(x)y(t`)

I Q(x) = [q1(x), q2(x), . . . qnm(x)] - Nonlinear POD basis- From Snapshots s` = f (Ψ(x)y(t`))

EIM: Numerical Example

f (y ;µ) = (1− y)cos(3πµ(y + 1))e−(1+y)µ,

6 POD basis fns from 50 snapshots µ ∈ [1, π] uniform

−1 −0.5 0 0.5 1−1.5

−1

−0.5

0

0.5

1

1.5

2

x

s(x;

µ)

Plot of Approximate Functions (dim = 10) with Exact Functions (in black solid line)

µ= 1 µ= 1.1713 µ= 1.3855 µ= 3.1416

Figure: Approximate Function from EIM with 10 POD basis

EIM Reduction of FitzHugh-Nagumo Fiber

click figure for movie

EIM Reduction of FitzHugh-Nagumo Fiber

Nonlinear Reduction N = 1024→ k = 40

click figure for movie

EIM for Hodgkin-Huxley Equations

Kellems

After FEM or FD discretization HH-equations yield ODE system2664

a(x)CmI 0 0 00 I 0 00 0 I 00 0 0 I

3775

2664

vtmthtnt

3775 =

2664

12Ri

H 0 0 0

0 0 0 00 0 0 00 0 0 0

3775

2664

vmhn

3775 +

2664

g0(t)000

3775 +

2664

N v (v, m, h, n)Nm(v, m)

N h(v, h)N n(v, n)

3775

succinctly written as Eyt = Ay + g(t) +N (y)

Implementation Issues

1) Choose input stimuli to generate snapshots over full wave

2) NL-snapshots generated for nonlinear term N v (v ,m, h, n)

3) Terms Nm(v ,m),N h(v , h),N n(v , n) evaluated only at EIM points

EIM Reduction of Hodgkin-Huxley Fiber

Three Inputs

0 10 20 30 40 50 60−80

−60

−40

−20

0

20

40

60Forked neuron: voltages at node 10

Time (ms)

Vol

tage

(m

V)

Full (N = 1198)

Reduced (k,nm = 30)

EIM Reduction of HH : 3 inputs

Voltage Profile at Various Times (Full vs ROM)

0 0.2 0.4−75

−70

−65

0 0.2 0.4−76

−74

−72

−70

0 0.2 0.4−76

−74

−72

−70

0 0.2 0.4−76

−74

−72

0 0.2 0.4−78

−76

−74

−72

0 0.2 0.4−78

−76

−74

−72

0 0.2 0.4−80

−70

−60

0 0.2 0.4−80

−60

−40

−20

0 0.2 0.4−100

−50

0

50

0 0.2 0.4−100

−50

0

50

0 0.2 0.4−100

−50

0

50

0 0.2 0.4−100

−50

0

50

0 0.2 0.4−100

−50

0

50

0 0.2 0.4−100

−50

0

50

0 0.2 0.4−100

−50

0

50

0 0.2 0.4−100

−50

0

50

0 0.2 0.4−100

−50

0

0 0.2 0.4−70

−60

−50

−40

0 0.2 0.4−70

−60

−50

0 0.2 0.4−70

−65

−60

−55

0 0.2 0.4−66

−65.5

−65

−64.5

0 0.2 0.4−66

−65.5

−65

−64.5

0 0.2 0.4−65.5

−65

−64.5

0 0.2 0.4−65.5

−65

−64.5

−64

0 0.2 0.4−65

−64.5

−64

0 0.2 0.4−65

−64.5

−64

−63.5

0 0.2 0.4−65

−64

−63

−62

0 0.2 0.4−66

−64

−62

−60

0 0.2 0.4−66

−64

−62

−60

0 0.2 0.4−66

−64

−62

−60

0 0.2 0.4−66

−64

−62

−60

0 0.2 0.4−65

−64

−63

−62

0 0.2 0.4−65

−64

−63

−62

Blade Variation → Turbine Reliability/Performance

Siemens press picture

Small variations among blades ⇒ large impact on forced response

Determines high-cycle fatigue properties

Parametric ROM for Geometric Shape Variations

T. Bui-Thanh, , K. Willcox, and O. Ghattas 2007

Efficient reduced order models for probabilisticanalysis of the effects of blade geometry

2D problem governed by the Euler equations

Three orders of magnitude reduction in number of states

Accurately reproduce CFD Monte Carlo simulation resultsat fraction of computational cost.

Results impossible without ROM

Euler Equations → DG Discretization

Mathematical Model governed by 2D Euler Equations

DG + BC’s ⇒

E(w)dy

dt+ R(y,u;w) = 0

R nonlinear function of y,u;w

u ∈ Rm m-external forcing inputs via BC’s

w - random vector associated with geometric variability

Parametrically Dependent Linear Model

Linearize about steady state yss(g(w)) corresponding to geometry g(w)y = yss + y

E(w)dy

dt= A(w)y + B(w)u

z = C(y)

Linearize E,A,B,C :

e.g. A(w) ≈ A0 + A1w1 + A2w2 + . . .Amwm

One time off-line cost: nominal geometry base

Construct POD Basis → ROM

Adaptive Sampling Method

Starting with existing POD basis Φ, construct ROMThen iterate:

1) Select new parameter value wj via

wj = argmaxw‖z(w)− zr (w)‖s.t. PDE Constraints

2) Collect additional snapshots y(t;wj) for w = wj

3) Update POD basis Φ

5) Project Ai ,Bi ,Ci via e.g. Ai = ΦTAiΦ

ROM vs Full Linear Response

Outputs:z(1) = CL Lift Coefficientz(2) = CM Moment Coefficient

ROM Preservation of Statistics

Linearized CFD (left)

ROM predictions of work per cycle (WPC)

Monte Carlo simulation results for 10,000 blade geometries

Same geometries analyzed in each case.

Prediction and Cost

CFD (Full) Reduced

Model Size 103,008 290

Number nonzeros 2,846,056 84,100

Offline cost — 10.92 hrs

Online cost 515.61 hrs 1.10 hrs

Blade 1 WPC mean -1.8583 -1.8515

Blade 1 WPC variance 0.0503 0.0506

Blade 2 WPC mean -1.8599 -1.8583

Blade 2 WPC variance 0.0136 0.0138

10 random parameters

Computations on 64 bit PC with 3.2GH Pentium Processor

WPC = integral of the blade motion times the lift force over one unsteady cycle

Summary

Gramian Based Model Reduction: POD, Balanced Reduction

Optimal H2 Reduction via IRKA

Neural Modeling - Single Cell ROM ⇒ Many InteractionsExample of important class of problems(including Monte Carlo of Stochastic Systems)

Nonlinear MOR via EIMDemonstrated effective reduction and feasibility

Process Variation - Blade Geometry EffectsExample of important future application of MOR