An Optimal Domain Decomposition Method for the … Chart 1 > GAMM’17 > J. Thies et al. Optimal C-Grid DD > 2017-09-08 An Optimal Domain Decomposition Method for the C-Grid Navier-Stokes

www.DLR.de • Chart 1 > GAMM’17 > J. Thies et al. • Optimal C-Grid DD > 2017-09-08

An Optimal Domain Decomposition Method for the C-GridNavier-Stokes Jacobian

Jonas Thies? Sven Baars Mark v.d. Klok Fred Wubs†

? German Aerospace CenterSimulation and Software Technology

† University of GroningenDepartment of Mathematics and Computer Science

project ESSEX


Problem Setting: Nonlinear PDE System

2nd order PDE after space discretization

• M∂Φ∂t = F(Φ, t)

• with suitable boundary and initial conditions

Steady state; Φ as t →∞.

Standard technique: time stepping

• may take very long

• no information about stability of solution

• low frequency modes affect solution on very long time scales


Problem Setting: Nonlinear PDE System

Example: 3D Boussinesq equations

∂u

∂t=− ((uu)x + (vu)y + (wu)z )− px + ν∇2u

∂v

∂t=− ((uv)x + (vv)y + (wv)z )− py + ν∇2v

∂w

∂t=− ((uw)x + (vw)y + (ww)z )− pz + ν∇2w + gαT

∂T

∂t=− ((uT )x + (vT )y + (wT )z ) + κ∇2T

ux + vy + wz = 0

Jacobian has saddlepoint structure: J =

(A −GradDiv 0

).


Example: Rayleigh-Benard Convection

• Cube-shaped domain

• heated from below

• Rayleigh-Number

Ra = αg∆Td3

νκ

Figure. Flow patterns near the

first three primary

bifurcations

(a) x/y roll,(b) diagonal roll,(c) four rolls,(d) toroidal roll


Continuation Methods

• step through parameter space ratherthan time

• predictor/corrector method

• Newton-Krylov

• linear stability analysis (Eigenproblems)

Bifurcation diagram

Depends crucially on strong preconditioning


C-grid and F-matrices

u u

v

v

p u

uv

v

w

w

p

• Discretization yields saddle point problem[A BBT O

] [~up

]=

[f1f2

]F-matrix: A is spd and B has at most two nonzeroes per row and

row sum zero.


Computing an LU Decomposition of an F-matrix

[A BBT 0

] [xvxp

]=

[fvfp

]V − nodesP− nodes

Algorithm: LU decomposition of an F -matrix.

• Compute a fill-reducing ordering for the graph F (A) ∪ F (BBT ),

• during Gaussian elimination, insert the P-nodes to form 2× 2 pivotswhenever a coupling between a V-node and a P-node is encountered.

Theorem (De Niet/Wubs 2009)

In every step of this algorithm, the resulting Schur complement is anF-matrix.


How is Fill Generated in the Direct Approach?

α β aT bT

β 0 bT 0

a b A B

b 0 BT O

. (1)

Elimination step:

• Multiple of bbT is added to A;

• b becomes denser as P-nodes are eliminated;

• So dropping in A doesn’t make sense.


Domain Decomposition

• Subdomains and separators;

• Retain one pressure per subdomain.

• This ordering exposesparallelism in the matrix:

K =⇒(

K11 K12

K21 K22

),

where K11 is block-diagonal.


Domain Decomposition

• Subdomains and separators;

• Retain one pressure per subdomain.

• This ordering exposesparallelism in the matrix:

K =⇒(

K11 K12

K21 K22

),

where K11 is block-diagonal.


The Schur Complement

• LU-decomposition of the matrices onthe subdomains, K11 = L11U11;

• Schur-complement:S = K22 − K21K

−111 K12;

• retains F-matrix-property of K;

• only a few rather dense ‘B’ columns(with at most two entries per row);

Schur complement

(four subdomains)


Sparse Approximation of the Schur Complement

• Still an F-matrix;

• All V-nodes on a separator are nowconnected to the same 2 P-nodes;

• Use orthogonal transformation todisconnect them.

=⇒ Only one V-node per separator remainsconnected to P-nodes (VΣ-nodes)


Sparse Approximation of the Schur Complement

• Still an F-matrix;

• All V-nodes on a separator are nowconnected to the same 2 P-nodes;

• Use orthogonal transformation todisconnect them.

=⇒ Only one V-node per separator remainsconnected to P-nodes (VΣ-nodes)


Dropping

• Use simple drop-by-position:• Drop all couplings between separator

groups• ... and all couplings between VΣ and

regular V-nodes.

=⇒ Block diagonal preconditioner with a‘reduced matrix’ S2 in the lower right.


Dropping

• Use simple drop-by-position:• Drop all couplings between separator

groups• ... and all couplings between VΣ and

regular V-nodes.

=⇒ Block diagonal preconditioner with a‘reduced matrix’ S2 in the lower right.


Robustness and O(N logN) with an ILU???

1. Orthogonal transformations:• Eliminate most V-P couplings to avoid fill;• ‘Transfer operators’ defining coarse problem S2.

2. Coarse problem S2: solve for flux VΣ through each separator;• Still an F-matrix in case of the Stokes equations;

3. Constraint preconditioning:• no approximations in ‘Grad’ or ‘Div’ part;• mass is conserved exactly throughout.

4. Drop-by-position• original properties preserved (symmetry, positiveness);• singular subsystems cannot occur.

5. No segregation of variables:• velocity and pressure kept together;• no nested iterations.





































Issue with Standard Cartesian Partitioning

isolated pressure unknowns must be retained in Schur-Complement

2D

Ω1 Ω2

Ω3 Ω4

1 retained P-node per subdomain

3D

O(s) retained P-nodes forsubdomain size s3


Solution in 2D: Skew Partitioning


Solution in 2D: Skew Partitioning


Skew Partitioning in 3D – the Parallelepipedal Subdomain

• space-filling single template

• stackable: suitable for multilevel

• structure-preserving: no isolated P-nodes

German: Spat

BSc. thesis of Mark v.d. Klok, implemented in HYMLS by Sven Baars






German: Spat







German: Spat







German: Spat



Summary: Staggered-grid Multi-Level ILU (SMILU)








A First Benchmark: 3D Stokes on Emmy @ RRZE

• Stokes-flow, cube-shaped domain

• 2× 10 core Intel(R) IvyBridge cluster (approx. 500 nodes)

• GMRES(50)+SMILU

• 3–7 levels

• separator length s = 8 (subdomain size s3 grid cells)

• coarsening factor c = 2 (8 subdomains combined)

• implementation: HYMLS (Trilinos/Epetra, flat MPI)


A First Benchmark: 3D Stokes on Emmy @ RRZE

64 128 256 512

0

0.2

0.4

0.6

0.8

grid size nx

CP

Um

s/

un

know

nfactor

100 iterationssolve

2 16 128 1024

number of cores np


Questions?

Contact

Jonas Thies

DLR Simulation and Software TechnologyHigh Performance Computing

[email protected]

Phone 02203 / 601 41 45

http://www.DLR.de/sc

Funding:DLR, NWO, DFG(SPPEXA/ESSEX)

.

Summary

• fully coupled preconditioner

• ‘geometric’ approach

• multi-level, O(N logN)

Next

• evaluate performance on biggermachines (e.g. SuperMUC)

• more complex flow problems(e.g. Rayleigh-Benard)

• fully implicit ocean model

[email protected]

http://www.DLR.de/sc

Documents

An Optimal Domain Decomposition Method for the … Chart 1 > GAMM’17 > J. Thies et al. Optimal C-Grid DD > 2017-09-08 An Optimal Domain Decomposition Method for the C-Grid Navier-Stokes