Parallel Preconditioners for the Incompressible Navier-Stokes Equations

04/19/23 AMSC Candidacy Presentation 1

Parallel Preconditioners for the Incompressible Navier-

Stokes Equations Robert Shuttleworth

Applied Math & Scientific Computation (AMSC)

University of Maryland


Outline Background

Incompressible Navier-Stokes Equations Discretization/Linearization

Preconditioning the N-S Equations General Preconditioners N-S Problem Specific

Pressure Correction Methods Pressure Convection-Diffusion

High Performance Computing Issues Preliminary Results

Lid driven cavity problem Flow over a diamond obstruction

Conclusions


Motivation/Focus

Motivation: Efficient and robust solution of steady state and transient flow problems Develop fully implicit solution methods to the

incompressible Navier-Stokes Solving the linear systems that arise can take upwards

of 70% of the CPU time of a given simulation Linear Solvers: Operator Based Block Preconditioning Focus: Adapt block preconditioners to the linear

subproblems that arise in realistic fluid flow problems

u)u(uu f inpt

in 0u


Introduction

uu old )( )(

Given the Incompressible Navier-Stokes Equations:

Nonlinear Term:

Oseen:

Newton:

Jacobian of Momentum Eq.

0

fu

pCB

BF T

Discretization and Linearization:

f pt

0u :Mass

u)u(uu :Momentum


Discretization and Linearization

0

fu

pCB

BF T

Mt

Fn

n

n

1

u

u

u

1

1

1

oseen

Mt

F

znn

yn

xn

zn

ynn

xn

zn

yn

xnn

1

)u(u)u()u(

)u()u(u)u(

)(u)(u)u(u

13

113

13

12

12

112

11

11

11

1

Newton


Complete Algorithm u(0) = initial condition or initial guessp(0) = initial condition or initial guess

for m = 1:Ntimestepsu(m) = u(m-1) , p(m) = p(m-1)

while || F (u(m) ,u(m-1) ,p(m) ,u(m) ) || > nonlin

ulag = u(m)

/* *//* Set up linear subproblem *//* *//* corresponding to F (u(m) ,u(m-1) ,p(m) ,ulag ) = 0. */

Iterate on A u(m) = b until || rk || /|| r0 || < saddle

Block Precondition

CB

BF T

A

linear solver

time loop

nonlinear loop


General Preconditioning Premise Preconditioning needed in solution of any large

scale PDE Bottleneck of solving N-S is the iterative solution of

the linear systems Given:

Preconditioning speeds up convergence by improving the spectral properties of a matrix

bQAxQ

AQAQ

bAx

11

11 )(

“Good”

“Cheaper”


Types of Preconditioners

General (Algebraic) Preconditioners Incomplete LU Factorization (ILU) Sparse Approximate Inverses (SPAI) Multigrid Domain Decomposition

N-S Problem Specific Preconditioners Pressure Correction Pressure Convection-Diffusion


Incomplete LU Factorization (ILU) Factoring a sparse matrix by Gaussian Elimination

generates fill-in. So, the L and U factors are less sparse than the original matrix.

By ignoring the fill-in that occurs within a certain tolerance, approximate factors to L and U are available.

Advantages – simple to implement, inexpensive, good for certain problems

Disadvantages – potential instabilities, lack of scalability, not good for CFD applications, and difficulty in parallelization


Block Preconditioners

Discretization

Consider: Optimal preconditioner is when X is the Schur

Complement, Question: How to approximate the Schur

complement?

00

fu

pB

BF T

X

BFP

T

.1 TBBFS

I

BFI

S

F

IBF

I

B

BF TT

00

00

0

1

1


Pressure Correction – (LD)U

11

11

1

1

1

)(ˆ0

0

ˆ

ˆˆ)(diagˆ

where0

ˆ

ˆ0

)(0

0

0

LDUSB

F

I

BFI

BFBS

FF

I

BFI

SB

F

ULDI

BFI

SB

F

B

BF

T

T

T

TT

So, we can apply a preconditioner to the saddle point matrix of the form:


Pressure Correction

SIMPLER:

111

1

11

1

)(ˆ0

ˆ00

0

ˆˆ)(diagˆ

whereˆ0ˆ0

)(0

0

0

ULDIFB

I

S

BF

I

P

BFBS

FF

S

BF

IFB

I

DULS

BF

IBF

I

B

BF

T

T

T

TT

Projection Matrix:

Enforces Incompressibility

(SIMPLEC) sumrow the of value absolute :F

(SIMPLE) diag(F) :F

ˆ

ˆ


Pressure Correction

Advantages Used in both transient and steady state Easy to implement and parallelize

Disadvantages Slower convergence – coupling of physics is

violated Choosing a relaxation parameter Inefficient for convection dominated flows


Pressure Convection-Diffusion – L(DU)

1

11

1

1

1

1

0

0

00

0

0

0

0

00

)(0

0

0

S

I

I

BI

I

F

S

BFQ

S

BFQ

IBF

I

S

BF

B

BF

DULS

BF

IBF

I

B

BF

TT

T

TT

TT

Therefore, a right oriented preconditioner can be applied to this problem:


Pressure Convection Diffusion - Fp

1

11

p22

)(ˆ

)(

grad))((grad))((

pT

pTT

pTT

FBBS

FBBBBF

FBFB

ww

p))grad(( wSuppose:

Suppose the velocity and pressure convection-diffusion terms commute with one another:

Then,


Pressure Convection Diffusion - BFB

.))(()(

).())((

.))((

)(

)(

.)(

.,

111

11

11

11

1

1

TTT

TTTT

TT

TT

TT

TT

TT

BBBFBBBS

BBBFBBBBBFS

BBFBBBBF

BBFBBBFB

BFBBBBFB

BFyByBBBFB

uByBy

So,

:becomes complement Schur The

: considerNow

Then, so vector, a Consider


Pressure Convection-Diffusion Advantages

Insensitive to mesh size, time step, and CFL number

Minor Reynolds number dependence Solves coupled system

Disadvantages Applications do not supply Fp

Designed for Oseen iterations Equations for Stabilized FEM are not developed Boundary Conditions


Time Loop

NonlinearLoop

LinearSolver

End NonLinLoop

End TimeLoop

BlockPrecond

Packages:

PackageMethodsComponent

NonlinearSolver

LinearSolver

blockprecondition

Fluid Flow

NOX

1

1

X

F Meros(TSF)

Newton-KrylovMethods

Finite ElementMPSalsa(Epetra)

GMRESRAztec00

(Epetra, TSF)

F-1 : GMRES/AMGX-1 : CG/AMG

Aztec00, MLEpetra


Epetra – Sparse Matrix Package Facilitates sparse matrix construction on both

parallel and serial machines Compressed Row Storage (CRS)

Double precision nonzero values are stored in contiguous memory locations

Builds a map (graph) – an array of integers corresponding to nonzero row/column entries

Rows are stored in consecutive order


Epetra - Sparse Matrix Library

Serial – interfaces the BLAS Parallel - handles distributed matrix details

Local versus global indices Global Map (graph) details which processor owns which

entries Local Map (graph) details how local data (and ghosts) is

represented Matrix-vector products Wrappers to Message Passing Interface (MPI)

(Note: The global map is normally determined by the user or other library.)


TSF Properties What is TSF?

TSF – Trilinos Solver Framework High level matrix/block matrix manipulation language Provides framework for integrating different

solvers/preconditioners Interface for representation-independent solvers

Why TSF? Abstract interfaces for vectors and operators Composable Block operators Deferred inverse and transpose Overloaded operators (matlab like syntax) Matlab-like simplicity, running on a supercomputer Transparent memory management


Benchmark Problems Lid Driven Cavity

Contains many features of harder flows

Steady and Unsteady Solutions

Flow over a diamond obstruction Inflow/Outflow boundary conditions Harder flow

MPSalsa Realistic massively parallel, chemically

reactive fluid flow code


MPSalsa Steady Problem Results

2D Lid driven cavity on a 64 x 64 gridRe ILU Simplec Simple Fp

10 88.0 52.5 46.8 25.4

50 92.8 56.6 50.2 30.8

100 95.7 59.2 53.0 40.8

200 95.9 70.2 61.3 56.6

The values in each column represent the average number of Outer Saddle Point

Solves per Newton Step.

Residual reduction for each of the preconditioners.



2D Lid driven cavity

Mesh Independence Preliminary Time Comparison

Re Mesh ILU Fp Proc

10 64 x 64 88.0 25.4 4

10 128 x 128 194.2 23.2 16

10 256 x 256 >1200 23.4 64

100 64 x 64 95.7 40.8 4

100 128 x 128 335.3 40.7 16

100 256 x 256 >1200 41.3 64

500 64 x 64 94.9 98.3 4

500 128 x 128 350.2 91.4 16

500 256 x 256 >1200 92.2 64


Implementation Challenge

Timings are not very good After profiling, upwards of 50% of the CPU

time is spent in an inefficient memory allocation routine

A multigrid smoother is inefficiently implemented



2D Flow over a Diamond ObstructionRe # Unknowns ILU Fp

10 16,000 45.0 36.0

64,000 110.8 37.8

250,000 332.2 36.2

25 16,000 45.5 48.8

64,000 101.7 51.5

250,000 297.2 47.0

The values in each column represent the average number of Outer Saddle Point Solves per Newton Step.


Future Work

Sparse Approximate Commutator (SPAC) for Fp

Compare/Optimize CPU Time amongst methods Tests on higher Re # for both steady/time

dependent problems More realistic problems

3D Problems Chemically reacting flow Turbulent flows


Conclusions Incompressible Navier-Stokes Equations Preconditioning the N-S Equations

General Preconditioners Problem Specific

Pressure Correction Methods Pressure Convection-Diffusion

Preliminary Remarks ILU preconditioner does not scale well Fp preconditioner is mesh independent and

competitive in CPU time


References A.J. Chorin, A numerical method for solving incompressible viscous

problems, Journal of Computational Physics, 2:12,1967. H. C. Elman, D. J. Silvester and A. J. Wathen, Finite Elements and Fast

Iterative Solvers, Oxford University Press, 2005. Howard Elman, V. E. Howle, John Shadid and Ray Tuminaro, A Parallel Block Multi-level Preconditioner for the 3D Incompressible Navier-Stokes Equations. Journal of Computational Physics 187:504-523, 2003.

D. Kay, D. Loghin, and A. J. Wathen, 2002, A preconditioner for the steady-state Navier-Stokes equations. SIAM J. Sci. Comput. 24, pp. 237-256.

M. Pernice and M.D. Tocci, A multigrid-preconditioned Newton Krylov method for the incompressible Navier-Stokes equations., SIAM J. Sci. Comput. 123, pp. 398-418.

S.V. Patankar, Numerical heat transfer and fluid flow, Hemisphere Pub. Corp, New York, 1980.

Documents

Parallel Preconditioners for the Incompressible Navier-Stokes Equations