Maillage adaptatif massivement parallèle et calculs ... · - Curie thin nodes, a French supercomputer, 80640 Intel cores (Sandy Bridge EP (E5-2680) 2.7 GHz), with 4 Go RAM per core

Maillage adaptatif massivement parallèle et calculs multiphasiques Massively Parallel Mesh Adaptation for Multiphase Flows

T. Coupez, L. Silva, H. Digonnet

Ecole Centrale de Nantes Institut de Calcul Intensif, High Performance Computing Institute [email protected]

Falling droplet with surface tension benchmark:

Viscosity ratio : 10-6/10-3

Density ratio : 1/103

Surface tension: 0.05

Computational Multiphase Flow Dynamics

L= 0.1 m

V ~ 2.0 m/s

Re ~ 20000

Droplet splashing with surface tension: the crown

phenomenon.

Navier Stokes, FEM, Level Set and dynamic

anisotropic adaptive meshing

Thierry Coupez - ICI - Ecole Centrale de Nantes - France

Incompressible multiphase Navier-Stokes:

• Discontinuity: Heaviside and Dirac function

• Regularization:

))(1()())((

))(1()())((

21

21

HHH

HHH

mixture law :

Incompressible Navier Stokes and Multiphase flow


Eulerian framework

Phase function by Level Set

0

. ( )

0,

( )

uv U u s g u

t

u t x u x

u u

( ) ( , ) ,

, ( ) 0

x d x x

x x

The phase function as a signed distance function

Convected level set* : redistancing is included

* Ville L., Silva L., Coupez T. Convected level set method for the numerical simulation of fluid buckling International Journal for Numerical Methods in Fluids 66, 3 (2011) Pages 324-344


Industrial Multiphase Flow Simulation Example of wiping: an extreme process

Heavy Liquid (molten Zinc) moving up along a vertical surface at 2 m/s

High surface tension coefficient (10 times the water/air value)

Geometry: 0.20 m large, 0.60 cm high

Air jet impacting the liquid surface: 200 m/s

Liquid thickness above the impacting jet: less that 10 µm = 10-5 m

Smallest edge of the mesh = 1 µm = 10-6 m

uniform mesh would require : 120 billions of nodes

Present anisotropic adaptive calculation: 50 000 nodes

Droplet formation

Film of liquid thinning less than 10 µm = 10-6 m

Air jet speed: 200 m/s Re= 106

Multiphase Meshing

• Multiphase: interface • Explicit boundary:

– Lagrangian and ALE framework – Surface tracking – Coupling: surface and interface meshing constraint – Adaptive meshing nightmare – Parallel meshing ?

• Implicit Boundary

– Eulerian framework and Level Set representation – Moving or fixed geometry – Automatic and implicit coupling – Coupled adaptivity, parallel meshing – Volume meshing only

– Advanced Solver


Interpolation error of the local level set and mesh adaptation

Signed distance

Filtering: local truncated level set

Interpolation and approximation error

- Calculate the distance function at the node of the mesh

- Apply the bandwidth filter

- Let the interpolation machinery doing the work

- Let vary




Immersed volume by anisotropic mesh adaptation

• flow past a cylinder

( ( )) 2 ( ( )) ( ( ))

0

dvH H v p H g

dt

v

Naca0012


Implict boundary with rotating structure:

Wind turbine:

Immersed boundary and moving mesh:

- Level representation

- Boundary layers


Meshing and Anisotropic adaptive Meshing by local mesh modification

• Cut and paste : Mi+1 = Mi – E + E’

• E = node or edge neighbouring, E’ Star operator

• General 2D,3D,…: – Conform meh: Minimal volume theorem

12 12

Mesh derivation process

Mesh generation by local modification

and volume minimization

2 2

1 1

1

22

( ) ( )

(2)

( ), ( )

' ( )( )( )

: Lagrange or Clement interpolation operator

1H(u) Hessian:the second derivative

2

h h h K

h h K KL LK

h

V v H v P K

u u C u u C H u x x x x

u

Approximation error and Interpolation error relationship

1 1

1 1

1

21

2

ˆ ˆ( ( ), ( )) ( , )

ˆ ˆ( , )

1

K K K K K K

t

K K K

K K K

H x x x x H A x A x

A H A x x

A H M H

Metric field based on

the unit mesh theory

and linked to Hessian:

• Céa lemma for elliptic problem, remains an open question for Navier Stokes

• Conjecture: Interpolation error is dominant for diffusion dominant problem

• viscous boundary layers are diffusion dominant flow region


New metric by rescaling the edges : • Edge unit vector mapping:

• Length distribution tensor and Metric

• Balance the error on the edges

ijX

T. Coupez, Metric construction by length distribution tensor and edge based error for anisotropic

adaptive meshing, Journal of Computational Physics 230 2391-2405, 2010


Incompressible Navier-Stokes problem

The instabilities in convection-dominated regime (?)

The velocity-pressure compatibility condition

Variational MultiScale Approach:

Small scale

Large scale

1- Small scale modeling by the residual estimate:

2-Large scale modified varational equations

• Stabilized Finite element


Stabilization parameters:

Three identified additional contributions

For convection dominated problems

For pressure instabilities

Incompressible flows at high Reynolds number

,h

C h h

K

u v

T

,h

K h c h

K

u u v

T

R

Stabilized finite element method for NS equations

A. Masud, R.A. Khurram, A multiscale finite element method for the incompressible Navier–Stokes equations , Computer Methods in Applied Mechanics and Engineering, Volume 195, (2006) 1750-1777

T.J.R. Hugues et al., The variational multiscale method - a paradigm for computational

mechanics , Computer Methods in Applied Mechanics and Engineering, Nov (1998)

V. Gravemeier, W.A. Wall, E. Ramm, A three-level finite element method for the instationary incompressible Navier-Stokes equations, Computer Methods in Applied Mechanics and Engineering 193 (2004) 1323-1366.

L.P. Franca, A. Nesliturk, On a two-level finite element method for the incompressible

Navier-Stokes equations, International Journal for Numerical Methods in Engineering 52 (2001) 433-453

R. Codina, Stab ilized finite element approximation of transient incompressible flows using orthogonal subscales, Comput. Methods Appl. Mech. Engrg. (2002)

,h

K h h

K

u q

T

R

E. Hachem, B. Rivaux, T. Kloczko, H. Digonnet, T. Coupez, Stabilized finite element method for incompressible flows with high Reynolds number, Journal of

Computational Physics, Volume 229, Issue 23, 2010, 8643-8665

1

2en

iK

i

V Nh

V x

V V

h

Anisotropic Finite Elements Thierry Coupez - ICI - Ecole Centrale de

Nantes - France

Adaptive anisotropic mesh at high Re

Anisotropic vs Uniform isotropic

J. L. Guermond and P. D. Minev, Start-up flow in a three-dimensional

lid-driven cavity by means of a massively parallel direction

splitting algorithmInt. J. Numer. Meth. Fluids (2011)

E. Hachem, B. Rivaux, T. Kloczko, H. Digonnet, T. Coupez, Stabilized

finite element method for incompressible flows with high

Reynolds number, Journal of Computational Physics, 2010,

229, 8643-8665

T. Coupez. E. Hachem, Solution of High Reynolds incompressible

flow with stabilised finite element and adaptive anisotropic

meshing, Comp. Meth. App. Mech. Engng 267,65-85,2013

2D Lid driven cavity:

Re: 1000, 5000, 10000, 20000, 33000, 50000

Reynolds 20000:

Reynolds 100000

(~20000 elements)

(~50000 elements)


2D Lid Driven cavity with mesh adaptation Zoom x 100

RE = 5000


Parallel Anisotropic Mesh Adaptation

Parallel adaptive meshing

• reMeshing per subdomain

– local mesh modification

– Generation by local modification under a minimal volume constraint

– Length constraint by the metric field

– Interfaces frozen

• Parallelization by repartitioning

– Partioning

– Move the interfaces

– iterate reMeshing and rePartioning

local remeshing: no communication

local repartioning : element mesh migration

[Coupez, 2000], [Digonnet, 2001]

Parallel strategy (II) Parallelization

Independent remeshing : non conform meshes

Incomplete remeshing at the frozen interfaces: reRepartioning

Initial partitioning

Remesh each subdomain independently on each core with

frozen interfaces

Repartitioning to move partition interfaces

iterate

load-balance the final partition

[[Digonnet, 2001]

Initial partition on 7 cores

Parallel strategy (III) Illustration: 2d case

1st remeshing with blocked interfaces


1st repartitioning to displace interfaces inside the subdomains


2nd remeshing


2nd repartitioning


3rd remeshing


Last repartitioning for FE load balancing


Anisotropic mesh adaptation (I) A short summary

• On each MPI process (core) the meshing work is driven by a metric field

• Metric field at nodes by the edge error estimate and the error distribution constrained a given number of nodes

• Adaptation to multi components (heuristical interpolation error estimate)

[Coupez, 2011], [Coupez, 2013]

Anisotropic mesh adaptation (II) Illustration

• Works for different types of solution field

• Example on image interpolation and adaptation

[Silva, 2014]

Image – 700 x 700 pixels

140 nodes 50 000 nodes

Direct P1 interpolation

Anisotropic mesh adaptation (III) Illustration

• Extension to multi-field: color images

• Extension to multi-field: velocity and image

[Silva, 2014]

Massively parallel performance (I) Machines used

- Curie thin nodes, a French supercomputer, 80640 Intel cores (Sandy Bridge EP (E5-2680) 2.7 GHz), with 4 Go RAM per core each connected by an Infiniband network, 1.359 Petaflops - JuQUEEN, a German supercomputer, 458752 IBM PowerPC® A2, 1.6 GHz cores, with 1 Go RAM per core and an IBM Tore 5D network, 5.0 Petaflops

1

10

100

1000

10000

1 10 100 1000 10000

Remesh

Remesh + FE_Repart

perfect

#cores

Sp

ee

d-U

p

1

10

100

1000

16 160 1600

Sp

ee

d-U

p

#cores

Space dimension #cores Initial mesh #nodes

Final mesh #nodes

Times (s)

2d 1 - 4096 5 millions 21 millions 3300 to 3,3

3d 16 - 4096 3.6 millions 30 millions 6800 to 122

In 2d In 3d

Massively parallel performance (II) Strong speed up

• Uniform mesh refinement by a factor of 2, on Curie

Sp

ee

d-U

p Massively parallel performance (III) Weak speed up

• Run from 1 to 131 072 cores, uniform mesh refinement by a factor 4, on Curie and JuQUEEN

• Constant load per core: 500 000 nodes on Curie and 250 000 on JuQUEEN

• Final mesh has 33.3 billions of nodes (67 billions of elements)

• Good performances up to 8192 cores

0

50

100

150

200

250

300

350

1 10 100 1000 10000 100000 1000000

Curie (Remeshing)

Curie (Remeshing + FE repart)

JuQUEEN (Remeshing)

JuQUEEN (Remeshing + FE repart)

# cores

Tim

e (

s)

Massively parallel illustration (I) Adaptation on a sharp test function

- Adapted mesh with 60 millions nodes - Smallest mesh size less

than 1e-4 - 70 iterations using 2 048

cores of Curie in (10h)

Massively parallel illustration (II) Adaptation for implicit functions

• Volume reconstruction by immersion of a surface mesh Musée du Château des Ducs de Bretagne

9 millions of triangles

Volume mesh: 6 millions of nodes on 256 cores

Parallel Multigrid Solver

Parallel multigrid solver Goals

• Adapted meshes can be produced with several billions of nodes and elements. Solving PDEs on such meshes?

• Complexity of iterative methods, such as conjugate gradient, gmres, bcgs,… is a breakdown to deal with such large meshes.

• To keep the method versatile and robust

– not using hierarchical refinement

– mesh partitioning may change between grid levels

– only the fine mesh is given

• PETSc library: several parallel iterative solvers and preconditionners, well suited for implicit solvers

– interface for a multigrid preconditionner type - PCMG

Parallel multigrid solver (II) Parallel interpolation / restriction operators

• One has to provide:

– The matrix for each level:

• Physical discretization (for Geometric MG)

• Recursive reduction of the fine matrix (Algebraic MG)

– Interpolation operator and/or restriction one between levels.

• Interpolation between levels needs performing a parallel localization of the nodes

• Thanks to the remeshing strategy, we reduce the number of external localizations

• A« pixel » mask filter optimization to reduce memory used for massive parallel computations

external localization

« Pixel » mask

Massively parallel performance (I) Weak speed-up

• Resolution of the incompressible Stokes equation using a mixed stabilized formulation on Curie and JuQUEEN

• Runs from 1 to 262 144 cores for a relative error of 1e-9

• Biggest run with 100 billions of unknowns (using around 200TB of RAM)

• Really good performance up-to 10 000 cores, worsening beyond (specially on Curie).

0

100

200

300

400

500

600

1 10 100 1000 10000 100000 1000000

Curie

JuQUEEN

# cores

1E-12

1E-11

1E-10

1E-09

1E-08

0,0000001

0,000001

0,00001

0,0001

0,001

0,01

0,1

1

0,000001 0,0001 0,01 1

Err

or

mesh size

FE error

Err V 1e-9

Err P 1e-9

Err Tot 1e-9

Err V 1e-12

Err P 1e-12

Err tot 1e-12

Err V Order2

Err P Order1

Err TotOrder 1

Linear convergence for the pressure and quadratic for the velocity

3D Direct Image-Based Flow Simulations

Applications: context and objectives Study scales: microscopic, mesoscopic, macroscopic

• Computations at the micro / meso scales, by considering randomness

Random reinforcements

UD yarn with 2000 fibres

• Direct simulation on the sample

3D results at mesoscales 3D Irregular reinforcement

• 3D sample obtained by 3D imaging [Orgéas et al, 3S-R]

• Reinforcement characteristiques: R=0.1mm, L=10mm, F=0.83

• Images obtained by X-ray tomography, voxel size= 10x10x10 m3

900x900x220 (350Mb)

• Flow computation on the volume

Metrics field norm from mesh adaptation Adapted mesh


• Generation of the finite element mesh : interpolation + adaptation, on 96 cores, in //, with a local remeshing with partition interface displacement

• Size of the mesh : ~5 millions of nodes (compared to 178.2 millions of voxels)

Isovalue 0 of the regularized Heaviside Adapted mesh

3D results 3D Irregular reinforcement at

• Generation of the finite element mesh : interpolation + adaptation, on 96 cores, in //, with a local remeshing with partition interface displacement

• Size of the mesh : ~5 millions of nodes (compared with 178.2 millions of voxels)


• Flow solveur: stabilized with multigrid

• Computation of permeability values: ~10e-9 dans les trois directions

Pressure Velocity norm

Larger images? Massively parallel simulation

Pressure

Velocity

- Positive Isovolume representing fibers - Adapted mesh with 400 millions nodes and 2.2 billions tetrahedra - Done using a 1200x1200x1800 voxels (2GB) 3d

Xray tomography image. (3SR Grenoble) - With Curie’s 4 096 cores :

- the mesh was generated in 5h with 10 iterations.

- the incompressible Stokes flow computation leads to a 1.5 billions of unknowns linear system solved in 1 100 seconds

Conclusions and perspectives • Anisotropic mesh adaptation and flow solution with a multigrid solver at

supercomputers scale.

• The largest generated mesh contained 33 billions of nodes (67 billions of elements) using 65 536 (Curie) and 262 144 (JuQUEEN) cores

• The largest linear system solved for 100 billions unknowns using 65 536 (Curie) and 262 144 (JuQUEEN) cores and 200 TB of RAM.

• Applications:

– CFD at large scale

– Computational Multiphase Flow Dynamics

– Image based flow computations (Simulation from real Data)

– Material science

– …

• Future :

– Use hardware information to fit the partition on the hardware (>8192 cores)

– Massively parallel scalability on complex unsteady and nonlinear flow simulations

– Move to higher order elements (P2/P3):

• Adaptive anisotropic P2 element with curved edges

• Convergence order would compensate additional costs on massive parallel machine

• Stability

• Acknowledgment to GENCI (Grand Equipement National de Calcul Intensif) and PRACE for the access to Curie and Juqueen

Documents

Maillage adaptatif massivement parallèle et calculs ... · - Curie thin nodes, a French supercomputer, 80640 Intel cores (Sandy Bridge EP (E5-2680) 2.7 GHz), with 4 Go RAM per core