PaMPA : Parallel Mesh Partitioning and Adaptation · Parallel Mesh Partitioning and Adaptation....

Cedric Lachat & Francois Pellegrini June 14, 2012

PaMPA :Parallel MeshPartitioning and Adaptation

Introduction

Contents

Introduction

Common needs of solvers regarding meshes

Data structures

Version 0.1

Example: Laplacian equation using finite element method

Some results

Upcoming features

Cedric Lachat & Francois Pellegrini - PaMPA June 14, 2012- 2

Introduction

Middleware

I PaMPA: Parallel Mesh Partitioning and AdaptationI Middleware library managing the parallel repartitioning and

remeshing of unstructured meshes modeled as interconnectedvaluated entities

I The user can focus on his/her “core business”:I SolverI Sequential remesher

I Coupling with MMG3D provided for tetrahedra

Physics, solver

Remeshing and redistribution

PT-Scotch

Seq. Qual.Measurement

Sequentialremesher

Contents

Introduction

Data structures

Version 0.1

Some results

Upcoming features

Common needs of solvers regarding meshes:

I Handling of mesh structuresI Distribution of meshes across the processors of a parallel

architectureI Handling of load balance

I Data exchange across neighboring entitiesI Iteration on mesh entities

I Entities of any kind: e.g. elements, faces, edges, nodes, . . .I Entity sub-classes: e.g. regular or boundary faces, . . .I Inner or frontier entities with respect to neighboring processorsI Maximization of cache effects thanks to proper data reordering

I Dynamic modification of mesh structureI Dynamic redistribution

I Adaptive remeshing

Data structures

Contents

Introduction

Data structures

Version 0.1

Some results

Upcoming features

Data structures

Constraints

I Distributed dataI Mesh data should not be replicated for the sake of scalabilityI Minimization of data exchangesI Abstraction from actual data structure implementations

Data structures

Definitions

I ElementI NodeI Edge

I InternalI Boundary

I VertexI RelationI EntityI Sub-entityI Enriched graph

Top-level mesh entityMay bear some data (volume,

pressure, etc.)

Data structures

Definitions

May bear some data (geometry, etc.)

Data structures

Definitions

May bear some data (flux, etc.)

Data structures

Definitions

Regular mesh edge

Data structures

Definitions

Boundary mesh edge

Data structures

Definitions

What all entities are in fact. . .

Data structures

Definitions

Subset of edges between verticesbelonging to prescribed entity types

Data structures

Definitions

Subset of vertices bearing the samedata

Data structures

Definitions

Subset of entity vertices that maybear specific data

Data structures

Definitions

Whole set of vertices and relationsEvery vertex belongs to one and only

one entity (and sub-entity)

Data structures

Global vue

baseval

enttglbnbr

proccnttab

procvrttab

1 4 8 11

Data structures

Local vision of process 0

vertgstnbr

vertlocnbr

edgelocnbr

ventloctab

edgeloctab

vertloctab

vendloctab

1 2 3 4 2

Data structures

Local vision of process 1

vertgstnbr

vertlocnbr

edgelocnbr

ventloctab

edgeloctab

vertloctab

vendloctab

3 13 2

1 2 1 3 2 1 2 1 3 1

Data structures

Per-entity data (1/2)

E.g. entity 1 on processor 0:

vertgstnbr

vertlocnbr

Data structures

Per-entity data (2/2)

E.g. node neighbors of element 1 on process 0:

vertloctab

edgeloctab

vendloctab

Data structures

Data linking

Version 0.1

Contents

Introduction

Data structures

Version 0.1

Some results

Upcoming features

Version 0.1

Entities

I Based (for C and Fortran interfaces)I SimpleI Stable with respect to sub-entities

Version 0.1

Partitioning

I Performed with respect to top-level entity

Version 0.1

Overlap

I Of size 1 at the time beingI Extensible to any distance nI Requires top-level relations

Version 0.1

Local vue of process 0

I Halo of size 1 I Overlap of size 1

Version 0.1

Local vue of process 1

I Halo of size 1I Overlap of size 1

Version 0.1

Iterators

I Entities - sub-entitiesI Decrease of cache missesI Debugging tools

PAMPA dmeshIt In i tStart ( mesh , vent locnum , PAMPA VERT ANY, & i t v e r t ) ;PAMPA dmeshItInit ( mesh , vent locnum , nentlocnum , &i t n g h b ) ;

w h i l e ( PAMPA itHasMore(& i t v e r t ) ) {v e r t l o c n u m = PAMPA itCurEnttvert locnum(& i t v e r t ) ;p r i n t f ( ” v e r t l o c n u m : %d\n” , v e r t l o c n u m ) ;PAMPA itStart(& i t n g h b , v e r t l o c n u m ) ;

w h i l e ( PAMPA itHasMore(& i t n g h b ) ) {PAMPA Num sngblocnum , engblocnum , mngblocnum , nsublocnum ;engblocnum = PAMPA itCurEnttvert locnum(& i t n g h b ) ;mngblocnum = PAMPA itCurMeshvertlocnum(& i t n g h b ) ;sngblocnum = PAMPA itCurSubEnttvert locnum(& i t n g h b ) ;nsublocnum = PAMPA itCurnsublocnum(& i t n g h b ) ;p r i n t f ( ”[%d ] ngb o f %d w i t h e n t i t y %d : %d ( sub : %d ) %d ( e n t : %d ) %d ( g l b )\n” ,

rank , ver t locnum , vent locnum , sngblocnum , nsublocnum ,engblocnum , nentlocnum , mngblocnum ) ;

PAMPA itNext(& i t n g h b ) ;}

Contents

Introduction

Data structures

Version 0.1

Some results

Upcoming features

All steps

! P a r a l l e l Mesh Computat ions!−−−−−−−−−−−−−−−−−−−−−−−−−−−

! P r o c e s s o r 0 o n l y :CALL ReadMesh ( ) ! Read mesh on f i l eCALL GraphPampa ( ) ! B u i l d graph o f v e r t e x n e i g h b o r s f o r PAMPACALL PampaCentra l izedMesh ( ) ! B u i l d PaMPA g l o b a l non d i s t r i b u t e d mesh

! On a l l p r o c e s s o r s :CALL PampaDistr ibutedMesh ( ) ! B u i l d PAMPA d i s t r i b u t e d mesh :! 1− S c a t t e r c e n t r a l i z e d mesh on a l l p r o c e s s o r s! 2− C a l l PAMPA mesh p a r t i o n n e r! 3− R e d i s t r i b u t e d i s t r i b u t e meshCALL ElementVolume ( )CALL I n i t i a l i z e M a t r i x C S R ( )

! S o l u t i o n computat ion!−−−−−−−−−−−−−−−−−−−−−

CALL I n i t S o l ( )CALL F i l l M a t r i x ( )CALL So lveSystem ( )

Solve system: Jacobi (1/2)

UaPrec = 0 . ! Suppose A = L + D + U, system to s o l v e : A x = bDO i r e l a x = 1 , N r e l a x

r e s = 0 .DO i s = 1 , MatCSR%Nin

r e s 0 = RHS( i s ) ! r e s 0 = bi 1 = MatCSR%I a ( i s )I 1 F i n = MatCSR%I a ( i s + 1) −1

DO i v = I1 , I 1 F i nj s = MatCSR%Ja ( i v )r e s 0 = r e s 0 − MatCSR%V a l s ( i v ) ∗ UaPrec ( j s ) ! r e s 0 = b − ( L + U) xˆn

END DOUa( i s ) = r e s 0 / MatCSR%Diag ( i s ) ! xˆn+1 = ( b − ( L + U) xˆn )/Dr e s 0 = r e s 0 − MatCSR%Diag ( i s ) ∗ UaPrec ( i s ) ! r e s 0 = ( b − ( L + D + U) xˆn )r e s 0 = r e s 0 ∗ r e s 0 ! r e s 0 = ( b − ( L + D + U) xˆn )∗∗2r e s = r e s + r e s 0

END DO

CALL PAMPAF dmeshHaloValue (dm, && enttnum = PAMPA ENTITY NODE , && tagnum = INT (PAMPA TAG SOL , PAMPAF NUM) , && r e t v a l = s t a t u t )

Solve system: Jacobi (2/2)

d e l t a U = 0 .d e l t a U = DOT PRODUCT( UaPrec ( : MatCSR%Nin ) − Ua ( : MatCSR%Nin ) ,

UaPrec ( : MatCSR%Nin ) − Ua ( : MatCSR%Nin ) )r e s 0 = 0 .r e s 0 = DOT PRODUCT( RHS ( : MatCSR%Nin ) ,

RHS ( : MatCSR%Nin ) )UaPrec = Ua

d e l t a U g l b = 0 .CALL MPI ALLREDUCE( de l taU , d e l t a U g l b , 1 , t y p e r e a l , MPI SUM ,MPI COMM WORLD, code )d e l t a U g l b = s q r t ( d e l t a U g l b )

r e s g l b = 0 .CALL MPI ALLREDUCE( r e s , r e s g l b , 1 , t y p e r e a l , MPI SUM , MPI COMM WORLD, code )r e s g l b = s q r t ( r e s g l b )

r e s 0 g l b = 0 .CALL MPI ALLREDUCE( r e s 0 , r e s 0 g l b , 1 , t y p e r e a l , MPI SUM , MPI COMM WORLD, code )r e s 0 g l b = s q r t ( r e s 0 g l b )

r e s g l b = r e s g l b / r e s 0 g l b

END DO ! end loop on i r e l a x

Some results

Contents

Introduction

Data structures

Version 0.1

Some results

Upcoming features

Some results

First results (1/2)

0 1 2 3 4 5 6 7 8 9 10

Number of processors (x1000)

Time for distributing centralized mesh and partitionning distributed mesh

ScatterPart

Some results

First results (2/2)

0 1 2 3 4 5 6 7 8 9 10

Number of processors (x1000)

Time for distributing centralized mesh and partitionning distributed mesh

Redistribution

Upcoming features

Contents

Introduction

Data structures

Version 0.1

Some results

Upcoming features

New features

I Main idea:I Parallel mesh adaptation

I Others:I Parallel I/O with HDF5I Overlap greater than 1I Unbreakable relations

Upcoming features

Franois Pellegrini June 14, 2012

PaMPA : Parallel Mesh Partitioning and Adaptation · Parallel Mesh Partitioning and Adaptation....

Documents

A Variationally Consistent Mesh Adaptation Method for

Variational Mesh Adaptation: Isotropy and Equidistributionpeople.ku.edu/~wzhuang/research/paper/H01b.pdfVariational Mesh Adaptation: Isotropy and Equidistribution Weizhang Huang Department

LEVELSETS AND ANISOTROPIC MESH ADAPTATION Alexandra ... · LEVELSETS AND ANISOTROPIC MESH ADAPTATION Alexandra Claisse and Vincent Ducrot UPMC Univ Paris 06, UMR 7598, Laboratoire

Anisotropic Delaunay mesh adaptation for unsteady simulationsdobrzyns/telechargement/DFIMR08.pdf · Anisotropic Delaunay mesh adaptation for unsteady simulations 5 To achieve this

8-3 Dynamic Mesh Adaptation for Three-Dimensional ...in4.iue.tuwien.ac.at/pdfs/sispad2005/1562046.pdf · Dynamic Mesh Adaptation for Three-Dimensional Electromigration Simulation

Strategies for Driving Mesh Adaptation in CFD (Invited)

MESH ADAPTATION IN FRACTURED RESERVOIR SIMULATION A …

Using Parallel Mesh Partitioning Strategies to …mmwolf/presentations/Conferences/SIAMPP04.pdfUsing Parallel Mesh Partitioning Strategies to Improve the Performance of Tau3P, an Electromagnetic

LEVELSETS AND ANISOTROPIC MESH ADAPTATION … AND ANISOTROPIC MESH ADAPTATION Alexandra Claisse and Vincent Ducrot UPMC Univ Paris 06, UMR 7598, Laboratoire J.L. Lions, F-75005 Paris,

CONVERGENT ERROR-CONTROLLED MESH ADAPTATION

Curvilinear Mesh Adaptation Using Radial Basis Function ...kirby/Publications/Kirby-112.pdfCurvilinear Mesh Adaptation Using Radial Basis Function Interpolation and Smoothing ... Abstract

A Geometry-aware Data Partitioning Algorithm for Parallel Quad … · 2015. 10. 5. · Keywords: Geometry-aware Data Partitioning, Parallel Mesh Generation, Large-scale Structural

Comparison of Mesh Adaptation Techniques for High-Speed

Parallel Tetrahedral Mesh Adaptation with Dynamic Load ...Parallel Tetrahedral Mesh Adaptation with Dynamic Load Balancing 1 Leonid Oliker NERSC, MS 50B-2239, Lawrence Berkeley National

Entropy dissipation of moving mesh adaptation · and the mesh adaptation used in the moving mesh approach. Let us point out that our approach allows us to represent the e ects of

New Numerical simulations and mesh adaptation : exercise sessions · 2017. 5. 9. · Numerical simulations and mesh adaptation : exercise sessions Charles Dapogny1, Pascal Frey2,

Mesh Adaptation Strategies for Discontinuous Galerkin Methods …web.kaust.edu.sa/faculty/ShuyuSun/pub/CCCT04_adaptDG.pdf · 2010-02-09 · Mesh Adaptation Strategies for Discontinuous

Adjoint-Based Error Estimation and Mesh Adaptation for ... · 20.01.2015 · Context and Motivation Output-based error estimation and mesh adaptation Demonstrated applicability to

Geometry Based Parallel Mesh Generation and Adaptation · 12 Parallel Mesh Adaptation • Predictive load balancing –New size field may lead to heavy refinement on one part and

Anisotropic and Dynamic Mesh Adaptation for Discontinuous ...element, and establish dynamic mesh adaptation strategies. Section 4 is devoted to numerical studies. It is demonstrated