1 Parallel Simulations of Underground Flow in Porous and Fractured Media H. Mustapha 1,2, A. Beaudoin 1, J. Erhel 1 and J.R. De Dreuzy 2 1 - IRISA – INRIA

1

Parallel Simulations Parallel Simulations of Underground Flow of Underground Flow

in Porous and Fractured in Porous and Fractured MediaMedia

H. MustaphaH. Mustapha1,21,2, A. Beaudoin, A. Beaudoin11, J. Erhel, J. Erhel11 and J.R. and J.R. De DreuzyDe Dreuzy22

1 - IRISA – INRIA de Rennes1 - IRISA – INRIA de Rennes

2 - Géosciences de Rennes2 - Géosciences de Rennes

2

OutlineOutline

Problem presentation Problem presentation Geometrical and flow fluid modelGeometrical and flow fluid model Numerical method and parallel matrix generationNumerical method and parallel matrix generation

Parallel linear solver Parallel linear solver Direct solver - PSPASESDirect solver - PSPASES Iterative solver – HYPRE Iterative solver – HYPRE

Results and performancesResults and performances Tests and architectureTests and architecture Complexity and scalability analysis with PSPASESComplexity and scalability analysis with PSPASES Comparison between PSPASES and HYPREComparison between PSPASES and HYPRE

Conclusions and future workConclusions and future work

H. Mustapha INRIA, Parco sept 2005

Parallel Simulations of Underground Flow in Porous Parallel Simulations of Underground Flow in Porous and Fractured Mediaand Fractured Media

3

OutlineOutline

Problem presentationProblem presentation Geometrical and flow fluid modelGeometrical and flow fluid model Numerical method and parallel matrix generationNumerical method and parallel matrix generation

Parallel linear solver Parallel linear solver Direct solver - PSPASESDirect solver - PSPASES Iterative solver – HYPRE Iterative solver – HYPRE





4

GeometricalGeometrical model model

Discrete fracture network Discrete fracture network

3D fracture network3D fracture network2D porous media2D porous media

Heterogeneous hydraulic Heterogeneous hydraulic conductivity fieldconductivity field



5

EquationsEquations

Q = - K*Q = - K*grad (hgrad (h) )

div (Q) = 0div (Q) = 0 BoundaryBoundary conditions conditions

Flow fluid modelFlow fluid model

Fixed head

Nul flux

3D fracture network3D fracture network

Fix

ed

head

Fix

ed

head

Nul flux

Nul flux

2D porous media2D porous media



6

OutlineOutline

Problem presentationProblem presentation Geometrical and flow fluid modelGeometrical and flow fluid model Numerical method and parallel matrix Numerical method and parallel matrix

generationgeneration

Parallel linear solver Parallel linear solver direct solver – PSPASES direct solver – PSPASES iterative solver – HYPRE iterative solver – HYPRE

Result and performancesResult and performances tests and architecturetests and architecture complexity and scalability analysis with PSPASEScomplexity and scalability analysis with PSPASES comparison between PSPASES and HYPREcomparison between PSPASES and HYPRE




7

Numerical method and parallel matrix generationNumerical method and parallel matrix generation

Conforming Conforming triangular triangular

meshmesh

Regular gridRegular grid

Assembled by all corresponding sub-domains linear systemsAssembled by all corresponding sub-domains linear systems



Mixed Hybrid Finite Element MethodMixed Hybrid Finite Element Method

Type of meshType of mesh

Global linear systemGlobal linear system

8

OutlineOutline


Parallel linear solverParallel linear solver Direct solver – PSPASES Direct solver – PSPASES Iterative solver – HYPRE Iterative solver – HYPRE





9

Parallel Linear solverParallel Linear solver Direct solver – PSPASESDirect solver – PSPASES

sparse matricessparse matrices

symmetric positive definite matricessymmetric positive definite matrices

distributed-memory paradigm and MPI librarydistributed-memory paradigm and MPI library

P (number of processors) = 2^NP (number of processors) = 2^N

Cholesky factorizationCholesky factorization

Iterative solver – HYPREIterative solver – HYPRE

sparse matricessparse matrices

V-cycle multi-grid algorithm (SMG)V-cycle multi-grid algorithm (SMG)

distributed-memory paradigm and MPI librarydistributed-memory paradigm and MPI library



10

OutlineOutline


Parallel linear solver Parallel linear solver Direct solver – PSPASES Direct solver – PSPASES Iterative solver - HYPREIterative solver - HYPRE

Results and performancesResults and performances Parallel architectureParallel architecture Complexity and scalability analysis with PSPASESComplexity and scalability analysis with PSPASES Comparison between PSPASES and HYPREComparison between PSPASES and HYPRE




11

Parallel architectureParallel architecturedistributed memorydistributed memory

2 nodes of 32 bi – processors 2 nodes of 32 bi – processors (Proc AMD Opteron 2Ghz with 2Go (Proc AMD Opteron 2Ghz with 2Go

of RAM)of RAM)

Parallel architectureParallel architecture



12

OutlineOutline

Problem presentation Problem presentation Geometrical and flow fluid modelGeometrical and flow fluid model Numerical method and parallel matrix generationNumerical method and parallel matrix generation

Parallel linear solver Parallel linear solver Direct solver – PSPASES Direct solver – PSPASES Iterative solver - HYPREIterative solver - HYPRE

Results and performancesResults and performances Parallel architectureParallel architecture Complexity and scalability analysis with Complexity and scalability analysis with

PSPASESPSPASES Comparison between PSPASES and HYPREComparison between PSPASES and HYPRE




13

Test 1Test 1 : CPU time of matrix generation, linear solving and flow : CPU time of matrix generation, linear solving and flow computation obtained with two processorscomputation obtained with two processors



Complexity analysis with PSPASESComplexity analysis with PSPASES

14

Test 2 :Test 2 : Memory requirements for both applications Memory requirements for both applications



Complexity analysis with PSPASESComplexity analysis with PSPASES

15

Scalability analysis with PSPASESScalability analysis with PSPASES



PTp

TE S

PTp

NR

P N Tp R

2 2,6 105 5.60 1.20 106

8 1,05 106 11.33 1.18 106

32 4,19 106 25.70 1,04 106

4 2,6 105 2.92 1.15 106

16 1,05 106 6.06 1.11 106

64 4,19 106 13.08 1,05 106

P N Tp R

2 2,6 105 13.10 10006

8 1,05 106 22.06 5942

32 4,19 106 38.41 341

4 2,6 105 7.94 16508

16 1,05 106 16.05 4083

64 4,19 106 No value No value

2D medium 3D fracture network

Test 1 :Test 1 : Efficiency Efficiency

16

Scalability analysis with PSPASESScalability analysis with PSPASES



TpT2S 2 Test 2 :Test 2 : Speed-up Speed-up

17

OutlineOutline


Parallel linear solver Parallel linear solver Direct solver - PSPASESDirect solver - PSPASES Iterative solver - HYPREIterative solver - HYPRE





18

Comparison between PSPASES and HYPREComparison between PSPASES and HYPRE

Test 1 :Test 1 : Linear solving time Linear solving time



PSPASESHYPRE

19

Comparison between PSPASES and HYPREComparison between PSPASES and HYPRE

Test 2 :Test 2 : Speed-up Speed-up

HYPRE PSPASES



20

OutlineOutline


Parallel linear solver Parallel linear solver Direct solver – PSPASES Direct solver – PSPASES Iterative solver – HYPRE Iterative solver – HYPRE





21

ConclusionsConclusions

Future workFuture work



Direct solver PSPASES

Iterative solver HYPRE

Scalable in the case of 2D medium

Not scalable in the case of 3D fracture network

• A 3D extension of parallel software in the case of 2D porous media

• Parallel software for simulating solute migration

Efficient for large linear systems

Documents

1 Parallel Simulations of Underground Flow in Porous and Fractured Media H. Mustapha 1,2, A. Beaudoin 1, J. Erhel 1 and J.R. De Dreuzy 2 1 - IRISA – INRIA