A. Beaudoin 1 , J.R. De Dreuzy 2 , J. Erhel 1 and H. Mustapha 1

Sparse linear solvers applied to Sparse linear solvers applied to parallel simulations of parallel simulations of

underground flow underground flow in porous and fractured mediain porous and fractured media

A. BeaudoinA. Beaudoin11, J.R. De Dreuzy, J.R. De Dreuzy22, J. Erhel, J. Erhel11 and H. and H. MustaphaMustapha11

1 - IRISA / INRIA, Rennes, France1 - IRISA / INRIA, Rennes, France

2 - Department of Geosciences, University of Rennes2 - Department of Geosciences, University of Rennes, , FranceFrance

Matrix Computations and Scientific Computing Seminar

Berkeley, 26 October 2005

2D heterogeneous porous medium2D heterogeneous porous medium

Heterogeneous Heterogeneous permeability fieldpermeability fieldY = ln(K)Y = ln(K)with correlation functionwith correlation function

Parallel Simulations of Underground Flow in Porous Parallel Simulations of Underground Flow in Porous and Fractured Mediaand Fractured Media

2( ) expY YY

C

rr

91 Y

3D fracture network with impervious matrix3D fracture network with impervious matrix


length distribution has a great impact : power law n(l) = l-a

3 types of networks based on the moments of length distribution

mean variation third moment3 < a < 4

mean variation2 < a < 3

mean variation third momenta > 4

EquationsEquations

Q = - K*Q = - K*grad (hgrad (h) )

div (Q) = 0div (Q) = 0 BoundaryBoundary conditions conditions

Flow modelFlow model

Fixed head

Nul flux

3D fracture network3D fracture network

Fix

ed

head

Fix

ed

head

Nul flux

Nul flux

2D porous medium2D porous medium


Numerical method for 2D heterogeneous Numerical method for 2D heterogeneous porous mediumporous medium


Finite Volume Method with a regular mesh

Large sparse structured matrix with 5 entries per row


n=32 zoom

Sparse matrix for 2D heterogeneous porous Sparse matrix for 2D heterogeneous porous mediummedium

Conforming Conforming triangular triangular

meshmesh


Mixed Hybrid Finite Element Method with unstructured mesh

Large sparse unstructured matrix with about 5 entries per row

Numerical method for 3D Numerical method for 3D fracture networkfracture network


Sparse matrix for 3D fracture Sparse matrix for 3D fracture networknetwork

N = 8181

Intersections and 7 fractures

zoom

Memory requirements for matrices A and LMemory requirements for matrices A and L


Complexity analysis with PSPASESComplexity analysis with PSPASES

CPU time of matrix generation, linear solving and flow computationCPU time of matrix generation, linear solving and flow computationobtained with two processorsobtained with two processors


Complexity analysis with PSPASESComplexity analysis with PSPASES


2D porous medium : memory size and CPU time 2D porous medium : memory size and CPU time with PSPASESwith PSPASES

Theory : NZ(L) = O(N logN) Theory : Time = O(N1.5)

Slope about 1 Slope about 1.5


3D fracture network : memory size and CPU time 3D fracture network : memory size and CPU time with PSPASESwith PSPASES

NZ(L) = O(N) ? Time = O(N) ?

Theory to be done


2D porous medium : condition number estimated by 2D porous medium : condition number estimated by MUMPSMUMPS

To be ckecked : scaling or not


2D porous medium : residuals with PSPASES 2D porous medium : residuals with PSPASES

Parallel architectureParallel architecturedistributed memorydistributed memory

2 nodes of 32 bi – processors 2 nodes of 32 bi – processors (Proc AMD Opteron 2Ghz with 2Go (Proc AMD Opteron 2Ghz with 2Go

of RAM)of RAM)

Parallel architectureParallel architecture


Scalability analysis with PSPASES : speed-upScalability analysis with PSPASES : speed-up


TpT2S 2

Scalability analysis with PSPASES : isoefficiencyScalability analysis with PSPASES : isoefficiency


PTp

TE S

PTp

NR

P N Tp R

2 0.26 106 5.60 1.20 106

8 1.05 106 11.33 1.18 106

32 4.19 106 25.70 1,04 106

4 0.26 106 2.92 1.15 106

16 1.05 106 6.06 1.11 106

64 4.19 106 13.08 1,05 106

P N Tp R

2 0.26 106 13.10

8 1.05 106 22.06

32 4.19 106 38.41

4 0.26 106 7.94

16 1.05 106 16.05

64 4.19 106 No value No value

2D medium 3D fracture network

5.1 ?


2D porous medium : number of V cycles with 2D porous medium : number of V cycles with HYPRE/SMGHYPRE/SMG

Comparison between PSPASES and HYPRE/SMG : Comparison between PSPASES and HYPRE/SMG : CPU timeCPU time


PSPASESHYPRE

Comparison between PSPASES and HYPRE/SMG : Comparison between PSPASES and HYPRE/SMG : speed-upspeed-up

HYPRE PSPASES


PerspectivesPerspectives


• porous medium : large sigma, up to 9 and large N, up to 108

• porous medium : 3D problems, N up to 1012

• porous medium : scaling, iterative refinement, multigrid adapted to heterogeneous permeability field

• 3D fracture networks : large N, up to 109

• model for complexity and scalability issues• 2-level nested dissection • subdomain method

• parallel architectures : up to 128 processors• Monte-Carlo simulations• grid computing with clusters for each random simulation

• parallel advection-diffusion numerical models

Documents

A. Beaudoin 1 , J.R. De Dreuzy 2 , J. Erhel 1 and H. Mustapha 1