A Fortran 95 object-oriented parallel code for the ...Introduction NEMO main features Examples A Fortran 95 object-oriented parallel code for the numerical solution of PDEs NEMO (Numerical

IntroductionNEMO main features

Examples

A Fortran 95 object-orientedparallel code for the

numerical solution of PDEsNEMO (Numerical Engine for Multiphysics Operators)

Marco Rorro

e-mail:[email protected] Inter-Univ. Computing Consortium

Seminario di Modellistica differenziale numericaRome, October 27, 2009

Marco Rorro NEMO: Numerical Engine for Multiphysics Operators


Examples

Outline

1 IntroductionObject-Oriented programming in Fortran 95The PSBLAS library

2 NEMO main featuresPartitioning and ReorderingGeometry and mesh capabilitiesPDE solver

3 ExamplesHeat transferConvection-diffusionIncompressible Navier-Stokes equations



Examples

Main references

S. Toninel, Development of a New Parallel Code for ComputationalContinuum Mechanics Using Object-Oriented Techniques, PhD Thesis,Universitá di Bologna, 2006.(http://www.ce.uniroma2.it/nemo/toninel_phd.pdf)

J.H. Ferziger and M. Peric, Computational Methods for Fluid Dynamics,Third ed. Springer, 2002.

S. Filippone and A. Buttari, PSBLAS-2.0 User’s Guide. A ReferenceGuide for Parallel Sparse BLAS library, 2006.

S. V. Patankar, Numerical Heat Transfer and Fluid Flow, First ed. Seriesin Computational Methods in Mechanics and Thermal Sciences. Taylor& Francis, 1980.

H. K. Versteeg and W. Malalasekera, An Introduction to ComputationalFluid Dynamics - The Finite Volume Method, First ed. Prentice Hall,1995.



Examples

Main references

S. Toninel, Development of a New Parallel Code for ComputationalContinuum Mechanics Using Object-Oriented Techniques, PhD Thesis,Universitá di Bologna, 2006.(http://www.ce.uniroma2.it/nemo/toninel_phd.pdf)

J.H. Ferziger and M. Peric, Computational Methods for Fluid Dynamics,Third ed. Springer, 2002.

S. Filippone and A. Buttari, PSBLAS-2.0 User’s Guide. A ReferenceGuide for Parallel Sparse BLAS library, 2006.

S. V. Patankar, Numerical Heat Transfer and Fluid Flow, First ed. Seriesin Computational Methods in Mechanics and Thermal Sciences. Taylor& Francis, 1980.

H. K. Versteeg and W. Malalasekera, An Introduction to ComputationalFluid Dynamics - The Finite Volume Method, First ed. Prentice Hall,1995.



Examples

Object-Oriented programming in Fortran 95The PSBLAS library

OO languages for scientific computing

The programming language chosen for supporting anobject-oriented formulation should satisfy the followingrequisites:

Data abstractionEncapsulationInheritancePolymorphismOperator overloading

Besides this, one should also account forcomputational efficiencydebugging and maintaining possibilitiescross-platform portability



Examples


NEMO and PSBLAS

NEMO is designed for running on distributed memoryparallel systems, according to the Message Passingparadigm. In particular the numerical algorithms whichrequire great effort in order to be parallelized are theiterative methods for the solution of the linear systemarising from the discretization of the PDE.This task is accomplished by means of the Parallel SparseBLAS library (http://www.ce.uniroma2.it/psblas)whose development was addressed by S. Filippone and M.Colajanni (2000) and S. Filippone and A. Buttari.

It provides a large set of very efficient preconditioners andnon-stationary, iterative, methods from the Krylov subspacefamily.Preconditioners and iterative methods are invoked bycalling single subroutines in a manner completelytransparent to the user.



Examples


NEMO and PSBLAS

NEMO is designed for running on distributed memoryparallel systems, according to the Message Passingparadigm. In particular the numerical algorithms whichrequire great effort in order to be parallelized are theiterative methods for the solution of the linear systemarising from the discretization of the PDE.This task is accomplished by means of the Parallel SparseBLAS library (http://www.ce.uniroma2.it/psblas)whose development was addressed by S. Filippone and M.Colajanni (2000) and S. Filippone and A. Buttari.

It provides a large set of very efficient preconditioners andnon-stationary, iterative, methods from the Krylov subspacefamily.Preconditioners and iterative methods are invoked bycalling single subroutines in a manner completelytransparent to the user.



Examples


The PSBLAS library consist of

computational routinessparse matrix by dense matrix productsparse triangular system solution for block diagonalmatricesvector and matrix normdense matrix sumsdot product

auxiliary routinescommunication descriptor allocationdense and sparse matrix allocationdense and sparse matrix build and updatesparse matrix and data distribution preprocessing



Examples


The PSBLAS library consist of

Preconditioner routinesDiagonal scalingIncomplete LU (ILU) factorizationAdditive SchwarzMultilevel Schwarz

Iterative methodsConjugate Gradient (CG)Generalized minimal Residual (GMRES)BiConjugate Gradient (BICG)Conjugate Gradient Stabilized (CGS)BiConjugate Gradient Stabilized (BiCGSTAB)



Examples

Partitioning and ReorderingGeometry and mesh capabilitiesPDE solver

Partitioning

case(-1)call bld_part_random(ncells,nprocs,part_cells)

case(0)call bld_part_block(ncells,nprocs,part_cells)

case(1)call get_conn_csr(c2c,xadj_glob,adjncy_glob)call bld_part_graph(xadj_glob,adjncy_glob,ipart=1)call get_part_graph(part_cells)

case defaultwrite(*,200)call abort_psblas

end select

call psb_cdall(icontxt,desc_c,info,vg=part_cells)



Examples


Partitioning example

Block partition Metis partitition



Examples


Reordering

In order to reduce the bandwith and profile of connectivitymatrix A

Bandwith = max{|i − j | : aij 6= 0

}(1)

Profile =N∑

i=1

di , di = i − fi , fi = min{j : aij 6= 0} (2)

NEMO implement the reordering algorithm developed by Gibbset al. (Gibbs, Poole and Stockmeyer An algorithm for reducingthe bandwith and the profile of a sparse matrix, SIAM J. ofNum. Anal., 1976).The renumbering improve also the partition.



Examples


Reordering example

sparsity pattern without reordering sparsity pattern with GPS reordering



Examples


Reordering + partitioning example

Block partition without reordering Block partitition with GPS reordering



Examples


Mesh main features I

triangles, quadrilaterals or generic polygons (2D)tetrahedra, pyramids, prism, hexahedra or genericpolyhedra (3D)

3D hybrid mesh with mixed tetrahedra, pyramids, prism and hexaedra



Examples


Mesh main features II

co-located arrangmentmaster-slave, face-based approach

Master-slave approach



Examples


Supported mesh formats

Input formats:Gambit Neutral Format (grid generator of FLUENT)CGNS format (CFD General Notation System)(http://cgns.sourceforge.net/)

Output formatDX (http://www.opendx.org/)VTK Visualization Toolkit (http://www.vtk.org/)In this case the results can be visualized for example with

VISIT (https://wci.llnl.gov/codes/visit/)ParaView (http://www.paraview.org/)



Examples


Vector, Face and Cell class

UML diagram of Vector, Face and Cell classes



Examples


The vector class I

module class_vectorprivate ! Defaultpublic :: vector ! Classpublic :: vector_, alloc_vector ! Constructorpublic :: free_vector ! Destructorpublic :: x_, y_, z_ ! Getterspublic :: operator(+), operator(-), &

operator(*) , ... ! Vector Algebra

type vectorprivatereal(kind(1.d0)) :: xreal(kind(1.d0)) :: yreal(kind(1.d0)) :: z

end type vectorMarco Rorro NEMO: Numerical Engine for Multiphysics Operators


Examples


The vector class II

interface operator(+)module procedure vec_sum

end interface...containselemental function vec_sum(a,b)type(vector) :: vec_sumtype(vector), intent(in) :: a, bvec_sum%x = a%x + b%xvec_sum%y = a%y + b%yvec_sum%z = a%z + b%z

end function vec_sum...end module class_vector



Examples


The mesh class

UML diagram of the mesh class



Examples


The field class

UML diagram of the field class



Examples


The PDE class

UML diagram of the PDE class



Examples


Operators

Operators acting on Field ObjectsGradient of scalar and vector fieldsDivergence of vector field

Operators acting on PDEsTime derivative of scalar and vector variablesLaplacian (div(Γgradφ)) of scalar and vector variablesDivergence of scalar variables and vector variablesScalar Source



Examples


subroutine scalar_pde_div(sign,pde,phi,phi2,flux,...)

...! Gets local to global conversion listcall psb_get_loc_to_glob(msh%desc_c,iloc_to_glob)

...call get_ith_conn(if2b,msh%f2b,0)insert_fluid: do

...! Local indicesi = i + 1if = if2b(i)im = master_(msh%faces(if))is = slave_(msh%faces(if))! Global indicesim_glob = iloc_to_glob(im)is_glob = iloc_to_glob(is)



Examples


subroutine scalar_pde_div(sign,pde,phi,phi2,flux,...)

coeff_a(1) = fact * (1-w); coeff_a(2) = fact * w! Diagonal coefficients ! MasterA(ka) = coeff_a(1)irow_a(ka) = im_glob; icol_a(ka) = im_glob! SlaveA(ka+1) = coeff_a(2)irow_a(ka+1) = im_glob; icol_a(ka+1) = is_glob! Off-diagonal coefficients ! Master rowA(ka+2) = -coeff_a(1);irow_a(ka+2) = is_glob; icol_a(ka+2) = is_glob! Slave rowA(ka+3) = -coeff_a(2)irow_a(ka+3) = is_glob; icol_a(ka+3) = im_glob

end do block_fluid_cdcall spins_pde(nel,irow_a,icol_a,A,pde)...

end do insert_fluidMarco Rorro NEMO: Numerical Engine for Multiphysics Operators


Examples

Heat transferConvection-diffusionIncompressible Navier-Stokes equations

div(λgradT) + ST = 0

- Number of cells: 6813804- Number of faces: 13696956- Number of vertices: 1169649

BC1 and BC2: adiabatic wall (∂T/∂n = 0) BC3: wall (T = 300)



Examples


Heat transfer program (variables declaration)

program steady_conductionuse tools_nemocharacter(len=30), parameter :: input_file = ’nemo.inp’type(mesh) :: mshtype(material) :: mattype(bc_poly), allocatable :: bc(:)type(scalar_field) :: T, lambdatype(scalar_pde) :: eqnTtype(scalar_source) :: srctype(iterating) :: itertype(output) :: outreal(kind(1.d0)) :: var



Examples


Heat transfer program (variables setup)

call start_psblas

call create_mesh(msh,input_file,’MESH’)call create_material(mat,input_file,’MATERIAL’)call create_bc(bc,input_file,’BC’,msh)

call create_field(T,msh,bc=bc,mat=mat,dim=temperature_)call create_field(lambda,msh,bc=bc,mat=mat,&

dim=conductivity_,on_faces=.true.)

call create_pde(eqnT,input_file,’Energy PDE’,msh,power_)call create_source(src,input_file,’SOURCE’,power_ / volume_)call create_iterating(iter,input_file,&

’BIG ITERATIONS’,it_convergence_)

call create_output(out,input_file,’OUTPUT’)



Examples


Heat transfer program (PDE solving)

big_iteration: docall increment(iter)if(mypnum_() == 0) write(*,100) current_iteration(iter)

call pde_laplacian(’+’,eqnT,lambda,T)call pde_source(’+’,eqnT,src)

call solve_pde(eqnT,T,var=var)

call update_field(lambda,T)

if(stop_iterating(iter,var)) exit big_iteration

end do big_iteration



Examples


Heat transfer

Execution time: 2728 s. 1 core, 810 8 cores on one node of thematrix cluster at CASPUR



Examples


Heat transfer

BC1: fixed flux (∂T/∂n = −1.6d4), BC2: fixed temp (T = 510), BC3: fixed temp (T = 360), BC4 and BC5:adiabatic (∂T/∂n = 0), BC6: fixed flux (∂T/∂n = −0.6d4), BC7: fixed temp (T = 540).

Nubmer of cells: 416469, Number of faces: 857563, Number of vertices: 81738



Examples


Heat transfer

Execution time: 76 s. on my laptopMarco Rorro NEMO: Numerical Engine for Multiphysics Operators


Examples


Convection-Diffusion

div(vφ) + div(Γgradφ) = 0

Geometry and boundary conditions for the scalar transport in a stagnation point flow



Examples


Convection-Diffusion

Γ = 0.01 Γ = 0.001



Examples


Navier-Stokes equations for an incompressible flow

div(v) = 0 continuity eq.∂v∂t

+ div(vv) = div(νgradv)− 1ρ

gradp momentum eq.

The second equation in Cartesian form reads:

∂ui

∂t+ div(uiv) = div(νgradui)−

1ρ

divpii

where ui (i = 1,2,3) are the Cartesian component of thevelocity vector v and ν = µ/ρ is the kinematic viscosity(dynamic viscosity / density).



Examples


Implicit pressure-correction methods

Take the divergence of the momentum equation:

div(

1ρ

gradp)

= −div[∂v∂t

+ div(vv − νgradv)

]exploit the continuity equation to get:

div(

1ρ

gradp)

= −div [div(vv)]

In Cartesian coordinates this equation reads:

∂

∂xi

(1ρ

∂p∂xi

)= − ∂

∂xi

(∂

∂xj(uiuj)

)



Examples


Semi-Implicit Method for Pressure-Linked Equations

Let um∗ the solution of the momentum eq. with p = pm−1:

AuiP um∗

i +∑

n

Auin um∗

i,n = Bm−1ui−(

1ρ

∂pm−1

∂xi

)P

Correction: umi = um∗

i + u′ and pm = pm−1 + p′

substitute this expressions in the momentum equation to get:

u′i,P =1

AuiP

[1ρ

(∂p′

∂xi

)P−∑

n

Anu′i,n

]apply the continuity eq. to um, by neglecting divu′ get:

∂

∂xi

(1

AuiP

1ρ

∂p′

∂xi

)=∂um∗

i∂xi



Examples


Lid-driven cavity

pressure for Re = vL ∗ L/ν = 1000 velocity magnitude for Re = vL ∗ L/ν = 1000



Examples


Lid-driven cavity movie



Examples

Future work

Test the SIMPLE schemeImplement other pressure correction schemes, like PISOAdd turbolent models


Documents

A Fortran 95 object-oriented parallel code for the ...Introduction NEMO main features Examples A Fortran 95 object-oriented parallel code for the numerical solution of PDEs NEMO (Numerical