December 9, 2008
Centre desMatériauxP.-M. Fourt
Post-seismic relaxation process in southeastern Asia(my first steps as a user of Z-Set in parallel)
Jean-Didier GaraudClub ZéBuLoN
Club ZéBuLoN
2
Centre desMatériauxP.-M. Fourt
Outline
Sumatra, December 26th 2004
General presentation
2D models
Parallel approach
Meshing & Splitting
BCs & MPCs
Linear solvers
Computations
Visualization
Conclusion
Club ZéBuLoN
3
Centre desMatériauxP.-M. Fourt
Context & objectives
ANR between CdM and ENS-géologie (Luce Fleitout & Christophe Vigny)
Fit finite element model to GPS data
Large computations (>106 nodes)
coco @ CdM (112x2 cores)
jade @ cines (1536x8 cores)
Club ZéBuLoN
4
Centre desMatériauxP.-M. Fourt
Sumatra, December 26th 2004
Club ZéBuLoN
5
Centre desMatériauxP.-M. Fourt
Sumatra, December 26th 2004
Giant earthquake, mag>9, causing the Tsunami
Epicenter ~200 km away from the coast
GPS measures:
27 cm at Phuket (500 km away)
1 cm at Chengdu (3000 km away)
Club ZéBuLoN
6
Centre desMatériauxP.-M. Fourt
Available data
Seismic cartography [Engdhal]
Subduction & transform faults
Club ZéBuLoN
7
Centre desMatériauxP.-M. Fourt
Available data
~60 GPS measurements [Vigny & al. 2005]
Co-seismic displacement
Inverse, elasticcomputation
Club ZéBuLoN
8
Centre desMatériauxP.-M. Fourt
Outline
Sumatra, December 26th 2004
General presentation
2D models
Parallel approach
Meshing & Splitting
BCs & MPCs
Linear solvers
Computations
Visualization
Conclusion
Club ZéBuLoN
9
Centre desMatériauxP.-M. Fourt
Material model
Elastic lithosphere (0►-80 km)
E = 400 GPa
= 0.3
Visco-elastic asthenosphere (-80►-1500 km)
E = 400 GPa
= 0.3
n = 1
K = 1019 -> 1022 Pa.s (depends on temperature & depth)
Club ZéBuLoN
10
Centre desMatériauxP.-M. Fourt
Geometry
Elsets :
Crust
Mantle
Serpentine
MPC
1500 km
4000 km
Club ZéBuLoN
11
Centre desMatériauxP.-M. Fourt
Input earthquake
Through MPC-2
Impose co-seismic displacement
Debonding elements (cohesive zone)
Gravity
Initial stress: gh
Club ZéBuLoN
12
Centre desMatériauxP.-M. Fourt
Results @3yr for different material parameters
Drift:40 cm
Club ZéBuLoN
13
Centre desMatériauxP.-M. Fourt
Outline
Sumatra, December 26th 2004
General presentation
2D models
Parallel approach
Meshing & Splitting
BCs & MPCs
Linear solvers
Computations
Visualization
Conclusion
Club ZéBuLoN
14
Centre desMatériauxP.-M. Fourt
Parallel computation: step-by-step
1- mesh & split
2- compute
3- glue
4- visu & post
Club ZéBuLoN
15
Centre desMatériauxP.-M. Fourt
3D meshing
Engdhal data too noisy
Extension along fault line
**xyz_to_r
**extension_along_nset_jd
**por
cupin
e
Club ZéBuLoN
16
Centre desMatériauxP.-M. Fourt
3D meshing: difference
On a flat Earth
“Hand-made” boolean operation
Yams
Club ZéBuLoN
17
Centre desMatériauxP.-M. Fourt
3D meshing: difference
Special care at -80km such that the union process goes fine
Yams 3d gives unsatisfying results
Laplacian on z-coordinate
Club ZéBuLoN
18
Centre desMatériauxP.-M. Fourt
3D meshing: union
Club ZéBuLoN
19
Centre desMatériauxP.-M. Fourt
3D meshing: ghs3d
c Requires 32 Gb RAM
ghs3d
Club ZéBuLoN
20
Centre desMatériauxP.-M. Fourt
Mesh splitting
Splitmesh
**split *domains 60 *mincon 3
**rigid_body *nset est est est ouest *dof U1 U2 U3 U1
Domain 36
Forbids insufficiently attached elements
Detects rigid body movements caused by split& creates the ***bc bloc in .cut file
Club ZéBuLoN
21
Centre desMatériauxP.-M. Fourt
Mesh splitting
Metis split (**metis_split)
Optimized mode is not reproducible !!!
**metis_split *domains 60 *check_domains
**rigid_body *nset est est est ouest *dof U1 U2 U3 U1
Club ZéBuLoN
22
Centre desMatériauxP.-M. Fourt
Mesh splitting
Split by elset ?
Rigid body movements ?
Load balancing ?
**elset_split *domain lithosphere *domain asthenosphere *domain serpentine *domain subduction
Club ZéBuLoN
23
Centre desMatériauxP.-M. Fourt
Outline
Sumatra, December 26th 2004
General presentation
2D models
Parallel approach
Meshing & Splitting
BCs & MPCs
Linear solvers
Computations
Visualization
Conclusion
Club ZéBuLoN
24
Centre desMatériauxP.-M. Fourt
BCs & MPCs
Local / not problematic:
**impose_nodal_dof
**gravity
Problematic:
**mpc2 currently requires the global mesh (old format)roundabout if coupled dofs are in the same subdomain
Club ZéBuLoN
25
Centre desMatériauxP.-M. Fourt
Outline
Sumatra, December 26th 2004
General presentation
2D models
Parallel approach
Meshing & Splitting
BCs & MPCs
Linear solvers
Computations
Visualization
Conclusion
Club ZéBuLoN
26
Centre desMatériauxP.-M. Fourt
Linear solvers
***linear_solver rigid **n_last 15 **local_solver sparse_dscpack
***linear_solver sparse_dscpack ***linear_solver sparse_direct
***linear_solver dissection **ldlt
Detects RBMSlower (x3)Memory consuming
Doesn’t detect rigid body movements (RBM)
Detects RBMStill in development
Detects RBMSlowerUnnecessary after 1st iteration
Club ZéBuLoN
27
Centre desMatériauxP.-M. Fourt
FETI solver
Global solver on the interface
Iterative (Conjugate Gradient)
Depends on conditioning number
Mesh quality
Splitting
Material heterogeneity
MPCs
...
Club ZéBuLoN
28
Centre desMatériauxP.-M. Fourt
Outline
Sumatra, December 26th 2004
General presentation
2D models
Parallel approach
Meshing & Splitting
BCs & MPCs
Linear solvers
Computations
Visualization
Conclusion
Club ZéBuLoN
29
Centre desMatériauxP.-M. Fourt
Elastic computations (fake)
To test the convergence / mesh / inp
Simple traction, 16 domains
FETI converges in
296 iterations (dscpack + mesher_rigid)
403 iterations (sparse_direct)
Pb with dissection (segfault in mkl, during 1st FETI iteration)
***feti **precond lumped **projector direct **reprojection **max_iteration 1000 **precision 1.e-9
**precond dirichlet doesn’t work (?)
Club ZéBuLoN
30
Centre desMatériauxP.-M. Fourt
Visco-elastic computations (fake)
Visco-elastic behavior in asthenosphere
Simple traction, 16 domains
FETI converges in
~500 it per increment (dscpack)
Club ZéBuLoN
31
Centre desMatériauxP.-M. Fourt
Visco-elastic computation
Realistic BCs
fixed outer boundaries
Earthquake localized in 1 sub-domain
Cohesive Zone Elements
**mpc2
Club ZéBuLoN
32
Centre desMatériauxP.-M. Fourt
Visco-elastic computation
Attempt 1: dscpack / 60+ domains
FETI ratio doesn’t decrease
Club ZéBuLoN
33
Centre desMatériauxP.-M. Fourt
Visco-elastic computation
Attempt 1: dscpack / 60+ domains
FETI ratio doesn’t decrease
Attempt 2: dscpack / 60 domains / handmade .cut
FETI ratio doesn’t decrease
Matrix becomes singular at iter 5 in domain 30 ( ?? )
Club ZéBuLoN
34
Centre desMatériauxP.-M. Fourt
Visco-elastic computation
Attempt 1: dscpack / 60+ domains
FETI ratio doesn’t decrease
Attempt 2: dscpack / 60 domains / handmade .cut
FETI ratio doesn’t decrease
Matrix becomes singular at iter 5 in domain 30 ( ?? )
Attempt 3: dissection / 60 domains
std::bad_alloc
Club ZéBuLoN
35
Centre desMatériauxP.-M. Fourt
Outline
Sumatra, December 26th 2004
General presentation
2D models
Parallel approach
Meshing & Splitting
BCs & MPCs
Linear solvers
Computations
Visualization
Conclusion
Club ZéBuLoN
36
Centre desMatériauxP.-M. Fourt
Visualization: Zglue
Most commonly used
Main drawbacks
Large output files
Zmaster becomes memory consuming
Club ZéBuLoN
37
Centre desMatériauxP.-M. Fourt
Visualization: Zmaster -PP
xhost +
mpirun -np 1 -hostfile mymachines Zrun -PP -s MPI mpi2 -s Parallel.Mpmd.RunMode -G -s Parallel.Display x3755 mycalcul.inp
Zmaster in parallel mode
Rendering is done on computation nodes
No longer requires the glue process
Experimental (?)
See SQ
Club ZéBuLoN
38
Centre desMatériauxP.-M. Fourt
Visualization: Paraview
Paraview
open source equivalent of Ensight
Vtk based
Features
Cut, probe, calculator
~100 other filters
Visualization by sub-domain(s)
Misses Zmaster’s posts & plots
***output Z7 [**other options]
***output ensight_gold
Club ZéBuLoN
39
Centre desMatériauxP.-M. Fourt
Outline
Sumatra, December 26th 2004
General presentation
2D models
Parallel approach
Meshing & Splitting
BCs & MPCs
Linear solvers
Computations
Visualization
Conclusion
Club ZéBuLoN
40
Centre desMatériauxP.-M. Fourt
Conclusion
Parallel computation is a routine on simple cases (elastic / small size)
Becomes difficult when:
Large size (> 50 sub-domains)
Non linearities
Mesh & split quality
Source(s) of problem is not clear
User-time consuming !
Club ZéBuLoN
41
Centre desMatériauxP.-M. Fourt
Useful (?) stuff
Zman parallel: troubleshooting
Obscure error messages
***verify_parallel_splitting
Error handler
Attaches a gdb to a dying process
Zrun -s Error.HandlingPolicy auto [...]
// MPMD - Waiting for ACK: 1 2 3 4 5 6 7 8 9 10 [stuck] Output directory doesn’t exist
// MPMD - 4 tasks spawned on sisulu. [stuck] export PVM_ARCH=LINUX