1

HPC Middleware on GRIDHPC Middleware on GRID… … as a material for discussion of as a material for discussion of

WG5WG5

GeoFEM/RISTGeoFEM/RISTAugust 2nd, 2001, ACES/GEM at MHPCCAugust 2nd, 2001, ACES/GEM at MHPCC

Kihei, Maui, HawaiiKihei, Maui, Hawaii

2

BackgroundBackground• Various Types of HPC Platforms– MPP, VPP– PC Clusters, Distributed Parallel MPPs, SMP Clusters– 8-Way SMP, 16-Way SMP, 256-Way SMP– Power, HP-RISC, Alpha/Itanium, Pentium, Vector PE• Parallel/Single PE Optimization is Important Issue for Efficiency– Everyone knows that ... but it's a big task especially for application experts such as geophysics people in ACES community.– Machine dependent optimization/tuning required.• Simulation Methods such as FEM/FDM/BEM/LSM/DEM etc. have Typical Processes for Computation.• How about "Hiding" these Processes from Users ?– code development : efficient, reliable, portable, maintenance-free• line number of the source codes will be reduced– accelerates advancement of the applications (= physics)

3

Background (cont.)Background (cont.)• Current GeoFEM provides this environment

– limited to FEM– not necessarily perfect

• GRID as next generation HPC infrastructure– Currently, middlewares and protocols are being developed t

o enable unified interface to treat various OS, computers, ultra-speed network and database.

– What are expected to GRID ?• Meta-computing : simultaneous use of supercomputers in the world• Volunteer-computing : efficient use of idling computers• Access Grid : research collaboration environment• Data Intensive Computing : computation with large-scale data• Grid ASP : application services on WEB

4

Similar Research GroupsSimilar Research Groups• ALICEALICE （（ ANLANL ））

• CCAforumCCAforum （（ Common Component ArchitectureCommon Component Architecture,,DOEDOE ））• DOE/ASCI/Distributed Computing Research TeamDOE/ASCI/Distributed Computing Research Team

– ESI(Equation Solver Interface Standards)ESI(Equation Solver Interface Standards)– FEI(The Finite Element/Equation Solver Interface Specification)FEI(The Finite Element/Equation Solver Interface Specification)

• ADRADR （（ Active Data RepositoryActive Data Repository ）（）（ NPACINPACI ））

5

Are they successful ? Are they successful ? It seems NOIt seems NO

• Very limited targets, processesVery limited targets, processes– Mainly for Optimization of Linear SolversMainly for Optimization of Linear Solvers

• Where are Interfaces between Applications and Libraries ?Where are Interfaces between Applications and Libraries ?– Approach from Computer/Computational Science PeopleApproach from Computer/Computational Science People– Not Really Easy to Use by Application PeopleNot Really Easy to Use by Application People

-Linear solvers -Numerical Algorithms -Parallel Programming -Optimization

Computer/Computational

Science

-FEM -FDM -Spectral Methods -MD, MC -BEM

Applications

6

Example of HPC Middleware Example of HPC Middleware (1)(1)

Simulation Methods include Some Simulation Methods include Some Typical ProcessesTypical Processes

Sparse Mat. Mult.Sparse Mat. Mult.

Nonlinear ProcedureNonlinear Procedure

FFTFFT

Eward TermsEward Terms

O(N) Ab Initio MDO(N) Ab Initio MD

7Example of HPC Middleware Example of HPC Middleware (2)(2)

Individual Process could be optimized Individual Process could be optimized for Various Types of MPP for Various Types of MPP

ArchitecturesArchitectures

Sparse Mat. Mult.

Nonlinear Proc.

FFT

Eward Terms

O(N) Ab Initio MD MPP-A

MPP-B

MPP-C

8

Sparse Matrix Mult.Sparse Matrix Mult.

Nonlinear Proc.Nonlinear Proc.

FFTFFT

Eward TermsEward Terms

Example of HPC Middleware Example of HPC Middleware (3)(3)

Use Optimized LibrariesUse Optimized Libraries

OO(N)(N)

abab--

initioinitio

MMDD

Sparse Matrix Mult.Sparse Matrix Mult.

Nonlinear Proc.Nonlinear Proc.

FFTFFT

Eward TermsEward Terms

Sparse Matrix Mult.Sparse Matrix Mult.

Nonlinear Alg.Nonlinear Alg.

FFTFFT

Eward TermsEward Terms

Sparse Matrix Mult.Sparse Matrix Mult.

Nonlinear Proc.Nonlinear Proc.

FFTFFT

Eward TermsEward Terms

9Example of HPC Middleware Example of HPC Middleware (4)(4)- Optimized code is generated by special language/ - Optimized code is generated by special language/ compiler based on analysis data and H/W compiler based on analysis data and H/W information.information.- Optimum algorithm can be adopted- Optimum algorithm can be adopted

OO(N)(N)

abab--

initioinitio

MMDD

MPP-AMPP-A

MPP-BMPP-B

MPP-CMPP-C

Data for Analysis Model

ParametersParametersof H/Wof H/W

SpecialSpecialCompilerCompiler

Sparse Matrix Mult.Sparse Matrix Mult.

Nonlinear Proc.Nonlinear Proc.

FFTFFT

Eward TermsEward Terms

10Example of HPC Middleware Example of HPC Middleware (5)(5)- On network-connected H/W's (meta-computing)- On network-connected H/W's (meta-computing)- Optimized for individual architecture- Optimized for individual architecture- Optimum load-balancing- Optimum load-balancing

OO(N)(N)

abab--

initioinitio

MMDD

analysisanalysismodelmodelspacespace

11Example of HPC Middleware Example of HPC Middleware (6)(6)

Multi Module Coupling through Multi Module Coupling through PlatformPlatform

HPC Platform/MiddlewareHPC Platform/Middleware

Ab-Initio MDAb-Initio MD Classical MDClassical MD FEMFEM

HPC Platform/MiddlewareHPC Platform/Middleware

Ab-Initio MDAb-Initio MD Classical MDClassical MD FEMFEM

ModelingModeling VisualizationVisualization LoadLoadBalancingBalancing

ResourceResourceManagementManagementOptimizationOptimizationDataData

AssimilationAssimilation

12

PETAFLOPS on GRIDPETAFLOPS on GRIDfrom GeoFEM's Point of Viewfrom GeoFEM's Point of View

• Why? When?Why? When?– Datasets (mesh, observation, result) Datasets (mesh, observation, result)

could be distributed.could be distributed.– Problem size could be too large for sProblem size could be too large for s

ingle MPP system.ingle MPP system.• according to G.C.Fox, according to G.C.Fox, (TOP500) is (TOP500) is

about 100 TFLOPS now ...about 100 TFLOPS now ...

• LegionLegion– Prof.Grimshaw (U.Virginia)Prof.Grimshaw (U.Virginia)– Grid OS, Global OSGrid OS, Global OS– Can handle MPP's connected througCan handle MPP's connected throug

h network as one huge MPP (= Supeh network as one huge MPP (= Super MPP)r MPP)

MPP-AMPP-A

MPP-BMPP-B

MPP-CMPP-C

– Optimization on Individual Optimization on Individual Architecture (H/W)Architecture (H/W)

– Load balancing according to Load balancing according to machine performance and machine performance and resource availabilityresource availability

13

PETAFLOPS on GRID (cont.)PETAFLOPS on GRID (cont.)

• GRID + (OS) + HPC MW/PFGRID + (OS) + HPC MW/PF• Environment for "Electronic CollaborationEnvironment for "Electronic Collaboration

14

15

"Parallel" FEM Procedure"Parallel" FEM Procedure

Initial Mesh DataInitial Mesh Data

PartitioningPartitioning

Post ProcPost Proc..Data Input/OutputData Input/Output

Domain Specific Domain Specific Algorithms/ModelsAlgorithms/Models

Matrix AssembleMatrix Assemble

Linear SolversLinear Solvers

VisualizationVisualization

Pre-ProcessingPre-Processing MainMain Post-ProcessingPost-Processing