Overview of the ACTS Toolkit For NERSC Users Brent Milne John Wu [email protected]

Overview of the ACTS ToolkitFor NERSC Users

Brent Milne

John Wu

[email protected]

http://acts.nersc.gov/

What is the ACTS Toolkit?

• Advanced Computational Testing and Simulation• Part of the DOE 2000 Project

• Will likely shift to base DOE funding

• Tools (roughly 20) for developing parallel applications• Developed (primarily) at DOE Labs

• Separate projects originally

• ACTS is an “umbrella” project• Collecting tools

• Leveraging numerous independently funded projects


ACTS Project Goals

• Bringing tools together into a “Toolkit”

• Making the tools interoperable

• Providing consistent application interfaces

• Promoting general (not application specific) solutions to parallel programming needs

• Encouraging code re-use

• Impacting DOE science• ASCI, IT2(SSP)

• Enabling large scale applications• example follows: Oil Reservoir Simulation


Prometheus

• Multigrid solver

• Unstructured meshes in solid mechanics

• Modified matrix graphs

• 26 million DOF, 640 PEs

• Use PETSc

http://www.cs.berkeley.edu/~madams/prometheus


NERSC Activities

• Make ACTS tools available on NERSC platforms

• Provide technical support

• Find new ER users who can benefit from the tools

• Work with users to integrate tools into applications

• Evaluate the tools

• Create ACTS information center (online)

ACTS Information Online

• One-stop Shopping • But coordinated and integrated with developer resources.

• User-oriented Information• Which tool should I use to solve my problem?

• Which tools are ready for prime time?

• Reviews: strengths, weaknesses, caveats, etc.

http://http://acts.nersc.gov/acts.nersc.gov/

ACTS Support at NERSC

• Support for application development

• Technical support

• Tool installation (by request)

• Leverage with developers

• Support is also available through traditional NERSC channels (e.g. [email protected]).

[email protected]


Risk

• ACTS Tools are still research projects• Can’t guarantee indefinite support

• Can’t guarantee fixed interface

• But commercial software doesn’t either

• A NERSC goal is to minimize risk to users• Provide unbiased advice

• Promote tools that work

• Support tools with the best chance of surviving

• Be an advocate for users


Tool Catagorization

• Numerical• Libraries/data structures that implement numerical algorithms

• I.e. your “classic” numerical packages

• PETSc, Aztec, Hypre, ScaLAPACK, SuperLU, PVODE, Opt++

• Structural (Frameworks)• Libraries/data structures that manage data, communicate

• Global Arrays, Overture, POOMA , Cumulvs, PAWS, InDEPS

• Infrastructural• TAU (performance analysis)

• Globus (meta-computing)

• PADRE, Nexus, Tulip (middleware -- mostly used by other tools)


Tool Status at NERSC

• Installed on the T3E and fully supported by NERSC:• Aztec

• PETSc

• ScaLAPACK

• TAU

• Known to have been successfully used on NERSC’s T3E:• CUMULVS

• Global Arrays

• Globus

• POOMA

• PVODE

Tools available on T3E


ScaLAPACK

• Installed on the T3E

• Parallel version of LAPACK

• Well known, trusted

• Basic dense linear algebra (BLAS)

• Direct solution of linear systems / Factorization• General, tri-diagonal, banded

• Dense matrix eigensolvers• Sustained 605 GFLOPS on ASCI Red in materials simulation

• Get started: “man scalapack” or “module help scalapack”


PETSc

• Installed on the T3E

• Functionality for solving PDEs in parallel

• The most widely used and well supported member of the ACTS toolkit

• Runs on serial and parallel machines (even NT)

• C and Fortran Programming Interfaces

• Modular, objected design methodology• Methodology extensible to User code

• Get started: “module help petsc” or “man petsc”


Aztec

• Installed on T3E

• Solving sparse linear systems on distributed machines

• Highly efficient and scalable• Some Apps have successfully scaled to 1000s of processors

• Krylov iterative methods:• CG, CGS, Bi-CG-Stab, GMRES, TFQMR

• Large preconditioning suite including• Jacobi, Gauss-Seidel, overlapping domain decomps (ILU et al)

• Get started: “module help aztec” or “man aztec”


Aztec

• User input matrix in DMSR or DVBR format

• Aztec sets up its own data structure for solving linear systems

• Used by groups of dedicated users• Salinas

• MPSalsa

• COYOTE II

• TOUGH

• ...

http://endo.sandia.gov/BB/comp_str_dyn/mp-fema.html


Using Aztec: basic steps

• Prepare your linear system• distribute the matrix

• call AZ_transform

• set up right-hand side and initial guess

• call AZ_reorder_vec on initial guess and right-hand side

• selective an iterative solver and a preconditioner

• call AZ_solve

• call AZ_invorder_vec on solution


Using Aztec: DMSR

PE 0

PE 1

PE 2

•local index to global index map

•ja: ja[0:nloc] - starting positionsja[nloc+1:] - global column indices

•valval[0:nloc-1] diagonal elementsval[nloc] not usedval[nloc+1:] off-diagonal nonzeros


Using Aztec

• Two re-ordering:• global re-ordering to distribute the matrix (done by user)

• local re-ordering to allow maximum overlapping of communication and computation (done by AZ_transform)

• Part of the right-hand side (elements beyond nloc) is used as scratch space internally

• more information at http://acts.nersc.gov/aztec/evaluation.html

• examples to look at• /usr/local/pkg/Aztec/Aztec-2.0/app

• /u1/kewu/azmm.tar


TAU

• Currently installed on the T3E

• Profiling of C++/C and Fortran programs

• Detailed information - much more than prof/gprof

• For C++: per-class and per-instance profiling

• Graphical display of profiling results• Built-in viewers, interface to VAMPIR

• Automatic instrumentation in the future

• Get started: “module help tau” or “man tau”


TAU Screen Shot


Using TAU: instrumentation

Template <class T>

class Templated {

private:

T Data;

public:

T SetData (T d) {

Data = d;

return Data;

}

}

TAU_TYPE_STRING(str, CT(Data)+“(“+CT(d)+“)”);TAU_PROFILE(“Templated::SetData()”, str, TAU_DEFAULT);


Using TAU: compilation

• Load module tau to define TAUROOTDIR

• include one of the Makefiles from ${TAUROOTDIR}/t3e/lib to define TAU_DEFS and TAULIBS

• Add ${TAULIBS} to link command

• Examples at ${TAUROOTDIR}/examples

• More help at http://acts.nersc.gov/tau/at-nersc.html


Using TAU: view traces

• Must use VAMPIR on T3E

• To turn TAU trace files to VAMPIR trace file• merge all TAU trace files into one (tau_merge)

• convert merged file with tau_convert -vampir

• see Makefile of examples in ${TAUROOTDIR}/examples

• To use VAMPIR• module load vampir

• vampir tracefile.pv

Tools not on T3E or under development


SuperLU

• Direct solution of large sparse linear systems

• Single processor and multithread versions are available http://www.nersc.gov/~xiaoye

• Distributed memory (MPI) version under development

• Attained 10 GFLOPS (speedup 100) on 512 node T3E• 2-D matrix distribution

• static pivoting (replace small pivot with larger value)

• iterative refinement to ensure accuracy


SuperLU

http://www.nersc.gov/~xiaoye/SuperLU


Hypre

• Family of problem specific preconditioners

• Physics and grid based• Algebraic and structured multigrids

• Sparse approximate inverse

• Will be becoming available for use soon

• Should be useful stand-alone

• …and in the context of other ACTS tools

• http://www.llnl.gov/CASC/linear_solvers/


PVODE

• Solves ODE & DAE• Adams multi-step for non-stiff

problems

• BDF multi-step for stiff problems

• Can use linear solvers from ACTS• Integration with PETSc planned

• Now includes non-linear solvers (KINSOL)• Newton based solvers

• Globalization strategies (line search, trust region)

Http://www.llnl.gov/CASC/PVODE


ATLAS and PHiPAC

• Automatic generation of machine optimized BLAS

• Can beat hand-tuned performance


Global Arrays

• “Shared memory” programming interface on distributed memory computers• But doesn’t hide remote access characteristics

• Global array object abstraction

• One-sided communication

• Used as infrastructure for several computational chemistry packages (e.g. NWChem on the NERSC T3E)


Global Arrays

http://www.emsl.pnl.gov/pub/proj/tms/hpcc_actinides


Overture

• Complex geometry

• C++ Class Library

• Overlapping composite grids


POOMA

• C++ framework (class library) for high performance parallel computing

• Abstractions for arrays, grids, particles

• Allows application to be written with high level “data-parallel” style

• T3E specific difficulties have apparently been overcome• Extensive expression templates overwhelmed compilers


CUMULVS

• Simple mechanisms for• Computational steering

• Interactive visualization

• Fault tolerance

• Handles, for multiple attached viewers,• Data collection, distribution

• Computational parameter control (“steering”)

• Can be used in heterogeneous environment (layered on PVM)


Globus

• Provides many services to enable development of meta-computing applications:• Resource discovery, monitoring, allocation

• Process (and IO) staging, management

• Communication - varied protocols - “Nexus”

• User and resource authentication, security (RSA)

• Hides implementation details from application• Provides common “middleware” layer of interfaces

• Transparent remote access to resources• User authentication, File staging, Batch queuing, etc.,

through common interfaces

• Perhaps of greatest immediate interest to application programmer


PAWS

• Distributing data between separate parallel applications

• Coupling parallel simulation to parallel visualization tools that• require different parallel data distributions

• run on same machine

• ...or run across heterogeneous machines

• Minimal application coding requirements

• PAWS “Controller” handles interactions, data mapping


PAWS Screen Shot

Controller

Sim. Code

Viz. Code


Future Directions

• Comparisons of parallel linear system solvers• AZTEC

• BLOCKSOLVE

• ISIS++

• PETSc

• PSPARSLIB

• Test problems, anyone ?


Future Directions

• CCA (Common Component Architecture)• Developing standardized ways of managing numerical

“components” to allow mixing-and-matching

• Frameworks for gluing components together

• Similarities to CORBA, DCOM, Java Beans

• But:

• Scientific Interface Description Language (allowing Fortran)

• Designed for zero-copy interconnections

• ESI (Equation Solver Interface)• Developing standardized interfaces for scalable linear algebra

• Specific test case for CCA component design


Conclusion

• DOE has funded a nice set of tools

• We are here to maintain them for you

Any questions?

http://http://acts.nersc.gov/acts.nersc.gov/[email protected]

Documents

Overview of the ACTS Toolkit For NERSC Users Brent Milne John Wu [email protected]