Terascale Simulation Tools and

Technologies Center

Jim Glimm (BNL/SB), Center Director

David Brown (LLNL), Ed D’Azevedo (ORNL), Joe Flaherty (RPI), Lori Freitag (ANL),

Patrick Knupp (SNL), Mark Shephard (RPI), Harold Trease (PNNL), Co-PIs

TSTT-2

The TSTT Center will bring terascale simulation technology to applications

scientists Observation: Terascale computing will enable high-

fidelity calculations based on multiple coupled physical processes and multiple physical scales Adaptive methods Composite or hybrid solution strategies High-Order discretization strategies

Barrier: The lack of easy-to-use interoperable meshing, discretization, and adaptive tools requires too much software expertise by application scientists

The TSTT recognizes this gap and will address the technical and human barriers preventing use of

adaptive, composite, hybrid methods

TSTT-3

TSTT will develop interoperable meshing and discretization technology components

Meshing and Discretization Research and Development high-quality, hybrid mesh generation for complex domains front tracking and other adaptive approaches high-order discretization techniques algorithms for terascale computing

Software interoperability is a pervading theme initial design will account for interoperability at all levels encapsulate research into software components define interfaces for plug-and-play experimentation

Application deployment and testing is paramount SciDAC collaborations in accelerator design, fusion,

climate and chemically reacting flows existing DOE application collaborations in biology, mixing

fluids, and many more

TSTT-4

Existing Tools for Mesh Generation

A wide variety of tools exist for thegeneration of …

… structured meshes Overture - high quality predominantly structured

meshes on complex CAD geometries (LLNL) Variational and Elliptic Grid Generators (ORNL, SNL)

… unstructured meshes MEGA (RPI) - primarily tetrahedral meshes, boundary

layer mesh generation, curved elements, AMR CUBIT (SNL) - primarily hexahedral meshes, automatic decomposition tools, common geometry module NWGrid (PNNL) - hybrid meshes using combined Delaunay, AMR and block structured algorithms

These tools all meet particular needs, but they do not interoperate to form hybrid, composite meshes

MEGAMEGA Boundary Layer Boundary Layer Mesh (RPI)Mesh (RPI)

OvertureOverture Diesel Diesel Engine Mesh (LLNL)Engine Mesh (LLNL)

TSTT-5

Geometric Hierarchy

Required to provide a common frame of

reference for all tools facilitate multilevel solvers facilitate transfer of

information in discretizations Level 0: Original problem

specification via high level geometric description

Level 1/2: Decomposition into subdomains and mesh components that refer back to Level 0

Level 3: Partitioning

Given GeometrySpecification

Domain Decomposition

Mesh Components

ParallelDecomposition

Level 3

Level 0

Level 2

Level 1

P0

P2 P3

P1 P4

P6 P7

P5 P8

Pa Pb

P9 Pc

Pe Pf

Pd

TSTT-6

Mesh Data Hierarchy Level A: Geometric

description of the domain Accessed via tools such as

CGM (SNL) or functional interfaces to solid modeling kernels (RPI)

Level B: Full geometry hybrid meshes mesh components communication mechanisms

that link them (key new research area)

allows structured and allows structured and unstructured meshes to be unstructured meshes to be combined in a single combined in a single computationcomputation

Level C: Mesh Components

GeometryInformation(Level A)

Full GeometryMeshes(Level B)

MeshComponents(Level C)

TSTT-7

Access to Mesh Data Hierarchy...

… as a single object (high-level common interfaces) TSTT will develop functions that provide, e.g.,

PDE discretization operators adaptive mesh refinement multilevel data transfer

Prototype provided by Overture framework Enables rapid development of new mesh-based

applications … through the mesh components (low-level

common interfaces) TSTT will provide, e.g.,

element-by-element access to mesh components fortran-callable routines that return interpolation coefficients

at a single point (or array of points) Facilitates incorporation into existing applications

TSTT-8

Common Interface Specification

Initially focus on low level access to static mesh components (Level C) Data: mesh geometry, topology, field data Efficiency though

Access patterns appropriate for each mesh type Caching strategies and agglomerated access

Appropriateness through working with Application scientists TOPS and CCA SciDAC ISICs

Application scientists program to the common interface and can than use any conforming tool without changing their code

High level interfaces to entire grid hierarchy which allows interoperable meshing by

creating a common view of geometry mesh adaptation including error estimators and curved

elements All TSTT tools will be interface compliant

TSTT-9

Mesh Data Hierarchy Construction Level 0 to Level 1 geometry

Leverage existing TSTT tools that provide graphical interfaces to decompose the initial geometry into subdomains

CGM (SNL), Overture (LLNL)

Level 1 mesh components Leverage exsiting mesh generation tools

Level C to Level B hybrid meshes Stitching algorithms Overlapping meshes

Start with a set of Start with a set of component meshes...component meshes...

… … Cut holes...Cut holes... … … Stitch togetherStitch togetherto form a hybrid meshto form a hybrid mesh

Overture Overture Stitching Stitching Algorithm Algorithm (LLNL)(LLNL)

CUBITCUBIT Geometry Geometry Decomposition (SNL)Decomposition (SNL)

TSTT-10

Enhancing Mesh Generation Capabilities

Will leverage most existing TSTT technology “as is”

Provisions for Creating interface compliant tools Improving mesh generation capabilities on

complex geometries for high order elements

Curvilinear elements Geometry approximations

Interoperability of appropriate tools e.g., ORNL elliptic and variational mesh

generators with Overture

Mesh quality control for hybrid meshes

Linear coarse elements Linear coarse elements verses high-order, curvilinear verses high-order, curvilinear P elements in P elements in MEGA MEGA (RPI)(RPI)

TSTT-11

Mesh Quality Control

Unstructured mesh quality research and development is provided by MESQUITE (SNL, ANL) optimization-based smoothing reconnection schemes development of quality metrics for

high order methods a posteriori quality control using

error estimators

PDE-solution based mesh optimization will be investigated for overlapping and hybrid meshes

ImprovedImprovedmeshmesh

8x error reduction by8x error reduction byselecting optimal mesh selecting optimal mesh generation parametersgeneration parameters

TSTT-12

Dynamic Mesh Evolution

Geometry evolves due to Adaptive mesh refinement Internally tracked interfaces (e.g.,

shocks) Motion of the domain boundary

MEGAMEGA Rayleigh- Rayleigh-Taylor Simulation Taylor Simulation (RPI)(RPI)

OvertureOverture simulation of simulation ofHele-Shaw flowHele-Shaw flow

TSTT-13

TSTT Research in Mesh Evolution

Requires evolution of both the hierarchy and the individual mesh components

TSTT will provide interfaces that allow the mesh tools to access the changing geometry the application programmer to access the changing mesh local or global modifications

New techniques will address Curvilinear geometries to preserve convergence rates of

high order discretizations abstraction of adaptive techniques to provide “plug and

play” adaptive techniques that use multiple criteria to extend

applicability automatic selection and application of optimal strategies

TSTT-14

Combining TSTT technologies will improve front tracking techniques

FronTier FronTier interfaceinterfacerepresentationrepresentation

Improve conservation properties and accuracy at the front by inserting a surface determined by front tracking into a volume mesh

Results in a front-adaptive space-time discretization

TSTT-15

TSTT will ease the use of high order discretization methods

Observation: Complexities of using high-order methods on adaptively evolving grids has hampered their widespread use Tedious low level dependence on grid infrastructure A source of subtle bugs during development Bottleneck to interoperability of applications with

different discretization strategies Difficult to implement in general way while

maintaining optimal performance Result has been a use of sub-optimal

strategies or lengthy implementation periods TSTT Goal: to eliminate these barriers by

developing a Discretization Library

TSTT-16

The Discretization Library Will...

… contain numerous mathematical operators Start with +, -, *, /, interpolation, prologation Move to div, grad, curl, etc. Both strong and weak (variational) forms of operators when

applicable … contain numerous discretization strategies

Finite Difference, Finite Volume, Finite Element, Discontinuous Galerkin, Spectral Element, Partition of Unity

Emphasize high-order and variable-order methods various boundary condition operators

… be independent of the underlying mesh infrastructure Utilizes the common low-level mesh interfaces All TSTT mesh tools will be available

… be extensible to allow user-defined operators and boundary conditions

TSTT-17

Additional Functionalities Support for Temporal Discretization

Method of Lines formulation (time steps and temporal methods are spatially independent)

Local Refinement Methods (time steps and methods vary in space) Space-Time techniques (unstructured meshes are used in both

space and time) Support for Adaptive Methods

Error estimators Richardson’s extrapolation (meshes of different resolution) P-refinement estimators Solution gradient and vorticity metrics

Optimal strategies for mesh enrichment (combinations of p- and h- adaptivity)

Combined with work on mesh quality improvement Support for Interpolation

Between meshes and operators Local conservation when mapping between meshes

TSTT-18

Performance of Discretization Library

Kernel operations imply good performance is critical Single Processor Performance

Compile time optimization of user-defined high level abstractions via ROSE (LLNL)

Consider hierarchical memory performance and cache usage

Terascale Computing scalability of local operations requires good partitioning

strategies Efficiency determined by the size of the partition boundary

relative to the partition volume Will leverage the experience of

LLNL’s Overture project that supports structured mesh topologies

RPI’s Trellis project for variational discretization

TSTT-19

Benefits of the Discretization Library

Lowers the time, cost, and effort to effectively deploy modern discretization tools High-level access for new application development

on TSTT Level B meshes Mid- and low-level access for insertion into existing

technology Increases reliability of application codes by

eliminating a common source of coding errors Enhances software reuse Permits easy experimentation with various

combinations of discretization strategies and mesh technologies for a given application

TSTT-20

Issues in Terascale Computing

Observation: Many tools exist that utilize hierarchical design principles to achieve good performance at the terascale e.g., multi-level partitioners, multigrid solvers, multiresolution

visualization tools

Barrier: Their union is not optimized often difficult to take advantage of the multiresolution

representations from one solution stage to the next

TSTT Goal: To design our hierarchy and tools so that downstream tools can take advantage of the multi-resolution information Actively consider trade-offs across the entire simulation Allow preservation of information as desired

e.g., subdomain decompositions used in creating a hybrid mesh may be similarly useful in preconditioning of iterative solvers

TSTT-21

Parallel Mesh Generation

Primarily leverage existing TSTT tools for parallel mesh generation

Current Techniques generate a coarse mesh on the geometry and

distribute that for further refinement distribute complete level 1 geometry information to

each processor New development focuses on the partitioning

and distribution of Level 1 geometry description Provides a start to finish scalable solution for mesh

generation

TSTT-22

Load Balancing

Use existing tools for partitioning Chaco and Metis for static partitioning Zoltan library (SNL) for dynamic partitioning

Develop and provide interfaces from TSTT software to Zoltan to ensure seamless operation

Augment Zoltan Research methods to accommodate hierarchical

machine models and heterogeneous parallel computers different processor speeds, memory capacities, cache

structures, networking speeds RPM (RPI) and PADRE (LLNL) serve as prototypes

Load balancing strategies for adaptive, structured, overlapping grids

MLB (LLNL) serves as a prototype

TSTT-23

SciDAC ISIC Collaborations

CCA: (PI: Armstrong) co-develop common interfaces for mesh and field data create CCA-compliant mesh components and provide them in the CCA

component repository explore the role of the component model in the composition of

numerous discrete operators performance critical operations extend ROSE project to explore component models

TOPS: (PI: Keyes) provide mesh representations for multilevel techniques co-develop well-defined interfaces to ensure that the meshes and

discretization strategies will be interoperable with solution software Performance: (PI: Bailey)

we will use ROSE preprocessor to develop highly-tuned discretization libraries

TSTT will provide benchmarks and a testing environment for developments in the performance ISIC

TSTT-24

SciDAC Applications: Accelerator Design Particle Forces and EM Field Calculations (Ko)

TSTT will provide… advanced mesh generation capabilities for complex geometries Hybrid meshes that match conformally orthogonal structured

grids to unstructured mixed element grids Mesh quality and improvement to accelerate solver

convergence TSTT Points of contact: D. Brown, P. Knupp

Particle Tracking (Luccio) TSTT will provide…

parallel decomposition tools to cluster particles into spatially coherent load balanced domains

assistance in the development of codes for adaptive solution to Poisson’s equation with realistic BC for rapid solution of the space charge

TSTT Point of Contact: J. Glimm

TSTT-25

SciDAC Applications: Fusion

Magnetohydrodynamics modeling with parM3d (Jardin) TSTT will provide…

higher-order finite element schemes for poloidal discretization

explore the use of TSTT mesh generation techniques for automating the process of flux-aligned unstructured meshes in the poloidal directions

Incorporation of adaptivity, mixed element meshes, and dynamic load balancing tools in the long term for resonant instability studies

Also will work with TOPS ISIC in the development of mesh abstractions for multilevel solvers

TSTT Point of contact: J. Flaherty

TSTT-26

SciDAC Applications: Chemically Reactive Flows

Computational Facility for Reacting Flow Science (Najm) TSTT will provide…

high-order spectral elements deployed in current toolkit collaborative development of CCA-compliant interfaces for

block-structured mesh adaptation using GrACE deployment of discretization library in GrACE (fourth order

schemes are desired) TSTT Points of contact: P. Fischer and L. Freitag

Model Jet Breakup and spray formation (non-SciDAC) TSTT will provide…

Frontier interface tracking capabilities to provide a more accurate model

TSTT Point of Contact: J. Glimm

TSTT-27

SciDAC Applications: Climate

Community Climate System Model (J. Drake) TSTT will provide…

Collaboration with model coupling toolkit (mct) developers to define locally conservative interpolation schemes between different mesh types

work with mct developers to include dynamic load balancing techniques for the case in which component models reside on dynamically changing sets of processors

TSTT Points of contact: L. Freitag Global Transport models

TSTT will provide adaptive capabilities for local, regional, and global transport of atmospheric species and aerosols

TSTT Point of Contact: J. Glimm

TSTT-28

Other DOE Applications

Biosimulation modeling Cardiac electrophysiology (BNL, PNNL) Biofluids (ANL, RPI, LLNL) Computational cell and organ physiology (PNNL, ORNL,

LLNL) Fluid instabilities in ICF applications (BNL, SB,

RPI, PNNL, LLNL) Jet breakup and spray modeling (BNL, SB, ANL,

PNNL) Free surface flow modeling for target design of a

muon collider accelerator and liquid metal cooling in a Tokamak (BNL)

Flow in porous media (SB, BNL) Accelerator tracking design (BNL)

TSTT-29

TSTT Institutional Roles ANL

Co-lead mesh quality and optimization, contribute to discretization library, interoperable meshing and terascale computing. Liaison with CCA, climate, reacting flows, and biology applications

BNL Leads the application effort and is liaison for climate and accelerator

design. Leads efforts to create interoperability between Frontier and TSTT mesh generators, contributes to discretization library

LLNL Co-leads design and implementation of mesh hierarchy and

component design. Contributes performance optimization tools to discretization library and is liaison to the accelerator design app

ORNL Contributes to mesh quality optimization, enhancement and

interoperability. Contributes to climate and chemically reacting flow applications

TSTT-30

TSTT Institutional Roles

SNL Co-leads efforts on mesh quality optimization, contributes

to interoperable meshing, domain decomposition and load balancing. Liaison with accelerator application.

RPI Co-leads the development of meshing and discretization

technologies for mesh hierarchy and discretization libraries. Contributes to the load balancing work and serves as liaison to the fusion application.

PNNL Contributes to interoperable meshing and terascale

computing areas, liaison for the biology applications. SUNY SB

Leads the interoperability of FronTier with meshing technologies and development of high-order versions. Liaison in spray simulations and oil reservoir applications.

TSTT-31

Contact Information

Jim Glimm, Center Director Brookhaven National Lab and SUNY Stony Brook glimm@ams.sunysb.edu 631-62-8355

David Brown Lawrence Livermore National Laboratory dlb@llnl.gov 925-424-3557

Patrick Knupp Sandia National Laboratories pknupp@sandia.gov 505-284-4565

Lori Freitag Argonne National Laboratory freitag@mcs.anl.gov 630-252-7246