Parallel, remote visualization of large data sets with VisIt

in collaboration with VACET and Prabhat (NERSC)

Wide range of tools

VisIt

Tecplot

General Mesh Viewer (GMV)

Plotting tools (gnuplot etc)

Sharing of visualization scripts

Job Launching Submits remote jobs to Chinook,

Franklin, and workstations; easily

supports new queue systems

Launches multiple jobs concurrently,

to multiple machines

Monitors remote job progress with

the option to terminate

Generates boilerplate text for job

submission scripts, which can be

customized on a per-machine per-task

basis

User control over where to store

output files, such as to an archive (left),

to the Content Management System,

or left on the compute server

Implemented with the Kepler

workflow system

Workflow and Tool Integration

Tasks are the execution of tools or

a sequence of tools. Computational

models are treated like tools

New tools are easily added

through a text file-based registry; a

registration user tool is planned

Tools can be executed locally or

remotely

Visualization tools are registered

just like any other task and can be

desktop tools like TecplotTM or those

requiring a supercomputer to run

A programming model is provided

to plug-in custom wxPython GUIs to

allow complex processing or

visualization

Many subsurface flow and transport problems of importance today

involve coupled non-linear flow, transport, and reaction in media

exhibiting complex heterogeneity. In particular, problems involving

biological mediation of reactions and/or mineral

precipitation/dissolution fall into this class of problems. We are

developing a hybrid multi-scale modeling framework that integrates

models with diverse representations of physics, chemistry and biology

at different scales (sub-pore, pore and continuum). The modeling

framework takes advantage of advanced computational technologies

including parallel code components using the Common Component

Architecture (CCA). A pore scale Smoothed Particle Hydrodynamics

(SPH) model has been built within the CCA framework.

The Support Architecture for Large-scale Subsurface Simulation and

Analysis (SALSSA) software framework addresses the macro

modeling processes such as running simulations and analyzing their

results. SALSSA integrates workflow, data management, and

visualization supporting the development, validation, and use of the

SPH model.

Hybrid Multi-scale Modeling Frameworks

Provenance Tracking and Data Management

Automatically keeps track of user activities including location of data files,

associated metadata, and provenance.

Enables generation of multiple views including provenance graph shown

above and simulation table shown below

Supports automatic metadata extraction to create context aware views

Provides clear record that can be used for verification

Record can be used to repeat activities such as validation testing

Flexible metadata model easily incorporates new metadata

Users can dynamically add/remove columns from the table for any

metadata that has been collected

Automatically transfers all inputs for staging to remote workstations and

then stores outputs back to the Alfresco Content Management System

Based on Sesame Resource Description Framework (RDF) and Alfresco

content management system

http://subsurface.pnl.gov/salssa

Data &

Provenance

Store

Franklin (NERSC)

Chinook (EMSL)

Workstations…

High Performance

Storage System

(HPSS)

Job Scripts Tool Registry Metadata

extractors

MIME type

extractors

Custom Tool

User InterfacesStandard tools

User preferred tools

User developed tools

Default scripts

Power user scriptsInput file parameters

Output file results

Extensibility Mechanisms

User Environment

Visualization

The SALSSA framework is comprised of a user environment that

supports the integration of a wide range of tools, a mechanism for

launching simulations to a variety of machines, a sophisticated data

sub-system that manages provenance and keeps references to

simulation data that may be located on archives or compute systems,

and a mechanism to invoke tools located with the data. Scientists can

extend the system to support new tools by modifying data files and

scripts. For more sophisticated integration, a Python-based

programming model is available.

Density distribution of SPH simulation in

porous media

Visualization of SPH simulation

provided by Kwan Lui Ma

(UltraVis)

Data / Job

Submission

Service

The “Organizer” is the primary user interaction tool. It is structured around the concept of a

“study” where a single study is a collection of setup, simulation execution, and analysis tasks. The

data associated with each task is automatically recorded along with provenance which results in

the graph-based view shown above. New tasks can be initiated based on the type of an existing

task and its data. The figure above shows the graph view of the simulation and analysis task

graph for the validation study. A range of graph viewing and manipulation options are available.

Inputs and outputs for each task can be displayed by selecting the task.

Simulation Execution

SALSSA Architecture

The set of available tools is

organized into categories. A wide

range of tools can be incorporated

including: models, visualization

tools, data transformation

tools/scripts, desktop tools, and

custom tools. Users initiate new

tasks by double-clicking on a task

within the tree. If a process or

data is selected, the task tree will

only show tools that can operate

on the specified data.

When no tools are “known”

to handle a particular data

type, the user is prompted

to select a tool. This allows

easy integration of desktop

tools such as users’

preferred editors, plotting

tools, or simulation codes.

A wizard will prompt for any

inputs that cannot be

determined from the user

selection. The prompting is

determined by input file and

parameters specified in the

tool registration file.

The wizard has been

customized to support

editing of SPH input

files and specification

of a range of

parameters for which

separate simulations

should be generated.

Kepler

Sesame /

Alfresco

Application of SALSSA Framework to the

Validation of Smoothed Particle Hydrodynamics

Simulations of Low Reynolds Number FlowsKaren Schuchardt, Jared Chase, Jeff Daily, Todd Elsethagen, Bruce Palmer, Tim Scheibe

About Pacific Northwest

National Laboratory

The Pacific Northwest National Laboratory,

located in southeastern Washington State, is a

U.S. Department of Energy Office of Science

laboratory that solves complex problems in

energy, national security and the environment,

and advances scientific frontiers in the chemical,

biological, materials, environmental and

computational sciences. The Laboratory

employs 4,000 staff members, has a $760

million annual budget, and has been managed

by Ohio-based Battelle since 1965.

For more information about the science you see

here, please contact:

Karen Schuchardt

Pacific Northwest National Laboratory

P.O. Box 999, K7-90

Richland, WA 99352

(509) 375-6525

Karen.Schuchardt@pnl.gov

Computational Model

Smoothed Particle Hydrodynamics (SPH):

A Lagrangian particle method for solving systems of partial differential equations

Chase JM, KL Schuchardt, G Chin, Jr, JA Daily, and TD Scheibe. 2008. "Iterative Workflows for Numerical Simulations in Subsurface Sciences." In Proceedings of 2008 IEEE Congress on Services Part 1, pp. 461-464. IEEE Computer Society, Los Alamitos, CA.

Gibson TD, KL Schuchardt, and EG Stephan. 2009. "Application of Named Graphs Towards Custom Provenance Views." In 1st Workshop on the Theory and Practice of Provenance (TaPP '09).

Schuchardt KL, GD Black, JM Chase, TO Elsethagen, and L Sun. 2007. "Process Integration, data management, and visualization framework for subsurface sciences." Journal of Physics: Conference Series 78:012064, 1-5.

Scheibe TD, AM Tartakovsky, DM Tartakovsky, GD Redden, P Meakin, BJ Palmer, and KL Schuchardt. 2008. "Hybrid numerical methods for multiscale simulations of subsurface biogeochemical processes." In Proceedings of SciDAC 2008: Journal of Physics, vol. 125, p. 012054. IOP Publishers, Bristol, United Kingdom.

Publications

Collaborators

Fluid flow in porous media at velocities common in natural porous media (i.e., groundwater

flow) occur at low Reynolds numbers and therefore it is important to verify that the SPH

model is producing accurate flow solutions in this regime. The SPH model contains

parameters that do not appear in the corresponding incompressible Navier-Stokes

equations and it is important to identify value ranges for these parameters that lead to good

solutions for the flow profiles in topologically complex media. These parameters include a

speed of sound c, which manifests itself by introducing a finite compressibility into the flow,

and a weighting function diameter h that is roughly analogous to the grid spacing in grid-

based numerical methods. Ideally, solutions should be independent of these parameters in

some limit.

The approach taken is to perform a series of simulations using relatively simple geometries

where the flow solutions can be compared against analytic results or numerical results

obtained using other methods, such as finite elements. The two flow geometries are slit

flow between two parallel walls and flow in a cubic array of periodic spheres. The slit flow

problem leads to a parabolic flow profile for which there is an analytic solution and the

periodic sphere problem can be modeled using standard grid-based numerical methods.

Both of the configurations will be simulated using SPH over a range of parameters and

compared to validated solutions.

Validation of Low Reynolds Number Flows

Preliminary Results

Validation of the SPH model is currently under way. Tens of parameter

configurations have already been executed and analyzed within the

SALSSA workflow environment. Results of a selected SPH validation

simulation run are shown above. Left: comparison of simulated velocity

profile along a cross-section through the slit; Center: Time variation of

simulated mean velocity as steady-state flow is approached; Right: Density

variations within the simulated flow field as a function of time. All variables

are presented in non-dimensional units.

0

0.01

0.02

0.03

0.04

0.05

0.06

0.07

0 5 10 15 20

x

ve

loc

ity

Simulated

Analytical

Solution

0.00E+00

5.00E-03

1.00E-02

1.50E-02

2.00E-02

2.50E-02

3.00E-02

3.50E-02

4.00E-02

4.50E-02

5.00E-02

0 100 200 300 400 500

Time

Mean

velo

cit

y

27.06

27.065

27.07

27.075

27.08

27.085

27.09

0 100 200 300 400 500

Time

De

ns

ity

Minimum

Maximum

Simulation of non-reactive contaminant transport

using SPH framework. Porous media surface is in

green, isosurfaces of velocity are red and

isosurfaces of concentration are blue. The

visualization is done using the VisIt parallel software

package.

The Organizer’s Simulation View provides a spreadsheet view of associated metadata and

provenance information for selected simulation runs. These views are customizable, allowing

a user to add, remove, or reorder the data columns.

PN

NL-S

A-6

6855