LLNL-PRES-813207This work was performed under the auspices of the U.S. Department of Energy by Lawrence Livermore National Laboratory under contract DE-AC52-07NA27344. Lawrence Livermore National Security, LLC LLNL-PRES-813207
ATPESC 2020Monday August 3rd, 2020
Visualization and Analysis of HPC Simulation Data with VisIt
This work was performed under the auspices of the U.S. Department of Energy by Lawrence Livermore National Laboratory under contract DE-AC52-07NA27344. Lawrence Livermore National Security, LLC
Cyrus Harrison ([email protected])
2LLNL-PRES-813207
Outline
Intro to VisIt
Simulation Exploration
§ VisIt Project Introduction (~30 min)
§ Hands-on: (~1 hour)— Guided tour of VisIt— Visualization of an Aneurysm
(Blood Flow) Simulation
3LLNL-PRES-813207
§ VisIt 3.1.2— https://github.com/visit-dav/visit/releases
§ Tutorial Materials— http://visitusers.org/index.php?title=VisIt_Tutorial
§ How to get in touch— GitHub: https://github.com/visit-dav/visit
— Email List: [email protected]
— Cyrus: [email protected]
Tutorial Resources
4LLNL-PRES-813207
Aneurysm Simulation Dataset Simulated using the LifeV (http://www.lifev.org/) finite element solver.Available thanks to:
— Gilles Fourestey and Jean FavreSwiss National Supercomputing Centre (http://www.cscs.ch/)
Potential Flow Simulation Dataset Simple tutorial simulation built using MFEM (https://mfem.org/)Available thanks to:
— Aaron Fisher and Mark Miller, LLNL
Tutorial Data Acknowledgements
5LLNL-PRES-813207
VisIt Project Introduction
6LLNL-PRES-813207
7LLNL-PRES-813207
§ Production end-user tool supporting scientific and engineering applications.
§ Provides an infrastructure for parallel post-processing that scales from desktops to massive HPC clusters.
§ Source released under a BSD style license.
VisIt is an open source, turnkey application for data analysis and visualization of mesh-based data
Pseudocolor plot of Density(27 billion element dataset)
8LLNL-PRES-813207
VisIt supports a wide range of use cases
Data Exploration
Visual Debugging
Quantitative Analysis
Comparative Analysis
=?
Presentation Graphics
9LLNL-PRES-813207
VisIt provides a wide range of plotting features for simulation data across many scientific domains
Molecular Visualization Parallel Coordinates
Pseudocolor RenderingVector / Tensor Glyphs
Volume Rendering
Streamlines / Pathlines
10LLNL-PRES-813207
§ The VisIt project started in 2000 to support LLNL’s large scale ASC physics codes.
§ The project grew beyond LLNL and ASC with development from DOE SciDACand other efforts.
§ VisIt is now supported by multiple organizations:— LLNL, LBNL, ORNL, Univ of Oregon, Univ of Utah, Intelligent Light, …
§ Over 100 person years of effort, 1.5+ million lines of code.
VisIt is a vibrant project with many participants
Project Started
2000
LLNL ASC users transitioned to VisIt
2003
2005 R&D 100
2005
Transition to Public NERSCSW repo
2008 2010
VisIt 2.0 Release
2006
DOE SciDAC:VACET Funded
2012 - 2016
R&D Collaborations
VisIt 3.0 Release
Transition to repo
2019
ECP ALPINE Funded
2017
11LLNL-PRES-813207
§ Mesh Types — Point, Curve, 2D/3D Rectilinear,
Curvilinear, Unstructured— Domain Decomposed, AMR— Time Varying— Primarily linear element support,
limited quadratic element support
§ Field Types— Scalar, Vector, Tensor, Material Volume
Fractions, Species
VisIt provides a flexible data model, suitable for many application domains
12LLNL-PRES-813207
“How do I obtain VisIt?”
§ Use an existing build:— For your Laptop or Workstation:
• Binaries for Windows, OSX, and Linux (RHEL + Ubuntu):(https://wci.llnl.gov/simulation/computer-codes/visit/executables)
— VisIt on ALCF’s Cooley:• https://www.alcf.anl.gov/user-guides/visit-cooley
— Several other HPC centers have VisIt installed
§ Build VisIt yourself:— “build_visit” is a script that automates the process of building VisIt and its third-party
dependencies. (also at: https://wci.llnl.gov/simulation/computer-codes/visit/executables)— Fledgling support for building via spack (https://github.com/spack/spack)
The VisIt team releases binaries for several platforms and a script that automates the build process
13LLNL-PRES-813207
“How do I get my data into VisIt?”
§ The PlainText database reader can read simple text files (CSV, etc)— http://visitusers.org/index.php?title=Using_the_PlainText_reader
§ Experiment with the visit_writer utility:— http://visitusers.org/index.php?title=VisItWriter
§ Write to a commonly used format:— VTK, Silo, Xdmf, PVTK
§ We are ramping up support for Mesh-based data in Conduit Blueprint:— http://llnl-conduit.readthedocs.io/en/latest/blueprint_mesh.html
§ Consult the Getting Data Into VisIt Manual and its associated source code examples.
VisIt supports more than 110 file formats
14LLNL-PRES-813207
§ C++ Plugin Architecture— Custom File formats, Plots, Operators— Interface for custom GUIs in Python,
C++ and Java
§ Python Interfaces— Python scripting and batch processing— Data analysis via Python Expressions and Queries
§ In-Situ Coupling— VisIt’s Libsim library allows simulation codes to link
in VisIt’s engine for in situ visualization
VisIt’s infrastructure provides a flexible platform for custom workflows
Simulation
VisIt
Libsim
Adap
tor
15LLNL-PRES-813207
§ DOE Research Collaborations
§ Research Focus Areas— Light weight In Situ Processing— Node Level Parallelism — Distributed Memory Parallel Algorithms
VisIt is used as a platform to deploy visualization research
Algorithms research:How to efficiently calculate
particle paths in parallel.
Scaling research:Scaling to 10Ks of cores and
trillions of cells.
Methods research:How to incorporate statistics
into visualization.
Algorithms research:Reconstructing material
interfaces for visualization
16LLNL-PRES-813207
§ We migrated our source repo from svn at NERSC to git on GitHub and our issue tracking from an ORNL Redmine instance to GitHub— https://github.com/visit-dav/visit
§ We ported our legacy docs to Sphinx, now hosted on Read the Docs— https://visit-sphinx-github-user-
manual.readthedocs.io/en/develop/
VisIt 3.0 (April 2019) included major updates to our software development process
VisIt source repo and issue tracking on GitHub
VisIt manuals on Read the Docs
17LLNL-PRES-813207
VisIt uses MPI for distributed-memory parallelism on HPC clusters
We are enhancing VisIt’s pipeline infrastructure to support threaded processing and many-core architectures
Full Dataset(27 billion total elements)
3072 sub-grids (each 192x129x256 cells)
18LLNL-PRES-813207
VisIt employs a parallelized client-server architecture
netw
ork
conn
ectio
n
Parallel HPC ClusterClient Computer
VisIt Viewer
VisIt GUI VisIt CLI Python Clients
JavaClients
(Files or Simulation)
DataPlugin
VisItEngine Data
DataPlugin
VisItEngine Data
DataPlugin
VisItEngine Data
MPI
19LLNL-PRES-813207
VisIt automatically switches to a scalable rendering mode when plotting large data sets on HPC clusters
Task 1 Task 2
Task 3 Task 4
ParallelCompositing
Final Composited Image
In addition to scalable surface rendering, VisIt also provides scalable volume rendering
20LLNL-PRES-813207
DOE’s visualization community is collaborating to create open source tools ready for Exascale simulationsAddressing node-level parallelism
§ VTK-m is an effort to provide a toolkit of visualization algorithms that leverage emerging node-level HPC architectures
§ We are also exploring using VTK-m and DIY to share more distributed-memory infrastructure across projects
http://m.vtk.org
Addressing I/O gaps with in-situ
§ There are several efforts focused on in-situ infrastructure and algorithms
http://www.paraview.org/in-situ
https://github.com/Alpine-DAV/ascenthttps://github.com/diatomic/diy
DIYVisIt LibSimhttps://visit.llnl.gov
http://www.sensei-insitu.org
http://alpine.dsscale.org
21LLNL-PRES-813207
The VisIt team is investing in Conduit and Ascent to create next generation in situ infrastructure
Flyweight in-situ visualization and analysis for HPC simulations
http://ascent-dav.orghttp://software.llnl.gov/conduit
Intuitive APIs for in-memory data description and exchange
22LLNL-PRES-813207
§ Provides an intuitive API for in-memory data description— Enables human-friendly hierarchical data organization
— Can describe in-memory arrays without copying
— Provides C++, C, Python, and Fortran APIs
§ Provides common conventions for exchanging complex data— Shared conventions for passing complex data (eg: Simulation Meshes) enable
modular interfaces across software libraries and simulation applications
§ Provides easy to use I/O interfaces for moving and storing data— Enables use cases like binary checkpoint restart
— Supports moving complex data with MPI (serialization)
Conduit provides intuitive APIs for in-memory data description and exchange
http://software.llnl.gov/conduithttp://github.com/llnl/conduit
Website and GitHub Repo
Hierarchical in-memory data description
Conventions for sharing in-memory mesh data
23LLNL-PRES-813207
§ Easy to use in-memory visualization and analysis— Use cases: Making Pictures, Transforming Data, and Capturing Data
— Young effort, yet already supports most common visualization operations
— Provides a simple infrastructure to integrate custom analysis
— Provides C++, C, Python, and Fortran APIs
§ Uses a flyweight design targeted at next-generation HPC platforms— Efficient distributed-memory (MPI) and many-core (CUDA or OpenMP) execution
• Demonstrated scaling: In situ filtering and ray tracing across 16,384 GPUs on LLNL's Sierra Cluster
— Has lower memory requirements than current tools
— Requires less dependencies than current tools (ex: no OpenGL)
• Builds with Spack https://spack.io/
Ascent is an easy to use flyweight in situ visualizationand analysis library for HPC simulations
Visualizations created using Ascent
Extracts supported by Ascent
http://ascent-dav.orghttps://github.com/Alpine-DAV/ascent
Website and GitHub Repo
24LLNL-PRES-813207
VisIt’s Visualization Building Blocks
25LLNL-PRES-813207
§ Databases: Read data
§ Plots: Render data
§ Operators: Manipulate data
§ Expressions: Generate derived quantities
§ Queries: Summarize data
VisIt’s interface is built around five core abstractions
26LLNL-PRES-813207
Examples of VisIt Pipelines
§ Databases: Read data
§ Plots: Render data
§ Operators: Manipulate data
§ Expressions: Generate derived quantities
§ Queries: Summarize data
Open a database, which reads from a file(Example: Open file1.hdf5)Database
Make a plot of a field in the database(Example: Pseudocolor plot of density)Plot
27LLNL-PRES-813207
Examples of VisIt Pipelines
§ Databases: Read data
§ Plots: Render data
§ Operators: Manipulate data
§ Expressions: Generate derived quantities
§ Queries: Summarize data
Open a database, which reads from a file(Example: Open file1.hdf5)Database
Make a plot of the result(Example: Pseudocolor plot of density)Plot
Apply an operator to transform the data(Example: Slice operator)Operator
28LLNL-PRES-813207
Examples of VisIt Pipelines
§ Databases: Read data
§ Plots: Render data
§ Operators: Manipulate data
§ Expressions: Generate derived quantities
§ Queries: Summarize data
Open a database, which reads from a file(Example: Open file1.hdf5)Database
Make a plot of the result(Example: Pseudocolor plot of density)Plot
Apply an operator to transform the data(Example: Slice operator)Operator 1
Apply a second operator to transform the data(Example: Elevate operator)Operator 2
29LLNL-PRES-813207
Examples of VisIt Pipelines
§ Databases: Read data
§ Plots: Render data
§ Operators: Manipulate data
§ Expressions: Generate derived quantities
§ Queries: Summarize data
Open a database, which reads from a file(Example: Open file1.hdf5)Database
Make a plot of the result(Example: Pseudocolor plot of speed)Plot
Create derived quantity from existing fields(Example: speed = magnitude(velocity)Expression
30LLNL-PRES-813207
Examples of VisIt Pipelines
§ Databases: Read data
§ Plots: Render data
§ Operators: Manipulate data
§ Expressions: Generate derived quantities
§ Queries: Summarize data
Open a database, which reads from a file(Example: Open file1.hdf5)Database
Extract quantitative information(Example: integrate density to find mass)Query
Make a plot of a field in the database(Example: Pseudocolor plot of density)Plot
31LLNL-PRES-813207
Examples of VisIt Pipelines
§ Databases: Read data
§ Plots: Render data
§ Operators: Manipulate data
§ Expressions: Generate derived quantities
§ Queries: Summarize data
Open a database, which reads from a file(Example: Open file1.hdf5)Database
Create derived quantity from existing fields(Example: speed = magnitude(velocity)Expression
Apply an operator to transform the data(Example: Slice operator)Operator 1
Apply a second operator to transform the data(Example: Elevate operator)Operator 2
Plot a field(Example: Pseudocolor plot of speed)Plot
Extract quantitative information(Example: Maximum speed over cross-section)Query
32LLNL-PRES-813207
Resources
Presenter Contact Info:
§ Cyrus Harrison: [email protected]
User Resources:
§ Main website: http://www.llnl.gov/visit
§ Wiki: http://www.visitusers.org
§ Email: [email protected]
Developer Resources:
§ Email: [email protected]
§ Github: https://github.com/visit-dav/visit
33LLNL-PRES-813207
Aneurysm Simulation Exploration
https://visit-sphinx-github-user-manual.readthedocs.io/en/develop/tutorials/Aneurysm.html
34LLNL-PRES-813207
Remote Usage Tips
https://visit-sphinx-github-user-manual.readthedocs.io/en/develop/tutorials/RemoteUsage.html
35LLNL-PRES-813207
Python Scripting Basics
https://visit-sphinx-github-user-manual.readthedocs.io/en/develop/tutorials/Scripting.html
36LLNL-PRES-813207
Connected Components
https://visit-sphinx-github-user-manual.readthedocs.io/en/develop/tutorials/CCL.html
37LLNL-PRES-813207
§ Potential Flow Simulation Exploration— https://visit-sphinx-github-user-manual.readthedocs.io/en/develop/tutorials/PotentialFlow.html
§ Water Flow Simulation Exploration— http://visitusers.org/index.php?title=Water_Flow_Tutorial
§ Volume Rendering— http://visitusers.org/index.php?title-Visit-tutorial-Volume-Rendering
§ Movie Making— https://visit-sphinx-github-user-manual.readthedocs.io/en/develop/tutorials/MakingMovies.html
§ Advanced Movie Making— http://visitusers.org/index.php?title=Visit-tutorial-Advanced-movie-making
Additional Hands-on Materials
38LLNL-PRES-813207
Resources
Presenter Contact Info:
§ Cyrus Harrison: [email protected]
User Resources:
§ Main website: http://www.llnl.gov/visit
§ Wiki: http://www.visitusers.org
§ Email: [email protected]
Developer Resources:
§ Email: [email protected]
§ Github: https://github.com/visit-dav/visit
39LLNL-PRES-813207
Additional Slides
40LLNL-PRES-813207
Visualization Techniques for Mesh-based Simulations
41LLNL-PRES-813207
Pseudocolor rendering maps scalar fields to a range of colors
Pseudocolor rendering of Elevation Pseudocolor rendering of Density
42LLNL-PRES-813207
Volume Rendering cast rays though data and applies transfer functions to produce an image
Emitter
Film/Image
43LLNL-PRES-813207
Isosurfacing (Contouring) extracts surfaces of that represent level sets of field values
44LLNL-PRES-813207
Particle advection is the foundation of several flow visualization techniques§ S(t) = position of particle at time t
§ S(t0) = p0— t0: initial time— p0: initial position
§ S’(t) = v(t, S(t))— v(t, p): velocity at time t and position p— S’(t): derivative of the integral curve at time t
This is an ordinary differential equation.
45LLNL-PRES-813207
§ Streamlines – Instantaneous paths
§ Pathlines – Time dependent paths
Streamline and Pathline computation are built on particle advection
46LLNL-PRES-813207
Meshes discretize continuous space
§ Simulations use a wide range of mesh types, defined in terms of: — A set of coordinates (“nodes” / “points” / “vertices”)— A collection of “zones” / “cells” / “elements” on the coordinate set
VisIt uses the “Zone” and “Node” nomenclature throughout its interface.
Points Uniform Curvilinear Unstructured
47LLNL-PRES-813207
§ Field values are associated with the zones or nodes of a mesh— Nodal: Linearly interpolated between the nodes of a zone— Zonal: Piecewise Constant across a zone
§ Field values for each zone or node can be scalar, or multi-valued (vectors, tensors, etc.)
Mesh fields are variables associated with the mesh that hold simulation state
Nodal Association Zonal Association Vector Field Tensor Field
48LLNL-PRES-813207
§ Multi-material simulations use volume/area fractions to capture disjoint spatial regions at a sub-grid level.
§ These fractions can be used as input to high-quality sub-grid material interface reconstruction algorithms.
Material volume fractions are used to capture sub-zonal interfaces
49LLNL-PRES-813207
§ Species describe sub-grid variable composition— Example: Material “Air” is made of species “N2” ,“O2”, “Ar”, “CO2”, etc.
§ Species are used for weighting, not to indicate sub-zonal interfaces. — They are typically used to capture fractions of “atomically mixed” values.
Species are used to capture sub-zonal weightings
50LLNL-PRES-813207
§ Simulation meshes may be composed of smaller mesh “blocks” or “domains”.
§ Domains are partitioned across MPI tasks for processing.
Domain decomposed meshes enable scalable parallel visualization and analysis algorithms
51LLNL-PRES-813207
§ Mesh domains are associated with patches and levels
§ Patches are nested to form a AMR hierarchy
Adaptive Mesh Refinement (AMR) refines meshes into patches that capture details across length scales
DisclaimerThis document was prepared as an account of work sponsored by an agency of the United States government. Neither the United States government nor Lawrence Livermore National Security, LLC, nor any of their employees makes any warranty, expressed or implied, or assumes any legal liability or responsibility for the accuracy, completeness, or usefulness of any information, apparatus, product, or process disclosed, or represents that its use would not infringe privately owned rights. Reference herein to any specific commercial product, process, or service by trade name, trademark, manufacturer, or otherwise does not necessarily constitute or imply its endorsement, recommendation, or favoring by the United States government or Lawrence Livermore National Security, LLC. The views and opinions of authors expressed herein do not necessarily state or reflect those of the United States government or Lawrence Livermore National Security, LLC, and shall not be used for advertising or product endorsement purposes.