Inner Workings And Architecture of FLASH

An Advanced Simulation & Computing (ASC) Academic Strategic Alliances Program (ASAP) Center

at The University of Chicago

The Center for Astrophysical Thermonuclear Flashes

Inner Workings And Architecture of FLASH

Anshu Dubey & Katie Antypas

June 4, 2006

The ASC/Alliances Center for Astrophysical Thermonuclear FlashesThe University of Chicago

Overview

What is FLASH

Basics of FLASH Architecture Similarities and differences between versions 2 and 3

Units

Inheritance

Naming

Basics behind a problem setup Enough of an idea of these concepts to be able to look at a

sample setup and understand what is happening.


The Flash Code

Cellular detonation

Compressed turbulence

Helium burning on neutron stars

Richtmyer-Meshkov instability

Laser-driven shock instabilitiesNova outbursts on white dwarfs Rayleigh-Taylor instability

Flame-vortex interactions

Gravitational collapse/Jeans instability

Wave breaking on white dwarfs

Shortly: Relativistic accretion onto NS

Orzag/Tang MHDvortex

Type Ia Supernova

Intracluster interactions

MagneticRayleigh-Taylor

The Flash code1. Parallel, adaptive-mesh simulation code2. Designed for compressible reactive flows3. Has a modern CS-influenced architecture4. Can solve a broad range of (astro)physics problems5. Portable- runs on many massively-parallel systems6. Scales and performs well7. Is available on the web: http://flash.uchicago.edu


What FLASH Provides

Physics Hydrodynamics

PPM MHD Relativistic PPM

Equation of State Ideal gas Multigamma Helmholtz

Nuclear Physics Gravity Cosmology Particles

Infrastructure Setup Mesh Management

AMR: Paramesh UG:

Parallel I/O hdf5, pnetcdf

Monitoring Runtime and post-

processing visualization Regular testing toolkit Unit test framework


A Little FLASH History

FLASH0 Paramesh2, Prometheus and EOS/Burn

FLASH1 Smoothing out the smash First form of module architecture & inheritance

FLASH2 Untangle modules from each other (Grid)

dBase Concept of levels of users

FLASH3 Stricter interface control & unit architecture Taming the database Better encapsulation and isolation Community supported and developed code

BAM


FLASH Audiences

FLASH Application

DeveloperApplicationProgrammer

End User

• Develop physics Unit• Publish API• Use other unit’s API

• Initial conditions• Boundary conditions• Runtime parameters• Select units, and their specific implementations• Implement functions specific to an application, if they are not provided by FLASH

• Works on just about everything• Grid development• Data access• Architecture


FLASH Code Basics

An application code, composed of units/modules. Particular modules are set up together to run different physics problems.

Performance, Testing, Usability, PortabilityFortran, C, Python, …

More than 560,000* lines of code75% code, 25% comment

Very portableScaling to tens of thousand procs

* Internal Release


FLASH on BG/L - Sod weak scaling

Source :K.M. Riley, Argonne National Lab

Hydro Time: Constant Work Per Processor using AMR

Number of Processors

To

tal

Tim

e


Hydro and Particles Scaling


Basic Computational Unit : Block

The grid is composed of blocks

Most often all blocks have same dimensions

Cover different fraction of the physical domain.

In AMR blocks at different levels of refinement have different grid spacing.


FLASH Architecture

Four Cornerstones

Setup toolassemble simulation

Config filesTell setup how to

Assemble simulation

DriverOrganize interactions

Between units

Unit ArchitectureAPI

InheritanceData management


What’s a FLASH Unit?

FLASH basic architecture unitComponent of the FLASH code providing a

particular functionalityDifferent combinations of units are used for

particular problem setupsPublishes a public interface for other units’ use.Ex: Driver, Grid, Hydro, IO etc

Fake inheritance by use of directory structureInteraction between units governed by the

DriverNot all units are included in all applications


FLASH Units

Driver

I/ORuntimeParams

Grid

Profiling

LogfileSimulation

Infrastructure

monitoring

Hydro

BurnGravity

MHD

Physics


Unit Architecture

Driver

Top level:•API

•Unit Test

GridData Module

block datatime step

etc

Wrapper

Kernel


Unit Hierarchy

UnitAPI/stubs

Impl_1Remaining

API impl

kernel

kernel

UnitMainCommon API

implementation

UnitSomethingAPI

implementation

kernel

Impl_2Remaining

API impl

kernel

Impl_3Remaining

API impl

commonMore common

impl


Inheritance Through Directories: Eos

Eos_init

EosEos_

wrapped

EosMain

Gamma

Multigamma

Stub Implementations of thethree functions at the top level

Eos/EosMain Replaces the stub with animplementation common to all formulations of EOS

Eos/EosMain/Gamma implements gamma versionsOf Eos_init and Eos

Eos_init

Eos_wrapped

There is only one subunit

Specific implementation

Eos

Another implementation, which will have its own Eos and Eos_init etc.


The Config File

Declare solution variables, fluxes

Declare runtime parametersSets defaults

Lists required, requested or exclusive

units

Config files are additive down the

directory tree - no replacements


FLASH Setup Script: Implements Architecture

Python code links together needed physics and tools for a problem

Parses Config files to Determine a self consistent set of units to include If a unit has multiple implementations, finds out which

implementation to include get list of parameters from units Determines solution data storage

Creates a file defining global constants set at build time

Builds infrastructure for mapping runtime parameters to constants as needed

Configures Makefiles properly


Pulling it All Together

Choose a problem simulationRun setup to configure that problem

Everything is in a new top-level directory ‘object’

MakeRun

flash.par for runtime parametersDefaults already set from particular units


Adding a new Simulation

A basic problem setupConfig file

Required physics modules

flash.par Default list runtime parameter configuration

Simulation_initBlock Initial conditions for the problem set block by block

Many other possible files: Driver, Refinement algorithms, User defined boundary

conditions

Any files in setup take precedence

More about setting up your own problem in the hands on session later this afternoon


Provided Driver

Provided: Second order, state form, strang split

InitializeLoop over timesteps evolve adjust dt output visualizeEnd loopFinalize

flash.F90

set time stepdo physicsset time steprepeat physicsMesh_updateGrid

evolve.F90

FLASH2 FLASH3

Flash.F90

Driver_initFlashDriver_evolveFlashDriver_finalizeFlash

Loop over timesteps set time step call physics set time step repeat physics Grid_updateRefinement adjust dt outputEnd loop

Driver_evolveFlash.F90

Driver Unit


Flash Grids : Paramesh

FLASH original design was Paramesh specific Block Structured All blocks have same dimensions Blocks at different refinement levels have

different grid spacings and thus cover different fractions of the physical domain

Fixed sized blocks specified at compile time Every section of the code assumed knowledge

of block size Good for capturing shocks Currently removing fixed block size

requirement from code to open door for other mesh packages like a uniform grid, squishy grid and patched base grid

In choosing Paramesh, the original FLASH code architects chose simplicity of the Paramesh structure over a patch based

mesh.


Infrastructure : Mesh Packages in Flash

Paramesh

Block Structured Fixed sized blocks Specified at compile time Not more than one level jump at fine

coarse boundaries

Uniform Grid

one block per proc No AMR related overhead


Building and Running an Application

Grid mesh

I/O

RuntimeInputs

Profiler

Simulation/setup

Driver

Physics


IO/Viz Outline

I/O Challenges with Scientific Computing speed, portability, file size serial, parallel multi-file, MPI-IO

I/O Libraries hdf5 parallel-netcdf

FLASH I/O specifics parameter input file checkpointing and restarts plotfiles

Visualization runtime visualization fidlr3 flashView


I/O Challenges: Portability

Portability big endian, little endian

size of types vary

data alignment in memory

4 byte int 2 byte int

0 4 8 12 16 20 24

int int int double

Taking a basic binary output file from one machine and trying to use it on another often causes problems


I/O challenges: File size and Speed

File size Files are huge --in FLASH a typical checkpoint file

typically ~5 GB and up to 50 GB for a research simulation (plotfiles ~2GB)

Need access to large storage areas Data transfer takes lots of time (~.5 MB/sec from labs)

I/O can be a major performance bottle neck slow file system writing to disk takes time

It isn’t hard to have speed, portability or usability. It is hard to have speed, portability and usability in the same

implementation


Serial I/O

0 1 2 3 4

File

processors

Each processor sends its data to the master who then writes the data to a file

Advantages Don’t need a parallel file system Simple

Disadvantages Not scalable Not Efficient

5


Parallel I/O Multi-file

0 1 2 3 4

File File File File File

processors

Each processor writes its own data to a separate file Advantages

Fast! Disadvantages

can quickly accumulate many files hard to manage requires post processing

5

File


Parallel I/O Single-file

0 1 2 3 4

File

processors

Each processor writes its own data to the same file using MPI-IO mapping

Advantages single file scalable (to how many procs is debatable...)

Disadvantages requires MPI-IO mapping or other higher level libraries

5


Parallel I/O single file

3 5 2 9 2 4 3 1 9 8 2 4

0 1 2 3 4 5processors

array of data

Each processor writes to a section of a data array. Each must know its offset from the beginning of the

array and the number of elements to write


I/O Libraries

FLASH works with 2 different I/O libraries HDF5 Parallel-NetCDF

Use MPI-IO mappings Both Portable libraries Both can translate data between systems Scientific Data mostly stored in multidimensional arrays

FLASH3 also supports a basic direct FORTRAN I/O -- use only as a last resort!


FLASH I/O

FLASH can use either HDF5 or Parallel-NetCDF for I/O

Initially very optimistic about PnetCDF because it appeared to be ~ twice as fast as HDF5

Source :J.B. Gallagher on ASC White


Parameter Input File - flash.par

To change parameters in a simulation edit the flash.par file

Defaults are set in Config files and any value in the flash.par overides the Config values

Parameter MUST be declared in Config file!!! Controls parameters for all units (examples below)

Grid parameters - lrefine_max, lrefine_min, geometry Physics parameters - flame_speed I/O parameters - restart, checkpointFileNumber


Flash Checkpoint (restart) Files

Checkpoint file -- a file that saves all the data you need to restart your simulation from the exact state it left off Grid Data

refinement level nodetype (parent, child) bound box coordinates unknown variables (density, pressure, temperature etc)

Simulation Data time dt timestep

MetaData time stamp build directory setup problem etc

For any number of reasons your simulation could be interrupted - the machine crashes, the queue window closes, the machine runs out of disk

space etc. Rather than starting again from the beginning of your simulation you can restart your problem using a

flash checkpoint files


Checkpoint Parameters

There are a number of ways to control the occurrence and spacing of checkpoint dumps (names have been clarified for FLASH3) timesteps between dumps - (parameter: checkpointFileIntervalStep) simulation time seconds between dumps - (parameter:

checkpointFileIntervalTime) wall clock seconds between dumps - (parameter: wall_clock_checkpoint)

very useful when running in a queue or when only a certain amount of time is available on a machine

creating a file called .dump_checkpoint will force a checkpoint file to be dumped immediately

creating a .dump_restart file will force a checkpoint and halt the code rolling checkpoint ability - (parameter: rolling_checkpoint)

checkpoint files are big! 5 - 50GB for a large science run this feature allows you to keep only a specified number of checkpoints to save disk

space

The FLASH code support website gives detailed explanations of all runtime parameters


Restarting a Simulation

Because the checkpoint files save the exact state of the simulation you can restart your simulation from any FLASH checkpoint file

Can start with Parallel IO if previous run was written in serial Can restart with a different number of processors than the

previous run

To restart a simulation, in your flash.par file set *the runtime parameter ‘restart’ = .true.

*the checkpointFileNumber parameter to the desired checkpoint number


Restarting on a different number of processors

3 5 2 9 2 4 3 1 9 8 2 4

0 1 2 3 4 5processors

array of data

3 5 2 9 2 4 3 1 9 8 2 4

0 1 2processors

array of data

restart


Flash Plotfiles and Particle Files

Plotfiles are smaller than checkpoints Only output variables needed for visualization or analysis Data stored in single precision

Control plotfile frequency with the runtime parameters plotfileNumber plotfileIntervalStep plotfileIntervalTime

Particle files stored in single precision Control particle file frequency with the runtime parameters

particleFileNumber particleFileIntervalStep particleFileIntervalTime

Plotfiles and particle files output data for visualization and analysis purposes


Visualization


Visualization

Runtime Visualization Fidlr3 FlashView


Runtime Visualization

Runtime Visualization module not automatically included -- can

add it into your Modules file to set up problem with Runtime Viz

Allows user a quick and dirty view of the simulation

Only requires the widely available png library

Creates png files in current directory during runtime


Fidlr3 (FLASH IDL Routines)

IDL tool for 2d visualization Available with FLASH directly Can step through plots to view

sequences or see movies Can visualize 3d data by looking at

slices Make histograms Additionally, if you have access to

idl and want to do more analysis you can use the routines to read in data, statistical analysis, max min vaules

Fidlr is an application written in idl to visualize Flash checkpoint and plotfiles


Example of 3d slice taken by FILDR3


FlashView

Features Reads native FLASH plotfiles XYZ cut planes features so entire range of data for a

variable can be projected on a plane Renders 3d iso-surfaces of Flash Data Fully parallel (both reading the file and calculations) Parallelism allows for the ability to look at very large data

sets interactively Displays mulitple (up to 5) variables at the same time various color maps, zoom, rotation, translation etc shows grid structure movies making Currently only supports hdf5 format

* Yes ... we think it is difficult to install too...* Key is to install exact version of recommended packages

* Ask Dan if you have questions

FlashView is a 3d visualization package developed for FLASH data at Argonne National Lab by Mike Papka and Randy Hudson


FlashView

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.


Example of an isosurface (single variable)


Another example with two variables


Another example/ isosurface with cut plane


Isosurface with Grid


Issues with Cell-centered vs. Corner Interpolated Data


Issues with cell-centered vs. corner interpolated data

Documents

Inner Workings And Architecture of FLASH