What’s FLASH? How does FLASH do it?

An Advanced Simulation & Computing (ASC) Academic Strategic Alliances Program (ASAP) Center

at The University of Chicago

The Center for Astrophysical Thermonuclear Flashes

What’s FLASH?How does FLASH do it?

Katherine M Riley

Code Group

The ASC/Alliances Center for Astrophysical Thermonuclear FlashesThe University of Chicago

Overview

What is FLASH

Basics of FLASH2 Architecture

Basics behind a problem setup

Walk away with: Enough of an idea of these concepts to be

able to look at a sample setup and

understand what is happening.


FLASH: The Application

To simulate matter accreted onto the surfaces of compact stars, nuclear ignition of the accumulated (and possibly stellar) material, and subsequent evolution of the star’s interior, surface, and exterior. Novae (on white dwarf surfaces) Type 1a supernovae (in white dwarf interiors) X-ray bursts (on neutron star surfaces)


FLASH Results

Cellular detonations

Compressible turbulence

Helium burning on neutron starsRayleigh-Taylor instability

Intracluster interactions

Flame-vortex interactions

Nova outbursts on white dwarfs

White Dwarf deflagration

Shocked cylinder

Compressible reactive flow Wide range of length of time scales Many interacting physical processes Only indirect validation possible for

the astrophysics Many people in collaboration


What FLASH Provides

PhysicsHydrodynamics

PPM MHD Relativistic PPM

Nuclear PhysicsGravityCosmologyParticles

InfrastructureSetupAMR: ParameshRegular testingParallel I/O

hdf5, pnetcdf,

ProfilingRuntime and post-

processing visualization


A Little FLASH History

FLASH0 Paramesh2, Prometheus and EOS/Burn

FLASH1 Smoothing out the smash First form of module architecture & inheritance

FLASH2 Untangle modules from each other (Grid)

dBase Concept of levels of users

FLASH3 Stricter interface control & module architecture Taming the database

BAM


FLASH2 Audiences

FLASH Application

DeveloperApplicationProgrammer

End User

• Develop physics modules• Talk to grid• Module communication

• Initialize setup• Boundary conditions• Basic data structure• (Might add a kernel)

• Works on just about everything• Grid development• Data access• Architecture


FLASH2 Code Basics

An application code, composed of units/modules. Particular modules are set up together to run different physics problems.

Performance, Testing, Usability, PortabilityFortran, C, Python, …

560,000* lines of code75% code, 25% comment

Very portableScaling to 1000’s of procs

* Internal Release


Basic Computational Unit : Block

The adaptive grid is composed of blocks

All blocks: same dimensions

Cover different fraction of the physical domain.

Kevin Olson will talk about this more.


First Look at FLASH

‘The tar-ball’‘source’ directories

modules or groups of modules

Post-processing tools, docs, setups

SetupFLASH architecture toolSelects and sets up these modules

Collecting variables, runtime parameters, etc

Meshmesh_init()

mesh_guardcells()mesh_updateRef()

mesh_fluxConserve()

MaterialsHydro Source_termsGravityinit()

tstep()hydro3d()

init()tstep()

grav3d()

init()tstep()

src_terms()

eos3d()eos1d()

eos()

Explicit mhd

PPM

Diffuse

Unsplit

WENO PPM

constant

point_mass

Poisson

Multigrid

Gamma

Helmholtz

...

burn

cool

heat

iso13

...

Multipole

DatabasedBase_init()

dBaseGetData()dBasePutData()dBaseProperty()

Particlesinit()

advance()

Visualizationinit()

render()

IOinit()

Checkpoint_write()Checkpoint_read()

MHDinit()

Driverinit()

dBaseGetData()dBasePutData()dBaseProperty()

Cosmologyinit()

Structure of FLASH Modules

(not exact!)


What’s a FLASH Module?

FLASH basic architecture unit: ModulesComponent of the FLASH code providing a

particular functionalityDifferent combinations of modules are used for

particular problem setupsEx: driver, hydro, mesh, dBase, I/O

Fake inheritance by use of directory structureModules communicate

DriverVariable Database


Abstract FLASH2 Module

1. Meta-data (Configuration Info)• Interface with driver and setup• Variable/parameter registration• Variable attributes• Module Requirements

2. Interface Wrapper• Exchange with variable database• Prep data for kernels

3. Physics Kernel(s)• Single patch, single proc functions written in any language• Can be sub-classed

FLASH Component

Database mesh

FLASH Application

driver

Collection of Flash2 Modules


Module Implementations

FLASH2 Modules are directory treessource/hydro/explicit/split/ppm

Each level might have sourceSource relevant for all

directories/implementations below

Preserves interfacesAllows flexible implementations


Inheritance Through Directories: Hydro

init hydro tstep

Explicit

implemtation

shocktstep

Split

DeltaForm

hydro

An ‘empty’ hydro init, hydro, tstep are defaults on top of the directory tree.

Hydro/Explicit Replaces tstep Introduces ‘shock’ No hydro Implemented yet!

Hydro/Explicit/Split hydro implemented Uses general explicit tstep Uses general shock Replaces init

init

hydro

implemtation

tstep


The Module Config File

Declare solution variables, fluxes

Declare runtime parametersSets defaults

Lists required, exclusive modules

Config files are additive down the

directory tree - no replacements


Setup Building an Application

MeshDatabase

Gravity

SourceTerms

Materials

HydroParticles

I/O Vis

MHD

ConfigurationTool

(Setup)

Driver


FLASH Setup: Implements Architecture

Python code links together needed physics and tools for a problemobject

Traverses modules to get implementations

Determines solution data storage listCreates list of parameters from modulesConfigures Makefiles properly


Accessing the Mesh

Physics and tools access some scope of meshDatabase

Unifies method for module to access data out of its scope

Mesh data Solution Data, Grid information

Runtime parameters

Mesh InterfaceWrappers to the public grid routines

initialize, guardcell, fluxconserve, updateRefinment


FLASH Audiences

FLASH Application

End User

• Initialize setup• Boundary conditions• Basic data structure• (Might add a kernel)

Solidify information so far• Modules and setup• How do they combine

to make create an application? That runs?

• Go through a little of that.


Simple FLASH Flow

DriverInit Evolve TimeStep

EvolveHydro

SourceTerms…

UpdateGrid

HydroPrepare data for kernels SourceTerms

Burn Heat Cool Ioniz…

BurnburnBlock

dBase

Mesh

Variable Database

ParameshupdateGridghostCellFillfluxConserve

hydroBlock

FunctionData


Pulling it All Together

Choose a problem setupRun setup to configure that problem

Everything is in a new top-level directory ‘object’

MakeRun

Flash.par for runtime parametersDefaults already set from particular modules


Setups

A basic problem setupConfig file

Required physics modules

Flash.par Default list runtime parameter configuration

Init_block Initial conditions for the problem set block by block

Many other possible files: Driver, Refinement algorithms, User defined boundary

conditions

Any files in setup take precedence


Provided Driver

Provided:Second order,

state form, strang split

Initialize()Loop over timesteps evolvePhysics() timestep() output() visualize()End loopFinalize()

flash.F90

New driversPut in setupsWelcome

contributions

set time stephydrosourceTermscosmologyradiationparticlesgravityset time step(repeat physics)Mesh_updateGrid

evolve.F90


What’s Next

Covered the basics of what FLASH can doNow - How does it do it?

Alan Calder

Documents

What’s FLASH? How does FLASH do it?