Outline

March 27-30, 2006Astronum2006 - Palm Springs 1

Numerical simulations of astrophysical plasmas :Numerical simulations of astrophysical plasmas :status and perspectives of the Saclay/DAPNIA status and perspectives of the Saclay/DAPNIA

software projectsoftware project

E. Audit, D. Pomarède, R. Teyssier, B. ThoorisCEA/DAPNIA - Saclay

Astronum2006 2D. Pomarède, CEA/DAPNIA

•Overview of numerical simulations in Saclay

•Cosmological structures formation : the RAMSES code

•Dynamics of the interstellar medium : the HERACLES code

•Numerical simulations software project

data handling

visualization

•Perspectives

Outline


Numerical simulations in Saclay•An extensive program of simulations of astrophysical plasmas is

conducted at the Saclay/DAPNIA Laboratory

COSMOLOGICALSTRUCTURESFORMATION

RAMSES

STELLAREVOLUTION

ASH

DYNAMICSOF THE ISMHERACLES

FORMATION OFPROTO-PLANETARY

DISKSFARGO

•It covers a vast spectrum of challenging problems at various scales and has its foundations in a suite of independent numerical codes :


Numerical simulations in Saclay : overview•Cosmological structures

– The formation of the large structures in the Universe– Interplay of the dark matter component and the baryon gas– RAMSES code (R. Teyssier) : N-body and hydrodynamical

simulations

•The Interstellar Medium– The formation and dynamics of molecular clouds– Radiation Hydrodynamics– HERACLES code (E. Audit et al.) : radiative transfer coupled to

hydrodynamics

•Stellar evolution– MHD of the Sun– Participation (A.S. Brun) to ASH developments (J. Toomre et al.)

•Proto-planetary systems– Study of the disk-planet tidal interactions– FARGO code (F. Masset) : 2D & 3D hydrodynamics


Numerical simulations in Saclay : overview•Computing resources

– CCRT (CEA Supercomputing Center) :• HP/OSF1 256 quadri-processors alpha nodes• Linux 128 quadri-processors AMD Opteron

•“Horizon Project” resources (cosmology)– IDRIS (CNRS Supercomputing Center)

• 1024 processors Power4 cluster– MareNostrum (Barcelona Supercomputing Center)

• 2406 dual 64-bit processor nodes @ 2.2GHz, ~42 TeraFlops• the most powerful supercomputer in Europe

•Software Project– Developments are now addressed in the context of a

software engineering project aiming at the optimization and sharing of expertise for various components :

Algorithms parallelization/optimization Data Handling Database Post-treatment Visualization


The RAMSES code•A hybrid simulation code

– N-body treatment to solve the Dark Matter (DM) dynamics– Hydrodynamical treatment of the baryonic component

•Position of the problem :– Objective is to study the structure formation in the Universe with high

spatial resolution– DM is believed to be the dominant component in mass of the

cosmological density field, with only a small fraction ~10% in baryons– At intermediate scales such as galaxy clusters DM stills dominates

but a gaseous component is introduced (constraints from observations of hot intracluster medium)

– At smaller scales, gas-cooling and fluid dynamics play a dominant role in the structure of galaxy-size objects

• at 1st order, baryons = hydrostatic ionized plasma trapped in DM gravitational potential wells

• complex hydrodynamical processes require accurate treatment : shock heating atomic radiation cooling star formation


The RAMSES code•Resolution requirements

– a volume of 100 Mpc h-1 side with haloes simulated with 10 resolution elements requires a spatial resolution of 10 kpc h-1

dynamical range of 104

– 100 particles/galaxy 2563 particles in total

•Algorithms– F90 parallelized with MPI or OpenMP

– Based entirely on Adaptive Mesh Refinement (AMR) technique

– Tree based data structure allowing recursive refinements on a cell-by-cell basis

– DM particles considered as a collisionless N-body system described by the Vlasov-Poisson equations

– Hydrodynamical solver based on a 2nd-order Godunov method

– With 4.1107 cells a formal resolution of 81923 is reached at the 14th Level

•Ref : R. Teyssier, A&A 385, 337-364 (2002)

The RAMSES AMRlevel 2

level 3

level 5

level 9

level 14

basic element of AMR structure :group of 2dim

sibling cells called “octs”

level 11

The RAMSES AMR

level 9 to level 14highest granularity reach

213 =8192 cells in each direction

RAMSES simulations of cosmological structuresRecent results obtained with RAMSES :“The History of the Baryon Budget – Cosmic Logistics in a Hierarchical Universe”, Y. Rasera and R.

Teyssier, accepted for publication in A&A, astro-ph/0505473 (23 May 2005)

“Kinematic Dynamos using Constrained Transport with High Order Godunov Schemes and Adaptive Mesh Refinement”, R. Teyssier, S. Fromang, E. Dormy, accepted for publication in Journal of Computa-tional Physics, astro-ph/0601715 (31 Jan 06)

extension to the modeling of magnetic field evolution (MHD)

gas density zooming sequence : from large scale filamentary structures to spiral discs


The HERACLES code

•A 3-D RHD code written in F90 parallelized with MPI•Implemented in Cartesian, cylindrical, and spherical coordinates•In current implementation only regular mesh grids•Solves the equations of radiative transfer coupled to

hydrodynamics : fluid evolution is determined by the classical conservations

equations (mass, momentum, energy) + source terms characterizing the momentum and energy

exchanges between the fluid and the radiation

•The transfer equations are solved by a second order Godunov type method and integrated implicitly using iterative solvers

•Reference : “HERACLES: a three dimensional radiation hydrodynamics code” ,M. González, E. Audit, and P. Huynh, to be published


HERACLES - simulations of ISM dynamics•Current studies focus on the thermal fragmentation of turbulent

flows of interstellar hydrogen neutral atomic hydrogen (HI) represents more than 50% of the ISM molecular clouds form through the condensation of a warm neutral

phase (WNM) into a cold medium (CNM)

•Thermal processes : heating : photo-electric effect on small grains and poly-aromatic

hydrocarbons due to the far-ultraviolet galactic radiation cooling by molecules and various processes

•Initial conditions : collision between two turbulent streams (converging flow) of WNM

triggers the formation of CNM structures

•Resolution requirements : 4 different spatial scales are involved1. cooling length of the WNM typically ~ 10-20 pc2. typical size of CNM fragments ~ 0.1 pc3. conduction length in the 10-1-10-3 pc range4. size of the shocked layer ~ 10-3 pc dynamic range of 104


HERACLES - simulations of ISM dynamics•First studies conducted with 2D simulations

•Analysis of structure morphology, internal velocity dispersion, fraction of gas in the various states in

E. Audit and P. Hennebelle, A&A 443,1-13 (2005)

weakly turbulent forcing very turbulent forcing

Fragments of cold, high-density gas connected by thin layers of low-density gas. Sharp thermal fronts bound the structures and connect them to the warm surrounding medium.

Complex density field. The different phases are highly interwoven with pockets of warm gas embedded in filaments of cooler gas.

New, high resolution 3D simulations

density fieldvolume of size 15 pc

grid 12003

HERACLES simulations of ISM dynamics

2D slice

the limited dynamical range leads to under-estimation of the highest density reached during supersonic collisions


Numerical simulations software project

•Software developments are managed in the context of an indepen-dent engineering project. The objective is to provide a core of software modules useable by the various simulations tools. This includes data handling, post-treatment, visualization, parallelization optimization,…

RAMSESA

SH

HE

RA

CL

ES

FARGO

CORE SOFTWAREMODULESI/O, Graphics,

Algorithms


Data Handling

•A unique format has been chosen for the data produced by all the simulation codes : HDF5, the Hierarchical Data Format developed by the NCSA National Center for Supercomputing Applications.

•HDF5 is a general purpose library and file format for storing scientific data

•Two primary objects are handled : Datasets = multidimensional array of data elements Groups = structures

•Efficient storage and I/O : the library is tuned and adapted to read and write data efficiently on parallel computing systems

•Free, open source software, including utilities (browser)•Plenty of applications have interfaces to HDF5, in particular in the

graphics/analysis area : IDL, ParaView, MATLAB, …

browser of HDF5 files

Visualization and browsing of HDF5 files in IDL


Visualization•Visualization is performed in the framework of IDL•Baseline is to use the object-oriented programming offered by IDL’s Object

Graphics•The visualization tool is a graphical widget

users act on the objects to modify their attributes through menus, droplist, sliders, buttons & dialog fields

•Input data : scalar and vector fields on regular grids and on AMR tree, particles•Once the data are loaded in memory, various implementations of the visualization

objects are proposed based on different classes of the IDL library : IDLgrPolygon : set of polygons and vertices, to hold an iso-surface IDLgrVolume : mapping from a 3D array of data to a 3D array of voxel colors projected

to two dimensions IDLgrPolyline : useful for hedgehog displays of vector fields IDLgrSurface, IDLgrImage, IDLgrPlot, IDLgrSymbol

•Basic graphics objects are inserted in Models that can be transformed (rotated, scaled and/or translated) : this forms the basis for spatial navigability around the 3D objects on display.

IDLgrLight objects represent sources of illumination : ambient, positional, directional, spotlight

•Two methods of rendering objects : hardware accelerator using OpenGL, highly efficient on a local machine software rendering applies when running on a distant computer

the graphical widget

menusslidersbuttonsdroplistdialog fields

surface object

axis object

plot objectimage object

navigation

profile

Select :densitypressurevelocityradiative energyradiative fluxparticlescustom variable

Visualization of regular grids and AMR data

Max grid :regular 3D 1200x1200x1200 = 1.728 109 cellsregular 2D 10000x10000AMR up to level 10 projected in grid 1024x1024x1024

image of 2D 10000x10000 HERACLES simulation

surface of a density slice on a restricted region of the RAMSES AMR (level 7 to level 11)

Visualization of iso-surfaces

Simulation of ISM hydrogen turbulencesdensity field - volume of size 15 pc

grid 12003

Interactive setting of the contour value on the density histogram

Visualization of AMR data

Volume ProjectionBox of size 100 h-1 Mpc

Visualization of 3D vector fields

RAMSES cosmological simulationbox of size 100 h-1 Mpc

HERACLES simulation of ISM turbulences

Hedgehog display of hydrodynamical velocity fieldBlue semi-transparent volume = iso-surface of the density

Visualization of particles

particle cloud display of a dark matter sampleyellow semi-transparent volume = iso-surface of the hydrodynami-cal density field

DM haloes


Conclusions and outlooks

•An extensive program of simulations of astrophysical plasmas is pursued•New developments are managed in the framework of a software engineering

project•Solutions to current limitations will be studied :

– HERACLES : implementation of multiple grid algorithm to reach the effective 104 mesh resolution in 3D simulations

– RAMSES : algorithm optimization to improve load balancing– Visualization :

• optimization of memory management to access larger data sets• parallelization to improve speed (fastDL/mpiDL solutions from RSI)• implementation of multiple grid

•New projects :– Extend the use of the core software modules to FARGO and ASH– Database : development of a Virtual Observatory for cosmological

simulations (Horizon project)– Openings towards other domains :

• HERACLES application to laser fusion and plasma physics

Documents

Outline