Embed Size (px)
Citation preview
CScADS 2012
FLASH, HEDP, and Future NeedsJuly 30th, 2012, CScADS, Snowbird, UT
Anthony ScopatzThe FLASH Center
The University of [email protected]
1
What is FLASH?
FLASH code is a modular, parallel multiphysicssimulation code for modeling terrestrial and astrophysicalplasmas. Features include:
2
What is FLASH?
FLASH code is a modular, parallel multiphysicssimulation code for modeling terrestrial and astrophysicalplasmas. Features include:
•Grid: Uniform Grid, AMR
2
What is FLASH?
FLASH code is a modular, parallel multiphysicssimulation code for modeling terrestrial and astrophysicalplasmas. Features include:
•Grid: Uniform Grid, AMR
•Equation of State: Ideal gas, Multimaterial
2
What is FLASH?
FLASH code is a modular, parallel multiphysicssimulation code for modeling terrestrial and astrophysicalplasmas. Features include:
•Grid: Uniform Grid, AMR
•Equation of State: Ideal gas, Multimaterial
•Laser ray trace package
2
What is FLASH?
FLASH code is a modular, parallel multiphysicssimulation code for modeling terrestrial and astrophysicalplasmas. Features include:
•Grid: Uniform Grid, AMR
•Equation of State: Ideal gas, Multimaterial
•Laser ray trace package
•Nuclear Burning
2
What is FLASH?
3
FLASH Architecture
FLASH compiles simulation-specific binaries from userdefined code Units.
4
FLASH Architecture
FLASH compiles simulation-specific binaries from userdefined code Units.
Code Units are mix of Fortran, C, C++, and Python andhave an OO-esque inheritance model via posix directories.
4
FLASH Architecture
FLASH compiles simulation-specific binaries from userdefined code Units.
Code Units are mix of Fortran, C, C++, and Python andhave an OO-esque inheritance model via posix directories.
To execute a FLASH simulation you must:
•setup (pure python),
4
FLASH Architecture
FLASH compiles simulation-specific binaries from userdefined code Units.
Code Units are mix of Fortran, C, C++, and Python andhave an OO-esque inheritance model via posix directories.
To execute a FLASH simulation you must:
•setup (pure python),
•build (make),
4
FLASH Architecture
FLASH compiles simulation-specific binaries from userdefined code Units.
Code Units are mix of Fortran, C, C++, and Python andhave an OO-esque inheritance model via posix directories.
To execute a FLASH simulation you must:
•setup (pure python),
•build (make),
•and run (flash binary).
4
What is HEDP?
(H)igh (E)nergy (D)ensity (P)hysics is the study of hot,dense plasmas where temp >10^6 C & pres >10^8 ATM.
5
What is HEDP?
(H)igh (E)nergy (D)ensity (P)hysics is the study of hot,dense plasmas where temp >10^6 C & pres >10^8 ATM.
Many features have been added to FLASH enabling HEDPsimulations in the past couple of years.
5
What is HEDP?
(H)igh (E)nergy (D)ensity (P)hysics is the study of hot,dense plasmas where temp >10^6 C & pres >10^8 ATM.
Many features have been added to FLASH enabling HEDPsimulations in the past couple of years.
Specifically, we are interested in the related field ofLaboratory Astrophysics.
5
What is HEDP?
(H)igh (E)nergy (D)ensity (P)hysics is the study of hot,dense plasmas where temp >10^6 C & pres >10^8 ATM.
Many features have been added to FLASH enabling HEDPsimulations in the past couple of years.
Specifically, we are interested in the related field ofLaboratory Astrophysics.
•Many of the fundamental laws of plasma physics haveno implicit scale.
5
Problem Statement
The is a small, but measurable intergalactic magnetic field(femto- to pico-gauss) [origin unexplained].
6
Problem Statement
The is a small, but measurable intergalactic magnetic field(femto- to pico-gauss) [origin unexplained].
One theory is that they are born from asymmetric shockscaused when galaxies or galaxy clusters collide.
6
Problem Statement
The is a small, but measurable intergalactic magnetic field(femto- to pico-gauss) [origin unexplained].
One theory is that they are born from asymmetric shockscaused when galaxies or galaxy clusters collide.
Generation of magnetic fields from a neutral plasma hasrecently been demonstrated in the lab [1].
1. G. Gregori, et al., Generation of scaled protogalactic seed magnetic fields in laser-produced shockwaves, Nature 481 (2012) 480-483.
6
Biermann Battery Mechanism
The generalized Ohm's law sets the strength of the electricfield in the MHD approximation. Only the Battery termcan produce magnetic fields from an initiallyunmagnetized plasma:
E =u£B + ´j + 1nee
j£B ¡ rPene e
7
Biermann Battery Mechanism
The generalized Ohm's law sets the strength of the electricfield in the MHD approximation. Only the Battery termcan produce magnetic fields from an initiallyunmagnetized plasma:
E =u£B + ´j + 1nee
j£B ¡ rPene e
Faraday's law relates the electric field to the rate of changeof the magnetic field:
³∂B∂t
´Biermann=c r£
³rPenee
´=c
rPe £rneen 2
e
7
Experimental Setup
8
Shadowgraphy: Simulation & Data
9
Shadowgraphy
Computationally, shadowgraphy is a little annoying. Theintensity in the image is proportional to:
I/Z n2
n1
µ∂2 ne
∂x2+
∂2 ne
∂y2
¶dn
10
Shadowgraphy
Computationally, shadowgraphy is a little annoying. Theintensity in the image is proportional to:
I/Z n2
n1
µ∂2 ne
∂x2+
∂2 ne
∂y2
¶dn
Some vector calculus later:
I/Z n2
n1
¡r2 ne¡r(rne ¢n̂) ¢n̂
¢dn
10
Shadowgraphy
Computationally, shadowgraphy is a little annoying. Theintensity in the image is proportional to:
I/Z n2
n1
µ∂2 ne
∂x2+
∂2 ne
∂y2
¶dn
Some vector calculus later:
I/Z n2
n1
¡r2 ne¡r(rne ¢n̂) ¢n̂
¢dn
This is not quite volumetric ray tracing...
10
FLASH on BG/P
• Intrepid BG/P: 4 cores/node, 2 GB/node, 40,960 nodes
•FLASH has run on Intrepid for the last several years
•Scales to the whole machine
•MPI-only is sufficient
•Run in VN mode (4 MPI ranks/node)
11
FLASH on BG/Q
•Mira BG/Q 4 hw threads/core, 16 cores/node, 16GB/node, 49,152 nodes
•MPI-only approach not suitable for BG/Q
•OpenMP directives have been added to FLASH totake advantage of the additional intra-nodeparallelism
12
Scaling
13
Resource Usage
•We foresee needing to use 20% of Mira (or 1 Intrepid)due to the increased number of physics models included.
14
Resource Usage
•We foresee needing to use 20% of Mira (or 1 Intrepid)due to the increased number of physics models included.
•During a full 3D run, the checkpoint files will be ~68GB and the plot files will be ~6.8 GB. We anticipateneeding 36 checkpoints and 144 plots.
14
Resource Usage
•We foresee needing to use 20% of Mira (or 1 Intrepid)due to the increased number of physics models included.
•During a full 3D run, the checkpoint files will be ~68GB and the plot files will be ~6.8 GB. We anticipateneeding 36 checkpoints and 144 plots.
•By performing in situ simulated diagnostics, huge datareductions are possible.
14
Rough Plan
Over the next 6 months, I will be attempting to:
•Add capabilities to yt to support simulated diagnosticcalculations,
15
Rough Plan
Over the next 6 months, I will be attempting to:
•Add capabilities to yt to support simulated diagnosticcalculations,
•Extend yt and/or h5py to a GLEAN backend,
15
Rough Plan
Over the next 6 months, I will be attempting to:
•Add capabilities to yt to support simulated diagnosticcalculations,
•Extend yt and/or h5py to a GLEAN backend,
•Create infrastructure for co- or post-processing ofFLASH data for initial diagnostics (Schlieren,shadowgraphy, and magnetic field probes).
15
Acknowledgements
•FLASH: Milad Fatenejad, Chris Daley, Don Lamb,Anshu Dubey
•The University of Oxford: Gianluca Gregori, JenaMeineke
•yt: Matt Turk, John ZuHone
•ANL: Venkat Vishwanath
16
Questions
17
![Towards an Appropriable CSCW Tool Ecology ... - …...Flash animations for the website Newgrounds [13]. Pipeline helped meet the needs of Flash animators and was later released as](https://img.pdfslide.us/doc/110x75/5f0522a57e708231d41171c0/towards-an-appropriable-cscw-tool-ecology-flash-animations-for-the-website.jpg)
