An Advanced Simulation and Computation (ASC) Academic Strategic Alliances Program (ASAP) Center at The University of Chicago The Center for Astrophysical

An Advanced Simulation and Computation (ASC) Academic Strategic Alliances Program (ASAP) Center

at The University of Chicago

The Center for Astrophysical Thermonuclear Flashes

Overview of the Flash Center

Don Lamb

Annual Site ReviewOctober 28-29, 2004(http://flash.uchicago.edu)

The ASC/Alliances Center for Astrophysical Thermonuclear FlashesThe University of Chicago

The Ultimate Goal: Understanding Astrophysical Flashes

Picture credits: NASA

Type Ia SN: SN 1994D

Nova Cygni 1992

X-ray burst: GS 1826-24

Cosmic nucleosynthesis: how is the stuff we’re made of produced?

Guideposts to the universe: using exploding stars as “rulers”for the universe


Our “Overarching Goals”

To build a new generation code for computing astrophysical thermonuclear flashes involving compact stars Incorporate advances from CS, applied math, physicsVerify and validate code and its components

To produce great scienceAstrophysics and physicsComputational sciences [CS, Applied math, Math]

Training/education


Flash Code Highlights During Past Year

Updated Code Project Plan Primary objectives for FY04 are to

Release Flash 2.4 – Done Support addition of modules to Flash 2 required for astro

simulations of Type Ia SNe – Done Develop and implement Flash 3 Framework – Done Implement a basic set of units in Flash 3 – Done Run first application of Flash 3 (“end-to-end” test) – Done

Identified and helped to solve a series of problems w. ASC platforms (e.g., ALC and QSC)


More Flash Code Highlights

Organized and hosted Flash Tutorial Researched, selected, purchased, and installed

“ellipse” Linux cluster: Sought and received invaluable advice from CS group

at ANL Cluster has 16 nodes (32 processors) Includes 13 Tbyte RAID system Processors and graphics cards selected are identical or

similar to those used at ANL Cluster will be used primarily for code development and

visualization


Flash Code Future Objectives

Primary objectives for FY05 are to Implement remaining units in Flash 3 Create Flash 3 Developer’s Guide Explore open design issues (I/O, gravity, particles) Interact successfully w.

Astro group to support, document, maintain, and

bug-track code; and to solve platform problems Comp Phys group to ensure framework support

for new algorithms Visualization group to bring Flashview into

Flash 2 and 3


Flash CS and Visualization Highlights During Past Year

Flash selected as Marquee application for LLNL BlueGene/L

Flash 3 is now running as “benchmark application”

on Blue Gene/L Flash 3 is driving developments in CS and Viz Steadily increasing importance of visualization for

astrophysics simulations has led to creation of Visualization group w. Mike Papka as leader


Validation Highlight: 3-D Simulations of Shock-Cylinder Experiment

Basic morphology of our 2-D simulations agrees with experimental data and simulations of others, but details differ across all groups With Todd Dupont (basic physics group), we have done extensive study to determine initial conditions consistent with experimental data We have developed quantitative, relevant, practical metrics Our 3-D simulations demonstrate that flow field is fully 3-D in nature Experimenters are planning to make measure- ments in vertical plane, prompted by our simulations

Weirs et al. (2004)


Flash Comp Phys and Validation Future Objectives

Primary objectives during FY05 are to:Continue development and testing of low-Mach

number, implicit hydro solverContinue development and testing of level-set

flame-tracking module Improve flame module, taking into account results

of “Astrophysical Flame Laboratory” simulationsContinue effort to build local radiation transfer

capability for Type Ia SN light curves and spectraPublish results of validation studies of LANL shock

cylinder experiments Initiate further validation studies w. LANL and LLNL


Flash Astrophysics Highlights During Past Year

Updated Astrophysics project plan Continued work on novae (Plewa, Rubini, Pan,

Truran) Won INCITE award of 2.7 million cpu-hours on

Seaborg for Flash simulations of Type Ia SNe Carried out studies of sub-grid flame model using

“Computational Astrophysical Flame Laboratory” (Messer)

Carried out whole-star simulations of Type Ia SN deflagration phase (Calder) Octant models of central ignition Whole-Star models of central ignition Whole-Star models of off-center ignition

Discovered possible new Type Ia SN mechanism (Plewa)


Discovery of New Type Ia SN Mechanism

Whole-star 3-D simulations show that slightly off-center ignition creates hot bubble of material that rises rapidly to surface of white dwarf

As bubble breaks through surface, hot material flows across surface of star at high velocity, pushing material in surface layers of star ahead of it

Flow converges at opposite point on surface of star; resulting compression raises temperature and density of surface material, initiating detonation

First model of Type Ia SN in which detonation occurs “naturally” (i.e., without being put in by hand)

Star pre-expands while hot bubble material is flowing across surface of star, so that when detonation occurs, density of star has decreased enough that both nickel and intermediate mass elements are produced


How does insensitivity to initial conditions happen? —off-center ignition accommodates some variations in initial

conditions, but more study definitely needed

What produces the range of Ni masses that are observed in Type Ia SNe that may be the origin of the Phillips relation?

—variation in initial conditions of off-center ignition produces range in masses of hot bubble, range in amounts of pre-expansion, and therefore range in amounts of Ni, but more study definitely needed

How does a detonation occur in a medium that has no walls (i.e., that is gravitationally confined)?

—gravitationally confined detonation (colliding shocks in focused flow of hot material across white dwarf surface), but more study definitely needed

Why do 1-D models do better than 2-D models and much better than 3-D models?

—bubble burns only a few percent of MWD,producing a pre-expanded star that is nearly spherically symmetric when detonation occurs; i.e., mimics 1-D model!

Answers to Key Questions Re Type Ia SNe In Gravitationally Confined Detonation Model


Criteria for Success

“Achieving scientific leadership in the chosen application discipline:” Have achieved scientific leadership in nova problem Are now a “player” in Type Ia supernova problem

“Demonstration of scientific breakthroughs and/or solutions to important problems through advanced computational science:” Demonstrated viable solution to previously unsolved problem of

pre-nova mixing Discovered promising new Type Ia supernova mechanism

(“Gravitationally Confined Detonation”)


Flash Center-Wide Team Effort on Type Ia SN Problem

Pre-2003 – Flash Code developed Pre-2003-present – Flash Code maintained 2003 – Flame capturing module developed and

implemented 2003-present – Flash Code migrated to new ASC

platforms (QSC, ALC, BlueGene/L,…) and problems solved

2003 – 3-D entire star slightly off-center simulations done


Off-Center Entire Star Simulation of Deflagration Phase of Type Ia SNe



Pre-2003 – Flash Code developed Pre-2003-present – Flash Code maintained 2003 – Flame capturing module developed and

implemented 2003-present – Flash Code migrated to new ASC

platforms (QSC, ALC, BlueGene/L,…) and problems solved

2003 – 3-D entire star slightly off-center simulations done

2003-2004 – Multi-pole gravity developed and implemented

2003-2004 – Multi-grid gravity speeded up



2004 – Tracer particles developed and implemented to determine nucleosynthetic yields

2004 – Distributed nuclear burning network being developed in order to follow detonation phase

2004 – Data transfer rate from ASC platforms improved 2004 – Cluster purchased, installed, and operating system

problems solved 2003-present – Flashview and visualization of Type Ia SN

deflagration phase simulations done (e.g., movies) 2004 – Type Ia SN post-deflagration phase 2-D simulations

done


2-D Simulation of Post-Deflagration Phase of Type Ia SNe



2004 – Tracer particles developed and implemented to determine nucleosynthetic yields

2004 – Distributed nuclear burning network being developed in order to follow detonation phase

2004 – Data transfer rate from ASC platforms improved 2004 – Cluster purchased, installed, and operating system

problems solved 2003-present – Flashview and visualization of Type Ia SN

deflagration phase simulations done (e.g., movies) 2004 – Type Ia SN post-deflagration phase 2-D simulations

done 2004 – Discovery of possible new Type Ia SN mechanism



2003-present – Boussinesq reactive flow simulations done, leading to new insights

2003-present – “Computational Astrophysical Flame Laboratory” simulations done, leading to new insights and improved sub-grid flame model


Self-Regulating Turbulent Flame

Steady-state turbulent flame speed does not depend on small-scale physics:

St gLA

St Sl

Sl=1.07 x 106 cm/s Sl=2.14 x 106 cm/s

= 2.84 x 1013 cm2 = 1.41 x 1013 cm2

Messer et al. (2004)





2004 – New diagnostics developed for “Computational Astrophysical Flame Laboratory”

2004 – First Monte Carlo radiation transfer code calculations of spectra for Flash Type Ia SN hydro simulations


First Contact Between Flash Hydro Simulations of Type Ia SNe and Observation

Kasen and Plewa (2004)





2004 – New diagnostics developed for “Computational Astrophysical Flame Laboratory”

2004 – First Monte Carlo radiation transfer code calculations of spectra for Flash Type Ia SN hydro simulations

2004-present – Development and testing of low-Mach number hydro solver (will allow us to investigate “smoldering phase”)

2004-present – Development of level-set interface capturing module


Future Flash Astrophysics Objectives

Primary scientific objectives during FY05 will be to carry out large-scale, integrated, multi-physics simulations of Type Ia SNeFocus on deflagration phase

Carry out verification studies Resolution studies of center and off-center ignition

Carry out studies of sensitivity to initial conditions Parameter study of initial offset distances and

velocities Study effect of turbulence

Carry out 2-D simulations of detonation phaseStudy nucleosynthetic yield, using tracer

particlesConstruct spectra for some Flash models


Program of Enrico Fermi Institute Mini-Symposium on Type Ia SNe


Reference Frame Column in Physics Today by Leo Kadanoff


Visibility of Flash Center in Scientific Community and

Beyond 2004-present – Award of INCITE time (2.7M cpu-hours) on Seaborg and selection of Flash code as Marquee application on BlueGene/L

Article on “Coming Crisis in Large-Scale Simulations” featuring Flash Center results will appear in a future Physics Today (Doug Post)

Educational and public outreach Major upgrade of Flash Center website (incl. menu of Flash movies) Showing of Flash Center movies at Adler Planetarium “Short Course”

for planetarium staff and directors and disseminated to many planetaria

Development of 3-D movies of wind-wave mixing in novae and deflagration phase (“jellyfish”) of Type Ia SNe

Creation of exhibit of Flash Center results and movies for Sci-Tech, Hands-On Science Museum in Aurora, IL

Invitation to provide multi-projector Flash Center movies for exhibition at Smithsonian Air and Space Museum in Washington, DC, and for dissemination to planetaria throughout U.S.


Visibility of Flash Center in Scientific Community – Refereed Papers

On Variations in the Peak Luminosity of Type Ia Supernovae (Timmes, Brown, and Truran, ApJ, 590, 83, 2003)

Response of Model and Astrophysical Flames to Curvature and Stretch (Dursi, et al., ApJ, 595, 955, 2003)

Linear Instability of Astrophysical Flames in Magnetic Fields (Dursi, ApJ, 606, 1039, 2004)

Morphology of Rising Hydrodynamic and magneto-Hydrodynamic Bubbles from Numerical Simulations (Robinson, et al. ApJ, 601, 621, 2004)

On Heavy Element Enrichment in Classical Novae (Alexakis, et al., ApJ, 602, 931, 2004)

A Comparison of High-Resolution 3D Numerical Simulations of Turbulent Rayleigh-Taylor Instability: Alpha-Group Collabpration (Dimonte, et al., Phys. Fluids, 16, 1668, 2004)

Validating Astrophysical Simulation Codes (Calder, et al. CiSE 6, 10, 2004) On the Nonlinear Evolution of Wind-Driven Gravity Waves (Alexakis, et al.

Phys. Fluids, 16, 3256, 2004) Type Ia Supernova Explosion: Gravitationally Confined Detonation (Plewa,

Calder, and Lamb, ApJ, 612, L37, 2004) Laboratory Astrophysics Experiments for Simulation Code Validation: A Case

Study (Calder, Ap&SS, in press, 2004) Type Ia Supernovae: Simulations and Nucleosynthesis (Brown, et al. Nuclear

Physics A, in press, 2004) Type Ia Supernovae: An Asymmetric Deflagration Model (Calder et al., ApJ,

submitted, 2004)


Visibility of Flash Center in Scientific Community – Talks

2 invited talks at APS/DPP in Albuquerque in October 2003 (Dwarkadas, Rosner)

3 invited talks at Chicago Workshop on Type Ia SNe in November 2003 (Khoklov, Calder, Brown)

Talk at LES Workshop at Caltech in December 2003 (Plewa) 2 oral talks at 192nd AAS Meeting in Atlanta, GA, in January 2004

(Alexakis, Calder) 4 invited talks at 5th HEDLA Meeting in Tucson, AZ, in March 2004

(Rosner, Khokhlov, Calder, Lamb) 3 invited talks at Ringberg Workshop on Nuclear Astrophysics in March

2004 (Calder, Brown, Truran) Invited talk at Workshop on Combustion and Type Ia SNe in Sedona, AZ,

in May 2004 (Plewa) 2 talks at ASC Alliance/Laboratory Verification and Validation Workshop,

La Jolla, CA, in July 2004 (Weirs, Plewa) Talk at 9th International Workshop on Physics of Compressible Turbulent

Mixing, Cambridge, UK, in July 2004 (Weirs) 3 invited talks at Conference on AMR Simulations in Cosmology, Durham,

UK, in June 2004 (Dubey, Gallagher, Sheeler)


Visibility of Flash Center in Scientific Community – Talks

(Cont.) 2 invited talks at INT Workshop on Supernova Theory and

Nucleosynthesis in Seattle in July 2004 (Lamb, Truran) 4 invited talks at INT Workshop on Type Ia SNe in Seattle in August 2004

(Bronson, Calder, Lamb, Truran) Astronomy Department seminar at Notre Dame in September 2004

(Calder) Astronomy Department colloquium at Illinois in September 2004 (Plewa) Physics Department seminar at Illinois in September 2004 (Plewa) Astronomy Department colloquium at Maryland in September 2004

(Plewa) Astronomy Department colloquium at Vanderbilt in November 2004

(Calder) Astrophysics seminar at Fermilab in Winter 2005 (Plewa) Invited talk and contributed talk at Texas Symposium on Relativistic

Astrophysics in Palo Alto, CA, December 2004 (Khokhlov, Messer) 4 contributed talks at AAS meeting in San Diego, CA, January 2005

(Brown, Messer, Truran, Zhang)


Flash Center Visitor Program

Marko Arienti (Caltech) visited for two weeks in February; he worked w. Alexei Khoklov on simulations of detonations in solids

Dave Dearborn (LLNL) visited in April; he gave a Flash Center colloquium on Earth-crossing asteroids

Peter Hoeflich (U. Texas at Austin) visited for two weeks in April and again in September; he worked w. Alexei Khokhlov to port radiation transport modules to Chicago for eventual use in simulating Type Ia SNe light curves and spectra, and participated in the EFI Mini-Symposium on Type Ia SNe

Snejana Abarzhi (Stanford Turbulence Center) visited in early May; she gave a talk on the Rayleigh-Taylor instability

Wolfgang Hillebrandt (Max Planck Institute fur Astrophysik), who is on the Flash Center External Advisory Committee, visited in early May; he gave a Cosmology Seminar on Type Ia SNe

Luigi Iapichino (Max Planck Institute fur Astrophysik), visited for a week in May; he worked w. people on the smoldering phase of Type Ia SNe


Flash Center Visitor Program (Cont.) Inma Dominquez (U. Granada, Spain) is visiting for a month;

she is learning to use the Flash Code and is working w. members of the Center on models of the progenitors of Type Ia SNe

Paul Woodward (U. Minnesota) visited in late May; he is interested in contributing his interface tracking algorithm to Flash; his group has great experience w. flow diagnostics from which we can learn a great deal; visit was very productive for both sides

Francesco Rubini (Italy) visited for six months and will return early next year to continue working on the smoldering (pre-outburst) phase of novae

Dan Kasen (Johns Hopkins) visited for one month this summer; he will be worked w. Tomek Plewa to do first radiation transfer calculations of spectra for new Type Ia SN mechanism

Marco Comparato (Italy) will visit in November and December

Yingjie Liu (Georgia Tech) will visit in August 2005 to work on level set methods for interface tracking


Flash Center Folks Are Joining the Labs

Lori Freitag (cs group) permanent staff member, LLNL Scott Wunch (basic phys group) research fellow, SNL (Livermore) Alex Heger (astro group) research fellow, T Division, LANL Andrew Siegel (code group) permanent staff member, MCS

Division, ANL Katherine Riley (code group) staff member MCS Division, ANL Frank Timmes (astro group) National Security Fellow and T-6 Group Leader, LANL Andrei Draganescu (basic phys group) staff member, SNL (Albuquerque) Greg Weirs (comp phys & validation group) staff member, SNL (Albuquerque)


Flash Center Folks Are Going To Top-Notch Positions

Ed Brown (astro group) Assistant Professor, Department of Physics and Astronomy, Michigan State University Jonathan Dursi (astro group) Postdoctoral Fellow, CITA, Toronto, Canada Andrea Mignone (astro group) postdoc in Torino, Italy Alexandros Alexakis (astro group) postdoc in NCAR Advanced Study Program Annie Sandman (solar MHD) did senior thesis w. Flash Center and graduated spring 2004, is now graduate student at Rice University


Recruitment in Past Year

New senior faculty member in A&A who is actively working on Type Ia SN problem (Alexei Khokhlov)

Visiting senior faculty member in math (Henri Berestycki)

Two new postdoctoral research associates in astro group from pool of ~ 40 applicants (Alexei Poludnenko and Ju Zhang)

Two new postdoctoral research associates in comp phys and V&V group from pool of ~ 80 applicants (Dahai Yu)

Three new staff members in code group (Katie Antypas, Jignesh Joshi, Noel Taylor)

New system administrator (Shawn Needham)

Two new graduate students to work w. Flash (first-year students Justin Johnsen in A&A and Andriy Zhyglo in Physics

Undergraduate student to work w. Flash (Jason Sherwin)


Education CS and computational science courses being taught:

A&A, CS and Math Department courses (scientific computation, numerical hydro, CFD, computational astrophysics, visualization)

CS Master’s Program courses (intro. to programming, parallel programming, adv. software engineering, Java, adv. Java)

CS Master’s Program has been a vital source for recruiting code group members (Gallagher, Needham, Sheeler)

Computer Science Department degree program in computational mathematics is in its first year and growing: Program involves courses in CS, Math, and other sciences Flash Center expects to support 2 students in this program in

2005-2006 PhD degrees awarded and expected:

Alexakis, Draganescu, Dursi, and Mignone graduated in 2004 Winn expected to graduate in June 2005


Vision for Future of Flash Center Beyond the Next Three Years

Flash Center Type Ia SN hydroand radiation transfer simulations

Supernova Acceleration Probe (SNAP) scheduled for 2013-2020


Type Ia SNe – An Overarching Theme of Flash Center

1997 – Flash Center proposed and funded 1998 – Observations of Type Ia SNe led to discovery that rate of

expansion of universe is accelerating 1997-present – Flash Code developed 2003 – Flash Center begins work on Type Ia SNe 2003 – DOE and NASA commit to Joint Dark Energy Mission (SNAP) 2004 – Flash Center discovers new Type Ia SN mechanism 2004 – First Flash Center spectra of Type Ia SNe 2005-2013 – Flash Center will be able to contribute to design of SNAP

instruments and to scientific observing strategy 2013-2020 – SNAP mission 2013-2020 – Flash Center will be able to contribute to

understanding Type Ia SNe through comparison of simulations and observations

minimizing systematic errors that will limit accuracy with which properties of Dark Energy can be determined


Specifics of Flash Center Beyond the Next Three Years

Code and CS Core support from University of Chicago Major additional support from ANL (CS and Viz) Flash code continues to be a benchmark application for world’s

fastest computers and driver for advances in CS and Viz

Comp Phys and Validation Seek support through collaborations with NNSA labs to do

validation experiments (e.g., at NIF) Strong effort this year to renew collaboration w. Bruce

Remington and others at LLNL New postdoctoral research associate has been hired to

increase activity in this area

Astrophysics Create “Type Ia SN center” at University of Chicago Support and participate in SNAP mission (e.g., in studies of

dark energy and cosmology)


Responses to 2003 Review Team Report

Recommendation Action

Strategic Direction: Develop 5-year plan for integrated simulations w. milestones Done Maintain strategic focus on Type Ia SN problem Done Prioritize implicit solvers, transport, and validation Done

Flash as a Community Code:

Develop and release explicit design documentation for Flash 3.0 Done Create a Developer’s Guide

Underway Ensure that new physics packages and code modules

are easily implemented Done Visualization should be a routine feature available

to Flash users Underway Flash Center should host annual workshops or conferences for Flash user/developer community Done Flash design and implementation plan needs to accommodate production usage of Flash as well as continuous development Done


More Responses to 2003 Review Team Report

Recommendation Action

Verification and Validation: Rigorously pursue Verification Underway Establish a V&V methodology that is an integral part of the astrophysics research and Flash development Underway

General: Keep effort to develop Flash 3.0 going Done Take advantage of existing numerical methods and techniques wherever possible Done Enhance mentoring of staff and students Underway Improve internal mechanisms for collaboration and communication, esp. wrt. graduate students Underway Strengthen visitor program Done Increase recruitment of graduate students Underway Develop “curriculum” for Flash graduate students, including formal courses as well as training and seminars Underway

More and improved office space for staff and post docs Underway


Issues and Challenges Code group needs increased manpower and resources to

Maintain Flash code Support large-scale astro simulations Make deep changes to Flash code & research challenging code issues

Comp phys group needs increased manpower and resources to Support new astro simulations Simulate experiments at NNSA labs – an activity vital to future of Flash

Center beyond the next three years Astrophysics group needs increased manpower and

resources to exploit ability Flash code now has to address key astrophysics problems

Major funding issue: Alliance Centers funding uncertain at a time when we need to

Support diversified code effort Expand validation effort Exploit Flash code for key astrophysics problems


Monthly Usage

0

100

200

300

400

500

600

700

800

900

CP

U h

ou

rs i

n 1

00

0

Frost ALC QSC Seaborg

Cumulative Usage

0

5

10

15

20

25

CP

U h

ou

rs i

n 1

000

00

Frost ALC QSC Seaborg

ASC Platform Usage


Issues and Challenges

Access to computational resources is a major issue: As was already true at the time of the 2003 site review, we can use every cycle

we can lay our hands on to do highly significant astrophysical simulations (i.e., we are now “cycle-bound” in the science that we are able to do).

We have aggressively sought to exploit ASC platforms to the full and have sought access to non-ASC platforms (e.g., INCITE proposal to use Seaborg).

We are aggressively seeking access to new ASC platforms (e.g., Blue Gene/L), but these machines will not come on line for a year or more.

ALC was designated as the main platform for Alliance centers in November 2003; however, it is an experimental machine. Close collaboration w. the ASAP CRT and w. consultants at LLNL Computing have been instrumental in our beginning to use ALC, but no production run has been successfully carried out to date. Recent progress encourages us to believe that we will be able to make full use of ALC in the near future.

The problems w. the ASC platforms temporarily placed a large burden on the code group, delaying the start of work on Flash 3 until February 2004.

This year we had a harder time exploiting ASC computational resources. We are therefore appreciative that Frost will continue to be available to the Alliance centers, and are working with LLNL Computing to achieve reliability on ALC for Flash runs.


… which brings us to

Questions and Discusson

Documents

An Advanced Simulation and Computation (ASC) Academic Strategic Alliances Program (ASAP) Center at The University of Chicago The Center for Astrophysical