SciDAC 6-30-05 M. L. Norman San Diego Supercomputer Center, UCSD Keck Observatory, HI Simulating the Cosmic History of Baryons Discoveries Using Advanced

SciDAC 6-30-05 M. L. Norman

San Diego Supercomputer Center, UCSDKeck Observatory, HI

Simulating the Cosmic History of BaryonsDiscoveries Using Advanced Computing

Michael L. Norman, Physics Dept., UC San Diego

validation


Cosmic History of Baryons

linear perturbation theory nonlinear simulations

phase transitions gravitational instability

Baryogensis: GUT phase transitiont~10(-12) s speculative

Nucleosynthesis: formation of light nucleit~1-100 s precision era (BBNS)

Recombination: matter & radiation decouplet~380,000 yr precision era (CMB)

Structure Formation: 50 Myr < t < 14 Gyr synthesis erasurveys


We are here


Cosmological N-body SimulationA. Evrard and the Virgo Consortium



Multiscale ChallengeMultiscale Challenge


Grand Challenges in Computational Hydrodynamic Cosmology

Formation and evolution of stellar systems on all scales and epochs

Chemical enrichment and reionization of intergalactic medium

Formation of massive black holes and nature of the quasar phenomenon

Cosmological constraints on nature of dark matter and dark energy


Outline

• Cosmology’s Standard Model

• Universe in a Box

• History of Baryons: Discoveries using Advanced Computing

• Exciting Opportunities Ahead– Cosmological limits on dark matter mass– Measuring dark energy equation of state


04.0~

3.0~ ,7.0~

3

820

b

M

ii H

G

• Concordance model– H0=72+/-7 km/s/Mpc

– expansion rate accelerating (q0<0)

– flat universe (k=0)– dominated by dark matter and

dark energy– baryons minor constituent

Cosmology’s Standard Model

Perlmutter (2003), Physics Today

SciDAC 6-30-05 M. L. NormanS. Perlmutter, Physics Today (2003)

Evidence for an Accelerating Universe


Cosmic Microwave BackgroundTemperature Fluctuations 380,000 yr ABB

T/T ~ ~ 10-4

NASA WMAP


CMB Angular Power Spectrum


Mass-Energy Budget of the Universe (WMAP+SNe+XRC)

Universe in a Box


The Universe is an IVP suitable for computation

• Globally, the universe evolves according to the Friedmann equation

33

8)(

2

22

a

kG

a

atH

Hubble parameter

mass-energydensity

spacetimecurvaturescale factor a(t)

cosmological constant


The Universe is an IVP...

• Locally*, its contents obey:– Newton’s laws of gravitational N-body

dynamics for stars and cold dark matter– Euler or MHD equations for baryonic

gas/plasma – Atomic and molecular processes important

for radiative cooling of gas and condensation to form stars and galaxies

– Radiative transfer equation for photons

(*scales << horizon scale ~ ct)

Numerical astrophysics on a cosmic scale


radiationbackground

galaxies IGM

photo-ionizationphoto-heating

absorption

feedback(energy, metals)SF-recipe

self-shieldingphoto-evaporation

infall

ionizingflux

multi-specieshydrodynamics

radiative transfer

N-body dynamics

cosmic expansion self-gravitydark matter

dynamics

baryonic universe


Cold Dark Matter• Dominant mass constituent: cdm~0.23• Only interacts gravitationally with ordinary matter

(baryons)• Candidates: WIMPs or axions• Collisionless dynamics governed by Vlasov-Poisson

equation

• Solved numerically using fast N-body methods

),,( 4

0),,(32 txfdG

fftxf xt


Gridding the Universe

• Transformation to comoving coordinates x=r/a(t)

a(t1) a(t2) a(t3)

• Triply-periodic boundary conditions

But what about initial conditions?


Matter Power Spectrum P(k)

http://www.hep.upenn.edu/~max

Concordancemodel


Gravitational Instability: Origin of Cosmic Structure

A

B

C

A

B

C

x

x

very small fluctuations

gravity amplifies fluctuations

Formation of the Cosmic Web:Sky Dome Rendering for DomeFest 2005

Michael Norman, Brian O’Shea, UCSD

Donna Cox, Robert Patterson, Stuart Levy, UIUCSteve Cutchin, Amit Chourasia, SDSC


Technical Details• Simulation (Enzo)

– Dark matter, gravity, multispecies gas dynamics, photo-ionization and, radiative cooling

– 1 billion cells, 1 billion particles– 512 cpu, NCSA TeraGrid cluster

• Data– 512x512x512 arrays of density– 2000 timesteps– 1 Terabyte of data

• Volume rendering– SDSC IBM DataStar



Structured Adaptive Mesh Refinement

(Berger and Colella 1989)


Cosmological Adaptive Mesh Refinement(Bryan & Norman 1997)

• Spatial dynamic range unlimited in principle

• Today:– L/ = 104 in statistical volumes– L/ =1010 single objects of interest

• Petascale:– L/ =106 in statistical volumes


http://cosmos.ucsd.edu/enzo

http://cosmos.ucsd.edu/enzo


Enzo Implementation Details • Multi-scale in space and time• Arbitrary # levels of refinement• Arbitrary # grids per level• Portable, MPI-parallel, C++/C/F77 hybrid• Nonlocal dynamic load balancing• Ported to IA64, SGI Altix, IBM SP, BG/L, your mother’s Linux cluster, …..


• Technical details– 2563 base grid– >32,000 grid patches @ 8 levels of refinement– 110,000 cpu-hrs on 128 cpu Origin2000– 0.5 TB of data– Run at NCSA in 1999

Image credit: D. Cox et al.Science credit: M. Norman, G. Bryan, B. O’Shea

Galaxy Formation and Large Scale Structure


Computational Discoveriesusing Advanced Computing

First baryoniccondensations


“Bottom-Up” Galaxy Formation

• large galaxies form from mergers of smaller galaxies

• where does this begin?

•What are the first objects to form?

Lacey & Cole (1993)


First objects: a well-posed problem

• Initial conditions specified: i, P(k)

• Macroscopic dynamics understood

• Microphysics of primordial gas known

• Have 3D solution-adaptive algorithms

• Have adequate computer power

February 2003


Formation of First StarsAdaptive Mesh Refinement Simulation

Abel, Bryan & Norman (2001)

1 x 10 x 100 x 1000 x

104 x105 x106 x107 x

Cosmic scales

Solar system scales


Birth and Death of the First Star in the Universe

Movie credit: R. Kaehler & T. AbelScience credit: T. Abel, G. Bryan, M. Norman


Impact of the first stars the first stars in the universe began forming around

50 million years after the big bang

they were exceptionally massive and bright, bringing an earlier end to the cosmic “dark ages” than previously thought

when they exploded as supernovae they seeded the universe with heavy elements essential for planets and life

they kick-started the cosmogonic sequence which eventually formed galaxies, clusters and superclusters



structure ofintergalacticmedium


The Intergalactic Medium

Source: M. Murphy


N=10243

L=54 Mpc/h

Structure of the IGM

Baryon Overdensity, z=3

quasar

Earth

Simulated HI absorption spectrum


Matter Power Spectrum P(k)

http://www.hep.upenn.edu/~max

CDM



whereaboutsof missingbaryons


Missing Baryons at z=0

• Galaxies in local universe account for only 10% of baryons we know exist due to three independent measurements, which all agree to 2– Big bang nucleosynthesis– CMB anisotropies– IGM absorption at high redshift

• Where are the baryons now?


Whereabouts of the missing baryons:

Warm-Hot intergalactic gas

Cen & Ostriker (1998)

warm-hot gas

“galaxies”

N=5123


Exciting Opportunities Ahead(require Terascale and beyond)

• Predicting properties of first galaxies

• Understanding quasar-galaxy connection

• Self-consistent simulation of the reionization era

• Cosmological limits on dark matter mass

• Measuring the dark energy equation of state


Effect of DM particle mass on first objects: critical threshold

25 keV

10 keV

O’Shea & Norman (2005)


Measuring Dark Energy EOS

• Principal goal of NASA/DOE JDEM mission

• Approach: precision measurements of expansion history of the universe using Type Ia SN standardizable candles

• Complimentary approach: redshift distribution of galaxy clusters


Lightcone Simulation(A. Evrard et al. 2003)

• Evrard et al. – Single, 10243 P3M– L/=104

– Dark matter only

• Our plan– Multiple, 5123 AMR– Optimal tiling of lightcone– L/=105

– Dark matter + gas

ct (Gyr)

0 -1 -2 -3 -4 -5


• Cosmic Simulator• A software facility for physical cosmology• A new collaboration between LLNL and UCSD• Scientific data management focus

– Simulations: LLNL Thunder, BG/L– Data management: SDSC SRB– Public archive @ UCSD

• Science driver: – LSST (Large Synoptic Survey Telescope)

Documents

SciDAC 6-30-05 M. L. Norman San Diego Supercomputer Center, UCSD Keck Observatory, HI Simulating the Cosmic History of Baryons Discoveries Using Advanced