November 1, 2007Scott Dodelson1 Computational Cosmology Initiative Report of the Task Force

November 1, 2007 Scott Dodelson 1

Computational Cosmology

Initiative

Report of the Task Force


Task Force Scott Dodelson (Center for Particle

Astrophysics), co-Chair Don Petravick (Computing Division), co-Chair James Amundson (Computing Division) Jim Annis (Experimental Astrophysics) Nick Gnedin (Theoretical Astrophysics) Don Holmgren (Computing Division) Jim Kowalkowski (Computing Division) Andrey Kravtsov (University of Chicago) Andreas Kronfeld (Particle Theory)


Task Force

Develop, as necessary, and articulate the case for a strategic initiative for a computational cosmology computing project as part of the Fermilab computing capability.

It will be important to: Articulate an outstanding scientific case. Discuss the existing and potential scientific collaboration across the US. Could this evolve into

a national program? put the proposed capability in the context of other computing capabilities such as that

available from large national computing centers such as NERSC, leadership DOE, Terascale NSF centers.

put the proposed capability in the context of worldwide developments discuss the technical aspects; this may include an R&D component. discuss the infrastructure needs. discuss the resource needs including the effort at different levels addressing not only the

hardware but also the algorithmic aspects of a possible project. discuss the connection to or synergies with any relevant other high performance computing or

SciDACC efforts. develop a business plan considering multiple funding sources.

Prepare a written report considering two 3-year periods with the first period not thoroughly developed. This should take into account the funding cycle for opportunities such as SciDACC. The report should be limited to 20 pages maximum including all tables and figures using a reasonable font size. It would be interesting to receive a draft report by mid-June with the final report by July 31.

Charge from Hugh Montgomery, April 10, 2007


Task Force

Weekly Meetings Briefings on: Lattice QCD (AK), Legal/Business

Issues (DH), Computing Resources (Space, power, etc.) (DP), Science (AK,NG), Software Issues (AK,NG), Refining Charge (Mont)

Delivered First Draft in June 2007 Sent for review to ~10 people; received

responses from ~5; incorporated these in final draft

Final draft delivered August, 2007


Science Case

We have learned a lot from the cosmic microwave background because deviations from uniformity are small

Measurements can be compared to theory which depends on parameters


Science Case

Gravitational Instability is nonlinear Baryons governed by hydrodynamics Radiation Field affects and is affected by structure Stars form, Supernovae explode, …

Extracting information from observations of galaxies is much more difficult


Science Case

QED QCD

Akin to the difference between


Science Case

Shape of the matter power spectrum depends on neutrino mass and inflation

Evolution of structure depends on dark energy Gravitational wells determined by dark matter Rich astrophysical questions: galaxy formation,

reionization, black holes, quasars,…

If we can obtain accurate predictions for a given set of cosmological parameters, the payoff will be enormous


Science Case

Computing power – software and hardware – has improved so that the

goal of obtaining accurate predictions for the distribution of matter in the

universe is within reach

Note: Focus on cosmological simulations from smooth initial conditions


Science Case

Example: Weak Lensing. Leaving out physics of star formation leads to incorrect predictions in the “dark energy” region.


Science Case: Finding #1

“Cosmological simulations are powerful tools to understand the growth of structure in the universe. This understanding is interesting and important in its own right, but also is essential for extracting precision information about fundamental physics - such as dark energy and dark matter - from upcoming astronomical surveys.”


Local Expertise

Ngak.htm

Nick Gnedin (FNAL) and Andrey Kravtsov (UChicago) are collaborators and two of the strongest numerical astrophysicists in the world. They have assembled excellent teams of students and postdocs. The two groups are extremely productive (25 papers/183 citations since 1/06).


Gnedin and Kravtsov developed Adaptive Refinement Tree (ART) code

Implementation of Adaptive Mesh Refinement method Refines on a cell-by-cell basis Full 4D adaptive Includes - dark matter dynamics - gas dynamics and chemistry - radiative transfer - star formation, etc

Local Expertise


Local Expertise


Local Expertise: Finding #2

“Groups at the University of Chicago and Fermilab have the tools in the form of state-of-the-art codes to contribute to this important area.”


As of today, Computing Division maintains an 8 core cluster for cosmology Other U.S. groups have many more resources: ITC, Harvard: 316 CPU Opteron, 264 CPU Athlon LANL: 294 CPU Pentium4 (just for cosmology) SLAC: 72 CPU SGI Altrix, 128 CPU Xeon cluster Princeton: 188 CPU Xeon cluster UWash: 64 CPU Pentium cluster International Groups are far ahead Virgo Consortium: 670 CPU SparcIII, 816 CPU Power4 CITA: 270 CPU Xeon cluster, 2000 CPU dual Pentium (shared)

Resources


Resources

In the two years since the inception of Distributed European Infrastructure for Supercomputing Applications, 8 large-scale allocations were made for computational cosmology projects at European supercomputer centers with an average allocation size of 1 million hours or more.

For comparison, only 1 computational cosmology project of similar size was approved in the last three years through the DOE INCITE program.


Resources: Finding #3

“The local resources for these groups are currently inadequate to run the largest, highest resolution simulations. This inadequacy is representative of a problem on a national scale. In contrast, European groups have pooled resources and currently run the most powerful cosmological simulations.”


Infrastructure @ FNAL Space/Power for Hardware not a problem CD currently supports many machines

(>20,000 cores) Software development for LQCD/Physics for

Accelerators Build/manage releases for software for

D0,CDF, CMS … Experience benchmarking code; optimizing

hardware Data Handling Project Management


Infrastructure @ FNAL: Finding #4

“Fermilab has the infrastructure to host a cosmological computing initiative on a national scale.”


Proposal

What have we done so far?

Received FRA grant ($75k) Received funds for hardware from Kavli Institute for Cosmological Physics at UChicago ($100k) Contributed funds from Theoretical Astrophysics (Rocky money) ($60k) Successfully obtained LDRD at Argonne for software development and ultimately joint postdocs Expect to have ~500 cores @ FNAL by December 2007


Proposal

What are we asking for from FNAL in FY08?

Funds to continue building local cluster ($200k) Support for cluster from CD (0.2 FTE) Software Developer from CD (0.5 FTE)


Proposal


ProposalLong Term Goal

Develop a national program in computational cosmology powerful enough to exploit fully Stage IV dark energy experiments


Back-up Slide 1: PAC Advice“Should opportunities for expanding the scope of Fermilab

R&D activities emerge in the future (noble liquid detectors for Dark Matter searches and water Cerenkov detectors for long-baseline neutrino experiments are among the various possibilities), they should be evaluated in the context of the overall national scientific strategy and funding constraints, taking into account the following criteria:

1. Scientific and technical excellence; 2. The uniqueness and impact of the Fermilab contribution,

given its existing infrastructure and technical expertise; and 3. The interest among Laboratory personnel in collaborating

on the project. Ideally, Laboratory management would help external proponents to identify relevant personnel who might be available to participate.“


Back-up Slide 2: Resources

Documents

November 1, 2007Scott Dodelson1 Computational Cosmology Initiative Report of the Task Force