U.S. Department of Energy’s Office of Science

U.S. Department of Energy

Office of Science

Advanced Scientific Computing Research Program

Walter M. [email protected]

Office of Advanced ScientificComputing Research

- Status -

Petascale SimulationsSciDAC

U.S. Department of Energy’s Office of Science


Office of Science


CASC- September 6, 2006 2

American Competitiveness Initiative

President Bush in the President’s State of the Union Address on January 31, 2006

“I propose to double the federal commitment to the most critical basic research program in the physical sciences over the next ten years. This funding will support the work of America’s most creative minds as they explore promising areas such as nanotechnology, supercomputing, and alternative energy sources.”

Secretary Bodman commenting on President Bush’s statement

“Developing revolutionary, science-driven technology is at the heart of the Department of Energy’s mission. To ensure that America remains at the forefront in an increasingly competitive world, our Department is pursuing transformational new technologies in the cutting-edge scientific fields of the 21st century—areas like nanotechnology, material science, biotechnology, and high-speed computing.”


Office of Science



Office of Science: Mission

“The Office of Science plays a critical role in ensuring America’s scientific leadership and economic dynamism…” The mission of the Science program is to deliver the discoveries and scientific tools that transform our understanding of energy and matter and advance the national, economic, and energy security of the United States.

Secretary Samuel Bodman, House Committee on Science Hearing, February 15, 2006

In support of it’s mission, the Science program has responsibilities in three main areas: selection and management of research; operation of world-class, state-of-the-art scientific facilities; and design and construction of new facilities.

“Investment in these facilities is much more than bricks and mortar: it is an investment in discovery, and in the future of our nation.”

Facilities for the Future of Science: a 20-Year Outlook


Office of Science


Office of Science


Office of Science -- FY2007 Budget Request- $4.1B --

FY2006 Appropriation- $3.6B


Office of Science



ASCR Vision

First in Computational Science “Best in class in advancing science and technology through modeling and simulation”

Facilities

Enabling Technologies

Computational Partnerships


Office of Science



ASCR Research Strategy

• Applied Mathematics and Computer Science deliver the operating systems, programming models, software tools, and mathematical algorithms and libraries needed for scientists to make effective use of petascale computing.

• Scientific Discovery through Advanced Computing (SciDAC) program, will strengthen activities at the software centers initiated in FY 2006 for petascale computing.

• SciDAC will foster additional research investments in Applied Mathematics and Computer Science to accelerate efforts in modeling and simulation on the petascale computing facilities.


Office of Science



ASCR Facilities

• Oak Ridge Leadership Computing Facility

• Argonne Leadership Computing Facility

• NERSC at Lawrence Berkeley National Lab

• ESnet at Lawrence Berkeley National Lab


Office of Science



FY 2006 ASCR Budget Distribution

Research and Evaluation Prototypes

$13.0

Applied Mathematics $29.4

NERSC $37.5

ESnet $19.0

Distributed Network Environment Research

$13.6

SciDAC $38.1

Computer Science $24.3

SBIR/STTR $6.3

LCF$53.7


Office of Science



FY 2007 ACSR Budget Status

0

50

100

150

200

250

300

350

FY06 FY07 Request FY07 House FY07 Senate

$235M

$319M $319M$319M

Do

llars

in M

illio

ns


Office of Science



FY 2007 - House

FY 2007 House Energy and Water Development Appropriations Bill passed both the Appropriations Committee and the House Report language:

“ The Committee commends the Office of Science and the Office of Advanced Scientific Computing Research for their efforts to provide cutting-edge capabilities to meet current scientific computational needs, and at the same time to extend the boundaries of that cutting edge into the next generation of high-performance scientific computers and supporting software.”


Office of Science



FY 2007 - Senate

FY 2007 Senate Energy and Water Development Appropriations Bill passed the Appropriations Committee Report language

“ …The Committee is concerned with the relationship between the Office of Science and the NNSA. As an example, the ASCR strategic plan discusses the need to work with other Federal agencies including several defense agencies, but only discusses in general terms three areas of research where NNSA and the Office of Science cooperated. In the area of basic research, the strategic plan states that it is an area that is `not important enough to justify ASCI investment at this time.' The Committee is also aware that the Office of Science has budgeted $13,000,000 for the DARPA to support a petaflop computer deployment by 2010. The Committee believes this funding would be better spent within the Department to support a petaflop initiative. The Department is directed to divide the funds equally between the Office of Science and the NNSA Advanced Simulation and Computing activities to support development of component architecture for high-performance software and storage”


Office of Science



Facilities Update

• NERSC – delivery of NERSC-5 in FY 2007• LCF at ORNL –

– Upgrade Cray XT3 upgrade • Now: 25Teraflop 50Teraflop• By the end of 2007: 50Teraflop 250Teraflop

– Acquire 1 Petascale Cray Baker system by end of 2008• LCF at Argonne –

– Acquire 100 Teraflop IBM Blue Gene/P in FY2007– Upgrade to 250-500 Teraflop IBM Blue Gene/P in 2008– On path to a Petascale IBM Blue Gene /Q by end of the

decade


Office of Science



Innovative and Novel Computational Impact on Theory

and Experiment- INCITE

• Initiated in 2004• Provides Office of Science computing resources to a small

number of computationally intensive research projects of large scale, that can make high-impact scientific advances through the use of a large allocation of computer time and data storage

• Open to national and international researchers, including industry

• No requirement of DOE Office of Science funding• Peer-reviewed• 2004 Awards: 4.9 Million processor hours at NERSC awarded

to three projects• 2005 Awards: 6.5 Million processor hours at NERSC awarded

to three projects


Office of Science



INCITE 2006

• Expanded to include SC high end computing resources at PNNL, ORNL and ANL in addition to LBNL and multiple year requests.

• Received 43 proposals requesting over 95 million processor hours.– 60% from Universities– 40% had funding from other

federal research agencies

• 15 Awards for over 18.2 million processor hours

Wired Magazine, August, 2006, pg. 42


Office of Science



INCITE 2007

• Expanded in 2007 to include 80% of resources at Leadership Computing Facilities in addition to 10% of NERSC and 5% of PNNL

• Call issued July 27, 2006

• Proposals due September 15, 2006• Award announcements in mid-December

http://hpc.science.doe.gov


Office of Science



Scientific Discoverythrough Advanced Computing

Structure

-- Scientific Application Areas - Breakthrough scientific advances through computer modeling and simulation that

are impossible using theoretical and experimental studies or improve experimental science.

-- Centers for Enabling Technology (CETs)- provide the essential computing and communications infrastructure to support SciDAC applications

-- Institutes- University-led centers of excellence; address specific software issues; Engage a broader community of scientists in modeling and simulations.

www.SciDAC.gov


Office of Science



Computing/Networking

Applications

Scientific Discovery

Leadership Computing- ANL

100 TF

Leadership Computing- ORNL

250 TF

Production Computing- NERSC

100-150 TF

ESnetOn path toward Dual rings 40Gbps/ 10 Gbps fault tolerant

Path to Petascale

- Centers for Enabling Technology

- Scientific Applications Partnerships - Institutes (University-lead)

- Accelerator science and simulation - Astrophysics- Climate modeling and simulation - Computational Biology- Fusion science - High-energy physics- Petabyte high-energy/nuclear physics - Materials science and chemistry- Nuclear physics - QCD- Radiation transport - Turbulence

- Groundwater reactive transport modeling and simulation

Sci

DA

C-2

INC

ITE

NE

RS

C A

llo

ca

tio

n


Office of Science



SciDAC-2 Institutes…

… University-led centers of excellence that focus on major software issues through a range of collaborative research interactions.

– Develop, test, maintain, and support optimal algorithms, programming environments, systems software and tools, and applications software.

– Focus on a single general method or technique.

– Focal point for bringing together a critical mass of experts from multiple disciplines to focus on key problems in a particular area of enabling technologies.

– Forge relationships between experts in software development, scientific application domains, high performance computing, and industrial partners.

– Reach out to engage a broader community of scientists in the activities of scientific discovery through advanced computation and collaboration.

– Incorporate training and outreach in high performance computing topics, including for graduate students and postdocs.


Office of Science



Centers for Enabling Technologies… (CETs)

… provide the essential computing and communications infrastructure to support SciDAC applications; multi-discipline approach with activities in:

– Algorithms, methods, and libraries.– Program development environments and tools -- terascale and petascale program

development and tools provide maximum ease-of-use to scientific end users. – Systems software that provides system stability and functionality needed by users for tera- to

petascale simulations. – Visualization and data management systems.

• CETs work directly with applications on: – Development and application of computing systems software that allows scientific simulation

codes to take full advantage of the extraordinary capabilities of terascale and petascale computers.

– Ensuring that the most critical computer science and applied mathematics issues are addressed in a timely and comprehensive fashion.

– Addressing all aspects of the successful research software lifecycle including transition of a research code into a robust production code and long term software evolution and maintenance and end user support.


Office of Science



SciDAC Accomplishments

• To better understand combustion which provides 80% of the energy used in the U.S., SciDAC teams created first laboratory-scale flame simulation in three dimensions

• Magnetic fusion scientists and applied mathematicians simulated techniques for re-fueling fusion reactors

• Teams developed new methods for simulating improvements in future particle accelerators

• Partnerships improved effectiveness of scientific applications codes between 275% to over 10,000%

Hydroxyl radical in a turbulent jet flame


Office of Science



The Power and Evolution of Partnerships

Today and Tomorrow

ASCR

HEP

NE

EM

NNSA

BES

BER

FES

NP

Industry

Methane Flames Diesel CombustionUltrafast Science

GCM Performance CloudsSimulation of Microbes

Plasma Core Full Simulation ofBurning Plasma

Weak Matrix ElementsPerformance of Supernova codesAccelerator Design

Increased Accuracy

Elements beyond Fe

LCLS System Design

Neutron Transport Advanced Fuel Cycles

Groundwater Transport

Libraries &Algorithms Science Based Materials

Improvements in Mesh Generation

Virtual Prototypes


Office of Science



Road to Petascale Computing: Combustion

A 50% increase in efficiency of automobiles could save 21% of U.S. oil consumption for transportation

Direct Numerical Simulation of Turbulent Combustion, a collaborative effort between Sandia National Laboratories, U. of Wisconsin, U. of Michigan, U. of Maryland, U.C. Davis, and Princeton U., has been focusing on advancing science for improved fuel efficiency of gas turbines and technologies for transportation.

Petascale computing will enable fundamental understanding of turbulence-chemistry interactions, e.g. multi-stage auto-ignition and flame stabilization in compression ignition engine environments, in regimes relevant to real devices. This large turbulent CO/H2 jet flame was run

on the CrayX1E at ORNL LCF.

(Credit for the image is to Kwan-Liu Ma, Hongfeng Yu, and Hiroshi Akiba of UC Davis.)


Office of Science



Volume rendering of scalar dissipation rates in a temporal plane jet flame. High values are shown in red, medium in yellow, and lower values in blue. (Credit: Jackie Chen/SNL)

Turbulence


Office of Science



Center for Simulation of Wave Interaction with Magneto-hydrodynamics, a collaboration of PPPL, ORNL, MIT, Columbia U., Indiana U., Wisconsin U., etc., is a project for very large plasma wave simulation for fusion-based research.

Using 4,096 processors of Cray XT3 at LCF (ORNL), it is the largest, most-detailed simulation ever done of plasma control waves in a tokamak, the reactor that will eventually form the core of the multinational ITER reactor.

Road to Petascale Computing: Fusion


Office of Science



A multi-institution project for large-scale simulation of Neutrino-driven Supernova moves forward to discover a new instability that is important to core collapse supernovae – the supernova shock wave instability (SASI) and a plausible mechanism for generating the initial neutron star spin required to give birth to pulsars.

Road to Petascale Computing: Astrophysics


Office of Science



Researchers at Argonne National Laboratory have recently developed a Portable, Extensive, Toolkit for Scientific (PETSc) numerical library used in dozen of scientific applications world-wide.

PETSc encapsulates the complexity of underlying parallel algorithms and presents them in terms of numerical abstractions familiar to scientists. It has been used to extend fully compressible flow codes, most notably the pressure-oriented implicit continuous Eulerian (PCICE) scheme developed at Idaho National Engineering Laboratory.

Road to Petascale Computing:Modeling for Reactor Design


Office of Science



Researchers at Lawrence Livermore National Laboratory have recently developed a suite of fast and scalable adaptive algorithms for modeling shocks and detonations interacting with movable rigid bodies in a fluid. Accurate 2D simulations tax existing computer resources; extending to well-resolved 3D simulations will require peta-scale resources.

This new algorithm combines efficient local representations of the embedded bodies with moving geometry and adaptive mesh refinement.

Road to Petascale Computing: Modeling the Interaction b/w Fluids & Embedded

Solid Particles

Shock hitting several movable rigid bodies using a Godunov method with adaptive mesh refinement; density and refinement grids are shown

The same fluid solver can model detonations using an ignition and growth model and JWL equation of state. This mock Schlieren shows a detonation interacting with solid bodies.

(Credit for the images is to Bill Henshaw (PI) and Kyle Chand, LLNL)

Documents

U.S. Department of Energy’s Office of Science