A SCIENCE-BASED CASE FOR LARGE-SCALE SIMULATIONscience.energy.gov/.../archive/Scales_report_vol1.pdf · A Science-Based Case for Large-Scale Simulation vii ... Phillip Colella, Lawrence

A SCIENCE-BASED CASE FOR LARGE-SCALE SIMULATION

Office of ScienceU.S. Department of Energy

July 30, 2003

“There will be opened a gateway and a road to a large andexcellent science, into which minds more piercing than mineshall penetrate to recesses still deeper.” Galileo (1564–1642)

[on the “experimental mathematical analysis of nature,”appropriated here for “computational simulation”]

TRANSMITTAL

July 30, 2003

Dr. Raymond L. Orbach, DirectorOffice of ScienceU.S. Department of EnergyWashington, DC

Dear Dr. Orbach,

On behalf of more than 300 contributing computational scientists from dozens of leadinguniversities, all of the multiprogrammatic laboratories of the Department of Energy, otherfederal agencies, and industry, I am pleased to deliver Volume 1 of a two-volume reportthat builds a Science-Based Case for Large-Scale Simulation.

We find herein (and detail further in the companion volume) that favorable trends incomputational power and networking, scientific software engineering and infrastructure,and modeling and algorithms are allowing several applications to approach thresholdsbeyond which lies new knowledge of both fundamental and practical kinds. A majorincrease in investment in computational modeling and simulation is appropriate at thistime, so that our citizens are the first to benefit from particular new fruits of scientificsimulation, and indeed, from an evolving culture of simulation science.

Based on the encouraging first two years of the Scientific Discovery through AdvancedComputing initiative, we believe that balanced investment in scientific applications,applied mathematics, and computer science, with cross-accountabilities between theseconstituents, is a program structure worthy of extension. Through it, techniques fromapplied mathematics and computer science are systematically being migrated into scientificcomputer codes. As this happens, new challenges in the use of these techniques flow backto the mathematics and computer science communities, rejuvenating their own basicresearch. Now is an especially opportune time to increase the number of groups working inthis multidisciplinary way, and to provide the computational platforms and environmentsthat will enable these teams to push their simulations to the next insight-yielding levels ofhigh resolution and large ensembles.

The Department of Energy can draw satisfaction today from its history of pathfinding ineach of the areas that are brought together in large-scale simulation. However, from thevantage point of tomorrow, the Department’s hand in their systematic fusion will beregarded as even more profound.

Best regards,

David E. KeyesFu Foundation Professor of Applied MathematicsColumbia UniversityNew York, NY

Executive Summary

A Science-Based Case for Large-Scale Simulation iii

Important advances in basic science crucialto the national well-being have beenbrought near by a “perfect fusion” ofsustained advances in scientific models,mathematical algorithms, computerarchitecture, and scientific softwareengineering. Computational simulation—ameans of scientific discovery that employsa computer system to simulate a physicalsystem according to laws derived fromtheory and experiment—has attained peerstatus with theory and experiment in manyareas of science. The United States iscurrently a world leader in computationalsimulation, a position that confers both anopportunity and a responsibility to mount avigorous campaign of research that bringsthe advancing power of simulation to manyscientific frontiers.

Computational simulation offers toenhance, as well as leapfrog, theoretical andexperimental progress in many areas ofscience critical to the scientific mission ofthe U.S. Department of Energy (DOE).Successes have been documented in suchareas as advanced energy systems (e.g., fuelcells, fusion), biotechnology (e.g., genomics,cellular dynamics), nanotechnology (e.g.,sensors, storage devices), andenvironmental modeling (e.g., climateprediction, pollution remediation).Computational simulation also offers thebest near-term hope for progress inanswering a number of scientific questionsin such areas as the fundamental structureof matter, the production of heavy elementsin supernovae, and the functions ofenzymes.

The ingredients required for success inadvancing scientific discovery are insights,models, and applications from scientists;

EXECUTIVE SUMMARY

theory, methods, and algorithms frommathematicians; and software andhardware infrastructure from computerscientists. Only major new investment inthese activities across the board, in theprogram areas of DOE’s Office of Scienceand other agencies, will enable the UnitedStates to be the first to realize the promiseof the scientific advances to be wrought bycomputational simulation.

In this two-volume report, prepared withdirect input from more than 300 of thenation’s leading computational scientists, ascience-based case is presented for major,new, carefully balanced investments in• scientific applications• algorithm research and development• computing system software

infrastructure• network infrastructure for access and

resource sharing• computational facilities• innovative computer architecture

research, for the facilities of the future• proactive recruitment and training of a

new generation of multidisciplinarycomputational scientists

The two-year-old Scientific Discoverythrough Advanced Computing (SciDAC)initiative in the Office of Science provides atemplate for such science-directed,multidisciplinary research campaigns.SciDAC’s successes in the first four of theseseven thrusts have illustrated the advancespossible with coordinated investments. It isnow time to take full advantage of therevolution in computational science withnew investments that address the mostchallenging scientific problems faced byDOE.

Contents

iv A Science-Based Case for Large-Scale Simulation

Contents

A Science-Based Case for Large-Scale Simulation v

A SCIENCE-BASED CASE FOR LARGE-SCALE SIMULATION

VOLUME 1

Attributions ..................................................................................................................................... vii

List of Participants .......................................................................................................................... xi

1. Introduction .............................................................................................................................. 1

2. Scientific Discovery through Advanced Computing:A Successful Pilot Program .................................................................................................... 9

3. Anatomy of a Large-Scale Simulation .................................................................................. 17

4. Opportunities at the Scientific Horizon ............................................................................... 23

5. Enabling Mathematics and Computer Science Tools ......................................................... 31

6. Recommendations and Discussion ....................................................................................... 37

Postscript and Acknowledgments ............................................................................................... 45

Appendix 1: A Brief Chronology of “A Science-Based Case forLarge-Scale Simulation”.......................................................................................................... 47

Appendix 2: Charge for “A Science-Based Case for Large-Scale Simulation”...................... 49

Acronyms and Abbreviations ....................................................................................................... 51

Attributions

vi A Science-Based Case for Large-Scale Simulation

Attributions

A Science-Based Case for Large-Scale Simulation vii

ATTRIBUTIONS

Report EditorsPhillip Colella, Lawrence Berkeley National Laboratory

Thom H. Dunning, Jr., University of Tennessee and Oak Ridge National LaboratoryWilliam D. Gropp, Argonne National Laboratory

David E. Keyes, Columbia University, Editor-in-Chief

Workshop OrganizersDavid L. Brown, Lawrence Livermore National Laboratory

Phillip Colella, Lawrence Berkeley National LaboratoryLori Freitag Diachin, Sandia National Laboratories

James Glimm, State University of New York–Stony Brook andBrookhaven National Laboratory

William D. Gropp, Argonne National LaboratorySteven W. Hammond, National Renewable Energy Laboratory

Robert Harrison, Oak Ridge National LaboratoryStephen C. Jardin, Princeton Plasma Physics Laboratory

David E. Keyes, Columbia University, ChairPaul C. Messina, Argonne National Laboratory

Juan C. Meza, Lawrence Berkeley National LaboratoryAnthony Mezzacappa, Oak Ridge National Laboratory

Robert Rosner, University of Chicago and Argonne National LaboratoryR. Roy Whitney, Thomas Jefferson National Accelerator Facility

Theresa L. Windus, Pacific Northwest National Laboratory

Topical Group Leaders

Science Applications

AcceleratorsKwok Ko, Stanford Linear Accelerator Center

Robert D. Ryne, Lawrence Berkeley National Laboratory

AstrophysicsAnthony Mezzacappa, Oak Ridge National Laboratory

Robert Rosner, University of Chicago and Argonne National Laboratory

BiologyMichael Colvin, Lawrence Livermore National LaboratoryGeorge Michaels, Pacific Northwest National Laboratory

ChemistryRobert Harrison, Oak Ridge National Laboratory

Theresa L. Windus, Pacific Northwest National Laboratory

Climate and Earth ScienceJohn B. Drake, Oak Ridge National Laboratory

Phillip W. Jones, Los Alamos National Laboratory Robert C. Malone, Los Alamos National Laboratory

Attributions

viii A Science-Based Case for Large-Scale Simulation

CombustionJohn B. Bell, Lawrence Berkeley National Laboratory

Larry A. Rahn, Sandia National Laboratories

Environmental Remediation and ProcessesMary F. Wheeler, University of Texas–Austin

Steve Yabusaki, Pacific Northwest National Laboratory

Materials ScienceFrancois Gygi, Lawrence Livermore National Laboratory

G. Malcolm Stocks, Oak Ridge National Laboratory

NanosciencePeter T. Cummings, Vanderbilt University and Oak Ridge National Laboratory

Lin-wang Wang, Lawrence Berkeley National Laboratory

Plasma ScienceStephen C. Jardin, Princeton Plasma Physics Laboratory

William M. Nevins, Lawrence Livermore National Laboratory

Quantum ChromodynamicsRobert Sugar, University of California–Santa Barbara

Mathematical Methods and Algorithms

Computational Fluid DynamicsPhilip Colella, Lawrence Berkeley National Laboratory

Paul F. Fischer, Argonne National Laboratory

Discrete Mathematics and AlgorithmsBruce A. Hendrickson, Sandia National Laboratories

Alex Pothen, Old Dominion University

Meshing MethodsLori Freitag Diachin, Sandia National Laboratories

David B. Serafini, Lawrence Berkeley National LaboratoryDavid L. Brown, Lawrence Livermore National Laboratory

Multi-physics Solution TechniquesDana A. Knoll, Los Alamos National Laboratory

John N. Shadid, Sandia National Laboratories

Multiscale TechniquesThomas J. R. Hughes, University of Texas–Austin

Mark S. Shephard, Rensselaer Polytechnic Institute

Solvers and “Fast” AlgorithmsVan E. Henson, Lawrence Livermore National Laboratory

Juan C. Meza, Lawrence Berkeley National Laboratory

Transport MethodsFrank R. Graziani, Lawrence Livermore National Laboratory

Gordon L. Olson, Los Alamos National Laboratory

Attributions

A Science-Based Case for Large-Scale Simulation ix

Uncertainty QuantificationJames Glimm, State University of New York–Stony Brook and

Brookhaven National LaboratorySallie Keller-McNulty, Los Alamos National Laboratory

Computer Science and Infrastructure

Access and Resource SharingIan Foster, Argonne National Laboratory

William E. Johnston, Lawrence Berkeley National Laboratory

ArchitectureWilliam D. Gropp, Argonne National Laboratory

Data Management and AnalysisArie Shoshani, Lawrence Berkeley National LaboratoryDoron Rotem, Lawrence Berkeley National Laboratory

Frameworks and EnvironmentsRobert Armstrong, Sandia National LaboratoriesKathy Yelick, University of California–Berkeley

Performance Tools and EvaluationDavid H. Bailey, Lawrence Berkeley National Laboratory

Software Engineering and ManagementSteven W. Hammond, National Renewal Energy Laboratory

Ewing Lusk, Argonne National Laboratory

System SoftwareAl Geist, Oak Ridge National Laboratory

VisualizationE. Wes Bethel, Lawrence Berkeley National Laboratory

Charles Hanson, University of Utah

Workshop Participants

x A Science-Based Case for Large-Scale Simulation


A Science-Based Case for Large-Scale Simulation xi

Tom AbelPennsylvania State University

Marvin Lee AdamsTexas A&M University

Srinivas AluruIowa State University

James F. AmundsonFermi National Accelerator Laboratory

Carl W. AndersonBrookhaven National Laboratory

Kurt Scott AndersonRensselaer Polytechnic Institute

Adam P. ArkinUniversity of California–Berkeley/LawrenceBerkeley National Laboratory

Rob ArmstrongSandia National Laboratories

Miguel ArqaezUniversity of Texas–El Paso

Steven F. AshbyLawrence Livermore National Laboratory

Paul R. AveryUniversity of Florida

Dave BaderDOE Office of Science

David H. BaileyLawrence Berkeley National Laboratory

Ray BairPacific Northwest National Laboratory

Kim K. BaldridgeSan Diego Supercomputer Center

Samuel Joseph BarishDOE Office of Science

Paul BayerDOE Office of Science

Grant BazanLawrence Livermore National Laboratory

Brian BehlendorfCollabNet

John B. BellLawrence Berkeley National Laboratory

Michael A. BenderState University of New York–Stony Brook

Jerry BernholcNorth Carolina State University

David E. BernholdtOak Ridge National Laboratory

Wes BethelLawrence Berkeley National Laboratory

Tom BettgeNational Center for Atmospheric Research

Amitava BhattacharjeeUniversity of Iowa

Eugene W. BierlyAmerican Geophysical Union

John BlondinNorth Carolina State University

Randall BramleyIndiana University

Marcia BranstetterOak Ridge National Laboratory

Ron BrightwellSandia National Laboratories

Mark BosloughSandia National Laboratories

Richard Comfort BrowerBoston University

David L. BrownLawrence Livermore National Laboratory

Steven BryantUniversity of Texas–Austin

Darius BuntinasOhio State University

Randall D. BurrisOak Ridge National Laboratory

Lawrence BujaNational Center for Atmospheric Research

Phillip Cameron-SmithLawrence Livermore National Laboratory

Andrew CanningLawrence Berkeley National Laboratory

WORKSHOP PARTICIPANTS


xii A Science-Based Case for Large-Scale Simulation

Christian Y. CardallOak Ridge National Laboratory

Mark CarpenterNASA Langley Research Center

John R. CaryUniversity of Colorado–Boulder

Fausto CattaneoUniversity of Chicago

Joan CentrellaNASA Goddard Space Flight Center

Kyle Khem ChandLawrence Livermore National Laboratory

Jacqueline H. ChenSandia National Laboratories

Shiyi ChenJohns Hopkins University

George Liang-Tai ChiuIBM T. J. Watson Research Center

K.-J. ChoStanford University

Alok ChoudharyNorthwestern University

Norman H. ChristColumbia University

Todd S. CoffeySandia National Laboratories

Phillip ColellaLawrence Berkeley National Laboratory

Michael ColvinLawrence Livermore National Laboratory

Sidney A. CoonDOE Office of Science

Tony CraigNational Center for Atmospheric Research

Michael CreutzBrookhaven National Laboratory

Robert Keith CrockettUniversity of California–Berkeley

Peter Thomas CummingsVanderbilt University

Dibyendu K. DattaRensselaer Polytechnic University

James W. DavenportBrookhaven National Laboratory

Eduardo F. D’AzevedoOak Ridge National Laboratory

Cecelia DelucaNational Center for Atmospheric Research

Joel Eugene DendyLos Alamos National Laboratory

Narayan DesaiArgonne National Laboratory

Carleton DeTarUniversity of Utah

Chris DingLawrence Berkeley National Laboratory

David Adams DixonPacific Northwest National Laboratory

John DrakeOak Ridge National Laboratory

Phil DudeOak Ridge National Laboratory

Jason DuellLawrence Berkeley National Laboratory

Phil DuffyLawrence Livermore National Laboratory

Thom H. Dunning, Jr.University of Tennessee/Oak Ridge NationalLaboratory

Stephen Alan EckstrandDOE Office of Science

Robert Glenn EdwardsThomas Jefferson National AcceleratorFacility

Howard C. ElmanUniversity of Maryland

David EricksonOak Ridge National Laboratory

George FannOak Ridge National Laboratory

Andrew R. FelmyPacific Northwest National Laboratory

Thomas A. FerrymanPacific Northwest National Laboratory

Lee S. FinnPennsylvania State University

Paul F. FischerArgonne National Laboratory


A Science-Based Case for Large-Scale Simulation xiii

Jacob FishRensselaer Polytechnic University

Ian FosterArgonne National Laboratory

Leopoldo P. FrancaUniversity of Colorado–Denver

Randall FrankLawrence Livermore National Laboratory

Lori A. Freitag DiachinSandia National Laboratories

Alex FriedmanLawrence Livermore and Lawrence BerkeleyNational Laboratories

Teresa FrybergerDOE Office of Science

Chris FryerLos Alamos National Laboratory

Sam FulcomerBrown University

Giulia GalliLawrence Livermore National Laboratory

Dennis GannonIndiana University

Angel E. GarciaLos Alamos National Laboratory

Al GeistOak Ridge National Laboratory

Robert Steven GermainIBM T. J. Watson Research Center

Steve GhanPacific Northwest National Laboratory

Roger G. GhanemJohns Hopkins University

Omar GhattasCarnegie Mellon University

Ahmed F. GhoniemMassachusetts Institute of Technology

John R. GilbertUniversity of California–Santa Barbara

Roscoe C. GilesBoston University

James GlimmBrookhaven National Laboratory

Sharon C. GlotzerUniversity of Michigan

Tom GoodaleMax Planck Institut für Gravitationsphysik

Brent GordaLawrence Berkeley National Laboratory

Mark S. GordonIowa State University

Steven Arthur GottliebIndiana University

Debbie K. GracioPacific Northwest National Laboratory

Frank Reno GrazianiLawrence Livermore National Laboratory

Joseph F. GrcarLawrence Berkeley National Laboratory

William D. GroppArgonne National Laboratory

Francois GygiLawrence Livermore National Laboratory

James HackNational Center for Atmospheric Research

Steve HammondNational Renewable Energy Laboratory

Chuck HansenUniversity of Utah

Bruce HarmonAmes Laboratory

Robert J. HarrisonOak Ridge National Laboratory

Teresa Head-GordonUniversity of California–Berkeley/LawrenceBerkeley National Laboratory

Martin Head-GordonLawrence Berkeley National Laboratory

Helen HeLawrence Berkeley National Laboratory

Tom HendersonNational Center for Atmospheric Research

Van Emden HensonLawrence Livermore National Laboratory

Richard L. HilderbrandtNational Science Foundation


xiv A Science-Based Case for Large-Scale Simulation

Rich HirshNational Science Foundation

Daniel HitchcockDOE Office of Science

Forrest HoffmanOak Ridge National Laboratory

Adolfy HoisieLos Alamos National Laboratory

Jeff HollingsworthUniversity of Maryland

Patricia D. HoughSandia National Laboratories

John C. HoughtonDOE Office of Science

Paul D. HovlandArgonne National Laboratory

Ivan HubenyNASA/Goddard Space Flight Center

Thomas J. R. HughesUniversity of Texas–Austin

Rob JacobArgonne National Laboratory

Curtis L. JanssenSandia National Laboratories

Stephen C. JardinPrinceton Plasma Physics Laboratory

Philip Michael JardineOak Ridge National Laboratory

Ken JarmanPacific Northwest National Laboratory

Fred JohnsonDOE Office of Science

William E. JohnstonLawrence Berkeley National Laboratory

Philip JonesLos Alamos National Laboratory

Kenneth I. JoyUniversity of California–Davis

Andreas C. KabelStanford Linear Accelerator Center

Laxmikant V. KaleUniversity of Illinois at Urbana-Champaign

Kab Seok KangOld Dominion University

Alan F. KarrNational Institute of Statistical Services

Tom KatsouleasUniversity of Southern California

Brian KauffmanNational Center for Atmospheric Research

Sallie Keller-McNultyLos Alamos National Laboratory

Carl Timothy KelleyNorth Carolina State University

Stephen KentFermi National Accelerator Laboratory

Darren James KerbysonLos Alamos National Laboratory

Jon KettenringTelcordia Technologies

David E. KeyesColumbia University

Alexei M. KhokhlovNaval Research Laboratory

Jeff KiehlNational Center for Atmospheric Research

William H. KirchhoffDOE Office of Science

Stephen J. KlippensteinSandia National Laboratories

Dana KnollLos Alamos National Laboratory

Michael L. KnotekDOE Office of Science

Kwok KoStanford Linear Accelerator Center

Karl KoehlerNational Science Foundation

Dale D. KoellingDOE Office of Science

Peter Michael KoggeUniversity of Notre Dame

James Arthur KohlOak Ridge National Laboratory

Tammy KoldaSandia National Laboratories

Marc Edward KowalskiStanford Linear Accelerator Center


A Science-Based Case for Large-Scale Simulation xv

Jean-Francois LamarqueNational Center for Atmospheric Research

David P. LandauUniversity of Georgia

Jay LarsonArgonne National Laboratory

Peter D. LaxNew York University

Lie-Quan LeeStanford Linear Accelerator Center

W. W. LeePrinceton Plasma Physics Laboratory

John Wesley LewellenArgonne National Laboratory

Sven LeyfferArgonne National Laboratory

Timothy Carl LillestolenUniversity of Tennessee

Zhihong LinUniversity of California–Irvine

Timur LindeUniversity of Chicago

Philip LoCascioOak Ridge National Laboratory

Steven G. LouieUniversity of California–Berkeley/LawrenceBerkeley National Laboratory

Steve LouisLawrence Livermore National Laboratory

Ewing L. LuskArgonne National laboratory

Mordecai-Mark Mac LowAmerican Museum of Natural History

Arthur B. MaccabeUniversity of New Mexico

David MalonArgonne National Laboratory

Bob MaloneLos Alamos National Laboratory

Madhav V. MaratheLos Alamos National Laboratory

William R. MartinUniversity of Michigan

Richard MatznerUniversity of Texas–Austin

Andrew McIlroySandia National Laboratories

Lois Curfman McInnesArgonne National Laboratory

Sally A. McKeeCornell University

Gregory J. McRaeMassachusetts Institute of Technology

Lia MermingaThomas Jefferson National AcceleratorFacility

Bronson MesserUniversity of Tennessee/Oak Ridge NationalLaboratory

Juan M. MezaLawrence Berkeley National Laboratory

Anthony MezzacappaOak Ridge National Laboratory

George MichaelsPacific Northwest National Laboratory

John MichalakesNational Center for Atmospheric Research

Don MiddletonNational Center for Atmospheric Research

William H. Miner, Jr.DOE Office of Science

Art MirinLawrence Livermore National Laboratory

Lubos MitasNorth Carolina State University

Shirley V. MooreUniversity of Tennessee

Jorge J. MoréArgonne National Laboratory

Warren B. MoriUniversity of California–Los Angeles

Jose L. MunozNational Nuclear Security Administration

Jim MyersPacific Northwest National Laboratory

Pavan K. NaickerVanderbilt University


xvi A Science-Based Case for Large-Scale Simulation

Habib Nasri NajmSandia National Laboratories

John W. NegeleMassachusetts Institute of Technology

Bill NevinsLawrence Livermore National Laboratory

Esmond G. NgLawrence Berkeley National Laboratory

Michael L. NormanUniversity of California–San Diego

Peter NugentLawrence Berkeley National Laboratory

Joseph C. OefeleinSandia National Laboratories

Rod OldehoeftLos Alamos National Laboratory

Gordon L. OlsonLos Alamos National Laboratory

George OstrouchovOak Ridge National Laboratory

Jack ParkerOak Ridge National Laboratory

Scott Edward ParkerUniversity of Colorado

Bahram ParvinLawrence Berkeley National Laboratory

Valerio PascucciLawrence Livermore National Laboratory

Peter PaulBrookhaven National Laboratory

William PennellPacific Northwest National Laboratory

Arnie PeskinBrookhaven National Laboratory

Anders PeterssonLawrence Livermore National Laboratory

Ali PinarLawrence Berkeley National Laboratory

Tomasz PlewaUniversity of Chicago

Douglas E. PostLos Alamos National Laboratory

Alex PothenOld Dominion University

Karsten PruessLawrence Berkeley National Laboratory

Hong QinPrinceton University

Larry A. RahnSandia National Laboratories

Craig E. RasmussenLos Alamos National Laboratory

Katherine M. RileyUniversity of Chicago

Martei RipeanuUniversity of Chicago

Charles H. RomineDOE Office of Science

John Van RosendaleDOE Office of Science

Robert RosnerUniversity of Chicago

Doron RotemLawrence Berkeley National Laboratory

Doug RotmanLawrence Livermore National Laboratory

Robert Douglas RyneLawrence Berkeley National Laboratory

P. SadayappanOhio State University

Roman SamulyakBrookhaven National Laboratory

Todd SatogataBrookhaven National Laboratory

Dolores A. ShafferScience and Technology Associates, Inc.

George C. SchatzNorthwestern University

David Paul SchisselGeneral Atomics

Thomas Christoph SchulthessOak Ridge National Laboratory

Mary Anne ScottDOE Office of Science

Stephen L. ScottOak Ridge National Laboratory

David B. SerafiniLawrence Berkeley National Laboratory


A Science-Based Case for Large-Scale Simulation xvii

James SethianUniversity of California–Berkeley

John Nicholas ShadidSandia National Laboratories

Mikhail ShashkovLos Alamos National Laboratory

John ShalfLawrence Berkeley National Laboratory

Henry ShawDOE Office of Science

Mark S. ShephardRensselaer Polytechnic Institute

Jason F. ShepherdSandia National Laboratories

Timothy ShippertPacific Northwest National Laboratory

Andrei P. ShishloOak Ridge National Laboratory

Arie ShoshaniLawrence Berkeley National Laboratory

Andrew R. SiegelArgonne National Laboratory/University ofChicago

Sonya Teresa SmithHoward University

Mitchell D. SmookeYale University

Allan Edward SnavelySan Diego Supercomputer Center

Matthew SottileLos Alamos National Laboratory

Carl R. SovinecUniversity of Wisconsin

Panagiotis SpentzourisFermi National Accelerator Facility

Daniel Shields SpicerNASA/Goddard Space Flight Center

Bill SpotzNational Center for Atmospheric Research

David J. SrolovitzPrinceton University

T. P. StraatsmaPacific Northwest National Laboratory

Michael Robert StrayerOak Ridge National Laboratory

Scott StudhamPacific Northwest National Laboratory

Robert SugarUniversity of California–Santa Barbara

Shuyu SunUniversity of Texas–Austin

F. Douglas SwestyState University of New York–Stony Brook

John TaylorArgonne National Laboratory

Ani R. ThakarJohns Hopkins University

Andrew F. B. TompsonLawrence Livermore National Laboratory

Charles H. TongLawrence Livermore National Laboratory

Josep TorrellasUniversity of Illinois

Harold Eugene TreasePacific Northwest National Laboratory

Albert J. ValocchiUniversity of Illinois

Debra van OpstalCouncil on Competitiveness

Leticia VelazquezUniversity of Texas–El Paso

Roelof Jan VersteegIdaho National Energy and EngineeringLaboratory

Jeffrey Scott VetterLawrence Livermore National Laboratory

Angela VioliUniversity of Utah

Robert VoigtCollege of William and Mary

Gregory A. VothUniversity of Utah

Albert F. WagnerArgonne National Laboratory

Homer F. WalkerWorcester Polytechnic Institute


xviii A Science-Based Case for Large-Scale Simulation

Lin-Wang WangLawrence Berkeley National Laboratory

Warren WashingtonNational Center for Atmospheric Research

Chip WatsonThomas Jefferson National AcceleratorFacility

Michael WehnerLawrence Livermore National Laboratory

Mary F. WheelerUniversity of Texas–Austin

Roy WhitneyThomas Jefferson National AcceleratorFacility

Theresa L. WindusPacific Northwest National Laboratory

Steve WojtkiewiczSandia National Laboratories

Pat WorleyOak Ridge National Laboratory

Margaret H. WrightNew York University

John WuLawrence Berkeley National Laboratory

Ying XuOak Ridge National Laboratory

Zhiliang XuState University of New York–Stony Brook

Steve YabusakiPacific Northwest National Laboratory

Woo-Sun YangLawrence Berkeley National Laboratory

Katherine YelickUniversity of California–Berkeley

Thomas ZachariaOak Ridge National Laboratory

Bernard ZakSandia National Laboratories

Shujia Zhou ZhouNorthrop Grumman/TASC

John P. ZiebarthLos Alamos National Laboratory

1. Introduction

A Science-Based Case for Large-Scale Simulation 1

National investment in large-scalecomputing can be justified on numerousgrounds. These include national security;economic competitiveness; technologicalleadership; formulation of strategic policyin defense, energy, environmental, healthcare, and transportation systems; impact oneducation and workforce development;impact on culture in its increasingly digitalforms; and, not least, supercomputing’sability to capture the imagination of thenation’s citizens. The public expects all tosee the fruits of these historicallyestablished benefits of large-scalecomputing, and it expects that the UnitedStates will have the earliest access to future,as yet unimagined benefits.

This report touches upon many of thesemotivations for investment in large-scalecomputing. However, it is directlyconcerned with a deeper one, which inmany ways controls the rest. The questionthis report seeks to answer is: What case canbe made for a national investment in large-scalecomputational modeling and simulation fromthe perspective of basic science? This can bebroken down into several smallerquestions: What scientific results are likely,given a significantly more powerful simulationcapability, say, a hundred to a thousand timespresent capability? What principal hurdles canbe identified along the path to realizing thesenew results? How can these hurdles besurmounted?

A WORKSHOP TO FIND FRESHANSWERS

These questions and others were posed to alarge gathering of the nation’s leadingcomputational scientists at a workshopconvened on June 24–25, 2003, in theshadow of the Pentagon. They were

answered in 27 topical breakout groups,whose responses constitute the bulk of thesecond volume of this two-volume report.

The computational scientists participatingin the workshop included those who regardthemselves primarily as natural scientists—e.g., physicists, chemists, biologists—butalso many others. By design, about one-third of the participants were computa-tional mathematicians. Another third werecomputer scientists who are orientedtoward research on the infrastructure onwhich computational modeling andsimulation depends.

A CASCADIC STRUCTURE

Computer scientists and mathematiciansare scientists whose research in the area oflarge-scale computing has direction andvalue of its own. However, the structure ofthe workshop, and of this report,emphasizes a cascadic flow from naturalscientists to mathematicians and from bothto computer scientists (Figure 1). We areprimarily concerned herein with theopportunities for large-scale simulation toachieve new understanding in the naturalsciences. In this context, the role of themathematical sciences is to provide ameans of passing from a physical model toa discrete computational representation ofthat model and of efficiently manipulatingthat representation to obtain results whosevalidity is well understood. In turn, thecomputer sciences allow the algorithms thatperform these manipulations to be executedefficiently on leading-edge computersystems. They also provide ways for theoften-monumental human efforts requiredin this process to be effectively abstracted,leveraged across other applications, andpropagated over a complex and ever-evolving set of hardware platforms.

1. INTRODUCTION

1. Introduction

2 A Science-Based Case for Large-Scale Simulation

Most computational scientists understandthat this layered set of dependencies—ofnatural science on mathematics and of bothon computer science—is anoversimplification of the full picture ofinformation flows that drive computationalscience. Indeed, one of the most exciting ofmany rapidly evolving trends incomputational science is a “phasetransition” (a concept to which we return inChapter 2) from a field of solo scientificinvestigators, who fulfill their needs byfinding computer software “thrown overthe wall” by computational tool builders, tolarge, coordinated groups of naturalscientists, mathematicians, and computerscientists who carry on a constantconversation about what can be done now,what is possible in the future, and how toachieve these goals most effectively.

Almost always, there are numerous ways totranslate a physical model intomathematical algorithms and to implementa computational program on a givencomputer. Uninformed decisions madeearly in the process can cut off highlyproductive options that may arise later on.Only a bi-directional dialog, up and downthe cascade, can systematically ensure thatthe best resources and technologies areemployed to solve the most challengingscientific problems.

Its acknowledged limitations aside, thecascadic model of contemporarycomputational science is useful for

understanding the importance of severalpractices in the contemporarycomputational research community. Theseinclude striving towards the abstraction oftechnologies and towards identification ofuniversal interfaces between well-definedfunctional layers (enabling reuse acrossmany applications). It also motivatessustained investment in the cross-cuttingtechnologies of computational mathematicsand computer science, so that increases inpower and capability may be continuallydelivered to a broad portfolio of scientificapplications on the other side of astandardized software interface.

SIMULATION AS A PEERMETHODOLOGY TO EXPERIMENTAND THEORY

Historians of science may pick differentdates for the development of the twinpillars of theory and experiment in the“scientific method,” but both are ancient,and their interrelationship is wellestablished. Each takes a turn leading theother as they mutually refine ourunderstanding of the natural world—experiment confronting theory with newpuzzles to explain and theory showingexperiment where to look next to test itsexplanations. Unfortunately, theory andexperiment both possess recognizedlimitations at the frontier of contemporaryscience.

The strains on theory were apparent to JohnVon Neumann (1903–1957) and drove him

Figure 1. The layered, cascadic structure of computational science.

1. Introduction


to develop the computational sciences offluid dynamics, radiation transport,weather prediction, and other fields.Models of these phenomena, whenexpressed as mathematical equations, areinevitably large-scale and nonlinear, whilethe bulk of the edifice of mathematicaltheory (for algebraic, differential, andintegral equations) is linear. Computationwas to Von Neumann, and remains today,the only truly systematic means of makingprogress in these and many other scientificarenas. Breakthroughs in the theory ofnonlinear systems come occasionally, butcomputational gains come steadily withincreased computational power andresolution.

The strains on experimentation, the goldstandard of scientific truth, have grownalong with expectations for it (Figure 2).Unfortunately, many systems and manyquestions are nearly inaccessible toexperiment. (We subsume under“experiment” both experiments designedand conducted by scientists, andobservations of natural phenomena out ofthe direct control of scientists, such as

celestial events.) The experiments thatscientists need to perform to answer themost pressing questions are sometimesdeemed unethical (e.g., because of theirimpact on living beings), hazardous (e.g.,because of their impact on the life-sustaining environment we have on earth),politically untenable (e.g., prohibited bytreaties to which the United States is aparty), difficult (e.g., requiringmeasurements that are too rapid ornumerous to be instrumentable or timeperiods too long to complete), or simplyexpensive.

The cost of experimentation is particularlyimportant to consider when consideringsimulation as an alternative, for it cannot bedenied that simulation can also beexpensive. Cost has not categoricallydenied experimentalists their favored tools,such as accelerators, orbital telescopes,high-power lasers, and the like, at pricessometimes in the billions of dollars perfacility, although such facilities are carefullyvetted and planned. Cost must also notcategorically deny computational scientiststheir foremost scientific instrument—the

Figure 2. Practical strains on experimentation that invite simulation as a peer modality of scientificinvestigation.

1. Introduction


supercomputer and its full complement ofsoftware, networking, and peripheraldevices. It will be argued in the body of thisreport that simulation is, in fact, a highlycost-effective lever for the experimentalprocess, allowing the same, or betterscience to be accomplished with fewer,better-conceived experiments.

Simulation has aspects in common withboth theory and experiment. It isfundamentally theoretical, in that it startswith a theoretical model—typically a set ofmathematical equations. A powerfulsimulation capability breathes new life intotheory by creating a demand forimprovements in mathematical models.Simulation is also fundamentallyexperimental, in that upon constructing andimplementing a model, one observes thetransformation of inputs (or controls) tooutputs (or observables).

Simulation effectively bridges theory andexperiment by allowing the execution of“theoretical experiments” on systems,including those that could never exist in thephysical world, such as a fluid withoutviscosity. Computation also bridges theoryand experiment by virtue of the computer’sserving as a universal and versatile datahost. Once experimental data have beendigitized, they can be compared side-by-side with simulated results in visualizationsystems built for the latter, reliablytransmitted, and retrievably archived.Moreover, simulation and experiment cancomplement each other by allowing acomplete picture of a system that neithercan provide as well alone. Some data maynot be measurable with the bestexperimental techniques available, andsome mathematical models may be toosensitive to unknown parameters to invokewith confidence. Simulation can be used to“fill in” the missing experimental fields,

using experimentally measured fields asinput data. Data assimilation can also besystematically employed throughout asimulation to keep it from “drifting” frommeasurements, thus overcoming the effectof modeling uncertainty.

Further comparing simulation toexperiment, one observes a sociological orpolitical advantage to increased investmentin the tools of simulation: they are widelyusable across the entire scientificcommunity. A micro-array analyzer is oflimited use to a plasma physicist and atokamak of limited use to a biologist.However, a large computer or anoptimization algorithm in the form ofportable software may be of use to both.

HURDLES TO SIMULATION

While the promise of simulation isprofound, so are its limitations. Thoselimitations often come down to a questionof resolution. Though of vast size,computers are triply finite: they representindividual quantities only to a finiteprecision, they keep track of only a finitenumber of such quantities, and they operateat a finite rate.

Although all matter is, in fact, composed ofa finite number of particles (atoms), thenumber of particles in a macroscopicsample of matter, on the order of a trillionparticles, places simulations at macroscopicscales from “first principles” (i.e., from thequantum theory of electronic structure)well beyond any conceivable computationalcapability. Similar problems arise whentime scales are considered. For example, therange of time scales in protein folding is12 orders of magnitude, since a process thattakes milliseconds occurs in moleculardance steps that last femtoseconds (atrillion times shorter)—again, far too wide a

1. Introduction


range to routinely simulate using firstprinciples.

Even simulation of systems adequatelydescribed by equations of the macroscopiccontinuum—fluids such as air or water—can be daunting for the most powerfulcomputers available today. Such computersare capable of execution rates in the tens ofteraflop/s (one teraflop/s is one trillionarithmetic operations per second) and cancost tens to hundreds of millions of dollarsto purchase and millions of dollars per yearto operate. However, to simulate fluid-mechanical turbulence in the boundary andwake regions of a typical vehicle using“first principles” of continuum modelingwould tie up such a computer for months,which makes this level of simulation tooexpensive for routine use.

One way to describe matter at themacroscopic scale is to divide space up intosmall cubes and to ascribe values fordensity, momentum, temperature, and soforth, to each cube. As the cubes becomesmaller and smaller, a more and moreaccurate description is obtained, at a cost ofincreasing memory to store the values andtime to visit and update each value inaccordance with natural laws, such as theconservation of energy applied to the cube,given fluxes in and out of neighboringcubes. If the number of cubes along eachside is doubled, the cost of the simulation,which is proportional to the total number ofcubes, increases by a factor of 23 or 8. This isthe “curse of dimensionality”: increasingthe resolution of a simulation quickly eatsup any increases in processor powerresulting from the well-known Moore’sLaw (a doubling of computer speed every18–24 months). Furthermore, for time-dependent problems in three dimensions,the cost of a simulation grows like thefourth power of the resolution. Therefore,

an increase in computer performance by afactor of 100 provides an increase inresolution in each spatial and temporaldimension by a factor of only about 3.

One way to bring more computing powerto bear on a problem is to divide the workto be done over many processors.Parallelism in computer architecture—theconcurrent use of many processors—canprovide additional factors of hundreds ormore, beyond Moore’s Law alone, to theaggregate performance available for thesolution of a given problem. However, thedesire for increased resolution and, hence,accuracy is seemingly insatiable. Thissimply expressed and simply understood“curse of dimensionality” means that“business as usual” in scaling up today’ssimulations by riding the computerperformance curve will never be cost-effective—and is too slow!

The “curse of knowledge explosion”—namely, that no one computational scientistcan hope to track advances in all of thefacets of the mathematical theories,computational models and algorithms,applications and computing systemssoftware, and computing hardware(computers, data stores, and networks) thatmay be needed in a successful simulationeffort—is another substantial hurdle to theprogress of simulation. Both the curse ofdimensionality and the curse of knowledgeexplosion can be addressed. In fact, theyhave begun to be addressed in aremarkably successful initiative calledScientific Discovery through AdvancedComputing (SciDAC), which was launchedby the U.S. Department of Energy’s (DOE’s)Office of Science in 2001. SciDAC was just astart; many difficult challenges remain.Nonetheless, SciDAC (discussed in moredetail in Chapter 2) established a model fortackling these “curses” that promises to

1. Introduction


bring the power of the most advancedcomputing technologies to bear on the mostchallenging scientific and engineeringproblems facing the nation.

In addition to the technical challenges inscientific computing, too few computationalcycles are currently available to fullycapitalize upon the progress of the SciDACinitiative. Moreover, the computationalscience talent emerging from the nationaleducational pipeline is just a trickle incomparison to what is needed to replicatethe successes of SciDAC throughout themany applications of science andengineering that lag behind. Cycle-starvation and human resource–starvationare serious problems that are felt across allof the computational science programssurveyed in creating this report. However,they are easier hurdles to overcome thanthe above-mentioned “curses.”

SUMMARY OF RECOMMENDATIONS

We believe that the most effective programto deliver new science by means of large-scale simulation will be one built over abroad base of disciplines in the naturalsciences and the associated enablingtechnologies, since there are numeroussynergisms between them, which arereadily apparent in the companion volume.

We also believe that a program to realizethe promises outlined in the companionvolume requires a balance of investments inhardware, software, and human resources.While quantifying and extrapolating thethirst for computing cycles and networkbandwidth is relatively straightforward, thescheduling of breakthroughs inmathematical theories and computationalmodel and algorithms has always beendifficult. Breakthroughs are more likely,however, where there is critical mass of all

types of computational scientists and goodcommunication between them.

The following recommendations areelaborated upon in Chapter 6 of thisvolume:

• Major new investments incomputational science are needed in allof the mission areas of the Office ofScience in DOE, as well as those ofmany other agencies, so that the UnitedStates may be the first, or among thefirst, to capture the new opportunitiespresented by the continuing advances incomputing power. Such investmentswill extend the important scientificopportunities that have been attainedby a fusion of sustained advances inscientific models, mathematicalalgorithms, computer architecture, andscientific software engineering.

• Multidisciplinary teams, with carefullyselected leadership, should beassembled to provide the broad range ofexpertise needed to address theintellectual challenges associated withtranslating advances in science,mathematics, and computer science intosimulations that can take full advantageof advanced computers.

• Extensive investment in newcomputational facilities is stronglyrecommended, since simulation nowcost-effectively complementsexperimentation in the pursuit of theanswers to numerous scientificquestions. New facilities should strike abalance between capability computingfor those “heroic simulations” thatcannot be performed any other way andcapacity computing for “production”simulations that contribute to the steadystream of progress.

1. Introduction


• Investment in hardware facilities shouldbe accompanied by sustained collateralinvestment in software infrastructurefor them. The efficient use of expensivecomputational facilities and the datathey produce depends directly uponmultiple layers of system software andscientific software, which, together withthe hardware, are the “engines ofscientific discovery” across a broadportfolio of scientific applications.

• Additional investments in hardwarefacilities and software infrastructureshould be accompanied by sustainedcollateral investments in algorithmresearch and theoretical development.Improvements in basic theory andalgorithms have contributed as much toincreases in computational simulationcapability as improvements in hardwareand software over the first six decadesof scientific computing.

• Computational scientists of all typesshould be proactively recruited withimproved reward structures andopportunities as early as possible in theeducational process so that the numberof trained computational scienceprofessionals is sufficient to meetpresent and future demands.

• Sustained investments must be made innetwork infrastructure for access andresource sharing as well as in thesoftware needed to support

collaborations among distributed teamsof scientists, in recognition of the factthat the best possible computationalscience teams will be widely separatedgeographically and that researchers willgenerally not be co-located withfacilities and data.

• Federal investment in innovative, high-risk computer architectures that are wellsuited to scientific and engineeringsimulations is both appropriate andneeded to complement commercialresearch and development. Thecommercial computing marketplace isno longer effectively driven by theneeds of computational science.

SCOPE

The purpose of this report is to capture theprojections of the nation’s leadingcomputational scientists and providers ofenabling technologies concerning what newscience lies around the corner with the nextincrease in delivered computational powerby a factor of 100 to 1000. It also identifiesthe issues that must be addressed if theseincreases in computing power are to lead toremarkable new scientific discoveries.

It is beyond the charter and scope of thisreport to form policy recommendations onhow this increased capability and capacityshould be procured, or how to prioritizeamong numerous apparently equally ripescientific opportunities.

2. Scientific Discovery through Advanced Computing: A Successful Pilot Program




THE BACKGROUND OF SciDAC

For more than half a century, visionariesanticipated the emergence of computationalmodeling and simulation as a peer totheory and experiment in pushing forwardthe frontiers of science, and DOE and itsantecedent agencies over this period haveinvested accordingly. Computationalmodeling and simulation made greatstrides between the late 1970s and the early1990s as Seymour Cray and the companythat he founded produced ever-fasterversions of Cray supercomputers. Between1976, when the Cray 1 was delivered toDOE’s Los Alamos National Laboratory,and 1994, when the last traditional Craysupercomputer, the Cray T90, wasintroduced, the speed of these computersincreased from 160 megaflop/s to32 gigaflop/s, a factor of 200! However, bythe early 1990s it became clear that buildingever-faster versions of Craysupercomputers using traditionalelectronics technologies was notsustainable.

In the 1980s DOE and other federalagencies began investing in alternativecomputer architectures for scientificcomputing featuring multiple processorsconnected to multiple memory unitsthrough a wide variety of mechanisms. Awild decade ensued during which manydifferent physical designs were explored, aswell as many programming models forcontrolling the flow of data in them, frommemory to processor and from one memoryunit to another. The mid-1990s saw asubstantial convergence, which foundcommercial microprocessors, each withdirect local access to only a relatively small

fraction of the total memory, connected inclusters numbering in the thousandsthrough dedicated networks or fastswitches. The speeds of the commercialmicroprocessors in these designs increasedaccording to Moore’s Law, producingdistributed-memory parallel computersthat rivaled the power of traditional Craysupercomputers. These new machinesrequired a major recasting of the algorithmsthat scientists had developed and polishedfor more than two decades on Craysupercomputers, but those who investedthe effort found that the new massivelyparallel computers provided impressivelevels of computing capability.

By the late 1990s it was evident to manythat the continuing, dramatic advances incomputing technologies had the potentialto revolutionize scientific computing.DOE’s National Nuclear Security Agency(DOE-NNSA) launched the AcceleratedStrategic Computing Initiative (ASCI) tohelp ensure the safety and reliability of thenation’s nuclear stockpile in the absence oftesting. In 1998, DOE’s Office of Science(DOE-SC) and the National ScienceFoundation (NSF) sponsored a workshop atthe National Academy of Sciences toidentify the opportunities and challenges ofrealizing major advances in computationalmodeling and simulation. The report fromthis workshop identified scientific advancesacross a broad range of science, ranging inscale from cosmology through nanoscience,which would be made possible by advancesin computational modeling and simulation.However, the report also noted challengesto be met if these advances were to berealized.

2. SCIENTIFIC DISCOVERY THROUGH ADVANCEDCOMPUTING: A SUCCESSFUL PILOT PROGRAM



By the time the workshop report waspublished, the Office of Science was wellalong the path to developing an Office-wide program designed to address thechallenges identified in the DOE-SC/NSFworkshop. The response to the NationalAcademy report, as well as to the 1999report from the President’s InformationTechnology Advisory Committee—theScientific Discovery through AdvancedComputing (SciDAC) program—wasdescribed in a 22-page March 2000 reportby the Office of Science to the House andSenate Appropriations Committees.Congress funded the program in thefollowing fiscal year, and by the end ofFY 2001, DOE-SC had established51 scientific projects, following a nationalpeer-reviewed competition.

SciDAC explicitly recognizes that theadvances in computing technologies thatprovide the foundation for the scientificopportunities identified in the DOE-SC/NSF report (as well as in Volume 2 of thecurrent report) are not, in themselves,sufficient for the “coming of age” ofcomputational simulation. Closeintegration of three disciplines—science,computer science, and appliedmathematics—and a new discipline at theirintersection called computational scienceare needed to realize the full benefits of thefast-moving advances in computingtechnologies. A change in the sociology ofthis area of science was necessary for thefusion of these streams of disciplinarydevelopments. While not uniquelyAmerican in its origins, the “new kind ofcomputational science” promoted in theSciDAC report found fertile soil ininstitutional arrangements for the conductof scientific research that are highlydeveloped in the United States, andseemingly much less so elsewhere:multiprogram, multidisciplinary nationallaboratories, and strong laboratory-

university partnerships in research andgraduate education. SciDAC is built onthese institutional foundations as well as onthe integration of computational modeling,computer algorithms, and computingsystems software.

In this chapter, we briefly examine the shortbut substantial legacy of SciDAC. Ourpurpose is to understand the implicationsof its successes, which will help us identifythe logical next steps in advancingcomputational modeling and simulation.

SciDAC’S INGREDIENTS FORSUCCESS

Aimed at Discovery

SciDAC boldly affirms the importance ofcomputational simulation for new scientificdiscovery, not just for “rationalizing” theresults of experiments. The latter role forsimulation is a time-honored one from thedays when it was largely subordinate toexperimentation, rather than peer to it. Thatrelationship has been steadily evolving. In afrontispiece quotation from J. S. Langer, thechair of the July 1998 multiagency NationalWorkshop on Advanced ScientificComputing, the SciDAC report declared:“The computer literally is providing a newwindow through which we can observe thenatural world in exquisite detail.”

Through advances in computationalmodeling and simulation, it will be possibleto extend dramatically the exploration ofthe fundamental processes of nature—e.g.,the interactions between elementaryparticles that give rise to the matter aroundus, the interactions between proteins andother molecules that sustain life—as well asadvance our ability to predict the behaviorof a broad range of complex natural andengineered systems—e.g., nanoscaledevices, microbial cells, fusion energyreactors, and the earth’s climate.



Multidisciplinary

The structure of SciDAC boldly reflects thefact that the development and applicationof leading-edge simulation capabilities hasbecome a multidisciplinary activity. Thisimplies, for instance, that physicists are ableto focus on the physics and not ondeveloping parallel libraries for coremathematical operations. It also impliesthat mathematicians and computerscientists have direct access to substantiveproblems of high impact, and not onlymodels loosely related to application-specific difficulties. The deliverables andaccountabilities of the collaborating partiesare cross-linked. The message of SciDAC isa “declaration of interdependence.”

Specifically, the SciDAC report called for• creation of a new generation of scientific

simulation codes that take fulladvantage of the extraordinarycomputing capabilities of terascalecomputers;

• creation of the mathematical andsystems software to enable the scientificsimulation codes to effectively andefficiently use terascale computers; and

• creation of collaboratory softwareinfrastructure to enable geographicallyseparated scientists to work togethereffectively as a team and to facilitateremote access to both facilities and data.

The $57 million per year made availableunder SciDAC is divided roughly evenlyinto these three categories. SciDAC alsocalled for upgrades to the scientificcomputing hardware infrastructure of theOffice of Science and outlined aprescription for building a robust, agile,and cost-effective computing infrastructure.These recommendations have thus far beenmodestly supported outside of the SciDACbudget.

Figure 3, based on a figure from the originalSciDAC report, shows the interrelationshipsbetween SciDAC program elements as

Figure 3. Relationship between the science application teams, the Integrated SoftwareInfrastructure Centers (ISICs), and networking and computing facilities in the SciDAC program.



implemented, particularizing Figure 1 ofthis report to the SciDAC program. All fiveprogram offices in the Office of Science areparticipating in SciDAC. The disciplinarycomputational science groups are beingfunded by the Offices of Basic EnergySciences, Biology and EnvironmentalResearch, Fusion Energy Sciences, and HighEnergy and Nuclear Physics, and theintegrated software infrastructure centersand networking infrastructure centers arebeing funded by the Office of AdvancedScientific Computing Research.

Multi-Institutional

Recognizing that the expertise needed tomake major advances in computationalmodeling and algorithms and computingsystems software was distributed amongvarious institutions across the UnitedStates, SciDAC established scientificallynatural collaborations without regard forgeographical or institutional boundaries—laboratory-university collaborations andlaboratory-laboratory collaborations.Laboratories and universities havecomplementary strengths in research.Laboratories are agile in reorganizing theirresearch assets to respond to missions andnew opportunities. Universities respond ona much slower time scale to structuralchange, but they are hotbeds of innovationwhere graduate students explore novelideas as an integral part of their education.In the area of simulation, universities areoften sources of new models, algorithms,and software and hardware concepts thatcan be evaluated in a dissertation-scaleeffort. However, such developments maynot survive the graduation of the doctoralcandidate and are rarely rendered intoreusable, portable, extensible infrastructurein the confines of the academicenvironment, where the reward structureemphasizes the original discovery and

proof of concept. Laboratory teams excel increating, refining, “hardening,” anddeploying computational scientificinfrastructure. Thirteen laboratories and50 universities were funded in the initialround of SciDAC projects.

Setting New Standards for Software

For many centuries, publication of originalresults has been the gold standard ofaccomplishment for scientific researchers.During the first 50 years of scientificcomputing, most research codes targeted afew applications on a small range ofhardware, and were designed to be used byexperts only (often just the author). Codedevelopment projects emphasized findingthe shortest path to the next publishableresult. Packaging code for reuse was not ajob for serious scientists. As the role ofcomputational modeling and simulations inadvancing scientific discovery grows, thisculture is changing. Scientists now realizethe value of high-quality, reusable,extensible, portable software, and theproducers of widely used packages arehighly respected.

Unfortunately, funding structures in thefederal agencies have evolved more slowlyand it has not always been easy forscientists to find support for thedevelopment of software for the researchcommunity. By dedicating more than halfits resources to the development of high-value scientific applications software andthe establishment of integrated softwareinfrastructure centers, SciDAC hassupported efforts to make advancedscientific packages easy to use, port them tonew hardware, maintain them, and makethem capable of interoperating with otherrelated packages that achieve critical massof use and offer complementary features.



SciDAC recognizes that large scientificsimulation codes have many needs incommon: modules that construct and adaptthe computational mesh on which theirmodels are based, modules that producethe discrete equations from themathematical model, modules that solvethe resulting equations, and modules thatvisualize the results and manage, mine, andanalyze the large data sets. By constructingthese modules as interoperable componentsand supporting the resulting componentlibraries so that they stay up to date withthe latest algorithmic advances and workon the latest hardware, SciDAC amortizescosts and creates specialized points ofcontact for a portfolio of applications.

The integrated software infrastructurecenters that develop these libraries aremotivated to ensure that their users haveavailable the full range of algorithmicoptions under a common interface, alongwith a complete description of theirresource tradeoffs (e.g., memory versustime). Their users can try all reasonableoptions easily and adapt to differentcomputational platforms without recodingthe applications. Furthermore, when anapplication group intelligently drivessoftware infrastructure research in a newand useful direction, the result is soonavailable to all users.

Large Scale

Because computational science is driventoward the large scale by requirements ofmodel fidelity and resolution, SciDACemphasizes the development of softwarefor the most capable computers available.In the United States, these are nowprimarily hierarchical distributed-memorycomputers. Enough experience wasacquired on such machines in heroicmission-driven simulation programs in the

latter half of the 1990s (in such programs asDOE’s ASCI) to justify them as cost-effective simulation hardware, providedthat certain generic difficulties inprogramming are amortized over manyuser groups and many generations ofhardware. Still, many challenges remain tomaking efficient and effective use of thesecomputers for the broad range of scientificapplications important to the mission ofDOE’s Office of Science.

Supporting Community Codes

As examples of simulations from Office ofScience programs, the SciDAC reporthighlighted problems of ascertainingchemical structure and the massivelyparallel simulation code NWCHEM, winnerof a 1999 R&D 100 Award and a FederalLaboratory Consortium Excellence inTechnology Transfer Award in 2000.

NWCHEM, currently installed at nearly900 research institutions worldwide, wascreated by an international team ofscientists from twelve institutions (two U.S.national laboratories, two Europeannational laboratories, five U.S. universities,and three European universities) over thecourse of five years. The core teamconsisted of seven full-time theoretical andcomputational chemists, computerscientists, and applied mathematicians,ably assisted by ten postdoctoral fellows ontemporary assignment. Overall, NWCHEMrepresents an investment in excess of100 person-years. Yet, the layered structureof NWCHEM permits it to easily absorbnew physics or algorithm modules, to runon environments from laptops to the fastestparallel architectures, and to be ported to anew supercomputing environment withisolated minor modifications, typically inless than a week. The scientific legacyembodied in NWCHEM goes back three



decades to the Gaussian code, for whichJohn Pople won the 1998 Nobel Prize inchemistry.

Community codes such as Gaussian andNWCHEM can consume hundreds ofperson-years of development (Gaussianprobably has 500–1000 person-yearsinvested in it), run at hundreds ofinstallations, are given large fractions ofcommunity computing resources fordecades, and acquire an authority that canenable or limit what is done and acceptedas science in their respective communities.Yet, historically, not all community codeshave been crafted with the care ofNWCHEM to achieve performance,portability, and extensibility. Except at thebeginning, it is difficult to promote majoralgorithmic changes in such codes, sincechange is expensive and pulls resourcesfrom core progress. SciDAC applicationgroups have been chartered to build newand improved community codes in areassuch as accelerator physics, materialsscience, ocean circulation, and plasmaphysics. The integrated softwareinfrastructure centers of SciDAC have achance, due to the interdependence builtinto the SciDAC program structure, tosimultaneously influence all of these newcommunity codes by delivering softwareincorporating optimal algorithms that maybe reused across many applications.Improvements driven by one applicationwill be available to all. While SciDACapplication groups are developing codes forcommunities of users, SciDAC enablingtechnology groups are building codes forcommunities of developers.

Enabling Success

The first two years of SciDAC provideample evidence that constructingmultidisciplinary teams around challengeproblems and providing centralized points

of contact for mathematical andcomputational technologies is a soundinvestment strategy. For instance, DaltonSchnack of General Atomics, whose groupoperates and maintains the simulation codeNIMROD to support experimental designof magnetically confined fusion energydevices, states that software recentlydeveloped by Xiaoye (Sherry) Li ofLawrence Berkeley National Laboratory hasenhanced his group’s ability to performfusion simulations to a degree equivalent tothree to five years of riding the wave ofhardware improvements alone. Ironically,the uniprocessor version of Li’s solver,originally developed while she was agraduate student at the University ofCalifornia–Berkeley, predated the SciDACinitiative; but it was not known to theNIMROD team until it was recommendedduring discussions between SciDAC fusionand solver software teams. In themeantime, SciDAC-funded researchimproved the parallel performance of Li’ssolver in time for use in NIMROD’s currentcontext.

M3D, a fusion simulation code developedat the Princeton Plasma Physics Laboratory,is complementary to NIMROD; it uses adifferent mathematical formulation. LikeNIMROD and many simulation codes, M3Dspends a large fraction of its time solvingsparse linear algebraic systems. In acollaboration that predated SciDAC, theM3D code group was using PETSc, aportable toolkit of sparse solvers developedat Argonne National Laboratory and inwide use around the world. Under SciDAC,the Terascale Optimal PDE Solversintegrated software center providedanother world-renowned class of solvers,Hypre, from Lawrence Livermore NationalLaboratory “underneath” the same codeinterface that M3D was already using to callPETSc’s solvers. The combined PETSc-



Hypre solver library allows M3D to solveits current linear systems two to three timesfaster than previously, and this SciDACcollaboration has identified several avenuesfor improving the PETSc-Hypre codesupport for M3D and other users.

While these are stories of performanceimprovement, allowing faster turnaroundand more production runs, there are alsostriking stories of altogether new physicsenabled by SciDAC collaborations. ThePrinceton fusion team has combined forceswith the Applied Partial DifferentialEquations Center, an integrated softwareinfrastructure center, to produce asimulation code capable of resolving thefine structure of the dynamic interfacebetween two fluids over which a shockwave passes—the so-called Richtmyer-Meshkov instability—in the presence of amagnetic field. This team has found that astrong magnetic field stabilizes theinterface, a result not merely of scientificinterest but also of potential importance inmany devices of practical interest to DOE.

SciDAC’s Terascale Supernova Initiative(TSI), led by Tony Mezzacappa of OakRidge National Laboratory, has achievedthe first stellar core collapse simulations forsupernovae with state-of-the-art neutrinotransport. The team has also producedstate-of-the-art models of nuclei in a stellarcore. These simulations have broughttogether two fundamental fields at theirrespective frontiers, demonstrated theimportance of accurate nuclear physics insupernova models, and motivated plannedexperiments to measure the nuclearprocesses occurring in stars. This group hasalso performed simulations of a rapidneutron capture process believed to occurin supernovae and to be responsible for thecreation of half the heavy elements in theuniverse. This has led to a surprising and

important result: this process can occur inenvironments very different than heretoforethought, perhaps providing a way aroundsome of the fundamental difficultiesencountered in past supernova models inexplaining element synthesis. To extendtheir model to higher dimensions, the TSIteam has engaged mathematicians andcomputer scientists to help them overcomethe “curse of dimensionality” and exploitmassively parallel machines. Mezzacappahas declared that he would never go back todoing computational physics outside of amultidisciplinary team.

Setting a New OrganizationalParadigm

By acknowledging sociological barriers tomultidisciplinary research up front andbuilding accountabilities into the projectawards to force the mixing of “scientificcultures” (e.g., physics with computerscience), SciDAC took a risk that has paidoff handsomely in technical results. As aresult, new DOE-SC initiatives are takingSciDAC-like forms—for instance, theFY 2003 initiative Nanoscience Theory,Modeling and Simulation. The roadmap forDOE’s Fusion Simulation Program likewiseenvisions multidisciplinary teamsconstructing complex simulations linkingtogether what are today variousfreestanding simulation codes for differentportions of a fusion reactor. Such complexsimulations are deemed essential to the on-time construction of a workingdemonstration International ThermonuclearExperimental Reactor.

In many critical application domains, suchteams are starting to self-assemble as anatural response to the demands of the nextlevel of scientific opportunity. This self-assembly represents a phase transition inthe computational science universe, fromautonomous individual investigators to



coordinated multidisciplinary teams. Whenexperimental physicists transitioned fromsolo investigators to members of largeteams centered at major facilities, theylearned to include the cost of otherprofessionals (statisticians, engineers,computer scientists, technicians,administrators, and others) in their projects.Computational physicists are now learningto build their projects in large teams built

around major computational facilities andto count the cost of other professionals. Allsuch transitions are sociologically difficult.In the case of computational simulation,there is sufficient infrastructure in commonbetween different scientific endeavors thatDOE and other science agencies canamortize its expense and provide organiza-tional paradigms for it. SciDAC is such apilot paradigm. It is time to extend it.



3. Anatomy of a Large-Scale Simulation

Because the methodologies and processesof computational simulation are still notfamiliar to many scientists and laypersons,we devote this brief chapter to theirillustration. Our goal is to acquaint thereader with a number of issues that arise incomputational simulation and to highlightthe benefits of collaboration across thetraditional boundaries of science. Thestructure of any program aimed atadvancing computational science must bebased on this understanding.

Scientific simulation is a verticallyintegrated process, as indicated in Figures 1and 3. It is also an iterative process, inwhich computational scientists traverseseveral times a hierarchy of issues inmodeling, discretization, solution, codeimplementation, hardware execution,visualization and interpretation of results,and comparison to expectations (theories,experiments, or other simulations). They dothis to develop a reliable “scientificinstrument” for simulation, namely amarriage of a code with a computing and

3. ANATOMY OF A LARGE-SCALE SIMULATION

user environment. This iteration isindicated in the two loops (light arrows) ofFigure 4 (reproduced from the SciDACreport).

One loop over this computational process isrelated to validation of the model andverification that the process is correctlyexecuting the model. With slight violence togrammar and with some oversimplification,validation poses the question “Are wesolving the right equations?” andverification, the question “Are we solvingthe equations right?”

The other loop over the process ofsimulation is related to algorithm andperformance tuning. A simulation that yieldshigh-fidelity results is of little use if it is tooexpensive to run or if it cannot be scaled upto the resolutions, simulation times, orensemble sizes required to describe thereal-world phenomena of interest. Againwith some oversimplification, algorithmtuning poses the question, “Are we gettingenough science per operation?” and

Figure 4. The process of scientific simulation, showing the validation and verification loop (left) andalgorithm and performance tuning loop (right).



performance tuning, “Are we gettingenough operations per cycle?”

It is clear from Figure 4 that a scientificsimulation depends upon numerouscomponents, all of which have to workproperly. For example, the theoreticalmodel must not overlook or misrepresentany physically important effect. The modelmust not be parameterized with data ofuncertain quality or provenance. Themathematical translation of the theoreticalmodel (which may be posed as adifferential equation, for instance) to adiscrete representation that a computer canmanipulate (a large system of algebraicrelationships based on the differentialequation, for instance) inevitably involvessome approximation. This approximationmust be controlled by a priori analysis or aposteriori checks. The algorithm that solvesthe discrete representation of the systembeing modeled must reliably converge. Thecompiler, operating system, and message-passing software that translate thearithmetic manipulations of the solutionalgorithm into loads and stores frommemory to registers, or into messages thattraverse communication links in a parallelcomputer, must be reliable and fault-tolerant. They must be employed correctlyby programmers and simulation scientistsso that the results are well defined andrepeatable. In a simulation that relies onmultiple interacting models (such as fluidsand structures in adjacent domains, orradiation and combustion in the samedomain), information fluxes at the interfaceof the respective models, which representphysical fluxes in the simulatedphenomenon, must be consistent. Insimulations that manipulate or produceenormous quantities of data on multipledisks, files must not be lost or mistakenlyoverwritten, nor misfiled results extractedfor visualization. Each potential pitfall may

require a different expert to detect it,control it at an acceptable level, orcompletely eliminate it as a possibility infuture designs. These are some of theactivities that exercise the left-hand loop inFigure 4.

During or after the validation of asimulation code, attention must be paid tothe performance of the code in eachhardware/system software environmentavailable to the research team. Efforts tocontrol discretization and solution error in asimulation may grossly increase thenumber of operations required to execute asimulation, far out of proportion to theirbenefit. Similarly, “conservative”approaches to moving data within aprocessor or between processors, to reducethe possibility of errors ormiscommunication, may require extensivesynchronization or minimal data replicationand may be inefficient on a modernsupercomputer. More elegant, oftenadaptive, strategies must be adopted for thesimulation to be execution-worthy on anexpensive massively parallel computer,even though these strategies may requirehighly complex implementation. These aresome of the activities that exercise the right-hand loop in Figure 4.

ILLUSTRATION: SIMULATION OF ATURBULENT REACTING FLOW

The dynamics of turbulent reacting flowshave long constituted an important topic incombustion research. Apart from thescientific richness of the combination offluid turbulence and chemical reaction,turbulent flames are at the heart of thedesign of equipment such as reciprocatingengines, turbines, furnaces, andincinerators, for efficiency and minimalenvironmental impact. Effective propertiesof turbulent flame dynamics are alsorequired as model inputs in a broad range



of larger-scale simulation challenges,including fire spread in buildings orwildfires, stellar dynamics, and chemicalprocessing.

Figure 5 shows a photograph of alaboratory-scale premixed methane flame,stabilized by a metal rod across the outlet ofa cylindrical burner. Although experimentaland theoretical investigations have beenable to provide substantial insights into thedynamics of these types of flames, there arestill a number of outstanding questionsabout their behavior. Recently, a team ofcomputational scientists supported bySciDAC, in collaboration withexperimentalists at Lawrence BerkeleyNational Laboratory, performed the firstsimulations of these types of flames usingdetailed chemistry and transport. Figure 6shows an instantaneous image of the flame

surface from the simulation. (Many suchinstantaneous configurations are averagedin the photographic image of Figure 5.)

The first step in simulations such as these isto specify the computational models thatwill be used to describe the reacting flows.The essential feature of reacting flows is theset of chemical reactions taking place in thefluid. Besides chemical products, thesereactions produce both temperature andpressure changes, which couple to thedynamics of the flow. Thus, an accuratedescription of the reactions is critical topredicting the properties of the flame,including turbulence. Conversely, it is thefluid flow that transports the reactingchemical species into the reaction zone andtransports the products of the reaction andthe released energy away from the reactionzone. The location and shape of the reactionzone are determined by a delicate balanceof species, energy, and momentum fluxesand are highly sensitive to how these fluxesare specified at the boundaries of thecomputational domain. Turbulence canwrinkle the reaction zone, giving it much

Figure 5. Photograph of an experimentalturbulent premixed rod-stabilized “V” methaneflame.

Figure 6. Instantaneous surface of a turbulentpremixed rod-stabilized V-flame, from simulation.



more area than it would have in its laminarstate, without turbulence. Hence, incorrectprediction of turbulence intensity mayunder- or over-represent the extent ofreaction.

From first principles, the reactions ofmolecules are described by the Schrödingerequation, and fluid flow by the Navier-Stokes equations. However, each of theseequation sets is too difficult to solvedirectly for the turbulent premixed rod-stabilized “V” methane flame illustrated inFigure 5. So we must rely onapproximations to these equations. Theseapproximations define the computationalmodels used to describe the methane flame.For the current simulations, the set of84 reactions describing the methane flameinvolves 21 chemical species (methane,oxygen, water, carbon dioxide, and manytrace products and reaction intermediates).The particular version of the compressibleNavier-Stokes model employed here allowsdetailed consideration of convection,diffusion, and expansion effects that shapethe reaction; but it is “filtered” to removesound waves, which pose complicationsthat do not measurably affect combustion inthis regime. The selection of models such asthese requires context-specific expertjudgment. For instance, in a jet turbinesimulation, sound waves might represent anon-negligible portion of the energy budgetof the problem and would need to bemodeled; but it might be valid (and verycost-effective) to use a less intricatechemical reaction mechanism to answer thescientific question of interest.

Development of the chemistry model andvalidation of its predictions, which wasfunded in part by DOE’s Chemical SciencesProgram, is an interesting scientific storythat is too long to be told here. Suffice it tosay that this process took many years. In

some scientific fields, high-fidelity modelsof many important processes involved inthe simulations are not available; andadditional scientific research will be neededto achieve a comparable level of confidence.

These computations pictured here wereperformed on the IBM SP, named“Seaborg,” at the National Energy ResearchScientific Computing Center, using up to2048 processors. They are among the mostdemanding combustion simulations everperformed. However, the ability to do themis not merely a result of improvements incomputer speed. Improvements inalgorithm technology funded by the DOEApplied Mathematical Sciences Programover the past decade have beeninstrumental in making these computationsfeasible. Mathematical analysis is used toreformulate the equations describing thefluid flow so that high-speed acoustictransients are removed analytically, whilecompressibility effects due to chemicalreactions are retained. The mathematicalmodel of the fluid flow is discretized usinghigh-resolution finite difference methods,combined with local adaptive meshrefinement by which regions of the finitedifference grid are automatically refined orcoarsened to maximize overallcomputational efficiency. Theimplementation of this methodology usesan object-oriented, message-passingsoftware framework that handles thecomplex data distribution and dynamicload balancing needed to effectively exploitmodern parallel supercomputers. The dataanalysis framework used to explore theresults of the simulation and create visualimages that lead to understanding is basedon recent developments in “scriptinglanguages” from computer science. Thecombination of these algorithmicinnovations reduces the computational costby a factor of 10,000 compared with a



standard uniform-grid compressible-flowapproach.

Researchers have just begun to explore thedata generated in these simulations.Figure 7 presents a comparison between alaboratory measurement and a simulation.The early stages of the analysis of this dataindicate that excellent agreement isobtained with observable quantities.Further analyses promise to shed new light

on the morphology and structure of thesetypes of flames.

This simulation achieves its remarkableresolution of the thin flame edge by meansof adaptive refinement, which places fineresolution around those features that needit and uses coarse resolution away fromsuch features, saving orders of magnitudein computer memory and processing time.As the reaction zone shifts, the refinementautomatically tracks it, adding andremoving resolution dynamically. Asdescribed earlier, the simulation alsoemploys a mathematical transformationthat enables it to skip over tiny time stepsthat would otherwise be required to resolvesound waves in the flame domain. The fourorders of magnitude saved by thesealgorithmic improvements, compared withprevious practices, is a larger factor thanthe performance gain from parallelizationon a computer system that costs tens ofmillions of dollars! It will be many yearsbefore such a high-fidelity simulation canbe cost-effectively run using previouspractices on any computer.

Unfortunately, the adaptivity andmathematical filtering employed to savememory and operations complicate thesoftware and throw the execution of thecode into a regime of computation thatprocesses fewer operations per second anduses the thousands of processors of theSeaborg system less uniformly than a“business as usual” algorithm would. As aresult, the simulation runs at a smallpercentage of the theoretical peak rate ofthe Seaborg machine. However, since itmodels the phenomenon of interest inmuch less execution time and delivers itmany years earlier than would otherwise bepossible, it is difficult to argue against its“scientific efficiency.” In this case, asimplistic efficiency metric like “percentageof theoretical peak” is misleading.

Figure 7. Comparison of experimental particleimage velocimetry crossview of the turbulent “V”flame (top) and a simulation (bottom).



The simulation highlighted earlier isremarkable for its physical fidelity in acomplex physics multiscale problem.Experimentalists are easily attracted tofruitful collaborations when simulationsreach this threshold of fidelity. Resultingcomparisons help refine both experimentand simulation in successive leaps.However, the simulation is equallynoteworthy for the combination of thesophisticated mathematical techniques itemploys and its parallel implementation onthousands of processors. Breakthroughssuch as this have caused scientists to referto leading-edge large-scale simulationenvironments as “time machines.” Theyenable us to pull into the present scientificunderstanding for which we wouldotherwise wait years if we were dependenton experiment or on commodity computingcapability alone.

The work that led to this simulationrecently won recognition for one of itscreators, John Bell. He and his colleaguePhil Colella, both of Lawrence BerkeleyNational Laboratory, were awarded the firstPrize in Computational Science andEngineering awarded by the Society forIndustrial and Applied Mathematics andthe Association for Computing Machineryin June 2003. The prize is awarded “in thearea of computational science in recognition

of outstanding contributions to thedevelopment and use of mathematical andcomputational tools and methods for thesolution of science and engineeringproblems.” Bell and Colella have appliedthe automated adaptive meshingmethodologies they developed in suchdiverse areas as shock physics,astrophysics, and flow in porous media, aswell as turbulent combustion.

The software used in this combustionsimulation is currently being extended inmany directions under SciDAC. Thegeometries that it can accuratelyaccommodate are being generalized foraccelerator applications. For magneticallyconfined fusion applications, a particledynamics module is being added.

Software of this complexity and versatilitycould never have been assembled in thetraditional mode of development, wherebyindividual researchers with separateconcerns asynchronously toss packageswritten to arbitrary specifications “over thetransom.” Behind any simulation ascomplex as the one discussed in thischapter stands a tightly coordinated,vertically integrated team. This processillustrates the computational science “phasetransition” that has found proof of conceptunder the SciDAC program.



4. Opportunities at the Scientific Horizon

4. OPPORTUNITIES AT THE SCIENTIFIC HORIZON

The availability of computers 100 to1000 times more powerful than thosecurrently available will have a profoundimpact on computational scientists’ abilityto simulate the fundamental physical,chemical, and biological processes thatunderlie the behavior of natural andengineered systems. Chemists will be ableto model the diverse set of molecularprocesses involved in the combustion ofhydrocarbon fuels and the catalyticproduction of chemicals. Materialsscientists will be able to predict theproperties of materials from knowledge oftheir structure, and then, inverting theprocess, design materials with a targetedset of properties. Physicists will be able tomodel a broad range of complexphenomena—from the fundamentalinteractions between elementary particles,to high-energy nuclear processes andparticle-electromagnetic field interactions,to the interactions that govern the behaviorof fusion plasmas. Biologists will be able topredict the structure and, eventually, thefunction of the 30–40,000 proteins coded inthe human DNA. They will also mine thevast reservoirs of quantitative data beingaccumulated using high-throughputexperimental techniques to obtain newinsights into the processes of life.

In this section, we highlight some of themost exciting opportunities for scientificdiscovery that will be made possible by anintegrated set of investments, following theSciDAC paradigm described in Chapter 2,in the programmatic offices in the Office ofScience in DOE—Advanced ScientificComputing Research, Basic EnergySciences, Biological and EnvironmentalResearch, Fusion Energy Sciences, HighEnergy Physics, and Nuclear Physics. InTable 1 we provide a list of the major

scientific accomplishments expected in eachscientific area from the investmentsdescribed in this report. The table isfollowed by an expanded description of oneof these accomplishments in each category.The progress described in these paragraphsillustrates the progress that will be madetoward solving many other scientificproblems and in many other scientificdisciplines as the methodologies ofcomputational science are refined throughapplication to these primary Office ofScience missions. Volume 2 of this reportprovides additional information on themany advances that, scientists predict,these investments will make possible.

PREDICTING FUTURE CLIMATES

While existing general circulation models(GCMs) used to simulate climate canprovide good estimates of continental- toglobal-scale climatic variability and change,they are less able to produce estimates ofregional changes that are essential forassessing environmental and economicconsequences of climatic variability andchange. For example, predictions of globalaverage atmospheric temperature changesare of limited utility unless the effects ofthese changes on regional space-heatingneeds, rainfall, and growing seasons can beascertained. As a result of the investmentsdescribed here, it will be possible tomarkedly improve both regionalpredictions of long-term climatic variabilityand change, and assessments of thepotential consequences of such variabilityand change on natural and managedecosystems and resources of value tohumans (see Figure 8).

Accurate prediction of regional climaticvariability and change would have



Table 1. If additional funding becomes available for Office of Science scientific, computerscience, and mathematics research programs, as well as for the needed additional

computing resources, scientists predict the following major advances in theresearch programs supported by the Office of Science within the

next 5–10 years

Research Programs Major Scientific Advances

Biological and • Provide global forecasts of Earth’s future climate at regional scales usingEnvironmental high-resolution, fully coupled and interactive climate, chemistry, landSciences cover, and carbon cycle models.

• Develop predictive understanding of subsurface contaminant behaviorthat provides the basis for scientifically sound and defensible cleanupdecisions and remediation strategies.

• Establish a mathematical and computational foundation for the study ofcellular and molecular systems and use computational modeling andsimulation to predict and simulate the behavior of complex microbialsystems for use in mission application areas.

Chemical and • Provide direct 3-dimensional simulations of a turbulent methane-air jetMaterials Sciences flame with detailed chemistry and direct 3-dimensional simulations of

autoignition of n-heptane at high pressure, leading to more-efficient,lower-emission combustion devices.

• Develop an understanding of the growth and structural, mechanical, andelectronic properties of carbon nanotubes, for such applications asstrengtheners of polymers and alloys, emitters for display screens, andconductors and nonlinear circuit elements for nanoelectronics.

• Provide simulations of the magnetic properties of nanoparticles andarrays of nanoparticles for use in the design of ultra-high-densitymagnetic information storage. (Nanomagnetism is the simplest of manyimportant advances in nanoscience that will impact future electronics,tailored magnetic response devices, and even catalysis.)

• Resolve the current disparity between theory and experiment forconduction through molecules with attached electronic leads. (This is thevery basis of molecular electronics and may well point to opportunitiesfor chemical sensor technology.)

Fusion Energy • Improve understanding of fundamental physical phenomena in high-Sciences temperature plasmas, including transport of energy and particles,

turbulence, global equilibrium and stability, magnetic reconnection,electromagnetic wave/particle interactions, boundary layer effects inplasmas, and plasma/material interactions.

• Simulate individual aspects of plasma behavior, such as energy andparticle confinement times, high-pressure stability limits in magneticallyconfined plasmas, efficiency of electromagnetic wave heating andcurrent drive, and heat and particle transport in the edge region of aplasma, for parameters relevant to magnetically-confined fusion plasmas.

• Develop a fully integrated capability for predicting the performance ofmagnetically confined fusion plasmas with high physics fidelity, initiallyfor tokamak configurations and ultimately for a broad range of practicalenergy-producing magnetic confinement configurations.

• Advance the fundamental understanding and predictability of high-energy density plasmas for inertial fusion energy. (Inertial fusion andmagnetically confined fusion are complementary technologicalapproaches to unlocking the power of the atomic nucleus.)



tremendous value to society, and that valuecould be realized if the most significantlimitations in the application of GCMs wereovercome. The proposed investmentswould allow climate scientists to• refine the horizontal resolution of the

atmospheric component from 160 to40 km and the resolution of the oceancomponent from 105 to 15 km (bothimprovements are feasible now, butthere is insufficient computing power toroutinely execute GCMs at theseresolutions);

• add interactive atmospheric chemistry,carbon cycle, and dynamic terrestrialvegetation components to simulateglobal and regional carbon, nitrogen,and sulfur cycles and simulate theeffects of land-cover and land-usechanges on climate (these cycles andeffects are known to be important toclimate, but their implementation inGCMs is rudimentary because ofcomputational limitations); and

• improve the representation of importantsubgrid processes, especially clouds andatmospheric radiative transfer.

High Energy • Establish the limits of the Standard Model of Elementary ParticlePhysics Physics by achieving a detailed understanding of the effects of strong

nuclear interactions in many different processes, so that the equality ofStandard Model parameters measured in different experiments can beverified. (Or, if verification fails, signal the discovery of new physicalphenomena at extreme sub-nuclear distances.)

• Develop realistic simulations of the performance of particle accelerators,the large and complex core scientific instruments of high-energy physicsresearch, both to optimize the design, technology and cost of futureaccelerators and to use existing accelerators more effectively andefficiently.

Nuclear Physics • Understand the characteristics of the quark-gluon plasma, especially inthe temperature-density region of the phase transition expected fromquantum chromodynamics (QCD) and currently sought experimentally.

• Obtain a quantitative, predictive understanding of the quark-gluonstructure of the nucleon and of interactions of nucleons.

• Understand the mechanism of core collapse supernovae and the natureof the nucleosynthesis in these spectacular stellar explosions.

Table 1 (continued)

Research Programs Major Scientific Advances

Figure 8. Surface chlorophyll distributionssimulated by the Parallel Ocean Program forconditions in late 1996 (a La Niña year) showingintense biological activity across the equatorialPacific due to upwelling and transport ofnutrient-rich cold water. Comparisons with satelliteocean color measurements validate model accuracyand inform our understanding of the regionalenvironmental response to global climate change.[S. Chu, S. Elliott, and M. Maltrud]



Even at 40-km resolution, cloud dynamicsare incompletely resolved. Recent advancesin the development of cloud sub-models,i.e., two-dimensional and quasi three-dimensional cloud-resolving models, canovercome much of this limitation; but theirimplementation in global GCMs willrequire substantially more computingpower.

SCIENCE FROM THE NANOSCALE UP

Our ability to characterize small moleculesor perfect solids as collections of atoms isvery advanced. So is our ability tocharacterize materials of engineeringdimensions. However, our abilities fall farshort of what is necessary because many ofthe properties and processes necessary todevelop new materials and optimizeexisting ones occur with characteristic sizes(or scales) that lie between the atomisticworld (less than 1 nanometer) and theworld perceived by humans (meters ormore), a spatial range of at least a billion.Bridging these different scales, whilekeeping the essential quantum descriptionat the smallest scale, will requireunprecedented theoretical innovation andcomputational effort, but it is an essentialstep toward a complete fundamental theoryof matter.

The nanoscale, 10–100 nanometers or so—the first step in the hierarchy of scalesleading from the atomic scale tomacroscopic scales—is also important in itsown right because, at the nanoscale, newproperties emerge that have no parallel ateither larger or smaller scales. Thus, thenew world of nanoscience offerstremendous opportunities for developinginnovative technological solutions to abroad range of real-world problems usingscientists’ increasing abilities to manipulatematter at this scale. Theory, modeling, andsimulation are essential to developing an

understanding of matter at the nanoscale,as well as to developing the newnanotechnologies (see Figure 9). At thenanoscale, experiment often cannot standalone, requiring sophisticated modelingand simulation to deconvoluteexperimental results and yield scientificinsight. To simulate nanoscale entities,scientists will have to perform ab initioatomistic calculations for more atoms thanever before, directly treat the correlatedbehavior of the electrons associated withthose atoms, and model nanoscale systemsas open systems rather than closed(isolated) ones. The computational effortrequired to simulate nanoscale systems farexceeds any computational efforts inmaterials and molecular science to date bya factor of at least 100 to 1000. The

Figure 9. To continue annual doubling of harddrive storage density (top: IBM micro-drive),improved understanding of existing materialsand development of new ones are needed. Firstprinciples spin dynamics and the use of massivelyparallel computing resources are enablingresearchers at Oak Ridge National Laboratory tounderstand magnetism at ferromagnetic-antiferromagnetic interfaces (bottom: magneticmoments at a Co/FeMn interface) with the goal oflearning how to store binary data in ever smallerdomains, resulting in further miniaturizationbreakthroughs. [M. Stocks]



investments envisioned in this area willmake nanoscale simulations possible byadvancing the theoretical understanding ofnanoscale systems and supporting thedevelopment of computational models andsoftware for simulating nanoscale systemsand processes, as well as allowing detailedsimulations of nanoscale phenomena andtheir interaction with the macroscopicworld.

DESIGNING AND OPTIMIZINGFUSION REACTORS

Fusion, the energy source that fuels the sunand stars, is a potentially inexhaustible andenvironmentally safe source of energy.However, to create the conditions underwhich hydrogen isotopes undergo nuclearfusion and release energy, high-temperature(100 million degrees centigrade) plasmas—a fluid of ions and electrons—must beproduced, sustained, and confined. Plasmaconfinement is recognized internationallyas a grand scientific challenge and aformidable technical task. With the researchprogram and computing resources outlinedin this report, it will be possible to developa fully integrated simulation capability forpredicting the performance of magneticconfinement systems, such as tokamaks.

In the past, specialized simulations wereused to explore selected key aspects of thephysics of magnetically confined plasmas,e.g., turbulence or micro-scale instabilities.Only an integrated model can bring all ofthe strongly interacting physical processestogether to make comprehensive, reliablepredictions about the real-world behaviorof fusion plasmas. The simulation of anentire magnetic fusion confinement systeminvolves simultaneous modeling of the coreand edge plasmas, as well as the plasma-wall inter-actions. In each region of theplasma, turbulence can cause anomalies inthe transport of the ionic species in the

plasma; there can also be abrupt changes inthe form of the plasma caused by large-scale instabilities (see Figure 10).Computational modeling of these keyprocesses requires large-scale simulationscovering a broad range of space and timescales. An integrated simulation capabilitywill dramatically enhance the efficientutilization of a burning fusion device, inparticular, and the optimization of fusionenergy development in general; and itwould serve as an intellectual integrator ofphysics knowledge ranging from advancedtokamaks to innovative confinementconcepts.

PREDICTING THE PROPERTIES OFMATTER

Fundamental questions in high-energyphysics and nuclear physics pivot on ourability to simulate so-called stronginteractions on a computational lattice viathe theory of quantum chromodynamics(QCD).

The ongoing goal of research in high-energyphysics is to find the elementaryconstituents of matter and to determinehow they interact. The current state of ourknowledge is summarized by the Standard

Figure 10. Simulation of the interchange ofplasma between the hot (red) and cold (green)regions of a tokamak via magnetic reconnection.[W. Park]



Model of Elementary Particle Physics. Acentral focus of experiments at U.S. high-energy physics facilities is precision testingof the Standard Model. The ultimate aim ofthis work is to find deviations from thismodel—a discovery that would require theintroduction of new forms of matter or newphysical principles to describe matter at theshortest distances. Determining whether ornot each new measurement is in accordwith the Standard Model requires anaccurate evaluation of the effects of thestrong interactions on the process understudy. These computations are a crucialcompanion to any experimental program inhigh-energy physics. QCD is the onlyknown method of systematically reducingall sources of theoretical error.

Lattice calculations lag well behindexperiment at present, and the precision ofexperiments under way or being plannedwill make the imbalance even worse. Theimpact of the magnitude of the lattice errorsis shown in Figure 11. For the StandardModel to be correct, the parameters ρ and ηmust lie in the region of overlap of all thecolored bands. Experiments measurevarious combinations of ρ and η, as shownby the colored bands. The reduction oftheory errors shown will require tens ofteraflop-years of simulations, which will be

available through the construction of newtypes of computers optimized for theproblems at hand. A further reduction,when current experiments are complete,will need another order of magnitude ofcomputational power.

A primary goal of the nuclear physicsprogram at the Relativistic Heavy IonCollider (RHIC) at Brookhaven NationalLaboratory is the discovery andcharacterization of the quark–gluonplasma—a state of matter predicted byQCD to exist at high densities andtemperatures. To confirm an observation ofa new state of matter, it is important todetermine the nature of the phase transitionbetween ordinary matter and a quark–gluon plasma, the properties of the plasma,and the equation of state. Numericalsimulations of QCD on a lattice haveproved to be the only source of a prioripredictions about this form of matter in thevicinity of the phase transition.

Recent work suggests that a major increasein computing power would result indefinitive predictions of the properties ofthe quark–gluon plasma, providing timelyand invaluable input into the RHIC heavyion program.

Figure 11. Constraints on the Standard Model parameters r and h. For the Standard Model tobe correct, they must be restricted to the region of overlap of the solidly colored bands. Thefigure on the left shows the constraints as they exist today. The figure on the right shows the constraintsas they would exist with no improvement in the experimental errors, but with lattice gauge theoryuncertainties reduced to 3%. [R. Patterson]



Similar lattice calculations are capable ofpredicting and providing an understandingof the inner structure of the nucleon wherethe quarks and gluons are confined, anothermajor nuclear physics goal. Precise latticecalculations of nucleon observables canplay a pivotal role in helping to guideexperiments. This methodology is wellunderstood, but a “typical” calculation of asingle observable requires over a year withall the computers presently available totheorists. A thousandfold increase incomputing power would make the sameresult available in about 10 hours. Makingthe turn-around time for simulationscommensurate with that of experimentswould lead to major progress in thequantitative, predictive understanding ofthe structure of the nucleon and of theinteractions between nucleons.

BRINGING A STAR TO EARTH

Nuclear astrophysics is a major researcharea within nuclear physics—in particular,the study of stellar explosions and theirelement production. Explosions of massivestars, known as core collapse supernovae,are the dominant source of elements in thePeriodic Table between oxygen and iron;and they are believed to be responsible forthe production of half the heavy elements(heavier than iron). Life in the universewould not exist without these catastrophicstellar deaths.

To understand these explosive events, theirbyproducts, and the phenomena associatedwith them will require large-scale,multidimensional, multiphysicssimulations (see Figure 12). Currentsupercomputers are enabling the firstdetailed two-dimensional simulations.Future petascale computers would provide,for the first time, the opportunity tosimulate supernovae realistically in threedimensions.

While observations from space are critical,the development of theoretical supernovamodels will also be grounded in terrestrialexperiment. Supernova science has playeda significant role in motivating theconstruction of the Rare Isotope Accelerator(RIA) and the National UndergroundLaboratory for Science and Engineering(NUSEL). As planned, RIA will performmeasurements that will shed light onmodels for heavy element production insupernovae; and NUSEL will housenumerous neutrino experiments, includinga half-megaton neutrino detector capable ofdetecting extra-galactic supernovaneutrinos out to Andromeda.

Supernovae have importance beyond theirplace in the cosmic hierarchy: the extremesof density, temperature, and compositionencountered in supernovae provide anopportunity to explore fundamental nuclearand particle physics that would otherwisebe inaccessible in terrestrial experiments.Supernovae therefore serve as cosmiclaboratories, and computational models ofsupernovae constitute the bridge betweenthe observations, which bring usinformation about these explosions, and thefundamental physics we seek.

Figure 12. Development of a newly discoveredshock wave instability and resulting stellarexplosions (supernovae), in two- and three-dimensional simulations. [J. Blondin,A. Mezzacappa, and C. DeMarino; visualization byR. Toedte]

5. Enabling Mathematics and Computer Science Tools




MATHEMATICAL TOOLS FORLARGE-SCALE SIMULATION

Mathematics is the bridge between physicalreality and computer simulations ofphysical reality. The starting point for acomputer simulation is a mathematicalmodel, expressed in terms of a system ofequations and based on scientificunderstanding of the problem beinginvestigated, such as the knowledge offorces acting on a particle or a parcel offluid. For such a model, there are threetypes of mathematical tools that may bebrought to bear in order to produce and usecomputer simulations.

• Model Analysis. Although themathematical model is motivated byscience, it must stand on its own as aninternally consistent mathematicalobject for its computer representation tobe well defined. For that reason, it isnecessary to resolve a number of issuesregarding the mathematical structure ofthe model. Do the equations have aunique solution for all physicallymeaningful inputs? Do small changes inthe inputs lead only to small changes inthe solution, or, as an indication ofpotential trouble, could a smalluncertainty be magnified by the modelinto large variability in the outcome?

• Approximation and Discretization. Formany systems, the number ofunknowns in the model is infinite, aswhen the solution is a function ofcontinuum variables such as space andtime. In such cases, it is necessary toapproximate the infinite number ofunknowns with a large but finitenumber of unknowns in order to

5. ENABLING MATHEMATICS AND COMPUTERSCIENCE TOOLS

represent it on a computer. Themathematical issues for this process,called “discretization,” include (1) theextent to which the finite approximationbetter agrees with the solution to theoriginal equations as the number ofcomputational unknowns increases and(2) the relationship between the choiceof approximation and qualitativemathematical properties of the solution,such as singularities.

• Solvers and Software. Once one hasrepresented the physical problem as thesolution to a finite number of equationsfor a finite number of unknowns, howdoes one make best use of thecomputational resources to calculate thesolution to those equations? Issues atthis stage include the development ofoptimally efficient algorithms, and themapping of computations onto acomplex hierarchy of processors andmemory systems.

Although these three facets representdistinct mathematical disciplines, thesedisciplines are typically employed inconcert to build complete simulations. Thechoice of appropriate mathematical toolscan make or break a simulation code. Forexample, over a four-decade period of ourbrief simulation era, algorithms alone havebrought a speed increase of a factor of morethan a million to computing theelectrostatic potential induced by a chargedistribution, typical of a computationalkernel found in a wide variety of problemsin the sciences. The improvement resultingfrom this algorithmic speedup iscomparable to that resulting from thehardware speedup due to Moore’s Law



over the same length of time (seeFigure 13). The series of algorithmicimprovements producing this factor are allbased on a fundamental mathematicalproperty of the underlying model—namely,that the function describing electrostaticcoupling between disjoint regions in spaceis very smooth. Expressed in the right way,this coupling can therefore be resolvedaccurately with little computational effort.The various improvements in algorithmsfor solving this problem came about by

successive redesigns of the discretizationmethods and solver algorithms to betterexploit this mathematical property of themodel.

Another trend in computational science isthe steady increase in the intellectualcomplexity of virtually all aspects of themodeling process. The scientistscontributing to this report identified anddeveloped (in Volume 2) eight cross-cuttingareas of applied mathematics for whichresearch and development is needed toaccommodate the rapidly increasingcomplexity of state-of-the-artcomputational science. They fall into thefollowing three categories.

• Managing Model Complexity.Scientists want to use increasingcomputing capability to improve thefidelity of their models. For manyproblems, this means introducingmodels with more physical effects, moreequations, and more unknowns. InMultiphysics Modeling, the goal is todevelop a combination of analytical andnumerical techniques to better representproblems with multiple physicalprocesses. These techniques may rangefrom analytical methods to determinehow to break a problem up into weaklyinteracting components, to newnumerical methods for exploiting such adecomposition of the problem to obtainefficient and accurate discretizations intime. A similar set of issues arises fromthe fact that many systems of interesthave processes that operate on lengthand time scales that vary over manyorders of magnitude. MultiscaleModeling addresses the representationand interaction of behaviors on multiplescales so that results of interest arerecovered without the (unaffordable)expense of representing all behaviors at

Figure 13. Top: A table of the scaling ofmemory and processing requirements for thesolution of the electrostatic potential equation ona uniform cubic grid of n ××××× n ××××× n cells. Bottom:The relative gains of some solution algorithmsfor this problem and Moore’s Law forimprovement of processing rates over the sameperiod (illustrated for the case where n = 64).Algorithms yield a factor comparable to that of thehardware, and the gains typically can be combined(that is, multiplied together). The algorithmic gainsbecome more important than the hardware gains forlarger problems. If adaptivity is exploited in thediscretization, algorithms may do better still, thoughcombining all of the gains becomes more subtle inthis case.



uniformly fine scales. Approachesinclude the development of adaptivemethods, i.e., discretization methods thatcan represent directly many orders ofmagnitude in length scales that mightappear in a single mathematical model,and hybrid methods for couplingradically different models (continuumvs. discrete, or stochastic vs.deterministic), each of which representsthe behavior on a different scale.Uncertainty Quantification addressesissues connected with mathematicalmodels that involve fits to experimentaldata, or that are derived from heuristicsthat may not be directly connected tophysical principles. Uncertaintyquantification uses techniques fromfields such as statistics and optimizationto determine the sensitivity of models toinputs with errors and to design modelsto minimize the effect of those errors.

• Discretizations of Spatial Models.Many of the applications described inthis document have, as corecomponents of their mathematicalmodels, the equations of fluid dynamicsor radiation transport, or both.Computational Fluid Dynamics andTransport and Kinetic Methods have astheir goal the development of the nextgeneration of spatial discretizationmethods for these problems. Issuesinclude the development ofdiscretization methods that are well-suited for use in multiphysicsapplications without loss of accuracy orrobustness. Meshing Methodsspecifically address the process ofdiscretizing the computational domain,itself, into a union of simple elements.Meshing is usually a prerequisite fordiscretizing the equations defined overthe domain. This process includes themanagement of complex geometrical

objects arising in technological devices,as well as in some areas of science, suchas biology.

• Managing Computational Complexity.Once the mathematical model has beenconverted into a system of equations fora finite collection of unknowns, it isnecessary to solve the equations. Thegoal of efforts in the area of Solvers and“Fast” Algorithms is to developalgorithms for solving these systems ofequations that balance computationalefficiency on hierarchicalmultiprocessor systems, scalability (theability to use effectively additionalcomputational resources to solveincreasingly larger problems), androbustness (insensitivity of thecomputational cost to details of theinputs). An algorithm is said to be“fast” if its cost grows, roughly, onlyproportionally to the size of theproblem. This is an ideal algorithmicproperty that is being obtained for moreand more types of equations. DiscreteMathematics and Algorithms make upa complementary set of tools formanaging the computationalcomplexity of the interactions ofdiscrete objects. Such issues arise, forexample, in traversing data structuresfor calculations on unstructured grids,in optimizing resource allocation onmultiprocessor architectures, or inscientific problems in areas such asbioinformatics that are posed directly as“combinatorial” problems.

COMPUTER SCIENCE TOOLS FORLARGE-SCALE SIMULATION

The role of computer science in ultrascalesimulation is to provide tools that addressthe issues of complexity, performance, andunderstanding. These issues cut across thecomputer science disciplines. An integrated



approach is required to solve the problemsfaced by applications.

COMPLEXITY

One of the major challenges for applicationsis the complexity of turning a mathematicalmodel into an effective simulation. Thereare many reasons for this, as indicated inFigure 4. Often, even after the principles ofa model and simulation algorithm are wellunderstood, too much effort still is requiredto turn this understanding into practicebecause of the complexity of code design.Current programming models andframeworks do not provide sufficientsupport to allow domain experts to beshielded from details of data structures andcomputer architecture. Even after anapplication code produces correct scientificresults, too much effort still is required toobtain high performance. Code tuningefforts needed to match algorithms tocurrent computer architectures requirelengthy analysis and experimentation.

Once an application runs effectively, oftenthe next hurdle is saving, accessing, andsharing data. Once the data are stored, sinceultrascale simulations often produceultrascale-sized datasets, it is still toodifficult to process, investigate, andvisualize the data in order to accomplishthe purpose of the simulation—to advancescience. These difficulties are compoundedby the problems faced in sharing resources,both human and computer hardware.

Despite this grim picture, prospects forplacing usable computing environmentsinto the hands of scientific domain expertsare improving. In the last few years, therehas been a growing understanding of theproblems of managing complexity incomputer science, and therefore of theirpotential solutions. For example, there is adeeper understanding of how to make

programming languages expressive andeasy to use without sacrificing highperformance on the sophisticated, adaptivealgorithms.

Another example is the success ofcomponent-oriented software in someapplication domains; such “components”have allowed computational scientists tofocus their own expertise on their sciencewhile exploiting the newest algorithmicdevelopments. Many groups in high-performance computing have tackled theseissues with significant leadership fromDOE. Fully integrated efforts are requiredto produce a qualitative change in the wayapplication groups cope with thecomplexity of designing, building, andusing ultrascale simulation codes.

PERFORMANCE

One of the drivers of software complexity isthe premium on performance. The mostobvious aspect of the performance problemis the performance of the computerhardware. Although there have beenastounding gains in arithmetic processingrates over the last five decades, users oftenreceive only a small fraction of thetheoretical peak of processing performance.There is a perception that this fraction is, infact, declining. This perception is correct insome respects. For many applications, thereason for a declining percentage of peakperformance is the relative imbalance in theperformance of the subsystems of high-endcomputers. While the raw performance ofcommodity processors has followedMoore’s Law and doubled every 18 months,the performance of other critical parts of thesystem, such as memory and interconnect,have improved much less rapidly, leadingto less-balanced overall systems. Solvingthis problem requires attention to system-scale architectural issues.



As with code complexity issues, there aremultiple on-going efforts to addresshardware architecture issues. Differentarchitectural solutions may be required fordifferent algorithms and applications. Asingle architectural convergence point, suchas that occupied by current commodity-based terascale systems, may not be themost cost-effective solution for all users. Acomprehensive simulation programrequires that several candidate architecturalapproaches receive sustained support toexplore their promise.

Performance is a cross-cutting issue, andcomputer science offers automatedapproaches to developing codes in waysthat allow computational scientists toconcentrate on their science. For example,techniques that allow a programmer toautomatically generate code for anapplication that is tuned to a specificcomputer architecture address both theissues of managing the complexity ofhighly-tuned code and the problem ofproviding effective portability betweenhigh-performance computing platforms.Such techniques begin with separateanalyses of the “signature” of theapplication (e.g., the patterns of localmemory accesses and inter-processorcommunication) and the parameters of thehardware (e.g., cache sizes, latencies,bandwidths). There is usually plenty ofalgorithmic freedom in scheduling andordering operations and exchanges whilepreserving correctness. This freedomshould not be arbitrarily limited byparticular expression in a low-levellanguage, but rather chosen close toruntime to best match a given application–architecture pair. Similarly, the performanceof I/O and dataset operations can beimproved significantly through the use ofwell-designed and adaptive algorithms.

UNDERSTANDING

Computer science also addresses the issueof understanding the results of acomputation. Ultrascale datasets are toolarge to be grasped directly. Applicationsthat generate such sets already currentlyrely on a variety of tools to attempt toextract patterns and features from the data.Computer science offers techniques in datamanagement and understanding that can beused to explore data sets, searching forparticular patterns. Visualizationtechniques help scientists explore theirdata, taking advantage of the uniquehuman visual cortex and visually-stimulated human insight. Current effortsin this area are often limited by the lack ofresources, in terms of both staffing andhardware.

Understanding requires harnessing theskills of many scientists. Collaborationtechnologies help scientists at differentinstitutions work together. Gridtechnologies that simplify data sharing andprovide access to both experimental andultrascale computing facilities allowcomputational scientists to work together tosolve the most difficult problems facing thenation today. Although these technologieshave been demonstrated, much workremains to make them a part of everyscientist’s toolbox. Key challenges are inscheduling multiple resources and in datasecurity.

In Volume 2 of this report, scientists frommany disciplines discuss critical computerscience technologies needed acrossapplications. Visual Data Exploration andAnalysis considers understanding theresults of a simulation throughvisualization. Computer Architecture looksat the systems on which ultrascalesimulations run and for whichprogramming environments and algorithms



(described in the section on computationalmathematics) must provide software tools.Programming Models and ComponentTechnology for High-End Computingdiscusses new techniques for turningalgorithms into efficient, maintainableprograms. Access and Resource Sharinglooks at how to both make resourcesavailable to the entire research communityand promote collaboration. SoftwareEngineering and Management is aboutdisciplines and tools—many adapted fromthe industrial software world—to managethe complexity of code development. DataManagement and Analysis addresses thechallenges of managing and understanding

the increasingly staggering volumes of dataproduced by ultrascale simulations.Performance Engineering is the art ofachieving high performance, which has thepotential of becoming a science. SystemSoftware considers the basic tools thatsupport all other software.

The universality of these issues calls forfocused research campaigns. At the sametime, the inter-relatedness of these issues toone another and to the application andalgorithmic issues above them in thesimulation hierarchy points to the potentialsynergy of integrated simulation efforts.



The preceding chapters of this volume havepresented a science-based case for large-scale simulation. Chapter 2 presents an on-going successful initiative, SciDAC—acollection of 51 projects operating at a levelof approximately $57 million per year—as atemplate for a much-expanded investmentin this area. To fully realize the promises ofSciDAC, even at its present size, requiresimmediate new investment in computerfacilities. Overall, the computational scienceresearch enterprise could make effectiveuse of as much as an order of magnitudeincrease in annual investment at this time.Funding at that level would enable animmediate tenfold increase in computercycles available and a doubling or triplingof the researchers in all contributingdisciplines. A longer, 5-year view shouldallow for another tenfold increase incomputing power (at less than a tenfoldincrease in expense). Meanwhile, theemerging workforce should be primed nowfor expansion by attracting and developingnew talent.

Chapter 3 describes the anatomy of a state-of-the-art large-scale simulation. Asimulation such as this one goes beyondestablished practices by exploitingadvances from many areas ofcomputational science to marry rawcomputational power to algorithmicintelligence. It conveys themultidisciplinary spirit intended in theexpanded initiative. Chapter 4 lists somecompelling scientific opportunities that arejust over the horizon—new scientificadvances that are predicted to be accessiblewith an increase in computational power ofa hundred- to a thousand-fold. Chapter 5lists some advances in the enablingtechnologies of computational mathematics

6. RECOMMENDATIONS AND DISCUSSION

and computer science that will be requiredor that could, if developed for widespreaduse, reduce the computational powerrequired to achieve a given scientificsimulation objective. The companionvolume details these opportunities moreextensively. This concluding section revisitseach of the recommendations listed at theend of Chapter 1 and expands upon themwith additional clarifying discussion.

1. Major new investments incomputational science are needed in all ofthe mission areas of DOE’s Office ofScience, as well as those of many otheragencies, so that the United States may bethe first, or among the first, to capture thenew opportunities presented by thecontinuing advances in computing power.Such investments will extend theimportant scientific opportunities thathave been attained by a fusion ofsustained advances in scientific models,mathematical algorithms, computerarchitecture, and scientific softwareengineering.

The United States is an acknowledgedinternational leader in computationalscience, both in its national laboratories andin its research universities. However, theUnited States could lose its leadershipposition in a short time because theopportunities represented bycomputational science for economic,military, and other types of leadership arewell recognized; the underlying technologyand training are available withoutrestrictions; and the technology continuesto evolve rapidly. As Professor JackDongarra from the University of Tennesseerecently noted:

6. Recommendations and Discussion



The rising tide of change [resulting fromadvances in information technology] showsno respect for the established order. Thosewho are unwilling or unable to adapt inresponse to this profound movement notonly lose access to the opportunities that theinformation technology revolution iscreating, they risk being rendered obsoleteby smarter, more agile, or more daringcompetitors.

The four streams of development broughttogether in a “perfect fusion” to create thecurrent opportunities are of different ages,but their confluence is recent. Modernscientific modeling still owes as much toNewton’s Principia (1686) as to anythingsince, although new models are continuallyproposed and tested. Modern simulationand attention to floating point arithmetichave been pursued with intensity since thepioneering work of Von Neumann at theend of World War II. Computationalscientists have been riding the technologicalcurve subsequently identified as “Moore’sLaw” for nearly three decades now.However, scientific software engineering asa discipline focused on componentization,extensibility, reusability, and platformportability within the realm of high-performance computing is scarcely adecade old. Multidisciplinary teams thatare savvy with respect to all four aspects ofcomputational simulation, and that areactively updating their codes with optimalresults from each aspect, have evolved onlyrecently. Not every contemporary“community code” satisfies the standardenvisioned for this new kind ofcomputational science, but the UnitedStates is host to a number of the leadingefforts and should capitalize upon them.

2. Multidisciplinary teams, with carefullyselected leadership, should be assembledto provide the broad range of expertiseneeded to address the intellectualchallenges associated with translating

advances in science, mathematics, andcomputer science into simulations that cantake full advantage of advancedcomputers.

Part of the genius of the SciDAC initiative isthe cross-linking of accountabilitiesbetween applications groups and thegroups providing enabling technologies.This accountability implies, for instance,that physicists are concentrating on physics,not on developing parallel libraries for coremathematical operations; and it impliesthat mathematicians are turning theirattention to methods directly related toapplication-specific difficulties, movingbeyond the model problems that motivatedearlier generations. Another aspect ofSciDAC is the cross-linking of laboratoryand academic investigators. Professors andtheir graduate students can inexpensivelyexplore many promising research directionsin parallel; however, the codes theyproduce rarely have much shelf life. On theother hand, laboratory-based scientists caneffectively interact with many academicpartners, testing, absorbing, and seedingnumerous research ideas, and effectively“hardening” the best of the ideas into long-lived projects and tested, maintained codes.This complementarity should be built intofuture computational science initiatives.

Future initiatives can go farther thanSciDAC by adding self-contained, verticallyintegrated teams to the existing SciDACstructure of specialist groups. The SciDACIntegrated Software Infrastructure Centers(ISICs), for instance, should continue to besupported, and even enhanced, to developinfrastructure that supports diverse scienceportfolios. However, mathematicians andcomputer scientists capable ofunderstanding and harnessing the outputof these ISICs in specific scientific areasshould also be bona fide members of the



next generation of computational scienceteams recommended by this report, just asthey are now part of a few of the largerSciDAC applications teams.

3. Extensive investment in newcomputational facilities is stronglyrecommended, since simulation now cost-effectively complements experimentationin the pursuit of the answers to numerousscientific questions. New facilities shouldstrike a balance between capabilitycomputing for those “heroic simulations”that cannot be performed any other way,and capacity computing for “production”simulations that contribute to the steadystream of progress.

The topical breakout groups assessingopportunities in the sciences at the June2003 SCaLeS Workshop have universallypleaded for more computational resourcesdedicated to their use. These pleas arequantified in Volume 2 and highlighted inChapter 4 of this volume, together with(partial) lists of the scientific investigationsthat can be undertaken with current know-how and an immediate influx of computingresources.

It is fiscally impossible to meet all of theseneeds, nor does this report recommend thatavailable resources be spent entirely on thecomputational technology existing in agiven year. Rather, steady investmentsshould be made over evolving generationsof computer technology, since eachsucceeding generation packages morecapability per dollar and evolves towardgreater usability.

To the extent that the needs for computingresources go unmet, there will remain adriving incentive to get more science donewith fewer cycles. This is a healthyincentive for multidisciplinary solutions

that involve increasingly optimalalgorithms and software solutions.However, this incentive existsindependently of the desperate need forcomputing resources. U.S. investigators areshockingly short on cycles to capitalize ontheir current know-how. Some workshopparticipants expressed a need for very largefacilities in which to perform “heroic”simulations to provide high-fidelitydescriptions of the phenomena of interest,or to establish the validity of acomputational model. With currentfacilities providing peak processing rates inthe tens of teraflop/s, a hundred- tothousand-fold improvement in processingrate would imply a petaflop/s machine.Other participants noted the need toperform ensembles of simulations of moremodest sizes, to develop quantitativestatistics, to perform parameteridentifications, or to optimize designs. Acarefully balanced set of investments incapability and capacity computing will berequired to meet these needs.

Since the most cost-effective cycles for thecapacity requirements are not necessarilythe same as the most cost-effective cyclesfor the capability requirements, a mix ofprocurements is natural and optimal. In theshort term, we recommend pursuing, foruse by unclassified scientific applications,at least one machine capable of sustainingone petaflop/s, plus the capacity foranother sustained petaflop/s distributedover tens of installations nationwide.

In earlier, more innocent times,recommendations such as these could havebeen justified on the principle of “build itand they will come,” with the hope ofencouraging the development of a largercomputational science activity carryingcertain anticipated benefits. In the currentatmosphere, the audience for this hardware



already exists in large numbers; and theoperative principle is, instead, “build it orthey will go!” Two plenary speakers at theworkshop noted that a major U.S.-basedcomputational geophysicist has alreadytaken his leading research program toJapan, where he has been made a guest ofthe Earth Simulator facility; this facilityoffers him a computational capability thathe cannot find in the United States. Becauseof the commissioning of the Japanese EarthSimulator last spring, this problem is mostacute in earth science. However, theproblem will rapidly spread to otherdisciplines.

4. Investment in hardware facilitiesshould be accompanied by sustainedcollateral investment in softwareinfrastructure for them. The efficient useof expensive computational facilities andthe data they produce depends directlyupon multiple layers of system softwareand scientific software, which, togetherwith the hardware, are the engines ofscientific discovery across a broadportfolio of scientific applications.

The President’s Information TechnologyAdvisory Committee (PITAC) report of1999 focused on the tendency to under-invest in software in many sectors, and thisis unfortunately also true in computationalscience. Vendors have sometimes deliveredleading-edge computing platformsdelivered with immature or incompletesoftware bases; and computational scienceusers, as well as third-party vendors, havehad to rise to fill this gap. In addition,scientific applications groups have oftenfound it difficult to make effective use ofthese computing platforms from day one,and revising their software for the newmachine can take several months to a yearor more. Many industry-standard librariesof systems software and scientific software

originated and are maintained at nationallaboratories, a situation that is a naturaladaptation to the paucity of scientific focusin the commercial marketplace.

The SciDAC initiative has identified fourareas of scientific software technology forfocused investment—scientific simulationcodes, mathematical software, computingsystems software, and collaboratorysoftware. Support for development andmaintenance of portable, extensiblesoftware is yet another aspect of the geniusof the SciDAC initiative, as the rewardstructure for these vital tasks is otherwiseabsent in university-based research anddifficult to defend over the long term inlaboratory-based research. AlthoughSciDAC is an excellent start, the currentfunding level cannot support many types ofsoftware that computational science usersrequire.

It will be necessary and cost-effective, forthe foreseeable future, for the agencies thatsponsor computational science to makebalanced investments in several of thelayers of software that come between thescientific problems they face and thecomputer hardware they own.

5. Additional investments in hardwarefacilities and software infrastructureshould be accompanied by sustainedcollateral investments in algorithmresearch and theoretical development.Improvements in basic theory andalgorithms have contributed as much toincreases in computational simulationcapability as improvements in hardwareand software over the first six decades ofscientific computing.

A recurring theme among the reports fromthe topical breakout groups for scientificapplications and mathematical methods is



that increases in computational capabilitycomparable to those coming directly fromMoore’s Law have come from improvedmodels and algorithms. This point wasnoted over 10 years ago in the famous“Figure 4” of a booklet entitled GrandChallenges: High-Performance Computing andCommunications published by FCCSET anddistributed as a supplement of thePresident’s FY 1992 budget. Acontemporary reworking of this chartappears as Figure 13 of this report. It showsthat the same factor of 16 million inperformance improvement that would beaccumulated by Moore’s Law in a 36-yearspan is achieved by improvements innumerical analysis for solving theelectrostatic potential equation on a cubeover a similar period, stretching from thedemonstration that Gaussian eliminationwas feasible in the limited arithmetic ofdigital computers to the publication of theso-called “full multigrid” algorithm.

There are numerous reasons to believe thatadvances in applied and computationalmathematics will make it possible to modela range of spatial and temporal scales muchwider than those reachable by establishedpractice. This has already beendemonstrated with automated adaptivemeshing, although the latter still requirestoo much specialized expertise to haveachieved universal use. Furthermore,multiscale theory can be used tocommunicate the net effect of processesoccurring on unresolvable scales to modelsthat are discretized on resolvable scales.Multiphysics techniques may permit codesto “step over” stability-limiting time scalesthat are dynamically irrelevant at largerscales.

Even for models using fully resolved scales,research in algorithms promises totransform methods whose costs increase

rapidly with increasing grid resolution intomethods that increase linearly or log-linearly with complexity, with reliablebounds on the approximations that areneeded to achieve these simulation-liberating gains in performance. It isessential to keep computational sciencecodes well stoked with the latestalgorithmic technology available. In turn,computational science applications mustactively drive research in algorithms.

6. Computational scientists of all typesshould be proactively recruited withimproved reward structures andopportunities as early as possible in theeducational process so that the number oftrained computational scienceprofessionals is sufficient to meet presentand future demands.

The United States is exemplary inintegrating its computational sciencegraduate students into internships in thenational laboratory system, and it has hadthe foresight to establish a variety offellowship programs to make the relativelynew career of “computational scientist”visible outside departmental boundaries atuniversities. These internships andfellowship programs are estimated to beunder-funded, relative to the number ofwell-qualified students, by a factor of 3.Even if all presently qualified andinterested U.S. citizens who are doctoralcandidates could be supported—and evenif this pool were doubled by aggressiverecruitment among groups in the native-born population who are underrepresentedin science and engineering, and/or theaddition of foreign-born and domesticallytrained graduate students—the range ofnewly available scientific opportunitieswould only begin to be grazed in newcomputational science doctoraldissertations.



The need to dedicate attention to earlierstages of the academic pipeline is welldocumented elsewhere and is not thecentral subject of this focused report,although it is a related concern. It is time tothoroughly assess the role of computationalsimulation in the undergraduatecurriculum. Today’s students of science andengineering, whose careers will extend tothe middle of the twenty-first century, mustunderstand the foundations ofcomputational simulation, must be awareof the advantages and limitations of thisapproach to scientific inquiry, and mustknow how to use computational techniquesto help solve scientific and engineeringproblems. Few universities can make thisclaim of their current graduates. Academicprograms in computational simulation canbe attractors for undergraduates who areincreasingly computer savvy but presentlyperceive more exciting career opportunitiesin non-scientific applications of informationtechnology.

7. Sustained investments must be made innetwork infrastructure for access andresource sharing, as well as in the softwareneeded to support collaboration amongdistributed teams of scientists,recognizing that the best possiblecomputational science teams will bewidely separated geographically and thatresearchers will generally not beco-located with facilities and data.

The topical breakout group on access andresource sharing addressed the question ofwhat technology must be developed and/or deployed to make a petaflop/s-scalecomputer useful to users not located in thesame building. Resource sharing constitutesa multilayered research endeavor that ispresently accorded approximately one-thirdof the resources available to SciDAC, whichare distributed over ten projects. Needs

addressed include network infrastructure,grid/networking middleware, andapplication-specific remote userenvironments.

The combination of central facilities and awidely distributed scientific workforcerequires, at a minimum, remote access tocomputation and data, as well as theintegration of data distributed over severalsites.

Network infrastructure research is alsorecommended for its promises of leading-edge capabilities that will revolutionizecomputational science, includingenvironments for coupling simulation andexperiment; environments formultidisciplinary simulation; orchestrationof multidisciplinary, multiscale, end-to-endscience process; and collaboration insupport of all these activities.

8. Federal investment in innovative, high-risk computer architectures that are wellsuited to scientific and engineeringsimulations is both appropriate andneeded to complement commercialresearch and development. Thecommercial computing marketplace is nolonger effectively driven by the needs ofcomputational science.

Scientific computing—which ushered in theinformation revolution over the past half-century by demanding digital products thateventually had successful commercial spin-offs—has now virtually been orphaned bythe information technology industry.Scientific computing occupies only a fewpercentage points of the total marketplacerevenue volume of information technology.This stark economic fact is a reminder ofone of the socially beneficial by-products ofcomputational science. It is also a wake-upcall regarding the necessity that computa-



tional science attend to its own needs vis-à-vis future computer hardwarerequirements. The most noteworthy neglectof the scientific computing industry is,arguably, the deteriorating ratio of memorybandwidth between processors and mainmemory to the computational speed of theprocessors. It does not matter how fast aprocessor is if it cannot get the data that itneeds from memory fast enough—a factrecognized very early by Seymour Cray,whom many regard as the father of thesupercomputer. Most commercialapplications do not demand high memorybandwidth; many scientific applications do.

There has been a reticence to accordcomputational science the sameconsideration given other types ofexperimental science when it comes tofederal assistance in the development offacilities. High-end physics facilities—suchas accelerators, colliders, lasers, andtelescopes—are, understandably, designed,constructed, and installed entirely attaxpayer expense. However, the design andfabrication of high-end computationalfacilities are expected to occur in the courseof profit-making enterprises; only thescience-specific purchase or lease andinstallation are covered at taxpayerexpense. To the extent that the marketplaceautomatically takes care of the needs of thecomputational science community, thisrepresents a wonderful savings for thetaxpayer. To the extent that it does not, itstunts the opportunity to accomplishimportant scientific objectives and to leadthe U.S. computer industry to furtherinnovation and greatness.

The need to dedicate attention to researchin advanced computer architecture isdocumented elsewhere in many places,including the concurrent interagency High-End Computing Revitalization Task Force

(HECRTF) study, and is not the centralconcern of this application-focused report.However, it is appropriate to reinforce thisrecommendation, which is as true of our eraas it was true of the era of the “Lax Report”(see the discussion immediately following).The SciDAC report called for theestablishment of experimental computingfacilities whose mission was to “providecritical experimental equipment forcomputer science and applied mathematicsresearchers as well as computationalscience pioneers to assess the potential ofnew computer systems and architecturesfor scientific computing.” Coupling thesefacilities with university-based researchprojects could be a very effective means tofill the architecture research gap.

CLOSING REMARKS

The contributors to this report would like tonote that they claim little original insightfor the eight recommendations presented.In fact, our last six recommendations maybe found, slightly regrouped, in the fourprescient recommendations of the LaxReport (Large-Scale Computing in Science andEngineering, National Science Board, 1982),which was written over two decades ago, invery different times, before the introductionof pervasive personal computing and wellbefore the emergence of the World WideWeb!

Our first two recommendations reflect thebeginnings of a “phase transition” in theway certain aspects of computationalscience are performed, from soloinvestigators to multidisciplinary groups.Most members of the Lax panel—Nobelistsand members of the National Academiesincluded—wrote their own codes, perhapsin their entirety, and were capable ofunderstanding nearly every step of theircompilation, execution, and interpretation.This is no longer possible: the scientific



applications, computer hardware, andcomputing systems software are now eachso complex that combining all three intotoday’s engines of scientific discoveryrequires the talents of many. The phasetransition to multidisciplinary groups,though not first crystallized by the SciDACreport (Scientific Discovery through AdvancedComputing, U.S. Department of Energy,March 2000), was recognized andeloquently and persuasively articulatedthere.

The original aspect of the current report isnot, therefore, its recommendations. It is,rather, its elaboration in the companionvolume of what the nation’s leadingcomputational scientists see just over thecomputational horizon, and thedescriptions by cross-cutting enablingtechnologists from mathematics andcomputer science of how to reach these newscientific vistas.

It is encouraging that the generalrecommendations of generations of reportssuch as this one, which indicate a desireddirection along a path of progress, arerelatively constant. They certainly adapt tolocal changes in the scientific terrain, butthey do not swing wildly in response to thesiren calls of academic fashion, the

international politics of informationtechnology, or the economics of sciencefunding. Meanwhile, the supportingarguments for the recommendations, whichindicate the distance traveled along thispath, have changed dramatically. As thecompanion volume illustrates, the twodecades since the Lax Report have beenextremely exciting and fruitful forcomputational science. The next 10 to20 years will see computational sciencefirmly embedded in the fabric of science—the most profound development in thescientific method in over three centuries!

The constancy of direction providesreassurance that the community is wellguided, while the dramatic advances incomputational science indicate that it iswell paced and making solid progress. TheLax Report measured needs in megaflop/s,the current report in teraflop/s—a unit onemillion times larger. Neither report,however, neglects to emphasize that it isnot the instruction cycles per second thatmeasure the quality of the computationaleffort, but the product of this rating timesthe scientific results per instruction cycle.The latter requires better computationalmodels, better simulation algorithms, andbetter software implementation.



This report is built on the efforts of acommunity of computational scientists,DOE program administrators, and technicaland logistical professionals situatedthroughout the country. Bringing togetherso many talented individuals from so manydiverse endeavors was stimulating. Doingso and reaching closure in less than fourmonths, mainly summer months, wasnothing short of exhilarating.

The participants were reminded oftenduring the planning and execution of thisreport that the process of creating it was asimportant as the product itself. If resourcesfor a multidisciplinary initiative in scientificsimulation were to be presented to thescientists, mathematicians, and computerscientists required for its successfulexecution—as individuals distributedacross universities and government andindustrial laboratories, lacking familiaritywith each other’s work across disciplinaryand sectorial boundaries—the return to thetaxpayer certainly would be positive, yetshort of its full potential. Investments inscience are traditionally made this way, andin this way, sound scientific building blocksare accumulated and tested. But as a pile ofblocks does not constitute a house, theoryand models from science, methods andalgorithms from mathematics, and softwareand hardware systems from computerscience do not constitute a scientificsimulation. The scientists, mathematicians,and computer scientists who met at theirown initiative to build this report are nowready to build greater things. Whileadvancing science through simulation, thegreatest thing they will build is the nextgeneration of computational scientists,whose training in multidisciplinaryenvironments will enable them to see much

POSTSCRIPT AND ACKNOWLEDGMENTS

further and live up to the expectations ofGalileo, invoked in the quotation on thetitle page of this volume.

The three plenary speakers for theworkshop during which the input for thisreport primarily was gathered—RaymondOrbach, Peter Lax, and John Grosh—perfectly set the perspective of legacy,opportunity, and practical challenges forlarge-scale simulation. In many ways, thechallenges we face come from the samedirections as those reported by the NationalScience Board panel chaired by Peter Lax in1982. However, progress along the path inthe past two decades is astounding—literally factors of millions in computercapabilities (storage capacity, processorspeed) and further factors of millions fromworking smarter (better models, optimaladaptive algorithms). Without themeasurable progress of which the speakersreminded us, the aggressive extrapolationsof this report might be less credible. Witheach order-of-magnitude increase incapabilities comes increased ability tounderstand or control aspects of theuniverse and to mitigate or improve our lotin it.

The DOE staff members who facilitated thisreport often put in the same unusual hoursas the scientific participants in breakfastmeetings, late-evening discussions, andlong teleconferences. They did not steer theoutcome, but they shared their knowledgeof how to get things accomplished inprojects that involve hundreds of people;and they were incredibly effective. Manyprovided assistance along the way, but DanHitchcock and Chuck Romine, in particular,labored with the scientific core committeefor four months.

Postscript and Acknowledgments



Other specialists have left their imprints onthis report. Devon Streit of DOEHeadquarters is a discussion leadernonpareil. Joanne Stover and Mark Baylessof Pacific Northwest National Laboratorybuilt and maintained web pages, oftenupdated daily, that anchored workshopcommunication among hundreds of people.Vanessa Jones of Oak Ridge NationalLaboratory’s Washington office providedcheerful hospitality to the core planners.Jody Shumpert of Oak Ridge Institute forScience and Education (ORISE) single-handedly did the work of a conference teamin preparing the workshop and a pre-workshop meeting in Washington, D.C. Thenecessity to shoehorn late-registeringparticipants into Arlington hotels duringhigh convention season, when the responseto the workshop outstripped planningexpectations, only enhanced her display ofunflappability. Rachel Smith and BruceWarford, also of ORISE, kept participantswell directed and well equipped in ten-wayparallel breakouts. Listed last, but of firstimportance for readers of this report,members of the Creative Media group atOak Ridge National Laboratory iteratedtirelessly with the editors in meeting

production deadlines and creating apresentation that outplays the content.

The editor-in-chief gratefully acknowledgesdepartmental chairs and administrators attwo different universities who freed hisschedule of standard obligations and thusdirectly subsidized this report. This projectconsumed his last weeks at Old DominionUniversity and his first weeks at ColumbiaUniversity and definitely took his mind offof the transition.

All who have labored to compress their (ortheir colleagues’) scientific labors of love intoa few paragraphs, or to express themselveswith minimal use of equations and jargon,will find ten additional readers for eachco-specialist colleague they therebyde-targeted. All involved in the preparationof this report understand that it is a snap-shot of opportunities that will soon beoutdated in their particulars, and that it mayprove either optimistic or pessimistic in itsextrapolations. None of these out-comesdetracts from the confidence of its con-tributors that it captures the current state ofscientific simulation at a historic inflectionpoint and is worth every sacrifice and risk.



Under the auspices of the Office of Scienceof the U.S. Department of Energy (DOE), agroup of 12 computational scientists fromacross the DOE laboratory complex met onApril 23, 2003, with 6 administrators fromDOE’s Washington, D.C., offices and DavidKeyes, chairman, to work out theorganization of a report to demonstrate thecase for increased investment incomputational simulation. They alsoplanned a workshop to be held in June togather input from the computationalscience community to be used to assemblethe report. The group was responding to acharge given three weeks earlier by Dr. WaltPolansky, the Chief Information Officer ofthe Office of Science (see Appendix 2). Theworking acronym “SCaLeS” was chosen forthis initiative.

From the April meeting, a slate of groupleaders for 27 breakout sessions (11 inapplications, 8 in mathematical methods,and 8 in computer science andinfrastructure) was assembled andrecruited. The short notice of the chosendates in June meant that two importantsub-communities, devoted to climateresearch and to Grid infrastructure, couldnot avoid conflicts with their own meetings.They agreed to conduct their breakoutsessions off-site at the same time as themain workshop meeting. Thus 25 of thebreakout groups would meet in greaterWashington, D.C., and 2 would meetelsewhere on the same dates andparticipate synchronously in the workshopinformation flows.

Acting in consultation with their scientificpeers, the group leaders drew up lists ofsuggested invitees to the workshop so as to

APPENDIX 1: A BRIEF CHRONOLOGY OF “A SCIENCE-BASED CASE FOR LARGE-SCALE SIMULATION”

Appendix 1: A Brief Chronology of “A Science-Based Case for Large-Scale Simulation”

guarantee a critical mass in each breakoutgroup, in the context of a fully openworkshop. Group leaders also typicallyrecruited one junior investigator “scribe”for each breakout session. This arrangementwas designed both to facilitate the assemblyof quick-look plenary reports frombreakouts at the meeting and to provide aunique educational experience to membersof the newest generation of DOE-orientedcomputational scientists.

Group leaders, invitees, and interestedmembers of the community were theninvited to register at the SCaLeS workshopwebsite, which was maintained as a serviceto the community by Pacific NorthwestNational Laboratory.

A core planning group of approximately adozen people, including Daniel Hitchcockand Charles Romine of the DOE Office ofAdvanced Scientific Computing Research,participated in weekly teleconferences toattend to logistics and to organize theindividual breakout sessions at theworkshop in a way that guaranteedmultidisciplinary viewpoints in eachdiscussion.

More than 280 computer scientists,computational mathematicians, andcomputational scientists attended the June24 and 25, 2003, workshop in Arlington,Virginia, while approximately 25 others metremotely. The workshop consisted of threeplenary talks and three sessions of topicalbreakout groups, followed by plenarysummary reports from each session, anddiscussion periods. Approximately 50% ofthe participant-contributors were from DOElaboratories, approximately 35% from



universities, and the remainder from otheragencies or from industry.

In the plenary talks, Raymond Orbach,director of the DOE Office of Science,charged the participants to document thescientific opportunities at the frontier ofsimulation. He also reviewed other Officeof Science investments in simulation,including a new program at the NationalEnergy Research Scientific ComputingCenter that allocates time on the 10 Tflop/sSeaborg machine for grand challenge runs.Peter D. Lax of the Courant Institute, editorof the 1982 report Large-Scale Computing inScience and Engineering, provided a personalhistorical perspective on the impact of thatearlier report and advised workshopparticipants that they have a similar callingin a new computational era. To further setthe stage for SCaLeS, John Grosh, co-chairof the High-End Computing RevitalizationTask Force (HECRTF), addressed theparticipants on the context of thisinteragency task force, which had met thepreceding week. (While the SCaLeSworkshop emphasized applications, theHECRTF workshop emphasized computerhardware and software.) Finally, DevonStreit of DOE’s Washington staff advisedthe participants on how to preserve their

focus in time-constrained multidisciplinarydiscussion groups so as to arrive at usefulresearch roadmaps.

From the brief breakout group plenaryreports at the SCaLeS workshop, a total of53 co-authors drafted 27 chaptersdocumenting scientific opportunitiesthrough simulation and how to surmountbarriers to realizing them throughcollaboration among scientists,mathematicians, and computer scientists.Written under an extremely tight deadline,these chapters were handed off to aneditorial team consisting of Thom Dunning,Phil Colella, William Gropp, and DavidKeyes. This team likewise worked under anextremely tight schedule, iterating withauthors to encourage and enforce a measureof consistency while allowing for variationsthat reflect the intrinsically different naturesof simulation throughout science.

Following a heroic job of technicalproduction by Creative Media at Oak RidgeNational Laboratory, A Science-Based Case forLarge-Scale Simulation is now in the hands ofthe public that is expected to be theultimate beneficiary not merely of aninteresting report but of the scientific gainstoward which it points.



APPENDIX 2: CHARGE FOR “A SCIENCE-BASED CASEFOR LARGE-SCALE SIMULATION”

Date: Wed, 2 Apr 2003 09:49:30 -0500From: “Polansky, Walt” <[email protected]>Subject: Computational sciences—Exploration of New Directions

Distribution:

I am pleased to report that David Keyes ([email protected]), of Old Dominion University,has agreed to lead a broad-based effort on behalf of the Office of Science to identify rich andfruitful directions for the computational sciences from the perspective of scientific andengineering applications. One early, expected outcome from this effort will be a strongscience case for an ultra-scale simulation capability for the Office of Science.

David will be organizing a workshop (early summer 2003, Washington, D.C. metro area) toformally launch this effort. I expect this workshop will address the major opportunities andchallenges facing computational sciences in areas of strategic importance to the Office ofScience. I envision a report from this workshop by July 30, 2003. Furthermore, thisworkshop should foster additional workshops, meetings, and discussions on specific topicsthat can be identified and analyzed over the course of the next year.

Please mention this undertaking to your colleagues. I am looking forward to their supportand their participation, along with researchers from academia and industry, in making thiseffort a success.

Thanks.

Walt P.

Appendix 2: Charge for “A Science-Based Case for Large-Scale Simulation”

Acronyms and Abbreviations


Acronyms and Abbreviations


ACRONYMS AND ABBREVIATIONS

ASCI Accelerated Strategic Computing Initiative

ASCR Office of Advanced Scientific Computing Research (DOE)

BER Office of Biology and Environmental Research (DOE)

BES Office of Basic Energy Sciences (DOE)

DOE U.S. Department of Energy

FES Office of Fusion Energy Sciences (DOE)

FCCSET Federal Coordinating Council for Science, Engineering, and Technology

GCM general circulation model

HECRTF High-End Computing Revitalization Task Force

HENP Office of High Energy and Nuclear Physics (DOE)

Hypre a solver library

ISIC Integrated Software Infrastructure Center

ITER International Thermonuclear Experimental Reactor

M3D a fusion simulation code

NIMROD a fusion simulation code

NNSA National Nuclear Security Agency (DOE)

NSF National Science Foundation

NUSEL National Underground Laboratory for Science and Engineering

NWCHEM a chemistry simulation code

ORISE Oak Ridge Institute for Science and Education

QCD quantum chromodynamics

PETSc Portable, Extensible Toolkit for Scientific Computing, a solver library

PITAC President’s Information Technology Advisory Committee

RHIC Relativistic Heavy Ion Collider

RIA Rare Isotope Accelerator

SC Office of Science (DOE)

SciDAC Scientific Discovery through Advanced Computing

TSI Terascale Supernova Initiative

“We can only see a short distance ahead,but we can see plenty there that needs to be done.”

Alan Turing (1912—1954)

Documents

A SCIENCE-BASED CASE FOR LARGE-SCALE SIMULATIONscience.energy.gov/.../archive/Scales_report_vol1.pdf · A Science-Based Case for Large-Scale Simulation vii ... Phillip Colella, Lawrence