27
February 25, 2008 The Emerging Front Range The Emerging Front Range HPC Collaboratory HPC Collaboratory Dr. Rich Loft: Dr. Rich Loft: [email protected] [email protected] Director, Technology Development Director, Technology Development Computational and Information Systems Laboratory Computational and Information Systems Laboratory National Center for Atmospheric Research National Center for Atmospheric Research

February 25, 2008 The Emerging Front Range HPC Collaboratory Dr. Rich Loft: [email protected] [email protected] Director, Technology Development Computational

Embed Size (px)

Citation preview

Page 1: February 25, 2008 The Emerging Front Range HPC Collaboratory Dr. Rich Loft: loft@ucar.edu loft@ucar.edu Director, Technology Development Computational

February 25, 2008

The Emerging Front The Emerging Front Range HPC CollaboratoryRange HPC Collaboratory

Dr. Rich Loft: Dr. Rich Loft: [email protected]@ucar.edu

Director, Technology DevelopmentDirector, Technology Development

Computational and Information Systems LaboratoryComputational and Information Systems Laboratory

National Center for Atmospheric Research National Center for Atmospheric Research

Page 2: February 25, 2008 The Emerging Front Range HPC Collaboratory Dr. Rich Loft: loft@ucar.edu loft@ucar.edu Director, Technology Development Computational

February 25, 2008

Front Range HPC CollaboratoryComponents…

Facilities

HPC Systems

Grid Technology

Applications/Workflows

Science Collaborations

Science Drivers

Networking

Page 3: February 25, 2008 The Emerging Front Range HPC Collaboratory Dr. Rich Loft: loft@ucar.edu loft@ucar.edu Director, Technology Development Computational

February 25, 2008

The National Center for Atmospheric Research

Climate Modeling Expertise •Nobel Prize winning IPCC AR-4•Community Modeling Approach•Scalable Models

Grid Expertise•TeraGrid Resource Provider•Earth System Grid•Grid-BGC (BioGeoChemistry)

New Facility Plans (2012)•>1 PFLOPS•8 MW•>100 PB archive

High Performance Computing•40 years of experience•25 TFLOPS peak computers•5 Pbyte archive

Page 4: February 25, 2008 The Emerging Front Range HPC Collaboratory Dr. Rich Loft: loft@ucar.edu loft@ucar.edu Director, Technology Development Computational

February 25, 2008

Climate Modeling…Climate Modeling…

40 km version credit: J. Hack, NCAR

Page 5: February 25, 2008 The Emerging Front Range HPC Collaboratory Dr. Rich Loft: loft@ucar.edu loft@ucar.edu Director, Technology Development Computational

February 25, 2008

Climate of the last Millennium

Credit: Caspar AmmanNCAR

Page 6: February 25, 2008 The Emerging Front Range HPC Collaboratory Dr. Rich Loft: loft@ucar.edu loft@ucar.edu Director, Technology Development Computational

February 25, 2008

This is a Critical Moment in This is a Critical Moment in Climate Change Research…Climate Change Research…

Reproduce historical trends

Investigate climate change

Run IPCC Scenarios

Investigate solutions

Assess impacts

Simulate adaptation strategies

Work with energy industry

Before IPCCBefore IPCC AR4 AfterAR4 After2007

Page 7: February 25, 2008 The Emerging Front Range HPC Collaboratory Dr. Rich Loft: loft@ucar.edu loft@ucar.edu Director, Technology Development Computational

February 25, 2008

Mitigation: What Mitigation: What we do nextwe do next can can change the amount of warming change the amount of warming that occurs.that occurs.

Likelihood of warmingLikelihood of warming

(Knutti et al. 2005)(Knutti et al. 2005)

Page 8: February 25, 2008 The Emerging Front Range HPC Collaboratory Dr. Rich Loft: loft@ucar.edu loft@ucar.edu Director, Technology Development Computational

February 25, 2008

L. Buja Dimensions SlideL. Buja Dimensions Slide

Page 9: February 25, 2008 The Emerging Front Range HPC Collaboratory Dr. Rich Loft: loft@ucar.edu loft@ucar.edu Director, Technology Development Computational

February 25, 2008

Why High Resolution?Why High Resolution?

Ocean component of CCSM (Collins et al, 2006)

Eddy-resolving POP (Maltrud & McClean,2005)

Page 10: February 25, 2008 The Emerging Front Range HPC Collaboratory Dr. Rich Loft: loft@ucar.edu loft@ucar.edu Director, Technology Development Computational

February 25, 2008

The Case for an HPC CollaboratoryThe Case for an HPC Collaboratoryin the Western Interiorin the Western Interior

An integrated approach to An integrated approach to climate change will involveclimate change will involve– Climate modeling with Climate modeling with

regional capabilitiesregional capabilities– Mitigation efforts via, for Mitigation efforts via, for

example, example, carbon carbon sequestrationsequestration

Most sequestration reservoir Most sequestration reservoir formations in the Western formations in the Western InteriorInterior

Unconventional oil and gas Unconventional oil and gas locations in Western Interiorlocations in Western Interior

Regional climate change Regional climate change effects on the Great Plains effects on the Great Plains biomassbiomass

Half of all EPSCOR states are Half of all EPSCOR states are in the Western Interiorin the Western Interior

Power CostsPower CostsDOE FutureGenProject in Texas

Page 11: February 25, 2008 The Emerging Front Range HPC Collaboratory Dr. Rich Loft: loft@ucar.edu loft@ucar.edu Director, Technology Development Computational

February 25, 2008

Experimental Program to Stimulate Experimental Program to Stimulate Competitive Research (EPSCoR) Competitive Research (EPSCoR)

Most of the Western Interior is EPSCoR

Page 12: February 25, 2008 The Emerging Front Range HPC Collaboratory Dr. Rich Loft: loft@ucar.edu loft@ucar.edu Director, Technology Development Computational

February 25, 2008

National Security Agency - The power consumption of today's advanced computing systems is rapidly becoming the limiting factor with respect to improved/increased computational ability." 

Page 13: February 25, 2008 The Emerging Front Range HPC Collaboratory Dr. Rich Loft: loft@ucar.edu loft@ucar.edu Director, Technology Development Computational

February 25, 2008

Moore’s Law = More Cores: Moore’s Law = More Cores: Quad Core “Barcelona” AMD Processor…Quad Core “Barcelona” AMD Processor…

Can 8, 16, 32 cores be far behind?

Page 14: February 25, 2008 The Emerging Front Range HPC Collaboratory Dr. Rich Loft: loft@ucar.edu loft@ucar.edu Director, Technology Development Computational

February 25, 2008

The Path to Petascale is Through The Path to Petascale is Through Massive Parallelism…Massive Parallelism…

Cray-2/8 1986 Blue Gene/L 2006130K CPU’s367 TFLOPS

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

3.9 GFLOPS

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

Page 15: February 25, 2008 The Emerging Front Range HPC Collaboratory Dr. Rich Loft: loft@ucar.edu loft@ucar.edu Director, Technology Development Computational

February 25, 2008

Page 16: February 25, 2008 The Emerging Front Range HPC Collaboratory Dr. Rich Loft: loft@ucar.edu loft@ucar.edu Director, Technology Development Computational

February 25, 2008

Average Electrical Cost by State(1990 - 2003)

Data From Energy Information Administrationhttp://www.eia.doe.gov/cneaf/electricity/epa/average_price_state.xls

0

2

4

6

8

10

12

14

IDWA ORWY KY WV NE MT UT COVA IN OK WI MO MN ND SC TN IA AR NC KS SD FL AL MD TX MS DE NV GA LA DC OH

ILNM MI AZ PA NJ AK RI MA CT ME VT CA NH NY HI

(cents per kWh)

Commercial

Industrial

Cheap Power in the Western Interior

Page 17: February 25, 2008 The Emerging Front Range HPC Collaboratory Dr. Rich Loft: loft@ucar.edu loft@ucar.edu Director, Technology Development Computational

February 25, 2008

Front-Range Collaboratory Front-Range Collaboratory Resources Resources

Current SystemsCurrent Systems GECO (CSM+NREL)GECO (CSM+NREL)

– 2144, 2.66 GHz Intel CPU Dell Cluster (22.8 TFLOPS)2144, 2.66 GHz Intel CPU Dell Cluster (22.8 TFLOPS) NCARNCAR

– 2048, 700 MHz PwrPC CPU Blue Gene/L (5.7 TFLOPS)2048, 700 MHz PwrPC CPU Blue Gene/L (5.7 TFLOPS)

Planned Systems (FY2009)Planned Systems (FY2009) NCAR

– ~30 TFLOPS system upgrade – 1.5 PB HPSS-based archive

CU– ~100 TFLOPS MRI Proposal to NSF

Farther OutNCAR Supercomputing Center (NSC)

Page 18: February 25, 2008 The Emerging Front Range HPC Collaboratory Dr. Rich Loft: loft@ucar.edu loft@ucar.edu Director, Technology Development Computational

February 25, 2008

Blue Gene/L @ NCAR:Blue Gene/L @ NCAR:A platform for scalability researchA platform for scalability research

“Frost”

Page 19: February 25, 2008 The Emerging Front Range HPC Collaboratory Dr. Rich Loft: loft@ucar.edu loft@ucar.edu Director, Technology Development Computational

February 25, 2008

NCAR is part of the TeraGrid - NCAR is part of the TeraGrid - NSF’s HPC GridNSF’s HPC Grid

SDSCTACC

UC/ANL

NCSA

ORNL

PU

IU

PSC

TeraGrid is a facility that integrates computational, information, and analysis resources at the San Diego Supercomputer Center, the Texas Advanced Computing Center, the University

of Chicago / Argonne National Laboratory, the National Center for Supercomputing Applications, Purdue University, Indiana University, Oak Ridge National Laboratory, the Pittsburgh Supercomputing Center, and the National Center for Atmospheric Research.

NCAR

Caltech

USC-ISI

UtahIowa

Cornell

Buffalo

UNC-RENCI

Wisc

Page 20: February 25, 2008 The Emerging Front Range HPC Collaboratory Dr. Rich Loft: loft@ucar.edu loft@ucar.edu Director, Technology Development Computational

February 25, 2008

HPC Infrastructure End State HPC Infrastructure End State

Integrated HPC Resource& Visualization Systems

E N T E R P R I S E6 0 0 0

-

E N T E R P R I S E6 0 0 0

-

WAN Network

WAN Network

HPCStorageCluster

10Gb/s

MSSArchive

E N T E R P R I S E6 0 0 0

-

TG Shared Filesystems

E N T E R P R I S E6 0 0 0

-

Sciencegateways

E N T E R P R I S E6 0 0 0

-

Data Preservation

Switch E N T E R P R I S E

6 0 0 0

-

E N T E R P R I S E6 0 0 0

-

E N T E R P R I S E6 0 0 0

-

Data SAN

Page 21: February 25, 2008 The Emerging Front Range HPC Collaboratory Dr. Rich Loft: loft@ucar.edu loft@ucar.edu Director, Technology Development Computational

February 25, 2008

Compute cabinets

I/O nodes

Interactive nodes

Network

Switch cabinets

Future HPC Systems Are Likely to be Heterogeneous

I/O Subsystem

Front End

Switch

Compute Accelerators

Accelerator cabinets

Page 22: February 25, 2008 The Emerging Front Range HPC Collaboratory Dr. Rich Loft: loft@ucar.edu loft@ucar.edu Director, Technology Development Computational

February 25, 2008

NCAR Summer Internships in NCAR Summer Internships in Parallel Computational Science Parallel Computational Science (SIParCS)(SIParCS)

Open to:Open to:– Upper division undergradsUpper division undergrads– Graduate studentsGraduate students

In Disciplines such as: In Disciplines such as: – CS, Software EngineeringCS, Software Engineering– Applied Math, StatisticsApplied Math, Statistics– ES ScienceES Science

Support:Support:– Travel, Housing, Per diemTravel, Housing, Per diem– 10 weeks salary10 weeks salary

2008 application deadline: 2008 application deadline: – Feb 1stFeb 1st

Number of interns selected:Number of interns selected:– 8-108-10

Members of traditionally Members of traditionally underrepresented groups encouraged to underrepresented groups encouraged to apply.apply.QuickTime™ and aTIFF (Uncompressed) decompressorare needed to see this picture.QuickTime™ and aTIFF (Uncompressed) decompressorare needed to see this picture.

QuickTime™ and aTIFF (Uncompressed) decompressorare needed to see this picture.

http://www.cisl.ucar.edu/siparcs

http://www.cisl.ucar.edu/siparcs

Page 23: February 25, 2008 The Emerging Front Range HPC Collaboratory Dr. Rich Loft: loft@ucar.edu loft@ucar.edu Director, Technology Development Computational

February 25, 2008

Thanks! Any Questions?

Page 24: February 25, 2008 The Emerging Front Range HPC Collaboratory Dr. Rich Loft: loft@ucar.edu loft@ucar.edu Director, Technology Development Computational

February 25, 2008

Front Range HPC Collaboratory: Front Range HPC Collaboratory: A geographical viewA geographical view

Ring TopologyMultiple Network Providers

•Level3•McCleod•Platte River Power Auth.•ICG

Upgradable to multiple 10 Gb/s waves

NCAR:~30 TflopsFY2009

CU:100 TflopsFY2009

GECO:11.4 Tflops

NSC:1 Pflops2011

Page 25: February 25, 2008 The Emerging Front Range HPC Collaboratory Dr. Rich Loft: loft@ucar.edu loft@ucar.edu Director, Technology Development Computational

February 25, 2008

Over 3000 registered users make it the Over 3000 registered users make it the most used Meteorological Research model;most used Meteorological Research model;

Operational use by the National Weather Operational use by the National Weather Service, US Navy, US Army, USAF, South Service, US Navy, US Army, USAF, South Korean Meteorological Service, Indian Korean Meteorological Service, Indian Meteorological Department;Meteorological Department;

Special forecasts are made by NCAR over Special forecasts are made by NCAR over the Antarctic in support of the Antarctic in support of international operations there.international operations there.

36 h Reflectivity

Forecast

4 km WRF Model

RadarReflectivity

10 June 2003 12Z

State-of-the-art Coupled Climate System ModelState-of-the-art Coupled Climate System Model Open Source Code and freely available data Open Source Code and freely available data Significant development collaborations with:Significant development collaborations with:

– 22 US universities22 US universities– US Department of Energy (DOE)US Department of Energy (DOE)

LANL, LLNL, LANL, ORNL, ANL, LBL, NERSCLANL, LLNL, LANL, ORNL, ANL, LBL, NERSC– NASANASA

Largest contributor to the 2007 IPCC AR4Largest contributor to the 2007 IPCC AR4 – 1.4° resolution1.4° resolution– 11,000 model years simulated >100 TBytes 11,000 model years simulated >100 TBytes – of output dataof output data

WRF: Weather and Research Forecast ModelWRF: Weather and Research Forecast Model

CCSM: Community Climate System ModelCCSM: Community Climate System Model

NCAR’s Community Model ApproachNCAR’s Community Model Approachto Enabling Collaborative Researchto Enabling Collaborative Research

Page 26: February 25, 2008 The Emerging Front Range HPC Collaboratory Dr. Rich Loft: loft@ucar.edu loft@ucar.edu Director, Technology Development Computational

February 25, 2008

IPCC

Page 27: February 25, 2008 The Emerging Front Range HPC Collaboratory Dr. Rich Loft: loft@ucar.edu loft@ucar.edu Director, Technology Development Computational

February 25, 2008

Atmospheric Resolution:Number of Northern Hemisphere Cyclones

T255

ERA

T159

T95

Credit: Jung et al. 2006