Upload
dangnhi
View
221
Download
0
Embed Size (px)
Citation preview
Blue Waters and CloudsIntense Computing at the Petascale and Beyond
William Kramer, Thom Dunning, Marc Snir,
William Gropp, Wen-mei Hwu
Bernie Acs.Cristina Beldica, Brett Bode, Robert Fiedler,
Scott Lathrop, Mike ShowermanNational Center for Supercomputing Applications, Department of Chemistry, Department of Computer
Science, and Department of Electrical & Computer Engineering
Molecular Science Weather & Climate Forecasting
Earth ScienceAstronomy Health
Sustained Petascale computing will enable advances in abroad range of science and engineering disciplines:
Astrophysics
Life Science Materials
2
Blue Waters Project Components
Blue Waters Base System – Processors, Memory, Interconnect, On-line Storage, System Software, Programming Environment
Value added Software – Collaborations
Value added hardware and software
Petascale Application Collaboration Team Support
Petascale Applications (Computing Resource Allocations)
Outstanding User and Production SupportWAN connections, Consulting, System Management, Security,
Operations, …
Petascale Computing Facility
PetascaleEducation,
Industry and
Outreach
Great Lakes Consortium
for Petascale
Computing
3
NSF Petascale Computing Resource
Allocation (PRAC) AwardeesPIs Field Institutions
Schulten Bio-molecular Dynamics Illinois
Sugar Quantum
Chromodynamics
UC-Santa Barbara
O’Shea Early galaxy formation MSU
Nagamine Cosmology UNLV
Bartlett Parallel language,
Chemistry
U. FL
Bisset, Brown, Roberts Social networks,
Contagion
VA Tech, CMU, Research
Triangle Inst.
Yeung Turbulent flows GA Tech.
Zhang Materials science Wm. & Mary
Wilhelmson Tornadoes Illinois
NSF Petascale Computing Resource
Allocation (PRAC) Awardees(Cont’d)
PIs Field Institutions
Jordan Geophysics U. So. CA
Lamm Chemistry IA St. U.
Woodward Stellar hydrodynamics U. of MN
Campanelli General relativity,
compact binaries
Rochester Inst. Tech.
Stan, Kirtman, Large,
Randall
Climate COLA (MD), U. Miami,
UCAR, CO St. U.
Savrasov, Haule Materials science UC-Davis, Rutgers
Schnetter Gamma-ray bursts LSU
Tagkopoulos Evolution Princeton
Wang Geophysics U. of WY
Testing Hypotheses About Climate Prediction
• Hypotheses
• The large errors in current-generation climate models are associated with fundamentally flawed assumptions in the parameterizations of cloud processes and ocean eddy mixing processes
• Climate predictability, a synthetic quality entirely associated with a given model, increases with increasing model resolution by virtue of the changing representation of atmospheric and oceanic noise
• Target Problems
• Annual Cycle Experiment using the Co. St. U. Global Cloud-Resolving Atmospheric General Circulation Model
• Test if annual cycle of quantities such as the precipitation and surface temperatures are more accurately reproduced when both cloud processes and the ocean meso-scale are resolved and coupled
Stan, Kirtman, Large, Randall
From Chip to Entire Integrated System
8
Chip
Quad Chip MCM
Rack/Building Block
Blue Waters System
PCF
On-line Storage
Near-line Storage
Color indicates relative
amount of public information
multiple MCMs
Blue Waters Computing System
9
System Attribute Ranger Blue Waters
Vendor Sun IBMProcessor AMD Barcelona IBM Power7Peak Performance (PF) 0.579 >10
Sustained Performance (PF) <0.05 >1Number of Cores/Chip 4 8Number of Processor Cores 62,976 >300,000Amount of Memory (TB) 123 >1Interconnect Bisection BW (TB/s) ~4Amount of Disk Storage (PB) 1.73 18I/O Aggregate BW (TB/s) ? 1.5Amount of Archival Storage (PB) 2.5 (20) >500External Bandwidth (Gbps) 10 100-400
17>20
2~3.5
>8>>10
>10
>200>10
10
POWER7 Processor Chip
• 567mm2 Technology: 45nm lithography, Cu, SOI, eDRAM
• 1.2B transistors
• Equivalent function of 2.7B due to eDRAMefficiency
• Eight processor cores
• 12 execution units per core
• 4 Way SMT per core
• 32 Threads per chip
• 256KB L2 per core
• 32MB on chip eDRAM shared L3
• Dual DDR3 Memory Controllers
• 100GB/s Memory bandwidth per chip sustained
• Scalability up to 32 Sockets
• 360GB/s SMP bandwidth/chip
• 20,000 coherent operations in flight
• Advanced pre-fetching for Data and Instruction
• Binary Compatibility with POWER6 * Statements regarding SMP servers do not imply that IBM will introduce a system with this capability.
Feeds and Speeds per QCM
1 TF/s QCM
• 32 cores
• 8 Flop/cycle per core
• 4 threads per core max
• 3.5 – 4 GHz
• 32 MB L3
• 512 GB/s memory BW
(0.5 Byte/flop)
• 800 W (0.8 W/flop)
1.1 TB/s Hub Chip
• 192 GB/s Host
Connection
• 336 GB/s to 7 other local
nodes
• 240 GB/s to local-remote
nodes
• 320 GB/s to remote
nodes
• 40 GB/s to general
purpose I/O
11
Caches• Low latency L1 (32KB) and L2 (256KB) dedicated caches per core
• ~45x lower latency than memory
• 32MB shared L3 cache
• ~3x lower latency than memory
• Automatically migrates per-core private working set footprints (up to 4MB) to fast
local region per core at ~15x lower latency than memory
• Automatically clones shared data to multiple per core private regions
• Enables subset of cores to utilize entire L3 when remaining cores are not using it
12
Cache Level Capacity Type Policy Comment
L1 Data 32 KB Fast SRAM Store-thru Local thread storage update
Private L2 256KB Fast SRAM Store-In Coherency maintained throughout system
Fast L3 “Private”
Up to 4 MB eDRAM Partial Victim Reduced latency & power consumption
Shared L3 32MB eDRAM Adaptive Coherency maintained throughout system
13
Two Level Interconnect SuperNode 1
No
de
1
No
de
2
No
de
3
No
de
4
No
de
1
No
de
2
No
de
3
No
de
4
Node 1
Node 2
Node 3
Node 4
Node 1
Node 2
Node 3
Node 4
1st level: L-Links, 24 or 6 GB/s
Connect 4 drawers together to
form a SuperNode
Copper or Optical Cable
2nd level: D-Links, 10 GB/s
Optical Cable
Connects SuperNodes to
all other SuperNodes
Up to 512 SuperNodes
fully connected
SuperNode 2
SuperNode 5
SuperNode 7
Imaginations unbound
High-level Schedule 2006-2010• March 4, 2010 Substantial completion
• 88,000 GSF over two stories—45’ tall
• 30,000+ GSF of raised floor
• LEED Gold/Platinum + PUE ~1.02 to 1.20
projected
• Free cooling (On site cooling towers)
used 70% of the year
• Higher operating temperature in the
computer room
• Initially capable of 24 MW of power
• Substantial security: biometrics, cable beam
barricade
• 300 gigabit external connectivity
• Five acre site allows room for facility
expansion
What does this have to do with Clouds
• Many definitions of what ―cloud computing‖ means
• I like the explanations in Above the Clouds: A Berkeley View of
Cloud Computing for a start
• What a cloud means depends on the viewer’s role
• Cloud providers range from bare metal to very specific software
services
• Cloud uses range from ―turn key‖ applications to direct application
development
• Motivations for cloud computing
• Much of the ―buzz‖ is business related
• Many are related to resource concentration
• Around data, facility benefits, expertise, mission
• Optimize utilization and cost
• Reduce risk
• Low bar to access for many
15
HPC and Clouds
• HPC and ―distributed‖ computing have a very long and mutually beneficial relationship
• Remote use of large resources
• Cluster computing
• Parallel computing
• Distributed computing
• Internet
• Web
• Grid
• Clouds
• What is different today that could make clouds dominate
• The wiring of the world
• Software as a Service
• Standard interfaces
• Shrinking hardware options
• Consolidation
• Peoples laptops and desktops note enough to do their analysis
16
So are HPC systems cloud resources?
• Yes – of course HPC can be parts of clouds
Large concentration of modern physical resources co-located for efficiency
Consumers are geographically separated
Time and component multiplexing
Users expect immediate feedback
Mostly for independent, unsynchronized tasks
Utility computing provider want to capture and retain user base
? Users expect low bar to access/Little HW specifics and little optimization
? Relies on under utilization to succeed
• Yes - HPC systems and applications can and will be cloud services
• Portals and data sharing
• Expansion of computing and storage resources
• Ability to solve societal scale problems
• Needed to train next generation of computational and analytical experts
17
Cloud Challenges
• Parallelism
• Can clouds thrive when parallelism dominates?
• Memory wall implications
• For science and academia – is there enough bandwidth?
• Are simple tools sufficient for complex analysis?
• Data movement costs
• Persistent data storage costs
• Cyber security and privacy
• Simple rule of outsourcing – don’t outsource your core business advantages
• Does this still apply?
18
Blue Wave:
K-12 Education Next Generation ICLCS/WebMO
A Cloud to Grid Conduit model for Next Generation Science Gateway
NCSA Enterprise Cloud & Virtual Machine Services
ICLCS/WebMO Cloud Science Gateway
Passive LB Node
Active LB Node
CentralizeRelational Database
Shared Network File System
WorkerNode
WorkerNode Worker
NodeWorkerQueue
GRIDQueue Resource
Node
ResouceNodeResource
Node
Internet Users
Conduit
Dedicated
Dynamically
Scalable
Proxy
• Cloud to Grid Conduit • Blue Wave (HPC for K-12 Education)
• ICLCS WebMO as an example
• Cloud to Cloud Conduit• Model for Eucalyptus Cluster integration (UIUC CS)
• Model for Hadoop integration (HP/Intel/Yahoo UIUC/CS)
• Model for Site to Site integration (Intra-Institutional)
• Mixed Services Integrations• Persistent Virtual Instances and Dynamic Instances
• Enable Experimentation into using Amazon & Azure for instant access to large-scale limited-time usage.• Account for development and experimentation
• Investigate Virtual Network Interconnection
NCSA Enterprise Cloud Conduits
Summary
• Clouds and HPC are both needed
• HPC and clouds are mutually beneficial
• Both HPC and clouds have to deal with the largest scale
• Understanding complex, highly interconnected systems that evolve dyanmically
• A challenge is to have good understanding of what a cloud means and when
it is the best value solutions
• Performance, Effectiveness, Resiliency, Consistency, Usability
• A challenge is data
• Analytics is key to transformative science, engineering and business at any scale
• Why should the large scale community be pushing "Exaflops" rather than
"Yottabytes― in order to improve science productivity and quality?
22
Acknowledgements
This research is part of the Blue Waters sustained-petascale computing
project, which is supported by the National Science Foundation
(award number OCI 07-25070) and the state of Illinois. Blue Waters is
a joint effort of the University of Illinois at Urbana-Champaign, its
National Center for Supercomputing Applications, IBM, and the Great
Lakes Consortium for Petascale Computation.
The work described is only achievable through the efforts of the Blue
Waters Project.
23
Questions?
24
Dr. William Kramer
NCSA/University of Illinois
Blue Waters Deputy Director
[email protected]/ - http://www.ncsa.uiuc.edu/BlueWaters
(217) 333-6260