Upload
others
View
2
Download
0
Embed Size (px)
Citation preview
System Development & Technology EnablementIBM Printed Circuit Board Symposium – Nov 2011
Michael G. NealonDirector – IBM Enterprise Systems Hardware Development
2
New ERA of Computing – Client Driven “Smart” Computing
� New Compute Environment Paradigm
� Performance Alone will not drive buying decision � BUT is a defacto requirement
� Heterogeneous Environment
� Utility Compute Delivery Models � CLOUD Computing � Grid Compute Networks
� Virtualization
� Workload Consolidation
� Data Center Energy Efficiency
� RASU – Reliability, Availability, Serviceability & USEABILITY
� System Management
� Security
� Workload Accelerators / Specialty Engines
� Integrated Networking & Storage
TCOTotal
Cost ofOwnership
& BUSINESSValue
3
Data Center Cost Factors
SW Licenses6%SW Maintenance
5%DB+OS Licenses
3%
Hardware Costs8%
Implementation Costs29%
Internal Implementation Costs
8%
Application Dev & Support Costs
25%
Infrastructure Support Costs
10%
Administrative Costs6%
Rising Costs and Complexity Threaten Profits and Competitiveness
Spending(US$B)
Source: IDC, Virtualization 2.0: The Next Phase in Customer Adoption, Doc #204904, Dec 2006
$0
$50
$100
$150
$200
$250
$300
1996
1997
1998
1999
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
Installed Base(M Units)
Spending(US$B)
New server spendingServer mgmt and admin costs x4Power and cooling costs x8
05101520253035404550
Total Cost of Ownership Factors - Example
4
Data Center Cooling Cost and Directions
5
Server Market Trend and Directions
Internet Data CentersUltra Low CostHuge Scale Out
Server Redundancy
Micro-ServersUltra Low Power
Workload Optimized Integrated SolutionsBundled Hardware &
Software w/ Application
Management
Cloud ComputingIT Utility Service
Model
6
System Hardware Initiatives for “Smarter” Computing
� So How Does New Compute Environment Drive System Hardware?
• System Integration Density (Volumetric Area)� Compute Density driving bandwidth, memory & IO density� Integration of Storage & Networks� Flash Memory
• System Design for RAS� Complexity, Density & Power drive reliability and serviceability� New RAS models??? Fail in Place???
• Energy Efficiency at Data Center Level� Power Management� High Voltage DC Power Distribution� Water Cooling� Heat exchangers
• COST� Cost of hardware� Cost of RAS� Cost Enablement of TCO Value
77
� All data center power & cooling infrastructure included in compute/storage/network rack� No need for external power distribution or computer room air handling equipment.� All components correctly sized for max efficiency – very good 1.18 Power Utilization Efficiency� Integrated management for all compute, storage, network, power, & thermal resources.� Scales to 512K P7 cores (192 racks) – without any other hardware except optical fiber cables
P7 775 System “Data-Center-In-A-Rack” System Architecture
Integrated Storage – 384 2.5” HDD or SSD drives /drawer230 TBytes\drawer (w/600 GB 10K SAS disks), 154 GB/s BW/drawer, software-controlled RAID, up to 6/rack (replacing server drawers) (up to 1.38 PBytes / rack)
Integrated Cooling – Water pumps and heat exchangersAll heat transferred directly to building chilled water – no thermal load on room
Integrated Power Regulation, Control, & DistributionRuns off any building voltage supply world-wide (200-480 VAC or 370-575VDC). Full in-rack redundancy and automatic fail-over.
Up to 252 kW/rack max / 163 kW Typ.
Servers – 256 Power7 cores / drawer, 1-12 drawers / rackCompute: 8-core Power7 CPU chip, 3.7 GHz, 45nm technology, 32 MB L3 eDRAM/chip, 4-way SMT, 4 FPUs/core, Quad-Chip Module; >90 TF / rack
No accelerators: normal CPU instruction set, robust cache/memory hierarchyEasy programmability, predictable performance, mature compilers & libraries
Memory: 512 Mbytes/sec per QCM (0.5 Byte/FLOP), 12 Terabytes / rackExternal IO: 16 PCIe Gen2 x16 slots / drawer; SAS or external connectionsNetwork: Integrated Hub (HCA/NIC & Switch) per each QCM (8 / drawer), with 54-port switch, including total of 12 Tbits/s (1.1 TByte/s net BW) per Hub:
Host connection: 4 links, (96+96) GB/s aggregate (0.2 Byte/FLOP) On-card electrical links: 7 links to other hubs, (168+168) GB/s aggregateLocal-remote optical links: 24 links to near hubs, (120+120) GB/s aggregateDistant optical links: 16 links to far hubs (to 100M), (160+160) GB/s aggregatePCI-Express: 2-3 per hub, (16+16) to (20+20) GB/s aggregate
FrontFront
RearRear
88
P7 775 System Node Hardware – Node Front View
P7 QCM (8x)
Hub Module (8x)
D-Link Optical InterfaceConnects to other Super Nodes
360VDC Input Power Supplies
Water Connection
L-Link Optical InterfaceConnects 4 Nodes to form Super Node
MemoryDIMM’s (64x)
PCIe Interconnect
1m W x 1.8m D x 10cm H
MLC Module
Hub Assembly
PCIe Interconnect
D-Link Optical InterfaceConnects to other Super Nodes
Avago microPODTM All off-node communication optical
9
Hub Module – MCM with Optical I/Os
�This shows the Hub module with full complement of Optical I/Os. �Module in photo is partially assembled, to show construction – full module HW is symmetric
Heat Spreader for Optical DevicesCooling / Load Saddle for Optical Devices
Optical Transmitter/Receiver Devices 12 channel x 10 Gb/s 28 pairs per Hub - (2,800+2,800) Gb/s of optical I/O BW
Heat Spreader over HUB ASIC
Strain Relief for Optical RibbonsTotal of 672 Fiber I/Os per Hub, 10 Gb/s each
Hub ASIC (Under Heat Spreader)
10
What’s Next?
0.1
1
10
100
1000
1996 1998 2000 2002 2004 2006 2008 2010 2012
Date of publish
Re
lati
ve
Pe
rfo
rma
nc
eSocket-Level Perfomanceis growing at 50% CGR
•Socket Compute density will continue to grow•Si Device Perf. & Operating Freq Flatten
•Performance Obtained Through:• Increased Cores per Die• Increased Cache per Core• Higher Levels of Integrated Function• Need Innovation to “Feed the Beast”
•Differentiation will come through Integration•At Processor & System Level•Data Center Integration in a Chassis•BW is Critical to Success
1
10
100
2005 2006 2007 2008 2009 2010 2011 2012
Date of publish
Re
lati
ve
Ca
pa
cit
y
Cores per socket
0.1
1
10
100
2006 2007 2008 2009 2010 2011 2012
Date of publish
Re
lati
ve
Ca
pa
cit
y
Memory Capacity is growing at about 60% per year.
4-Socket GB/skt
2-Socket GB/skt
4-Socket GB/core
2-Socket GB/core
11
Innovation that Matters – Future Technology Directions
• Improved Materials & Process Controls• Low K Materials• Moisture Resilient • Built in Quality
• Stacked Memory (TSV Enabled)• 4-8X Density Improvement• RAS Enhancements to Enable
• Solid State Memory Exploitation• Reduced Cost• PCI Attached• Disk Drives• DIMMS
• Optics / Silicon Photonics• Reduced Cost Tx/Rx • Integrated Silicon Photonics• Advanced Packaging
• Integrated Modules• PCB Waveguides
TSV Si carrier Optochips assembled
6.4mm x 10.4mm
2x12 PD
array
2x12 RX IC
2x12 LDD IC
2x12 VCSEL array
RXTXLDD RX
Si CarrierVCSEL
Lens Arrays
PD
Organic Carrier
PCB
Polymer Waveguides
or Flex
To optical connector
TSV Si carrier Optochips assembled
6.4mm x 10.4mm
2x12 PD
array
2x12 RX IC
2x12 LDD IC
2x12 VCSEL array
RXTXLDD RX
Si CarrierVCSEL
Lens Arrays
PD
Organic Carrier
PCB
Polymer Waveguides
or Flex
To optical connector
Optical PCB Polymer waveguides
12
System Hardware Initiatives for “Smarter” Computing
Summary
� IT Compute Environment is Changing� New Deployment Models� Holistic Client Perspective – E2E Cost � Opens New Ways to Integrate Solutions
� Integration Wins!!!� Integration at all levels drives IT density / value� HW/SW Integration provides Workload & Management Optimized Solutions
� Technology Development which provides VALUE is required� Need Integrated Solution Based Technology
� Drives Need for Collaboration across Technology Development� Cost & Reliability are critical
Thank You