Internet-Scale Datacenter Economics: Costs & Opportunities

Internet-Scale Datacenter Economics: Costs & Opportunities

High Performance Transaction Systems 2011

James Hamilton, 2011/10/24

VP & Distinguished Engineer, Amazon Web Services

email: [email protected]

web: mvdirona.com/jrh/work

blog: perspectives.mvdirona.com

Agenda

• Quickening Pace Infrastructure Innovation – Influence of Cloud computing

• Power Distribution

• Cooling & Building Designs

• Networking & Server Innovation

2 2011/10/24 http://perspectives.mvdirona.com

Talk does not necessarily represent positions of current or past employers

Pace of Innovation • Datacenter pace of innovation increasing

– More innovation in last 5 years than previous 15

– Driven by cloud services & extraordinary-scale internet applications like search

– Cost of infrastructure dominates service cost

– Not just a cost center

• High focus on infrastructure innovation – Driving down cost

– Increasing aggregate reliability

– Reducing resource consumption footprint

2011/10/24 3 http://perspectives.mvdirona.com

Perspective on Scaling

Each day Amazon Web Services adds enough new capacity to support all of Amazon.com’s global infrastructure through the company’s

first 5 years, when it was a $2.76B annual revenue enterprise

2011/10/24 http://perspectives.mvdirona.com 4

Where Does the Money Go? • Assumptions:

– Facility: ~$88M for 8MW critical power

– Servers: 46,000 @ $1.45k each

– Commercial Power: ~$0.07/kWhr

– Power Usage Effectiveness: 1.45

5 http://perspectives.mvdirona.com

• Observations: • 31% costs functionally related to power (trending up while server costs down) • Networking high at 8% of overall costs & 19% of total server cost (often more)

2011/10/27

From: http://perspectives.mvdirona.com/2010/09/18/OverallDataCenterCosts.aspx

57%

8%

18%

13%

4% Servers

NetworkingEquipment

Power Distribution& Cooling

Power

Other Infrastructure

Monthly Costs

3yr server & 10 yr infrastructure amortization

Agenda






Power Distribution


13.2kv

11

5kv

13

.2kv

13.2kv 480V

20

8V

0.3% loss 99.7% efficient

2% loss 98% efficient

2% loss 98% efficient

6% loss 94% efficient, ~97% available

~1% loss in switch gear & conductors

UPS: Rotary or Battery

Sub-station Transformers Transformers

High Voltage Utility Distribution

IT Load (servers, storage, Net, …)

Generators

~11% lost in distribution .997*.94*.98*.98*.99 = 89%

UPS & Gen often on 480V

Note: Two more levels of power conversion at server

Removing Power Conversions

• Remove final conversion prior to server

– 480VAC line-to-neutral yields 277VAC

– 400VAC line-to-neutral yields 230VAC

– Both good but later supports standard PSUs

• Another option is HVDC distribution

– 400DC an interesting option

– Improved efficiency but higher capital cost


Power Distribution Efficiency Summary

• 2 more power conversions at servers 5. Power Supply: often under 80% at typical load

6. On board voltage regulators (VRMs or VRDs)

• Rules to minimize power distribution losses: – Oversell power (more load than provisioned power)

– Avoid conversions (fewer & better)

– Increase efficiency of conversions

– High voltage as close to load as possible

– Size voltage regulators to load & use efficient parts

– High voltage direct current a small potential gain


Agenda






Mechanical Systems


Computer Room Air Handler

Cooling Tower

CWS Pump

Heat Exchanger

(Water-Side Economizer)

A/C Condenser

Primary Pump

A/C Evaporator

leakage

cold

Ho

t

Diluted Hot/Cold Mix

Co

ld

fans

Air Impeller

Server fans 6 to 9W each

Overall Mechanical Losses

~22%

A/C Compressor

Blo

w d

ow

n &

Evapo

rative Loss at

8M

W facility: ~2

00

,00

0 gal/d

ay

20% of total power

Hot Aisle Containment

2011/10/24 12

Facebook Open Compute

Intel

http://perspectives.mvdirona.com

Intel

WriteLine

ASHRAE 2008 Recommendations


ASHRAE 2008 Allowable Class 1

59F-90F

ASHRAE 2008 Recommended Class 1

64-81F

ASHRAE 2011 Recommendations


59F-90F

64-81F


ASHRAE 2011 Recommended Class 1 81F

89F-95F

Network Equipment Building Standard



59F-90F

ASHRAE 2008 Recommended Class 1

64-81F


ASHRAE 2011 Recommended Class 1 81F

89F-95F

NEBS Telco Standard (~1970)

Most Datacenters Still Run Cold


59F-90F

64-81F

Most datacenters run down in this range

81F

89F-95F

ASHRAE 2008 Recommended Class 1 ~77F

Hard Drives: 7W- 25W

Temp Spec: 50C-60C

Processors/Chipset: 40W - 200W

Temp Spec: 60C-70C I/O: 5W - 25W

Temp Spec: 50C-60C

Air Cooling • Component temps specs higher than historically

hottest place on earth – Al Aziziyah, Libya: 136F/58C (1922)

• Just a mechanical engineering problem – More air or better mechanical designs

• Tradeoff: semi-conductor leakage & power to move more air vs cooling savings

• Currently available equipment temp limits: – 40C/104F: CloudRack C2 & most net gear

– 35C/95F: Most of the server industry


Memory: 3W - 20W

Temp Spec: 85C-105C

Thanks to Ty Schmitt& Giovanni Coglitore

Innovative Shell Designs


• Evaporative cooling only

– High pressure misting on right

– Damp media design below

• Full building ductless cooling

Facebook Prineville above & below

EcoCooling

Modular and Pre-fab DC Designs

• Fast & economic deployments

• Sub-1.2 PUE designs

• Air-side economized – In some cases no mechanical cooling

• ISO standard shipping containers offered by Dell, HP, SGI, IBM, …


Microsoft ITPAC

Amazon Perdix

Cooling Looking Forward

• River water or lake water cooling – Google Belgium & Finland, Deepgreen Switzerland

– Not new: Toronto metro area cooling

• Water direct to the rack – IBM iDataPlex

• Water direct to components – Done before 80’s IBM S/370 308x & 3090 series

– Again when heat densities climb back to that level

• Direct on component spray cooling


Agenda






Sea Change in Networking • Current networks over-subscribed

– Forces workload placement restrictions

– Goal: all points in datacenter equidistant

• Mainframe model goes commodity

– Competition at each layer over vertical integ.

• Get onto networking on Moore’s Law path

– ASIC port count growth at near constant cost

– Competition: Broadcom, Marvell, Fulcrum,…


Simple Packet Forwarding Hardware



Simple Packet Forwarding Hardware Simple Packet

Forwarding Hardware

Network Operating System (Onix)

Software Defined Networking

Openflow

Openflow

Openflow

Openflow

Openflow

Metering Routing QOS Monitoring

2011/10/24 23 http://perspectives.mvdirona.com

Nick McKeown

• Distributed control plane with central control

• Research examples: VL2, PortLand, & others

• Onix/OpenFlow gaining industry support & traction quickly

Server Innovation • Removing all unnecessary cost & power

– Omit lid, Depop board, efficient components, 12V-only PSUs

• Form factor: fractional RU & multi-server modules

• Shared power supplies – N supplies for M servers

– Run supplies at most efficient load

• Shared large back-of-rack fans • Cell phone technology predicts future server generations

• Super high-density storage platforms – Increasing server to disk ratio for cold storage

• Soon: ARM architecture, low power servers, multiple servers on board, proprietary net fabric


Documents

Internet-Scale Datacenter Economics: Costs & Opportunities