Data Center PlanningData Center Planning

SpaceSpace PowerPower CoolingCooling Data Connections – CapacityData Connections – Capacity

Redundancy}} Density

Typical Space UsageTypical Space Usage

60%Support

40%Equipment

CYH A100 Space Usage

Total = 5,402 Sq. Ft

Servers & Storage 21%

Command/Console 11%Printers 10%

Networking 4%

Aisles 28%

Parts & Supplies 8%Power Distribution 6%

Growth 5%

HVAC 5%

Office 5%

Columns -3%Doors/Ramps -1%

Non-IT Space 58%

IT Space 42%

Our Space Usage (A100)Our Space Usage (A100)

How can we make more space?How can we make more space?

Curtail current usageCurtail current usage Shrink bulky systems, for example:Shrink bulky systems, for example:

Converting Unix timesharing machines to blades makes room for 40 Converting Unix timesharing machines to blades makes room for 40 new 2U serversnew 2U servers

““SAN-itizing” storage makes room for 75 new 2U serversSAN-itizing” storage makes room for 75 new 2U servers Relocate non-critical and development equipment to A82 or B7Relocate non-critical and development equipment to A82 or B7 Relocate UPS, HVAC equipmentRelocate UPS, HVAC equipment

Create more spaceCreate more space Add more racksAdd more racks Utilize idle, underused space in A100 (ACIS area, tape library)Utilize idle, underused space in A100 (ACIS area, tape library) Rearrange existing functionsRearrange existing functions

PrintingPrinting Repair/StagingRepair/Staging ColocationColocation

Build out B level -- Probably cost-prohibitive at ~$1,000/sq ft.Build out B level -- Probably cost-prohibitive at ~$1,000/sq ft.

PowerPower

Watts (single phase)Watts (single phase)W = Volts x AmpsW = Volts x Amps

Apparent Power (3 phase)Apparent Power (3 phase)kVA = V x Avg Amps x √3kVA = V x Avg Amps x √3

Real Power (3 phase)Real Power (3 phase)kW = kVA x Power factor ( < 1)kW = kVA x Power factor ( < 1)

PowerDensity

Watts/Sq FtOf ServerFloor Space

Other trends in computer technologyOther trends in computer technology

Disks – migration to 2.5” form factorDisks – migration to 2.5” form factor Greater manufacturer focus on heatGreater manufacturer focus on heat

Multi-core processor chipsMulti-core processor chips On-chip power managementOn-chip power management More exotic heat sinks, airflowMore exotic heat sinks, airflow Water coolingWater cooling

Denser blade servers (16 blades/2U today)Denser blade servers (16 blades/2U today) More “composite hosts” (clusters/grids)More “composite hosts” (clusters/grids)

Blade and Disk Equipment Densities

Products currently shipping

0

5

10

15

20

25

30

kW p

er R

ack

Blade Servers

Disk Arrays

Dell PowerEdge 1855

HP Proliant BL

IBM Bladecenter H20

EqualLogic PS / NetApp FAS 200

AC&NC JetStor III SATA

CYH Data Center Demand Densities CYH Data Center Demand Densities (A100+A84)(A100+A84)

100100 108 kW108 kW 3,0003,000 575575

RacksRacks PowerPower Sq FtSq Ft DevicesDevices

Per rackPer rack   -- 1.08kW1.08kW 3030 5.755.75

Per kWPer kW 0.930.93   -- 27.827.8 5.35.3

Per sq ftPer sq ft 0.030.03 40W40W   -- 0.190.19

Per devicePer device 0.170.17 190W190W 5.225.22   --

Device = Anything that consumes rack space and power(Computer, disk, switch, UPS)

CMU Campus Server Room SurveyFeb 2005

0

50

100

150

200

250

300

350

400

450

Wat

ts/S

q Ft

CYH A100

Beowulf clusterPDL/Cylab (2005)

Wean 3611

ECE HH-211B

Chemistry MI-502

PDL/Cylab (2011)

Qatar (Day 1)

Qatar (Maximum)

Power Density

CYH Emergency Electrical Power BudgetCYH Emergency Electrical Power Budget

Maximum LoadsMaximum Loads kVAkVA

3 UPS units (50 3 UPS units (50 kVA ea)kVA ea) 150150

4 (of 6) Edpacs4 (of 6) Edpacs 3737

Life safetyLife safety 33

HumidifiersHumidifiers 3737

LightingLighting 33

ECE UPSECE UPS 6060

ECE EdpacECE Edpac 6060

100 Ton rooftop 100 Ton rooftop chillerchiller 130130

Chilled water pumpChilled water pump 5252

Booster pumpBooster pump 1414

WH Hot water WH Hot water pumpspumps 2020

WH Air compressorWH Air compressor 1010

WH Life safetyWH Life safety 1010

586 kva586 kva

Estimated normal load500 – 550 kVA

Generator capacity625 kVA (500 kW)

A100/A84 Power Usage Projection

0.0

20.0

40.0

60.0

80.0

100.0

120.0

140.0

Po

wer

(kV

A)

Capacity – How much power?Capacity – How much power?

Redundancy – How much?Redundancy – How much?

DesignDesign LoadLoad PDU/UPSPDU/UPS

NN 11 11

N+1N+1 11 1.5*1.5*

2N2N 11 22

* Multiplier depends on topology

A100 Power DistributionA100 Power Distribution

Row 90Powered By PDUs 3 & 4

Row 80Powered By PDUs 3 & 4

Row 70Powered By PDUs 3 & 4

Row 60Powered by PDUs 2 & 4

Row 50Powered by PDUs 2 & 4

Row 40Powered by PDUs 2 & 4

Row 30Powered by PDUs 2 & 3

Row 20Powered by PDUs 2 & 3

Row 10Powered by PDUs 2 & 3

PDU 4

PDU 3

PDU 2

A-100 Power Distribution

How can we add more power capacity?How can we add more power capacity?

No upgrade to building transformerNo upgrade to building transformer Emergency (Generator + UPS)Emergency (Generator + UPS)

Increase UPS, PDU capacitiesIncrease UPS, PDU capacities Increase generator capacityIncrease generator capacity Increase panel sizes, feeder capacityIncrease panel sizes, feeder capacity StrategiesStrategies

Separate computer, HVAC loads for power qualitySeparate computer, HVAC loads for power quality Begin removing non-critical equipment from emergency Begin removing non-critical equipment from emergency

power to manage emergency capacitypower to manage emergency capacity Install pathway for future growthInstall pathway for future growth

Replacing distribution infrastructure is costly and Replacing distribution infrastructure is costly and disruptivedisruptive

Emergency CoolingEmergency Cooling

Actual LoadsActual Loads ServersServers DisksDisks Networking equipmentNetworking equipment Water pumpsWater pumps HVAC (chillers, Edpacs)HVAC (chillers, Edpacs) Humidifiers Humidifiers Emergency lightingEmergency lighting Life safety equipmentLife safety equipment SunlightSunlight Power distribution (PDU, UPS)Power distribution (PDU, UPS) Equipment inefficienciesEquipment inefficiencies PeoplePeople

Supply DependenciesSupply Dependencies Chiller capacityChiller capacity Generator capacityGenerator capacity Pipe diameterPipe diameter Chilled water temperatureChilled water temperature Circulating pump gpmCirculating pump gpm Chilled water pressureChilled water pressure Fan cfmFan cfm

Power In (kW) = Heat Out (BTUs)1 kW = 3,412 BTUs

1 kW requires 160 cfm for 20° ∆T1 Ton requires 3 gpm of chilled water

How can we add more cooling capacity?How can we add more cooling capacity?

Emergency chiller and generator must be upgraded Emergency chiller and generator must be upgraded together as critical load increasestogether as critical load increases

Options for upgrading HVAC capacityOptions for upgrading HVAC capacity As needed, per rack, using Liebert’s Xtreme Density As needed, per rack, using Liebert’s Xtreme Density

supplemental cooling products [supplemental cooling products [moremore]] As needed, per pair of rows, using APC’s NetworkAIR IR As needed, per pair of rows, using APC’s NetworkAIR IR

products [products [moremore]] Entire room capacity may be upgraded by adding additional Entire room capacity may be upgraded by adding additional

CRAC units, up to 60 T eachCRAC units, up to 60 T each

All options supplement existing HVACAll options supplement existing HVAC All options can be added incrementally, by rack(s)All options can be added incrementally, by rack(s) One high-density project does not “break” entire roomOne high-density project does not “break” entire room

Data ConnectionsData Connections

How many systems?How many systems? Types of network connectionsTypes of network connections

PrimaryPrimary RedundantRedundant StorageStorage Cluster heartbeatCluster heartbeat ManagementManagement ConsoleConsole Other?Other?

A100 Equipment Installations by Year

0

20

40

60

80

100

120

140

160

180

2002 2003 2004 2005

A100/A84 Server Installations FY 2004-05

0

20

40

60

80

100

120

Jul-0

4

Aug-0

4

Sep-0

4

Oct-04

Nov-04

Dec-04

Jan-

05

Feb-0

5

Mar

-05

Apr-0

5

May

-05

Jun-

05

Jul-0

5

Cumulative

By Month

Data Center ProjectionsData Center Projections(A100 + A84)(A100 + A84)

NowNow 5 Years5 Years 10 Years10 Years

DevicesDevices 575575

Data Data connections connections per systemper system

1.11.1

Power/CoolingPower/Cooling

(kW)(kW) 108108

RacksRacks 100100

Space (sq ft)Space (sq ft) 3,0003,000