View
257
Download
0
Category
Tags:
Preview:
DESCRIPTION
Learn how to optimize data center cabinet power distribution designs for higher density loads.
Citation preview
Designing Critical Power Chainsfor Higher Density Loads
SURVEY QUESTION
What Is the Average Power Density
(in kW) Per Rack In Your Datacenter?
(If you’re a consultant, “… in datacenters you advise?”)
• < 2kW• 2 – 4kW• 4 – 8kW• 8 – 12kW
• 12 – 16kW• 16kW +• Unsure
Common Misnomer: “Datacenter Average Density Increasing”
Datacenter average density increases have been predicted formany years now, and as a general statement has not happened.
Same Survey Question … ~5 years ago!
Spring 2006 Spring 2007 Spring 2008 Spring 2009 2011 Prediction 2019 Prediction0
4
8
12
16
20
6 6.17.3 7.4
12
16.5
Source: Data Center Users Group (DCUG) – Emerson Network Power
Common Misnomer: “Datacenter Average Density Increasing”
Datacenter average density increases have been predicted formany years now, and as a general statement has not happened.
Same Survey Question … LAST MONTH
Spring 2009 Fall 20130
4
8
12
16
20
7.46.3
Source: Data Center Users Group (DCUG) – Emerson Network Power
And again, users predict average power densities ~12kW
in less than 3 years…
… what is going on?
The Real Story / Problem: Bifurcated Hi + Low Density Cabinets
Higher Density (15kW++)Lower Density (<8kW)
• Storage
• 1U / 2U Servers
• IDF/MDF (Networking)
Two different application profiles in critical computing environments,
each with correspondingly variant energy densities.
• Blade Server
• Compute Clusters (HPC)
1U / 2U Server Power Consumption Is Actually Decreasing
Year Watts / U
2004 (DL360G3)
360 W
2008 (DL360G5)
238 W
2011 (DL360G7)
157 W
2013 (DL360eG8)
170 W
HP DL360 @ 100% Utilization
Source: Data Center Pulse. Wiersma, Jan. “Where is the Rack Density Trend Going?”
Combining…
• major CPU improvements in power/MIPS
• better underlying h/w architecture
– e.g. High-efficiency PSUs
– e.g. Move to SSDs
– e.g. Improved front-side bus architectures (RAM)
… standard 1U / 2U servers are actually less power-hungry per U.
NET:~6 kW for 35 servers
Real Challenge = Deploying High, ALONGSIDE Lower Densities
HP C7000
• 16 blades
• 256 cores
• 10U
Cisco UCS 5108
• 8 blades
• 128 cores
• 6U
4551W 2636W
Source: SPECpower ssj_2008 @ 90% Loads
HP DL360G7
• 1 server
• 12 cores
• 1U
192W
~13.5kW (3 chassis)~18kW (4 chassis)
~15.8kW (6 chassis)~18.5kW (7 chassis)
~6.8kW (35 servers)~8.1kW (42 servers)
…But Rewards of Server Consolidation / Virtualization Worth It
8:1
Typical (Minimal)Physical Server Ratio
Post-Virtualization
– Dramatic Reduction in New Server Provisioning Time
– Increased Redundancy Characteristics
– Reduced Server Administration Workload
– Typical Investment Payback ~1 year or Less
• The compelling business case has driven significant IT virtualization initiatives for almost ten years.
• Scaling these projects upward now affect the power chain.
Sources: Gartner, IDC
Some EXAMPLE Design Challenges
– Cooling Mixed Densities• In-Row cooling?
• Actively cooled cabinets?
• Containment?
• Placement of High vs Low Density Cabinets?
– Correctly Sizing UPS
– RPP-to-Cabinet Topology (busway vs whip)
Two very different problemsa) Design for 8kW Average Rack Power Density
b) 80% of my racks are ~5kW or 6kW… but 20% are 18kW;
– Practical Cabling Concerns• # Whips per cabinet?
• 3-phase as standard?
• Investigate 400V?
– Impact on Co-Location Environments
– Hot Aisle temperature impact
– etc., etc.
The latter raises all sorts of questions. To name a few...
Agenda: Unique Design Challenges Faced by Peers >12kVA
• Safely deploy 400v/3ph distribution to the cabinet;
• Reduce # whips required per cabinet for high density;
• Six L6-30R [208V / 1ph] from one 208V / 3ph whip;
• Required breadth of 17.3kVA configurations[208V / 3ph / 60A input on IEC 460R]
Major FinancialServices Provider
• Increase Rack Density (Servers / Cabinet)
• Reduce high-density (17kVA+) costs with UL Listed power strips utilizing conduit input;
• Develop solution for 50kW cabinet;
• Enforce best-practices for load-balancing:
o blade chassis + other high-power devices;
o on high-power PDUs (208V / 3ph / 60a);
Major InternetSite
• Place five c7000 chassis in a single cabinet;
• While still using economizer for cooling;
Major International
Bank
Agenda: Unique Design Challenges Faced by Peers >12kVA
MAJOR INTERNATIONAL BANK
Free Cooling With 5x HP c7000 per cabinet
Challenge
• Appropriate overcurrent protection breaker count & placement;
• Extremely high ambient temperatures;
RPP / Breaker Sizing
• 4551kW per blade chassis x 5 blades per cabinet
• ÷ 97.5% power factor
• = 23.3kVA per cabinet
• @ 208v / 3phase = 23338VA ÷ 208V ÷ √3 = 64.8A continuous / line;
• x 125% (NFPA 70, Article 645) = 81A per line;
• Rounding up …
• 100A breaker
• 80A rated continuous (28.8kVA)
• @ 208v / 3ph;
MAJOR INTERNATIONAL BANK
Free Cooling With 5x HP c7000 per cabinetDesign 1 - NEC Minimum
• Each breaker (1-phase, 20A, 208V) rated to 4160kVA – so in principle, 9 breakers is sufficient;
A A A
B B B
C C C
D D D
E E E
MAJOR INTERNATIONAL BANK
Free Cooling With 5x HP c7000 per cabinetDesign 2 - Three-Pole Breakers
A
A
A
B
B
B
C
C
C
D
D
D
E
E
E
MAJOR INTERNATIONAL BANK
Free Cooling With 5x HP c7000 per cabinetDesign 3 - Individual 2-pole Breakers
A A AB B BC C C
D D DE E E
Reduced transmission loss (Ohm’s Law V=IR)
Reduced step-down transformation loss
Less copper, more power
Already in widespread deployment worldwide
18kW @400V
4P 5W, 8# (24A)27.3mm (1.08”) OD
17.3kW@208V
Source: General Cable SOOW Spec Sheet
3P 4W, 4# (48A)32mm (1.26”) OD
AC Power Cable Cross-Section @ ~17.5kW
Disney, et al: 400V / 3ph to Cabinet >12.6kVA
Disney: Deploy 400v/3ph Power to Rack Safely
Challenge
• New datacenter in Kings Mountain, NC;
• Client wished to harness cost savings / electrical benefits of 400v/3ph to the rack;
• Still relatively new in North American market;
Raritan Solution
• Leading pioneer of 400v/3ph rack PDUs in North America;
• Largest install base of relatively “exotic” emerging standard [“European”];
• At client’s request, fabricated chassis to match UPS color scheme [blue, green, magenta, and yellow];
208V vs. 400V Intelligent PDUs – Power Density Review
208v / 1ph / 30a 208v / 3ph / 30a 400v / 3ph / 30a
Phase Single phase Three phase Delta Three phase Wye
Input Voltage 208V 208V, 3-phase 400V, 3-phase
Output Voltage 208V 208V 230V
Apparent power 5.0 kVA 8.6 kVA 16.6 kVA
1U servers supported* 24 41 80
2u servers supported* 24 41 80
Blade chassis supported* 1 1-2 3-4
*HP Proliant DL360G7 1U server, 192 watts at 100% load on SPECpower_ssj2008HP Proliant DL380G6 2U server, 208 watts at 80% load on SPECpower_ssj2008HP Proliant C7000, ProLiant BL2x220c G6, 4551 watts at 80% load on SPECpower_ssj2008
…Delivering More Power at Same Ampacity
…but 400v/3ph Power Raises Unique ConsiderationsUntil very recently, 400v/3ph power distribution considered “European”.True, practical expertise at this standard is rare in North America
Arc Flash Hazard. Unlike status quo, 400v/3ph technically poses an arc flash hazard. Special care required by electricians, as well as connected hardware;
Input Plug Preference. Relatively new to North America, confusion abounds regarding the correct input plug to utilize:
• NEMA L22? or NEMA L7?
• IEC 60309 (“Pin-and-Sleeve”)? If so…
- Ground pin at 6HR (415v max) : red plug? typically considered a European plug, can it pass UL?
- Ground pin at 7HR (480v max) : blue plug? typically considered a North American plug, but 480v?
Harmonics. Line-to-Neutral wiring can induce harmonics / current on the neutral line, if load is not properly balanced.
Product Selection (UPS, RPP, Rack PDU). Other vendors lack the install base, and therefore the breadth of product selection, in the 400v/3ph electrical specification.
Cisco: Reduce #Whips for High-Density Cabinet
Challenge
• Common problem for certain blade servers, and many Cisco devices [e.g. Nexus 7000 series]
• Power supplies hardwired with L6-30P power supplies;
• Customers assume rack PDUs cannot be used, run 6x whips;
Raritan Solution
• Design rack PDU with 30A breakers and NEMA L6-30R outputs;
Floor PDU / RPP
Cisco Example: Exadata Cabinets
12 x Single phase 30A Circuits
Floor PDU / RPP
BEFORE
Floor PDU / RPP
Cisco Example: Exadata Cabinets
2x Three-Phase, 50A circuits
Floor PDU / RPP
AFTER
Cisco: Electrical Equivalence with Fewer WhipsIn concept, a remote “Branch Panel”… metered + monitored for fault prevention
MAJOR FINANCIAL SERVICES COMPANY:
Maximize Deployment Flexibility @ 60A/3ph
Challenge
• Relatively small source of vendors specializing in configurations above 12.6kVA;
• Need a 3ph / 60a solution flexible to accommodate blades OR 1U/2U/storage;
• Most vendors assume blades; or 1U/2U/storage – not flexible;
Raritan Solution
• 17.3kVA (208v / 3ph / 60a)
• 12x C19’s allow deployment of up to 4x blade servers;
• 24x C13’s allow deployment of ~20x 2U servers + switches;
• Six circuit breakers protect load against bad power supplies;
• Cable exits on end of PDU to simplify cable management of very large SO cord;
• Each “bank” of outlets alternates phases to enable easier blade server deployment (see “Major Internet Company” case study, later in this slide deck)
IEC 60309 3-Pole 4-Wire
“Pin-and-Sleeve / CEEform”IP67 (“Watertight”)
“460C9W or 460R9W”
MAJOR FINANCIAL SERVICES COMPANY:
Maximize Deployment Flexibility @60A/3ph• 208v / 60A / 3ph power strip requires 4-conductor / #4 AWG input cable;
• SOOW UL rated cable is 1.26” in diameter, and weighs 11.5 pounds*!!
• Appropriate plug selection required to ensure reliable connection;
* assumes 10ft input cable** photo shows 2P3W
IEC 60309 3-Pole 4-Wire**
“Pin-and-Sleeve / CEEform”IP44 (“Splashproof”)“460C9 or 460R9”
NEMA 15-60P
IP44 (“Splashproof”)
DeFACTO STANDARD RARITAN STANDARD(secure locking ring)
CLIENT’S CUSTOM CHOICE(physically smallest)
Twitter: Increase Rack Capacity with Confidence
Challenge
• Client standard = ~35 servers / rack. Power believed stranded / under-utilized;
• Required solution to:
• provide enough outlets to accommodate more servers (42+)
• ensure no power faults due to overloading [monitoring accuracy];
• fits in same exact form factor as incumbent;`
Raritan Solution
NEMA L21-30P208v / 3ph / 30A
8.6kVA
48xC13 and 3x NEMA 5-20R in 2.1” wide chassis
Twitter: Increase Rack Capacity with Confidence
+/- 1% Metering per Breaker Assures Fault Prevention & Detection…
… at same price as competitor’s 36-outlet model
eBay: Reduce High-Density Cost w/ Conduit
Challenge
• Client regularly deploys cabinets exceeding 20kW;
• Above 16.2kVA (208v / 3ph / 45a), SOOW cord and plug becomes very expensive;
Raritan Solution
• Design rack PDU option for “hard-wiring” to user-accessible terminal block;
• Worked with certification authorities to attain UL Listing;
• Worked with StarLine (Universal Electric) to provide appropriate tap box;
• Deployed SealTite flexible conduit at 1” Trade Size;
eBay: Reduce High-Density Cost w/ Conduit
SealTiteflexibleconduit@ 1” Trade Size
Standardconduitfittings mateto punchouton PDU
Electrician-accessibleterminal block
Savings (at ~17kVA)
Part Spec Price
PDU Cord10’ SOOW #4 AWG
$88
PDU Plug IEC 60309 $173
Tap Box Connector
IEC 60309 $205
Tap Box Cord2’ SOOW #4 AWG
$17
Tap Box Strain Relief
Kellems Mesh Grip, 1.5” O.D.
$48
~$1000 per rack
New design far cheaper:
• ~1hr of electrician time (max);
• “Bare” wire conductors;
• Inexpensive conduit;
• “Permanent” installation;
eBay: World’s Highest Density Power Strip*
Challenge
• New Dell container datacenter (“MDC”) to be deployed next month;
• Can consume nearly 50kW per cabinet at peak;
Solution
• Dual inputs: top and bottom, each 27kVA;
* (I think)
MAJOR INTERNET COMPANY
Eliminate Stranded Capacity by DesignChallenge
• Due to weight and exhaust heat, users typically place high-power equipment (e.g. blade servers) toward the bottom of the rack;
• Client deploys 3-phase power;
• Results in overutilization of one electrical phase [outlets closest to bottom of rack]; and underutilization of other two phases = Stranded capacity.
Raritan Solution
Phase 3(L3/L1)
Phase 1(L1/L2)
Phase 2(L2/L3)
Phase 3(L3/L1)
Phase 1(L1/L2)
Phase 2(L2/L3)
Every OutletEvery BreakerClearly labeled
Key Discussion Items
1. “Average Power Density” vs. Select Cabinets @ Very High Density.
2. Traditional Line of Advisory Demarcation Has Moved. At very high densities, clients require your expertise to advise on operational implications of the electrical distribution inside the rack (not just at the RPP).
3. Genuine Strategy for High Ambient Temperatures Required. BMS monitoring of set points, return temperatures, and a few sensors per row may not be sufficient.
• True UL ratings for all equipment;
• Cost-effective cabinet-level monitoring;
• Consider qualifying vendor equipment with load testing + temperature gun;
4. Consider Physical Limitations of Copper. Above 12.6kVA, whips themselves can become operational constraints;
• SO vs. Conduit vs. Overhead Busway;
• 400v vs 208v;
• Don’t forget cabinet openings and bend radius implications;
Ready to Lean More?
Click Here to Visit Raritan.Com and Learn About Our Power Solutions
Recommended