1
Critical Design Elements for High-Power Density Data Centers
2
Critical Design Elements for High-Power Density Data Centers
When it comes to high-power density data centers, all are not created equal. As organizations increasingly focus on heavy-duty, transactional workloads like big data analytics, they’re seeking space that can support upwards of 17kW per rack. Delivering these super high-power densities while ensuring tolerable working conditions requires careful planning and significant attention to air flow management, temperature control and electricity.
#DataCenter
3
Artificial intelligence will exceed human intellectual capacity and control, thus radically changing civilization in an event called the singularity.
50th Anniversary of Moore’s Law
#DataCenter
4
• In 1984, the number of Internet devices was 1000
• In 2015, the number of Internet devices
exceed 30 Billion
• By 2020, it will exceed 50 Billion
Did you know?
#DataCenter
5
There are two main issues regarding faster processors and those are:
1. Transmission delays on the chip
2. Heat build-up on the chip
Transmission delays As the size of the wires and transistors have gotten smaller over the years, the time required to change states has gotten smaller, too. But there is some limit -- charging and draining the wires takes time. That limit imposes a speed limit on the chip.
HEAT - Every time the transistors in a gate change state, they leak a little electricity. The faster a chip goes, the more heat it generates.
What is the issue with Faster Processing? HEAT!
6
High power density computing is here if you can design for it
Smaller and More Servers = Lots of Heat & Power!
#DataCenter
7
Courtesy of Schneider Electric Most data centers average to be low density with some high density cabinets only.
Most efficient density range 5-8kW
Enterprise kW / Rack High Density Study
8
• A decade ago, standard densities were 3-5 kW per rack
• Today, it is not uncommon to see 8-12 kW per rack with some deployments reaching 15-20 kW per rack
• For this presentation, 10-20 kW is considered as high density in a colocation facility
Colocation Data Center Power Density Trends
#DataCenter
9
High Density Benefits
Lowered Total Cost of Ownership (TCO) • Emerson Network Power Study [2] – New Data Center
• Compare: 5 kw / rack (400 racks) vs. 20 kW / rack (100 racks) – Low Density vs. High Density
• Floor Space: 10, 000 SF vs. 2,500 SF • TCO difference after 5 years is $ 3, 660, 000 less for high density scenario
assuming similar maintenance cost
Categories Description 400 Racks @5 kW/rack
100 Racks @20 kW/rack
Capital Costs Building shell, rack, PDU 3,500,000 875,000 Cooling Capital Costs CRAC, high density modules, installation 830,000 1,900,000 Annual Cooling Operating Costs Cooling energy costs 946,080 525,600 Maintenance No Data No Data No Data
Estimated TCO ( End of 5th Year) 9,060,400 5,403,000
Reduced Space @ $150-200/SF Reduced # of PDUs @ $80-100K/ PDU
#DataCenter
10
High Density Benefits
Increased Scalability • Example: Customer starts up with 6 kW / rack, and after a few years has need
for 18 kW for their equipment
Current
Future
High Density Capable
11
High Density Challenges
• More cooling required in smaller footprint
• Higher cooling equipment initial costs
• Stronger weight bearing structures required for heavier racks • Response time in the event of a cooling failure
• Example: If the chiller pump failure or CRAH fan fails, data center operator will have less than 1 minute for 20 kW / cab regions before temperatures exceed ASHRAE recommended temperatures
• Solution: Feed CRAH fans and chiller pumps from UPS
• Power distribution
#DataCenter
12
Not all racks in colocation data center are high density: equipment from the past and present creates a mixed environment
• Can have 2 kW per rack and 20 kW per cabinet in same data center space
• These servers have different operating requirements (e.g. rack inlet temperatures and air flow rates)
• From a cost / watt perspective, average densities of 5-8 kW per rack is optimal [3]
Stranded Capacity (Space, Power & Cooling) • Available floor space and racks but no remaining power or cooling
• Air handlers have remaining cooling capacity but not enough air flow rates to match IT equipment requirement
• PDU has remaining electrical capacity but RPPs out of breaker positions, caused by low density racks
Resource Management (Reducing Stranded Capacity) • Quantity and location of air handlers driven based on cabinet densities in each region
• Proper planning and book-keeping for power distribution to avoid overloading PDUS and to balance phases
• Build out in modular pods to minimize stranded resources
Mixed Environment Challenges
13
Cooling Strategies for Mixed Environments
• Blanking Panels & Rack Skirt • Data Center Zoning • Vertical Exhaust Ducts (VED) • Cold or Hot Aisle Containment • In Row Coolers (IRC) • Other Strategies
#DataCenter
14
Blanking Panels & Rack Skirts
Recommended for any kW / rack for proper air flow management
12 kW / rack with and without blanking panels & rack skirts
15
Data Center Zoning (Underfloor)
Similarly, owner can build out high density cabinets in contained pods (hot and/or cold aisle)
16
High Density Cooling Solutions
• No Raised Floor – (although it could be used for routing chilled water pipes)
• Row Cooling (Close-coupled)
• Hot Aisle Containment • 25 kW+ • Scalable (add cooling units
with increased rack densities)
The challenge with any high density application is how to provide that much volume of air (both supplying cold air and rejecting hot air), typically 120-160 CFM/kW
17
High Density Cooling Solutions
• Raised Floor Supply • Chimney Racks • Ceiling Plenum Return • Cooling units can be located outside • Up to 20-25 kW
• Raised Floor Supply • Fully Contained Hot Aisle • Ceiling Plenum Return • 25 kW+ • Needs extra space (to increase volume)
18
High Density Cooling Solutions
• Hybrid Cooling • Raised Floor Supply • Row Cooling • Hot Aisle Containment • Ceiling Plenum Return • 25 kW+
#DataCenter
19
High Density Cooling Solutions
• Rear Door Heat Exchangers
• Overhead Cooling Units • Hybrid – combination of
different cooling architectures
• Pipe routing – raised floor, ceiling plenum
• 20kW+
#DataCenter
20
High Density Cooling Solutions
• Direct Server Liquid Cooling • Water can absorb about 4000
times more heat than air for the same volume
• Supercomputers (100 kW+) • Immersion Cooling (Dielectric
Fluid)
21
• Minimize underfloor obstruction by planning route of conduit and piping to limit the maximum obstruction in any given area
• Strict hot aisle-cold aisle configuration with containment when necessary
• Balance tile flow rates to match required rack flow rates: manually or use tile with automatic dampers
• Overhead coolers: similar to in row coolers but do not occupy floor space
• Direct Liquid Cooling – provide water based cooling directly to chip level
Other Strategies:
#DataCenter
22
The Electrical Issue of High Power Density
• Proper management of power cables and electrical distribution system
• Management of air flow due to # of conduits if placed under floor
• Flexible distribution design plan is needed
• Number of remote power panels and their location
• Cost
#DataCenter
23
• The very low loads are mainly rack enclosures with wiring patch panels, switches, and hubs
• Loads in the 1 kW range are mainly sparsely populated rack enclosures • Loads in the 2-3 kW range are mainly rack enclosures that are populated
with typical equipment but with significant unfilled rack space • Loads in the 5 kW range are partially loaded with 1U servers, or contain a
mix of technologies • Loads in the 7 kW + range are extremely rare but, according to customers,
are going to become more common with the recent density increases resulting from server technology advancements
• Colocation Facilities must provide for all the above and in INAP’s case up to 20kW/ Cabinet
Power densities in an average Enterprise Data Center
#DataCenter
24
The Electrical Issue of High Power Density
Checklist 1. Know your equipment power requirement (Voltage, Amps, Phase,
Primary, Redundant) needs to allow electrical design planning. If you’re a collocation provider you must prepare for all types of circuits
2. Be sure your team understands how to manage circuits and determining overloading situations before they are installed
3. Pre-plan your electrical architecture with your rack plan layout to manage conduit and circuit locations
4. Design your system to be flexible to avoid stranded capacity, installation cost sticker shock and risk reduction
#DataCenter
25
The Electrical Issue of High Power Density
Infrastructure deployment needs to consider: • Sold / reserved customer power capacity • Actual capacity being drawn by the racks • Ensuring enough circuit / pole positions are available
Infrastructure topology should: • Minimize upfront / Day 1 installation costs • Avoid prematurely installing or stranding infrastructure • Scale without significant cost premium to accommodate more racks & more power capacity
Customer load profiles at INAP data centers are trending towards an INCREASE in kW / Rack or Power Density (w/sq ft)
• Increased risk of inadvertently overloading the infrastructure • Increased risk of Customer power capacity demand out-pacing the ability to
deploy infrastructure
#DataCenter
26
The Electrical Issue
We want to ensure that our power infrastructure does not have STRANDED capacity
• The generators, switchgear, UPS are the most costly components; we want the capacity of those to be “fully COMMITTED”
The Customer power circuit demands vary throughout the Data Hall
• Each circuit requires a circuit breaker that uses “pole positions” within an RPP panel
• To ensure we do not strand capacity, we need enough circuits to handle “low density” racks
• “High density” racks require fewer pole positions and fewer RPPs
#DataCenter
27
1. There are different SIZES of Circuits (capacity) that can be provided:
• The table below indicates a number of different breaker configurations and the resulting kW
• The most popular circuits are 208V/30A, 2-pole (5kW) & 208V/30A, 3-pole (8.6kW)
• Although the normal power draw across these circuits may be lower, there is nothing to prevent a customer from drawing up to the maximum circuit load at any point in time
2. The sold or COMMITTED capacity is always significantly higher than the ACTUAL capacity
• If COMMITTED capacity EXCEEDS the infrastructure capacity then it is OVERSUBSCRIBED
• Even if you monitor the actual power consumption, a customer may increase their power draw without warning
• There is no means to automatically shed load from the RPP, PDU, or UPS
The Electrical Issue
28
1. Each Data Hall can be sub-divided into 3-zones (upper / middle / lower)
• Initially install ONE UPS and 3 PDU pairs (6 PDUs) for a data hall • Capacity across the total datacenter is up to 1200kW
2. As customer racks are deployed in a given zone install additional RPPs to support LOW power density
• Each zone initially provisioned with up to 500kW of capacity • Up to 10 RPPs (4 panel with dual 400A inputs, RPP Option 3) per zone (5
A-side, 5 B-side) means there will be sufficient circuit capacity • RPPs to be floor mounted to reduce length of branch circuits
3. Monitor the overall power consumption and install an additional UPS as required
4. As the power density in a given zone increases deploy additional PDUs and RPPs to serve that increase in load
• Additional RPPs to be wall-mounted (2 panel with single 400A input, RPP Option 1) for easier retrofit installation
The following slides indicate a phased deployment approach
#DataCenter
29
4-‐Panel RPP (A-‐side)
4-‐Panel RPP (B-‐side)
PDU Pair (Fed by UPS 1)
The Electrical Issue
30
2-‐Panel RPP (A-‐side)
PDU Pair (Fed by UPS 2)
The Electrical Issue
31
3 Key Takeaways
1. The High Power Density option is available, but for the average enterprise it remains a small footprint. Internap provides this option for all customers
2. There are various types of cooling solutions dependent upon the actual density that needs to be addressed
3. Electrical distribution in a high power density data center requires pre-design and proper circuit management to avoid issues
#DataCenter
32
References
1. Clark, J. (2013, October 24). Raising Data Center Power Density. Retrieved March 19, 2015, from http://www.datacenterjournal.com/it/raising-data-center-power-density/
2. High Density = Lower Cost ( Efficiency Without Compromise E-Book Series, Chapter 2). (n.d.). Retrieved March 19, 2015, from http://www.emersonnetworkpower.com/documentation/en-us/brands/liebert/documents/white papers/emerson network power higher density equals lower cost.pdf
3. Brown, K., Torell, W., & Avelar, V. (n.d.). Retrieved March 19, 2015, from http://www.apcmedia.com/salestools/VAVR-8B3VJQ/VAVR-8B3VJQ_R0_EN.pdf?sdirect=true
#DataCenter