Intel® Server System S9200WK...Intel® Server System S9200WK Liquid Cooled Rack Reference Design Guide Guidelines and requirements to design rack and cabinet level solutions with

Intel® Server System S9200WK Liquid Cooled Rack Reference Design Guide

Guidelines and requirements to design rack and cabinet level solutions with the Intel® Server System S9200WK in a liquid cooled environment. Rev 1.2 February 2020

S9200WK


2

<Blank page>


3

Document Revision History Date Revision Changes

August 2019 1.0 First release.

December 2019 1.1 • Updated chapter 6 • Added Additional references (Appendix A)

February 2020 1.2 • Updated chapter 6


4

Disclaimers Intel technologies’ features and benefits depend on system configuration and may require enabled hardware, software, or service activation. Performance varies depending on system configuration. No computer system can be absolutely secure. Check with your system manufacturer or retailer or learn more at intel.com.

You may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel products described herein. You agree to grant Intel a non-exclusive, royalty-free license to any patent claim thereafter drafted which includes subject matter disclosed herein.

No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document.

The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request.

Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade.

Copies of documents which have an order number and are referenced in this document may be obtained by calling 1-800-548-4725 or by visiting www.intel.com/design/literature.htm.

Intel and the Intel logo are trademarks of Intel Corporation or its subsidiaries in the U.S. and/or other countries.

*Other names and brands may be claimed as the property of others.

© Intel Corporation

http://www.intel.com/design/literature.htm


5

Table of Contents 1. Introduction .................................................................................................................................................................. 9

1.1 Usage disclaimer ............................................................................................................................................................ 10 1.2 Target audience .............................................................................................................................................................. 10

2. Components Overview ............................................................................................................................................ 11 2.1 Rack and Cabinet Configuration .............................................................................................................................. 11 2.2 Facility Power ................................................................................................................................................................... 15 2.3 Configuration Components ....................................................................................................................................... 15 2.4 Server System Configuration Requirements ...................................................................................................... 15

3. Electrical Specifications .......................................................................................................................................... 17 3.1 Overall Power Budget .................................................................................................................................................. 17 3.2 Power Distribution Unit (PDU) Selection .............................................................................................................. 18

3.2.1 EU Standards Compliant PDU .................................................................................................................................. 18 3.2.2 US Standards Compliant PDU .................................................................................................................................. 18

3.3 Power Mapping ............................................................................................................................................................... 19 3.4 4 PDU Reference Design Power Mapping ............................................................................................................ 19 3.5 3 PDU Reference Design Power Mapping ............................................................................................................ 22

4. Management Network Specifications .................................................................................................................. 28 5. Mechanical Specifications ...................................................................................................................................... 32

5.1 Rack Dimensions ............................................................................................................................................................ 32 5.2 PDU ...................................................................................................................................................................................... 32 5.3 Management Switch ...................................................................................................................................................... 32 5.4 Data Switch (Fabric) ....................................................................................................................................................... 32 5.5 Liquid Cooling Manifold .............................................................................................................................................. 32 5.6 Equipment Weight Data .............................................................................................................................................. 33 5.7 Crating ................................................................................................................................................................................. 33

5.7.1 Crate Compliance Guidelines for Shipping ......................................................................................................... 34 5.8 Drawings and CAD ......................................................................................................................................................... 34

6. Thermal Specifications and Requirements ........................................................................................................ 35 6.1 Fluid Network Concept ................................................................................................................................................ 35 6.2 Boundary conditions ..................................................................................................................................................... 35 6.3 Thermal Requirements Based on CDU Selection (End of Row-Based) ................................................... 38

6.3.1 End-of-row CDU ............................................................................................................................................................. 38 6.4 Air cooling recommendations .................................................................................................................................. 38

Appendix A. Glossary ................................................................................................................................................. 40


6

List of Figures Figure 1. Rack concept ............................................................................................................................................................................... 9 Figure 2. Cabinet concept ...................................................................................................................................................................... 10 Figure 3. Reference rack layout ........................................................................................................................................................... 12 Figure 4. Reference cabinet layout – front view ........................................................................................................................... 13 Figure 5. Reference cabinet layout – rear view ............................................................................................................................. 14 Figure 6. Power supply layout for 4 PDU design reference..................................................................................................... 20 Figure 7. Power mapping detail for 4 PDU design reference (PDUs 1 and 3) .................................................................. 21 Figure 8. Power mapping detail for 4 PDU design reference (PDUs 2 and 4) .................................................................. 22 Figure 9. Power supply layout for 3 PDU design reference rack ........................................................................................... 23 Figure 10. Power mapping detail (top) for 3 PDU design reference (PDUs 3 and 1) .................................................... 24 Figure 11. Power mapping detail (top) for 3 PDU design reference (PDU 2) ................................................................... 25 Figure 12. Power mapping detail (bottom) for 3 PDU design reference (PDUs 3 and 1) ............................................ 26 Figure 13. Power mapping detail (bottom) for 3 PDU design reference (PDU 2) ........................................................... 27 Figure 14. Network Cable Length Reference Layout .................................................................................................................. 28 Figure 15. Port mapping reference for Intel Omni-Path switches 3 and 4 ....................................................................... 29 Figure 16. Port mapping reference for Ethernet switches 1 and 2 ...................................................................................... 30 Figure 17. Port mapping reference for Intel Omni-Path switches 1 and 2 ....................................................................... 31 Figure 18. Rack level diagram .............................................................................................................................................................. 35 Figure 19. Minimum flow rate per rack ........................................................................................................................................... 36 Figure 20. Minimum flow rate per rack ........................................................................................................................................... 37 Figure 21. Pressure impedance per system .................................................................................................................................. 37

List of Tables Table 1. Facility power requirements ............................................................................................................................................... 15 Table 2. Recommended rack and cabinet configuration components ............................................................................. 15 Table 3. Intel® Server System S9200WK configuration ............................................................................................................ 16 Table 4. Rack power budget ................................................................................................................................................................ 17 Table 5. Cabinet power budget ........................................................................................................................................................... 17 Table 6. EU PDU ........................................................................................................................................................................................ 18 Table 7. US PDU ....................................................................................................................................................................................... 18 Table 8. Typical Intel® Server System S9200WK Configuration ........................................................................................... 19 Table 9. Quantity and type of cables used in the recommended configuration ............................................................ 29 Table 10. EU Compliant PDUs ............................................................................................................................................................. 32 Table 11. US Compliant PDUs ............................................................................................................................................................. 32 Table 12. Management Switch Spec ................................................................................................................................................. 32 Table 13. Management Switch Spec ................................................................................................................................................. 32 Table 14. Liquid cooling manifold features ................................................................................................................................... 33 Table 15. Rack weight estimates ........................................................................................................................................................ 33


7

Table 16. Cabinet weight estimates .................................................................................................................................................. 33 Table 17. Equipment used for simulation ...................................................................................................................................... 36 Table 18. End-of-row CDU cooling for systems with 16GB DIMMs ..................................................................................... 38 Table 19. End-of-row CDU cooling for systems with 64GB DIMMs ..................................................................................... 38 Table 20. Product family reference collaterals............................................................................................................................. 39


8

<This page is intentionally left blank>


9

1. Introduction

This reference design guide provides an overview of the architecture components, guidelines, and other requirements for designing a rack or cabinet solution with the Intel® Server System S9200WK in a liquid cooled environment.

For the purpose of this guide, the concepts of "Rack", "Cabinet", and "Cluster" are referred to in the following context:

• Rack: A Server system rack configured with liquid cooled 2U 4-compute module systems, Ethernet switches, high speed QSFP switches, Power Distribution Units (PDUs) and a liquid cooling manifold.

Figure 1. Rack concept

• Cabinet: Two racks according to the definition presented in this guide, accompanied by an Air

Handler Unit (AHU) between them.


10

Figure 2. Cabinet concept

• Cluster: A group of three or more racks or cabinets according to the definition presented in this guide.

For the rack and cabinet solution described within this guide, Intel has selected Stulz* liquid cooling products. This guide includes information concerning the hardware inside the rack, the power, cooling and network infrastructure, and the physical space it occupies.

1.1 Usage disclaimer This guide includes references to both Intel and non-Intel products but have been tested and are compatible as described. It is the responsibility of the integrator that decides against using the recommended products herein described to consult vendor datasheets and operating parameters to determine the compatibility and viability of a solution for their specific application and operating environment.

Intel Corporation cannot be held responsible for the performance of the resulting IT infrastructure configuration, failure of non-Intel components, or if any components do not operate correctly when used outside of the parameters described within this guide.

1.2 Target audience The information contained in this guide is intended for IT professionals with advanced knowledge of thermal, electrical and compute performance analysis for datacenter IT infrastructure deployments. The terminology used in this guide requires that the reader has experience performing this type of analysis and planning for datacenter IT infrastructure.


11

2. Components Overview

This chapter provides an overview of the components installed inside the reference rack and cabinet as well as the power and cooling requirements to be met when designing the reference rack and cabinet configuration.

2.1 Rack and Cabinet Configuration The solution described within this guide is presented in two different configurations: rack and cabinet. The following is a high-level overview of the major components in both configurations:

Rack configuration:

• 24 server systems • 4 – Power distribution units (PDU) • 2 – Ethernet management switches • 4 - Intel® Omni-Path edge switches • 1 - Liquid cooling rack manifold

Cabinet configuration:

• 2 – Racks in accordance with the rack configuration • 1 – Air handler unit (AHU) • 1 – Rack frame for AHU

The following figures show the recommended hardware configuration when populating a rack or cabinet. Figure 3 shows the configuration for a 48U rack with 24 chassis, Ethernet switches and Intel® Omni-Path Edge Switches, enabling an optimal configuration with the shortest cables.


12

Figure 3. Reference rack layout

Figure 4 and Figure 5 show the configuration for a cabinet comprised of 2 racks with systems and a cooling rack in between.


13

Figure 4. Reference cabinet layout – front view


14

Figure 5. Reference cabinet layout – rear view


15

2.2 Facility Power The following table lists the requirements for the facility in which the rack or cabinet will be installed:

Table 1. Facility power requirements

Requirement Details

Rack input power 400VAC 3P

Power receptacles IEC 60309 3P+N+E 63A/60A OR Terminal Block

Max input current 63 A

Single Rack power load 127 KW maximum

See Section 3.1 for details on the power budget for rack and cabinet configurations

2.3 Configuration Components The following table lists the recommended components for each rack and cabinet configuration.

Table 2. Recommended rack and cabinet configuration components

Vendor Item Type Description QTY per Rack

QTY per

Cabinet

Chatsworth* Rack Frame 48U 800mm Wide 1457mm Deep Rack - Solid Doors 1 2


Stulz* Air Handler Cyber Row Cooling for Air in Cabinet 0 1 Intel Server 2U, 4-compute module system 24 48

Enlogic* EU Std certified PDU 400V 3Ph 45.36 KVA 63A 50Hz PDU 4 8

Raritan* US Std certified PDU 400V 3Ph 55.4 KVA 80A 50Hz PDU 4 8

Cisco*

Switch - Base Unit Catalyst 2960 1Gb Network Switch 2 4

Fan Assembly Switch Fan (front to back) 2 4 PSU Redundant PSU 4 8

Intel 100G Switch Intel® Omni-Path Edge Switch 100 Series 48 Port 2 PSU 4 8

CoolIT* Liquid Cooling part

Inlet Vertical Manifolds (Cold Water) 1 2 Outlet Vertical Manifolds (Hot Water) 1 2

Intel QSFP Cables Intel Omni-Path 100G QSFP Cables 96 192 Amphenol* Ethernet Cable CAT6 Cable - Blue 100 200

Bizlink* Power Cable C13-C14 Power Cord 84 168 --- Power Cable C14 - DIN Type 2508 Power Cord 1 2

King Slide* Rail Rack mount rail kit 24 48

Stulz*/CoolIT* Coolant

Distribution Unit (CDU)

Liquid to Liquid Cooling from Rack 0 1


2.4 Server System Configuration Requirements The rack and cabinet solutions described in this guide are built around the capabilities of the Intel® Server System S9200WK.


16

A liquid cooled server system configured with 1U 48C 350W compute modules is recommended for a typical HPC application. However, other configurations can be utilized based on application and workload requirements. For other SKU options, contact your local Intel representative.

The following table lists the recommended configuration for the Intel® Server System S9200WK when selected to be installed in the reference rack or cabinet.

Table 3. Intel® Server System S9200WK configuration

Description Qty. per Node Qty. per System

Intel Server System S9200WK 1

Consisting of:

Intel® Server Chassis FC2000 Liquid-Cooled (2100W) 1

Intel® Compute Module S9200WK 1U Half-Width Liquid-Cooled CPU 48C 350W 4

Memory: DDR4 16GB 2R 2933 RDIMM 24 96

Storage: Intel SSD DC DCP4101 Series, M.2 PCIe Gen3 x4 512GB 22x80mm SSDPEKKA512G801 1 4

Fabric: Intel Omni-Path Host Fabric Interface Adapter 100 Series, 1 Port PCIe x16, Low Profile 100HFA016LS 1 4

2U system Rail kit 2

For more details on each configuration and other specifications, refer to the Intel® Server System S9200WK Technical Product Specification.


17

3. Electrical Specifications

This chapter provides information on power budget for the:

• Rack • Air handling unit (AHU) • Cooling distribution unit (CDU) • Power distribution unit (PDU) planning and selection • Power mapping for all equipment • Network switch mapping

3.1 Overall Power Budget Table 5. Cabinet power budget lists the recommended components for each rack and cabinet configuration. The total de-rated power for a rack is 159.6KVA, and the total available power in-rack is 220KW divided into 4 PDUs at 55KW each. The following tables list the estimated power budget for both the rack and cabinet. The values are based on the following assumptions:

• Node configuration: 2 x 350W CPUs, 24 X 14W DIMMs, 2 x 15W PCIE cards • Cluster Configuration: 5 Cabinets = 10 Racks with 24 Servers each, 4 Nodes per Server, 3 end-of-row

CDUs in Parallel • Secondary Fluid: 25% Propylene Glycol Water (PG25) • Connection Points: CBG20 QD at Rack level and SCG06 at Chassis level. • Primary Fluid: ASHRAE W3 (32°C) Water at 360lpm per each end-of-row-based CDU. • 3CDUs recommended for every 960 nodes (10 racks with 24 chassis each) • 4.6lpm per chassis or 1.15lpm per compute module • Server exit temp = 50°C.

Table 4. Rack power budget

Compute Rack Qty. per rack Device Power Aggregate Power Amps @ 240V

1 2U four node server system 24 5225W 125,400W 21.7708333

2 Intel Omni-Path Edge 48-port Switch 4 230W 920W 1

3 Cisco* 2960 Catalyst 1G Network Switch 2 125W 250W 0.54

Total 126,570W

De-rated power 159,625VA

Available power in rack per PDU capacity 4 55,000W 220,000W

Table 5. Cabinet power budget

L11 Cabinet (MDC - 2 x Compute IT Racks + 1 x Cooling Rack) Qty per cabinet Device Power Aggregate Power 1 Compute Rack - 1 (24 x server systems) 2 127000W 254,000W 2 Air Handler (Cooling Rack) 1 6,100W 6,100W

Total 260,100W De-rated power 325,125VA

Cluster level Power consumption

1 Coolant Distribution Unit (CDU CHx750) 3 7500W 22,500W 2 L11 Cabinet 5 260,100W 1,300,500W

Total 1,323,000W


18

De-rated power 1,653,750VA

3.2 Power Distribution Unit (PDU) Selection PDU selection is a crucial step in system design. When selecting a PDU, consider both the physical attributes (mechanical dimensions, mounting features, cord length, etc.) and electrical attributes (voltage type, power capacity, circuit breaker counts, etc.).

To power a single rack with 24 x 2U/4-Compute module systems, the PDU must be three-phase 400/415 V rated and have a power capacity range between 38 kVA to 46 kVA with 12 or more circuit breakers/branches. A higher power capacity, three-phase 400 V rated PDU is also acceptable.

With an estimate of the total actual power for all equipment around 127 kW in a single rack, the de-rated power budget for the rack is 159 kVA. A 20% safety margin is added into the rack level power design.

The following table lists the recommended components for each rack and cabinet configuration. Table 6. EU PDU and Table 7. US PDU recommended PDUs for the rack or cabinet configurations

3.2.1 EU Standards Compliant PDU

Table 6. EU PDU

Manufacturer Enlogic* 3 Phase WYE Model # EN 1185

PDU remote management YES (Non-Switched PDU) Power cord length 3.0 m

Plug type & quantity IEC309 563p6 Output receptacle/socket 36 outlets - 24 x C13, 12 x C19

Rated Input voltage 380V - 415V Nominal Input voltage 400V

Input Frequency 50 Hz Input Power Capacity 45.36 kVA

Input current (derated/max) 63A

3.2.2 US Standards Compliant PDU

Table 7. US PDU

Manufacturer Raritan* Model # PX3-5157XU-01V2

PDU remote management YES (Switched PDU) Power cord length N/A

Plug type & quantity Terminal Block Output receptacle/socket 48 x C13 (12A)

Rated Input voltage 380V - 415V Nominal Input voltage 400V

Input Frequency 50/60 Hz Input Power Capacity 52.7 kVA@380V, 57.5kVA@415V

Input current(derated/max) 80A


19

3.3 Power Mapping The configuration consists of approximately 84 connections for individual power supply units for servers and switches. The power load must be distributed equally across the PDUs for the entire rack.

When the entire rack is used for a high-performance computing (HPC) application, all nodes consume approximately the same amount of power in a typically loaded configuration. The common redundant power supply units used in the Intel® Server System S9200WK operate at the higher end of their capability by splitting the load equally or accordingly when running at maximum power. Generally, if each system chassis is plugged into three different PDUs, a single PDU failure prevents all systems from running at full power. However, if two or three PSUs within the same system are plugged into one PDU and phases are balanced by evenly loading all PDUs from top to bottom, a single PDU failure affects fewer systems in the rack.

Serviceability of the server rack is improved if the chassis is plugged into power at the same unit level in the rack using the shortest possible power cable routes. The tables in this section show the recommended mapping between power supply modules and power outlets, beginning with the rack and then passing to the different PDUs. The tables are color coded for convenience.

The Intel® S9200WK Server Systems were typically configured for both the 4 PDU and 3 PDU design concepts. The configuration and measured power are shown in Table 8 below.

Table 8. Typical Intel® Server System S9200WK Configuration

Intel® Server System S9200WK Configuration

Calculated Power Consumption

(WAC)

Measured Power Consumption

- typical ambient (WAC)

Measured Power Consumption

- worst ambient (WAC)

+5% guard band

(WAC)

CPUs- 2 x 48C DIMM- 24 x 16GB 2933MTs

Fabric- 1 x OPA 4500W 4000W 4160W 4370W

Note: Deviation from the typical configuration to higher TDP configurations could result in the loss of additional server systems during PDU failures. It will require power budgeting recalculations due to the increased loading of the greater TDP configuration.

3.4 4 PDU Reference Design Power Mapping In the 4 PDU design concept, eight Intel® S9200WK Server Platforms are connected to a single PDU. Each system power supply is connected to a different phase, e.g. PSU1 (phase 1), PSU2 (phase 2), and PSU3 (phase 3). With this distribution, the loading and phases across the PDUs are well balanced and offers the greatest redundancy and uptime during a PDU failure (refer to the 4 PDU configuration images). During a single PDU failure, it is expected that the 3 remaining operational PDUs will provide enough power to the impacted server systems to remain operational without impact to system performance.


20

Figure 6. Power supply layout for 4 PDU design reference


21

Figure 7. Power mapping detail for 4 PDU design reference (PDUs 1 and 3)


22

Figure 8. Power mapping detail for 4 PDU design reference (PDUs 2 and 4)

3.5 3 PDU Reference Design Power Mapping In a 3PDU design, there is an increased risk of losing additional server systems due to the loading and mapping of the PDUs. In this design, both the Ethernet and OPA switches are connected in a redundant fashion across different PDUs. However, for this configuration the switches share power with several server systems, thereby increasing risk of losing the switches during a single PDU failure. This risk is mitigated in the 4 PDU rack design since power to the switches is separate from the server systems.

In theory, during a single PSU failure, the 2 remaining PSUs will support all system loading unless throttling triggers (e.g. OC or OT CLST) or shutdown (e.g. OCP) occurs. The increased current drawn by the remaining PSUs will increase current loading for each PDU and its associated phase. Ideally, loss of a single PSU (and single phase) will increase loading to other 2 phases, thereby reaching the input current OC trigger. Secondly, each PDU circuit breaker is set to 20A which means that as PSU loading increases, the risk for a circuit breaker trip increases as well.

Two power mappings have been considered depending on the availability of the main power lines. Figure 10 and Figure 11 show the power mapping when the power lines come from the top of the reference rack. Figure 12 and Figure 13 show the power mapping when the power lines come from the bottom of the reference rack.


23

Figure 9. Power supply layout for 3 PDU design reference rack

U#4847 PSU 1 PSU 2 PSU 34645 PSU 1 PSU 2 PSU 3

4443 PSU 1 PSU 2 PSU 34241 PSU 1 PSU 2 PSU 34039 PSU 1 PSU 2 PSU 33837 PSU 1 PSU 2 PSU 33635 PSU 1 PSU 2 PSU 33433 PSU 1 PSU 2 PSU 33231 PSU 1 PSU 2 PSU 33029 PSU 1 PSU 2 PSU 32827 PSU 1 PSU 2 PSU 32625 PSU 1 PSU 2 PSU 32423 PSU 1 PSU 2 PSU 32221 PSU 1 PSU 2 PSU 32019 PSU 1 PSU 2 PSU 31817 PSU 1 PSU 2 PSU 31615 PSU 1 PSU 2 PSU 31413 PSU 1 PSU 2 PSU 31211 PSU 1 PSU 2 PSU 3109 PSU 1 PSU 2 PSU 387 PSU 1 PSU 2 PSU 365 PSU 1 PSU 2 PSU 343 PSU 1 PSU 2 PSU 321 PSU 1 PSU 2 PSU 3

Liquid Liquid

Liquid Liquid

Liquid Liquid

Liquid Liquid

Liquid Liquid

Liquid Liquid

Liquid Liquid

Liquid Liquid

Liquid Liquid

Liquid Liquid

Liquid Liquid

Liquid Liquid

Liquid Liquid

Liquid Liquid

Liquid Liquid

Liquid Liquid

Liquid Liquid

Liquid Liquid

Liquid Liquid

Liquid Liquid

Liquid Liquid

Rear View of the Rack

Liquid Liquid

Liquid Liquid

Liquid Liquid


24

TOP FEED ORIENTATION

Figure 10. Power mapping detail (top) for 3 PDU design reference (PDUs 3 and 1)

Port type Port # Port # Port type Port type Port # Port # Port type

C19 1 6 C19 C19 1 6 C19C19 2 7 C19 C19 2 7 C19

6.52 C13 3 U47 - PSU1 U47 - PSU3 8 C13 6.52 6.52 C13 3 U43 - PSU1 U43 - PSU3 8 C13 6.52

C13 4 9 C13 C13 4 OPA1-PSU2 9 C13 1.00


13.04 C19 11 16 C19 13.04 13.04 C19 11 16 C19 14.04

C19 12 17 C19 C19 12 17 C196.52 C13 13 U35 - PSU3 U47 - PSU2 18 C13 6.52 6.52 C13 13 U31 - PSU3 U43 - PSU2 18 C13 6.52


13.04 C19 21 26 C19 13.04 13.04 C19 21 26 C19 13.04

C19 22 27 C19 C19 22 27 C196.52 C13 23 U35 - PSU2U35 - PSU1 28 C13 6.52 6.52 C13 23 U31 - PSU2 U31 - PSU1 28 C13 6.52

C13 24 ETH2-PSU2 29 C13 0.54 0.54 C13 24 ETH2-PSU1 ETH1-PSU2 29 C13 0.54

6.52 C13 25 U29 - PSU2U29 - PSU1 30 C13 6.52 6.52 C13 25 U25 - PSU2 U25 - PSU1 30 C13 6.52

13.04 C19 31 36 C19 13.58 13.58 C19 31 36 C19 13.58



13.04 C19 41 46 C19 13.04 13.04 C19 41 46 C19 13.04


C13 44 49 C13 C13 44 OPA2-PSU1 49 C13 1.00


13.04 C19 51 56 C19 13.04 13.04 C19 51 56 C19 14.04


C13 54 OPA1-PSU2 59 C13 C13 54 59 C136.52 C13 55 U5 - PSU2 U5 - PSU1 60 C13 6.52 6.52 C13 55 U1 - PSU2 U1 - PSU1 60 C13 6.52

13.04 13.04 13.04 13.04

PDU 3 PDU 152.17 Amp 53.17 Amp52.71 Amp 52.71 Amp52.17 Amp 53.71 Amp

Total Phase Amp per Rack158.52159.60158.60

Phase 2Phase 3

Phase 3 Phase 3

Phase 1

Phase 1 Phase 1Phase 2 Phase 2

Circ

uit 1

2

Circ

uit 1

1

Circ

uit 1

2

Circ

uit 1

1

Circ

uit 8

Circ

uit 7

Circ

uit 8

Circ

uit 7

Circ

uit 1

0

Circ

uit 9

Circ

uit 1

0

Circ

uit 9

Circ

uit 4

Circ

uit 3

Circ

uit 4

Circ

uit 3

Circ

uit 6

Circ

uit 5

Circ

uit 6

Circ

uit 5

Circ

uit 2

Circ

uit 1

Circ

uit 2

Circ

uit 1

PDU1 - TopPDU3 - Top Amp@230V

Amp@230V


25

TOP FEED ORIENTATION

Figure 11. Power mapping detail (top) for 3 PDU design reference (PDU 2)

Port type Port # Port # Port type

C19 1 6 C19C19 2 7 C19

6.52 C13 3 U45 - PSU1 U45 - PSU3 8 C13 6.52

1.00 C13 4 OPA1-PSU1 9 C136.52 C13 5 U39 - PSU1 U39 - PSU3 10 C13 6.52

14.04 C19 11 16 C19 13.04

C19 12 17 C196.52 C13 13 U33 - PSU3 U45 - PSU2 18 C13 6.52

C13 14 19 C136.52 C13 15 U27 - PSU3 U39 - PSU2 20 C13 6.52

13.04 C19 21 26 C19 13.04

C19 22 27 C196.52 C13 23 U33 - PSU2 U33 - PSU1 28 C13 6.52

0.54 C13 24 ETH1-PSU1 29 C136.52 C13 25 U27 - PSU2 U27 - PSU1 30 C13 6.52

13.58 C19 31 36 C19 13.04

C19 32 37 C196.52 C13 33 U21 - PSU1 U21 - PSU3 38 C13 6.52

C13 34 39 C136.52 C13 35 U15 - PSU1 U15 - PSU3 40 C13 6.52

13.04 C19 41 46 C19 13.04

C19 42 47 C196.52 C13 43 U9 - PSU3 U21 - PSU2 48 C13 6.52


14.04 C19 51 56 C19 13.04

C19 52 57 C196.52 C13 53 U9 - PSU2 U9 - PSU1 58 C13 6.52

C13 54 OPA1-PSU1 59 C13 1.00

6.52 C13 55 U3 - PSU2 U3 - PSU1 60 C13 6.52

13.04 14.04

PDU 253.17 Amp54.17 Amp52.71 AmpPhase 3

Phase 1Phase 2

Circ

uit 1

2

Circ

uit 1

1

Circ

uit 8

Circ

uit 7

Circ

uit 1

0

Circ

uit 9

Circ

uit 4

Circ

uit 3

Circ

uit 6

Circ

uit 5

Circ

uit 2

Circ

uit 1

PDU2 - TopAmp@230V


26

BOTTOM FEED ORIENTATION

Figure 12. Power mapping detail (bottom) for 3 PDU design reference (PDUs 3 and 1)

Port type Port # Port # Port type Port type Port # Port # Port type6.52 C13 60 U47 - PSU1 U47 - PSU3 55 C13 6.52 6.52 C13 60 U43 - PSU1 U43 - PSU3 55 C13 6.52

C13 59 54 C13 C13 59 OPA1-PSU2 54 C13 1.00


C19 57 52 C19 C19 57 52 C1913.04 C19 56 51 C19 13.04 13.04 C19 56 51 C19 14.04



13.04 C19 47 42 C19 13.04 13.04 C19 47 42 C19 13.04


C13 39 ETH2-PSU2 34 C13 0.54 0.54 C13 39 ETH2-PSU1 ETH1-PSU2 34 C13 0.54

6.52 C13 38 U29 - PSU2U29 - PSU1 33 C13 6.52 6.52 C13 38 U25 - PSU2 U25 - PSU1 33 C13 6.52

13.04 C19 37 32 C19 13.58 13.58 C19 37 32 C19 13.58



13.04 C19 27 22 C19 13.04 13.04 C19 27 22 C19 13.04


C13 19 14 C13 C13 19 OPA2-PSU1 14 C13 1.00


13.04 C19 17 12 C19 13.04 13.04 C19 17 12 C19 14.04



13.04 C19 7 2 C19 13.04 13.04 C19 7 2 C19 13.04

C19 6 1 C19 C19 6 1 C19

PDU 3 PDU 152.17 Amp 53.71 Amp52.71 Amp 52.71 Amp52.17 Amp 53.17 Amp

Total Phase Amp per Rack158.60158.60158.52

Phase 3 Phase 3

Phase 1Phase 2Phase 3

Phase 1 Phase 1Phase 2 Phase 2

Circ

uit 1

Circ

uit 2

Circ

uit 1

Circ

uit 2

Circ

uit 3

Circ

uit 4

Circ

uit 3

Circ

uit 4

Circ

uit 5

Circ

uit 6

Circ

uit 5

Circ

uit 6

Circ

uit 7

Circ

uit 8

Circ

uit 7

Circ

uit 8

Circ

uit 9

Circ

uit 1

0

Circ

uit 9

Circ

uit 1

0

PDU3 - Bottom PDU1 - Bottom

Circ

uit 1

1

Circ

uit 1

2

Circ

uit 1

1

Circ

uit 1

2

Amp@230V

Amp@230V


27

BOTTOM FEED ORIENTATION

Figure 13. Power mapping detail (bottom) for 3 PDU design reference (PDU 2)

Port type Port # Port # Port type6.52 C13 60 U45 - PSU1 U45 - PSU3 55 C13 6.52


C19 57 52 C1914.04 C19 56 51 C19 13.04

6.52 C13 50 U33 - PSU3 U45 - PSU2 45 C13 6.52

C13 49 44 C136.52 C13 48 U27 - PSU3 U39 - PSU2 43 C13 6.52

13.04 C19 47 42 C19 13.04

C19 46 41 C196.52 C13 40 U33 - PSU2 U33 - PSU1 35 C13 6.52

0.54 C13 39 ETH1-PSU1 34 C136.52 C13 38 U27 - PSU2 U27 - PSU1 33 C13 6.52

13.58 C19 37 32 C19 13.04

C19 36 31 C196.52 C13 30 U21 - PSU1 U21 - PSU3 25 C13 6.52

C13 29 24 C136.52 C13 28 U15 - PSU1 U15 - PSU3 23 C13 6.52

13.04 C19 27 22 C19 13.04

C19 26 21 C196.52 C13 20 U9 - PSU3 U21 - PSU2 15 C13 6.52


14.04 C19 17 12 C19 13.04

C19 16 11 C196.52 C13 10 U9 - PSU2 U9 - PSU1 5 C13 6.52

C13 9 4 C136.52 C13 8 U3 - PSU2 U3 - PSU1 3 C13 6.52

13.04 C19 7 2 C19 13.04

C19 6 1 C19

PDU 252.71 Amp53.17 Amp53.17 AmpPhase 3

Phase 1Phase 2

Circ

uit 1

Circ

uit 2

Circ

uit 3

Circ

uit 4

Circ

uit 5

Circ

uit 6

Circ

uit 7

Circ

uit 8

Circ

uit 1

2

Circ

uit 9

Circ

uit 1

0

PDU2 - Bottom

Circ

uit 1

1

Amp@230V


28

4. Management Network Specifications

This chapter provides recommendations on the network mapping for the reference rack. With the cabinet being comprised of racks, the information for the reference cabinet is covered as well.

All network switches can be mounted on the side of the rack with special cutouts on the rack’s EIA vertical rail. The following figures and tables show the required port mappings for six vertically mounted switches (two Ethernet switches and four Intel® OPA switches), and the cabling layout for the rack as seen from the front. The figures are listed starting from the top of the rack and are color coded for convenience.

Figure 14. Network Cable Length Reference Layout


29

Table 9. Quantity and type of cables used in the recommended configuration

Cable Type Length Left Side Right Side

QSFP28 Cables 0.5 m 24 24

1.0 m 24 24

CAT6 Ethernet cables 3 ft 20 29

6 ft 28 28

Figure 15. Port mapping reference for Intel Omni-Path switches 3 and 4


30

Figure 16. Port mapping reference for Ethernet switches 1 and 2

51

Upl

ink

52 Ope

nServer

Port #

Server

Port #49 PD

U2

50 Ope

nU

36/S72

2

U48/S96

147

U48

/S95

48

U36

/S71

U35/S70

4

U47/S94

345

U47

/S93

46

U35

/S69

U34/S68

6

U46/S92

543

U46

/S91

44

U34

/S67

U33/S66

8

U45/S90

741

U45

/S89

42

U33

/S65

U32/S64

10

U44/S88

939

U44

/S87

40

U32

/S63

U31/S62

12

U43/S86

1137

U43

/S85

38

U31

/S61

U30/S60

14

U42/S84

1335

U42

/S83

36

U30

/S59

U29/S58

16

U41/S82

1533

U41

/S81

34

U29

/S57

U28/S56

18

U40/S80

1731

U40

/S79

32

U28

/S55

U27/S54

20

U39/S78

1929

U39

/S77

30

U27

/S53

U26/S52

22

U38/S76

2127

U38

/S75

28

U26

/S51

U25/S50

24

U37/S74

2325

U37

/S73

26

U25

/S49

U12/S24

26

U24/S48

2523

U24

/S47

24

U12

/S23

U11/S22

28

U23/S46

2721

U23

/S45

22

U11

/S21

U10/S20

30

U22/S44

2919

U22

/S43

20

U10

/S19

U9/S18

32

U21/S42

3117

U21

/S41

18

U9/

S17 U

8/S16

34

U20/S40

3315

U20

/S39

16

U8/

S15 U

7/S14

36

U19/S38

3513

U19

/S37

14

U7/

S13 U

6/S12

38

U18/S36

3711

U18

/S35

12

U6/

S11 U

5/S10

40

U17/S34

399

U17

/S33

10

U5/

S9U

4/S8

42

U16/S32

417

U16

/S31

8

U4/

S7U

3/S6

44

U15/S30

435

U15

/S29

6

U3/

S5U

2/S4

46

U14/S28

453

U14

/S27

4

U2/

S3U

1/S2

48

U13/S26

471

U13

/S25

2

U1/

S1 AHU

50

PDU1

49Po

rt #

Serv

er

Port

#

Serv

erPDU

3

52

CDU

51

SFP+

Top servers' ethernet cable routing direction

RJ45RJ

45

Top

serv

ers'

ethe

rnet

cab

le ro

utin

g di

rect

ion

Bottom servers' ethernet cable routing

Bott

om se

rver

s' et

hern

et c

able

rout

ing

SFP+


31

Figure 17. Port mapping reference for Intel Omni-Path switches 1 and 2

48

Empt

y

32

U2/

SLD3

16

U14

/SLD

27 Server

Port #

Server

Port #

Server

Port #47

Empt

y

31

U1/

SLD1

15

U13

/SLD

25 USB

MG

MT

46

Empt

y

30

U24

/SLD

47

14

Empt

yU

14/SLD28

1

U24/SLD48

17

Empty

3345

Empt

y

29

U23

/SLD

45

13

Empt

yU

13/SLD26

2

U23/SLD46

18

Empty

3444

Empt

y

28

U12

/SLD

23

12

Empt

y Empty

3

U22/SLD44

19

Empty

3543

Empt

y

27

U11

/SLD

21

11

Empt

y Empty

4

U21/SLD42

20

Empty

3642

Empt

y

26

U22

/SLD

43

10

U10

/SLD

19U

12/SLD24

5

U2/SLD4

21

Empty

3741

Empt

y

25

U21

/SLD

41

9

U9/

SLD1

7U

11/SLD22

6

U1/SLD2

22

Empty

3840

Empt

y

24

U8/

SLD1

5

8

Empt

y Empty

7

U10/SLD20

23

Empty

3939

Empt

y

23

U7/

SLD1

3

7

Empt

y Empty

8

U9/SLD18

24

Empty

4038

Empt

y

22

U20

/SLD

39

6

U6/

SLD1

1 U8/SLD16

9

U20/SLD40

25

Empty

4137

Empt

y

21

U19

/SLD

37

5

U5/

SLD9

U7/SLD14

10

U19/SLD38

26

Empty

4236

Empt

y

20

U18

/SLD

35

4

Empt

y Empty

11

U18/SLD36

27

Empty

4335

Empt

y

19

U17

/SLD

33

3

Empt

y Empty

12

U17/SLD34

28

Empty

4434

Empt

y

18

U16

/SLD

31

2

U4/

SLD7

U6/SLD12

13

U16/SLD32

29

Empty

4533

Empt

y

17

U15

/SLD

29

1

U3/

SLD5

U5/SLD10

14

U15/SLD30

30

Empty

46M

GM

T

USB

Empty

15

U4/SLD8

31

Empty

47Po

rt #

Serv

er

Port

#

Serv

er

Port

#

Serv

erEm

pty

16

U3/SLD6

32

Empty

48

QSF

P28 Q

SFP28


32

5. Mechanical Specifications

This chapter provides dimensions, weight and other mechanical specifications for some of the mentioned components used in the reference rack and cabinet configurations. Information on crating is also included.

5.1 Rack Dimensions Dimensions for the rack described in this guide are 800mm x 1450mm x 48U.

5.2 PDU The following tables list the mechanical specifications for the EU and US reference Power Distribution Units

Table 10. EU Compliant PDUs

Manufacturer Enlogic* 3 Phase WYE PDU weight 11.5 Kg

PDU dimensions (W x D x L) 59mm x 100mm x 1981mm; 2.3" x 3.9" x 78" PDU mounting Tool-less button

Table 11. US Compliant PDUs

Manufacturer Raritan* PDU weight 11.5 Kg

PDU dimensions (W x D x L) 59mm x 100mm x 1981mm; 2.3" x 3.9" x 78" PDU mounting Tool-less button

5.3 Management Switch The following table lists the Management Switch’s mechanical specifications

Table 12. Management Switch Spec

Manufacturer Cisco Model 2690

Dimensions 17.5in (W) x 16.94 in (D) x 1.75 in (H) Weight 12.9 lbs. (5.8 Kg)

5.4 Data Switch (Fabric) The following table lists the Data Switch’s mechanical specifications

Table 13. Management Switch Spec

Manufacturer Intel Model Intel® Omni-Path Edge Switch 100 Series 48

Dimensions 17.15 in x 17.3 in x 1.72 in Weight 13.64 lb. (6.2 kg)

5.5 Liquid Cooling Manifold The following table lists the features of the recommended liquid cooling manifold for the rack and cabinet configurations:


33

Table 14. Liquid cooling manifold features

Feature Description Ports 24 ports 90 degree SCG06 Stäubli* liquid quick disconnects on tubes affixed to barbed fittings on manifolds Feed

Orientation Can be configured for either a top or bottom feed configuration. The bottom feed includes a Shrader* valve on

top of the inlet (cold) manifold. Leak

Mitigation Includes a solenoid on the inlet manifold (cold) and a check valve on output (hot).

Mounting The liquid cooling manifold uses mounting plates provided by CoolIT*. The mounting plates are then affixed to both the inner side of the rack and the manifolds.

5.6 Equipment Weight Data The following tables list the weight estimates for both the rack and cabinet:

Table 15. Rack weight estimates

# Part name Weight in lbs. Qty per Rack Total (lbs.) Total (kg) 1 48U Rack Frame (Solid Doors) 685 1 685 311 2 Server system (2U / 4N) 116 24 2784 1263 3 OPA Edge Switch 15 2 30 14 4 Cisco 2690 Switch 13 2 26 12 5 PDU 20 4 80 36 6 Accessories - Rails, Brackets etc. 100 1 100 45 7 Cables 50 1 50 23 8 Rack manifold 50 2 100 45 Populated compute rack Total 3855 1766

Table 16. Cabinet weight estimates

# Part name Weight in lbs. Qty per Rack Total (lbs.) Total (kg) 1 Populated compute rack 3839 2 7678 3483 2 CyberRow* 792 1 792 359 3 Empty 700mmW 1457mmD Rack 250 1 250 113 Complete L11 Cabinet Total 8720 3955

5.7 Crating All crate materials are approved per SPEC ISTA-2B. The following steps are recommended when crating the server rack:

1. Inspect the rack caster prior to loading the rack into the crate. 2. Inspect the crates before loading the rack, noting any structural defects (per factory quality

specifications).

Note: Crate includes minimum padding material to ensure a tight fit of the rack product.

3. Ensure straps are included to facilitate rack unloading. Metal straps are required for shipping. 4. Place at least 2 tilt and shock indicators inside the crate rear and side panels. 5. Place at least 1 tilt and shock indicator on external side panels. 6. Ensure pallet bolts are secured. 7. Load the rack into crate front side first. 8. Ensure crate locks are secured properly.

Note: Add accelerometers or GPS devices if required.


34

5.7.1 Crate Compliance Guidelines for Shipping The crate will comply with certification requirements for overseas shipment and any certification labels required per shipping logistics will be added. Treatment and marking of the pallet must conform to the International Standards for Phytosanitary Measures, Publication no. 15 (ISPM 15) guidelines for regulating wood packaging materials in international trade.

5.8 Drawings and CAD Drawings and CAD files may be supplied upon request and on an as needed basis. Please contact your Intel support representative for details.


35

6. Thermal Specifications and Requirements

This chapter provides guidelines and recommendations based on a solution configuration with an end-of-row CDU measured and cooling simulation data for cabinet configurations.

The information presented considers differences in heat produced by system memory, the power requirements presented in Chapter 3, as well as the combination of the liquid coolant flow rate with the ambient temperature. These factors directly impact the number of supported systems per rack, thus affecting the number of supported systems per cabinet.

6.1 Fluid Network Concept The pictorial diagram below shows the basic layout and elements of the fluid system.

Figure 18. Rack level diagram

6.2 Boundary conditions The operation limits of the recommended configuration for rack and cabinet are based on the following assumptions:

• 48U rack with 24, 2U form factor server systems • Direct Liquid Cooling with a Heat Capture Rate (HCR) > 80% • Computer Room Air Conditioning Unit (CRAC) to cool ~15 to 20% of heat for rack configurations • Air Handler Unit (AHU) for cabinet configurations • Fluid: 25% Propylene Glycol Water (PGW) on the secondary loop and water in the primary loop of the

heat exchanger. • Heat generated per memory DIMM: Minimum of 6W and a maximum of 14W • Facilities flow rate for primary fluid loop: 360lpm for an end-of-row CDU. • Server systems fluid exit temperature (Tfluid): below 65°C • Components cooled by Passive Cooling Loop (PCL) are assumed to have 100% heat to fluid.


36

The chassis minimum flow rate and maximum inlet temperature numbers are shown in Figure 19 and Figure 20 and must not be exceeded. For example, a combination of 10°C inlet temperature and 1lpm passes but 30C and 1lpm automatically fails.

The following table lists the specific equipment used in testing the thermal requirements:

Table 17. Equipment used for simulation

Part type Vendor Model Quick disconnect for the chassis Staubli* SCG06

Quick disconnect for the rack Staubli* SCG20 End-of-row heat exchanger CoolIT* CHX750

Caution: Configuration for maximum coolant flow rate must not exceed 4.6lpm per chassis or 1.15lpm per compute module. Coolant flow rates above this value can lead to premature wear and corrosion in the chassis plumbing and the liquid cooling loop in the compute modules.

Figure 19. Minimum flow rate per rack


37

Figure 20. Minimum flow rate per rack

Figure 21. Pressure impedance per system

Note: Pressure drop from rack manifold quick connects are not included in Figure 21.

y = 0.4619x2 + 2.5706x

0

2

4

6

8

10

12

14

16

0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5

Pres

sure

Dro

p (p

si)

FLowrate (lpm) per system


38

6.3 Thermal Requirements Based on CDU Selection (End of Row-Based)

6.3

The tables in this section show the maximum number of chassis that can be cooled using a single end-of-row CDU at different ASHRAE W conditions for facility water.

6.3.1 End-of-row CDU When installed in an ASHRAE compliant facility, the end-of-row CDU (CoolIT* CHx750) can cool between 4 to 8 racks with 24 systems each based on DIMM thermal power and primary fluid supply temperature. The following tables list the details of the required parameters for each DIMM type’s configuration. The values in the tables apply to racks with perforated or solid doors, and cabinet configurations where the AHU is on a separate fluid loop that does not affect the CDU supply temperature. The values are also based on a two pump, redundant CDU configuration, with fault tolerant fail-over in the event of a CDU failure.

Table 18. End-of-row CDU cooling for systems with 16GB DIMMs

Table 19. End-of-row CDU cooling for systems with 64GB DIMMs

6.4 Air cooling recommendations The reference rack and cabinet configurations described assume 84% of the heat generated is captured by the liquid cooling elements. It is recommended to select an adequate air-cooling solution to capture the remaining heat. The following is a list of requirements for air cooling planning:

• Minimum system inlet temperature: 5°C • Maximum system inlet temperature: 35°C • Maximum system exit temperature: 55°C (from nodes into power supplies) • Air flow rate per system:

o Systems with 1U compute modules: 145 cfm o Systems with 2U compute modules: 168 cfm

• 88 cubic feet per minute (cfm) per system = 52m3/hr. (at 35°C ambient temperature) • Maximum supported cfm at 100% fan speed is ~290 per chassis = 171 m3/hr.


39

Additional References

For additional information about the Intel® Server System S9200WK Product Family or any of its supported accessories, refer to the following resources available at http://www.intel.com/support.

Table 20. Product family reference collaterals

For this information or software Use this document or software

For in-depth technical information about the product family

Intel® Server System S9200WK Product Family Technical Product Specification Intel® Remote Management Module 4 (Intel® RMM4) and BMC Embedded Web

Console User Guide Intel® Remote Management Module 4 Technical Product Specification Intel® Server System BIOS Setup Utility Guide Intel® Server Platform Specification Update

For system integration and service instructions Intel® Server System S9200WK Product Family Setup and Service Guide

For server configuration guidance and compatibility

Intel® Data Center Blocks Configurator (orderconfigurator.intel.com) Intel® Server System S9200WK Product Family Configuration Guide

For system firmware updates, onboard device drivers, and software

Download Center (https://downloadcenter.intel.com/search?keyword=S9200WK)

For system power budget guidance Intel® Server System S9200WK Product Family Power Budget and Thermal Configuration Tool

For a complete list of supported processors, memory, add-in cards, and peripherals

Intel® Server System S9200WK Product Family support site https://www.intel.com/content/www/us/en/support/products/192334/server-products/server-systems/intel-server-system-s9200wk-family.html

Product warranty Information Warranty Terms and Conditions (https://www.intel.com/content/www/us/en/support/services/000005886.html)

Safety and Regulatory Compliance Information Intel® Server System S9200WK Product Family Technical Product Specification

http://www.intel.com/support

https://orderconfigurator.intel.com/

https://downloadcenter.intel.com/search?keyword=S9200WK

https://www.intel.com/content/www/us/en/support/products/192334/server-products/server-systems/intel-server-system-s9200wk-family.html?

https://www.intel.com/content/www/us/en/support/products/192334/server-products/server-systems/intel-server-system-s9200wk-family.html?

https://www.intel.com/content/www/us/en/support/services/000005886.html


40

Appendix A. Glossary

Term Definition

2U Two rack units (3.5 in.)

AHU Air Handler Unit

Cabinet Two racks with an Air Handler Unit

CDU Cooling distribution unit

Chassis Casing containing the server compute modules and fans/liquid cooling plumbing

Cluster A group of three or more racks or cabinets

CRAC Computer Room Air Conditioning Unit

DIMM Dual inline memory module

HCR Heat Capture Rate

HPC High Performance Computing

PDU Power Distribution Unit

PGW Propylene Glycol Water

PSU Power Supply Unit

Rack A server system rack

TDP Thermal Design Power

Documents

Intel® Server System S9200WK...Intel® Server System S9200WK Liquid Cooled Rack Reference Design Guide Guidelines and requirements to design rack and cabinet level solutions with