Optimizing data centers for high-density computing technology brief, 2nd edition

Abstract.............................................................................................................................................. 2 Introduction......................................................................................................................................... 2 Power consumption and heat load ......................................................................................................... 2

Power consumption .......................................................................................................................... 2 Heat load........................................................................................................................................ 5

The Need for Planning ......................................................................................................................... 7 Optimizing the effectiveness of cooling resources .................................................................................... 8

Raised floors.................................................................................................................................... 8 Perforated tiles ............................................................................................................................. 8 Air supply plenum ........................................................................................................................ 9

Racks.............................................................................................................................................. 9 Cooling footprint ........................................................................................................................ 10 Internal airflow........................................................................................................................... 10 Hot and cold aisles..................................................................................................................... 11 Rack geometry ........................................................................................................................... 11

Computer room air conditioners....................................................................................................... 12 Capacity of CRAC units .............................................................................................................. 12 Placement of CRAC units ............................................................................................................. 12 Discharge velocity ...................................................................................................................... 13

Airflow distribution for high-density data centers.................................................................................... 14 Ceiling return air plenum ................................................................................................................ 15 Dual supply air plenums.................................................................................................................. 15 Ceiling-mounted heat exchangers .................................................................................................... 16

Advanced thermal management techniques .......................................................................................... 16 Static Smart Cooling....................................................................................................................... 17 Dynamic Smart Cooling .................................................................................................................. 18

Conclusion........................................................................................................................................ 19 For more information.......................................................................................................................... 20

Abstract This paper describes factors causing the increase in power consumption and heat generation of computing hardware. It identifies methods to optimize the effectiveness of cooling resources in data centers that are deploying high-density equipment or that are already fully populated with high-density equipment. The intended audience for this paper includes IT managers, IT administrators, facility planners, and operations staff.

Introduction From generation to generation, the power consumption and heat loads of computing, storage, and networking hardware in the data center have drastically increased. The ability of data centers to meet increasing power and cooling demands is constrained by their designs. Most data centers were designed using average (per unit area) or "rule of thumb" criteria, which assume that power and cooling requirements are uniform across the facility. In actuality, power and heat load within data centers are asymmetric due to the heterogeneous mix of hardware and the varying workload on computing, storage, and networking hardware. These factors can create "hot spots" that cause problems related to overheating (equipment failures, reduced performance, and shortened equipment life) and drastically increase operating costs.

The dynamic nature of data center infrastructures creates air distribution problems that cannot always be solved by installing more cooling capacity or by using localized cooling technologies. A more sophisticated scientific method can help to find the most effective solutions. Research at HP Laboratories has found that proper data center layout and improved Computer Room Air Conditioner (CRAC)1 utilization can prevent hot spots and yield substantial energy savings.

This paper is intended to raise awareness of present and future challenges facing data centers beginning to deploy or already fully populated with high-density equipment. This paper describes power consumption and heat load, recommends methods to optimize the effectiveness of data center cooling resources, and introduces thermal management methods for high-density data centers.

Power consumption and heat load In the past when data centers mainly housed large mainframe computers, power and cooling design criteria were designated in average wattage per unit area (W/ft2 or W/m2) and British Thermal Units per hour (BTU/hr), respectively. These design criteria were based on the assumption that power and cooling requirements were uniform across the entire data center. Today, IT managers are populating data centers with a heterogeneous mix of high-density hardware as they try to extend the life of their existing space. This high-density hardware requires enormous amounts of electricity and produces previously unimaginable amounts of heat.

For example, IT infrastructures are now using 1U dual-processor and 4U quad-processor ProLiant blade servers that can be installed together in a rack-mounted enclosure, interconnected, and easily managed. This high-density server technology lowers the operating cost per CPU by reducing management expenses and the requirements for floor space. Despite speculation that high-density server technology has the opposite effect on power consumption and heat load, a closer server-to-server comparison reveals that HP p-Class blades consume less power and generate less heat load.

Power consumption HP provides online power calculators to estimate power consumption for each ProLiant server. The power calculators provide information based on actual system measurements, which are more

1 CRAC units are sometimes referred to as air handlers.

2

accurate than using nameplate ratings. Figure 1 displays an HP power calculator, which is a macro-driven Microsoft Excel spreadsheet. Power calculators for all current HP ProLiant servers can be found at http://www.hp.com/configurator/calc/Power Calculator Catalog.xls.

Figure 1. Screen shot of ProLiant DL380 G4 server power calculator

3

From generation to generation, the power consumption of high-density servers is increasing due to the extra power needed for faster and higher capacity internal components. For example, the power required by a ProLiant DL360 G3 featuring a 3.0-GHz Intel® Xeon™ processor is 58 percent higher than its predecessor with a Pentium III 1.4 GHz processor (see Table 1).

Table 1. Increase in power consumption from generation to generation of ProLiant DL and BL servers

Rack Unit

(CPUs/Memory/Drives/Adapters)

Power Supply Rating per Server Generation Power Increase

1U

(2P, 4 GB, 2 HDD, 1 PCI)

DL360 G2

246 W

1.2 A @ 208V

840 BTU/hr

DL360 G3

389 W

1.6 A @ 208V

1328 BTU/hr

DL360 G4

460 W

1.9 A @208V

1570 BTU/hr

2U

(2P, 4 GB, 6 HDD, 2 PCI)

DL380 G2

362 W

1.8 A @ 208V

1233 BTU/hr

DL380 G3

452 W

2.2 A @ 208V

1540 BTU/hr

DL380 G4

605 W

3.0 A @ 208V

2065 BTU/hr

4U

(4P, 8 GB, 4 HDD, 3 PCI)

DL580 G1

456 W

2.3 A @ 208V

1556 BTU/hr

DL580 G2

754 W

2.7 A @ 208V

2573 BTU/hr

DL580 G3

910 W

4.0 A @ 208V

3106 BTU/hr

The table above compares the power consumption of individual servers; however, standard racks can house several of these servers. Estimating the power consumption of a rack of servers is more difficult because several variables (number of servers per rack, type and number of components in each server, etc.) contribute to the amount of power consumed. For racks, a very useful metric is power density, or power consumption per rack unit (W/U). Power density captures all of the key variables that contribute to rack densification. Figure 2 illustrates the overall power density trend from generation to generation of HP ProLiant BL and DL servers.

Figure 2. Power density of HP ProLiant BL and DL servers (Watts/U)

4

Heat load Virtually all power consumed by a computer is converted to heat. The heat generated by the computer is typically expressed in BTU/hr, where 1 W equals 3.413 BTU/hr. Therefore, once the power consumption of a computer or a rack of computers is known, its heat load can be calculated as follows:

Heat Load = Power [W] × 3.413 BTU/hr per watt

For example, the heat load for a DL360 G4 server is 460 W × 3.413 BTU/hr/W =1,570 mBTU/hr. The heat load of a 42U rack of DL 360 G4 servers is almost 65,939 BTU/hr, which is more than that of a typical one-story house. Table 2 lists the power requirements and heat loads of racks of density-optimized ProLiant DL and BL class servers. The table shows the trend toward higher power and heat load with rack densification.

Table 2. Power and heat loads of fully-configured, density-optimized ProLiant servers*

DL 580 G3 DL380 G4 DL360 G4 BL20p G2

ProLiant Server

Servers per Rack

10 21 42 48

Power 10 × 900W =

9.0 kW

21 × 575W =

12.08 kW

42 × 460W =

19.32 kW

6 × 2458W =

14.75 kW

Heat load 30,717 BTU/hr 41,212 BTU/hr 65,939 BTU/hr 50,374 BTU/hr

* These calculations are based on the product nameplate values for fully configured racks and therefore may be higher than the actual power consumption and heat load.

IT equipment manufacturers typically provide power and heat load information in their product specifications. HP provides a Rack/Site Installation Preparation Utility to assist customers in approximating the power and heat load per rack for facilities planning (Figure 3). The Site Installation Preparation Utility uses the power calculators for individual platforms so that customers can calculate the full environmental impact of racks with varying configurations and loads. This utility can be downloaded from http://h30099.www3.hp.com/configurator/calc/Site%20Preparation%20Utility.xls.

5

Figure 3. The ProLiant Class, Rack/Site Installation Preparation Utility available on the HP website

6

The Need for Planning The server densification trend is being driven by customers’ need to maximize the use of valuable data center floor space. Because concentrated heat generation is an inevitable byproduct of concentrated computing power, data centers must ensure adequate localized cooling capacity to match non-uniformly distributed heat loads. Most data centers have sufficient cooling resources already in place; their main challenge is directing cooling to racks that generate high heat loads.

For data centers that cannot afford to upgrade cooling capacity to handle concentrated heat loads, HP’s small form factor servers offer the flexibility to limit rack power density (kW/rack) based on the capacity of nearby cooling resources. For example, Figure 4 shows several rack configurations using six generations of ProLiant DL360 servers, each limited to 10 kW by controlling the number of servers per rack.

Some data center personnel believe that limiting rack power density neutralizes the space-saving benefits of densification because the racks are not full. However, from generation to generation the compute power per server is increasing faster than the corresponding increase in power consumption, resulting in higher efficiency. The ability to limit rack power density while increasing computing power can prolong the lifecycle of space-constrained infrastructures. Data centers that have sufficient capacity or that can afford to add cooling capacity can use HP small form factor servers to maximize rack computing power so that different facilities can be consolidated to reduce overall operating costs.

Figure 4. Limiting rack power density based on power consumption results in lower rack utilization.

The goal of all data centers is to optimize the effectiveness of existing cooling resources. The following section describes proven methods to achieve better airflow distribution. Given that power consumption and cooling demands will continue to increase, future data center designs will have to take a more holistic approach that examines cooling from the chip level to the facility level. The section titled “Advanced thermal management techniques” outlines breakthrough research by HP Laboratories to create intelligent data centers that dynamically redistribute compute workloads and provision cooling resources for optimum operating efficiency.

7

Optimizing the effectiveness of cooling resources This section recommends methods to optimize the effectiveness of cooling resources in raised floor infrastructures, a common configuration used in today's data centers.

Raised floors Most data centers use a down draft airflow pattern in which air currents are cooled and heated in a continuous convection cycle. The down draft airflow pattern requires a raised floor configuration that forms an air supply plenum beneath the raised floor (Figure 5). The CRAC unit draws in warm air from the top, cools the air, and discharges it into the supply plenum beneath the floor. Raised floors typically measure 18 inches (46 cm) to 36 inches (91 cm) from the building floor to the top of the floor tiles, which are supported by a grounded grid structure. The static pressure in the supply plenum pushes the air up through perforated floor tiles to cool the racks. Most equipment draws in the cold supply air and exhausts warm air out the rear of the racks. Ideally, the warm exhaust air rises to the ceiling and returns along the ceiling back to the top of the CRAC units to repeat the cycle. Many traditional data centers arrange rows of racks in the front-to-back layout shown in Figure 5. The mixing of cold and hot air in the aisles is very inefficient and wastes valuable cooling resources and energy. While this layout can work with lower power densities and heat loads, as the power density and heat load increase, the equipment inlet temperatures will begin to rise (shown in the figure) and overheat critical resources.

Figure 5. Traditional raised floor configuration with high-density racks arranged front to back

Perforated tiles Floor tiles range from 18 inches (46 cm) to 24 inches (61 cm) square. The percentage and placement of perforated floor tiles are major factors in maintaining static pressure. Perforated tiles should be placed in front of at least every other rack. In higher density environments, perforated tiles may be necessary in the front of each rack. Perforated tiles are classified by their open area, which may vary from 25 percent (the most common) to 56 percent (for high airflow). A 25 percent perforated tile provides approximately 500 cubic feet per minute (cfm) at a 5 percent static pressure drop, while a 56 percent perforated tile provides approximately 2000 cfm.

8

Air supply plenum The air supply plenum must be a totally enclosed space to achieve pressurization for efficient air distribution. The integrity of the subfloor perimeter (walls) is critical to prevent moisture retention and to maintain supply plenum pressure. This means that openings in the plenum perimeter and raised floor must be filled or sealed. Subfloor plenum dividers should be constructed in areas with large openings or with no subfloor perimeter walls.

The plenum is also used to route piping, conduit, and cables that bring power and network connections to the racks. In some data centers, cables are simply laid on the floor in the plenum where they can become badly tangled (Figure 6). This can result in cable dams that block airflow or cause turbulence that minimizes airflow and creates hot spots above the floor. U-shaped “basket” cable trays or cable hangers can be used to manage cable paths, prevent blockage of airflow, and provide a path for future cable additions. Another option is to use overhead cable trays to route network and data cables so that only power cables remain in the floor plenum.

Electrical and network cables from devices in the racks pass through cutouts in the tile floor to wireways and cable trays beneath the floor. Oversized or unsealed cable cutouts allow supply air to escape from the plenum, thereby reducing the static pressure. Self-sealing cable cutouts are required to maintain the static pressure in the plenum (Figure 7).

Figure 6. Unorganized cables (left) and organized cables (right) beneath a raised floor.

Figure 7. Self-sealing cable cutout in raised floor

Racks Racks (cabinets) are a critical part of the overall cooling infrastructure. HP enterprise-class cabinets provide 65 percent open ventilation using perforated front and rear door assemblies (Figure 8). To support the newer high-performance equipment, glass doors must be removed from older HP racks and from any third-party racks.

9

Figure 8. HP enterprise-class cabinets

Cooling footprint The floor area that each rack requires must include an unobstructed area to draw in and discharge air. Almost all HP equipment cools from front to rear so that it can be placed in racks positioned side-by-side. The cooling footprint (Figure 9) includes width and depth of the rack plus the area in front for drawing in cool air and the area in back for exhausting hot air.

Equipment that draws in air from the bottom or side or that exhausts air from the side or top will have a different cooling footprint. The total physical space required for the data center includes the cooling footprint of all the racks plus free space for aisles, ramps, and air distribution. Typically, a width of two floor tiles is needed in front of the rack, and a width of at least one unobstructed floor tile is needed behind the rack to facilitate cable routing.

Figure 9. Cooling footprint

Internal airflow Front and rear cabinet doors that are 65 percent open to incoming airflow also present a 35 percent restriction to air discharged by the equipment in the rack. Servers will intake air from the path of least resistance. Therefore, they will access the higher-pressure discharge air flowing inside the cabinet easier than they will access cooling air coming through the front of the cabinet. Some configurations such as those with extreme cable or server density may create a backpressure situation forcing heated exhaust air around the side of a server and back into its inlet. In addition, air from the cold isle or hot

10

isle can flow straight through a rack with open "U" spaces. Gaskets or blanking panels must be installed in any open spaces in the front of the rack to support the front-to-back airflow design and prevent these negative effects (Figure 10).

Figure 10. Airflow in rack without blanking panels (top) and with blanking panels (bottom)

Hot and cold aisles The front-to-rear airflow through HP equipment allows racks to be arranged in rows front-to-front and back-to-back to form alternating hot and cold aisles. The equipment draws in the cold supply air and exhausts warm air out the rear of the rack into hot aisles (Figure 11). The amount of space between rows of racks is determined as follows.

• Cold aisle spacing should be 48 inches, two full tiles, and hot isle spacing should be at least one full tile, 24 inches minimum. This spacing is required for equipment installation and removal and for access beneath the floor.

• Cold aisles should be a minimum of 14 feet apart center-to-center, or seven full tiles.

Figure 11. Airflow pattern for raised floor configuration with hot aisles and cold aisles

Rack geometry Designing the data center layout to form hot and cold aisles is one step in the cooling optimization process. Also critical is the geometry of the rack layout. Research by HP Laboratories has revealed that minor changes in rack placement can change the fluid mechanics inside a data center and lead to inefficient utilization of CRAC units. See the "Static Smart Cooling" section for more information.

11

Computer room air conditioners A common question with respect to cooling resources is how many kilowatts a particular CRAC unit can cool. Assuming a fixed heat load from the equipment in its airflow pattern, the answer depends largely on the capacity of the CRAC unit, its placement in the facility, and its discharge velocity.

Capacity of CRAC units The heat load of equipment is normally specified in BTU/hr. However, in the U.S., CRAC unit capacity is often expressed in "tons" of refrigeration, where one ton corresponds to a heat absorption rate of 12,000 BTU/hr. The "tons" capacity rating is measured at 80˚F; however, the recommended operating conditions for CRAC units are 70˚ to 72˚F and 50 percent relative humidity (RH). At 72˚F, the CRAC unit output capacity is considerably reduced. Furthermore, the tons rating is very subjective because it is based on total cooling, which is comprised of "sensible cooling" and "latent cooling."

Computer equipment produces sensible heat only; therefore, the sensible cooling capacity of a CRAC unit is the most useful value. For this reason, CRAC unit manufacturers typically provide cooling capacities as "total BTU/Hr" and "sensible BTU/Hr" at various temperatures and RH values. Customers should review the manufacturer's specifications and then divide the sensible cooling capacity (at the desired operating temperature and humidity) by 12,000 BTU/Hr per ton to calculate the useable capacity of a given CRAC unit, expressed in tons of cooling.

Cooling capacity is also expressed in volume as cubic feet per minute (cfm). The volume of air required is related to the moisture content of the air and the temperature difference between the supply air and return air (ΔT):

Cubic feet per minute = BTU/hr ÷ (1.08 × ΔT)

The cooling capacity calculations presented here are theoretical, so other factors must be considered to determine the effective range of a particular CRAC unit. The effective cooling range is determined by the capacity of the CRAC unit and the heat load of the equipment in its airflow pattern. Typically, the most effective cooling begins about 8 feet (2.4 m) from the CRAC unit. Although units with capacities greater than 20 tons are available, the increased heat density of today's servers limits the cooling range to approximately 30 feet or 9.1 m (Figure 12).

Figure 12. Cooling ranges of CRAC units

Placement of CRAC units The geometry of the room and the heat load distribution of the equipment determine the best placement of the CRAC units. CRAC units can be placed inside or outside the data center walls. Customers should consider placing liquid-cooled units outside the data center to avoid damage to electrical equipment that could be caused by coolant leaks.

CRAC units should be placed perpendicular to the rows of equipment and aligned with the hot aisles, discharging air into the supply plenum in same direction (Figure 13). This configuration provides the

12

shortest possible distance for the hot air to return to the CRAC units. Discharging in the same direction eliminates dead zones that can occur beneath the floor when blowers oppose each other. Rooms that are long and narrow may be cooled effectively by placing CRAC units around the perimeter. Large, square rooms may require CRAC units to be placed around the perimeter and through the center of the room.

Figure 13. CRAC units should be placed perpendicular to hot aisles so that they discharge cool air beneath the floor in the same direction.

Discharge velocity To force air from beneath the raised floor through the perforated tiles, the static pressure in the supply air plenum must be greater than the pressure above the raised floor. The velocity of the cooled air is highest near the CRAC unit because the entire flow is delivered through this area. The air velocity decreases as air flows through the perforated tiles away from the CRAC unit. The decrease in velocity is accompanied by an increase in static pressure with distance from the CRAC unit.

Excessive discharge velocity from the CRAC unit reduces the static pressure through perforated tiles nearest the unit, causing inadequate airflow (Figure 14). The static pressure increases as the high-velocity discharge moves away from the unit, thereby increasing the airflow through the perforated tiles. To counter this situation, airfoils under the raised floor can be used to divert air through the perforated tiles.2 Another option is to use a fan-assisted perforated tile to increase the supply air circulation to a particular rack or hot spot. Fan-assisted tiles can provide 200 to 1500 cfm of supply air.

2 From Changing Cooling Requirements Leave Many Data Centers at Risk. W. Pitt Turner IV, P.E. and Edward C. Koplin, P.E. ComputerSite Engineering, Inc.

13

Figure 14. Plenum static pressure greater than pressure above the floor (left). High-velocity discharge reduces static pressure closest to the unit (right).

Airflow distribution for high-density data centers To achieve an optimum down draft airflow pattern, warm exhaust air must be returned to the CRAC unit with minimal obstruction or redirection. Ideally, the warm exhaust air will rise to the ceiling and return to the CRAC unit intake. In reality, only the warm air closest to the intake may be captured; the rest may mix with the supply air. Mixing occurs if exhaust air goes into the cold aisles, if cold air goes into the hot aisles, or if there is insufficient ceiling height to allow for separation of the cold and warm air zones (Figure 15). When warm exhaust air mixes with supply air, two things can happen:·

• The temperature of the exhaust air decreases, thereby lowering the useable capacity of the CRAC unit.

• The temperature of the supply increases, which causes warmer air to be recirculated through computer equipment.

Figure 15. Mixing of supply air and exhaust air

14

Ceiling return air plenum In recent years, raised floor computer rooms with very high heat density loads have begun to use a ceiling return air plenum to direct exhaust air back to the CRAC intake. As shown on the right of Figure 16, the ceiling return air plenum removes heat while abating the mixing of cold air and exhaust air. Once the heated air is in the return air plenum, it can travel to the nearest CRAC unit intake. The return air grilles in the ceiling can be relocated if the layout of computer equipment changes.

Figure 16. Ceiling return air plenum

Dual supply air plenums As power and heat densities climb, a single supply air plenum under the raised floor may be insufficient to remove the heat that will be generated. High-density solutions may require dual supply air plenums, one above and one below (see Figure 17). In this configuration, additional supply air is forced downward in the cold aisle.

Figure 17. Dual air supply plenum configuration for high-density solutions

15

Ceiling-mounted heat exchangers The drive to maximize the compute density of data centers has prompted research and development of alternative cooling methods that do not require floor space.3 Figure 18 shows a representation of an alternate cooling method using modular air-to-liquid heat exchangers in the ceiling. The heat exchanger units collect the hot air exhaust from the racks and cool it using circulated chilled water. The units eject the cool air downward by using fan trays located over on the intake side of each rack.

The advantage of this approach is the proximity of the heat exchangers to the racks. With this scheme, rack cooling can be localized. Unique mechanical design ideas, such as the ability to move the intake and exhaust sections, have been implemented in the heat exchangers to direct airflow to and from the racks. Additionally, modular heat exchangers offer the flexibility to scale the cooling as needed. More importantly, ceiling-mounted heat exchangers save revenue-generating floor space and allow the raised floor to be mainly used for cable distribution.

Figure 18. Ceiling-mounted air-to-liquid heat exchangers

Advanced thermal management techniques Heat loads vary throughout a data center due to the heterogeneous mix of hardware types and models, changing compute workloads, and the addition or removal of racks over time. The variation in heat load is too complex to predict intuitively or to address by adding cooling capacity.

HP Laboratories has devised two thermal analysis approaches—Static Smart Cooling4 and Dynamic Smart Cooling—that model heat distribution throughout a data center using computational fluid dynamics (CFD). Static Smart Cooling uses CFD modeling to aid planners in designing the physical layout of the data center for optimum distribution of cooling resources and heat loads. Static Smart Cooling can also predict the changes in heat extraction of each CRAC unit when the rack layout and equipment heat load are varied.

Dynamic Smart Cooling offers a higher level of automated facility management. It enables intelligent data centers that dynamically provision cooling resources to match the changing heat dissipation of computing, networking, and storage equipment. It also redistributes compute workloads based on the most efficient use of cooling resources within a data center or a global network of data centers.

3 For more information, please read “Computational Fluid Dynamics Modeling of High Compute Density Data Centers to Assure System Inlet Air Specifications” at http://www.hpl.hp.com/research/papers/power.pdf. 4 For more information, please read “Thermal Considerations in Cooling Large Scale High Compute Density Data Centers” at http://www.hpl.hp.com/research/papers/2002/thermal_may02.pdf.

16

Static Smart Cooling Static Smart Cooling uses CFD modeling to determine the best layout and provisioning of cooling resources based on fixed heat loads from data center equipment. The heat extraction of each CRAC unit is compared to its rated capacity to determine how efficiently (or inefficiently) the CRAC unit is being used, or "provisioned."

The provisioning of each unit in the data center is presented as a positive or negative percentage as follows:

• An under-provisioned CRAC unit (positive percentage) indicates that the cooling load is higher than the capacity of the unit.

• A closely provisioned CRAC unit (small negative percentage) signifies that the cooling load is less than but reasonably close to the capacity of the unit, leading to efficient use of energy resources.

• An over-provisioned CRAC unit (large negative percentage) operates significantly below the capacity of the unit. This results in wasted energy if operation of the unit cannot be adjusted to match the lower cooling load.

For example, Figure 19 shows the row-wise distribution of heat loads (41 kW to 182 kW) for a combination of compute, storage, and networking equipment in a typical raised floor data center with four CRAC units. The CFD model shows that the provisioning of the CRAC units is completely out of balance.

Figure 19. Poorly provisioned CRAC units

17

In Figure 19, the 102-kW row and the 182-kW row have been repositioned to better distribute the heat load. This CFD model shows that the CRAC units are now provisioned within 15 percent of their capacity.

Figure 19. Statically provisioned CRAC units

Dynamic Smart Cooling A “smart” data center requires a distributed monitoring system and a feedback control system that continually provisions the cooling resources based on the workload distribution.5 Computing resources must be pooled and virtualized—rather than dedicated to a particular user or application—so that workloads can be allocated dynamically. The control system schedules compute workloads across racks of servers in a way that minimizes energy use and maximizes cooling efficiency. The computing resources not in use are put on standby to improve operating efficiency.

Due to the high airflow rates in data centers, thermal management is achievable only if hot and cold air mixing is minimal. The hot air must return to the CRAC units with minimal infiltration into the cold zones because such mixing increases the inlet temperatures at the racks. Researchers have developed dimensionless parameters, known as Return Heat Index (RHI) and Supply Heat Index (SHI) that can be used as control points to allocate compute workloads and cooling to minimize energy use.6 RHI denotes the degree of mixing of the cold air with the hot return air to the CRAC units. SHI is a measure of heat infiltration into cold aisles. SHI is the primary control parameter used to minimize the energy required to meet inlet air specifications.

Dynamic Smart Cooling is possible for a data center with the following features:

• Distributed sensors such as - temperature sensors on the racks measuring supply and exhaust air temperature of the systems - temperature sensors in the aisles measuring three dimensional temperature distribution in the

data center - temperature sensors at the CRAC return and supply - pressure sensors in the air distribution plenum

5 Patel, C.D., Sharma, R.K, Bash, C.E., Beitelmal, A, Friedrich, R., “Smart Cooling of Data Centers,” July 2003, IPACK2003-35059, Proceedings of IPACK03- International Electronics Packaging Technical Conference and Exhibition, Maui, Hawaii. 6 Sharma R, Bash, C.E, Patel, C.D, June 2002, “Dimensionless Parameters for Evaluation of Thermal Design and Performance of Large Scale Data Centers,” AIAA-2002-3091, American Institute of Aeronautics and Astronautics Conference, St. Louis, MO.

18

- sensors that measure power drawn by machines • Variable air flow devices to modulate flow work, and variable cooling coil temperature in the

CRACs to modulate thermodynamic work • Data aggregation system that

- collects sensed data from all locations - visually presents the real time power draw - calculates data center control parameters: RHI and SHI

• Control system that modulates the variable air conditioning resources through a control algorithm for a given distribution of workloads (heat loads)

• Data center manager (computerized system) that uses thermal policies to distribute workloads in the data center

At the time of this writing, HP Laboratories is developing the control system and the data center manager, and plans to report its progress in future papers.

Conclusion The growing power consumption of computing components requires modular data center designs with sufficient headroom to handle increasing power and cooling requirements. To determine actual requirements, facilities planners must consider several factors, including room geometry and the capacity and placement of the CRAC units. Planners must also give special attention to factors that affect airflow distribution, such as supply plenum static pressure, airflow blockages beneath raised floors, and configurations that result in airflow mixing in the data center.

HP is a leader in the thermal modeling of data centers. HP Professional Services can work directly with customers to optimize existing data centers for more efficient cooling and energy consumption. The modeling services can also be used to confirm new data center designs or predict what will happen in a room when certain equipment fails. As long as the data center has the power and cooling resources to support the expected loads, Static Smart Cooling can rectify cooling problems as well as enhance the overall efficiency of air conditioning resources. In most cases, the energy savings alone may pay for the cost of the service in a relatively short period.

19

For more information For additional information, refer to the resources detailed below.

Resource description Web address

Thermal Considerations in Cooling Large Scale High Compute Density Data Centers white paper

http://www.hpl.hp.com/research/papers/2002/thermal_may02.pdf

HP Rack/Site Installation Preparation Utility

http://h30099.www3.hp.com/configurator/calc/Site%20Preparation%20Utility.xls

Power calculators http://h30099.www3.hp.com/configurator/calc/Power%20Calculator%20Catalog.xls

© Copyright 2004, 2005 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. The only warranties for HP products and services are set forth in the express warranty statements accompanying such products and services. Nothing herein should be construed as constituting an additional warranty. HP shall not be liable for technical or editorial errors or omissions contained herein.

Intel and Xeon are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and other countries.

TC050901TB, 9/2005