Modeling Rack and Server Heat Capacity in a Physics Based Dynamic

CFD Model of Data Centers

Sami Alkharabsheh, Bahgat Sammakia 10/28/2013

ES2 Vision

2

Toward a full physics-based experimentally verified 3D

computational fluid dynamics model for data centers

To create electronic systems that are self sensing and regulating, and are optimized for energy efficiency at any desired performance level

Project Vision

Outline

Introduction Physics Based Steady State Baseline Model CRAC model

Server model

Tile model

Dynamic Model- Server Heat Capacity Effect Server level model

Room level model

Case studies

Conclusions and Future Work

3

Introduction

EPA (2007):1.5 % of total U.S. electricity consumption in 2006. (Total cost of $4.5 billion)

Datacenter Dynamics (2012) Global Census : power requirements grew by 63% globally to 38 GW from 24 GW in 2011.

Uptime Institute 2012 Data Center Industry Survey: PUE>1.8 for more than 55% of data centers

4

0%

2%

4%

6%

8%

10%

12%

14%

2.5

<=

2.4

-2.4

9

2.3

-2.3

9

2.2

-2.2

9

2.1

-2.1

9

2.0

-2.0

9

1.9

-1.9

9

1.8

-1.8

9

1.7

-1.7

9

1.6

-1.6

9

1.5

-1.5

9

1.4

-1.4

9

1.3

-1.3

9

1.2

-1.2

9

1.1

-1.1

9

1.0

9>=

Re

spo

nse

pe

rce

nta

ge

PUE

M. Iyengar and R. Schmidt, “Energy Consumption of Information Technology Data Centers”, 2010.

M. Stansberry and J. Kudritzki, “Uptime Institute 2012 Data Center Industry Survey,” Uptime Institute, 2013.

IT HVAC Cooling

Others

Nature of Problem

5

Solutions for improving the energy efficiency in data centers have been isolated

• Performance is not proportional to power

• Server overprovisioning is a common practice

• In real time, cooling is

difficult to control due to long lag times

•Complexity of transport in data centers

•Overprovisioning is commonly used for safe operation

Cooling Power

System-level and holistic solutions are a MUST

Fromtimes.com treehugger.com

Bench Mark Numerical Model

6

Raised Floor

Rack

Perforated tile

CRAC

Parameter Value

Room size 6.05 m x 13.42 m x 3.65 m

Plenum depth 0.6 m

Tile perforation ratio 50%

Perforated tiles area 0.61 m x 0.61 m

CRAC fan speed 100%

CRAC Model

Based on manufacturer data

Liebert 114D CW

Operating fan curve is obtained from the manufacturer, Liebert Consulting

The CRAC model is calibrated such that the flow rate can be predicted accurately at different operating pressures

7

* Alkharabsheh et al. “Utilizing Practical Fan Curves in CFD Modeling of Data Centers,” SEMITHERM2013.

Emersonnetworkpower.com

0 0.5 1 1.5 2 2.5 3 3.5 4

x 104

0

0.5

1

1.5

2

2.5

3

3.5

Flow rate (CFM)

Sta

tic p

ressu

re (

in. H

2O

)

CR

AC

in

tern

al re

sis

tan

ce

Calibrated operating point

Uncalibrated operating point

Server Model

8

2U server for

testing

A standard testing procedure following the AMCA 210-99 guidelines are used to measure the pressure fan curves

9 RU server simulators (load banks) and a 2 RU commercial server are tested

The measured fan curves include the internal resistance of the server

The measured fan curve can be imbedded directly into the CFD

Flow bench

apparatus

0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45-50

0

50

100

150

200

250

Flow rate (m3/s)

Sta

tic p

ressu

re (

Pa

)

2 RU server

9 RU load bank

Tile Model

The CFD tile model is validated using experimental data in Schmidt et al.*

The CFD tile model is modified to compensate for the momentum loss in the CFD flow resistance model

The CFD tile model is able to capture the tile flow distribution and can be used in room level simulations

9

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15-0.1

0

0.1

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15-0.1

0

0.1

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15-0.1

0

0.1

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15-0.1

0

0.1

Tile

Airflow

rate

(m

3/s

)

Row A

Row B

Row D

Row C

Solid line: experimental data, Dashed line: CFD results *Experimental data: Schmidt et al, “Measurements and Predictions of The Flow Distribution Through Perforated Tiles in Raised-Floor Data Center,” InterPACK2001

Computerfloorpros.com

Steady State Room Level Simulations

In addition to affecting the power dissipation, the servers power scenario also the airflow pattern by operating

The room can be underprovisioned/ overprovisioned based on the servers power level

Several parametric studies can be conducted using this model

10

15 17.5 20.5 24 28.1 32.9 38.5 45

Tem

pera

ture

(C)

15 kW/ rack

20 kW/ rack

32 kW/ rack

* Alkharabsheh et al. “Numerical Steady State and Dynamic Study in a Data Center Using Calibrated Fan Curves for CRACs and Servers,” InterPACK2013

Simple Dynamic Model

The thermal capacity of the equipment is not taken into account

Complete CRAC failure simulated at 20 seconds

Supporting the CRAC blower with backup power provides the room with extra cooling and time that can be utilized in increasing the reliability of operation

11

0 10 20 30 40 50 60 70 80 9020

30

40

50

60

70

80

90

100

110

Time (s)

Inle

t te

mpera

ture

(degC

)

No backup power

Blower backup power

Failure

Critical temperature

Unused plenum cold air

* Alkharabsheh et al. “Numerical Steady State and Dynamic Study in a Data Center Using Calibrated Fan Curves for CRACs and Servers,” InterPACK2013

Server Heat Capacity

The server level CFD model is developed based on the lumped mass approximation

Experimental data is used to calibrate and validate the server level CFD model

An increase in the rate of change in temperature is observed at low values of heat capacity until instantaneous change in temperature is noticed when server heat capacity is completely neglected

12

0 100 200 300 400 500 600 700 800 9003

3.5

4

4.5

5

5.5

6

Time (s)

T

se

rve

r

Exp. data [*]

CFD model

0 100 200 300 400 500 600 700 8003

3.25

3.5

3.75

4

4.25

4.5

4.75

5

Time (s)

T s

erv

er

Exp. data [*]

CFD model

0 100 200 300 4003

3.5

4

4.5

5

Time (s)

T

se

rve

r

No HC

1% Cap.

10% Cap.

50% Cap.

100% Cap.

120% Cap.

150% Cap.

*Ibrahim et al., "Thermal Mass Characterization of a Server at Different Fan Speeds," ITHERM2012.

Room Level Model

The detailed rack model is capable of hosting the server model, blanking panels, leakage through the mounting rails, and internal supports

Each server consists of an experimentally characterized fan curve and thermally calibrated heated mass

Each rack is populated with twenty of the 2 RU servers

13

n=20

n=1n=2

Server

Blanking panel

Mounting rails

Raised

Floor

Rack

Perforated

tile

CRAC

Detailed rack model

Case I: Servers Shutdown

It is assumed that all the servers inside the modular data center are shutdown at time 20 seconds

Three different room level models are compared in this transient analysis

Including the servers heat capacity is crucial in dynamic modeling. However, the heat capacity of the rack chassis can be neglected without affecting the accuracy of the results and reducing the computational time

14

0 200

20

Time (s)

Pow

er

(kW

/rack)

0 500 1000 1500 2000 25000

0.2

0.4

0.6

0.8

1

Time (s)

Rack inle

t te

mpeatu

re

No HC

Servers HC Only

Servers & Racks HC

Where: sso

ss

TT

TTT

ˆ

Case II: Server Power Short Pulses

Fluctuations in the dissipated power is simulated in the form of 30 second pulses

The temperature increases immediately in the model if we ignore the heat capacity

The heat capacity damps down the effect of short duration power fluctuations on the inlet temperatures

15

30 120180 100010

15

Time (s)

Pow

er

(kW

)

0 200 400 600 800 10000

0.2

0.4

0.6

0.8

1

Time (s)

Rack A

1 inle

t te

mpera

ture

Temperature without HC

Temperature with HC

Conclusions and Future Work

Experimentally validated models of different data center components are developed

A steady state and dynamic, physics based, room level CFD model for a bench mark data center is developed

It is found that the heat capacity of the servers affects the rate of change in temperature significantly

The effect of rack frames heat capacity is found to be small and can be neglected in room level simulations

Future work will include adding cooling unit heat capacity

16

Acknowledgement

This material is based upon work supported by the National Science Foundation under Grants No.1134867 and CNS-1040666

17