Sustainable Computing: Informatics and Systems 1 (2011) 134150

Contents lists available at ScienceDirect

Sustainable Computing: Informatics and Systems

journa l homepage: www.e lsev ier .com

Integrating cooling awareness with thermal awadata centers

Ayan Ban , SaImpact Lab Sch sity, Te

a r t i c l

Article history:Received 19 DReceived in reAccepted 17 F

Keywords:Cooling awareSpatial job schedulingEnergy efciencyDynamic cooling modelEnergy efciency metrics and bounds

ata cetheir; grehis efenessaper

cooling management algorithm, Highest Thermostat Setting (HTS). HTS is aware of dynamic behavior ofthe Computer Room Air Conditioner (CRAC) units and places jobs to reduce cooling demands from theCRACs. HTS also dynamically updates the CRAC thermostat set point to reduce cooling energy consump-tion. Further, the Energy Inefciency Ratio of SPatial job scheduling (a.k.a. job placement) algorithms, alsoreferred as SP-EIR, is analyzedby comparing the total (computing+ cooling) energy consumption incurredby the algorithms with the minimum possible energy consumption, while assuming that the job start

1. Introdu

High Percomputatioissue in contion facilitieIndeed, the2% of the toreach 4% inhas been ofenergy canenergy) to k

This workand Intel Corpand Thermal-Ain IGCC 2010 wdifferent typessimulation res

CorresponE-mail add

2210-5379/$ doi:10.1016/j.times are already decided tomeet the Service Level Agreements (SLAs). This analysis is performed for twocooling models, constant and dynamic, to show how the constant cooling model assumption in previousresearch misses out on opportunities to save energy. Simulation results based on power measurementsand job traces from the ASU HPC data center show that: (i) HTS has 15% lower SP-EIR compared to LRH,a thermal-aware spatial scheduling algorithm; and (ii) in conjunction with FCFS-Backll, HTS increasesthe throughput per unit energy by 6.89% and 5.56%, respectively, over LRH andMTDP (an energy-efcientspatial scheduling algorithm with server consolidation).

2011 Elsevier Inc. All rights reserved.

ction

formance Computing (HPC) applications require highn capabilities, often in the range of teraops. A majortemporary data centers, hosting such high computa-s, is the high energy consumption in their operations.data centers energy consumption amounted to nearlytal energy budget of the US in 2007 and is expected to2011 [1]; as such, greening the data center operationsutmost interest over the years [210]. Up to half of thisbe attributed to cooling the data centers (i.e. coolingeep the operating temperatures within the computing

was funded in parts by NSF (CNS#0855277 and CSR#0834797), SFAz. This paper extends the conference version titled Cooling-Awareware Workload Placement for Green HPC Data Centers publishedith: (i) comprehensive thermodynamic details, (ii) incorporation ofof CRAC behavior in the scheduling decision making, (iii) extensive

ults, and (iv) improved related works.ding author.ress: sandeep.gupta@asu.edu (S.K.S. Gupta).

servers manufacturer specied redline temperatures. This paperfocuses on a cyber-physical oriented coordinated job and coolingmanagement in HPC data centers to reduce the total (i.e. computingand cooling) energy consumption of the data centers.

The cooling energy depends on two factors: (i) the coolingdemand, which is driven by the power distribution and the redlinetemperature; and (ii) the cooling behavior, i.e. the behavior of theComputer RoomAir Conditioner (CRAC) unit (controlled by varyingthe thermostat setting), to meet the demand. A major concern inthis regard is the possible recirculation and intermixing of hot air,generatedby the servers running the jobs,with the cold air suppliedfrom the CRAC [7]. Recirculation of hot air depends on the data cen-ter layout and can cause hot-spots; thus potentially increasing thecooling demand.

Techniques to reduce the recirculation include: (i) thermal-aware workload (job) management in the cyber domain [58] thatis aware of the heat recirculation; and (ii) design layout and expen-sive physical infrastructure [11] to minimize, or even remove, anypotential recirculation. However, cooling energy reduction is notguaranteed by thermal-aware jobmanagement unless coordinatedwith the dynamicmanagement of CRAC; suchmanagement should

see front matter 2011 Elsevier Inc. All rights reserved.suscom.2011.02.003erjee, Tridib Mukherjee, Georgios Varsamopoulosool of Computing, Informatics, and Decision Systems Engineering Arizona State Univer

e i n f o

ecember 2010vised form 15 February 2011ebruary 2011

a b s t r a c t

High Performance Computing (HPC) ddensity and energy consumption ofattributed to cooling the data centerand cooling energy is imperative. To tof the cooling unit with thermal awarplacement) in HPC data centers. The p/ locate /suscom

re workload placement for HPC

ndeep K.S. Gupta

mpe, Arizona, USA

nters are becoming increasingly dense; the associated power-operation are increasing. Up to half of the total energy is

ening the data center operations to reduce both computingfect, this paper integrates awareness of the dynamic behaviorwhile performing spatial workload scheduling (i.e. workloadrst proposes a coordinated cooling-aware job placement and

A. Banerjee et al. / Sustainable Computing: Informatics and Systems 1 (2011) 134150 135

be cooling-aware, i.e. aware of the dynamic cooling behavior. Assuch, inmany existing data centers the cooling is over-provisioned;resulting in a high Power Usage Efciency (PUE).1 Further, there is nodenite way to evaluate the energy inefciencies of the jobmanage-ment algorithms and their dependencies on the heat recirculationand the coolingbehavior of theCRAC.Hence, the cost-benet trade-off of incorporating better physical and cooling infrastructure toreduce the

This papaware spatHPC dataceCRAC, (2) thtial schedulbehaviors,algorithm t

1.1. Overvie

The con

(i) Highestperforming, and(i.e. dytions wof enerHeat (Lschedusame tare bastraces o

(ii) An enealgorithrecircuciency Rthe totrespectassumithe Serof anyconsumlation (

(iii) Spatio-server twithditwo difFirst Seused sc[13]; anpreviouceptionto ensumented

SP-EIR cbenet of rof the infrasset temperament and coHTS, can incing SP-EIR aHPC data ce

1 PUE is thesumption to se

than that for LRH. Simulation results for the same data center fur-ther show that HTS, when combined with EDF (i.e. EDF-HTS), canachieve up to 15% energy savings over EDF-LRH [12], an energy-efcient and thermal-aware online spatio-temporal schedulingalgorithm ipower conMTDP, i.e. E

(MTconsl (i.ehputberpro

rform

ervi

diffstantantltiplnitthouliteraxtracon knper

ACsmherperaolinghis renestatraturhe Cespecers sstatsiderbasespats ofle ofgurevailantertmos-therst seAC tn in astat

spotss shod plastatlingto h2 con datemens thaomehedscherecirculation and improve cooling is unknown.er focuses on: (1) developing a cooling and thermalial job scheduling (i.e. job placement) algorithm fornters taking into account the dynamic behavior of theeoretically analyzing the energy inefciency of the spa-ing algorithm, when operated under different coolingand (3) developing a spatio-temporal job schedulinghat is cognizant of the cooling behavior of the CRAC.

w of contributions and results

tributions of this paper are summarized as follows.

Thermostat Setting (HTS) algorithm is developed, whichs cooling-aware and thermal-aware spatial job schedul-integrates such scheduling with cooling management

namic variation of CRAC thermostat setting). Simula-ere performed to compare the performance per unitgy consumption of HTS with that of Least RecirculatedRH), an energy-efcient thermal-aware online spatialling algorithm [12], by separately using them with theemporal scheduling algorithm. The simulation resultsed on the layout, equipment prole, and actual jobf the ASU HPC data center.rgy inefciency analysis of spatial job schedulingms is performed by taking into account the heatlation and cooling behavior. First, ametric, Energy Inef-atio of SPatial scheduling (SP-EIR), is dened to compare

al energy consumption incurred by the algorithm withto the minimum possible energy consumption, whileng that the job start times are already decided to meetvice LevelAgreements (SLAs). It is shown that the SP-EIRspatial scheduling algorithm decreases (i.e. the energyption is reduced) with a decrease in the heat recircu-i.e. with better data center design).temporal scheduling, i.e. deciding on when and in whicho execute a job, can be performed by integrating HTSfferent temporal scheduling algorithms. This paper usesferent temporal scheduling algorithms: (i) First Comerve with Backlling (FCFS-Backll), the most commonlyheduling algorithm in contemporary HPC data centersd (ii) Earliest Deadline First (EDF), which has been usedsly for energy efciency while meeting the users per-of job turn-around time (i.e. thedeadline) [12]. Further,re reduction in the computing energy, HTS can be aug-with power control techniques.

an be used by the data center designers to quantify theeducing the heat recirculation with respect to the costtructure. Further, the increase in the CRAC thermostattures can decrease the SP-EIR. Cooling-aware job place-ordinating with the dynamic CRAC management, as inrease CRAC thermostat set temperatures; hence reduc-nd consequently the energy consumption. For the ASUnter, HTS has an SP-EIR of 1.013, which is 15% lower

ratio of total power consumption over the computing power con-rvice the jobs in a data center.

PowerserverBacklthrougof numSuch imand pe

1.2. Ov

Two(1) cona conshas muat a deCRAC,viousheat-ecommThe temtheCRThe higair temless co

In tter, thethermotempefrom ttheir rin servthermobe conplaced

Theupdateexampin the three adata cethe lefof nonleftmolow CR

Eveusuallyof hottime. Ato avoithermothe coomostat

Fig.ment imanagensuretined tdispatcspatialf there is no power control in the servers. Even withtrol, EDF-HTS achieves 9% energy savings over EDF-DFused in conjunctionwithMinimumTotalData centerDP) spatial scheduling, which performs thermal-awareolidation [7]. Further, HTS, when combined with FCFS-. FCFS-Backll-HTS), can achieve up to 5.56% higherper unit of energy consumption (measured in terms

of jobs per second per Joule) over FCFS-Backll-MTDP.vement is signicantwhen scaled to annual cost savingsance benets of a data center.

ew of approach

erent models of CRAC unit operation are considered:t model, where the CRAC acts as a source of cool air attemperature, and (2) dynamic model, where the CRACe modes of operation and in each mode extracts heate rate from the data center. The constant model of thegh unrealistic, has been a common assumption in pre-ture [58,10,12]. However, CRACs can have multipletingmodes, as an empirical study shows [14] and as it isowledge in the refrigeration engineering community.ature of the cold supply air from the CRAC depends onode of operation and the thermostat set temperatures.

the thermostat set temperature the higher is the supplyture; thus producing less cooling and hence consumesenergy.egard, it is observed that, in an over-cooled data cen-rgy consumption can be reduced by increasing the CRACset temperatures. An upper bound on the thermostat setes is obtained from the required supply air temperatureRAC to ensure that all the servers can operate withintive redline temperatures. HTS is designed to place jobsuch that the heat recirculation effect is reduced and theset temperature requirements are increased. LRH [12] caned as a special case of HTS since LRH allows jobs to bed only on the heat recirculation effects.ial job scheduling in HTS is augmented with dynamicthe CRAC thermostat set temperatures. Fig. 1 shows ancooling-aware spatial job scheduling in HTS. As shown, there are two jobs requested to be placed in two of theble server. The leftmost server is in the portion of thewith high heat recirculation. As such, placing a job int server may lead to a hot spot. A traditional approachmal-aware spatial job scheduling can place a job in therver (Fig. 1a); thus leading to hot spots and requiringhermostat set temperature (18 C in the gure).dynamic cooling environment, the CRAC thermostat is

ically set to a lowvalue, to counterworst case situations(Fig. 1a); thus cooling is over-provisioned most of thewn in Fig. 1b, cooling-aware spatial job scheduling hascing jobs in the leftmost server; and thus the requiredtemperatures are increased. Further, coordinationwithmanagement can enable dynamically setting the ther-igh values.nceptualizes the coordinated job and cooling manage-a centers. The submittedworkload is provided to the jobtmodulewhere it rst gets temporally scheduledwhicht the jobs get the requested types of servers and are des-et the deadline. The temporally scheduled jobs are thenamong the requested servers in the data center by theduling module. The spatial scheduling attempts to place

136 A. Banerjee et al. / Sustainable Computing: Informatics and Systems 1 (2011) 134150

the jobs inthermostatenergy (comnation of this called theof which istwo perspe

(i) Spatial ptial jobwith resimizes tdependin questFig. 1. Example of cooling-aware job managem

Fig. 2. Information ow during coordinated job and cooling man

the requested types of servers such that: (1) the CRACcan be dynamically set, and (2) the total data centerputing + cooling) consumption is reduced. The combi-e spatial scheduling with dynamic thermostat settingcoordinated job and coolingmanagement, an example

the HTS algorithm.We evaluate HTS from the followingctives:

erspective: A generic expression for SP-EIR of any spa-scheduling (i.e. job placement) technique is obtainedpect to the optimal energy-efcient approach thatmin-he energy consumption. The actual values of the SP-EIRon the spatial scheduling algorithm and the data centerion.

(ii) Spatio-tall spatusing embenetswhen cing algo

1.3. Paper o

The restthe system

2 http://hpcent in HTS.

agement in HPC data centers.

emporal perspective: SP-EIR does not evaluate the over-io-temporal performance. Simulations were performedpirical data from theASUHPCdata center2 to show theof being aware of the CRAC dynamic behavior in HTS

ombined with FCFS-Backll and EDF temporal schedul-rithms.

rganization

of the paper is organized as follows. Section 2 describesmodel of the data center. The cooling-aware job place-

.asu.edu.

A. Banerjee et al. / Sustainable Computing: Informatics and Systems 1 (2011) 134150 137

ment probledescriptionspatial jobrecirculatiosimulationtemporal penergy-efcconcludes tfuture resea

2. System

This sectused in desrithm (i.e. HSP-EIR). Firs2.1, followeSection 2.3Section 2.4job manage

2.1. Physica

The data(Section 2.1tion 2.1.2).

2.1.1. DataContemp

ceilings fororganized iwith alternequipmentOften, in dawhich coolit enters the

The coolthe heatingair into theows throufrom the cooutlet. The hHowever, dhot air mayat various p

2.1.2. EquipThe CRA

each chassiotherwise wputing unit

hassiscapace CRAC

ancf a chAC seat fenteffe

C airhe ouhes tvatio

rat

actio

ir on

j=1(f

ir

r o

airn

j=1(a

sumeame, if racg outresul

al capacity of air owing through chassis i

nFig. 3. Heat transfer mechanisms in data center.

m is dened in Section 3 followedby theHTS algorithmin Section 4. Section 5 denes the SP-EIR metric forscheduling and describes its dependency on the heatn and CRAC thermostat setting. Section 6 presents thebased evaluation of HTS from both spatial and spatio-erspectives. Section 7 presents the related work onient job scheduling in data centers. Finally, Section 8he paper with a discussion on the various issues andrch directions.

model

ion provides the system model and assumptions to beigning the cooling aware spatial job scheduling algo-TS) and computing algorithm energy inefciency (i.e.t, the data center physicalmodel is described in Sectiond by the job and machine environment in Section 2.2.presents the constant and dynamic CRAC models andpresents their inter-dependencies with the data centerment and server power management.

l model

center physical model includes the physical lay-out of.1) and the computing equipment (server) safety (Sec-

center lay-outorary HPC data centers use raised oors and loweredcooling air circulation, with the computing equipmentn rows of 42U racks arranged in an aisle-based layout,ating cold aisles and hot aisles (Fig. 3). The computingis usually in blade-server form, organized in 7U chassis.ta centers, server racks are provided with chiller doors,down the hot air coming out of the blade servers beforedata center room [15].

ing of the data center room is done by the CRAC, a.k.aand ventilation air conditioner (HVAC). They supply cooldata center through the raised oor vents. The cool airgh the chassis inlet and gets heated up by convection

Fig. 4. Cthermalair at th

performature o(1) CRlated hplacemcoolingof CRAfrom tit reacconser

air ow

= fr a

+

a ai= li

+

It is asis the sHenceowinabove

Thermmputing equipment andhot air comes out of the chassisot air goes to the inlet of the CRACwhich cools it down.epending on the design of the data center, parts of therecirculate within the room affecting the thermal mapositions including the inlet of the CRAC and the chassis.

ment safetyC has to supply cold air so that the inlet temperature ofs does not exceed the redline temperature (Tred), whichould lead to throttling of the operation of the com-

an undesirable phenomenon with respect to HPC job

= li rac

If the inletamount ofplied by Tin

iobtained byCRAC to chathe thermachassis i isat time t, To

jlevel view of the air inlet, outlet and re-circulation. Here mCp is theity of air at the chassis outlet, r, and macCp is the thermal capacity ofoutlet, rac.

e (throughput and turnaround time). The inlet temper-assis i is a result of heat contributed from two sources:

upply at a temperature of T sup and (2) the recircu-rom the outlet of all other chassis. Due to equipmentwithin the data center the CRAC does not have uniformct everywhere. Hence, we assume that only li fractionsupply reaches to a chassis i. Further, the heated airtlet of a chassis j gets recirculated and a fraction aji ofhe inlet of the chassis i as shown in Fig. 4. Followingn of mass principle we get,

e through chassis i

n of air owing from CRAC to chassis i

wrate at theCRACoutlet

raction of air recirculation from chassis j

ow rate through chassis j)

w rate through chassis i

owrate at CRACoutlet

ji air ow rate through chassis j)

d that the mass ow rate at the outlet of each chassisand the specic heat is uniform throughout the room.and r are the thermal capacities per unit time of the airof the CRAC and a chassis, respectively, then from thet we get,+j=1

(aji r) (1)

temperature at the chassis i is Tini(t) at time t, then the

heat entering chassis i is the thermal capacity multi-(t). The heat input to chassis i from CRAC is similarlymultiplying the thermal capacity of air owing fromssis i by the supply temperature. For recirculated heat,l capacity of air coming out from chassis j and enteringto be multiplied by the outlet temperature of chassis jut(t). Hence using Eq. (1) we get,

138 A. Banerjee et al. / Sustainable Computing: Informatics and Systems 1 (2011) 134150

Heat input to chassis i in timedt

= heat input from CRAC at chassis i in timedt+ recirculated heat to chassis i from all other chassis in timedt

lirac

where dt isthe data cenbe constant

Further,its inlet temconservatio

Heat input

+Heat= Heat o

lirac

where Pi(t)execution, w

Vectorizdetermine T

Tin(t) = FT

where F =R[(R AR)i-th elemenan nn dTsup(t) is adimensionaonal matrixrespectivelydepends onthe chassiscenter.

For theshould be bredline con

Chassis inle

where T re

temperaturdata centerpaper.

2.2. Job and

Given tdescribes thstate-of-thesoftware fothat focusesbeing throucenter is thwhich decirun at.

A job suinput data,

runtime, and (d) other constraints such as a priority, and speciccomputing node preferences. A computing node is a chassis con-taining multiple blade servers. The job run-times are normallyoverestimated by the users [12]. We consider user estimated job

oundat dohasn.re aporay imtial (thr

s witureperakinsion.ther,eredingmple(dene jocen

hine.l is chaviodencptio

AC b

o typ

Consonstatemtheted iserv

utpu

at in

cTsup

Pcomh

at evork

lowsuchariaraturig. 5)raturlearl. In ry th

DynaCRA

, in tmo

h andt of+n

j=1ajir

Tini (t)dt = liracTsup(t)dt +

nj=1

ajirToutj (t)dt,

(2)

a small time interval where temperature throughoutter and power consumptions of chassis are assumed to.the outlet temperature of the chassis i can be related toperature and power dissipation using the principle ofn of energy as follows:

to chassis i in timedt

generated from chassis i in timedt

utput from chassis i in timedt

+n

j=1ajir

Tini (t)dt + Pi(t)dt = rTouti (t)dt, (3)

is the power generated by chassis i due to workloadhich is constant in the time interval dt.

ing Eqs. (2) and (3) and simultaneously solving them toini

for each i we get,

sup(t) + DP(t), (4)

(LRac + BR)1 R[(R AR)1 LRac], D = (LRac + BR)11

R1], L is an nn diagonal matrix in which thet of the i-th column contains the li for all i, B is

iagonal matrix whose diagonal elements are

jaji,n n dimensional vector {Tsup(t)}n, Tout(t) is an nl vector {Tout

i(t)}

i=1...n, and R and R ac are nn diag-populated with the r and rac values in the diagonal,. Eq. (4) shows that the chassis inlet temperaturethe fraction of CRAC supply temperature reachingand the amount of heat recirculation in the data

safe operation of each chassis, the inlet temperatureelow the redline temperature. Thus, using Eq. (4), thestraint can be expressed as follows,

t temp redline temp FT sup(t) + DP(t) Tred, (5)d is an n dimensional vector {Tred

i}n, Tred

iis the redline

e for chassis i, and n is the total number of chassis in the. Table 1 provides the list of scalar variables used in this

machine environment

he data center physical model above, this sectione job andmachine environment in the data centers. The-art in commercially availabledata centermanagementllows a conventional job queuing and issuing paradigmonoptimizing performance policymetrics, those usually

ghput and turn-around time. The user front-end of a datae submission interface, i.e. the interface of the scheduler,des when and where (i.e. what servers) the jobs to be

bmission usually provides: (a) the executable, (b) the(c) the number of servers it requires and the estimated

turnarjobs thrithmabortio

The(i) temdirectl(ii) spathe jobserverTo ensthis pasion msubmis

Furconsidbeginn(job coperiodsecutiva dataa macintervathe bedepenconsum

2.3. CR

Tw

2.3.1.In c

supplyture atgenerathe con

Heat o

= he ra

wherecenterviouswand altools sever, vtempeters (Ftempeunit) cmodestured b

2.3.2.The

plicityand lowthe higamountimes as the jobs deadlines. The scheduler aborts thenot complete by the deadline. Thus, a scheduling algo-

to ensure meeting of the job deadlines to avoid job

re two types of decision making for job scheduling:l (i.e. when to start the execution of the jobs), whichpacts the job throughput and turnaround times; andi.e. where to execute the jobs), which can also impactoughput and turnaround times if jobs are assigned toh low computing capabilities (e.g. processor speed).no degradation in throughput and turnaround time,focuses only on energy-aware spatial scheduling deci-g among the servers requested by the users during job

an event based decision making for job scheduling is. An event comprises arrival of new jobs (job arrival),of job execution (job start) and end of job executiontion). Interevent interval, also referred to as eventoted by the symbol h), is the time between two con-

b start and completion events. Computing power inter changes when a job starts or ends execution onTherefore, the computing power in any inter-eventonstant over time. The following sections will describer of the CRAC unit in the data center and the inter-

y of the cooling behavior with the computing powern.

ehavior

es of CRAC behavior is considered in this paper.

tant cooling modelnt coolingmodel, theCRAC, at all time, keeps a constantperature Tsup(t) at its outlet regardless of the tempera-inlet (i.e., Tsen(t) in Fig. 3). This means the entire power,n the data center room, gets removed by the CRAC. Fromation of energy at the CRAC, we get:

t from CRAC + heat generated in the data center roomput to the CRAC from the room

(t)dt + Pcomph

dt = racTsen(t)dt, (6)p is the total computing power consumption in the dataentperiodh. Thismodel is a commonassumption is pre-[58,12]. Such an assumptionmakes the analysis easiersteady state simulation of the data center using CFDas Flovent (http://www.mentor.com/products/). How-tions in the CRAC input and output over time frome sensor measurements in the ASU HPC data cen-suggest otherwise. The difference between these twoes (that determines the power extracted by the CRACy shows two distinct values indicating two operationaleality, the CRAC has a dynamic behavior, which is cap-e next model.

mic cooling modelC can havemany different modes of operation. For sim-his paper, we assume two operational modes viz. highdes. During its operation the CRAC oscillates betweenlowmodes. In eachmode, the CRAC extracts a constant

power Phighex and Plowex , respectively (Phighex > P

lowex ). From

A. Banerjee et al. / Sustainable Computing: Informatics and Systems 1 (2011) 134150 139

40

45

50

55

60

65

70

75

80

12/1800:00

35

40

tem

pera

ture

(F)

CRAC supply and input temperature over time

CRAC input temperature (left axis)CRAC supply temperature (left axis)

difference (right axis)

Fig. 5. Variatiomeasurementindicates the t

the energy

Heat input

= heat ex+heat

racTsen

where the CA mode

ates in a paextracts a ction of the ivariation isthe room Pwe get,

CRAC inlet tem

= CRAC inle

+ rate of c

= Tsen(0) +

Tsen(

where Tsen(the CRAC mCRAC is in mature Tth

high,

Tsen(t) goesswitches froence betweonly thehigbehavior.

Further,

below Tthlow

switch betwin the sameend of each

temperatures are given as follows:

max CRAC inlet tempRAC high threshold temptemp change due to operation in low mode for tsw time

senmax = Tthhigh +

Pcomph

Plowexrroom

tsw,

RACRACtemp

senmin =

CRACthe dCRAe CoeAC tlly gi

sup(t)

Tsen(The aput

ing mp(t))/n by:

= Ra

=ra

chnie inas fathe phe toPcom

h

ter-d

en th12/1812:00

12/1900:00

12/1912:00

12/2000:00

12/2012:00

12/2100:00

12/2112:00

12/2200:00

0

5

10

15

20

25

30

tem

pera

ture

diff

eren

ce (

F)

time (month/day)

ns in the CRAC input and output temperature based on actual sensors in the ASU HPC data center. The difference in these temperatureswo operational modes of the CRAC.

conservation in the CRAC, we get,

to the CRAC from the room

tracted by the CRAC

output by the CRAC to the room

(t)dt = Pxexdt + racTsup(t)dt, (7)

RAC operates in mode x {low, high}.epoch is the time duration for which the CRAC oper-rticular mode. In a mode epoch, since the CRAC onlyonstant amount of power Pxex, we expect a linear varia-nlet temperature over time [14]. The rate of the linearproportional to the difference in power generated incomph

, and the power extracted by the CRAC. Hence

perature at time t

t temperature at time 0

hange of temperature t Tsen(t)Rate of heat generation in room Rate of heat extraction by CRAC

Thermal capacity of room

= C+

Tmin C

= C+

T

2.3.3.For

on theand ththe CRnorma

COP(T

wheret [16].the comoperat(racTsu

is give

PAC (t)

Any teoperatTsup(t)CRACwith tperiod

2.4. In

Givt) = Tsen(0) +Pcomp

h Pxex

rroom t, (8)

0) is the initial input temperature of the CRAC whenoved to mode x and t is the duration for which theode x. If Tsen(t) goes above the high threshold temper-the CRAC switches from low to high mode and when

below the low threshold temperature Tthlow

, the CRACm high to lowmode (Tth

high> Tth

low). Generally the differ-

en the two threshold temperature is kept constant andh threshold temperature is varied tomodulate theCRAC

once the input temperature goes above Tthhigh

or goes

the CRAC takes an additional tsw amount of time toeen modes. During this tsw time the CRAC operatesmode as in the previous mode epoch. Hence, at themode epoch the maximum and minimum CRAC inlet

and the CRAsummarizepowermaningmodel fjob and powwork [12].

It shouldpower consply tempertemperaturutilization.heat inputkeep the tefore, it is imfrom the Cature alsotemperaturperature cais low, i.e.inlet templow threshold tempchange due to operation in high mode for tswtime

Tthlow

+ Pcomph

Phighexrroom

tsw.

(9)

power consumptionynamic mode the CRAC power consumption dependsC power mode (i.e. the power extracted by the CRAC)fcient Of Performance (COP) of the CRAC. The COP ofo supply air at temperature Tsup(t) at an instance t isven by

) = Tsup(t)

Tsen(t) Tsup(t)

t) is the CRAC input temperature (Fig. 3) at the instancebove assumption on the COP of the CRAC unit enablesation of the energy consumed by a cooling unit in theode x. Replacing Tsen(t) from Eq. (7), COP is given byPxex. The power consumption to run the CRAC at time t

te of heat extraction by CRACCoefcient of performance

= Pxex

COP(Tsup(t))

(Pxex)2

cTsup(t). (10)

que to reduce the CRAC power consumption has tolower modes, reducing the Pxex, and increase ther as possible. For the constant cooling model of theower consumption can be obtained by replacing Pxextal power dissipated in the data center at an eventp.

ependency of cooling and job management

e data center physical model, machine environment,C behavior in the previous subsections, this subsections the inter-dependencies among the cooling, job, andagement. In this section, we focus on the dynamic cool-or the CRAC as the dependencies of the constant cooling,er management have been discussed in our previous

be noted that as the data center utilization increases,umption at the chassis increases; requiring lower sup-ature to meet the redline (Eq. (5)) [12]. The supplye is maximum for 0% utilization andminimum for 100%Generally, however, if there is heat recirculation, theto the chassis increases; thus requiring lower Tsup(t) tomperature within the redline temperature [12]. There-portant to predict the maximum supply temperature

RAC. In a particular CRAC mode, the supply temper-changes linearly at the same rate as the CRAC inpute (Eq. (7)). It should be noted that the maximum tem-n be reachedwhen the power extraction from the CRACwhen it is operating in the low mode and the CRAC

140 A. Banerjee et al. / Sustainable Computing: Informatics and Systems 1 (2011) 134150

input temperature is maximum. Hence combining Eqs. (7) and (9)we get,

Tsupmax = Tthhigh +Pcomp

h Plowex

rroomtsw P

lowex

rac, (11)

It can be concluded from Eq. (11) that the maximum supplytemperature from the CRAC depends on the high thermo-stat settingHence, jobserver powputing powmanagemeneed to besupply tem(Eq. (5)).

We assuatures can ba constantsettings3 (i

this differefrom the CEq. (11)) whset point aextracted b

Tsupmin = Tthlow

= Tthhigh

Eq. (12)temperaturtially reducThis paperrithm, HTSwhen integsettings.

3. Problem

This papjobs on thedata center

Given aenvironmnd thewhich thCRAC wienergy ctimes are

It shouldmaking aspjobs to thetrol (i.e. dejob scheduNP-compleconsumptioheuristics p

3 In the restchanging both

servers [24] to reducing heat recirculation [7,12]. A heuristic spa-tial job-scheduling algorithm, called LRH, was developed in [12]that has two distinct characteristics: (i) LRH statically ranks thenodes so that the choice of nodes to place a job can be veryfast (ideal situation of dynamically re-calculating the nodes rankscan increase the complexity); and (ii) the static rank of a nodeis assigned based on the nodes contribution in heat recircula-

.e. the total recirculated heat from the node to all other

rankof ththe Hon:odesothemeratur

hest

s sectHTSpatiacan

ratur

gorit

prine serrequsign)namie jostat

l owepenions.

ServeounCRACon thmpefrom

powbounquattheximservevershe pthe

of otheyieldws:

thms and the computing power in the data center.management (i.e. scheduling and placement) and

er management, both of which determine the com-er consumption, in conjunction with the CRAC

nt (i.e. dynamically updating the thermostat settings),performed in such a way that for the maximumperature, the redline temperature is not violated

me a programmable thermostat where the set temper-e dynamically changed. However, the CRAC maintains

difference, Tth, between the high and low thermostat.e. Tth = Tth

high Tth

lowand Tth

high> Tth

low). Depending on

nce, the minimum possible supply temperature, Tsupmin,RAC can be determined (in the same way as Tsupmax inen the input temperature reaches the low thermostat

nd the CRAC operates at a high mode (i.e. the powery the CRAC is higher):

+ Pcomph

Phighexrroom

tsw Phighex

rac Tsupmin

Tth + Pcomph

Phighexrroom

tsw Phighex

rac. (12)

shows how an increase in the high thermostat sete can increase the supply temperature; thus poten-ing the CRAC power consumption (from Eq. (10)).designs a cooling-aware spatial job scheduling algo-, that allows higher thermostat set temperaturesrated with dynamic updates to the CRAC thermostat

denition

er deals with energy-efcient spatial scheduling of HPCheterogeneous parallel machine environment in the

s. The problem is dened as:

heterogeneous parallel machine environment and a jobent such that a job k requires ctot

knumber of processors,

spatial distribution of jobs (a vector of chassis numbers ine job k is executed), and the thermostat set points of theth every job start and completion to minimize the totalonsumption such that the job throughput and turnaroundwithin the user expectations.

be noted that the problem has two major decision-ects: (i) spatial job scheduling (i.e. placement of theappropriate servers), and (ii) CRAC thermostat con-termining the thermostat set temperatures). Spatialling is similar to the Knapsack problem, which iste [17]. The requirement for minimizing the energyn makes the problem harder. There has been variousroposed over the years ranging from using low-power

of the paper, changing the high thermostat set temperature refers tothe set temperatures.

tion, inodes.

Theimpactposesbasedent mall themanagtempe

4. Hig

Thirithm.with smostattempe

4.1. Al

Therank thi.e. the(i.e. as(iii) dyafter ththermoationainter-doperat

4.1.1.To c

on thebasedinlet terankedment.

Theupperfrom Eture tothe maof thethe sersince tndinghoursing ofwhichas follo

Algoriing mechanism in LRH however does not consider thee cool air from the CRAC to the nodes. This paper pro-TS algorithm that enables cooling-aware node ranking(i) the supplied cool air from the CRAC with differ-of operations, and (ii) the recirculated hot air fromer nodes. Further, HTS performs coordinated coolingnt by dynamically updating the CRAC thermostat setes.

Thermostat Setting (HTS)

ionpresents theHighest Thermostat Setting (HTS) algo-integrates the control of the CRAC thermostat settingsl job scheduling (i.e. job placement) so that the ther-be set as high as possible without violating the redlinees and the SLAs.

hm design

cipal intuition behind the algorithm is to: (i) staticallyvers from best to worst (according to the potential load,ired thermostat setting, incurredon theCRAC); (ii)placetemporally scheduled jobs to the best ranked servers;c determination of the required CRAC thermostat settingb placement is performed; and (iv) dynamically set theto the required value. Fig. 6 presents the intuitive oper-showing the aforementioned four operations and theirdencies. The following subsections describe these four

r rankingter the energy inefciencies because of the dependencythermostat setting, theHTSalgorithmranks theserverseir requirement on the CRAC thermostat to keep therature within the redline temperature. The servers are

highest to lowest thermostat temperature require-

er consumption of the servers themselves impact thed on the CRAC thermostat settings. This can be observedion 11, which relates the maximum supply tempera-high CRAC threshold and Equation 5, which relates

um supply temperature with the power consumptionrs through the redline constraint. Dynamically rankingdepending on the placement is however not efcient,lacement itself is dependent on the ranking. Further,optimal placement is NP-complete and may requireperation [12]. As such HTS performs a static rank-servers based on full utilization of the data center,s the thermostat setting requirement for a server i

1 (HTS integrated in the spatio-temporal job scheduling).

A. Banerjee et al. / Sustainable Computing: Informatics and Systems 1 (2011) 134150 141

w of HTS.

procedure IniGroup nodeSort groupsPerform serv et the redline for 100% utilization (Eq. (13)).

end procedur

procedure HtPlace job toDetermine tSet the CRAC

end procedur

procedure SetSet high the

F1Tred [

P

end procedur

procedure UpDispatch theDetermine tRemove theSet the CRAC

end procedur

procedure Upif job comes

Insert theelse

Insert theend iffor each nod

if jobs n(1) Inser(2) If Ste(3) If Ste

end if

if requiredDispatchRemove

end ifend for

end procedurFig. 6. Architecture and work-o

tialization()s with respect to power specications.with respect to computing efciency (i.e. MIPS/watt).er rankings, R, according to the requirement of thermostat set temperature to mee

s()available node(s) with the lowest rank in R.he power distribution vector, P h .thermostat using SetThermostat ( P ).h

e

Thermostat ( P h)rmostat setting (Thigh

th) as

comp

hPlowex

rroomtsw P

lowexrac

] F1DPh (Eq. (20))

e

onJobCompletion()next job in this groups queue using HTS().

he power distribution vector, P h .job from the queue.thermostat using SetThermostat ( P h).

e

onJobArrival()with node restrictions thenjob in the queue of the specied node group based on a scheduling policy (e.g. FCFS or ED

job in the most energy-efcient group queue based on a scheduling policy (e.g. FCFS or E

e group, from the most to least efcient node doish estimation>deadline thent the job in an opening having enough free servers for enough time. Continue with nexp 1 fails, push-t the job at an earlier time if shifting jobs still make the deadline. Contp 2 fails, add the job to next groups queue.

nodes in this group are idle thenthe job in this groups queue. using HTS().the job from the queue.

eF).

DF).

t job.inue with next job.

142 A. Banerjee et al. / Sustainable Computing: Informatics and Systems 1 (2011) 134150

CRAC high threshold requirements for chassis i

+ temperature rise due to switching time of the CRAC,difference in temperature due to CRAC power extraction

= maximum allowed inlet temperature of chassis i,

(Tthhigh)

(Tthhigh)

where Pfullj

tion. The seof (Tth

high)i. T

(performedrepresented

4.1.2. Job pAs show

from the scbased on thbased job pmanagememanagemesoftwares cprevious sepriority-basment.

4.1.3. DynaAfter pla

the highestrst the reqlowed by ac(20) is usedjob placem4.1.1 (wherthermostatutilization a

4.2. Spatio-

Algorithtemporal jotaken wherevents:

Arrival Earrives, i.Backll odecision,it is schedcenter ruJobArriva

Start Eveuled to sthe job, wHTS in A

4 Here the ajobs start tim

setting of CRAC. Given the placement of jobs, HTS sets the Tthhigh

of the CRAC to its upper bound, which can be obtained from Eq.(20).

End Event: When a job nishes execution, the placementchanges sthe therm

r bo

e thacasas t

y simtem

=Tr

s imphea

, wenkinnami

me co

n ame oNh isling,e thexityexityre aio-templey O(

EIR f

s secto catialthe

h, thuatedof thSP-Econ

pt, foptio

odeshm Aion fmal c

eneene

EhigAlg

Ehigopt

ExAlg

lgoriptioi+ (P

comph

)full Plowex

rroomtsw P

lowex

rac= Tini ,

i=

Tred

jdijPfullj

jfij+ P

lowex

rac (P

comph

)full Plowex

rroomtsw,

(13)

is the power consumption of chassis j at 100% utiliza-rvers are then statically ranked in the decreasing orderhe server ranking, is an one-time initialization processin procedure Initialization in Algorithm 1. The ranks areby ranking vector R.

lacementn in Fig. 6, the Spatial Job Scheduler takes the jobsheduled job queue4 and places them to the serverseir ranks (procedure HTS in Algorithm 1). Such rank-lacement can be easily incorporated in the current jobnt softwares. For example, the widely used Moab jobnt software allows setting up server priorities in theonguration [18]. The server ranking presented in thection can be used to prioritize the servers in Moab. Theed job assignment can then be enabled for job place-

mic thermostat settingcing the jobs, EDF-HTS sets the thermostat setting topossible value given by Eq. (20). As shown in Fig. 6,uired thermostat set temperature is determined fol-tually setting the thermostat to the required value. Eq.to determine the required thermostat setting after theent is performed. Unlike the server ranking in Sectione the data center was assumed to be fully utilized), therequirement is computed based on the actual serverfter job placement.

temporal job scheduling with HTS

m 1 presents how HTS is integrated with the spatio-b scheduling. An event based scheduling approach ise decision making is performed for the following three

vent: Temporal scheduling decision is takenwhen a jobe. the job is submitted by the user to the queue. FCFS-r the EDF approach is for temporal scheduling. After thisif some job is projected to nish after its deadline, thenuled to execute at an earlier time, thus having the datan multiple jobs at the same time. The procedure Upon-l in Algorithm 1 performs the temporal scheduling.nt: Two decisions are involved when a job is sched-tart execution: (1) determination of the placement ofhich is done following the HTS approach (procedure

lgorithm 1); and (2) determination of the thermostat

ssumption is that the jobs are already temporally scheduled, i.e. thees are decided by some temporal scheduling algorithm.

uppe

Notcoolingsideredstrategsupply

(Tsupmin)i

ThiculatedHenceLRH rathe dy

4.3. Ti

Upostart tiwherescheduto placcomplcomplthere aof spatThe cogiven b

5. SP-

Thirithmsthat splong asAs sucis evalenergy

Theenergytion, Eoconsumlow malgoritsumptin optias:

EAlgEopt

=

=

wherex for aconsumince the job is removed from the server and HTS resetsostat setting of the CRAC to a new value following the

und in Eq. (20).

t if we consider the HTS ranking in Eq. (13) for constante, then tsw will be zero. Tthhigh (Plowex /rac) can be con-he supply temperature of the CRAC. Thus, the rankingplies to ordering the servers in decreasing order of

perature requirements (Eq. (14)).

ed

jdijPfullj

jfij, (14)

lies sorting the servers in increasing order of the recir-t

jdijPfullj

, which is same as the LRH ranking [12].see that for constant cooling case HTS reduces to theg strategy. Hence, constant cooling is a special case ofc model.

mplexity

job arrival the job queue is searched to nd a properf the job. This has a complexity of O(Nh) at the worst,the total number of jobs in the event period. For spatialsince the static server ranking is used, it is only requirede job in the required number of servers, which has aof O(n), where n is number of chassis. Thus, the totalfor spatio-temporal scheduling of a job O(n+Nh). SincetotalNh job arrivals in an event period h, the complexitymporal scheduling in the event period is O(Nh(n+Nh)).xity for only spatial scheduling in the event period isnNh).

or spatial job scheduling

tion denes the SP-EIR of the spatial scheduling algo-ompare the quality of the solutions. It should be notedjob scheduling does not impact the job performance asafnity of the HPC jobs to the servers are maintained.e quality of the solution provided by an algorithm Algwith respect to the objective of minimizing the totale schedule.IR of Alg is dened as the ratio, EAlg/Eopt, of the totalsumption, EAlg, for Alg, to the total energy consump-r the optimal algorithm thatminimizes the total energyn. Note that the CRAC oscillates between the high andduring its operation (Section 2.3). The SP-EIR for anlg can then be obtained by calculating the energy con-or Alg and nding its ratio to the energy consumptionase. The SP-EIR for any algorithm Alg can thus be given

rgy consumption ofAlg under all CRAC modesrgy consumption of opt under all CRAC modes

EAlgEopt

h + ElowAlg

h + Elowopt, (15)

and Exopt are the energy consumptions of CRAC in modethm Alg and the optimal case, respectively. The energyn at a particular CRAC mode can be obtained by inte-

A. Banerjee et al. / Sustainable Computing: Informatics and Systems 1 (2011) 134150 143

grating the CRAC power consumption (Eq. (10)) at that mode overthe period for which the CRAC remains in the mode.

Ex = tx

0

(Px )2

ra

where tx isx. From theand the varform expresperiod can b

Ex =Pxex Tsx

where Txacbetween itsbe obtainedtemperaturcan be obtaTsenmin respectvalues andminimum scan be obta

It can bsumption inapproachesTsenmax/T

senmin a

the ratio beture in a moThe ratio omode perio

(Tsupmax)x

(Tsupmin)x =

T

From Eq. (1threshold tsetting morpower extrdifferencebswitching tdifferentmon thedynatemperatur

For the cand the poHence, intein an event

EconstCRAC =(P

h

rac

Differeneventperioon the ratioalgorithm, astant coolinon the heatdepend onHTS and LRwhich is no

5.1. Thermostat set point limits and its dependency on heatrecirculation

As discussed in Section 5, an algorithmwith SP-EIR near to unitytempconstat

pperobtaconobta

ax

ice hs onconsationthe hirculaon th

ollowcreareasenteren the thenterievede eners. Tof thnter

und

maxfor aimizervedes swernomboun

owehighon th

eneene

numcentefcperwithbe dding

ulat

prelingstat

differex

cTsup(t)dt, (16)

the time for the CRAC to remain in a particular modelinear relationship of Tsup(t) with Tsen(t) from Eq. (7)

iation of Tsen(t) over time (Eq. (11)) the following closedsion for the energy consumption of the CRAC in amodee obtained after performing the integration in Eq. (16):

Txacen

[ln

{(Tsupmin)

(Tsupmax)

}] , (17)is the temperature difference maintained by the CRACinlet and the supply in a particular mode x (this canfrom Eq. (7)), T senx is the rate of change of CRAC input

e obtained from Eq. (8). The values of (Tsupmin) and (Tsupmax)

ined from Eq. (12) and Eq. (11) in terms of Tsenmax andively. The SP-EIRdependson these energy consumptionhence is directly related to the ratio of maximum toupply temperature. The closed form equation for SP-EIRined by substituting Eq. (17) in (15).e observed from Eq. (17) that the CRAC energy con-a mode period nears zero as the ratio of Tsenmax to T

senmin

unity. The optimal algorithm will thus have the ratios close to unity as possible. It can be easily observed astween the maximum and minimum supply tempera-de period gets closer to unity the SP-EIR also improves.f the maximum to minimum supply temperature in ad can be obtained from Eqs. (11) and (12),

Tthhigh

+ ((Pcomph

Plowex )/(rroom))tsw (Pxex/rac)thhigh

Tth + ((Pcomph

Phighex )/(rroom))tsw (Pxex/rac),

(18)

8), it can be seen that the ratio depends on: (1) highemperature setting of the CRAC (higher the thresholde close the ratio to unity); and (2) the difference in theacted by the CRAC in its different modes (higher theetweenPhighex andPlowex lower is the ratio). However, if theime of the CRAC is low, the effect of extracted power inodes on the ratio is negligible. The SP-EIRhence dependsmicbehaviorof theCRACandgetsbetter if the thresholde is increased to a high value.onstant cooling case, during an event period the Tsup(t)wer extracted by the CRAC, Pcomp

h, remains constant.

grating Eq. (10), the energy consumption of the CRACperiod is given by,

comp)2

Tsup(t) h. (19)

t spatial scheduling algorithms affect the Tsup(t) in and. TheSP-EIR for constant coolingcase thusonlydependsof the supply temperature requirements imposed by anlg, to that by the optimal algorithm, opt. Hence for con-g, the supply temperature requirement solely dependsrecirculation in the data center and the SP-EIR does notthe dynamic behavior of the CRAC. In such a case, theH algorithms will have the same energy inefciency,t observed in our simulation results (Section 6).

will atredlinethermois an ucanberedlinecan be

(Tthhigh)m

Notdependpowera utilizattainno recbound

ter. It fwith inthe inc

An isolely otrols thdata cebe achand thdesignbounddata ce

5.2. Bo

TheSP-EIRto maxbe obsincreaswith lothe deupper

to the lto thebound

EAlgEopt

Thea datation cothe upcenterwouldof spen

6. Sim

Thescheduthermoundert to set the thermostat to a high value. However, thestraint in the data center puts an upper limit on thesetting that can be obtained from Eq. (5). Hence therebound on the maximum Tsup(t) in a mode epoch, whichined fromEq. (11); andusing thismaximumTsup(t) in thestraint, an upper bound on the high thermostat settingined as follows:

min(F1(Tred DPh)) +P lowexrac

Pcomph

P lowexrroom

tsw (20)

ere that the highest Tthhigh

that an algorithm can setthe recirculation in the data center, the computing

umption (Pcomph

), and the placement vector, Ph. Givenof the data center, the optimal algorithm will alwaysighest allowable Tth

highsetting. For a data center with

tion, D becomes a matrix of all zeroes and the uppere Tth

highonly depends on the utilization of the data cen-

s from Eq. (20) that the upper bound on Tthhigh

decreasesse in the recirculation in the data center. Hence, within the recirculation the SP-EIR also decreases.sting observation in this regard is that the SP-EIR dependse recirculation in the data center assuming that Alg con-rmostat set point. If investments aremade inmaking thefree of recirculation, then greater energy savings can. The tradeoff between the infrastructure investmentsergy savings has to be considered by the data centerhe following sectionpresents how to compute anuppere SP-EIR for any algorithm given the recirculation in a.

on energy inefciency

imum value of EAlg/Eopt gives the upper bound, on theny algorithm. In order to get the upper bound, we neede the numerator and minimize its denominator. It canthat if Tth

highfor Alg is decreased, then the numerator

ince the cooling energy consumption would increasethermostat settings. Similarly, if Tth

highfor opt is increased

inator decreases, hence increasing the SP-EIR. So and on the SP-EIR can be obtained if Tth

highfor Alg is set

st possible value, i.e. at 100% utilizationwhile Tthhigh

is setest possible value, i.e. at 0% utilization. Thus, the uppere SP-EIR can be given as follows:

rgy consumption ofAlg under 100%utilizationrgy consumption ofOpt under 0%utilization

, (21)

eric value of the upper bound can be calculated forer by plugging in the specic values of the recircula-ient for the data center. For the ASU HPC data center,bound for any algorithm is 1.69. For a different datadifferent heat recirculation pattern, this upper bound

ifferent; thus providing a metric to analyze the benetin data center infrastructure to reduce recirculation.

ion study

vious sections presented: (i) a new integrated jobalgorithm, EDF-HTS, that combines dynamic control ofset temperatures with spatio-temporal job schedulingent coolingmodels; and (ii) ananalytical study todeter-

144 A. Banerjee et al. / Sustainable Computing: Informatics and Systems 1 (2011) 134150

mine energy inefciencies of any spatial job scheduling algorithmin data centers. This section performs simulation study to showthat HTS indeed achieves better SP-EIR than other spatial schedul-ing algorithms, and EDF-HTS reduces the total data center energyconsumptioresults for dstant CRACavoid repet

6.1. Evalua

Evaluation and (2)an analysisalgorithms.

6.1.1. SP-EIThe SP-

schedulingrecirculatioenergy savischedulingconsolidatioa fair compturned off cbe set to ais used. Thition, i.e. formore casesest value alparticular jwhen usedand MTDPmmostat accowith such dMTDPd, resthe SP-EIR,are heuristulation, thecalculated bThe minimitribution toof

jdijPj f

6.1.2. TotalThe HTS

temporal scEDF-HTS, repractice inand utilizat[12]. The EDlowing spat

(i) FCFS-Bporal s

(ii) EDF-LRand LRedge, tjob schand LREDF-LRtrollingtheir coof doin

(iii) EDF-MMTDPm

doing integrated cooling over cooling-oblivious server consol-idation.

6.1.3. Throughput per unit energys is dened as the number of jobs serviced (i.e. completed)it timeperunit of energyconsumed. The throughputdependstemporal scheduling algorithm since the spatial schedul-erfoparthms,unctredackof coidati

mula

ENTmulaU HPis crhas tcoldnit wlenuconch rao diffthe

acks

uipm

er ms weyberysteassispow

], ase thaion ohe acwer c

+(1

ata ce

usedr the(ii) thquireacke selring

AC c

CRAetwe

job tran. In this simulation study, we only present simulationynamic CRAC behavior. The simulation results for con-model were presented in [12] and hence are omitted toition.

tion methodology

ons are performed for two cases: (1) idle servers keptidle servers turned off. For each of these cases, we doon the SP-EIR and the total energy consumption of the

REIR of HTS is compared with that of two spatial jobalgorithms: (i) LRH, which tries to reduce the heatn in the data center and is shown to have the bestngs when used in conjunction with the EDF temporalalgorithm [12], and (ii) MTDP, which performs servern in the data center (turning servers on or off) [7]. Forarison, HTS and MTDP are compared for idle chassisase only. The CRAC thermostat setting is assumed toconstant value when either LRH or MTDP algorithms value is usually set to handle the worst case situa-100% data center utilization. Further, we consider twofor a fair comparison: (i) thermostat setting to the high-lowable for the maximum data center utilization for aob trace considered in the simulation (LRH and MTDP,with such thermostat settings, are referred as LRHm, respectively), and (ii) dynamically varying the ther-rding to the requirement (LRH and MTDP, when usedynamic thermostat settings, are referred as LRHd andpectively). The computation of Eopt, in order to calculateis non-trivial since the most energy efcient solutionsics without guaranteeing optimality [12]. In the sim-lower bound on the optimal energy consumption isyminimizing the value ofmaxi( DP) for any utilization.zation of DP is done by assigning a separate power dis-each row in the D matrix so as to minimize the value

or all i.

energy consumptionis used in conjunction with both FCFS-Backll and EDFheduling algorithm (referred as FCFS-Backll-HTS andspectively). While FCFS-Backll is the most commonthe contemporary data centers to improve throughpution, EDF has been shown to be more energy-efcientF-HTS and FCFS-Backll-HTS are veried with the fol-io-temporal job scheduling algorithms.

ackll-LRH: This algorithm uses FCFS-Backll for tem-cheduling and LRH algorithm for spatial scheduling.H: This algorithm uses the EDF for temporal schedulingH for spatial scheduling. To the best of our knowl-his is the best online energy-efcient spatio-temporaleduling algorithm for HPC data centers [12]. LRHmHd are also used with EDF (referred as EDF-LRHm andHd, respectively). However, noneof these consider con-the cooling set points in the schedulingdecisions. Thus,mparison with EDF-HTS will hint towards the benetsg integrated cooling control.TDP: Comparison of EDF-HTS with EDF-MTDP, EDF-, and EDF-MTDP m will highlight the advantages of

Thiperunon theing is p4.3). Aalgoritin conjcompaFCFS-Btagesconsol

6.2. Si

FloVmal sithe ASmodelwhichtypicalCRAC uoor pthe airand eaare twAmongthree r

6.3. Eq

Powserverfrom Cblade sidle chserver[21,22assumsumptfrom tthe po

P =

6.4. D

Weyear fotimes,sors reFCFS-Bjobs artion du

6.5. CR

Theence b

5 Thermed among the requested servers of the jobs (Sectionfrom the aforementioned spatio-temporal schedulingthe FCFS-Backll-MTDP (i.e. MTDP spatial schedulingion with the FCFS-Backll temporal scheduling) is alsowith the FCFS-Backll-HTS algorithm. Comparison ofll-MTDP with FCFS-Backll-HTS will show the advan-ordinated job and cooling management over serveron in terms of the throughput per unit energy.

tion setup

[19], aCFDsimulationsoftware, isused toconduct ther-tions for obtaining heat distribution matrix. Based onCI data center physical layout, a data center simulationeated with physical dimensions 9.6m8.4m3.6m,wo rows of industry standard 42U racks arranged in aaisle and hot aisle layout. The cold air is supplied by oneith the ow rate 8m3/s. The cold air rises from raised

m through vent tiles, and exhausted hot air returns toditioner through ceiling vent tiles. There are ten racksck is equipped with ve 7U (12.25-in.) chassis. Thereerent types of computing equipment in the data center.fty chassis, there are thirty Dell PowerEdge 1955 (i.e.) and twenty Dell PowerEdge 1855 chassis (Table 1).

ent power consumption

easurements of Dell PowerEdge 1855 and 1955 bladere performed using the DUALCOM [20] power meterSwitching Inc. Using the power measurements of thems and performing linear regressions on the data, thepower consumption () and the single fully-utilizeder consumption () values for the simulation runsgiven in Table 2, were computed. The simulationst the jobs are CPU-intensive. The estimated power con-f the resulting linear function has an error of 0.49%tual measurements. For a different utilization u

A. Banerjee et al. / Sustainable Computing: Informatics and Systems 1 (2011) 134150 145

Table 1Scalar symbols and denitions.

Symbol Denition

n Total number of chassis

h Inter event interval or event period

Tthhigh

The number of servers (blades) job k requires

rac Thermal capacity of air owing out of the CRAC per unit time

rroom Thermal capacity of air in the data center room

fii Cold supply air fraction going from CRAC to chassis i

dij Heat recirculation coefcient from chassis i to j

x Mode of operation of CRAC, x {high, low}tsw Time taken by the CRAC to switch from one mode to the other

Idle chassis power consumption

Power consumption of a chassis per unit of utilization

u Chassis utilization

Nh total number of jobs in event period h

Tsup(t) Air temperature as supplied from the CRAC at time t

Tsen(t) Air temperature at the input of the CRAC at time t

Tred Manufacturers redline inlet temperature

Tthhigh

Tthlow

(Tthhigh

)max

(Tthhigh

)i

Tth

Pxex

Pfullj

Pcomph

PAC(t)

Ey

Exy

Table 2Power consum

PowerEdge 1PowerEdge 1

value of 15the CRAC w15 C. The p350kW wh

10 20 30 40 50 60 70 801.005

1.01

1.015

1.02

1.025

1.03

1.035

1.04

Peak Utilization (%)

SP-E

IR

LRH

LRHm

LRHd

HTS

Fig. 7. Energy inefciency of the total energy of HTS and LRH when idle chassis arekept on. The plots are interpolated from the energy consumption for 5%, 40%, and80% peak utilization in ASU HPC data center.

the data center. The power consumption in the lowmodewas keptkWwta ce

sults

Table 3Percentage of

Data center

5%40%80%

Table 4Percentage of

Data center

5%40%80%High thermostat setting for the CRAC

Low thermostat setting for the CRAC

Upper bound on high thermostat setting for CRAC

CRAC high thermostat setting requirement for server i

at 100HPC da

6.6. Re

Temperature difference between the CRAC high and lowthermostat settings

Power extracted by the CRAC in mode x

Power dissipation of chassis j at 100% percent utilization. For anyvariable z, zfull denotes the value of z at 100% utilization and zempty

denotes that at empty data center

Total computing power at inter-event period h

Power consumption of CRAC at time t

Energy consumption for algorithm y

Energy consumption of CRAC in mode x for algorithm y

ption parameters.

855 1820 72955 2420 175

C. For EDF-LRH the high threshold temperature ofas kept at 30 C while the low threshold was kept atower consumption of the CRAC at the high mode wasich is greater than the maximum computing power of

EDF-HTSmost energaugmentedper unit of

The perrespect to oin Tables 3ings of EDF-for higher uare more optings. Furthkept constasumption isthermostatgiven set of100% utiliza

The SP-Ethe utilizatrespectivelyreaches amis expectedboth the alg

total energy savings in EDF-HTS for different utilization (idle chassis kept on).

utilization Percentage of energy savings of EDF-HTS with respect to

FCFS-Backll-LRH FCFS-Backll-HTS

12.41 10.655.70 3.273.30 0.85

total energy savings in EDF-HTS for different utilization (Idle chassis turned off).

utilization Percentage of energy savings of EDF-HTS with respect to

FCFS-Backll-LRH FCFS-Backll-HTS EDF-LRH EDF-LRHm

23.78 21.30 12.53 12.3021.50 17.22 16.00 15.5615.80 10.81 9.03 8.86hich is equal to the idle power consumption of the ASUnter. The CRAC mode switching time was kept at 3 s.

, when augmented with idle server turn-off, is they-efcient algorithm while FCFS-Backll-HTS, whenwith idle server turn-off, has themaximum throughputenergy consumption.centage savings in energy obtained by EDF-HTS withther algorithms for idle on and idle off cases are shownand 4, respectively. For lower peak utilization, the sav-HTSwith respect to EDF-LRH is high, while it decreasestilization. This is because at low peak utilization theretions to place a job and achieve higher thermostat set-er, it can be observed that if the thermostat setting isnt (as in EDF-LRH and EDF-LRHm) then the energy con-higher than that of EDF-LRHd. Moreover, keeping thesetting for the maximum data center utilization for ajobs (as in EDF-LRHm) is benecial than setting it fortion (as in EDF-LRH).IR of the HTS and LRH algorithms is plotted against

ion in Figs. 7 and 8 for the idle on and idle off cases,. With increase in the utilization the SP-EIR increases,aximumvalue, and thenagain goesdown. This behaviorfrom the formulation where at 0% and 100% utilizationorithms will be optimal. The theoretical maximum SP-EDF-LRH EDF-LRHm EDF-LRHd

3.70 3.32 0.871.85 1.49 0.831.40 1.31 0.73

EDF-LRHd EDF-MTDP EDF-MTDPm EDF-MTDPd

5.17 8.54 8.17 5.1710.84 9.00 8.73 5.730.66 3.81 3.47 0.47

146 A. Banerjee et al. / Sustainable Computing: Informatics and Systems 1 (2011) 134150

10 20 30 40 50 60 70 80

1.02

1.04

1.06

1.08

1.1

1.12

1.14

1.16

1.18

1.2

Peak Utilization (%)

SP-E

IR

LRH LRHm

LRHd

MTDPd

MTDP

MTDPm

HTS

Fig. 8. Energy inefciency of the total energy of HTS, LRH, and MTDP when idlechassis are turned off. The plots are interpolated from the energy consumption for5%, 40%, and 80% peak utilization in ASU HPC data center.

1.6x 10-11

1.4

1.2MTDP

1HTS

LRH0.8

FF

0.4

0.6

0 10 20 30 40 50 60 70 80Utilization (%)

Thro

ughp

ut P

er U

nit o

f Ene

rgy

Con

sum

ptio

n (jo

bs/s

.J)

Fig. 9. Job throughput per unit of energy consumption when different spatialscheduling is used with FCFS-Backll temporal scheduling. The plots are inter-polated from the throughput per energy consumption for 5%, 40%, and 80% peakutilization in ASU HPC data center.

5 40 800

50

100

150

200

250

Ene

rgy

cons

umpt

ion

in G

J

Energy consumption for different peak utilizations

Peak Utilization (%)

FCFS - LRHFCFS - HTSEDF - LRHEDF - HTSOPT

CoolingEnergy

ComputingEnergy

Fig. 11. Total energy consumption comparison between EDF-LRH, EDF-HTS andlower bound on optimal solution at different data center peak utilizations for idlechassis kept on.

EIR of HTS is 1.013 (obtained by using synthetic workload in therange of 0-100% data center utilization), which is around 15% lowerthan the LRH algorithm.

FCFS based algorithms perform poorer than the EDF based algo-rithms withthe temporthe comparidle on andcenter enerthe break uresearch isCRAC contr

On the ohigher throThis is mainot temporin throughpwhen theincreases creduction in

As showthroughputthe througBackll-MTFCFS-Backunit energy

0 20

5

10

15

20

25

30

35

40

45

Time (seconds)

EDF LRH

0 2 4x 10 5

0

5

10

15

20

25

30

35

40

45

Time (seconds)

FCFS-BackFill - HTS

Max

imum

Cha

ssis

Inle

t Tem

pera

ture

(deg

ree

C)

Redline Temperature Redline Temperature

Fig. 10. Maximum chassis inlet temperature for 80% utilization under dynamic cooling model for diffEDF-HTS are compared in this gure to represent the effect of only cooling aware, only job and thermaIt can be seen that EDF-HTS runs the data center hottest yet avoiding redline violations and hence achirespect to the total energy consumption, because ofal spreading of the jobs in EDF [12]. Tables 3 and 4 giveison of the FCFS based approacheswith EDF-HTS for theidle off cases, respectively. Figs. 11 and 12 show the datagy consumption for the different algorithms along withp of the computing energy and cooling energy. Futureneeded to investigate the energy savingswhen dynamicol is integrated with server consolidation.ther hand, FCFS-Backll based algorithms can achieveughput per unit energy over the EDF based algorithms.nly because of very high job throughput since jobs areally spread in FCFS-Backll. Fig. 9 shows the variationut per unit energy for FCFS-Backll based algorithms

idle servers are turned off. The energy consumptiononsiderably with increase in utilization; thus causingthe throughput per unit energy for higher utilization.n in Fig. 9, FCFS-Backll-HTS achieves the highestper unit energy for all utilizations and can increase

hput per unit energy by up to 5.56% over the FCFS-DP algorithm. When compared to FCFS-Backll-LRH,ll-HTS can achieve up to 6.89% higher throughput perconsumption.

40

45EDF HTS

Redline Temperature0 2 4x 10 5

0

5

10

15

20

25

30

35

Time (seconds)4

x 10 5

erent energy management policies. FCFS-Backll-HTS, EDF-LRH andl aware and job thermal and cooling aware scheduling mechanisms.eves the most energy efciency.

A. Banerjee et al. / Sustainable Computing: Informatics and Systems 1 (2011) 134150 147

Fig. 12. Totallower bound ochassis turned

046.5

47

47.5

48

48.5

49

49.5

50

50.5

Ther

mos

tat S

et T

empe

ratu

res

(deg

ree

C)

Fig. 13. CRACmanagement p

Further,ness, job awthe simulatalgorithm, wing FCFS-Bawhich incornumber ofHTS algoritthe maximuthe three alfor EDF-LRHThis showsing to the thutilized bying assumpset by the tEDF-LRH, mhigher CRAslack in jobthe compar

7. Related

This secenergy efc

has been tackled by both academia and industry from differentdomains. For example, mechanical engineers focus on modi-cation in physical layout of data centers [11], liquid cooling

/www.42u.com/liquid-cooling-article.htm), and introduc-chillcal ecingsts frkloaphyslingCRACearchinto

ling; (2) tta cen-awanterrmalRangpowrgy c], dute eneof sed th[25]haveenergy consumption comparison between EDF-LRH, EDF-HTS andn optimal solution at different data center peak utilizations for idleoff case.

(http:/tion ofelectriintroduscientilel wocyber-scheduin the

Resdividedscheducenterthe dacoolingdata ce

Thewhereon theall ene[23,24achievcyclingproposenergytheory0.5 1 1.5 2 2.5 3 3.5 4

x 105Time (seconds)

EDFHTSEDFLRHFCFSHTS

thermostat set temperatures with respect to time when differentolicies are employed (peak utilization=80%).

the relative benets of incorporating cooling aware-areness and thermal awareness can be obtained from

ions. To this effect we choose: (1) the FCFS-Backll-HTShich just incorporates cooling awareness to the exist-

ckll scheduling algorithm, (2) the EDF-LRH algorithm,porates thermal awareness and ensures thatmaximumjobs meet their respective deadline, and (3) the EDF-hm, which integrates all three aspects. Fig. 10 showsm chassis inlet temperature as a function of time for

gorithms. It can be seen that the data center runs hotterthan FCFS-Backll-HTS and the hottest for EDF-HTS.

that EDF-HTS benets from ranking the servers accord-ermostat temperature requirements. This benet is notEDF-LRH, which was designed based on constant cool-tions. Fig. 13 shows the CRAC thermostat temperatureshree algorithms. From the gure it is clearly seen thatisses out on adjusting the CRAC thermostat. Further,C thermostat set temperatures can be achieved if thes are utilized to spread them apart in time as seen fromison between FCFS-Backll-HTS and EDF-HTS.

work

tion presents the related research on data centeriency. Reduction in data center energy consumption

and performthe thermain the data

In [28],account fomal effectsbetween serithms are ta similar apIn [30], aneffects of inthe heat reunder-estimare awareconsider its

Integratsuch as supmanagemeet al. propoeffects in aas to reducetaken by Abservers areincreased. Tet al. haveing and coodecision prmal state toHowever, aof the CRACthe CRAC dyaware, andclassicatio

Further,gives a meferent manmanagemeer doors (http://www.42u.com/42u-rack-cooling.htm);ngineers focus on designing low power servers andnew power states in processors; and computer

ocus on energy-efcient spatial scheduling of paral-d on heterogeneous servers. This paper focused onical decision making through: (i) spatio-temporal jobto reduce cooling demands; and (ii) dynamic variationsthermostat settings.in the area of energy-efcient job scheduling can bethree categories: (1) thermally oblivious, in which, job

decisions are not aware of the heating effects in the datahermal-aware, which considers the thermal behavior ofter but does not consider its cooling behavior; and (3)re, which considers both the thermal behavior of theand also considers its cooling behavior.ly obliviousmanagement strategies include work in [2]anathan et al. propose a load balancing scheme baseder consumption of machines so as to minimize over-onsumption of the data center while meeting SLAs. Iny cycling of servers (turning off and on) is proposed torgy efciency. A unication of load balancing and dutyervers is further proposed in [3]. Researchers have alsoe use of dynamic voltage scaling of servers to reducewhile maintaining SLA. In this regard, ideas from gamebeen employed to account for the trade off in energyance [26,27]. However, all theseworks do not consider

l effects and the associated cooling energy expenditurecenter.the thermal behavior of the data center is taken intor job placement algorithms. In this work, the ther-in the data center are abstracted to heat recirculationrvers that cause local hot spots. Job placement algo-hen proposed to avoid servers in the hot spots. In [29],proach is taken to avoid hot spots while placing jobs.RC thermal model is used to characterize the thermaldividual server on the environment. However, it ignorescirculation from other servers which may result ination of temperature rise. Although these techniques

of the thermal effects in the data center, they do notcooling behavior.

ion of cooling control (controlling cooling parametersply temperature and set points of the CRAC)with energynt in data centers is of recent focus. In [12], Mukherjeese job scheduling algorithms that consider the thermaldata center and try to schedule as well as place jobs sothe cooling demands. Similar efforts have been under-basi et al. [8] for the Internet data centers where activeso chosen that the supply temperature requirements arehis in turn causes reduction in cooling energy. Parolinialso proposed an integrated approach to job schedul-ling control in [6]. Data center is modeled as a Markovocess where the actions taken to move from one ther-another is controlled by the CRAC thermostat settings.ll these techniques do not model the dynamic behavior. The HTS algorithm proposed in this paper considerednamic behavior in performing cooling-aware, thermal-energy-efcient job management. Table 5 shows thisn and highlights the novelty in HTS.given these three types of scheduling algorithms, [9]chanism to decide on which algorithm to use for dif-agement requirements such as, job, thermal or coolingnt. Recent works have further, evaluated the energy

148 A. Banerjee et al. / Sustainable Computing: Informatics and Systems 1 (2011) 134150

Table 5Classication of previous research on data center energy efciency.

Techniques for energy efciency Thermally oblivious Thermal aware Cooling aware Cooling model

Ensemble level power management [2],Load balanGame theo

MinHR, XIntTemperaturecirculati

SCINT and LMDP base

HTS

efciency oence of eneconstant cohow energyof heat reciperformed.

In this rwork to mmanagemeconsumptioior. It denof energy inpossible covious reseaconsideredrithms. Linof online also as to savaware scheeffects in thdata centerpetitive bouRather an ois more releaware jobhave beenshort of conerogeneousframeworkrithms for da specic eof the dataior is still igformulationcyber-physIt incorporathe formulaformance e

8. Conclus

IntegratschedulingFirst, a coois proposedCRAC thresCRAC thresout violatinanalysis ofstant and lmetric is ina spatial scknown coo

, condynreshitchils tos tolingtensitems FCFH. Fg offompidatiulatist entedrgy-eduhieveion tacks.cooith ch inSF).6

lop,urrennatedebehfrom

ny earchshorpapedatae numadathe) coolingrmals witamicing [3,23,24], CPUMISER [25],retic approach [26,27], Minimax Optimization [22],re based allocation withouton [29,30]RH [12], TASP and TAWD [8], andd scheduling [6]

f these algorithms through simulations in the pres-rgy proportional servers in the data centers [10] underoling environment. However, a theoretical analysis oninefcient these algorithms can get in the presence

rculation and dynamic cooling behavior, has not been

egard, this paper also provides a theoretical frame-athematically evaluate the performance of energynt algorithms in terms of their inefciency in energyn under heat recirculation and dynamic cooling behav-es a new metric, SP-EIR, which determines the degreeefciency of a given algorithm with respect to the bestoling and thermal aware job placement (optimal). Pre-rch in algorithm performance evaluation has mostlyenergy efciency of thermal and cooling unaware algo-et al. [31] have performed competitive bound analysisgorithms for dynamically selecting servers to turn offe energy. However, such an analysis is only for energyduling algorithms and does not consider the thermale data center or the cooling behavior. Further, for HPCs, the workload is often known before hand so a com-nd analysis for online job scheduling is not applicable.fine analysis using the SP-EIR metric, as in this paper,vant. Ofine approximation bound analysis of thermalscheduling algorithms for homogeneous data centersconsidered [32]. However, such an analysis again fallssidering cooling behavior and thermal proles of het-data center. Tang et al. [33] proposes a theoretical

for the development and evaluation of scheduling algo-istributed cyber physical systems with data center asxample. Although it incorporates the thermal prolescenter in the algorithm development, cooling behav-nored. Further, Gupta et al. [34] propose a theoreticalof the scheduling for energy efciency problem in

ical systems with data center as a specic case study.tes the heat recirculation and cooling behavior intotion but does not give any metric for algorithm per-valuation.

ions and future work

ion of cooling and thermal awareness for spatialof jobs in HPC data centers is discussed in the paper.ling and thermal aware job placement algorithm HTS

regardcase ofhigh thand swthat fatunitiescheduing. Exspatio-such aEDF-LRturninbeen cconsol

Simthe moaugme9% enetial schcan acsumptFCFS-Bcenter

Theing it wresearction (Nto deveters. Ccoorditure thenergy

Ma(e.g. semostlyin thisbasedof largcan being ofand (iischeduon thecenter

Dyn

, which ranks the servers in decreasing order of theirhold temperature requirements and then adjusts thehold temperature to the highest possible value with-g the redline constraint. Second, an energy inefciencythe existing spatial scheduling algorithms under con-inear cooling modes is performed, where the SP-EIRtroduces. This metric shows how much energy savingsheduling algorithm can have with respect to the bestling and thermal aware scheduling algorithm. In this

impact andsaturatingcycle (seve(especiallychanges in

6 http://wwConstant

Dynamic

stant cooling is mathematically shown to be a specialamic cooling. Further, SP-EIR is shown to depend on theold temperature, power extraction in different modesing time of the CRAC. It is concluded that any algorithmcontrol the threshold temperaturemisses out onoppor-save cooling energy. Third, the proposed HTS spatialalgorithm is augmented with EDF temporal schedul-ve simulations are performed to compare the EDF-HTSporal scheduling against other common approachesS-Backll-FF, FCFS-Backll-LRH, FCFS-Backll-HTS, andurther, in simulations, potential for energy efciency byservers has been studied. In this regard EDF-HTS hasaredwith EDF-MTDP, which performs intelligent serveron.on results showHTScan reduceSP-EIRbyup to15%overnergy-efcient scheduling algorithm, LRH. HTS, whenwith idle server turn-offs, can further achieve up to

savings compared to the MTDP, which performs spa-ling based on thermal-aware server consolidation. HTSup to 5.56% higher throughput per unit energy con-

han MTDP, when both HTS and MTDP are used withll, a widely used temporal scheduling algorithm in data

ling-aware spatial job scheduling in HTS (and integrat-ooling management) is developed as part of a BlueToolfrastructure funded by the National Science Founda-A miniature data center test-bed is being developed

test and evaluate energy reduction policies in data cen-t and future works in this regard are geared towardsjob scheduling and cooling management that can cap-avior of novel future coolingunits designed to scavengealternate sources (e.g. solar power).

nterprise data centers further host Internet servicesengines, data retrieval) where the applications are

t-duration transactions (unlike the HPC jobs consideredr, which are normally long running). In such transactioncenters, job management usually involves distributionber of workloads depending on their arrival rate. HTS

pted for such data centers in two ways: (i) provision-servers based on the HTS server ranking mechanism;ling-aware workload distribution similar to the spatialfor HPC data centers. Indeed, we have been working-aware workload distribution in transaction-based datahout any dynamic setting of the CRAC thermostat.cally setting the thermostatmaynot have instantaneous

may take effect with a delay, principally because of

the heat capacity of the agent in the internal coolingral seconds, perhaps minutes). As such, in many casesin case of transaction based data centers with frequentthe thermostat requirements), it may be important to

w.nsf.gov/awardsearch/showAward.do?AwardNumber=0855277.

A. Banerjee et al. / Sustainable Computing: Informatics and Systems 1 (2011) 134150 149

make sure that any dynamic update of the CRAC thermostat doesnot become stale by the time it takes effect. Future work in thisregard needs to performmanagement decisionmaking that employsproper energy-management policies depending on the state of thedata center

Further,based sincerequire shuan ongoingBlueTool/wgrant (CNStestbed. Sucrithms in rework.

References

[1] U. E. P. Agpublic law

[2] P. Ranganment forSymposiu6677.

[3] E. Pinheiring for poCompiler

[4] P. Bohrermony, L.C

[5] J. Moore,temperatAnnual Te

[6] L. Parolindata centback ContSan Franc

[7] E. Pakbazcosts, in:sium on L145150.

[8] Z. Abbasiandworkposium o

[9] T. Mukheordinated(2010) 28

[10] G. Varsamtionalityon High P

[11] M. Stangies go0,289142

[12] T. MukhSpatio-tesumptionhttp://dx

[13] E. Krevattem, in:Job Sched2002, pp.

[14] G.Varsamjob schedIndia, Aug

[15] S. R. LaPC. Cavanater. Availnov a1.ph

[16] M.J. Moraed., Wiley

[17] T.H. Cormed., McGr

[18] Moab grclusterres

[19] A. Flomeomerics

[20] Cyber Swpdf/Dualc

[21] T. Mukhepower prence on CComputin

[22] Q. Tang, et al., Energy-efcient thermal-aware task scheduling for homoge-neous high-performance computing data centers: a cyber-physical approach,IEEE TPDS 19 (11) (2008) 14581472.

[23] E. Pinheiro, R. Bianchini, E.V. Carrera, T.Heath,DynamicClusterRecongurationfor Power and Performance, 2003, pp. 7593.

ikita,are jobDistre, X. F-timeallel Phan,ationE Tranubratcom

(2) (2J4MF

Tang,ion mrnatio6. ICISash,ings o. HPLards-Hsiu/trunkLin, A.portioAissi,-max

timizaang,in diputin

tware.S. Gupnergy: Infor.S. GupngAcosustapmanwhile meeting the SLAs and server redlines.the results presented in this paper are simulationactual experiments in physical data centers may

tting down servers, which in turn may affect SLAs. Inproject named BlueTool [35] (http://impact.asu.edu/

iki/index.php/Main Page) funded by NSF infrastructure#0855277) Impact Lab is developing a data centerh a testbed can be used to evaluate the proposed algo-al scenarios and is considered as an important future

ency, Report to congress on server and data center energy efciency109-431, ENERGY STAR Program, 2007.

athan, P. Leech, D. Irwin, J. Chase, Ensemble-level power manage-dense blade servers, in: IEEE Proceedings of the 33rd Internationalm on Computer Architecture (ISCA06), Boston, MA, May, 2006, pp.

o, R. Bianchini, E.V. Carrera, T. Heath, Load balancing and unbalanc-wer and performance in cluster-based systems, in: In Workshop ons and Operating Systems for Low Power, 2001., E.N. Elnozahy, T. Keller, M. Kistler, C. Lefurgy, C. McDowell, R. Raja-. Mcdowell, The case for power management in web servers, 2002.J. Chase, P. Ranganathan, R. Sharma, Making scheduling cool:

ure-aware resource assignment in data centers, in: 2005 Usenixchnical Conference, April, 2005.

i, B. Sinopoli, B.H.Krogh,Aunied thermal-computational approach toer energy management, in: Fourth International Workshop on Feed-rol Implementation andDesign in Computing Systems andNetworks,isco, California, USA, April, 2009.nia, M. Pedram, Minimizing data center cooling and server powerISLPED09: Proceedings of the 14th ACM/IEEE International Sympo-ow Power Electronics and Design, ACM, NewYork, NY, USA, 2009, pp.

, G. Varsamopoulos, S.K.S. Gupta, Thermal aware server provisioningloaddistribution for internet data centers, in: ACM International Sym-n High Performance Distributed Computing (HPDC10), June, 2010.rjee, A. Banerjee, G. Varsamopoulos, S.K.S. Gupta, Model-driven co-management of data centers, (Elsevier) Computer Networks 54 (16)692886.opoulos, Z. Abbasi, S.K.S. Gupta, Trends and effects of energy propor-on server provisioning in data centers, in: International Conferenceerformance Computing Conference (HiPC2010), December, 2010.sberry, Hot-aisle/cold-aisle containment and plenum strate-big-time, http://searchdatacenter.techtarget. com/news/article/

,sid80 gci1320452,00.html, 2008.erjee, A. Banerjee, G. Varsamopoulos, S.K.S. Gupta, S. Rungta,mporal thermal-aware job scheduling to minimize energy con-in virtualized heterogeneous data centers? Computer Networks.

.doi.org/10.1016/j.comnet.2009.06.008, June 2009., J.G. Casta nos, J.E. Moreira, Job scheduling for the bluegene/l sys-JSSPP02: Revised Papers from the 8th International Workshop onuling Strategies for Parallel Processing, Springer-Verlag, London, UK,3854.opoulos,A. Banerjee, S.K.S.Gupta, Energyefciencyof thermal-awareuling algorithms under various cooling models, in: IC32009, Noida,ust, 2009.lante, N. Aubry, L. Rosa, P. Levesque, B.S. Aboumrad, D. Porter,ugh, J. Johnston, Liquid cooling of a high density computer clus-able from: http://www.electronics-cooling.com/articles/2006/2006p, 2006.n, H.N. Shapiro, Fundamentals of Engineering Thermodynamics, 6th, 2007.en, C.E. Leiserson, R.L. Rivest, C. Stein, Introduction toAlgorithms, 2ndaw-Hill Book Company, Cambridge, London, 2001, 1. editon 1993.id suite of ClusterResources Inc. Available from: http://www.ources.com/.rics Ltd., Flovent Version 2.1. Available from: http://www..com/, 1999.itching, DUALCOM user manual. http://www.cyberswitching.com/omManual.pdf.rjee, G. Varsamopoulos, S.K.S. Gupta, S. Rungta, Measurement-basedoling of data center equipment, in: Proceedings of the IEEE Confer-lustered and Grid Computing (Cluster 2007), Workshop on Greeng (GreenCom07), Austin, TX, September, 2007.

[24] J. Hawand

[25] R. GrunPar

[26] S. KmizIEE

[27] R. Sgrid704XC

[28] Q.uatInte200

[29] C. BsavRep

[30] TowKaosvn

[31] M.pro

[32] H.minOp

[33] Q. TingComSof

[34] S.Kin eing

[35] S.KusiandChaA. Hirano, H