Penn State University

Penn State University

Computer Science &Engineering (CSE)Mechanical & Nuclear Engineering (MNE)

Energy Storage in Datacenters: What, Where, and How much?

Di Wang (CSE)Chuangang Ren (CSE) Anand Sivasubramaniam (CSE) Bhuvan Urgaonkar (CSE)Hosam Fathy (MNE)

Datacenters Are Heavy Power Consumers

2

• Increase in number/size of datacenters due to heavy reliance on Internet services

• Datacenters, if treated as a country, fifth in the world for electricity use

• Datacenter electricity usage expected to double in next 5 years and requires 12 new power plants

Monthly Costs for a 10MW Datacenter

3

$921,172

$1,137,615

$730,000

$249,720

Servers

Power Infrastructure

Utility BillOther

8%

30.5%

24%

37.5%

All cost are amortizedat a monthly granularity

Chart:Source: Book by Barroso et al.,

Assumption: 20,000 servers, 1.5 PUE, 15$/W Cap-ex, Duke Energy Op-ex,

4yr server & 12 yr infrastructure amortization (Tier-2)

Pow

er d

raw

(W)

Energy consumption(area under this curve)

Month

Peak power draw

4


$921,172

$1,137,615

$730,000

$249,720

Servers


(Cap-ex)

Utility Bill(Op-ex)

Other

8%

30.5%

24%

37.5%


Cost is heavily impacted by peak

power draw.(Barroso,

Ranganathan,Hamilton,

Bandarkar)

Peak Power Impact on Op-ex

5 c/KWh

Pow

er d

raw

(W)

Energy consumption(area under this curve)

Note: Tariff rates collected from Duke Energy Utility. 5

Month

Duke Utility Tariffs(12 $/KW, 5 c/KWh)

15-min

Average draw

Peak power draw

Peak toAverage

ratio

3:1

12 $/KW

$ Peak 50%

$ Energy 50%

Peak Power Impact on Cap-ex

6

UtilitySubstation

Diesel Generator

(DG)

UPS

…

…

PowerDistributionUnit (PDU)

ServerRacks

Auto TransferSwitch (ATS)



7

Diesel Generator

(DG)

Rated Peakcapacity

UtilitySubstation

UPS

…

…


ServerRacks



8

Rated Peakcapacity

Pow

er (W

)

Time

UtilitySubstation

UPS

…

…


ServerRacks



9

Rated Peakcapacity

Pow

er (W

)

Time

UtilitySubstation

UPS

…

…


ServerRacks


Key Lesson

10

• Reducing peak power draw helps– Lower Op-ex– Lower Cap-ex

How do we reduce peak draws ?

Demand Response Knobs in a Datacenter

11Time

Pow

er c

onsu

mpt

ion

(W)

Peak

Energy Storage Device (ESD)

Power Cap

Newdraw

Originaldraw

DVFS throttling: Fan et al., [2007], Felter et al., [2005], Meisner et al., [2011]Consolidation:Chase et al., [2001], Pinheiro et al., [2001], Lim et al., [2011]Migration/Scheduling:Moore et al., [2005],Ganesh et al., [2009], Lin et al., [2011]

How realize energy storage in datacenters?

Energy storage:Govindan et al. [2011,2012],Urgaonkar et al. [2011], Culler et al., [2012],

12

UtilitySubstation Diesel

Generator(DG)

ESD

Energy Storage Device (ESD) in Current Datacenters

…

…


ServerRacks


Cost Saving

13

Distributed UPS Configurations

…

Server level UPS

…

…ServerRacks

PDU

ESD

Rack level UPSUtility

Substation


UtilitySubstation Diesel

Generator(DG)


Diesel Generator

(DG)

Cost Saving

Similar to the ones in Google, Microsoft and Facebook datacenters

…

Why should we be restricted to single ESD technology

(e.g., Lead acid battery)?

Why should we restrict ESDs to any one level of the datacenter power hierarchy (e.g., central or server)?

Talk Outline

• Motivation• Efficacy of Different ESDs• Framework for Provisioning and Control• Evaluation• Conclusions

14

Which ESD to choose for peak shaving?

15

Pow

er

Time Time

Pow

er

Time

Pow

er

E E E

http://www.valandre.com/blog/wp-content/uploads/2011/06/K2_south_routes_m1.jpg


16

Timepo

wer


17

pow

er

Time Timepo

wer

Ragone Plot

18

Spec

ific E

nerg

y (W

h/kg

)

Specific Power (W/kg)

0Batteries

Capacitors

Compressed Air (CAES)

Supercapacitors

Combustion Engine,

Gas Turbine

10,000

1,000

100

10

1

0.1

10 100 1,000 10,000 100,000 1,000,000

LA

Fuel Cell

Flywheels (FW)

LI

Ultracapacitors(UC)

Ragone Plot

19

Spec

ific E

nerg

y (W

h/kg

)

Specific Power (W/kg)

0

Compressed Air (CAES)

Supercapacitors

10,000

1,000

100

10

1

0.1

10 100 1,000 10,000 100,000 1,000,000

LA

Flywheels (FW)

LI

Ultracapacitors(UC)

# 1: Capital Cost (Energy and Power)

FlywheelUltracapacitor Lead-acid battery

Lithium ion battery

Compressed air

20

Energy Cost ($/kWh)k

10,000 5,000 525 200 50

Power Cost

($/kW)

100 250 175 125 600

# 2: Volume Density (Energy and Power)

21


Lithium ion battery

Compressed air

Energy Density(Wh/L)

30 80 150 80 6

Power Density (W/L)

3000 1600 450 128 0.5

𝒗𝒌𝒆𝒏𝒈

𝒗𝒌𝒑𝒐𝒘𝒆𝒓

# 3: Discharge Time vs. Charge Time

22

Flywheel UltracapacitorLead-acid

batteryLithium ion

batteryCompressed

air

𝑟𝑘 h𝑑𝑖𝑠𝑐 𝑎𝑟𝑔𝑒

𝑟𝑘h𝑐 𝑎𝑟𝑔𝑒Po

wer

Time

Peak cap

Pow

er

Time

Peak cap

# 4: Lifetime

ESD Health• Charge-discharge life cycles• Depth of discharge ()


Lithium ion battery

Compressed air

Life Cycles(x1000)

100 200 15 5 2

Depth of discharge

Dead

23

# 4: Lifetime

ESD Health• Charge-discharge life cycles• Depth of discharge ()


Lithium ion battery

Compressed air

Life Cycles(x1000)

100 200 15 5 2

Depth of discharge

Dead

# 5: Energy Efficiency

25

Energy WastageInput > Output

Flywheel Ultracapacitor Lead-acid battery

Lithium ion battery

Compressed air

Energy Efficiency(%)

95 95 85 75 68

# 6: Self-Discharge Losses

26

Lose charge even not being discharged


Lithium ion battery

Compressed air

Self-discharge per dayμk

100% 20% 0.3% 0.1% low

# 7: Ramp Time

27

Start up time to change the power output

2727


Lithium ion battery

Compressed air

Ramp Time

Millisec Millisec Millisec Millisec Min

Pow

er

outp

ut

TimeRamp time

28

Given a workload, which ESD is best suited for reducing its peak?

29

1 10 100

32

UltraCapacitorFlywheelLead AcidCAES

Peak Width: W (min)

8

2

0.5

Inte

r-pea

k di

stan

ce: D

(hou

r)

0.1

Pow

er

Time

Peak cap

W

D

Cost-effective ESD for Different Demands“No Single ESD to Shave Them All !”

“No Single ESD to Shave Them All !”

30

1 10 100

32


Peak Width: W (min)

8

2

0.5

Inte

r-pea

k di

stan

ce: D

(hou

r)

0.1

Pow

er

Time

UC

UC


31

1 10 100

32


Peak Width (min)

8

2

0.5

Inte

r-pea

k di

stan

ce(h

our)

0.1

CAES

Pow

er

Time

Ultracapacitor

Pow

er

Time

CAES


32

1 10 100

32


Peak Width (min)

8

2

0.5

Inte

r-pea

k di

stan

ce(h

our)

0.1

FW

Pow

er

Time (W=1min)

Time (W=10min, D=0.5h)

Pow

er

Time (W=100min)

Ultracapacitor

CAES

Pow

er FW

Time(W=10min, D=5h)

Pow

er

LA

LA

Hybrid ESD solution may be desirable

33

Compressed Air

Battery

Ultracapacitor/flywheelPo

wer

Time

Multi-level Multi-technology ESDs

34

ATS

ESD

PDU PDU PDU…

…

UtilityDiesel Generator

ESD

……

ESD…

ESD

ESDServerH/W

Battery

Capacitor

Rack Rack Rack

Flywheel

Battery

Compressed Air

Talk Outline


35

Which ESDs?

How much capacity?

Where in hierarchy?

Linear program

Object: Optimize cost S.T. : ESD and workload

constraints

Methodology

36

Pow

er (W

)

Time

Power cap()

t=30s

Max. Demand()

Methodology

37

𝑃 𝐿,1𝑟𝑒𝑎𝑙𝑖𝑧𝑒

𝑃 𝐿−1 , 1𝑟𝑒𝑎𝑙𝑖𝑧𝑒

𝑅𝑘, 𝑙 , 𝑖 ,𝑡

𝐸𝑘, 𝑙 , 𝑖 ,𝑡

𝐸𝑘, 𝑙 , 𝑖 ,𝑡𝑆𝑘 ,𝑙 ,𝑖

, i=1

𝑙=𝐿−1

𝑙=1

𝑛1𝑖1

𝑃 𝐿,1𝑚𝑎𝑥

𝑃 𝐿−1 , 1𝑚𝑎𝑥 𝑃 𝐿−1 ,𝑛𝐿 −1

𝑚𝑎𝑥

𝑃 𝐿−1 ,𝑛𝐿 −1

𝑟𝑒𝑎𝑙𝑖𝑧𝑒

𝑃1 ,1𝑟𝑒𝑎𝑙𝑖𝑧𝑒

𝑃1 ,1𝑚𝑎𝑥

𝑃1 ,𝑖𝑟𝑒𝑎𝑙𝑖𝑧𝑒

𝑃1 ,𝑖𝑚𝑎𝑥

𝑃1 ,𝑛1𝑟𝑒𝑎𝑙𝑖𝑧𝑒

𝑃1 ,𝑛1𝑚𝑎𝑥

Optimization Problem: Objective

38

Max (CapExSaving + OpExSaving – ESDCost)– CapExSaving = (– OpExSaving = b ( + Dk,l,i,t - R)) – ESDCost = k,l,i k,l,i)

𝑃 𝐿,1𝑟𝑒𝑎𝑙𝑖𝑧𝑒

𝑃 𝐿−1 , 1𝑟𝑒𝑎𝑙𝑖𝑧𝑒

𝑛1𝑖1

𝑃 𝐿,1𝑚𝑎𝑥

𝑃 𝐿−1 , 1𝑚𝑎𝑥 𝑃 𝐿−1 ,𝑛𝐿 −1

𝑚𝑎𝑥

𝑃 𝐿−1 ,𝑛𝐿 −1

𝑟𝑒𝑎𝑙𝑖𝑧𝑒

𝑃1 ,1𝑟𝑒𝑎𝑙𝑖𝑧𝑒

𝑃1 ,1𝑚𝑎𝑥

𝑃1 ,𝑖𝑟𝑒𝑎𝑙𝑖𝑧𝑒

𝑃1 ,𝑖𝑚𝑎𝑥

𝑃1 ,𝑛1𝑟𝑒𝑎𝑙𝑖𝑧𝑒

𝑃1 ,𝑛1𝑚𝑎𝑥

Optimization Problem : Constraints

• State of charge is bounded by depth of discharge and maximum capacity:

(1 - ≤ k,l,i,t ≤

39

𝑫𝒐𝑫𝒌𝒎𝒂𝒙

𝑺 k , l , i


• State of charge is bounded by depth of discharge and maximum capacity:

(1 - ≤ k,l,i,t ≤ • Discharge/charge power are bounded by maximum

capacity and discharge/charge rates:0 ≤ ≤ 0 ≤ R ≤

40


• Net power draw at each Li is bounded by the cap: 0 ≤ P + - ≤

41

Optimization Problem Formulation: Constraints

• Net power draw at each Li is bounded by the cap: 0 ≤ P + - ≤ • Account for energy losses due to self-discharge

k,l,i,1 + δ - δ - μk

42



k,l,i,1 + δ - δ - μk

• Constrained by ramp rate of discharge

43



k,l,i,1 + δ - δ - μk

• Constrained by ramp rate of discharge

• Volumetric constraints (10% server-level, 20% rack-level, no datacenter-level constraint)

≤

≤

44

Talk Outline


45

Realistic Power Profiles

46

(a) TCS (Indian IT Company) (b) Google

(c) MSN (d) Streaming Media

Cost Savings for Google Workloads

47

Server: LA

(Savings, ESD cost)

Datacenter: FW+CAESServer: LA

Datacenter: CAES

Savi

ngs (

$/da

y)

Single-tech,Datacenter-level

Multi-tech,Server Level

Multi-tech,Multi-level

Total cost without ESD is $12k/day

Single-tech,Server-level

0

1,000

2,000

3,000

4,000

5,000

6,000

(3.9k, 0.2k)25% 30%(4.9k, 0.4k) (4.7k, 0.3k)

Server:UC + LA

(5.2k, 0.3k)20%

0

1,000

2,000

3,000

4,000

5,000

Cost Savings for MSN Workloads

48

(Savings, ESD cost)

Savi

ngs (

$/da

y)

(4.0k, 0.5k)

Single-tech, Single-level Multi-tech, Single-level

Multi-tech,Multi-level

Total cost without ESD is $15k/day

Server: LA

Rack:UC + LA

Datacenter: FW+CAESServer: UC

Rack: LA

(3.8k, 0.3k)

Datacenter: LA

(4.3k, 0.3k) (4.2k, 0.2k) (4.4k, 0.3k)

Server:UC + LA

(3.4k, 0.1k)

Charge/Discharge Control for MSN demand

49

• CAES takes a bulk of the gap for significant portions of time• Ultra-capacitor is used for sudden spikes and gets charged

from CAES

Concluding Remarks

• Framework for holistic energy storage based Cap-ex and Op-ex optimization

• Representative results• ESD technologies beyond battery also useful in

datacenter context • ESD technologies employed at multiple-levels of

datacenter• Multiple technologies at multiple level

50

Penn State University

Thank you!

Our other related papers on Energy Storage: [ISCA 2011]: opex savings [SIGMETRICS 2011]: time of day

price variations [ASPLOS 2012]: under-provisioning

for capex savings

http://csl.cse.psu.edu/

52

Backup Slides

Model for a single ESD

Power requirement: Energy requirement: Charging requirement: Lifetime: = min() Amortized Cost = max (, , )

53

Pow

er

Time

hshave

hvalley

hpeak

wvalleywpeak

Optimization Problem Formulation: decision variables

• ESD capacity of type k at (l,i): k,l,i

• State of charge at time t: k,l,i,t

• Discharge rate: Dk,l,i,t

• Recharge rate: R• Realized peak in sub-hierarchy (l,i):

54

l+1

l1 i n

… …

R

D

ESPrealize

Concern 2: Battery Health

55

Battery Health

Frequent discharges

Depth of discharge

56


$921,172

$1,137,615

$730,000

$249,720

Servers


Utility BillOther

8%

30.5%

24%

37.5%


Chart:Source: Book by Barroso et al.,

Assumption: 20,000 servers, 1.5 PUE, 15$/W Cap-ex, Duke Energy Op-ex,

4yr server & 12 yr infrastructure amortization (Tier-2)

Sriram G. et.al. propose a novel solution using energy storage

(ISCA’11, Sigmetrics’11, ASPLOS’12)


57

1 Day

Pow

er (W

)

Power Cap

Pow

er (W

)

Power Cap

1 Day

Shallowdischarge

Deepdischarge


5858

Time

Pow

er (W

) Power Cap

Time

Pow

er (W

)

Power Cap

Shallowdischarge

Deepdischarge

Day1

Day1

…

…

Year 1

Year 1

…

Year 3

Dead

0 20 40 60 80 1000

1000

2000

3000

4000

5000

6000

DoD - Depth of Discharge (%)

# Ch

arge

/Dis

char

ge cy

cles


59

Lead-acid Battery Lifetime Chart charge/discharges sustained

before requiring replacementHow to keep battery alive

for 4 years?Deeper Discharges

= Quicker Death

60

Battery operational rules(4 year lifetime constraint)

0 10 20 30 40 500

2

4

6

8

% Power drawn from UPS

No. o

f Hou

rs o

f UPS

op

erati

on p

er d

ay 20% of peak load can be sourced from

UPS for 2.5 hours every day


Restrict battery usage to meet lifetime constraint


• Fully charged at the beginning and end states:k,l,i,1

61

𝑺 k , l , i 𝑺 k , l , i 𝑺 k , l , i

𝑬 k , l , i ,𝟏 𝑬 k , l , i ,𝒕 𝐄 k , l , i , T+1

Documents

Penn State University