Execution Environments for Distributed Computing

Intelligent Placement of Datacenters for Internet

Services

EEDC

343

30

Master in Computer Architecture, Networks and Systems - CANS

Homework number: 6

Members: Roger Rafanell roger.rafanell@bsc.es

2

Outline

Introduction Motivation Placing Datacenters Evaluation Conclusion

2

3

Introduction: Datacenter construction costs

Each datacenter costs >$100M to construct– The smaller datacenters are rated at ~25MW

Examples:– Microsoft DCs in Virginia & Chicago: $500M each

3

4

Outline

Introduction Motivation Placing Datacenters Evaluation Conclusion

4

5

Motivation

Internet services require thousands of servers

Use multiple “mirror” datacenters– High availability and fault tolerance

– Low response time

Spend millions building and operating datacenters

Consume enormous amounts of brown energy!!

6

Outline

Introduction Motivation Placing Datacenters Evaluation Conclusion

6

7

Intelligent placement of datacenters

Goal: Manage the monetary and environmental costs

Define framework Model costs and datacenter characteristics Create solution approaches Collect cost and location-related data Create placement tool

7

8

Selecting datacenter locations

Model datacenter placement– Network latencies

– Availability

8

9

Selecting datacenter locations

Model datacenter placement– Network latencies

– Availability

CAPEX costs– Distance to electricity and networking infrastructure

– Land and construction (maximum PUE)

– Power delivery, cooling, backup equipment

– Servers and networking equipment

9

10

Selecting datacenter locations

Model datacenter placement– Network latencies

– Availability

CAPEX costs– Distance to electricity and networking infrastructure

– Land and construction (maximum PUE)

– Power delivery, cooling, backup equipment

– Servers and networking equipment

OPEX costs– Maintenance and administration

– Electricity and water prices (average PUE)

10

11

Selecting datacenter locations

Model datacenter placement– Network latencies

– Availability

CAPEX costs– Distance to electricity and networking infrastructure

– Land and construction (maximum PUE)

– Power delivery, cooling, backup equipment

– Servers and networking equipment

OPEX costs– Maintenance and administration

– Electricity and water prices (average PUE)

Incentives (taxes)

11

12

Selecting datacenter locations

Model datacenter placement– Network latencies

– Availability

CAPEX costs– Distance to electricity and networking infrastructure

– Land and construction (maximum PUE)

– Power delivery, cooling, backup equipment

– Servers and networking equipment

OPEX costs– Maintenance and administration

– Electricity and water prices (average PUE)

Incentives (taxes)

12

13

The problem formulation

Goal– Minimize CAPEX and OPEX

Constraints– Response times < MAX LATENCY for all users– Min consistency delay between 2 DCs < MAX DELAY– Min system availability > MIN AVAILABILITY

Output– Number of servers at each location– Minimum cost

13

14

Solving the (non-linear) problem

Linear Programming– Does not support non-linear costs

Brute force– Too slow

Simple heuristics– May not produce accurate results efficiently

14

15

Our approach for solving the problem

Evaluate each potential solution– Quickly via Linear Programming (LP)

Consider neighboring configurations– Simulated annealing (SA)

Cost optimization process– Combine SA and LP

15

Current solution Near neighbor

LP

SA

LP

16

Our approach for solving the problem

16

LP

SA

LP

LP

SA

LP

SA

$13.8M/month

$9.2M/month $10.7M/month

$10.3M/month

17

Summary of our approach

1) Generate a grid of tentative locations

2) Collect data about each location

3) Define datacenter characteristics

4) Instantiate optimization problem

5) Solve optimization problem

17

18

Outline

Introduction Motivation Placing Datacenters Evaluation Conclusion

18

19

Comparing locations for 60k-server DC

0100020003000400050006000700080009000

Austin Bismarck Los Angeles

New York Orlando Seattle St. LouisCost

(tho

usan

d do

llars

per

mon

th)

Servers Land Building Connection Energy Water Staff Networking

19

20

Interesting questions

How much does…… lower latency cost?

… higher availability cost?

… faster consistency cost?

… a green DC network cost?

… a chiller-less DC network cost?

20

21

Cost of 60k-server green DC network

21

Green DC network costs $100k/month more, except when latency <70ms

22

Cost of a 60k-server chiller-less DC network

0

2

4

6

8

10

12

14

30 50 70 90 110

Cost

(in

mill

ion

dolla

rs)

Maximum latency (milliseconds)

Chiller-less

Traditional

22

Chiller-less DC network is cheaper but it cannot achieve low latencies

23

Conclusions

First scientific work on smart datacenter placement– Proposed framework and optimization problem

– Proposed solution approach

– Characterized many locations across the US

– Built a tool to automate the process

– Answered many interesting questions

Results show that smart placement can save millions Work enables smaller companies to reap the benefits

23

24

Future work

To extend with data from Europe

Include tax incentives

Test the tool with data from real services

24

25

Questions

26

Specially thanks to the real authors of the work …

Íñigo Goiri, Kien Le, Jordi Guitart,Jordi Torres, and Ricardo Bianchini

26

27

Energy costs and carbon emissions

Company #ServersEnergy/year

(MWh)Energy

cost/yearCO2/year

(Metric tons)

eBay 16K 0.6 x 105 $3.7M 0.4 x 105

Akamai 40K 1.7 x 105 $10M 1.0 x 105

Rackspace 50K 2 x 105 $12M 1.2 x 105

Microsoft >200K >6 x 105 >$36M >3.6 x 105

Google >500K >6.3 x 105 >$38M >3.8 x 105

Sources: [Qureshi’09], EPA

27