Intelligent placement of datacenters for Internet Services

Inigo Goiriyz, Kien Lez, Jordi Guitart, Jordi Torres, and Ricardo Bianchini

Presenter: Zafar Gilani

Introduction

• Selection of suitable datacenter locations is very important.

• Why?

– Running and maintenance costs.

– Network latency.

– Environmental factors (renewable energy vs carbon-intensive).

Important considerations for location selection

• Proximity to

– population centers,

– power plants, and

– network backbones.

• Source of electricity in the region.

• Electricity, land and water prices.

• Average temperatures of the location.

Framework

Framework for placement

• Goal:

– Minimize overall cost, while respecting response time, consistency and availability.

• Objectives:

– Formalize the process as a non-linear cost optimization problem.

– Automated datacenter location selection process.

Framework: Parameters

• Capital costs: investments made upfront.

Type of capital cost Description

Independent of number of servers

Electricity, external networking.

Maximum number of servers

Land acquisition, datacenter construction, power delivery, backup, cooling systems.

Actual number of servers

Purchase of servers, internal networking.

Framework: Parameters

• Operational costs: incurred during operation.

Type of operational cost

Description

Actual number of servers

Maintenance of equipment, external bandwidth usage.

Utilization of hosted servers

Electricity and water costs.

Framework: Parameters

• Response time.

• Consistency delay.

• Availability.

• CO2 emissions.

Framework: Optimization problem Placement of a datacenter at

locaton d, either 1 or 0.

Maximum number of servers at location d.

Number of servers that service population center c

at location d.

Framework: Optimization problem Placement of a datacenter at

locaton d, either 1 or 0.

Maximum number of servers at location d.

Number of servers that service population center c

at location d.

Framework: Solution approaches

• Make it linear.

Use linear version of CAP_max.

Remove Sd and Pd,c.

PBd,c is use of servers at location d to serve population center c, either 1 or 0.

This is actual number of servers at each

location d.

Framework: Solution approaches

• Using Heuristics:

1. Use simple linear program to generate M1 datacenter networks for 1 to D datacenters. We have M1 * D configurations.

2. Use SBd (placement) and PBd,c (use to meet demand) to derive pre-set linear program.

3. Select most popular locations and run brute force.

Framework: Solution approaches

• Simulated Annealing:

– For each candidate solution we have values for each location d and population center c.

– Optimization starts with a configuration and datacenter at each location.

– Each iteration evaluates a neighboring configuration.

– Iterate until no more cost reductions observed for n iterations.

Input data and datacenter characteristics for placement tool

Input data

OR

LA

NY

AU

SE BI

SL

Input data

Input data

Datacenter characteristics

• Datacenter size, cooling and PUEs. – 8% power delivery losses.

• Connection costs.

– $500K/mile for transmission. – $480K/mile for fiber optic. – $1 per Mbps. 1Mbps per server.

• Building costs. – As a function of maximum power: $15 per watt (small),

$12 per watt (large). – Availability: 99.827%

Datacenter characteristics

• Land cost. – 6K sq. ft. per MW

• Water cost. – 24K gallons of water per MW per day.

• Server and internal networking hardware. – $2K per server. – $20K per switch.

• Staff costs. – An admin can manage 1K servers for an average salary

of $100K/year.

Results from the tool, a few characterizations

Location characteristics

Location characteristics: observations

City PUE/Temp Land/Water cost

Network cost

CO2

emissions

Austin H L L L

Bismarck L L H H

Los Angeles H H L L

New York H H L L

Orlando H H L L

Seattle L H L L

St. Louis H L H H

A case study: placing a datacenter network

Evaluation

Evaluating solution approaches Heuristic was run for 3 days and then forcefully terminated, results were extrapolated.

OSA+LP1 is: •2x faster than Heuristic. •5x faster than Brute.

Datacenter placement tradeoffs: Latency

2x difference in price between

desired latency of 33ms and 50ms

$7.8M/month for latency 70ms or more

Datacenter placement tradeoffs: Availability

Cheaper to have 3 Tier II than 2 Tier IV

datacenters.

Overall Tier II datacenters are the

best option.

Datacenter placement tradeoffs: Consistency delay

Consistency delay and latency are conflicting

goals.

Acceptable ranges for consistency delay and

latency.

Datacenter placement tradeoffs: Green datacenters

A network of 8 datacenters with 60K

servers produces 8K tons of CO2/month.

With relatively higher latency of 70ms, it will cost $100K/month

more for green energy.

Will cost a lot more for lower latencies.

Datacenter placement tradeoffs: chiller-less datacenters

Avoiding chillers can reduce costs by 8% for

latencies > 70ms.

Conclusion

In a nutshell

• Intelligent placement of datacenters can save millions of $/€ .

• Cost of networks of datacenters doubles when maximum acceptable response time is reduce from 50ms to 35ms.

Intelligent placement of datacenters for Internet Services

Inigo Goiriyz, Kien Lez, Jordi Guitart, Jordi Torres, and Ricardo Bianchini

Presenter: Zafar Gilani