A New Building Data Center Upgrade capacity and technology step

A New Building Data Center Upgrade capacity and technology step

2011 – May the 4 th– Hepix spring meeting

Darmstadt (de)

Pascal Trouvé (Facility Manager CC IN2P3)

Summary :

Latest improvements of the current computing room New computing room - Project scope New Building – New architecture Technical commissioning

– Virtual tour– Technical data & figures

On going projects– Technical infrastructures phase II– Electrical power supply challenge

Future development project

[email protected] Hepix Spring Meeting - Darmstadt - Mai 4 th 2011

Latest improvements of the computing room

POWER

– New transformers have been installed (01/2010) : 3 X 1 MVA – Diesel generator (1.1 MVA)– Second electrical low voltage master distribution panel installed

(one dedicated for Operation power [racks and cooling], one dedicated to infrastructure power)

– Reactive energy compensation panel ( 4 capacitors banks) - savings– 4 Power distribution panels replaced + Source Transfer Switch for

critical Network racks installed– News batteries farm – sealed lead battery – cell watch system

[email protected] Hepix Spring Meeting - Darmstadt - Mai 4 th 2011v

Last improvements of the computing room

COOLING : – A third cooling unit (600 kW) has been installed on roof top,1800 kW

available– Cooling capacity is now maximum complying with environmental

regulation HAZARDS

– News fire safety detection system and monitoring ( SSI/CMSI)– New extinguishers fleet and reviewed lay out

INFRASTRUCTURE – High density confined racks standardization– InRow air-conditioning racks– PDUs suited to equipment specifications, and monitored


IMPROVMENT INPUTS

POWER No power distribution problems occurred over the last year Better batteries back up capacity coupled with the GE

COOLING Still not fully redundant (short of 1600 kW) on the existing

computing room No cooling issues over the last summer, redundancy will be

reached with the new computing room commissioning HAZARDS

Better fire safety with modern fire detection system Decrease of battery’s Hydrogen Hazard with sealed technology

INFRASTRUCTURE Better global efficiency and sustainability Higher general density


Power exemplification

6

New aeros on roof top

2 New chillers unit


Confinement and Water cooled racks

7

confinement

InRaws cooled racks

All the new hardware is installed in “confined racks”, including disks


Project scope New computing room

The identified needs :~ 60 users groups increased demand Meet LHC’s requirements To meet the planned upgrade of LHC Important identified request for « astroparticules” research activity


Project scope - Evolution plan – new

The new computing room is an extension of the existing computing room (3 MW power + cooling in 2011)

Target plan:

201150 racks0.6 MW

2015125 racks1.5 MW

2019216 racks3.2 MW

9

Total power 3 MW

Total power 6 MW

Total power 9 MW


A careful hardware selection

10

IBM iDataplexIntel 5430 LV –

2.66 GHz – 50 W16 GB

DELL 1U 1950b 3 GHz

16 GBBlades DELL

E5540 (2.53 Ghz)48 GB

Dell C6100X5650 – 2.66 GHz

72 GB


New Building – New architecture


31 mars 2011

Chiller units

Cold Water pumps stations

Special construction design – heavy load roof structure

Heavy duty Elevator 3 tons load

12Dominique Boutigny

Chilled water tank 30 m3

Infrastructure design scope

13

Fully confined architecture

All the water pipes, electrical distribution and other services are routed from the ceiling

[email protected] Hepix Spring Meeting - Darmsatdt - Mai 4 th 2011

New computing room

3 levels of 850 Sq. Meters each [ machine room for 850 sq. meters and 2 technical levels including the roof top] + heavy load elevator

We adopted a simplified “Design and build procedure” (single company contracted) delivered the building within18 months

Due to budget constraints the new computer room start with a limited technical infrastructure, but has a modular design that allowed scalable upgrades

First years, room dedicated to worker nodes only

Chilled water and electricity distribution and back up, technical rooms are designed for the 2019 target value


The New Data Center Building


Setting up of a new building live !

• 12 months construction work• >= 630 tons of concrete• >= 70 tons of steel framework•

Building work started end of April 2010

Targeted delivery date targeted February 1st 2011Final delivery date : March 14 th 2011Technical commissioning : undergoingGo live production ~ Mai 24 th 2011

16

Virtual Tour

24 APC racks : InRaw 20 KW eachConfined + workers + 2 Penduit racks with Nexus + switch Cisco

To be completed by end of year :

26 APC racks InRaw 20 KW and fully

confined area

Still 600 sqr meters available ……


Computing room phase I

17

Technical equipment and installation on roof top

• 2 Chillers units 800 KW each• Chilled water tank 30 tons –

30 min back up• Insulated piping and electrical

tracing• Spare place for future

upgrades


18

Power Distribution

• 2 UPS Galaxy 7000 – 600 KVa• Main Switch board 2 MW• Redundant Air cooling unit

[email protected] Hepix Spring Meeting - Darmstdt - Mai 4 th 2011

Fire protection

• “Vesda” permanent detection • Conventional fire detection system level one• Sprinkler (automatic water extinguisher system)

level 2• Monitoring

Power distribution structure

19

Cellules HTA

TGBT UPS critiques +

services

2 500 kVA

TGBT UPS

critiques

400 kVA

TGBT Workers

2 400 kVA

N Modules

clim

Critics

ServicesWorkers600 kW HQ

115 kW HQ

574 kW HQ

N+1

600 kVA

N

600 kVA

N NN

N

1 000 kVA

N+1

1 000 kVA

N+1

800 kVA

N

630 kVA

N

200 kVA

N Modules

clim

N Modules

climCritiques

N x (1+1/5)Modules

climServicesN

N+1

800 kW

N+1

800 kW


20

Phase II

Additional Equipment & facilities

• Additional power transformer• Third chiller unit 400 KW• Second Batteries farm• Piping and water pumps• Additional power switch boards


Project launched to be delivered by end of October 2011

A “ready to plug in” for 50 more racks units ……

On going projects

Main Electrical Supply redundancy project

21

2013 – 1,95/ 4 MW 1,95 MW will be backed

up by the power supplier

2016 -2019 - 4 / 9MW 4 MW will be guaranteed

by the power supplier

2011 - 1,7/ 6 MW

Final decision taken to face a new challenge


In the first phase the electrical redundancy will be minimal

On going projects

Future Development project

22

Amphitheatre accessible to the public

Multimedia center

Outreach center

Dedicated Training center

A “Hub” for fundamental

research, University and

public information

An office space was part of the original extension project but dropped due to budget limitation

Project Re-formatted with new functionalities added

Very well perceived by the Region and the city community (Grand Lyon)


Documents

A New Building Data Center Upgrade capacity and technology step