MEDIUM VOLTAGE ELECTRICAL SYSTEMSELECTRICAL SYSTEMS
for DATA CENTERSfor DATA CENTERS
Michael Mosman, PEChief Technical OfficerChief Technical Officer
CCG Facilities Integration Incorporated FACILITIES INTEGRATION
INCORPORATED
© CCG Facilities Integration Incorporated 15 September 2012
INTRODUCTION
CCG Facilities Integration Incorporated:CCG Facilities Integration Incorporated:• Headquartered in Baltimore, MD since 1992.• An Engineering/Architecture firm specialized
in high-reliability facilities.• Engineer and Program Manager for hundreds
of data centers from 1 to 72 Megawatts.• Programmed facilities both within the USA
and internationally.• Developed or employed several electrical
system innovations for critical facilities,
MEDIUM VOLTAGE ELECTRICAL SYSTEMS FOR DATA CENTERS Page 2
including Isolated-Parallel UPS.
AN OBSERVATION
The first choice for service andThe first choice for service and distribution voltage is 480 volts. Why? Traditional, safe design. Specs are already written around 480 V.p y UPS equipment is readily available at 480 V. Data centers are relatively compact for their Data centers are relatively compact for their
high power density, therefore don’t often require long runs at Medium Voltage forrequire long runs at Medium Voltage for voltage drop reasons like campus-wide electrical distribution systems
MEDIUM VOLTAGE ELECTRICAL SYSTEMS FOR DATA CENTERS Page 3
electrical distribution systems.
TYPICAL COMMERCIAL DESIGN Utility service commonlyUTILITY Utility service, commonly
Medium Voltage. Pad-mounted step-down
UTILITYSERVICE
SERVICE Pad mounted step down service Transformer with LV underground secondary feeder.GDE
SERVICE TRANSFORMER
Main Switchboard with service main breaker.
MAIN SWB
Optional backup Diesel Engine-Generator and transfer devices.
ESSENTIALLOAD
UPS
Optional UPS system to carry critical loads through switching t f
CRITICALLOAD
MEDIUM VOLTAGE ELECTRICAL SYSTEMS FOR DATA CENTERS Page 4
transfers.
LARGE LOW VOLTAGE DATA CENTERLarge Low Voltage electrical systems are certainlyLarge Low Voltage electrical systems are certainly possible, as in this 36 MW critical load data center.
32 UPS SYSTEMSSYSTEMS @ 600V, 1.33 MW
TWO UTILITY SERVICES
@ 35 kV, 30 MW
32 DIESEL ENGINE 16 SERVICE TRANSFORMERS
MEDIUM VOLTAGE ELECTRICAL SYSTEMS FOR DATA CENTERS Page 5
32 DIESEL ENGINE-GENERATORS @ 2.25 MW
16 SERVICE TRANSFORMERS @ 600 V, 5 MVA
MEDIUM VOLTAGE APPLICATION
Where are Medium Voltage distributionWhere are Medium Voltage distribution systems typically considered? In campus distribution where service runs
are quite long. At Engine-Generator Switchgear when
they are paralleled in massive numbers y pfor capacity. In high-rise buildings where shaft space isIn high rise buildings where shaft space is
a premium. NEVER in a computer room
MEDIUM VOLTAGE ELECTRICAL SYSTEMS FOR DATA CENTERS Page 6
NEVER in a computer room.
MEDIUM VOLTAGE APPLICATION
What are the perceptions of MediumWhat are the perceptions of Medium Voltage switchgear when compared to Low Voltage switchboards? More expensive.
Low Voltage switchboards? Less reliable.
More dangerous. Harder to maintain
Not as flexible. Needs more space Harder to maintain.
However, recent advancements in the
Needs more space.
However, recent advancements in the technology of certain Medium Voltage equipment are changing the realityMEDIUM VOLTAGE ELECTRICAL SYSTEMS FOR DATA CENTERS Page 7
equipment are changing the reality…
LINEAR ACTUATED MV BREAKERS Relatively new on market Relatively new on market. Have the “look and feel” of
Low Voltage breakers.g Can have integral trip units. Can be found in 5kV, 15kV
and 25kV Class. Fast tripping, typically 50ms. Far fewer moving parts. Endurance tests indicate
they may operate up tothey may operate up to 100,000 times without failure of breaker operating
MEDIUM VOLTAGE ELECTRICAL SYSTEMS FOR DATA CENTERS Page 8
p gmechanisms.
FRONT ACCESSIBLE SWITCHGEAR New enclosure architectureNew enclosure architecture
requires ⅔ the space of traditional MV switchgear,
ll i f i i leven allowing for minimal space behind switchgear.
Flexibility nearly equal to LV Flexibility nearly equal to LV switchboards.
Can be built with arc containment enclosures that vent explosive gas out the top or to the building
1.5’
5’FRONT ACCESS3.5’
top or to the building exterior.
Less costly than traditional
5
5’
FRONT ACCESSMV SWITCHGEAR
TRADITIONALMV SWITCHGEAR 8’
6’
MEDIUM VOLTAGE ELECTRICAL SYSTEMS FOR DATA CENTERS Page 9
Less costly than traditional switchgear.
6
UPGRADED DISTRIBUTION TRANSFORMERS New class of liquid-cooledNew class of liquid cooled
transformers incorporated into unit substations.
High flash point, biodegradable fluid (FR3®) safe for indoor applicationsafe for indoor application.
High BIL (150 kV) suitable for vacuum breaker control without snubbers.
Internal, fluid-immersed i d i hiprotection and switching.
High efficiency, 99% or better
MEDIUM VOLTAGE ELECTRICAL SYSTEMS FOR DATA CENTERS Page 10
better.Photos courtesy of Cooper Power Systems
TYPICAL LV & MV SYSTEMSUTILITY
SERVICEUTILITY
SERVICE
GDE
SERVICE
SERVICE TRANSFORMER GDE
SERVICE
SERVICE TRANSFORMER
MAIN SWBD MAIN SWGR
DISTRIBUTION TRANSFORMERS
ESSENTIALLOAD
UPS
CRITICALESSENTIAL
LOAD
UPS
(IN UNIT SUBSTATIONS)
LOADLOAD
CRITICALLOAD
TYPICAL 480 VOLT SYSTEM TYPICAL 4I60 VOLT SYSTEMTYPICAL 480 VOLT SYSTEM2500 kVA SERVICE TRANSFORMERSECONDARY: 277/480 V, 3-PH, 4-W
2000 kW, 480 V STANDBY E-G
TYPICAL 4I60 VOLT SYSTEM2500 kVA SERVICE TRANSFORMER
SECONDARY: 4160 V, 3-PH, 3-W2000 kW STANDBY E-G
MEDIUM VOLTAGE ELECTRICAL SYSTEMS FOR DATA CENTERS Page 11
4000 AMP SWITCHBOARDw/ 4000 A INSULATED CASE MAIN
1200 AMP SWITCHGEARw/ 600 A VACUUM BREAKER MAIN
COST COMPARISON #1 - FEEDERSThe greatest cost difference between systemsThe greatest cost difference between systems is in the feeders. Both of these service entrance ductbanks are rated 2500 kVA.
VS
4160 VOLT 3-PHASE 3-WIRE480 VOLT 3-PHASE 4-WIRE
VS
4160 VOLT, 3-PHASE, 3-WIRE500 AMP, 75’ LONG
(3) 500kCM 5kV XLPE in 5” PVC75’ of 5” PVC
480 VOLT, 3-PHASE, 4-WIRE4000 AMP, 75’ LONG
(8) SETS: (4) 750kCM XHHW in 4” PVC600’ of 4” PVC
180’ of 500kCM CU SHIELDED CABLE2 CU YD of CONCRETE(6) 5 kV TERMINATIONS
$9 000*
2720’ of 750kCM CU WIRE6 CU YD of CONCRETE
(64) 600 V TERMINATIONS$67 000*
MEDIUM VOLTAGE ELECTRICAL SYSTEMS FOR DATA CENTERS Page 12
$9,000$67,000* INCL. LABOR, MATERIAL, O&P (YOUR MILEAGE MAY VARY)
COST COMPARISON #2 - SYSTEMSTYPICAL 480 VOLT SYSTEM TYPICAL 4I60 VOLT SYSTEMVS
ENGINE-GENERATOR:2000 kVA, 4160 V
$700k
ENGINE-GENERATOR:2000 kVA, 480 V
$650k
TYPICAL 480 VOLT SYSTEM TYPICAL 4I60 VOLT SYSTEMVS
$700kSWITCHGEAR LINEUP:
1200 A, 4160 V, w/ MAIN BKR,EG BKR & (2) FDR BKRS
$650kSWITCHBOARD:
4000 A, 480 V, w/ MAIN BKR,EG BKR & (2) FDR BKRS
$250kMAJOR FEEDERS:
75’ 500 A MAIN FDR, 100’ 500 A EG FDR &
$350kMAJOR FEEDERS:
75’ 4000 A MAIN FDR, 100’ 3000 A EG FDR, 100 500 A EG FDR &
150’ of LOAD FDRS @ 250 A$50k
DISTR. TRANSFORMERS:1500 kVA 480 V
100 3000 A EG FDR,100’ 1200 A LOAD FDR &50’ 2000 A LOAD FDR
$200kDISTR TRANSFORMERS 1500 kVA 480 V
1000 kVA 480 V$100k
_____________
$1 100k
DISTR. TRANSFORMERS:NONE
_____________
$1 200k
MEDIUM VOLTAGE ELECTRICAL SYSTEMS FOR DATA CENTERS Page 13
$1,100k$1,200k
COST COMPARISON #3 – LARGE SYSTEMSLarge systems allow transformer consolidation.
35 kV OUTDOOR SWITCHGEAR
UTILITY SERVICE35 kV, 15 MVA
UTILITY SERVICE35 kV, 15 MVA
15 MVAVS
Large systems allow transformer consolidation.
4160 V OUTDOOR SWITCHGEAR
35 kV OUTDOOR SWITCHGEAR
(6) 2500 kVA
15 MVAVS
MAIN SWGR
MAIN SWGR
MAIN SWGR
MAIN SWGR
MAIN SWGR
MAIN SWGR
MAIN SWBD
MAIN SWBD
MAIN SWBD
MAIN SWBD
MAIN SWBD
MAIN SWBD4160 V
(TYPICAL)
(1) 35 kV SWGR, 1200 A: $450k(6) 35 kV FDRs, 200 A, 150’: $80k
$
(1) 15 MVA TRANSF, 4160V: $250k(1) 5 kV SWGR, 3000 A: $550k(1) 3000 100 $20
MAIN SWGR MAIN SWGR MAIN SWGRMAIN SWBD MAIN SWBD MAIN SWBD (TYPICAL)
(6) 2500 kVA TRANSF, 480V: $400k(6) 480 V FDRs, 4000 A, 75’: $400k(6) TYPICAL 480 V SWBD &
DISTRIBUTION SYSTEM: $7 200k
(1) 5 kV FDR, 3000 A, 100’: $20k(6) 5 kV FDR’S, 500 A, 150’: $120k(6) TYPICAL 4160V SWGR &
DISTRIBUTION SYSTEM: $6 600k
MEDIUM VOLTAGE ELECTRICAL SYSTEMS FOR DATA CENTERS Page 14
DISTRIBUTION SYSTEM: $7,200kTOTAL $8,530k
DISTRIBUTION SYSTEM: $6,600kTOTAL $7,540k
MEDIUM VOLTAGE ADVANTAGEBesides the cost advantage there are
Greater safety – lower arc flash
Besides the cost advantage, there are other reasons to consider MV systems:
Greater safety lower arc flash. Smaller footprint – new compact switchgear. Higher reliability longer lasting breakers Higher reliability – longer lasting breakers,
cooler running ductbanks. Lower maintenance simpler breakers Lower maintenance – simpler breakers,
fewer terminations. Greater flexibility choice of secondary Greater flexibility – choice of secondary
voltages, longer feeders without huge cost. Greener building less copper PVC steel
MEDIUM VOLTAGE ELECTRICAL SYSTEMS FOR DATA CENTERS Page 15
Greener building – less copper, PVC, steel.
ARC FLASH SAFETY
Counterintuitive to common belief MVCounterintuitive to common belief, MV systems can have far less arc flash than equivalent power LV systems: Currents are lower in MV systems.equivalent power LV systems:
The new MV linear breakers are faster than LV breakers of equal power rating. Arc currents in MV systems are typically
closer to available fault levels than in LV t B k t i f t li blsystems. Breakers trip faster, more reliably.
Faster tripping + lower currents produce far l
MEDIUM VOLTAGE ELECTRICAL SYSTEMS FOR DATA CENTERS Page 16
less arc energy.
ARC FLASH STUDYMAXIMUM ARCING FAULT CASE #1 (LV) CASE #2 (MV)Utility Characteristics 34.5kV, 750MVA, X/R = 8 34.5kV, 750MVA, X/R = 8
Primary Protection Vacuum fault interrupter, 400 A
Vacuum fault interrupter, 400 A
Service Transformer 2500 kVA 2500 kVA
Secondary Voltage 480 volt, 3-ph, 4-wire, grounded
4160 volt, 3-ph, 3-wire, grounded
Service Entrance Feeder 75' (8) 750kCM/ph, 600V XHHW in PVC
75' (1) 750kCM/ph, 5kV XLPE in PVC
Main Breaker(w/ integral long time & Insulated Case 4000A Solenoid-Actuated (w/ integral long time &short time electronic trip)
Insulated Case, 4000A Vacuum Bottle, 600A
Max Bolted 3-ph Fault Current 53.52 kA 6.41 kAMax Arcing Fault Current at Bus 25.47 kA 6.27 kAMax Arcing Fault Current at Bus 25.47 kA 6.27 kANormal Mode Arcing Time 0.4 seconds 0.4 seconds
Arc Flash Boundary Distance 180 inches 128 inches
Incident Energy 23 1 cal/cm² 4 1 cal/cm²
MEDIUM VOLTAGE ELECTRICAL SYSTEMS FOR DATA CENTERS Page 17
Incident Energy 23.1 cal/cm 4.1 cal/cm
PPE Category 3 1
ARC FLASH STUDYLIMITED ARCING FAULT CASE #3 (LV) CASE #4 (MV)LIMITED ARCING FAULT CASE #3 (LV) CASE #4 (MV)
85% of Maximum Arcing Fault Current 21.65 kA 5.33 kA
A i Ti (@ 85% f lt) 1 0 d 0 4 dArcing Time (@ 85% fault) 1.0 seconds 0.4 seconds
Arc Flash Boundary Distance(@ 85% fault) 297 inches 107 inches
Incident Energy (@ 85% fault) 48.5 cal/cm² 3.5 cal/cm²
PPE Category (@ 85% fault) Dangerous 1
MAINTENANCE MODE CASE #5 (LV) CASE #6 (MV)MAINTENANCE MODE CASE #5 (LV) CASE #6 (MV)
Maintenance Mode Arcing Time (max. fault) 0.06 seconds 0.06 seconds
A Fl h B d Di tArc Flash Boundary Distance (maint. mode) 50 inches 18 inches
Incident Energy (maint. mode) 3.5 cal/cm² 0.6 cal/cm²
MEDIUM VOLTAGE ELECTRICAL SYSTEMS FOR DATA CENTERS Page 18
PPE Category (maint. mode) 1 0
SWITCHGEAR FOOTPRINT
1.5’
5’FRONT ACCESSMV SWITCHGEAR
METAL-CLAD
3.5’
TYPICAL
3.5’
6’5’
METAL-CLADMV SWITCHGEAR 8’
TYPICALLV SWITCHGEAR
6
5’ 6’5’
F t ibl M di V ltFront accessible Medium Voltage switchgear is smaller than both t diti l t l l d it h ANDtraditional metal-clad switchgear AND Low Voltage switchboards of equivalent
MEDIUM VOLTAGE ELECTRICAL SYSTEMS FOR DATA CENTERS Page 19
power.
DUCTBANK INTEGRITY
MUTUAL HEATING
NO MUTUAL HEATING
Multiple underground LV ductbanks close together (common in large facilities) can
NO MUTUAL HEATING
together (common in large facilities) can be destroyed by mutual heating.MV ductbanks are far less susceptible
MEDIUM VOLTAGE ELECTRICAL SYSTEMS FOR DATA CENTERS Page 20
MV ductbanks are far less susceptible.
BREAKER EVALUATION600 volt breaker endurance testing switchboard w/four brands of insulated-case breakers, 9/2005.
5 kV b k5 kV breaker endurance testing switchgear lineup w/two metal-clad b k d tbreakers and two linear-actuator breakers, 2/2007.
MEDIUM VOLTAGE ELECTRICAL SYSTEMS FOR DATA CENTERS Page 21
Photos courtesy of Power Distribution Systems.
BREAKER RELIABILITY
VS
600 VOLT INSULATED CASE BREAKERSQUARE D MASTERPACT NW
MV MAG-ACTUATED VACUUM BREAKERABB VM-1
Endurance test results: The best 480 V breaker performed nearly
25 000 i b f f il25,000 operations before failure. The best 4160 V breaker went over 70,000
ti b f t ti t dMEDIUM VOLTAGE ELECTRICAL SYSTEMS FOR DATA CENTERS Page 22
operations before testing was stopped.
MAINTAINABILITY
Linear actuator type Linear actuator type breakers have fewer parts, simpler construction than
YES
pmolded case LV breakers.
Medium Voltage switchgear have fewer bus connections and wire terminations than LVterminations than LV switchboards.
MV switchgear have lessMV switchgear have less heat related issues than LV switchboards.
NO
MEDIUM VOLTAGE ELECTRICAL SYSTEMS FOR DATA CENTERS Page 23
FLEXIBILITY
4160 VOLT4160 VOLT MAINS & EG’s
230/400 V. UNIT SUBSTATION
600 V. UNIT SUBSTATION
Providing separate transformation for essential
MECHANICAL PLANT UPS SYSTEMS
Providing separate transformation for essential loads and critical loads allows the secondary voltage to be tailored to load requirements.g q
Medium Voltage main and generator feeders allow the distance to load to grow without huge increase
MEDIUM VOLTAGE ELECTRICAL SYSTEMS FOR DATA CENTERS Page 24
in cost and losses.
SUSTAINABLE CONSTRUCTION?
MEDIUM VOLTAGE ELECTRICAL SYSTEMS FOR DATA CENTERS Page 25
GREEN IDEAS
Eliminate 80% of the PVC and copper Eliminate 80% of the PVC and copper content in data center electrical distribution.R d t f h t l b 60% Reduce transformer heat losses by 60%, and major feeder losses by a factor of 50. Make electrical equipment rooms 25%
smaller and require less building structure. Bring total critical power losses between
utility grid and computer rack under 5% by:y g p y Developing 4160 volt UPS systems and, Using 4160 – 230/400 volt PDU’s in the
MEDIUM VOLTAGE ELECTRICAL SYSTEMS FOR DATA CENTERS Page 26
gcomputer room.
THE TRULY GREEN DATA CENTERUTILITY SERVICE (GRID)
GDE
SERVICE TRANSFORMER WITH 4160V SECONDARY & 99%+ EFFICIENCY
STANDBY EG w/ 4160V ALTERNATOR
MAIN SWGR
4160V MAIN SERVICE SWITCHGEAR WITH FAST LINEAR BREAKERS AND LOW ARC FLASH4160-600V UNIT
UPS
SUBSTATION w/ 99%+ EFFICIENT TRANSFORMER FOR MECHANICAL
4160V UPS SYSTEM w/ 99% EFFICIENT ECONOMY MODE
ESSENTIALLOAD
600 VOLT
EQUIPMENT 4160-400/230V PDU’sw/ 98%+ EFFICIENT TRANSFORMERS
600 VOLT
CRITICALLOAD
400/230 VOLT
MECHANICAL PLANT USING FREE-AIR COOLING, HOT-AISLE CONTAINMENT AND OTHER
400/230V DISTRIBUTION TORACKS w/ 99%+ EFFICIENT BUSWAYS AND TAPS
MEDIUM VOLTAGE ELECTRICAL SYSTEMS FOR DATA CENTERS Page 27
400/230 VOLTCONTAINMENT AND OTHER ENERGY REDUCTION METHODS CRIT PWR LOSS = 5% MAX GRID TO RACK
COMMENTS OR QUESTIONS?
If you have any comments or questions about this presentation please contact
Mi h l M PEMichael Mosman, PE [email protected]
or visit our websiteor visit our website www.ccgfacilities.com
FACILITIES INTEGRATIONINCORPORATED
MEDIUM VOLTAGE ELECTRICAL SYSTEMS FOR DATA CENTERS Page 28