PowerPoint from January 27th 2012 Forum

Friday January 27th, 2012 Comcast Boot Road Data Center

Agenda

• 8:30-9:00 Breakfast and networking

• 9:00-9:15 Welcome» Logistics» Introductions» Topics for Discussion

• 9:15-10:30 Session One:

– New Year, New objectives and priorities… – Vendor Spotlight:

» Peter Panfil, Emerson Network Power VP Global Power Sales• Current trends and future challenges

• 10:30-10:40 Break

• 10:40-11:30 Data Center Tour

• 11:30-12:00 Session Two:

– User Spotlight: Donna Manley – Post mortem procedures

• 12:00-1:00 Lunch

2012 Goals• Expansion of data centers• Audit with RFID• Bussway

Vendor Spotlight:

Peter A. Panfil VP Global Power Sales

Emerson Network Power

The State of the Data Center:Current trends and future challenges

AC Power Technology AC Power Technology

DVL DCUG

Jan 2012

Page 6COMPANY CONFIDENTIAL

AgendaAgenda Top data center challenges

Distributed (1+N) and central (N+1) static switch

Transformer based and transformer free UPS

Static switches improve MTBF in distribution

Distribution voltage trends and considerations


Infrastructure Management

Heat Density

Availability

Energy Efficiency

Power Density

Top Data Center Challenges Top Data Center Challenges

Source: Data Center Users’ Group Survey

Virtualization,Cloud

Increasing Demand

Reduced Budget

Higher Density

Regulation Compliance

IT Outsourcing Consolidation

Efficiency & Green

initiatives

External forces

changing the business climate

Business & technology

forces pressing on the data

center

Facility Challenges

7


Top Data Center Manager ConcernsTop Data Center Manager ConcernsRank Spring 2005 Fall 2007 Spring 2008 Spring 2009 Fall 2009 Spring 2010 Spring 2011

1Heat Density

78%Heat Density

64%Heat Density

56%Heat Density

55%Availability

56%

Monitoring Infrastructure

Mgt51%

Availability53%

2Power Density

64%

Power Density

55%

Power Density

50%

Energy Efficiency

47%


Mgt49%

Heat Density49%


Mgt52%

3Availability

57%

Energy Efficiency

39%

Availability45%


Mgt46%

Heat Density46%

Availability47%

Heat Density47%

4Space

Constraints32%

Availability33%


Mgt43%

Availability41%

Energy Efficiency

40%

Energy Efficiency

44%

Energy Efficiency

44%

5Change

Management28%

Space Constraints

29%

Energy Efficiency

40%

Power Density

35%

Power Density

25%

Power Density

36%

Power Density

29%

6


Mgt18%


Mgt27%

Space Constraints

26%

Space Constraints

29%

Space Constraints

25%

Space Constraints

21%

Space Constraints

18%

Monitoring / Infrastructure Management properly balances the needs of efficiency and availability

Data Center Users’ Group Survey


Trend Liebert Solution

Energy Efficienc

y

High-Efficiency Products

Liebert APMLiebert NXLiebert NXL

Features Improving Efficiency

Softscale Intelligent Eco ModesTP1 (Energy Star) Rated Distribution TransformersDistribution Voltages (240/139 and 415/240V)

Services Data Center Power & Cooling Assessments

Increased Density

Increasing Power Requirements

2 Stage (Segmented) Distribution400A Panel Boards w/ 100% Rated MainsBusway SolutionsMPX – up to 60A Rack PDU575V NXL

System Focus

Partnerships & Marketing Materials

Universal Switchgear ProgramLarge Systems Design Guide

High Availability Topologies

Development of 1+N for Large SystemsModular Systems with Internal Redundancy

Renewable Energy

Existing Products Flywheel Systems

New Products

Alternative Energy StorageSolar – ENPC: Solar Controller, EP: Solar

Inverter; DOE Funded Smart Grid ResearchWind – ENPC: Wind Converter

Liebert AC PowerTrends and Strategies


Dual Corded Dual BusRequires custom switchgear for power tie

Maximum Loading N/2For 4x1000 kVA=2000 kVA Max Load

UPS 1 UPS 2

PDU PDU

STS STS

UPS 3 UPS 4

PDU PDU

STS STS

Interleaved Dual BusDoes not require complex switchgear

STS does the power tieMaximum Loading N/2

For 4x1000 kVA=2000 kVA Max Load

UPS 1 UPS 2 UPS 3 UPS 4

PDU PDU PDU PDU

STS STS STS STS

UPS 1 UPS 2 UPS 3 Reserve

STS STS STS

PDU PDU PDU

Reserve/Catcher Dual BusDoes not require complex switchgear

STS does the power tieMaximum Loading N-R


50% Utilization

75% Utilization

50% Utilization

66% Utilization

Ring Dual Bus (Distributed Reserve)Does not require complex switchgear

STS does the power tieMaximum Loading (N-1)/N


STS STSSTS STSSTSSTS

UPS 1 UPS 3

PDU PDUPDU PDUPDUPDU

UPS 2

High Availability ConfigurationsHigh Availability Configurations


UPSCore

SS SS SS

Paralleling Cabinet

UPSCore

UPSCore

IT Load

UPSCore STS

System Control Cabinet

UPSCore

UPS Core

IT Load

Distributed Bypass (1+N)

Distributed static switches

Individual modules manage load transfers

Cannot parallel different sized UPS

Central Bypass (N+1)

Centralized static transfer switch

System-level control, fault tolerant

Size of STS determines total capacity

Options for parallel redundant UPSOptions for parallel redundant UPS


N+1 vs 1+NN+1 vs 1+N For a system of 4x750kVA

– 1+N will cost $1.2M , max aic 100kaic– N+1 will cost $1.4M , max aic 200kaic– If specifications allow both the 1+N will always be cheaper

When operating on inverter both have identical performance– N+1 has better fault transfer to bypass due to one 3000/4000amp breaker– 1+N has more sag due to parallel SS/inductors/1200a CB during fault transfer.– Since MTBF of NXL module is 200,000 hours the 4 module system will transfer to

bypass every 50,000 hr or 6 years if capacity and statistically never if redundant

1+N since it is composed of SMS can easily be split and sent to different locations– Requires two upstream feeder breakers or single input kit versus one for N+1

NEC70E requires both to have downstream ROB to be able to service one module while system is energized


Transformer Based UPS System Single Module, Topology Three-LineTransformer Based UPS System Single Module, Topology Three-Line

MIB

OutputMBB

3P

CB2

BFB

E

BIB

EG

FBO

AC

FBO

AC

GEC

MBJ

N N

EG

A

Trap Disconnect

CB1

12P isolated

12P non isolated

or

To Batteries

Negative DC bus

Positive DC bus

Output -AC

+

+

+

+

+

+

Battery and DC Bus

Isolation

Input Isolation

Output Isolation

Neutral-Gnd Management;Low Common Mode Noise;

Separately Derived Source

Bypass can be connected to separate utility source


Transformer Less UPS System Single Module, Topology Three-LineTransformer Less UPS System Single Module, Topology Three-Line

No Input Isolation

Additional DC Converter

Less Eff Rectifier

High DC Bus

Neutral Mgt / Control Required

Some topologies require the bypass to be connected to

the same utility source

No Output Isolation


Application PhilosophyTransformer Based & Transformer Free UPSApplication PhilosophyTransformer Based & Transformer Free UPS There are appropriate applications for both transformer

based and transformer free UPS– Many customers have multiple applications with different priorities

Transformer based enterprise UPS’s offer the highest availability – Galvanic isolation is provided for DC fault protection– Output isolation protects the critical load and simplifies fault

management– Use ultra-reliable, efficient SCR-based rectifiers and simple lower

voltage inverters– Can feed rectifier and bypass from dual Separately Derived Sources– Inherently compatible with High Resistance Grounded systems

Transformer-free UPS’s offer low TCO with high availability– Double conversion efficiency up to 96% – Uniformly Low input harmonics with consistent high power factor.– Power distribution provides complete solution for transformer free UPS


Characteristic Transformer Free

Transformer Based

AC-AC Double Conversion Efficiency

96% Range 94% Range

Eco Mode Efficiency Up to 99% Up to 99%

Ground Fault Protection Coordination

External or Incremental

Inherent

Arc Flash Mitigation External or Incremental

Inherent

> 480 volt ratings for high power density

Additional External Xformers Required

No Additional External Xformers Needed

Reduction in Common mode noise and EMI

No Yes

Rectifier Resiliency IGBT vs. SCR

Lower Higher

High Resistance Ground Compatibility

No Yes

Transformer Based –vs-Transformer Free Design

Transformer Based –vs-Transformer Free Design


Tier 3-4 UPS Power configuration into a Dual Input IT Load

MTBF UPS Out = UPS 1 = UPS2 Primary AC Input MTBF = S10 UPS MTBF out = > 1.6M hr MTBF Field ObservedBypass AC Input =

100 Hr MTBF

Bypass AC Input PDU MTBF > 9 M hrUPS 1 Field-Observed

Primary AC Input SMS MTBF > 1.7 M hr PDU

IT each AC Input MTBF =Simplified - Components in series

= 1/((1/MTBF UPS) + (1/MTBF PDU)) IT LoadMTBF = 1.4 M hr

Primary AC Input PDUUPS 2

Bypass AC Input SMS

System MTBFWithout Static SwitchesSystem MTBFWithout Static Switches

Module Demonstrated MTBF Block Diagram


Tier 3-4 UPS Power configuration with STS 2 into a Dual Input IT Load

MTBF UPS Out = UPS 1 = UPS2 Primary AC Input MTBF = S10 UPS MTBF out = > 1.6M hr MTBF Field ObservedBypass AC Input =

100 Hr MTBF

Bypass AC Input STS MTBF > 7.2 M hr PDU MTBF > 9 M hrUPS 1 MTBF > 1.7 M hr Field-Observed

Primary AC Input SMS STS2 PDU

enter

UPS MTBF = 1.7Combined MTBF of two STS 2 Field-Observed MTBF:Paralleled UPS outputs Field-observed STS MTFB output

= MTBF1+MTBF2+((MTBF1*MTBF2)/(MTTR)) ≈ 7.2 M hr MTBF IT LoadMTBF = 113,441 M hr IT each AC Input MTBF =

Simplified - Components in series where Mean Time to Repair [MTTR] = 24 hrs = 1/((1/Para UPS Out) + 1/MTBF STS) + (1/MTBF PDU)) where UPS repair is less than 8 hrs enter MTBF = 4.0 M hr

Primary AC Input STS2 PDUUPS 2

Bypass AC Input SMS

System MTBF Improvement With Static SwitchesSystem MTBF Improvement With Static Switches

Module Demonstrated MTBF Block Diagram


STS2

Multiple Rows

Service Feed

Surge Suppression

Engine Generators

Generator Paralleling Switchgear

Feed to UPS

Input Switchgear

Precision Cooling

LBS

RDC/FDC

RDC/FDC

Racks

UPS A

UPS B

Alte

rna

te

Prim

ary

Prim

ary

Alte

rna

te PDU: PPC/FPC

PDU: PPC/FPC

STS2

Dual Bus = twice as many power cables

Traditional Dual Bus, 2NTraditional Dual Bus, 2N


480VAC • RDC• UPS • Rack480/277V 480/277V


480VAC • UPS • PDU • Rack480V 208/120V

600VAC • UPS • PDU • Rack600V 415V

600VAC • UPS • PDU • Rack600V 208/120V


Tod

ay

Em

erg

ing

Efficiency improvement ???

Distribution VoltagesDistribution Voltages


Isolation Transformers At The PDUIsolation Transformers At The PDUPROS Single point ground, separately derived source with

safety ground closer to the load reduces susceptibility to lightning and other transients

Only requires a 3 wire system to the PDU input Provide impedance which reduces available fault currents

~ and Arc Flash potential at distribution points

CONS Size – PDU’s with transformers can be larger Transformation losses …However…today’s TP-1

transformers are typically 98.5% + efficient Higher weight and cost


PDU Transformer Efficiency

PDU Transformer Efficiency

96.00

96.50

97.00

97.50

98.00

98.50

99.00

15% 25% 35% 50% 65% 75% 100%

% LOAD

% E

FF

ICIE

NC

Y

300 kVA K20

300 kVA STD

300 kVA K20 TP1

300 kVA STD TP1

$$$


A Fresh Look at the 400-415v SystemA Fresh Look at the 400-415v System Modern Power supplies are wide ranging 208v to 240v test

– Higher voltage equates to higher efficiency – about 0.3% gain Line to neutral connection – 230/400 or 240/415v

– Can be transformer free saving energy-1-3% gain, plus cooling savings– Fault current HAS been a major concern if transformer free

• 480 or 600v to 240/415 v with Auto (efficiency) or Iso. (aic and N-G) • Historically, vendors supplied pieces and parts, but not an end-to-

end solution for 400-415V in North America.– Neutral fault path and neutral noise are concerns with transformer free– No Rack PRU balancing issue

Line to Line connection – 120/208 and 127/240v– New copper TP-1 Transformers have 1.5% losses– Fault current is controlled by the transformer

• Panels, breakers, power cords, rack PDU and servers rated for fault current (aic) are readily available

– Neutral fault path and neutral noise are from server to isolation transformer only


Short Circuit Considerations (Historical)Short Circuit Considerations (Historical)Panelboards 208/120 & 240/139 Volt Panels

Rated at 250V

– Type NQ

– Available to 22kAIC

480/277 & 415/240 Requires Panels Rated to 600V

– Type NF

– Series rated with main CB at• 35,65 and 100kaic

– Physically larger

– More costly (10-25%)

Are your Rack PDUs and servers rated for this high AIC?


Fault CurrentArc Flash ConsiderationsFault CurrentArc Flash Considerations Arc flash?

– Bolted vs. arcing faults

– Significant incident energy released during the arcing event and is considered the “arc flash hazard”

NFPA 70E-2004 “A flash hazard analysis be done in order to protect personnel from the possibility of being injured by an arc flash”

Determination of required PPE - Personal Protective Equipment

Calculation of incidence of energy

– Ampere rating of over current protective device

– Operating time of the device

– Available fault current is key!!!


Solving the 415V AIC IssueSolving the 415V AIC Issue

PDU

UPSSYSTEM

3250 kVA34.5 kV – 480/277Z >= 5.32%

Isc ~ 73,480A

Isc ~ 17,576 A

300 kVA480V – 208/120VZ >= 4%

X

X

208 Volt Distribution

PDU

UPSSYSTEM

3250 kVA34.5 kV – 415/240Z >= 5.32%

Isc ~ 84,989A

Isc ~ 56,144 A

X

X

415 Volt Distribution

RACKIsc < 5kA

RACKIsc ~ 10-12kA

Problem• AC Distribution panels

• Lighting panels• Exposed buss (arch flash)• AIC of UL approved “touch

safe”• Rack PDUs

• AIC may exceed safe design

Solutions• Introduce impedance such as inductor or transformer

• Disadvantage efficiency• Advantage grounding and

fault management (tx)• I-Line Panels offer higher AIC (100k) and safer design• Higher AIC capable RPDU’s


Maximize your investment in breakers and gear with higher UPS System voltages

The higher the chosen voltage - the greater the potential capacity – 15% to 25%

5000 Amp System UPS System Voltage

415V 480V 600V

Max. Bus Capacity 3590 kVA 4152 kVA 5190 kVA

Distribution Voltage ConsiderationsUPS System Voltage and Capacity Distribution Voltage ConsiderationsUPS System Voltage and Capacity


Distribution Voltage Pros & ConsDistribution Voltage Pros & ConsPROS CONS

480 – 208/120

600 – 208/120

Most commonly accepted application Reduced aic – fault curent Uses standard 240V panelboard & breakers N-G bond at PDU

2-3% transformation energy loss 208V requires 2 pole breaker Reduces the number of poles

480 – 400/230

600 – 400/230

N-G bond at PDU( iso) Higher energy efficiency Higher energy density Higher UPS capacity - kVA Reduced AIC – fault current

0.5 to 1.3%% transformer energy loss Can’t power 120V equipment More circuits due to1-pole N-G bond (auto) at service entrance

480 – 480

480 – 480/277

No transformation energy losses No neutral required (unless 277V loads)

Can’t power 120V or 240V equipment Requires 480V panelboard & breakers Few servers at 480V & 277V Higher AIC – fault current at load

480 – 415/240

No transformation energy losses 240V load requires 1 pole breaker More useable pole spaces Higher energy efficiency

N-G bond at bypass transformer Requires 480V panelboard& breakers Requires UPS Maint Bypass Xfmr Higher AIC – fault current at load

415 – 415/240

No transformation energy losses Reduced cooling load 240V load requires 1 pole breaker More useable pole spaces Higher energy efficiency Save cost and weight of transformers in

PDUs

Can’t power 120V equipment Requires 480V panelboard& breakers Needs different approach to fault

current management N-G bond at service entrance Increase cost of full neutral and higher

ampacity – lower system kVA

Page 29COMPANY CONFIDENTIAL 29

29


Product Operating Hours Mod. MTBF Sys. MTBF

(Cur # Units) Oct 00 – Mar 11 Code 14 Code 15

Unfiltered Filtered

STS 2 (4,082) 121,462,656 Hrs. 886,589 Hrs. 8,675,904 Hrs.

S610 (6,642) 419,488,584 Hrs. 47,838 Hrs. 1,712,198 Hrs.

NXb (2,843)105,092,280 Hrs. 202,490 Hrs. 2,563,226 Hrs.

PPC (10,474) 878,674,536 Hrs. 2,670,743 Hrs. 10,217,146 Hrs.

FPC (1,951) 41,363,976 Hrs. 10,340,994 Hrs. 13,787,992 Hrs.

NXL (612) 5,581,704 Hrs. 206,730 Hrs. 1,395,426 Hrs.*

APM (113) 384,432 Hrs. 384,432 Hrs. 384,432 Hrs.

Liebert STS2, S610, NXb, PPC, FPC,NXL Reliability Summary May 2011Liebert STS2, S610, NXb, PPC, FPC,NXL Reliability Summary May 2011

Transfer To BypassCritical Bus Failure

* Updated Nov 2011 30


Power Business SegmentsPower Business Segments

Scale-OutCore Enterprise

Availability Capital/Operational Savings

Primary Technology

Transformer Based UPS

Transformer Based UPS in Eco-Mode

Transformer Free UPS

Transformer Free UPS in Eco-Mode

Battery on Server Model

Customer Type

Traditionalist Opportunist Experimentalist

Customer Behavior & Motivation

Firmly adheres to long-held, proven industry standards to

maximize infrastructure availability

Operates on the edge of acceptable operating

recommendations taking calculated risks to balance

financial costs and availability

Ventures outside of industry standards and best-practices with the goal to significantly

reduce financial costs

Customer Attributes

• Providing mission critical computing to customers

• Required uptime based on government regulations

• Extremely high cost of application downtime

• Providing less critical computing to internal or external customers

• Need to meet customer SLAs for uptime with limited penalties

• Balancing cost of downtime with OPEX

• Customers still expect application high availability

• Basic services are provided for limited fees with no guarantees

• High compute volume demands lowest computing costs possible

Critical Infrastructure

Primary Motivation

TCOAvailability Capital/Operational Savings


Dual Corded Dual BusRequires custom switchgear for power tie

Maximum Loading N/2For 4x1000 kVA=2000 kVA Max Load

UPS 1 UPS 2

PDU PDU

STS STS

UPS 3 UPS 4

PDU PDU

STS STS

Interleaved Dual BusDoes not require complex switchgear

STS does the power tieMaximum Loading N/2


UPS 1 UPS 2 UPS 3 UPS 4

PDU PDU PDU PDU

STS STS STS STS

UPS 1 UPS 2 UPS 3 Reserve

STS STS STS

PDU PDU PDU

Reserve/Catcher Dual BusDoes not require complex switchgear

STS does the power tieMaximum Loading N-R


50% Utilization

75% Utilization

50% Utilization

66% Utilization

Ring Dual Bus (Distributed Reserve)Does not require complex switchgear

STS does the power tieMaximum Loading (N-1)/N


STS STSSTS STSSTSSTS

UPS 1 UPS 3

PDU PDUPDU PDUPDUPDU

UPS 2

High Availability ConfigurationsHigh Availability Configurations

Break:See you in 15!

User Spotlight: Donna Manley

Post mortem procedures

Managing a Full Data Center Power Down

Donna M. Manley, MBAIT Senior Director, Computer Operations

University of Pennsylvania

July 3-4, 2009

36

July 3-4, 2009

37

January 2010 (Morning)

38

January 2010 (Noon)

39

40

41

Before you get started….

• Agree upon scope• Documentation • Validate infrastructure and architecture• Asset identification and application dependencies• Understand what pre-work can be completed• DR site and Vital Records storage providers on

standby• What’s the weather forecast?

Take the opportunity to do stuff you wouldn’t normally be able to without an outage!

42

43

44

45

Logistics….

• Coordination with Public Safety• Coordination with Facilities• Command Center• Know who will be there and when• Vendor Expectations• Accommodations, food, and beverage

46

47

48

Managing the Outage…

• Communication (Bridges, Web, Email)• Playbook• Change Freeze• Action Items to be remediated prior• Test plan• Points of Contact - Data Center/Facilities/Vendors• Who gives the “GO”

49

50

51

52

Post Outage…

• Communication (Bridges, Web, Email)• Execute Test Plan• Lessons Learned• Process and procedural modifications• Automation opportunities• Sleep!

Lots of great information was compiled for this event – keep it current!

53

Thank you for allowing me to share my thoughts with

you today!

Donna M. Manley, MBAIT Sr. Director, Computer OperationsITIL V3 Foundations CertifiedUniversity of [email protected]

Survey– Was this forum beneficial?– Was this the proper number of end users or

should the forum grow? If grow, please give a number you feel would be appropriate.

– How often should this forum meet? When?– What topics would you be interested in

discussing at the next meeting?– What other venues for this event would you like

to see?

Thank you for attending!