Data Center Commissioning: What you need to know

Data Center Commissioning:

What you need to know

Course # CXENERGY1511

AIA Provider # 50111116

AABC Commissioning GroupCxEnergy ConferenceHenderson, NVApril 30, 2015

Presented by: Judson H. Adams, P.E., CxA, ATDPower Management Corporation

AIA Stuff:

Credit(s) earned on completion of this course will be reported to AIA CES for AIA members. Certificates of Completion for both AIA members and non-AIA members are available upon request.

This course is registered with AIA CES for continuing professional education. As such, it does not include content that may be deemed or construed to be an approval or endorsement by the AIA of any material of construction or any method or manner ofhandling, using, distributing, or dealing in any material or product.

2

Course Description

Data centers have become a critically important

facility type in our modern economy. Because of

their unique requirements, commissioning them

properly requires some specific knowledge and

skills. This practical session will discuss uptime tier

classifications, working in live data centers,

balancing reliability vs efficiency, and much

more.

Also featured will be a review of example data-

center-specific functional performance and

integrated system tests.

3

Learning Objectives:

At the end of this course, participants will ….

1. Be familiar with the Uptime Institute® Tier

classifications

2. Understand the 5 levels of data center Cx

3. Be able to differentiate Integrated Systems

Testing from Functional Performance Testing

4. Understand challenges associated with Cx in a

live data center

4

Data Center

A facility used to house computer systems and associated components, such as telecommunications and storage systems. It generally includes redundant or backup power supplies, redundant data communications connections, environmental controls (e.g., air conditioning, fire suppression) and various security devices.-

-Wikipedia 2015

Internet

Enterprise

Telecommunications

Colocation (wholesale and retail)

Server/computer room

Telecom closets

5

Critical Infrastructure

Physical systems designed and constructed specifically

to support the hardware, network, and applications

that make up the “data center” :

Electrical Power

Mechanical Cooling

Fire Suppression

Automation

Monitoring

6

Critical Infrastructure Building Automation System

Chillers

Cooling Towers

Pumps/VFDs/Loop

Outside Air Units

Economizers

CRAHs/CRACs/IRCs

Makeup water and Pressure reducer valves

Electrical Substations

Generators and Fuel Delivery Systems

Transfer gear/ATS

Distribution Switchboards/Switchgear

UPS systems

Power Distribution Modules

Remote Power Panels

Critical Distribution Busway

EPO

Power and Environmental Monitoring Systems

Liquid Detection Systems

Lighting Control Systems

Fire Alarm and Detection Systems

Preaction Sprinkler Systems

Gaseous Fire Suppression Systems

7

Capacity Component

Capacity Component is any power producing and

energy storage equipment, such as:

Electrical power generator

UPS and Battery system

Chiller

Cooling Tower

Chilled water pump

CRAC/CRAH units

Fuel storage

Water Storage

8

Distribution path

A distribution path is the means by which the power is

transferred from the capacity component to the load,

such as:

Transformers

Electrical feeders

ATS, Switchgear, PDUs

Hydronic piping

Refrigerant piping

Valves

9

ITEQ Load

The load (expressed in Kilo-Watts) of the users

Information Technology Equipment. What’s in “The

Rack.”

Telecommunications provider equipment

Firewalls

Routers

Network switches

Servers

Storage arrays (SANs)

And others…..

10

Uptime Institute®

Tier Classification System

11

Uptime Institute®


An International Standard developed by The UTI that

defines criteria for the critical infrastructure design

based on four levels of classification:

Tier I – Basic Capacity

Tier II – Redundant Capacity Components

Tier III – Concurrently Maintainable

Tier IV – Fault Tolerant/Continuous Cooling

12

Uptime Institute®


Tier levels are defined at a specific ITEQ load using

ASHRAE Handbook Extreme outdoor design conditions

for the specific location (n=20 years)

Nashville, TN

Typical Cooling Design (0.4%) = 94.4 DB

Extreme (20 year high) = 103.1 DB

Las Vegas, NV

Typical Cooling Design (0.4%) = 108.3 DB

Extreme (20 year high) = 116.1 DB

13

Uptime Institute® - Tier I

Tier I – Basic Capacity

Self sustaining for 12 hours - Does not rely on utilities

Number and size of capacity components is

sufficient to maintain the operation of the business

between maintenance intervals

Maintenance requires shutting down the data

center

Subject to planned and un-planned outages

14


12 hour Water Storage N Generator(s)

15


N Chiller(s) N AC Unit(s)

16


Electrical Oneline

17

– From UTI white paper: Tier Classifications Define Site Infrastructure Performance –

Note: Example only, not prescriptive

Uptime Institute® - Tier II

Tier II – Redundant Capacity Components

Tier I, plus:

Number of capacity components provides

redundancy (N+1) for:

unexpected failure

maintenance without system shutdown

Maintenance of pathways requires shutdown

Subject to planned and un-planned outages

18


N+1 UPS N+1 Generators

19


N+1 Chillers N+1 Cooling Towers

20


Electrical Oneline

21

– From UTI white paper: Tier Classifications Define Site Infrastructure Performance

– Note: Example only, not prescriptive

Uptime Institute® - Tier III

Tier III – Concurrently Maintainable

Tier II, plus:

Multiple distribution pathways for power and

cooling (one may be passive)

All IT equipment has redundant power supplies

Generator must be rated for continuous operation

Each and Every single component and distribution

pathway can be removed for service/replaced

without shutting down the data center

Maintenance requires shudown

22


Double Isolation Valves

23

Uptime Institute® - Tier IIIRedundant Pathways

24

Uptime Institute® - Tier IIIRedundant Pathways

25


Main-Tie-Tie-Main

26


Electrical Oneline

27

– From UTI white paper: Tier Classifications Define Site Infrastructure Performance – Note:

Example only, not prescriptive

Uptime Institute® - Tier IV

Tier IV – Fault Tolerant

Tier III, plus:

Multiple distribution paths - both active

Compartmentalization between redundant

components/pathways

Class-A Continuous Cooling is required

Failures are self healing

Unexpected failure of any single component will

not interrupt the operation of the data center

The site is not subject to outages from a planned

event, or a single unplanned event

28


Continuous Cooling during loss of utility

29


Compartmentalization

30


Electrical Oneline

– From UTI white paper: Tier Classifications Define Site Infrastructure

Performance – Note: Example only, not prescriptive

31

Uptime Institute®

Tier Availability Matrix

* Based on recommended practice** Historical data from Uptime Institute membersSource: Data Center Site Infrastructure Tier Standard: Topology © 2012 - Uptime Institute, LLC

32

Tier I Tier II Tier III Tier IV

Planned Shutdowns for

maintenance*

2 every

year

3 every 2

years

Not

required

Not

Required

Unplanned Equipment or

Distribution Failures **

1.2 per

year

1 per year 1 every

2.5 years

1 every 5

years

Annual hours of

downtime**

28.8 22 1.6 0.8

Availability 99.67% 99.75% 99.98% 99.99%

Data Center Commissioning

aka

”Level 5 Commissioning”

33

“Level 5 Commissioning”

Level 0: Design Review

Level 1: Planning

Level 2: Factory Acceptance Testing

Level 3: Pre-Functional Inspections/Startup

Level 4: Functional Testing

Level 5: Integrated Systems Testing

34


CxA will assist with defining the OPR:

What is your target ITEQ Load?

What are your Resiliency Goals (Tier Level)?

Phasing?

How will the site be tested?

Make sure testing requirements are in the specifications

35

Photo source: www.zeniumdatacenters.com/the-risk-of-short-term-data-center-planning/


Review the Basis of Design:

Compare to OPR

Sequence of operations

Piping flow diagram

Electrical oneline diagram

Back-check at CDs

36

Level 0: Design ReviewA few pointers regarding controls:

Keep sequences simple

Fewer control points means fewer failures

Consider resiliency vs. efficiency

The fallacy of “UPS backup”

All capacity components should self sustainable

37

Level 1: Planning

Begins in design

phase

Prepare outline of

testing scripts

Test for

maintenance

modes (Tier III)

Test for failure

modes (Tier IV)

38

Level 1: Planning

Load bank placement for heat run test?

How will the different failure modes be tested?

How does the testing plan integrate with construction schedule?

39

Level 1: Planning

Existing Data Center Modifications:

Identify Critical MOPS (Methods of Procedure)

Is the sequence of construction integrated with Cx activities?

What is the user’s tolerance for risk?

40

Level 2: Factory Acceptance

Generators

Paralleling Gear

UPS Systems

Static Switches

Chillers

Custom HVAC equipment

41


Full sequence demonstrated

Safeties and alarms

Heat Run at full load

Transients recorded

Chillers:

Quick Start Low load

42


Why?

Testing performed by factory technicians

Deficiencies identified and corrected prior to shipment

Test Drive: Owner becomes familiar with system before owning it

43

Level 3: Pre-FunctionalOverall readiness for FPTs

Equipment matches submittals

Installation per construction drawings

SVCs complete

Manufacturer startup complete

Electrical Acceptance Testing

44

Photo source: http://www.tequipment.net/


Cooling Equipment:

Controls interface

Alarm notification Team-work mode

Heat Run for containment systems

Failure modes

Run Enable Failsafe Loss of Comm Power restore

45

Level 4:

Functional Performance Test

Example: Chiller Plant – Add/Drop Chiller

New DC3 Expansion (1,800 kW total ITEQ)

Added a 5th chiller to existing plant of 4

Replaced all primary CHW pumps (2N)

Installed active secondary CHW loop (Tier III)

Live Data Center

46

47

48

49


Generator/Transfer Equipment:

Test individual generators

Transients, load steps

Alarm notification

Thermal Imaging

Test parallel gear/ATS

Full Heat Run at rated load – Entire plant

50


UPS and STS Equipment:

Test individual modules

Inverter/Bypass Maint Bypass Battery discharge Alarm notification Thermal Imaging

Test parallel gear

Full Heat Run at rated load –Entire plant

Interaction with Generator(s)

51


Emergency Power Off (EPO):

Document how to restore

Confirm control power source

Avoid normally closed circuits

for UPS/PDU equipment

Interface with smoke dampers

Is there a way to bypass for

maintenance?

52

Level 5:

Integrated Systems Testing

Integrated System Testing is a

comprehensive test protocol that

incorporates all electrical power, cooling,

and control functions under design load

IST is typically performed over multiple

days

IST requires participation from all

contractors, vendors, the owner’s agent,

and the CxA

53

Level 5:


The “Prove it” Phase:

Heat Run: Demonstrate ability for the IT

equipment rooms to carry design load

under steady state design conditions

Low Load Conditions: Demonstrate

ability of the infrastructure to carry “Day

One” loads at steady state design

conditions

54

Level 5:



Loss of Utilities: Demonstrate the ability to

automatically react to utility outages (this

is NOT a failure – Tier I)

Maintenance: At load, demonstrate the

ability to remove capacity components

(Tier II) and pathways (Tier III) from service

without interruption

55

Level 5:


Perform maintenance procedures:

Remove a generator/UPS from the bus

Power down and isolate one electrical

switchboard/UPS/PDU completely

Shut off one Distribution panel serving

multiple CRAC units

Isolate a chiller /pump

Simulate a valve replacement

Simulate a complete BAS controls outage

56

Level 5:



Equipment Failure: Demonstrate the ability to automatically react to unplanned equipment failures (Tier II)

Pathway Failure: Demonstrate the ability to automatically react to unplanned pathway failures (Tier IV)

Example: Fire in a chiller room or electrical room – Assume total loss of room

57

Level 5:


Example: Pull the Plug – Low Load

3 Diesel Generators in parallel

Dual bus UPS system (A /B)

Dual Static Transfer Switches (A/B)

Air cooled Chillers (N+1)

Variable Primary pumping

Computer Room Air Handlers (CRAH)

58

59

60

61

62

Level 5:


Example: Heat Run and BAS failure

New DC3 Expansion (1,800 kW total ITEQ)

Five air-cooled chillers in parallel

Four primary CHW pumps (2N)

DP control (five sensors – fail safe)

Computer Room Air Handlers (CRAH)

Live Data Center – Tier III Design

63

64

65

66

Commissioning

within a “live” data center

67

Commissioning within a live

data centerNew Equipment Commissioning

Outage/failures during the testing process impact the business

Full load Systems Integration Testing is sometimes not feasible without scheduled downtime (Tier I and Tier II)

Medium risks

Full Facility Retro-Commissioning

It’s was never really commissioned before

Usually using “live load” for the test load

High Risks – Especially Tier I and Tier II Facilities

68

Commissioning within a live

data center

The CxA in a mission critical project will often be expected to prepare MOPs (Methods of Procedure) for all critical steps in construction and testing

The CxA is typically best suited for this role, as he/she has the best understanding of the overall risks related to the procedure

Critical MOPS must be identified early in the design process

Team effort – Must have sign off by owner, engineers, and contractors

Perform a Dry Run in advance

69

70Example MOP

71Example MOP LOG

Questions and Discussion

72

Documents

Data Center Commissioning: What you need to know