Uptime Institute Fall 2008 EPO alternatives

All Information in This Report is Confidential and Covered by an Information Exchange Agreement1© 2008 The Uptime Institute, Inc.

EPO Alternatives Project2008 Accomplishments

Site Uptime® Network

2008 Fall Conference


Project History

In early 2007, a special project was initiated by the Network to address the significant risk to uptime attributed to disconnecting means controls, generally known as emergency power off (EPO) switches.

At each Network group’s spring 2007 conference, an EPO Alternatives working session was held.


Spring 2007 Working Sessions (Continued)

Participants volunteered ideas for each of two approaches designed to minimize EPO downtime:

Best practices for working within existing code requirementsRecommended changes to the codes

The cumulative list of ideas was recorded and reviewed last fall by the team of individual Network members who volunteered to pursue the project’s ongoing objectives.


Project Team Members

These Network members participated on the project team: Mike Bell of Office DepotDan Bonner of HPMatt Brown of HPJohn Clemens of HP Matt Gustafson of SSAJack Knavel of HSBCMike Lavazza of CiscoJerell Myers of Wachovia


Project Team Members (Continued)

Paul Peoples of Target Gregg Rudinski of Morgan StanleyLarry Rushing of HPJoe Stephenson of WachoviaWayne Whitcomb of BoeingDavid Boston of the InstituteRichard Schlosser of the Institute


Project Objectives

During the fall 2007 conference, project team members identified four sub-teams that would work concurrently on the following deliverables:

Task Team 1: Develop list of safety enhancements that will result from accepting our proposed code changes.Task Team 2: Refine and complete list of recommended code changes. Task Team 3: Create best practices paper for minimizing risks with installed EPO systems.


Project Objectives (Continued)

Task Team 4: Create a primer for those who wish to construct a data center to comply with the general requirements of NFPA 70 (no EPO requirement).

These teams worked on their objectives independently as the collective project team met monthly thereafter.In the meantime, AFCOM representatives confirmed they wished to join our project team. AFCOM had recently been developing their own plan to propose code changes.


AFCOM Project Team Members

The following AFCOM individuals participated:Steve McCluer of APCTom Roberts of Trinity HealthRick Sawyer of EYPWilliam DiBella, AFCOM’s President

Steve was the only project team member with previous code proposal writing experience.

AFCOM team members met in person with the Network twice in January. Included was a working session for the combined project team in Dallas.


AFCOM

AFCOM was originally the acronym for the Association for Computer Operations Management. Now it’s simply AFCOM.AFCOM members are professionals related to data center management and operations, including data center managers, operations managers, MIS directors, CIOs, CTOs and other IS/IT professionals. AFCOM members represent government, information services, insurance, healthcare, manufacturing and distribution, financial services, universities, high-tech, consulting, telecommunications, utilities, and more.


AFCOM (Continued)

AFCOM provides data center professionals with networking opportunities and educational forums and resources through its annual Data Center World® conferences, published magazines, regional chapters, and industry alliances. The AFCOM membership base represents more than 4,000 of the world’s largest data centers.


Optimizing the Code Change Process

Task Team 2 sought a personal connection on Code Making Panel 12 to advise us. After the 2007 fall Network conference, Greg Dolence of Progressive provided the project team with a contact from the NEC’s Code Making Panel 12: Lori Tennant from Schneider Electric. This was the connection we needed. Lori was supportive and recommended we talk with Joe Sheehan, the NFPA’s staff liaison for the NEC, to learn how to optimally work within CMP-12’s process.


Optimizing the Code Change Process (Continued)

We met in January 2008 with Joe Sheehan, who advised us to request a Task Group from CMP-12 to provide feedback on our ideas. This would allow us to hear objections and find compromises before actually submitting our proposal.Joe also advised that we inform local electrical inspectors (often the Authority Having Jurisdiction) of our intended code change proposal by attending their regional meetings in the fall of 2008. (This approach had helped another group’s proposal achieve acceptance.)



Joe further assisted us with our desire for a Task Group by officially introducing us to Tim Croushore, the CMP-12 chairperson.In late January, Tim Croushore reviewed a description of our code change proposal and assigned Bob Johnson, a member of CMP-12, to head a Task Group to work with us.The Task Group initially included the following CMP-12 members:

An electrical utility representativeA UL representativeAn electrical inspector



An electrical safety engineerAn electrical equipment manufacturer rep.A manufacturing company’s electrical eng.

In addition:Two members of NFPA 75 (protection ofinformation technology equipment) Four members of NFPA 76 (fire protection of telecommunications facilities)One NFPA staff memberSteve McCluer - representing AFCOMRichard Schlosser and David Boston -representing the Network


Project Team Progress

Over the winter and spring months, the four Task Teams made continued progress. Task Team 1 was the first to complete their work. Mike Bell, Matt Gustafson, Paul Peoples, and Rick Sawyer provided a list of safety enhancements (for firefighters, data center occupants, and data processing customers) that would likely be achieved if the code change proposal is accepted. This list was passed on to Task Team 2 to include with their code change proposal.


Project Team Progress (Continued)

This list included:Emergency services respond unnecessarily (taken away from real emergencies) if disconnecting means are accidentally deployedPeople on site are trained not to use it –may be a life safety issue in a real emergencyMany data centers directly threaten life safety if accidentally powered down (those that support VOIP, 911, medical, security companies, etc.)



Wayne Whitcomb was a one man “Task Team 3.” He developed the Best Practices for Minimizing Downtime Risks with Installed EPO Systems paper that you have now as a handout. Wayne utilized the information collected from spring 2007 Network working sessions in combination with his own experience and ideas from the other Project Team members.This document should be an immediate benefit for the majority of members who operate facilities with installed EPO systems.



This paper addresses various methods and designs that have been used to satisfy a wide range of “Authorities Having Jurisdiction (AHJs)” in their interpretation of the requirements of NEC Article 645.10. In particular, it focuses on established “best practices,” with particular attention given to the ones that provide the greatest protection from accidental, inadvertent, or malicious EPO activation (and are the most likely to be acceptable to most AHJ’s).



An optimal EPO design must include the following elements:Tamper resistant switching device(s) located only at each principal exit from the room.Switching devices that are installed at the same location at each door and away from other devices like light switches, fire alarms and door opening devices.A highly audible alarmed cover mounted over the switch. (This is documented to be the greatest deterrent to accidental and malicious operation and can help to quickly identify which switch was activated.)



All conduits and j-boxes used for the EPO system wiring should be clearly marked to distinguish them from other systems.Clear and concise labels and procedures posted at each switch location, in all applicable languages.A normally open activation circuit requiring the application of power to shut down the room.A means of bypassing the system in order to make modifications or to perform maintenance.



Manual operation only, with no interconnection to the smoke detection system or other means of automatic activation.A single source of power for tripping the circuit breakers feeding the IT equipment in the room.A simple and reliable design with good documentation as well as an obvious configuration.Segregation into the smallest “zones” or quantity of affected IT equipment that is allowed by the code.



The ability to quickly identify which device has been activated so that the appropriate response can be given and the device can be reset quickly.Cameras focused on each exit door, including the EPO switch. Training, including EPO specifics, for all who work in the data center. This should be provided the first day on site and at least annually thereafter. Install switches that have a unique appearance instead of a button –type device.


Disconnecting Means Device Cover


Disconnecting Means Device


Disconnecting Means Device With Cover



These methods have been approved by a small number of AHJs and may be worth considering:

A separate EPO system for each of two dual power paths, when dual power is provided. Switches and covers may have different colors for each power path.Install a single EPO in the main electrical room or the Command Center instead of at exit doors.



In summary, these guidelines should help existing data center owners with EPO systems optimize EPO related uptime. Some of the steps listed may be implemented only when upgrades to the facility are planned.



Joe Stephenson, Matt Brown, Dan Bonner, John Clemens, Larry Rushing, and JerellMyers worked together to develop Task Team 4’s primer for Constructing a Data Center Facility To Comply With the General Requirements of NFPA 70.

This is the second handout that you have.



Data centers can be constructed to meet enterprise IT requirements without conforming to NFPA 70 Chapter 6, Article 645, which requires DISCONNECTING MEANS (commonly referred to as EPO or Emergency Power Off) controls.Only chapters 1 through 4 of the NEC are mandatory. Chapters 5, 6 and 7 apply to special occupancies, special equipment or other special conditions in which additional requirements are necessary or in which exceptions can be made to the requirements of the previous four chapters.



Most data centers today are designed to conform to Chapter 645, which permits additional flexibility, cost savings, and convenience when installing communications and power cabling, with the provision that a disconnecting means control be provided at principal exits.If the electrical system designer follows the requirements of chapters 1 through 4 only, the disconnecting means is not a required part of the electrical distribution system, if the local Authority Having Jurisdiction (AHJ) concurs.



More than one Network member has elected to design and construct their data center facility to comply with the general requirements of NFPA 70 (chapters 1-4).The design team and owner should receive awritten commitment from the local AHJ before proceeding with this type of data center design.



Optional design choices to conform to Chapters 1 through 4 (subject to local jurisdictions):

Use non-raised floor design with overhead power and IT cabling distribution (low voltage overhead cabling) in the computer rooms. Or, install only wiring types permissible below a raised floor used for air handling (Chapter 3), such as:

Type MI cable (Article 332)



Type MC cable without a covering (Article 330)Type AC cable without a nonmetallic sheath (Article 320)Other non-metallic cabling shall be permitted to be installed in EMT conduit tubing. (Article 300)Other raceways without a nonmetallic sheath



If a raised floor is used, power and communication cabling under the floor must be enclosed, secured and protected from physical damage. Article 300.22(c) (Articles 320,330,332)

Power receptacles, conduit, and flexible cables are secured in place.



Secure plenum rated power and communication cabling in solid bottom metal cable tray with solid metal covers. Cables shall be secured in cable trays in accordance with Article 392.8.Enclosure and equipment ground must be in accordance with Article 250; cable trays in accordance with Article 392.7



Develop a corporate philosophy that identifies the cost of data center downtime events for the company and that prescribes the need to invest the extra labor and materials necessary to meet one of the two optional cabling applications defined in the previous section. The additional cost will be minimal in comparison to the potential downtime costs avoided.



Build data centers to conform to Chapters 1 through 4 wherever AHJ’s support data center designs without “disconnecting means.”Life/safety design strategies that include incipient and pre-action fire protection systems are preferred. Continuous shift Maintenance/Security, robust BAS monitoring, and comprehensive personnel training are preferred practices.



Challenges to Success:Insurance requirements.Regional/city specific building codes.Cost/budget constraints.Education and agreement of AHJ.Code interpretations, reasonable and prudent alignment.



In summary, Article 645 is an option that allows more lenient wiring rules in the IT equipment space as compared to the other Chapters of the NEC. A decision to design and build a data center solely within the requirements of Chapters 1 through 4 of the NEC (National Electrical Code) should be made only after the costs and physical cabling limitations described above are carefully weighed vs. the potential costs of downtime related to an EPO event.



Task Team 2 included Gregg Rudinski, John Knavel, Mike Lavazza, and Steve McCluer.This team significantly refined and added to the list of code change proposal ideas that the Network originally developed in spring 2007 working sessions.The team’s objective was to provide a document that captures the concept for the changes requested, as well as the details on how the changes could be implemented.Steve McCluer then incorporated this information in the first draft of the actual change proposal.



All Task Teams have completed their original deliverables, but their members remain as participants on the overall Project Team.

They will continue to provide input and feedback to AFCOM/Network CMP-12 Task Group members as the code change proposal development progresses.

As a next step, the EPO Alternatives Project Team selected Steve McCluer, Richard Schlosser, and David Boston to represent them as members of the CMP-12 Task Group.



CMP-12’s Task Group met for the first time on May 15th. At this point, only CMP-12 members and AFCOM/Network members were included.CMP-12 asked us to share the history of the Disconnecting Means requirement, which Richard Schlosser had previously researched.We then presented a matrix and draft proposal forms which Steve McCluer authored, based on input from Task Team 2.



CMP-12 members offered a number of ideas and suggested edits, most designed to simplify the proposal and make it easier to understand and enforce for AHJs.They also asked for several definitions to be added and for the substitution of preferred terminology in a few areas.The Task Group suggested that the chairperson invite NFPA members to participate now, instead of making them aware after CMP-12 reviewed the final proposal, as they normally would.



NFPA 75 and NFPA 76 representatives agreed to participate as part of the Task Group. A second Task Group meeting was held July 30th by conference call. The group now numbered fifteen. The chairperson asked the group what purpose they felt the disconnecting means requirement served.

The group was split between those who thought it no longer served a purpose and those who did.



The Task Group will therefore not try to eliminate the requirement at this time.Much of the meeting’s discussion focused on aligning definitions and wording between Article 645, NFPA 75, and NFPA 76. This was very tedious.Feedback on the AFCOM/Network change proposal was only received during the last hour of a five hour call. Most was detail oriented, rather than conceptual.



Seven subsequent Task Group meetings were held by conference call from August through October. Each lasted four hours.The process involved gaining consensus on each line of Article 645 of the Code. CMP-12’s chairperson had dictated that the group review the entire article.Unfortunately, this is indicative of the exhaustive committee work required to effect changes for codes (and other critical industry initiatives like energy efficiency policy).


Final Proposal

The Task Group’s final change proposal for 645.10 reads as follows:An approved means shall be provided to

disconnect power to all electronic equipment in the information technology equipment room or in designated zones within the room. There shall also be a similar approved means to disconnect the power to all dedicated HVAC systems serving the room or designated zones and shall cause all required fire/smoke dampers to close. Disconnecting means shall be implemented by either (A) or (B) below.


Final Proposal (Continued)

(A) Remote Controls for Disconnecting Means

1. Remote Controls for Disconnecting Means shall be at approved locationsreadily accessible in case of fire to authorized personnel and emergency responders.

2. The Remote Controls for Disconnecting Means for the control of electronic equipment power and HVAC systems shall be grouped and identified. A single means to control both shall be permitted.



3. Where multiple zones are created, each zone shall have an approved means to confine fire or products of combustion to within the zone.

4. Additional means to prevent unintentional operations of remote disconnect controls shall be permitted.

FPN: For further information see NFPA 75-2008,Standard for the Protection of Information Technology Equipment.



(B) No Remote Controls for Disconnecting Means

Remote Controls for Disconnecting Means shall not be required for critical operations data systems when all of the following are met:

1. An approved procedure has been established and maintained for removing power and air movement within the room or zone.

2. Qualified personnel are continuously available to meet emergency responders and to advise them of disconnecting methods.



3. A smoke sensing fire detection system is in place.

FPN: For further information on early warning fire detection systems, see NFPA 72-2007, National Fire Alarm Code.

4. An approved fire suppression system suitable for the application is in place.

5. Cables under a raised floor other than branch circuit wiring and power cords are in compliance with 300.22(C), 725.154(A), 770.154(A), or 800.154(A).(As required in Chapters 1-4.)


IAEI Education

Separately, AFCOM and Network project team members have handed out a summary of our proposal effort to electrical inspectors at five regional IAEI conferences this fall.

At each of these conferences, we worked from a desk or booth in the exhibit hall.We encouraged discussion and input from the inspectors (many of whom are AHJs) regarding our proposal effort. We believe this effort will help when the proposal review process occurs.


Additional Assistance - ITIC

The Information Technology Industry Council (ITIC) is made up of computer hardware and software manufacturers.They are frequently involved in change efforts with national organizations and code bodies.ITIC has indicated they will support our effort. We recently sent them a copy of our final proposal and asked that they help where they can.They have at least two individuals with experience participating on the code panels.


Next Steps

Because we reached consensus on the proposal wording before the proposal deadline, the CMP-12 Task Group has adopted our proposal and recently submitted it themselves.

This is considered a significant benefit, as it should carry a strong endorsement with the rest of the proposal review group.

As members of the Task Group, we will have one AFCOM/Network representative present at the Code Making Panel meetings (review of proposals) in January, 2009.


Next Steps (Continued)

From that point forward, we will monitor the proceedings and update Network members as we learn of decisions or feedback on our proposal.

If we see opportunities to help by asking Network, AFCOM, and ITIC members to share comments in support of the proposal, we will notify members of each group.

The projected schedule for the Code change process follows:


Projected Schedule – Code Changes

This cycle for changes is the 2011 NEC.The closing date for public proposals for changes to the 2011 NEC is November 7, 2008. Code Making Panel meetings will conclude January 24, 2009.NFPA’s Report on Proposals - July 14, 2009.NFPA’s closing date for receipt of comments – October 23, 2009.


Projected Schedule (Continued)

Code Making Panel meetings close (review of comments) – December 14, 2009.NFPA publishes Report on Comments –March 28, 2010.Intent to Make a Motion closing date –May 7, 2010.Association Meeting – June 5-9, 2010.Implementation of Changes Accepted –issued January 2011. Adopted into Law by Local Jurisdictions –months to years later.


Annual Reminder

Each year, our AIR trending report shows that several member sites still:Do not disable local PDU EPOs. Do not ensure the fire system has no connection to the EPO.Do not provide audibly alarmed covers, labels and consistent training (to all who will enter the computer room) for EPO switches.

Have you verified these conditions do not exist at your facility?


In Closing

If successful, it will be 2011 or later before our proposed Code changes are effective. The benefits will be substantial for the industry.

The immediate benefits for our members are the two documents you received today.You may implement Best Practices for Minimizing Downtime Risks with Installed EPO Systems as soon as you identify a plan.You may choose to build future data centers that do not require Disconnecting Means, as described in Constructing a Data Center Facility To Comply With the General Requirements of NFPA 70.


In Closing (Continued)

Please join me in thanking all our Project Team members, who have contributed many hours in addition to their “real” jobs:

Mike Bell of Office DepotDan Bonner of HPMatt Brown of HPJohn Clemens of HP Matt Gustafson of SSAJack Knavel of HSBCMike Lavazza of CiscoSteve McCluer of APC



Jerell Myers of WachoviaPaul Peoples of Target Tom Roberts of Trinity Health Gregg Rudinski of Morgan StanleyRick Sawyer of EYP Richard Schlosser of the Institute Joe Stephenson of WachoviaWayne Whitcomb of Boeing

It has been a particular pleasure to team with members of AFCOM for the first time.



Questions?