© 2014 Uptime Institute

Is your data center on the verge of a crisis?

Julian Kudritzki Chief Operating Officer

Uptime Institute

What Defines a Crisis?

2

Tour of Operational Computer Room

3

Looking for Clues

4

Tour of ‘Live’ Critical Spaces

5

Daily Practices Compromise Uptime, Safety, and Security

6

•  Overtime hours exceeding 10% •  Voice mail boxes full •  Emails not responded to •  Email inbox size limit exceeded •  Meetings missed or routinely cancelled •  No time for training •  Shortage of qualified staff •  Personnel performing work outside their competency •  Everything is an emergency •  Personnel turnover

What Else Is Going On?

7

•  Break fix budget exceeded •  Maintenance budget exceeded •  Energy cost estimate exceeded or unknown •  Last minute deployment requirements •  No organization chart •  No responsibilities matrix •  No records of maintenance activities •  No written policies & procedures •  No preventive maintenance schedule •  Back of the server looks like a spaghetti pot exploded

The Issues Add Up

8

•  Cabling is not labeled or worse incorrectly labeled •  Equipment is not uniquely labeled •  Loads are consistently out of balance •  Capacities are not managed or tracked •  Deferred maintenance exceeds 10% •  Housekeeping: if it looks like a mess, it is a mess Maybe you don’t have a crisis, but how do you know how well your data center operation compares to rest of industry?

The Issues Add Up

9

Are you confident in your Facilities team’s capability to manage a technologically advanced and highly efficient design to your 24 x 7 uptime requirements?

•  Can you easily replace any member of that team? •  Are you protected against poor operations practices

migrating from older sites to higher criticality data centers? •  Do you have sites that operate in isolation, ignoring global

corporate standards? •  Do you even have corporate global standards? •  If you outsource any aspect of your data center operations,

how do you avoid losing responsibility and accountability? •  Do you manage an outsourcing contract. . . . or direct an

expert team?

Ask the Tough Questions

10

•  Initial review •  Gap analysis against industry best practices

§  Staffing and Organization §  Maintenance §  Training §  Planning, Coordination & Management §  Operating Conditions

•  Roadmap to operational excellence •  Plan changes •  Implement changes •  Monitor & refine •  Annual review

Path to Data Center Operations Success

11

Key Elements of Facilities Management Staffing and Organization

•  Staffing •  Qualifications •  Organization

Maintenance •  Preventative Maintenance (PM)

Program •  Housekeeping Policies •  Maintenance Management

System (MMS) •  Vendor Support •  Deferred Maint. Program •  Predictive Maintenance •  Life-Cycle Planning •  Failure Analysis Program

12

Key Elements of Facilities Management Training

•  Data Center Staff •  Vendors

Planning, Coordination, and Management

•  Site Policies •  Financial Management •  Reference Library •  Computer Room Mgmt.

Operating Conditions •  Load Management •  Operating Set Points •  Alternating Use of

Infrastructure Equipment

13

The Uptime Institute over the years has observed management issues posing the largest risk to uptime physical infrastructure

•  Inadequate staffing •  Ineffective or non-existing maintenance and training programs •  Lacking processes and procedures •  Resulting in the majority of outages being caused by

‘human error’ No standard existed to help Owners/Operators determine

•  Common language/vocabulary  of  data  center  operations •  Focus of data center management •  Resource allocation •  Resource requirements

Genesis of Industry Best Practices

14

Data Center Owners / Operators / End Users •  Increased availability and cost savings •  Multi-site consistency •  Benchmark for continuous monitoring and refinement

Colocation / Managed Services Sites

•  All of the above plus… •  Customer assurance of consistency •  Competitive differentiator (attain & retain certification)

Industry Benchmark

•  No need to reply on opinions and anecdotes

Value of Industry Best Practices

15

Uptime Institute has been conducting Operational Sustainability Reviews for approximately 3 years— based upon decades of site operations knowledge and experience:

•  Operational Sustainability Certifications: Tier + Gold, Silver, or Bronze •  Management & Operations (M&O) Stamps of Approval

See http://uptimeinstitute.com/publications for Tier Standard: Operational Sustainability

Best Practices Reviews

16

Staffing •  Inadequate staffing •  Excessive overtime (over 10%) •  No escalation process

Qualification

•  No list of required qualifications •  No experience with data center specific equipment

Organization

•  Roles and Responsibilities not documented •  Data center organization not integrated

Staffing and Organization Significant Findings

17

Preventive Maintenance (PM) •  No list of required PM activities •  PM activities not fully scripted •  No quality control process

Housekeeping

•  Combustibles in the data center •  No documented housekeeping policy

Maintenance Management System (MMS)

•  No list of equipment •  Missing critical data: warranty info, maintenance history, performance

data, etc.

Maintenance Significant Findings

18

Vendor Support •  Contracts missing response times, call-in process, detail SOW, or

technician qualifications Deferred Maintenance

•  Unable to produce Deferred maintenance report from MMS Predictive Maintenance

•  No predictive maintenance program •  Not comparing current results with previous results

Maintenance Significant Findings

19

Life-Cycle Planning •  No life-cycle plan •  Not using MMS data to develop plan

Failure Analysis •  No record of outages or near misses

Maintenance Significant Findings

20

Data Center Staff •  Undocumented On-the-Job (OJT) programs •  No formal qualification program •  No list of training required by position •  No formal training program with lesson plans, etc.

Vendors •  No briefing for escorted vendors

Training Significant Findings

21

Load Management •  Alarm settings not documented •  Alarms not set on PDUs to ensure maximum loads are not exceeded

Operating Set Points •  Cooling set points are not document or part of

Change Management Process •  Changing of set points is not controlled

Operating Conditions Significant Findings

22

Site Policies •  Missing Site Policies •  Especially Site Configuration Policy

Reference Library

•  No process for keeping documents up-to-date

Capacity Management •  No process for forecasting future space, power, and cooling

requirements •  No active tracking of cooling capacity •  Ineffective management of Cold Aisles /Hot Aisles •  Electrical power monitoring (balancing phases)

Planning, Coordination, and Management Significant Findings

23

Facilities •  Operate and maintain the critical facility infrastructure •  Support the installation of IT equipment (space, power, & cooling)

IT Management •  Operate and maintain IT hardware, software, applications, and

network connectivity •  Manage the installation/de-installation of IT equipment

Security •  Access Control •  Physical Security

Typical Data Center Disciplines

24

Functionally Separate Organization •  Corporate Real Estate (Facilities) •  IT •  Security

Communication between organizations was typically poor

•  Data center activities conducted without coordination •  Poor future space, power, and cooling planning

No individual responsible for all aspects of operating a data center

Past Organizational Structures

25

Factors driving changes to organizational structure •  Rapid changes in technology and speed at which capacity must be

brought online •  Increased costs associate with IT and Facilities •  Business objectives of continuous computing availability

Legacy organizations could not accommodate quickly evolving business requirements

•  Slow to respond •  Not integrated

Evolving Organizational Structure

26

The value of industry best practices is in the process of continuous improvement

•  Discovery leads to learning •  Learning leads to change •  Change leads to improvement •  Regular reviews leads to discovery •  Crises can be avoided

Summary

27

For more information contact: Julian Kudritzki

jkudritzki@uptimeinstitute.com 206.706.4143

Questions?

© 2014 Uptime Institute 28