Operational Excellence Data Center Solutions Day, 5 July 2013, Martin Hogberg

Increasing Uptime in Data Centers by

Addressing Human Factors

Human Factors It’s not About Pointing Fingers

Mike, are you

racking the new

server?

Yea, some guy

installed more

power last night.

Ok put it in. Right, here

we go!

Common Issues in the NOC

Why is it important to focus on the operator?

1. Rick Schuknecht, The 451 Group Uptime Institute, June 2011

Availability/Downtime 75% of downtime due to human error1

• Over the total life cycle, most important decisions are made by the

operator.

• The amount of information from the process has increased dramatically

with risk for stress in critical situations.

• Experience and operator skills are invaluable assets.

• It is difficult and expensive to recruit and train new operators among the

young generation.

Traditional NOC environments Issues we are seeing across every industry

• A complex environment with a multitude of different systems and a low level of

integration.

• Very difficult to get an overview of what’s going on in the data center and what’s

about to happen.

• Lack of attention to human factors result in an unhealthy working environment.

Traditional NOC environments Issues that degrade reliability and profitability

• Poorly integrated CCTV and telecom equipment.

• Too many keyboards and other input devices.

• None ergonomic operator desks with built-in computers that generate heat,

noise and are difficult to keep clean.

Newer control rooms with traditional approach Issues that degrade safety, quality and profitability

• A newer centralized control

room built according to

traditional principles.

• Very low level of

integration results in total

screen and keyboard

overload.

• Overview information

screens very difficult to

view.

Operations Environment

University evaluating the positive alertness effect by working with Modern Concepts In the spring of

2012 Chalmers

has been testing

the

effectiveness of

the EOW, and

compared it to

the traditional

way of working

with control

room design. To

facilitate the test

Chalmers has I

corporation with

ABB created a

simulation of a

“color factory”

State of the art ergonomic and high

level of integration environment

Research Trials Traditional Operations

Basic level of integration that fulfill

standard traditional requirements

How to measure the alertness effect ?

Chalmers testning the Eow Concept

The tests has been conducted but using subjective and

Objective measuring tools as well as interviews

The subjective Tools measure

the perceived comfort and stress

level of the operator.

The SAM form one of several different

forms that use to evaluate the result.

The objective tools measure

the reaction time, cursor path

and number of problems

resolved

The specially developed facility

How to measure the alertness effect?

Chalmers evaluating the effect of the EOW Concept

Basic level of integration

that fulfill standard and

traditional requirements

Subjective Objective Interviews

State of the art

ergonomic and high

level of integration

environment

Subjective Objective Interviews Analysis and evaluations of

operators

environment and

technology solutions

resulting in expected

findings of

5-10 % increased operator

effectiveness

Integrated operations High information density and ergonomic workplaces

Interactive, close large overview ”owned” by the NOC operators

Easy to use individual ergonomics

Built-in communication tools simplifies collaboration

Saves up to 30% floor space compared to traditional solutions

Adaptive NOC Environment

Extended Operator Workplace as a centerpiece

of control room.

Cover all operator needs from one

console.

Futuristic NOC equipment.

Desk raises up on critical alarms so

Operator has to stand to deal with

situation.

Blows cold air to maintain Operator

alertness.

Sound shower directs operator through

audible alarms.

Can link to lights & window-blinds.

Going beyond the operator workplace

Overall control room environment is critical

Dedicated space for control, removes distractions and focus operators

on task of controlling the facility.

Visitor area to keep non-essential personnel out of the control area

Collaboration space. Meetings, troubleshooting, problem solving with

A/V tied to control center visualization.

Relaxation area for extended shift recharging, separate eating area to

avoid noise, segregated printers to remove noise

Helping the Teams and Communication - Integration

Data Centers – Management Today

Isolated Silos with Individual Planning,

Applications and Reporting

System and Network Management, Servers, Network and Storage

Building, Power and Environmental Management

Change Management, Help Desks and Trouble

Tickets

Integrated Reporting: Monthly Spreadsheets and Slides

IT Facility Operations

Utility Services

ABB Decathlon™ System

Security

Switchgear RDU CRAC VESDA/NOVEC

Chiller

Control

Fire

Protection

Integration Command Center – Single Pane of Glass

Alarm List

Trend Display Documentation

Faceplate

Navigation

Display Operator Note

One display management system

Consistent methods for navigation and display access

Supports power, cooling, and IT

Provides local and remote display access

Improves operational efficiency with better visibility

Creating a Collaboration Culture

CCTV

IT, planning the moves adds

changes.

Utility & Facility, verifying the

necessary space, power and

cooling is available.

Possible though an integrated

system.

Integration of Facility and IT Work Faster and Avoid the Mistakes

Creating a Collaboration Culture

Effective Software which Supports the Operator

Reduce the Isolated Systems Information Kept With the Asset

Data center based information

access

One Click to all information

Navigation based on job function

Real-time decisions and action

Control

Faceplates

Trend

Graphics

Video

Reports

Documentation ERP/ CMMS

Asset

Health

Integrated Information – CRAC Example

Trends

Alarm Instructions

Documentation

Operator Note

Overview of Effective Data Center HMI

High Performance Design

User Interface - The human performance improvement is

supported by selection of color scheme, shape of elements

and icons and tab based navigation.

Addressing Alarm Flooding - The effectiveness of new

operator interface concepts for improving operators’ ability

to handle data center disturbances that generate alarm

floods.

Integrated Alarm Management Alarm Management - A key concern for data centers

Too Many Alarms! Operators acknowledge/silence

alarms without looking at or

acting on them.

Incidents or near-incidents

where operators missed alarms.

Acoustic alarms turned off.

Operators don't know what

particular alarms mean.

Alarms disabled/suppressed for

long periods without review.

© ABB / PA / Control Systems

July 8, 2013 | Slide 25

Integrated Alarm Management Mission-Critical performance

View alarms from cooling, power, and IT in a

single list master list as well as individual lists

sorted chronologically, by priority, or by equipment

and filtered by location, functionality, or network

connectivity

Embedded analysis tools for determining the most

frequent offenders, the ones that take the longest

to remedy, or those that result in the greatest

downtime

Integrated Alarm Management

Enables a starting point for alarm rationalization

Analysis of ”worst cases”

Allows for continuous analysis and optimization

View alarms from cooling, power, and IT in a single list

sorted chronologically, by priority, or by equipment and

filtered by location, functionality, or network connectivity

Consistent alarm-response navigation

Fast navigation from an alarm reveals detailed information

needed to handle it correctly.

© ABB / PA / Control Technologies July 8, 2013 | Slide 28 3BSE068040 en A

Alarm grouping replaces long lists

Damage limitation no longer depends on an individual’s

ability to ‘piece together’ information from several

simultaneous alarms

© ABB / PA / Control Technologies July 8, 2013 | Slide 29 3BSE068040 en A

Condition Based Maintenance

The System Guides the User

Common format to determine faults in real time including Description of fault Possible cause Suggested action Priority / Severity of fault

SMS / e-mail messaging

Online snapshot reports for better change management Reduces time for shift handovers and maintenance stops

Reduced time to get right information Thanks to the integration capabilities of the system

the ”snapshot” covers all DDC, PLC’s etc regardless of brand

Enables real-time shift reports for example Navigation with the context menu that’s accessible from the

report

Decathlon Users Future Operator

Future Operator - Possibilities

Augmented reality

3D visualization

Gestures

Social media

Sensor technologies

Mobile devices

Touch

Depth cameras

Eye-tracking

© ABB Inc

3BSE074371 en.

July 8, 2013 | Slide 33

Future Operator – Visualizing in 3D When it makes sense

Before After

Future Operator - Controlling by Eye-Tracking Gesture Interaction

Future Maintenance Engineer Augmented Reality

A coffee cup tag shows

information related to the

coffee cup

An ABB logo shows the

latest news related to

Decathlon

A factory logo brings up a

3D plant