Alarm Training Making the transition from alh to CSS Alarm Tools

XFELThe EuropeanX-Ray Laser Project X-Ray Free-Electron Laser

Matthias Clausen, DESYXFEL Refrigerator Controls – April 27 2010

Alarm Training

Making the transition from alh to CSS Alarm Tools

EPICS Collaboration Meeting Aix-en-ProvenceJune 1st , 2010

Matthias Clausen - DESY

Matthias Clausen, DESYXFEL Refrigerator Controls – April 27, 2010 2


Alarm Training

• Record Processing• Monitor Types• Setting Alarms• Alarm Handling



EPICS Record Processing

P T

EPICS (Experimental Physics and Industrial Control System)

I/O System(Field bus)

I/O Controller(control loops, Sequence programs)

Client(Configuration- and Operator Interfaces)

Channel AccessIP based Communication Protocol

I/O Controller is independent from clients



EPICS Information exchange

P T

Client initiating basic requstes: get/ set/ monitor

Whenever Displa/ Archive/ Alarm-Deadband/ Status changes -> check whether clients have been registered.If true: -> send updated data

ClientRegister for Changes•Display-Deadband•Archive-Deadband•Alarm-Status Sending request via

Channel Access

Register Client for chanegs in specified record for Alarm changes



EPICS default: focus on ‘requesting’ data-> not ‘expecting’ data

P T

Client initiating basic requstes: get/ set/ monitor

get/

set

register

for

monitors

New/ Changed

Alarm Limits

Alarm/Status

How to stay ‘in synch’ with alarm deadbands/ status in IOC?



Introducing a Message based Protocol: JMS(keeping Channel Access for ‘basic’ monitors)

P T

Basic: get/ set/ monitor

get/

set

register

for

monitors

New/ Changed

Alarm Limits

Alarm/Status

Accepting any alarms from IOC

InterConnectionServer

JMS



JMS: For Alarms and all Log-based Messages

P T


get/

set

register

for

monitors

New/ Changed

Alarm Limits

Alarm/Status


JMS

SysLog-Messages

PutLog-Messages

SNL-Log-Messages



Existing Log Server Implementations

P T


get/

set

register

for

monitors

PutLog-MessagesSysLog-Messages

Sys-LogMessage

Server

Put-LogMessage

Server



JMS does not add more levels of complexity(just new ones …)

P T


get/

set

register

for

monitors

New/ Changed

Alarm Limits

Alarm/Status


JMS

SysLog-Messages

PutLog-Messages

SNL-Log-Messages



How to implement the new Logging Mechanism

• Using the existing Alarm Hook in Record Processing

• Using the existing Hooks in System Logging• Using the existing Hooks in (ca)Put Logging• Using the new Function in SNL Logging



Alarm Logging

Changes in iocCore for Alarm Logging• very small hook in recGbl recGblResetAlarms/* Hook Routines */RECGBL_ALARM_HOOK_ROUTINE recGblAlarmHook = NULL;.......

• loading and starting the alarm logging software redefines recGblAlarmHook....... if(sevr!=nsev || stat!=nsta) { /* HOOK for alarm logging */ if (recGblAlarmHook) (*recGblAlarmHook)(pdbc, stat, sevr);.......



Alarm Logging

Hook Routine

Control task

SCAN context

Message tasks

Port B Port A

The special queue has sequential input ....

and parallel access if space is rare and the record is stored in the overflow area.

Beacon task

add

replace

implenented as ring buffer



asTrapWriteRegisterListener(caPutLog2Array)

caPutLog2Array (...) (caller context)

caPutLog2Q (...) (interrupt context)

caPutLogTask ()

EPICS

copy into Queue

register Hook Routine for caPut operation

write caPut-Message into Array

timeout

Channel-Access-Put-Logging

index queue

send Message to Network



SNL Log

‘Just’ a new supported function in the snl executive.

Code example:

when(TV001zu){

snlMsgLog( SEVR_INFO, "<Start> TV001 has been closed");

} state Auf50

Sent to the log table in CSS.

Different severy types can be set -> used to color-code the messages



EPICS

IOC-Logging Overview

Beacon

Alarms caPut SysLog SNL-Log

AlarmAlarm caPutLog sysMsg snlMsg

Command Reply

ioc log control

ICSICSICS



Alarm Table



Looking back into the Archive (24h)



Selecting the Time Span



Any Message in the System



Log Messages (System, Put, SNL)



Properties View (default Eclipse view)



Alarm System - Alarm Tree view

• Shows the current status of the alarm state from the IOC• Delete and create records and subcomponents by context menu• Configuration-Changes are stored in the LDAP server (or in XML file)• Alarm status is propagated to root component• Property view to display and edit tree items



Demo Alarm Views

• Changing / setting alarm Limits• Severity Settings• Different kind of Displays



How to get there without JMS?

The internal Alarm Model in CSS (DESY) has been modified to accept Alarm-Messages from two different sources:

- JMS- DAL (Channel Access)(currently only available in the

development environmentforeseen for the next release)



How to display an Alarm Tree without LDAP?

• Use the new alternative to configure your LDAP Tree from an XML file– Create XML file by Hand– Create XML file from DCT (next release)– Create XML file from ‘your’ database creation tool– Use LDAP Tree to manually configure your tree and

save your configuration from tree ‘save as’



How to integrate IOC’s without iocLogClient-Support?

A new headless application will handle this:• Reading an XML config file• Use channel access to register for Alarm

Changes• Send alarm changes to JMS



The transition Plan:

• Start using the ‘DAL’ option for Alarm Tables and the Alarm Tree– Get your configuration file e.g. from DCT– Get the hierarchy visualized in the Alarm Tree– Get all configured alarms in the Alarm Table

• Install JMS Servers– Immediately get your CSS log messages into you CSS Alarm views– Use dal2jms to send Alarm State changes to JMS– Install iocLogClient-Support on your IOCs

• Install interConnection-Server(s)• Get ALL your messages (alarm and log) from your IOCs

– Install jms2Ora (rdb) to store your messages



CSS Instance

Acknowledge

Alarm message

Alarm System - Acknowledgement

Ack.

Message

(JMS)

Update

Persistant Store (LDAP)

Persistant Store (LDAP)

JMS Server

Ack

Ack

Ack

Ack

CSS InstanceCSS InstanceCSS InstanceCSS Instance

Ack



CSS Alarm

Configu-

rator

Alarm System - Alarm Management System (AMS)

DBDB

Filter

ManagerFilter

SMS

JMS

Read

configuration

Action

Alarm

Message

(JMS)

Write

Configuration

SMS Connector Voice Mail Connector Mail Connector

JMS

Mail

JMS

Voice

Mail



Alarm Management System

Slides from Markus Möller



AMS - Overview

ActiveMQJMS-Server

DepartmentDecision(FilterManager)

MessageMinder

Distributor

Derby Db

Filter-Konfiguration

VoicemailConnector

EMailConnector

SmsConnector

JmsConnector

ALARM

T

T_AMS_MESSAGEMINDER

T

T_AMS_DISTRIBUTE

T

T_AMS_CON_SMS

T

T_AMS_CON_MAIL

T

T_AMS_CON_JMS

T

T_AMS_CON_VOICEMAIL

T

Derby Db

Filter-Konfiguration

ActiveMQJMS-Server

MessageSources



AMS - Applications

• DepartmentDecision (FilterManager)

Compares the incoming message with the filter configuration

• MessageMinder

Reduces message storms

• Distributor

Creates the alarm messages and sends them to the responsible Connector

• Connector

Sends the message to the user via

SMS

e-Mail

telephone call

JMS message producer to a user defined JMS topic



User – Configuration (GUI)



User – Configuration

• Name

- The name of the user

• Rubrik (rubric, category)

- Name of the configuration rubric

(just for display puposes)

• E-Mail

- E-Mail address of the user

• SMS number

- Number of the user‘s cell phone

• Voicemail number

- Number that is used for the telephone call

• Prefered alarming type

• Status code

- To deactivate a user or a group

• Confirm code

- To confirm a alarm chain

• User is active

- Only active users receive a alarm message



Group – Configuration (GUI)



Group – Configuration

• Name

- The name of the group




• Minimale Anzahl aktiver Mitglieder

- Minimum number of active users in this group

• Wartezeit bis Rückmeldung

- Number of seconds to wait for an alarm chain

answer

• Alarmgruppe aktiv

- Only the member of an active group receive

messages

• List of group members

- Every member of a group can be set active

or not active

- Only active member of a group receive

messages



Condition – Configuration (GUI)



Condition – Configuration

• Name

- The name of the condition




• Description

- Description for this condition

• Filtertype

- Type of this filter

(String, String Array, PV, Time based)

• Compare Key

- Name of the message property that is

compared by this condition

• Operator

- Type of operator that is used to compare the

value

• Compare Value

- The value that is used for comparison



Filter – Configuration (GUI)



Filter – Configuration

• Name

- The name of the condition




• Templates

- Message template (not used yet)

• Default Message

- Alarm message that will be send to the users

• List of receivers

- Name of the receiver

- Type of action (SMS, Mail, Voicemail, JMS)

• Operator

- Type of operator that is used to compare the

value

• Compare Value

- The value that is used for comparison

• Filterconditions

- The configuration / combination of the conditions



Questions?



Active ICS Connection OverviewICS: InterConnection Server (between IOC and JMS)



Disabling Messages (for Testing/ Reboot …)





Alarm / Log message

Sources

Alarm system - Structure

EPICS IOC D3 PCM CSS Instance

JMS ServerPersistent

Store (LDAP)Persistent

Store (LDAP)Archive DBArchive DB

CSS Alarm

Tools

(Views, Con-

figuration, …)

Message

Table

Message

Archive

Alarm Management

System

AMS

ConfigurationAlarm Tree

SM

S

Mai

l

Updated from IC



JMS Communication

Message sources

Special Format

Other Sources

• EPICS IOC and D3 PCM send alarm messages in special format

• Interconnection Server (EPICS) and D3 Alarms (D3) translates alarm messages in JMS format

• CSS uses log4j and sends log messages in JMS format

• Generic message system for alarm messages

• Easy to add other sources

EPICS IOC D3 PCM

CSS Instance

InterconnectionServer

D3 AlarmReader

Special Format

JMS Server(Active MQ)



Alarm System - AMS Filter

Filter:• Checks if the filter matches• Creates a new message with the

relevant information of the alarm message

• Forwards the message to an actionFilter condition:• A Filter is a combination of filter

conditions• Filter conditions can be connected

with AND and OR• Available condition types are:

Compare strings, Check current PV, Time based condition, …



Alarm System - AMS operators and groups

Operators:• Receive alarm messages via mail, sms, …• Status active or inactive can be set• PIN Code to acknowledge alarm messages

Groups:• Operators responsible for specific facilities• Defines priority who should be informed

first, second, …• Maximum delay for acknowledgment of

alarm messages



Logging : 2X Log Server Verbindung

CSS mit direct Channel Access (DAL) connectionAs a transition to JMSDemo DALDemo JMSICS und JMS FolienAMS FolienConfig in CSS Log Table Topics



Application Development

Integrating new I/O devices Configuration of I/O HardwareAnd Process Controls

Programmer Process Engineer

Get driverfrom

EPICS site

Prepare newIOC

Load Image

ConfigureDatabase

DatabaseCreation

Tool

Write text fileto load

directory

Load IOC with-Operating System-EPICS core-Database

Hardware Engineer

ProfibusConfiguration

Tool

ConfigureProfibus

Documents

Alarm Training Making the transition from alh to CSS Alarm Tools