Creating an Alarm Management System with KSX

Mike Hales, Lynn Hales and Dustin Collins

2010-09-13

Contents

1 Introduction 3

1.1 Objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

1.2 Brief Introduction to KSX . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

2 Building the Alarm Management Framework 5

2.1 Connecting to an OPC AE Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

2.2 Processing Events with KSX Rule Scripts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

2.3 Creating Filters and Selectively Responding to Events . . . . . . . . . . . . . . . . . . . . . . 7

2.3.1 Source Filter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

2.3.2 Message Filter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

2.3.3 Severity Filter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

2.3.4 Filter Graph Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

2.4 Logging Events and Statistical Calculations . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

3 Building the Alarm Management User Interface 15

3.1 Trends, Logs and Histograms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

3.2 Queries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

1

3.3 Creating the Custom View . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

4 Summary 22

2

Chapter 1

Introduction

1.1 Objectives

One great challenge that process control and automation engineers face is the frequent and overwhelmingnumber of alarms that are generated by modern regulatory control systems. Today’s DCS and PLC systemshave thousands of tags, and hundreds of control blocks, all of which can generate alarms. Many times thesealarms are received by control room operators who view them as a nuisance at best. Important notificationsare lost in the “noise” of the alarm system. Several alarm management packages have become commerciallyavailable. Most are ugly, complicated, and limited in their utility. KSX implements the OPC Alarms andEvents client protocol and can be used to subscribe to, and process alarm messages. The objective of thiswork is to implement a simple and elegant framework for processing and managing alarms. This frameworkshould be minimalistic, and stay out of the user’s way, yet provide extensibility so that custom responsescan be crafted. The functional requirements of the system are:

• Subscribe to any number of OPC AE sources

• Receive alarm messages

• Filter alarm messages based on source, message, or severity

• Log, count and perform statistical calculations on alarms and alarm sources

• Provide graphical feedback such as trends, histograms, tables and text messages

• In addition the alerts in the graphical user interface, alert the user via email when certain conditionsexist, such as alarms over a severity or frequency threshold, abnormal alarm frequency or suddenchanges in alarm frequency

• Provide daily and monthly reports and automatic identification of the most problematic control blocksto target process control and instrumentation resources

3

1.2 Brief Introduction to KSX

KSX is the worlds most advanced expert control system. It allows for the rapid deployment of expertcontrol strategies using an easy to understand english-based rule syntax and brings to bear the power ofintegrated fuzzy logic, neural-network modeling, genetic algorithm optimizers as well as a plethora of toolsdesigned to make process control application development as fast and as easy as possible. A full descriptionof KSX is beyond the scope of this paper, but more information can be found on the world wide web athttp://www.kscape.com. KSX will be the foundation for the development of the alarm management system.Because the KSX Alarms and Events System is built with all the capabilities of KSX it is certainly the mostcapable Alarm and Event system available.

4

Chapter 2

Building the Alarm ManagementFramework

The construction of the alarm management system proceeds in steps. Each step will add new functionalityto the system and will build on the current functionality. For the sake of brevity, details of the basic use ofKSX will be omitted. It is assumed that the reader is familiar with the use of KSX. Detailed help can befound in the tutorial document KSXBasicTrainingGuide.pdf.

2.1 Connecting to an OPC AE Server

Connecting KSX to an OPC AE Server is straight forward and includes the following steps.

1. Create a new KSX server and save the configuration to disk.

2. Connect to the server and navigate to the Communication pane and select the OPC Alarms and Eventstab.

3. Add an OPC AE server and enter the appropriate server name, and host name.

Once done, KSX will connect to the OPC AE Server and will begin to receive event messages. The screenshot in figure 2.1 below shows the configuration. A message indicating the connection status will be shownin the KSX system transcript.

2.2 Processing Events with KSX Rule Scripts

Once the OPC AE Server is configured, KSX will receive event messages. In order to process them, we mustconfigure the OPC AE Server as an event source for a KSX rule script, so that the rule script will be invoked

5

Figure 2.1: Configure KSX to receive events from an OPC Alarms and Events Server

when new event messages are received. This is very simple. Configure a rule script that will receive four (4)arguments. This is shown below in figure 2.2. The four arguments that are passed when an event messageis received are:

• Source - source of the message, typically a DCS control block or module

• Message - text body of the message

• Severity - an integer value between 0 and 999

• Time Index - an integer representing the time in microseconds since Jan 1, 1900. The time indexrepresents the time of the event, as indicated by the OPC AE Server

Once the rule has been accepted, click on the Schedule tab, click the Add Event button, then drag the AE1OPC AE Server from the tree to the Event Triggers sub pane as shown in figure 2.3 below.

Once done, the rule will be invoked automatically each time that an event message is received from the OPCAE Server.

6

Figure 2.2: Create a rule script that receives four arguments

Figure 2.3: Add an event trigger to the rule script

2.3 Creating Filters and Selectively Responding to Events

Based on our stated objectives, once we are receiving event messages in KSX, we want to be able to filterthem and selectively respond to messages of interest. We will create a simple framework where filters canbe easily configured to respond to messages that match certain criteria based on their source, message andseverity. To do this we create several new nodes in our KSX configuration. We add a Lib node with four childnodes: SourceFilter, MessageFilter, SeverityFilter and EventLogger. These nodes will act as prototypes,and will be copied for use in configuring our filter graph for event processing. We also add an EventFiltersnode that will be the root of our filter graph. See the screen shot in figure 2.4 below.

The filter graph concept is easy to implement with KSX. We simply pass all events to the root of the filtergraph. The root passes all events to all of its children. Each child applies its filter logic. If the even matches,it is passed on to the children, if not nothing happens and the event is effectively filtered or dropped. Topass all events to the EventFilters node, we first implement an event rule script, then call it from the eventrule script implemented above. KnowledgeScape/EventFilters/event should be implemented as:

args source message severity timeIndex;for each child do { args ea ; ea event ( source , message , severity , timeIndex ) }

7

Figure 2.4: Add filter nodes to the configuration

Then we implement KnowledgeScape/event as:

args source message severity timeIndex;EventFilters event ( source , message , severity , timeIndex )

Now each event that is received is passed to the KnowledgeScape/EventFilters/event rule script. We cannow begin to construct our filters and assemble them into a filter graph.

We have created three types of filter nodes, let us examine each one individually.

2.3.1 Source Filter

The source filter will match the event source with a pattern set by the user. If the match is successful theevent is passed on, if not it is filtered. We will take advantage of the matchesRegex() built in functionprovided by KSX to do the pattern matching. This function uses a regular expression engine to do complexpattern matching. Regular expressions are the subject of entire books, but a good starting place for furtherstudy can be found on the world wide web at http://en.wikipedia.org/wiki/Regular_expression.

We add an attribute named sourceExpression, set its type to StringAttribute and give it an initial value of“.*”. This will match anything. Next we implement the event rule script as shown below:

8

args source message severity timeIndex ;if matchesRegex ( source , sourceExpression ) then for each child do { args ea ; ea event (source, message, severity, timeIndex ) }

To use the SourceFilter, just add it as a child of the EventFilters node, or any filter below it, and enter thedesired pattern in the sourceExpression attribute. All events that match the pattern will be passed on.

2.3.2 Message Filter

The message filter will work in the same way as the source filter above, but it will match on the message text,rather than the event source. This will allow us to capture the same alarms but from different controllers.

We add an attribute named messageExpression, set its type to StringAttribute and give it an initial valueof “.*”. This will match anything. Next we implement the event rule script as shown below:

args source message severity timeIndex ;if matchesRegex ( message , messageExpression ) then for each child do { args ea ; ea event( source, message, severity, timeIndex ) }

To use the MessageFilter, just add it as a child of the EventFilters node, or any filter below it, and enterthe desired pattern in the messageExpression attribute. All events that match the pattern will be passed on.

2.3.3 Severity Filter

The severity filter will allow us to filter events based on the numerical severity value. Rather than matchpatterns in a string as above, we will match using simple greater than or less than comparison.

We add two attributes named high and low, set their types to IntegerAttribute and give them initial valuesof 1000 and 0 respectively. This will match anything. Next we implement the event rule script as shownbelow:

args source message severity timeIndex ;if severity >= low and severity <= high then for each child do { args ea ; ea event ( source,message, severity, timeIndex ) }

To use the SeverityFilter, just add it as a child of the EventFilters node, or any filter below it, and enterthe desired high and low severity values. Any message with a severity in the range will be passed on.

2.3.4 Filter Graph Example

Let us consider an example case, where we are interested in any events from two controllers, Controller1and Controller2. From these controllers, we are interested in events concerning sump levels, but only if the

9

severity is over 200. It is very easy to configure a filter graph to select only those events that interest us. Wewill use each of the three filter types we configured above. Add a SourceFilter node to the EventFilters node.Then add a MessageFilter to it, and finally add a SeverityFilter to the end of the chain. This configurationis shown below in figure 2.5.

Figure 2.5: A filter graph for selectively responding to events

The regular expression for matching the text “Controller1” or “Controller2” is as follows:

(Controller1)|(Controller2)

We enter that as the value for the sourceExpression attribute in our SourceFilter node.

The regular expression for matching the text “Sump” or “sump” in any part of a larger string is slightly morecomplicated. It is is as follows:

.*[sS]ump.*

We enter that as the value for the messageExpression attribute in our MessageFilter node.

Finally, we enter 200 as the value of the low attribute of our SeverityFilter node, leaving the high value setat 1000.

Our filter graph is now totally configured, and will pass on only the events that specifically interest us. Weneed one more node however to process the messages that are received. For our example we will just writeout the message to the system transcript.

10

Add a new node to the SeverityFilter node at the bottom of our filter graph. Implement a rule script namedevent as follows:

args source message severity timeIndex ;out (message)

This will write the message text from any event satisfying our conditions to the system transcript.

Now that we have created a mechanism for easily configuring filter graphs to selective respond to eventswe will proceed to our next system objective, logging and performing statistical calculations on the eventsreceived.

2.4 Logging Events and Statistical Calculations

The next functional component of the alarms and events manager should log, count and perform statisticalcalculations on alarms and alarm sources. In order to intelligently indentify abnormal events, we need tofirst establish a baseline to know what is normal. The important tasks of the logging component are:

• Record events in a historical database, making them available for search and statistical analysis

• Keep running statistics on event sources such as count, frequency, upper and lower control bands (2sigma)

• Display events to the user

The next step is to implement the functionality described above in the EventLogger node that we previouslycreated. We add several attributes that will store the event data (source, message, severity and timeIndex)as well as attributes to contain the running statistics. The screen shot in figure 2.6 shows the fully configuredattributes.

Once the attributes have been configured, we will build the functionality required using KSX rule scripts. Allof the EventLogger rule scripts are shown in figure 2.7 below. As established in the pattern demonstratedabove, the node will have an event rule script that will receive the events. It is implemented as follows:

args source message severity timeIndex ;transcript ( source , message , severity , timeIndex ) ;record ( source , message , severity , timeIndex ) ;notify ( )

This rule script calls the appropriate rule scripts in the EventLogger node to do the work described above.Specifically, the transcript rule script displays the event information to the user in the system transcript,and is implemented as follows:

args source message severity timeIndex ;out ( cat ( ’OPC Event Received - ’ , timestamp ( timeIndex ) , ’

11

Figure 2.6: Configured attributes of the EventLogger node

Source - ’ , source , ’Message - ’ , message , ’Severity - ’ , severity ) )

The record rule script records the event details as attribute values, and calculates the running statisticsnecessary for identifying abnormal alarm activity and acting on it. The implementation for record is asfollows:

args source message severity timeIndex ;simulate ( eventSource , source , timeIndex ) ;simulate ( eventMessage , message , timeIndex ) ;simulate ( eventSeverity , severity , timeIndex ) ;simulate ( eventTimeIndex , timeIndex , timeIndex ) ;eventCount = eventCount + 1 ;eventFrequency = rateOfChange per hour of eventCount over 1 hour ;avgEventFrequency = avg of eventFrequency over 1 hour ;stdEventFrequency = stdev of eventFrequency over 1 hour ;hiBand = avgEventFrequency + 2 * stdEventFrequency ;loBand = avgEventFrequency - 2 * stdEventFrequency ;

We use the simulate() built in function to record the attribute values. simulate() is like regular assignmentusing the “=” sign, except that it specifies the time index for recording in the historical database. This isimportant, because we want the events to be recorded at the time provided by the OPC AE Server, notthe wall time when they were recorded in KSX. This will allow us to reconcile events in KSX with events

12

Figure 2.7: Configured rule scripts of the EventLogger node

recorded in other tools like DCS system historians or Pi.

Finally, the notify rule script evaluates the recent activity of the event source, and raises notification ifabnormal activity is identified. It is implemented as follows:

temps subject body ;if eventSeverity > getSeverityLimit ( ) oreventFrequency > getFrequencyLimit ( ) oreventFrequency > hiBand thensubject = printFlagReason ( ) andbody = printSummary ( ) andemail ( ks recipients ( ) , subject , body ) andlog ( cat ( subject , ’’ , body ) , ’Notifications’ )

This rule script checks for severity or event frequency greater than a defined limit. It also checks if theevent frequency is statistically abnormal, meaning that it is greater than two standard deviations above theaverage for a specified period. If any of these conditions are true, then an email is sent and a message islogged to a permanent log. Note that several helper rule scripts are used in the rule script above to simplifythe rule and add readability. They are self describing and can be examined by refering to the referenceimplementation, provided with this paper.

The EventLogger node can be attached at any point on the filter graph described above to log specific eventsof interest. A more general use will be to use it as a prototype to be dynamically copied when events from

13

new sources are received. The initial event and all subsequent events will be logged. This is one of the morepowerful features of KSX. It allows the configuration to change at run time and adapt to conditions thathave never been seen before. It is very easy to dynamically create new nodes. First we add a new nodenamed EventSources to the KnowledgeScape root. We implement the event rule script as per our pattern,and accept it with the following source:

args source message severity timeIndex ;temps sink ;sink = find ( { args ea ; ea eventSource = source } , children ( ) ) ;if sink = nil then sink = create ( #EventLogger ) ;sink event ( source , message , severity , timeIndex )

We must modify KnowledgeScape/event to call our new rule script when new events are received. Wesimply add the following line to the end of the script:

EventSources event ( source , message , severity , timeIndex )

Now when an event is received by the system, the EventSources node searches for an existing node with amatching eventSource. If it is found the event is passed to it for logging and notification. If a matching nodeis not found, then a new EventLogger node is created, and the event is passed to it. To make EventLoggeravailable as a prototype, we must use KSX’ template functionality. This simply means that we must changethe node’s template from Base to EventLogger. This will enable it to be automatically discovered by thesystem and copied using the create() built in function. This is shown below in figure 2.8.

Figure 2.8: Set the template to EventLogger

At this point the base foundation of our alarm management system is complete. We can selectively filterand respond to events and we have a framework to log all the events that occur, displaying them to the userin the transcript and emailing when abnormal conditions arise. The next step is to configure the graphicalaspects of our system so that the user may visualize, query and interact with the system.

14

Chapter 3

Building the Alarm Management UserInterface

3.1 Trends, Logs and Histograms

Our alarm management system should allow the user to dynamically interact with it. Specifically, we willimplement user interfaces to visualize the system (trends, logs, histograms) and to query the sytem.

First we configure a trend of the event frequency of a given source. As shown below in figure 3.1 we trendthe current event frequency, the average event frequency and the high and low bands, as calculated in therecord rule script shown above. You may recognize this as a variation of the Shewhart control chart used instatistical process control.

This gives a trend for the event frequency for a specific event source. To trend total system events, we addthe same rule scripts and attributes to the EventSources node, and configure the same trend there. We alsoadd a log named Notifications to the EventSources node. This log will act as an aggregator for all of thenotification messages generated by the event logger nodes dynamically created as events when new sourcesare received.

In addition to visualizing event frequency in real time, it is desireable to see the distribution of eventfrequency of all the event sources. We will use the KSX histogram functionality to view this frequency, toquickly identify outlying sources with very high event frequency. The same thing will be done for eventseverity. To prepare the data for the histograms we write a pair of rule scripts to aggregate the datafrom all the sources. On the EventSources node we implement rule scripts named frequencyValues andseverityValues with the following source respectively:

map ( { args ea ; ea avgEventFrequency } , children ( ) )

map ( { args ea ; ea avgEventSeverity } , children ( ) )

15

Figure 3.1: A trend of event frequency

These rules return a collection of the average event frequency and average event severity from all the eventsources. The data can then be visualized in histogram format. We add two histograms named frequencyHistand severityHist and add the rule scripts configured above to each respectively. The frequencyHist histogramis shown below in figure 3.2. A similar rule and histogram is constructed for the EventLogger node, so thateach event source can display a histogram of its event severity values individually.

3.2 Queries

The final element of the alarms and events system is report generation. The system should automaticallygenerate daily reports identifying event sources with the highest event frequency or severity so that processcontrol and instrumentation resources can be efficiently allocated for solving problems. The user must beable to generate these reports interactively also.

We will implement four report types and will schedule them to run daily, logging their results, and emailinga composite summary. The report types are:

• Highest Frequency Sources

• Highest Severity Souces

• Highest Count Sources

• Recently Added Sources

16

Figure 3.2: The configured frequencyHist histogram

The rule scripts that execute these reports must iterate over all the event sources, collect and sort them bythe desired parameters and create a report identifying the source and the parameter value. We first createattributes to hold the report parameter values, then create the rule scripts to generate the reports from thoseparameters. The update attributes for the EventSources node is shown below in figure 3.3. An example ofsuch a script to create a report of the event sources with the highest event frequency is shown below:

temps sources report ;report = ” ;sources = filter ( { args ea ; ea avgFrequency < highFrequency } , children ( ) ) ;sources = sort ( { args s1 s2 ; s1 > s2 } , sources ) ;forEach ( {args idx ;temps source ;source = nth ( sources , idx ) ;report = cat ( name ( source ) , ’ - ’ , source eventSource , ’ - Frequency: ’ , source avgFrequency, ’’ ) } , sequence ( 1 , min ( size ( sources ) , resultLimit ) ) ) ;report

Similar rule scripts are created for event severity, event count and most recent events. They can then beaggregated together for a daily report, which is logged and emailed. The following script does this.

temps report ;report = cat ( ’Daily Alarms and Events Report ’ , timestamp ( ) , ’’ , ’High Frequency Sources

17

Figure 3.3: Attributes hold the parameters for the dynamic queries used to generate reports

+––––––––––––––––––––––––––––––––––––––––––––––+’ , frequencyHigh ( ) , ’

High Severity Sources+––––––––––––––––––––––––––––––––––––––––––––––+’ , severityHigh ( ) , ’

High Count Sources+––––––––––––––––––––––––––––––––––––––––––––––+’ , countHigh ( ) , ’

Newly Added Sources+––––––––––––––––––––––––––––––––––––––––––––––+’ , newInLastXDays ( ) ) ;log ( report , ’DailyReports’ ) ;email ( ’report.recipient@yourcompany.com’ , ’Daily Alarms Report’ , report )

Finally, we want to be able to interactively run the reports and adjust the report parameters. To do this weadd four more rule scripts to log the report, and then set the appropriate query parameter attribute as anevent trigger. This way, when the user changes the query parameter, the script is automatically run, andthe results logged.

18

3.3 Creating the Custom View

Now that the events framework is complete and we have created trends, histograms and logs, and caninteractively perform dynamic queries, the last step is to arrange the user interface using KSX’ customview feature. We will create a top level user interface with tabs to display the real time components, thepre-configured histograms and the interactive report results.

We first add two tabs to the EventSources custom view. We will name the first tab “Trends”, the secondtab “Reports”. We split the first tab horizontally, and drag the frequencyTrend to the left side, and theNotifications log to the right. We split the first tab horizontally, then split each side vertically. We drag thefrequencyHist to the top left, the severityHist to the bottom left, the queryParameters value table to thetop right, and the QueryResults log to the bottom right. We resize the splitters for the best results thentear off the custom view, leaving us with an interactive user interface for our alarms and events managerapplication. Several screen shots show the configuration, and the finished application below.

Figure 3.4: Configuring the custom view

19

Figure 3.5: The trends tab of the interactive user interface

20

Figure 3.6: The reports tab of the interactive user interface

21

Chapter 4

Summary

We were able to easily create a custom alarms and events manager application with KSX. All of our systemrequirements were met by using the built in features and functionality of KSX. Now we will be able to usethe alarms and events of our automation system to identify abnormal conditions and warn the appropriatepersonnel to take action. We can easily identify troublesome controllers and event sources and target ourprocess control and instrumentation resources effectively to implement a system of continuous improvement.This system is simple and elegant, and is infinitely extendable to meet any specific need that may arise. It istrivial to integrate the system with the existing automation or expert control system to take immediate actionupon the process if necessary when certain alarm conditions occur. KSX provides all of the functionality ofother commercial alarm managers, yet can be extended with any conceivable logic and made to perform anyother control task, something that no other alarm management system can do.

22