SCADA System Call Center Training Manual-Revised Sept 2013

7/26/2019 SCADA System Call Center Training Manual-Revised Sept 2013

1/112

Aimee Pulido, C.E.T

Fall 2013


2/112

Table Of Contents

Description

Functions of the SCADA system ..7

Hardware Components .8

Software Components ...13

Automatic Failover of SCADA Servers ..19

Failover of Historian Servers ..21

Starting iFIX

Starting iFIX on the Primary Dispatch Computer ..22

Starting iFIX on the Backup Dispatch Computer ..24Navigating the SCADA System

Initial Startup ...28

Normal Error Messages And Notifications Encountered 29

2


3/112

Table Of Contents

Navigating the SCADA System (continued)

Lower Menu Bar ....32

- Dispatch View Screen 32

- Main Menu ..33

- Help Menu 34- City Overview .34

- Report Menu ..35

- Early Warning System For High Flows At Gold Bar ..36

- Current Alarm Summary .37

- Alarm / Event History 39

- Notepad .40

- Tag List .41

- Dropdown Lists ....42

- Statuses .42

3


4/112

Table Of Contents

Navigating the SCADA System (continued)

City Overview Screen ..43

Station Pictures ..47

- Communication Screen .47

- Drains View Screen ..48- Annunciator Screen .49

- Trend Screen .51

- Site Info Screen 53

- Control Settings Screen .54

- Flow Diagram Screen ..55

- System Layout Screen .56

- Sectional View Screen 58

- Tank Summary Screen 59

- Schematic Screen ..60

4


5/112

Table Of Contents

Functions of the Dispatcher

24 hours/day, 7 days/week ....61

Task 3: Monitoring the SCADA Systems Health 63

Task 4: Monitor the SCADA System, Acknowledge and Dispatch

Alarms To the Appropriate Staff For Resolution ..66- A) Gather The Facts About The Alarm ...69

- B) Respond To The Alarm .78

Task 5: Document Steps Taken For Resolution of Alarms .92

Task 6: Verify Correct Functioning of the CommunicationsNetwork ..98

Task 7: Communicate With Pumpwells Field Staff And ContractorsDaily and Document Who Is At Site, Performing What Tasks, SoFalse Alarms Can Be Accounted For .113

5


6/112

Table Of Contents

Functions of the Dispatcher (continued)

Task 8: Document problems with the SCADA SystemFunctionality....117

Task 9: Communicating Problems With The Functionality Of The

Call Centers SCADA Computer Hardware or Software .120

6


7/112

Description

Functions of the SCADA system:

Collects and displays and monitors over 5000 real-time alarms, events,statuses and readings, and displays graphically animated and non-animated data used to verify the operation of >130 mechanical and

electrical drainage facilities, including pump stations, storm waterpond automatic gates, level sensors, odour filters, etc.

- Shows how facilities should work in accordance with the design

specification

- Shows how facilities are actually working (and if not properly working,

shows what is wrong)- Helps with the maintenance and troubleshooting of facilities

7


8/112

Description

Hardware components:

SCADA servers:

- Runs the SCADA application (iFIX software)

- Master KENSCA01: Located in Century Place

- Backup KENSCA02: Located in Century Place Historian servers:

- Runs Proficy Historian software

- Master COEACLA1: Located in Century Place

- Backup COEACLB1: Located in Century Place

- Servers have a shared clustered drive known as CEPACL171 for the

purposes of data management

8


9/112

Description


SQL server:

- Runs Alarm and Events SQL database for Proficy Historian software

- CEPASQL81: Located in Century Place

- Server is shared with other SQL databases from other COE departments Terminal Services server:

- Runs iFix Terminal Services software

- CEPACTX02E: Located in Century Place

- iFIX on this server is accessed through a Citrix Session

Real-Time Information Portal server:

- Runs Proficy Real-Time Information Portal software

- CEPACTX02E: Located in Century Place

- Accessed through a webpage at http://proficy/proficyportal/ orhttp://rwa.edmonton.ca/proficy

9


10/112

Description


T1 Switch:- Connection to the Rogers Communications GPRS cellular network

Developers workstations:

- Assembly and programming of the graphics, database, scripts, etc.- Troubleshooting of the system- In Aimee Pulidos and Tony Liaos office

View Client workstations:

- Various personal computers used to view the SCADA System

- Located in Kennedale, Coronation Yard, Century Place, EmergencyOperations Centre, Davies Yard

10


11/112

Description


Programmable logic controllers (PLC):

- Located at remote sites (mechanical and electrical drainage facilities,

including pump stations, storm water pond automatic gates, level sensors,

odour filters, etc.)- Controls Processes at sites based on input coming from instrumentation

monitoring site conditions

- Relays all important alarms, events, statuses and readings from the site to

the SCADA system

General Packet Radio Service (GPRS) radio modems:- Connected to PLCs at remote sites and to a wireless antenna

- Transmits data received from the PLCs through the Rogers GPRS wireless

network to the City of Edmonton networks T1 switch, through the firewall

and to the SCADA servers

11


12/11212


13/112

DescriptionSoftware components:

iFIX:

- Human Machine Interface software for the Supervisory Control And DataAcquisition (SCADA) system

- Allows development and presentation of and monitoring tools and sitegraphics for alarms and processes at each station in the WastewaterCollection system.

- Contains the following sub-applications:

Proficy iFIX Workspace:

- Presents interactive animated graphics containing real-time station data fromprocesses and alarms, and allows for monitoring and dispatch of alarm condition

- Contains development tools for creation of graphics, animations, etc.

Process Database:

- Records current values of alarms, statuses and readings from sites

13


14/112

DescriptionSoftware components:

iFIX:

- Contains the following sub-applications:

Visual Basic Editor:

- Allows for programming computer code to automate functions and provide

additional functionality to the system, not provided for by the out-of-the-box iFIXsoftware

Fix Background Server:

- Runs on the primary and backup SCADA servers to monitor door alarms,communications alarms, transfer of PLC communications to alternate server onserver failure, database reading and writing tasks, alarm response functionality,report generation, and rain gauge calculations.

- Runs on one of the development nodes to automate the Early Warning System formonitoring and preparing for high flows at Goldbar Wastewater Treatment Plant,sends the information to Goldbars control center

14


15/112

DescriptionSoftware components: iFIX:

- Contains the following sub-applications

OPC Powertool:

- Programs addresses of tags in the Process Database corresponding to addresses in

the Programmable Logic Controller in each station for data transfer of real-timedata process and alarm values into iFIX

IGS Driver:

- Translates communications from Pump test tanks PLC into iFIX directly ratherthan through a modem

SCADA Sync Monitor:

- Runs on the primary and backup SCADA Servers to handle failovers of servers,synchronization of databases, pictures and alarm areas between the two servers

15


16/112

DescriptionSoftware components: IO Server:

- Runs on primary and backup SCADA Servers to translate communications protocolsover the Rogers network from the PLCs into iFIX

Proficy Historian:

- Runs on dedicated primary and backup Historian Servers in a virtual clusteredenvironment to archive and retrieve:

Real-time data generated from stations equipment and processes andhandled in iFIXs Process database (excluding alarm statuses from sites)

Alarms and Events generated in iFIX based on values from stations, issues withthe SCADA Servers, Historian servers, or security issues

16


17/112

Description

Software components: Proficy Historian:

- Contains several sub-applications for its processes:

iFIX OPC AEServer / Collector:

- Handles the transfer of alarms and events from iFIX into Proficy Historian, which

are then routed through to a SQL Server database (on a server shared with otherCOE departments SQL databases) for archival

iFIX OPC Server / Collector:

- Handles the transfer of station Real-time data values from iFIX into ProficyHistorian, which are archived in Historians proprietary formatted archive files.

Calculation Collector:

- Performs calculations on custom-made tags based on real-time values in ProficyHistorian for a given tag

17


18/112

Description

Software components: iFix Terminal Services:

- runs on a dedicated server to provide client iFIX sessions to field crews andremote users

- Performs the same functions as a standalone (desktop) version of iFIXwithout needing to install iFIX on that machine

- Is accessible through a secure Citrix Session (running through InternetExplorer), in which permissions for iFIX have been granted by InformationTechnology on the users profile

- Requires that Citrix Client software is installed on the computer

Real-Time Information Portal:- Runs on a web server to provide access to SCADA data and data

presentation tools for remote users and external contractors

18


19/112

Description

Automatic Failover of SCADA Servers:

All remote site PLCs are programmed to transmit data to KENSCA01

(the master SCADA server) at IP Address 10.191.1.51, which is

translated by the network switch and routing equipment within the

City as 162.106.222.107 The SCADA servers monitor each others status

If iFix stops running on the KENSCA01 server, KENSCA02 (the standby

server) detects this, takeover as master SCADA server, activate an

alarm, and transmit a command to all remote sites to reset their target

IP address to 10.191.1.52, the current server, which is translated by

the network switch and routing equipment within the City as

165.106.222.210

19


20/112

Description

Automatic Failover of SCADA Servers:

When iFix is reinstated on the KENSCA01 server, it will transmit a

command to all remote sites to reset their target IP address to

10.191.1.51

If iFix stops running on the KENSCA02 (the backup server), the masterserver will detect this and activate an alarm

20


21/112

Description

Failover of Historian Servers: All SCADA nodes are programmed to access archived data and

archived Alarms and Events via the CEPACL171 cluster for historian

Both COEACLA1 and COEACLB1 historian servers maintain acontinuous representation of the data in the cluster so if eitherhistorian fails, the data cluster will be unchanged.

If both COEACLA1 and COEACLB1 servers fail, archiving will cease anddata will not be available to the SCADA nodes.

Upon reinstatement of one of the historian servers:

- the collectors located on the current master SCADA Server (KENSCA01 orKENSCA02) will automatically forward the missing data to the historian

- the historical archives will be updated

- access to their data and to Alarm and Events data by the SCADA nodes will bereinstated

21


22/112

Starting iFIX

Starting iFIX on the Primary Dispatch SCADA Computer: The primary Dispatch computer is to be logged on and functional 24

hours/day, 7 days/week using a generic logon, which doesntautomatically log off after a given timeout period and does not havescreen saver turned on

Log onto the primary dispatch SCADA computer with the followingcredentials:

- User name: isifix

- Password: Pumpwe!!01

- Domain: COE

22


23/112

Starting iFIX

Starting iFIX on the Primary Dispatch Computer: The iFIX software should commence automatically

- If iFIX does not commence within about a minute, on the desktop, double-click the iFIX icon

- In the event that a dialog box pops up,

looking like the one on the right, ensure

that the checkbox at the bottom is

checked,

then click the iFIX icon at the top

left in the box

23


24/112

Starting iFIX

Starting iFIX on the Backup Dispatch SCADA Computer: The Backup Dispatch SCADA computer can be either a regular

workstation used by the dispatch staff or the backup dispatch SCADAlaptop

It is only used when the primary SCADA dispatch computer:

- is having technical difficulty

- is in the process of being sent away for hardware related issues withAcrodex

- is in the process of being replaced by another computer

It performs the same tasks as the primary dispatch SCADA computer,however, instead of the iFIX software being installed on the machinedirectly, access is to iFIX terminal services on the iFIX terminal serverthrough the Citrix web client.

The Citrix Web client is accessible from any computer in the worldthrough an internet browser.

24


25/112

Starting iFIX

Starting iFIX on the Backup SCADA Computer:Method 1: If the City network is having no problems:

On a regular dispatch workstation, login to Windows with the dispatchemployees credentials (user name and password, Domain: COE)

On the desktop, double-click the Citrix VM External

The Citrix Web Browser will open and navigate to :

http://citrix.edmonton.ca

This is a web address within the City of Edmontons network

25
http://citrix/http://citrix/http://citrix/http://citrix/http://citrix/


26/112

Starting iFIX

Starting iFIX on the Backup Dispatch Computer:Method 2: If the City network having problems:

On the backup SCADA laptop (Serial number:R99MD36 currentlystored in Jan Cebulas office in the far left side bottom drawer), loginto Windows with the dispatch employees credentials (user name andpassword, Domain: COE

Plug the aircard (either Rogers or Telus, pick one) into the laptop

Note: There is one is for each network provider so if one providers network is

down then you can access using the other network.

26


27/112

Starting iFIX


IF USING ROGERS AIRCARD

- The Mobilink Lite software should open automatically.

- If it does not double-click on the Mobilink Lite icon on the desktop or press the windows key

On the taskbar at the bottom right side of the screen, right-click on themobilink icon

Select Open Mobilink Lite

- Click the Connect button and wait until the aircard is connected to the

network

27


28/112

Starting iFIX


IF USING TELUS AIRCARD

- The Connection Manager software should open automatically.

- If it does not double-click on the Connection Manager icon on the desktop- Click the Connect button and wait until the aircard is connected to the

network

- NO MATTER WHICH AIRCARD IS USED:

- Double-click the Citrix VM External icon

- Or, in an Internet browser window, enter the address:

https://citrix.edmonton.ca/

28
https://citrix.edmonton.ca/https://citrix.edmonton.ca/


29/112

Starting iFIX

Starting iFIX on the Backup Dispatch Computer:Methods 1 and 2:

When prompted, enter the login credentials and click Log On

- User name: isifix

- Password: Pumpwe!!01

29


30/112

Starting iFIX

Starting iFIX on the Backup Dispatch Computer: The Citrix Web Interface window will open:

- Double-click the iFIX5_0 Operator icon

- Logon scripts will run, they

map the directories so that

the application will function

correctly

30


31/112

Navigating the SCADA system

Initial Startup: Once the iFIX software is started, the City Overview screen will be

displayed

32


32/112


Normal Error Messages and Notifications Encountered: Occasionally an error message is

displayed similar to the one at the

right.

This tells the user that the networkhas not finished intializing its

connections and the tags are unable to get their data from the processdatabase.

There is no need to be concerned with these errors, as they will be

resolved as the system stablizes. Click the Skip All button

If any other error message is displayed, this is not normal AimeePulido should be notified during regular working hours so the problemcan be rectified. This can be done by documenting the error on the

SCADA Concerns form, as per SCADA Dispatcher Functions Task 8.32


33/112


Lower Menu Bar:

Buttons are present for:

- Dispatch View Screen

The screen where the

dispatch staff will spend the

majority of their time

Acknowledge, and document

alarms and their resolutions Place and view alarms on

hold

33


34/112


Lower Menu Bar:


- Main Menu

Contains:

- several quick-navigation

buttons to various station

pictures

- Buttons for administrativetasks (these are only

functional with

administrative and

supervisor security

clearances, so they will

not function for thedispatchers) 34


35/112


Lower Menu Bar:


- Help Menu

Not yet functional

Will be used to provide help information for users with respect to the SCADAsystem functionality

- City Overview

Navigates back to the City Overview (main map) screen

35


36/112


Lower Menu Bar:


- Current Alarm Summary

A list of current alarms in the SCADA

system, which is able to be printed

Used by pumpwells staff and

supervisors to check on current alarm

statuses Clicking the View # Held Alarmsbutton opens Alarms On Hold screen:

- Clicking on an alarm row in this screen generates message, clicking Yes to this

message re-enables alarming for the tag in the selected row36


37/112


Lower Menu Bar:


- Current Alarm Summary

Right-clicking on the alarm screen

displays a menu, which enables the

dispatcher to:

- Enable/Disable Alarm Horn - turns the

audible alarm signal on/off on thecurrent SCADA node for the entire time that the node is being

used until it is manually re-enabled/disabled

Note: menu changes depending on which option is currently

selected prior to right-clicking)

- Silence Alarm Horn silences the alarm horn for the current

alarm instance only IMPORTANT : Dont select Pause Alarm Read - alarming will cease in SCADA 37


38/112


Lower Menu Bar:


- Alarm / Event History

Displays a list of alarms and events

in the past

Queries Historian Alarm and Events

SQL database based on:

- Start Date - End Date- Station # - Type (Alarm or

Event)

Clicking Print will print the list

Dragging column headers left/right

adjusts the width of the column

for easy reading of field values 38


39/112


Lower Menu Bar:


- Notepad

Used by Pumpwells staff and

supervisors to modify and save

operator message (a message that

pertains to the SCADA system in

general, rather than a particularstation)

Provides additional information

with respect to conditions in the

SCADA system, or in the field and

response requirements39


40/112


Lower Menu Bar:

Dropdown lists are present for navigating to station pictures based on:

- Station #

- Station Name (not organized alphabetically)

Statuses (in bright blue) are read from the Process Database anddisplayed for:

- PLC Communication Path (The current server where the stationscommunications are directed to)

- KENSCA01 status

- KENSCA02 status

- Current Date and Time

40


41/112


City Overview Screen: The City Overview screen contains:

- A right menu panel with:

Buttons that, when clicked, show (black button face text)

or hide (grey button face text) different layers on the map

to enhance its visual functionality

A Legend corresponding to the animated station icons on

the map based on:

- Station type (icon shape)

- A stations influent type (color of icon)

41


42/112



- Animated blinking red indicator for unacknowledged alarms in the DispatchView System, consisting of:

Outline around the

map

Lower Navigation Area

Background

Note to the Operator

below the navigation

buttons

42


43/112



- An animated map that displays stations, their current statuses (monitoredby the pumpwells department) and their geographic locations within theCity of Edmonton boundaries

When a station is clicked, navigates to station or system pictures A stations status will either be:

- Normal (no yellow or blinking oval

surrounding the station number)

- In alarm (a yellow or blinking oval

surrounding the station number)

43


44/112


City Overview Screen: The City Overview Screen contains:

- An animated map that highlights (displays a box outlining) the 4 quadrantsof the city when the mouse hovers over each individual quadrant

When a quadrant is double-clicked, it will zoom in for easier clicking

on stations When a quadrant is double-

clicked a second time, it will

return to its normal size

44


45/112


Station Pictures: Station Pictures contain:

- An alarm banner (at top of picture)

outlining all current alarms within

the station

- A navigation bar with buttons for:

Communication

- A screen that allows for changing the station refresh rate (The maximum time to pass between communications from the

stations PLC to the SCADA servers. This is used to generate an automatic send of data

from the site to the SCADA Servers in the event that no process data is sent from the

site to the server prior to the expiry of this time. Therefresh rates at each station are different and depend on

the stations criticality.)

Server that the PLC is supposed to communicate to

- Allows for request of refresh of site data transmissions

- Easy viewing of heartbeats from the site to both the

primary and backup servers

45


46/112




Drains View

- A screen that displays the site in a screenshot from the DRAINS application

- Displays flow directions and paths for the Stations incoming and outgoing flow- Has zoom in/out capability for easy identification of site details and DRAINS IDs

- Screen is not yet constructed for many stations

46


47/112




Annunciator Screen

- Accessed by button with station address

- Allows for viewing of all tags in the Process database and their current values,which correspond to this station

- Tags are displayed by their type (as

selected by the left menu)

47


48/112




Annunciator Screen

- Double-clicking on the individual tag

opens its tag profile (configuredsettings for the selected

tag), which is helpful to

electricians and

instrumentation staff for

troubleshooting

48


49/112




Trend

- Displays a graph of one of more

pens (colored lines), associatedwith data for each selected tag

from Proficy Historian

- Allows for easy viewing

of potential site problems and

troubleshooting

- Displays data in text format belowthe graph for each tag, based on:

- Cursor Position (the thick vertical grey line on the trend), which can be dragged left or

right, to get reading values for any place on the trend

- Maximum reading on the screen

- Average Value of readings on the screen

- Minimum reading on the screen

49


50/112




Trend

- Double-clicking on the trend opens chart

configuration dialog box- Allows for temporarily changing the chart

(reverts to original state when restarting iFIX) General tab:

- Change the visual appearance of the chart

Chart tab:- Data Sources section: add/remove pens (tags)

- Pens tab: change pen high/low limits, colors, linestyles and widths

- Time tab: change chart start date or start time

to a fixed date, or a number of days before now, change the duration of the chart (from

beginning to end of displayed data) based on how many days to view at a time, change the data

interval (how often the data is sampled to create the trend)

- X-Axis and Y-axis tabs: change the color, or number of labels or ticks to display on the

horizontal axis (X-axis) or vertical axis (Y-axis), add titles to axis

- Grid tab: change # vertical or horizontal lines, line color/style for the chart grid

- Legend tab: Change the pens (tags) text data values to be displayed below the chart50


51/112



- A navigation bar with buttons for: Site Info

- Allows for changing of operator

message viewed for a given station

(station selected in the dropdown listat the top center of the screen) , which

then is displayed in the Dispatch View

System screen in the Temporary

Notes box

- Clicking Site Description button Toggles Site Description button to a Site Notes button (takes you back to the Site Info

screen Changes text displayed in window to the description of the site, including :

- Address, and how to locate the station- General functionality of the station- Pump, rated discharge, power supply, active volume, storage volume, emergency overflow

information, and other important-to-know information pertinent to the station

The information entered in this window is displayed in the Site Info tab in the DispatchView System

51


52/112




Control Settings

- Displays the control sequences for the

station, with values coming from thestation (in bright blue) or hardcoded

values (in black) indicating its pump

equipment and alarm setpoints, as

well as any other pertinent control

information for the site

- Values are color coded to correspondto the color of the animations on the

main station screen

52


53/112




Flow Diagram (optional)

- Displays the layout and flow directions of chambers and/or pipe-work within the

station to enhanceunderstanding of flow

patterns

- Individual pipes or

chambers are different

colors for easy

identification

53


54/112


55/112




System Layout (optional)

- This picture contains:

Flow basin information (Layout of Pipes within the system and their corresponding flowdirections)

Clickable station icons for all stations within the system (Clicking on the station icons

opens the corresponding

station picture )

55


56/112




Sectional View (optional)

- A sidelong view of a station with multiple flow paths

- Outlines depths and instrumentation readings for control philosophy at thevarious points throughout

the flow branches and site

components

- Provides a simplified view

of the site to assist in

better understanding of thesite layout and functionality

56


57/112




Tank Summary (optional)

- A report screen for

stations that requirereporting on

hydrocarbon level

control within

particular stations

based on:

Gradual increases Rapid increases

Emptying of tank

Start and stop of

oil collection

57


58/112




Schematic (optional)

- Electrical and water line schematic for the site to assist in understanding

operation of the sitewhere complex

processes are

taking place

- Used by pumpwell

electricians,

instrumentationstaff, and millwrights

to troubleshoot the

site.

58


59/112


24 hours /day, 7 days/week: Task 1: Ensure that the dispatch monitoring computer is logged in and

iFIX is running at all times (see slides 22-30 for methodology)

Task 2: If audible signal is turned on (i.e. on evening shift, etc.), ensurethat volume is turned on and loud enough for audible alarm signals to

be heard by dispatch staff (also see slide 37)

Task 3: Monitor the SCADA systems health and Communicating withPumpwells Departments Electrical Engineer (Tony Liao) or ProcessControl Specialist (Aimee Pulido) when server or software issues arepresent (see slides 61-64 for methodology)

Task 4: Monitor the SCADA system, acknowledge and dispatch alarmsto the appropriate staff for resolution (see slides 65-90 formethodology)

Task 5: Document steps taken for resolution of alarms (see slides 91-97 for methodology)

59


60/112


24 hour /day, 7 days/week: Task 6: Verify correct functioning of the Rogers Communications GPRS

network. (see slides 98-105 for methodology)

Task 7: Communicate with Pumpwells Field staff and contractors dailyand document who is at what site, performing what tasks, so false

alarms can be accounted for (see slides 106-108 for methodology)

Task 8: Document problems with the SCADA system functionality (seeslides 109-110 for methodology)

Task 9: Communicating problems with the functionality of the callcenters SCADA computer hardware or software (see slide 111 for

methodology)

60


61/112


Task 3: Monitoring the SCADA systems health: On the lower menu bar:

- State 1: Verify SCADA system is in

Normal Operating Mode:

PLC Communication Path: KENSCA01

KENSCA01: ACTIVE KENSCA02: STANDBY

- State 2: If not in Normal Operating Mode, verify SCADA system is inMaintenance Mode (Tony or Aimee is currently working on the Servers):


KENSCA01: ACTIVE KENSCA02: ACTIVE

and it is currently in the regular working day time period (7:30am-3:30pm,Monday to Friday)

- These two states are considered normal and nothing needs to be done atthis time

61


62/112



- State 1: If the SCADA system has

failed over:


KENSCA01: ?????? KENSCA02: ACTIVE

Or KENSCA01: STANDBY KENSCA02: ACTIVE

- This state indicates trouble with the primary server, however the systemhas failed over correctly and is functioning is it is intended for backup

purposes call Aimee Pulido or Tony Liao to investigate the problem if

during the working day, or wait until the next working day to call,provided no other server problems occur.

- If other SCADA is not functioning correctly in the above states, call Tonyimmediately.

62


63/112



- State 1: If the SCADA system

is not functioning correctly:

PLC Communication Path: KENSCA01 or KENSCA02 or ??????

KENSCA01: ??????

And, KENSCA02: STANDBY

- State 2: If the PLC Communication Path does not match the current activeserver

KENSCA01: ACTIVE and PLC Communication Path: KENSCA02 or ??????

KENSCA02: ACTIVE and PLC Communication Path: KENSCA01 or ??????- State 3: If the SCADA system is in Maintenance Mode (see previous slide)

and it is currently outside the regular working day time period (after3:30pm and before 7:30am (Monday to Friday), or all daySaturday/Sunday)

- These three states are not considered normal call Aimee Pulido or TonyLiao immediately to investigate the problem (Tony after hours) 63


64/112


Task 3: Monitoring the SCADA systems health:

During Regular Working Hours:Tony Liao Office: 780-423-3257

Cell: 780-868-2469Aimee Pulido Office: 780-496-1768Cell: 780-554-0824

Outside Regular Working Hours:

Tony Liao Cell: 780-868-2469Aimee Pulido Cell: 780-554-0824

64


65/112


Task 4: Monitor the SCADA system, acknowledge and dispatchalarms to the appropriate staff for resolution :

The Dispatch View Screen:

65


66/112



When an unacknowledged alarm is present:

- The City Overview Screen will display:

A blinking red outline

around the map A blinking red background in the

center of the lower navigation bar

A high-visibility message at the

bottom of the screen indicating

the presence of an

unacknowledged alarm

66


67/112



When an alarm is present:

- If the Alarm Horn is enabled, beeping will sound from the computer

- A row in the alarm banner at the top of the screen will blink yellow/blue,

indicating the tag that is in alarm- A row that is solid yellow indicates that a tag is still in alarm, but that the

alarm has been acknowledged (as viewed also by the in the Ack column

of the alarm banner)

A rows text will go green and then disappear once an alarm condition

has cleared (see next slide) 67


68/112


Task 4: Monitor the SCADA system, acknowledge and dispatchalarms to the appropriate staff for resolution:

A) Gather the facts about the alarm:

- Step 1: Double-click on the blinking alarm row

If the Alarm Horn is enabled, the audible beeping will stop, provided there areno other unacknowledged alarms in the alarm banner

The alarm will be changed to a solid yellow row, and the will be populatedinto theAckcolumn of the alarm banner.

The Date In and Time In columns will be populated with the current Date andTime when the alarm was generated. If the alarm was previously on hold thistime is the time the alarm was re-enabled (came off hold).

68


69/112




The Description column indicates the tag that is in alarms english description,rather than its technical name and is used in conjunction with the Value (thecurrent state of the tag in question) to determine the alarm condition

TheAlarm Extension Field2 column indicates the last task performed with

respect to alarm resolution for this tags current state to determine the alarmcondition

The alarm information below the alarm banner will be updated to theselected tags information

69


70/112




- Step 2: Carefullyread the information in the Dispatch Info tab

This outlines the steps the dispatcher is to take to resolve the alarm Each tag has different dispatch information, depending on the requirements

of the site, the severity of the alarm, the influent type, and many otherpertinent considerations

The selected tags dispatch information depends on the day, the current timeof day, and the season, and changes automatically

70


71/112




- Step 3: Carefullyread the information in the Temporary Notes

This informs the dispatcher of any temporary concerns or changed conditionsat the station that correspond to this alarm

It provides additional temporary alarm handling and response procedures,which supplement or overrule the Dispatch Info

If the note is more than 6 months old, contact the writer of the note durign

regular working hours to verify its accuracy and have them change the date totodays date 71


72/112




- Step 4: If there is no expiry date on a note, assume that the note is stillpertinent and:

Verify the alarm type:- For MECHANICAL ALARM, send

an email to the Pumpwell

Millwright Foreman (Denis Hillier) requesting that an expiry date is added to the

temporary notes for this station

- For ELECTRICAL ALARM, send an

email to the Electrical Engineer(Tony Liao) and the Electrical foreman (Heng Khey) requesting that an expiry date

is added to the temporary notes for this station

Document this deficiency for this tag value on the SCADA Concerns form inthe the SCADA Dispatch Forms Binder (Section 2) as per Functions of theDispatcher, Task 8

72


73/112




- Step 5: Read the information in the Site Info

This provides the dispatcher a basic understanding of :

- The stations address- Directions to get to the station

- The stations control philosophy (basic expectations of the sites function)

- The stations major equipment

- Interactions with other stations

- Other important information pertaining to the station

This information helps the dispatcher to understand the potential impact ofan alarm at the site and its surrounding areas

73


74/112




- Step 6: If, based on the Site Info and the Temporary Notes, the DispatchInfo seems incorrect or strange:

Respond to the alarm with due diligence, using the information provided asan outline for guiding your response decision (err on the side of safety)

Document the steps taken for alarm response and the reasons for deviatingfrom the listed procedure by typing the steps taken in the Log Note Box, andclicking the Log Only button

Document this deficiency for this tag value on the SCADA Concerns form inthe the SCADA Dispatch Forms Binder (Section 2) as per Functions of the

Dispatcher, Task 8

74


75/112




- Step 7: Carefullyread the information in the Calling Info

This outlines a list of people to call in order to report the alarm It is important to follow these steps fully and in sequential order

Pay close attention to call only the people on the list

75


76/112




- Step 8: Review the information in the Employee/Contractor Working InPumpwell Stations documentation for in the SCADA Dispatch FormsBinder (Section 1)

This outlines a list of employees or contractors

currently working at sites

Look for the station # to verify if anyone is listed

as working at that site

If an employee or contractor is on site:

- call them first prior to verify if the alarm isgenerated by the work they are completing

- Call them first for any alarms of the same type (electrical or mechanical)

At that site

76


77/112



B) Respond to the Alarm

- Step 1: If the Dispatch Info states toplace the alarm on hold

a) Determine if this is the initial alarming of the tag:

- In the alarm banner, Date In and Time In will be the current date and time

- TheAlarm/Event History will have no previous record of this alarm in the past 112hours

b) If it is the initial alarming of the tag, follow the Dispatch Info instructions todetermine how long to place the alarm on hold

c) Determine if the alarm waspreviously placed on hold (indicated by):

- Date In and Time In will be anywhere from 5 minutes to 112 hours ago (dependingon how long the alarm was placed on hold previously)

- TheAlarm/Event History will have record of this alarm being placed on hold in thepast 112 hours

d) If it was previously placed on hold, DO NOT PLACE IT ON HOLD A SECONDTIME, follow callout procedure.

77


78/112





To check the Alarm/Event History for records of this alarm:

- The Alarm/Event History comprises several fields of data queried from the ProficyHistorian, outlining any alarms or events that were generated in SCADA in a giventime frame

- Click theAlarm / Event History button on the bottom menu bar

- On theAlarm/ Event HistoryScreen, click the Down Arrow besideAlarm Typeuntil the Alarm Type reads Events

78


79/112






- Click the dropdown arrowbeside the Select Station dropdown box, scroll downand select the station pertaining to the alarm in question

- Change the start date to 4 days prior to today (the end date), using the startdatesdown arrow

79


80/112






- Drag the right edge of the column headers for the following columns to the left toshrink the columns

alarmid

source

eventcategory,

subcondition

start_value

end_value

acktime

acked

80


81/112


82/112






- If an alarm was previously placed on hold, the following lines will appear for thegiven alarm (in this case, the alarm was PW131_P1_PHASE):

- They are read from the bottom up, as the newest time is at the top

e) Click the Place On Hold Button

82


83/112





f) Select the correct date and time

(including AM|PM) according tothe instructions reviewed on the

Dispatch Info tab, Temporary Notes, or

Instructions received from person

called to deal with the alarm

g) Click OK

83


84/112





i) Confirm the disabling of the alarm by clicking OK

j) If the incorrect amount of time was selected, click Cancel and repeat theprevious process of placing the alarm on hold, using the correct amount oftime

- The alarm will be placed in the Held Alarms list and alarming will be disabled in theProcess Database until the end of the hold time

- The alarm will be logged automatically and the log status will be updated with thehold time selected

84


85/112




- Step 2: If the Dispatch Info states to call someone, determine who to callusing the Calling Info, and determine their phone number

During regular working hours:

- pick a NAME from the dropdown box, to where the call will be made

- The phone number will be displayed below the name selection box

85


86/112




- Step 2: If the Dispatch Info states to call someone, determine who to callusing the Calling Info, and determine their phone number

Outside regular working hours:

- Click the View Weekly Standby Listbutton

- The Weekly Off-Hours Response Information document will be opened in aseparate window

- This document provides a primary and an alternate phone number for each personon call for the week (commencing Wednesday at 3:30pm and ending the followingWednesday at 7:30am)

- The list changes weekly

- If the Alarm Type is Mechanical Alarm:

The first contact on the list will be the Pumpwell Millwright

The second contact on the list will be the Pumpwell Repairman

- If the Alarm Type is Electrical Alarm:

The primary contact on the list will be the Electrician 86


87/112




- Step 3: Make the phone call

Inform the contact of the:

- Alarm Description (in the Alarm Banner)

- Its current Value (in the Alarm Banner)

- Its start Date and Start Time (in the Alarm Banner)

- If it has previously been placed and come off the hold list

87


88/112





During regular working hours:

- The names in the dropdown list will change based on whether the alarm ismechanical or electrical in nature

- If the first person on the name dropdown list doesnt answer, according to theCalling Info instructions, try the next number on the dropdown list

- In the event that nobody on the list can be reached:

Leave a message. Wait 5 minutes and call through the list again

Repeat calling at 5 minute intervals until someone is reached

- In the event that nobody on the list can be reached after 20 minutes: Contact Pumpwells General Supervisor (Sid Lodewyk)

If the Pumpwells General Supervisor cannot be contacted, then contact the Director ofDrainage Services (Derek Melmoth)

Phone numbers for these are listed at the end of the name and phone numberdropdown lists

88


89/112






- The standby crews are listed on the standby list, according to the weekly standbyschedule

- For Mechanical Alarms:

Call the Pumpwell Millwright Primary #

If no answer, leave a message, call the Pumpwell Millwright Alternate #

If no answer, leave a message, call the Pumpwell Repairman Primary #

If no answer, leave a message, call the Pumpwell Repairman Alternate#

Repeat calling at 5 minute intervals until someone is reached- For Electrical Alarms:

Call the Electrician Primary #

If no answer, leave a message, call the Electrician Alternate #


89


90/112






- In the event that nobody on the above list can be reached:

Leave a message. Wait 5 minutes and call through the list again


- In the event that nobody on the above list can be reached after 20 minutes:

Contact the Standby Supervisor listed on the Weekly Off Hours Response Informationdocument

90


91/112


Task 5: Document steps taken for resolution of alarms- Step 1: Log the resolution:

Select from the LOG NOTE list:

- Notified Millwright Foreman:

If the millwright was spoken to, and it was stated that he would take care of it

- Notified Electricians:

The alarm was listed as an electrical alarm, the electrician was contacted and will betaking care of the problem

- Notified Maintenance Crew:

The millwright foreman could not be contacted, the alarm was listed as an mechanicalalarm, so a pumpwell maintenance crew was contacted and will be taking care of theproblem

- Notified Carmacks:

For stations PW5016, PW5017, PW5018 alarms 91


92/112




- Notified Edmonton Catholic Schools

- Notified Edmonton Public Schools:

For high levels in ponds on school properties, as per alarm response instructions on

certain stations- Notified Bob Richards/ Greg Bergsten:

For alarm response with respect to Rain Gauges

- Notified Shaune Rivard:

For alarm response with respect to certain alarms at PW5024

- Notified Tony Liao

- Notified Aimee Pulido

For alarm response with respect to SCADA or server issues 92


93/112



Select from the LOG NOTE dropdown list:

- Notified Supervisor:

If the responsible personnel could not be contacted and the pumpwell supervisor(during regular working hours) or the standby supervisor (outside regular workinghours) was contacted instead

- Notified Director:

If the responsible personnel could not be contacted and the pumpwell supervisor(during regular working hours) or the standby supervisor (outside regular workinghours) could not be contacted, and the director was contacted instead

- Notified Standby Crew:

If it is outside normal working hours, standby personnel was spoken to and it wasmentioned that they would go and take care of the issue

93


94/112




- Notified Drainage Operations Preventative Maintenance

If the alarm is related to:

- high water level on the North Saskatchewan River

- Notified EPCOR dispatch: If the alarm is related to:

- high water level on the North Saskatchewan River

- EWSs (Early Warning Systems) Mix chemical now or high level alarms

- No Answer:

Telephoning personnel resulted in no answer (telephone line rang (was not busy),nobody answered the call)

94


95/112




- Busy Phoneline

Unable to speak with the required personnel, as the phone line is busy

Wait 2 minutes and try the call again

If unable to get through, then call the next person on the list- As Instructed by Supervisor:

If the supervisorinstructed the alarm to be placed on hold

- As Instructed by Temp Notes:

If the instructions in the Temporary Notes were followed rather than the instructions inthe Dispatch Info

95


96/112




- Crew On Site:

A crew has called in and is working at the site (as documented on theEmployee/Contractor Working In Pumpwell Stations form, and they have generatedthe alarm

- Central Station Fault- False Alarm:

A false alarm was generated by an error in SCADA or on the servers

- Contractor Tripped False Alarm:

A contractor is working at the specified site, they have been contacted (i.e. as instructedby temporary notes), and they have mentioned that it is a false alarm caused by theirwork

- Accidental Trip:

The personnel on site has mentioned that they accidentally tripped the alarm 96


97/112




- Testing

A new alarm is being implemented and tested or an existing alarm is being tested , asnotified by the contacted personnel

- Point Message Not Configured

Message for the alarm states Point Message Not Configured, which indicates that

there are no current procedures documented for this alarm in the Dispatch Info If this message is displayed in the Dispatch Info, document this deficiency for this tag

value on the SCADA Concerns form in the the SCADA Dispatch Forms Binder (Section 2)as per Functions of the Dispatcher, Task 8.

Send an email to Denis Hiller or Tony Liao outlining the alarm, the current state of thealarm and the Point Message Not Configured status in Dispatch Info for that particularalarm

Tony and Denis, will review the email, and will make the change in the system

97


98/112


Task 6: Verify correct functioning of Communications network All communications from the PLCs at remote stations are transmitted

via the Rogers Mobility General Packet Radio Service cellular network

Communications are monitored by the SCADA system for everyremote station

The Communications refresh rate

(minimum time between PLC to

SCADA Server communications) is

set in the communications settings

picture of each given site Each station has a different refresh

rate depending on the criticality of the

site

98


99/112


Task 6: Verify correct functioning of the Communications network If the SCADA servers fail to receive station communications after 4

times the refresh rate, a COMMUNICATION FAILURE ALARM will begenerated for that station

There are a quite few possible causes for a Communication FailureAlarm

99


100/112


Task 6: Verify correct functioning of the Communications network Cause 1: Simultaneous occurrences of communication failure alarms

over a large geographic area of sites is unusual, and is likely a networkfailure with:

- The Rogers Mobility Cellular Network

If the progression of faults appears to stall at a dozen or so sites, it is more likely theproblem is in the Rogers Communications network, where an individual cellular towerhas a fault

If the entire city is in communication failure (and you are able to use your computer toaccess your H:\ drive or another network location, indicating that the Citys network isstill fine) It is likely a larger outage with Rogers network

- The City of Edmonton Network A complete breakdown may occur in the event of failure of the City of Edmonton

- Access point

- Networking equipment

- Extended Power failure

100


101/112


Task 6: Verify correct functioning of the Communications network For network-related communications failure:

- Immediately Call Tony Liao (for entire city in alarm) or one of theelectricians (if not entire city in alarm), working your way through theelectricians callout list

- If they require the Rogers APN Customer Support phone number, provideit to them: 1-877-939-3282 (24 hours)

101


102/112


Task 6: Verify correct functioning of the Communications network Cause 2: Simultaneous occurrences of a few communication failure

alarms of sites (excluding rain gauge RG### communicationfailures) in no specific geographic pattern is not unusual, and is likely aproblem with:

- The modems or PLCs at the affected stations- Radio signal interference

For site-related communications failures:

-Step 1: On the station dropdown menu in the lower menu bar of the SCADAwindow, select the station affected by the alarm.

- Step 2: If a system graphic opens, click on the affected station

- Step 3: Once the station graphic opens, click the COMMUNICATION button

102


103/112


Task 6: Verify correct functioning of the Communications network For site-related communications failure:

- Step 4: On the Communications Settings Window check that the refreshrate matches the refresh rate in the note at the bottom of the screen

- Step 5: In the event of a discrepancy, type the number of minutes into therefresh rate minutes box

103


104/112


Task 6: Verify correct functioning of the Communications network For site-related communications failure:

- Step 6: On the lower menu bar, verify which SCADA Server is ACTIVE

- Step 7: If KENSCA01: Click the button

- If KENSCA02: Click the button

- Step 8: Click the button

- Step 9: Return to the Dispatch View system and continue monitoringalarms

- Step 10: If alarm doesnt go away after 5 minutes, follow instructions aslisted in the Dispatch Info

104


105/112


Task 6: Verify correct functioning of the Communications network Cause 3: Rain gauge (RG###) communication failures in no specific

geographic pattern is notunusual, and is likely a problem with:

- The modems or PLCs at the rain gauge sites

- Radio signal interference

- For rain gauge-related (sites RG037, RG038, RG039) communicationsfailures:

- Step 1: During Regular working hours, call

- Bob Richards or Greg Bergsten 780-4965697

After Hours:

Place on hold until next working day , as per alarm response instructions

105


106/112


Task 7: Communicate with Pumpwells Field staff andcontractors daily and document who is at what site,performing what tasks, so false alarms can be accountedfor

When an employee or contractor calls to informdispatch that an entry into a site will take place:

- Use the Employee/Contractor Working In PumpwellStations documentation for in the SCADA DispatchForms Binder (Section 1) to document the following:

Date In/Out

Time In/Out

Pumpstation # Employee/Contractor Name

Work Being Performed

- Remind the employee / contractor to call when theyare leaving the site

- Inform other dispatchers of the person on site, so

modifications in alarm response can be made 106


107/112


107


108/112


Task 7: Communicate with Pumpwells Field staff andcontractors daily and document who is at what site,performing what tasks, so false alarms can be accountedfor

When a sheet is full:

- Place it in the Completed Employee/Contractor InStations Forms section (section 3) of the SCADADispatch Forms Binder

- Start a new sheet

If new sheets are needed for printing:

- The template is located in:

N:\Operations\Customer Services\HIGH PRIORITYREPAIRS\Call Management Centre\SCADA TrackingForms\Employee or Contractor Working In PumpwellStations Tracking Form.pdf

108


109/112


Task 8: Document problems with the SCADA system functionality The SCADA Concerns that are to be documented on this form are:

- If an alarm has a point message not configured

- If an alarm has a strange response message in the Dispatch Info

- If an alarm has missing information either in the Dispatch Info or the Site

Info- If there are problems with acknowledging an alarm

- If there are error messages generated

- If there are extra features that would work well, or make the job easier

109


110/112


Task 8: Document problems with the SCADA system functionality

110


111/112


Task 8: Document problems with the SCADA system functionality When a sheet is full:

- Place it in the Completed SCADA Concern Forms section (section 4) of theSCADA Dispatch Forms Binder

- Start a new sheet

- On a monthly basis, give these sheets to Aimee Pulido or Tony Liao forreview

If new sheets are needed for printing:

- The template is located in:

N:\Operations\Customer Services\HIGH PRIORITY REPAIRS\CallManagement Centre\SCADA Tracking Forms\SCADA Concerns TrackingForm.pdf

111


112/112


Task 9: Communicating problems with the functionality of the callcenters SCADA computer hardware or software When a problem arises with respect to the functionality of the call centers

SCADA computer (hardware/software) or the communications network:

- During normal working hours, contact Aimee Pulido or Tony Liao forsupport

- Outside regular working hours, contact the Standby Technologist

Documents

SCADA System Call Center Training Manual-Revised Sept 2013