IA Series LifeLiner Session 6

Invensys and the Invensys logo are trade marks of Invensys plc

EMEA LIFELINE Session 6

Irma Aguilar Jaroslaw Dubacki

Introduction

© Invensys 00/00/00Invensys proprietary & confidentialSlide 2

I am Irma Aguilar from Invensys Qatar. I am a principal service engineer and I have been working with Invensys systems for 18 years.

And I am Jarek Dubacki from the Invensys Poland. I am a control system engineer and have worked on IA system for the last 15 years. Last 8 years – as Invensys employee

© Invensys May 2, 2023Invensys proprietary & confidential3

Lifeline Session 61. What we usually do as preventive maintenance is:

• Physical inspection of equipment• Review of software releases and fixes• Technical and/or product alerts• Analysis of system conditions (counters, loading, etc.)

2. The scope usually does not include:• Network load and traffic analysis

That’s why on this session we are going to collect and organize knowledge that may be usefull in network maintenance.

FERRET + SAV


1Why to focus on multicast and broadcast

Broadcast and Multicast DefinitionBROADCAST

In computer networking, broadcasting refers to transmitting a packet that will be received by every device on the network

MULTICASTIn computer networking, multicast is the delivery of a message or information to a group of destination computers simultaneously in a single transmission from the source.



Broadcasts/Multicasts vs I/A System

1. I/A Series system network by design shouldn’t use high amount of broadcast/multicast.

2. It is send by a process on one station, that is looking for info on other station on the network without knowing where it is.

3. Each station on the network is disturbed.4. Sustained overloading can cause excessive collisions, buffer

overruns, timeouts etc.5. High level of broadcasts is indicating that some programs and/or

services are not working efficiently/correctly and your network need closer look. That can be caused by station running out of resources, engineering not using best practice.

Engineering practice

BAD GOOD


omget

Script executed every 1s

omgetimp

Script executed every 1s

omget vs omgetimp

Script executing omget every 1s Script executing omgetimp every 1s

© Invensys 05/02/2023Invensys proprietary & confidential8

AVG no of _BCAST/s Jump from 1,32 to 4,28 !!!

Running scrip Running scrip

No impact on AVG no of _BCAST/s Only two pkgs sent – shown by blue arrow

Sequence code with external references:

Sequence code with internal references:

Engineering practiceBAD GOOD


VARIABLESa : R;b : R;c : R;d : I;

USER_LABELSflow : RO0001;

STATEMENTSa := :COMPOUND1:CALC1.RO0001;b := :COMPOUND2:CALC2.RO0001;c := :COMPOUND3:CALC3.RO0001;d := :COMPOUND4:CALC4.IO0001;

flow := (a + b + c) / d;

ENDSEQUENCE

VARIABLESUSER_LABELSa : RI0001;b : RI0002;c : RI0003;d : IO0001;flow : RO0001;

STATEMENTS

flow := (a + b + c) / d;

ENDSEQUENCE....


Natural Broadcasts/Multicasts

1.Displays opened for the first time2.Reboot of a station3.Starting historian4.Device Monitor heartbeat5.Applications requesting data in the Network (omget)


Native I/A Series MulticastsName MAC ADDRESS Description

_BCAST 01006C000000 to all stations except the sender_COMM 01006C010101 to all communication processors (COMM Servers)_DEVMN 01006C020202 to all device monitors_DIAG 01006C030303 to all diagnostic supervisor_ELDB 01006C040404 to ELDB manager_ENDS 01006C050505 to all end systems (all stations except CBLIs)_CP 01006C060606 to all control processors_FSG 01006C070707 to all foxnet slave gateways_WP 01006C080808 to all workstation processors (GCs)_HIST 01006C090909 to all historians_INTS 01006C0A0A0A to all intermediate systems (CBLIs)_TMKPR 01006C0B0B0B to master time keeper_RSVLK 01006C0C0C0C for unresolved linkage_SMDSP 01006C0D0D0D to all system monitor display handlers_SYSMN 01006C0E0E0E to all system monitors_AP 01006C0F0F0F to all application processors (File Servers)CABLETST 01006C101010 _SYNC_PULSED 01006C111111 to all sync pulsed stations (ZCP270, FCP270, FCM100Et, see B0700AQ)--LI-- 00006C0001FF Label for the CBLI local to Foxwatch


Unwanted Broadcast/Multicast

1.Display configured with C:B.P does not exist2.Non existing Alarm destination (printer, WP, historian) in CP database3.Typos/errors in C:B.P links in blocks (P2P)4.Bad engineering practice5.Ghost lists6._AP multicast ‘boot me’ request on a stable network


I/A Series Architecture Overviewo Protocol IEEE802.3o Speed/Bandwidth 10 Mb/so for 150 bytes packets at 10% of full bandwidth

will give 833 pkts/so Single share collision domaino Half Duplexo Up to 3 segments via NBE or FONBEo Max. 64 stations (32 in a segment)

o Protocol IEEE802.3u Fast Ethernet and IEEE802.3z Gigabit

o Speed/Bandwidth: 100 Mb/s – 1 Gb/s (between uplinks)

o Switched Etherneto Full Duplexo Redundant Network accesso Collision Domain not longer existo Rapid Spanning Tree Protocol (RSTP)o Root and Backup managed switcheso Max 1920 stations

o Address Translation Station ATS designed to merge new MESH with old Nodebus systems

o separates unnecesary network traffic between MESH and NODEBUS.

o 10 Mb/s (NODEBUS) and 100 Mb/s on MESH

o one Collision domain on NODEBUSo Half Duplex (NB), Full duplex (MESH)o Many Nodebus networks can be

connected to the MESH using ATS

Critical Broadcast Values


REMEMBER!•Those numbers are not fixed!•It can be different on your stations•It always depends on CP load in I/O scan and block processing load level.•It’s based on the experience of TAC – not docummented

When the number of Multcasts/sec does reach a level of 25 /sec for a period of more than .5 minute that may affect the operation of Nodebus FT stations. ACM’s, CP30/40 stations may go single.When the number goes up it may cause these station types to stop communicating across the network.CP270, CP60 and 30B/40B stations will show this behavior at higher values.So we need to stay away from this type of MultiCast levels.

Broadcast Alarm Values REMEMBER!

•Values are not fixed!•It is independent of the size of the network•It’s based on the experience – it can be found on GCS web


Normal = 2 Mcast/secError = 8 Mcast/secCritical = 14 Mcast/sec


2How to properly collect data

TOOLS/SOFTWARE

1. Some Network Analyzer software that can be used to collect traces from the Network are:

1.EtherPeek - Licensed2.WireShark – Free 3.MS Network Monitor - Free

2. FERRET/SAV as additional tools to get aliases/letterbug list of the stations in the Network and Data Topology Report.



NODEBUS To collect traces from Nodebus is required to use DNBT + PC/Laptop with WireShark

This procedure is documented in our GCS under SOL1276


MESHIn Mesh Network the most easy way to collect traces is by installing WireShark in one of the Workstation connected to the Mesh.

In both scenarios Nodebus and Mesh it is not an automatic process. It is on demand.

Calculation of average is manual.

Analysis is manual.


Mixed NODEBUS / MESHIn same way is done for mixed Networks. Collecting traces from a Workstation connected to the MESH.

If it is required to collect traces from a specific Nodebus and do more analysis, then the DNBT is used.(

Remote Watch 4.0 has a Multicast Monitor tool, which basically is Wireshark running in background collecting traces every 10 minutes from a Mesh Workstation.

It is automatic process!!RW has two alarms set for Multicast, additionally it creates emails/alerts to the system engineer when the threshold has been exceeded.


3Analysis

Generate the Multicast report


WiresharkCapture

ofMulticast’s

Parser

A2xml.bat

XML

report

XLST

template

Using Multicast report


Multicast reportSection 1 :• show the statistics of the Wireshark Capture file

Section 2:• Lists the individual Multicasts types with the number of Multicasts found for the

individual types

Invensys Multicasts


Inv_BCAST_(00) Invensys BCAST’s are using the 01:00:6C:00:00:00 multicast address.

An operator station calling a display and attempting to Locate the C:B.P’s

These type of Multicasts are an example of using the Multicast mechanism correctly, however this should not happen all the time (preference is to use Display Optimization or in case om OMGET use the OMGET_Imp).

In this example we have many multicasts in the trace over a period in an hour.

What do we see in the Inv_BCAST_(00) packet

• To_address : 01:00:6C:00:00:00

• An application looking for PAMELATIJMEN:AIN1.PNT

Let’s look closer”_BCAST are not easy ones”


Look into a packageThere it is!!!. C:B. That is not in the DB. A display opened in AWLADY has the problem.

*********Cyan Point

Detecting Boot_Me requests


Inv_AP_(0F) Boot_Me requests are using the 01:00:6C:0F:0F:0F multicast address.

An IA station that does require a Host to booted will sent a set of Boot_Me request to the network via this Multicast type.

In a healthy system these Multicasts should only be there if a station is performing a startup.

In this example we have 4 Inv_AP_(0F) multicasts in the trace over a period in an hour.

What do we see in the _AP packet

• To_address : 01:00:6C:0F:0F:0F

• Station Letterbug for the station sending the Boot_Me request HJUDIT

Let’s look closer


Only _AP Multicast

Locate Logical Name


Locate LN If an Application or Station is unable to find a configured Logical Name (Historian Name, Alarm Destination, Printer Name), it will use the Invensys Multicast (01:00:6C:00:00:00) to Locate it. These packets have a typical size of 143 Bytes.

Under Locate LN we report the number of packets found matching the above.

Preference is to have this number as 0, if not the we should understand why.

In this example we have 35486 Locate LN multicasts in the trace over a period in an hour. So time to checkout why!

What do we see in the Locate LN packet

• To_address : 01:00:6C:00:00:00• Packet Size: 143 bytes• Historian name HSLADY (uppercase) should be hslady (Lower case)

Introducing a wrong device alarming name and a Display with bad C:B


There it is!!!. Historian name is in UPPERCASE (HSLADY)

Results/Conclusion:

By looking at the Report under Locate LN and filtering the trace with Inv_BCAST_(00) it was possible to find the root cause. The historian name was corrected to hslady and this Multicast disappeared from the network traffic.

Detecting DEVMON Heartbeats


Inv_DEVMN_(02) The Device Monitor Master uses this Multicast to as heartbeat and check the Device Monitor slaves.

In a healthy system this Multicast is expected every 60 sec.

In this example we see 240 Inv_DEVMN_(02) multicasts in the trace over a period in an hour.

What do we see in the _AP packet

• To_address : 01 00 6C 02 02 02

• 2 pakets with 1 second apart, each one echoed by Mesh.

_DEVMN heartbeat


Results/Conclusion:

By looking at the Report and filtering the trace with Inv_DEVMN_(02) it was possible to verify the good performance of the Device Monitor.

ARP Multicasts


ARP Bcast partAddress Resolution Protocol (ARP) is a telecommunications protocol used for resolution of network layer addresses into link layer addresses

ARP packets will be found In all systems using the IP protocol and will be used to convert an IP address to a physical address such as an Ethernet address.

We are currently investigating why the number of ARP’s are constantly there and do not seem to be able to get the required ARP info.

QUESTIONS


Invensys and the Invensys logo are trade marks of Invensys plcMay 2, 2023

Invensys proprietary & confidentialSlide 34

Documents

IA Series LifeLiner Session 6