Transforming Mission-Critical Networks · 7705 SAR 7705 SAR R B B B 7705 SAR Figure 2. Alcatel-Lucent IP/MPLS-network. transforming Mission-Critical networks alCaTel-luCenT appliCaTion

Transforming mission-CriTiCal neTworks IP/MPLS network tranSforMatIon to SuPPort SCaDa aPPLICatIon MIgratIonaPPLICatIon note

absTraCTOur modern society is fully dependent on smooth and safe operations of industries such as power utilities, oil and gas as well as public transport and safety authorities, whose operations are usually widespread geographically. To ensure smooth and safe operations, their mission-critical networks need to collect data from and monitor processes at all remote stations using supervisory control and data acquisition (SCADA) systems.

As mission-critical networks rapidly adopt IP/MPLS as part of the converged network transformation program, continued support for the low-speed serial data generated by widely-deployed SCADA systems is a challenge. Even though IP/MPLS is capable of carrying TDM data transparently, there is no obvious solution to merge traffic from the many remote locations at the control center. Merging traffic requires an advanced TDM data bridging capability called Multi-Drop Data Bridge (MDDB) to be available on an IP/MPLS platform.

Rising to this challenge, Alcatel-Lucent delivers a converged IP/MPLS-based com-munications solution with an integrated MDDB that enables an IP/MPLS network to be compatible with legacy TDM applications. This paper describes how this solution can be deployed to carry legacy SCADA application traffic in an IP/MPLS network.

Table of ConTenTs

Introduction / 1

Challenges for mission-critical networks / 1

SCaDa overview / 1

alcatel-Lucent IP/MPLS network Solution / 2

Many VPnS, one network / 3

alcatel-Lucent IP/MPLS solution components overview / 4

alcatel-Lucent MDDB Solution for SCaDa application Migration to IP/MPLS / 4

Communication between master and slave equipment / 4

Solution architecture and overview / 5

Master equipment redundancy protection / 7

Branch squelching / 9

Conclusion / 10

acronyms / 10

transforming Mission-Critical networksalCaTel-luCenT appliCaTion noTe

1

inTroduCTionChallenges for mission-critical networksOur modern society is fully dependent on smooth and safe operations of industries such as power utilities, oil and gas as well as public transport and safety authorities, whose operations are usually widespread geographically and at times extend even to uninhabit-able terrain. To ensure smooth and safe operations, their mission-critical networks need to collect data from and monitor industrial processes at all remote stations. SCADA systems are designed to fulfill this need. SCADA is sometimes called telecontrol equip-ment in power utilities, as in the IEC 60870 standard suite, as well as interlocking system by rail operators.

A SCADA system has a master station in a control center and many slave stations in the field communicating with each other over a low-speed serial interface. As mission-critical communications networks are rapidly adopting IP/MPLS as part of the converged network transformation program, one challenge is continued support of low-speed serial data generated by widely-deployed SCADA systems. These systems can have a service life as long as 20 to 25 years, and they use proprietary TDM protocols. Even though IP/MPLS is capable of carrying TDM data transparently with TDM pseudowire technology, there is no obvious solution to merge the traffic received from many remote locations received at the control center. To do this requires an advanced TDM data bridging capability called MDDB to be available on an IP/MPLS platform.

sCada overviewBy centrally monitoring alarms and processing status data in the field continually with SCADA systems, the efficiency and uptime of the industrial process can be increased, resulting in substantial operational savings. Common applications include voltage, current and frequency reading in power grids, pressure measurement in oil and gas pipelines, as well as automation of traffic lights and railroad crossing gates.

A typical SCADA system has four components:

• Sensors: Devices that monitor the managed process and equipment.

• Remote Telemetry Units (RTUs): Devices in the field that collect information from sensors and transmit it to a SCADA master. RTUs are commonly called slaves.

• SCADA master: The main end equipment at the control center that the user interacts with, usually through a Human Machine Interface (HMI) that runs on a computer. The SCADA master controls and communicates with a number of RTUs/slaves periodically.

• Communications network: Sits between the central master and remote slaves, providing reliable and resilient communications between them. Because it carries information critical to safe and efficient operation of the industrial process, the communications network is considered to be a mission-critical network.


2

Figure 1 shows the four SCADA system components.

Figure 1. SCADA system components

REMOTE SITE

CONTROL CENTER

Sensors

RTU/slave

Master

HMI

Sensors

RTU/slave

Mission-criticalcommunications network

alCaTel-luCenT ip/mpls neTwork soluTion Many operators of mission-critical networks have started to consider deploying, or have already deployed, converged next-generation networks to support all their com-munications needs. However, not all next-generation solutions are appropriate. To simultaneously support all mission-critical and non-mission-critical traffic, an IP/MPLS-based communications network is needed.

Non-MPLS-based IP networks have grown significantly in recent years, but they often lack the necessary traffic management capability to support traffic that requires strict quality of service (QoS) for mission-critical operations. They also lack the flexibility to optimize the use of network resources and the capability to react to network events fast enough to guarantee end-to-end QoS per application.

By using an Alcatel-Lucent IP/MPLS network, operators get the best of both worlds — the versatility of an IP network and the predictability of a circuit-based network along with high capacity and support for packet-based traffic with high QoS. An IP/MPLS network enables the deployment of new IP/Ethernet applications and also supports exist-ing TDM-based applications. Because IP/MPLS networks can continue to carry existing TDM services, operators can now flexibly choose when to migrate the applications from TDM to IP.


3

With an IP/MPLS network, operators have a network with the following features:• High scalability and robustness with full redundancy and rapid recovery mechanism

such as MPLS Fast Reroute (FRR)• A solution that addresses a wide range of QoS and Service Level Agreement (SLA)

requirements, from circuit emulation to best-effort Internet surfing• Optimized bandwidth usage of all links and avoidance of common modes through

traffic engineering• An extensive operations, administration and maintenance (OAM) suite for performance

monitoring, troubleshooting and maintenance at all protocol layers

• Advanced network and service management to simplify operations

Each application run on the network has its unique requirements for bandwidth, QoS and availability. An IP/MPLS network enables operators to configure service parameters for each service and traffic type according to operational requirements. This includes multiple types of voice, video and data traffic. The network can also support low jitter and delay to handle all traffic types effectively and reliably in real time. In addition, an Alcatel-Lucent IP/MPLS network supports advanced capabilities, including non-stop routing, non-stop services and FRR, to maintain high network resiliency.

many Vpns, one networkAn Alcatel-Lucent IP/MPLS network provides for the virtual isolation of various traffic types on a single infrastructure supporting many Virtual Private Networks (VPNs) simultaneously. As shown in Figure 2, whether the network is a Virtual Leased Line (VLL) of various types, a Virtual Private LAN Service (VPLS) or a Virtual Private Routed Network (VPRN), deploying Alcatel-Lucent IP/MPLS allows full separation of control and data traffic in each VPN from other applications or operations in the network. The results are a fully secured environment, effective infrastructure sharing and optimal bandwidth allocation. With this advanced capability, the same IP/MPLS network infrastructure can be leveraged to also carry corporate business data.

VPLservice

Virtual bridge

VPLSLayer 2 bridged multipoint Ethernet service

IP/MPLSnetwork

B B

B B

ATMservice

VLLPoint-to-Point pseudowire(such as TDM or frame relay)

IP/MPLSnetwork

TDMservice

Ethernetservice

VPRNLayer 3 IP VPN

Virtual router

R R

R

R

R

7705 SAR

7705 SAR

R

BB

B

7705 SAR

Figure 2. Alcatel-Lucent IP/MPLS-network


4

alcatel-lucent ip/mpls solution components overviewThe Alcatel-Lucent IP/MPLS implementation provides a service-oriented approach that focuses on service scalability and quality as well as per-service OAM. A service-aware infrastructure enables the operator to tailor services such as mission-critical applications so that the network has the guaranteed bandwidth to meet peak requirements. The Alcatel-Lucent service routers support IP routing and switching, which enables the network to support real-time Layer 2 and Layer 3 applications.

The Alcatel-Lucent converged IP/MPLS network leverages multiple state-of-the-art technologies. The network extends IP/MPLS capabilities from the core to access and can include the following main components:

• Alcatel-Lucent 7750 Service Router (SR)

• Alcatel-Lucent 7705 Service Aggregation Router (SAR)

• Alcatel-Lucent 7450 Ethernet Services Switch (ESS)

• Alcatel-Lucent 7210 Service Access Switch (SAS)

• Alcatel-Lucent 9500 Microwave Packet Radio (MPR) providing packet microwave link connecting MPLS nodes

• Alcatel-Lucent 1830 Photonic Service Switch (PSS) as optical layer underlying the IP/MPLS network

• Alcatel-Lucent 5620 Service Aware Manager (SAM) for service and network management

alCaTel-luCenT mddb soluTion for sCada appliCaTion migraTion To ip/mplsThe Alcatel-Lucent MDDB solution for SCADA is supported on the Alcatel-Lucent 7705 SAR product family to help operators migrate traffic from current and legacy SCADA application on a TDM network to an IP/MPLS network.

Communication between master and slave equipmentIn a SCADA system, one master can be responsible for tens or hundreds of slaves. The interface used to connect to the communications network is usually a serial interface such as V.24 or X.21 with a bit rate ranging from 300 b/s to 19.2 kb/s. The communica-tion between the master and slaves is as follows:

1. The master queries the individual slaves sequentially using a broadcast query message embedded with a unique slave address encoded inside the message.

2. Although the broadcast query message is sent to all the slaves, only the addressed slave processes the query and responds by sending data back in a reply message.

3. Through another broadcast message encoded with another slave address, the master then queries another slave. The slave responds as in Step 2.

4. These steps are repeated continuously.


5

During the idling time, all slaves are transmitting an all ones pattern back to the master. To filter the response message, a data bridge performing an “AND” gate logic function is required.

Figure 3 shows this process with five slaves being queried sequentially.

Figure 3. SCADA communications between master and slaves

Master polls individual slave sequentially through broadcast message

Polled slave replies to master through unicasting

2

1

3

5

4

AND

Unlike some real-time applications, such as voice or teleprotection, SCADA data is not delay-sensitive.

solution architecture and overviewFigure 4 shows the architecture of the Alcatel-Lucent 7705 SAR-based MDDB solution. A 7705 SAR at the control center aggregates traffic from slaves for the master 7705 SARs at the remote sites that interface with the slaves. Figure 4. 7705 SAR-based MDDB solution

IP/MPLSnetwork

REMOTESITES

CONTROL CENTER

IP/MPLS routerwith MDDB

Serialinterface

1

2

Serialinterface

7705 SAR

Serialinterface

1

C-PIPEC-PIPE


6

As labeled in the figure, there are two key functions in the communication between master and slaves that the MDDB solution provides:

1. Using TDM pseudowires, also called C-pipes, the 7705 SARs at remote sites packetize and transport traffic generated by the low-speed serial interface of slaves across the IP/MPLS network towards the control center gateway router.

2. At the control center, a 7705 SAR router acts as the MDDB. The MDDB is implemented in a dedicated resource card called the Integrated Services Card (ISC). It receives all traffic from various slaves through individual TDM pseudowires, filters out the idling traffic and sends a reply message to a particular slave.

The master communicates with the slaves using the same steps in reverse order. The master sends traffic to the MDDB, then the traffic is broadcast over individual pseudow-ires to each slave.

The traffic from slaves is transported across the IP/MPLS network using TDM pseudow-ire technology as described in IETF RFC50861.

Because the interface speed usually ranges from 300 b/s to 19.2 kb/s, the traffic needs to first be rate-adapted to 64 kb/s. It is then packetized into an MPLS packet. The packet is carried over a pseudowire inside a Label Switched Path (LSP) tunnel established by MPLS signaling.

The ISC is a powerful resource that can be virtualized to support multiple applications simultaneously. Multiple MDDBs can be supported on the same ISC (see Figure 5).

Figure 5. A two-MDDB deployment scenario

Two virtual MDDB instances

7705SAR

C-PIPEC-PIPEC-PIPEC-PIPE

1 http://tools.ietf.org/html/rfc5086

http://tools.ietf.org/html/rfc5086


7

masTer equipmenT redundanCy proTeCTionMaster equipment is critical to SCADA operation. If it becomes faulty, no field data and alarms can be recorded and processed. This situation could potentially cause catastrophic damage if the failure lasts for a long time. Therefore, redundancy protection is required to maximize uptime.

SCADA solutions support redundant masters with both an active and a standby master listening to replies from slaves but only the active master transmitting. The Alcatel-Lucent MDDB solution is designed to work with this master redundancy behavior. There are various protection models for network operators to choose from. Each model provides a different level of protection and requires a different amount of resources to implement. Depending on the network’s reliability and robustness requirements as well as other logistics constraints, an operator can choose accordingly.

Model 1: Active/standby master pair with A/B switchModel 1 has one control center router with one MDDB and an A/B switch connected to active and standby SCADA master equipment, to provide redundancy protection. If the active master fails, the operator intervenes manually to activate the A/B switch to connect to the standby master.

Model 2: Active/standby master pair with two interfaces over single MDDBModel 2 is similar to Model 1 except that instead of using an external A/B switch, each master connects to the MDDB with its own serial interface. The standby master’s inter-face status is configured to standby. If the active master fails, the operator intervenes to toggle the standby master status to active using the Command Line Interface (CLI) or the network manager.

Figure 6 shows Model 1 and Model 2. Figure 6. Protection models 1 and 2

A/B SWITCH

Serialinterface

Serialinterface

7705 SAR

MODEL 2:Active/standby master pair withtwo interfaces over single MDDB

MODEL 1:Active/standby master pair with A/B switch over single MDDB



8

Model 3: Active/standby master pair with two MDDBs over one routerIn Model 3, in addition to the two masters, there are two MDDB instances running on two ISCs in the control center router. The router is also typically equipped with redundant control and fabric complex and dual power feed to eliminate any single point of failure.

Model 4: Active/standby master pair with two MDDBs over two routersIn Model 4, the two MDDB instances run on two control center routers, each connecting to a different master. The two routers and two masters can be located in different racks or even on different floors of the control center to provide a limited degree of space diversity protection (but no geographic diversity protection).

Figure 7 shows Model 3 and Model 4.

Figure 7. Protection models 3 and 4

MODEL 3:Active/standby master pairwith two MDDBs over one router

7705SAR

ISC

MODEL 4:Active/standby pair withtwo MDDBs over two routers


Model 5: Primary/backup control center with active/standby/standby master trioWhile Model 4, with two control center routers, can provide router redundancy, when a disaster such as an earthquake or hurricane strikes, the whole control center building can be damaged, affecting both routers and master equipment.

Model 5 provides protection in this scenario by placing a complete set of equipment (SCADA master and router) in a backup control center that can be tens or hundreds of kilometers away (see Figure 8). In case of active master failure, the standby master becomes the active master, as in Model 3.If the operating control center is seriously damaged, staff can quickly move to the backup undamaged control center to continue operations.


9

Figure 8. Protection Model 5

MODEL 5:Active/standby/standby master triowith primary/backup control center

PRIMARY CONTROL CENTER BACKUP CONTROL CENTER

ISC

C-PIPEC-PIPE

Table 1 provides a concise comparison of the five protection models. Each model has its own merits and associated costs. To choose the model that best suits their needs, opera-tors should assess their reliability and robustness requirements.

Table 1. Comparison of the five protection models

model 1 model 2 model 3 model 4 model 5

required equipment at Control Center

2 x Master1 x a/B Switch1 x 7705 Sar1 x ISC Card (for MDDB)

2 x Master1 x 7705 Sar1 x ISC Card




geographic diversity

no; all in the same Control Center




Yes; located in two Control Centers

Protected element

Master7705 Sar Control/fabric/power

Master7705 Sar Control/fabric/power

Master7705 SarControl/fabric/powerISC Card

Masterwhole 7705Sar node

Masterwhole 7705 Sar nodeControl Center building

branch squelchingWhen a slave goes out of order and continues to send data after the allotted response time, it can lock up the MDDB so that it can no longer properly filter responses from other slaves. With branch squelching, after a user configurable period expires, incoming data from the slave is overridden and replaced by an all ones pattern so that the MDDB can continue to service other slaves. Meanwhile, an alarm is raised.

www.alcatel-lucent.com alcatel, Lucent, alcatel-Lucent and the alcatel-Lucent logo are trademarks of alcatel-Lucent. all other trademarks are the property of their respective owners. the information presented is subject to change without notice. alcatel-Lucent assumes no responsibility for inaccuracies contained herein. Copyright © 2013 alcatel-Lucent. all rights reserved. nP2013113458en (December)

ConClusionSince its advent in communications networks, IP/MPLS technology has proven its ultimate versatility and adaptability to operators of mission-critical networks worldwide. Network operators can rest assured that both future and legacy applications such as SCADA are able to run smoothly and seamlessly in an Alcatel-Lucent IP/MPLS communications network.

aCronymsCLI Command Line Interface

frr fast reroute

HMI Human Machine Interface

IP/MPLS Internet Protocol/Multiprotocol Label Switching

Lan Local area network

MDDB Multi-drop Data Bridge

oaM operations, administration and maintenance

QoS Quality of Service

rtu remote telemetry unit

SCaDa supervisory control and data acquisition

SLa Service Level agreement

tDM time Division Multiplexing

VLL virtual leased line

VPLS Virtual private Lan service

VPn Virtual Private network

VPrn Virtual Private routed network

Documents

Transforming Mission-Critical Networks · 7705 SAR 7705 SAR R B B B 7705 SAR Figure 2. Alcatel-Lucent IP/MPLS-network. transforming Mission-Critical networks alCaTel-luCenT appliCaTion