1

Challenges in the Migration to Packet Switched Networks for Teleprotection Service of Power Transmission Lines

L. H. M. LEITE* R. A. FERNANDES L. F. F. ALMEIDA FITec, Brazil FITec, Brazil INATEL, Brazil

A. M. ALBERTI S. H. SOUZA R. S. J. SANTOS INATEL, Brazil CEMIG, Brazil CEMIG, Brazil

SUMMARY

This paper aims to present challenges in the migration of TDM (Time Division

Multiplexing) network to Packet Switched network, based on IP-MPLS, and the results of data

communication network performance based on packet-switched networks, applicable to the

teleprotection service over high-voltage power lines. The tests were performed in the laboratory

of CEMIG (Companhia Energética de Minas Gerais), an energy distribution company located

in the southeast of Brazil with over 8 million consumers.

The tests comprised practical implementation of network topologies in different

scenarios and showed the potential of statistical networks to support the teleprotection service

in terms of its performance related to the technical requirements such as data rate, response time

and transmission errors.

KEYWORDS

Operational Communication Networks; Mission-Critical Services; Teleprotection with MPLS

ethernet communications; Statistical Networks

Paris 2020

D2-301

2

INTRODUCTION

Historically, the communication networks used to support mission-critical services in

power systems, such as teleprotection, self-healing, communications with reclosers, power

plants and substations, communication with operation centers and others, are based on

telecommunication networks based on TDM (Time Division Multiplexing). This scenario is

justified by several factors, including low latency and high availability of communication

networks, in addition to the dedicated bandwidth configuration applied to network channel, that

is, there is no channel sharing between several clients or services. These types of networks have

been used for a long time in several applications, such as voice, video and data in most countries

and, therefore, are reliable and easy to operate and maintain by telecommunications networks

operators.

Although TDM networks are well accepted from an operational standpoint, the need for

evolution is necessary due to several advantages when compared with other systems, in

particular to packet-switched networks. The development of telecommunication networks has

as premise to improve the cost-benefit of the data transport networks and increase their

transmission efficiency and safety to match the demands of critical services. The emergence

and growth of packet-switched networks, also called statistical networks, driven by the use of

the Internet, has provided various researches and development of communication protocols and

applications for services integration with several operational and commercial requirements.

The Internet, based on TCP-IP/ETHERNET protocols, provides the integration of

various technologies and applications used in corporate and operate networks. Unlike TDM

networks, the statistics networks have as their main characteristic the bandwidth link sharing

between clients and services, that is, there is no dedicated circuit or reserved bandwidth for the

customer's application, generating better cost-benefit of the transmission network. Another

advantage is the increase in the guarantee of data delivery in case of failures on links or elements

of telecommunications networks.

Specifically, for mission-critical services, such as teleprotection, MPLS (MultiProtocol

Label Switching), MPLS-TP (MultiProtocol Label Switching-Transport Profile) and MPLS-TE

(Multiprotocol Label Switching-Traffic Engineering) are adequate to meet the operational and

technical requirements of transport networks.

This paper aims to present the results of data communication network performance

based on packet-switched networks, applicable to the teleprotection service of the transmission

energy lines (≥ 230 kV) of CEMIG (Companhia Energética de Minas Gerais) from an practical

implementation of network topologies in different scenarios.

1. TELEPROTECTION FUNCTION

Teleprotection systems, in transmission or distribution power systems, use

communication channels to establish a link between protection relays in the substations of line

terminals. In case of faults in power lines or equipment failure, the equipment protection with

an available and reliable communication system offers the possibility to isolate damaged

sections from the entire network.

The teleprotection function, which converts the signals and protection messages into

signals and messages compatible with communication channels and vice versa, can be

performed by the protection relays themselves or dedicated equipment known as teleprotection

equipment. The protection commands of the teleprotection system can be sent to a remote

substation through the telecommunications network according to the teleprotection scheme.

The Figure 1 illustrates the architecture of a protection link between the substations A and B.

3

Figure 1 - General architecture for teleprotection of transmission network

The good performance of the protection system is achieved when the requirements of

security, reliability and speed of the teleprotection system are reached. So, each teleprotection

scheme has different requirements on the communication channel to ensure its proper operation

and to meet the desired performance of the protection system.

1.1 Teleprotection Requirements

The main teleprotection requirements are summarized in Table 1.

Table 1 - Teleprotection requirements between substations

Requirements Values Notes

Total time for fault

extinction < 100 ms

Includes means of communication,

teleprotection equipment and electrical

protection. [5]

Latency/Teleprotection < 10 ms At communication channel. [9]

Availability of

telecommunications

network

99,98% per year

High

Service class A

Available Time – AT [5]

Telecommunication

channel

Distinct and

redundant

Distinct main and alternated route and with

identical equipment per route, when compared to

channel A and B. [5]

Sharing the data channel Not allowed Dedicated to each teleprotection circuit. [5]

4

Packet error rate

→ Minimum BER:

10-3 bps 1

→ 0 2

→ Minimum BER:

10-9 bps 3

1 10 consecutive second period [14] 2 Measured for fifteen (15) minutes for

transmission rates equal to or greater than 64

Kbps in at least one measurement out of three)

[5] 3 C37.94 (applicable to fiber optics)

Jitter (UIpp) – Unit

Interval – peak to peak

amplitude

64kbps (0,25 UIpp) 1

64kbps (0,05 UIpp) 2

2Mbps (1,5) UIpp) 3

2Mbps (1,5) UIpp) 4

(20 a 20 kHz) 1 ; (3 to 20 kHz) 2 ; (20 a 100 kHz) 3 ; (18 a 100kHz) 4 Measurement bandwidth, –3

dB frequencies (Hz)

64 kbit/s: 1 UI = 15.6 µs

2048 kbit/s: 1 UI = 488 ns [13]

Channel Symmetry < 4 ms Presented at IEC 60834-1[9]

Reliability High* Probability of Missing Command (Pmc) [9]

Security High* Probability of Unwanted Command (Puc) [9]

Electromagnetic

Interference (EMI) Yes

Mainly relevant for telecommunications

equipment used in transmission power systems.

* Requirements compared to telecommunications services such as monitoring, control and telephony.

1.2 Matrix of the Main Teleprotection Technologies

The Table 2 summarizes the technologies and interfaces of telecommunications

networks used in teleprotection, as well as some advantages and disadvantages of using them.

Table 2: Summary of technologies applied to teleprotection

Technology Interfaces

Teleprotection Main advantages Main disadvantages

PLC

G.703; G.703

codirectional;

RS-422-V.11;

(64k;2Mbps)

X.21/X.24; RS-

232.

- Uses the own transmission

lines as network / access;

- Mature technology;

- Long distance covered

without repeaters;

- Normally the shortest

communication path between

substations;

- Little risk of unwanted re-

routing, switching or

tampering.

- Signal-to-noise ratio and line

parameters altered adverse conditions

and when transmission line is under

fault;

- Subjected to several interferences;

- Limited data transmission capacity;

- Greater delay compared to

technologies that use fiber optics;

- Limited frequency band available,

limiting the number of PLC links that

can work in a given network (frequency

congestion);

- Not applicable for current differential

protection.

TDM

(Rádios /

Muxes/PDH

/SDH)

Optical fiber;

DWDM; OPGW;

G.703/ G704

(Nx64k;2Mbps);

G.703

codirectional

(64kbps);

V.11/X.21/X.24.

- Simplicity;

- Circuit protection;

- Ease of maintenance;

- Mature tecnology;

- Dedicated band;

- Fiber optic immune to

interference and low error rate.

- Low cost-benefit;

- Non-optimized link band;

- Restrictions for integration with new

technologies;

- Repair is difficult for OPGW (high

voltage);

- Radio: frequency bans constitute a

limited resource and may not be

5

available as desired; influence of

atmospheric conditions.

IP / Packets

MPLS-TE

MPLS-TP

(Links:

10/100Mbps

,1/10 Gbps)

Optical fiber;

DWDM; OPGW;

C37.94 optical

(Nx64k,768Kbps)

;

G.703/G704

(Nx64k);

G.703

codirectional

(64kbps);

V.11/X.21/X.24.

- Low latency;

- High cost-benefit;

- Ease of integration with new

technologies;

- Optimized link band

- Fiber optic immune to

interference and low error rate.

- Network resilience;

- Static route and circuit

protection (MPLS-TE and

MPLS-TP functionalities.)

- Deterministic behavior

(MPLS-TP)

- Depends on high QoS network for

good performance (MPLS-TE or

proprietary solutions from

manufacturers) and to guarantee

Deterministic behavior;

- Subject to asymmetric latency

(MPLS-TE);

- Incipient technology for

Teleprotection;

- Difficulty of fiber maintenance in

OPGW (high voltage).

1.3 Progress of statistical networks for mission-critical services

The providers of equipment and solutions based on statistical networks are testing the

MPLS and MPLS-TP networks, demonstrating advantages and disadvantages of these

technologies, as a network backbone for mission-critical services. At first, a slower migration

is perceived from a hybrid solution of the TDM and packet-switched networks. The advance of

statistical networks is a real trend in the corporate environment, but it is still an incipient

solution in power utilities for mission critical services.

The evolution of the communication networks directly involves protocols of statistical

networks, based on packet switching, with implementations of functionalities to supply mission

critical application requirements, mainly with the implementation of MPLS-TP and MPLS-TE

protocols. Recently tests reported by international benchmarking show the MPLS-TP and

MPLS-TE protocols present greater advantages over MPLS for Teleprotection service.

The next sections show the tests results involving telecommunications network and

teleprotection equipment from two different manufacturers, according to the architecture

presented in Figure 2. The tests covered different topologies (ring and radial) with up to six

hops, simulation of different types of teleprotection commands and simulation of traffic

insertion to cause congestion in the network to test the performance in terms of data rate,

response time and transmission errors.

6

Figure 2 - Teleprotection architecture based on IP/MPLS packet network

2. STATISTICAL NETWORK TESTING FOR TELEPROTECTION SERVICE

To test and evaluate the use of statistical networks for teleprotection services, a

laboratory test setup was installed at the CEMIG company. Solutions offered by two IP network

vendors were evaluated, which will be referenced in this work by vendors 1 and 2, respectively.

The statistical network solution from both suppliers is based on a combination of

features of MPLS-TE and MPLS-TP protocols.

In the following topics, the evaluated scenarios and the results obtained will be

described. The physical availability of equipment in the laboratory for both suppliers can be

seen in Figure 3.

Figure 3 - Test scenarios with vendors 1 and 2

High Voltage LineSubstation A Substation B

MPLS, MPLS-TE, MPLS-TP

Router

TeleprotectionEquip./Function

ProtectionEquipment

TeleprotectionEquip./Function

ProtectionEquipment

Router

Interface Type: E1/C37.94/G703 Co-dir

7

2.1 Test Setup

A common test book was designed, so the tests performed on both solutions

presented the same procedures. Both solutions were evaluated using communication

through G.703 Codir 64 Kbps and G.703 E1 interfaces between teleprotection

equipment and routers. For the implementation of this test setup, a teleprotection

command generator, two DIP 5000 teleprotection equipment and a data network

composed of six routers from each supplier were employed. Teleprotection equipment

is responsible for emulating the commands triggered by the protection relays.

The generation of traffic on the network was performed using the JPerf software

and the TSW900ETH equipment from the WISE company. The generators were

connected to the edge routers 1 and 6, through ethernet ports, with 1Gbps traffic. The

primary path was established through a direct connection between routers 1 and 6, and

the alternative path was established through the 6 routers that make up the topology,

aiming at simulating a longer path and, consequently, with more hops, to evaluate

changes in the transmission time (latency) of the teleprotection commands.

The topologies used for suppliers 1 and 2 can be seen in Figure 4 and Figure 5,

respectively. The topologies of both suppliers are similar, except for the implementation

of interfaces between routers and DIP5000 teleprotection equipment, due to the

unavailability in Brazil of some interfaces directly on the routers.

Figure 4 - Test setup employed for vendor 1

To carry out the tests with the G.703 E1 interface at vendor 1, it was necessary

to implement the V.35 / G.703 E1 interface converter model DM704C from the

manufacturer Datacom from Brazil.

Router-03 Router-04

Router-02

Router-01 Router-06

Router-05

Primary Path

Alternative Path

WISETSW900ETH

WISETSW900ETH

DIP 5000 A

DIP 5000 B

Clock

Teleprotection Command Generator

G.703 Codir64 kbps

G.703 2Mbps G.703 2MbpsG.703 Codir

64 kbpsDM 704C DM 704C

8

Figure 5 - Test setup employed for vendor 2

In vendor 2, to perform the tests with the G.703 Codir 64 Kbps interface, it was

necessary to convert the g.703 64kbps interface to G.703 E1 through a TDM Mux model

AMDII from the manufacturer Digitel.

2.2 Tests Performed

The following tests were performed during the experiment:

• Measurement of the operating time of teleprotection equipment and

telecommunications network (IP network) performing without traffic

insertion in the routers.

• Measurement of the operating time of teleprotection equipment and

telecommunications network (IP network) performing with traffic

insertion in the routers

• Failure of the main channel and recovery of the network through the

redundant channel.

• Channel latency asymmetry.

• Analysis of performance requirements such as jitter, loss of frames and

bit error rate (BER) in the network.

For the scenarios described above, the teleprotection command generator was

configured to emulate the distance protection function (function 21), by sending 200

samples from the DIP 5000 equipment (tip A) to the DIP 5000 equipment (tip B). Each

sample contains 4 commands, 2 DTT (Direct Transfer Trip) and 2 DCB (Directional

Comparison Blocking) commands. The commands of the permissive type POTT

(Permissive Overreaching Transfer trip) presented a similar behavior to the DTT, so

Router-03 Router-04

Router-02

Router-01 Router-06

Router-05

Primary Path

Alternative Path

WISETSW900ETH

WISETSW900ETH

DIP 5000 A

DIP 5000 B

Clock

Teleprotection Command Generator

G.703 2Mbps

G.703 Codir64 kbps

G.703 Codir64 kbps

G.703 2MbpsMX AMDII MX AMDII

9

they were not considered in the results. The samples were selected for a period of 100

ms with an interval of 1s between the sending of each sample.

Briefly, the configuration of the telecommunications network was performed

through the insertion of the IP / MPLS protocol, with the implementation of proprietary

functionalities of the vendors, aiming to meet the technical requirements of the

teleprotection service, and the OSPF routing protocol, among other configurations in

the routers. The network synchronism was established through an external clock from

the SDH. Tests were performed with an internal clock, SyncE and IEEE 1588V2 PTP

standards on the routers.

To identify the latency of the teleprotection commands in the

telecommunications network, without considering the time spent for sending and

processing the commands by the DIP 5000 teleprotection equipment, a test with the

back to back topology was performed as shown in Figure 6. In this scenario, the DIP

5000 A equipment was directly connected to the DIP 5000 B equipment. From the

comparison between the delay values found for the complete system (teleprotection

equipment and IP network as shown in Figure 4 and Figure 5) and the values found for

this test, it was possible to obtain latency in the telecommunications network, which is

the main purpose of the tests.

Figure 6 - Back-to-Back topology

The channel failure test was performed by dropping the main link when the

teleprotection command generator reached the fiftieth sample. After this procedure, the

traffic was automatically directed to the alternative path, allowing the evaluation of the

system when submitted to a path with more hops (6 hops).

The asymmetry latency was obtained in two stages. Initially, the delay of

teleprotection commands was evaluated and sent from teleprotection equipment A to

teleprotection equipment B and then from equipment B to A. In both scenarios, traffic

was sent by the same LSP. Through a simple subtraction, it was possible to obtain the

asymmetric latency value of the channel.

The tests described above allowed to verify the behavior of the teleprotection

system when operating in a statistical network with and without the insertion of external

data traffic, the impact caused by the switching between primary and secondary paths

and the sensitivity to channel asymmetry.

2.3 Results

In this section, the test results of each of the evaluated scenarios are presented.

Initially, the best results from the complete scenario of IP networks are addressed

without the insertion of external traffic. In this setup, considering the solutions with

10

interface G.703 Codir 64kbps and G.703 E1-2Mbps, the results for the commands DTT

(A and B) and DCB (C and D) can be seen in Figure 7.

Figure 7 - Network latency (teleprotection equipament + IP network)

In the complete scenario (teleprotection equipment + IP network), using the

64kbps interface, the DTT teleprotection commands had an average latency of 9.96 ms

for A and an average of 9.95 ms for B. The DCB commands had an average latency of

7.52ms for C and an average of 7.51ms for D. For the 2Mbps interface, the DTT

commands had an average latency of 4.59 ms for A and an average of 4.57 ms for B.

The DCB commands had an average latency of 4.33 ms for C and an average of 4.29

ms for D.

The values for the operating time of the DIP 5000 teleprotection equipment can

be seen in Figure 8.

Figure 8 - Back-to-Back Teleprotection equipment latency Test

In the scenario with only the DIP 5000 teleprotection equipment, using the

64kbps interface, the DTT teleprotection commands had an average latency of 7.45 ms

for A and an average of 7.44 ms for B. The DCB commands had an average latency of

5 ms for C and an average of 4.99 ms for D. For the 2Mbps interface, DTT commands

had an average latency of 3.11 ms for A and an average of 3.1 ms for B. The DCB

commands had an average latency of 2.83 ms for C and an average of 2.81 ms for D.

11

To calculate the latency of the IP network, the total latency shown in Figure 7

was subtracted from the latency of the back to back teleprotection equipment, shown in

Figure 8. Therefore, the average latency of the IP network is 2.5 ms with a 64kbps

interface and 1.48 ms with an E1 interface (2Mbps).

Table 3 summarizes the tests performed and their respective results. The tested

solutions from both vendors used a combination of features of MPLS-TE and MPLS-

TP protocols.

Table 3 – Tests results for both vendors

Test Scenario Vendor

Latency for

DTT

Command

(ms)

Latency for

DCB

Command

(ms)

Note

Test - 01

IP network with

64kbps interface

without traffic.

A 9.96 7.52 Vendor B uses a

mux to convert

the g.703 64

kbps interface to

G.703 E1 (~ 1

ms delay) B 10.96 8.5

Test - 02 IP network with

64kbps interface

and traffic

A 9.92 7.48 Vendor B uses a

mux to convert

the g.703 64

kbps interface to

G.703 E1 (~ 1

ms delay) B 11.42 8.94

Test-03 IP network with 2

Mbps interface

without traffic

A 5.75 5.49 Vendor A uses a

V.35 / G.703 E1

interface

converter (~ 1

ms delay) B 4.61 4.33

Test-04 IP network with

2Mbps interface

and traffic

A 5.82 5.55 Vendor A uses a

V.35 / G.703 E1

interface

converter (~ 1

ms delay) B 4,8 4,5

Test-05 IP network and

failover on the

main link.

A ~ 0 ~ 0

B ~0 ~0

Test -06 IP network and

asymmetric latency

test

A 0.07 0.08

B 0.03 0.035

The Jitter tests, error rate and frame loss were satisfactory, as the values obtained

were close to zero, in this laboratory environment.

3. OPERATIONAL CHALLENGES FOR MIGRATION OF THE

TELEPROTECTION SERVICE FOR STATIC NETWORKS

The migration from deterministic communication networks to statistical networks,

although with significant progress in the corporate environment, presents some challenges for

12

the operating environment, especially those related to mission critical services. The following

are the main points of attention to be considered by energy companies for the adoption of

statistical networks in data traffic related to the teleprotection function of power transmission

lines.

A. Reproduce the tests performed in the field, in order to assess the impact of external

factors that cannot be performed in the laboratory;

B. Elaboration of a logical project, creation and management of circuits configured

with bidirectional static routes and their due protection, aiming at a “deterministic

behavior” in the statistical network;

C. Interoperability with existing network management systems - OAM (Operation

Administration Management) at NOC (Network Operations Center), including IT

(information technology) system legacy functionality and alarm collection systems;

D. Evolve to systems where routers are directly connected to protection relays,

avoiding the processing and data sending latency inserted in the system by

teleprotection equipment

E. Integration with other mission critical functions, corporate data network functions

and other applications in Ethernet and IP protocols;

F. Training of teams focused on network protocols such as IP / MPLS / MPLS-TE /

MPLS-TP and other resources such as management testing functionality;

G. Promote alignment with the modernization of the protection systems for

transmission lines and substations of energy system (Smart Grid).

4. CONCLUSION

As shown in the test results (Table 3), the technical requirements required by the

telecommunications network for the teleprotection service, such as latency, jitter, asymmetric

latency, loss of frames, failover and error rate, were presented according to normative

parameters, demonstrating the potential of IP for the teleprotection service, with performance

characteristics similar to traditional TDM networks. The latency of the IP network was 2.5 ms

for the 64kbps interface and 1.48 ms for the 2Mbps interface, without any other external traffic

on the network altering the performance of the teleprotection command. It is concluded that it

is already possible to compare the performance between the statistical networks and TDM for

the teleprotection service, and tests should be carried out in real production environments and

assess the particularities of each energy infrastructure and the protection schemes used. It is

important to highlight other advantages of the statistical network, such as lower CAPEX,

OPEX, less obsolescence and easier integration of other convergent operating and corporate

services in the same network.

ACKNOWLEDGMENT

The authors thank the Research and Development Program of the Brazilian electricity sector

regulated by ANEEL and CEMIG - Companhia Energética de Minas Gerais, for the financial

13

support to the project. This work is related to the project " P&D – D640 – Modelo de Referência

para a Rede Operativa de Dados da CEMIG”.

BIBLIOGRAPHY

[1] DOE - Department of Energy – DOE Global Energy Storage Database provides free, up-

to-date information on grid-connected energy storage projects and relevant state and

federal policies. US, Abril, 2019. [Online]. Disponível em:

http://www.energystorageexchange.org

[2] ANEEL - CHAMADA NO 021/2016 – Projeto Estratégico: Arranjos Técnicos e

Comerciais para a Inserção de Sistemas de Armazenamento de Energia no Setor Elétrico

Brasileiro. Brasília, DF, Julho, 2016

[3] CIGRÉ, Protection using Telecomunications, Cigré Joint Working Group 44/35.11 Final

Report, 2000

[4] CIGRÉ, Telecommunication Service Provisioning and Delivery in the Electrical Power,

Working Group D2.26, 2011

[5] System National Operator in Brazil - ONS (Operador Nacional do Sistema Elétrico) -

Módulo 2 dos Procedimentos de Rede, 2016,<www.ons.org.br>

[6] Tan, V., & Cole, J. (2018). Teleprotection over Multiprotocol Label Switching (MPLS):

Experiences from an Australian Electric Power Utility. CIGRE - Session Papers &

Proceedings

[7] IEEE Std C37.113-2015 - IEEE Guide for Protective Relay Applications to Transmission

Lines

[8] IEEE Std C37.236-2013 - IEEE Guide for Power System Protective Relay Applications

Over Digital Communication Channels

[9] IEC 60870-5 - Telecontrol equipment and systems - Part 5: Transmission protocols

[10] IEC 60834-1 – Teleprotection Equipment of Power Systems - Performance and testing.

Part 1: Command systems, 1999

[11] IEC 61850 - Communication protocols for inteligente electronic devices at electrical

substations, 2013

[12] IEEE Std C37.94-2017 - IEEE Standard for N times 64 kbps Optical Fiber Interfaces

between Teleprotection and Multiplexer Equipment

[13] RFC 7625, “Architecture of an IP/MPLS Network with hardened Pipes”, 2015,

http://www.rfc-editor.org/info/rfc7625

[14] ITU-T G.823 The control of jitter and wander within digital networks which are based on

the 2048 kbit/s hierarchy

[15] ITU-T G.821/ G.826 End-to-end error performance parameters and objectives for

international, constant bit-rate digital paths and connections