Teleprotectionover an IP/MPLS Network - ETSI an IP/MPLS Network Technical validation ... ND of 3 ms, JBD of 4 ms – Total Latency = 2 + 3 + 4 = 9 ms. Using MPLS Network for Teleprotection

Teleprotection over an IP/MPLS

Network

Technical validationTechnical validation

ir. Lieven LEVRAU

Alcatel-Lucent IP Division

April 4th, 2011

• IP/MPLS-based Utility Networks

• Challenges for IP/MPLS in teleprotection

• Other design considerations

• Conclusion

© Alcatel-Lucent 2011 All Rights Reserved

IP/MPLS-based Utility Networks1

All Rights Reserved © Alcatel-Lucent © 2011All Rights Reserved © Alcatel-Lucent © 2011

IP/MPLS-based Utility Networks1

IP/MPLS-Based Utility Network

5620 SAM SCADAOmni PCX

NOC

Power Generation

7750 SR

Corporate HQ / NOC

IED

CollaborationServers

Internet

IP/MPLSNetwork

7750 SR

LAN

OS6855

7705 SAR

All Rights Reserved © Alcatel-Lucent © 20114 | MPLS Network for Teleprotection | Sep 2009

NOC

Transmission

7705 SAR

CollaborationTool

Servers

SubstationSubstation

Network

VideoSurveillance

7705 SAR

TPR

LANRTU

7750 SR

VideoSurveillance

TDMLANTPR

IED

7705 SAR

7750 SR

OS6855

7705 SAR

How IP/MPLS Address the challenges of the Utility Networks?

Reliability

Traffic Isolation

Multiple Legacy technologies

High Availability, Fast Reroute, end-to-end,…

IP/MPLS based - L2-VPNs, VLL, L3-VPNs, PWE

FR, ATM, TDM, Serial, analog voice, synchronous, …

Traffic isolation, Rtng protocols authentication, L2/L3

All Rights Reserved © Alcatel-Lucent © 2011

Security

Unified management

Quality of Service `

Traffic isolation, Rtng protocols authentication, L2/L3

Encryption + dynamic secure Key distribution, DoS

Guaranteed bandwidth, low latency and jitter

Enhanced QoS on different network layers

End-to-end secure management of network and

services

Challenges for IP/MPLS in 2


Challenges for IP/MPLS in

teleprotection2

Using MPLS Network for Teleprotection

∆t

kV

7750 SR

IP/MPLS

TPR

7705 SAR 7710 SR

E1/T1

Substation

TPR

7705 SAR

E&MRS-232

E1/T1

Substation

G.703G.703

Ethernet

E&MRS-232

Ethernet

MUXMUX


Teleprotection relay (TPR) signals must be transferred reliably and fast with low latency

End to end delay = telecom network latency + teleprotection equipment delay

� TDM over MPLS for legacy support (requires integration of legacy interface in MPLS node

to limit and control end to end delay)

� VPLS for IEC 61850 (requires high reliable Layer 2 transport)

Main Challenges for Teleprotection

End to End Delay

Jitter

Asymmetry

Depending on vendor equipmentMay include packetisation/depacketisation

Variation of delay in certain circumstances such as

traffic burst on network

Delay variations between transmit and receive


Resiliency

Denial of Services

Synchronization

Quality of Service

The impact of a failure in the network shouldn’t be noticed by application

TDM applications need end to end synchronization

How DoS/DDoS can affect applications

Guaranteed bandwidth, low latency and jitter

Enhanced QoS on different network layers

End-to-End TDM Transport Model (only left-to-right direction shown)

Packet SwitchedNetwork(PSN)

DS1LIU

DS1LIU

DS1 Data

Sig

PacketizationGigE GigE Data

Sig

Jitter Buffer

PacketizationNetwork

• Fixed delayPlayout

• TDM PW packets are

DS1

AccessCircuit

AccessCircuit

TDM Packets moving in this direction


• As TDM traffic from the Access Circuit (AC) is received, it is packetized and transmitted into the PSN

• Two modes of operation:

• CESoPSN (RFC5086) for structured nxDS0/64k channels

• SAToP (RFC4553) for unstructured T1

• Fixed delay

• Packet transfer delay based on link speeds and distances from end to end

• Variable delay

• the number of and type of switches

• queuing point in the switches

• QoS is key to ensure effective service delivery

• TDM PW packets are received from the PSN and stored into its associated configurable jitter buffer

• Play-out of the TDM data back into the AC when it’s at least 50% full

Using MPLS Network for Teleprotection

∆t

kV

7750 SR

IP/MPLS

TPR

7705 SAR 7750 SR

E1/T1

Substation

TPR

7705 SAR

E&MRS-232

E1/T1

Substation

G.703G.703

Ethernet

E&MRS-232

Ethernet

MUXMUX

Latency


TPR relay signals must be transferred in < ∆t

∆T = 1 cycle at 50Hz or 60Hz = 20ms or 16ms

End to end delay = telecom network latency

+ teleprotection equipment time

10 | Teleprotection over IP/MPLS - Validation - March2011

latency = packetization delay + network

delay + jitter buffer delay

The total end-to-end latency is calculated by summing the packetization delay (PD), network delay (ND) and jitter buffer delay (JBD) as shown here:

Total Latency = PD + ND + JBD

– e.g. PD of 2 ms (16 T1 frames/packet), ND of 3 ms, JBD of 4 ms

– Total Latency = 2 + 3 + 4 = 9 ms

Using MPLS Network for TeleprotectionLatency

∆t

kV

7750 SR

IP/MPLS

TPR

7705 SAR 7750 SR

E1/T1

Substation

TPR

7705 SAR

E&MRS-232

E1/T1

Substation

G.703G.703

Ethernet

E&MRS-232

Ethernet

MUXMUX


TPR relay signals must be transferred in < ∆t

∆T = 1 cycle at 50Hz or 60Hz = 20ms or 16ms

End to end delay = telecom network latency + teleprotection equipment time

Telecom network latency = packetization delay + network delay + jitter buffer delay


Calculation of Latency for TeleprotectionLatency

Latency is mainly at the edge where low speeds are present (Serial / E1 / 100 FX)

Latency in the core depends on number of nodes but mainly transmission delays

Each node adds a maximum of :

� 150µs (eqpt latency)


� 150µs (eqpt latency)

� 10µs (transmission of 1500 Bytes over a Gigabit link)

� 3µs / km : speed of light transmission over fiber

Example of a connexion between 2 TPRs :

� Over 1000km – traversing 10 nodes : 4ms


Jitter

� Jitter in normal operations may come from the equipments themselves

� This is minimal (less than 1% of router latency) and compensated by jitter buffer.

� Jitter happens in non normal conditions such as congestion.

� This can be solved by applying the correct QoS parameters to the node.

� The implementation should allow total control of the bandwidth required per application, minimizing jitter.`

All Rights Reserved © Alcatel-Lucent © 201113 | Teleprotection over IP/MPLS - Validation - March2011

TPR

Rate Limit TPR to 100Kbps

Rate Limit CCTV to 6Mbps

Ingress

Make Sure that no application can go higher than expected bandwidth. Via Rate limit per SAP

Egress

Ensure that TPR application always gets the priority

TPR gets to EF class

CCTV to BE Class

Per SAP policing Per interface queuing

H-QoS

Impact of failure

� Today’s teleprotection applications were developed towards SDH/PDH 50ms

failover time.

� The impact of a failure (node or link) can have huge impact in case the failure

exceeds the 50ms.

� MPLS FRR technology allows 50ms failover time in any failure scenario.

� These 50ms apply only in the core of MPLS network, but not in the case of the


� These 50ms apply only in the core of MPLS network, but not in the case of the

Access switch failing.


Conclusion on network failure impact

� Failover in the backbone can be limited to 50ms with FRR

� FRR has to be implemented in the first node to minimize failure risks.

� All ALU Service routers support FRR.

� Integrating TDM in MPLS (as in 7705 SAR) brings the FRR to the application and

limits the failure risks (compared to a 2 box solution)

� Providing extended rapid failover scenarios may be key in many applications.


� Providing extended rapid failover scenarios may be key in many applications.

� Support very rapid convergence time even in case of non direct connectivity

(BFD), for example with Microwave links

� Implementing MC-LAG, G.8032 and BGP-MH for IEC 61850 greatly enhances

total availability of the applications.

3


Other design considerations for Teleprotection


3

Packet flow asymmetry

LSPs are unidirectional

Topology changes or mis-configuration may result in different path being used in both directions.

Only trouble shooting tools may discover that.

5620 CPAM allows this detection and raises alarms.Bypass Green and

All Rights Reserved © Alcatel-Lucent © 201117 | Teleprotection over IP/MPLS - Validation - March2011

Active

Path

Bypass

Logical

Link

actual

path

Tracking of LSP status and history

Red LSP

follow

different

Path

Tracking of LSP Path for a given service

Synchronization

Information need to be delivered with time precision from an application standpoint.

� Many TDM applications require clock synchronization

(as SDH)

� Some applications require Time of Day type of

synchronization


Packet networks can deliver clocking through different techniques :


Summary4


Summary4

Conclusion

Teleprotection is the most stringent application that

can be transported over networks due to :

� Low delays requirements, very low jitter requirements

� Impact of a failure in the application

Alcatel-Lucent demonstrates that its IP/MPLS can be used as

the next generation network for such applications with :


the next generation network for such applications with :

� Native TDM integration (incl. interfaces, synchronization, …)

� Very high resistance to potential failures in network

� Several management tools to anticipate, control and trouble

shoot network


IP/MPLS is the foundation to prepare the Smart Grid data explosion, and manage the transition of existing mission critical applications

www.alcatel-lucent.comwww.alcatel-lucent.com

Documents

Teleprotectionover an IP/MPLS Network - ETSI an IP/MPLS Network Technical validation ... ND of 3 ms, JBD of 4 ms – Total Latency = 2 + 3 + 4 = 9 ms. Using MPLS Network for Teleprotection