Scalable Communication Transport Solutions Over Optical Networks

8/19/2019 Scalable Communication Transport Solutions Over Optical Networks

1/95

618

Scalable Communication TransportSolutions Over Optical Networks

Working GroupD2.35

May 2015


2/95

1 | P a g e

SCALABLE COMMUNICATIONTRANSPORT SOLUTIONSOVER OPTICAL NETWORKS

Type WG D2.35

Members

J.SANDHAMConvenor (IE), M.SCHATZSecretary (CH), M.ACACIA (BE), F.CASTRO (ES),

J.DARNE (ES), J.FEIJOO MARTINEZ (ES), M.FLOHIL (NL), R.IRONS-MCLEAN (UK),

M.JANSSEN (NL), A.MOAINI (FR), T.V.PEDERSEN (NO), H.RIIS (DK), A.RUNESSON (SE),

S.TANNER (FI)

Corresponding Members

C.DI PALMA (AR), C.EVERITT (AU), J.MENDES (PT), P.SCHWYTER (CH), M.SEEWALD (DE),

A.SILFVERBERG (FI), V.TAN (AU)

Copyright © 2015

“Ownership of a CIGRE publication, whether in paper form or on electronic support only infersright of use for personal purposes. Other uses are prohibited, except if explicitly agreed byCIGRE, total or partial reproduction of the publication for use other than personal and transferto a third party; hence circulation on any intranet or other company network is forbidden”.

Disclaimer notice

“CIGRE gives no warranty or assurance about the contents of this publication, nor does itaccept any responsibility, as to the accuracy or exhaustiveness of the information. All implied

warranties and conditions are excluded to the maximum extent permitted by law”.

ISBN :978-2-85873-320-0


3/95

3 | P a g e

SCALABLE COMMUNICATIONTRANSPORT SOLUTIONS OVEROPTICAL NETWORKS Table of Contents

1 INTRODUCTION ................................................................................................................ 8

2 REQUIREMENTS FROM USERS .......................................................................................... 9 2.1 Survey Results .................................................................................................................................... 9

2.1.1 Substation applications using IP ............................................................................................ 11 2.1.2 Main operational challenges using IP as communication transport solutions ................ 12 2.1.3 Main psychological barriers with using IP protocols in Substation applications .......... 13 2.1.4 Are all applications compatible with IP at the moment? .................................................. 14 2.1.5 Prediction to migration of all operational communications into IP ................................. 14 2.1.6 How to deal with legacy protocols and equipment? ........................................................ 15 2.1.7 Requisites and concerns for the telecommunication network ........................................... 16 2.1.8 Most promising technologies to provide communications in the access network? ........ 17 2.1.9 Most promising technologies to provide communications in the core network?............ 18 2.1.10 Should the IP network be for operational service or also for corporate services? . 19

2.1.11 Survey Conclusions ................................................................................................................ 19

2.2 List of Services ................................................................................................................................ 20

3 GENERAL CONSIDERATIONS ......................................................................................... 22 3.1 Why are Utilities migrating? ........................................................................................................ 22 3.2 Migration considerations ............................................................................................................... 24

3.2.1 Timing ......................................................................................................................................... 26 3.2.4 Communication Requirements Decomposition ..................................................................... 26 3.2.5 Security ...................................................................................................................................... 27 3.2.6 Communication Network Management ................................................................................ 28 3.2.7 IPv4 and IPv6 co-existence .................................................................................................... 29

3.2.8 Segmentation and Virtualization .......................................................................................... 29 3.2.9 Scalability ................................................................................................................................. 30 3.2.10 Availability ............................................................................................................................. 30

4 ASSESMENT OF TECHNOLOGIES .................................................................................... 31 4.1 Synchronous Digital Hierarchy (SDH) ......................................................................................... 32

Life Cycle ............................................................................................................................................. 32 Scalability ............................................................................................................................................ 32 Capability to support EPU Services ................................................................................................ 33 Ease of Implementation and Operation ......................................................................................... 33 Interoperability (other services/technologies)............................................................................... 33 Redundancy, Availability and Reliability ...................................................................................... 34 Quality of Service (QoS)................................................................................................................... 35 Network Management System and Overall Manageability (NMS) ......................................... 35


4/95

4 | P a g e

4.2 Dense Wavelength Division Multiplexing & Coarse Wavelength Division Multiplexing . 36 Life Cycle ............................................................................................................................................. 36 Scalability ............................................................................................................................................ 36 Capability to support EPU Services ................................................................................................ 39 Ease of Implementation and Operation ......................................................................................... 39 Interoperability (other services/technologies)............................................................................... 40 Redundancy, Availability and Reliability ...................................................................................... 41 Quality of Service (QoS)................................................................................................................... 41 Network Management System and Overall Manageability (NMS) ......................................... 42 CWDM versus DWDM ....................................................................................................................... 42

4.3 Optical Transport Network (OTN) .............................................................................................. 43 Life Cycle ............................................................................................................................................. 43 Scalability ............................................................................................................................................ 43 Capability to support EPU Services ................................................................................................ 45 Ease of Implementation and Operation ......................................................................................... 46 Interoperability (other services/technologies)............................................................................... 46

Redundancy, Availability and Reliability ...................................................................................... 46 Quality of Service (QoS)................................................................................................................... 46 Network Management System and Overall Manageability (NMS) ......................................... 46

4.4 Dynamic Internet protocol based Multiprotocol Label Switching (IP/MPLS) ..................... 47 Life Cycle ............................................................................................................................................. 47 Scalability ............................................................................................................................................ 47 Capability to support EPU Services (as defined in chapter 2) .................................................. 47 Ease of Implementation and Operation ......................................................................................... 48 Interoperability (other services/technologies)............................................................................... 48 Redundancy, Availability and Reliability ...................................................................................... 48

Quality of Service (QoS)................................................................................................................... 49 Network Management System and Overall Manageability (NMS) ......................................... 49 4.5 Static Multiprotocol Label Switching – Transport Profile (MPLS-TP) .................................... 50

Life Cycle ............................................................................................................................................. 50 Scalability ............................................................................................................................................ 50 Capability to support EPU Services (as defined in chapter 2) .................................................. 50 Ease of Implementation and Operation ......................................................................................... 51 Interoperability (other services/technologies)............................................................................... 51 Redundancy, Availability and Reliability ...................................................................................... 51 Quality of Service (QoS)................................................................................................................... 52 Network Management System and Overall Manageability (NMS) ......................................... 52

4.7 Provider Backbone Bridging (PBB) ............................................................................................. 54 4.8 Ethernet ............................................................................................................................................ 54

Life Cycle ............................................................................................................................................. 54 Scalability ............................................................................................................................................ 55 Capability to support EPU Services (as defined in chapter 2) .................................................. 55 Ease of Implementation and Operation ......................................................................................... 55 Interoperability (other services/technologies)............................................................................... 55 Redundancy, Availability and Reliability ...................................................................................... 55 Quality of Service (QoS)................................................................................................................... 55 Network Management System and Overall Manageability (NMS) ......................................... 55

4.9 Comparison matrix ......................................................................................................................... 56

5 CONCLUSIONS AND PROPOSAL FOR FUTURE WORK .................................................. 58


5/95

5 | P a g e

Survey ...................................................................................................................................................... 58 Services .................................................................................................................................................... 58 Technologies ........................................................................................................................................... 58 Future Works 1 ...................................................................................................................................... 60 Future Works 2 ...................................................................................................................................... 60 Future Works 3 ...................................................................................................................................... 61

ANNEX 1 SURVEY D2.35 ................................................................................................... 62

ANNEX 2 SURVEY D2.28 IP-BASED SUBSTATION APPLICATIONS.................................... 71

ANNEX 3 WGD2.35 SURVEY RESULTS............................................................................... 74 1 Substation applications using IP ...................................................................................................... 74 2 Substation applications eligible for use with IP ........................................................................... 76 3 Applications outside the substation eligible for use with IP ....................................................... 78 4 Main operational challenges using IP a communication transport solutions ........................... 78 5 Main psychological barriers with using IP protocols in Substation applications .................... 79 6 Are all applications compatible with IP at the moment? ............................................................ 80 7 Prediction to migration of all operational communications into IP ........................................... 81 8 Percentage of the existing applications using IP as communication transport solution? ...... 83 9 How to deal with legacy protocols and equipment? .................................................................. 83 10 Requisites and concerns for the telecommunication network ................................................... 84 11 Most promising technologies are to provide communications in the access network? ........ 86 12 Most promising technologies to provide communications in the core network? ................... 86 13 Should the IP network be for operational service or also be for corporate services? ...... 87 14 What type of scalability needs to be addressed? .................................................................. 88

ANNEX 4 NETWORK TIMING ............................................................................................ 90

Synchronisation Types ........................................................................................................................... 91 IRIG-B Overview .................................................................................................................................... 92 Global Positioning System (GPS) Overview .................................................................................... 92 One Pulse Per Second (1PPS) Overview .......................................................................................... 94 NTP/SNTP Overview ............................................................................................................................ 94 IEEE 1588 PTP / IEC 61588 PTP ........................................................................................................ 94

Versions of 1588 ................................................................................................................................ 95 NTP vs. SNTP vs. PTP: A Summary ...................................................................................................... 95


6/95

6 | P a g e

FiguresFigure 1 Companies represented in the survey responses ............................................................................. 10 Figure 2 Countries represented in the survey responses ................................................................................ 10 Figure 3 Trends in substation applications using IP ........................................................................................ 11 Figure 4 Trends in operational challenges ...................................................................................................... 12

Figure 5 Trends in psychological barriers ........................................................................................................ 13 Figure 6 Trends in current IP compatibility ...................................................................................................... 14 Figure 7 Trends in estimation of all operational traffic through IP ................................................................... 15 Figure 8 Trends in how to deal with legacy equipment and protocols ............................................................ 15 Figure 9 Trends in requisites and concerns .................................................................................................... 16 Figure 10 Trends in most promising access network technologies ................................................................. 17 Figure 11 Trends in most promising core network technologies ..................................................................... 18 Figure 12 Trends in network separation .......................................................................................................... 19 Figure 13 Traditional Grid Communications Architecture ................................................................................ 23 Figure 14 Tiered approach for New grid applications ...................................................................................... 23 Figure 15 Shift from TDM to packet ................................................................................................................. 24 Figure 16 Migration decision review process .................................................................................................. 25 Figure 17 Communications decomposition table for packet oriented traffic .................................................... 26 Figure 19 SDH Capacity Comparison ............................................................................................................. 32 Figure 18 SDH Add-Drop-Multiplexer .............................................................................................................. 33 Figure 20 SDH Sub-Network Connection Protection ...................................................................................... 34 Figure 22 Optical spectra in fiber ..................................................................................................................... 37

Figure 23 examples of Add-Drop-Multiplexer .................................................................................................. 37 Figure 24 Reconfigurable Optical Add Drop Multiplexer ................................................................................. 38 Figure 25 Working area for light source and receiver ..................................................................................... 38 Figure 21 WDM Multiplexing............................................................................................................................ 39 Figure 26 OTN Architecture ............................................................................................................................. 44 Figure 27 OTN Capacities ............................................................................................................................... 45 Figure 28 MPLS-TP Hierarchy ........................................................................................................................ 51 Figure 29 Protection configuration ................................................................................................................... 52

Figure 30: Substation applications using IP .................................................................................................... 75

Figure 31: Applications eligible for using IP..................................................................................................... 77 Figure 32: Operational challenges ................................................................................................................... 79 Figure 33: Psychological Barriers .................................................................................................................... 80 Figure 34: All applications IP compatible?....................................................................................................... 81 Figure 35: Predication of All operational traffic through IP .............................................................................. 82 Figure 36 Percentage of existing applications using IP ................................................................................... 83 Figure 37How to deal with legacy protocols/equipment .................................................................................. 84 Figure 38 Requisites and concerns for using only IP ...................................................................................... 85

Figure 39 Most promising access network technology .................................................................................... 86 Figure 40 Most promising core network technology ........................................................................................ 87

http://headfop023v/Sandham_J$/My%20Documents/Planning/Cigre/Versions/WG_2_35_TB_Ver_0_10l.docx%23_Toc408394154http://headfop023v/Sandham_J$/My%20Documents/Planning/Cigre/Versions/WG_2_35_TB_Ver_0_10l.docx%23_Toc408394154http://headfop023v/Sandham_J$/My%20Documents/Planning/Cigre/Versions/WG_2_35_TB_Ver_0_10l.docx%23_Toc408394158http://headfop023v/Sandham_J$/My%20Documents/Planning/Cigre/Versions/WG_2_35_TB_Ver_0_10l.docx%23_Toc408394158http://headfop023v/Sandham_J$/My%20Documents/Planning/Cigre/Versions/WG_2_35_TB_Ver_0_10l.docx%23_Toc408394158http://headfop023v/Sandham_J$/My%20Documents/Planning/Cigre/Versions/WG_2_35_TB_Ver_0_10l.docx%23_Toc408394154


7/95

7 | P a g e

Figure 41 Seperate operational services from corporate services? ................................................................ 88 Figure 42 What type of scalability to address? ................................................................................................ 89 Figure 43 Applications sensitive to time and frequency .................................................................................. 91 Figure 44 Time synchronisation application .................................................................................................... 92

TablesTable 1: (Future) IP usage ............................................................................................................................... 78


8/95

8 | P a g e

1 INTRODUCTION

The introduction of smart applications in the Electrical Power Utility (EPU) and consequentdispersed intelligence result in a tremendous growth of information exchange across the powersystem. This implies, in many cases, a change of scale in the requirements of thetelecommunication infrastructure and often the deployment of a core data transport network.This may be implemented through a number of different technologies and architectures.

The present network of most power utilities is extensively composed of TDM (e.g. PDH/SDH)technology. Packet communication and in particular Ethernet connections are growing very fastand may bring the necessity to adapt and /or replace network technologies.This Technical Brochure aims to identify and analyze solutions and migration plans in the light ofdata network technology evolutions, new application requirements and EPU’s capability tomaintain the system’s operation.

During 2009 and 2010, Cigré working group D2.28 conducted a survey amongst Cigré membersto identify the current and expected future use of IP networks within Electric Power Utilities. Forthis Technical Brochure a follow-up survey was conducted to identify trends in the use of networksand network technologies

This Technical Brochure contains: The results of the follow-up survey held for this Technical Brochure An analysis of trends in the use of networks and network technologies based on a

comparison of the survey held for working group D2.28 and the follow-up survey heldfor this Technical Brochure

General considerations for replacing, refurbishing or extending networks An assessment of available physical technologies and transport protocols Recommendations for future works


9/95

9 | P a g e

2 REQUIREMENTS FROM USERS

To make sure the contents of this Technical Brochure are relevant Working Group D2.35 startedby identifying what kind of information would be required by the intended audience. To identify the userrequirements a survey was held amongst Cigré members. Working Group D2.28 held a similar survey afew years before so the results of the new survey can be compared to those results to identify trends.

2.1 Survey ResultsThis next section describes the trends that can be identified by comparing the results from the survey heldby working group D2.28 with the survey held by working group D2.35. The complete results from thesurvey held by working group D2.35 can be found in Annex 3

One of the goals of this survey was to try to identify trends in the adoption of IP networks. To that extent,some of the questions in the survey held in 2009-2010 by D2.28 have been re-issued in the survey thatwas held for this Technical Brochure.

This section provides an indication of the trends between the D2.28 survey held in 2009-2010 and theD2.35 survey held in 2012. When comparing the results of the survey held for D2.28 and the surveyheld for this technical brochure, keep in mind that:

- The survey that was held for working group D2.28, while similar, is not exactly the same as thesurvey held for this Technical Brochure. For a detailed comparison both surveys can be found asan Annex to this TB.

- The survey for D2.35 was held under all Cigré members, not just the participants in the D2.28survey. This means that there may be differences between the results of the two different surveysdue to differing companies participating in the surveys.

Only questions that were similar or exactly the same between the two surveys are compared below.In total, 83 filled out surveys were received from 29 different countries. Figure 1shows which companiesfrom which countries have responded.

ANSWERS FROM:TRANELSA ArgentinaENERGINET.DK DenmarkEGAT ThailandITAIPU BINACIONAL BrazilScottish Power United KingdomREN PortugalSEAS-NVE DenmarkAXPO AG SwitzerlandHUAWEI BelgiumINEXUS United KingdomABB Switzerland SwitzerlandAUSGRID AustraliaPOWERLINK AustraliaESSENTIAL ENERGY AustraliaETSA Utilities AustraliaTRANSEND AustraliaELECTRANET AustraliaAURORA ENERGY AustraliaCOMMTEL AustraliaENDAVOR AustraliaENERGEX AustraliaERGON AustraliaFRESNEL NETWORKS AustraliaSNOWY HYDRO AustraliaSP AUSNET AustraliaTRANSGRID AustraliaTRANSPOWER NZ New ZealandGDH AustraliaWESTERN POWER AustraliaPOWER COR AustraliaELECTROBRAS BrazilPower Grid Corperation of Bangladesh BangladeshBhutan Power Corporation Limited IndiaSikkim power IndiaGETCO IndiaMSETCL IndiaPTCUL IndiaTSECL IndiaNokia Siemens Networks FinlandNokia Siemens Networks (2) FinlandESKOM South AfricaDNV KEMA The Netherlands

ANSWERS FROM:IBERDROLA SpainFINGRID FinlandISREAL ELECTRIC CORP IsraelRUGGEDCOM GermanySTATKRAFT NorwaySTATNETT NorwaySO-UPS RussiaELECTRIC RESEARCH INSTITUTE MexicoNARI RELAYS ChinaRTE FranceAGDER ENERGI Norway

EGAT ThailandZTE ThailandNational Grid Saudi Aribia Suadi ArabiaHUAWEI ThailandVERBUND SwitzerlandElektroprenos BiH Bosnia and HerzegovinaElektroprivreda BiH Bosnia and HerzegovinaHidroelektrane na Trebisnjici Bosnia and Herz egovinaABB CHINA(1) ChinaABB CHINA(2) ChinaABB CHINA(3) ChinaCFE Transmission Yucatan Peninsula MexicoCFE Transmnission north MexicoCFE Transmission north-east MexicoCFE Transmission central MexicoTata Power Co. Ltd IndiaEEPCO EthiopiaSaskPower CanadaAltalink Management CanadaUnknown ChinaPSE Operator PolandHokkaido Electric Power Company JapanHokuriku Electric Power Company JapanTokyo Electric Power Company JapanChubu Electric Power Company JapanChugoku Electric Power Company JapanShikoku Electric Power Company JapanKyushu Electric Power Company JapanNEC Corporation JapanLiandon The Netherlands


10/95

10 | P a g e

Figure 1 Companies represented in the survey responses

Figure 2shows the different countries from which responses were received and, per country, thepercentage that the country contributed to the total results.

Figure 2 Countries represented in the survey responses

The vast majority of the respondents work for Transmission-(49%) and/or Distribution-(28%) SystemOperators.


11/95

11 | P a g e

2.1.1 Substation applications using IP

Figure 3 Trends in substation applications using IP

While many of the results from the two surveys are very similar, there are a few differences worthnoting. The use of Wireless LAN access has significantly increased since the last survey. There also seemsto be an increase in the use of IP for:

- Building control- Telephony system- Substation RTU to SCADA system

The comparison indicates that there is a decreasing use of IP for:

- Remote IP access to substation assets- Inter control center connections- Time synchronization

While this could mean that, since 2010, IP communication for these applications has been replaced bynon-IP communication, it’s more likely that the differences in these results come from the difference inrespondents.

59%

71%

34%

49% 49% 51%

64%

34%

12%

81%

66%

12%

44%

56%

19%

67%

57%

24%

52%

34%

61% 64%

31%

22%

83%

57%

7%

28%

64%

49%

Substation applications using IP:2010

2012


12/95

12 | P a g e

2.1.2 Main operational challenges using IP as a communication transport solutions overoptical network

Figure 4 Trends in operational challenges

Again, the results of the new survey are very similar to the results of the previous survey. In the surveyheld for this technical brochure, fewer of the participants indicated that they see operational challengesin:

- Ruggedness- Scope of responsibility

Compared to 2010, more participants indicated that they see Quality of Service (QoS) as anoperational challenge.

86%

66%

41% 41%46%

58%

39%

82%

58%

34%

27%

59%57%

23%

Operational challenges:

2010

2012


13/95

13 | P a g e

2.1.3 Main psychological barriers with using IP protocols in Substation applications

Figure 5 Trends in psychological barriers

The results for both surveys are very similar. The most notable difference in psychological barriers is thatthe participants in the 2012 survey seem to be more concerned about the Quality of Service (QoS) of IPProtocols

85%

47%

32%

66%

29%

10%

77%

53%

48%

57%

23%

11%

Psychological barriers:

2010

2012


14/95

14 | P a g e

2.1.4 Are all applications compatible with IP at the moment?

Figure 6 Trends in current IP compatibility

There do not seem to be notable differences between the results of the 2010 and the 2012 surveys.

2.1.5 Prediction to migration of all operational communications into IP

21%

79%

18%

76%

Yes No

All applications IP compatible?

20102012

5%

0%

13%

36% 36%

11%

2% 2%

8%

60%

25%

2%

All operational traffic through IP?

2010

2012


15/95

15 | P a g e

Figure 7 Trends in estimation of all operational traffic through IP

Compared to 2010, participants seem much more convinced that they will be using IP basedcommunication in the foreseeable future. Only 3% of the 2012 survey participants indicated that theybelieve IP networks will never host all of their operational traffic. This is a significant decrease comparedto the 11% of the 2010 survey participants.

2.1.6 How to deal with legacy protocols and equipment?

Figure 8 Trends in how to deal with legacy equipment and protocols

While the popularity for each of the different options is comparable for both surveys, participants of the2012 survey seem to be more selective.

58%

12%

64%

17%

41%

17%

40%

17%

55%

14%

43%

10%

How to deal with legacy protocols andequipment?

20102012


16/95

16 | P a g e

2.1.7 Requisites and concerns for the telecommunication network

Figure 9 Trends in requisites and concerns

The results for the different surveys are again comparable. The percentages in Figure 9 for each optionare lower in the 2012 survey then they were in the 2010 survey, creating the impression that theparticipants of the 2012 survey have fewer requisites and concerns. FIGURE 9 however does not showoption “Suitable substation applications”, because this option was only available in the 2012 survey.68% of the participants of the 2012 survey chose this option.

69% 66%

34%39%

54%63%

30%23%

Existence of physical

infrastructure(fibre, copper,wireless, etc)

Suitabletransport

technology (PDH,SDH, Ethernet,

etc.)

Suitable IPaddressing spaceand scheme (IPv4

/ IPv6)

Identification of Applications

Requisites and concerns

2010

2012


17/95

17 | P a g e

2.1.8 Most promising technologies to provide secure and reliable transport communicationsin the access network?

Figure 10 Trends in most promising access network technologies

While similar, the scope of this question differs between the two surveys. For the 2010 survey, thequestion was: “Which underlying IP technology is the most promising to provide secure reliable IPcommunications?”, while the scope for the D2.35 survey was limited to access networks. Nevertheless,MPLS and Ethernet/IP over SDH seem to be popular in both surveys.

39%

35%

14%

41%

46%

13%

MPLS (IP overfibre)

Ethernet/IPover SDH

Ethernet/IPover DWDM

Most promising secure/relaibletechnology access network

20102012


18/95

18 | P a g e

2.1.9 Most promising technologies to provide secure and reliable transport communicationsin the core network?

Figure 11 Trends in most promising core network technologies

Again, the scope of this question in the 2010 survey differed from the scope in the 2012 survey. For the2010 survey, the question was: “ Which underlying IP technology is the most promising to provide securereliable IP communications?”,

39%

35%

14%

52%

45%

51%

MPLS (IP overfibre)

Ethernet/IPover SDH

Ethernet/IPover DWDM

Most promising secure/relaibletechnology core network

2010

2012


19/95

19 | P a g e

2.1.10 Should the IP network be reserved for operational service or also be used forcorporate services?

Figure 12 Trends in network separation

There does not seem to be any significant difference between the results for the 2010 survey and theresults of the 2012 survey.

2.1.11 Survey Conclusions

Many utilities indicate they have concerns and are facing challenges(2.1.2 Main operational challengesusing IP as a communication transport solutions over optical network) in implementing and maintaining anIP only network. Still, three out of four utilities indicated that they expect to migrate all their operationaltraffic to IP within the next 10 years(2.1.5 Prediction to migration of all operational communications intoIP). Chapter 3 explains why utilities are migrating to IP despite their concerns and the challenges theyface and provides guidelines on what to consider when migrating to IP.

2%

54%58%

5%

48% 47%

One network,no separation

Separatenetwork

(physical)

Separatenetwork

(virtual/logical)

Seperate operational services fromcorporate services? 2010

2012


20/95

20 | P a g e

2.2 List of Services

This section outlines, at a functional level, the service requirements of a typical EPU. Over twentydifferent systems are identified as supporting operational infrastructure, with many of these systemscritical to the operation of the electricity network. The integration or the support for the services listedhere will have to be considered by the utilities when they migrate to new network technologies.Bandwidth provisioned on many EPU telecommunications networks fall into two categories - Operationalservices and Group/Corporate services.

Operational services may be as follows:

Teleprotection Outage Management Systems Network and Access management SCADA services

Energy Management Systems Energy metering Event recorders Switchover of services between control centers Disturbance recorders Real-time PMU Polling telemetry operational communications and Black Start telephony Operational Voice Services Maintenance & Support Private Mobile Radio General site alarms, supervision and surveillance Time distribution using IEEE 1588 – alternative to GPS. Video services Physical site security using access control mechanisms. Smart Metering communications. IEC 61850 based communications. Dynamic Line Rating Weather Monitoring & Lightening Detection

Corporate services may be as follow: Corporate IP network. Corporate fixed telephony Corporate mobile telephony Corporate voicemail Corporate Video Conferencing

In a lot of existing deployments, where adequate spare capacity exists and economically advantageous

to do so, non-operational traffic is accommodated on operational networks. Non-operational traffic isusually a secondary consideration and in general is not a determinant of network build capacity.


21/95

21 | P a g e

Many documents from various organizations (e.g. CIGRE, IEC) already include detailed descriptions ofthe communication / performance requirements for individual applications. It is not the purpose of thisdocument to re-write or repeat these requirements. The following is a (non-exhaustive) list of documentsthat include such information:

CIGRE D2.23: The use of Ethernet technology in the EPU environment

CIGRE JWG D2B5.30: Line and System Protection using Digital Circuit and PacketCommunications CIGRE JWG 34/35.11- TB192: Protection using Telecommunications IEC 61850-90-1: Use of IEC 61850 for the communication between substations IEC 61850-90-2: Use of IEC 61850 for the communication between substations and control

centers (under preparation) IEC 61850-90-5: Use of IEC 61850 to transmit synchrophasor information according to IEEE

C37.118 IEC 61850-90-12: Wide Area Engineering Guidelines (under preparation)

o 90-12 compiles a significant amount of information related to applications and

requirements. It is envisaged to become a document which covers a lot of requirementssurrounding IEC 61850

Deriving a single mechanism to accurately classify applications based on the information above is quitedifficult. For this reason the Technical Brochure categorises differing applications into groups which canbe hereafter referred to as traffic types. Traffic types can be classified into 6 areas

Samples of possible classifications are listed below:

1. Very low latency, loss intolerant, sequence and symmetry determinant traffic:over the same path

a. Differential teleprotection schemes

2. Very low latency, loss intoleranta. Distance Protection3. Very low latency, loss tolerant traffic:

a. Time distributionb. Real-time PMUc. Inter substation event distribution

4. Low latency sequence determinant traffic:a. RS485b. IEC 61870-101 – IEC 61870-103c. DNP3

5. Low latency, loss tolerant traffic

a. Voiceb. Videoc. WAMPAC

6. Latency tolerant, loss toleranta. Packet based SCADA protocolsb. File transferc. Device management


22/95

22 | P a g e

3 GENERAL CONSIDERATIONS

This section provides an overview of the requirements utilities need to consider as part of a WANmigration strategy, and highlights key services and their performance requirements. Further details of thetechnologies and services are explained in Chapter 4.

3.1 Why are Utili t ies migrating?

Electrical power grids continue to evolve to meet the challenges of standardisation and gridmodernisation in order to enhance existing use cases and applications, while simultaneously enabling newuse cases driven by the concept of the smart grid. The addition of new technologies has the potential toallow an EPU to realise a more stable, reliable, efficient and visible power grid capable of handling anytype of communication flow.

Electric utilities are among the largest users of privately owned and operated wide area networkinfrastructures. These are built on a hybrid mix of fiber optics, microwave, power line carrier, a varietyof licensed and unlicensed wireless incl. Wi-Fi, PDH/SDH, plus more recently packet-based technologies.Communication flow models have largely followed a one-way power delivery flow from generation toconsumer. As a result most communication networks consist of multiple point-to-point circuits connectingsubstations to control centers for SCADA and EMS applications, and point-to-point circuits betweensubstations for protection applications. Many are built and operated for specific applications or solutions,making it more challenging to integrate new use cases and operational processes.

The evolution of Smart Grids will result in a large number of intelligent devices being dispersed

throughout the electricity grid. New last mile architectures, protocols and technologies will be required toservice these devices, many of which will have modest bandwidth requirements, but may be located inareas which traditional networks struggle to reach. New emerging protocols, such as IEC61850, requiresub systems based on IP technologies to operate on a local, regional and national level.

Telecommunications will be an essential enabler for many Smart Grid applications. The requirements ofSmart Grid telecommunications will be met by a combination of traditional networks and mediasupplemented by emerging last mile technologies, many of which are as yet not fully matured. It isnonetheless almost certain based on the distribution of physical electrical assets that last mile technologieswill be wireless based with a greater or lesser role for transmission/distribution line communications. Cost,reliability, criticality, capability, security and performance will be the primary determinant of the

technologies selected.The introduction of packet has mostly been restricted to IT, while OT communications have largelyremained on traditional TDM networks. The industry is experiencing a steady uptake in TDM to packetmigration due to a number of reasons including end of life of many TDM solutions, emerging standardssuch as IEC 61850, interoperability and the use of IP to future-proof grid communications. Utilities arebeginning to consolidate networks on a common infrastructure as part of ongoing refresh cycles, tominimise operational expenditure, and to introduce new use cases.

Newer, advanced multipoint use cases, for example, listed below are also driving the need for morebandwidth, multipoint connectivity, forcing us away from the model depicted in FIGURE 13, towards thatdescribed in FIGURE 14. This is the case for many traffics other than SCADA also.


23/95

23 | P a g e

Wide Area Measurements Synchro-check Adaptive relaying Situational awareness State Estimation and on-line security assessment Wide Area Controls - Special Protection Schemes Wide Area Controls - Predictive Dynamic Stability Maintaining System Wide Area Controls - Under Voltage Load Shedding Wide Area Controls - Phenomenon assumption type WAMPAC

Figure 13 Traditional Grid Communications Architecture

Figure 14 Tiered approach for New grid applications

The result is that new applications are driving transport models, transport models dictate the WAN design andtechnology choices.

There are a number of drivers to consider as part of a grid WAN migration process. Economic drivers includecash optimisation and margin control, cost effectiveness in grid optimization, OPEX reduction and the opportunito influence costs of investments for the future. Technology drivers include growing obsolescence of currecommunication infrastructure due to technology constraints in current platforms and the location of equipment product life cycles, grid flexibility and scalability, and grid monitoring, control and protection. In additionincreasing regulation (e.g. EU 20/20/20 ad M/490 mandates and NERC-CIP for North America) andstandardization efforts such as IEC 61850 and IEC 62351, mean that utilities are looking at migration in newways carefully considering all implications

This drives the need for an open network architecture based on open standards, for the implementation of amultiservice network which accommodates multi-vendors, tailored to meet the requirements for acommunications services and future ones, with the highest levels of quality, reliability, scalability and costeffectiveness.

TDM migration is one of the biggest changes in the electricity network and while many newer applications anduse cases are demanding flexibility that TDM does not have, existing applications such as Teleprotection

schemes require that packet based transport mechanisms need substantial testing over time to ensure they meetthe performance requirements of legacy, traditional and newer services.


24/95

24 | P a g e

Figure 15 shows a high level overview of the state of migration. Traditionally most WANs were based on TDMtechnologies. Over recent years packet technologies have emerged supporting some operational and ITapplications. These have either been a combination of packet over TDM, or TDM over packet. There is agradual shift in the industry towards a hybrid or all packet network architecture.

Figure 15 Shift from TDM to packet

Why are we seeing this transition? The reality is that the communications network is evolving due to changincommunications requirements such as precision timing and IEC 61850, increased network functionality includenhanced QoS features and multicast support, and the addition of multiservice capabilities like workforceenablement and physical access solutions.

3.2 Migration considerations

There are a number of decision factors to consider and questions to be asked as part of a transition

New technologies require new skillsets and engineering approaches. (one of the operational challengesaccording to the survey results). Is the required skill set and mind set available at utilities for atransition?

How does the organizational structure (with OT and IT responsibilities) influence a migration process? Substation equipment refresh: What is the time line for refreshing substation equipment to a new

technology? Is this process in progress? Does existing equipment have physical interfaces available toconnect using a new technology such as Ethernet/IP?

Control Centre capability: Does the EPU management system/s support the new technology choice?What is the time line for this support? Does this require a rebuild and what is the cost for the upgrade?

Is an upgrade of existing substation equipment possible, or does it need to be replaced? Cost ofsubstation equipment replacement/upgrade: If no refresh is taking place what is the cost of upgradingequipment to enable adequate interfaces? Can this be justified and is it giving additional functionality?

What is the disruptive risk to the overall electrical network and overall electrical supply due to thistechnology change?


25/95

25 | P a g e

Additional cost to integrate legacy applications which cannot be converted to new technology: What isthe additional cost to include TDM and traditional SCADA interfaces into the a newer network? Will thinclude legacy interface support, tunneling and translation of traditional SCADA traffic? Can this cost bejustified as the additional capabilities may not be required in 5-10 years?

Cost to maintain existing TDM based infrastructure: What is the associated cost to maintain the existingTDM/SDH infrastructure to allow the migration of the legacy devices? Is this infrastructure stisupported? What are the security and availability risks in maintaining this Infrastructure?

What is the cost of not migrating? What applications and use cases cannot be deployed? What is thecost of supporting two or more infrastructures?

In addition there are a number of network influencing factors. The TDM infrastructure, the business and functionrequirements, existing and future communication requirements, and existing infrastructure. Reviewing thquestions and influencing factors following a process such as that outlined in Figure 16, and taking intoconsideration the guidelines already documented in the Technical Brochure from D2.28, the outcome will be threquired communications infrastructure.

Figure 16 Migration decision review process


26/95

26 | P a g e

An architectural approach, consistent with industry and standards organisations (e.g. NIST, IEC, EPRI, CENELnetworking companies, and also power vendors is essential to understand the interdependencies andcommunications between systems that exist, but equally important to provide a seamless platform for future usecase integration. An architectural process defines a set of artifacts required to describe a system so that it canbe reproduced and maintained over its lifespan. These artifacts provide components, structure,interdependencies, and the guidelines determining the design and evolution over time. This is important as each

EPU may have subtly or very different requirements and WAN design will only reflect the use cases andservices an EPU wishes to deploy. All of this should be captured using relevant architecture frameworks such aTOGAF or equivalent.

3.2.1 Timing

Substation automation and system control are mission-critical processes and electric power utilities musynchronize applications and use cases across large-scale distributed power grid substation networks to ensuregrid stability. Precise timing is used to improve reliability, better understand operations of the power system,predict and prevent local and system-wide faults, for testing and verification, and to reduce costs.

Refer to ANNEX 4 Network Timingfor more details.

3.2.4 Communication Requirements Decomposition

When designing any communications network it is essential to consider the various elements that make upcommunications for an application or service. The communications decomposition table below gives agood starting point for areas to consider and build into the design process.

Figure 17 Communications decomposition table for packet oriented traffic


27/95

27 | P a g e

3.2.5 Security

Electrical power grids continue to evolve to meet the requirements of standardization efforts and gridmodernization in order to enhance existing use cases [1] and applications, while simultaneously enablingnew use cases driven by the concept of the smart grid. In addition there has been a surge in the number

of non-power applications introduced across the grid, as well as links between networks inside an EPU,and links to external networks. The addition of new technologies allows an EPU to realize a more stable,reliable, efficient and visible power grid capable of handling any type of communication flow.

Through the integration of new systems, applications, and technologies to facilitate new use cases acrossnetwork infrastructure (sometimes common), anEPU’s network becomes multi-service. However with theintroduction of any new technology there is an increased risk of security threats which must also be takeninto consideration.

The highly connected nature of a smart grid has the potential to provide unauthorized users with agreater opportunity to identify and exploit vulnerabilities to attack the electricity grid. There are manyexamples of attacks on networks worldwide and constant vigilance is necessary to ensure that adequateprotective mechanisms are used to ensure the electricity grid is not compromised.

While designing and implementing operational networks it will be necessary to comprehensively addresscyber security issues. A “defense in depth” (security measures deployed at various physical andconceptual layers of a deployment) approach should be adopted throughout the deployment. Securitycapabilities must therefore be layered such that defense mechanisms have multiple points to detect andmitigate breaches for all service types that an EPU wishes to deploy. These capabilities should beintegral to all segments of the grid infrastructure and address the full set of logical functionalrequirements, including:

Physical security Identity and access control policies Hardened network devices and systems Threat defense Data protection for transmission, distribution and storage Real-time monitoring, management, and correlation

Secureness of a technology. All technologies outlined within this document possess specific attributes whichat a basic level reduce or introduce additional risks to cyber security, information separation, dataprivacy and data authenticity. All technologies should be assessed based upon these criterion toevaluate suitability and to highlight any additional activities required to ensure a secure introduction of

the technology.Cyber Security encompasses the overall security landscape from a physical level to a user andelectrical/electronic level.

Information separation is the ability for a technology to transport a specific data stream in a fashionwhich can be deemed isolated from dependencies of other data streams.

Other security considerations:.

Access Control: Authentication and authorization of all personnel management tools and physicaldevices

Authentication, Authorization and Accounting for data and devices Data Confidentiality and Privacy Securely scalable Tools: Zones, Segmentation, Logging


28/95

28 | P a g e

Tools: Tamper-resistant design, authenticity and integrity of hardware and software Integrity of Platforms and Devices: Secure devices over the entire life-cycle

3.2.6 Communication Network Management

A unified security and network management architecture will be based on a set of applications

delivering the Fault, Configuration, Accounting, Performance and Security (FCAPS) functions as definedby the OSI Network Management reference model.

A key challenge is the definition of “unified management”. It is unrealistic to expect that a single application can provide all functions and services required when managing such diversity of equipmentand communications technologies. More recently applications have been able to manage multi-vendorand multi-technology communications networks, but there is no guarantee to cover all equipment,communication types, and future requirements. It is more realistic defining a set of requirements andfeatures that enables network management and power grid applications to interoperate and exchangeinformation, such as:

Network and security management and power grid management applications must be able tocommunicate over TDM, IPv4 and IPv6 network layers over the expected lifetime of thedeployment.

Role Based Access Control for administrators and operators must be able to get user’sauthentication from corporate directory services, thus showing interaction between IT and OTnetworks.

Network and security management and power grid management solutions applicable to usecases requiring large scalability, must be able to work in a clustering and load-balancernetworking environment. It allows growing the solution over time without reconsidering thearchitecture.

Data flow and API (Application Programing Interface) must be identified, specified anddocumented for communications between OT and NMS applications.

An unified Geographic Information System (GIS) solution should be shared between applications Recommended database (DB) properties should be defined when storing all operational state,

device configuration, network event alarm, performance metric, etc. Recommended Historian properties should be defined to store data on a long period of time as

required by regulations and operations. Standard protocols, including publicvendors’ extensions, should be recommended to facilitate

management data exchanges between a device and multiple management applications. Remote Operations Management capabilities should provide utilities asset management functions

with real-time communications, improved availability of information, and enhanced visualdisplays, all of which enable the asset planners and field workers to perform their work withimproved efficiency

Benefits

A powerful Network Management System enables both condition-based and predictive maintenancestrategies so the EPU does not have to rely upon reacting to failures after they happen and schedulinginspections, which normally have to take place even if there are no equipment health issues.

In addition, asset data can be used as input for grid state determination and remote power qualitymonitoring.

These benefits also contribute to improved System Reliability


29/95

29 | P a g e

Increase Service Restoration Speed: Improvements in the speed and availability of information inthe field allows technicians to diagnose and repair assets more quickly, resulting in a more rapidrestoration of service.

Increase Reliability Metrics: Increases in the speed and efficiency of delivering service, andincreased service restoration speeds will result in higher reliability metrics.

The technologies discussed within this document will be responsible for supporting critical electrical assets,systems and services within an EPU. To ensure that adequate clarity and control of a technology,dynamic, scalable and intuitive management systems need to be put in place. Management systemsshould be able to accurately present, configure, report and alert on network state in a close to real-timefashion while ensuring compliance with change control and management methodologies.

3.2.7 IPv4 and IPv6 co-existence

Even if in future networks the core network devices may be running IPv4, the networks may need to offerIPv6 support for applications.. Co-existence of these protocols is a consideration as part of any migrationor accommodation strategy involving Ethernet/IP.

3.2.8 Segmentation and Virtualization

With the growing adoption of packet based technologies to converge all data, voice and video trafficfrom both an operational and IT perspective, there has been a growing need for logical segregationrather than physical. The IEEE and IETF have standardized Layer-2 and Layer-3 mechanisms, enablinglinks to be shared by all data types which have been in operation for many years.

A segmented or virtual network can be seen as decoupled from the physical network but it doesn’tchange its basic characteristics. It represents an abstraction that requires proper design, deployment,security and management to be successfully operated.

There is a broad range of mechanisms to achieve segmentation and virtualization, in TDM, andEthernet/IP environments. Any design should match the associated applications requirements in regardsof bandwidth, prioritization, security, latency and so on.

Example applications could be Serial SCADA over IP, Layer-2 Ethernet or Serial circuit and segregationof operational and corporate data via MPLS VRFs or a separate VC container in SDH.

Whatever the selected mechanisms, the “Virtualized Network” building blocks must be designed in

considering the following topics: Segmentation versus virtualization Separate vs “virtually” separate Network management – Network management applications must be able to retrieve data from

a ll “Virtualized Networks” Capacity estimate – when designing and operating the network, capacity estimate allows proper

selection of physical and data link layers in order to provide enough bandwidth and optimizedlatency – particularly important for Teleprotection – for all applications. Capacity estimate of thephysical infrastructure must include the network protocol overhead related to the segmentationand virtualization mechanisms. This must include the performance and scalability impacts of themechanisms on the devices at the various network places.


30/95

30 | P a g e

3.2.9 Scalability

The migration of EPU Services and applications to new communication technologies will take place over along period. While future requirements are captured where possible, even more demands and

requirements will be presented post deployment. For this reason it is necessary that all technologies areassessed based on the ability to meet:

future needs of applications future deployment models future capacities future geographical expansion

3.2.10 Availability

The availability concept is related to time. That is, availability can be defined as the time in which the

system is reliable. In Optical systems, highest availability is achieved by means of certain degree ofRedundancy. That redundancy must cover not only the electronic devices such are transmitters, receiversor amplifiers, also must cover the fibre path implementing protected routes between nodes.

Availability is also strongly influenced by the network recovery time which depends on the failuredetection time and the availability of a protection path.


31/95

31 | P a g e

4 ASSESMENT OF TECHNOLOGIES

This section discusses a number of technologies which are currently already in use in EPU networks ormay be considered for future extensions of the network:

Synchronous Digital Hierarchy (SDH) Dense Wavelength Division Multiplexing/Coarse Wavelength Division Multiplexing

(CWDM/DWDM) Optical Transport Network (OTN) Dynamic Internet Protocol Multiprotocol Label Switching (IP/MPLS) Static Multiprotocol Label Switching (MPLS-TP) Ethernet

This is not a detailed technology tutorial, but has a focus on specific characteristics which are relevant forEPU applications. Therefore for each technology the following is discussed:

Life Cycle Scalability Capability to support EPU Services Ease of Implementation & Operation Interoperability (other services/technologies) Redundancy, Availability, Reliability Quality of Service (QoS) Network Management System and Overall Manageability (NMS)


32/95

32 | P a g e

4.1 Synchronous Digital Hierarchy (SDH)

Life Cycle

SDH is a mature technology. While SDH is a Legacy technology for Public Telephony, it continuesto be the predominant technology for utilities. Thus, it is most likely that SDH prolongs its existencefor years to come. This is evident within the survey.

SDH is a standardized transmission technology that allows multiple digital bit transfers overoptical and electrical links. It was established by a set of ITU-T standards in 1989. Whenintroduced, one of the main concerns was to assure compatibility with existing PDH serviceswithout synchronization problems. This was the reason to incorporate standard PDH interfaces.Right now SDH is still the transmission infrastructure of a large number of telecommunicationoperators and utilities. The most suitable media for SDH is fiber optic, nevertheless electrical SDHline interfaces as well as microwave radio are also common.

SDH is a widely adopted technology, it is possible to build SDH equipment using generalpurpose components. This reduces the risk and the effect of life cycle decisions within thevendor/manufacturing market.

ScalabilityHuge networks have been built with SDH technology and several European Public TelecomOperators still use SDH as the backbone network. Rates of 10 Gbps (STM-64) are commonlyinstalled and 40 Gbps (STM-256) rates are beginning to appear in the market. SDH is commonlyused to build extensive wide area networks. In the following table most common SDH rates areshown. That said, SDH still provides the ability to interface and connect networks and devices atvery low data rates (64kbps) in t seemless fashion.

Figure 18 SDH Capacity Comparison

SynchronousTransport Module

Hierachy (Mbps) Interfaceelectrical/optical

STM-1 1 155,520 G.703/G.957

STM-4 4 622,020 ---/G.957

STM-16 16 2 488,320 ---/G.957

STM-64 64 9 953,280 ---/G.957


33/95

33 | P a g e

Capability to support EPU Services

SDH supports a variety of different payload types enabling transport of TDM classical servicessuch as E1 Circuits. SDH network can tunnel point to point Ethernet connections (Ethernet over SDH)and SDH equipment can provide vendor specific Ethernet capabilities - even multipointconnections as an overlay service.

A key element in SDH networks is the Add and Drop Multiplexer (ADM) which is the equipmentthat provides basic capacity for traffic insert and cross-connections.

Figure 19 SDH Add-Drop-Multiplexer

This capability allows the efficient transport and support supports of legacy and TDM servicessuch as tele control and protections.

Ease of Implementation and Operation

From an EPUperspective, SDH’s concepts are familiar and known. They are commonly regardedas “simpler” than data-packet based technologies. This is evident in the questionnaire and insection 3 skillset Moreover, with a lot of matured equipment deployed, there are a lot oftechnicians familiar with the operation and maintenance of SDH and its theoretical concepts. Fromthis point of view it is relatively simple to establish maintenance strategies due to the fact thatknow-how is widely available.

The fact that SDH is a mature technology that has been very popular ensures also that a lot ofspare parts are available.

Due to its TDM characteristics, SDH lacks modern data technologies flexibility, and trafficengineering is hard to optimize in big networks generating the fact that some traffic capacity iswasted in meshed networks.

Interoperability (other services/technologies)

In comparison with other “ newer” technologies, interoperability with legacy interfaces is a strong

West East

Tributaries

STM-1,STM-4,STM-16,STM-64 STM-1,STM-4,STM-16,STM-64

STM-1,STM-4,STM-162M,34M,45M,140M

Electricalintefaces

Opticalinterfaces


34/95

34 | P a g e

point. Especially in utilities, SDH in combination with PDH offers interfaces for legacy electricalprotection and control devices that can hardly be obtained with other technologies in a nativeway. SDH provides also a deterministic behavior, which is widely appreciated by trafficengineers.

Recent developments have been made in order to accommodate SDH with data based network,such as Ethernet over SDH. SDH works efficiently with WDM.

Redundancy, Availability and Reliability

SDH incorporates native protection and resilience mechanisms such as SNCP or MS-SPRING thatcan provide a 1+1 and N:1 traffic protection. Restoration times of around 50 ms can beachieved.

Overall concept for SDH traffic redundancy is to offer an alternative path for connecting originand destination in a telecommunication circuit. So, in a typical ring configuration two physicaldifferent paths are provided. In the following figure an example of SNCP protection isprovided.

Figure 20 SDH Sub-Network Connection Protection

* DXC in Figure 20 refers to a Digital Cross Connect

Card protection redundancy schemes are also available. This architecture ensures that in the caseof one card failing, the remaining cards can avoid service interruption and single points of failurein equipment. In general, if adequate traffic redundancy is provided, SDH networks have highlevels of availability of telecommunication services.

SubNetwork ConnectionProtection: Simultaneoustranmission by two paths

DXC

ADM

ADM

working protection


35/95

35 | P a g e

Quality of Service (QoS)

As a TDM transmission technology SDH supports only reserved bandwidth for each circuit, thismechanism insures a guaranteed QoS especially for “ missioncritical” applications, but SDH is not

optimized in case of multi-services transport. These features are inherently given within SDHstandards. The ability to classify traffic within Ethernet circuits is not common across all vendors.

Network Management System and Overall Manageability (NMS)

SDH management technology is very well understood, providing inbuilt OAM and monitoringfunctionalities. SDH management provides alarms supervision and remote provision of services.

SDH networks management platforms offer automated management functions. These platformsprovide the data that is needed in the day-to-day running of a telecom network and have thepossibility of issuing commands to the network infrastructure to activate new services and detectand correct network faults.

One of the drawbacks is that SDH management platforms are proprietary. It is possible toconnect SDH equipment from different vendors, nevertheless is neither possible to manage anetwork element from other vendor platform nor feasible to establish automatically a point topoint SDH circuit that runs on different vendors regions.


36/95

36 | P a g e

4.2 Dense Wavelength Division Multiplexing & Coarse WavelengthDivision Multiplexing (DWDM & CWDM)

Life Cycle

It comes as no surprise that traffic growth is the underlying driver for 100 Gigabit (Gbit) WDMnetwork transport. This was the case with the transition from 2.5G to 10G and from 10G to 40Gand now 100G optical channel rates.

WDM technology is a physical technology allowing transporting any protocol with variety ofbandwidth granularity till 100 Gbps (or more in next future), this capacity of WDM technologywill allow covering actual and future data stream needs.

Currently Wavelength-division multiplexing WDM transport technology is assuring carrier servicesfor both TDM technology and Ethernet technology. In essence WDM is just a support technology,an infrastructure for the other technologies.

More recently WDM is a technology that allows the coexistence of various transmission systemssharing the same fibre pair.

SFP optical interfaces are one of the major contributors to the popularity of the WDM becauseit’s intrinsic easiest way to change the transmitting wavelength simply changing the plug -inmodule. In fact actually the major part of equipment manufacturers offers a wide variety of SFPpossibilities on its optical interface.

In the last years the extensive use of the OADM devices, which allows some degree of opticalrouting, in the Metro networks increases the popularity of that technology. In fact, at the present,all the Optical Transmission Vendors have WDM devices on their portfolio.

Scalability

There are two types of WDM architectures: CWDM, and DWDM:

A CWDM system typically provides 18 Channel-Wavelength, separated by 20nm, from 1260nmto 1630nm.

DWDM is a technology used for a very high capacity links (ITU-T G.694.1). DWDM systems,mainly used by Public Telecom Operators, can multiplex from 32 to more than 100 channels inthe range of 1530 – 1624 nm.


37/95

37 | P a g e

.The number of channels, the channel selection (center frequency), and the frequency width of eachchannel, as well as the channel separation, are important parameters in WDM system design.

With a channel spacing of 1nm (or below), DWDM operation requires extremely precise opticalsources (laser temperature stabilization) resulting in significantly higher complexity and cost.

WDM technologies can accommodate any bit rate providing necessary bandwidth: SDH (STM1,STM4, …), Ethernet (1Gbit/s, 10 Gbit/s) …

WDM technology has brought more than an order-of-magnitude increase in the amount ofbandwidth that can be transported over fiber. Early implementations were point-to-point only.However, while transport networks may be thought of as the roads of the network, theintersection points are also critical.

Natively, WDM technology provides a point to point links, however it is possible to have differenttype of architectures (ring, mesh …) by mixing WDM with other protocols (Layer 2, Layer 2.5,Layer 3…).

Today’s Optical Ethernet networks are much more dynamic than in the past and demand greaterflexibility. In order to provide the necessary flexibility, Reconfigurable Optical Add DropMultiplexers (ROADMs) were developed. They allow operators to access any wavelength at anynode at any time – replicating the operational simplicity and flexibility of SONET/SDH networks

at the wavelength level.

Figure 22 examples of Add-Drop-Multiplexer

Figure 21 Optical spectra in fiber


38/95

38 | P a g e

State of the art ROADM technology and the use of tunable lasers and filters, allows operators tonot only drop any wavelength at any node and any time, but to also send any wavelengths inany direction (directionless) using any available port on the network node (colorless).

This so-called triple-A architecture (AAA – any wavelength, any node, any time) is fully flexibleand non-blocking. It requires very little technical skill from the operational staff and eliminates the

need for meticulous pre-planning.Triple-A ROADMs are the foundation of a fully automated optical network, and they allow forintelligent interworking using the two layer architecture concept, which despite other technologies,have the facility to deploy a network architecture with two stacked switching layers.

Figure 23 Reconfigurable Optical Add Drop Multiplexer

LONG DISTANCE TRANSMISSION

In the transmitting side the laser diodes have its output power limited by the signaling speed and,for the same physical reasons, in the receiver side, the quantum energy necessary to reach thedetection threshold increases following the signaling speed.

TX/Laser Diodes

TX/LED

Power[mW]

Power[dBm]

10 mW

1 mW

1 µW

1 nW

L o w e r l i m i t f

o r p h o t o d i o d

e s i n R X

L o w e r l i m i t

f o r a v a l a n c h

e p h o t o d i o d

e s i n R X

Non detectable signal in RX

+10

0- 10

- 20

- 30

- 40

- 50

- 60

Bit Rate [Bit/s]

10 M 100 M1 M 1 G 10 G 100 G

Figure 24 Working area for light source and receiver


39/95

39 | P a g e

For example for 10 Gbit/s bite rate, standard photodiode sensitivity limit is higher than -35 dBm(Red point in the figure above) because the limited energy of the incoming optical pulses.

Obviously these characteristics imply that the optical amplification becomes necessary for highspeed transmission. An alternative solution is to spread the high speed stream in some medium /low streams and use passive WDM to transmit simultaneously those streams in a single fiber pair

Capability to support EPU Services

WDM is a method of combining multiple signals on laser beams at various infrared (IR)wavelengths for transmission along fiber optic media. Each laser is modulated by an independentset of signals. Wavelength-sensitive filters are used at the receiving end to separate the signals.

This is an attractive multiplexing technique that allows:

Does not directly support termination of many legacy or EPU specificconnections (RS-232/485, E1 etc)

High bit rate without high speed electronics or modulation Mixing legacy and new network technologies on a single optical fibre Very useful for upgrades to installed fibers Suitable for Mission Critical Networks Extensive use of passive components Loss, crosstalk and non-linear effects are potential problems.

Ease of Implementation and Operation

CWDM technologies offer rapid deployments with relative ease in comparison to DWDMsolutions. However, many CWDM solutions relay upon subsequent layers of technology to enable“vision” on the network. For this reason, ongoing operations and maintenance of a CWDMsolution can become laborious as a network scales.

Some CDWM and most DWDM systems include integrated, network management programs thatare designed to work in conjunction with other operations support systems (OSSs) and arecompliant with the standards the International Telecommunication Union (ITU) has established forTelecommunications Management Network (TMN). Current systems utilize an optical service

Figure 25 WDM Multiplexing


40/95

40 | P a g e

channel that is independent of the working channels of the WDM product to create a standards – based data communications network that allows service providers to remotely monitor and controlsystem performance and use. Meeting ITU standards and utilizing a Q3 interfaces ensure thatend users retain high Operations, Administration, Maintenance, and Provisioning (OAM&P)service. Most systems use SNMP as a standard but in-depth configurations of devices may not beachievable with open standards.

Interoperability (other services/technologies)

WDM networks provide purely optical transport for different transmission technologies anddifferent transport protocols.

The WDM technology is designed to allow the coexistence of diverse technologies over the samemedia. As a carrier system many wavelengths can be implemented inside a fibre with totalindependence, carrying each one the appropriate services.This ensures protocol and format transparency in the network. A major advantage is coexistencewith several protocols (SDH, Ethernet …), for example SDH/WDM, Ethernet/WDM, IP/WDM,MPLS/WDM…


41/95

41 | P a g e

Redundancy, Availability and Reliability

In terms of Availability and Reliability, there are some differences between the WDMimplementations.

With a channel spacing of 20 nm, CWDM systems tolerate the temperature drift over the wholeindustrial temperature range of the laser emitter without leaving their allocated channel andwithout use of special cooling system.

The strict frequency stability requirement to match with the DWDM channelling is achievedkeeping the laser cavity at constant temperature and limiting the emitter power output. Thisimplies that temperature needs to be kept constant and lowers the MTBF of a system incomparison with CWDM

CWDM ruggedness, in addition to considerably lower cost, constitute significant advantages ofthis technology in the EPU communication network where wavelength multiplexing is not often usedfor attaining maximum bandwidth but rather for separating networks.

Circuit redundancy and failover are not characteristics associated with CWDM/DWDM but areoften accomplished via technologies carried on top of WDM

Quality of Service (QoS)

QoS provided by WDM technology is based on resource reservation (equivalent to Constant BitRate): Each connection has reserved bandwidth (wavelength) without any overflow mechanism.

There are a lot of discussions about the suitability of the transmission of Teleprotection signals

and / or Differential protection over non dedicated transmission systems. Both Synchronous andPacketized systems present some degree of inconvenience which are discussed in other parts ofthis Brochure.

Direct transmission systems or dedicated point to point equipment. The most suitable media formission critical communications is fibre optic. There are methods to deploy multiple services viaseparate optical channels: The use of dedicated optical fibre or the use of a dedicatedwavelength. Because optical mixer and receiver filter banks are passive elements the reliabilityof the transmission path is not affected by choosing one solution or another.

WDM technology allows a deterministic end to end delay and jitter.


42/95

42 | P a g e

Network Management System and Overall Manageability (NMS)

WDM management technology varies a large amount due to the passive nature of some solutionsin comparison to the comprehensive nature of other types of solutions. CWDM devices usuallyrely on subsequent technology layers above to verify connectivity and availability. DWDM canprovide extensive optical and circuit based managers providing granular control and vision ofprovisioned services.

CWDM versus DWDM

The election of the optical channelling method is a decision which cannot be taken in early stagesof a design. This is mainly because both C and DWDM channels may coexist over the same fibreand CWDM is recommended for Mission Critical applications because its highly reliable nature incomparison with DWDM


43/95

43 | P a g e

4.3 Optical Transport Network (OTN)

Life Cycle

OTN adoption was initially slow; the first deployments of OTN were in Japan with little interest inEurope and North American. The reason for this slow adoption is that carriers have invested hugeamounts of capital in the existing SDH/SONET and WDM networks.

Since the mid 2000s, OTN technology has been essentially proposed for point to point opticallinks requiring enhanced Forward Error Correction (FEC) capability and higher data rates.

Today, OTN technology is deployed in Telco carriers for several network topologies (ring, mesh…), with the possibility to transport transparently any client traffic offering some additionalfeatures like enhanced OAM, Redundancy & Resiliency.

OTN is supplied by carrier network vendors, mainly the same vendors as SDH. The carrier marketfor OTN is envisaged to nearly double from 2012 to 2016.

Scalability

The basic signal architecture of OTN is shown in the FIGURE 26 below:


44/95

44 | P a g e

Client Signal

Optical Channel Payload Unit(OPU-k)

Optical Channel Data Unit(ODU-k)

Optical Channel Transport Unit(OTU-k)

Optical Channel(Och)

Optical Multiplex Unit(OMU)

Optical Transport Module(OTM)

E l e c t r i c a

l D o m a i n

O p t i c a

l D o m a i n

Figure 26 OTN Architecture

The client signal is passed through two main domains: Electrical domain and optical domain. TheO