A Hybrid Communications Network

Approach for Advanced Applications on the Modern Grid

Mark A. Carella, PEManager – Distribution Control and Integration

nationalgrid

2

Private Telecom Network

Key Stats Private Network

Fiber Route Miles……... 1311

T-1 Circuits …….…….….400

Protection ……………….250

Radio Circuits ………….350

EMS Circuits …………500

Voice Circuits …………400

Total Circuits ….. ..…… 1,900

Radio Sites ……………. 81

Comm Rooms ………… 74

Alarm Status Systems ……125

Batteries & Chargers………125

NE & Upstate NY Leased Circuits

T-1's ………. 500

EMS……. 1473

GEMS ……. 243

Radio………. 124

Other……. 1311

Total Lines…... 3651

% BW Leased vs Private = 53% (approximate)

3

Private Telecom Network

More than 1300 route miles of backbone fiber

Most backbone fiber is either wholly owned by National Grid or by others with National Grid having Indefeasible

Right to Use (IRU)

Optical transport is SONET platform with DWDM on backbone

Legacy Equipment

Alcatel Lucent DMX Platform with some DDM-2000

Approximately 141 optical nodes deployed, mostly at transmission substations and Control Centers

GE JMUX in substations for protective relay applications

RFL IMUX in substations for protective relay applications

Digital cross connect systems Tellabs 532 (1/0)

Microwave Sites

Approximately 81 radio tower sites

Mostly Alcatel MDR and Aviat (Harris) microwave equipment deployed

Microwave radios use TDM platform at 155Mbs or 45Mbs

4

Electric Substations

5

Microwave / Radio Tower Sites

6

Transmission Substation

Communications

Physical Security

AMAG card readers and access control system

Video cameras

Monitoring and Control:

Energy Managements System (EMS) - EMS RTU data & Wide Area Network (WAN)

Metering – Wired and Wireless for Power Control

Gas Energy Managements System (GEMS) – RTU and Wide Area Network (WAN)

Digital Fault Recorders (DFR’s) for event analysis

System Protection:

Bulk power system protection per NPCC Criteria

Dispatch Radio

Radio communications from Dispatch and Control Centers to field operations mobile radio

Telephone

Talk lines for NPCC Black Start Procedures

E911

7

Typical Transmission Substation

SONET Ring

All fiber routes separated by 3 feet minimum

Control Center

National Grid Fiber

SONET Optical Transport

SUBSTATION B

Protective

Relay A-BProtective

Relay A-B

Protective

Relay B-A

Protective

Relay B-APrimary

SCADA

Backup

SCADA

Backup Control

Center

SUBSTATION A

Protective

Relay A-C

SUBSTATION C

Protective

Relay C-A

POTS

POTS

POTS

Distribution & Sub-Transmission

Private RF Network RI VVO

Trilliant Private 5.8GHz. Radio System - ~35 devices

4RF Private Licensed 900MHz. Point-to-Multipoint Radio

Areas in MECO / NECO regions

Upgrades to older GE MAS systems w/ ~60 remote units

New IP based 4RF units currently considered where network availability is reasonable to achieve

4RF Private Licensed 700 MHz. Point-to-Multipoint Radio System

Nantucket Island

GE MDS Private (Unlicensed) 900MHz. Point-to-Multipoint Radio System

Western New York, ~ 200 Devices

Upstate NY DA Schemes with Private Point-to-Point 900MHz. Radio Systems

UtiliNET Radios ~ 35 Devices

S&C SpeedNet Radios ~ 30 Devices

Worcester Smart Energy Solutions Demonstration

~200 “Grid” Devices on 3.65GHz. Point-to-Multipoint Private WiMAX Network

~15,000 electric meters on Private 900MHz. mesh network / cellular backhaul

8

9

Distribution & Sub-Transmission

Legacy Communications

National Grid has also remotely monitored and controlled thousands of reclosers utilizing cellular communications that

were provided in a hosted, Software as a Service (SaaS), environment.

Annual SaaS cost was ~ $400,000 for communication with 1732 reclosers at 70kb/device/month.

Analog updates once a week.

Binary updates twice a day statically, with unsolicited report-by-exception.

SaaS offerings provided to National Grid:

Cellular communications

AT&T 2G & 3G network, monitoring and maintenance

Web portal

Visibility to the recloser data, event based actions, group notifications and commissioning aid.

DNP Server

SCADA interface.

The AT&T 2G service was sunset at the end of 2016.

To maintain communications beyond 2016, all 1732 recloser locations required a site visit to upgrade the hardware and also

upgrade our SaaS environment.

10

New Hybrid Communication

Network Architecture

Due to end of 2G service, we had to make a choice:

Upgrade existing SaaS environment

Take it internal for more control and flexibility

Our service territory already has:

Existing pockets of private RF infrastructure.

Remote locations where RF network buildout is not feasible.

The concept of a hybrid connectivity model became critical for sustainability.

11

New Hybrid Communication

Network Architecture

Experience gained through our “Smart Grid” and Volt / VAr Optimization (VVO) pilots also illustrated the need for a flexible communications model.

The model must support various types of communications technologies, as application requirements can vary greatly.

Modern network architectures for system critical operations require:

Low-latency

High degree of availability, reliability, serviceability

Security

Redundancy

These systems will vary in degrees of cost and complexity.

Private IP Network Overview

12

Layer 3 MPLS virtual private network solution facilitates

secure connectivity to multiple hub locations, enabling

system redundancy, and failover capability.

13

High-Level MPLS VRF Topology

This network architecture would allow data from field devices to

integrate to both primary and secondary back-office systems (e.g.,

data concentrators, SCADA, management portal, etc.)

Cyber security was a primary focus during the entire network

design and was embedded from inception to implementation,

complete with PEN testing and validation.

New virtual routing and forwarding (VRF) solutions were identified

to establish desired connectivity to all required facilities.

High-Level MPLS VRF Topology

14

New VRF for Access

from VSTIGs to Site #1

and Site #2 for New York

Reclosers Only

VSTIGS #2VSTIGS #1

Primary Path

High Level MPLS VRF Topology

Site #2

New VRF for Access

from VSTIGs to Site #2

and Site #3 for New

England Reclosers Only

New VRF for New York

VzW Access

Site #1

National Grid

CNI

Each VzW VRF will have a

Separate IP Address Pool

New VRF for New

England VzW Access

Secondary Path

Site #3

Existing VRF Existing VRF

Data Center

Device Connectivity

Both legacy and modern control equipment deployed across the service territory, both Serial and Ethernet

connectivity was required.

Verizon 4G / LTE Cellular based multi-service connected routers were used to replace the legacy cellular based

modems.

The router’s port forwarding functionality, on-board terminal server, and firewall, was leveraged to

simultaneously accommodate both serial and Ethernet communications in a single, secure, package.

This approach results in a truly “plug and play”, IP addressable, solution for any device on the network.

These new edge routers communicate through the Verizon Cloud (routed via MPLS) to National Grid owned

RTUs / data concentrators located at System Control Centers.

The PI Data Historian system is being used to archive operational data for analytics.

Currently have 700Mhz system integrated into MPLS/PIP network

Currently integrating satellite links to the MPLS/ PIP network for additional flexibility.

15

Device Connectivity Example

16

Device Redundency Example

17

18

Implementation Challenges

Strict risk management policies resulted in extensive time required to implement and test individual network changes.

90+ required network changes

Each network ‘change’ was implemented individually and required a subsequent 48hr window for testing and validation.

Limited Staffing level to build and maintain this new network.

Training and Documentation

New technology (i.e., hardware) required training for all impacted Engineering and Operations personnel.

New processes rolled out to all impacted stakeholders.

Engineering

Protection and Telecom Operations

Meter and Test

O&M

Overhead Operation

Extensive reference documentation required for sustainability.

19

Functional Network Benefits

Remote management of field devices through a secure Corporate portal:

Device configuration

Device security

Remote Record Retrieval

2-way file transfer (remote firmware, settings, etc.)

Network and security management embedded in the design of the network architecture allowing for real-time monitoring and proactive action should anomalies be detected or when general maintenance is required (e.g., Syslog, SNMP, etc.)

20

FUNCTIONALITY

SCADANear Real-Time

Data

Remote

Record

Retrieval

Remote

Configuration

Remote Device

Troubleshooting

Advanced

Applications

Support

Data

Historian

Legacy Telemetry Solution YES NO NO NO NO NO LIMITED

New Network Architecture YES YES YES YES YES YES YES

THANK YOU