SCADA Network & Data Redundancy at Guelph Water · SCADA = Supervisory Control and Data Acquisition. 8 ... •BGP switches from fibre to backup wireless within 45 seconds of a failure

1

SCADA Network & Data Redundancyat Guelph Water

Ensuring Security of Supply for

Operations and Compliance

OWWA Conference – Apr 29 to May 2, 2018 – Niagara Falls, Ontario, Canada

Graham NasbyWater SCADA & Security Specialist, Guelph Water Services

Derek WongPrincipal Consultant, Symbiotic Networks

2Backup SCADA Networks & Data‐loggingOWWA 2018 – Niagara Falls, Ontario, Canada

About the Speakers

2

Graham Nasby, P.Eng., PMP, CAPWater SCADA & Security Specialist City of Guelph Water Services

• 10 years in the consulting sector• Joined Guelph Water Services in 2015• Co-chair of ISA112 SCADA Systems standards committee• Member of the OWWA Automation Committee• Contact: [email protected]

Derek WongPrincipal Consultant, Symbiotic NetworksFounder, Zabu

• 20+ years building and managing IT infrastructure• Extensive experience supporting Communication Service Providers• Active member in the information security community• Outspoken advocate of open source software• Contact: [email protected]


Presentation Outline• SCADA Systems Review• Guelph SCADA Network Pre‐2017• Datalogging Requirements: O.Reg. 170• Impact of SCADA Network & Data‐Logging Outages

• SCADA network design goals• Guelph Water Solution: Fibre & Wireless with BGP• SCADA Data‐logging design goals• Guelph Water Solution: 3 layers of data‐logging

• Lessons Learned• Best Practices Moving Forward

3


City of Guelph Water Services

• Guelph, Ontario, Canada• 140,000 residents• 21 groundwater wells• 3 water towers

• 549 km of water mains• 49,000 service connections• 2,750 fire hydrants• 35 unmanned facilities• 46,000 m3/day [12 MGD]

4


Map of Guelph Water

5

• 35 Unmanned sites

• Central Ops Centre is staffed 8am‐4pm Mon‐Fri

• On‐call operators on evenings and weekends

• 30‐40 minutes to drive across the city (with traffic)

ARKELLSPRINGS


Water SupplyFacilities

6

• Arkell 1 & 7• Arkell 6• Arkell 8• Arkell 14• Arkell 15• Diversion Chamber• Water Street Well• Emma Well• Park Well• Clythe Station• Paisley Station• Membro Well• Carter Well

• Downey Well• Verney Tower• Burkes Well• Arkell Well• Clair Tower• Clair Booster• Robertson Booster• Helmar Well• University Well• Dean Well• Calico Well• Speedvale Tower• Scout Camp• Queensdale Well

All sites are linked together with SCADA Networkfor remote control, automatic control, monitoring and logging.

• Woods Station(Chlorine Building, UV Building, Booster Station)


What is SCADA?

7

SCADA = Supervisory Control and Data Acquisition


Typical SCADA Architecture

8


Typical SCADA Network

9


Regulatory Requirements • O.Reg. 170 – Drinking Water Systems• Free Chlorine Residuals Must be Logged Minimum Every 5 minutes• Filter Plants: Turbidity must be logged every 15 minutes• Low chlorine alarms must be communicated promptly

• Most IT Systems only reach 90 ‐ 95% uptime (outages at night ok)• Very challenging to meet this uptime for a SCADA network• SCADA system must be online 24/7, 365 days/year• Less than 5 min downtime/year = 99.9995% uptime• SCADA system has to be 10,000 X more reliable than an IT system

10


From O.Reg. 170, Section 6.5:

11


“Nines” Availability

12

I.T.

SCADA


Guelph SCADA Network: Pre-2017

13


Impact of SCADA Network Outages• Data Logging

– Meeting the 5 minute data‐logging requirement

• Process Visibility– Ability of Operators to “see” the status of remote sites

• Speed of Response– Ability to make adjustments remotely vs. Driving a Truck to Site

• Automatic Control of Pumps based on Water Tower Levels– Requires PLC‐to‐PLC communications to work

• Can be very disruptive to Operations– Operator has to be on site to physically write‐down residuals every 5 min– Operator stationed at a Water Tower to call in pump starts/stops via radio

14


Goals of SCADA Network Upgrade• Maintain SCADA Network Connectivity at All Times• Be able to keep operating if a Fibre Cable is damaged/cut• Be able to keep operating during fibre provider maintenance

• Reduce reliance on store/forward data‐loggers during outages• Reduce the impact on Operations during a Fibre Outage• Reduce the compliance risk associated with SCADA outages

• Implement continuous network monitoring for SCADA network• Gather network statistics to monitor SCADA network performance

15


SCADA Network Technology Options• Existing

– MPLS on Fibre (dedicated private network, via a Service Provider)

• Options to Supplement– Back‐up Fibre Connections– DSL– Ethernet‐over‐cable– Cellular– Private Cellular– Radios (unlicensed)– Radios (licensed)

• How do we activate the backup network link when it is needed?16


Redundant SCADA Network Design• Hub Site

– MPLS Primary & Secondary Fibre Optic Connections– Two different cable routings to the building(“diversity”)– HSRP = Hot Standby Router Protocol– Two Routers that use HSRP to create a floating “IP Address” that is shared between routers– UPS‐backed power, with Diesel Generator Backup

• Remote Sites– MPLS Primary Link: Fibre – Backup MPLS Link: Encrypted 3G/LTE Wireless on Private Cellular Network– One Router Used running BGP: Border Gateway Protocol– BGP is configured to switch to backup link in 45 seconds if Primary Fibre connection lost– BGP will automatically switch back to fibre when fibre connectivity is restored– UPS‐backed power

17


HSRP: Hot Standby Router Protocol• Technology developed by Cisco• Provides redundant Gateway using 2 identical fibre network links• Used for Main Site because two links are both fibre & identical• Does not do routing (could not be used for Remote Sites)

• Router One: xxx.xxx.0.1• Router Two: xxx.xxx.0.2• “Virtual Network Address” that shows up at: xxx.xxx.1.1• Takes about 2 minutes to switch from Primary to Secondary• Does both “fail‐over” and “fail‐back” automatically

18


BGP: Border Gateway Protocol• BGP is the Protocol that runs the Internet

– Used by Internet Service Providers to connect their networks together– For example, the connection between Bell and Rogers uses BGP

• We use BGP to transparently and automatically use fibre & backup wireless links• BGP switches from fibre to backup wireless within 45 seconds of a failure

• Border Gateway Protocol (BGP) is a standardized exterior gateway protocoldesigned to exchange routing and reachability information among autonomous systems (AS) on the Internet. The protocol is classified as a path vector protocol. The Border Gateway Protocol makes routing decisions based on paths, network policies, or rule‐sets configured by a network administrator and is involved in making core routing decisions. (source: Wikipedia)

19


SCADA Network - Before

20


Upgraded Network

21

MAIN SITE


SCADA Network Monitoring Dashboard

22


SCADA Network Monitoring Graphs

23


SCADA Network Uptime Results• 2016 SCADA network uptime: 99.6%• 2017 SCADA network uptime (post upgrade): 99.995% + increasing

• Pre‐upgrade when fibre provider had to do network maintenance it was very disruptive and required sites to be shutdown/manned

• Post‐upgrade when fibre provider has to do network maintenance, the network at that site automatically switches to backup wireless

• Fibre failures are no longer a major operational inconvenience• Network monitoring allows us to proactively fix problems

24


Back up Data-Logging• No technology or person is perfect…• So we have backup data‐logging to guard against failures/mistakes

25

• Primary Logging– Facility PLC– SCADA server reads data in “real‐time”

• Backup Data‐logging– “QuickPanel” store/forward data‐logger– Continuously logs into a buffer– Pushes data up to SCADA server– If no connectivity, stores timestamped data– When network connectivity restored, forwards

logged data up to server

• DNP3‐based Data‐loggers New Technology


SCADA Network

Existing Data Logging Infrastructure

26

Historian

Flowmeters, Analyzers, Level, Pressure

Site PLC QuickPanelData‐logger


New Data Logging Infrastructure

27

MainHistorian


Site PLC QuickPanel Data‐logger(store & forward)

Critical Instrumentation: Flow, Analyzers

DNP3 Data‐logger(store & forward)

Backup Historian

SCADA Network


Wiring the DNP3 Logger Into the Loop

28


DNP3 Protocol for Data-Logging• Automatic store/forward data logging is built into the protocol• Timestamping data at the device/instrument• Automatic Time synchronization • Both “Change on Value” and “Polling Interval” data gathering• Data Quality Flags

• Supports both Encryption and Authentication• Automatic Error checking of data packets• Prioritized Communications (Tags can be grouped in classes 0‐4)• Variety of networks supported: Ethernet, serial, ATM, cellular, etc.• Supports wide range of analog, digital and structured datatypes

29


New Data Logging Infrastructure

30

MainHistorian


Site PLC QuickPanel Data‐logger(store & forward)

Critical Instrumentation: Flow, Analyzers

DNP3 Data‐logger(store & forward)

Backup Historian

SCADA Network


Accessing your Logged Data

31


Take Aways• A functioning SCADA system is critical for operations

– SCADA Network allows for remote data‐logging & control– Data‐logging is needed to meet requirements of O.Reg. 170– SCADA outages can be very disruptive to operations and have compliance risks

• Uptime requirements for SCADA systems are very high • A typical SCADA system has to be 10,000X more reliable than an IT system• O.Reg. 170 requires us to log chlorine residual every 5 minutes (no exceptions!)

1. Consider using a redundant wire‐area SCADA network with automatic failover2. Use redundant data‐logging to guard against equipment failures3. Look at DNP3 protocol for doing back‐up store & forward data logging4. Think about how your operations/compliance team will get access to the data

32

33Backup SCADA Networks & Data‐loggingOWWA 2018 – Niagara Falls, Ontario, Canada 33

* Not a High Performance SCADA System

Any Questions?

Graham Nasby – graham.nasby@guelphca

Derek Wong – [email protected]

Documents

SCADA Network & Data Redundancy at Guelph Water · SCADA = Supervisory Control and Data Acquisition. 8 ... •BGP switches from fibre to backup wireless within 45 seconds of a failure