JEA Lessons Learned?

$ whoami

Daniel D. MishraJEA – Director of CIP Compliancemishdd@jea.com

Electric Utility Experience – 20 yearsCritical Infrastructure Experience – 10 YearsSpecialization – Innovative and efficient Cyber Security Infrastructure that provides highest rate of return on investment without compromising security or reliability

Event 1 – HMI Sync & loss of situational awarenessEvent 2 – JEA Firewall Hardware and Failover

Agenda

Protect Identity: Preserve Privacy

Economic Security: System Integrity

• Network Integrity:A state in which the network performs its intended function in an unimpaired manner, free from deliberate or inadvertent unauthorized manipulation of the system.

Operations Security: Service

Availability

What’s the Goal of Security?

Event 1 – HMI Sync & loss of situational awareness

• Time duration 40 Minutes (May 17, 2013, 20:08 – 20:48)

• Impact – Loss of situational Awareness• HMI – EMS Server session Failure• EMS Data Acquisition and Aggregation – OK• RC data communication – OK• Root Cause – Time Sync issue between HMIs

and EMS Servers

HMI Sync Failure

Event/Impact Summary

6

HMI Sync Error

APP HISDAC

APP HISDAC

APP HISDAC

GPS Clock

GPS Clock

GPS Clock

UF Cloud

Backup Control Center Primary Control Center

Domain Controller

Domain Controller

1. Serially connect GPS Clocks2. System 32 Time Sync – Domain control3. NTP Agent time sync

7

HMI Sync (Current State)

APP HISDAC

APP HISDAC

APP HISDAC

GPS Clock

GPS Clock

GPS Clock

Backup Control Center Primary Control Center

1. Time is queried from all the three GPS clocks and system time is synchronized with the best time (True-Chimer vs False-Ticker).

2. AD also synchronizes to the same source as the HMI NTP client.

8

• True Chimer – Most accurate time • False Ticker – Exceeds time difference

toleranceIn JEA’s current implementation, ntp client queries time from all the three available clocks and synchronizes with most accurate time source. All sources are synchronized with single source limiting any other sources for conflict.

HMI Sync Error

9

Event 2 – JEA Firewall Hardware and Failover

Network Topology – EMS Firewall Event

Firewall was single point of failure for –1. AD

Communication2. FEP

Communication3. MPLS (RC)

Communication4. SOCC – BUCC

Communication

Redundant High Availability Firewalls

JEA Private Fiber Network using

SONET and Carrier Ethernet

11

SOCC – BUCC Network TopologySOCC Data Center

EMSFirewall

Electronic Security Perimeter (EMS)

Ethernet

Electronic Security Perimeter (CORP)

ManagementFirewall

CC-3 Data Center

EMSFirewall

Electronic Security Perimeter (EMS)

Ethernet

Electronic Security Perimeter (CORP)

Point to PointT1

Point to PointT1

BadgingIdentityManager

Active Directory Anti-Virus

Badging Active Directory Anti-Virus

ManagementFirewall

SONETRing

Terminal Server

Terminal Server

IdentityManager

SIEM

To Substations

To SubstationsFirewall is single point of failure for –1. MPLS (RC)

Communication2. SOCC – BUCC

Communication

Improvement – SOCC operate BES normally, even if Firewall is not functioning. Only RC functionality will be impacted.

JEA Private Fiber Network using

SONET and Carrier Ethernet

12

Firewall Events – Immediate alerting

105009• Error Message %ASA-1-105009: (Primary) Testing on interface

interface_name {Passed|Failed}.• Explanation The result (either Passed or Failed) of a previous

interface test has been reported. Primary can also be listed as Secondary for the secondary unit.

• Recommended Action None required if the result is Passed. If the result is Failed, you should check the network cable connection to both failover units, that the network itself is functioning correctly, and verify the status of the standby unit.

8/13/2013 19:34 23:35 3 100278-105009 Testing on interface failure X.X.254.18/13/2013 4:14 4:14 2 100278-105009 Testing on interface success X.X.254.18/13/2013 4:14 4:14 2 100278-105008 Testing interface alert X.X.254.1

13

System Status MonitoringSolarWinds

14

Corrective ActionsMonitoring / Visibility1. Configuration change – Add configuration to monitor- interface health of

switches (trunked ports) and Firewall State (Refer to CISCO)2. Implement Health and Visibility Monitoring Solution for EMSCommunication1. Technology Services (TS) Network group will prepare steps/checklist used

for network problem diagnosis and train BPO support staff2. Use direct calling or conference for severe events resolution 3. CIP Compliance Department will conduct two mandatory training

(Incident Response & Disaster Recovery)Logging & Monitoring1. TS has added multiple alerts that in future will provide advance warning2. TS will review all devices to verify that logging is enabled.3. Network team will maintain complete testing records as per the checklist

15

Questions?

Clock Select Algorithm -https://www.eecis.udel.edu/~mills/ntp/html/select.htmlCisco ASA Series Syslog Messages -http://www.cisco.com/c/en/us/td/docs/security/asa/syslog-guide/syslogs.pdf

References

17