7 Networking Wrapup

Critical Infrastructure Security: The Emerging Smart Grid

In Class: Cpt S 580-03, Cpt S 483-01, EE 582-02, EE 483-01

Online or Tri-Cities: Cpt S 580-03, Cpt S 483-01, Cpt S 483-02, EE 582-01, EE 483-01

Dr. Anurag K. Srivastava, Dr. Carl Hauser, Dr. Dave Bakken

Putting it together: network security practices for the (smart) grid

April 24, 2014

1

Acknowledgement

• I am enormously indebted to Andrew Wright of n-dimension solutions (www.n-dimension.com) for the work he has put into educating the power community about security and for permission to use materials and figures drawn from his presentations, including his TCIPG 2011 Summer School Lecture.

2

http://www.n-dimension.com/



The most important thing!

… all else follows!

3

Source: NIST Smart Grid Framework and Roadmap, Conceptual Reference Diagram, January 2010

SCADA

HAN

Control Systems in the Power Grid

4

Historical ICS

• Proprietary

• Complete vertical solutions

• Custom

• Specialized communications – Wired, fiber, microwave, dialup, serial, etc.

– 100s of different protocols

– Slow; e.g. 1200 baud

• Long service lifetimes: 15–20 years

• Not designed with security in mind

5

Third Party Controllers, Servers, etc.

Serial, OPC or Fieldbus

Engineering Workplace

Device Network

Firewall

Services

Network

Third Party Application Server

Application Server

Historian Server

Workplaces Enterprise Optimization Suite

Mobile Operator

Connectivity Server

Control

Network

Redundant

Enterprise Network

Serial RS485

Modern ICS

IP

Internet

Enterprise Network

6

Technology Trends in ICS

• COTS (Commercial-Off-The-Shelf) technologies – Operating systems—Windows, WinCE, embedded RTOSes

– Applications—Databases, web servers, web browsers, etc.

– IT protocols—HTTP, SMTP, FTP, DCOM, XML, SNMP, etc.

• Connectivity of ICS to enterprise LAN – Improved business visibility, business process efficiency, meter to cash

– Remote access to control center and field devices

• IP Networking – Common in higher level networks, gaining in lower levels

– Many legacy protocols wrapped in TCP or UDP

– Most new industrial devices have Ethernet ports

– Most new ICS architectures are IP-based

7

• Some vulnerabilities are known and increasing; others unknown

• Consequences range from nuisance to severe

• Threat is ever changing

• Interdependencies and interconnections are the risk multipliers

Threat

Consequence Vulnerability

The Risk Equation

8

Sources of vulnerability in modern ICS

• COTS + IP + connectivity = many security risks

• All of those of Enterprise networks and more: Poor separation from enterprise nwk Legacy OSes and applications

No security monitoring Inability to limit access

Poorly secured 3rd party access Inability to revoke access quickly

Dialup modems Unexamined system logs

Unpatched systems Accidental misconfiguration

Limited use of anti-virus Improperly secured devices

Limited use of host-based firewalls Lack of security features

Improper use of ICS workstations Improperly secured wireless

Unauthorized applications Unencrypted links to remote sites

Unnecessary applications Passwords sent in clear text

Open FTP, Telnet, SNMP, HTML ports Password management problems

Fragile control devices Default OS security configurations

Network scans by IT staff Unpatched routers / switches

9

Known Security Problems in the Power Grid

• From NISTIR 7628

• 7.2 EVIDENT AND SPECIFIC CYBER SECURITY PROBLEMS 7.2.1 Authenticating and Authorizing Users to Substation IEDs 7.2.2 Authenticating and Authorizing Users to Outdoor Field Equipment 7.2.3 Authenticating and Authorizing Maintenance Personnel to Meters 7.2.4 Authenticating and Authorizing Consumers to Meters 7.2.5 Authenticating Meters to/from AMI Head Ends 7.2.6 Authenticating HAN Devices to/from HAN Gateways 7.2.7 Authenticating Meters to/from AMI Networks 7.2.8 Securing Serial SCADA Communications 7.2.9 Securing Engineering Dial-up Access 7.2.10 Secure End-to-End Meter to Head End Communication 7.2.11 Access Logs for IEDs 7.2.12 Remote Attestation of Meters 7.2.13 Protection of Routing Protocols in AMI Layer 2/3 Networks 7.2.14 Protection of Dial-up Meters 7.2.15 Outsourced WAN Links 7.2.16 Insecure Firmware Updates 7.2.17 Side Channel Attacks on Smart Grid Field Equipment 7.2.18 Securing and Validating Field Device Settings 7.2.19 Absolute & Accurate Time Information 7.2.20 Personnel Issues in Field Service of Security Technology 7.2.21 Weak Authentication of Devices in Substations 7.2.22 Weak Security for Radio-Controlled Distribution Devices 7.2.23 Weak Protocol Stack Implementations

7.2.24 Insecure Protocols 7.2.25 License Enforcement Functions 7.2.26 Unmanaged Call Home Functions

10

Known Security Problems in the Power Grid

• From NISTIR 7628 • 7.3 NONSPECIFIC CYBER SECURITY ISSUES

7.3.1 IT vs. Smart Grid Security 7.3.2 Patch Management 7.3.3 Authentication 7.3.4 System Trust Model 7.3.5 User Trust Model 7.3.6 Security Levels ... 7.3.33 Cyber Security Governance

NISTIR 7628, Guidelines for, Smart Grid Cyber Security, Volume 3, Chapter 7 11

Consequences • Loss of production

• Penalties

• Market breakdown (e.g. Enron)

• Lawsuits

• Loss of public trust

• Loss of market value

• Physical damage

• Environmental damage

• Injury

• Loss of life

• USSR pipeline explosion, 1982

• Bellingham pipeline rupture, 1999

• Queensland sewage release, 2000

• Davis Besse nuclear plant infection, 2003

• Northeast USA blackout, 2003

• Browns Ferry nuclear plant scram, 2006

• Stuxnet, 2010 $$$.$$ 12

Adversaries

• Script kiddies • Hackers • Organized crime • Competitors • Terrorists • Hactivists • Eco-terrorists • Nation states • Disgruntled insiders

– (security failures can make any of the others into an insider!)

13

Threat Model

• Targeted and untargeted threats – Targeted: human, specifically crafted worm/virus,

botnet

– Untargeted: generic worm/virus, script kiddy

• Assume adversary has: – Complete knowledge of network

– Beachhead in enterprise network

– Limited access to control network

– Limited physical access to field equipment

14

Of recent concern: APTs

• APT: Advanced Persistent Threats • Highly motivated teams of attackers with strong skills and

tenacity – Targeted: specific economic or political aims – Nation state backing – Exploit vulnerabilities very effectively

• Social engineering – spear-phishing • OS and other software vulnerabilities, including 0-day • Supply chain

• Goal: a beach-head in the victim’s systems from which further attacks can be launched at will – E.g. to steal data on an ongoing basis – To disrupt operations when the time is ripe

15

Attack Vectors into Control Systems

Source: 2003–2006 data from Eric Byres, BCIT

16

Poor Separation from Enterprise Network

17

Poorly Secured 3rd Party Connections

18

Fragile ICS Devices

• Many IP stack implementations are fragile

– Some devices lockup on ping sweep or NMAP scan

– Numerous incidents of ICS shut down by uninformed IT staff running a well-intentioned vulnerability scan

• Modern ICS devices are much more complex

– Some PLCs include web server for configuration and status

– More lines of code leads to more bugs

– Modern PLCs require patching just like servers

19

Unpatched Systems

• Many ICS systems are not patched current – Particularly Windows servers – No patches available for older versions of windows

• OS and application patches can break ICS • Uncertified patches can invalidate warranty • Patching often requires server reboot • Before installation of a patch:

– Vendor certification—typically one week – Lab testing by operator – Staged deployment on less critical systems first – Avoid interrupting any critical process phases

20

No Anti-Virus

• AV operations can cause significant system disruption at inopportune times

– 3am is no better than any other time for a full disk scan on a system that operates 24x7x365

• ICS vendors only beginning to support anti-virus

– Anti-virus is only as good as the signature set

– Signatures may require testing just like patches

21

Poor Authentication and Authorization

• Machine-to-machine comms involve no “user”

• Many ICS have poor authentication mechanisms and very limited authorization mechanisms

• Many protocols use cleartext passwords

• Many ICS devices lack crypto support

• Device passwords are hard to manage appropriately – Often one password is shared amongst all devices

and all users and seldom if ever changed

22

Poor Audit and Logging

• Many ICS have poor or non-existent support for logging security-related actions – Attempted or successful intrusions may go unnoticed

• Where IDS logs are kept, they are often not reviewed

• Various regulatory requirements are driving some change in this area – NERC—North American Electric Reliability Corporation

– FERC—Federal Energy Regulatory Commission

– DOE ARRA Funding cyber security requirements

23

Unmanned Field Sites

• Many unmanned field sites

• Some with high-speed connectivity to control center

• Most with poor authentication and authorization

• Many with dialup or wireless access

• Can be easy backdoor to the control center

24

Legacy Equipment

• Much legacy equipment

• Usually impossible to update to add security features

• Difficult to protect legacy communications

– but see IEEE P1711 for serial encryption

25

Unauthorized Applications

• Unauthorized apps installed on ICS systems can interfere with ICS operation

• Many types of unauthorized apps have been found during security audits – Instant messaging

– P2P file sharing

– DVD and MPEG video players

– Games, including Internet-based

– Web browsers

26

Inappropriate Use of ICS Systems

• Web browsing from HMI can infect ICS – Browser vulnerabilities

– Downloads

– Cross-site scripting

– Spyware

• Email to/from control servers can infect ICS – Sendmail and outlook vulnerabilities

• Disk storage exhaustion can crash OS – Storage of music, videos

27

Requirement for 3rd Party Access

• Firmware updates and PLC programming are sometimes done by vendor

– Many ICS have open maintenance ports

– Infected vendor laptops can bring down ICS

28

People Issues

• ICS network often managed by “Operations Department”, distinct from “IT Department” running enterprise network – ICS personnel are not IT or networking experts

– IT personnel are not ICS experts

• Significant fraction of control systems workforce is older and nearing retirement – Few young people entering this field

– Few academic programs

29

Security Assessments on ICS

• Various groups perform security assessments and penetration tests on ICS (generally under NDA) – Idaho National Labs – Sandia National Labs – N-Dimension Solutions – Other private organizations

• They always get in • Not a question of “if”, but “how long”

• N-Dimension prefers white-box assessments over

black-box penetration tests

30

Limited Information About Incidents

• Little information sharing about actual attacks – BCIT incident database has about 30 incidents per year vs.

100s of thousands of incidents per year in CERT database – Few cyber attacks on ICS for which details are public – National Electric Sector Cyber Security Organization

(NESCO) funded by DOE is ramping up a controlled sharing portal

• Difficult to estimate risk – Difficult to demonstrate ROI for security spending

• But… lots of data about significant financial losses in enterprise and e-commerce – Why would control systems be immune?

31

Other Issues

• Extreme environments

• Unusual physical and geographical topologies

• Many special purpose, limited function devices

• Static network configurations

• Multicast

• Long service lifetimes

• Much legacy equipment

32

How an Attack Proceeds—Step #1

Internet

Modem Pool

Web Server

Email Server

Business Workstation

Data Historian

Engineering Workstation

SCADA

RTU Control System Network

Enterprise Network

Database Server

Domain Name Server (DNS)

Enterprise Firewall

ICS Firewall

Attacker

PLC

PLC

Web Server

Management Console HMI

Initiated by Phishing or Spearphishing attack

33


Internet

Modem Pool

Web Server


Data Historian


SCADA


Enterprise Network


Enterprise Firewall

ICS Firewall

Attacker

PLC

PLC

Web Server


Email Server

Database Server

34

How an Attack Proceeds—Step #2b

Internet

Modem Pool

Web Server


Data Historian


SCADA


Enterprise Network


Enterprise Firewall

ICS Firewall

Attacker

PLC

PLC

Web Server


Email Server

Database Server

Initiated by Flash Drive

35


Internet

Modem Pool

Web Server


Data Historian


SCADA


Enterprise Network


enterprise Firewall

ICS Firewall

Attacker

PLC

PLC

Web Server


Email Server

Database Server

36


Internet

Modem Pool

Web Server

Web Server


Data Historian



SCADA


Enterprise Network


enterprise Firewall

ICS Firewall

Attacker

PLC

PLC

Vendor Web Server Email Server

Database Server

37


Internet

Modem Pool

Web Server

Web Server


Data Historian



FEP


Enterprise Network


enterprise Firewall

ICS Firewall

Attacker

PLC

PLC


Database Server

38

How an Attack Proceeds—Step #5b

Internet

Modem Pool

Web Server

Web Server


Data Historian



SCADA


Enterprise Network


enterprise Firewall

ICS Firewall

Attacker

PLC

PLC


Database Server

Initiated by Flash Drive on Control Network

39


Internet

Modem Pool

Web Server

Web Server


Data Historian



SCADA


Enterprise Network


enterprise Firewall

ICS Firewall

Attacker

PLC

PLC

Email Server

Database Server

40


Internet

Modem Pool

Web Server

Web Server


Data Historian



SCADA


Enterprise Network


enterprise Firewall

ICS Firewall

Attacker

PLC

Email Server

PLC

Database Server

41


Internet

Modem Pool

Web Server

Web Server


Data Historian



SCADA


Enterprise Network


enterprise Firewall

ICS Firewall

Command and Control

PLC

Email Server

PLC

Database Server

42

How are Power Grid ICS being Defended?

• Regulatory requirements

• Network design practices

• (Defense technology) – Firewalls, IDS & IPS, automated recovery systems, etc. (no time to discuss today)

43

Regulations: NERC CIP

• NERC – North American Electric Reliability Corporation

• FERC – Federal Energy Regulatory Commission

• NERC CIP – Critical Infrastructure Protection Standards

– Just one of several NERC standards related to reliably operating complex, interconnected power grids

44

NERC CIP Standards

• Industry writes standards with coordination by NERC; FERC approves them – Utilities have input to the process

– But ultimately FERC gets to say what is acceptable, as does its Canadian counterpart

• Standards aimed at protecting the Bulk Power System (transmission system) – therefore evolving how they apply to distribution entities

• Nearly 200 required documentation, management, and security practices; all must be auditable

• Fines for violation up to $1M/day!

45

NERC CIP Standards: Contents

• CIP-001-2a: Required reporting of incidents

– To ISO, to DOE, to NERC

• CIP-002-4: Critical Cyber Asset Identification

– Recognizes that some assets require greater protection than others: critical vs not critical

– Critical cyber assets are those that relate to critical power assets which are the things whose failure could disrupt reliable operation of the BPS

46

NERC CIP Standards: Contents (cont’d)

• CIP-003-4: Security Management Controls – Company executives are responsible

– Requires change control and configuration management

• CIP-004-4: Personnel and Training – People need to be aware of cybersecurity

– Background checks? Ongoing assessment of personnel for security risks

– Access control (cyber and physical)

47


• CIP-005-4a: Electronic Security Perimeters

– Identification of perimeters and ports

– Access control, logging, and monitoring

– Annual vulnerability assessments

• CIP-006-4d: Physical security of critical cyber assets

– Access control, logging and monitoring

48


• CIP-007-4: Systems Security Management – Testing when making changes

– Hardening prior to production use

– Patch management

– Malicious software prevention

– Account management

– Continuous security status monitoring

– Secure device disposal or redeployment

– Annual vulnerability assessments

49


• CIP-008-4: Incident Reporting and Response Planning – Have a plan to deal with cyber security incidents

– Know what has to be reported

– Document incidents

• CIP-009-4: Recovery Plans for Critical Cyber Assets – Have plan for backup and recovery

– Perform backup and recovery testing

50

Problem

• Standard largely addresses documentation and practices, not achievement of security – Nobody knows how to measure security

– If you can’t measure it, you can’t regulate it

• Consequence: attacks may succeed even when you are completely compliant with NERC CIP standards – Still worthwhile – less vulnerable, smaller

consequences, faster recovery (at least in theory!)

51

Defending ICS

• Separate control network from enterprise network – Harden connection to enterprise network

– Protect all points of entry with strong authentication

– Make reconnaissance difficult from outside

• Harden interior of control network – Make reconnaissance difficult from inside

– Limit single points of vulnerability

– Frustrate opportunities to expand a compromise

• Harden field sites and partner connections

• Monitor both perimeter and inside security events

• Monitor server and network behavior

• Periodically scan for changes in security posture

52

Availability, Integrity and Confidentiality

• Enterprise networks require C-I-A – Confidentiality of intellectual property matters most

• ICS requires A-I-C – Availability and integrity of control matters most – control data has low entropy—little need for

confidentiality – Many ICS vendors provide six 9’s of availability

• Ensuring availability is hard – Cryptography does not help (directly) – DOS protection, rate limiting, resource management,

QoS, redundancy, robust hardware with high MTBF

53

Defense in Depth

• Perimeter Protection – Firewall, IPS, VPN, AV

– Host IDS, Host AV

– DMZ

• Interior Security – Firewall, VPN, AV

– Host IDS, Host AV

– Application Whitelisting

– IEEE P1711 (AGA 12)

– NAC

• Monitoring – IDS

– Scanning

• Management

IDS Intrusion Detection System IPS Intrusion Prevention System DMZ DeMilitarized Zone VPN Virtual Private Network (cryptographic) AV Anti-Virus (anti-malware) NAC Network Admission Control

54

50000 Foot View Internet

Control Network

Field Site Field Site Field Site

Partner Site

VPN

VPN

FW

FW

IPS

IDS

IT Stuff

Scan

AV

FW IPS

P1711

FW

AV Host IPS Host AV

Proxy

Host IDS Host AV

IDS Scan

Enterprise Network

55

Control DMZ Architecture

• Enterprise Network contains typical business systems – Email, web, office apps, etc.

• Control DMZ provides business connectivity – Contains only non-critical systems that provide

connectivity btween Control and Enterprise Networks – Enforces separation between Enterprise and Control

Networks – May consist of multiple functional zones

• Separated by Firewall, IPS, Anti-Virus, etc.

• Control Network demarcates critical control systems – May consist of multiple functional zones

• Internally protected by Firewall, IDS, Anti-Virus, etc.

56

Control DMZ Perimeter Protection

Firewall with NAT Remote Access VPN Network Anti-Malware Intrusion Prevention

57

Control DMZ Design Principles

• Multiple functional security zones

• Traffic between zones undergoes firewall & IPS

• Only path in/out of Control Network

• Default deny for all firewall interfaces

• No/Minimal direct traffic across DMZ

• No common ports between outside & inside

• No control traffic to outside

• Highly limited outbound traffic

• No connections initiated from DMZ into Control Network

• Emergency disconnect at inside or outside

• No network management from outside

58

Control Network Design Principles

• Minimal number of connections to DMZ

• Control Network independent of DMZ, Enterprise – Separate Networking Hardware from DMZ

– Separate Time Server

– Separate AAA

– Allows emergency disconnect from DMZ

• QoS where applicable

• Redundancy where appropriate

59

Management

• Out-of-band management in DMZ

– Network and security management

– AAA servers

• Out-of-band local management in Control Network

– For disconnected operation

• In-band management for field sites

– Where local management is impractical

60

Securing Converged Smart Grid Networks

• no separate control network from enterprise network

• will use IP-based protocols, thus need IP-based defenses

• must consider defenses at all layers: – physical layer

– link layer

– internet layer

– transport layer

– application layer

• security will be a serious challenge!

• see Wright, Kalv, Sibery, “Interoperability and Security for Converged Smart Grid Networks”, Grid Interop 2010

61

Questions?

62

References and links

• Andrew Wright, “Cyber Security for Grid Control Systems”, EEI Transmission, Distribution and Metering Conference, April 2009. http://www.eei.org/meetings/Meeting%20Documents/2009-04-05-Tues-3-Wright.pdf

• Jim Brenton. “Advanced Persistent Threats to the Electric Sector”, Grid Security Conference 2011, New Orleans, LA. http://www.nerc.com/files/7_Brenton_APT_NERC_GridSecCon_2011-10-19_DRAFT_1.pdf

• Robert H. McLanahan, “CIP Standards & Grid Reliability”, Grid Security Conference 2011, New Orleans, LA. http://www.nerc.com/files/3_GridSecCon_2011_10_19-McClanahan.pdf

• Andrew Wright, Paul Kalv, and Rodrick Sibery, “Interoperability and Security for Converged Smart Grid Networks”, Gridwise Forum 2010, http://www.gridwiseac.org/pdfs/forum_papers10/wrightpre_gi10.pdf

63

http://www.eei.org/meetings/Meeting Documents/2009-04-05-Tues-3-Wright.pdf











http://www.nerc.com/files/7_Brenton_APT_NERC_GridSecCon_2011-10-19_DRAFT_1.pdf





http://www.nerc.com/files/3_GridSecCon_2011_10_19-McClanahan.pdf



http://www.gridwiseac.org/pdfs/forum_papers10/wrightpre_gi10.pdf