Department of Homeland Security Control Systems Security Program

Seán Paul McGurk

Director, Control Systems Security

National Cyber Security Division

U.S. Department of Homeland Security

Overview of Control Systems

U.S. Critical Infrastructure SectorsHomeland Security Presidential Directive 7 (HSPD-7) along with the National Infrastructure Protection Plan (NIPP) identified and categorized U.S. critical infrastructure into the following 18 CIKR sectors

• Agriculture and Food • Banking and Finance • Chemical • Commercial Facilities • Critical Manufacturing• Dams • Defense Industrial Base • Emergency Services • Energy • Government Facilities• Information Technology

• National Monuments and Icons

• Nuclear Reactors, Materials, and Waste

• Postal and Shipping • Public Health and

Healthcare • Telecommunications • Transportation • Water and Water Treatment

Many of the processes controlled by computerized control systems have advanced to the point that they can no longer be operated without the control system.

Risk Drivers: Modernization and Globalization Connections between

Information Technology and Control System networks (inheriting vulnerabilities)

Shift from isolated systems to open protocols

Access to remote sites through the use of modems, wireless, private, and public networks

Shared or joint use systems for e-commerce

Vulnerability LifecycleJanuary 2008, Core Security Technologies discovers a vulnerability in the CitectSCADA product, and works with Citect and US-CERT

June 2008, Citect releases patches for affected products

June 11, 2008, US-CERT publishes Vulnerability Note regarding Citect buffer overflow

Vulnerability LifecycleSeptember 5, 2008, Metasploit exploit code posted

September 6, 2008, Traffic increases for specified port

Control Systems Site Assessments

Since 2002 over 100 site assessments conducted

Electric, Oil and Natural Gas, Chemical, Water, and Transportation (pipeline)

Over 38,000 vulnerabilities were identified and categorized

ISA 99 Control Systems Security Model

ICS Vulnerabilities categorized by ISA99 Security Zones Level 0-5

Data provided by

ICS Security Zones of Interest

Level 3 - Operational Zone Network Device vulnerabilities

9.3% (1677 vulnerabilities)

Host based/application system vulnerabilities 90.7%(16288 vulnerabilities)

Primary security issue with: Web Server Applications

Database Servers (MS SQL, mySQL, Oracle)

Business Applications

Data provided by

ICS Security Zones of Interest

Level 2 – Supervisory HMI LAN Network Device vulnerabilities 35.4% (1614 vulnerabilities)

Host based/application system vulnerabilities 64.6%(2914 vulnerabilities)

Primary security issue with: Microsoft-based Operating System (migration)

Sun Solaris Operating Systems

General Findings Default vendor accounts and passwords still in use

Some systems unable to be changed!

Guest accounts still available

Unused software and services still on systems

No security-level agreement with peer sites

No security-level agreement with vendors

Poor patch management (or patch programs)

Extensive auto-logon capability

General Findings continued Typical IT protections not widely used (firewalls, IDS, etc.). This

has been improving in the last 6 months

Little emphasis on reviewing security logs (Change management)

Common use of dynamic ARP tables with no ARP monitoring

Control system use of enterprise services (DNS, etc.)

Shared passwords

Writeable shares between hosts

User permissions allow for admin level access

Direct VPN from offsite to control systems

Web enabled field devices

Cyber Incidents and Consequences

Italian Traffic LightsEvent: Feb, 2009 Italian authorities investigating unauthorized changes to traffic enforcement system

Impact: Rise of over 1,400 traffic tickets costing > 250K Euros in two month period

Specifics: Engineer accused of conspiring with local authorities to rig traffic lights to have shorter yellow light causing spike in camera enforced traffic tickets

Lessons learned:

Do not underestimate the insider threat

Ensure separation of duties and auditing

Transportation – Road Signs

15

Lessons learned:

Use robust physical access controls

Change all default passwords

Work with manufacturers to identify and protect password reset procedures

Event: Jan 2009, Texas road signs compromised

Impact: Motorists distracted and provided false information

Specifics: Some commercial road signs, can be easily altered because their instrument panels are frequently left unlocked and their default passwords are not changed. "Programming is as simple as scrolling down the menu selection," a blog reports. "Type whatever you want to display … In all likelihood, the crew will not have changed [the password]."

DaimlerChryslerEvent: Aug, 2005 Internet worms infect DaimlerChrysler’s systems

Impact: Workers were idle as infected Microsoft Windows systems were patched

Specifics: A round of Internet worms knocked 13 of DaimlerChrysler’s U.S. automobile manufacturing plants offline

Recovery time: Took manufacturing plants offline

for one hour

Lessons learned: Critical patches need to be

applied Provide adequate network

segmentation between control and business networks

Place controls between segments to limit congestion and cascading effects

Polish Trains

Lessons learned:

Do not rely on protocol obscurity for security

Apply appropriate access controls to all field devices

Event: A Polish teenager modifies a TV remote and hacks Lodz Tram system

Impact: 12 people injured, 4 derailments

Specifics: The 14-year-old modified a TV remote control so that it could be used to change track points. Local police said the youngster trespassed in tram depots to gather information needed to build the device. The teenager told police that he modified track setting for a prank.

Maroochy Waste WaterEvent: More than 750,000 gallons of untreated sewage intentionally released into parks, rivers, and hotel grounds

Impact: Loss of marine life, public health jeopardized, $200,000 in cleanup and monitoring costsSpecifics: SCADA system had 300 nodes (142 pumping stations) governing sewage and drinking water Used OPC ActiveX controls, DNP3, and ModBus protocols Used packet radio communications to RTUs Used commercially available radios and stolen SCADA software to make laptop appear as a pumping station Caused as many as 46 different incidents over a 3-month period (Feb 9 to April 23)

Lessons learned: Suspend all access after

terminations Investigate anomalous system

behavior Secure radio and wireless

transmissions

Browns Ferry Power PlantEvent: Aug, 2006 Two circulation pumps at Unit 3 of the nuclear power plant failed

Impact: The unit had to be shut down manually

Specifics: The failure of the pumps was traced to excessive traffic on the control system network, possibly caused by the failure of another control system device

Recovery time: SPDS – 4hours 50 minutes PPC – 6 hours 9 minutes

Lessons learned: Provide adequate network

segmentation Place controls on multiple

segments to limit congestion and cascading effects

Provide active network monitoring tools

Hatch Nuclear Power Plant

…there was full two-way communication between certain computers on the plant's

corporate and control networks. 20

Lessons learned: Patch management policy

must address testing requirements before integration in production environment

IT and ICS must be aware of connectivity

Event: A software update caused control system to initiate plant shutdown.

Impact: The Plant was shutdown for 48 hours

Specifics: . An engineer installed a software update on a computer operating on the plant's business network. When the updated computer rebooted, it reset the data on the control system, causing safety systems to errantly interpret the lack of data as a drop in coolant water reservoirs

Recovery time: 48 Hours

Davis Besse Nuclear Power PlantEvent: Aug 20, 2003 Slammer worm infects plant

Impact: Complete shutdown of digital portion of Safety Parameter Display System (SPDS) and Plant Process Computer (PPC)

Specifics: Worm started at contractors site

Worm jumped from corporate to plant network and found an unpatched server

Patch had been available for 6 months

Recovery time: SPDS – 4hours 50 minutes PPC – 6 hours 9 minutes

Lessons learned: Secure remote (trusted) access

channels Ensure Defense-in-depth

strategies with appropriate procurement requirements

Critical patches need to be applied

Olympic Pipeline ExplosionEvent: 16-inch gasoline pipeline explosion and fire, exacerbated by inability of SCADA system to perform control and monitoring functions.

Impact: 3 fatalities, property damage >$45M, matching fines of $7.86M against two companies.

Specifics: Erroneous changes to live historical database caused critical slowdown in system responsiveness (evidenced by sensor scan rate changing from 3 second poll to over 6 minutes!)

Communication link between main computer, field sensors, and controllers was a combination of leased phone lines and frame relay.

photo by David Willoughby copyright Bellingham Herald

Lessons learned:

Identify controls to Critical Assets

Do not use administrative controls to solve system anomalies

Do not perform database updates on live systems

Apply appropriate security to remote access

Arizona Salt River ProjectEvent: 1994 - Unauthorized access into network of the Salt River Project Water Utility

Impact: Estimated losses of $40,000, and lost productivity due to the compromise

Specifics: A programmer and software developer, using a dial-up modem, was able to break into the SRP network with the intention of retrieving billing information. Compromised server monitored the water levels of canals in the Phoenix area.

Accessed data included monitoring and delivery information for water and power processes, in addition to financial and customer data. Data exfiltrated or altered included login/ password files and system log files.

Lessons learned: Provide adequate network

segmentation Place controls on another

segment with no direct outside access

Provide active network monitoring tools

Ensure defense-in-depth strategies, firewalls & Intrusion Detection Systems

Big Bang Experiment is Hacked Event: Sept, 2008 - Computer hackers broke into the Large Hadron Collider and defaced one of the project websites.

Impact: “There seems to be no harm done. From what they can tell, it was someone making the point that CMS was hackable," said James Gillies, spokesman for European Organization for Nuclear Research (also known as CERN)

Specifics: Hackers targeted the Compact Muon Solenoid Experiment, or CMS, one of the experiments at facility that will be analyzing the fallout of the Big Bang

CERN expressed concerned over what the hackers could do as they were “one step away” from the computer control system

Lessons learned: Provide adequate network

segmentation Place controls on another

segment with no direct outside access

Provide active network monitoring tools

Ensure defense-in-depth strategies, firewalls & Intrusion Detection Systems

Space Station – Air Gap Bridged

25

Lessons learned: Due to the human factor – there is no

true airgap, for example, thumb drives, laptop connection, modems, VPN, CD/DVD, etc.

Event: Aug. 2008, Viruses intended to steal passwords and send them to a remote server infected laptops in the International Space Station (again).

Impact: Created a “nuisance” to non-critical space station laptops

Specifics:The virus did make it onto more than one laptop -- suggesting that it spread via some sort of intranet on the space station or via a thumb drive.

Highlights

Control system security can no longer hide behind proprietary configurations and special training (Security by Obscurity)

Control systems are no longer isolated systems that require special skills; open systems and protocols

Control systems are no longer isolated from corporate and other networks

Hackers are smart, and the prevalence of information available via the Internet makes attacking control systems easier

Control systems are migrating away from their traditional shared and unrestricted configurations to more secure ones

Functional Areas ICS Analysis and informational products Training – Instructor and web base Subject Matter Expertise support Standards support ICS Assessments

On site / Control Systems Analysis Center (CSAC) Interviewing control system operators, engineers, and IT

staff on configuration and use “Table top” review of network and security (firewalls,

IDS/IPS, etc.) R&D gap analysis Sector Agency Support

Government Coordinating Council Sector Coordinating Council

ICS - CERT CSAC analysis shared across all sectors

through products and trainings

Mitigate vulnerability in partnership with vendors

Vulnerabilities patched by vendors

CSSP web site links US-CERT control systems “Vulnerability Notes”

Vulnerability reports submitted via US-CERT web site and entered into National Vulnerability Database (NVD)

PCII is an information-protection tool that facilitates private sector information sharing with the government

Cyber Security Self Assessment Tools Assessment Covers Policy, Plans and

Procedures in 10 Categories

Creates baseline security posture

Provides recommended solutions to improve security posture

Standards specific reports (e.g. NERC CIP, DOD 8500.2, NIST SP800-53)

Control Systems Forensics

Addresses the issues encountered in developing and maintaining a cyber forensics plan

Supports forensic practitioners in creating a control systems forensics plan

Assumes evidentiary data collection and preservation using forensic best practices.

Provides users with the appropriate foundation

Recommended Practices for Cyber Forensics for Control Systems

Control Systems Security PublicationsProcurement language

New SCADA / control systems

Legacy systems

Maintenance contracts

Patch Management

Network integration of Control Systems

Differences in patch deployment

Reliable patch information

Embedded commercial off-the-shelf packages

32

Standards Improvement Collaborations to evolve national

and international standards for control system security

DHS Control System Security Program (CSSP)

DOE National SCADA Test Bed (NSTB)

Instrumentation, Systems, and Automation (ISA)

National Institute of Standards and Technology (NIST)

International Electrotechnical Commission (IEC)

Web Based Training

“Cyber Security for Control Systems Engineers and Operators”

“Operational Security (OPSEC) for Control Systems”*

Instructor Led Courses

Cyber Security Who Needs It?

Control Systems Security for Managers

Solutions for Process Control Security

Introduction to Control Systems Security for the Information Technology Professionals

Intermediate Control Systems Security

Cyber Security Advanced Training and Workshop

Education & Training

*IOSS first place award

Industrial Control Systems Partnerships Industrial Control Systems

Joint Working Group (ICS-JWG) formed under the National Infrastructure Protection Plan framework to engage government and private sector control systems stakeholders

Private Sector Council

Government Council

Vendor Council

International Community

Cyber Security is a Shared ResponsibilityReport cyber incidents and vulnerabilities

www.us-cert.gov

Or send email to:

soc@us-cert.gov,

ics.cert@dhs.gov

Or call:

888-282-0870

Get more information at:

www.us-cert.gov/control_systems

Partnerships – Industry

37

Definition - Industrial Control SystemThe term Industrial Control System (ICS) refers to a broad set of control systems, which include:

SCADA (Supervisory Control and Data Acquisition) DCS (Distributed Control System) PCS (Process Control System) EMS (Energy Management System) AS (Automation System) SIS (Safety Instrumented System) Any other automated control system

Los Angles Traffic LightsEvent: Aug 21, 2006 Disgruntled traffic engineer hacked into the city's traffic control computer

Impact: Shut down traffic signals at four critical points in the road network, causing crippling delays

Specifics: Thought to have been part of a pay-bargaining procedure between employers and the Engineers and Architects Association

Recovery time: Four days until return to normal

operations

Lessons learned:

Do not underestimate the insider threat

Ensure separation of duties and auditing

Change passwords regularly

Texas City Explosion 3/23/05

Lessons learned:• Key alarms, indicators, and control

logic must be protected from cyber subversion

Event: An explosion occurred during the restart of a hydrocarbon isomerization unit.

Impact: 15 workers Killed, 180 Injured

Specifics: At approximately 1:20 p.m. on March 23, 2005, a series of explosions occurred at the BP Texas City refinery during the restarting of a hydrocarbon isomerization unit. Fifteen workers were killed and 180 others were injured. The explosions occurred when a distillation tower flooded with hydrocarbons and was over pressurized, causing a geyser-like release from the vent stack.

CSX Train Signaling SystemEvent: Aug, 2003 Sobig computer virus was blamed for shutting down train signaling systems

Impact: The virus infected the computer system at CSX Corporation’s Jacksonville, Florida, headquarters, shutting down signaling, dispatching, and other systems

Specifics: Ten Amtrak trains were affected

Recovery time: Train service was shut down or

delayed for six hours

Lessons learned: Critical patches and Anti-Virus

needs to be applied and updated regularly

Defense-in-depth strategies, Firewalls

Isolate control networks from corporate networks

Taum Sauk Water Storage Dam FailureEvent: Dec, 2005 Dam suffered a catastrophic failure

Impact: Billion gallons of water was released 100 miles south of St. Louis, Missouri

Specifics: Malfunction in gauges affected automated monitoring system

Lessons learned: Calibrate instrumentation

regularly Add fail safe redundancy to

critical safety systems. Update contingency plans

Recovery time: Replacement Dam scheduled for

completion in the Fall of 2009

43

Control Systems Cyber Security Defense in Depth Strategies

Creating Cyber Forensics Plans for Control Systems

Good Practice Guide on Firewall Deployment

Hardening Guidelines for OPC Hosts

Mitigations for Security Vulnerabilities Found in Control System Networks

Recommended Practice for Patch Management of Control Systems

Securing Control System Modems

Securing WLANs Using 802.11i

Securing ZigBee Wireless Networks in Process Control System Environments

Using Operational Security (OPSEC) to Support a Cyber Security Culture in Control Systems Environments

Recommended Practices

http://csrp.inl.gov/Recommended_Practices.html

Significance of ICS Many of the processes controlled by computerized control

systems have advanced to the point that they can no longer be operated without the control system.

Control systems are migrating away from their traditional shared and unrestricted configurations to more secure ones

Currently, a cyber attack on Industrial Control Systems is one of the only ways to induce real-world physical actions from the cyber realm

Programmable Logic Controller (PLC) based systems

Ties back to integrated systems

Can control critical systems

Usually remote systems

Example systems

Railcar loading/unloading

Chemical loading/unloading

Water treatment

Conveyer/shipping

Hazardous Materials storage/filtering

Industrial Control Systems

Harrisburg Pennsylvania Water System Event: Oct, 2006 Foreign hacker penetrated security at a water filtering plant

Impact: The intruder planted malicious software that was capable of affecting the plant’s water treatment operations

Specifics: The infection occurred through the Internet and did not seem to be an attack that directly targeted the control system

Lessons learned: Secure remote computers Defense-in-depth strategies,

Firewalls & Intrusion Detection Systems

Critical patches and Anti-virus needs to be applied and updated regularly