Electricity for Free? - Black Hat Briefings...SCADA Vulnerability Trends •! Our team has performed over 120 security assessments of critical infrastructure facilities such as electric

© Copyright Red Tiger Security – Do not print or distribute without consent.

Electricity for Free? The Dirty Underbelly of SCADA and Smart Meters

1

generation (DCS)

transmission (EMS)

distribution (DMS, AMR, AMI)


next hour of your life…

•! High level review of how the electric grid works

• ! the role SCADA systems play in the generation, transmission, and distribution of electric power

• ! typical network diagrams and data flow for each part of the process

•! SCADA VA Assessment Methodology 101

• ! 6-layer approach to ensure all system components are checked

• ! “point-click-scan” == system shutdown

•! SCADA and Smart Grid Vulns

• ! SCADA Vulnerabilities from a data set of over 38,000 vulnerabilities from live SCADA assessments

• ! Vulnerabilities with new SMART Grid Technology

2


random thoughts

•! Obama was quoted as stating a figure of $1 trillion lost last year to cybercrime—a bigger underworld than the drugs trade

•! Banks and other companies do not like to admit how much data they loose. In 2008 alone Verizon recorded the loss of 285 million personal-data records, including credit-card and bank-account details, in investigations conducted for clients.

•! SCADA systems that control all critical functions required for civilization (water, food supply, power, fuel, etc…) are far less secure than Enterprise IT systems.

3


…more random thoughts

•! SCADA system administrators can no longer hide behind obscurity

• ! Most common SCADA protocols are one Google-click away – try it at home for yourself “Modbus Protocol Specification”

• ! Fuzzing / hacking SCADA is simplistic

• ! Malware writers are already targeting SCADA

• ! July 15th: “Trojan makes database queries that point towards the WinCC

SCADA system by Siemens”

• ! Smart meter hacking kits rumored to be released in other countries…

•! Remember that little APT (Advanced Persistent Threat) problem… well that impacts SCADA too

4


5

So are we winning the cyber war?

…hard to when we are asleep…

© Copyright Red Tiger Security – Do not print or distribute without consent. 6

…especially with important stuff…


so let’s learn how the stuff works

• ! power generation

• ! power transmission

• ! power distribution

•! Disclaimer:: Any network diagrams contained here are only typical / common architectures and not reflective of any one particular system.

7


power system overview

8


first choke point

DCS firewalls

typically weak and often contains real-

time ICCP connections to other

systems

DCS operator

stations, servers, and applications

typically not hardened. DCS

controllers prone to

weak stacks.


first choke point

EMS firewalls

typically weak.

EMS systems integrate data from

many diverse systems,

including Internet sources

RTUs and IEDs at the

substation level can drop load to sections

of entire cities


first choke point

DMS firewalls

typically weak.

DMS systems integrate data from

many diverse systems,

including Internet sources. DMS also

have load shedding capabilities.

AMR systems often

integrated with DMS systems and

Enterprise IT applications for

billing, etc..

Smart

meters have known flaws

RTUs and IEDs can

drop load to entire neighborhoods


•! Cyber security standards and regulations, including NERC CIP and ISA S99, are calling for specific perimeters or security levels

•! Access between security levels must be controlled, monitored, and logged.

•! At least now we have the start of a model to go by for securing critical infrastructure.

12


SCADA VA Methodology 101

• ! Over the past 10 years, we have built a proven process for safely conducting security assessments on live SCADA systems

• ! Follow a logical 6-layer approach that covers all system components in the following areas:

1.! Physical Security

2.! Network Infrastructure

3.! SCADA DMZ

4.! Control Room Assets (Servers/Hosts)

5.! SCADA Communications and Protocols

6.! Field Devices

13




quick keys to a successful SCADA assessment

•! Know thy tools

•! Actively scanning SCADA systems and devices can cause system outages

•! Use a passive approach

•! Capture traffic, config files, and data for offline analysis

•! If possible scan secondary systems, test systems, or development environments

16


SCADA Vulnerability Trends

•! Our team has performed over 120 security assessments of critical infrastructure facilities such as electric power generation plants, transmission energy control centers, chemical plants, water plants, and oil/gas production, refining, and pipeline systems

•! Vulnerability analysis and classification conducted under research project facilitated by INL and funded through the DHS Control Systems Security Program contract #240704

•! ISA99 architecture model used to classify where the vulnerabilities were discovered in the systems

17



data source – what was collected?

• ! From mid-2002 to 2008, vulnerability data was stripped of any client information and the raw vulnerabilities were captured in a database

• ! Vulnerability ID (auto-numbered from entry number 1)

• ! Vulnerability Title (title for the vulnerability)

• ! Security Zone or Location (location based on the ISA99 model where the vulnerability was located)

• ! Disclosure Date (date when vulnerability was disclosed)

• ! Discovery Date (date when vulnerability was discovered by the team and entered into the database)

• ! Days Between Disclosure and Discovery (time between disclosure and detection)

• ! Vulnerability Detailed Description

• ! Vulnerability Suggested Remediation Steps

19


We Don’t Need No Stinking SCADA 0-Days

•! avg. # of days between vulnerability disclosure and discovery • ! all field data was exported from the database to an excel spreadsheet

containing over 38,000 rows, and much of the analysis had to be performed manually

•! since we captured when the vulnerability was disclosed in the public, and also captured when the vulnerability was discovered and entered into the database, we were able to perform a simple diff against these two fields

•! vulnerabilities that were never disclosed in the public were thrown out of this particular exercise since negative or zero entries would throw off the calculations

•! the maximum number of days between when a vulnerability was disclosed in the public and when it was found during an assessment was over 3 years!

• ! the average was 331 days, or close to 1 year. this means that on average most SCADA and process control environments contained latent vulnerabilities, probably with known already-compiled exploits, and were not discovered until almost a year later, and would not have been discovered had they not decided to have a security assessment performed on their system.

20


where are most of the vulnerabilities being discovered?

21


Operational or SCADA DMZ most vulnerable

• ! Almost half of the total vulnerabilities were found in the DMZ between the Enterprise IT and SCADA systems

• ! Often we find that SCADA system owners struggle with which group in their company has the ownership and responsibility for maintaining the systems in this part of the network

• ! The Operational DMZ network is the first stepping-stone from the Enterprise IT network, and is the most common threat vector for attacks against SCADA systems

• ! Now we have found the “Perfect Storm” whereby the most connected area of the SCADA system also contains the most vulnerabilities, and is often overlooked by system administrators

22


why is this so important…

•! The Operational or SCADA DMZ network is the last line of defense before any traffic hits the SCADA and Industrial Process Control systems

•! In many cases, the servers, workstations, and applications in this middle area are all authorized and trusted by the SCADA systems

•! By dissecting the vulnerabilities in this level of the network, we can determine how the vulnerabilities at this level in the architecture can be exploited

23


systems impacted at the Operations DMZ zone

24


SCADA, how can I own thee, let me count the ways

25



workstation HMI vulnerabilities ranked by OS

27


want to run that critical plant on windows?

28


only logged 105 controller LAN vulnerabilities, but QnX showed up as the most typical source

29


interesting security findings on control system networks

30

•! VOIP (Voice over IP) Systems

•! Network Video Recording Devices

•! Network Surveillance Equipment and Software

•! Adult Video Directory Scripts

•! Online Dating Service Databases

•! Advanced Forensics Format (AFF) archives

•! Gaming Software Servers

•! aGSM - a freeware game server info monitoring utility

•! Alien Arena 2006 Gold Edition

•! Counter Strike

•! Brood Wars

•! Battlefield 1942 Server and Clients

•! Quake 2 and Quake 3 Game Servers found in Supervisor HMI LAN

•! Soldier of Fortune II

! ! Software license cracking executables (CD-key

generators)

! ! Torrent client software on Supervisor HMI LAN

! ! Paging Software Server (i.e. Air Messenger Server connected to both the SCADA and Internet for

SMTP relay out)

! ! America Online Clients

! ! MP3 Music and Video Playing Software including

iTunes

! ! Streaming Music and Radio software with

vulnerabilities

! ! BitTorrent Clients (for peer-to-peer file sharing)

! ! MSN and other IM chat clients

! ! Anonymous FTP Servers running waiting for connections


but wait…there’s more

31

! ! Windows NT found installed on hosts in

Supervisory HMI LAN (no longer

supported by MS)

! ! Windows Vista found used as OS for

operator consoles in Supervisory HMI

LAN

! ! IRC Chat Servers found installed on

hosts in the Operational DMZ LAN

!! Nintendo Entertainment System (NES)

Game Simulator

!! Netscape Browser vulnerabilities

detected in Supervisor HMI LAN

! ! Multi-function Printer/Fax/Scanner

device vulnerabilities

! ! Botnet code and Remote Command/

Control malware

•! Apache Web Servers and Linux hosts un-patched for over 2 years

• ! APC Battery Backup UPS systems with vulnerable Web Interface

• ! Several web blog site engines running in control system DMZ

•! Office grade Linksys, Belkin, and D-Link WiFi devices on Supervisory HMI LAN

• ! IM clients found installed and contained vulnerabilities on Supervisory HMI LAN

•! Windows 95 found installed on hosts in Supervisory HMI LAN (no longer supported by MS)


SCADA Vulnerability Summary / take away points

•! 331 = the average time in days between when a vulnerability was disclosed in the public versus when it was discovered in an industrial control systems assessment

•! the intermediate Operations DMZ network that sites between the Enterprise network and the industrial control systems had the most vulnerabilities attributed to its zone

•! web server and back-end database vulnerability findings comprised the largest number of vulnerabilities found in these Operations DMZ network – we need more web app testing!

•! number of client workstation vulnerabilities also increased deeper into the real-time operations networks, thus proving we still have a patch problem with SCADA systems

•! vulnerabilities with Windows operating systems or Windows applications also accounted for the overwhelming majority of vulnerabilities for systems in the Supervisory HMI LAN

•! almost every assessment uncovered unnecessary software installed on the SCADA systems, and in some cases this included botnet and malware code

32


still think we’re doing a good job at securing our critical infrastructure?

33


but wait… we have NERC CIP to save the day !

34


35

controls are often bypassed…


so are we collectively feeling better now?

•! NERC CIP at least forces the hand of Electric Power Utilities to implement a common baseline set of minimum physical and cyber controls

•! Most controls in NERC CIP are logical, make sense, and map back to other existing best practices and International standards

•! So then why are so many Utilities declaring ZERO (0) critical assets, or playing games with wording in the standard to make loopholes to de-list critical assets?

•! Now even if everyone was playing fair and in compliance, now insert shiny new nifty Smart Grid technology into the mix…


couple of thoughts about Smart Grid tech…

•! Ami / AMR systems have similar vulnerabilities as SCADA Systems

•! History is about to repeat itself

•! Old Threats, New Impacts

•! Attacks at the communication layer

•! Attacks at the device layer


we’ve seen this before… • ! Perimeter issues > these systems are interconnected with business

applications (billing, work-order, account management systems, etc..), AND also often connected to operational SCADA and Energy Management systems for load shedding and remote tripping

•! Back-end Server/Application issues > similar web and database app vulns as business applications, less secure implementation of protocols, and old versions of application frameworks

•! Too much trust in the Protocol > Most AMI / AMR vendors are simply trusting that the 802.15.4 protocol security implementation will save them, and have not given much thought about scenarios when a communications mote is compromised

•! End Devices have limited resources / weak stacks > The meters themselves do not typically have the resources to handle security features. Basically, the hardware cannot handle more computationally demanding processes, like upgrading their encryption handling capabilities once deployed. Limited tamper-detection capabilities cited, but not found operational in testing.


field life of 15-20 years – Déjà Vu

•! Due to high implementation costs, most AMI / AMR projects have long ROI cost recovery models, and are designed to operate for up to 20 years without requiring system upgrades

•! Combine this with patching and firmware upgradability issues, and we are building into place the conditions that created much of the issues with SCADA and Process Control Systems Security

• ! “Once these devices get deployed, they aren't going to get upgraded due to cost unless there is a major, crippling vulnerability found in them, and people are shamed into fixing it.” – quote by Jacob Kitchel (security researcher)

• ! “All it will take is someone to get bored and go shut a city down by telling all the communication motes that everyone didn't pay their bill, then half flash the firmware and brick them all.”

- quote by Nick DePetrillo (security researcher)


old threats, new impacts

• ! Data Enumeration (read-time grid data)

• ! Host Enumeration (what systems can we connect to?)

• ! Service Enumeration (what services are exposed?)

• ! Change Data on the fly (can the data be manipulated in flight?)

• ! Steal accounts and passwords (system admin access anyone?)

• ! Damage core system components (cause meters to fail…)

• ! Denial of Service (PING FLOOD, Malformed Packets, etc…)


Man-in-Middle Packet Capture


Write over any data in the stream (real time)

last24KWh=250;

Change

usage or

billing data…

last24KWh=25.0;


sometimes it is not even that hard…

43


Bricking PLCs and RTUs are relatively easy… “Smart Meters” have similar stack issues

PING Flood Often

Results in Faulted PLC Processor.

PLC Looses

Configuration, and

Must be Connected Locally with a Serial

Cable to Upload Configuration.


simple ping flood attack…

45


denial of access

•! Embedded device has a Login/Write Access password option

•! 16 character limit

•! Vendor specific Modbus/TCP function code

•! Password stored in the Flash of the controller

•! “This procedure cannot be undone if you forget the password. The device must be sent for repair”

46


thirsty for more? training is available…

•! Week of 6 September: • ! http://www.redtigersecurity.com/uk-fall-2010/

• ! Kent, UK

• ! If you click on the link provided on our web site, students can register for the course online

•! Week of 7 October: • ! http://www.sans.org/eu-scada-security-summit-2010/ • ! London, UK

• ! This event is being handled through SANS, so students have to register through the SANS web site. There is a register button on the page, and students can pick and select which optional training they are interested in. Our course is the one listed as: SPECIAL HOSTED: SCADA Security Advanced Training Oct 7th-11th

47


last few comments…

• ! Power generation, transmission, and distribution systems all require functioning SCADA, EMS, and DMS systems that are available with 24x7 uptime

• ! In the past, these systems were isolated systems that used serial protocols and obscure system components

• ! Recently, SCADA and Control Systems have evolved to the point where they are deployed with network infrastructure components used by Enterprise IT networks (Cisco, Juniper, 3COM, etc.. They also leverage the same Microsoft operating systems and .NET application frameworks

• ! However, they do not have the security features taken for granted on Enterprise IT systems

48


last few comments…

• ! The NERC CIP security compliance regulations is making a small dent in the problem, but not enough to hold back a determined attacker

• ! With the advancement of the “Smart Grid” and AMR systems, without the proper security precautions, the electric grid is now more vulnerable than ever

• ! Vulnerability Assessments are still the best first step in securing critical infrastructure

• ! The research that we, and other independent security firms, have performed on SCADA, Smart Meters, and AMR systems expose vulnerabilities that can lead to a situation whereby electricity is free… for those who have the intent and motivation

49


contact info / q & a

Jonathan Pollet, CAP, CISSP, PCIP

Founder, Principal Consultant

Red Tiger Security, USA

office: +1.877.387.7733

mobile: +1.281.748.6401

fax: +1.800.864.6249

[email protected]

www.redtigersecurity.com

50

Documents

Electricity for Free? - Black Hat Briefings...SCADA Vulnerability Trends •! Our team has performed over 120 security assessments of critical infrastructure facilities such as electric