ISSA CYBERSECURITY LEADERS GLOBALLY c.ymcdn.com/sites/ · PDF file · 2013-01-08NERC Critical Infrastructure Protection Challenges for the Power Industry 12 – ISSA Journal | June

NERC Critical Infrastructure Protection Challenges for the Power Industry

12 – ISSA Journal | June 2012

ISSA DEVELOPING AND CONNECTING

CYBERSECURITY LEADERS GLOBALLY

Abstract

Any information security governance effort in US critical infrastructure industries must include successfully meeting industry and governmental standards and requirements such as the North American Electric Reliability Corporation’s Critical Infrastructure Protection (NERC CIP) requirements in the electric power sector. Furthermore, the governance program must provide for the hands-on management of the technical and administrative controls that ensure opera-tional compliance and security in support of organizational objectives. Focusing on NERC CIP and the power industry, this article presents lessons learned about the most common technical and administrative risks to security and standards compliance as collected from vulnerability assessments and compliance audits across the US power industry.

The IT Governance Institute describes information se-curity governance as consisting of the leadership, or-ganizational structures, and processes that safeguard

information with several basic outcomes or deliverables.1 Those outcomes, highly summarized, focus on (1) aligning an enterprise’s information security effort with its business strategy to meet its organizational objectives, (2) managing risk in an efficient and measurable fashion in support of those objectives, and (3) delivering value in security investments.

By its very nature, governance must have an internal focus as it addresses internal resources, processes, and property. However, it also has a growing external component that must be addressed: industry and government security regulations and standards. Although there are differences across indus-tries, most effective information security governance pro-grams must have an effective external compliance compo-nent because, with significant fines for noncompliance, the result of an inefficient and ineffective compliance program can be every bit as financially disastrous as failing to meet an

1 IT Governance Institute, Information Security Governance, Guidance for Boards of Directors and Executive Management 2nd Edition, 2006.

Focusing on North American Electric Reliability Corporation’s Critical Infrastructure Protec-tion and the power industry, this article presents lessons learned about the most common technical and administrative risks to security and standards compliance as collected from vulnerability assessments and compliance audits across the US power industry.

By Jacob Kitchel and Stephen McIntosh – ISSA member, New England, USA Chapter

NERC Critical Infrastructure Protection

Challenges for the Power Industry

©2012 ISSA • www.issa.org • [email protected] • All rights reserved.

• CIP-002 – Cyber Security - Critical Cyber Asset Identifi-cation

• CIP-003 – Cyber Security - Security Management Con-trols

• CIP-004 – Cyber Security - Personnel & Training

• CIP-005 – Cyber Security - Electronic Security Perimeter(s)

• CIP-006 – Cyber Security - Physical Security of Critical Cyber Assets

• CIP-007 – Cyber Security - Systems Security Management

• CIP-008 – Cyber Security - Incident Reporting and Re-sponse Planning

• CIP-009 – Cyber Security - Recovery Plans for Critical Cyber Assets

Compliance with the NERC CIP standards is enforceable in the US and in several Canadian provinces. If a US electric util-ity is audited and found to be non-compliant with the stan-dards after a designated compliance deadline, it can be fined up to $1 million per day per violation.4 Thus, utilities com-mitting to successful NERC CIP compliance not only stand to gain the benefits of operational excellence – improved and predictable operating margin and reduced downtime – but they also reduce the risk of being financially penalized.

Vulnerability assessment and compliance mock audit lessons learned

With responsibility for executing and documenting cyber penetration tests, vulnerability assessments, and NERC CIP compliance gap analyses and mock audits, the authors have acquired a thorough understanding of the state of cyber se-curity in the electrical utility sector. Besides our own direct, hands-on experience, we also draw upon our organization’s broader real-time process control and Supervisory Control and Data Acquisition (SCADA) industry cyber security expe-rience to offer important lessons learned from well over 100 critical infrastructure assessments.

In a mock audit, security consultants walk through the NERC CIP standards and Reliability Standards Audit Work-sheets (RSAWs) just as a real auditor would. This process re-veals potential points of non-compliance for which we can recommend mitigation measures to achieve compliance and avoid potentially substantial fines. Below, we present the top threats to NERC CIP compliance across many consulting en-gagements. This information has been thoroughly sanitized to protect the identity of clients.

Top compliance challenges and best practicesThrough many NERC CIP mock audits, we identified a pat-tern of gaps that appeared more frequently across electric

4 Sanction Guidelines of the North American Electric Reliability Corporation, http://www.nerc.com/files/Appendix4B_Sanctions_Guidelines_Effective_20080115.pdf.

internal business objective. And this is especially true with the North American Electric Reliability Corporation’s Criti-cal Infrastructure Protection (NERC CIP) standards for in-formation security in the power industry.

The bottom line for any successful information security governance program is that, at the end of every day, it must have effectively prevented asset and revenue loss by ensuring nonstop business process operation, by preventing theft and damage to both property and reputation, and by avoiding costly noncompliance fines.

In this article we do not discuss how senior management should create frameworks for governance, but how the people responsible for day-to-day utility operations can take cyber-security measures that will help keep the power flowing, prevent damage, and maintain regulatory compliance. We seek to help electric utilities fortify their cyber security and standards compliance posture by presenting lessons learned across the electric power industry. The result of hands-on consulting engagements – vulnerability assessments, pen-etration tests, and NERC CIP mock audits, we summarize security gaps found most frequently across the electric utility industry that jeopardize NERC CIP compliance and, there-fore, reliable operation of the North American power grid.

NERC CIP standardsThe North American Electric Reliability Corporation,2 a nonprofit corporation, was created after the Energy Policy Act of 2005 authorized the Federal Energy Regulatory Com-mission (FERC) to designate a national Electric Reliability Organization (ERO). NERC, the ERO, is chartered to work with electric utilities to implement mandatory reliability standards in North America.

One fundamental requirement for reliable power is secure control systems. This is clearly recognized in NERC’s Reli-ability Functional Model, a framework for the development and implementation of NERC’s reliability standards for the bulk electric system. A key part of that model is Reliability Assurance, a basic system function that spawns the Critical Infrastructure Protection3 or CIP standards, a set of cyber security standards and requirements. To foster the imple-mentation of secure cyber protections in electric utilities, the CIP standards focus on securing the computers and networks critical to sustained electric power flow. Specifically, NERC CIP provides standards for a utility’s electronic security pe-rimeter (ESP), critical assets, security controls for the systems and networks, personnel background checking and training, physical security of the ESP, vulnerability assessments, inci-dent response, and incident recovery.

NERC Critical Infrastructure Protection

• CIP-001 – Sabotage Reporting

2 http://www.nerc.co.

3 http://www.nerc.com/page.php?cid=2%7C20.

July 2012 | ISSA Journal – 13

NERC Critical Infrastructure Protection Challenges for the Power Industry | Jacob Kitchel and Stephen McIntosh


corporate IT resources temporarily or by hiring more control system IT resources.

In addition, utilities should investigate work-saving compli-ance data collection, measurement, and reporting solutions and then select one that meets their particular requirements. Such solutions can greatly reduce the data collection and compliance assessment work load, particularly with respect to the more cyber-centric NERC CIP-005 and 007 standards.

Insecure perimeter firewall and router configurations

Discouraging unauthorized access to your electronic security pe-rimeter (ESP) requires secure firewall configurations and rules.

Standard and requirement:

CIP-005 R2.2

The gap:

Firewalls and routers are typically the access points to a util-ity’s ESP. The rules for routing traffic and the transparency of the traffic must be examined. For example, some organiza-tions have not confirmed that the firewalls and routers are configured such that rules deny and log the traffic that is not predefined. While these rules need to be documented and stored for CIP-005 R2.2, many older firewalls typically lack the audit capabilities needed for NERC CIP compliance.

In addition, router access control lists (ACLs) – and the ports and services they enable – often allow all traffic from vari-ous devices, network groups, and object groups which reside outside of the ESP into the ESP. We have found that ACLs are often too permissive and should be restricted as much as pos-sible to the hosts and needed ports and services.

A third perimeter security consideration is the degree of transparency of traffic passing through access points. Many organizations, for example, allow clear-text traffic such as telnet, rcp, rlogin, tftp, and ftp traffic through ESP network firewalls, switches, or routers. Clear-text services could also allow an attacker to easily obtain credentials and other infor-mation through packet capture. The attacker could then use these valid credentials to further exploit the system, perhaps using a man-in-the-middle attack.

Best practice:

The utility control staff should be able to generate reports of the firewall rules and the ACLs of the internal routers at any time and review them on a regular basis. This report will help achieve compliance by documenting the access allowed across the ESP. The report should include the source and des-tination IP information, the ports allowed, any time param-eters, and an easily understood description of the access. It can also include the approval of the access request. One of the biggest challenges for many utilities is to document the existing rules. Be sure to allow sufficient time for the review of existing rules.

“All” or “any” type rules should be reviewed and pared down to the essential ports and services which are required for op-eration of any system. While it is possible that all ports and

utility clients. As such, these issues introduce significantly more risk for a utility because auditors and ill-intentioned attackers are both more likely to be on the lookout for these common vulnerabilities.

These potential gaps in compliance arise from issues with personnel, access control devices such as firewalls, software patching practices, network isolation, access credentials, ports and services, and unnecessary software.

Inadequate security/compliance staffing

Effectively addressing security and compliance standards re-quires dedicating adequate resources to the task.


All standards

The gap:

An essential first step that is not often taken is applying suf-ficient resources to successfully meet the goal. When a util-ity commits to the goal of NERC CIP compliance, the com-pliance management work, such as coordinating audits and managing technical feasibility exception (TFE) generation and submission, can be extremely labor intensive, although there are tools designed to render compliance management much less onerous.

This is addressed to some degree by the standard audit pro-cess which requires the participation of subject matter ex-perts (SMEs). The SME for physical security is typically not the SME for configuration management, and neither of those is the SME for personnel training, and so forth. At a mini-mum, we recommend that the following staff be assigned to NERC CIP compliance efforts:

• A compliance or audit manager

• As many SMEs as are necessary to adequately cover the standard areas

• A SCADA cyber systems operations manager

We have repeatedly found that the amount of daily work to ensure CIP compliance is underestimated and, thus, proves too large a task for the resources assigned at the start. A large portion of this work involves data collection and detailed report generation. Other notable tasks include TFE develop-ment and submission, hosting auditors, and more.

For example, NERC CIP-007 requires that all ports and ser-vices be identified on Critical Cyber Assets (CCAs). Ports not needed for operation must be closed and a report of ports and services must be generated for monitoring and audit purpos-es. Implementing a consistent port and service data gathering and review methodology on a frequent time schedule – such as weekly or monthly – can be very costly, particularly if ac-complished manually.

Best practice:

To address this issue, utilities should carefully consider the work needed to prepare for an audit. They should then as-sign an appropriate number of people, perhaps drawing from

14 – ISSA Journal | July 2012



applied patches have been critical in nature and could allow an attacker who gets past the first line of defense to easily gain access to the control network. In addition, malware is quite prevalent so this is an unacceptable approach from a malware protection standpoint.

By taking advantage of an inconsistent patch policy, an at-tacker would be free to leverage gained access and use any number of easily obtainable, reliable exploits to take control of unpatched machines. This ultimately enables the attacker to gain the same privileges as the very people who provide technical support or have access to the control system from the corporate network.

Best practice:

Electric utilities should review their patch policy with an eye towards improving timeliness, regularity, and testing. Time-liness and regularity are critical because of the vulnerability of unpatched systems. Testing is important because the sys-tems being patched are critical control systems and should remain stable and available through the patching process.

A common recommendation is that utilities work with their SCADA, Distributive Control Systems (DCS), and Emergen-cy Management Services (EMS) system vendors on a regu-lar basis to determine which patches can be applied. These patches should then be tested in a development or test en-vironment prior to implementation on production systems.

Monitoring the patch level of systems on a regular basis be-comes significantly simpler and more accurate with a com-pliance tool that collects and reports on patch inventory. Additional security and compliance benefits accrue if the compliance tool can compare actual patch levels on a device to patch levels on a baseline or “gold standard” device.

Inadequate separation between corporate and plant control networks

Keeping skilled attackers from traversing from a corporate net-work to a control network requires a strong network architecture that includes a control DMZ and re-architected dual-homed hosts.


CIP-002 R3 CIP-005 R2.2

The gap:

Plant information systems (PIS) and historians aggregate control system information so the business can better direct operations toward increased profitability and productivity, and to conduct effective business planning. They also rep-resent paths between the corporate and control system net-works, and many utilities have implemented such systems.

Closer examination of these systems often reveals host sys-tems with more than one network card connecting the host directly to more than one network at a time (figure 1). A vulnerability in a dual-homed machine can offer an attacker

services are required, experience and practice suggest that the number of actual ports and services required is a very small subset of the possible 65,535 TCP or UDP values. In addi-tion, the rules should specifically state the ports and services which are required.

Regarding clear-text protocols such as telnet, rcp, and rlogin, any unencrypted protocols should be phased out and re-placed with secure administration protocols such as HTTPS,

SSH, or SCP. With wireless, all traf-fic should be at least encrypted with WPA2 AES-based encryption with a strong5 key on wireless network devices used to bridge physical net-work segments. The network name should not be broadcast in order to make network discovery harder for an attacker. Disabling SSID broad-casts will not completely prevent an attacker from discovering the net-work name, but it will require sig-

nificantly more time and effort for its discovery.

Here also, a compliance management tool that collects and reports on firewall data can save significant time and improve data accuracy. Such reports should list the firewall rules for a given device, on different devices, and at two different times to reveal changes over time. It should also support configura-tion and control management to aid in the timely review and tracking of firewall rules.

Insufficient patching

Assessment and implementation of the latest software patches is required to help prevent malicious, unauthorized incursion into your electronic security perimeters and critical cyber assets.


CIP-007 R3

The gap:

Vendor-supplied software patches frequently fix security vulnerabilities and improve usability or performance. Our assessments frequently identified numerous missing service packs and patches on the control system workstations and servers. In fact, the number of missing patches averages over 20 per site. Patches for network devices and third-party ap-plications are equally important to the security posture of SCADA networks.

Typically, systems are brought up to date when deployed. Subsequent patches are reviewed for applicability and com-patibility, and are then applied as necessary. Some of the un-

5 FIPS 197 specifies that all sensitive but unclassified government documents should be encrypted with AES which comes in three key-length versions: 128, 192, and 256 bits. All three are considered adequate for federal government applications. - NIST SP 800-57, Recommendations for Key Management – Part 1: General; Sect. 4.2.2.1.

In fact, the number of missing patches averages over 20 per site.




Reviewing network diagrams is the obvious way to identify network configuration issues, but larger networks change frequently, making network-diagram accuracy short-lived. Maintaining current network diagrams manually is unnec-essarily laborious and error-prone. Much greater efficiency, cost effectiveness, and accuracy would accrue from the de-ployment of a compliance management tool that automati-cally tracks device configurations (including network inter-faces), periodically compares them with a baseline, and then issues an alert when actual data deviates from the desired baseline configuration.

Weak, repeated passwords

Authorized access to accounts in the ESP must require strong passwords.


CIP-005 R4.4 CIP-007 R5.3

The gap:

Weak passwords represent a common and severe vulnerabil-ity, including across systems and network devices, that can be exploited to gain access. Pen testers will uncover the overall use of weak passwords after successfully compromising the directory service domain controller and cracking the hashed passwords of nearly all the domain accounts (including ad-ministrator level accounts).

Best practice:

Implementing and enforcing strong password policies across all environments will help ensure strong security. Strong password policies should be applied at the host and server lev-

direct access from the corporate network to the control net-work. Furthermore, these connections are often not audited since they do not traverse normal network infrastructure de-vices.

A typical response to this network configuration risk is to implement a control systems DMZ (figure 2). However, the control system’s ESP may well need to incorporate some DMZ devices.

Best practice:

A common recommendation advises clients to re-position dual-homed devices in the network or consider the imple-mentation of a control systems DMZ to provide greater ac-cess control and auditing of the connections into and out of the device. The DMZ allows the access from corporate, but maintains a level of separation from the control network. This separation will help prevent an attacker from freely ac-cessing the control network in the event a security hole in the PI server or in terminal services is found.

If systems and data network machines exist with interfaces on the business network, they should be placed in a DMZ, which serves as the only gateway to the rest of the CCA de-vices. While not explicitly a NERC CIP requirement, a con-trol systems DMZ is considered a security best practice and is recommended by NIST SP800-826 to enhance the security of a control system. The recommendation to include DMZ switches in the ESP, however, does fall under the NERC CIP regulations. Specifically, the perimeter of the ESP must in-clude these switches and hosts as defined by NERC CIP-002 R3.

6 NIST SP 800-82, Guide to Industrial Control Systems (ICS) Security, http://csrc.nist.gov/publications/nistpubs/800-82/SP800-82-final.pdf.

Figure 1 – Dual-homed Figure 2 – With DMZ




http://csrc.nist.gov/publications/nistpubs/800-82/SP800-82-final.pdf.

http://csrc.nist.gov/publications/nistpubs/800-82/SP800-82-final.pdf.

el with local security policies on Windows and by configur-ing the Pluggable Authentication Modules (PAM) on UNIX variants. In addition, a centralized authentication solution

such as Active Directory or LDAP should be considered to help en-force strong password policies and log access.

With respect to strength, NERC CIP recommends that passwords be a minimum of six characters long; consist of a combination of alpha, numeric, and special char-acters; and be changed at least an-nually, or more frequently based on risk. Some security experts advocate for even stronger pass-words that are 12 to 14 characters if permitted and that mix upper and lower case letters if the system recognizes case.

Regardless of password policies, a system that monitors and reports on the implementation of those policies adds significant value. A compliance system that actually examines the strength of individual passwords may be more intrusive than desired. However, one that at least automatically verifies that a given password policy has been implemented on a device is probably sufficient to deter-mine compliance posture vis-a-vis password policies.

Unnecessary third-party products installed with weak default configurations

Unauthorized traversal across an internal network is greatly hindered if unnecessary, weakly protected applications do not exist.


CIP-007 R5.2

The gap:

Accounts are considered “default accounts” if they were cre-ated by a vendor for maintenance or startup purposes. If left installed and available, these accounts can be used to access CCAs within a client’s ESP. Our team has often found ap-plications, database platforms, or other third-party software or firmware installed and running in default configurations with default accounts and default passwords still in place.

Microsoft, for example, creates a default administrator ac-count automatically that is both the most powerful and most risky account on a system. The password lockout policy does not apply to the administrator and it is most likely to be the first account an attacker would attempt to crack. An attacker who successfully cracks the administrator password could take complete control of the affected system and possibly the network.

In another example, numerous machines in an Active Direc-tory domain within the ESP have been found to be running the MS SQL Server services listening on port TCP 1433. These machines were also found to have the MS SQL Server “sa” account with a blank password. Such security oversights have been leveraged in penetration testing to execute administra-tor-level commands on various machines in order to gain ad-ministrative access.

Best practice:

The action for mitigating this compliance gap is clear: change any default user names to unique user names and change de-fault passwords to appropriately complex, unique passwords. With the Microsoft administrator and guest accounts, how-ever, renaming the original accounts and changing the text in the description to eliminate anything that indicates that these are the administrator and guest accounts is insufficient. Default administrator and guest accounts can be discovered regardless of renaming because the underlying SIDs of the accounts remains the same. Thus, best practice here is to add a customer-specific administrative-level account for each ad-ministrative user and disable both administrator and guest default accounts, where possible.

Collecting the current software inventory on a device could be done by running an installed applications report, but sig-nificant efficiencies can be realized with the use of a com-pliance management tool that automatically runs a device software inventory report on a predetermined schedule. The report would compare the actual software inventory on the device with a baseline inventory, highlight differences from the baseline, and then issue an alert on the difference.

Inadequate ports and services documentation

Documentation showing that only necessary ports and services are open on a CCA demonstrates commitment to complying with NERC CIP and to reducing the penetration opportunities for an attacker.


CIP-007 R2

The gap:

A NERC auditor expects open ports and services on CCAs to be documented so that compliance with the requirement to close unnecessary ports and services can be determined. As previously mentioned, unnecessary ports and services are often enabled by default when devices ship from vendors. It is not uncommon to find services such as name, comsat, talk, uucp, finger, time, echo, discard, daytime, chargen, rquotad, ruserd, spray, walld, and rstatd enabled by default. Any un-necessary services expose a device to vulnerabilities and at-tacks that would normally not be available if the services were not enabled.

Leaving unnecessary services running provides a potential path for an attacker attempting to compromise the system. So, by only running services and software required to run the

Regardless of password policies, a system that monitors and reports on the implementation of those policies adds significant value.




control system, the risk of attack is reduced. Utilities should work with their vendors to identify the ports and services re-quired for operation and disable unnecessary services. Un-fortunately, it is frequently very difficult to clearly document which ports and services are really necessary.

Best practice:

Inadequate ports and services documentation can be miti-gated by identifying all ports and services necessary for the normal operation of each server and applying them to all hosts that need access. Next, disable all services that are un-necessary for normal operations to reduce the attack surface of a device. This hardening process is industry best practice for securing critical systems.

Once ports and services are reduced to those required for normal and/or emergency operations, ports and services should be reviewed frequently to ensure that compliance is sustained. A compliance management solution that periodi-cally collects data on the open ports and services on a given device and then compares that data with a desired baseline will dramatically improve security and compliance sustain-ability.

ConclusionThis paper presents a compilation of the most common threats, in our experience, to achieving NERC CIP compli-ance and, in each case, discusses multiple actions that would help mitigate those threats to compliance. If you are a system administrator or security knowledgeable, none of these issues will come as a surprise to you. They are all basic practices that are considered the fundamental “blocking and tackling” of cyber security, so the success of your IT governance efforts absolutely depends on sound execution of these actions in a consistent and timely fashion.

This report also highlights where and how an automated compliance management tool can help mitigate these threats to compliance. Utilities can certainly continue to correct compliance issues the conventional “old fashioned way” and attempt to achieve compliancy of dubious accuracy and at high labor expense, or they can address nearly all these issues more rapidly, efficiently, and accurately with an automated compliance management tool that can be customized to meet specific requirements and address specific environments…today.

The decision to automate, however, leads to more questions about which tool or tools to use. There already exist individu-al tools for firewall rule monitoring, software patch manage-ment, and user monitoring. However, all of these tools are frequently offered by multiple vendors and are not designed for interoperability. Thus, a utility could choose a mixture of tools to help achieve compliance, but often this choice comes at a higher time and resource investment required to run the disparate tools. Tool management, already an important and significant task, can grow exponentially when using a differ-

ent tool for each requirement area, and the efficiency poten-tial of automation erodes.

In addition, multiple tools present a challenge for compliance auditors as report ease-of-use decreases with multiple, differ-ent report formats. In fact, manual report generation could easily achieve a more common look-and-feel for reports. Ta-ble 1 shows the relative impact of single versus multiple tools on:

• The efficiencies and accuracy of an automated compli-ance tool

• The ease-of-use inherent in reports with a common look-and-feel

(An up arrow indicates a positive impact and two arrows indi-cate greater relative impact.)

Multiple tools Single tool

Compliance efficiency

Audit report ease of use

Table 1 – Change from Manual Compliance Management

Whether through manual labor, through automation with multiple tools, or through automation with a single tool, the threats to NERC CIP compliance and, therefore, to bulk elec-tric system reliability must be addressed. We hope that this presentation of common threats and mitigating actions will help you create a sound foundation for a successful informa-tion security governance program.

About the AuthorsJacob Kitchel, Senior Manager of Security and Compliance at Industrial Defender, has experience with the most recent hack-ing techniques and has performed external and internal penetration tests of critical infrastructure networks as well as vulner-ability and risk assessments in SCADA and DCS environments. He is skilled in NERC CIP compliance gap analysis and in determining critical assets and critical cyber assets. Besides Industrial Defender, Jacob has worked in information security at Roche Diagnostics and In-fotex. He may be reached at [email protected].

Stephen McIntosh, CISSP, MBA, works in Development and Technical Communi-cations for Industrial Defender. His more that fifteen years in information security includes project management on power in-dustry compliance audits and security so-lution deployments at HP, Barclays Bank, and the US Department of Defense. Steve has held product management and techni-cal marketing positions at HP, nCipher, Certco, Broadcom, and eIQnetworks. He may be reached at [email protected].




Documents

ISSA CYBERSECURITY LEADERS GLOBALLY c.ymcdn.com/sites/ · PDF file · 2013-01-08NERC Critical Infrastructure Protection Challenges for the Power Industry 12 – ISSA Journal | June