Intellibind Top Ten Most Violated Standards Presentation 2011 01 27 (F)

WECC Compliance User Group Meeting

Top 10 Most Violated Standards in WECCApproaches to Compliance

February 9, 2011

Thanks to Los Angeles Department of Water and Power for sponsoring this session

Agenda

Speaker Intro Sources of Information Top 10 Most Violated Standards NERC and WECC Common Violation Findings Evidence and Proof of Compliance Details for 693 (Reliability) Standards

Primary Non-Compliance Factors Recommendations

Details for 706 (CIP) Standards Primary Non-Compliance Factors Recommendations

Introduction

Kevin Conway – Intellibind, LLC

26 Years in the Industry NERC Reliability Program NERC OC Representative NERC Functional Model

Workgroup NERC Standards Drafting

Team System Reliability

Manager Marketing and Trading NERC Certified System

Operator

Bill Addington – Intellibind, LLC

Over 20 years Cyber Security expertise

Involved in creation of original cyber security standard BS7799

Over 10 years with Electric Power Utilities

Principle author and speaker, NERC Cyber Security Workshop

Author of EPRI papers on Cyber Security

Former Interim Security Manager at ERCOT

Sources of Information

NERC published research and reviews

WECC documents Research into Violation Reports for

details and common causes Field Experience

Audit and readiness team experience across U.S.

Experience as SME’s in audit in various utilities in WECC

Interviews with utilities

NERC and WECC Top 10 Most Violated StandardsMay 1, 2009 to April 30, 2010 – Summary Table

1 2 3 4 5 6 7 8 9 10NERC PRC-005 CIP-004 CIP-007 CIP-001 EOP-005 CIP-003 FAC-008 CIP-006 FAC-001 CIP-002

WECC PRC-005 EOP-005 CIP-001 TOP-002 CIP-004 CIP-007 PER-002 EOP-001 CIP-003 COM-001

Violation RankRegion

Top Ten Most Violated WECC Standards

• PRC-005 Transmission Protection System Maintenance and Testing

• EOP-005 System Restoration Plans• CIP-001 Sabotage Reporting• TOP-002 Normal Operations Planning• CIP-004 Cyber Security – Personnel and training• CIP-007 Cyber Security – System Security

Management• PER-002 Operating Personnel Training• EOP-001 Emergency Operations Planning• CIP-003 Cyber Security – Security Management

Controls• COM-001 Telecommunications

Common Violation Findings

Poorly written or non-existent procedures and processes.

No evidence of testing procedures.

Documentation – inability to prove that processes where in place to prove compliance with the standard.

Common Violation Findings

Poorly written RSAWs RSAWS do not clearly describe

processes RSAWS do not describe the

relevance of evidence SME interviews do not match RSAW

statementsUnder or over documented

evidence

Evidence – Proof of Compliance

Procedures and Processes

Output of Procedures and Processes

Proof of Compliance (Evidence)

Top Most Violated Operational Standards (693)

Top Most Violated WECC Operational Standards

CIP-001 Sabotage Reporting COM-001Telecommunications EOP-001 Emergency Operations

Planning EOP-005 System Restoration Plans PER-002 Operating Personnel

Training PRC-005 Transmission Protection

System Maintenance and Testing

TOP-002 Normal Operations Planning

CIP-001 Sabotage Reporting

CIP-001 – Sabotage Reporting

Overall Non-Compliance Analysis Statement Problem areas identified for non-

compliance: ▪ Procedures▪ Reporting▪ Communication▪ Documentation


CIP-001 Primary Non-Compliance Factors Lack of required records demonstrating

compliance. Procedures were missing or deficient for

reporting events of sabotage.


CIP-001 Primary Non-Compliance Factors (cont.) Deficiencies were found in procedures to

communicate information regarding sabotage events to other appropriate personnel.

Contact list was too incomplete to report sabotage events to the local FBI and appropriate personnel.


Recommendations Have a thoroughly documented and tested

procedure in place for dealing with sabotage events. This includes steps for recognizing and making sure relevant personnel and entities are informed of a sabotage event.

Perform periodic review of the communication reporting procedure and confirm that contact lists are complete and current. This includes making sure all “operating personnel” and local government contacts are correctly identified and included.


Recommendations (cont.)

Verify that procedures are clearly identified and current in all Operator Procedure manuals.

COM-001 Telecommunications

COM-001 – Telecommunications


compliance: ▪ Procedures▪ Testing▪ Coordination▪ Documentation


COM-001 Primary Non-Compliance Factors Lack of required documentation

demonstrating that entity was managing, alarming, testing, and monitoring its vital telecommunications facilities.

Failure to test its vital telecommunications facilities was evident with its documented testing procedures.


COM-001 Primary Non-Compliance Factors (cont.) Entity unable to provide evidence that it

had the ability to investigate and recommend solutions to telecommunications problems within its own area and other areas, nor that it had procedures in place that confirm it would be able to continue operation of the system during a loss of telecommunications facilities.


Recommendations Design, develop, implement and

maintain procedures addressing the process of managing, alarming, testing, and monitoring vital telecommunications facilities and methods of documentation for recording this process.

Supply evidence that you test vital telecommunications facilities consistent with the testing procedures and document test records at the time testing occurred.


Recommendations (cont.) Design, develop, implement and

maintain written operating instructions and procedures to provide a means to coordinate telecommunications among respective areas.

This coordination shall include the ability to investigate and recommend solutions to telecommunications problems within the area and with other areas.

EOP-001 Emergency Operations Planning

EOP-001 – Emergency Operations Planning


compliance: ▪ Procedures▪ Documentation▪ Incomplete Emergency Operations Plans


EOP-001 Primary Non-Compliance Factors Failed to have in place an operating

agreement with provisions to obtain emergency assistance from remote and adjacent Balancing Authorities.

Emergency Plan did not directly address all of the necessary elements, such as, system restoration plans, communication protocol, mitigate operating emergencies, tasks to be coordinated and staffing levels during emergencies.


EOP-001 Primary Non-Compliance Factors Failure to provide evidence that it

reviews and annually updates its emergency plans.

Failure to provide updated emergency plans to all of the required entities at the time the plans were updated.


Recommendations Obtain an operating agreement with provisions

to obtain emergency assistance from remote and adjacent Balancing Authorities.

Design, develop, implement and maintain an Emergency Plan and procedures that directly address all of the necessary elements, such as, system restoration plans, communication protocol, mitigate operating emergencies, tasks to be coordinated and staffing levels during emergencies.



maintain a procedure that requires review and annually update to emergency plans.

Design, develop, implement and maintain a procedure to provide updated emergency plans to all of the required entities at the time the plans were updated.

EOP-005 System Restoration Plans

EOP-005 – System Restoration Plans


compliance: ▪ Procedures▪ Annual Review▪ Testing▪ Documentation▪ Coordination ▪ Training


EOP-005 Primary Non-Compliance Factors Failure to provide a Restoration Plan that

would re-establish its electric system in a stable and orderly manner in cases where there is a partial or total shutdown of its system.

Restoration Plan was not reviewed and updated at the least annually.


EOP-005 Primary Non-Compliance Factors (cont.) Failure to coordinate restoration plan with

the Generator Owners and Balancing Authorities within its area, its Reliability Coordinator, and neighboring Transmission Operators and Balancing Authorities.

Restoration Plan did not contain procedures for the loss of vital telecommunication channels and had not periodically tested its telecommunication facilities that are required to implement the restoration plan.


EOP-005 Primary Non-Compliance Factors (cont.) Failure to properly train operating

personnel on how to implement the restoration plan.

Restoration plan was not verified by actual testing or simulation.


Recommendations Design, develop, implement and maintain a

Restoration Plan that will re-establish its electric system in a stable and orderly manner in cases where there is a partial or total shutdown of its system.

Review and update the Restoration Plan at least annually and document this process.

Coordinate the Restoration Plan with the Generator Owners and Balancing Authorities within its area, its Reliability Coordinator, and neighboring Transmission Operators and Balancing Authorities.


Recommendations (cont.) Design, develop, implement and maintain a

Restoration Plan that contains operating instructions and procedures for the loss of vital telecommunication channels and periodically test its telecommunication facilities that are required to implement the restoration plan.

Properly train operating personnel on how to implement the restoration plan.

Verify restoration procedure by actual testing or by simulation.

PER-002 Operating Personnel Training

PER-002 – Operating Personnel Training


compliance: ▪ Documentation▪ Training ▪ Competency


PER-002 Primary Non-Compliance Factors Failure to staff operating personnel that

have direct impact on the real-time operation of the BES with adequately trained and qualified people.

Training program lacked specificity with respect to the training of operating personnel.


PER-002 Primary Non-Compliance Factors (cont.) Training program did not effectively

identify objectives, but merely provided a list of skills.

Failure to identify in the training plans the necessary knowledge, skills, or competencies for system operators to conduct reliable operations.

Failure to include a plan for initial and continuing training of operating personnel.


PER-002 Primary Non-Compliance Factors (cont.) Failure to include training time for

operating personnel. Lack of organized records for the

completion of training.


PER-002 Primary Non-Compliance Factors (cont.) Failure to prove the competencies of

training staff in both knowledge of system operations and/or formal training or other evidence of instructional skill competencies.

Failure to conduct annual training and drills using realistic simulations of system emergencies at least five days per year.

PER-002 – Operating Personnel Training Recommendations

Provide required training to staff operating personnel that have direct impact on the real-time operation of the BES.

A well-designed training program must start with the identification of job tasks. From this identification of the tasks required, learning objectives can be developed to give operators the abilities to perform the tasks. The knowledge, skills, and abilities are identified as required to meet the objectives, Training is then designed to the objectives and the related knowledge, skills, and abilities that are associated with them.


Recommendations (cont.) Program objectives should:

▪ Be based on NERC and regional Reliability Standards, entity operating procedures, and applicable regulatory requirements.

▪ Reference the knowledge and competencies needed to apply these standards, procedures and requirement.

▪ Consider normal, emergency, and restoration conditions.


Recommendations (cont.) Training programs must cover and allot

time for the operating personnel who have primary responsibility for real-time operations, or who are directly responsible for complying with NERC and regional Reliability Standards.

It is essential to carry out training according to plans for all operators to whom this reliability standard is applicable.

Document the records of all trained operators.

PER-002 – Operating Personnel Training Recommendations (cont.)

Provide training staff with training programs for instructional methods.

Annual practice sessions should use practice simulations of real emergency conditions, and records should be logged with date, participants, and events.

Overall training program should be based on a systemic approach to training, and address all aspects of the requirements, execute them and provide evidence.

PRC-005

Transmission Protection System Maintenance and Testing

PRC-005 - Transmission Protection System Maintenance and Testing


compliance: ▪ Understanding ▪ Documentation ▪ Organization

MOST FREQUENTLY VIOLATED STANDARD


PRC-005 Primary Non-Compliance Factors Documentation of testing and

maintenance results missing or inadequate.

Not all components of the protection systems were identified or tested.

Inventory lists of applicable devices are incomplete and therefore, devices were not scheduled appropriately.


PRC-005 Primary Non-Compliance Factors (cont.) Lacking basis to determine the

appropriate testing intervals. Failure to complete maintenance and

testing activities on time. Lack of complete and thorough

monitoring of testing and maintenance programs.


Recommendations Entities subject to standard PRC-005 need to

have a thorough and rigorous documented maintenance and testing plan in place for devices that qualify as protection systems.

Perform periodic physical inventories, including walkthroughs where needed, to ensure the active device inventory list is complete and accurate, and all pertinent devices appear on maintenance and testing schedules. CHANGE MANAGEMENT.


Recommendations (cont.) Verify that testing programs include the

appropriate basis of testing to ensure the reliability of the Bulk Electric System.

Complete maintenance and testing programs on schedule and within defined intervals.

Emphasis on the urgency to meet the specified time intervals must be made explicitly clear regardless of what situations the company may encounter that interfere with planned maintenance.

TOP-002 Normal Operations Planning

TOP-002 – Normal Operations Planning


compliance: ▪ Coordination▪ Notification▪ Documentation


TOP-002 Primary Non-Compliance Factors Failure to coordinate current-day, next-

day, and seasonal operations with appropriate entities.

Failure to provide documentation showing that it was providing forecasts to appropriate entities.

Failure to coordinate planning and operations with neighboring entities


TOP-002 Primary Non-Compliance Factors (cont.) Failure to notify appropriate entities of

changes in capabilities and characteristics, such as changes in real output capabilities.

Failure to provide documentation demonstrating that it used uniform line identifiers when discussing transmission facilities among a shared interconnect.


Recommendations Design, develop, implement and maintain

procedures to coordinate current-day, next-day, and seasonal operations with appropriate entities.

Create a standardized document to be utilized for providing forecasts with a mutually agreed upon format when providing forecasts to appropriate entities. Create an electronic file system for storage of these documents.

Design, develop, implement and maintain procedures to coordinate planning and operations with neighboring entities.


Recommendations (cont.) Confirm all line identifiers are consistent

with the identifiers used when discussing transmission facilities among a shared interconnect. Obtain a letter of agreement between entities on identifiers used when discussing transmission facilities among a shared interconnect or obtain an agreed upon one-line diagram demonstrating uniform line identifiers.



maintain procedures to utilize uniform line identifiers when discussing transmission facilities among a shared interconnect.

Design, develop, implement and maintain procedures to notify appropriate entities of changes in capabilities and characteristics, such as changes in real output capabilities.

Top Violated Cyber Security CIP Standards (706)

Top Most Violated WECC Cyber Security Standards

CIP-003 Cyber Security - Security Management Controls

CIP-004 Cyber Security - Personnel and Training

CIP-007 Cyber Security - Systems Security Management

CIP-003 Cyber Security - Security Management Controls

CIP-003 Cyber Security – Security Management Controls


compliance: ▪ Documentation▪ Review


CIP-003 Primary Non-Compliance Factors Failure to document or incomplete

documentation of designated CIP Sr. Manager or failure to update documentation upon changes to designated CIP Sr. Manager.

Failure to properly document exceptions to the Cyber Security Policy.


CIP-003 Primary Non-Compliance Factors (cont.) Failure to properly review and approve

exceptions annually or failure to properly document the review.

Failure to perform or properly document annual review of Cyber Security Policy.


Recommendations Document a procedure that ensures a

CIP Sr. Manager is designated by Name, Title, and Date of Designation. Ensure procedure requires an update to the documentation upon any changes.

Implementation of Cyber Security Policy review of procedures and develop methods such as a compliance calendar to ensure the review occurs annually.


Recommendations (cont.) Procedures should exist to manage so

they are properly documented and reported.

Personnel should be trained on exception management procedures that ensure all responsible parties follow through with their obligations to identify, document and mitigate instances where exceptions to the Cyber Security Policy must be made.

CIP-004 Cyber Security - Personnel and Training

CIP-004 Cyber Security – Personnel & Training


compliance: ▪ Access▪ Training▪ Documentation▪ Risk Assessment


CIP-004 Primary Non-Compliance Factors Lack of required records demonstrating

compliance. Employees or contractors were granted

access to critical cyber assets without documented proof of clearance or escorted access.


CIP-004 Primary Non-Compliance Factors (cont.) Unable to prove training was offered

and/or completed in a timely manner by personnel.

Background checks for employees or contractors with access to critical cyber assets were missing or incomplete.


Recommendations Ensure and verify that all employees with

access to Critical Cyber Assets, including contractors and service vendors, have the appropriate training prior to access. Train annually thereafter.

Entity procedures should have control points designed to prevent granting access to untrained individuals or individuals who have not passed the Personnel Risk Assessment as well as methods to ensure annual re-training requirements are met.


Recommendations (cont.) Entities need to ensure and verify that risk

assessments on employees, contractors, and service vendors with access to Critical Cyber Assets are not only completed prior to access, but that the assessment focuses on relevant information.

Entities need to ensure and verify that the training provided to employees, contractors, and service vendors being granted access to Critical Cyber Assets focuses on the relevant information.


Recommendations (cont.) Entities need to ensure that appropriate

changes are made to access lists upon termination, or transfer of employees from or to areas that contain Critical Cyber Assets, and that access lists are frequently updated to contain contractors or service vendors.

Procedure should exist to ensure all access lists are current and properly maintained within the provided timeframe required by the standard.

CIP-007 System Security Management

CIP-007 Cyber Security – System Security Management


compliance: ▪ Procedures▪ Documentation▪ Testing▪ Logical Account Management


CIP-007 Primary Non-Compliance Factors Failure to demonstrate that testing is

conducted to ensure new Cyber Assets and significant changes to existing Cyber Assets within an ESP do not adversely affect ALL existing cyber security controls.

Documented procedures insufficient to prove only ports and services required for normal and emergency operations are enabled.


CIP-007 Primary Non-Compliance Factors (cont.) Failure to document the assessment of

security patches and upgrade availability within thirty calendar days of availability of the patches or updates.

Failure to document and implement a process for the update of anti-virus and malware prevention tools (including “signatures”).


CIP-007 Primary Non-Compliance Factors (cont.) Failure to properly document and control

access to shared and system accounts. Failure to enable logging on cyber assets

located within the ESP and/or lack of operational processes to manually monitor system events related to cyber security.


CIP-007 Primary Non-Compliance Factors (cont.) Failure to destroy or erase data storage

media to prevent unauthorized retrieval of sensitive cyber security or reliability data.

Failure to document disposal/redeployment activities.


CIP-007 Primary Non-Compliance Factors (cont.) Failure to perform cyber vulnerability

assessment of all Cyber Assets at least annually.

Failure to perform cyber vulnerability assessment of all Cyber Assets at least annually.


Recommendations Require testing documentation be

retained to prove testing was performed in accordance with the test plans.

Testing should focus on the impact to cyber security controls rather than functionality. The standard does not require functionality testing but requires testing of ALL security controls.


Recommendations (cont.) If tools are used for implementing changes,

these tools also need to be tested to ensure they will not adversely affect the systems.

Work with all vendors of systems and applications of applicable cyber assets to identify and document which ports and services are required for normal and emergency operations. Consider running manual or automated ports and services scans as part of normally scheduled maintenance on cyber assets (as part of the test plan for example).


Recommendations (cont.) Entities should consider leveraging

corporate level patch management program for tracking, evaluating, testing, and installing applicable cyber security patches required for all Cyber Security Assets within the ESP.

Entities should understand the scope of their patch management programs and ensure that security patches for all applications, operating systems, databases and firmware is being actively tracked.


Recommendations (cont.) For patches that cannot be tracked

automatically, develop a tracking form listing the URL where patches and updates are posted to help streamline the identification of new patches.

Develop procedures designed to identify and evaluate patches and updates.


Recommendations (cont.) Ensure testing of anti-virus and malware

signatures is part of the process. Consider leveraging a corporate level program for updating anti-virus and malware if one does not exist. Successful installation of signatures on corporate systems can be leveraged to satisfy the testing requirements of the standard.


Recommendations (cont.) Ensure technical and procedural controls

exist to minimize the risk of unauthorized system access by shared and system accounts and that these procedures are followed when specified events occur.

Employee training and accountability is critical to ensuring adherence to security practices.


Recommendations (cont.) Test cyber asset logging capabilities in a

pre-production environment prior to moving to production to ensure the assets are properly configured to send automated events to a centralized logging server and/or generate event logs for manual review.

Implement automated review systems to reduce manual log reviews.


Recommendations (cont.) Implement and document procedures

that ensure all cyber assets within the ESP are properly configured to log to a centralized location or identified as a system requiring manual review.

File a TFE if logging capabilities do not exist.

Documented, repeatable methods help ensure consistency in your compliance program.


Recommendations (cont.) Implement documented procedures for

disposal or redeployment of sensitive electronic media which include the records proving erasure or destruction occurred.

Tracking electronic media (acquisition, deployment, destruction) is necessary to prove compliance.

Refer to NIST Special Publication 800-88; Guidelines for Media Sanitation for methods for destroying or erasing electronic media.


Recommendations (cont.) Consider leveraging corporate level

vulnerability assessment programs where they exist so long as they meet the requirements of CIP-007 R8.

Vulnerability assessments can be included in the test plans to help minimize the scope of the annual assessment. As systems are changed and tested, vulnerability assessments are performed to satisfy the annual assessment requirement.


Recommendations (cont.) Ensure the documented vulnerability

assessments explicitly cover the specific items required in the Standard.

Create comprehensive document review procedures to ensure review of all CIP-007 documentation is performed and implement methods to ensure the review occurs annually.

Summary

Need to have an overall approach to compliance.

There is no substitute for documented procedures.

Procedures must show implementation. Notable rise in Violations of CIP-006

Physical Security in 2010, heading for the Top 10 List.

QUESTIONS ?

http://www.google.com/imgres?imgurl=http://dreamfactoryblog.files.wordpress.com/2009/09/question-marks.jpg&imgrefurl=http://blog.dreamfactoryproductions.com/2009/09/15/marketing-vs-advertising/question-marks/&h=800&w=534&sz=214&tbnid=aqGsRg7quHNxfM:&tbnh=275&tbnw=183&prev=/images?q=question+marks&zoom=1&q=question+marks&usg=__DVbSqgROueutwDuT13ouCxEUnXA=&sa=X&ei=bzSQTImpBIGB8gbp1qDNDQ&ved=0CCAQ9QEwAA

Technology

Intellibind Top Ten Most Violated Standards Presentation 2011 01 27 (F)