Presentation to the EMS USER GROUP Meeting

Presentation to the EMS USER GROUP MeetingCompiled byChuck NewtonNewton-Evans Research CompanySeptember 2012

2012 Usage Patterns and Trends in Control Center SOP, Visualization and Cyber Security Welcome to this briefing session:

A Review of Findings from Three Studies Conducted in 2012 by Newton-Evans Research(1) Standard Operating Procedures For Control Room Operations. Larger utilities and ISO/RTOs Small Sample (26 IOUs, RTO/ISOs and Large Public Utilities)Study commissioned by American Engineering University

(2) NERC CIP Compliance Topical Study Small-Midsize utilities Up to 200,000 customersMore than 100 U.S. and Canadian Utilities ParticipatedStudy commissioned by Cyber Security Specialist Firm

(3) Newton-Evans Study of Cyber Security for Protection and ControlLarger Utilities (North America and International)More than 60 utilities from 30+ countries ParticipatingCommissioned by CIGRE JWG B5 D2.46

Standard Operating ProceduresControl Systems Operations Managers and Senior Staffers

% driven internally%Regulatory and reliability organizationsPublic Power44%56%Cooperative64%36%Investor-Owned51%49%Vendors48%53%ISO/RTO40%60%Summary48%52%1. How much of the details of real-time operating procedures are driven internally from the organization, how much driven from reliability (Reliability Coordinator, Transmission Operator, Balancing Authority) or regulatory organizations (NERC, FERC, state)?2. Please rank the following types of information based on their importance in making decisions for various real-time procedures in each scenario. Use a scale of 1-5, with 1=most important and 5=least important (Using each number only once.)1.123.083.522.644.641.403.003.562.604.441.483.603.761.924.241.122.923.442.804.720.001.002.003.004.005.00SCADA dataContingency Analysis dataState Estimation dataVerbal communicationsOther information sources

Normal operationEmergency operationRestorative operationPost contingencyNormal operationSCADA dataContingency Analysis dataState Estimation dataVerbal communicationsOther information sourcesPublic Power1.113.334.222.004.33Cooperative1.252.753.752.255.00Investor-Owned1.142.863.432.864.71Europe1.003.002.004.005.00Vendor1.003.002.004.504.50ISO/RTO1.003.502.503.005.00Summary1.123.083.522.644.64Emergency operationSCADA dataContingency Analysis dataState Estimation dataVerbal communicationsOther information sourcesPublic Power1.333.114.002.444.11Cooperative2.252.753.751.754.50Investor-Owned1.142.713.572.864.71Europe2.004.001.003.005.00Vendor1.002.503.004.504.00ISO/RTO1.004.003.002.005.00Summary1.403.003.562.604.44Summary1.122.923.442.804.727Restorative operationSCADA dataContingency Analysis dataState Estimation dataVerbal communicationsOther information sourcesPublic Power1.563.674.001.893.89Cooperative2.003.004.001.005.00Investor-Owned1.143.433.712.144.57Europe1.004.003.002.005.00Vendor1.504.003.003.503.00ISO/RTO1.504.503.501.504.00Summary1.483.603.761.924.24Post contingencySCADA dataContingency Analysis dataState Estimation dataVerbal communicationsOther information sourcesPublic Power1.113.003.892.564.44Cooperative1.252.753.752.255.00Investor-Owned1.142.713.712.714.71Europe1.003.002.004.005.00Vendor1.003.002.004.005.00ISO/RTO1.003.502.003.505.00Summary1.122.923.442.804.72 4. Who in the operational hierarchy executes the EMS applications, the higher or lower reliability authority? (Check all that apply)

5. Do any of the entities checked above in question #4 run EMS applications and compare results?

Yes, all involved71%No17%Other12%

6a. How are actions coordinated for events near the boundaries of Balancing Authority Areas or Reliability Coordinating Areas? (Check all that apply)

For the survey group as a whole, verbal communications is the dominant method for coordinating events near the boundaries of BAAs or RCAs. ICCP, however, is also frequently used among all of the domestic utilities and RTOs.

6b. How is corrective action decided and carried out?By a impressive margin (88%), corrective action is decided and carried out through cooperative decisions and actions by both the Balancing Authority and Reliability Coordinating Areas.88%8%4%0%20%40%60%80%100%Cooperative decisions and actions by both the Balancing Authority and Reliability Coordinating areasOnly the Balancing Authority decides and takes action Only the Reliability Coordinating area decides and takes action

6c. If just one entity decides and takes corrective action, what is the MAIN driver of this decision? (Pick one)

Equipment responsibility or ownership (44%) is the main driver for the eighteen respondents to this question. However, this value increases to seventy-one percent (71%) if only the responses from the seven investor owned utilities are considered.0%44%17%22%17%0%20%40%60%Proximity of event to boundary (i.e. further away) Equipment responsibility or ownershipSeverity of eventTime- criticalness of responseOther8. What type of control center wall board do you use?

tile/magnetic2D video3D videootherTotalPublic Power33028Cooperative03014Investor-Owned43017Europe01001Vendor22002ISO/RTO02103Summary91414259. What visualizations are most relevant during NORMAL SECURE (NORMAL) OPERATION of the grid?

Ninety-two percent (24 out of 26) of the survey respondents rated Topological Visuals as Extremely Important during Normal Secure (Normal) Operation of the grid. Dynamically Colored Visuals were also viewed as Extremely Important to 58% of respondents.

10. What visualizations are most relevant during NORMAL INSECURE (ALERT/CONTINGENCY) OPERATION of the grid?

27%92%15%54%31%69%8%58%8%31%35%15%12%31%15%0%54%12%54%19%62%0%20%40%60%80%100%Geographical VisualsTopological VisualsContour VisualsTabular VisualsAnimated VisualsDynamically ColoredDynamically SizedExtremely ImportantSomewhat ImportantNot Important to Our Operations11. What visualizations are most relevant during EMERGENCY OPERATION of the grid?

12. What visualizations are most relevant during RESTORATION OPERATION of the grid?

13. Which type of display (large control center board or desk top screen) is an operator more likely to use in the following situations?

Desk top computer screens are more likely to be used by a control room operator during all operation phases. However, during Normal and Post Contingency Operations the respondents tend to use them slightly more (69% and 73% respectively) than during Emergency and Restorative conditions where the use of a control center board significantly Increases (from 19% to 31%).

Responses from the survey group find that visualizations on the operators desktop computer screen offer significantly more benefits than the control center board. The control center board is cited as being better for wide area viewing.

14. What is the difference between the visualizations on the control center board and the operators desk top computer screen?

15. Are certain visualizations more appropriate or efficient on the control center board or operators desk top computer screen?

Overall, responses to this question provided a little more balance when comparing visualizations on the control center board to operator desk computer screens. However, there are some significant differences in three areas: Tabular Visuals efficiency on operators desk computer screen received an overwhelming preference (84%) to those of control center boards, Geographical Visuals were found to be more appropriate for control center board use by a 3 to 1 margin, while Topological Visuals were better suited to operator desk top computers by a 2 to 1 margin. 16. How are critical events visualized (e.g. operating limit violations, line tripping, generator tripping, etc)? (Check all that apply)The two visualization methods predominantly used by the twenty-six respondents for critical events are Blinking Values (81%) and Highlighted and Blinking Values (73%). IOUs indicated the highest use of Character Tag and Exception List.81%38%73%12%42%8%23%0%20%40%60%80%100%Blinking valuesCharacter TagHighlighted and Blinking ValuesZoomingException ListPanningOther20. Please rank the relevancy of the following data in the visualization of the grid from 1-6 where 1=most relevant and 6=least relevant (Using each number only once).Device states (2.21), Voltage (2.54) and Power (2.79) were ranked the most relevant data in the visualization of the grid by survey respondents. For the most part, this was pretty consistent among all of the survey groups with the exception of the one European utility.2.544.002.793.582.215.880.002.004.006.008.00VoltageCurrentPowerFrequencyDevice statesOther

-->Study of Small-to-Mid-Size Utilities Regarding NERC CIP TopicsStudy undertaken Jan-Apr 2012More than 100 Utilities ParticipatedHaving from 20,000 to 200,000 customers

1. Does your utility have Critical Cyber Assets under NERC CIP?

In spite of the seeming change in definitions of what is a critical cyber asset, two-thirds of the respondents indicated that they had NO cyber assets that are considered critical under current NERC CIP definitions.2. How much did your utility spend on cyber security Operations and Maintenance in 2011?

Responses here were reported across all dollar ranges. More than one-third spent less than $25,000 per year on cyber security O&M in 2011. Just over one third spent from $25,000 to $200,000. Thirteen percent spent more than $200,000. Four respondents indicated that cyber security O&M was not a budgeted item.4. What were your utilitys capital expenditures for cyber security in 2011?

One half of the respondents to this question reported spending less than $25,000 in capital expenditures for cyber security during 2011. Nearly one quarter stated that expenditures ranged from $25,000 to $200,000. Thirteen percent replied that they had invested more than $200,000 for cyber security items. Again four respondents replied that cyber security was not a separately budgeted CAPEX line item.6. Have utility work practices and procedures changed as a result of NERC CIP requirements?

Seventy percent of all respondents indicated that utility work practices and procedures have changed as a result of NERC CIP requirements. Importantly, 40% of all respondents stated that work practices and procedures have changed significantly due to NER CIP requirements. Most of the 30% reporting no change in work practices and procedures hold the view that they do not have critical cyber assets as currently defined by NERC. If NERC CIP requirements have caused changes, please explain:Respondent #1We have added workflows to the process to demonstrate/document compliance.Respondent #4Station access proceduresRespondent #5Limiting and logging access to dispatch & other source areas; lots of documentation & audit preparation; lots of effort to ensure compliance but not necessarily improve security.Respondent #6Installation of new SCADA system required improvements in physical access requirementsRespondent #7Maintenance of CIP rules is a massive and continuous undertaking. It took 14 FTE's to get through the most recent auditRespondent #8Device installation, testing, access management, patch managementRespondent #9NERC CIP does not applyRespondent #10Level of reporting and documentation requirements required have increased significantly.Respondent #12All actions and occurrences have to be verified under CIP regulationsRespondent #13While our utility does not currently fall under version 4 of the CIP standards we are actively preparing for full compliance because we anticipate version 5 will affect us significantly. At the very least a proactive cyber security program is a good practice and enhances the reliability of both the BES and non-BES power systems. Respondent #19Add a tremendous burden for security and logging of activitiesRespondent #20Process to develop a security program has begin. We are taking small incremental steps.Respondent #24Sign in sheets required into certain areas during certain time frames; escorts required into certain areasRespondent #25Since NERC CIP requirements are not yet applicable, our efforts are in anticipation of future changes. However, such future changes shown in pending drafts of the NERC standards will have a VERY significant impact to work practices and procedures.Respondent #26At this time, our utility will not consider substation LANs for IEDs & RTUs due to pending NERC requirements.Respondent #27We are distribution but have made changes in anticipation of NERC/CIPRespondent #28We evaluate each new technology initiative for its ability to put us into CIP requirements. Using communicating faulted circuit indicators as an example, we chose a hosted service rather than bringing the data in-house to avoid any potential CIP changes.7. Does your utility offer in-house training for cyber security?

Nearly one half (47%) of the survey respondents reported that their utility offers some form of in-house training for cyber security. Another 17% plan to offer such in-house training by 2014. More than one third (37%) of the survey sample do not offer cyber security training on an in-house basis.9. Do you currently outsource any cyber security tasks to a third party?

More than one half (53%) of the survey participants indicated that they DO outsource at least some cyber security tasks to outside services to third parties. Another 10% plan to do so by year-end 2014.12. What are the certification requirements that your employees must have to work with projects involving cyber security

Perhaps surprisingly, more than three quarters of the responding utility officials reported that there are currently NO certification requirements for employees in order to work with projects involving cyber security topics.Of the handful of utilities that indicated one or more certifications as requirements, CISSP (15%), CISM (11%), CISA (7%) and Comp TIA (4%) were specifically cited. A few reported other requirements such as CISCO Systems in the listing below the chart.16. Have NERC CIP requirements caused your utility to increase the number of full time employees dedicated to cyber security activity?Yes, 23%No, 77%Have NERC CIP requirements caused your utility to increase the number of full time employees dedicated to cyber security activity? 19. Which of the following cyber security technologies/methods do you currently use?

Newton-Evans Survey for CIGRE JWG B5 D2.46

Cyber Security for P&C Systems

Larger Utilities (North America and International)Participation from More than 60 utilities in 30+ countriesConducted with P&C Managers; Operations Managers; Some IT Management Involvement

1a. Are you offering your P&C System personnel (engineers and field technicians) any cybersecurity training for their job? ?responsibilities?

3. Have your P&C system personnel signed acceptable use policies?

1a. Are you offering your P&C System personnel (engineers and field technicians) any cybersecurity training for their job? responsibilities?

1b. Do you tailor cybersecurity training to address the issues related to job responsibility?

2a. How would you consider the quality and completeness of cybersecurity training in your organization?

2b. In your opinion, could your cybersecurity training be improved?

3. Have your P&C system personnel signed acceptable use policies?

4. Do you have a cybersecurity incident response plan for your P&C system?

5. Do you monitor P&C system personnel access to and use of P&C system components?

6. Do you test P&C system patches to correct cybersecurity defects prior to deployment?

7a. Do you have adequate controls in place to monitor P&C system behavior in order to indicate that a security incident has taken place?

7b. Do you benchmark or maintain a scorecard of P&C system cybersecurity incidents?

8. What is your short list of cybersecurity solutions needed to protect your P&C systems?Need to understand vulnerabilitiesNeed to plan to fix themSegmentation of networksUpdated password maintenanceActually, we've implemented cybersecurity measures by following NERC-CIP guideline since 2007. However, only control system has been secured but not for protection system. Do not connect relays to the network.Currently as little as possible connection to the outside Web. All personnel have dedicated pc's to connect to P&C systems.Anti-virus systemPhysical separated networkRestricted firewallPassword Policy - # of characters and 90 day expiration was implementedAccess control (authorization, etc.), closed network configuration (net separation)Audit Log, BackupAntivirus & Firewall software etc.Security of the network access to P&CSecurity of the computer devices used as toolsSecurity practices of the personnelRemote secure access (through SCADA or IP solution)Secure mobile local access to devicesRestricted physical accessRestricted electronic accessTraining and awarenessEnfording NERC-CIP standards usageImplementing firewalls in the substationImplementing anti-malware software in DCSP&C devices have no connectivity to any system. None at this time outside the substation.Remote accessProtocolIPSSecurity gatewaysFirewallsHardwired telephone switchPerimeter access control, both physical and electronicIntrusion detection and prevention softwareCentralized software patches and password managementSecurity enforcement pointsCentralized configuration management systemExtend our existing remote access system11a. Do you allow employees to use their personal devices (i.e. personal flash drive, smart phone, tablet, etc.) for P&C maintenance or configuring P&C components?

11b. If NEITHER to the above, are you planning to allow employees to use their personal devices (i.e. personal flash drive, smart phone, tablet, etc.) for P&C maintenance or configuring P&C components?

BYOD could affect views11c. Do you support programs loaded on employee personal devices?

11d. What is your estimate of the percent of employees using personal devices for P&C maintenance?

11e. What is your estimate of the percent of employees using personal devices for configuring P&C components?

11f. Do you enforce security policies and encryption for employee personal devices?

12a. Do you allow third party support technicians to use their personal devices (i.e. personal flash drive, smart phone, tablet, etc.) for P&C maintenance or configuring P&C components?

12b. If NEITHER to the above, are you planning to allow third party support technicians to use their personal devices (i.e. personal flash drive, smart phone, tablet, etc.) for P&C maintenance or configuring P&C components?

12c. Do you support programs loaded on third party support technicians' personal devices?

12d. What is your estimate of the percent of third party support technicians using personal devices for P&C maintenance?

12e. What is your estimate of the percent of third party support technicians using personal devices for configuring P&C components?

12f. Do you enforce security policies and encryption for third party support technicians' personal device?

13a. Are your P&C cybersecurity policies and procedures derived from regulatory requirements?

13c. If no, from where are your cybersecurity policies derived? 49%40%52%54%60%52%22%0%29%22%40%16%76%80%74%7%0%10%0%10%20%30%40%50%60%70%80%90%100%SummaryNorth AmInternationalUtility guidelinesGenerally accepted industry guidelinesProfessional association recommendationsP&C staff recommendationsIT department guidelinesOther14. From the list below, rank the inhibitors for implementing strong security policies for P&C system operations in order from 1-4, with 1=strongest inhibitor and 4=weakest inhibitor. SummaryNorth AmInternationalCost to maintain and operate a strong security system2.482.042.79Perimeter security provided by and supported by IT is adequate2.952.833.03Lack of interoperability between P&C system components2.472.542.41P&C system components do not incorporate strong security mechanisms2.102.581.7616a. Does your utility have the technology or business processes needed to manage role-based access control (RBAC) for P&C systems?

Upcoming EMS/SCADA/DMS Study4th Quarter 2012 Study of Control Systems usage patterns and plans among the worlds electric power delivery utilities.We need your help for this study to serve as the bridge between what YOU need and want in control systems and what systems providers-vendors need to know in order to develop solutions to meet your needs.

Other 2012 Research TopicsU.S. Manufacturing Readiness for Smart GridCloud Computing Outlook for Small-Midsize Utilities and Usage of Specific IT/OT Applications Packages Substation Processing Platform OptionsFault Current Limiting DevicesU.S. Market for Bus DuctAssessment of American Manufacturing Industry Readiness for Smart Grid Roll Out

Presentation to the EMS USER GROUP MeetingPrepared byChuck NewtonNewton-Evans Research CompanySeptember 2012

Thanks for sitting in on this briefing!

Appendix Slides 2012 Findings on Cyber Security and 61850 Usage and Plans

Looking at Smart Grid Opportunities for Growth at Mid-Year 2012 .What stands in the Way? by Chuck on June 25, 2012

Why we believe the near-term investment priority for utilities of all types must be cyber security-related!(Security is not always considered part of smart grid spending)!This year, Newton-Evans Research has already undertaken a number of national and international studies of cybersecurity issues, and the findings lead us to believe that the single most critical issue facing utilities of all types is the near-term requirement to shore up cyber defenses, policies and procedures. Unfortunately, these cyber security investments will likely continue to usurp funding from other smart grid activities, but this investment must be a priority, in my opinion. Findings from Jan-Mar 2012 Survey of Protection Engineers Relay Protocol UseNorth AmericaInternational

Extent of Use of IEC 61850 in SubstationsNorth AmericaInternational

Findings from Jan-Mar 2012 Survey of Protection Engineers

Features of IEC 61850 Being Used/PlannedNorth AmericaInternational