2nd Annual NERC Monitoring and Situational Awareness Conference:FPL’s Operational Technology Center

Ed Batalla – Director of Grid Control SystemsFlorida Power & Light CompanySept. 24, 2014

2

• Florida Power & Light Company (FPL) Overview– FPL EMS and Control Center Facilities Overview– FPL Technical Support Services Overview

• Problem Statement– What is an effective technology support structure to enhance grid

reliability?• FPL Approach

– Support Structure Evolution– Operational Technology Center (OTC)

Technology Monitoring Examples• Q&A

Agenda

3

• FPL is a subsidiary of NextEra Energy, Inc. (NEE)

• One of the largest U.S. electric utilities

• Vertically integrated, retail rate-regulated

• Approximately 4.7 million customer accounts

• 24,273 MW generating capacity

• $10.4 B in operating revenues

• $36.5 B in total assets

FPL is the largest electric utility in Florida and one of the largest rate-regulated utilities in the United States

FPL Overview

NOTE: All data extracted from NextEra Energy, Inc., Annual Report 2013.

4

FPL has geographically diverse and redundant control centers which contain the Grid Control Systems (GCS) – an integral part of our business continuity and recovery plans

System Control Center (SCC) FacilityLocal Backup Control

Center

Backup Control Center (BUCC)

Facility

Fully functional backup control center, capable of being activated per NERC requirements and has tools that closely replicate the

primary control center

The control room has emergency communication

methods for loss of the primary communication tools; also has remote access to the Backup

Control Center

Remote Backup Control Center

– Distribution Control Centers (use and access same EMS)

– Transmission Control Centers and EMS Facilities (FPL EMS is also used by FRCC Reliability Coordinator)

5 Confidential Information

Facilities and Systems Overview

• Geographically diverse primary and backup control centers facilities

• Interim provision control centers

• Redundant building power supply (with emergency generators)

Facilities

• Redundant systems at both facilities with fast-failover capability

•• Upgraded: November

2012• Linked to Distribution

Management System

Energy Management System

• Fully redundant with voice recording capabilities

• Backup Systems:(a) Satellite Phones (b) Hoot and Holler, and (c) Corporate Phone System

Voice Communications

• Real-Time Data Links (a)

• FPL data direct from FPL’s internal sources

Data Links and SCADA Data

• Full EMS/SCADA applications suite

•

• Customized Wide-Area views and situational analysis displays

Reliability Tools

• FPL's EMS models the pertinent FRCC BES elements within the FRCC footprint and neighboring reliability area to provide Wide-Area situational awareness

Power System Model

6

Technology support and services are provided to continuously improve the “operational certainty” of the operator tools, applications, and infrastructure

Technology Support and Services

PEOPLE• Operator tools and

applications support provided by Grid Control Systems (GCS) team

• Infrastructure support provided by FPL’s Information Management (IM) team

• Mission Priority: Availability IntegrityConfidentiality

PROCESS• 24X7 Operational

Technology Center for GCS applications and infrastructure monitoring

• Six Sigma concepts applied to daily work and performance monitoring –continuous improvement

• Rigid change management and configuration control process

TECHNOLOGY•

Fast-failover using “multi-site” technology

Advanced reliability tools• Cybersecurity technology

improvements to further protect control networks

• Technology upgrades underway to provide improved resiliency and robustness

7

• Technology support structures for EMS/SCADA systems have been prevalently accomplished through “callout”– Most have dedicated technical support staff for the EMS/SCADA

systems (and reporting up through the business unit responsible for grid monitoring and control -- transmission operations)

– Some have integrated the technology support structure with information technology (IT) personnel

– Few deployed 24X7 technology support• To optimize costs, most EMS/SCADA vendors are moving

toward commercial off-the-shelf (COTS) technology infrastructure, which is common to most IT environments– Support structure needs to adapt to business needs

What is an effective technology support structure to enhance grid reliability? Need to be able to PREVENT, MITIGATE, RESTORE technology-related events/incidents

Problem Statement

No silver bullet – need to consider business alignment with both grid operations and technology support objectives

8

FPL’s support model continues to evolve in order to keep up with technology complexity and to continuously improve the “operational certainty” of its Grid Control Systems

FPL’s Support Model Evolution

Callout Support (1979 – 2011)Off hours and weekends supported by “callout” SMEs

Operational Technology Center (2012 – 2013)24X7 dedicated shift personnelStill complemented with “callout” support as necessaryIncident response and triage Technology surveillance and monitoring (KPIs)

Enterprise Integration(2014 – Beyond)24X7 OTC integrated with corporate Network Operations Center (NOC) and Information SecurityProceed to incorporate proactive monitoring of events- Applications- Infrastructure- Cybersecurity

Reactive

Proactive

Situational Awareness

9

• 24/7 Technology Surveillance via OTC– Applications and infrastructure

performance monitoring– Improved incident response for:

GCS application issues Technology-related issues

• Plans are in place to further improve capability in infrastructure and cybersecurity monitoring– FPL’s IM team is now adopting this

support model enterprise-wide– GCS specific training of OTC operators

FPL continues to implement initiatives to improve the service level for the availability of the Operators’ monitoring and real-time assessment tools

Initiatives to Improve Service Level

These initiatives will provide service level margin with regards to the Operators’ monitoring and real-time assessment tools

10

OTC is currently positioned to complement grid operations (TOP and RC) through active technology surveillance

As we move forward, FPL plans to enhance OTC capabilities to improve technology situational awareness and analytics capabilities

OTC: Current and Planned InitiativesCurrent (On-Production) Planned• EMS (and related infrastructure)

monitoring • Technology surveillance• Alarms

• Telemetry data source monitoring• Communication circuits

(substation data)• Ticket Management

• Contingency Analysis (CA) performance

• Critical interfaces / throughput monitoring

• OTC integration with NextEraEnergy NOC

• Coordinated cybersecurity monitoring (between control and business networks)

• Dashboards / Control Charts• Analytics for technology-related

events• Event analysis• Predictive capability

11

Health of the system is monitored continuously by 24/7 technology personnel – via technology specific alarms

Coordination with grid operations (TOP and RC) is essential to ensure that “operator tools” are functioning as designed

System and Application Health Check

Technology-specific alarms

12

FPL implemented improved monitoring process capabilities (with alarming) on Contingency Analysis (CA) failures

Support processes are triggered based on CA “last run” elapsed time

13

OTC provides for the continuous monitoring of the technology components of SCADA data availability

As necessary, OTC will contact circuit vendor for repairs

14

OTC relieves technology-related issues from operatorsso operators can concentrate on grid monitoring and control functions

Trends are analyzed as necessary for continuous improvement

Ticket Management

Tickets are monitoredfor resolution

15

OTC is integrated with grid operations – provides the opportunity for operators and field personnel to be aware of technology issues

Status Report

Pending or Overnight Issues# Incident

Report #Issues and Actions Taken to Date Date &

Time Issue Began

1 BARBVLTP (0)

IR-10028

BARBVLTP (0) 30FDXX505532 Circuit is down. ATT#sf002814. ATT TOK initially, OTC requested to re-test cct and dispatch if need it. Still being worked by AT&T

8/11/2014 11:14 PM

2

Scheduled Maintenance# Provider Description Status

12

Weekly Scheduled Events

# EventDate & Time of

Event

1 Failover Schedule: EMS failover 8/27/2014

2

RTU’s Down# Site

NameIncident Report #

Service Provider Customer

Count

Issues & Actions Taken to Date

Date & Time Issue

Began

1 MELROSE TAP

IR-10098

AT&T AT&T circuit being retested to confirm last mile line is good.

8/17/2014

2

Stations on EDGE (Wireless Backup) Circuit1 TRAIN

(807) IR-10039RESOLV

ED

AT&T TRAIN (807) Circuit down. Wireless ok. RTU currently on EDGE; ATT tkt# 481206041(/DHEC/888232/ATI/)

8/12/2014 11:42 PM

2 JETPORT (573)

IR-10092

Century Link

Century Link update: Tested ok to the OCU; Need to notify P&C

8/16/2014 6:54 AM

3 MOTOROLA (405)

IR-10144RESOLV

ED

Fibernet Circuit Down - MOTOROLA (405) - FNET CCT=CV80804 TKT=FN01831

8/21/2014 6:07 AM

16

• Entities need to continue to strengthen the capabilities of their technology support structure– Essential component to maintain reliability– Need to deal with technology complexity– Cybersecurity is a major part of the mix

• No silver bullet – need to consider business alignment with both grid operations and technology support objectives

• Entities must continue to share best practices

FPL believes that the concept of the Operational Technology Center (OTC) keeps up with technology complexity andcontinuously improves the “operational certainty” of its Grid Control Systems

Final Note