Presented by Members: Tom Botello - Southern California Edison Terry Baker - Platte River Power Authority Reliability Coordination Task Force Report to

Presented by Members:Tom Botello - Southern California Edison

Terry Baker - Platte River Power Authority

Reliability Coordination Task ForceReport to Finance and Audit Committee

December 5, 2012

2

• FAC Assignment• Summary of the Issue/Finding/Conclusions• Real-time Operations• Operations Support• Western Interconnection Synchrophasor

Program (WISP)• Compliance• Training• Recommendations

Report Agenda

3

• Terry Baker• Tom Botello• John Cupparo• Graham Edwards• John Meyer

Reliability Coordination Task Force (RCTF) Members

4

• July 18 Kick-off meeting in Portland, Oregon• August 15 In-person meeting in Denver• August 24 Meeting by Webinar• October 22 In-person meeting at the Vancouver

RC• November 20 Meeting by Webinar• November 23 Meeting by Webinar• November 27 Meeting by Webinar• December 4 In-person meeting in Salt Lake City

RCTF 2012 Working Sessions

5

• RCTF to examine the following: o Original scope of RC duties and expenseso Current scope of duties including expanded

duties since 2009o Future scope of duties and expenses

• Analyze key drivers for operational and budgetary change, WECC’s response to those drivers, and provide recommendations

FAC Assignment

Operations Planning

• Current -day/Real-time Support

• Next-day Study Data Validation and Mapping

• Seasonal SOL Coordination and Validation

• Emergency Plan Coordination

Real-time Operations

• Monitor and analyze real-time system conditions

• Coordinate system restoration activities

• Proactively respond to and mitigate potential adverse impacts to the BES

• Communicate with and direct corrective actions of registered entities during actual SOL exceedances

• Identify and mitigate potential/actual IROLs in real-time operations horizon

Compliance

• Document and Evidence Management

• Audit Preparation

• Event Analysis• Standards

Development• Compliance

Reporting• Disturbance

Reporting

Operations Support

• Application Architects

• Application Support

• DTS Support• EMS Modeling

• RAS Engineer

• Architecture Support

• Communications• IT Infrastructure• Systems

Administrator

WISP

• Broaden deployment of phasor measurement units throughout the Western Interconnection

• Implement telecommunication network necessary to support a large-scale production synchrophasor system

• Install synchrophasor applications for Western Interconnection

• Increase ability to integrate variable generation

• Enhance power system analysis capability/model validation

Training

• Develop/Deliver in-house training to new RC trainees and current RC staff to ensure compliance with mandatory standards and retention of NERC RC System Operator certification

• Conduct just-in-time training for new tools, applications, procedures, and processes

• Update job task analysis

• Develop/deliver NERC-certified restoration training to BAs, TOPs, and GOs

Other

• Direction and Oversight

• Administrative Support

• WECC and NERC Committee Work

• WECCRC.org Development

• Membership Request Management

Existing WECC RC Functions and Current Staffing

Current FTE – FilledCurrent FTE Approved – Unfilled

8216

Totals

7

• FERC/NERC Requirementso Highest level of authority responsible for the

reliable operation of the BESo Authority to prevent or mitigate emergencies in

next-day analysis/real-timeo Wide Area view of BES (situational awareness)o Tools, processes, and procedureso Broad purview to enable IROL calculation(s) that

may be based on parameters of transmission systems beyond any TOPs vision

Summary of the Issue

8

• Reliability Coordination in the Western Interconnectiono Past – Reliability through collaboration

Hosted by utilities – CAISO, BPA, WAPA Use of facilities, tools, and personnel

o Present – WECC’s Responsibility Independent facilities, tools, and responsibilities Compliance obligations questioned Reliability vulnerabilities exist

o Future – Implement RCTF Recommendations Increase staffing in critical areas Meet regulatory compliance requirements Reduce reliability vulnerabilities

Summary of the Issue

9

Summary of RCTF Findings

• Additional personnel are recommended in the following areas:o Real-time operations

Broad purview to enable IROL calculations (granularity, specific focus)

Additional real-time tools, processes, procedures

o Operations support Creation of operating plans, procedures (seasonal and next-

day studies) System Modeling and model validation IT application support requirements

o Compliance supporto Training

10

Summary Conclusions

• WECC has a high level of responsibility and is at risk for reliability and compliance issueso Staffing is a major factoro Tools are a major factor

• Emergent tools and Standards will add to these risksoWISP-related displays, data, trainingo PER-005 and EOP-005

11

Real-Time Operations

• RC Contingency Analysis • Four RC System Operators on shift on

September 8, 2011o Prior to loss of North Gila-Hassayampa 500-kV line,

there were 287 harmful contingencies identified by RTCA

o RTCA run following line loss identified 632 harmful contingencies

o Between start of event and SONGS units tripping (less than 11 minutes) 1,185 alarms were generated

12


• Similar events to SW Outage reveal RC real- time staffing concerns

• Ability to monitor current system specifics questioned by regulatorso Increase in required monitoring of RAS o Emergent tools such as WISPo Sub-100-kV facilities

• Specific interconnection area knowledge by RC System Operators needed to meet requirements

• RC phone calls per quarter have increased to over 37,000

13


2009 2010 2011 2012 (Projected)

0

2000

4000

6000

8000

10000

12000

14000

16000

18000

20000

8455

992810528

1200712263

15660 15336

19730

RC Log Totals

Loveland RC

Vancouver RC

Loveland RCTotal Load: 48888 MWTotal BAs: 16Total TOPs: 25

Vancouver RCTotal Load: 104506 MW Total BAs: 21Total TOPs: 29

Vancouver RCTotal Load: 49666 MWTotal BAs: 16Total TOPs: 17

Loveland RCTotal Load: 47888 MWTotal BAs: 15Total TOPs: 24

CA/MX RCTotal Load: 54840 MWTotal BAs: 6Total TOPs: 13

17

• The RC must create and maintain operating plans, procedures, and processeso This documentation must:

Identify RC System Operator actions taken to address specific system conditions and events

Reflect compliance to mandatory NERC, FERC, and Regional requirements, policies, and procedures

• Currently no specific position to perform this function

Real-Time Operations - Documentation

18

• Additional personnel are recommended in order to:o Monitor the Western Interconnection to the level

of granularity expected by FERC, NERC, and the NPCC

o Respond effectively to the volume of alarms and RTCA results during an event to determine appropriate corrective action

o Create and maintain Operating Plans, Procedures, and Processes as required by NERC Standards

Real-Time Operations − Findings







33 0 14FTE Totals

Current FTE – FilledCurrent FTE Approved – UnfilledFuture FTE Projection

WECC RC Function and Recommended FTE Increase

20

• SW Outage event and regulatory findings revealed seasonal planning gapso Not all WECC paths studied seasonallyo TOPs coordination weako Incomplete WECC contingency analysis

• Mitigating reliability vulnerabilitieso With an enhanced seasonal process and more

thorough assessments, the event may have been predicted

Operational Planning (Seasonal)

21

Operational Planning (Next-Day)

• Why did the RC next-day study not see the problem that caused SW Outage event?o Only study next-day peako Study automation incompleteo Need to assess sub-100-kV facilitieso Forecast data quality inputs need improvement

and are too numerous Number of scheduled outages continues to increase A million rows of unit commitment data daily

22

• Additional personnel are recommended to perform the following activities: o Seasonal Planning:

Take lead in implementation of an enhanced process with more thorough assessments

Participate in the resolution of identified gaps in the WECC Regional process

o Next-Day Study: Complete study automation Assess sub-100-kV facilities Ensure quality of data inputs

Operations Planning - RCTF Findings

Operations Planning

•Current -day/Real-time Support

•Next-day Study Data Validation and Mapping

•Seasonal SOL Coordination and Validation

•Emergency Plan Coordination

9 5 11

FTE Totals



24

• Model Impacts to Situational Awareness• Three ways to improve accuracy of RTCA:o Increase accuracy of facility ratings and SOLso Increase number of RAS modeledo Expand modeling of lower-voltage networks

that impact the BES Sub-100-kV systems that parallel higher voltage

networks Sub-100-kV systems with both automatic and

manually-switched reactive devices

Operations Support – System Modeling

25


• RAS Engineer and EMS Modeling –• WECC RC System Operators are required to

improve awareness of the impacts of RAS

• Continuing increase in the number of WECC RAS

• A key component of improved awareness –• RAS modeling in the EMS and real-time

contingency analysis (RTCA)• Only 45 of 206 RAS modeled in RTCA

26


Jan-08 Jan-09 Jan-10 Jan-11 Jan-120

50

100

150

200

250

0 0

18

46 47

161

180195 200 206

RAS Modeled vs. Identified WECC RAS

Jan-08 Jan-09 Jan-10 Jan-11 Jan-120

500

1000

1500

2000

2500

3000

3500

702 718 722

1176

1604

3012 31093284 3270

3368

RAS Real-time Data Points in EMS

RT RAS Points in EMS

Minimum Expected RT Points

27


11/18/2010 2/26/2011 6/6/2011 9/14/2011 12/23/2011 4/1/2012 7/10/2012 10/18/20120

2000

4000

6000

8000

Model/Data/Tool Issues Identified and Addressed

Closed IssuesTotal Issues

Nu

mb

er o

f Is

sues

Jan-

09

Apr

-09

Jul-0

9

Oct

-09

Jan-

10

Apr

-10

Jul-1

0

Oct

-10

Jan-

11

Apr

-11

Jul-1

1

Oct

-11

Jan-

12

Apr

-12

Jul-1

2

Oct

-124000

5000

6000

7000

8000

4328 4192

4929 4737

5606 5677 55885081

5498 5566 5734 5872 59576421

7121

7775Contingencies Defined

# of Contingencies

28

• Addressing Model Deficiencies• WECC RC needs to be more proactive in the

modeling process by increasing the frequency of model reviews with TOPso West-wide System Model and Base Case

Reconciliation Task Force (WBRTF)o Develop process for periodic review and sign off of

model accuracyo Add more power system measurements

• Share data and application results


29

• Applications and IT areas of responsibility include, but are not limited to: o coordinated outage scheduling o EMS system support o load and interchange forecasting o operating system maintenance o change managemento cyber security administrationo networking

Operations Support - Applications and IT

30

• Application Architects and Support areas of responsibility include, but are not limited to: o support planning of IT initiatives to evaluate and determine

new tools, technology, and integration with external drivers such as SAFNR or data requests

o attend working groupso review new software releases o set system directiono upgrade pathso ensure a timely response for application support o ensure continued compliance with new CIP Standards


31

• IT and Applications increases since 2009:o Number of applications increased from seven to 16o Number of users increased from 15 to 34

• As a cost savings measure, RC tool applications have been developed in-house o Creating a burden: volume of custom applications

requiring on-going maintenance have increased o Number of developed applications increased from

two to 53

• Support staff increased from five to six


32

Operations Support - IT Infrastructure

0

20

40

60

80

100

120

140

160

148

171216 14

23

42

17 20

37

114

23

58 61

156

IT Infrastructure

2009

2010

2011

2012

33

Operations Support - IT Infrastructure

2009 2010 2011 20120

50

100

150

200

250

40

8090

232

0 0

50

218

1 2 4 4

Server Count Increase

Non-WISP Servers

WISP Servers

# of Dedicated Systems Administrators

• In 2009 average was 40 servers/IT employee• Today, it is over 112 servers/IT employee

34

WISP − IT Infrastructure

• WISP is cutting edge technologyo WECC staff to develop and maintain custom

software for the WISP programo There will be a 40 percent increase in the

amount of data storage in SQL databases with the synchrophasor systems

o Voltage Stability Analysis will be integrated with 1500 points of synchrophasor data that will be maintained and tuned

35

• Additional personnel needed in the following in order to:o Identify and address model deficiencieso Make necessary improvements to the modelo Ensure proper model validation with TOPso Model all WECC Region RAS and real-time RAS

data points in the WECC RC EMSo Provide maintenance and support for an

increasing number of users, applications, and servers

o Ensure compliance

Operations Support − RCTF Findings

Operations Support

•Application Architects•Application Support•DTS Support•EMS Modeling•RAS Engineer•Architecture Support•Communications•IT Infrastructure•Systems Administrator

20 3 12

FTE Totals



37

• WECC RC Compliance staffing has experienced high turnover in the last three years

• As of 11/26/12 only one staff position is dedicated to Complianceo Compliance monitoring and documentation

undertaken by upper-level RC management resulting in additional work often outside normal duty hours

Compliance

38

• The WECC RC increased staffing will improve the ability to:o Meet the March 2012 WECC Board Resolution

that established clear compliance expectations for WECC

o Meet increased compliance and reliability responsibilities

o Meet increased regulator expectationso Mitigate increasing regulatory and

financial risks

Compliance - RCTF Findings

Compliance

•Document and Evidence Management

•Audit Preparation•Event Analysis•Standards Development•Compliance Reporting•Disturbance Reporting

111

FTE Totals



40

• Reliability Coordination Trainerso Training responsibilities are identified in NERC

Reliability Standards

• DTS Engineers o Support increase in system restoration drills,

exercises, and simulationso Assist in developing implementation of new

TOP and region-specific restoration training scenarios

Training

41

NOTE: Includes projected hours for remainder of 2012

Internal -Training Hours

CEHs (Hrs) Standards (Hrs) Simulation (Hrs) Professional (Hrs)

2009 109 14 66 0

2010 843.5 579 501.5 0

2011 1810 1235.5 826 1473

2012 2406 1128 995 1822

250

750

1250

1750

2250 2009

2010

2011

2012

42

• Additional personnel recommended to:o Provide required training to RC staffo Provide required training to regional reliability

entitieso Ensure training documentation is maintained as

a providero Ensure compliance with NERC Reliability

Standards

Training - RCTF Findings

Training

•Develop/Deliver in-house training to new RC trainees and current RC staff to ensure compliance with mandatory standards and retention of NERC RC System Operator certification

•Conduct just-in-time training for new tools, applications, procedures, and processes

• Update job task analysis •Develop/deliver NERC certified restoration training to BAs, TOPs, and GOs

456

FTE Totals



44

Conclusions

• WECC has a high level of responsibility and is at risk for reliability and compliance issueso Staffing is a major factoro Tools are a major factor

• Emergent tools and Standards will add to these risksoWISP related displays, data, trainingo PER-005 and EOP-005

45

• Increase monitoring areas from two to three• Add 47 WECC RC FTEs to enhance and

improve the capability of the RC as follows:o 2013 - 17 FTEso 2014 - 22 FTEso 2015 - 8 FTEs

• Acquire additional tools as identified in the projected budget

RCTF Recommendations


821647

Totals

Operations Planning

• Current-day/ real-time Support




9 5 11

FTE Totals







33 0 14

FTE Totals

Compliance






Reporting

111

FTE Totals

IT Applications, Engineering

Support, & IT Support




• RAS Engineer



Administrator

FTE Totals

1220 3

WISP






029

FTE Totals

Training




• Develop/deliver NERC certified restoration training to BAs, TOPs, and GOs

456

FTE Totals

Other






504

FTE Totals


47

Budget Projections1 BASE BUDGET

2012 2013 2014 2015 2016 20172 Includes WISP, RC tools, approved FTE's and an across

the board 3% inflation increase $14,438,927 $19,024,411 $20,400,136 $21,010,320 $21,588,266 $23,169,891

3 TOTAL DOLLAR INCREASE FROM PREVIOUS YEAR $1,490,803 $3,585,484 $2,375,725 $610,184 $577,946 $1,581,6254 PERCENT INCREASE FROM PREVIOUS YEAR 10% 19% 12% 3% 3% 7%5 FUTURE REQUIREMENTS 6

Approved but unbudgeted FTE salaries, loading and set up cost. Not budgeted, staggered hiring in 2012 but included in 2013 base budget. And, over-runs for Operating Expenses and Fixed Assets. $1,520,569

7 18 unbudgeted requested 2013 FTE salary & loading costs. Staggered hiring in 2013. Full cost in 2014 and beyond. Includes 3% inflation increase on salaries.

$1,648,480 $1,702,759 $1,757,038 $1,811,317 $1,865,5968 Build out unbudgeted construction cost for 2013

requested 12 FTE. (6 FTE are in real-time control room)

$157,500 9 22 requested 2014 FTE salary & loading costs.

Staggered hiring in 2014. Full cost in 2015 and beyond. Includes 3% inflation increase on salaries.

$2,098,286 $2,164,627 $2,230,968 $2,297,30910 Build out construction cost for 2014 requested 14 FTE

(8 FTE are in real-time control room) $183,750 11 8 requested 2015 FTE salary & loading cost.

Staggered hiring in 2015. Full cost in 2016 and beyond. Includes 3% inflation increase on salaries.

$854,394 $878,518 $902,64212 Build out construction cost for 2015 requested 5 FTE

(3 FTE are in real time control room) $65,625 13 We will need to acquire 11250 Sq feet of space in

2013 as it becomes available in LRCC and VRCC so we don't lose the option in the current buildings. This is an added $17,000 per month. Assuming full cost beginning in 2013

$204,000 $204,000 $204,000 $204,000 $204,00014 Increase each year $1,520,569 $2,009,980 $4,188,795 $5,045,684 $5,124,803 $5,269,54715 Totals for base & future budgets (line 14 plus line 2) $15,959,496 $21,034,391 $24,588,931 $26,056,004 $26,713,069 $28,439,43816 PERCENT INCREASE FROM BASE BUDGET 10% 10% 17% 19% 19% 19%

NOTE -- 2013 DOE expansion of control rooms is $3.5 million dollars, anything outside of the $3.5 million will be a WECC cost.

NOTE -- the Enhanced Curtailment Calculator is in 2014 WECC Business Plan but not in the budget because it has not been approved by the WECC OC.

NOTE -- 2012 budget is projected to exceed base approved budget by $1,520,569 due to cost overrun in Personnel, Operating Expenses and Fixed Assets.

NOTE -- 2013 and 2014 Total Dollar Increase From Previous Year (Line 4) is due to change in WISP funding from DOE to WECC

NOTE -- 2017 Total Dollar Increase From Previous Year is due to hardware refresh for WISP support

Questions?

49

Back Up Slides

50

Control Room Facilities Comparison

MISO

WECC RC Vancouver

51

Acronym GuideBA – Balancing AuthorityBES – Bulk Electric SystemCIP – Critical Infrastructure ProtectionDTS – Dispatcher Training SimulatorEMS – Energy Management SystemFTE – Full-time EquivalentIROL – Interconnection Reliability Operating LimitNERC – North American Electric Reliability CoordinationRAS – Remedial Action SchemeRC – Reliability CoordinatorRTCA – Real-time Contingency AnalysisSOL – System Operating LimitSONGS – San Onofre Nuclear Generating StationTOP – Transmission OperatorUSF – Unscheduled FlowWBRTF – West-wide System Model and Base Case Reconciliation Task Force WISP – Western Interconnection Synchrophasor ProjectWSM – West-wide System Model

52

Real Time Operations

• Subregional strategic shift from two to three areas:o More RC System Operators monitoring the systemo Wide area coordination with specific and targeted

increase in focus o Dedicated on-shift engineer in each control room to

ensure accuracy of data and provide study supporto Improved and more specific regional procedures,

operating guides, and corresponding displays

53

2009 2010 2011 20120

2

4

6

8

10

12

14

16

Applications Support 2009 - 2012

Number of Applica-tions

Support Staff

Operations Support - Applications

7

5

10

56 6

11

16

54


2009 2010 2011 20120

5

10

15

20

25

30

35

Support Staff to Users Ratio 2009 - 2012

Users

Support Staff

15

24

28

34

5 56 6

55


2009 2010 2011 20120

10

20

30

40

50

60

25

24

53

0 1 2 2

Application Development 2009 - 2012

Developed Applications

Development Staff

Operations Support - RCWorkbook – Monitoring RTCA

CONFIDENTIAL

Post-contingent branch, interface, and bus voltage limit exceedances presented. Island contingencies as well as unsolved contingencies (potential IROLs) are presented using this WECC RC developed tool.

57

External -Training Hours

CEHs (Hrs) Standards (Hrs) Simulation (Hrs) Professional (Hrs)

2009 1890 326 1414 0

2010 3903 1781 2916.5 0

2011 6438 2538 5321.5 3010

2012 8345 2870 7462 8345

500

1500

2500

3500

4500

5500

6500

7500

8500

2009

2010

2011

2012

58

Training - External Savings

Airfare and car rental are not included in the estimated savings

2009 2010 2011 2012 2013 Restora-tion 4 Regions

(Projected)

Registration 59000 122000 201250 260750 250000

Per Diem + Hotel 47200 97600 161000 208600 200000

Grand Total Savings 106200 219600 362250 469350 450000

$25,000

$75,000

$125,000

$175,000

$225,000

$275,000

$325,000

$375,000

$425,000

$475,000

Registration

Per Diem + Hotel

Grand Total Savings

59

Training - Internal Cost Savings

Airfare and car rental are not included in the estimated savings

2009 2010 2011 2012

Registration 3500 26250 56500 60000

Per Diem + Hotel 2800 21000 45200 48000

Grand Total Savings 6300 47250 101700 108000

$10,000

$30,000

$50,000

$70,000

$90,000

$110,000

Registration

Per Diem + Hotel

Grand Total Savings

60


2009 2010 2011 2012Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3

0

5000

10000

15000

20000

25000

Number of Recorded Calls

Vancouver RC

Loveland RC

Are these numbers by Quarter?

61

Real Time Operations

BA TOP0

5

10

15

20

25

6

13

15

24

1617

BA/TOP Sub-Regional Assignments

(CA/MEX)

Loveland RC (Rocky Mountain and Desert SW)

Vancouver RC (Pacific NW)

Current FTE – FilledCurrent FTE Approved – Unfilled

8115

Totals

Operations Planning

• Current -day/real-time Support




9 5

FTE Totals







33 0

FTE Totals

Compliance






Reporting

00

FTE Totals

Operations Support




• RAS Engineer



Administrator

FTE Totals

20 3

WISP






29

FTE Totals

Training




• Develop/deliver NERC certified restoration training to BAs, TOPs, and GOs

56

FTE Totals

Other






04

FTE Totals

Existing WECC RC Functions and Current Staffing

63

Improving Models - Adding Measurements

Feb-0

9

Apr-0

9

Jun-

09

Aug-0

9

Oct-09

Dec-0

9

Feb-1

0

Apr-1

0

Jun-

10

Aug-1

0

Oct-10

Dec-1

0

Feb-1

1

Apr-1

1

Jun-

11

Aug-1

1

Oct-11

Dec-1

1

Feb-1

2

Apr-1

2

Jun-

12

Aug-1

20

10000

20000

30000

40000

50000

60000

Data Points Monitored by WECC RC

MW, MVAR, tap posi-tions

CB & Switch Status

Adding measurements is necessary but translates to more alarms for the RC System Operators. Many measurements are still needed.

Documents

Presented by Members: Tom Botello - Southern California Edison Terry Baker - Platte River Power Authority Reliability Coordination Task Force Report to