1

August 14, 2003 Blackout

Gerry CauleyDirector – Reliability Performance

North American Electric Reliability Council

2

NERC Response

●First hoursWorking with reliability coordinators –assessing restoration efforts and identifying what had initially tripped

●Began organizing investigation effortOutage covered three NERC regionsEstablished a Steering Group of industry experts

●Became technical lead for the U.S.-Canada Power System Outage Task Force

2

3

Investigation Organization OverviewSteering Group

MAAC/ECAR/NPCC Coordinating Group

MAAC

ECAR

NPCC

MEN Study Group

Project Planning and Support

Sequence of Events

Data Requests and Management

Investigation Team Lead

System Modeling and Simulation Analysis

NERC & Regional Standards/Procedures

& Compliance

Transmission System Performance,

Protection, ControlMaintenance & Damage

Operations - Tools, SCADA/EMS

Communications Op Planning

System Planning, Design, & Studies

Root Cause AnalysisCooper Systems

Generator Performance, Protection, Controls

Maintenance & Damage

U.S – CanadaTask Force

Vegetation/ROW Management

Frequency/ACE

Restoration

Investigation Process Review

4

Transmission Map Key

ONTARIO

Transmission Lines

765 kV500 kV345 kV230 kV

Transmission Lines

765 kV500 kV345 kV230 kV

3

5

Footprints of Reliability Coordinators in Midwest

6

Warm But Not Unusual for August

4

7

August 14 Imports to Northeast (Except ISO-NE, and Maritimes) Compared to 6/1 to 8/13/2003

-8000

-6000

-4000

-2000

0

2000

4000

6000

0:00

2:00

4:00

6:00

8:00

10:00

12:00

14:00

16:00

18:00

20:00

22:00

Hour (EDT)

MW

Max ImportsMax ExportsAverage14-Aug

Impo

rtsEx

ports

16:00

8

Voltages Prior to 15:05 EDT August 14

5

9

Transmission Lines

765 kV500 kV345 kV230 kV

Transmission Lines

765 kV500 kV345 kV230 kV

411

399

689

8241240

2411

1518

629

1588

2753

702

18841865

Reactive Power Margins from V-Q Analysis

158

144

178

303

466468

1923

681

10702226

578

18161803

Reactive Power Margin @ 15:32

Reactive Power Margin @ 15:41

430

143

10

Avon 345kV

-1000

-900

-800

-700

-600

-500

-400

-300

-200

-100

0

100

200

300

400

500

0.7 0.8 0.9 1 1.1

Voltage (p.u.)

Rea

ctiv

e M

argi

n

15-05 Chamberlin-Harding

15-32 Hanna-Juniper

15-41 Star-S Canton

15-45-40 Canton Central-Tidd

15-59 West Akron 138kV Lines

6

11

East Lake 5 Trip: 1:31:34 PM

ONTARIO

2

1

ONTARIO

12

East Lake 5 Exciter Failure Causes Trip

7

13

Stuart Atlanta Trip: 2:02 PM

ONTARIO

14

MISO State Estimator and Reliability Analysis

●MISO state estimator and contingency analysis ineffective from 12:37 to 16:04

State estimator not solving due to missing information on lines out in Cinergy then DPLHuman error in not resetting SE automatic trigger

●Using Flowgate Monitoring tool to monitor conditions on previously identified critical flowgates

8

15

FirstEnergy Computer Failures● 14:14 Alarm logger fails and operators are not aware

No further alarms to FE operators● 14:20 Several remote consoles fail● 14:41 EMS server hosting alarm processor and other

functions fails to backup● 14:54 Backup server fails

EMS continues to function but with very degraded performance (59 second refresh)FE system data passed normally to others: MISO and AEPAGC function degraded and strip charts flat-lined

● 15:08 IT warm reboot of EMS appears to work but alarm process not tested and still in failed condition

● No contingency analysis of events during the day including loss of East Lake 5 and subsequent line trips

16

Star- S. Canton (2:27:15)

9

17

Chamberlin-Harding (3:05:41)

18

(3:05:41) Hanna-Juniper(3:32:03)

10

19

Hanna - Juniper Tree ContactInsufficient Clearance with Trees

20

Chamberlin-Harding Indication of Ground Fault Due to Tree Contact as Measured by DFR at Juniper

11

21

Effects of Ambient Conditions on Ratings

22

(3:05:41)(3:32:03)

Star- S. Canton (3:41:35)

12

23

Actual Loading on Critical Lines

0

400

800

1200

1600

12:00 13:00 14:00 15:00 16:00

Time - EDT

Flow

s (M

W)

Harding - ChamberlinHanna - JuniperStar - South CantonSammis - Star

East Lake 5 Trip

Harding - Chamberlin Line Trip

Hanna - Juniper Line Trip

Sammis - Star Line Trip

Star - South Canton Line Trip

24

Situation after Initial Trips 3:05:41 – 3:41:35

13

25

Actual Voltages Leading to Sammis-Star

250

270

290

310

330

350

370

15:00 16:00Time - EDT

Vol

tage

(kV)

Star

Hanna

Beaver

Perry

100% Voltage

95% Voltage

90% Voltage

Sammis - Star 345 kVLine Trip

Star - South Canton345 kV Line Trip

Hanna - Juniper345 kV line Trip

Harding - Chamberlain345 kV Line Trip

26

Phone Calls to FirstEnergy● FE received calls from MISO, AEP, and PJM

indicating problems on the FE system but did not recognize evolving emergency

14:32 AEP calls regarding trip and reclose of Star-S. Canton15:19 AEP calls again confirming Star-S. Canton trip and reclose15:35 Calls received about “spikes” seen on system15:36 MISO calls FE regarding contingency overload on Star-Juniper for loss of Hanna-Juniper15:45 FE tree trimming crew calls in regarding Hanna-Juniper flashover to a treePJM called MISO at 15:48 and FE at 15:56 regarding overloads on FE system

14

27

138 kV Lines Overload and Cascade Near Akron

Simulated 138 kV Line Loadings

0

20

40

60

80

100

120

140

160

180

200

Outages

% o

f Nor

mal

Rat

ings

(Am

ps)

Dale-W.Can138 kV

W.Ak-PVQ22 138 kV

Cham-W.Ak138 kV

E.Lima-N.Fin 138 kV

CantC Xfmr

W.Ak-PVQ21 138 kV

Babb-W.Ak138 kV

E.Lima-N.Lib 138 kV

Clov-Torrey138 kV

Dale-W

.Can

138 kV

W.A

k 138 kVB

kr Failure

E.Lima-N

.Fi n138 kV

CantC

Xfmr

W.A

k-PV Q21

138 kV

Babb-W

.Ak

138 kV

E.Lima-N

.Lib138 kV

Clov-Torrey

138 kV

Star-S.Cant

345 kV

Hanna-Jun

345 kV

Hard-C

hamb

345 kV

Cham

-W.A

k138 kV

28

15:05:41 EDT

15:32:03 EDT

15:41:35 EDT

15:51:41 EDT

16:05:55 EDT

0

20

40

60

80

100

120

140

% o

f Nor

mal

Rat

ings

Canton C

entral Transformer

Babb-W

.Akron 138 kV H

arding-C

hamberlin

Hanna-

Juniper

Star-S.Canton

Cloverdale-Torrey 138 kV

E.Lima-N

ew Liberty 138 kV

W.A

kron-Pleasant Valley 138 kV

E.Lima-N

.Finlay 138 kV

Cham

berlin-W.A

kron 138 kV

W.A

kron 138 kV Breaker

Dale-W

.Canton 138 kV

Sammis-Star

138 kV Cascade Contributes Furtherto Overload of Sammis-Star

15

29

Sammis-Star(4:05:57.5)

30

Sammis-Star Zone 3 Relay Operateson Steady State Overload

16

31

Last Major Path to Cleveland Blocked after Loss of Sammis-Star 4:05:57.5 PM

32

Blackout Root Cause Group 1FirstEnergy Lack of Situational Awareness● Did not ensure a reliable system after

contingencies occurred because it did not have an effective contingency analysis capability

● Did not have effective procedures to ensure operators were aware of the status of critical monitoring tools

● Did not have effective procedures to test monitoring tools after repairs

● Did not have additional high level monitoring tools after alarm system failed

17

33

Blackout Out Root Cause Group 2FirstEnergy Ineffective Vegetation Management

● Did not adequately manage ground clearance (tree clearance) in its transmission rights of way

34

Blackout Cause Group 3Reliability Coordinator Ineffective Diagnostics

●Reliability Coordinator (MISO for FE)State estimator failed due to a data error.Flowgate monitoring tool didn’t have real-time line information to detect growing overloadsOperators couldn’t easily link breaker status to line status to understand changing conditions.Did not declare emergency or take any action

●PJM & MISO ineffective procedures & wide grid visibility to coordinate problems affecting their common boundaries

18

35

345 kV Lines Trip Across Ohio to West4:08:59 - 4:09:07 PM

ONTARIO

36

Generation Trips 4:09:08 – 4:10:27 PM

ONTARIO

19

37

345 kV Transmission Cascade Moves North into Michigan 4:10:36 – 4:10:37 PM

38

Northern Ohio and Eastern Michigan Served Only from Ontario after 4:10:37.5 – 4:10:38.6 PM

20

39

Power Transfers Shift at 4:10:38.6 PM

40

NY to Ontario 345kV Line Flows at NiagaraProgressively Worsening Stability Conditions

New York to Ontario 345 kV Line Flow at Niagara(does not include 230 kV line flow)

0.00

200.00

400.00

600.00

800.00

1000.00

1200.00

1400.00

1600.00

1800.00

2000.00

16:0

5:43

16:0

5:49

16:0

5:54

16:0

6:00

16:0

6:05

16:0

6:11

16:0

6:16

16:0

6:22

16:0

6:27

16:0

6:33

16:0

6:38

16:0

6:44

16:0

6:49

16:0

6:55

16:0

8:55

16:0

9:00

16:0

9:06

16:0

9:11

16:0

9:17

16:0

9:23

16:0

9:28

16:0

9:33

16:0

9:39

16:0

9:45

16:0

9:50

16:0

9:56

16:1

0:01

16:1

0:20

16:1

0:26

16:1

0:31

16:1

0:37

16:1

0:42

16:1

0:48

MW

200.0

220.0

240.0

260.0

280.0

300.0

320.0

340.0

360.0

380.0

KV

Niagara KV

PA301&2 MW

Sammis-Star 345 trip

East Lima - Fostoria Central 345 trip

Thetford-Jewel, Hampton Pontiac,& Perry - Ashtabula 345kV lines trip

Argenta - Battle Creekdouble circuit 345 trip

21

41

Power Surge on PJM – NY Ties 4:10:39 PM

42

PJM – NY Separating 4:10:44 PM

22

43

Cleveland – Toledo Island 4:10:39 - 4:10:46 PMCleveland Blacks Out

44

Eastern Eastern Michigan (Detroit) UnstableVoltage and Frequency Collapse and Pole Slipping

Ontario – Michigan Interface Flow and Voltages Beginning 16:10:38

23

45

Severe Under Frequency Condition

46

View Into Detroit from Lambton

24

47

Northeast Completes Separation from Eastern Interconnection 4:10:43 – 4:10:45 PM

North of LakeSuperior

48

Power plants affected

1. 531 units shut down at 263 plants2. During the conclusion of the cascading

failure, generation tripped off in three general categories:1. Excitation system overload or

extreme low voltage – 35%2. Generator protection or control

system action – 34%3. Consequential result of the broken

transmission system – 31%3. Some prolonged out-of-step conditions

are evident4. To date, little damage has been

discovered as a result of the cascade

25

49

Frequency in Ontario and New York

50

Island Breaks Up: 4:10:46 – 4:13 PM

26

51

Areas Affected by the BlackoutService maintained

in some area

Some Local Load Interrupted

End of the Cascade

52

When the Cascade Was Over50 million people8 states and 2 provinces60-65,000 MW of loadinitially interrupted− Approximately 11% of Eastern Interconnection

Sammis – Star trip at 4:06 PM – Blackout essentially complete by 4:13 PMHigh speed cascading lasted approximately 12 secondsThousands of discrete events to evaluate− Time stamping - critical

27

53

Violations of NERC Reliability Standards

● FE did not return the system to safe operating state within 30 minutes (OP-2)

● FE did not notify others of impending emergency (OP-5)

● FE did not have effective monitoring capability (OP-5)

OperatingPolicies

● FE did not adequately train operating personnel for emergency response (OP-8)

● MISO did not notify others of impending emergency (OP-9)

54

Other Key Findings of Investigation●Compliance with

reliability rules requires objective measurements and firm actions to resolve violations

●NERC policies were not sufficiently specific regarding reliability coordinator and control area functions, responsibilities, authorities, tools

●Problems from prior wide-area blackouts are being repeated: trees, operator tools, training

28

55

Other Key Findings of Investigation● System planning and design

studies, operations planning, facilities ratings, and modeling data accuracy were ineffective preparations for 8/14 event

● Power system in northeastern Ohio was being operated with insufficient reactive margins to meet NERC criteria

● Protection and controls could be more effectively used to slow or minimize spread of cascade

56

Corrective Actions - FE

●Voltage criteria and reactive resources

●Operational preparedness and action plan

●Emergency response capabilities and preparedness

●Control center and operator training

Juniper 345kV

-1000

-900

-800

-700

-600

-500

-400

-300

-200

-100

0

100

200

300

400

500

0.7 0.8 0.9 1 1.1

Voltage PU

Reac

tive

Pow

er (M

VAR)

15-05 Chamberlin-Harding

15-32 Hanna-Juniper

15-41 Star-S Canton

15-45-40 Canton Central-Tidd

15-59 West Akron 138kV Lines

29

57

Corrective Actions – Reliability Coordinators

●PJMCommunications protocols and procedures

●MISOReliability toolsVisualization toolsOperator trainingCommunications protocols and proceduresOperating agreements

58

NERC Strategic Initiatives

● Performance reviews● Readiness audits● Vegetation-related

outage reporting● Recommendations

implementation tracking

30

59

NERC Technical Initiatives - 1● Operator and reliability

coordinator emergency response training

● Reactive power and voltage control

● Cascade mitigation● Reliability coordinator

and control area functions, authorities, and requirements

● Real-time operating tools● Restoration review

40 20 0 20 40 60

20

40

60

80

100

Sammis-Star 345kV @ Sammis 1606:03110

10−

Im ZBCx( )Im Z1l( )

7040− Re ZBCx( ) Re Z1l( ),

60

NERC Technical Initiatives - 2

● Time-synchronized measurements for disturbance analysis and operations

● Reevaluate system design, planning and operating criteria

● System modeling and data exchange standards