August 14, 2003 Blackout Summary Based on Interim Report of the United States – Canada Power...

August 14, 2003 Blackout

Summary Based on Interim Report of theUnited States – Canada Power Outage Task Force

November 19, 2003

2

U.S.-Canada Interim Report

● Released November 19, 2003● Result of an exhaustive bi-national

investigation Working groups on electric system, nuclear

plant performance and security Hundreds of professionals on investigation

teams performed extensive analysis

● Interim report produced by the teams and accepted by the bi-national Task Force

3

Overview

● Overview of power system and reliability

● Pre-outage conditions on August 14● Trigger events and start of cascade● Wide area cascade● Root causes● Next steps

4

Power System Overview

5

Reliability Overview

● Balance generation and demand● Balance reactive power supply and

demand● Monitor flows and observe thermal limits● Observe power and voltage stability limits● Operate for unplanned contingencies● Plan, design and maintain a reliable

system● Prepare for emergencies

Reliably operate the system you have!

6

3 Interconnections / 10 NERC Regions

7

NERC Control Areas

8

NERC Reliability Coordinators

9

Footprints of Reliability Coordinators in Midwest

10

NERC Immediate Response to Blackout

● First hours Worked closely with NERC Reliability

Coordinators Identified what had tripped and extent

of outage Assessed restoration efforts Maintained open line with DOE/FERC Communicated with DHS, White

House, and NRC

● First days Assigned project manager Established Steering Group with

industry executive experts Began organizing investigation teams 90+ volunteers + entire NERC staff

11

Investigation Organization Overview

Steering Group

MAAC/ECAR/NPCC Coordinating Group

MAAC

ECAR

NPCC

MEN Study Group

Project Planning and Support

Sequence of Events

Data Requests and Management

Investigation Team Lead

System Modeling and Simulation Analysis

NERC & Regional Standards/Procedures

& Compliance

Transmission System Performance,

Protection, ControlMaintenance & Damage

Operations - Tools, SCADA/EMS

Communications Op Planning

System Planning, Design, & Studies

Root Cause AnalysisCooper Systems

Generator Performance, Protection, Controls

Maintenance & Damage

U.S – CanadaTask Force

Vegetation/ROW Management

Frequency/ACE

Restoration

Investigation Process Review

12

Data Gathering and Analysis

● Three fact-finding meetings August 22 September 8-9 October 1-3

● Onsite interviews and inspections● Secure database of outage information● Extensive corroboration of data to

determine facts● Analysis by teams of technical experts

13

Root Cause Analysis

● Logical structure for investigating complex problems

● Identifies changes, conditions, actions, or inactions at each causal step

● Starts with final event and drills back through each branch of causal tree.

● Asks “why?” at each step.● Accurate, reliable, defensible understanding

of the root causes.

Successfully used to investigate root causes of PJM voltage stability condition in July 1999 and established history in nuclear and defense industries.

14

Root Cause Analysis Phases

Pre-Existing ConditionsE.g. voltages, wide- area transfers,

line and generator outages, etc.

Sammis – StarStar – South Canton

Hanna – JuniperChamberlin - Harding

BLACKOUT16:15

16:06

15:05

Initial Focus

15

August 14 Conditions Prior to Blackout● Planned outages

Cook 2, Davis Besse nuclear plants East Lake 4, and Monroe 1

● Transfers high to northeast U.S. + Ontario Not unusually so and not above transfer limits

● Critical voltage day Voltages within limits Operators taking action to boost voltages

● Frequency Typical for a summer day

● System was within limits prior to 15:05, on both actual and contingency basis

16

Warm But Not Unusual for August

17

August 14 Imports to Northeast-Central Compared to 6/1 to 8/13/2003

18

Voltages Prior to 15:05 EDT August 14

19

Frequency Typical for Summer Day

20

Blackout was NOT Caused by

● Heavy wide-area transfers● Low voltages, voltage collapse● Lack of IPP voltage/reactive support● Frequency anomalies● Cinergy outages starting at 12:08● East Lake 5 trip at 13:31

Contributing factor to later events, but not by itself causal to the blackout

● DPL Stuart-Atlanta trip at 14:02 Contributing factor to loss of MISO real-time

monitoring, but not electrically significant

21

Outage Sequence of EventsTransmission Map Key

O N T A R I O

Transmission Lines

765 kV

500 kV

345 kV

230 kV

Transmission Lines

765 kV

500 kV

345 kV

230 kV

22

East Lake 5 Trip: 1:31:34 PM

ONTARIO

2

1

ONTARIO

23

East Lake 5 Exciter Failure Causes Trip

24

Stuart Atlanta Trip: 2:02 PM

ONTARIO

25

MISO State Estimator and Reliability Analysis

● MISO state estimator and contingency analysis ineffective from 12:37 to 16:04 State estimator not solving due to missing

information on lines out in Cinergy then DPL

Human error in not resetting SE automatic trigger

● Using Flowgate Monitoring tool to monitor conditions on previously identified critical flowgates

26

FirstEnergy Computer Failures

● 14:14 Alarm logger fails and operators are not aware No further alarms to FE operators

● 14:20 Several remote consoles fail● 14:41 EMS server hosting alarm processor and other

functions fails to backup● 14:54 Backup server fails

EMS continues to function but with very degraded performance (59 second refresh)

FE system data passed normally to others: MISO and AEP AGC function degraded and strip charts flat-lined

● 15:08 IT warm reboot of EMS appears to work but alarm process not tested and still in failed condition

● No contingency analysis of events during the day including loss of East Lake 5 and subsequent line trips

27

Phone Calls to FirstEnergy

● FE received calls from MISO, AEP, and PJM indicating problems on the FE system but did not recognize evolving emergency 14:32 AEP calls regarding trip and reclose of Star-

S. Canton 15:19 AEP calls again confirming Star-S. Canton

trip and reclose 15:35 Calls received about “spikes” seen on

system 15:36 MISO calls FE regarding contingency

overload on Star-Juniper for loss of Hanna-Juniper 15:45 FE tree trimming crew calls in regarding

Hanna-Juniper flashover to a tree PJM called MISO at 15:48 and FE at 15:56

regarding overloads on FE system

28

Chamberlin-Harding (3:05:41)

29

Chamberlin-Harding Indication of Ground Fault Due to Tree Contact as Measured by DFR at Juniper

30

(3:05:41) Hanna-Juniper(3:32:03)

31

Hanna Juniper Confirmed as Tree Contact atLess than Emergency Ratings of Line

32

Effects of Ambient Conditions on Ratings

33

(3:05:41)(3:32:03)

Star- S. Canton (3:41:35)

34

Situation after Initial Trips 3:05:41 – 3:41:35

ONTARIO

35

Canton Central – Tidd(3:45:41)

36

138 kV Lines Overload and Cascade Near Akron

Simulated 138 kV Line Loadings

0

20

40

60

80

100

120

140

160

180

200

Outages

% o

f N

orm

al R

atin

gs

(Am

ps)

Dale-W.Can138 kV

W.Ak-PVQ22 138 kV

Cham-W.Ak138 kV

E.Lima-N.Fin 138 kV

CantC Xfmr

W.Ak-PVQ21 138 kV

Babb-W.Ak138 kV

E.Lima-N.Lib 138 kV

Clov-Torrey138 kV

Da

le-W

.Ca

n1

38

kV

W.A

k 1

38

kV

Bk

r Fa

ilure

E.L

ima

-N.F

in1

38

kV

Ca

ntC

Xfm

r

W.A

k-P

V Q

21

13

8 k

V

Ba

bb

-W.A

k1

38

kV

E.L

ima

-N.L

ib1

38

kV

Clo

v-T

orre

y1

38

kV

Sta

r-S.C

an

t3

45

kV

Ha

nn

a-J

un

34

5 k

V

Ha

rd-C

ha

mb

34

5 k

V

Ch

am

-W.A

k1

38

kV

37

15

:05

:41

ED

T

15

:32

:03

ED

T

15

:41

:35

ED

T

15

:51

:41

ED

T

16

:05

:55

ED

T

0

20

40

60

80

100

120

140

% o

f N

orm

al R

atin

gs

Can

ton

Cen

tral Tran

sform

er

Bab

b-W

.Akro

n 138 kV

Hard

ing

-C

ham

berlin

Han

na-

Jun

iper

Star-S

.Can

ton

Clo

verdale-T

orrey 138 kV

E.L

ima-N

ew L

iberty 138 kV

W.A

kron

-Pleasan

t Valley 138 kV

E.L

ima-N

.Fin

lay 138 kV

Ch

amb

erlin-W

.Akro

n 138 kV

W.A

kron

138 kV B

reaker

Dale-W

.Can

ton

138 kV

Sammis-Star

138 kV Cascade Contributes Furtherto Overload of Sammis-Star

38

Sammis-Star(4:05:57.5)

39

Sammis-Star Zone 3 Relay Operateson Steady State Overload

40

Actual Loading on Critical Lines

0

400

800

1200

1600

12:00 13:00 14:00 15:00 16:00

Time - EDT

Flo

ws

(M

W)

Harding - Chamberlin

Hanna - JuniperStar - South Canton

Sammis - Star

East Lake 5 Trip

Harding - Chamberlin Line Trip

Hanna - Juniper Line Trip

Sammis - Star Line Trip

Star - South Canton Line Trip

41

Actual Voltages Leading to Sammis-Star

250

270

290

310

330

350

370

15:00 16:00Time - EDT

Vo

ltag

e (k

V)

Star

Hanna

Beaver

Perry

100% Voltage

95% Voltage

90% Voltage

Sammis - Star

345 kV Line Trip

Star - South Canton345 kV Line Trip

Hanna - Juniper345 kV Line Trip

Harding - Chamberlin345 kV Line Trip

Gaps in Data Records

42

Major Path to Cleveland Blocked after Loss of Sammis-Star 4:05:57.5 PM

RemainingPaths

43

345 kV Lines Trip Across Ohio to West

ONTARIO

44

Generation Trips 4:09:08 – 4:10:27 PM

ONTARIO

45

345 kV Transmission Cascade Moves North into Michigan 4:10:36 – 4:10:37 PM

46

Northern Ohio and Eastern Michigan Served Only from Ontario after 4:10:37.5 – 4:10:38.6 PM

47

Power Transfers Shift at 4:10:38.6 PM

48

Eastern Eastern Michigan (Detroit) UnstableVoltage and Frequency Collapse and Pole Slipping

Ontario – Michigan Interface Flow and Voltages Beginning 16:10:38

49

Generator Trips to 16:10:38

50

Generator Trips – Next 7 Seconds

51

Overloads on PJM – NY Ties 4:10:39 PM

52

PJM – NY Separating 4:10:44 PM

53

Cleveland – Toledo Island 4:10:39 - 4:10:46 PMCleveland Blacks Out

54

Northeast Completes Separation from Eastern Interconnection 4:10:43 – 4:10:45 PM

North of LakeSuperior

55

Conditions at Niagara Indicate Progressively Worsening Stability Conditions with Prior Events

56

Island Breaks Up: 4:10:46 – 4:13 PM

57

Frequency in Ontario and New York during BreakupNiagara Generation Stays with Western NY

58

Generator Trips – After 16:10:44

59

Areas Affected by the BlackoutService maintained

in some area

Some Local Load Interrupted

End of the Cascade

60

Blackout Root Cause Group 1FE Situational Awareness

● FE did not ensure a reliable system after contingencies occurred because it did not have an effective contingency analysis capability

● FE did not have effective procedures to ensure operators were aware of the status of critical monitoring tools

● FE did not have effective procedures to test monitoring tools after repairs

● FE did not have additional high level monitoring tools after alarm system failed

61

Blackout Out Root Cause Group 2Vegetation Management

● FE did not adequately manage tree growth in its transmission rights of way

62

Blackout Cause Group 3Reliability Coordinator Diagnostics

● MISO’s state estimator failed due to a data error.

● MISO’s flowgate monitoring tool didn’t have real-time line information to detect growing overloads

● MISO operators couldn’t easily link breaker status to line status to understand changing conditions.

● PJM and MISO ineffective procedures and wide grid visibility to coordinate problems affecting their common boundaries

63

Voltage support/reactive supply

Reliability communications

Computer failure response & notifications

Emergency action plans and capabilities

Operator training for emergencies

Vegetation management

Near-Term Industry ActionsResponses from Control Areas and

Reliability Coordinators Due December 15

64

Next Steps

● U.S./Canada Power Outage TF hearings Public hearings to allow comment on report and

input on recommendations December 4 December 5 December 8 – Toronto

Industry technical conference December 10 – Philadelphia

● NERC next steps NERC executive committees December 11 NERC committees meet January 13-14 Continue investigation

Near term analysis and recommendations in support of U.S. Canada Task Force

Long term analysis and recommendations for NERC