1 Lessons Learned 2011 Cold Snap Event System Operator Conference September 11-13 and September 25-27, 2012 Memphis, TN Steve Corbin Reliability Specialist

1

Lessons Learned2011 Cold Snap Event

System Operator ConferenceSeptember 11-13 and September 25-27, 2012

Memphis, TN

Steve CorbinReliability Specialist

SERC Reliability Corp.

2012 System Operator Conference

2

At the conclusion of this training session, you should be able to:

• Give examples of preparations needed for impending cold weather at a generating plant

• Identify issues an operator must know about manual load shedding

• Discuss the importance of Lessons Learned Recommendations relative to institutionalizing and training on such recommendations.

Learning Objectives


3

Outline of Topics

• Highlights of previous winter events in ERCOT

• Highlights of the cold-weather events of February 1, 2, 2011 in the Southwest

• Review the 4 of the 8 NERC published Lessons Learned from the 2011 cold snap event


4

Cold Front • The arctic cold front that descended on the Southwest

during the first week of February 2011 was indisputably severe.

• Many cities in Texas and New Mexico experienced a 50 degree drop in temperature over an eighteen-hour period.

• Temperatures dropped to the low teens in Texas and below zero in New Mexico. Much of north Texas experienced record setting sleet and snow, totaling up to seven inches.

• Exacerbating the effects of the cold temperatures were accompanying sustained winds of 30-40 mph, with gusts as high as 51 mph.


5

This had happened before, no sweat

• 1983, 1989, 2003, 2006, 2008, and 2010


6

December 21-23, 1989 February 1-2, 2011

Min. Temps & WindChills in Dallas Area

Temperature: -1 degrees F

Wind Chill: -12 degrees F

Temperature: 13 degrees F

Wind Chill: -6 degrees F

Peak System Load 38,300 MW 56,334 MW

Net GeneratingCapacity Reduction

11,809 MW31% of peak load

14,702 MW26% of peak load

Gross GeneratingCapacity Reduction

56+ units16,805 MW

193 units29,729 MW

Firm Load Shed 1710 MW4.5% of peak load

4900 MW8.7% of peak load

Overall Duration ofFirm Load Shedding

5 hours, 47 minutes 7 hours, 24 minutes


7

PUCT Recommendations in 1989

• All utilities should ensure that they incorporate the lessons learned during December of 1989 into the design of new facilities in order to ensure their reliability in extreme weather conditions.

• All utilities should implement procedures requiring a timely annual (each Fall) review of unit equipment and procedures to ensure readiness for cold weather operations.


8


• All utilities should ensure that procedures are implemented to correct defective freeze protection equipment prior to the onset of cold weather.

• All utilities should maintain insulation integrity and heat tracing systems in proper working order. Generating unit control systems and equipment essential to cold weather operations should be included in a correctly managed preventive maintenance program.

• Additional training programs for plant personnel on the emergency cold weather procedures, including periodic drills, should be implemented by each responsible utility.


9


• PUC Engineering Staff should modify procedures for power plant CCN [Certificates of Convenience and Necessity] reviews to include a specific review for plant reliability under adverse weather conditions.

• Of special interest would be the selection of proper design temperature ranges for the power plant site.


10

PUCT Recommendations of 2003• The Market Oversight/Reliability Division of the PUCT

investigated the 2003 cold weather event and issued a number of recommendations. Notable among these are the following:– Stricter enforcement of Resource Plan accuracy.– Improved weather and electric demand forecasting.– Consider providing financial incentives for fuel oil inventories to be

maintained for use by dual fueled units.– Curtailment prioritization – development of a joint curtailment

methodology for natural gas and electricity production.– ERCOT should communicate with both QSEs and Transmission /

Distribution Service Providers in the future when the power system is under stress.


11

2011 Lessons Learned4 LLs Reviewed of 8 Currently Posted by NERC

• Rotational load shed• Transmission facilities and winter weather operations• Plant fuel switching and cold weather• Generating unit temperature design parameters and

extreme weather conditions


12

Rotational Load Shed

Primary Interest Groups (PIGs): TOP, LSE, DP, TO Problem Statement Registered entities implemented rolling blackouts in response to directives issued by the reliability coordinator (RC) during an Energy Emergency Alert (EEA) event. During the rolling blackouts, operators experienced difficulties that may have been avoided with the use of improved operator tools and processes.


13


Details • The following challenges and issues were identified:• SCADA alarms were not distinguishable between non-rolling blackout

circuits and rolling blackout circuits. • The number of circuits available for rolling blackouts was limited by

exempt customers, critical loads, and under-frequency or under-voltage load-shedding circuits.

• Several circuits were not correctly identified as critical load, including hospitals and jails

• Issues were identified with the long duration of the rolling blackouts for some individual circuits, i.e. there were problems with re-energizing circuits due to high cold-load pickup inrush currents


14

Rotational Load Shed - Details• Real-time communications were primarily by phone,

making it difficult to keep circuit status documented. • Entities used manual spreadsheets to track and update

rolling blackout circuit status and had trouble keeping the spreadsheets updated.

• Maps of area outages were not immediately available to meet the needs of management, stakeholders, and media.

• Communications processes were not fully defined or fully implemented with other local government agencies, local law enforcement, local emergency management teams, and other critical facilities.


15


Lesson Learned • Periodic system operator training proved to be invaluable

for this event. No issues were identified due to lack of personnel training.

• Load-Serving Entities (LSE) should ensure they have identified enough load circuits that can be used in manual load sheds to meet the needs of the area load shed, to meet the regions criteria for under-frequency relaying requirements and to protect critical loads. These lists should be reviewed and update periodically.


16


Lessons Learned, continued• Operational tools and EMS SCADA displays which show

and track load shed outages and outage durations must be available to system operators to help manage load shed events.

• Registered entities must have procedures in place to periodically review and update circuit data for load shed events, to keep critical customer lists updated and to ensure under-frequency or under-voltage load-shed circuit lists are updated.


17


Lessons Learned, continued• Registered entities should have fully defined

communication processes with other local government agencies, local law enforcement, and local emergency management teams. Periodic exercises should be in-place to test these communication processes.

• Maps showing outage areas should be readily available for management, stakeholders and media. It is critical in such an event that these entities are kept updated. This responsibility should be performed by support personnel so as not to distract operators from operations.


18

Transmission Facilities and Winter Weather Operations

PIGs: GO, GOP, TO, TOP

Problem Statement• Unit 1 at a 1700 Mw plant was forced off line during a

severe weather event due to the failure of the disconnect "B" phase switch.


19


Details• After a winter storm at a generating plant, the "B" phase

disconnect switch connection started to arc in the plant switchyard.

• The Control Room was notified immediately and operations manually tripped the turbine and took the generator off line. It was found that the arc originated toward the pivot point of the switch just inside the contact region. The disconnect switch did not arc in the contact area of the switch. Once electricians removed the disconnect switch arm it was found to be full of ice.


20


Details, continued• Under normal circumstances water should not be inside the

disconnect switch arm. Water, which had ingress in the disconnect arm, froze during the cold weather, causing the weld on the end of the arm to crack and reduced the current carrying capabilities of the arm. This added heat to the area, which progressed to an arc.


21


Lessons Learned• All plant and transmission electrical disconnect equipment

should be inspected for areas where water could collect and freeze – especially in geographic areas not accustomed to freezing temperatures. Any areas found with the potential to retain moisture should be modified as prescribed by the equipment manufacturer or by an adequate engineer design.

• With regards to transmission and substation equipment, other lessons which the GO, GOP, TO and TOP should consider in their winter preparation plans are:


22


Lessons Learned, continued • Power transformers should have their equipment checked

to ensure:– Control cabinet heaters are working correctly– Oil is at its appropriate level for the oil temperature.– Bushing oil levels are appropriate– Nitrogen levels are correct– SF6 gas‐insulated devices should be checked to ensure:

SF6 gas in breakers, current transformers and other electrical equipment is at the correct pressure and temperature to operate safely during extreme cold


23


– Heaters (when so‐equipped) and associated support circuits are functioning correctly

• The minimum operating ambient temperature should be known on all transmission facilities. This temperature should also take into account fire protection systems.

Lessons Learned, continued


24

Plant Fuel Switching and Cold Weather

• PIGs: RC, BA, GO, GOP

• Problem Statement

• During a cold weather event, both equipment freezing and infrequent use of alternative fuels caused a plant’s generation to become unreliable, contributing to the BA and RC having to shed load.


25


Details• A two unit 900 MW natural gas powered plant developed

significant fuel switching issues and unit reliability problems during a severe winter weather event.

• The plant experienced temperatures low as 18 degrees (F), wind chill as low as 1 degree (F). For three days, the daytime temperatures stayed below freezing. During the winter weather event, the plant’s gas supply was interrupted.


26


• Operation on fuel oil was infrequent:– fuel oil burning equipment remained dormant much of

the time. – As a result, equipment required for burning fuel oil had

high failure rates, in part due to sub‐freezing temperatures, but also due to infrequent use.


27


The plant experienced:– Instrumentation sensor lines freezing– Condensation freezing in the air inlet needle valve for a

Gas Turbine (GT)– Reduced unit output due to gas transmission

curtailments– Failed oil cooler fan due to freezing issues– Unit tripping multiple times when attempting to fire on

fuel oil due to equipment malfunctioning and operator error


28


Lessons Learned• GOs and GOPs should review their winter maintenance

plans and ensure:– critical sensing and operating equipment are protected

and can operate at the plant’s designed minimum operating ambient temperature for all fuels the plant is capable of burning.


29


Lessons Learned, continued• GO/GOPs should consider the following if they have dual

fuel operation capability:– Reliable unit fuel oil operation requires dormant

equipment to be tested and maintained. Entities should evaluate their testing programs to insure all systems can operate reliably at all times and provide training to appropriate personnel on the maintaining and operation of the equipment.


30

Plant Fuel Switching and Cold WeatherLessons Learned, continued• GO/GOPs should consider the following if they have dual

fuel operation capability:– Perform review and update of procedures periodically for

unit fuel oil operation and maintenance based upon expected operation capabilities to ensure reliable operation.

• Training plans:– consider incorporation of processes to institutionalize the

knowledge of procedures for fuel switching and operation for the non primary fuel mode of operation.


31


• Alternate fuel supplies should be tested for contamination or degradation

• BAs and RCs should consider the following if they have plants which have dual fuel capability:– Verification of the output of each fuel capability provides

more accurate unit commitment planning.– Verification might include the following:

Document time to switch Document unit capacity and maximum runtimes while on

alternate fuel


32

Generating Unit Temperature Design Parametersand Extreme Winter Conditions

PIGs: GO, GOP, RC, BA

Problem Statement• Extreme low winter temperatures forced a plant offline

which had been providing over 1000 Mw. The plant was not designed to operate under such conditions. The Balancing Authority (BA) and Reliability Coordinator (RC) were not informed that the plant was not designed to operate during the low temperature conditions.


33


Details• During a period of extreme cold, a large power plant experienced

temperatures below freezing for as much as 100 hours. – Sustained winds of over 20 mph with gusts much higher created

wind chills in the single digits. • These weather conditions had been predicted a week earlier by

weather services in long range forecasts. The plant had taken some precautions before the cold period to protect equipment.

• Precaution steps included applying covering at various places, using heating equipment and adding items such as checks of equipment for freezing to operator “rounds” checklists, etc.


34


• Yet the plant was not designed to handle these temperatures and had numerous freezing issues. Critical pieces of equipment froze. For example:– the plant had instrumentation freeze on its Zero Liquid

Discharge Facility (ZLDF) which stopped it from making demineralized water.

– a freezing regulator on one unit caused oil to leak out of the seal on the generator. Oil from the seal got on the field brushes which started arcing. Units, some of which had just started to come online, tripped offline.


35


– The plant implemented further freeze protection such as: operating heaters for wind-sensitive transmitters

continuously to stop further freezing adding more tarps as windbreaks for the ZLDF tower. obtaining demineralization trailers from various

sources in other states, bringing them in to replace their ZLDF.


36


– The BA and RC did not know this plant had these low temperature design limitations and were counting on the plant for its generation to help meet the cold weather load peaks.

– As a result of the loss of this generation and other generation, the BA and RC were forced to implement rolling load sheds in order to reliability manage the load.


37


Lesson LearnedGenerator Owner/Operator should:• Know the ambient temperature limitations of their plant• If they don’t know…….

– they should perform a detailed engineering design analysis of their plants to determine their ability to operate during extreme cold conditions.

– A low ambient temperature limit according to the design of the plant should be established.


38


Lesson Learned continuedGenerator Owner/Operator should:• Fix issues which do not allow the plant to operate at

extreme low ambient temperature:– Identify freezing places and ensure insulation, heat

tracing, lagging and wind breaks are sufficient to maintain water temperature at 40 degrees or above.

– This practice should also be performed to protect the plant if it is offline.


39


• Establish ongoing maintenance practices which stress identification of possible freeze areas after maintenance or design changes

• Ensure those areas can meet the requirements to continue to operate during low ambient temperature periods.

• If ambient air temperatures exist which could cause a unit to runback its output, these temperature limitations should be documented and procedures and training modified accordingly.


40


• Work with the BA and RC to establish communications practices by which the GO and GOP informs the BA and RC of the plant design low ambient temperature limitation and when the plant cannot meet the design specification.

• These communications should include informing the BA and RC if weather conditions dictate that the plant must be kept online to be able to operate the next day.

• Include such notification in winter operation drills which involve the plants and other entities.


41


AND FINALLY……..• Document and institutionalize knowledge and

experiences from previous severe winter weather events, and apply this learning to the winterization procedures of plant equipment.

Severe Weather Procedures


42

Summary• Some examples of recommended preparation for impending

cold weather at a generating plant:• Perform analyses to determine low ambient temperature limit• Identify freezing places and ensure sufficient insulation & other fixes• Identify sufficient manual load shed circuits• Test alternate fuel supplies• Conduct cold weather drills for operators• Work with the BA and RC to establish communications practices

regarding temperature limitations, and when the plant cannot meet design specs.

• It is important to learn from past events – document & institutionalize your severe weather procedures based on lessons learned!


43

Questions?

Documents

1 Lessons Learned 2011 Cold Snap Event System Operator Conference September 11-13 and September 25-27, 2012 Memphis, TN Steve Corbin Reliability Specialist