Harris Plant - Heater Drain System TransientsINPO ICES #305654 HNP Root Cause Evaluation 592995

Presentation Outline

System Overview

System History

Atypical System Characteristics

Recent Problems

Cause(s) of Problems

Generic Lessons Learned

2

Heater Drain System Schematic – ‘A’ Train

3

Heater Drain System Schematic – ‘A’ Train (Cont’d)

4

System History

Initial operation experienced significant system instability issues. Unstable HD and FRV control valve operation due to excessive pressure drops across the

valves. 10 plant trips during first five months of power ascension testing and commercial operation,

most of which were caused by BOP problems. A Study recommended seven modifications to the BOP. These included:

Removing an impeller stage from the Heater Drain Pumps. Replacing the pneumatic controllers on the #4 FWHs, #5 FWHs, and MSR Drain Tanks with

electronic level controllers. Trimming the Main Feedwater Pump impellers.

The result was significantly more stable and reliable plant operation. The modifications continued to work effectively for the next 25 years of operation.

5

Atypical System Characteristics

No Heater Drain Tank. Small level transients often result in a Heater Drain Pump trip.

Variable speed Condensate Booster Pumps. Heater Drain Pump trips have a small impact on overall plant output.

Make and model of electronic controllers are unique to the Heater Drain System. Knowledge of controllers eventually began to fade. Knowledgeable personnel (with few exceptions) moved on.

First 20 years - System largely operated and maintained by tribal knowledge and as-needed corrective maintenance.

6

Overall Timeline of NCRs December 1, 2010 – March 5, 2013

7

Attachment 3 Overall Timeline of NCRs

December 1, 2010 – March 5, 2013

Note: Three Clusters of Events

4379

8643

8110

4397

29

4397

7443

9824

4398

28

4404

7944

0623

4417

2044

1734

4445

37

4531

8145

4157

4672

87

4695

7146

9850 4742

08

4791

9347

9198

4798

9247

9893

4919

30

5032

0150

3429

5115

03

5186

03

5331

98

5423

7354

3175

5458

18

5517

71 5546

85

5622

08

5652

14

5675

82

5717

2757

2026

5728

3257

2835

5728

7657

3113

5734

3057

3457

5734

7157

3473

5737

2057

3783

5759

96

5862

6958

8010

5884

6258

9179

5895

6358

9576

5897

5259

0483

5915

6759

2296

5923

3659

2995

5938

43

11/1

4/10

11/2

8/10

12/1

2/10

12/2

6/10

1/9/

11

1/23

/11

2/6/

11

2/20

/11

3/6/

11

3/20

/11

4/3/

11

4/17

/11

5/1/

11

5/15

/11

5/29

/11

6/12

/11

6/26

/11

7/10

/11

7/24

/11

8/7/

11

8/21

/11

9/4/

11

9/18

/11

10/2

/11

10/1

6/11

10/3

0/11

11/1

3/11

11/2

7/11

12/1

1/11

12/2

5/11

1/8/

12

1/22

/12

2/5/

12

2/19

/12

3/4/

12

3/18

/12

4/1/

12

4/15

/12

4/29

/12

5/13

/12

5/27

/12

6/10

/12

6/24

/12

7/8/

12

7/22

/12

8/5/

12

8/19

/12

9/2/

12

9/16

/12

9/30

/12

10/1

4/12

10/2

8/12

11/1

1/12

11/2

5/12

12/9

/12

12/2

3/12

1/6/

13

1/20

/13

2/3/

13

2/17

/13

3/3/

13

3/17

/13

3/31

/13

Key: Black Font – NCRs that were used for event investigation. Red Font – Reactivity Management NCRs Green Font – Quick Hit Self-Assessments

Note distribution of NCRs. Numbers and dates are expected to be illegible.

2010 - 2011 Timeline

8

12/14/104A FWH level

transient

12/22/104A FWH level

transient

12/25/104A FWH level transient &'A' HDP trip

1/2/114A FWH level transient &'A' HDP trip

1/4/114A FWH level transient & 'A' HDP trip

Actions taken: - Instrument functional checks - Tightened tubing on LC-1251A CAUSE NOT IDENTIFIED

Actions taken: - Functional check of 1HD-17 - Visual instrument inspections CAUSE NOT IDENTIFIED

Actions taken: - LC-1251A setpoint adjusted - Monitor LC-1251A output CAUSE NOT IDENTIFIED

Actions taken: - Replaced failed LC-1250A with controller found in TTF - Created PM to replace all electronic LTs and LCs in R17

Summary of 2010/2011 events:

- 5 transients, 3 HDP trips - Initiated by equipment failure - Cause not identified 3 times - New failure introduced resulting in 1/4/11 event

Degraded controller installed

Replaced LC-1250A

(not refuted)

Replaced LT-1250A (cause)

2012 - 2013 Timeline

9

RFO 17Initial PM due for replacement of level controllers and transmitters

11/15/124A FWH level transient &'A' HDP trip

11/16/12Unsuccessful 'A' HDP start

2/11/134B FWH level

transient

2/15/134B FWH level

transient

2/27/134B FWH level

transient

3/2/13Unsuccessful

4BFWH restoration

resulting in ES isolation and 4BFWH level

transient

Old controller LC-1250A not

replaced in R17

Replaced LC-1250A

From 2010/2011 events: PMR processed to replace electronic level controllers and transmitters in R17. LT/LC-1250A was screen out from initial scope due to replacement on 1/4/11.

Obsolete, aged level transmitters installed of unknown shelf life

Defective controller installed

Defective transmitter

installed

Replaced LC-1250B

Replaced LT-1250B

Replaced LC-1250B

(not refuted)

Replaced LT-1250B (cause)

Replace aged controllers and obsolete transmitters in R18 per WO 2217334

Revised OP-136

Revised OP-136

Replaced LS-1251B

OP-136 not revised

Unsuccessful restoration of 4BFWH to 1HD-323 control

(clearance not lifted)

OP-136 inadequacy identified

10

Initiating10 CRs

42%Resulting12 CRs

50%

Not Applicable2 CRs

8%

Event Type

11

12

Part Quality37%

Setpoint Control12%

Design13%

Preventative Maintenance

38%

Plant

13

HU Tool Usage46%

Troubleshooting15%

Risk Perception

39%

People

14

Procurement & Obsolescence

55%

Communication & Task Preview

9%

Procedure Quality

36%

Process

Root Cause

15

• Previous actions taken to address Heater Drain System equipment failures, while appropriate, have not been thorough and intrusive enough to prevent additional failures and associated reactivity management events.

• Until the most recent series of failures, insufficient attention was given to the source, quality, and age of replacement parts.

• In addition, considering the number of Heater Drain System components that can cause a reactivity management event and the goal of minimizing the number of reactivity management events, the establishment of preventative maintenance tasks has not been broad and comprehensive enough.

Contributing Causes

Untimely resolution of obsolescence issues and inadequate shelf life controls resulted in defective components being installed.

Components impacting Reactivity Management were not considered in the initial PM basis development work for the Heater Drain System. The resulting lack of PMs resulted in several initial transients and reactivity management events.

Troubleshooting of Heater Drain System events were not managed effectively. 3 of 10 initiating events resulted in a total of 12 additional events before the System was successfully returned to service.

The portion of OP-136 used to set the alternate level controller setpoints for the 4A and 4B FWHs was found to have deficiencies that contributed to the recurrence and severity of Heater Drain System transients and reactivity management events.

16

Generic Lessons Learned

Systematic troubleshooting is vital. If you don’t have enough data to determine cause, seriously consider restarting equipment to

collect more data, rather than replacing all the parts that potentially could have caused the problem.

Clearly define the roles and responsibilities of personnel on your troubleshooting team and insist management respect and follow the organization established.

Divide data collection and other field work from troubleshooting to allow the troubleshooting team to stay focused and function well as a multi-discipline team.

Don’t be so focused on producing a Support/Refute Matrix or Operational Decision Recommendation, that you short change the data collection and analysis.

17

18