SNS Reliability Program

SNS Reliability Program

George Dodson

Research Accelerator Division

Spallation Neutron Source

2 Managed by UT-Battellefor the U.S. Department of Energy Sustainable Neutron Production Availability at SNS

Topics

• Goals

• RAMI Modeling

• Management Information Systems

• Metrics

• Maintenance Management

• Configuration Control


SNS Accelerator Complex

Monthly Metrics for August, 2006

945 ns

1 ms macropulse

Cur

rent mini-pulse

H- stripped to protons

Cur

rent

1ms

Front-End: Produce a 1-msec long, chopped, H-

beam

LINAC: Accelerates the beam to

1 GeV

Accumulator Ring: Compress 1 msec

long pulse to 700 nsec

Deliver beam to Target

Chopper system makes

gaps

Ion Source2.5 MeV 1000

MeV87 MeV

CCL SRF, b=0.61SRF, b=0.81

186 MeV 387 MeV

DTLRFQ


SNS GoalsYear Neutron Production

AvailabilityIntegrated Beam Power

(MW-hrs) Commitment Actual Commitment Actual

FY2007 68.0% 65.7% 117 159FY2008 74.0% 72.0% 877 945FY2009 80.0% 80.7% 2031 2166FY2010 85.0% 85.6% N/AFY2011 88.0% 91%YTD N/AFY2012 90.0% TBD N/A

Year Neutron Production Hours Total Operating Hours

Commitment Actual Commitment ActualFY2007 1500 2078 3500 3779FY2008 2700 2807 4000 4032FY2009 3500 3553 4500 4916FY2010 3900 4250 4800 5310FY2011 4300 TBD 5000 TBDFY2012 4500 TBD 5000 TBD


ModelingPredict the Performance Data

• Modeling sets Your Expectations for Reliability/Availability:

• Static Model– Markov Chain Model – R(t) is Constant

• MTBF/MTTR inputs from Vendor Information and Industrial Standards

• Monte Carlo Model– R(t) is an input function. You get to pick where you are on

the function.

• Use Actual Performance Data to Validate the Model


Management Information Systems (Oracle) Acquire the Data

• Beam Time Accounting – Operations Accounting System (OAS)– Shift by Shift account of downtime

• Electronic Logbook– Narrative account of shift activities including threaded discussion of breakdown and repair

• CMMS – DataStream 7i (Infor)– Equipment Tracking

• Asset Structure tables with parent-child relationships• “Cradle to Grave” tracking by position, location, asset• Asset status (Installed, In-Repair, Spare, Disposed Of)

– Work Control

• Use the same “Data Structures” for each: System, Sub-System, Sub-Sub-System , Sub-Sub-Sub-System, Asset, Position. Location

• All 3 MIS Systems “Tied Together” through the Work Order Numbers


Data Management Analyze and Use the Data

• Robust data system for tracking and trending, including MTTF, MTTR, Spares Inventory, Fault Tracking, etc.

• Comparison of MTBF/MTTR data with the Reliability Model and industrial standards with an eye to the root cause of failures with higher than expected failure rates.

• Go after the highest sources of downtime • Effectively utilize Control System Monitoring Data –

filtering and pattern analysis to Detect the Onset of Pre-Failure Behavior so that you can replace the component in a Maintenance Period


Go after the Highest Sources of Downtime

RF

Ion Source

E-HVCM

Vacuum

Prot. Sys

.

Controls

E-Mag

PS

E-other

Mech. S

ys.

Cooling

E-choppers Ops AP

Target

Cryo0

4

8

12

16

20

24

28

32

36

40 FY11 YTD Breakdown Hours by Sys-tem

First Tier

Second Tier Third Tier


Operations Report for

December 20-26, 2010(Run FY11-1)

Research Accelerator Division

Spallation Neutron Source


Presentation_name

Operating Statistics – December 20-26, 2010Scheduled Activities (Published Schedule) Hours % ActivityShutdown 96.0 57.1Maintenance 0.0 0.0Beam Startup 0.0 0.0Neutron Production 8.0 4.8Accelerator Physics 64.0 38.1----------------------------------------------------------------- ----------Operational ScheduleShutdown 102.0Maintenance 0.0Beam Startup 0.0Neutron Production (Published Schedule) 2.0Neutron Production (Additional Option Time) 0.0Neutron Production Total 2.0Accelerator Physics (Published Schedule) 64.0Accelerator Physics (Additional Option Time) 0.0Accelerator Physics Total 64.0

Recorded Activity Hours % Available Shutdown 108.0 ----------Maintenance 0.0Beam Startup (SU) 0.0 ----------Neutron Production (NP) 2.0 100.0Accelerator Physics (AP) 55.9 87.3Downtime 2.1 ---------------------------------------------------------------------------Total 168.0 ----------Total Scheduled Beam Time (AP+NP) 66.0 ----------Total Scheduled Operating Time (AP+NP+SU) 66.0 ----------Total Beam Time Delivered (AP+NP) 57.9 87.7Total Operating Time Delivered (AP+NP+SU) 57.9 87.7Total Downtime 6.2 9.4Neutron Beam Efficiency 100.0%


Unscheduled Downtime, December 20-26, 2010

RF

Prot. S

ys.

Controls

E-Mag

PS0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

1.1

1.2Breakdown Hours by System


Unscheduled downtime ≥ 0.1 hr. for the last weekCategory SubCat. Hrs. Notes - blue indicates non-beam downtime Date(Shift)RF HPRF 0.7 SCL XMTR 15 trips on cath 5 OI x2 2010-12-20 (N)

Prot. Sys. PPS 0.6 Front End Trip Due to PPS testing 2010-12-21 (D)

RF HPRF 0.4 SCL 17d chatter f lts on Cav. Fld & HOM_A x4 2010-12-21 (D)

Controls MagPS 0.2 CCL Mag PS Q312t407 Channel Details Comm Loss 2010-12-20 (N)

Electrical MagPS 0.1 CCL Mag PS QH00 tripped 2010-12-21 (N)

RF HPRF 0.1 SCL 15a trip on Cav. f ld & Cav. fw d 2010-12-21 (D)

Category SubCat. Hrs. Notes #RF HPRF 0.4 SCL 17d chatter f lts on Cav. Fld & HOM_A x4 4RF HPRF 0.7 SCL XMTR 15 trips on cath 5 OI x2 2

Unscheduled downtime by number of occurrences >1 (beam and non-beam downtime combined)


Operating Statistics – FY11 to dateScheduled Activities (Published Schedule) Hours % ActivityShutdown 96.0 1.1Maintenance 88.0 1.0Beam Startup 176.0 2.0Neutron Production 1576.0 18.0Accelerator Physics 152.0 1.7----------------------------------------------------------------- ----------Operational ScheduleShutdown 102.0Maintenance 48.8Beam Startup 25.9Neutron Production (Published Schedule) 1570.0Neutron Production (Additional Option Time) 168.6Neutron Production Total 1738.6Accelerator Physics (Published Schedule) 152.0Accelerator Physics (Additional Option Time) 20.7Accelerator Physics Total 172.7

Recorded Activity Hours % Available Shutdown 108.0 ----------Maintenance 48.8Beam Startup (SU) 25.9 ----------Neutron Production (NP) 1600.4 92.1Accelerator Physics (AP) 162.3 94.0Downtime 142.6 ---------------------------------------------------------------------------Total 2088.0 ----------Total Scheduled Beam Time (AP+NP) 1911.3 ----------Total Scheduled Operating Time (AP+NP+SU) 1937.2 ----------Total Beam Time Delivered (AP+NP) 1762.7 92.2Total Operating Time Delivered (AP+NP+SU) 1788.6 92.3Total Downtime 142.6 7.5Neutron Beam Efficiency 92.1%


Unscheduled Downtime, FY11 to date

RF

Ion Source

E-HVCM

Vacuum

Prot. Sys

.

Controls

E-Mag

PS

E-other

Mech. S

ys.

Cooling

E-choppers Ops AP

Targ

etCryo

0

4

8

12

16

20

24

28

32

36

40FY11 YTD Breakdown Hours by System

Group Hours % of breakdown totalRF 38.5 27.0Ion Source 36.8 25.8E-HVCM 24.6 17.3Vacuum 18.4 12.9Prot. Sys. 6.5 4.6Controls 4.7 3.3E-MagPS 4.4 3.1E-other 2.9 2.0Mech. Sys. 1.7 1.2Cooling 1.7 1.2E-choppers 1.6 1.1Ops 0.3 0.2AP 0.3 0.2Target 0.1 0.1Cryo 0.1 0.1


Hours / week - Target / Down / AP

7/11/2

010

7/25/2

010

8/8/20

10

8/22/2

010

9/5/20

10

9/19/2

010

10/3/

2010

10/17

/2010

10/31

/2010

11/14

/2010

11/28

/2010

12/12

/2010

12/26

/2010

01224364860728496

108120132144156168

Neutron production (hrs) AP Unplanned Downtime (hrs)


Down Time – Pareto Chart for FY10-3 & 11-1

Targe

t

E-Mag

PS

Ion S

ource RF

E-HVCM

E-othe

r

Vacuu

m

Prot.

Sys.

E-Cho

pper

s

Contro

ls

Coolin

g

Physic

s

Mech.

OpsCryo

0

10

20

30

40

50

60

70

80

90

100 FY10-3 & 11-1 Pareto26-Dec19-Dec12-Dec5-Dec28-Nov21-Nov14-Nov7-Nov31-Oct24-Oct17-Oct10-OctFY10-3

Hou

rs d

own

System FY10-3 10-Oct 17-Oct 24-Oct 31-Oct 7-Nov 14-Nov 21-Nov 28-Nov 5-Dec 12-Dec 19-Dec 26-DecTarget 98.7 0.1 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0E-MagPS 51.7 0.8 0.0 0.3 1.1 0.3 0.1 0.0 0.0 0.9 0.3 0.5 0.1Ion Source 14.4 4.8 0.5 1.6 1.5 0.3 0.4 12.2 5.7 9.8 0.0 0.0 0.0RF 10.7 0.8 13.8 0.5 1.6 1.7 5.8 2.3 0.5 2.4 2.5 4.0 1.2E-HVCM 15.3 0.7 0.0 17.1 0.7 1.7 0.9 0.3 0.9 0.3 0.9 0.8 0.0E-other 22.3 0.2 0.0 0.0 1.8 0.0 0.0 0.0 0.6 0.0 0.0 0.3 0.0Vacuum 0.6 0.1 0.0 0.6 0.5 1.8 15.3 0.1 0.0 0.0 0.0 0.0 0.0Prot. Sys. 9.3 0.0 0.0 0.0 0.0 3.1 0.0 0.0 0.0 0.0 0.0 0.0 0.6E-Choppers 5.2 0.0 0.0 0.0 0.0 0.0 0.0 1.6 0.0 0.0 0.0 0.0 0.0Controls 0.9 0.0 0.0 0.0 0.9 3.3 0.0 0.0 0.0 0.0 0.0 0.0 0.2Cooling 1.7 0.0 0.0 0.5 0.0 0.3 0.4 0.0 0.0 0.0 0.0 0.5 0.0Physics 1.7 0.2 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.1 0.0 0.0 0.0Mech. 0.1 1.6 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.1 0.0Ops 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.3 0.0 0.0 0.0 0.0 0.0Cryo 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.1 0.0 0.0 0.0


NP availability by week7/

5/20

10

7/19

/201

0

8/2/

2010

8/16

/201

0

8/30

/201

0

9/13

/201

0

9/27

/201

010

/11/

2010

10/2

5/20

10

11/8

/201

011

/22/

2010

12/6

/201

0

40%

45%

50%

55%

60%

65%

70%

75%

80%

85%

90%

95%

100% NP availability by week, FY10-2&3 & 11-1

Neutron Beam % AvailableRun Running Avg.Commitment FY running Avg.

% a

vaila

ble


Maintenance Management• Predictive/Preventive maintenance schedules based on accepted

practices for standard equipment and experience/MTTF data for specialized equipment– Manufacturer data is NOT always the best– EPRI Database

• Proactive replacement of equipment showing pre-failure behavior

• Effective use of scheduled and discretionary weekly maintenance opportunities

• Avoid “run to failure” – “replace/repair when possible”

• Spares inventory

• Proactive replacement of equipment at a pre-determined % of measured lifetime – mature facilities with lots of data


Configuration Control

One of the worst things that you can do at a mature, operating facility is allow changes to the design basis that, though the Law of Unintended Consequences, causes a failure that prevents the facility from operating. – Corollary – Smart People Sometimes Do Dumb Things.


Configuration Control Policy• Configuration management (CM) is defined as a process for

establishing and maintaining consistency of a configuration item’s performance, functional and physical attributes, and its documented configuration with its requirements, design and operation information throughout its lifetime.

• Configuration management control begins with baselining of requirements, the Design Criteria Document (DCD and DCN) process, and ends with decommissioning of equipment in the operational SNS.

• Responsibility for Configuration Control of Systems, Structures, Components and Software (SSCS) resides with the System Engineer.


Configuration Control Objectives

• To document and provide full evidence of an SSCS’s previous history (when available) and present configuration including the status of compliance of an item to its physical and functional requirements.

• To ensure that staff who operate, use, repair or maintain an SSCS or who have the potential to affect its configuration use correct, accurate, and current documentation.

• To ensure that new designs and changes to existing designs for systems, structures, components and software utilize best engineering practice, follow from an approved set of specifications, and are appropriately documented.

• To ensure that the deployment of a new SSCS or a change to an existing SSCS is authorized.

• To ensure that the impact on performance due to the deployment of a new SSCS or a change to an existing SSCS is fully understood, and that the risks associated with the deployment are considered.


Configuration Management PolicySNS-OPM 9.A-1 SNS Configuration Management Policy

Configuration IdentificationConfiguration Change Management Configuration Status AccountingConfiguration Verification and Audit Commercial Off-The-Shelf, Non-Developmental Items, and

Commercially Available Software Procedures for Design Development and Design Change SNS-OPM 9.A-2 SNS Design Development Procedure

SNS-OPM 9.A-3 SNS Design Change Procedure


Since 2006 operational performance improvement at SNS has been dramatic


Initial and ongoing operation revealed system weakness that have been substantially addressed

FY07-FY10 Downtime by group

0

50

100

150

200

250

300

350

400

450

System

Hour

s of

dow

ntim

e

FY07 FY08 FY09 FY10

System FY07 FY08 FY09 FY10E-HVCM 379.4 421.2 309 240.0Ion Source 394.7 142.2 101.8 113.5RF 162.3 230.2 226.4 126.3Target 140.1 158.9 12 140.5E-MagPS 78.6 162.2 68.1 88.5E-chopper 241.8 50.3 58.3 30.4Controls 161.3 58 90.4 24.1Vacuum 90.2 124 33.7 11.4Cooling 165.2 31.2 27.5 24.3E-other 85.9 45.6 40 36.1Cryo 15.2 4.7 38.9 21.1AP 19 27.3 20.2 7Prot. Sys. 19 9.4 26.4 13.6Fac./Mech. Sys. 5.7 3.1 31.7 2.9BI 15.2 8.4 16.2 0.6Ops 8.8 13.4 6.6 5.1Misc./Mag/RS/ESH 7.1 2 3.8 1.4Neut. Inst. 0.2 2 0 0Total 1989.7 1494.1 1111 886.8


1 1 2 2 3 3 4 4 5 5 6 6 7 7 8 8 9 9 10 10 11 11 12 120

10

20

30

40

50

60

70

Normalized Downtime per shift

FY07

FY08

FY09

FY10

FY11

Perc

ent


SNS Management in FY10 decided to emphasize availability improvement while holding proton beam power at or near 1MW• Resources were allocated to address major contributors to down time,

particularly the HVCM– Replacement of some highly stressed oil filled capacitors with less lossy solid

units that led to fewer and lower consequence capacitor failures and easier fault recovery.

– IGBT drive gate synchronization turn off that reduced IGBT failures by more than a factor of 10.

• The single largest downtime contributor to RF systems, the MEBT RF Power Amplifiers, were replaced with new solid state devices.

• The 2MHz RF amplifier that drives the ion source plasma was removed from the 65KV floating deck to ground potential and is now powered through an isolation transformer, an improvement that allows for better diagnosis of failures and quicker repair.


Recent targeted actions based on detailed system engineering analysis have substantially reduced HVCM failure contributions to down time

Performance improvement from system modifications installed during 2010 Winter and Summer outages –For the FY 2011-1 run period HVCM

downtime accounted for only 1.36% of the scheduled beam time compared to

5.7% for all of FY 2010


Analysis of FY10 and FY11 availability data indicate that a new approach to performance improvement is needed• Events of duration 1

minute to 1 hour contribute to about 4% down time

• Events of duration >3 hours drive the overall structure and contribute the majority of recent cumulative down time

These longer events have the greatest impact on experiment outcomes


Summary• The SNS has an evolving Reliability Program• We are making good progress• We realize that we are “young” and that we have not

reached Terminal Mortality for many systems. • The final goal is 95% availability.

– A Plan has been developed. – It may be too costly to be implemented. Why? – Going from 90% to 95% is only another 5.5% in beam

delivery, but it is a factor of 2 in downtime reduction. Diminishing returns! The facility Science impact will likely be larger from another beamline instrument (Spectrometer).

– We will likely make more modest evolutionary (not revolutionary) changes to our operating base.


Backup Slides


Analysis

• Of operating trends to act on leading indicators of failure in a predictable way

• Of equipment failures to understand root causes (design, installation, fabrication, etc.) and to fold into computation of spare inventories or anticipate obsolescence issues

• Of vulnerabilities (single-point failure modes)

• Of day-to-day internal, less frequent formal internal and external assessments and off-normal events.

• Of fault correction data to determine needed changes to the response model (on-call, recovery algorithms, etc.)

• 00.00 hours - Diagnosis

• 00.00 hours - Transit

• 00.00 hours - Repair Time

• 00.00 hours - Recovery Time

•

Documents

SNS Reliability Program