SNS Reliability and Maintenance Programs George Dodson Research Accelerator Division Spallation Neutron Source

SNS Reliability and Maintenance Programs

George Dodson

Research Accelerator Division

Spallation Neutron Source

2 Managed by UT-Battellefor the U.S. Department of Energy Sustainable Neutron Production Availability at SNS

Topics

• Vision and Goals

• Enablers

• Performance Metrics

• Management Information Systems

• Continuous Improvement

• RAMI Modeling

• Maintenance Management

• Spares/Obsolescence/Vulnerability

• Configuration Control


Goals• The goals for Accelerator systems include: 4500 Hours of neutron production beam, at greater

than 90% availability at or close to the nominal power delivery capacity of the SNS.

• As the funding landscape shifts, achieving these goals will become more challenging. Increasingly greater demands are being placed on facility even as those staff are becoming leaner and in some cases less experienced due to retirements. As time passes, conditions change. Older equipment becomes obsolete and new equipment is added on a continuous basis. As a result, facilities are being operated and maintained under continually changing conditions. These changes will produce a new dynamic for our organization that adds to the facility maintenance challenges that we will face.

• Our goals can be met in this challenging environment by developing best practices associated with an Integrated Maintenance Program structure and functionality. We must develop a maintenance processes that identifies causes of potential equipment failures, effectively monitors and assesses equipment condition, and proactively plans for equipment maintenance. This organization will more effectively utilize our staff by increasing their proficiency by applying standard processes, facilitating peer collaboration, completing databases to support condition-based maintenance, and documenting case histories.

VisionThe vision of the SNS Reliability and Maintenance Programs is an efficient,

effective, reliable science facility throughout the lifetime of the SNS, currently expected to be ~40 years.


SNS Accelerator Complex

Monthly Metrics for August, 2006

945 ns

1 ms macropulse

Cur

rent

mini-pulse

H- stripped to protons

Cur

rent

1ms

Front-End: Produce a 1-msec long, chopped, H-

beam

LINAC: Accelerates the beam to

1 GeV

Accumulator Ring: Compress 1 msec

long pulse to 700 nsec

Deliver beam to Target

Chopper system makes

gaps

Ion Source2.5 MeV 1000

MeV87 MeV

CCL SRF, b=0.61SRF, b=0.81

186 MeV 387 MeV

DTLRFQ


SNS GoalsYear Neutron Production

AvailabilityIntegrated Beam Power

(MW-hrs)

Commitment Actual Commitment ActualFY2007 68.0% 65.7% 117 159FY2008 74.0% 72.0% 877 945FY2009 80.0% 80.7% 2031 2166FY2010 85.0% 85.6% FY2011 88.0% 92.0FY2012 90.0% 92.7(94.0%)FY2013 90.0% 72.4(89.4%)

Year Neutron Production Hours Total Operating Hours

Commitment Actual Commitment ActualFY2007 1500 2078 3500 3779FY2008 2700 2807 4000 4032FY2009 3500 3553 4500 4916FY2010 3900 4250 4800 5310FY2011 4300 5437 5000 5941FY2012 4500 5098 5000 5746FY2013 4000 4202 5000 5120


• The SNS Reliability and Maintenance Program is a facility-wide program for achieving the SNS primary beam delivery goals while maintaining and improving SNS Facilities in a cost-effective manner over the lifetime of the facility. The core of this program is a Reliability Centered Maintenance program. It is surrounded by a number of linked Management Information Systems (MIS), Other Systems and specific Policies and Procedures using applicable industrial standards. These systems include;– A Beam-time/Downtime Tracking System and Electronic Logbook– A Performance Metrics Reporting System– A Computerized Maintenance Management System (CMMS)– A Document Control System (DCS) linked to the CMMS– A Work Request/Planning/Scheduling System in or linked to the CMMS– A Reliability (RAMI) Modeling System– A Spares Plan linked to an Equipment Obsolescence Plan– A Vulnerability analysis of “single point” and/or “long time to recover” failures– A process for driving continued improvement in Equipment Design and Operation– A Configuration Control System to keep you from doing STUPID THINGS

Enablers


Major Components

SNS Integrated Maintenance Program

RAMI Model

Spares Plan

Reactive Maintenance

< 10%

Predictive Maintenance

45-55%

Preventative Maintenance

25-35%

Reliability Centered

Maintenance

Testing and Inspection

Equipment Obsolescence

Plan

Performance Metrics

CMMS

Goals

FMEA Equipment

Design Considerations

Equipment Operations

Considerations

Configuration Control for

Upgrades and New

Equipment /Systems

Fault Reporting

Work Planning - Scheduling

Document Control


Management Information Systems (Oracle) Acquire the Data

• Beam Time Accounting

– Operations Administration System (OAS)

– Shift by Shift account of downtime

• Electronic Logbook– Narrative account of shift activities including threaded discussion of breakdown and repair

• CMMS – DataStream 7i (Infor)

– Equipment Tracking

• Asset Structure tables with parent-child relationships

• “Cradle to Grave” tracking by position, location, asset

• Asset status (Installed, In-Repair, Spare, Disposed Of)

– Work Control

• Use the same “Data Structures” for each: System, Sub-System, Sub-Sub-System , Sub-Sub-Sub-System, Asset, Position. Location

• All 3 MIS Systems “Tied Together” through the Work Order Numbers

9 Managed by UT-Battellefor the U.S. Department of Energy SNS Reliability and Maintenance Programs

OAS Shift Closeout

Operations Metrics Report

forSeptember 23-29, 2013

(Run FY13-2)

Research Accelerator Division

Spallation Neutron Source

11 Managed by UT-Battellefor the U.S. Department of Energy Presentation_name


Operating Statistics – September 23-29, 2013


Unscheduled Downtime – September 23-29, 2013

Ion S

ource

RF

E-Mag

PS0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

4.5

5.0

5.5

6.0Breakdown Hours by System


Unscheduled downtime for the last week ≥ 0.2 hrs.

Unscheduled downtime by number of occurrences >1 (beam and non-beam downtime combined)


MPS trip summary


Hours / week - Target / Down / AP

0

12

24

36

48

60

72

84

96

108

120

132

144

156

168

Neutron production (hrs) AP Unplanned Downtime (hrs) Planned shutdown + recovery


Operating Statistics – FY13 to date


Down Time – Pareto Chart for FY 13 to date


RTBT_Diag:BCM25I:Power60Beam power on Target (60 sec. average) for the last week

1.41792 MW peak


Energy and power on target from October 2006


Beam hrs. to Target & Avg. kW/hr as of Sept. 29, 2013

0

50

100

150

200

250

300

350

400

450

500

550

600

650

700

750

800

850

900

950

1000

1050

1100

1150

1200

0

24

48

72

96

120

144

168

Avg. kW/h

Neutron Prod. Hrs.


NP availability by week4/

1/20

134/

7/20

134/

13/2

013

4/19

/201

34/

25/2

013

5/1/

2013

5/7/

2013

5/13

/201

35/

19/2

013

5/25

/201

35/

31/2

013

6/6/

2013

6/12

/201

36/

18/2

013

6/24

/201

36/

30/2

013

7/6/

2013

7/12

/201

37/

18/2

013

7/24

/201

37/

30/2

013

8/5/

2013

8/11

/201

38/

17/2

013

8/23

/201

38/

29/2

013

9/4/

2013

9/10

/201

39/

16/2

013

9/22

/201

39/

28/2

013

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%NP availability by week, FY13

Neutron Beam % Available

Commitment to DOE

FY running Avg.

Run Running Avg.

% a

vaila

ble







Machine Issues:• Ion source

– Arcing causing 13 MHz and Edmp power supply trips

• RFQ

– Chiller 2 PID tuning (0.5 C overshoot when RF is turned off and back on)

– Cryopump regen

• Verify all warm linac arc detectors are working properly

– No ion pump faults in DTL2 without RF

• DTL3 winair arcs and vacuum burst in the tank

– If venting is necessary during 2 week shutdown then replace DTL2 IP202

• CCL2 klystron window arcs (not sure there is enough time)

– Arcs have returned after waveguide polishing

• CCL2 modulator

– Still tripping (last trip was 9/30 on DFDC B flux saturated fault)

• DTL6 tank turbo pump is off


Analysis Identifies Problem Areas

Fault Reporting

Performance Metrics

Operations Administration System

(OAS) Shift Reports

Electronic Logbook (E-Log)

E-Log entries and OAS Downtime are reported. Work Orders are created in the CMMS and entered in the E-Log. Downtime linked to Work Order Number in the OAS is reported in the Metrics

Downtime and Trip Rates are evaluated in the Weekly Machine Health Report, The trend from the past week, 2 weeks ago and 3 week ago.

Weekly Metrics and Machine Health Report

List of Machine “Issues”

Operational and Design

Considerations

Failure precursors are identified by increased trip rates


Management Information Systems (Oracle) Acquire the Data

• Beam Time Accounting

– Operations Accounting System (OAS)

– Shift by Shift account of downtime

• Electronic Logbook– Narrative account of shift activities including threaded discussion of breakdown and repair

• CMMS – DataStream 7i (Infor)

– Equipment Tracking

• Asset Structure tables with parent-child relationships

• “Cradle to Grave” tracking by position, location, asset

• Asset status (Installed, In-Repair, Spare, Disposed Of)

– Work Control

• Use the same “Data Structures” for each: System, Sub-System, Sub-Sub-System , Sub-Sub-Sub-System, Asset, Position. Location

• All 3 MIS Systems “Tied Together” through the Work Order Numbers


What Equipment Must be Tracked?1. Is the equipment safety-related?2. Is the cost of the equipment $2500 or more?3. Is the equipment categorized as a Quality Level 1 or Level 2 item (Safety Related)4. Does the equipment require preventative/predictive maintenance?5. Does the equipment require periodic calibration?6. Does the equipment contain electrical components, which are categorized as “unlisted electrical equipment,” and require inspection and approval?

• Manufacturer, Model, Version and Serial Number

• When was it built

• What did it arrive

• When and where was it installed (position, location)

• When it was maintained and who maintained it

• When did it fail, what was the root cause, who repaired it

• Where is it, where has it been and when (position and location)


Devices(Position /Location)

MIS Database of Equipmentand Spares

(Assets)

Receiving Tracking ID

Number(barcode #)

Vendor Data(Traveler)

Test Data

Installation Data

Vendor Documents

Maintenance History

Fault History

EPICS Control System

Cradle-to Grave Equipment Tracking Data in the CMMS

Data are in Document

Control Systemby Tracking

Number

Example CCL_Vac:IP204


CMMSCradle-to-Grave

Equipment Tracking

Inventory Control

Work Execution

Work Planning

Equipment Status Position-Location History

Cradle-to-Grave Asset History

Work Requests/AuthorizationsWork Prioritization and

Scheduling

Resource Allocation and Scheduling

Automated Time-Based PMs

Work DocumentationPost Maintenance Testing

Equipment Swaps

Inspections/Testing Based PMs Automated Meter-Based PMs

Spares and Parts Management Warranty Information Tracking

Equipment Repair

Maintenance Costs Tracking Maintenance Hours Tracking


Data Management Analyze and Use the Data

• Build a robust data system for tracking and trending, including MTTF, MTTR, Spares Inventory, Fault Tracking, etc.

• Comparison of MTBF/MTTR data with the Reliability Model and industrial standards with an eye to the root cause of failures with higher than expected failure rates.

• Go after the highest sources of downtime

• Effectively utilize Control System Monitoring Data – filtering and pattern analysis to Detect the Onset of Pre-Failure Behavior so that you can replace the component in a Maintenance Period


Modeling:Predict the Performance Data

• Modeling sets Your Expectations for Reliability/Availability for a given design:

• Static Model– Markov Chain Model – R(t) is Constant

• MTBF/MTTR inputs from Vendor Information and Industrial Standards

• Monte Carlo Model (many commercial models available)– R(t) is an input function. You get to pick where you are on

the function.

• Use Actual Performance Data to Validate the Model


ReliaSoft BlockSim7 – Full Accelerator Complex


Front End

Ion Source


Antenna and Front End Simulation


Use the Model:

• Model subsystems, systems, eventually the whole machine

• Initially use vendor data and commercial standards for MTBF

• Play “what if’s” with redundant systems (Hot Spares)

• Be certain that what you are building meets the customer’s requirements

• As equipment breaks you can immediately assess the impact of the measured lifetime on overall availability

• Use Weibull distributions with guesses at failure onset, failure rate after onset, initial stock of spares and resupply rate to predict Mean Time to Out of Stock.

• With actual performance data, carefully monitor transitions in performance data from Infant Mortality to Reliable Operation to the onset of Terminal Mortality to refine model parameters and your spares inventory


Maintenance Management

• Predictive/Preventive maintenance schedules based on accepted practices for standard equipment and experience/MTTF data for specialized equipment– Manufacturer data is NOT always the best

– EPRI Database

• Proactive replacement of equipment showing pre-failure behavior

• Effective use of scheduled and discretionary weekly maintenance opportunities

• Avoid “run to failure” – “replace/repair when possible”

• Spares inventory, not too big, not too small, just right!

• Proactive replacement of equipment at a pre-determined % of measured lifetime – mature facilities with lots of data


Configuration Control

One of the worst things that you can do at a mature, operating facility is allow changes to the design basis that, though the Law of Unintended Consequences, causes a failure that prevents the facility from operating. – Corollary – Smart People Sometimes Do Dumb Things.


Work Control

• The SNS Work Control System is based around Safety then Complexity

• Regardless of the work being performed, the basic approach is the same: – Define the Scope of Work

– Analyze the Hazards

– Develop and implement Hazard Controls

– Perform the Work

– Perform Post Work Testing

– Provide feedback and continuous improvement

• Work is requested, approved, planned, executed, completed and closed out using the CMMS.


Work Levels

Level If Work Involves1 Changes to a Credited Engineered Control2 Work on a Credited Engineered Control but no

change to the control3 Work done in accordance with:

Approved procedures (Operations Procedure Manual, Internal Operating Procedures, Complex Work Instructions, etc.)

Drawings or specifications Permits (except Radiological Work Permits –

see section 3.4) Complex LO/TO (multiple energy source

and/or stored hazardous energy release required)

Change to a component or system that may involve an unplanned operational impact (i.e. loss of an unrelated mission-critical system)

Requiring engineering evaluation/concurrence,

Unique or unusual hazards Special waste disposal requirements Requirements for access to radiological areas

with Facilities and Operations (F&O) support/resources

4 Skill of the Worker, routine activities including simple LO/TO (single energy source with no stored hazardous energy release required).

5 Operational adjustments outside of Control Rooms, Evaluation, Inspection and Troubleshooting

Documentation Level 1

Level 2

Level 3

Level 4

Level 5

Work Request X X X X X **

Initial Job Hazard Analysis (JHA)

X X X X**** X ***

Permits (e.g. Penetration Permit, etc.) and/or complex LO/TO – as appropriate

X X X

Complex Work Information (i.e. Procedures, Instructions, Drawings, Specifications, DCN/DCD, etc.)

X X X

Equivalency Evaluation- as appropriate (see section 7.3)

X *

Unreviewed Safety Issue Determination Screen (USID Screen or Determination that a Screening is not needed)

X

Inspection/Test / Acceptance Criteria and results, as appropriate

X X X

Approved Design Criteria Document (DCD) or Design Change Notice (DCN), as appropriate

X X

Class 1 Safety Systems (Personnel Safety)



Configuration Control Policy

• Configuration management (CM) is defined as a process for establishing and maintaining consistency of a configuration item’s performance, functional and physical attributes, and its documented configuration with its requirements, design and operation information throughout its lifetime.

• Configuration management control begins with baselining of requirements, the Design Criteria Document (DCD and Design Change Notification DCN) processes, and ends with decommissioning of equipment.

• Responsibility for Configuration Control of Systems, Structures, Components and Software (SSCS) resides (at the SNS) with the System Engineer.


Configuration Control Objectives• To document and provide full evidence of an SSCS’s previous history (when available) and

present configuration including the status of compliance of an item to its physical and functional requirements.

• To ensure that staff who operate, use, repair or maintain an SSCS or who have the potential to affect its configuration use correct, accurate, and current documentation.

• To ensure that new designs and changes to existing designs for systems, structures, components and software utilize best engineering practice, follow from an approved set of specifications, and are appropriately documented.

• To ensure that the deployment of a new SSCS or a change to an existing SSCS is authorized.

• To ensure that the impact on performance due to the deployment of a new SSCS or a change to an existing SSCS is fully understood, and that the risks associated with the deployment are considered.

• SNS Procedures• OPM 9.A-1 SNS Configuration Management Policy

• OPM 9.A-2 Design Development Policy

• OPM 9.A-3 SNS System, Structure, Component or Software Change Procedure


Spares – Cold SparesCritical Equipment is equipment which is essential to the facility mission, which is traditionally defines as greater than the nominal beam delivery at greater than ~90% availability for some number of operating hours per year. Spares must be identified for critical equipment.

Classes of Spares1. A “true spare” consisting of a “like for like or equivalent” “on the shelf, tested and ready to go “,

“plug compatible” replacement unit.

2. A “like for like or equivalent” that is installed in some other system that is not required for operation of the accelerator systems e.g. a Test Stand that must be removed from where it is being used so that it can be used as a replacement for the failed unit.

3. A system structure or component that must be modified to be used as a spare.

4. A system structure or component that must be purchased to be used as a spare.

Only a level 1“true spare” will not contribute to down time. In all other classes, demounting, modification or procurement of the replacement will necessarily contribute to downtime. Class 4 is referred to as an “out of stock” condition

The number of spares should be based on a calculation but should never be 1 ( or you are guaranteed to break it while installing it).

SNS OPM 9B.-1 RAD Spares Management Policy (DRAFT)


Obsolescence• You probably don’t want to think about this now but the MTTO is on the order of 3 years

for some classes of electronics.

• With the manufacturing world changing rapidly, companies go out of business or are bought up and their product lines discontinued at an alarming rate. When they do your new replacements and product support may go to zero.

• Obsolescence Definitions:– Supported:

• Identical New Items/Repair/Parts are available from the OEM

– Obsolescent: • New/Repair/Parts will no longer be supplied by the OEM after a given date. Sometimes you are even notified in

advance!

– Obsolete:• New Items/Repair/Parts are no longer available from the OEM

• Obsolescence issues should be considered in the item life cycle to avoid risk. This means:

– Assess the impact, cost and probability of obsolescence

– Derive a Strategy

• Reactive – do nothing until the need arises - Emulate/Partial Redesign/Replace

• Proactive – Adopt a proactive strategy – Partial Redesign/Technology Transparency/Contract Support/Lifetime Buy

– Periodically review and monitor the situation and act accordingly.


Types of MaintenanceReactive Maintenance

Reactive maintenance is basically the “run it till it breaks” maintenance mode. No actions or efforts are taken to maintain the equipment as the designer originally intended to ensure design life is reached.

– Advantages

• Low initial cost.

• Less staff.

– Disadvantages

• Increased unplanned downtime of equipment.

• Increased labor cost due to overtime needed for call-in repairs

• Possible secondary equipment or process damage from equipment failure.

• Inefficient use of staff resources.


Preventive Maintenance Preventive maintenance can be defined as follows: Actions performed on a

time- or machine-run-based schedule that detect, preclude, or mitigate degradation of a component or system with the aim of sustaining or extending its useful life through controlling degradation to an acceptable level.

• Advantages

• Cost effective in many capital-intensive processes.

• Flexibility allows for the adjustment of maintenance periodicity.

• Increased component life cycle.

• Energy savings.

• Reduced equipment or process failure.

• Estimated 12% to 18% cost savings over reactive maintenance program.

• Disadvantages

• Catastrophic failures still likely to occur.

• Labor intensive.

• Includes performance of unneeded maintenance.

• Potential for incidental damage to components in conducting unneeded maintenance.



Predictive maintenance can be defined as follows: Measurements that detect the onset of system degradation (lower functional state), thereby allowing causal stressors to be eliminated or controlled prior to any significant deterioration in the component physical state. Results indicate current and future functional capability.

Advantages

Increased component operational life/availability.

Allows for preemptive corrective actions.

Decrease in equipment or process downtime.

Decrease in costs for parts and labor.

Improved worker and environmental safety.

Improved worker morale.

Estimated 8% to 12% cost savings over preventive maintenance program.

Disadvantages

Increased investment in diagnostic equipment.

Increased investment in staff training.

Savings potential not readily seen by management.


Reliability Centered Maintenance

• Reliability centered maintenance (RCM),RCM is a systematic approach to evaluate a facility’s equipment and resources to best mate the two and result in a high degree of facility reliability and cost-effectiveness

• The RCM methodology recognizes that 1. all equipment in a facility is not of equal importance to either the

process or facility safety.

2. equipment design and operation differs and that some will have a higher probability to undergo failures from different degradation mechanisms than others.

It also approaches the structuring of a maintenance program recognizing that a facility does not have unlimited financial and personnel resources and that the use of both need to be prioritized and optimized


• Advantages

• Can be the most efficient maintenance program.

• Lower costs by eliminating unnecessary maintenance or overhauls.

• Minimize frequency of overhauls.

• Reduced probability of sudden equipment failures.

• Able to focus maintenance activities on critical components.

• Increased component reliability.

• Incorporates root cause analysis.

• Disadvantages

• Can have significant startup cost, training, equipment, etc.

• Savings potential not readily seen by management


Reactive

Maintenance< 10%


45-55%

Preventative Maintenance

25-35%

Reliability Centered

Maintenance

Testing and Inspection

FMEA

Industrial Standards for Reliability Centered Maintenance (RCM) References:•DOE EERE O&M Best Practices Guide Rev. 3•NASA RCM Guide 2008

Accelerator Systems have more Reactive Maintenance due to the high percentage of digital electronic systems which fail with no precursor events.


Since 2006 operational performance improvement at SNS has been dramatic

FY07 FY08 FY09 FY10 FY11 FY12 FY130

500

1000

1500

2000

2500

3000

3500

4000

4500

5000

5500

60

65

70

75

80

85

90

95

NP Hrs. delivered

MWh delivered to target

NP Downtime

NP Availability

Without Target Failures


FY07-FY13 Downtime by group

Targ

etE

-HV

CM RF

Ion

Sou

rce

E-M

agP

SE

-cho

pper

Con

trol

sV

acuu

m

E-o

ther

Coo

ling

Cry

o

AP

Pro

t. S

ys.

Mis

c./M

ag/R

S/E

SH BI

Fac.

/Mec

h. S

ys.

Ops

CM

/SR

FN

eut.

Inst

.

0

50

100

150

200

250

300

350

400

450

FY07

FY08

FY09

FY10

FY11

FY12

FY13

System

Ho

urs

of

do

wn

tim

e



In the end, you have to satisfy your customer.

• SNS Management in FY10 decided to emphasize availability improvement while holding proton beam power at or near 1MW

– Resources were allocated to address major contributors to down time, particularly the HVCM

• Replacement of some highly stressed oil filled capacitors with less lossy solid units that led to fewer and lower consequence capacitor failures and easier fault recovery.

• IGBT drive gate synchronization turn off that reduced IGBT failures by more than a factor of 10.

– The single largest downtime contributor to RF systems, the MEBT RF Power Amplifiers, were replaced with new solid state devices.

– The 2MHz RF amplifier that drives the ion source plasma was removed from the 65KV floating deck to ground potential and is now powered through an isolation transformer, an improvement that allows for better diagnosis of failures and quicker repair.

• SNS Management, following the 2 unexpected target failures at the and of FY12 decided to emphasize target availability by holding proton beam power at or near 850KLW which was considered to be a safe power level for extended running.

– Extensive analysis of the targets was done

– Orders were placed for new “jet flow” targets and the original style targets, but ALL will have removable water shrouds to allow for inspection of the failure location and mode.


Summary• The SNS has evolving Reliability and Integrated Maintenance

Management Programs

• We are making progress

• We are no longer a “young” facility and that we may soon reach Terminal Mortality for many systems.

• The final goal is 95% availability.– A Plan has been developed. – It may be too costly to be implemented. Why? – Going from 90% to 95% is only another 5.5% in beam

delivery, but it is a factor of 2 in downtime reduction. Diminishing returns! The facility Science impact will likely be larger from another beamline instrument (Spectrometer).

– We will likely make more modest evolutionary (not revolutionary) changes to our operating base.


Backup Slides


Power plots from Sep. 29