An Overview of the Reliability and the Fault Trends of the SRS

WAO, Trieste, 24th - 28th September 2007

An Overview of the Reliability and the Fault Trends of the

SRS

Cheryl HodgkinsonDaresbury Laboratory

Synchrotron Radiation Department


Contents• Introduction• SRS Operational Statistics• Analysis of the Operational Data

– Efficiency– Fault Statistics– Ageing and Obsolete Equipment

• Maintaining Reliability– Capital Investment and Redundancy– Risk Analysis– Post Incident Investigation and Modifications– Preventative Maintenance and Scheduling

• Conclusions and Acknowledgments


Introduction

• The Synchrotron Radiation Source– 2GeV, 2nd Generation Light Source– Constructed between 1975 and 1980– Began scheduled user operations in 1981– Some SRS systems, mainly injector

components were salvaged from a previous accelerator

– Operational user facility for 26 years and scheduled for closure December 2008


1984 – 1992•A number of upgrades

•HBL•Undulator•SC Wavelength Shifters

•Erratic efficiency profile due to new systems

2001 -2007•Dominated by water – vacuum incidents

•01/02 (65%) SCW 9 Absorber•04/05 (85%) RF Cavity Window & Quadrupole Vessel Absorber•06/07 (78%) TVM 9

•Efficiency in 06/07 was also significantly effected by Linac RF and HT systems and Gun Cathode issues.•Periods between incidents in excess of 90%.

1993 - 2001Stable period of operationGradual increase in efficiency to figures in excess of 90%

SRS Reliability Data


SRS Fault Statistics

Vacuum Incidents


Vacuum Incidents• Usually an isolated single failure• Difficult to predict and prevent

– failed RGA head– Quadrupole absorber – failed flowmeter

• Common Theme?– SCW Absorber & TVM 9– Both very different failure mechanisms– Both historical – not needed in the

current SRS configuration


SRS Fault Statistics

Beam Loss Cause UnknownRF Klystron PSURF SystemControls


Ageing and Obsolete Equipment

• Early Equipment– Built in-house– Repairs in-situ (time)– Can be repaired

• Later Equipment– Plug-in– Replaced with a spare– Return to manufacturer for repair– Support issues



• Concern regarding condition of in vacuum water circuits– 17 year old

absorber removed from storage ring

– Area’s milled away to examine any wall thinning

RESULT

No evidence of wall thinning or erosion



• External water circuits– Elbows– Near feedthroughs– Potential for

vacuum incident during repair

– Potential to cause more serious failure


Ageing Machine – It’s not just equipment!

• It’s about people too!– Loss of design and build knowledge, as staff retire

• Less documentation for older equipment, drawings, maintenance schedules, etc.

• Can’t teach 20 years design, build and operational knowledge.– Succession planning important– You only know what’s been missed when the fault occurs

• Lapses in maintenance schedules• Longer fault rectification


Capital Investment and Redundancy

• Capital investment– Must be bid for against

other priority items– Only essential systems

and components which would significantly improve efficiency

• Spares– Hot

• Power Converters• Storage Ring Klystron

– Vacuum vessels

Old RF PSU

New Commercial Unit


Risk Analysis

• Benefit difficult to assess– The biggest risk is the one you don’t know

about

• At top level– Difficult to compare risk and focus resource– How do you compare extremely reliable long

lead time items against components that fail frequently, but are a fast repair?

– Does highlight major issues• Linac Klystron


Risk Analysis• Most beneficial at individual technical group level• Direct comparison can be made between

equipment and fault hours contributed• Led to successful modernisation in key area’s

based on safety and operational risk

Risk Exposure Location Equipment Likelihood of Failure

Impact

09/06 10/05 09/04 01/03

Resources man

weeks

Cost Comments

Temperature Interlock Panel 1 1 1 1 1

4

2

10

10

6

2

3

4

2

6

1

2

2

0 £0K Replaced Aug 04 and drawings updated

Septum Extraction 1 1 1 1 2 6 0 £0K Replaced Sept 04, drawing updated. Proven reliability.

Septum Injection 1 2 2 4 4 6 2 £0K Spares available, Drawing need updating

Kickers Extraction 1 1 1 2 6 6 0 £0K Replaced Oct 05, drawing updated. Proven reliability.

Booster Accelerator

Kickers Injection 1 2 2 4 4 6 2 £0K Spares available, Drawing need updating


Post Incident Investigation

• Learning from incidents is important• Committee is always formed to recover

and also determine actions to prevent reoccurrence.

• Actions from recent faults– Modification to TVM cooling pipes– Revised interlock testing schedule– Test rig to test thermionic valves on-site


Preventative Maintenance and

Scheduling


Conclusions

• The SRS has provided reliable user operations for over 26 years

• Vacuum incidents are isolated incidents, which do not indicate any form of systematic failure which would limit the life of the SRS

• Only evidence of any systematic failure due to ageing is the failure of copper elbows on water circuits

• The high operating efficiency demanded by the light source community can be maintained on an ageing machine using a variety of techniques.


Acknowledgments

• The author would like to thank– Previous SRS Operations Managers for

compiling these statistics– Current and past Operations Teams for

providing the data– Steve Griffiths of EE&PS for providing his

Groups Risk Analysis DataAnd Finally...........

Daresbury Staff past and present for their contributions to SRS Operations

Documents

An Overview of the Reliability and the Fault Trends of the SRS