prepared by R.A.Barlow (May 2014)

prepared by R.A.Barlow (May 2014)

TE-ABT-EC Maintenance methods and policyEdms nb.

TE-ABT-EC

Maintenance methods and policy in EC section

2

Outline EC equipment location Equipment failures EC environment

Hardware Software Maintenance and procedures Repairs Documentation Storage of components Quality control

CONCLUSION

TE-ABT-EC Maintenance methods and policy

3

EC equipment location (expert/piquet equipment)


LHCMKI2-Beam1MKI8-Beam2LBDS-MKD-Beam1LBDS-MKB-Beam1LBDS-MKD-Beam2LBDS-MKB-Beam2MKQATCDQ-Beam1TCDQ-Beam2TCDS

PSPI.KFA45PI.KFA28PE.BFA9-21PE.KFA71-79PE.KFA4-9-13-21PE.SMH16PE.SEH23PI.SMH26PE.SEH31PI.SMH42PE.SMH57PE.SMH61PS.SMH61

PSBBI.DISBE.BSWBI.KSWBE.KFABT.KFA10,20BI.SMVBI.SMHBE.SMHBT.SMV10BT.SMV20

LEIRER.KFH31-32-34BI.DFH11-12-21-42SMH11,40ER.SEH10

ADAD.HORNADE/KFE50ADI/KFI55-56NIKHEF 5305-5307

SPSMKPMKDMKQMKE4MKE6ZS2TPSNMST & MSE 2TPSG4MSE4TPSG6MST & MSE 6ZS6

ISOLDEHRS/GPSLINAC4Pre-ChopperCTFCR.KHB550CC.KHC0250CD.SHA110/CD.SHB0120CD.SHB0480/CD.SHA0490TT2FT16.DFA242-254

Equipment Responsible and Deputy

4TE-ABT-EC Maintenance methods and policy

HV genera

tor

Actuato

rs an

d sensors

Power su

pply

Power ele

ctronics

Magnet

HV genera

tor elem

ents

Cablin

g

Controls

Hydrau

licsTim

ing

Vacuum

0

10

20

30

40

50

60

70

80

Components Kickers + Septa, all facilities

Failu

res b

etw

een

( 200

9-20

13)

HV GENERATOR IGBT’s/GTO’s/Ignitrons, power triggersThyratrons + Heaters

CONTROLSPLC’s, Fast Controls, Electronic cards, Software, INTL

ACTUATORS and SENSORS:Electrovalves, Sensors, Gauges, Temp measurements, etc ...

VACUUMFailure of vac gauges, Vac levels

HYDRAULICSWater cooling systems, oil systems

POWER SUPPLYLarge power supplies, Small and Medium type,


BB

ABT EquipmentPS Complex, SPS & LHC

Kickers & Septa

TE/ABT/ECElectronics& Controls

BE/CO & EN/ICEControl Infrastructure

BE/OPOperation

TE/ABTSpecialist

Electronic Modules Power

electronicsHV GeneratorElements

ControlsPLC’s&Pxi systems Power

distribution

Software

Actuators&Sensors Power

suppliesControlchassis

Cabling & associated hardware

6

The EC environment

TE-ABT-ECMaintenance,

methods & policy

Hardwaremodules

Software

Maintenance & procedures

Storage

Documentation

Quality control


7

Electronic design data Electronic design is done in-house with the schematics PCB routing with exterior companies or TE-MPE-EM Manufacturing produced outside or by TE-MPE-EM + assembly AED/EDA numbering system for card versioning If done by TE-MPE-EM then all documentation goes into EDMS

under the EDA numbering scheme If done by EC section then documentation is stored in EDMS and

numbered with AED marking Sometimes these are mirrored ! (WE wish to keep OUR ‘AED’

numbering system) Each working manufacturing portfolio contains: Schematic, layout, gerber files, mechanical drawings, drillings file and serigraphy, BOM’s, test procedures, photo’s...


8

Equipment coding of modules


Electronic cards labeling with bar code system MTF (Manufacturing and Test Folder) + AED/EDA number + Tested label

The complete MTF system (judged not best suited for our equipment), with LIMITATIONS but...on-going

9

Programmable Logic Controllers EC has a vast array of industrial automation in its

controls park PLC distributed systems are based around H/W design

with associated software Some chassis design requires custom made electronics

coupled to PLC’s H/W Each PLC chassis is created with technical drawing,

cabling scheme, connector scheme (See_electrical) and software code (Step7)

In recent years industrial PXi systems (N.I) are becoming an additional resource to our supervisory control systems


10

Power elements/HV elements Many semi-conductor switches type (IGBT,

GTO, Behlke) Trigger systems in the medium power range HV discrete components HV generators (EC/FPS/SE) Electro Mechanical elements


11

Power distribution systems EC responsible for the design of system power

distribution These are documented systems and undergo

approval from HSE unit with outside companies such a DEKRA who come along for conformity testing and place approval seals on the systems. This includes Electrical circuit and Electrical protection checks.

We are currently looking at how to produce these elements on the outside using rigorous procedures and guidelines compatible with expected industry standards.


12

Software maintainability PLC codes (Ladder, SCL, STL, Graph7) + WinCC

(scada)Stored onto DFS volume, currently looking at other systems for software maintainability (Autosave product) FPGA ‘s codes (mainly VHDL, sim files, chipscope,

etc....)/SVN C++, Java, other/SVN Hardware drivers/SVN PXI systems software/SVN Fesa equipment classes/SVN


13

Preventive maintenance Annual recalibration Annual rotation of DCPS Capacitor exchange (ex: Heater crates) System consolidation Timing system Firmware upgrades Interlock checks Software release Piquet weekly inspections of equipment where possible (ex: oil leaks, HV cable

degradation, flasher not functioning, etc...)And many more!Preventive maintenance is difficult in areas with difficult access, no clear evaluation of possible cyclic maintenance, lack of data and analysis for overall systems.

Also MTBF and MTTR are unknown quantities at the present (only LBDS LHC has undergone reliability studies)


14

Procedures Magnet conditioning procedures (SPS

MKE4 & 6, MKP, MKD and LHC MKI) ‘Consignation’ procedures Electronic module installation procedures LHC LBDS generator testing + energy scan Test procedures for electronics card Procedures for loading BETS energy tables

et al ...


15

Procedures – movies ?

http://www.youtube.com/watch?v=AwugbEjSxAI

The requirements are:‘To create a multimedia support that shows and demonstrate some typical intervention carried during piquet time and outside of piquet time on TE/ABT equipment (Kickers & Septa)





16

Procedures – JIRA“ Plan, track, work – smarter and faster ”

ABT-EC has started to use JIRA as an accomplished work and planning management tool for the following actions:


Creating and tracking FSU jobsCreating and tracking Hardware production jobs

Notification of repairs

Notification of troubleshooting

Notification of general maintenance of equipment related issues

Intervention tracking

17

Interventions/Repairs Daytime requires -> Experts Outside working hours -> Piquet/Experts

These events can last 5mins (ex: Remote solving) up to extreme cases such as 48 hours or more (intervention in-situ involving longsome trouble shooting, special equipment handling, heavy maneuvering. Lots of specialists, etc.,,,)

And occasionally a new spare part will be required !!!


18

Anatomy of a typical intervention


Logintervention

CCC calls

‘online’ monitoring

of faulty system

Go to systemlocation

Solveproblem

Call CCC

Intervene remotely

Problem solved

IdentifyFault

Stay around

Leavepremises

Logintervention

Controls, Hardware,

Procedural ?

TimeMaterialsPressureHuman factorUnknownsTechnicalexpertise sharingEtc. ...

19

Surface interventions vs tunnel interventions

Surface equipment interventions Items and spares are located in-situ (storage

cupboards, racks, Faraday cages) Missing items have to be fetched from 865 in EC

section AND other storage areas

Tunnel equipment interventions Items are invariably located in 865 or other storage

areas, no spares in UA’s for LHC kicker equipment (except certain power elements)


20

Intervention examples LHC LBDS fault: The ratio between DVs1 [v] and DVs2 [v] was out of

specification. A possible GTO failure inside the GTO stack. Decision taken to mask the interlock due to built in redundancy. Clearly a procedural intervention requiring permission from various sources and ultimately the chief LHC coordinator on that day

SPS MKE4 heater problem: after analysis of a thyratron related problem a decision was taken to replace the suspected damaged heater module This required locating a spare module, and the removal of the faulty module for future repairs

SPS MKD thyratron switch replacement: heavy maneuvering, specialists (not an evident one man job). This operation is a complex one, requiring transport, crane maneuvering, specialists, potentially dangerous and costly, re-conditioning of system (procedure) and follow up of faulty thyratron switch

‘The technical diversity of the systems under EC responsibility presents a challenge and requires high levels of expertise from individuals. All members of the team cannot always intervene on equal competence levels’


21

Written follow up of intervention Usually carried out in the logbook system, operation and EC

logbook Discussed at weekly ‘Piquet’ meetings Actions follow-up are required


22

Storage of spare equipment PLC spare parts critical stock (en-ice-plc, b.864) Electronic modules/cards (b.865) Power supplies (b.865) + building HV elements spares (b.865, b.867, in-situ) Power distribution components (b.865) Spares in-situ for surface controls (BA’s) (Electronic

modules, power electronics, thyratron switches) Spare part lists (Excel file) with components

and spare location for each installation



24

Storage of spares at 865


865 2-A50

25

Documentation types Electronic designs schematics, layouts, PCB

masks, drilling drawings, mechanical features Functional specifications and Engineering

specifications, internal notes, papers Minutes of meetings, summaries Mechanical designs System layouts, timing schemes, cabling

schemes, procedures, stock lists Many more .... !


26

Documentation Storage


Much of previous list still stored on DFS volume (should be avoided), some in file holders/no electronic versions.

Ongoing guidelines still to be finalised for the EDMS framework.

27

Pre-installation quality control Electronic diagrams reviewed by third party Surface and visual checks for H/W Weekly rounds on surface points of

equipment controls Visual inspection of produced PCB’s Electronic hardware test procedures


28

Recapitulative summaryHardware Software

Maintenance OK OK

Procedures OK OK

Storage OK GOOD

Documentation GOOD GOOD

Quality check OK OK

Module tracking POOR n.a

Returned hardware for repairs

OK n.a


POOR, OK, GOOD, EXCELLENT

29

To take away today Maintenance and methods is a vast complex

subject ! There are no all ‘encompassing’ solution This leaves room for bigger ideas and concepts What is OK today will not suffice in the future There are maintenance methods systems, they

are long and tedious (massive legacy of data entry) to put in place and require resources


Documents

prepared by R.A.Barlow (May 2014)