Accelerator Operations and Accelerator Operations and EfficiencyEfficiency
Presentation by Roger EricksonPresentation by Roger Erickson
SLAC Operations ReviewSLAC Operations ReviewSLAC, June 15-16, 2004SLAC, June 15-16, 2004
06/16/04 Accelerator Operations Review
OutlineOutline
Accelerator Operations Section Accelerator Operations Section
and Matrix Organizationand Matrix Organization Communications with Stakeholders Communications with Stakeholders MCC Control Room StaffingMCC Control Room Staffing Operator Profile and TrainingOperator Profile and Training Conduct of Operations: Conduct of Operations:
Formality of Safety ProceduresFormality of Safety Procedures Maintenance ManagementMaintenance Management ROD and Down Time PlanningROD and Down Time Planning Hardware AvailabilityHardware Availability
06/16/04 Accelerator Operations Review
Accelerator Operations SectionAccelerator Operations Section
Operations Section
R. Erickson, HeadM. Stanek, Deputy
G. Stewart, Admin. Assoc.
AcceleratorSystems
Operators
P. Schuh
A. AlarconJ. AmannC. BlanchetteM. BoyesL. ChristensenM. CyterskiK. DonnellyK. GoldsmithC. HollosiZ. KevitkyN. LipkowitzL. OttsS. SchuhT. SommerJ. Warren
OperationsEngineers
M. Stanek
H. BholatS. HooblerK. KublerG. MilanovichH. SmithZ. Van HooverS. Weathersby
Maintenance
C. W. Allen
Documentation
C. Hultquist
MCC AreaManager
W. Colocho
06/16/04 Accelerator Operations Review
Matrix ManagementMatrix Management
Accelerator programs are managed through a matrix organization Accelerator programs are managed through a matrix organization structure with links to:structure with links to:
Program Coordinators for PEP-II and Linac Pulsed Programs.Program Coordinators for PEP-II and Linac Pulsed Programs. Liaison persons representing each experiment.Liaison persons representing each experiment. System Physicists for each major section of the accelerator facility.System Physicists for each major section of the accelerator facility. Area Managers – one for each major section, who report to the Area Managers – one for each major section, who report to the
System Physicists.System Physicists. Accelerator physicists and engineers responsible for subsystems or Accelerator physicists and engineers responsible for subsystems or
commissioning tasks.commissioning tasks. Technical support departments with engineering and technician Technical support departments with engineering and technician
support for on-call maintenance.support for on-call maintenance. Accelerator Department Safety Office.Accelerator Department Safety Office. ES&H Departments, including Radiation Physics and OHP.ES&H Departments, including Radiation Physics and OHP.
06/16/04 Accelerator Operations Review
Communications with StakeholdersCommunications with Stakeholders
8:00 meeting every day, seven days a week, in MCC conference room.8:00 meeting every day, seven days a week, in MCC conference room.• Operations report by EOIC, including run time statistics, progress in carrying out
scheduled program, descriptions of problems, etc.• Reports by representatives of major experimental user groups.• Attended by Tech Div, Accel Dept, and Operations management reps, Program
Deputy, System Physicists, Area Managers, representatives of major technical support departments, and interested users; typical attendance: 45 persons.
8:15 Maintenance meeting, five days each week.8:15 Maintenance meeting, five days each week.• Run by Operations Section maintenance managers.• Attended by supervisors of all maintenance groups, Area Managers, and interested
physicists and engineers.• Attendance: typically 45 persons.
Accelerator Plans Meeting, weekly.Accelerator Plans Meeting, weekly.• Review of operational progress, problems, and plans.• Attended by Tech Div AD and Dept Heads, Program Coordinators, Accelerator
Section Heads and ADSO.
06/16/04 Accelerator Operations Review
Control Room StaffingControl Room Staffing
Program DeputyProgram Deputy• Duty rotates among Accelerator Department physicists.• Oversees program for 3 or 4 day period.• Not always present in control room.
Operations EngineerOperations Engineer• 1 EOIC per shift (from group of 7), on rotating 7-week cycle. • Responsible for safely carrying out the scheduled program.• Exempt position; directs the Accelerator Systems Operators.• Must be present if accelerator is capable of producing beam.• All have experience as accelerator systems operators.
Accelerator Systems OperatorsAccelerator Systems OperatorsTypically 3 operators per shift on rotating 5-week cycle: • Linac and sources • PEP-II rings• Secondary program, PPS control, and backup.
06/16/04 Accelerator Operations Review
Typical Operator ProfileTypical Operator Profile
Typical applicant is a recent graduate with a Typical applicant is a recent graduate with a bachelor’s degree in physics .bachelor’s degree in physics .
• 15 have bachelor’s degrees• 4 have master’s degrees• 4 have a technical background and extensive SLAC
experience.
• Accelerator Systems Operator 1• 1-2 years of experience
• Accelerator Systems Operator 2• 2-5 years of experience
• Accelerator Systems Operator 3• 5+ years of experience
• Operations Engineer• 2+ years of experience
06/16/04 Accelerator Operations Review
Operator Training ProgramOperator Training Program
Workbook and evaluation for each classification:Workbook and evaluation for each classification:• Accelerator Systems Operator 1• Accelerator Systems Operator 2• Accelerator Systems Operator 3• Operations Engineer
ES&H Training (including RWT-II and LRCA)ES&H Training (including RWT-II and LRCA)
Accelerator Operations Safety Procedures TrainingAccelerator Operations Safety Procedures Training
(including PPS operation, searching, interlock testing for each zone.)(including PPS operation, searching, interlock testing for each zone.)
US Particle Accelerator SchoolUS Particle Accelerator School Downtime seminars and special projectsDowntime seminars and special projects
06/16/04 Accelerator Operations Review
Documentation HierarchyDocumentation Hierarchy
A dm in is tra tive D ocum en ts example: A d m in istra tive C h e ck lis ts
total num ber of documents: (6 )
P rog ram C on tro l Docum en tsB e a m T im e R e q ue s ts
(17)
A cce le ra to r O pe ra tions D ocum en tsO p era tio n s L ina c T u rn -on C h e ck list
(71)
S a fe ty D ocum en tsS e arch P ro ce d ures
(41)
M ain tenance D ocum en tsC a ll- in L is ts
(4 )
T ra in ing D ocum en tsA S O -2 Q ua lifica tio n W orkbo ok
(19)
D ocum en ta tion D ocum en tsA D D o cu m e n ta tion A rch ive P lan
(5 )
A cce le ra to r D epa rtm en t O pe rations D irec tives
06/16/04 Accelerator Operations Review
Safety ProceduresSafety Procedures
Operators are trained to carry out certain well-defined tasks, Operators are trained to carry out certain well-defined tasks, following formally controlled procedures.following formally controlled procedures.
Examples:Examples:• Interlock Checklists• Search Procedures• Entry and Exit Procedures• Safety Inspection Checklists
Safety procedures:Safety procedures:• undergo a formal review and approval process.• are updated as conditions change.• are available on the AD Documentation web site, although the signed
originals are preserved in paper form.
Completed checklists and log books are archived.Completed checklists and log books are archived.
06/16/04 Accelerator Operations Review
AcceleratorAccelerator Department Safety Department Safety DocumentsDocuments
Accelerator Operations BAS Key and Lock Procedures Hot SheetAccelerator Operations BAS Instructions Incident Response ProceduresAlarm Response Procedures Laser Safety ProceduresArcs PPS Log Linac etc Building Emergency PlanBeam Containment System Procedures Linac PPS LogBSOIC Certification Checklists Logging Requirement for Work…BSY PPS Log MCC Building Manager Checklists CID PPS Log MCC Emergency PlanDRIP/DR PPS Log MCC Key Safe CheckoutElectrical Hazard Test Procedures PEP PPS LogEmergency Notification Form PEP PPS OverviewEntry and Exit Procedures PLS Keys Procedures and LogEquipment Checkout Log Positron PPS LogESA PPS Log PPS Interlock ChecklistsFFTB PPS Log PPS Zone MapsFinal Focus PPS Log Safety Configuration ControlFire Alarm System Reference Manual Safety Inspection ChecklistsGuidelines for Work in SLAC Accelerator Housings Search ProceduresGun Test Lab Safety Procedure VVS Remote Turn-on ProcedureHigh Voltage Test Facility Safety Procedures Warning Response Procedures
06/16/04 Accelerator Operations Review
Maintenance ManagementMaintenance Management
Problems that interrupt the scheduled program trigger immediate repairs Problems that interrupt the scheduled program trigger immediate repairs to restore operations.to restore operations.
24-hour on-site staff to repair:24-hour on-site staff to repair:• Mechanical utilities• Power supplies• Modulators• Cryogenics
On-call staff for:On-call staff for:• Electrical power• Magnets and other mechanical systems• RF systems• Vacuum systems• OHP
Extensive call-in lists identify first responders, alternates, and experts for Extensive call-in lists identify first responders, alternates, and experts for all systems.all systems.
06/16/04 Accelerator Operations Review
Opportunistic MaintenanceOpportunistic Maintenance
Opportunistic ApproachOpportunistic Approach• Whenever possible, identified maintenance, repair, and installation jobs are pre-
planned, but deferred until an opportunity arises.
CoordinationCoordination• Daily 8:15am meeting.• Coordinated by Operations Maintenance Office.• Area Managers are advocates for their areas.
Standby Maintenance List (SML)Standby Maintenance List (SML)• Jobs that are planned and can be launched on short notice.• Provides options for EOIC to take advantage of unscheduled interruptions.
Repair Opportunity Day (ROD)Repair Opportunity Day (ROD)• Launched only when required by a driving event.• Carefully planned, and typically involve 100 or more jobs.
ARTEMIS Electronic DatabaseARTEMIS Electronic Database
06/16/04 Accelerator Operations Review
ARTEMISARTEMIS
ARTEMIS (Accelerator Remedy Trouble Entry and Maintenance ARTEMIS (Accelerator Remedy Trouble Entry and Maintenance Information System) is an Oracle database with a Remedy interface, Information System) is an Oracle database with a Remedy interface, accessible over the internet by everyone involved with SLAC accessible over the internet by everyone involved with SLAC accelerator maintenance.accelerator maintenance.
Operators report problems as they occur by making entries in Operators report problems as they occur by making entries in ARTEMIS.ARTEMIS.
Repair work is launched immediately if the problem is inhibiting the Repair work is launched immediately if the problem is inhibiting the scheduled accelerator program.scheduled accelerator program.
New ARTEMIS entries are reviewed at the daily maintenance meeting. New ARTEMIS entries are reviewed at the daily maintenance meeting. Open questions are addressed and jobs are assigned.Open questions are addressed and jobs are assigned.
Jobs may be assigned to the Standby Maintenance List (SML), or to the Jobs may be assigned to the Standby Maintenance List (SML), or to the Repair Opportunity Day (ROD) list, or marked for the next long down Repair Opportunity Day (ROD) list, or marked for the next long down time.time.
Jobs are tracked using ARTEMIS until they are finally closed.Jobs are tracked using ARTEMIS until they are finally closed.
06/16/04 Accelerator Operations Review
Review of New ProblemsReview of New ProblemsSample from the June 10 maintenance meeting.
06/16/04 Accelerator Operations Review
Repair Opportunity DaysRepair Opportunity Days
The accelerator systems are The accelerator systems are notnot turned off for maintenance turned off for maintenance on a routine, scheduled basis.on a routine, scheduled basis.
A ROD list of pending repair jobs is derived from the A ROD list of pending repair jobs is derived from the ARTEMIS database and maintained by the Operations ARTEMIS database and maintained by the Operations Section. Installation and PM jobs are added as needed.Section. Installation and PM jobs are added as needed.
All ROD jobs are reviewed in the daily maintenance All ROD jobs are reviewed in the daily maintenance meetings.meetings.
The ROD list can be viewed over the internet.The ROD list can be viewed over the internet. When a “driving event” occurs, a detailed plan can be When a “driving event” occurs, a detailed plan can be
assembled and launched quickly, using the ROD list.assembled and launched quickly, using the ROD list. A typical ROD day includes about 100 distinct jobs.A typical ROD day includes about 100 distinct jobs. Three ROD days so far in 2004.Three ROD days so far in 2004.
06/16/04 Accelerator Operations Review
Long Down Time PlanningLong Down Time Planning
Long Down Times usually occur once each year (although the linac Long Down Times usually occur once each year (although the linac hasn’t seen a long down time for two years) and typically last two to hasn’t seen a long down time for two years) and typically last two to three months.three months.
Area Managers collect job requests from system physicists, project Area Managers collect job requests from system physicists, project engineers, maintenance groups, and the ARTEMIS database.engineers, maintenance groups, and the ARTEMIS database.
A comprehensive schedule is prepared, updated, and reviewed at A comprehensive schedule is prepared, updated, and reviewed at weekly meetings, starting about three months before the down time.weekly meetings, starting about three months before the down time.
This schedule lists each job with an assigned shop, and lists the Project This schedule lists each job with an assigned shop, and lists the Project or Group Manager, Responsible Task Person, and Area Manager. or Group Manager, Responsible Task Person, and Area Manager.
Schedule specifies time and duration of each job (with 1 day time Schedule specifies time and duration of each job (with 1 day time resolution), and number of persons required.resolution), and number of persons required.
Detailed (shift by shift) schedule of pre-run testing, safety certification, Detailed (shift by shift) schedule of pre-run testing, safety certification, and turn-on activities is prepared by the Operations Section and revised and turn-on activities is prepared by the Operations Section and revised as needed on a real-time basis.as needed on a real-time basis.
06/16/04 Accelerator Operations Review
Maintenance StatisticsMaintenance Statistics
ARTEMIS entries since Jan 1989: 66,883 ARTEMIS entries since Jan 1989: 66,883 (as of June 14, 2004)(as of June 14, 2004)
Total identified problems not yet resolved: 620Total identified problems not yet resolved: 620Many of these involve non-critical problems that were observed, but went away without Many of these involve non-critical problems that were observed, but went away without clear explanation; e.g., a flow switch trip or a power supply found to be out of tolerance.clear explanation; e.g., a flow switch trip or a power supply found to be out of tolerance.
ARTEMIS entries since September 1, 2003:ARTEMIS entries since September 1, 2003:• 2306 new problems (8.0 per day).• 339 problems resulted in lost PEP-II run time.• 2222 problems closed (7.7 per day).
Active jobs:Active jobs:• SML: 83• Next ROD: 89• Aug-Sept down time: 109
06/16/04 Accelerator Operations Review
Accelerator Performance StatisticsAccelerator Performance Statistics
Revealed failures: malfunctions resulting in lost beam time. Also called “events”.Revealed failures: malfunctions resulting in lost beam time. Also called “events”.
Unscheduled down time:Unscheduled down time: hours lost from scheduled program due to malfunctions.hours lost from scheduled program due to malfunctions.
Mean Time to Fail:Mean Time to Fail:
MTTF = MTTF = Scheduled beam timeScheduled beam timeEventsEvents
Mean Time to Repair:Mean Time to Repair:
MTTR = MTTR = Unscheduled down timeUnscheduled down timeEventsEvents
Availability = 1 - Availability = 1 - Unscheduled down timeUnscheduled down timeScheduled beam timeScheduled beam time
NOTE:NOTE: PEP-II aborts are not counted as downtime, unless the event is reported; PEP-II aborts are not counted as downtime, unless the event is reported; i.e., unless we stop to fix something and make a database entry. i.e., unless we stop to fix something and make a database entry.
06/16/04 Accelerator Operations Review
Unscheduled Down StatisticsUnscheduled Down Statistics
Jan 10, 2000 – June 29, 2003Jan 10, 2000 – June 29, 2003
06/16/04 Accelerator Operations Review
Down Time Analysis DetailsDown Time Analysis DetailsExample: PEP-II systems, Run 3Example: PEP-II systems, Run 3