Designing a HEP Experiment Control System, Lessons to be Learned From 10 Years Evolution and Operation of the DELPHI Experiment. André Augustinus 8 February

Designing a HEP Experiment Control System,

Lessons to be Learned From 10 Years Evolution and Operation of

the DELPHI Experiment.

André Augustinus8 February 2000

CHEP 2000 8 February 2000 2

Overview

The DELPHI experiment control system

The evolution of the DELPHI ECS

Observations and lessons learned from over 10 years of operation

Conclusions


What is an ECS ?

What is an Experiment Control System ? All systems controlling or monitoring

(parts of) an experiment ‘Classical’ controls

Power supplies, temperatures Data-flow controls

Start/stop of a run Interface with external systems

Accelerator, safety system


The DELPHI ECS

What is the DELPHI Experiment Control System ?

Two main components: Slow Controls system (SC) Data Acquisition System (DAS)

Other components: Trigger system Communication with LEP accelerator Data quality monitoring


The DELPHI ECS (SC)

Slow Controls system Several 1000 channels Power supplies, temperatures, pressures Independent per sub-detector

G64/G96 crates (~100), PLCM6809, M68340 processorsOS9, RPC/OSI, TCP/IP


The DELPHI ECS (SC)

The ECS is used to Prepare sub-detectors for data taking Report (and correct) anomalies Integrate ancillary systems

Gas, magnet, safety Store detector status on database(s) Control & monitor the hardware


The DELPHI ECS (DAS)

Data Acquisition System: Over 250 000 channels 20 partitions + central partition

Can run independentlyFastbus (~180), OS9, TCP/IP

The ECS is used to Configure the readout Initialise the partitions Control and monitor the data flow


The DELPHI ECS (Other)

Trigger system Decision and timing Hierarchical: local and central

ECS is used to Configure and initialise the trigger

system Download look-up tables Monitoring of counters


The DELPHI ECS (Other)

LEP communications Bi-directional exchange of experiment

and LEP machine parameters Luminosity, background, machine settings

Data quality monitoring Several 100 histograms Event processing farm


The DELPHI ECS (Software)

Two main packages:SMI++

Models the behaviour of a system as a FiniteState Machine in a dedicated language (SML) Objects

Associated: represent concrete entitiesAbstract: behaviour defined in SMLObjects are in a definite stateObjects can receive action requestsLogically related objects are grouped in domains


The DELPHI ECS (Software)

User interface State of objects are presented to the

operator Commands are given by sending action

requests

DIM Publish/Subscribe paradigm Universal data exchange package

Refer to previous talk by Clara Gaspar


The Evolution of the ECS

Started taking data in 1989 No ‘real’ ECS yet Only a rudimentary version of SMI Line mode interfaces ‘Human’ synchronisation



Taking advantage of SMI (1990-1991) Completion of SC and DAS domains Centralised control of the experiment Abstraction

Variety of hardware can be represented by one type of object

Logically related objects can be summarised in one single object

Uniformity across sub-detectors



Introduction of DIM (1993 onward) Make use of user interfaces more flexible Solve communication problems inside SMI Evolved to a universal data exchange

package in the experiment DIM really started an integrated ECS :

Trigger, LEP communications became integrated part of ECS



Reengineering of SMI (1996) Improve maintainability and portability Using OO techniques Smooth transition because of well defined

interfaces (already in design phase) Hardware and Software upgrades

New sub-detectors New technologies New versions of operating systems


Automation in the ECS

Automation is unavoidable and imperative for efficient running of a complex experiment Too complex for a non-expert operator Gain in time, efficiency Ensure consistency

Relatively easy to implement because of the use of SMI in all domains


Automation in the ECS

Examples of automation Trip recovery

Automatic reaction to trips DAS auto-pilot

Automatic reaction to a variety of anomalies SMI analyser

Analyse combination of SMI states ‘Big Brother’

Interconnection of various domains ‘Hands-Free’ running of the experiment


Future ECS

LEP experiments were probably the first generation that needed an ECS

One should take advantage of the expertise gained in running these big experiments


Future ECS

Partitioning Well thought out to allow stand-alone

running (debugging, calibration) Unhindered operation when part of the

experiment is off Central control

Small crew of operators Well structured commands


Future ECS

Uniformity across ECS subsystems Will ease integration and automation Will reduce maintenance efforts

Uniformity across sub-detectors Common hardware and software

reduced costs and maintenance effort reduced development efforts easier operation

Use of commercial solutions


Future ECS

Central support team Strong central support is a great benefit Provide guidelines and frameworks

enforce uniformity Provide common solutions for common

problems Will ease maintenance over lifetime


Future ECS

Flexibility ECS is never ‘finished’ Many changes will happen over the lifetime

of an experiment, the ECS should cope with Modification or addition of a sub-detector Upgrades with new technology Change of working points or operational

procedures Easy to modify or reconfigure

Good documentation


Future ECS

Efficient day-to-day operation Abstraction

Non-expert operators can run the experiment Hide detailed information Uniform representation and commands

Automation Automatic error recovery Automate ‘standard operations’

Proper training and adequate documentation


Summary

Designing an ECS Strong central support Partitioning Uniformity Flexibility Abstraction Automation

Documents

Designing a HEP Experiment Control System, Lessons to be Learned From 10 Years Evolution and Operation of the DELPHI Experiment. André Augustinus 8 February