Offline shifter training tutorial

L. BetevFebruary 19, 2009

2

Outline

• Offline shifter basic responsibilities• The shifter check list• Systems and tools (separate talks)• The Dashboard• The Shuttle• Offline Shifter Information System• Event display

3

Basic responsibilities – RAW data

• The RAW data path

DAQ online buffer @P2

Fast optical link to CERN CC, maximum rates: 500MB/sec (p+p), 1.25GB/sec (Pb+Pb)

CASTOR2 disk buffer CASTOR2 tape buffer

Step A

Step B

Reduced100 MB/sec (p+p)

4

Step A – Online buffer -> CASTOR buffer

• Automatic and well-exercised (it almost never goes wrong)

• At this step, the files are also registered in the AliEn catalogue through a gateway

• DAQ is nominally responsible for the transfers

• Offline provides the registration gateway

• If not working, DAQ/SL notifies the shifter and/or the alice-shift-alarms@cern.ch expert list

5

Step A – Shifter responsibilities• Monitors the fill of the CASTOR buffer

(dashboard)• Notify the run coordinator/shift leader if more than 80%

full• Clear disk space following instructions received from

the SL

• Follow the registration of RAW (dashboard)• All runs in PHYSICS partition are typically written

to CASTOR• Follow the run screen and grow suspicious if none of

the runs are being registered• Contact the SL and ask what is going on

6

Step B – CASTOR buffer -> Tape storage

• Selective copying of runs to tape• Part of the p+p data stream (depends on the acquisition

rate, max 100MB/sec)• Full data stream in Pb+Pb (1.25GB/sec)

• The selection of runs to be copied/removed is provided by the SL• Offline shifter is responsible for the copy procedure

(dashboard)• And for the deletion of data from the CASTOR buffer

7

Basic responsibilities – Shuttle

• Covered in Shuttle presentation• Here just to put it in the context of the basic responsibilities

8

Basic responsibilities – event display

• Covered in Event Display presentation• Here just to put it in the context of the basic responsibilities

9

Basic responsibilities – data replication

• After RAW is recorded to tape in CASTOR• A copy is made to a remote T1 centre (out of 6

possible) for custodial storage and processing

• The replication is an automatic process, triggered at EoR• Progress is displayed on the dashboard, the

shifter follows the transfers and reports problems

• Presently (muon/calibration runs) – automatic replication is disabled

10

Basic responsibilities – offline processing pass 1 (at CERN T0)

• After RAW is recorded to tape in CASTOR + Shuttle is done• Processing is launched automatically

• Progress is displayed on the dashboard

• Automatic processing – only for PHYSICS runs• Detector calibration runs are processed on request

• The Offline shifter (if asked by detector groups/run coordination) collects the run numbers and writes them in the shifter report

11

Offline shifter check list• Registration of RAW (dashboard)

• Periodic check of status

• Follow PHYSICS runs (start/stop in DAQ logbook) and registration to CASTOR

• Ask shift leader in case of doubt

• Report registration errors to on-call expert (list of experts in aloshi)

• Run copy and removal procedure (dashboard)

• Shuttle (dashboard)

• Follow on processing of all runs + global Shuttle messages

• In case of preprocessor failures, escalate to (concerned) detector shifters, note in shifter report (aloshi)

• In case of Shuttle failures first follow the restart/debug procedures, then report to on-call expert

12

Offline shifter check list (2)• Data replication (dashboard)

• Periodic check of replication status

• Note ‘stuck’ runs – not replicated 12 hours after registration – in the shifter report pages and sent list to alice-shift-alarms@cern.ch

• Data processing pass 1 (dashboard)

• Periodic check of processing status

• Note ‘stuck’ runs – not processed 12 hours after registration – in the shifter report pages and sent list to alice-shift-alarms@cern.ch

• Shift report (aloshi)

• At end of shift – summary of the operation and noteworthy events

13

System Run Coordination meeting

Evening shifter onlyAttend the daily @17:30 System Run

Coordination (SRC) meetingPrepare and present a 24-hour Offline status

reportTemplate for the report is given in aloshi

14

General shifter rules• Before pressing the

• Read the procedures and rules, defined for each error type• aloshi has a search feature, use it to look for

similar problems and solutions• Try out the remedies• If all fails, inform the on-call expert

15

Information sources for the shifter

• The shifter manual – instructions

• Shifter interface (http://aloshi.cern.ch)

• Monitoring – MonALISA (http://alimonitor.cern.ch/)

• Dashboard

• Shuttle

• Processing and data management