TPC shift guide1 04.09.2008 TPC shift guide Run period 10.09-31.12.2008 Updated on 04.09.2008

TPC shift guide 104.09.2008

TPC shift guideTPC shift guide

Run period 10.09-31.12.2008Run period 10.09-31.12.2008

Updated on 04.09.2008Updated on 04.09.2008


ForewordThis shift guide is intended for the TPC shifter to be able to operate, run and

monitor the TPC:• Identify the relevant machines:

• aldaqacr34, the local TPC linux machine in the Working Room (TPC.501)• alitpcon001, the Windows PVSS Operator Node• alitpcwn008, the Busy Box linux machine

• Power on/off and configure the FEE for any specific run• Configure and initialise the Busy Box• Use the Local Trigger Unit• Start/stop various standalone runs, or get ready for a global run

• Periodic pedestal and cal. pulser runs (DAs); laser runs

• Use the online monitor to look at event displays, and use AMORE• Monitor the TPC: LV, FEC temperatures, cooling, ROC HV, Drift HV, gas, run

status• Use the electronics logbook• Troubleshooting, list of expert


Connecting to the Operator Node

1) Make sure there is a remote desktop session to alitpcon001; if there is

not one, a registered user does:

– rdesktop –g1280x1024 –a16 –u username alitpcon001

– Enter your NICE username and password

2) Start TPC.bat (or just TPC)

3) Click on the key icon (top left) to login as tpc tpc

4) You now have the PVSS panel

– On the left there is the tree with all the panels

– Bottom left, a synoptic with the colour-coded status of all systems

5) The top level node, TPC_DCS, will be the most extensively used

Note: if you can't type in a field, go to a linux terminal session, type something

there, and go back to PVSS


Functionality of top node1) TPC_DCS: Top node of the TPC FSM Tree

Take the lock of the FSM Tree from here. If you need to CLOSE the main

panel, first release the lock with the option RELEASE ALL

FSM Commands:

State: OFF Action: GO_STANDBY -> Switches ON LV PS

State: XXX Action: CONFIGURE -> Configuration of the FEC according to

the sent parameter (usually RESET and then COSMICS or pp)

State: XXX Action: SOR or EOR -> Actual condition values are send to

database for Shuttle

2) If the FSM has to be restarted (it also releases the lock) press

'Start/Restart All'. Wait until the 12 boxes (tpc_ui etc.) on the lower part of

the panels are green.

3) Forced restart: Press 'Stop All' wait ~20 seconds, press 'KILL FSM' wait ~5

seconds, press 'Start/Restart All'


Cooling1) TPC_COOL (in Operator Node) (next page)

2) FEC Cooling plant must be RUNNING

3) Resistor Rod Cooling plant must be RUNNING

4) In case of problems ('Standby' or 'Stopped') call expert

Experts: Cooling Piquet: CCC 72201 (ask for CV/Detector Cooling,

answer ALICE TPC)

Rudi Schmidt 16 3962

Chilo Garabatos 16 5327

Danilo Vranic 16 3691

Note: if the cooling goes to standby, an interlock will turn of the FECs after

2 min, and the LV power supplies after 7 min



Power up and configure FEE1) Switch ON LV Power Supplies from PVSS

– This operation should be done by an expert

2) Go to the FED MONITOR, select FSM state and wait for required partitions to be blue

3) In the (sub)system state tab, select CONFIGURE and enter, e.g., COSMICS as parameter. This will turn on the FECs and configure them for running with cosmic triggers.

4) Other commands are – GO_IDLE (FEC power off, do not do this for fun)– CONFIGURE with entry RESET (a popular command)– CONFIGURE with entry PEDESTAL_RUN (for pedestal run)

Before each run it is recommended to CONFIGURE with entry RESET and once more with, e.g., entry COSMICS (and reinitialise the BB, see later)

Experts: Christian Lippmann 162430, Luciano Musa 163119, Chilo Garabatos 165327


FEE Monitoring1) In TPC_FERO you can select different monitors:

– FED Monitor: shows by default the temperature of the FECs, which

should be lower than ~30 oC.

• In the lower third of the panel, all boxes should be green. If one is red, try

Restart. If still red, call expert (CL).

• The tab FSM State shows you the state of all readout partitions

– LV Monitor: shows by default the analog currents, which should be 2

Amp (FECs off) to 130 Amp (all FECs on).

– Cooling Monitor: shows the status of the cooling: it should be OK and

RUNNING (all green).

2) In case of problems call Christian Lippmann 16 2430, Chilo Garabatos

165327, or Luciano Musa 16 3119


FEE Configuration (after power up)

1) On the operator node (alitpcon001) right-click on TPC_FERO in the FSM tree to the left.

2) Choose view panel.

3) Click on the FSM button (just above).

4) Take the lock by clicking on the lock symbol.

5) In the System TPC_FERO (on the top) click on the state and choose CONFIGURE and type RESET as run parameter. The states should all be (or change to) STANDBY.

6) Choose again CONFIGURE ,and type as run parameter the right name from the table of next slide.

7) Wait for all states to change to ON. This may take ~10 seconds.

8) In case of ERRORs choose RECOVER and repeat steps 6 and 7.


ConfigurationsName Pedestal

Subtr.Zero

SuppressionRemarks

PEDESTAL_RUN - No For Pedestal DA

BLACK_FPED - No Black evens for, e.g., signal studies

ZS_FPED Fixed Yes(Threshold 3)

1 Pre- & 2 Postsamples.FPED values are part of

configuration.

ZS_PEDMEM PedestalMemory

Yes(Threshold 3)

1 Pre- and2 Postsamples

COSMICS Fixed Yes(Threshold 3)

0 Pre-,0 Postsamples, Gating Grid Suppressed,Glitch Filter On

pp Fixed Yes(Threshold 3)

Like cosmics. Only relevant acceptance readout


Configuration via DCS of single RCUs

• On the operator node (alitpcon001) open the FSM tree and

find the desired partition (RCU): TPC_RCU_xxx_p.

• Check monitoring values and change states here.

• You get the same functionality by clicking on the individual

partitions (RCUs) on the FED Monitor panel.


Troubleshooting: FEE Configuration / Monitoring via DCS (1)

The FSM tree is locked:

1) Find out who has the lock, and ask him/her to release it.

2) From TPC_DCS top node, Restart ALL, Kill FSM, Start/Restart all

No response from FEE:

1) If it is a single Partition (DCS board / RCU), try– Restart feeserver on this DCS board (ask expert),

– Reboot this DCS board (ask expert) It takes 20 to 60s for the DCS board to reboot and for the data from this RCU to appear on the monitoring panel.

2) Restart PVSSdim by clicking on “Kill PVSSdim“ in the FED Monitor panel.

3) Restart interComLayer(s) (see next slide).


Troubleshooting: FEE Configuration / Monitoring via DCS (2)

Restart interComLayers:

1) Log in to alitpcwn008 (A side) or alitpcwn009 (C side) (user tpc, no password needed from aldaqacr34).

2) Check if interComLayer is running (type „ps –a“ in a shell).

3) Restart it by executing „killall interComLayer“.

Additional information:

1) The interComLayer is an intermediate Layer between the FeeServer on the DCS boards and the PVSS and FSM Control Layer you are using. There is one per side.

2) It runs in the directory /home/tpc/icl-rundir.

3) In case it exits (or is killed), the interComLayer is automatically restarted by the script /home/tpc/icl-rundir/startInterComLayer.sh.


Busy Box• The busy box has to be reconfigured before a run starts.

• It is now done automatically from the central ECS

• It can be done from our FSM by sending a SOR command with the option RECONFIGURE

• Click on BUSY BOX button of the TPC_FERO node to see the status

of each partition

• The selected partitions should show green in the setup tab

• clicking on a partition toggles inclusion/exclusion

• clicking the label of a sector toggles the whole sector

• Click APPLY to load a new configuration

• You can reconfigure manually –not during a run!- the BB by clicking

RECONFIGURE in the STATUS tab; selected partitions should go

green


should not b

e neededFEE Configuration via PERL scripts

• Go to aldaqacr34 and type the command:

./SECTORS_do.pl has the following options:

reset/config/on/off/armSYNC

choose the appropriate one, the side, and the sector number

• Examples of commands:

./SECTORS_do.pl A config 0-17 (to configure all FEE on A side)

./SECTORS_do.pl C reset 0-4 5 6 7 8-17

• The normal sequence is on, reset, config

• The option armSYNC is used in stand alone runs only, in case

one has just reconfigured a part of the system


should not b

e neededFEE Configuration via scripts in each LDC

• If you log into one LDC, you should go to /local/home/tpc/TPCrdo

and type:

source TPCrdo_commands.sh

• Commands: SECTOR_on SECTOR_off SECTOR_reset

SECTOR_configuration SECTOR_armSYNC

• If this failed for a partition, you can address it individually:

rpx_on rpx_off rpx_reset rpx_config rpx_armSYNC

x=1,2,3,4,5,6


Not yet fu

lly operatio

nal

FEE Configuration via TPCC (1)

1) Find a desktop with a session in aldaqecs01 (/date/tpcc/Linux)

2) Type: ./tpcc_server (and maybe type after this:

./dumpOracle.tcl)

3) Find a desktop in /local/home/tpc/rundir

4) For each LDC type

/date/tpcc/Linux/tpcc_client

5) Move to aldaqpc126 (should be in /local/home/tpc/TPCrdo)

-should see configurations (1-15), different commands for FECs(100-

103), check the status of server (200) and exit (300)


Not yet fu

lly operational

FEE Configuration via TPCC (2)

If you don't see the above options, type:

/date/tpcc/Linux/tpcc_control

To see online the informations online go to ALICE DAQ

window, press infoBrowser and another window should

appear; in this new window change from Stream

(should be TPC and you have to clear this field) to

Facility (write TPCC) and then press Online


Trigger (Standalone)

To run CTP Emulator (LTU, trigger window):• Go to alidcscom026 (tpc tpc145) and type vmecrate tpc

• The VME crate window should appear

• select Configuration -> TTCinit

• (Busy box is used by default)

• Open the CTP Emulator window and here

• choose sequence l2a.slm

• load it by pressing the button Load sequence

• select Signal

• for pulser trigger choose BC and write the rate, e.g., 1s

• for laser runs choose pulser/edge•In case the CTP emulator gets stuck, kill the 'ltu client' process from

alidcscom026, and use the command 'xkill' and mouse-click to close the dead

windows. In case of doubt or problems, please ask the CTP shifter.


DAQ (Standalone)

Select LDCs for Readout

1. make sure you have the lock

2. press left arrows until you get to Configuration

3. press Define

4. go to GDC and choose e.g. aldaqpc046

5. go to Detector: choose TPC -> A/C side -> choose LDCs

6. save configuration and press ok

7. Select the HLT mode (Mode A: no HLT)

8. Select the equipment (readout partitions) in the equipment button of the

DAQ menu)

In DAQ Run Control window (click Date HI button in task bar to open):

In DCA window (for DAs for example):

1. Make sure you have the locks2. Select the run type: pedestal, laser, cosmics, etc3. Start a run


Now in D

CSConfiguration of Busy Box

• Log in to alitpcwn008 (user tpc, no password)

- cd /home/tpc/busy

- execute ./bbox-conf.pl

- select the partitions

- save and exit

• From here log into busy box dcs board: ssh

alidcsdcb1555

- cd /mnt/dcbrw/busy/busybox_files

- execute ./bbinit.sh (before each run!)

- to check status, execute ./regpoll.sh statusIf one of the two top registers is zero, execute

./program busybox_fpga1.bit busybox_fpga2.bit


Start/Stop stand-alone run

In DAQ Run Control

1) press right arrows to get from Configuration to Ready to Start

2) press 'Start process', wait for ready and check for errors

3) press 'Start'

4) press 'Start emulation' in LTU window

5) Note: Each sector (6 RCUs) is readout via optical fibre with a Local

Data Concentrator, a DAQ machine. The data is then transferred to

one or several Global Data Concentrators which may or may not

perform Event Building and Data Recording

6) To stop the run, quit the emulator (LTU) first, then click stop in Run

Control


Run control from ECS (standalone)• Use the ECS panel TPC Detector Control Agent (DCA) to control the

run.

– get the lock from the „TPC DAQ – Run Control“ (release it there first)

– Select the operation of the GDC• No event building

• No data recording

• mStream

– get the LTU, DAQ_RC, DETECOR, (HLT) locks

– Select the HLT mode (ask the HLT shifter)

– Select the run type (pedestal, pulser, laser, cosmic, ...)• this will start a run

• pedestal and pulser will stop automatically

– Eventually stop the runNote: if your lock is red and nobody seems to have it, make sure there isn't a

second, hidden window in your panel: this one might have the lock!


HV Monitoring1) Drift HV (SIM_TPC_FC next page)

– check Voltage (100 kV) and Current (355 A)

2) ROC HV (SIM_TPC_ROC next page)

– In FSM state, all sectors must be green

– If red (trip) call expert

– If electronics is turned off >1 hour• click on the corresponding chambers

• Switch ON button to INTERMEDIATE

– Check the currents: call expert if > 1 A

3) Experts:

- Danilo Vranic 16 3691

- Chilo Garabatos 16 5327




Gas

1) TPC_GAS (Operator Node)

2) Status must be 'OK' and 'Connected'

3) Mixer, Exhaust, Pump and Distribution modules should be run or circulating; if stop, call expert.

4) You may navigate around and look at trends

In case of problems, call expert:

Chilo Garabatos 16 5327

Gas Piquet: 16 2516


Gating Grid pulsers

The Gating Grid Pulser voltages are operated by experts.

• Experts: Danilo Vranic 16 3691, Rainer Renfordt 16 3969, Chilo Garabatos 16 5327


Lasers

One PVSS panel per side. To turn on and run do:GO_STANDBYLOAD RECIPYGO_ON (this will calibrate and launch the 'tracker' to get optimal intensity)SUTTER_OPEN

There is a trigger box which allows one to select the trigger rate –implementation in progress

• Experts: Borge Nielsen 16 4221, C. Garabatos 16 5327


Calibration pulser

There is one unit per side. In each of them click

• Open COM port

• INIT

• All ON

The corresponding automatic run (DA) is called

Pulser_Calibration_run

Click on All OFF when you are done!


Online monitoringLog in as daquser@aldaqpc081 and go to /local/home/daquser/OM• type root• in root type: .x Monitor.C• another window appears:

• select as format (DATE file/stream)•press Select and OK

• select File/Stream -> write LDC or GDC name (e.g. @aldaqpc046:).

•Monitoring has to be enabled on that machine• press Next Event button

• If root crashed or hangs, open another window in aldaqpc081 and type: killall root root.exe • To take a screenshot, open a terminal on aldaqacr34 and run ksnapshot; paste it in the logbook


How to run AMORE /11) Open two terminal windows on aldaqacr33 (as user tpc)

2) In both do: ssh aldaqdqm04, source .balice, cd /tmp

3) Check that you get the monitoring data by doing e.g.

eventDump –i –b @aldaqpc046:

4) On the first machine start the amoreAgent:

amoreAgent -aTPCQA -s @aldaqpc046:

5) On the second machine start the display:

amore -dTPC -mUIQA

- select init

- select start

- when you receive data, make the window bigger and you should

see the different variables


How to run AMORE /2The variables are:

MaxCharge – Qmax of the cluster

MeanCharge – Qtot (integral over the 5x5 cell)

NlocalMaxima – total number of clusters in this pad

NPads – average width of cluster in pad direction

NTimebins – average width of cluster in time direction (Noisy pads will

typically have a small average since the noise is not shaped, see

example in guide)

NoThreshold – fraction of digits above threshold (ZS reduction) This is

the only variable which is not a cluster variable but shows the average

occupancy for each pad (between 0 and 1)

TimePosition – mean Z (time) position of clusters (if you have many

noise clusters in a pad this will give a peak centered at the middle of

the time bin interval because the noisy clusters are typically evenly

spread out in time - perhaps a bit counterintuitive).

Some more info can be found here:

http://www.hep.lu.se/staff/christiansen/tpc_online_monitor_guide.ppt


Logbook

1) Use it.

2) For each run add an entry with the run

conditions:– FEC configuration

– Missing partitions

– Gain

– Magnet status

3) Define run quality for TPC (good, bad)


List of on-call experts

• DCS, FEE, Busy Box, DAQ, Trigger– U. Frankenfeld 164048 (DCS), C. Lippmann 162430, L. Musa 163119

• FC HV, ROC HV, Cooling– D. Vranic 163691, C. Garabatos 165327

• Gas– C. Garabatos 165327, Gas piquet 162516

• Laser– B. Nielsen 164221, C. Garabatos 165327

• GG Pulser, Calibration Pulser– R. Renfordt 163969, D. Vranic 163691

• Online, Offline– M. Ivanov 77761, P. Christiansen 78956

Documents

TPC shift guide1 04.09.2008 TPC shift guide Run period 10.09-31.12.2008 Updated on 04.09.2008