View from ROC-West !
DRC report 17 Feb 2016
Nuno Barros
1
FNAL, Feb 17, 2016
• Noise hunting
• Increased prescale
• Running with all components in disk writing mode (later part of the week)
• SSP tests
• PTB testing
• DAQ stress testing
• More stable after “fixing” the prescale
• Run control “hanging”
• Typically can take ~30 min of data until DAQ crashes2
Major activities
• At about 6 pm on Friday the noise suddenly increased
• Shifter noticed and posted in elog, but nobody noticed the post until next day
• Need to define better procedure to react to events during shifts
3
Noise problems
• High frequency modes showed up again
• First at 500 kHz
• Then “moved” to 650 kHz
• Usually got better after applying Alan’s “trick”
• Ramping bias voltage 100 V and then ramping down again
• Seemed to work until yesterday
4
Noise observationsGood
Really badNot as bad
• Noise seems to increase suddenly
• Small increase over time followed by a sudden shot up
• Run where the noise is high usually crashes (still trying to get this noise increase on file)
5
Noise observations (cont)
• When noise is high RCE’s seem to miss more sync pulses
• When noise was lower failure rate was ~10%
• When noise is high several runs in a row fail to start (miss sync pulse)
• Saturday increase trigger prescale to 8 so that we could take data continuously with disk writing
• DAQ became more unstable
• Turns out the trigger divider was broken (prescale 8 didn’t work). Now prescale set to 7.
6
Still on noise
• RCE’s (or the FEMB attached) 3 and 14 still misbehaving.
7
Recurring issues
• New firmware update tested on PTB
• Fix two remaining glitches
• Fix problem in trigger prescale
• Fix missing microslice at begin of run
• Need to update lbne-artdaq at same time firmware is updated
• Holding it until sync update with John Freeman
• PTB high statistics run taken overnight on Monday (shifter stopped the DAQ at 8 am next day)
• SSP tests being run hourly by shifter
• J. Insler running other tests during the night after DAQ crashes
• SSP01 removed from configuration (lbne-artdaq does not start when it is included)8
Other testing
9
SSP noise
• Behaving better after sorting out the prescale issue
• RCE timeouts still the major reason for crashing
• Occasionally still see crashes showing signs of back pressure
• Happened a couple of times yesterday during the evening shift
• Does not make sense given a prescale of 7 (trigger rate ~0.2 Hz)
• Run control hanged a couple of times
• Seems to be more stable now10
DAQ stability
• Need to get the noise low
• Right now data is almost unusable
• Update lbne-artdaq and PTB firmware
• Will provide better control over the trigger prescales
• Probably temporarily replace vertical (BSU) trigger by other horizontal trigger
• Need expert call list
• Currently responsibility of DRC to know who to contact
• Need available experts11
Next steps
• Could implement a shower trigger
• Based on theory that a cosmic shower sends the TPC into the oscillation pattern
• The current BSU trigger is not being used
• Trigger consists in coincidence of a single counter hit in the roof (RM) with a single counter hit in the cryostat (CL)
• Complement of this trigger would be more than one counter in the roof in coincidence with more than one counter in the cryostat
• It exists in the firmware already
• Implementing the current trigger requires implementing this one
• Need to estimate the rate of this trigger to see if it would be of any use12
Next steps (proposal)