D0 Collaboration Meeting July 2002 L2: The Road Ahead James T. Linnemann MSU D0 Oklahoma Workshop July 9, 2002 Special Thanks to Terry Toole for status

D0 Collaboration Meeting July 2002

L2: The Road Ahead

James T. Linnemann

MSU

D0 Oklahoma Workshop

July 9, 2002Special Thanks to Terry Toole for status slides


Trigger Organization

CTTPS

CAL

CTTCFT/CPS

STT

MUON

FPD

CAL

MUON

PS

CTT

GLOBAL

Detectors L1 Trigger

Framework/SCL

L2 Trigger

FPSCPS/St

CAL

CFTCPS Ax

L0

SILICON

SCINT

MDT/PDT


● Making decisions since May 1st (On since Nov. 2001)● Rejection done on em, jet, and muon objects ● 4 L2 Crates online: Global, Cal, Mu Forward, Mu Central● 1 Alpha per crate

– Limits L2 input to ~.5kHz debug– Probably ~1.5-2KHz maxopt if no optimizer problems)

● Ability to disable individual L2 inputs● L2 Software Framework improved

– More robust operation in real system– Downloading works!

● Crate configurations, Trigger scripts ● Expert & Non-expert monitoring GUIs in control room ● L2 Experts on call

Level 2: Status


Level 2: StatusL2Cal

● Em and Jet algorithms, Admin run simultaneously on 1 alpha

● coverage to 0.8 (commissioning 2.4 in progress)

● Adjusting L1Cal inputs with L2 Software (performance hit)

– pedestals, gains, seed mask, L2 headers/trailers

– Recently updated pedestals, gains to reflect L1Cal mods● Jet parameters Et cut

● Em parameters Et cut, EM fraction, (isolation)

L2Muon: Rejecting on Muons

● Loose, Medium, Tight

● Pt cut

● No track matching yet


L2 Future: Short Term

● Continue improving smoothness of operation● Online bit-by-bit verification● Improvements to online monitoring and debugging

– Start running L2 trigger examines– Error logging

● Add New Triggers: dimuon trigger soon; others to follow. ● Work to Increase Rate Capabilities

– Speed up code– Overlap data input & processing

● Currently in lockstep mode– Want to overlap DMA, L3 readout, and processing

● Multiple alphas per crate


Rate Limits:It takes a task force

● Trigger framework and muon FE crate bug fixes recently– Big decrease de-synch problems– Now would like automatic response to SCL_INIT requests (now hangs!)

● L2 code fixes – properly responds to SCL_INIT, config_crate w/o reboot!– Was causing wakeup call to L2 expert every 2 hours—not acceptable

● Limit at 200 Hz: de-synch problems in muon pixels FE– Muon group working on this —many players, not all on site

● Some indications of Cal problems around 400 Hz● L2 probably can run >500 Hz—no known problems● Currently: running w/o L2Mu in trigger

– Crate in run; diagnose + fix problems, but not in global– Hope ready to turn back on this week– Special running conditions in this configuration ALSO needed debugging

● FE crates couldn’t raise busy if L2Global not making decisions


What makes progress possible● Improving error messages

– Details from SLIC DSP’s of which channels de-synched– Errlogger messages tagged with tick and turn by framework– DIAGNOSIS—you can’t fix it by guessing whose fault it is

● L2 expects same inputs every event, and all fragments from same event● Central L2 error logs (eventually, to SES)

● TCC (oversubscribed); working on phase 1:– collect messages from all crates’ dualport memories– Time stamp them (5 sec blocks)– Merge into single file

● Better timing information now available (offline, online)– Speedup in I/O, cleanup of many layers of jet code (Roger)– Consulting from Computing Division (Fischler)– L2 headers and candidate bit masks avail from L1Cal would help


1100 with all 10 inputs?

maxopt ~3 x faster

Slower than prototype code tests—are you surprised?

Add < 50usec for DMA, L3, monitoring, etc


Level 2: Extended Plans● Full coverage for L2Cal (as inputs become available)● Commission L2CTT, L2PS, L2STT

– Mid-August: L1CTT added to L1Muons● L2muon rejection drop?

– L2CTT: Inputs may be available September– L2PS inputs

● CPS stereo may be available in August● CPS axial, FPS available this Fall (Oct 1 earliest)

– L2STT begins commissioning this Fall; ● 30 degree sector by Oct 1; all by Nov?

– Expect 6-8 weeks to commission a given processor

once inputs are available—we will need ID group help– Will expand capabilities of L2Global (code works on MC)

● , e- triggers from track matching ● , e- triggers from PS


Level 2: Extended Plans II

Alpha PIO rewrite (August??)

DMA overlap with processing

Multiple alphas per crate (L2Cal, e.g.)

Betas (850 MHz PIII cpu; Replacements for Alphas)● 2nd prototype expected July 20th

– Test at D0 during August● Upgrade to 1 GHz PIII 3x faster than Alphas● Assuming success with pre-production tests:

– Start production in September– Hardware checks in November– Begin installation in December


Assessment● Good:

– Rejecting where we have inputs (since May)– All code running from releases– Trigsim code matched with online code

● Ntuples can be used for studies● Bad

– Late Start (we had our own problems, but we also had “help”)● alpha hardware, and arrival of inputs● 7EE’s left the project before full commissioning of their subsystem● Commissioning took longer than we’d hoped (remember this phrase)

– L2 Monitoring behind commissioning needs– Can’t run at spec of 5kHz– Can’t look at SLIC inputs yet– Trigsim needs an expert to run it (even to make ntuples)– No thumbnail


Assessment, II: The Ugly(Frustrations, mostly)

No tracking inputs

Helping debug missing or de-synched inputs

“we can’t run because of Level 2” (!)

Operating without designed-for environment

reasonable priority decisions, but costs time, enthusiasm

SCL_INIT request connected to aetherCode not designed to chat with outside world while for SCL_INIT

Some functionality missing from framework/COORVarious flavor of Mark&Pass support

Monitoring collection

No common object interface across trigger levels to Reco, MC

Makes certification harder than necessary

Run summary, including pass fractions for L2 (not yet…)


L2 Manpower● 1 postdoc leaving soon, 1 joining ● Loss of experience● Muon group very thin● 2 other postdocs ramping down● I’m on sabbatical starting January

– There is a certain level of fatigue● This is the cost of slow turn-on

– Alpha hardware; no tracking inputs● People who have been doing near 100% service for several

years need to be turning more of their attention towards physics

– We can build hardware and code– We can’t do all the studies


Areas where help needed● Thumbnail for L2 in particular, Trigger in general● Release Regression Testing (base on examine?)

– Share technology with L3, RECO?– Need data samples from physics groups

● Common Object Interface for certification● Certification for CPS, FPS, CTT, STT!● Code maintenance for CPS, FPS?● IOGen: multiple version reading; long-term support● Hardware maintenance—we can provide training

– FIC (and Alpha to lesser extent)● L2 expert shift—we can provide training


Trigger/DAQ Priorities (Lipton)1. Keep experiment running (but take necessary time for development)

1. come out of the October shutdown with the D0 Trigger/DAQ operating as designed

2. Improvement of DAQ rates

1. Find problems in multibuffer mode

2. Implement automatic SCL inits, error logging

3. CTT – L1muon, L1, L2 track trigger terms

4. L1 Cal extensions and debugging

5. L2 Improvements

1. Tune filters, improve timing

2. L1 seeds (and headers) to L2 (speedup for L2)

3. Multiple alphas/crate (issue at ~2 kHz)

4. L2

6. Improve monitoring/examines/error logging …


Some L2 Priority Choices● Multiple CPU/crate, DMA overlap with Alpha or Beta?

– Rate pressure, and who does which work– Could try maxopt and wait for betas, hoping we don’t:

● Find optimizer bugs● Have to interpret debug output (maybe use test stand)

● How best to get started with tracks in L2?– L2CTT, then switch to L2STT, or skip L2CTT?– Tracks vs. displaced vertices within L2STT

● We may be short of manpower for L2PS– L2axial on L2CTT tracks for electrons– L2CPS stereo arrives before independent L2PS axial (for photons)– L2FPS arrives last– Anyone interested enough in photons to help?– Defer in favor of tracks?

● New functionality vs. Smoother Running, Verification


Readout


CTT

● L1 CTT used as basis for L1 tracks– ~6 week schedule delay (~mid August)

● Rough schedule for other components:– L1, L2 Tracks: No sooner than September 1

– CPS stereo clusters: No sooner than August 1

– FPS clusters: No sooner than October 1.


Level 2● L2 requires inputs for every crate for which L2 is configured in any run

– Sensitive to problems in other systems– Needed substantial work for trigger database and COOR to provide proper support Adding better error logging

to diagnose FE problems that lead to L2 hangs● L2 Cal

– Em and Jet algorithms, Admin run simultaneously on 1 alpha– coverage to 0.8 (commissioning 2.4 in progress)

● L2Muon– Rejecting on Loose, Medium, Tight muons wit Pt cut

● Future● Full coverage for L2Cal

– Commission L2CTT, L2PS, L2STT

– Automate SCL init requests – L2 beta – increase L2 input rate to >2.5 Khz



Documents

D0 Collaboration Meeting July 2002 L2: The Road Ahead James T. Linnemann MSU D0 Oklahoma Workshop July 9, 2002 Special Thanks to Terry Toole for status