Detector (Tracker) monitoring: introduction to COSINE and the Filter Farm Giacomo BRUNO

Detector (Tracker) monitoring: introduction to COSINE and the Filter Farm

Giacomo BRUNO

UCL Louvain

Tracker Monitoring Workshop – 3 September 2004

Tracker Monitoring Workshop 3 September 2004 CERN - Geneva

2Detector monitoring with COSINE G. Bruno

COSINE links

Web : http://cmsevf.web.cern.ch/cmsevf/Mailing list : [email protected]

Filter Unit core (COSINE) E.MeschiFilter Unit HLT applications (COSINE) TBDSubFarm Manager (COSINE) V.BrigljevicSM Problem Solver (COSINE) TBDPhysics Monitor (COSINE) E.MeschiDaqPrototype framework extensions (COBRA) G.Bruno, E. Meschi, P. Kreuzer

COSINE Release Manager (COSINE) N.MarinelliXDAQ Release liaison (XDAQ) E.MeschiRun Control liaison (XDAQ) V.BrigljevicTracker liaison (ORCA) G.BrunoMuon liaison (ORCA) G. Bruno, P.KreuzerCalorimetry liaison (ORCA) N.Marinelli

Responsibles



Outline

• Introduction to COSINE• Special ORCA features when running online

applications• Monitoring scheme foreseen for the filter farm• Prototype monitoring tool in COSINE



DAQ and Filter Farm

Being components of the DAQ, the Filter Units (FUs) run a XDAQ application (intra-node communication, control, configuration…)

For the filtering task the FUs use the same reconstruction software as the off-line analysis: ORCA

Event building

Event filtering

Detector readout

BU

MSS

FU FU FU FU SM

Filter Data Network

BUBU

FU FU FU FUFilterControlNetwork

Filter farm



Bringing Reconstruction online

1. On-line software (XDAQ)– Basic core executive ( intra-application communication,

memory management, control, configuration)

2. Offline software (ORCA, COBRA)– Reconstruction code– HLT steering algorithms– Data persistency

Integration of 1) and 2) has been accomplished by means of

• extensions of the off-line software

• new software integration environment



Online WorldOffline World

COBRA

ORCA

COBRADaq

Prototype

ORCA*Daq*

COSINE

XDAQ

Dependency

Integration software project



COSINE

• Consistent Online Software Integration Environment – afs location: /afs/cern.ch/cms/Releases/COSINE – CVS/SCRAM managed– Latest release: 1_0_0_pre5– External tools: XDAQ, ORCA, COBRA, Geometry..– Basic software:

• FU implementation,

• SubFarm Manager implementation: configuration, control, monitoring (first prototype based on CDF model)



Modes of operation

Object DB

exchangeFile

exchangeFile

FU

1. New Offline ORCA application: simulated raw data production

2. New Offline ORCA application: test beam data analysis, data format studies

3. New Online COSINE application: CMS filter farm, test beams, local daq

ObjectDB

ObjectDB

Event Builder

Object DB Standard Offline

application ( Sim/Rec Application)



Event building

The full event information is made available in ORCA as a collection of FED raw data fragments, irrespective of the details of the hardware components involved in the data flow.

The event building software (http://cmsdoc.cern.ch/cms/TRIDAS/DAQkit/doc/html/index.html) is fully configurable. Example: there can be a single FED-builder, a single RU and a single BU application, all running on the same machine (interesting for local DAQ)

LTCDSN

GbE

Service PC(s)



Sub-detector specific tasks

• Setup description (xml geometry file)– Dimensions and location in space of the detectors (DetUnits)

• Readout description/partitioning (“DetPartitioner” class) – Connection between Detectors and FEDs (perhaps this

information should be in the xml geometry files, in the configuration DB …?)

– Grouping of the sub-detector FEDs into “RecoRUs” (logical software construct): subject to optimization

• Transformation of FED raw data into DIGIs (optionally also their delivery from the BU to the FU) are “on demand” (reconstruction driven).

• Persistent data are organized into collections of DIGIs associated to the RecoRUs

• Raw data – DIGI conversion (“DataFormat” package)• Application-specific code in the form of a standard ORCA user

analysis class (Observer<G3EventProxy *> …):– HLT event selection code for CMS– Test beam data analysis– Monitoring task: booking and filling of histograms



RecoRUs-FEDs-DetUnits association

RU1 RU2

FED1 FED2

Det Partitioner Geometry

creates

Det subgroup1

Det subgroup2

SubDetector SetUp

Det group 1

FED3 FED4

Det subgroup1

Det subgroup2

Det group 2

• Dets-FED association: must reflect the actual hardware connections

• FEDs-RecoRU: logical software construct subject to optimization



Online reconstruction working scheme

Data source/destinationORCA + COBRA Reconstruction

algorithm

Standard or Daq Readout Unit

POOL DB

Detectors

Raw data source (BU/file)

Formatter

D1

RecoRU1

RecoRU2

D2

BU

OODB algo

FE

D 1

FE

D 2

FE

D 3

FE

D 4

int (FED ID

)FED formatter

FED formatter

String (RU name)



Current default (CMS geometry) implementation

• RecoRUs– 77 SiStrip, 8 Pixel (determined by a division in phi

by groups of 200 DetUnits)• FEDs

– 440 SiStrip, 38 Pixel (numbers from the DAQ TDR)– Connection to DetUnits: arbitrary, but balanced

repartition.– Association to RecoRUs: balanced repartition.



• Silicon Strip FEDs can be readout in zero suppressed (ZS) mode or non-ZS mode, concurrently.

ZS RecoRU

Non-ZS RecoRU Non-ZS Formatter

ZS FED1 ZS FED2 Non-ZS FED3

ZS Formatter

Silicon Strip SetUp Geometry

createscreates

The Silicon Strip case

Still to implement: persistency for the NON-ZS DIGIs



Data formats

• Arbitrary and unrealistic data formats released since a long time for both Pixel and Silicon Strip sub-detectors

• Silicon Strip: data – formatter for test beam data analysis from Root files (M. Pearson, I. Reid, I. Tomalin)

• Pixel: work started for the final CMS data format (M. Konecki, D. Kotlinsky)



Test beams applications

The framework allows interchange of geometries and data formats. An obvious advantage of this feature is the immediate extension to test beam applications

OO DB

Silicon modules

Monitoring info

FEDS PC (FEDS in PCI slots)

Builder Unit PC

Filter Unit PC

Root data file

ORCA FU

In 2003 attempt to have an ORCA-based monitoring (when COSINE did not exist..)

– Implementation with patched code (test beam “official” FU was pushing events into the ORCA-based FU)

– Failed due to different versions of the same library used by ORCA and XDAQ

Light-weight monitoring



2004 test beam

OO DB

Silicon modules

OO DB

FEDS PC (FEDS in PCI slots)

Builder Unit PC

COSINE Filter Unit PC

Root data file Online monitor

ORCA monitor

– Direct use of the COSINE-based FU in the test-beam setup has been proved to be possible (FED emulator tests), though minor modifications to the code were needed

– Due to lack of manpower the data formatter is still not ready



Open issues

Short term (being addressed):

• Final porting to ORCA/src/CommonDet of the remaining code common to all sub-detectors

• Functionality extensions, minor bug fixes

• Memory leak when running track reconstruction in the FU

• Allow persistency of NON-ZS silicon strip digis

• Prove smooth running of a COSINE-based FU within the silicon strip test beam DAQ through the FED emulator (need to complete the data formatter)

Longer term:

• Documentation

• Release of realistic CMS data formats



Monitoring

The following slides are mostly based on the presentation by E. Meschi at the Event Filter meeting during the May 2004 CPT week (http://agenda.cern.ch/askArchive.php?base=agenda&categ=a041651&id=a041651s6t1/transparencies)



Monitoring: Global vs. Local DAQ

Global DAQ:• First access to the full event data: in the FU• It should be possible to observe and analyze any given subset

of events accepted by the Level 1 • The FU runs an ORCA application where access to geometry,

calibrations, conditions is guaranteed

The FU is a natural candidate to run monitoring tasks Local DAQ:• Fundamental rate/throughput limitations• In principle lots of freedom as to:

– Choice of processing and analysis framework– Feedback scheme and access to central information repositories– Clients

Could profit from implementing the monitoring tasks in the software framework of the global DAQ



Options for Online Monitors

• Every FU is able to run certain monitor tasks: • Monitor "algorithms" need to be maintained on each FU, and handled like an

HLT algorithm• Need an API (not necessarily different from the DAQ monitoring) to publish

and update complex monitor objects to a central server

• Every FU is able to forward specific event types to dedicated processing nodes (either directly or through a concentrator):• One more interface in the FU• More network traffic, more complex topology (but need not be 100% reliable.

At worst consumers can be ranked by relevance)• What protocol for event forwarding ? See this in relationship to event storage

• Why might BOTH 1 and 2 needed ?• Some monitor process may need to see events at large rate

• An obvious example is HLT monitoring

• A lot of processing is already performed on the farm



Storage Manager and Consumers

FDN1

FCNSM

BU

FU Storage manager

"Event Consumer”

Server

To tier0 (events?/files?)

FCN

"Monitor Consumer"

RecoAbstract

Layer

StreamingCommunication

Server

SM

”Quasi-online Monitor”

Server

RegistryPublish

FU



Monitor Task Location

> (say) ~10 Hz(> 10 MB/s aggregate)

On FU

Requires > (say) 1% FU resources (CPU/mem)

yes yes

no

???

no

lowlatency

(0.1÷1 min)

Requires > (say) 1% FU resources (CPU/mem)

yes

no

yes

Dedicated processor off online

stream

no

On localmass

storage

On tier 0

Requires < ~hourslatency

yes

no



Monitoring tool in COSINE

• Prototype ROOT-based tool for monitoring already present in DaqPrototype + COSINE (based on the CDF monitoring model): COSINE_1_0_0_pre5/src/SubFarmManager/ COBRA/DaqPrototype/DaqMonitor

• Directly applicable to tasks where histograms are filled by the FUs

• A server application collects the histograms from several FUs and presents them

• A GUI allows histograms to be visualized, updated, printed….

• No documentation yet, but use is quite straightforward



Exercises

• Try to follow something like flowchart above (with numbers) for each task to:• Define rate and resource budget for online monitors

(similar to HLT)• Define local storage needs (in addition to normal

event buffering)• Examine current design of the FU physics

monitor tool in light of possible additional uses.• Can we propose a uniform framework for all use

cases (maybe even including local DAQ?)• In connection with DAQ monitoring (currently also in

the requirement collection phase, see http://cmsdoc.cern.ch/cms/TRIDAS/distribution/Meetings/DAQ.weekly/0DW2004/SRSMON_V1_0.pdf



BACKUP SLIDES



Global vs. Local DAQ

• Global DAQ Trigger uses GTP: GTP trigger distributed to FEDs via TTC FED / GTP readout via Slink into FRL FED builders assemble “superfragments” RU builders in Surface-Bld: full events assembled in BU from superfragments EventFilter farm in Surface-bld: complete event record available in FU

– Intended for physics data taking– Throughput: 200 MB/s per FED, or 12.5 kHz for 2kB fragments per RU

builder, or 200 events/s per BU– Can operate up to 8 partitions in parallel

• Local DAQ Trigger uses LTC: LTC inputs distributed to FEDs via TTC FED readout via VME into crate controller PC RU: crate controller PCs. Also BU / FU / monitor if no event building Event building over DSN (GbE) FU / monitor on network PCs

– Throughput: something like 20 MB/s per FED crate, few 100 Hz rates for 2kB fragments (less if events built)

– Operation completely independent of central DAQ and central Trigger (apart for central services such as boot, file servers, databases ..)



Examples

• Farm specific monitoring (HLT performance, rates etc., control plots like geographic distributions of objects etc. for rejected/accepted events): on FU• SM collects, summarizes and serves directly monitor objects

(histograms etc.) to consumers • Event display: use event forwarding (?) • Most calibration tasks which can’t run on tier 0 should

qualify to run off data buffered locally (provided they don’t require a kHz)• And even then, might be only parts of an event so actual bw

limited (how much in total ?) say want to allow another 100 MB/s in addition to data?

• These data have a short life cycle, they never make it as such to tier 0 and/or databases, only summary information persists

Documents

Detector (Tracker) monitoring: introduction to COSINE and the Filter Farm Giacomo BRUNO