LHCb Computing Status Report Meeting with LHCC Referees October 19th, 1999 John Harvey CERN/ EP-ALC

LHCb Computing Status Report

Meeting with LHCC RefereesOctober 19th, 1999

John HarveyCERN/ EP-ALC

Report to LHCC Referees October 1999 Slide 2

Outline

Frontend / DAQ workshop DAQ - TFC, Readout Unit, Readout Network ECS - Fieldbus, OPC, SCADA Software - Migration to GAUDI Computing Model Studies Status and plans for next production of simulated events


Frontend DAQ Workshop An LHCb workshop was held 12-14 October at CERN to

discuss issues of Front-End Electronics and DAQ. 35 participants from all the various subsystems and institutes A great deal of discussion and a positive outcome :

establishing a common language between all participants underlining advantages of taking a uniform approach and adopting

uniform solutions whenever possible agreement on the need to review specifications before building

hardware (qualification) global decisions (choice of parameters) to be easily/widely visible areas where clarification of ideas are needed were identified

Agenda and slides :http://lhcb.cern.ch/electronics/meetings/FE-DAQ_workshop_october99/agenda.htm


Trigger/DAQ ArchitectureTrigger/DAQ Architecture

Read-out Network (RN)

RU RU

Control &

Monitoring

RU

4 GB/s

20 MB/s

Variable latencyL2 ~10 ms

L3 ~200 ms

LA

N

Read-out units (RU)

Timing&

FastControl

Front-End Electronics

VDET TRACK ECAL HCAL MUON RICH

LHCb Detector

L0

L1

Level 0Trigger

Level 1Trigger

40 MHz

1 MHz

40 kHz

Fixed latency 4.0 s

Variable latency <2 ms

Datarates

40 TB/s

1 TB/s

Front-End Multiplexers (FEM)1 MHz

Front End Links

Trigger Level 2 & 3Event Filter

SFC SFC

CPU

CPU

CPU

CPU

Sub-Farm Controllers (SFC)

Storage

Th

rott

le


Timing and Fast Control Responsibilities

transmission of LHC clock, L0 and L1 triggers to front-end electronics monitor state of buffers at all stages of readout - throttle trigger to avoid overflows react to control commands such as ‘resets” support partitioning

The Readout Supervisor (RS) is the heart of the TFC system. It must model state of frontend electronics and throttle trigger according to rules It must react to control commands, DAQ throttle, resets etc. It must encode channel A (L0) and channel B (L1, resets) for TTC

Effort has been started to make a detailed specification of RS functions next step is to define use cases to be able to specify format of channel B types, frequency and timing of resets, commands to pulse detectors etc.

Warsaw group starting to work on prototype for RS switch Effort will shortly be put on making a detailed specification of RS itself

gL0gL1

ReadoutSupervisor

L0 L1

gL0gL1

ReadoutSupervisor

Localtrig.

gL0gL1

LHCClock

Clk Fanout

SD1 TTCtx SD2 TTCtx SDn TTCtx

Readout SupervisorSwitch

L1 TTCtx

ReadoutSupervisor

L-0

Localtrig.

L0 L1

L0 L1

L0 TTCtx

L-1

LHCTurn Signal

LHCTurn Signal

LHCTurn Signal

TFC : Signal Generation and Treatment

RD12


Trigger/DAQ Architecture


RU RU

Control &

Monitoring

RU

4 GB/s

20 MB/s


L3 ~200 ms

LA

N

Read-out units (RU)

Timing&

FastControl



LHCb Detector

L0

L1

Level 0Trigger

Level 1Trigger

40 MHz

1 MHz

40 kHz

Fixed latency 4.0 s


Datarates

40 TB/s

1 TB/s


Front End Links


SFC SFC

CPU

CPU

CPU

CPU


Storage

Th

rott

le


Front-End Multiplexing/Readout Unit

There are ~2000 sources of data from L1 electronics and these have to be combined onto ~200 links and fed to the Readout Units (RU)

There is agreement to aim to have a common module for this based on RU The RU project is well underway. A number of issues were identified and

some resolved single fragment per event (simple protocol, performance) maximum size of fragments to be determined - issue is cost of fast memory adopted a single data frame format for transport through various readout stages data payload to be self describing project starting to investigate use of fieldbus for remote control/monitoring

A first prototype of RU should be ready by the beginning of 2000 Auxiliary equipment and software to allow testing is also being built


Trigger/DAQ ArchitectureTrigger/DAQ Architecture


RU RU

Control &

Monitoring

RU

4 GB/s

20 MB/s


L3 ~200 ms

LA

N

Read-out units (RU)

Timing&

FastControl



LHCb Detector

L0

L1

Level 0Trigger

Level 1Trigger

40 MHz

1 MHz

40 kHz

Fixed latency 4.0 s


Datarates

40 TB/s

1 TB/s


Front End Links


SFC SFC

CPU

CPU

CPU

CPU


Storage

Th

rott

le


Event Building

After discussion with representatives from industry we are convinced that a switching network supporting 4 GB/s sustained rate will not be a problem to acquire at affordable cost on the timescale of the LHCb DAQ.

Several technologies are in principle available Gb Ethernet or 10Gb Ethernet Myrinet latest estimate of cost per port for 1 Gbps technology is now 500$ - 1000$

In this light we see no reason to deviate from our ‘full readout protocol’ as described in the Technical Proposal.

We are currently concentrating on studying ‘intelligent Network Interfaces’ that would allow to perform the assembly of the incoming fragments locally and only send complete events to the host CPU. A project based on an ATM card from IBM has been started beginning of October


Experiment Control System


RU RU

Control &

Monitoring

RU

4 GB/s

20 MB/s


L3 ~200 ms

LA

N

Read-out units (RU)

Timing&

FastControl



LHC-B Detector

L0

L1

Level 0Trigger

Level 1Trigger

40 MHz

1 MHz

40 kHz

Fixed latency 4.0 s


Datarates

40 TB/s

1 TB/s


Front End Links


SFC SFC

CPU

CPU

CPU

CPU


Storage

Th

rott

le


Control/Monitoring structure

Experimental equipment

. . .

LAN

WAN

Storage

Oth

er

syst

em

s(L

HC

, S

afe

ty,

...)

Configuration DB, Archives,Logfiles, etc.

Controller/PLC VME

FieldBus

LAN

Supervision

ProcessManagement

FieldManagement

Sensors/devices

Field buses

PLC/ PC

OPC

CommunicationProtocols

SCADA

Technologies

SCADA = supervisory control and data acquisitionOPC = OLE for Process ControlPLC = Programmable Logic ControllerField buses = CAN, ProfiBus, WorldFip, ...


LHCb Priorities Follow standards at all levels

LHCb, CERN, HEP, Industry

First priority is to adopt guidelines for choice of fieldbus and integration with hardware Milestone : end of this year

Use PLCs for specialised domains (gas, magnet…), where safety is issue

Use PCs for other domains cheaper, programming flexibility

Adopt OPC JCOP recommendation (Sept ‘99)

Adopt SCADA JCOP tender under discussion timescale target is ~ end 2000

Hardware resources

Field Bus controller

Field bus

I/O Interfaces

Software Interface

Device behavior

GUI/MMI

Network comm.

Sub-system supervision

High level services

Sensors/devices

Field buses

PLC / PC

OPC

CommunicationProtocols

SCADA

TechnologiesLayers

Process control Pri

ori

ty

high

low


Fieldbus

Collect requirements in terms of #boards, #bytes/board, frequency and speed types of boards and interface to electronics technical constraints (radiation environment, power consumption, space) issue questionnaire (next week)

Match requirements to fieldbus capabilities and produce LHCb guidelines Concrete projects to get hands-on experience : fieldbus controller for RU

Profibus CANbus Worldfip Ethernet Firewire

SpeedMbps

12 1 2.5/5 100 400

MessagesizeBytes

256 8 128

#nodes 127 127 256

Distancemetres

100 40 1000


Supervisory Software (SCADA) LHCb requirements :

The number of different device types is of the order of a 100 The number of devices is of the order of 17’000 The number of parameters is of the order of n•107

The number of monitored quantities is of the order of n•105

Implications for choice of SCADA system : A tag-oriented system is unrealistic if each parameter is an entry. We need a namespace hierarchy (Device->Parameters). For highly repetitive items (e.g. individual detector channels in an electronics board)

arrays are needed (don’t want to name each of them individually).

Short term SCADA - Bridgeview Longer term - adopt scalable device-oriented commercial product (JCOP tender) Invest in OPC to ease transition


Migration to GAUDI

Final objective: Produce a complete set of fully functional data processing applications

using exclusively OO technology

Sub-objectives: Provide a fully functional framework (GAUDI) Assemble new and old algorithms into a single and complete suite of data

processing applications. Be able to run productions. Convert all the existing FORTRAN code to C++


Possible strategiesFortran

C++

SICb

Gaudi?1

SICb

Gaudi2

Fast translation of Fortran into C++

SICb

Gaudi3

Wrapping Fortran

Framework development phase

Transition phase

Hybrid phase

Consolidation phase


Framework development phase

At the end of this phase the GAUDI framework should be functionally complete Data access services Generic event model Generic detector description model Data visualization Basic set of services

Develop some physics algorithms to prove architecture concept We started this phase one year ago We expect to be completed by middle of November 1999 with the

release of v3 of GAUDI


Transition phase

At the end of this phase we should be able to reconstruct and analyze simulated data within the GAUDI framework. The Monte Carlo data production will still be done using SICb.

Incorporate reconstruction and analysis parts of SICb in the GAUDI framework - wrap FORTRAN code Analyse SICb to identify all modules, their inputs and outputs Develop a complete OO event data model Write converters to allow access to data in both formats

Development of new algorithms can proceed within GAUDI Caveats

lot of work to make converters in both directions we could discover technical difficulties (size, commons, initialization,…)


GeneratorDet. Simul.

MCHits

DST

HistoNtuples

Recon-struction +

Analysis

Transition phase -2

SICbGEANT 3

FrameworkGAUDI

Framework


Transition Phase - 3

GeneratorDet. Simul.

(Sicb)MCHits

FA

Bank 1 ZEBRA

GAUDI store

FB FC

C++A C++B

Cnv

Commons

Bank 2 Bank4

Obj 2 Obj 3 Obj 4

Cnv

DST

HistoNtuples

Reconstruction + Analysis


Hybrid Phase

One single program with FORTRAN and C++ cooperating to produce physics results.

Replace wrapped FORTRAN code incrementally. At the end of this phase we should be able to retire the FORTRAN

compiler and libraries Already known problems:

Two different detector descriptions. Difficult to maintain. Output file format. Which one to choose? Different set of input “card files”. Large memory needs for data and code.

The hybrid phase should be as short as possible to minimize pain.


Consolidation phase

Complete the new detector description Re-iterate with the O-O Event Model Re-engineer some of algorithms Incorporate Geant4 with GAUDI to make framework for new

simulation program …..


Planning

Qtr 1Qtr 3 Qtr 2 Qtr 3 Qtr 41998 1999

Architecture Design

Gaudi Development v1



Qtr 1 Qtr 2 Qtr 3 Qtr 42000

Qtr 4

Framework Functional

Analysis Sicb

Transition phase

Production program

Hybrid phase


Computing Model Studies We are revising our computing requirements

event sizes and storage requirements triggering cpu requirements (optimisation strategies) availability of calibration/alignment constants simulation requirements

We are looking at 'use cases’for the analysis of selected physics channels (+-, J/ Data retained at each stage Processing required at each stage Frequency of processing cycles

Generate a baseline model for distribution and role of experiment-specific, CERN and regional computing facilities

Timescales fixed by impending Hoffmann Review


SICb Status

Generators: Pythia 6.125 + QQ (CLEO) for decays Full GEANT simulation Realistic detector desciption:

WARM magnet design realistic beampipe model calorimeters,muons and RICHes follow current optimisation

Reconstruction: no pattern recognition in tracking full pattern recognition in RICHes


PLANs up to end 1999

Trigger optimisation CERN (PCSF) 200k mbias - 4 different settings(luminosity, generator) 20k b -> mX 20k b -> eX 20k B0

d -> +-

20k B0d -> J/()K0

s

20k B0d -> J/(ee) K0

s

Background studies with GCALOR at Lyon 50k mbias

Physics production 500k b inclusive at Rutherford lab (PCSF) 100k other physics channels at CERN (PCSF)


Training Formal training through CERN Technical Training services

Hands-on Object-Oriented Analysis, Design and Programming with C++ C++ for Particle Physicists (Kunz) at least 40 people followed courses

Follow-up training books - Design Patterns (Gamma), Software Design (Lakos),..

Learning on the job - we already do following : design discussions where developers present their designs for review tutorials on new GAUDI releases during LHCb software weeks

Learning on the job - ideas for future include : code walkthroughs document patterns (those well-tried and successful and those that failed)

Consultancy and mentoring have GAUDI team spend 50% of their time helping algorithm builders

Documents

LHCb Computing Status Report Meeting with LHCC Referees October 19th, 1999 John Harvey CERN/ EP-ALC