1AN

DREA

NEGRI

, INFN

PAVIA

– NUC

LEAR S

CIENCE

SYMPO

SIUM –

ROME

20th O

ctober

2004

011101000111011110110010101111011000110111111110111000111010100100111011011100100010110111110100010101111100

111101001111010110000101

H

t

W

Z0

Andrea Negri, INFN Pavia Andrea Negri, INFN Pavia

on behalf of the ATLAS HLT Groupon behalf of the ATLAS HLT Group

2AN

DREA

NEGRI

, INFN

PAVIA

– NUC

LEAR S

CIENCE

SYMPO

SIUM –

ROME

20th O

ctober

2004

Level 1 triggerHardware basedCoarse granularity calo/muon data

Event FilterFull event access“Seeded” by LVL2 resultAlgorithms inherited from offline

Level 2 triggerDetector sub-region processedFull granularity for all subdetectorsFast rejection steering

40 MHz

~75 kHz

~2 s

~2 kHz

~10 ms

~ 1 s

~200 Hz

Muon

ROD ROD ROD

LVL1

Calo Inner

PipelineMemories

ReadoutDrivers

RatesLatency

RoI

ATLAS T/DAQ systemATLAS T/DAQ system

LVL2

Event builder network

Storage: ~ 300 MB/s

ROBROB ROBROB ROBROBReadoutBuffers~1600

EF farm~1000 CPUs

1 selected eventevery millionTDAQ( )

EF

FUNCTIONAL TESTSFUNCTIONAL TESTS DEPLOYMENTDEPLOYMENT CONCLUSIONSCONCLUSIONSINTRODUCTIONINTRODUCTION DESIGNDESIGN

CM energy 14 TeV Luminosity 1034 cm-2s-1

Collision rate 40 MHz Event rate ~ 1 GHz Detector channels ~ 108

3AN

DREA

NEGRI

, INFN

PAVIA

– NUC

LEAR S

CIENCE

SYMPO

SIUM –

ROME

20th O

ctober

2004

A common framework for offline and onlineand similar reconstruction algorithms

Avoids duplication of work Simplify performance/validation studiesAvoid selection biasesCommon database access tools

General requirements

Scalability, flexibility and modularity

Hardware independence in order to follow technology trends

Reliability and fault tolerance

Avoid data losses

Could be critical: EF algorithms

inherited from the offline ones

EF

SFI

SFO

SFI

SFO

SFI

SFO

SFI

SFO

SubFarmInput

SubFarmOutput

EFSubFarm

Event Filter system: Constraints and RequirementsEvent Filter system: Constraints and Requirements

The computing instrument of the EF is organized

as a set of independent subFarms, connected to

different output ports of the EB switch

Possibility to partition the EF resources and run multiple concurrent DAQs instances (e.g.: calibration and commissioning purposes)

Event builder network

Storage

Read out system

FUNCTIONAL TESTSFUNCTIONAL TESTS DEPLOYMENTDEPLOYMENT CONCLUSIONSCONCLUSIONSINTRODUCTIONINTRODUCTION DESIGNDESIGN

4AN

DREA

NEGRI

, INFN

PAVIA

– NUC

LEAR S

CIENCE

SYMPO

SIUM –

ROME

20th O

ctober

2004

data processing data flow functionalities

Design featuresDesign features

Read out systemEach processing node manages its own connection with the

SFI and SFO elements that implement the server part of

the communication protocol

Allows dynamic insertion/removal of sub-farms in the EF or of processing hosts in a sub-farm

Allows geographically distributed implementations

Supports multiple SFI connections:

dynamic re-routing in case of SFI malfunction (depends on the network topology)

Avoids single point of failure: a faulty processing host do not interfere with the

operations of other sub-farm elements

In order to assure data security in case of event processing problems the design has been based on the decoupling between:

SFI

SFO

SFI

SFO

SFI

SFO

SFI

SFO

Event builder network

StorageRemote

Farm

FUNCTIONAL TESTSFUNCTIONAL TESTS DEPLOYMENTDEPLOYMENT CONCLUSIONSCONCLUSIONSINTRODUCTIONINTRODUCTION DESIGNDESIGN

5AN

DREA

NEGRI

, INFN

PAVIA

– NUC

LEAR S

CIENCE

SYMPO

SIUM –

ROME

20th O

ctober

2004

In each EF processing host

Data flow functionalities are provided by

the Event Filter Dataflow process that:

Manages the communication with SFI and SFO

Stores the events during their transit in the Event Filter

Makes the events available to

the Processing Tasks that perform the

data processing and event selection operations

running the EF algorithms in the standard

ATLAS offline framework

A pluggable interface (PTIO) allows PTs to access

the dataFlow part via a unix domain socket ( )

DataFlow DataFlow DataProcessing decoupling DataProcessing decoupling

Node n

DataFlow

DataProcessing

EFD

SFO

SFI

AcceptedEvents

IncomingEvents

PT #1

PT #n

PTIOPTIO

FUNCTIONAL TESTSFUNCTIONAL TESTS DEPLOYMENTDEPLOYMENT CONCLUSIONSCONCLUSIONSINTRODUCTIONINTRODUCTION DESIGNDESIGN

6AN

DREA

NEGRI

, INFN

PAVIA

– NUC

LEAR S

CIENCE

SYMPO

SIUM –

ROME

20th O

ctober

2004

When an event enters the processing node it is stored in a shared

memory (sharedHeap) used to provide events to the PTs

A PT, using the PTIO interface (socket)

Requests an event

Obtains a pointer to sharedHeap portion that

contain the event to be processed

(The PTIO maps this portion in memory)

Processes the event

Communicates back to the EFD the filtering decisions

PT cannot corrupt the events because the map is read only

Only the EFD manages the sharedHeap

If the PT crashes the event is still owned by the EFD,

that may assign the event to another PT or force accept it

Fault Tolerance: the sharedHeap (1)Fault Tolerance: the sharedHeap (1)

Node n

EFD

SFO

PT #1

PTIO

PT #n

PTIO

SFI

100111010100010010010001000101000111101000100101001000100101000100001000100101010111100000101110011001001001001010011010101000100010001000100100010010100010000100010010101011110000010111001100100100100101001101010100010001000101010101010100010111101001101001110001

01000111

Evy

SharedHeap

Evx

Evz

RO map

FUNCTIONAL TESTSFUNCTIONAL TESTS DEPLOYMENTDEPLOYMENT CONCLUSIONSCONCLUSIONSINTRODUCTIONINTRODUCTION DESIGNDESIGN

7AN

DREA

NEGRI

, INFN

PAVIA

– NUC

LEAR S

CIENCE

SYMPO

SIUM –

ROME

20th O

ctober

2004

To provide fault tolerance also in case of EFD crash the

sharedHeap is implemented as a memory mapped file

The OS itself manages directly the actual write

operations avoiding useless disk I/O over-heading

The raw events can be recovered reloading

the sharedHeap file at EFD restart

The system could be out of sync only in case of

power cut, OS crash or disk failure

these occurrences are completely decoupled from

the event types and topology and therefore do not

entail physics biases on the recorded data

Fault tolerance: the sharedHeap (2)Fault tolerance: the sharedHeap (2)

Node n

EFD

SFO

PT #1

PTIO

PT #n

PTIO

SFI

100111010100010010010001000101000111101000100101001000100101000100001000100101010111100000101110011001001001001010011010101000100010001000100100010010100010000100010010101011110000010111001100100100100101001101010100010001000101010101010100010111101001101001110001

01000111

Evy

SharedHeap

Evx

Evz

FUNCTIONAL TESTSFUNCTIONAL TESTS DEPLOYMENTDEPLOYMENT CONCLUSIONSCONCLUSIONSINTRODUCTIONINTRODUCTION DESIGNDESIGN

8AN

DREA

NEGRI

, INFN

PAVIA

– NUC

LEAR S

CIENCE

SYMPO

SIUM –

ROME

20th O

ctober

2004

Node n

EFD

SFO

PT#1

PTIO

PT#2

PTIO

SFI

Input

Monitoring

Sorting

ExtPTs ExtPTs

Output Output Output

Trash

SFI

Input

PT#3

PTIO

PT#a

PTIO

PT#b

PTIO

SFOSFO

Calibration data

Debuggingchannel

Main outputstream

Cal

ibra

tion

Implementation

Implementationexample

example

The EFD function is divided into

different specific tasks that could be

dynamically interconnected to form a

configurable EF dataflow network

The internal dataflow is based on

reference passing

Only the pointer to the event (stored in the

sharedHeap) flows among the different tasks

Tasks that implement interfaces to external

components are executed by

independent threads (Multi Thread design)

In order to absorb communication latencies

and enhance performance

Flexibility and ModularityFlexibility and ModularityFUNCTIONAL TESTSFUNCTIONAL TESTS DEPLOYMENTDEPLOYMENT CONCLUSIONSCONCLUSIONSINTRODUCTIONINTRODUCTION DESIGNDESIGN

9AN

DREA

NEGRI

, INFN

PAVIA

– NUC

LEAR S

CIENCE

SYMPO

SIUM –

ROME

20th O

ctober

2004

Throughput

0

20

40

60

80

0,1 1 10 100 1000 10000Event Size (kB)

MB/

sec

Verified the robustness of the architecture

Week long runs (>109 events) without crashes or event losses (even randomly killing PTs)

EFD PT communication mechanism scales with the number of running PTs

SFIEFDSFO communication protocol

Exploit gigabit links for realistic event sizes

Rate limitations for small event sizes (or remote farm implementations)

EFD asks for a new event only after the previous one has been receivedRate limited by the round trip timeImprovements under evaluation

Scalability tests carried out on 230 nodes

Up to: 21 subFarms, 230 EFDs, 16000 PTs

0102030405060708090

100

1 10 100Number of PTs

Rat

e (H

z)

40003600320028002400

Real PT

Dummy PT

Memory limit

Quad xeon 2.5GHz, 4GB

Functional TestsFunctional TestsFUNCTIONAL TESTSFUNCTIONAL TESTS DEPLOYMENTDEPLOYMENT CONCLUSIONSCONCLUSIONSINTRODUCTIONINTRODUCTION DESIGNDESIGN

10AN

DREA

NEGRI

, INFN

PAVIA

– NUC

LEAR S

CIENCE

SYMPO

SIUM –

ROME

20th O

ctober

2004 ATLAS Combined Test BeamATLAS Combined Test Beam

TRT LAr

Tilecal

MDT-RPC BOS

TRT LAr

Tilecal

MDT-RPC BOS

FUNCTIONAL TESTSFUNCTIONAL TESTS DEPLOYMENTDEPLOYMENT CONCLUSIONSCONCLUSIONSINTRODUCTIONINTRODUCTION DESIGNDESIGN

11AN

DREA

NEGRI

, INFN

PAVIA

– NUC

LEAR S

CIENCE

SYMPO

SIUM –

ROME

20th O

ctober

2004

pROS

LocalLVL2 farm

Contains the LVL2 result that steers/seeds the EF processing

Trac

ker

Cal

oM

uon

monitoringrun control

101010100010001001001000100010110

ROS

LVL1calo

101010100010001001001000100010110

ROS

LVL1mu

101010100010001001001000100010110

ROS

RPC

101010100010001001001000100010110

ROS

TGC

101010100010001001001000100010110

ROS

CSC

101010100010001001001000100010110

ROS

MDT

101010100010001001001000100010110

ROS

Tile

101010100010001001001000100010110

ROS

LAr

101010100010001001001000100010110

ROS

TRT

101010100010001001001000100010110

ROS

SCT

101010100010001001001000100010110

ROS

Pixel

EventBuilder

DFM

SFI

data

net

wor

k (G

bE)

EF farm @ Meyrin(few Km)

gateway

Remote Farms:PolandCanada

DenmarkInfrastructure

tests only

Test Beam LayoutTest Beam Layout

Local EF farm SFO

FUNCTIONAL TESTSFUNCTIONAL TESTS DEPLOYMENTDEPLOYMENT CONCLUSIONSCONCLUSIONSINTRODUCTIONINTRODUCTION DESIGNDESIGN

12AN

DREA

NEGRI

, INFN

PAVIA

– NUC

LEAR S

CIENCE

SYMPO

SIUM –

ROME

20th O

ctober

2004

Online event monitoring

Online histograms obtained merging data published by different PTs and gathered by a TDAQ monitoring process (the Gatherer)

Online event reconstruction

E.g.: Track fitting

Online event selection

Beam composed of , , e

Track reconstruction in muon chamber allowed the selection of events

Events labelled according to the selection and/or sent to different output streams

Validation of the HLT muon slice (work in progress)

Transfer LVL2 result to EF (via pROS) and decoding

Steering and seeding of the EF algorithm

Test Beam Online Event ProcessingTest Beam Online Event ProcessingFUNCTIONAL TESTSFUNCTIONAL TESTS DEPLOYMENTDEPLOYMENT CONCLUSIONSCONCLUSIONSINTRODUCTIONINTRODUCTION DESIGNDESIGN

Presenter Main Window

13AN

DREA

NEGRI

, INFN

PAVIA

– NUC

LEAR S

CIENCE

SYMPO

SIUM –

ROME

20th O

ctober

2004

Online event monitoring

Online histograms obtained merging data published by different PTs and gathered by a TDAQ monitoring process (the Gatherer)

Online event reconstruction

E.g.: Track fitting

Online event selection

Beam composed of , , e

Track reconstruction in muon chamber allowed the selection of events

Events labelled according to the selection and/or sent to different output streams

Validation of the HLT muon slice (work in progress)

Transfer LVL2 result to EF (via pROS) and decoding

Steering and seeding of the EF algorithm

Online Event ProcessingOnline Event ProcessingFUNCTIONAL TESTSFUNCTIONAL TESTS DEPLOYMENTDEPLOYMENT CONCLUSIONSCONCLUSIONSINTRODUCTIONINTRODUCTION DESIGNDESIGN

= 61 m

mm

Residuals of segments fit

in muon chambers

14AN

DREA

NEGRI

, INFN

PAVIA

– NUC

LEAR S

CIENCE

SYMPO

SIUM –

ROME

20th O

ctober

2004

Online event monitoring

Online histograms obtained merging data published by different PTs and gathered by a TDAQ monitoring process (the Gatherer)

Online event reconstruction

E.g.: Track fitting

Online event selection

Beam composed of , , e

Track reconstruction in muon chamber allowed the selection of events

Events labelled according to the selection and/or sent to different output streams

Validation of the HLT muon slice (work in progress)

Transfer LVL2 result to EF (via pROS) and decoding

Steering and seeding of the EF algorithm

Online Event ProcessingOnline Event ProcessingFUNCTIONAL TESTSFUNCTIONAL TESTS DEPLOYMENTDEPLOYMENT CONCLUSIONSCONCLUSIONSINTRODUCTIONINTRODUCTION DESIGNDESIGN

Energy deposition in calo cells

Hits

in m

uon

cham

ber

15AN

DREA

NEGRI

, INFN

PAVIA

– NUC

LEAR S

CIENCE

SYMPO

SIUM –

ROME

20th O

ctober

2004

Online event monitoring

Online histograms obtained merging data published by different PTs and gathered by a TDAQ monitoring process (the Gatherer)

Online event reconstruction

E.g.: Track fitting

Online event selection

Beam composed of , , e

Track reconstruction in muon chamber allowed the selection of events

Events labelled according to the selection and/or sent to different output streams

Validation of the HLT muon slice (work in progress)

Transfer LVL2 result to EF (via pROS) and decoding

Steering and seeding of the EF algorithm

Online Event ProcessingOnline Event Processing

pROS

LocalLVL2 farm

ROS

ROS

ROS

ROS

ROS

DFM

SFI

data

net

wor

k

Local EF farm

FUNCTIONAL TESTSFUNCTIONAL TESTS DEPLOYMENTDEPLOYMENT CONCLUSIONSCONCLUSIONSINTRODUCTIONINTRODUCTION DESIGNDESIGN

16AN

DREA

NEGRI

, INFN

PAVIA

– NUC

LEAR S

CIENCE

SYMPO

SIUM –

ROME

20th O

ctober

2004

Design: EF designed to cope with the challenging on-line requirements

Scalable design in order to allow dynamic hot-plug of processing resources, to

follow technology trend and to allow geographically distributed implementations

High level of data security and fault tolerance via decoupling between data

processing and data flow functionalities and the use of memory mapped file

Modularity and flexibility in order to allow different EF data-flows

Functional tests: design validated on different test beds

Proven design robustness, design scalability and data security mechanisms

No design limitations observed

Deployment on test beam setup

Online event processing, reconstruction and selection

Online validation of the HLT muon full slice

ConclusionsConclusionsFUNCTIONAL TESTSFUNCTIONAL TESTS DEPLOYMENTDEPLOYMENT CONCLUSIONSCONCLUSIONSINTRODUCTIONINTRODUCTION DESIGNDESIGN