Data Acquisition Backbone Core

J. Adamczewski-Musch, N. Kurz, S. Linev

GSI, Experiment Electronics, Data processing group

9.02.2011

DABC

S. Linev http://dabc.gsi.de 2

DAQ architectures

Detector

FEE buffer

Readoutbuffer

Switch

Processorfarm

Storage

Self-triggered Front-endall hits shipped to DAQ.Data push architecture

High-throughputEvent building

First event selectiondone in processor farm.

Readout buffer outside radiation area. Many Gbyte

storage easily possible. Allows L1 decision times of

10-100 ms

Fast links

Conventional DAQ

L1trigger

L1trigger

L1trigger

L1trigger

HLTHLTHLTHLTHLTHLTHLTL1

CBM DAQ

9.02.2011

DABC

S. Linev http://dabc.gsi.de 3

InfiniBand performance studies

0 1 2 3 4

Nominal QDR

Unidir. QDR

Bidir. QDR

Nominal DDR

Unidir. DDR

Bidir. DDR

Nominal SDR

Unidir. SDR

Bidir. SDR

GByte/s

2006*

2007**

2011***

Measurements: * GSI, 4-nodes SDR IB cluster

** Uni Mainz, 120-nodes DDR IB cluster

*** LOEWE-CSC, Höchst, 836-nodes QDR IB cluster (preliminary)

9.02.2011

DABC

S. Linev http://dabc.gsi.de 4

DAQ requirements

Flexible – adopt different kinds of soft- and hardware

Compact – use only necessary code

Scalable – from small detector tests to many-nodes clusters

Performing – low framework overhead

Multiprocessing & multithreading

9.02.2011

DABC

S. Linev http://dabc.gsi.de 5

Data Acquisition Backbone Core

Controls

core-testnet-testbnet-testmbsbnet-mbs

Slim(batch)

DIM

Java GUI

ApplicationsPlugins

socketIB verbsbnetmbsbnet-mbs

Core

Plugins: Implementation of applications (programmers)

Applications: Mainly setup or testing programs (users)

DABC

Download via http://dabc.gsi.de

9.02.2011

DABC

S. Linev http://dabc.gsi.de 6

DABC dataflow concept

UserModule

“Pool”

“Input”

“Output0”

“Output1”

File

UserModule

“Pool”

“Input”

“Output0”

“Output1”

UserModule

“Pool”

“Input”

“Output0”

“Output1”

File

DataServer

NetworkDevice

NetworkDevice

PCIeDevice

node1

node2

Transport

Transport

Transport

Transport

Transport

Transport

Transport

Transport:• manages buffers queue• runs in own thread• decouples user code from actual transport functionality

Device:• represents hardware items• manages several transports

9.02.2011

DABC

S. Linev http://dabc.gsi.de 7

User code - modules

dabc::Module

dabc::PoolHandle

dabc::Timer

dabc::Port

dabc::Port

dabc::Port

dabc::Parameter dabc::Module provides:• I/O ports for communications

• Pools handles to request memory

• Timers for timeouts processing

• Configuration & monitoring parameters

9.02.2011

DABC

S. Linev http://dabc.gsi.de 8

Many other features

Configuration with XML files

Controlling interface and GUI

Factories (plugins) architecture

Application and state machine

Event building network (BNET)

Runs on 32/64 bit Linux

9.02.2011

DABC

S. Linev http://dabc.gsi.de 9

CBM hardware prototypes

nXYTER, 128-channel self-triggered

readout chip with few ns time

resolution, DEFNI collaboration

CBM Readout controller (ROC),

Kirchhoff Institut für Physik, Heidelberg

9.02.2011

DABC

S. Linev http://dabc.gsi.de 10

DABC as access layer to ROC

file I/O online monitor

ROC-udpplugin

ROC-opticplugin

Use

r acc

ess

la

yer

DABCUsers

scripts, GUIs

optic

PCIe

Ethernet

9.02.2011

DABC

S. Linev http://dabc.gsi.de

ROC software components

DABC-based components roc-udp plugin roc-optic plugin combiner and sorter modules readout application

rocGui and rocutil for configuration

go4-based online analysis

https://subversion.gsi.de/cbm/ROC

9.02.2011

DABC

S. Linev http://dabc.gsi.de

CBM beam test in CERN, Nov.10

ROCEth

SYNC

A

B ROCEth

SYNC

A

B

ROCEth

SYNC

A

B

STS

stati

on

RIC

H

MBSEth

SYNCsender

SusiboUSB

SYNCSPA

DIC

SusiboUSB

SYNCSPA

DIC

SusiboUSB

SYNCSPA

DIC

SusiboUSB

SYNCSPA

DIC

SusiboUSB

SYNCSPA

DIC

SusiboUSB

SYNCSPA

DIC

TR

D1

TR

D2

TR

D3

TR

D4

Beam

monit

or

PC

DABC

Eth

USB

Control room

storagego4 online monitor

• Three different readouts:• ROC UDP-based readout• SPADIC/Susibo USB readout• MBS readout

• Combination of free-running and triggered electronics

Cave

9.02.2011

DABC

S. Linev http://dabc.gsi.de

node3

node2

node1

DABC application

SPADICcombiner

Input1

OutputInput0

Input2

Input3

Input4

Input5

Supercombiner

MBS

Output0ROC

SPADIC

Output1

ROCcombiner

Input1

OutputInput0

Input2go4 online

monitor

lmd filestorage

MBS

RO

Cs

SPA

DIC

s/Susi

bo

9.02.2011

DABC

S. Linev http://dabc.gsi.de

CBM beam test in COSY, Dec.10

ROCSFP

SYNC

A

BData

combinerboard

SFP

SFP

SFP

SFP

SFP

SFP

ROCSFP

SYNC

A

B

ROCSFP

SYNC

A

B ROCSFP

SYNC

A

B

ROCSFP

SYNC

A

B ROCSFP

SYNC

A

B

ROCSFP

SYNC

A

B ROCSFP

SYNC

A

B

Datacombiner

boardSFP

SFP

SFP

SFP

SFP

SFP

STS

stati

on 1

STS

stati

on 2

GEM

stati

on 1

GEM

stati

on 2

MBSEth

SYNCsender

Beam

monit

or

PC

DABC

AVNet

PCIeSFP

SFP

Eth

PC

DABC/ EPICS

AVNet

PCIeSFP

SFP

Eth

• data readout and control via optic• clock synchronization via optic• optional MBS readout• optional EPICS readout

Cave Control room20 m

9.02.2011

DABC

S. Linev http://dabc.gsi.de

DABC application

Supercombiner

MBS

Output0ROC

EPICS

Output1

go4 onlinemonitor

Lmd filestorage

MBS

RO

Cs

ROCcombiner

Input1

OutputInput0

Input2

Input3

Input4

Input5

Input6

Input7

DABC/EPICS

ROC datasplitter

Output1

Input Output0

Output2

Output3

Output4

Output5

Output6

Output7

opticreadout

9.02.2011

DABC

S. Linev http://dabc.gsi.de

Plans for version 2

More scalability and flexibility

Failure detection and recovery

Redesign of control interface

9.02.2011

DABC

S. Linev http://dabc.gsi.de 17

Conclusion

DABC is general-purpose DAQ framework

Can be used for different purposes – from small detectors tests to multi-nodes application

Because of its plugin architecture can be easy extended to specific needs

Open for improvements and new ideas