SBS VME DAQhallaweb.jlab.org/12GeV/SuperBigBite/meetings/col... · (MC studies in progress) Calorimeter Rates HCAL ECAL Electron rate estimate w/2.5 GeV ... TI Trigger Interface SD

SBS VME DAQ

SBS collaboration Meeting

Alexandre Camsonne

July 7th 2014

Outline

• SBS DAQ overview

• GEM readout

• Fastbus readout

• HCAL readout

• Plan

• Manpower

• Conclusion

7/7/2014 2 SuperBigbite DAQ and Electronics

Alexandre Camsonne

SBS DAQ Overview

• Calorimeter – ECAL : Fastbus – HCAL

• SBS GEM

– APV25 INFN MPD

• BigBite – Scintillator – Shower preshower

• Coordinate detector


Alexandre Camsonne

SuperBigbite Spectrometer Focal Plane Polarimeter setup


Alexandre Camsonne

Background rate vs. cut on deposited energy (MC studies in progress)

Calorimeter Rates

HCAL

ECAL

Electron rate estimate w/2.5 GeV threshold (73% of Eelas): ≈ 200 kHz

(CDR section 5.1.7) Most demanding

Hadron rate estimate using SLAC & DESY data, Wiser code: w/4.5 GeV threshold: ≈ 1.5 MHz

≈ 9 kHz coincidence rate w/ 30 ns window

NB: Good resolution ≈ 16%/E


Alexandre Camsonne

From Hall A Real Compton Scattering experiment

DAQ concept


Alexandre Camsonne

• Hybrid Fastbus and pipelined electronics • Level 1

– ≈100 ns latency by analog summing and discrimination – Generated by electron arm (≈200 kHz rate) – Gate for Fastbus & non-pipelined VME for BigCal

• Level 2 : coincidence proton in HCAL and electron in ECAL – Assume up to ≈1.8 μs latency ( L2 800 ns max + Fast Clear 1 μs) – 9 KHz with 30 ns coincidence windows – FPGA-based coincidence logic using geometrical constraints reduction

by factor 5 ≈ 2 kHz physics DAQ rate

– Fast Clear FB & VME after L2 timeout ≈ 13% Electronics Dead Time

e’-p Kinematic Correlation 11 x 22 HCAL blocks 20 x 76 ECAL blocks

(CDR section F.3)

7/7/2014 7

SuperBigbite DAQ and Electronics Alexandre Camsonne

Using geometric correlations from elastic kinematic one can reduce final rate by a factor of 5 and

tracker data by at least a factor of 3

Data flow


Alexandre Camsonne

GEM

AP

V2

5

VM

E d

igitizer

BigCal Bigbite

HCAL

VM

E C

PU

Summing

Fastbus

FADC

Digital Sum

Logic

VME CPU

VME CPU

Event builder

Event recorder

Data file

Silo

L2 trigger

TS L1 trigger

GigE

SAS RAID

Compression +

10 GigE

CODA

GigE

100 Mb/s

L1 trigger


Alexandre Camsonne

GEM

AP

V2

5

VM

E d

igitizer

BigCal Bigbite

HCAL

VM

E C

PU

Summing

Fastbus

FADC

Digital Sum

Logic

VME CPU

VME CPU

Event builder

Event recorder

Data file

Silo

L2 trigger

TS L1 trigger

GigE

SAS RAID

Compression +

10 GigE

CODA

GigE

100 Mb/s

L1 trigger

GEM

AP

V2

5

FEC SRS SRU

GigE 10 GigE

PC

10 GigE

GEM readout • INFN MPD

– VME64X board – New version with fast VME protocol – Working with CAEN controller – Adapting software package to Intel VME CPU to use with

CODA

• SRS readout – Using Mississippi State SRS – Running with DATE, starting to look integration into CODA – Standard UDP based protocol

• Request to use GEM for PREX and Tritium in addition to A1n

Will have baseline performance number before November 7/7/2014 10


Front Tracker layout

7/7/2014 SuperBigbite DAQ and Electronics

Alexandre Camsonne 11

20

48

ch

ann

el

1 M

PD

2048 channel 1 MPD

Trackers layout



1 MPD 2048

1 M

PD

2

04

8

Front tracker Region of

Interest from BigCal position

Back tracker Region of

Interest from HCal position

Middle tracker Interpolated from both front

and back information

Trackers layout



Worst case

configuration

Suppression schemes

• Several algorithm can be implemented in FPGA for further data reduction

– Thresholds

– Timing

– Fitting c2

– slope



FPP Tracker layout



512

2560

20

48

Tracker event size with 3 samples readout



Detector Rate Channels Occupancy Time

window Hits

Geometrical

factor Data size

Effective data size

Data rate 3KHz Mb/s

Front Tracker 400 49000 13.5% 75ns 6615 3 52Kb 14.3Kb 43

Second Tracker

130 13600 7.4% 50ns 1010 5 8Kb 1.6Kb 5

Third Tracker 64 13600 3.6% 50ns 490 5 4Kb 0.8Kb 2

Electron arm GEM

173 12000 2.4% 50ns 288 1 2.3Kb 2.3Kb 6

Calorimeters 125 0.5Kb 0.5Kb 1.5

Total 67.8Kb 19.5Kb 58.5 Mb/s

Data rates

Detector Rate Channels Occupancy Hits

Data size Bytes

Data rate

MB/s 5 KHz

Geometrical factor

Data size no sup

Data rate

MB/s

Front Tracker

400.00 49000.00 0.50 6615.00 161929 809.65 3.00 269.88 134.94

Second Tracker

130.00 61440.00 0.50 1010.00 203040. 1015.20 5.00 203.04 101.52

Third Tracker 64.00 61440.00 0.50 490.00 203040 1015.20 5.00 203.04 101.52

Total 171880.00 8115.00 568009 2840.05 675.96 337.98


Alexandre Camsonne

Fastbus readout

• Time : 1877S

• Amplitude 1881M or MQT+1877S

• Fastbus max transfer speed : 40 MB/s can use either Intel or Old vxworks VME CPU

• Test Lab : 4 sets of 3 crates, will be able to test performance


Alexandre Camsonne

HCAL readout

• 288 channels

• 2 VXS crates , 18 FADCs

• 1.5 MHz singles

• 16 block clusters

• FADC 250 MHz 12 bit = 2 bytes

• 10 samples : 320 bytes

• Need HCAL occupancy

7/7/2014


19

Tape price

Experiment Days Rate

(MB/s) seconds Total

TB Double DLO5 DLO6

GEp 45 250 3888000 972 1944 97200 58320

GEn 50 250 4320000 1080 2160 108000 64800

GMn 25 250 2160000 540 1080 54000 32400

SIDIS 64 250 5529600 1382.4 2764.8 138240 82944

Total 397 K$ 239K$


Alexandre Camsonne

Man power

• A. Camsonne : General , MPD readout

• S. Abraham , M. Jones : Fastbus

• Students

– Jessica Campbell (SMU) : Fastbus

– Jessie Twigger (FIT SULI) ,?: SRS CODA readout

• HCAL trigger readout : ?


Alexandre Camsonne

Plan summary

• Setup on 3 Fastbus crate setup and test performance

• APV25 performance with SRS and MPD and integration with CODA

• Getting started with FADC for HCAL

• Start write-up about DAQ


Alexandre Camsonne

Conclusion

• Need good ways to reduce data

• Fastbus setup almost ready for testing

• Need to start thinking about SIDIS

• Start writeup


Alexandre Camsonne

Backup

VETROC JLAB Electronics group


Alexandre Camsonne

CP

U

SSP

SS

P

SSP

SS

P

SSP

SS

P

GTP

SD

SS

P

SSP

SS

P

SSP

SS

P

SSP

TI

VXS Crate

SSP

C

PU

FA

DC

FA

DC

FA

DC

FA

DC

FA

DC

FA

DC

C

TP

SD

FAD

C

FAD

C

FAD

C

FAD

C

FAD

C

FAD

C

TI

VXS Crate C

PU

TD

TD

SD

TS

VXS Crate

TD

TD

TD

TD

TD

TD

TD

TD

TD

TD

SD

L1 Trigger Diagram

VXS Serial Link • 16 bit @ 250 MHz: 4 Gbps

FADC250 • 12 bit @ 250 MHz, 16 ch • Sums amplitude from all channels • Transfer total energy or hit pattern to CTP Crate Trigger Processor

• Sums energies from FADCs • Transfer total energy or hit pattern to SSP

Fiber Optics • 64 bit @ 125 MHz

SSP

SS

P

CTP CTP

7/7/2014 SuperBigbite DAQ and Electronics Alexandre Camsonne

26

CP

U

SSP

SS

P

SSP

SS

P

SSP

SS

P

GTP

SD

SS

P

SSP

SS

P

SSP

SS

P

SSP

TI

VXS Crate

SSP

C

PU

FA

DC

FA

DC

FA

DC

FA

DC

FA

DC

FA

DC

C

TP

SD

FAD

C

FAD

C

FAD

C

FAD

C

FAD

C

FAD

C

TI

VXS Crate C

PU

TD

TD

SD

TS

VXS Crate

TD

TD

TD

TD

TD

TD

TD

TD

TD

TD

SD

L1 Trigger Diagram

CTP

C

TP

TS

TS

Global Trigger Processor • Collect L1 data from SSPs • Calculate trigger equations • Transfer 32 bit trigger pattern to TS


Sub-System Processor • Consolidates multiple crate subsystems • Report total energy or hit pattern to GTP

Copper Ribbon Cable • 32 bit @ 250 MHz: 8 Gbps

SSP SSP


27

fADC250

CTP Crate Trigger Processor

TI Trigger Interface

SD Signal Distribution

Detector Signals

Fiber Optic Link (~100 m)

(64bits @ 125 MHz)

(8) (2)

(12)

(1)

Copper Ribbon Cable (~1.5 m)

(32bits @ 250 MHz)

Fiber Optic Links Clock/Trigger

(16bits @ 62.5MHz

VXS Backplane

(16) (1) (1)

(1)

(1)

(1)

• Trigger Latency ~ 3 μs

( ) – Number in parentheses refer to number of modules

Custom Designed Boards at JLAB

Pipelined detector readout electronics: fADC

Level-1 Trigger Electronics


7/7/2014 28

Pipelined Hall D DAQ Calorimeter

Light Gas Cerenkov

Heavy Gas Cerenkov

FADC

FADC

FADC

CTP

CTP

CTP

CTP

SSP

G

TP

29 7/7/2014 SuperBigbite DAQ and Electronics

Alexandre Camsonne

Pipelined Hall D DAQ

3 us latency

Above threshold

Electron shower

accidental accidental

30 SuperBigbite DAQ and Electronics

Alexandre Camsonne 7/7/2014

ROC ROC

ROC ROC

ROC ROC

ROC ROC

ROC ROC

ROC ROC

ROC ROC

ROC ROC

ROC ROC

Front-End

Crates

Re

ad

Ou

t C

on

tro

llers

~60 crates ~50MB/s out

per crate

EB1 Event Builder

stage 1

EB1 Event Builder

stage 1

EB1 Event Builder

stage 1

EB2 Event Builder

stage 2

EB2 Event Builder

stage 2

Staged Event

Building

blocked event

fragments partially recombined

event fragments

N x M array of nodes (exact number to be determined by available hardware at time of purchase)

Level-3

Trigger

and

monitoring

full events

L3 Farm L3 Farm

node node

node node

node node node node

node node node node

node node

Ra

id D

isk

ER Event Recorder

Event

Recording

300MB/s in 300MB/s out

• All nodes connected with 1GB/s links

• Switches connected with 10GB/s fiber optics

L3 Farm


Alexandre Camsonne

Hall D L1 Trigger-DAQ Rate • Low luminosity (107 g/s in 8.4 < Eg < 9.0 GeV)

– 20 kHz L1

• High luminosity (108 g/s in 8.4 < Eg < 9.0 GeV)

– 200 kHz L1

– Reduced to 20 kHz L3 by online farm

• Event size: 15 kB; Rate to disk: 3 GB/s

Detectors which can be used in the Level-1 trigger:

SC

Forward Calorimeter (FCAL) Energy

Barrel Calorimeter (BCAL) Energy

Start Counter (SC) Hits

Time of Flight (TOF) Hits

Photon Tagger Hits

Energy FCAL (GeV) Energy FCAL (GeV) Energy FCAL (GeV)

Ener

gy B

CA

L (

GeV

)

Ener

gy B

CA

L (

GeV

)

Ener

gy B

CA

L (

GeV

)

Electromagnetic background Hadronic Eg < 8 GeV Hadronic Eg > 8 GeV

Basic Trigger Requirement:

EBCAL + 4 ∙ EFCAL > 2 GeV and a hit in Start Counter


32

Custom Electronics for JLab • VME Switched Serial (VXS) backplate

– 10 Gbps to switch module (J0)

– 320 MB/s VME-2eSST (J1/J2)

• All payload modules are fully pipelined

– FADC125 (12 bit, 72 ch)

– FADC250 (12 bit, 16 ch)

– F1-TDC (60 ps, 32 ch or 115 ps, 48 ch)

• Trigger Related Modules

– Crate Trigger Processor (CTP)

– Sub-System Processor (SSP)

– Global Trigger Processor (GTP)

– Trigger Supervisor (TS)

– Trigger Interface/Distribution(TI/D)

– Signal Distribution (SD)

FADC125 FADC125

F1-TDC F1-TDC


33

CP

U

FAD

C

FAD

C

FAD

C

FAD

C

FAD

C

FAD

C

CTP

SD

FA

DC

FA

DC

FA

DC

FA

DC

FA

DC

FA

DC

TI

VXS Crate

TI

TI

CP

U

SSP

SS

P

SSP

SS

P

SSP

SS

P

GTP

SD

SS

P

SSP

SS

P

SSP

SS

P

SSP

TI

VXS Crate

SSP

CP

U

TD

TD

SD

TS

VXS Crate

TD

TD

TD

TD

TD

TD

TD

TD

TD

TD

SD

L1 Trigger Diagram

Trigger Supervisor • Calculate 8 bit trigger types from 32 bit trigger pattern • Prescale triggers • Transfer trigger and sync signal to TD (16 bit total)

VXS Serial Link • 16 bit @ 62.5 MHz: 1 Gbps

Trigger Distribution • Distribute trigger, clock and synchronize signals to TI in each Crate

Fiber Optics • 16 bit @ 62.5 MHz: 1 Gbps

GTP

G

TP


34

VME Readout Controller • Gigabit ethernet

CP

U

TD

TD

SD

TS

VXS Crate

TD

TD

TD

TD

TD

TD

TD

TD

TD

TD

SD

TD

TD

CP

U

SSP

SS

P

SSP

SS

P

SSP

SS

P

GTP

SD

SS

P

SSP

SS

P

SSP

SS

P

SSP

TI

VXS Crate

SSP

C

PU

FA

DC

FA

DC

FA

DC

FA

DC

FA

DC

FA

DC

C

TP

SD

FAD

C

FAD

C

FAD

C

FAD

C

FAD

C

FAD

C

TI

VXS Crate

L1 Trigger Diagram

Signal Distribution • Distribute common signals to all modules: busy, sync and trigger 1/2


Trigger Interface • Receive trigger, clock and sync signals from TD • Make crate trigger decision • Pass signals to SD

TID TID


35

TOF

time of flight

SC

start counter

• 2.2T superconducting solenoidal magnet • Fixed target (LH2) • 108 tagged g/s (8.4-9.0GeV) • hermetic

2.2 Tesla Solenoid

Calorimetry • Barrel Calorimeter (lead, fiber sandwich) • Forward Calorimeter (lead-glass blocks)

PID • Time of Flight wall (scintillators) • Start counter • Barrel Calorimeter

Charged particle tracking • Central drift chamber (straw tube) • Forward drift chamber (cathode strip)

The GlueX Detector


36

Front End DAQ Rate

Event Size

L1 Trigger Rate

Bandwidth to mass Storage

GlueX 3 GB/s 15 kB 200 kHz 300 MB/s

CLAS12 0.1 GB/s 20 kB 10 kHz 100 MB/s

ALICE 500 GB/s 2,500 kB 200 kHz 200 MB/s

ATLAS 113 GB/s 1,500 kB 75 kHz 300 MB/s

CMS 200 GB/s 1,000 kB 100 kHz 100 MB/s

LHCb 40 GB/s 40 kB 1000 kHz 100 MB/s

STAR 50 GB/s 1,000 kB 0.6 kHz 450 MB/s

PHENIX 0.9 GB/s ~60 kB ~ 15 kHz 450 MB/s

LHC

JL

ab

BN

L *

CH

EP2

00

7 t

alk

Sylv

ain

Ch

apel

in

pri

vate

co

mm

.

* Jeff Landgraf Private Comm. 2/11/2010 ** CHEP2006 talk MartinL. Purschke

**

GlueX Data Rate


37

CODA3 – What’s different

CODA 2.5 CODA 3

Run Control (X, Motif, C++) (rcServer, runcontrol)

Experiment Control – AFECS (pure JAVA) (rcPlatform, rcgui)

Communication/Database (msql, cdev, dptcl, CMLOG)

cMsg – CODA Publish/Subscribe messaging

Event I/O C-based simple API (open/close read/write)

EVIO – JAVA/C++/C APIs Tools for creating data objects, serializing, etc…

Event Builder / ET System / Event Recorder (single build stream)

EMU (Event Management Unit) Parallel/Staged event building

Front-End – vxWorks ROC (Interrupt driven – event by event readout)

Linux ROC, Multithreaded (polling – event blocking)

Triggering: 32 ROC limit, (12 trigger bits -> 16 types) TS required for buffered mode

128 ROC limit, (32 trigger bits -> 256 types) TI supports TS functionality. Timestamping (4ns)


38

FADC Encoding Example


39

GTP Trigger Bit Example


40

Documents

SBS VME DAQhallaweb.jlab.org/12GeV/SuperBigBite/meetings/col... · (MC studies in progress) Calorimeter Rates HCAL ECAL Electron rate estimate w/2.5 GeV ... TI Trigger Interface SD