FELIX-based Vertical Slice TestH. Chen, K. Chen, J. Fried, S. Gao, B. Viren, S. Tang, J. Zhang

DUNE Far Detector DAQ WorkshopBrookhaven National Laboratory

July 20th, 2018

Outline❏ Current Test Setup❏ Vertical Slice Test Setup for DUNE❏ Development Plan

2Shaochun Tang (BNL)

Current Test Setup

❏ ProtoDUNE at CERN❏ Test setup at BNL❏ Test Setup at Nikhef

3Shaochun Tang (BNL)

ProtoDUNE Test Setup at CERN

FEMBx20

WIBx5

FELIXx1

4

ProtoDUNE-SP 5 WIBs

FELIX FLX712

APAx1

❏ LArTPC → ionisation tracks are collected by the wires of the Anode Plane Assemblies (APAs).

❏ 20 Front End Mother Boards (FEMB) digitize signals recorded by wires at 2 MHz. Each has up to 128 channels.

❏ 5 Warm Interface Boards (WIB), each WIB controls up to four 128-ch FEMBs❏ Output via optical links to DAQ:

❏ 2x 9.6 Gb/s or 4x 5 Gb/s supported, depending on readout solution❏ Continuous time-stamped data frame streams

❏ Each APA (2,560 channels) is read out by 5x WIBs for a total payload of about 74 Gb/s.

ProtoDUNE Test Setup at CERN Block Diagram FEMB

Shaochun Tang (BNL)

Block Diagram of ProtoDUNE Data flow

5

❏ Each FEMB has four links to WIB at 1.28 Gbps

❏ Each WIB accept 16 links from four FEMBs

❏ Each WIB is read out by two 9.6 Gbps FELIX links. Each link covers two FEMBs

Shaochun Tang (BNL)

Test Setup at BNL

❏ 4 FEMBs + 1 WIB + 1 FELIX (2 links are used)❏ Calibration data and pedestal data are tested.

❏ Each WIB has 2 links @ 9.6 Gbps❏ 1 FELIX has 24 Links @ 9.6 Gbps

6

FEMBx4

WIBx1

FELIXx1

Block Diagram of Test Setup at BNL

Shaochun Tang (BNL)

Test Setup at BNL - Test Results

7

❏ Left two figures are pedestal data of two FEMBs;

❏ The upright figure is one channel calibration data;

❏ the down right one is the time stamp data.

❏ Two links are running at 9.6Gb/s to FELIX successfully with real data and calibration data without CRC error.

❏ The WIB is freely running for 2 weeks, Both links are stable. We have set up a stability test for the WIB firmware. Till now, the firmware is very stable.

❏ Detailed results: https://indico.cern.ch/event/741215/contributions/3063991/attachments/1680963/2700648/WIB-FELIXmeeting-070418v1.pdf

Pedestal FEMB 1 Calibration data Channel 10 of FEMB2

Pedestal FEMB 2 Time stamp

Shaochun Tang (BNL)

Test Setup at Nikhef

❏ 1 Mini-FELIX + 1 FELIX or Mini-FELIX (4 links are used)❏ Data emulator is used at the Mini-FELIX (VC709) side.❏ 1 FELIX (FLX-711) or another Mini-FELIX (VC709) is used to receive

data. 4 Links @ 9.6 Gbps.

8

Mini-FELIX (VC709) x1FELIX (FLX-711) x1

orMini-FELIX (VC709) x1

Block Diagram of Test Setup at Nikhef

Shaochun Tang (BNL)

Vertical Slice Test for DUNE

Step 1: FELIX FLX-712 → FELIX FLX-712

Step 2: FELIX FLX-712 → New FELIX + DPM

Step 3: New FELIX + DPM → New FELIX + DPM

Step 4: Replacing the FLX-712 in ProtoDUNE setup with New FELIX + DPM

9Shaochun Tang (BNL)

Vertical Slice Test for DUNE (Step 1)

❏ Using two FELIX FLX-712 to do the test❏ Simulation data is generated at one FELIX.❏ Another FELIX is used to receive data.

10

FELIX (FLX-712) x1 FELIX (FLX-712) x1

The first step of VST based on the FLX-712

Shaochun Tang (BNL)

Vertical Slice Test for DUNE (Step 2)

❏ Using one FELIX FLX-712 and one new FELIX with DPM to do the test.❏ Initial test setup for new FELIX+DPM debug.❏ Simulation data is generated at the FELIX FLX-712.❏ The new FELIX with DPM is used to receive data.

11

FELIX (FLX-712) x1 FELIX + DPM x1

The second step of VST based on the FLX-712 and Modified FELIX +DPM

Shaochun Tang (BNL)

Vertical Slice Test for DUNE (Step 3)

❏ Using two new FELIX with DPM to do the test.❏ Simulation data is generated at one FELIX+DPM.

❏ This may take advantage of storage available on DPM.❏ The new FELIX with DPM is used to receive data.

12

FELIX + DPM x1 FELIX + DPM x1

The third step of VST based on the Modified FELIX +DPM

Shaochun Tang (BNL)

Vertical Slice Test for DUNE (Step 4)

❏ After verifying the FELIX+DPM, then can replace the FELIX in the ProtoDUNE test setup at CERN to do more tests.

13

The step 4: replacing the FELIX with FELIX+DPM in the protoDUNE test setup at CERN

FEMBx20

WIBx5

FELIX + DPM x1

APAx1

Shaochun Tang (BNL)

Development Plan

❏ Hardware:❏ Modification is minimal,❏ 10% work.

❏ Firmware:❏ Need to design lots of firmware blocks. Details refer to Kai and Frans slides. ❏ 40%.

❏ Software:❏ Need lots of software development❏ 50%.

❏ Integration:❏ Test… Test … Test...❏ 200%.

14Shaochun Tang (BNL)

Summary

❏ Vertical slice test takes real effort to make it successful (useful).❏ Firmware/software development should start as early as possible to meet tight

schedule (and goals).❏ It would be too late to start FW/SW development after hardware becomes available.❏ FLX-712 is available today to facilitate the setup of lab test stand, get familiar with firmware and

software design.❏ One (management) shall start to plan how to organize the various FW/SW developments, and

integration test ❏ Identify resources who will have (obtain) experiment specific expertise❏ Team building along the VST development serves as the backbone for DUNE FD DAQ system

❏ ProtoDUNE is the most important test platform to validate system design concepts.❏ It would be the best to exercise the VST with real detector in cosmic run, provided the early steps of

VST are sufficiently developed15Shaochun Tang (BNL)

Thank you for your attention

16Shaochun Tang (BNL)

THAN