Integrating Timepix(3)

Szymon Kulis, Mathieu Benoit, Bas van der Heijden, Frans Schreuder, Henk Boterenbrood, MvB

and the Timepix3 designersXavi Llopart, Tuomas Poikela, Vladimir Gromov, Francesco Zappon

Massimiliano de Gaspari

and others

10 December 2013

Timepix versus Timepix3

Timepix(3) integration, AIDA common DAQ, 10 Dec. 2013Martin van Beuzekom 2

Timepix Timepix3

55x55 um2 pixels 55x55 um2 pixels

256x256 matrix = 1.4x1.4 cm2 256x256 matrix = 1.4x1.4 cm2

ToA or ToT measurement ToA and ToT simultaneously

frame based readout data driven readoutzero-suppressed frame readout possible

max. 120 frames/sec, with 100 tracks/frame= ~10k tracks/s (more is possible)

(tens of) millions of tracks/stheoretical max of 80 Mhits/s

dead time due to read out of single frame: 8+ ms

dead time less

not radiation hard radiation hard; several hundred Mrad but not SEU robust

FitPix or Relaxd readout SPIDR or FitPix readout

Timepix integration in common DAQ

Detailed talk given by Mathieu Benoit in WP9.3 meeting on Nov. 21 Lots of info in that talk, will not repeat it here. Please look at https://indico.desy.de/conferenceDisplay.py?confId=8849 Main bottle-neck: Timepix is either in acquisition mode, or in readout

mode Readout time of 8 (or more) ms per frame -> long dead time Operation of FitPix and Relaxd is similar

Timepix(3) integration, AIDA common DAQ, 10 Dec. 2013Martin van Beuzekom 3

Our (Nikhef/LHCb) focus is on Timepix3

In view of the LHCb VeloPix developments (ASIC submission aimed for Q3 2014) VeloPix which has to cope with 600 Mhits/s But we also have a complete and working Timepix/Relaxd telescope

Timepix(3) integration, AIDA common DAQ, 10 Dec. 2013Martin van Beuzekom 4

SPIDR

Speedy PIxel Detector Readout

Readout system for Timepix3 and MPX3.1/RXover 10 Gb or 1 Gb Ethernet

Timepix(3) integration, AIDA common DAQ, 10 Dec. 2013Martin van Beuzekom 5

SPIDR

Timepix(3) integration, AIDA common DAQ, 10 Dec. 2013Martin van Beuzekom 6

Firmware properties: Vendor independent and highly configurable 1 Timepix3 at full (80 Mhits/s) speed, or multiple

Timepix3 chips at lower speed LFSR lookup tables in FPGA Pixel data over UDP/IP, slow control over TCP/IP

Currently running on a development system (Xilinx VC707) Multiple setups running at Nikhef and CERN

Development of Compact SPIDR ongoing (but at a slower pace)

FGPA USB12VDC/DC

1V2DC/DC

1V5DC/DC

2V5

FMC PHY

Ext bias

+/- 400V Bias supply I2C

SFP+ cage 1G/10G

Expansion header

Trigger/busy

Multiple TPX3 hardware configurations

Timepix(3) integration, AIDA common DAQ, 10 Dec. 2013Martin van Beuzekom 7

NikhefCompact readout

NikhefCompact readout

Cern Chipboard

NikhefChipboard

FMC extender

cable50cm FMC 2

VHDCI

Xilinx VC707 development board

NikhefChipboard

FMC extender

cable50cm

NikhefChipboard

FMC extender

cable50cm

(Thanks to Jerome Alozy)

FitPix

Cern Chipboard

Data acquisition

To run at high speed, each SPIDR needs its own DAQ PC A single timepix3 at max. speed produces 5.12 Gbits/s Including the extended time-stamp and overhead this fills ~70% of a

10 GbE link Data is streamed to disk, without looking at the data

Not possible to evaluate each packet at maximum speed Separate monitoring stream which samples (copies) the main data stream

DAQ Format: header of several kBytes (settings etc. ) followed by a stream of pixel packets, 8 bytes each

Header format not yet frozen No trigger/event number or alike Pixels packets are unique by their time-stamp Synchronisation and checking of different SPIDR data streams is

important

Timepix(3) integration, AIDA common DAQ, 10 Dec. 2013Martin van Beuzekom 8

Data acquisition II

Timepix(3) integration, AIDA common DAQ, 10 Dec. 2013Martin van Beuzekom 9

FTOA 4b SPIDR time 16bToA 14b ToT 10bhdr 4b PixAddr 16b

FTOA 4bSPIDR time 16b ToA 14b

time stamp30 bits, 25ns resolution

+ 4 bits, 1.56 ns resolutionrange: 26.844 sec

01516192029304344596063

034171833

However 26 seconds is too short for a normal run Timepix3 has an internal 48 bit counter, which is reset with the T0-sync

resolution 25 ns, range 81.44 days This timer can be read on request, and has unique packet header -> timer data will end up in the data-stream Hence to cover a larger time range, the (Leon) processor in the SPIDR

FPGA will request readout of this timer every second.

Running with multiple SPIDRs

Required TLU functionality can be very basic Provide clock, shutter, T0-sync and combine busy signals (OR) Somewhat more advanced functionality required for high speed

monitoring (next slides)

Timepix(3) integration, AIDA common DAQ, 10 Dec. 2013Martin van Beuzekom 10

TLU

SPIDR SPIDRSPIDR

repeater repeater

busy

clock, T0-sync, shutter

Interface to TLU, sync time-stamp Synchronising time-stamps

Disable shutter Send T0-sync, min. 25 ns Enable shutter

Timepix(3) integration, AIDA common DAQ, 10 Dec. 2013Martin van Beuzekom 11

25 ns (or more)

> 1.6 us

shutter (ext)

tpx3-reset (int, optional)

T0-sync (ext)

0 1 2 3 4 5 6 7 8nn-1n-2time-stamp

no data packets

no data packets

data-outw.o reset

data-outwith reset

> 25 ns

TLU interface, Busy Data flow is controlled/halted via shutter/busy mechanism Busy is tied to ‘almost full flag’ of SPIDR ethernet buffer DAQ PC signals busy/overflow by sending pause-frames to SPIDR

-> SPIDR buffers fill up, leading to the SPIDR pulling the busy

In addition DAQ PC can send a ‘halt’ command to Leon processor This will pull also pull the busy line

Will be implemented soon

Timepix(3) integration, AIDA common DAQ, 10 Dec. 2013Martin van Beuzekom 12

shutter (ext-in)

busy-0 (out)

no data packetsdata-out1

busy-1 (out)

no data packetsdata-out0

Monitoring of data streams Monitoring of data streams by copying a fraction of the data Send data to a dedicated monitoring/slow control PC Two sampling methods Software sampling

Each DAQ PC takes a snapshot at a given time (CPU timer triggered, or via run control?)

However DAQ streams are not synchronised -> need a large snapshot to guarantee overlap between streams

Hardware sampling Dedicated ethernet packets (different destination IP address, and/or port number)

are generated by SPIDR Copies data from a range of timestamps to this dedicated monitoring packet Start of monitoring sample could be controlled by TLU Length of monitoring sample defined by SPIDR setting This is foreseen, but not yet implemented

Timepix(3) integration, AIDA common DAQ, 10 Dec. 2013Martin van Beuzekom 13

Summary & Plans Development of Timepix3 readout is actively ongoing (Nikhef + CERN) But building a high speed readout is not trivial (so we have to be

patient) We (LHCb) are building a high speed Timepix3 telescope

Timepix3 is good exercise for VeloPix which produces 8x more hits/tracks VeloPix has to be submitted Q3 next year

Integrating Timepix3 into an AIDA telescope seems simpler than integrating Timepix

Matching of data from different SPIDR streams using time-stamps Checking of synchronicity is key Separate monitoring data stream

Timepix(3) integration, AIDA common DAQ, 10 Dec. 2013Martin van Beuzekom 14