17 January 2002The Beam Test DAQ Design Status of the design and implementation of the DAQ for the beam test G. Alimonti, G. Chiodini, S. Magni, D. Menasce,

17 January 2002 The Beam Test DAQ Design

Status of the design andimplementation of theDAQ for the beam test

G. Alimonti, G. Chiodini, S. Magni, D. Menasce, L. Uplegger


The basic idea (1)• We would like to be able to read out events in a stream-like mode: each pixel element that has pulse-height data, above threshold, sends them to a memory along with space and time coordinates (row, col and time stamp).

• We will make use of standard PCI bus cards with programmable logic and local memories

The DAQPC

The DAQPC

DetectorDetector Mezzanine-cardMezzanine-card PCI cardPCI card



PCI extender

Read-out &monitor processes

Read-out &monitor processes

• The DAQ should be broad in scope enough to cover not only pixel needs, but in principle other detectors as well


The basic idea (2)• We are concentrating our efforts to develop a system to read-out PCI local memories to the DAQ computer and provide users with the ability to get events and do with them whatever they like.

• We would like to build the simplest possible system having in mind an architecture where each component is loosely (or not at all) coupled with others. This approach reduces cross dependencies and allows concurrent devlopers to contribute with sofwtare components with few or no logical incompatibilites.

• Language of choice is C++. Reasons are: 1) the Jungo Driver is written in C 2) we would like to keep the number of languages to a minimum and C++ offers all the flexibility we’ll possibly need 3) As a physics analysis tool we plan to adopt Root: interface with it becomes then trivial


Read-out process

Read-out process

Components (0)• Basic mechanism of operation of the PCI card and the read-out:

Bank0 Bank1

FPGAFPGA

0

Interrupt handlerInterrupt handler

Reset interruptReset interrupt

Shared memory

Disk stagerDisk stager

ConsumerProcess AConsumerProcess A

ConsumerProcess BConsumerProcess B



Bank0

FPGAFPGA

Time0


Bank1Read-out process

Read-out process

Shared memory

Reset interruptReset interruptDisk stagerDisk stager



Let’s see how thisworks with a cartoon...



Bank0

FPGAFPGA

Time0



Read-out process

Shared memory






Bank0

FPGAFPGA

Time0



Read-out process

Shared memory






Bank0

FPGAFPGA

Time0



Read-out process

Shared memory






Bank0

FPGAFPGA

Time0



Read-out process

Shared memory






Bank0

FPGAFPGA

Time0



Read-out process

Shared memory






Bank0

FPGAFPGA

Time0



Read-out process

Shared memory






Bank0

FPGAFPGA

Time0



Read-out process

Shared memory






Bank0

FPGAFPGA

Time0



Read-out process

Shared memory






Bank0

FPGAFPGA

Time0



Read-out process

Shared memory






Bank0

FPGAFPGA

Time0



Read-out process

Shared memory






Bank0

FPGAFPGA

Time0



Read-out process

Shared memory






Bank0

FPGAFPGA

Time0



Read-out process

Shared memory






Bank0

FPGAFPGA

Time0



Read-out process

Shared memory






Bank0

FPGAFPGA

Time0



Read-out process

Shared memory





DetailsThis process of periodic memory swap and transfer to a sharedmemory continues indefinetely. The shared memory is a large chunk of memory that acts as a balancing buffer to compensate rate fluctuations in the read-out chain.

ProblemWe have several PCI cards playing this swap game in parallel:in order to be able to rebuild events at a later stage, keeping track ofevents with the same time-stamp, we have to make some architecturalchoice at this point (an event consists, is defined, by of all hits marked by a same time-stamp, which have therefore occurred together in time)

Let’s examine pro and cons of two alternative possibilities:


More details (1)

Interrupt handler AInterrupt handler A

Interrupt handler BInterrupt handler B

Interrupt handler CInterrupt handler C

Interrupt handler nInterrupt handler n

Each PCI card has it’s owninterrupt-handler process listening

for the memory-full signal

As soon as one PCI fills up one of it’stwo 1Mb memories, it’s interrupthandler gets notified and a series ofactions occur: let’s examine a typicalscenario. Suppose, for instance, thatthe PCI C is the first being filled up:


More details (2)





• The memories of the PCI C get swapped


More details (3)





• The memories of the PCI C get swapped


More details (4)





• The memories of the PCI C get swapped• At this point we would like to swap all other memories togheter and write their content to a shared memory. In principle the first card that gets filled can become the one that commands all others to swap (becoming a de-facto master swapper)

• At any given time, though, only one master can be in charge, so we need a mechanism to establish a hierarchy among boards (only one board at a time can order swaps, otherwise chaos can ensue)


More details (5)





Auxiliaryshared

memory1

• An auxiliary shared memory can be used for this purpose. When an IH gets notified that a swap has occurred, it first checks whether another master is already at work.

• if FALSE, the aux. shared memory is flagged, the PCI C becomes the master swapper and, whether or not each of the other PCI card memories are currently full, it forces them to swap and generate an interrupt

• if TRUE, nothing occurs, this board is just passive and does nothing else.


More details (6)





Auxiliaryshared

memory1

• Problem is that IH C needs a few CPU cycles to swap all other PCI memories and during this time one of them can become full and swap. In this case we have a locking mechanism that prevents additional unintended swaps: if the pointer of a bank is not zero, the swap simply does NOT occur.

This pointer is reset by the read-out process when the last bank has been read out.

Let’s picture this process by a cartoon:


More details (7)





Auxiliaryshared

memoryTime: t0

0 1

At this time all banks 0 arefilling up, and the board C is reaching the full status first

Banks


More details (8)





Auxiliaryshared

memoryTime: t1

0 1

banks 0 of board C is finallyfull: at the next cycle it willswap banks and raise an interrupt

Banks


More details (9)





Auxiliaryshared

memoryTime: t2

0 1

Bank 1C starts filling and the interrupthandler checks if it can become master.

Banks

Since the auxliary shared memory isset to zero the board C can become master by flagging it to 1.

1

It then begins the process of swappingall other board’s banks starting withboard A.


More details (10)





Auxiliaryshared

memoryTime: t3

0 1Banks 1

During the cycles needed to accomplishall this jazz about swapping, it can happen that another board becomesfull and swaps (e.g. Board B).


More details (11)





Auxiliaryshared

memoryTime: t4

0 1Banks 1

When board B receives the command toswap, nothing happens, since it’s pointerhas not been reset by the read-out process. With this mechanism we aresure that all swaps are locked togetheruntil a full read-out has completed. Ifone detector is too noisy and requeststoo many swaps, faster than the read-outcan absorb, we start loosing events.The rate at which we can read events is2 Mb/s (more about this later)

Status register to hold informationabout banks already emptied


More details (12)





Auxiliaryshared

memoryTime: tn

0 1Banks 1

In the end of this swapping frenzy allboards either swap by themselves orare commanded to do so by board C

When all banks 0 have been emptiedand their content transfered to the large shared memory, all pointers are zeroed and a new swapping cycle can eventually restart.


More details (13)

• This architecture guarantees that events with nearby time-stamps belong to buffers which are contiguous in the read-out process, and thereby contiguous in the global shared-memory. As a result the event builder becomes relatively easy to implement.





Auxiliaryshared

memory0 1Banks 1

A B ... n


More details (14)

• With this read-out architecture, events with the same timestamp are contained within the boundaries of this overall buffer (BUFi), or at least in the next one, BUFi+1, but not in BUFi+2, making the event-builder a rather trivial implementation of a sorting algorithm.





Auxiliaryshared

memory0 1Banks 1

A B ... n

BUFi

A B ... n

BUFi+1


More details (15)





Auxiliaryshared

memory0 1Banks 1

• The drawback of this architecture, which we call asynchronous, is that each board must be knowledgeable about each other’s status in order to start a new cycle.

• A possible way out is by implementing a hardwired cable that connects all interrupts together: when a board emits an interrupt all boards do the same at the same time and swap in unison

• There no longer is the possibility of time glitches whereby a swap is forced just after one occurred because the memory reached the limit.


What has already been accomplished

The architecture mentioned above poses several problems (each boardmust be knowledgeable about each other’s status in order to force asynchronous memory swap, then there is the possibility of time-glitches between swap commands to take into account and other factors), but we think we have found a correct solution for all these problems.

• The first component we developed has been an abstract layer to the underlying PCI device driver (our code is written in C++). This allows us to swap to other device drivers, besides the Jungo Driver, eventually skipping license fees problems (we could even write our own light-weight driver, we think we know how to do that)

Let’s see what has already been built:


FPGA programming• The second major component has been the micro-programming of the ALTERA FPGA. Key issues here are:

2. ability to generate an interrupt upon a memory full condition

3. Possibility to generate specific patterns to feed the memory, thus allowing for debugging tests of the system (no need for an actual detector)

Code generated by the Quartus software (the FPGA firmware)can also be uploaded to the PCI board we have in Milano, thus allowing tests and code development to be carried out in parallel

1. Swap of the two memory banks when a full condition is reached

3. I/O activity and synchronization between banks and detector


The Interrupt Handler • When an interrupt is generated, a thread, spawned by the read-out overall process, listens for it and issues the command to swap all other memory banks eventually starting the read-out. This component is rather complex, since it must perform a whole list of checks before, eventually, becoming the master-swapper:

1. only one board at any given time can be allowed to perform this operation, in order to avoid unwanted multiple swaps)

2. care must be taken to avoid commanding a board to swap just after it swapped on it’s own because its memory got filled. (the FPGA has been programmed to reject a swap if its read-out pointer has not been reset, and this happens only when ALL memories have been flushed to the overall shared memory)


The Shared Memory• The heart of the system is a piece of code that creates a shared memory (or attaches to it if it’s already present). This shared memory has been implemented as a circular buffer, with all the utilities needed to inspect it’s content and synchronize the fill-in and read-out to disk.

• The obvious advantage of such an architecture is that there is a complete logical decoupling between the activity of the PCI board and the activity of the host computer. The timing is NOT dictated by the read-out process but by the detector itself.

• Another advantage is that a process to check the internal consistency of the data flow is just a consumer of this shared memory, but a completely separate and detached process (this component is still missing, though)


The Message Reporter• In order for all the processes, acting as the DAQ, to synchronize their activity, a light-weight package has been developed to act as as a Message Reporter (based on the IPC protocol). More refined packages could also be used, but at this stage we would like to keep the total amount of code to a minimum for better control.


The Code Management• Since our activity is already split between Milano and Fermilab, we adopted CVS as a code management system since the beginning. Periodic tags and releases are produced on AFS space, making the code available to users in real time.


Preliminary benchmarks (1)• We have carried out preliminary tests to check the sustainable data flow we can expect. We still have to understand several issues, but we can summarize here the key facts:

• The PLX 9030 PCI Controller on the PTA card we are using does NOT allow for DMA transfer. It could in principle be used in burst-mode, but this feature is not available on the INTEL architecture. We are thus forced to run in single word transfer mode, but this seems more than adequate for our beam test needs.• We have tried three different setups:

1. A PCI card directly connected to a 1.3 GHz PC motherboard2. A PCI card directly connected to a 1.8 GHz PC motherboard

3. A PCI card connected to a 1.8 GHz PC motherboard by means of the PCI extender bus


Preliminary benchmarks (2)• We observe the following data transfer rate from the PCI board to the shared memory in the three cases mentioned above:

1.3 GHz 1.8 GHz

Direct connection to motherboard

6 Mb/sec 4 Mb/sec

Connection to motherboard by PCI extender

- 2 Mb/sec

• We still do not fully understand these numbers, but studied are under way


To do• In this overall architecture several components are still missing:

1. A sophisticated error handling, encompassing both errors generated by the hardware (suppression of extremely noisy channels, dead PCI boards, faulty memories etc...) and by run time conditions (such as excessively high rate due to beam conditions and such)

2. Detailed benchmark test to determine the upper limit of the sustainable data rate. We already have made measurements and determined bottlenecks but a systematic work has still to be done.

3.

Documents

17 January 2002The Beam Test DAQ Design Status of the design and implementation of the DAQ for the beam test G. Alimonti, G. Chiodini, S. Magni, D. Menasce,