LHCb DAQ Protocol Independent Performance

Evaluator: DAQPIPE

Daniel Hugo Cámpora Pérez LHCb DAQ workshop, 27-05-2013

1

What is it, anyway

● DAQPIPE is an evaluation tool designed to measure the performance of a generic DAQ system.

Daniel Hugo Cámpora Pérez LHCb DAQ workshop, 27-05-2013

2

EM

What we want to measure

● Network transfer protocols● Network topologies● Event-Building protocols

Daniel Hugo Cámpora Pérez LHCb DAQ workshop, 27-05-2013

3

Network transfer protocols

● Preliminary tests with TCP/UDP over 40G.● Preparing IB FDR tests.● Testing

– 40G Ethernet UDP– iWarp– ROCE– IB OpenMPI (full hardware support)– IB RDMA

Daniel Hugo Cámpora Pérez LHCb DAQ workshop, 27-05-2013

4

InfiniBand? Ethernet? Upcoming ones?

Network topologies (recap)

Unidirectional flow

● Simple

● Current solution, scale up

Daniel Hugo Cámpora Pérez LHCb DAQ workshop, 27-05-2013

5

Network topologies

Bidirectional uniform flow

● Full-Duplex on core – mixed-servers.

● Implementation details: RUs sit on servers.

● Fraction of computing done in receiving nodes, offloading done to FUs.

Daniel Hugo Cámpora Pérez LHCb DAQ workshop, 27-05-2013

6

Event-Building protocols

● PUSH● PUSH with Barrel Shifting● PULL

Daniel Hugo Cámpora Pérez LHCb DAQ workshop, 27-05-2013

7

Push

Daniel Hugo Cámpora Pérez LHCb DAQ workshop, 27-05-2013

8

Barrel shifting

● Equal-length receiving queues, asymmetric sending queues.

● Potential dramatic decrease of congestion in DAQ network.

Daniel Hugo Cámpora Pérez LHCb DAQ workshop, 27-05-2013

9

Pull

Daniel Hugo Cámpora Pérez LHCb DAQ workshop, 27-05-2013

10

Let's talk software

● Test suite should be– Portable– Flexible– C-like entry point (interfaceable)– Scalable– Efficient!

Daniel Hugo Cámpora Pérez LHCb DAQ workshop, 27-05-2013

11

Output

Daniel Hugo Cámpora Pérez LHCb DAQ workshop, 27-05-2013

● Logs & rates– Effective throughput– Maximum trigger rate– Error rate– Incomplete event rate

12

Preliminary tests

Daniel Hugo Cámpora Pérez LHCb DAQ workshop, 27-05-2013

● PTP allows for microsecond-level synchronization.

● RDMA over IB CPU consumption is negligible.

13

Preliminary tests

Daniel Hugo Cámpora Pérez LHCb DAQ workshop, 27-05-2013

● 40GbE

14

Tx Rate CPU Consumption

40G (TCP), eight streams

36 Gb/s 5 – 8%

4 x 10G (TCP)

36 Gb/s

40G (UDP), four streams

32 Gb/s 4%

HP 59002x Mellanox Technologies MT27500 Family ConnectX-3

8x Chelsio Communications Inc T420-CR

Thanks!

Daniel Hugo Cámpora Pérez LHCb DAQ workshop, 27-05-2013

15

Backup

Daniel Hugo Cámpora Pérez LHCb DAQ workshop, 27-05-2013

16

Network topologies

Bidirectional mixed flow

● Full-Duplex on core - TOR connections.

● Port-density requirements for core switches significantly reduced.

● Server-wise identical to unidirectional solution.

Daniel Hugo Cámpora Pérez LHCb DAQ workshop, 27-05-2013

17

Network topologies

Bidirectional uniform flow

● Relatively high load per mixed server (RU/BU/FU) – 4 x rate

– Input – RU

– Input / Output – BU

– Output - FU

Daniel Hugo Cámpora Pérez LHCb DAQ workshop, 27-05-2013

18