1

Upgrades for the PHENIX Data Acquisition System

Martin L. Purschke, Brookhaven National Laboratoryfor the PHENIX Collaboration

RHIC from space

Long Island, NY

2

RHIC/PHENIX at a glance

RHIC:2 independent rings, one beam clockwise, the other counterclockwisesqrt(S)= 500GeV * Z/A~200 GeV for Heavy Ions~500 GeV for proton-proton (polarized)

PHENIX:

4 spectrometer arms

15 Detector subsystems

500,000 detector channels

Lots of readout electronics

Uncompressed Event size typically 280 -220 - 130 KB for AuAu, CuCu, pp

Data rate ~6KHz (Au+Au)

Front-end data rate 0.8 - 1.5 GB/s

Data Logging rate ~500MB/s, 700 MB/s max

3

TOF-W

RXNP MPC-N

Our youngest Detector Systems

HBD

4

Need for Speed: Where we are

ATLAS

CMS

LHCb

ALICE

CDF

~25 ~40

~100

~300All in MB/sall approximate~100

~150

600

~1250

Lvl1-Triggers in Heavy Ions have a notoriously low rejection factorthat's because so many events have something that's interesting (different from LHC)But hey, we could write out almost everything that RHIC gave us, so why bother... this approach has served us really well. It also opened up access to processes that you can't exactly trigger on, it “just” takes some more work offline.

5

600++ MB/sThis shows the aggregated data rate from the DAQ to disk in a RHIC fillWe are very proud of this performance...

Decay of RHIC Luminosity

Length of a DAQ run

It's not the best, it's one where I was there... the best RHIC fill best went up to 650MB/s

6

Run 10 Event statistics

3150 TB

950 TBPhysics

PHENIX Raw Data100 TB

7

Upgrade Programs

RHIC will give us several luminosity and beam livetime upgrades

The era where could mostly write out “everything” is coming to an end

The Future

we will add detectors in the central region which will significantly increase our data volume

8

Upgrades

3 main new detectors (that's in addition to the ones I showed before as “on board”):

• The Vertex/Forward Vertex detectors• A Muon trigger upgrade• RPC detectors

800-pound gorilla

23911Total

1.071002.8%6FVTX

1.92390.54%/0.16%3VTX pixel

1.25904.5%/2.5%2VTX strip

Data rate (Gbps)*

Event size (kbyte)

OccupancyDCM groups

Detector

Triples current event size 23911Total

1.071002.8%6FVTX

1.92390.54%/0.16%3VTX pixel

1.25904.5%/2.5%2VTX strip

Data rate (Gbps)*

Event size (kbyte)

OccupancyDCM groups

Detector

VTX pixel 3 0.54%/0.16% 39 1.92

VTX Pixels will be installed this summer for Run 11

9

Upgrades

Central Silicon Vertex Trackers

“VTX”

Pixel

Strippixel

10

Silicon Pixel in Run 11ALICE1LHCb readout chip:

Pixel: 50 µm (f) x 425 µm (Z). Channels: 256 x 32.Output: binary, read-out in 25.6s@10MHz.Radiation Hardness: ~ 30Mrad

Sensor module:

4 ALICE1LHCb readout chips.Bump-bonded (VTT) to silicon sensor.Thickness: 200 mThickness: r/o chips 150 µm

Half-ladder (2 sensor modules+bus)

1.36 cm x 10.9 cm.Thickness bus: < 240 µm.

SPIRO module • Control/read-out a half ladder

Send the data to FEM

FEM (interface to PHENIX DAQ)Read/control two SPIROs

• Interface to PHENIX DAQ

active arear

1.28 cm = 50mm x 256z

1.36 cm = 425mm x 32

Solder bump

~20m

All chips on Ladder #6 has good hit map by beta-ray source test

1st Complete Pixel Ladder on Dec 25, 2009

The hitmaps are a great success, but from where I stand, the fact that we are

reading out the ladder is most important

12

Buffer Box

Data Flow

ATP

ATP

ATP

ATP

ATP

ATP

ATP

ATP

ATP

ATP

SEB

SEB

SEB

SEB

SEB

SEB

SEB

SEB

SEB

SEB

Gigabit

Crossbar

Switch

To

HPSS

Buffer Box

Buffer Box

Buffer Box

Buffer Box

Buffer Box

Buffer Box

Classic Event builder architecture

DCMDCMDCMDCM

DCMDCMDCMDCM

DCMDCMDCMDCM

Data Collection Modules (100's)

Sub Event Buffers

(~35)

Crossbar Switch

Assembly & Trigger

Processors (~60)

Buffer Boxes (7)

Data ConcentrationInteraction

Region Rack Room

13

The A”T”P ATP

• “Assembly and Trigger Processor” was deemed important at the time when

PHENIX was designed

• Meant to run what is today known as a HLT

• We soon learned that we can do without it

• Data logging capability has kept pace with the MB data rate, which is great

• Opens access to processes which you simply can't trigger on

With all the upgrades I'll show, we are determined to keep

it that way

14

Need for Speed

• Data Collection Module (DCM) was modern in its

days, DSP-based + some FPGA

• DCM II uses latest FPGA technology, FPGA is the

main component

• 10G networks are becoming a commodity

• Allows to make better use of multi-core machines

• Saves money and power and A/C in the end

• In the same spirit, replace PCI with PCI Express

ATP

ATP

ATP

ATP

ATP

ATP

ATP

ATP

ATP

ATP

SEB

SEB

SEB

SEB

SEB

SEB

SEB

SEB

SEB

SEB

Gigabit

10G

Crossbar

Switch

Buffer Box

Buffer Box

Buffer Box

Buffer Box

Buffer Box

Buffer Box

Buffer Box

15

DCM-II and “jSEB”-II

• jSEB (“SubEvent Buffer”) is the PCI-Express card that reads a number of DCM's

• About a factor of 15 more bandwidth than the older generation

DC

M

II DC

M

II DC

M

II

Partitioner

II

INT

ER

FA

CE

BUSY

DATAL1

PC

JSE

B

II PCSEB

JSE

B

II

GTM

L1

Custom Backplane

FEM

SEBDCMDCMDCMDCM

16

Force-10 Switch

Standard Gigabit ports

“MRJ-20” cable bundles with 6 Gig ports each go to patch panels (or directly to the machines) May make our cable distribution easier, bring a few bundles to the racks

10GbE portsStandard fibers with “SFP+” connectors

17

Will add buffer boxes as needed

Buffer Box

Buffer Box

Buffer Box

Buffer Box

Buffer Box

Buffer Box

Putting it together

DC

M

II DC

M

II DC

M

II

Partitioner

IIINT

ER

FA

CE

BUSYDATAL1

PC

JSE

B II PC

SEB

JSE

B II

GTM

L1

Custom Backplane

FEM

ATP

ATP

ATP

ATP

ATP

ATP

ATP

ATP

ATP

ATP

SEB

SEB

SEB

SEB

SEB

SEB

Buffer Box

Buffer Box

Buffer Box

Buffer Box

Buffer Box

Buffer Box

DC

M

II DC

M

II DC

M

II

Partitioner

IIINT

ER

FA

CE

BUSYDATAL1

PC

JSE

B II PC

SEB

JSE

B II

GTM

L1

Custom Backplane

FEM

DC

M

II DC

M

II DC

M

II

Partitioner

IIINT

ER

FA

CE

BUSYDATAL1

PC

JSE

B II PC

SEB

JSE

B II

GTM

L1

Custom Backplane

FEM

The new jSEB-II's will exceed the limits of a Gigabit connection, need 10GbE

Will have 5 such jSEB-IIs in Run 11

20 for the full system in Run 13

The existing detectors will continue to use the current readout

The DAQ upgrades are geared towards maintaining the current event rate, not to increase it – for now. The existing detectors will keep their readout electronics and will limit us to the current rate

18

RPC (resistive plate counter)

RPC3

Test assembly of RPC-3 half octant support structure at UIUC

Adds timing resolution to the Muon detectors so we can distinguish muons from the vertex fron those traversing the IR

19

Outlook

PCI Express 10GB/s

Networks

DCM IIUpgrade

New Detectors on board or coming to us to search for dedicated signals

New hardware components to help maintain our current speed

The End

20

Backup

21

Pictures

22

Cabling “plan”

• Current Switch

23

Building up to record speed

• Over the previous runs we have been adding improvements

• Had lighter systems, d+Au, p-p, Cu-Cu in previous runs, less of a challenge than 200GeV Au+Au (most challenging)

• Distributed data compression (run 4)

• Multi-Event buffering (run 5)

• Mostly consolidating the achievements/tuning/etc in run 6, also lots of improvements in operations (increased uptime)

• 10G Network upgrade in run 7, added Lvl2 filtering

Ingredients:

24

Data Compression

LZO

algorithmNew buffer with the compressed one as payload

Add new

buffer hdr

buffer buffer buffer buffer buffer buffer

LZO

UnpackOriginal uncompressed buffer restored

This is what a file then looks like

On readback:

This is what a file normally looks like

All this is handled completely in the I/O layer, the higher-level routines just receive a buffer as before.

Found that the raw data are still gzip-compressible after zero-suppression and

other data reduction techniques

Introduced a compressed raw data format that supports a late-stage

compression

Run 11 Counts

strips pixel FVTX Total

Fibers 60 40 54

DCM2 module 8 5 7 21

Partitioner 3 5 3 4 12

JSEB 2 5 3 4 20 ** Include JSEB II for the crate controllers

The build for the final system running at full bandwidth.

The ratio between partitioner 3 to DCM is 2 to 1

16 fibers 1 fiber

26

Data Collection Module-II

• Front-end cards on the detector are getting read out by those (8 per card)

• DCM-II connects to a custom bus on the right

• A number of DCM-II's are read out via a PCI Express Card

48V in(Isolated)

5V in (control)

Download(readback)

L1

data

JTAG

(640MB/sec)

( 160 MB/sec)JTAG

clock320MB/sec

1.1V2.5V3.3V

27

Multi-Event Buffering: DAQ Evolution

PHENIX is a rare-event experiment, after all -- you don’t want to go down this path

Without MEB

Multi-Event buffering means to start the AMU sampling again while the current sample is still being digitized.

Trigger busy released much earlier

deadtime is greatly reduced

28

The Multi-Event Buffering Effect

29

Upgrades

• All new detectors have electronics with high rate capability• However, older detector readout limits the level 1 rate• no way to upgrade any time soon - $$$$$• We will need to focus on rare events more

Front end pipelines

Readout buffers

Processor farms

Switching network

Detectors

Lvl-1

HLT

40MHz

100KHz

100Hz

Remember, our Lvl1 is not the LHC Lvl1... ours is before digitizationHLT is no solution

CMS

Hence: FVTX has Lvl1 trigger “hookup” for displaced vertex triggersother upgrade is a trigger to begin with (W->muon)

30

u d W

d uW−

u u dd

ν μ /e

μ/eW

ALW

= σ−σ

σ +σ

Δ d x1 u x2 −Δu x 1 d x2u x 1 d x2d x1 u x2

Similar expression for W- to get Δῡ and Δd…

Since W is maximally parity violating large measured Δu and Δd require large asymmetries.

W Production Basics

No Fragmentation!

31

MuonTrigger and RPC upgrade

RPCs

μμ+/-+/-

uu

dd

ν μ /e

μ/eW

W physics with polarized

protons

• Trigger will allow to enhance the sample if high-momentum/straight-track muons

• RPC adds timing to reduce large background from non-collision muons (beam, cosmics)

FVTX

• Fitted track provides a DCA to the primary vertex (measured by central arm barrel VTX detector)

prompt

Pinpoint the decay vertexto eliminate backgrounds!

Endcap detects the following by displaced vertex (∆r, ∆z) of muons: D (charm) μ + X B (beauty) μ + X B J/ ψ + X μ+ μ-

21-October-2008 Jon S. Kapustinsky 2008 IEEE NSS-MIC Dresden 33

FVTX Section View

80 cmBarrel

• 4 discs of Si sensor in acceptance of each Muon Arm • Microstrips to accurately measure R coordinate of track

• Scheduled to be installed in FY11

Two endcap halves½ of one endcap

½ disks