XFEL Large Pixel DetectorDAQ

Project TeamTechnical Team:

STFC Rutherford DAQGlasgow UniversitySurrey University

Science Team:UCLDaresburyBath Universityothers …

Project Outline

1) Phase 1: Develop a digitising pipelined XFEL detector (1k by 1k pixels)

• 3Y Project given approval Dec 2007

2) Phase 2: Construct complete XFEL instruments as required before 2013

- mass produce electronics- match XFEL DAQ

Phase 1 Detector 1M Pixel

4 x 4 Super Modules

Module Concept

XFEL ASIC

New design matched to the XFEL:

- Time Structure- Dynamic Range- Channel Count- System Interfaces etc.

XFEL ASICPreamplifiers

Dynamic range stages

Pixel Resetting

Power supply Conditioning

Deep pipeline memory (786,432 samples)

IO to DAQOverload Control

Sequencing and control

ADC Stages

Power supplies

Store in Pipeline during bunch trainReadout during long gap

XFEL Structure

600 s

99.4 ms

100 ms 100 ms

200 ns

FELprocess

X-ray photons100 fs

Electron bunch trains; up to 3000 bunches in 600 sec, repeated 10 times per second.Producing 100 fsec X-ray pulses (up to 30 000 bunches per second).

XFEL ~ 30 000 bunches/sbut

99.4 ms (%) emptiness

Data Sampling to Memory Serialise and Transmit to DAQ

Multi-gain concept

Required dynamic range compression

– Experience with calorimetry at CERN

– Relaxes ADC requirements

– Fits with CMOS complexity

On detector electronics

Detector Summary

•Phase 1– Common super module design– Economic mass production– Eases test and maintenance– Scalable DAQ

•Phase 2– Mechanical design– Large scale replication– Industrial technology

Tracker View showing Higgs decay to 4 muons

LHC Example

The CMS Tracker• ~210 m2 of silicon, 10M channels• 75000 FE chips, 40000 optical links• 15000 modules mass produced using automatic

assembly techniques• Hybrids and assembly at CERN, FE ASIC Design at RAL

Radiation environment ~10Mrad ionising~1014 hadrons.cm-2

Inner barrel layerRod insertion

Petal assemblyCERN assembly

APV25

>500 cards >20,000 BGAs

• Collaboration with Imperial College and CERN• Massively Parallel Processing ~ 30 VME crates• 10 TERA-bits / sec• 15 Exa-Bytes of raw input per year!

The CMS Tracker DAQ

Project Management

•One overall project manager– Report to XFEL and provide information as

required

•Each workpackage– run as subproject in our QA system

•Approved ISO9000 system– Formal design review processes– Drawing and record control

The First 3 Years:

•Included– Sensor proving tests at LCLS– XFEL ASIC development– Build and test of 1Mpixel system

•Excluded– Off detector DAQ– Sensor R&D

Work Packages

WP1: SensorsWP2: Front End ElectronicsWP3: Mechanical DesignWP4: On Detector ElectronicsWP5: Data AcquisitionWP6: Software, controls and

integration

WP5: Data Acquisition

Leader – John Coughlan

DAQ ApproachUp to now effort has been concentrated on WPs for Mechanics / Sensors / ASICs rather than Readout.

We intend to exploit Commercial Off The Shelf (COTS) based equipment where practical (e.g. FPGA Development boards, vendor/commercial FPGA cores)

Xilinx (Virtex 5) System on ChipRocket IO serial data links , Embedded Ethernet coresEmbedded PowerPCs, Fast memory interfacesEmbedded Development Kit

Industry standards for interface protocols vs custom(GEthernet, PCIe, sFPDP …)

Develop a Scaleable system. Final detectors.

DAQ XFEL IntegrationStay Flexible to adapt to FE ASIC and common XFEL DAQ Architecture specifications.

DAQ/Timing/Trigger Interface Standards & Protocols need to be agreed with XFEL DAQ group.

STFC/Rutherford has a long history of successful partnerships with DESY on Particle Physics projects (e.g. H1 DAQ)

DAQ ExperienceSTFC has a large established base of DAQ hardware and firmware expertise (wide variety of projects… Particle Physics, Xray, Neutron).

STFC has a proven track record on the delivery and long term support of large scale readout systems (e.g. H1 DAQ, CMS Silicon Tracker)

DAQ On Detector

3 Year Plan includes only elements local to detector.

Module Support Cards (MSC) : FPGA Gain Selection

COTS : FPGA Dev BoardsXilinx Virtex 5 + EDK PPCCOTS : FPGA-> PC cores (Qx UDP)

FPGA

Electrical LVDS Links

Switch Inputs / Farm

FEMs : FirmwareData FormattingSample selectionTraffic Shaping

Data Rate Challenge

1 M Pixels x 512 x ~ 2 bytes x 10 Hz ~ 10 GBytes/secProtocol overheadsFixed length fragments?Data Selection, Sparsification?

=> 1 TeraByte recorded every 2 minutes !

Modularity 1 MPixNr Super Modules 16

Nr Hybrids 256

Nr Module Support Cards 32

Nr Front End Modules? 16 x 8 links

Nr Links? (GbE) 80 MB/s 128

x N MPixel x N

Each FEM Fragment = 128 KB (64 MB / train)128 KB x 512 x 10 = 640 MB/s @ 80 MB/s link = 8 links / FEM

DAQ EoI Off Detector

E.g. ATCA crate for Surface & Nuclear Science AGATA Daresbury & PadovaATCA card withFPGA->PC PCIe readout

Event BuilderAdvanced TeleComms ArchitectureCOTS Carriers + AMC MezzaninesOr MicroTCA

6 TB SFPDP Disk Storage and Servers

WP6: Software, controls and integration

Leader – Tim Nicholls

Software, Controls & Integration

Delivery and Operation of Detector in beam-line environments.

System Operation Timing and Controls.

Software Development and Integration with Beam-line scientists.

Real Time Data Monitoring and Analysis.

Team of Integration engineers, leader with experience on DESY H1 experiment.

Summary

1) Phase 1: Develop a digitising pipelined XFEL detector (1k by 1k pixels) 3Y starting Jan 2008

2) Phase 2: Construct complete XFEL instruments as required before 2013

- mass produce electronics- match XFEL DAQ Architecture