Upload
nguyenthien
View
219
Download
6
Embed Size (px)
Citation preview
Concepts for Detector Data AcquisitionRyan HerbstDepartment Head, Advanced Electronics Systems
Trigger/Streaming Readout Workshop - January 27, 2017
SLAC TID-AIRTechnology Innovation Directorate
Advanced Instrumentation for Research Division
TID-AIR
2
Outline
Trigger/Streaming Workshop - January 27, 2017
● SLAC Overview● Detector Front End
○ PGP & RSSI (Reliable UDP)● Inline Data Processing
○ RCE and Its Applications● Detector Data Handling● Detector-DAQ Interface
○ Timing Interface○ Readout
● Online Analysis● Accelerator Controls● Software
3
Overview of SLAC Structure
LSST
ATLAS
Business &Operations
Fermi-GLAST Gamma-Ray Telescope
LCLS
Biosciences, Chemical Science, Computer ScienceElementary Particle Physics, High Energy Density ScienceMaterial Science, Particle Astrophysics & Cosmology
Trigger/Streaming Workshop - January 27, 2017
1.65 m
4
SLAC’s Core Instrumentation R&D Capability in AIR provides Systems & Components for SLAC & National/International Projects
Technology Innovation Directorate
• Provides engineering for SLAC directorates
• All SLAC developed LCLS/LCLS-II detectors
• HEP: ATLAS, CDMS, HPS, etc.
• LSST, CMB, Fermi …• Non-SLAC Projects
LCLS Directorate
SSRL Directorate
Science Directorate
LCLS-II Project
LSST Project
Accelerator Directorate
Super CDMS Project Advanced
Instrumentation for Research
(AIR)Division
Gunther Haller
LSST: World’s largest digital camera
Work for OthersEuXFEL, KEK,
LANL, commercial companies incl. start-ups, etc.
Trigger/Streaming Workshop - January 27, 2017
TID-AIR
5
Accelerator & DAQ Overview
Trigger/Streaming Workshop - January 27, 2017
DetectorFront End
Inline Processing DAQ
Gateway
OnlineAnalysis
OnlineData Cache
To Offline
LocalTimingGateway
AcceleratorControls & Timing
● Items in blue should be provided by facility● Allows for a common set of APIs
○ Detector control○ Scripting API○ Online analysis plug ins○ Offlines analysis APIs
TID-AIR
6
Detector Front End
Trigger/Streaming Workshop - January 27, 2017
DetectorFront End
Inline Processing DAQ
Gateway
OnlineAnalysis
OnlineData Cache
To Offline
LocalTimingGateway
AcceleratorControls & Timing
● A variety of detectors must be supported by facility DAQ system
● Helps to have a small set of standard interfaces
○ PGP (SLAC)○ Aurora (Xilinx)○ Cameralink○ USB○ Ethernet
TID-AIR
7
Detector Front EndKPIX ASIC For ILC Calorimetry & Tracking
Trigger/Streaming Workshop - January 27, 2017
KPIX ASIC bump bonded to Si sensor (6 inch wafer)1024 pixel sensor
Readout motherboardsupporting 30-layers1Gbps Ethernet optical readout
Readout flex cable bonded to Si sensorfor calorimeter
● Power pulsing● Auto ranging● Timestamped analog memory for bunch interactions● Designed for bunch structure with multiple pulses
followed by readout cycle during idle time
Test beam single electron shower event
Minimal interface:CLK, RST, CMD, DOUT
TID-AIR
8
Typical Front End Firmware
Trigger/Streaming Workshop - January 27, 2017
Reg Control(Master)
AXIL Crossbar
ADC Readout(Slave)
ADC Config(Slave)
Stream Tester(Slave)
EPIX Regs
(Slave)
Microblaze(Master)
SPI
LVDSPGPOr
Ethernet
SACI
VC
ASIC Acquisition
Control
To ASICs
ASIC Readout Control
VC
Embedded processor allows for automatic ADC & detector sample point tuning
High speed link for data path & timing/event delivery. Standard facility interface.
Camera specific data processing, reorganization & compression (HLS/Matlab)
Standard reusable blocks for accelerated development
Common flexible ASIC configuration interface
TID-AIRPGP2B
Trigger/Streaming Workshop - January 27, 2017
• PGP = Pretty Good Protocol- PGP2 = Version 2- PGP2B = Version 2 with AXI-Stream interface (compatible with PGP2)
• Architecture independent – can be deployed on any 8B/10B SERDES- Copper or optical- Speed defined by SERDES and internal data path
• Unlimited frame size• 4 virtual channels, each with separate firmware interface
- 4 frames in flight at any given time- Avoid head of line blocking for configuration messages
• Built in flow control support• Cell based protocol
- Large frames segmented into 512byte cells for transport- Guaranteed cell ordering- CRC protected cells, errors forwarded to end point - Per VC arbitration at the cell level
• Low overhead (96% efficient after 8B/10B conversion)• Can be used to transport timing and trigger
- Low latency, deterministic trigger transport interface• Unidirectional & bidirectional protocol
- Example: 1 downstream link and 4 upstream links- Upstream and downstream links can be different line rates
• Supports lane bonding for wider data path• Standard interface for many experiments
- LCLS, SID, LSST, ATLAS & Others
SLAC PGPCard4 lane x 3.125GbpsFull duplex 9.6Gbps
TID-AIRRSSI
Trigger/Streaming Workshop - January 27, 2017
Application FPGAApplication FPGAIOC CPU
• Reliable SLAC streaming interface• Reliable message delivery over UDP: RSSI
• Firmware to firmware as well as firmware to software• Based upon RUDP protocol
- RFC-908, RFC-1151, draft-ietf-sigtran-reliable-udp-00• Error detection and retransmission• Flow control support• Additional features added to support flow control• Facilitates breakup of large transfers into MTU sized messages
IP/UDP
Ethernet Switched Network
RSSI
FirmwareApplication
IP/UDP
RSSI
FirmwareApplication
IP/UDP
RSSI
SoftwareApplication
TID-AIR
11
Inline Data Processing
Trigger/Streaming Workshop - January 27, 2017
DetectorFront End
Inline Processing DAQ
Gateway
OnlineAnalysis
OnlineData Cache
To Offline
LocalTimingGateway
AcceleratorControls & Timing
● Inline data processing for larger cameras and complex front ends
○ Additional level of processing between detector front end and back end DAQ
○ Can provide data reorganization or image pre-processing
○ Can generate trigger decisions for trigger based systems
TID-AIR
12
Optional Inline Data ProcessingHigh Density / High Performance Processing In ATCA
On board 10G Ethernet switch with 10G to each processing FPGASupports 14 slot full mesh backplane interconnect!
Data processing daughter board with dual Zynq 045 FPGAs
(modular for camera integration)
Front panel Ethernet2 x 4, 10-GE SFP+
Application specific RTM for experiment interfaces 96 High Speed bi-dir links to SOCs
Trigger/Streaming Workshop - January 27, 2017
SOC platform combines stable base firmware / sw with application specific cores
● HLS for C based algorithms & compression
● Matlab for RF processing
Deployed in numerous experiments● LSST● Heavy Photon Search● LSST● Dune● ATLAS Muon● Koto
High performance platform with 9 clustered processing elements (SOC)
● Dual core ARM A-9 processor ● 1GB DDR3 memory● Large FPGA fabric with numerous
DSP processing elements
13
DUNE 35-Ton
• SLAC RCE Platform is readout for 35Ton detector• Fully integrated with timing system• Integrated with ARTDAQ backend system• Successful deployment of zero suppression algorithm
- Written in C++ via Vivado HLS design flow- Support for 128 channels with 25% FPGA utilization- Fully adjustable to optimize performance depending on
S/N conditions- Successful handling of data quality issues
Noisy front end data links Cold ASIC stuck bit problem
Trigger/Streaming Workshop - January 27, 2017
14
Heavy Photon Search
Hybrids:Pulse shapingPulse sampling
Buffering
FE boards:Amplification
Analog to digitalHybrid control/power
DAQ PlatformSLAC RCE (ATCA blade)
JLab DAQ
Flex cables:Impedance controlled, low mass signal/bias/control to
hybrids
Power suppliesLow voltageSensor bias
• Integrated with JLAB’s timing and back end DAQ system (CODA)
• Took data during 2015 & 2016• Expect more data runs in 2018
- Add new layer 0 hybridsNew layer 0
hybrid design for 2018 run
Trigger/Streaming Workshop - January 27, 2017
TID-AIR
15
Planned Upgrades To RCE Platform
Upgrade current 24 port 10G switch to 96 port 40G capable switch.Support 10Gbps or 40Gps to DPMsSupport 120Gbps front connectionCost reduction and lower power
DPM Upgrade:Upgrade Zynq-7000 to Zynq Ultrscale+ MPSoC
3 layers of processing, CPU, RPU & GPUAdditional processor memory up to 32GBAdd direct attached memory to Fabric (Collaboration with Oxford)
Cost reduction re-spin of COB coincides with core switch upgrade. Less layers & component cost optimization.
Trigger/Streaming Workshop - January 27, 2017
TID-AIR
16
Detector Data Handling
Trigger/Streaming Workshop - January 27, 2017
DetectorFront End
Inline Processing DAQ
Gateway
OnlineAnalysis
OnlineData Cache
To Offline
LocalTimingGateway
AcceleratorControls & Timing
● Online data cache provides temporary storage for user experiment data
○ Solid state low latency storage○ Cumulative data > 210GBps
● Allows for first level analysis soon after data taking
● Allows users to filter data before moving to costly long term retention or transferring to home institution
Large cameras at high data rates create data handling challenges
● 1Mpixel @ 100Khz ~ 210GBps
TID-AIR
17
Data Path Processing
Trigger/Streaming Workshop - January 27, 2017
Detector Segment
Detector Segment
Detector Segment
DAQ Node
Detector Segment
DAQ Node
DAQ Node
DAQ Node
AcceleratorControls & Timing
Timestamped beam conditions
Detector segment data
Event data stored in HDF5 format
Event reconstruction & analysis
PC based flash storage allowing direct DMA from receiving blade & low latency data access
TID-AIR
18
Detector - DAQ Interface & Timestamps
Trigger/Streaming Workshop - January 27, 2017
DetectorFront End
Inline Processing DAQ
Gateway
OnlineAnalysis
OnlineData Cache
To Offline
LocalTimingGateway
AcceleratorControls & Timing
● Tight coupling to timing system required to properly timestamp detector data
○ Method for applying timestamps depends on the type of detector
○ Custom cameras are more flexible○ Commercial cameras come with
challenges○ Feedback path for multiple camera
coordination (busy, almost full, etc)
TID-AIR
19
Methods Of Timestamping Data For A Variety Of Detector Types
Trigger/Streaming Workshop - January 27, 2017
Three models of sensor integration:
1) CommercialSeparation of trigger, data, and timestamp path. Minimal support for busy, almost full. Data & timestamp combined at software level.
2) Custom-IPaths joined by an interface card. Timestamp added in hardware as data arrives at PC. PC card provides buffering, generates almost full etc. Camera uses facility approved protocol.
3) Custom-IISensor receives timing directly. Sensor contains buffering and provides almost full directly to trigger system. Data read out with custom PC card. Camera uses facility approved protocol.
Fast Control& Timing
Sensor CPU
TTL Timestamp
Data
TID-AIR
20
LCLS-2 PGP Interface CardFor Custom Cameras With Fiber Interface
Trigger/Streaming Workshop - January 27, 2017
Dual QUAD SFPs for 8 lanes of up to 6.125Gbps
(upgrade to 12Gbps)
LCLS-1 or LCLS-2Timing Interface
● PCI-Express card supports bi-directional links to camera front end○ Hardware handles protocol layers to reduce CPU loading
● Supports bi-direction timing interface supporting busy and trigger feedback○ Timing receiver firmware built into FPGA
● Timestamp and trigger forwarded down optical link with PGP○ Timestamp applied directly on camera
● Firmware can also apply trigger on incoming data○ Gate event acceptance based upon event code / trigger information
● Optical interface supports a number of detector protocols
○ PGP (SLAC)○ Aurora○ CameraLink○ Raw Ethernet○ User Custom!
TID-AIR
21
LCLS-1 / LCLS-2 Timing ReceiverFor Commercial Cameras With TTL Triggers
Trigger/Streaming Workshop - January 27, 2017
LCLS-I Timing input (119MHz)
LCLS-II Timing input (186MHz)
TTL Outputs
Receive andtransmit (feedback)
Feedback inputs can be forwarded as “almost full” or L1 Trigger inputs.
Presumes sensor can generate “almost full” and L1 Trigger signals. Sensor “almost full” could otherwise
be modeled but not guaranteed.
FeedbackInputs
Card developed for use in LCLS-I and LCLS-II controlsto interface to non-SLAC electronics.
TID-AIR
22
Online Analysis
Trigger/Streaming Workshop - January 27, 2017
DetectorFront End
Inline Processing DAQ
Gateway
OnlineAnalysis
OnlineData Cache
To Offline
LocalTimingGateway
AcceleratorControls & Timing
● Online analysis is extremely important● Provides confidence that experiment is
taking proper data● First feedback about timing, noise and data
quality● Large, high rate detector data bandwidth
creates a challenge
TID-AIR
23
Online Analysis Coordination MinimizesAnalysis Load & Network Traffic
Trigger/Streaming Workshop - January 27, 2017
● Facility based online analysis uses local timing system to coordinate online analysis
○ Event codes direct readout machines to send data for analysis
○ Minimizes network traffic and analysis load
● Analysis engines define rules for event selection at user request
● Timing system tags events for analysis by online nodes
● User API allows experiment specific analysis
○ Facility supports generic analysis types
TID-AIR
24
Accelerator Controls
Trigger/Streaming Workshop - January 27, 2017
DetectorFront End
Inline Processing DAQ
Gateway
OnlineAnalysis
OnlineData Cache
To Offline
LocalTimingGateway
AcceleratorControls & Timing
● A flexible accelerator controls platform ensure long term maintainability and ease of facility upgrades & expansion
● A common platform architecture for multiple controls sub-systems reduces both development costs and maintenance costs
TID-AIR
25
SLAC High Performance Accelerator Controls
Trigger/Streaming Workshop - January 27, 2017
Analog / RFCommon Platform
Architecture
● TID-AIR developed an ATCA based common platform for accelerator controls○ LCLS-2, LCLS-1 and SSRL○ Combines high performance digital processing with low noise RF design
LCLS-II High Performance Controls Systems● Beam Position Monitor (BPM)● Bunch Charge Monitor (BCM)● Bunch Length Monitor (BLEN)● Machine Protection System (MPS)● Timing Delivery System
LCLS-I Controls Upgrade Systems● Beam Position Monitor (BPM)● Bunch Charge Monitor (BCM)● Bunch Length Monitor (BLEN)● Low Level RF (LLRF)
Other Systems & Experiments● SSRL Low Level RF Upgrade● Transition Edge Sensor Detector
TID-AIR
26
ATCA Based Platform ForLCLS-I & LCLS-II High Performance Controls
Trigger/Streaming Workshop - January 27, 2017LCSL-2 BPM board
LCLS-1 LLRF Down Convert
ATCA AMCCarrier CardSupporting 2Analog AMCs
ATCA provides the space, power & cooling required for LCLS-2!
10Gbps EthernetBackplane
TES Readout Electronics
Trigger/Streaming Workshop - January 27, 2017 27
• Each TES element is coupled to a superconducting resonator with a SQUID that results in a resonant frequency dependent on current• Continuous current ramp induces sinusoidal frequency driver• TES sensor current modifies the frequency vs ramp current
• “Comb” of frequencies sent down to TES array• Triplett of frequencies per TES sensor
- One below and above resonant frequency for reference
• ATCA based readout electronics generates comb of frequencies• 1 at TES resonant point (dip in received
frequency)• 2 sideband frequencies for reference
• Design goal of 4K channels per FPGA• 12K total tx and rx frequencies (resonant
plus 2 reference)
EvalBoardManagement
Tree
Rogue
● Provide a system to facilitate hardware development and daq systems for interfacing to hardware
● Support a number of interface technologies between hardware and software
○ Including ones that don’t yet exist● Easy to understand mechanisms for connecting
independent modules together using a set of well defined, easy to understand interfaces
● Allow data paths to exist in independent high performance threads
● Flexible structure for creating a hierarchy of system components
○ Independent of the network and hardware hierarchies
● Direct interface to hardware development○ Generate variable definitions directly from
VHDL/Verilog● Non-professional software engineers must be able
to maintain the system○ Both core rogue components and application
software○ Easy to add functions, device control routines
and run control sequences
PGPVC0
SRPV0
AxiVersion
PGPVC1
AxiPrbsTx
PGPVC2
dataWriter
PRBS Rx
Register Access
PRBSData
MicroBlaze Console
Con
figur
atio
n S
tream
PyrogueMesh EPICS
GUIs Scripts
OtherServers
EPICSClients
RunControl
Lines in red are pure C++ threads & data paths
TID-AIR
29
The End
Trigger/Streaming Workshop - January 27, 2017