View
213
Download
1
Category
Tags:
Preview:
Citation preview
A. Homs-Puron, NOBUGS 2008, Sydney, 3-5 Nov 2008
Fast 2D Detector Control at ESRF beamlines
David Fernández Alejandro Homs
Emmanuel PapillonManuel Pérez
Vicente ReyCyril Guilloud V. Armando SoleAssen Kirov
A. Homs-Puron, NOBUGS 2008, Sydney, 3-5 Nov 2008
Talk outline
•Introduction
ESRF FReLoN CCDs
2D control arquitecture
•Espia card based systems
Espia card
FReLoN 2k, Maxipix
Dalsa, Sarnoff, Aviex
•Latest developments
Driver improvements
Data saving
Common CCD acq. platform
A. Homs-Puron, NOBUGS 2008, Sydney, 3-5 Nov 2008
FReLoN 1000 (1996):
• 1024 x 1024 - 14 bit• 4 parallel readout channels• Readout time: 100 ms
FReLoN 2000 (1998):
• 2048 x 2048 - 14 bit• Readout time: 220 ms
FReLoN – Fast Readout & Low Noise
A. Homs-Puron, NOBUGS 2008, Sydney, 3-5 Nov 2008
FReLoN 1000/2000 acquisition setup
Gigabit Ethernet
TACO Device serverTACO Device server
SPECSPEC
A. Homs-Puron, NOBUGS 2008, Sydney, 3-5 Nov 2008
Princeton Sensicam, PCO Photometrics Matrox Meteor II [H. Witsch, J. Meyer] Trixell Pixium [L. Claustre] Andor ImagePro [A. Gotz] Direct-Show
Other CCD-like devices supported
Mar CCD [L. Claustre] IDS Falcon [D. Miraut, J. Meyer] Firewire (IEEE 1394) [A. Gotz] Dalsa Dalstar
A. Homs-Puron, NOBUGS 2008, Sydney, 3-5 Nov 2008
ADSC Q315r for MX beamlines
Gigabit Ethernet
Local Gigabit
A. Homs-Puron, NOBUGS 2008, Sydney, 3-5 Nov 2008
Designed to interface the FReLoN 2k Collaboration ESRF + SECAD, S.A. 2 Gbps fiber optic link (> 100 m) PCI 64 bits / 66 MHz ~180 MB/s maximum data rate Bus master + Scatter Gather + IRQ Driver source code available rewritten!
Can write on 32-bit addresses only (4 GB)
Espia card
A. Homs-Puron, NOBUGS 2008, Sydney, 3-5 Nov 2008
Control workstation (2005)
Transtec (2005): Dual Intel Xeon 2.6 GHz, HT 2 GB ECC DDR-RAM 3 x 73.4 GB Ultra320 SCSI 15Krpm SuSE 8.2 + Linux 2.4.20-64GB-SMP
• Remote NFS through 1 Gbps link 90 MB/s• 2 disks in software RAID 0 100 MB/s (8 MB files)• 4 disks in software RAID 0 150 MB/s (1 GB files)
for ultra-fast tomography
A. Homs-Puron, NOBUGS 2008, Sydney, 3-5 Nov 2008
New FReLoN Generation
FReLoN 2k (ATMEL): No Mux – only one fiber optic! Frame Transfer Mode 2048 x 1024 @ 16 fps 66 MB/s
FReLoN 4M (KODAK): Only 4 channels – Full Frame Mode
FReLoN HD (PSB-2): 2048 x 1024 @ 32 fps ~125 MB/s
A. Homs-Puron, NOBUGS 2008, Sydney, 3-5 Nov 2008
• Connects to the Espia• 2 x Camera Link connectors:
• Base configurationMultiplexed• Multiple pixel packing formats
• Test image generator @ ~180 MB/s
Camera Link: FOCLA
A. Homs-Puron, NOBUGS 2008, Sydney, 3-5 Nov 2008
Frame transfer technology 1024 x 1024 @ 60 fps 120 MB/s 1 x Camera Link [Base]
Connector Compact! Image not reconstructed FOCLA does reconstruction
Dalsa Pantera 1M60
A. Homs-Puron, NOBUGS 2008, Sydney, 3-5 Nov 2008
Dalsa setup
Fast online displayImage reconstruction
A. Homs-Puron, NOBUGS 2008, Sydney, 3-5 Nov 2008
512 x 512 @ 300 fps 150 MB/s 512 x 128 @ 500 fps 125 MB/s 2 x 8 ADCs – 12 bit 2 x Camera Link [Base]
Connectors Separate RS-232 serial line
Sarnoff CAM512
A. Homs-Puron, NOBUGS 2008, Sydney, 3-5 Nov 2008
Sarnoff setup
Camera Link
Fiber optic
RS-232
A. Homs-Puron, NOBUGS 2008, Sydney, 3-5 Nov 2008
Sarnoff results (2007)• 300 fps• 75 frames• 250 ms• 40x
slowdown
No sound! B&W Flickering
(room light)
ID15: Tomography
in 0.5 sec @ 500 fps
A. Homs-Puron, NOBUGS 2008, Sydney, 3-5 Nov 2008
The real image
Espia #1
Espia #2
A. Homs-Puron, NOBUGS 2008, Sydney, 3-5 Nov 2008
Image reconstruction in kernel
Espia #1
Espia #2
A. Homs-Puron, NOBUGS 2008, Sydney, 3-5 Nov 2008
“Very High memory” support
Physical Memory
Low RAM0 MB
3.2 GBPCI
Very high RAM
800 MB
4 GB
5.8 GB
High RAM
32 bit
boundary
• Driver always has 32-bit aux. buffers• If next target frame has 64-bit pages:
Program each Espia to write on aux buffer
Frame end (all adapters finished) copy from aux. into target buffer
Inform waiting threads• Adapters are not synchronous• Memory copy with IRQ disabled
Kernel thread
Implementation in driver
High frame rate many image buffers
A. Homs-Puron, NOBUGS 2008, Sydney, 3-5 Nov 2008
Aviex PCCD-8448
2048 x 3584 @ 1.2 fps 18 MB/s
Binning 16 x 16 8 fps
A. Homs-Puron, NOBUGS 2008, Sydney, 3-5 Nov 2008
Software image reconstruction
ADC
ADC
ADC
ADC
• Per frame operation• Warranted only once in a multi-frame (big) buffer• Callback thread safe
A. Homs-Puron, NOBUGS 2008, Sydney, 3-5 Nov 2008
Medipix2 / Maxipix
Pixel detector photon counting 256 x 256 – 13 bit 0.3 ms readout time 1000 fps 130 MB/sMax: 1400 fps 180 MB/s
Chipboard: University of Geneve PRIAM: ISG/ESRF Interfaced to the Espia
Espia
A. Homs-Puron, NOBUGS 2008, Sydney, 3-5 Nov 2008
Maxipix results (2007)
• 1000 fps !!
A. Homs-Puron, NOBUGS 2008, Sydney, 3-5 Nov 2008
Multichip MaxipixMedipix2 array 5 x 1 1280 x 256
1000 fps 650 MB/s PRIAM multiple FO links Future configs: 2 x 2, 5 x 2
A. Homs-Puron, NOBUGS 2008, Sydney, 3-5 Nov 2008
New control workstation
OPALE server from Ecrin Systems (FR):
Trenton Single Host Board and backplane Dual Quad Core Intel Xeon @ 2.33 GHz 8 GB of FB DDR2 3 x 140 GB SAS disks @ 15 Krpm
Chelsio T320 10 Gbps Ethernet controller
A. Homs-Puron, NOBUGS 2008, Sydney, 3-5 Nov 2008
Driver evolution
32-bit application 3GB address space 8GB RAM Dynamic mapping of buffersMove to 64-bit architecture No so trivial ESRF/Linux 2.1 [Red Hat Enterprise Linux 5]Kernel version 2.6.18Copy to very-high-memory: 1.3 GB/s
A. Homs-Puron, NOBUGS 2008, Sydney, 3-5 Nov 2008
Sarnoff @ ID15
512 x 512 12 bits @ 300 fps 150 MB/sWrite to local disks: 300 MB/sWrite to remote disks: 300 MB/s
NetApp FAS6070NetApp
FAS6070
10 Gbps Ethernet10 Gbps Ethernet
A. Homs-Puron, NOBUGS 2008, Sydney, 3-5 Nov 2008
“La Bestia”Dell PowerEdge 2900 Similar to Ecrin/Trenton + 16 GB RAM 4 + 6 = 10 disks 530 MB/s (800 MB/s raw) Super Sarnoff: 2x FOCLA test imageAcquisition rate: ~360 MB/s
Running
A. Homs-Puron, NOBUGS 2008, Sydney, 3-5 Nov 2008
Turbo Bestia 4 + 6 + 8 = 18 disks Single RAID 0 array 840 MB/s raw 3 RAID 0 arrays + parallel write 1.5 GB/s raw
4x
6x
8x
A. Homs-Puron, NOBUGS 2008, Sydney, 3-5 Nov 2008
Espia Next Generation
PCI-Express 8x lanes 4 fiber optic links @ 250 MB/s 1
GB/s Full access to 64-bit memory Maximum register compatibility
A. Homs-Puron, NOBUGS 2008, Sydney, 3-5 Nov 2008
Common CCD acq. Library
Reuse of common code generic procedures + interfaces Software “features” fallback if hardware has limited
capabilitiesHigh level Python implementationCommon TACO CCD interface Interfaces: implicit explicit Image processing and data reduction as plugins (C++) Extensible interface through user-defined “features”
(Maxipix) Generic TANGO interface Collaboration framework (ALBA)
A. Homs-Puron, NOBUGS 2008, Sydney, 3-5 Nov 2008
Current status
x 10 + 1[+1] (C++)
x 1 (Python)
x 20
x 5 + 1
A. Homs-Puron, NOBUGS 2008, Sydney, 3-5 Nov 2008
Conclusions
Significant number of supported 2D detectors This number keeps raising … New detectors demand high performance control Experience in high speed acquisitions
Stable low level driver (Sarnoff failed only once…)Functional high level interface
Never-ending project … (sub-projects)Common image acquisition library will help!
A. Homs-Puron, NOBUGS 2008, Sydney, 3-5 Nov 2008
Acknowledgements
SECAD, S.A. F. Hereson
Instrument Support Group:
J.C. Labiche E. Collet L. Siron J.J. Thevenin D. Pothin C. Ponchut J.M. Rigal J. Clement P. Fajardo
Computing Services:
P. Makijarvi B. Regad P. Pinel L. Garçon F. Andrei D. Gervaise S. Ohlsson B. Lebayle
A. Homs-Puron, NOBUGS 2008, Sydney, 3-5 Nov 2008
Thank you!
Recommended