An FX software correlator for VLBI
Adam DellerSwinburne University
Australia Telescope National Facility (ATNF)
Outline
• Function of a correlator
• Reasons for software correlation
• Code and platforms
• Progress and current performance
• Immediate and near future plans
• Facilitation of real-time gigabit eVLBI
Correlator functionality
• Averaging the product of signals at two telescopes gives visibility
Correlator functionality
• Averaging the product of signals at two telescopes gives visibility
• To get frequency information, average additional multiplications by lagged signal, then FFT (or reverse for FX)
• What does it all boil down to? Gigabit interferometry, 1024 channels with
6 telescopes, FFT cost ~ 200 Gflops (FX)
Why software correlate?
• Flexibility Handle disparate file formats and
bands High spectral resolution + high b/w Real time fringe checking
• Rapid development• For Australian LBA
8 x bandwidth, using disk-based recorders instead of S2 tape system
Correlator code structure
• Developed in C++
• Disk read/write threaded to hide latency
• Currently using IPP for vector mathematics
fxcorr
Telescope Baseline
Mode InputStream
Platform options
• Initial development to run on Swinburne cluster: 200+ 3GHz Pentium 4 machines
• Cray XD-1 chassis: 12 AMD Opteron processors, 6 Xilinx FPGAs
Hybrid Architecture
• PC flexibility with FPGA power
• FPGA used for unpacking/fringe rotating and FFT
• Possibly incorporate cross-multiply• FPGA code compiled into library,
available as C subroutine call• Cray contributing via development
of FPGA subroutine library
Milestones to date
• ‘Basic’ and ‘complex’ FX algorithms implemented and verified
• Fortran driver for CALC ported to C for delay modeling
• Used in search for first trans-Tasman fringes to 6m antenna in NZ
• Range of support/analysis packages created
Correlator output
Current performance
• LBA: Complex correlation, 6 telescopes, 2x 16MHz bands (128 Mb/s), 4 products/band, one node of cluster: 512 channels: 21x real time 2048 channels: 25x real time 32786 channels: 32x real time
• 15 telescopes, 128 Mb/s, 4 products 512 channels: 88x real time
Immediate applications
• Improved LBA sensitivity: parallax of double pulsar J0737-3039
• High time and frequency resolution + high bandwidth: Wide field imaging
• Masers, SNRs, AGNs…NGC253 image credit:Emil Lenc (Swinburne)
Real-time gigabit eVLBI?
• All ATNF telescopes of LBA capable of 1 Gb/s, others 512 Mb/s
• Technical/cabling issues to overcome: disk-based recorders
• Software correlator ready now• Simulated Gb/s data, 512 channels
<160x real time Current system has disk transfer
limitation
In the near future…
• Fibre links overcome disk limitations?• Investigate alternate architecture:
‘Cell’ processor array at VPAC• Code improvements:
Sophisticated pulsar gating Polyphase filterbank alternative to FFT Automated configuration from schedule,
GUI frontend, output FITS compatible…
Integration into the LBA
• Available for use from now on a limited basis
• Final release planned before March 2006 observing session
• Will incorporate automatic configuration, GUI frontend etc
• Feedback on desired features encouraged
Applications outside the LBA
• Potentially useful for niche applications anywhere
• Applicable to connected-element as well as VLBI• “Piggy-backing” on existing interferometers - only need facility to get data to disk, and some processors
Existing correlator limited + disk space
available + processors = SOFTWARE CORRELATE
Conclusions
• Fast, flexible new FX software correlator for generic clusters developed
• Hybrid architecture will be explored• Will provide significant increase in
LBA sensitivity until next upgrade• Real-time gigabit eVLBI using the
software correlator is feasible on Swinburne cluster
• Final release planned in early 2006
Questions??
XD-1 Specifications
• 12x 2.2GHz Opteron processors - 106 Gflops total
• 1-8 GB RAM per processor, 12.8GB/sec• 4-8 GB/s interconnect (between nodes)• 6 Xilinx Virtex 4 FPGAs• 4 PCI-X slots for ethernet/fibre channel• Max 1.5 TB local storage