View
7
Download
0
Category
Preview:
Citation preview
LOFAR offline interference detection
André Offringa1
Ger de Bruyn1,2
Saleem Zaroubi1
Michael Biehl3
1Kapteyn Astronomical Institute, Groningen2ASTRON
3Faculty of Computing Science, University of Groningen
Contents
● About interference detection algorithms● Results for LOFAR● Perspectives for SKA
About detecting interference
● Motivation for automated detection:– Enormous data sizes in new telescopes
such as LOFAR
– No possibilities for manually “browsing” the data for bad baselines, bad antennae, contaminated frequencies, or any other bad dimension of the cube
About RFI detection (2)
● Many methods for RFI removal:– Blanking before correlation at highest time
resolution
– Spatial filtering before correlation
– Flagging after correlation
– RFI modeling and subtraction before and/or after correlation
● After: e.g. fringe fitting● Before: e.g. cyclostationair filtering, satellite
subtraction
● Most of the methods are complementary
Flagging methods
Approach of automatic (baseline) flaggers:● Estimate the value of a sample using its
neighboring time/frequency samples.
● If this deviates substantially from its real value, flag it.
Flagging methods
Approach of automatic (baseline) flaggers:● Estimate the value of a sample using its
neighboring time/frequency samples.– Median value, polynomial fit, weighted
Gaussian filter
● If this deviates substantially from its real value, flag it.
– By thresholding, line detection or combinatorial thresholding
See: Bhat et al. (2005), Winkel et al. (2006), Offringet al. (2010)
Freq
Time (Offringa et al., MNRAS, 2010, in press)
Flagging methods
● Combinatorial thresholding strategy● Idea:
– Sum samples and use different thresholds
A > threshold1? → FLAG AA+B > threshold2? → FLAG A, BA+B+C > threshold3? → FLAG A, B, CA+B+C+D > threshold4? → FLAG A, B, C, DA+E > threshold2? → FLAG A, EA+E+F > threshold3? → FLAG A, E, FA+E+F+G > threshold4? → FLAG A, E, F, GB > threshold1? → FLAG BB+C > threshold2? → FLAG B, C.......
Method comparison
● Compare methods with the help of test sets and ROC curves
● One of the test sets:
(Offringa et al., MNRAS, 2010, in press)
Without flagging After flagging:
LOFAR results
● Data quality is very good
● Flagging somewhat between 1% to 3%
● No show-stopping RFI
LOFAR results
● We see some in sito RFI, but it is very weak
● Too weak to subtract– Done at the GMRT
● Effects after long time integration not yet known
(Athreya, AJ, 2009)
LOFAR results (2)
● Many different flaggers are needed
Different needs per KSP for flagger sensitivity and flagging strategy:
– EoR needs a very stringent approach
– Pulsars need an approach not based on time smoothness
– For transients the same for frequency
– Other option is to quickly scan for RFI to compress the data immediately after
SKA and RFI
SKA and RFI
● The SKA will also be influenced by RFI:– Satellites, aeroplanes
– Lightning
– In situ RFI
– Car engines, (imperfect) transmitters, electrical fences
– Classes of RFI currently not observed
● ...and more detection requirements... :– Failing hardware and software
SKA and RFI
● We are not sure yet what it takes to flag LOFAR data sufficiently
● >>5002 cross-correlations with 10 x LOFAR time resolution and 50 x LOFAR beams is not trivial, if at all possible in 2020
● The order of the data might be a crucial problem:
– Accurate flagging needs large chunks of the cubes in T/F directions
– Calibration needs chunks in the baseline-direction
SKA and RFI
● If necessary, detectors can always trade off speed for accuracy, e.g.:
– Flag on time/frequency integrated data
– Flag on auto correlations
– Flag on groups of antennae
● But is it wise to build more antennae than we can possible process?
Conclusions
● LOFAR can very well deal with its RFI● For SKA, processing will be a major issue● IO and transport might be problematic● The more antennae, the less accurate you
can deal with RFI.● Carefully watch the pathfinders...
● If anyone wants to automatically flag data... => offringa@astro.rug.nl
Recommended