Theme 7 Group 1

Goal of Workshop

To develop a joint Sino-Australian proposal to ERIDANUS (波江座) for RFI mitigation in Radio Astronomy data using Big Data approaches.

Workplan

• Motivation

• Target

• Tentative solutions

Motivation

• RFI : a generic problem for radio astronomical data analysis...

• Challenges• Very high sensitivity for SKA, MWA, ASKAP, FAST, …• Appearing in many steps of the SKA processing system • For the SKA it would be nice to have (near) real-time requirement

• Useful for • Imaging pipeline • Non-imaging pipeline • Identifying local plant equipment generating RFI

RFI in IMAGINELocal RFI

RFI in IMAGINE

RFI in pulsar data

narrow and wideband RFI in pulsar data

RFI sources Specific mitigation

techniques

Common

techniques

Man made

interference

Local plant equipment Knowledge base Pattern recognition,

machine learning, esp.

deep learning

(parameter/feature

models)

Pseudo-random

sources (e.g., military,

local planes, cameras

etc.)

Anomaly detection,

data mining

Regular sources (e.g.,

planes/satellites, etc.)

Knowledge base

Pattern recognition

Natural

interference

Environmental / space

noise (e.g., lightning,

the sun, mice, etc.)

Data mining

Knowledge base

RFI categorization

ADC FIR+FFT BeamformingBands

FIR+FFTBeams Channels

Corrections

Correlate

Baselines

VisibilitiesRFI mitigation

Initial calibration & A-team removal

VisibilitiesRFI mitigationGriddingDirty image

ImagingSources

Sky modelSky image

De-gridding

Data Collection

Central Signal Processing

Imaging pipeline

Finer frequency channels Time-frequency Source fusion

SDP subsystem

Station subsystem

Calibration

De-dispersionSingle PulseFFT

Single pulse RFI mitigation

Periodic RFI mitigation

detectionSift/ general RFIPulsars / FRBs

RFI mitigationNon-imaging pipeline

Data and Methodology

• Data Methods• Unified architecture —— Big data technique, machine learning, data mining,

Spark, Keras, Tensorflow… for SKA Low and Mid

• Data available from telescopes• Interferometer——LOFAR, MWA, MeerKat , ASKAP, jVLA, ATCA

• Single Dish —— FAST (sensitivity same as SKA), Parkes, Arecibo,

Lessons from different telescopes for SKA

Research topics

• Building knowledge base based on different RFI sources

• Feature models by Deep Neural Networks (DNNs) (GPU, (Keras)Tensorflow/Theano/MXNET/Torch/Caffe(CNN))

• Machine learning, data mining, pattern recognition techniques for RFI detection

• Parallel computing in Spark platform

• Science use cases (imaging and non-imaging)

• …

Unified architecture

Longer Term Goals

Write two papers:1. The development of a database to be hosted at Fudan University (复旦大学) to characterise RFI

2. Overview paper of the existing approaches to RFI and why our approach will deliver better results for the community

Develop a joint proposal to ERIDANUS to use Big Astronomy Data techniques to mitigate RFI in radio astronomy data. This includes:• Pulsar data• Spectral line and Continuum dataThis will work on single dish and interferometers

Deliverables

• Data Files with Spark/Keras/…. Format (meta data)

• RFI knowledge base

• Feature models learnt by deep neural networks

• Machine learning/data mining/pattern recognition techniques suitable for RFI detection

• Computing platform on Spark (CPU-based) and DL (GPU-based) platform

A unified architecture for RFI mitigation especially for SKA Low and Mid

Conclusion

• We feel this is a problem that Big Data approaches including Machine Learning, Deep Learning and Adaptive Learning can help mitigate.

• Over the last three days of this workshop our group cannot be accused, in any way, of floccinaucinihilipilification about the workshop.

Thank you -谢谢

Kevin's important lessons learnt

1. When using Google translate remember to switch from English to 普通话, to 普通话 to English when translating menus

2. For a Vegetarian being able to say 我是一個素食者 is really handy at Restaurants

3. Chinese food somehow tastes better in China

4. Tones are important Wěi wěi is 炜玮's name, but said with a different tone wèi wèi (喂喂) it means "hello, hello" and is someone testing a microphone not someone calling him from the other side of the room