Imaging and computing for PAFscsas.ee.byu.edu/PAFSKA2011/Presentations/cornwell... · 2011. 5. 27. · • Antenna A’ receives radiation by Fresnel diffraction from AB plane •

$Page 1: Imaging and computing for PAFscsas.ee.byu.edu/PAFSKA2011/Presentations/cornwell... · 2011. 5. 27. · • Antenna A’ receives radiation by Fresnel diffraction from AB plane •$
Imaging and computing for PAFs

Tim Cornwell, ASKAP Computing Project LeadAustralian Square Kilometre Array Pathfinder

Friday, 27 May 2011

PAFs for SKA May 2010

Twitter feed ASKAP_computing

Friday, 27 May 2011


Calibration

• ASKAP Front end calibrated with on-surface radiators• Takes out inter-element gain drifts

• Astronomical calibration on 1 - 5 min timescale• On field being observed - no special calibration field• 55 Jy in every field• Assemble Global Sky Model in first few weeks of observing

• Calibration not a major computing load• Except if we need to solve for and apply beam parameters

• Concerns• Beam shape stability• Bandpass stability• Polarisation solution

Friday, 27 May 2011


Dynamic range

• ASKAP requirement is 50dB• SPF:

• Possible if bright source at pointing centre• Most effects become second order

• Not demonstrated with source at e.g. half power point• PAF

• Not possible to put all bright sources at beam centre• Effects become first order

• PAF element calibration only possible at front-end• e.g. radiators on surface

• PAF beam calibration possible astronomically• but beam shape can vary

• Adaptive removal of sources possible (peeling)• New calibration approaches probably necessary

• Waiting for data from BETA and then ASKAP• Stability of the system should be a very high priority

Friday, 27 May 2011


New imaging capabilities

• Highly parallel code• Aiming for ~ 9000 cores• Necessary for 10TB/hour throughput

• Snapshot imaging + WProjection + AProjection• Wide field imaging with low memory costs• AProjection allows frequency dependent primary beam models

• Post-gridding preconditioning• To avoid multiple passes through the data

• Urvashi’s Multi-Frequency Multi-Scale deconvolution algorithm• For wide-band (300MHz) imaging

• SNR-based CLEAN• To avoid cleaning low sensitivity regions

Friday, 27 May 2011


SST2 (run9)• 30” 8 hour synthesis• SKADS model• Peak = 2.6Jy• Edge effects due to rolloff

in sensitivity

• Data set ~ 1.1 TB

• ~ 1800 CPU-hours• ~ 190 GB memory

Friday, 27 May 2011


SST2 (run9) zoomed

Friday, 27 May 2011


SST2 (run9) zoomed

Friday, 27 May 2011


Radio telescope imaging

• Spatial coherence of electric field (visibility) is Fourier transform of sky brightness

• Measures for many values of the Fourier components u,v

• Invert Fourier relationship to get image of sky brightness

• Typical problems• Incomplete u,v, sampling• Calibration• Wide field of view: no longer Fourier

transform

VA 'B = EA 'EB*

t

= e−2π jw I(l,m)e−2π j ul+vm( )dldm∫

A

BA'

vu w

Friday, 27 May 2011

• Visibility on plane A’B is Fourier transform of sky

• Antenna A’ receives radiation by Fresnel diffraction from AB plane

• Visibility between A’ and B is Fresnel/Fourier transform

• Not invertible• Use iterative algorithm

• Predict forward with high accuracy• Reverse calculation is approximate• Apply prior information e.g. sparse


Wide field imaging

VA 'B = e−2π jw I(l,m)e−2π j ul+vm( )dldm∫

VA 'B = I(l,m)e−2π j ul+vm+w 1− l2 −m2( ) dldm

1− l2 − m2∫

Fresnel diffraction

A

BA'

vu w

Friday, 27 May 2011


W projection algorithm

• Re-arrange imaging equation

• Algorithm• Given a trial image I(l,m), Fourier transform to single grid• Tabulate w• Calculate convolution function for each w• For each visibility, calculate trial value using anti-aliasing

filter• Problems

• Large number of flops• Memory for convolution function can be very large• Scales as number of visibilities

V (u,v,w) = G(u,v,w)⊗V (u,v)

G(u,v,w) = e

j2πw 1− l2 −m2 −1( )1− l2 − m2∫ e j2π ul+vm( )dldm

Friday, 27 May 2011


Snapshot imaging

• Instantaneously arrays are mostly coplanar

• Grid on two dimensional plane, FFT, correct for coordinate distortion

• Combine snapshot imaging with w projection• Fit plane to instantaneously u,v,w sampling• Use w projection to project down onto fitted u,v plane • Set threshold in w e.g. 10% of maximum w• Refit plane when error in fit exceeds that threshold

• Scales as number of pixels

VA 'B = I(l,m)e−2π j ul+vm+w 1− l2 −m2( ) dldm

1− l2 − m2∫

VA 'B = I(l,m)e−2π j u l−a 1− l2 −m2( )+v m−b 1− l2 −m2( )( ) dldm

1− l2 − m2∫

w as a function of u,v and hour angle +/- 4 hours

Friday, 27 May 2011


Multiscale Multifrequency Synthesis

• Necessary to correct for source changes over bandwidth• MFClean in MIRIAD• Multi-scale multi-frequency synthesis algorithm• Tested extensively for Urvashi Rau PhD• Demonstration here on EVLA simulated data 1.4 - 5.4 GHz

• Also developing Compressive Sampling algorithm

Moment 0 Moment 1 Moment 2

Friday, 27 May 2011


Sensitivity without interlacing

Friday, 27 May 2011


Sensitivity with 16 way interlacing

Friday, 27 May 2011


Sensitivity with 4 way (fine) interlacing

Friday, 27 May 2011

1.9Tb/s

0.6Tb/s2.5GB/s

Thirty six antennas Beamformers

CorrelatorCentral

processor

Operations data archive27Tflop/s

340Tflop/s

0.5 - 1 Pflop/s

1.9Tb/s

1.9Tb/s

0.6Tb/s

0.6Tb/s

27Tflop/s

27Tflop/s

ASKAP Science Data

Archive Facility

ASKAP Science

products via VO protocols

PAF filterbank samples

Beamformed filterbank samples18Tflop/s

18Tflop/s

18Tflop/s

Filterbanks

Astronomers

Virtual Observatory

10GB/s

Pawsey High Performance Centre for SKA

MRO-Perth link

Murchison Radioastronomical Observatory

T. Cornwell, July 9 2010


ASKAP data flow

• From observing to archive with no human decision making• Calibrate automatically• Image automatically ~ 80 TB per 8 hour observation• Form science oriented catalogues automatically

Friday, 27 May 2011


Pawsey High Performance Computing Centre for SKA Science, Perth, Western Australia

• A$80M, funded by Australian Federal government• 8800 core machine being commissioned

• HP cluster in a box at Murdoch University: EPIC• ~ 88 on Top 500• ASKAP now using EPIC as early adopters• Use 10TF partition in late 2011 for early telescope testing

• Petascale system by 2013• 25% for radio astronomy

Friday, 27 May 2011


Improvement of scaling since project initiation

Friday, 27 May 2011


Importance of load balancing

Channel number

Time to complete (s)

Friday, 27 May 2011


Necessary changes in algorithm during scaling

• AWProject (2007)• W projection + A projection (for primary beam)• Too much CPU• Too much memory for convolution function

• AProjectWStack (2008)• Apply W term in image space• Much less CPU• Too much memory for w stack

• AWProject + trimmed convolution function (2009)• Only apply and keep non-zero part of convolution function• Still too much memory for convolution function

• AWProject + trimmed convolution function + multiple snapshot planes (2011)• Fit and remove w=au+bv plane every 30 - 60 min• Small memory for convolution function

• No current algorithm will scale as-is to full field longer baselines

Friday, 27 May 2011


Scaling with baseline length

• Data volume• Spectral line ~ B• Continuum ~ B2

• W term ~ B

• Example• SKA SPF

Friday, 27 May 2011


Scaling from ASKAP numbers

• 10,000 cores to process • 16K channels, 36 antennas, 30 beams, 2km baselines in real time

• 12,500,000 cores• 32K channels, 180 antennas, 30 beams, 10km baselines in real time

• Roughly 125 PFlops

• Cost estimate• 1PFlop ~ $10M in 2012• 125 PFlops ~ $20M in 2018

• Steepness of scaling curves can be problem and an aid

• Need scientific use cases to firm up these numbers

Friday, 27 May 2011


Climbing Mount Exaflop

ASKAP

SKA1

SKA2

ASKAP dev cluster

NCI Altix tests

Note that Flops numbers are not achieved - we actually get much lower efficiency because of memory bandwidth - so scaling is relative

NCI NF tests

Pawsey Centre tests

Friday, 27 May 2011


Using ASKAPsoft for SKA1 imaging

• Dish array• Single Pixel Feed• Core + inner, 1-2GHz straightforward

• 170 antennas• 8 hour observation = 234 GB• 10 * 3 * 4 * 12 cores ~ 1600 cores

• Longer baselines still to be tested• Easy compared to ASKAP!

Distribution of antennas Single frequency uv coverage All frequencies uv coverage

Friday, 27 May 2011


Continuum simulations

• 8 hour observation - core and inner• Confusion limited• -100uJy to +1000uJy

Contour at 1% sensitivity

Friday, 27 May 2011


Exascale=GigaHz KiloCore MegaNode

• Strawman Exascale architecture• GigaHz KiloCore MegaNode

• Flops are free• Memory and memory bandwidth are the real cost

Friday, 27 May 2011

WP2.6.2 energy cost input / output

Registers

L1 Cache

L2 Cache

Main memoryPhase change memory

LAN/WAN

Source: Energy at ExaFlops, Peter M. Kogge, SC09 Exa Panel

We're streaming all dataSo all accesses WILL require max. amount of power (and when data is reused additional power from more local access)

Friday, 27 May 2011


Summary

• Scientific performance• 50dB must be demonstrated (BETA, ASKAP)• Will learn more about 60 - 70dB

• Imaging• Software framework exists and demonstrated• Need to scale up by 3+ orders of magnitude• Algorithms must track changing computing architectures and configurations• Significant effort requiring access to > 10PFlop computers

• Not boring!

Friday, 27 May 2011

Contact UsPhone: 1300 363 400 or +61 3 9545 2176

Email: [email protected] Web: www.csiro.au

Thank you

ATNF/ASKAPTim CornwellASKAP Computing Project Lead

Phone: +61 2 9372 4261Email: [email protected]: www.atnf.csiro.au

Friday, 27 May 2011

mailto:[email protected]

mailto:[email protected]


Field of view versus beam separation

Friday, 27 May 2011

Documents

Imaging and computing for PAFscsas.ee.byu.edu/PAFSKA2011/Presentations/cornwell... · 2011. 5. 27. · • Antenna A’ receives radiation by Fresnel diffraction from AB plane •