EVLA Correlator DRAO NovelTel Jun14-11halpern/chime/workshop... · 2011. 6. 21. · EVLA P j t O...

Th EVLA C lThe EVLA CorrelatorB. Carlson

National Research CouncilCanada

Conseil national de recherchesCanada Novel Tel. Workshop, DRAO June 14, 2011

O tliOutline• EVLA project overview.

• What is a correlator?

• EVLA correlator capabilities.EVLA correlator capabilities.

• Basic signal processing.

• System architecture.

• A view inside the correlator room.

• A (very) short survey of current developments/directions.

B. Carlson June 14, 2001 Novel Tel. Workshop DRAO – The EVLA Correlator

2

A (very) short survey of current developments/directions.

• Peter Dewdney (now SPDO)• Peter Dewdney (now SPDO)• Sean Dougherty (NRC-DRAO)• Dave Fort (NRC-DRAO)• Sonja Vrcic (NRC DRAO)

• Bruce Rowen (NRAO-Socorro)• Kevin Ryan (NRAO-Socorro)• Hichem Ben Frej (NRAO-Socorro)

• Sonja Vrcic (NRC-DRAO)• Heng Zhang (NRC-DRAO)• Zoran Ljusic (NRC-DRAO)• Brent Carlson (NRC DRAO)

• Martin Pokorny (NRAO-Socorro)• Ken Sowinski (NRAO-Socorro)• Michael Rupen (NRAO-Socorro)

• Brent Carlson (NRC-DRAO)• Dave Del Rizzo (NRC-DRAO)• Ralph Webber (NRC-DRAO)• Mark Halman (NRC DAO)

• Rick Perley (NRAO-Socorro)• Barry Clark (NRAO-Socorro)• Kerry Shores (NRAO-Socorro)

• Mark Halman (NRC-DAO)• Dale Basnett (Contractor)• Rick Smegal (Contractor)• Amy Fink (frmr NRC DRAO)

• Pete Pelletier (Temp/prod)• Shelley Deakin (Temp/prod)

• Amy Fink (frmr NRC-DRAO)• Donna Morgan (NRC-DRAO)• BreconRidge/Sanmina• iSine Inc ; Altera; Xilinx

• Josh Fink (Temp/ship-rec’v)

• iSine Inc.; Altera; Xilinx

B. Carlson, 2011-Jun 14 Novel Tel. Workshop DRAO – EVLA Correlator 3

B. Carlson, 2011-Jun 14 Novel Tel. Workshop DRAO – EVLA Correlator 4

B. Carlson, 2011-Jun 14 Novel Tel. Workshop DRAO – EVLA Correlator 5

EVLA P j t O iEVLA Project Overview

• Upgrade started ~2000-2001.pg• All new lower-noise receivers and feeds. Antennas

completely gutted.• Replace waveguide system with digital fiber-optic system.

Samplers at the antennas for stable performance.• Increase BW from ~100 MHz/pol’n to 8 GHz/pol’n• Increase BW from ~100 MHz/pol n to 8 GHz/pol n.• Contiguous frequency coverage from 1-50 GHz.• New frequency-agile, high spectral and temporal resolution q y g , g p p

correlator.• Project completion end of 2012.

B. Carlson, 2011-Jun 14 Novel Tel. Workshop DRAO – EVLA Correlator 6

• Actually achieved SEFDs (initial spec):

– L-band – 400 Jy (325)– S-band – 350 Jy (235)

C band 300 Jy (245)– C-band – 300 Jy (245)– X-band – 250 Jy (300)– Ku-band – 280 Jy (385)– K-band – 450 Jy (650)– Ka-band – 620 Jy (760)– Q-band – 1100 Jy (1220/2760)y ( )

(Source: R. Perley, upcoming ApJLet)

B. Carlson, 2011-Jun 14 Novel Tel. Workshop DRAO – EVLA Correlator 7

Source: Rick Perley,NRAO

B. Carlson, 2011-Jun 14 Novel Tel. Workshop DRAO – EVLA Correlator 8

Wh t i C l t ?What is a Correlator?• Calculates the cross-correlation (cross-power) coefficient for every pair of

t i thantennas in the array.– XF: delay-domain cross-correlation and integration followed by FFT.– FX: real-time FFT, multiply channels and integrate.– These days, its most often all digital…and FX.

• More bandwidth yields more sensitivity…but higher-speed electronics.y y g p

• Spectral channels open up spectral-line observing (velocity, spectral index info)info).

B. Carlson, 2011-Jun 14 Novel Tel. Workshop DRAO – EVLA Correlator 9

B. Carlson, 2011-Jun 14 Novel Tel. Workshop DRAO – EVLA Correlator 10

B. Carlson, 2011-Jun 14 Novel Tel. Workshop DRAO – EVLA Correlator 11

EVLA C l t C bilitiEVLA Correlator Capabilities

• 32 antennas (EVLA 28 + spare capacity).( p p y)

• 8 GHz/pol’n, in 2 GHz bands.– 8-bit sampling, 1 GHz/pol’n x 2 = 2 GHz/pol’n – operational.– 3-bit sampling, 2 GHz/pol’n x 4 = 8 GHz/pol’n – samplers undergoing

testing…g

• 16,384 channels/baseline…up to 4 Mchannels/baseline at narrower bandwidths (≤ 0 5 MHz/sub band x 16 @1 pol’nnarrower bandwidths (≤ 0.5 MHz/sub-band x 16 @1 pol n, 256k channels ea.—2 Hz resolution) with “recirculation”.

B. Carlson, 2011-Jun 14 Novel Tel. Workshop DRAO – EVLA Correlator 12

EVLA C l t C bilitiEVLA Correlator Capabilities

• Stable digital sub-band filtering. Original concept was method for efficient parallel processing.– Sub-bands (a.k.a. “spectral windows”) 128 MHz, 64 MHz, …, 31.25 Sub ba ds (a. .a. spec a w dows ) 8 , 6 , …, 3 . 5

kHz.– Implemented in multi-stage digital filter in Xilinx FPGA.– Each sub-band can be on different delay center on the sky (within theEach sub-band can be on different delay center on the sky (within the

main beam of the antenna).

All 16 GH di it l h d t t il bl ll th ti• All 16 GHz digital phased-array output available all the time.

B. Carlson, 2011-Jun 14 Novel Tel. Workshop DRAO – EVLA Correlator 13

EVLA C l t C bilitiEVLA Correlator Capabilities

• Quasi-flexible sub-band tuning. “WIDAR” technique limited Q g qto sub-band integer slots (but stage-2 SSB mixer relaxes limitation somewhat).

• Pulsar phase binning: 2 banks of 2000 bins ~200 usec bin• Pulsar phase binning: 2 banks of 2000 bins, ~200 usec bin resolution on every product. Down to ~15 usec bin resolution with reduced bandwidth.

B. Carlson, 2011-Jun 14 Novel Tel. Workshop DRAO – EVLA Correlator 14

EVLA C l t C biliti

• High time resolution output: 100 msec (limitation due to data

EVLA Correlator Capabilities

g p (storage and backend processing) 10 msec 1GE output capable, 1 msec with 10GE output field upgrade.

Agile dumping/integration Each sub band can have different– Agile dumping/integration. Each sub-band can have different parameters…locked to system timing, or model of “something else” like pulsar ephemeris.

• Sub-arraying, with granularity of 4 antennas.

B. Carlson, 2011-Jun 14 Novel Tel. Workshop DRAO – EVLA Correlator 15

B i Si l P iBasic Signal Processing

• Introduce small epsilon frequency shift in signal in each antenna before sampler.

• Remove after sub-band filtering during correlation.

• Aliased sub-band transition bands de-correlate with time.– “Moral equivalent” of Walsh-function switching.

• Allows seamless “stitching” of sub-bands.– But, 2X sensitivity loss at sub-band edges.

• All-digital sub-sample delay correction. Stable, predictable.

B. Carlson, 2011-Jun 14 Novel Tel. Workshop DRAO – EVLA Correlator 16

B. Carlson, 2011-Jun 14 Novel Tel. Workshop DRAO – EVLA Correlator 17

B. Carlson, 2011-Jun 14 Novel Tel. Workshop DRAO – EVLA Correlator 18

S t A hit tSystem Architecture

• 8 “Station” racks – Antenna-based processing (wideband p g (delay, digital filtering, signalling generation).

• 16 “Station Boards” in each rack; 4 boards per antenna.

Di t ib t d b it hi f fl ibl ll ti f• Distributed cross-bar switching for flexible allocation of correlation hardware to sub-bands. (a.k.a. coarse filterbank“corner-turner”)

• Point-to-point cables from Station racks to Baseline racks.

B. Carlson, 2011-Jun 14 Novel Tel. Workshop DRAO – EVLA Correlator 19

Outputs to Baseline Board Pairs in Quadrants 1-4

F:0

F:0

F:2F:8

F:8

F:8

F:8

F:8

F:8

F:8

F:8

F:8

F:8

F:8

F:8

F:8

F:8

F:8

F:8

F:8

F:8

F:2

F:2

F:10

F:4

F:4

F:4

Outputs to Baseline Board Pairs in Quadrants 1 4

Q4

Q3

Q2

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 BlB Pair

F:0

F:0F:5

F:5 F:3

F:3

Bar

FP

GA

F:8F:8 F:8 F:8 F:8 F:8 F:8F:10F:4

-Bar

Boa

rd-0

-U

Q1

-Bar

Boa

rd-0

-L

-Bar

Boa

rd-1

-U

-Bar

Boa

rd-2

-U

-Bar

Boa

rd-3

-U

-Bar

Boa

rd-4

-U

-Bar

Boa

rd-5

-U

-Bar

Boa

rd-6

-U

-Bar

Boa

rd-7

-U

-Bar

Boa

rd-1

-L

-Bar

Boa

rd-2

-L

-Bar

Boa

rd-3

-L

-Bar

Boa

rd-4

-L

-Bar

Boa

rd-5

-L

-Bar

Boa

rd-6

-L

-Bar

Boa

rd-7

-L

X-Bar BoardConnectionsF:3

1 co

lum

n/sl

ice

= 1

X-B

for 4

ant

enna

s

F:4

F 3

F:10

F:2

F 5 F 5 F 8

F:0

F:5

F 9

F:3

F:0

F:8

F:2

F 0F 8 F 8 F 8 F 8 F 8 Station B

oard O

utputsX- X-X- X- X- X- X- X- X- X- X- X- X- X- X- X-

StationBoard

Output Chip

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 wafer (SBOID)

F:3 F:5 F:5 F:8 F:9 F:0F:8 F:8 F:8 F:8 F:8 Sta

D0,B2,D2

B0,B1,D1 0 1 2 3 9 10 11 12 13 14 154 5 6 7 8

0 1 2 3 9 10 11 12 13 14 154 5 6 7 8

p pX-bars

ards/an

t

C0,A2,C2

A0,A1,C1 0 1 2 3 9 10 11 12 13 14 154 5 6 7 8

0 1 2 3 9 10 11 12 13 14 154 5 6 7 8

Station Board Digital Filters

4 Stat

ion Boar

Inputs from antennas

Author, 2003-Jan 27-28 EVLA Correlator Meeting - Talk Title 20

Inputs from antennas

B. Carlson, 2011-Jun 14 Novel Tel. Workshop DRAO – EVLA Correlator 21

S t A hit t

• 8 “Baseline” racks—baseline-based processing (cross-

System Architecture

p g (correlation and phased-array summing).

• 16 boards in each rack, 2 boards correlate a sub-band pair, all 32 antennas.– Can trade-off bandwidth for number of spectral channels.Can trade off bandwidth for number of spectral channels.

• 1 Gigabit Ethernet correlated data output from each board. – Phased-array output on 1 Gigabit Ethernet as well in “VDIF” format

for VLBI recording.

B. Carlson, 2011-Jun 14 Novel Tel. Workshop DRAO – EVLA Correlator 22

B. Carlson, 2011-Jun 14 Novel Tel. Workshop DRAO – EVLA Correlator 23

24B. Carlson, 2011-Jun 14 Novel Tel. Workshop DRAO – EVLA Correlator

25B. Carlson, 2011-Jun 14 Novel Tel. Workshop DRAO – EVLA Correlator

Author, 2003-Jan 27-28 EVLA Correlator Meeting - Talk Title 26

27B. Carlson, 2011-Jun 14 Novel Tel. Workshop DRAO – EVLA Correlator

B. Carlson, 2011-Jun 14 Novel Tel. Workshop DRAO – EVLA Correlator 28

B. Carlson, 2011-Jun 14 Novel Tel. Workshop DRAO – EVLA Correlator 29

Author, 2003-Jan 27-28 EVLA Correlator Meeting - Talk Title 30

B. Carlson, 2011-Jun 14 Novel Tel. Workshop DRAO – EVLA Correlator 31

B. Carlson, 2011-Jun 14 Novel Tel. Workshop DRAO – EVLA Correlator 32

B. Carlson, 2011-Jun 14 Novel Tel. Workshop DRAO – EVLA Correlator 33

B. Carlson, 2011-Jun 14 Novel Tel. Workshop DRAO – EVLA Correlator 34

B. Carlson, 2011-Jun 14 Novel Tel. Workshop DRAO – EVLA Correlator 35

B. Carlson, 2011-Jun 14 Novel Tel. Workshop DRAO – EVLA Correlator 36

B. Carlson, 2011-Jun 14 Novel Tel. Workshop DRAO – EVLA Correlator 37

B. Carlson, 2011-Jun 14 Novel Tel. Workshop DRAO – EVLA Correlator 38

B. Carlson, 2011-Jun 14 Novel Tel. Workshop DRAO – EVLA Correlator 39

B. Carlson, 2011-Jun 14 Novel Tel. Workshop DRAO – EVLA Correlator 40

B. Carlson, 2011-Jun 14 Novel Tel. Workshop DRAO – EVLA Correlator 41Credit: Paul Harden, NRAO

B. Carlson, 2011-Jun 14 Novel Tel. Workshop DRAO – EVLA Correlator 42Credit: Paul Harden, NRAO

B. Carlson, 2011-Jun 14 Novel Tel. Workshop DRAO – EVLA Correlator 43

Author, 2003-Jan 27-28 EVLA Correlator Meeting - Talk Title 44

Author, 2003-Jan 27-28 EVLA Correlator Meeting - Talk Title 45

Author, 2003-Jan 27-28 EVLA Correlator Meeting - Talk Title 46

Author, 2003-Jan 27-28 EVLA Correlator Meeting - Talk Title 47

C t D l t /Di tiCurrent Developments/Directions• S/W correlator (NRAO): DiFX – Deller, Brisken et. al. VLBA multi-(~400)

phase center s/w flexibility…new parameter space. 4 Gbps, ~10 antennas.• ASKAP correlator (CSIRO)– multi-Xilinx FPGA ATCA standard.• CASPER (Berkeley) – single-Xilinx FPGA + PPC non-standard FF, cable

interconnect, libraries, fast to “get going”, keep up with tech curve.• Uniboard (JIVE/Astron) – 8 x Altera Stratix-IV recently completed/released.

non-standard FF. Uniboard-2 planned (I believe).• JPL ASIC (D’Addario) – high efficiency/compact T-MUX CMAC array with no

acc m memor Plan on b ilding protoaccum memory. Plan on building proto.• GSA/X421 (DRAO) – physical layout for very large corr, with no cable

interconnects, X421 ASIC baseline-based, time+channel int, VBA for RT IP.• Kermode (DRAO) – multi-Xilinx FPGA ATCA standard with flexibilities for• Kermode (DRAO) – multi-Xilinx FPGA ATCA standard, with flexibilities for

other sig-proc + compute tasks, and interfaces. Commercial BSP planned.• ALL poly-phase FX.

B. Carlson, 2011-Jun 14 Novel Tel. Workshop DRAO – EVLA Correlator 48

G l t dGeneral trends…• S/W correlation for small-to-medium jobs…High degree of flexibility. If

t ft th i k d l t FPGA l tfnot software, then quick-development FPGA platforms.

• Quasi-general purpose FPGA boards, of varying densities, and varying interconnect schemes for small-to-large jobs.– Keeping up with the technology curve is still a challenge, and is most

easily done with few devices per board, but then limits performance.– Must factor in total system development costs and time, not just board

design and test time.– Boards with FPGAs + memory + MGT fabric.

• Custom designs only for very large systems (e.g. SKA-2)

B. Carlson, 2011-Jun 14 Novel Tel. Workshop DRAO – EVLA Correlator 49

SSummary

• EVLA project overview.

• What is a correlator? EVLA correlator capabilities.

• Basic signal processing... filterbank followed by c-lag corr.

S hi k bli &C li• System architecture…racks, cabling, M&C, power, cooling.

• A view inside the correlator room…lotsa pictures.

• S/W (small-med); FPGA boards (small-large); custom (very large).

B. Carlson, 2011-Jun 14 Novel Tel. Workshop DRAO – EVLA Correlator 50

B. Carlson, 2011-Jun 14 Novel Tel. Workshop DRAO – EVLA Correlator 51

B i Si l P iBasic Signal Processing• All-digital sub-sample delay correction. Stable, predictable.

B. Carlson, 2011-Jun 14 Novel Tel. Workshop DRAO – EVLA Correlator 52

53B. Carlson, 2011-Jun 14 Novel Tel. Workshop DRAO – EVLA Correlator

Credit: Paul Harden, NRAO

B. Carlson, 2011-Jun 14 Novel Tel. Workshop DRAO – EVLA Correlator 54

B. Carlson, 2011-Jun 14 Novel Tel. Workshop DRAO – EVLA Correlator 55

2009 – 12 antennas

Author, 2003-Jan 27-28 EVLA Correlator Meeting - Talk Title 56

Awaiting longest baseline (A config) data(A-config) data.

L and Ka-band,2 GHz BW.

B. Carlson, 2011-Jun 14 Novel Tel. Workshop DRAO – EVLA Correlator 57