Daniel Charlet Paris Twepp2009 BAO project in radio CEA/Irfu IN2P3/LAL CMU Fermilab colaboration Fast aquisition system for 3D mapping of cosmological

Daniel Charlet Paris Twepp2009

BAO project in radio

CEA/Irfu IN2P3/LAL CMU Fermilab colaboration

Fast aquisition systemfor 3D mapping of cosmological

matter distribution in radio


BAO : Baryonic Acoustic Oscillations

Imprints left by the baryon-photon fluid (before recombination) in the distribution of ordinary (baryonic) matter

Slight modulation of the distribution of matter, (and galaxies as tracers). Structure formation being mainly driven by CDM which dominates structure formation

In Radio : Use 21 cm HI emission 3D HI mass distribution measurement through

total 21 cm emission intensity mapping (No individual galaxy detection) Hyperfine transition (spin-orbit) of atomic

hydrogen: ν ≈ 1,420405 GHz →λ ≈ 21 cm


BAO Radio-CRT Instrument principle

1-2km

N

S

x

y

Large field of view (10-100 deg^2)

≥ 1000 simultaneous lobes → Digital interferometer

Wide band receivers 250 MHz

Digital interferometer (correlator / beam-former)

Resolution 10 arcmin, Surface ≥ 10 000 m^2

Reflectors over an area of ~ 1000-2000 m × 1000-2000 m

Cylindrical Radio Telescope concept : 12 cylinders 100 m x 8 m

256-512 electronic channel by cylinder

4 bands of 250Mhz between 0.5-1.5GHz

Sampling at 500MHz

Possible implantation in Morocco


Standard Interferometry

Array: résolution gain /d /D we can observe only some directions sin(N) = N./L For =30cm, () = 1’ D= 1km !!!

Antenna array Array and corrélation

Same gain in resolution With a correlator, all directions are available N(N-1)/2 indépendant correlations


Interferometry

Array: résolution gain /d /D we can observe only some directions sin(N) = N./L For =30cm, () = 1’ D= 1km !!!

Antenna array Array and corrélation

Same gain in resolution With a correlator, all directions are available N(N-1)/2 indépendant correlations

REPLACE ANALOG RADIO INTERFEROMETERSELECTRONIC ELEMENTS(filters, mixer, multipliers...) by

DIGITAL ELECTRONICS


Bea

m from

ingF

or each FF

T

frequency

Numerical correlator

Each channel:Fast Fourier Transform

All channel: Combination of the Same frequency

3 dimensions:2 angles : FFT beam formingDistance : frequency red-shift


Electronics chain

128dipoles

2x4Gbit/s

Antenna board4 dipoles

Down conversion

filtering

ADC (500 MHz)

4 channelsX 4

…

…

X 4

Antenna board4 dipoles

2nd

……

……

2nd

PC

ADC (500MHz)4 channels

PC

X 8

……

……

Clockdistribution

Down conversion

filtering

24 Gbit/s

Time FFTFPGA

Time FFTFPGA

Spatial FFT(FPGA)

24.Gbit/s

24 Gbit/s

…

…

X 4

Analog module

PCIExpress board

ADC board


Analog chain for RF prototype boardCEA irfu


LAL ADC board Synoptic


ADC board FPGA Synoptic

SpyFIFO

SerialLite2

FIFO2

FIFO3

mux2

mux3

Header_Spare

Config Acq

timer_Counter MSB

timer_Counter LSB

Header Tag id MSB

Header Tag id LSB

mux4

Trailer_Spare

Frame_Status

Trailer Tag id MSB

Trailer Tag id LSB

chI

chId

chQ

chQd

00

10

01

001

010

011

100

101

110

001

010

011

100

FFT0

mux1

00

11

01

10FIFO1

FIFO0ch0

ch1

500MHz

250MHz

FPGA:Altera Stratix EP2SGX90EF1152C3

ADC 08D500

125MHz

5GHz

[15:0] [15:0]

[15:0] [15:0]

[31:0]

[31:0]

2k x 16 bits

2k x 16 bits

2k x 16 bits

2k x 16 bits

[31:0]

[31:0]FFT1

[15:0] [15:0]

[15:0] [15:0]

[31:0]

[31:0]

[31:0]

[7:0]

[7:0]

[7:0]

[7:0]

[31:0]

[31:0]

[31:0]

[31:0]

[31:0]

[31:0]

[31:0]

[31:0]

[31:0]

[31:0]

[31:0]


data[2*i]

data[2*i+1]

Out_re

Out_im

Complex-FFT

N/2 points

data[i]

0

Out_re

Out_im

Complex-FFT

N points

1 x 500 MSPS 2 x 250 MSPS

FPGA Implementation of real-FFT using a IP Core:to save hardware resources, we use a N/2 points complex-FFT to compute a N-points real-FFT

Post ADC processing: Temporal FFT

the real signal s[i=0:N-1] is transformed into a complex signal : c[i] = s[2*i]+j*s[2*i+1]


2 step computation:

• First step: N/4 clock cycles, we feed the block RAM with FFTN/2(c)[k]

• Second step: N/4+1 clock cycles, the output of complex-FFT and data stored in block RAM feed 2 butterfly operators,for 0k N/4

Post ADC processing: Temporal real-FFT

In_re

In_im

Out_re

Out_im

Complex-FFT

N/2 points

ADC output: 16 bits@250MHz (2 time samples)

Block RAM(Write mode)

16bits x N/4

FFTN/2(c)[k]

FFTN/2(c)[N/2-k]*

jWNk

FFTN(s)[k]

FFTN(s)[N/2-k]


Analog characteristic: 1Time resolution

Lin

~0.1ps ~6ps


Analog characteristic :2Linearity

Lin

Linearity at 1.4GHz


Analog characteristic : 3Bode diagram

Lin


LAL ADC board


PCIExpress evaluation board

Integrity framecontroller

+PCIExpress

Interface

serdes4Gb/s

serdes4Gb/s

4x

STRATIXIIGX

serdes 2Gb/s

~500Mbyte/s

serdesserdesserdes

2Gb/s 2Gb/s 2Gb/s


Synoptic FPGA PCIExpress

Evaluation board

Serdes

PCIExpress4x

DMAController

SerialLiteII

Fifo 32x16KByteFram filteringData ordering

Pattern generator

Fifo 64x8KByteFlow contol

Descriptor Ram

SerdesSerdesSerdes

DMAController

SerialLiteII

Fifo 32 x16KByteFrame filtering +Data ordering

Pattern generator


PCIExpress interfaceFront-end data computation


FPGA synoptic: Front-end data computation

SerialLiteII

Fifo 32x16KByteFram filteringData ordering

Pattern generator

SerialLiteII

Fifo 32 x16KByteData ordering

Pattern generator

Fram filteringData orderingFrame filtering

Link interface: SerialLiteII.Pattern generator.Data ordering.Frame filtering.Buffering: FIF0 64KByt


FPGA synoptic: PCIExpress interface

Serdes

PCIExpress4x

DMAController


Descriptor Ram

SerdesSerdesSerdes

DMAController


System On Programmable Chip module

FIFO bufferingScatter-gather DMAPCIExpress:

•End point•4X


Aquisition test

Poweredge 2900 2 hardware configuration:

– 1 double core cpu, 2GB ram, 1 raid controller with 8 sata2 disc of 160GB (Western Digital)

– 1 quadruple core cpu, 4GB ram 1 raid controller with 8 sata2 disc of 160GB (Western Digital)

Software configuration Multi thread acquisition:

Data reading in DMA mode PCIExpress board and

Reading the shared memory and writing on disk

PERC 5IController

PCIX

SATA 2

CPU Ram

SATA 2

FSB2000MB/s

x8

PCIx bus

South bridge.....

North bridge.....

ALTERAPCIExpress

Ram


PC Data flow

0

100000

200000

300000

400000

500000

600000

1 106 211 316 421 526 631 736 841 946 1051 1156 1261 1366 1471 1576 1681 1786 1891 1996

Série1

PCIEpress DMA transfert :

Average throughput 430MB/s

Multi-thread acquisition software:PCIExpress DMADisck write

300MB/s substained data rate from digitilizer to PC-disck

MByte/S

Files


Test with Nançay Radio Telescope (NRT)

Focal carriage


Test in Pittsburgh

Re(a1a2*) Re(a1a2*)

Interference


In conclusion

Powerful & versatile multi-channel acquisition system Distributed over several hundred meters Sample rate 500MHz with a input bandwidth of 1,5Gz 500MB/s Streaming data Time resolution between channels better than 10ps 300MB/s data processing & hard disk writing one 1 PC

Ongoing Multiprocessor synchronisation Real time operating system Interrupt mode on PCIExpress

Documents

Daniel Charlet Paris Twepp2009 BAO project in radio CEA/Irfu IN2P3/LAL CMU Fermilab colaboration Fast aquisition system for 3D mapping of cosmological