eVLBI Developments at Jodrell Bank Observatory

1

eVLBI Developments at Jodrell Bank Observatory

Ralph Spencer, Richard Hughes-Jones, Simon Casey, Paul Burgess,

The University of Manchester

2

eVLBI Development at JBO and Manchester:

• eVLBI correlation tests using actual astronomy data; both pre-recorded and real time data (see talk by Arpad)

• Network research:– Why?– How?– Results?

3

Why should a radio astronomer be interested in network research?

• Optical fibres have huge bandwidth capability: eMERLIN, eVLA, ALMA, SKA will use >>GHz bandwidths: we need increased bandwidth for VLBI

• Fibre networks are (were) under utilized – can VLBI use spare capacity?

So why study networks?

• What are the bandwidth limits?• How reliable are the links?• What’s the best protocol?• Interaction with end hosts?• What’s happening as technology changes?

Can we get more throughput using switched light paths?

4

How? Network Tests: Manchester/JBO to Elsewhere

• High Energy physics (LHC data) and VLBI have the same aims for internet data usage – collaboration!

• iGRID 2002 Manchester-Amsterdam-JIVE, showed that >500 Mpbs flows are possible

• UDP tests on production network in 2003/4• ESLEA Project 2005- use of UKLight• GEANT2 Launch 2005 RESULTS ------------------

5

WesterborkNetherlands

Dedicated

Gbit link

EVN-NREN

OnsalaSweden

Gbit link

Jodrell BankUK

DwingelooDWDM link

CambridgeUK

MERLIN

MedicinaItaly

Chalmers University

of Technolo

gy, Gothenbu

rg

TorunPoland

Gbit link

6

Gnt5-DwMk5 11Nov03-1472 bytes

0

2

4

6

8

10

12

0 5 10 15 20 25 30 35 40Spacing between frames us

% P

acket

loss

Gnt5-DwMk5

DwMk5-Gnt5

Throughput vs packet spacing Manchester: 2.0G Hz Xeon Dwingeloo: 1.2 GHz PIII Near wire rate, 950 Mbps Tests done at different times

Packet loss

CPU Kernel Load sender

CPU Kernel Load receiver

4th Year project Adam Mathews Steve O’Toole

UDP Throughput Manchester-Dwingeloo (Nov 2003)

Gnt5-DwMk5 11Nov03/DwMk5-Gnt5 13Nov03-1472bytes

0

200

400

600

800

1000

1200


Recv W

ire r

ate

Mbits/s

Gnt5-DwMk5

DwMk5-Gnt5

Gnt5-DwMk5 11Nov03 1472 bytes

020406080

100


% K

erne

l S

ende

r

Gnt5-DwMk5 11Nov03 1472 bytes

020406080

100


% K

erne

l R

ecei

ver

7

Packet loss distribution:

( )t

p d

Cumulative distribution

Cumulative distribution of packet loss, each bin is 12 sec wide

Long range effects inthe data?

Poisson

8

Exploitation of Switched Lightpaths for E Science

Applications:

• Multi disciplinary project involve collaboration between many research groups: network scientists, computer science, medical science, high energy physicists and radio astronomers: using UKLight network

• Protocol and control plane development• High performance computing• eHealth (e.g. radiology)• High Energy Physics data transfer (LHC)• eVLBI: funds a post-doc (ad out – apply now!)

9

26th January 2005 UDP Tests

Simon Casey (PhD project) Between JBO and JIVE in Dwingeloo, using production networkPeriod of high packet loss (3%):

10

The GÉANT2 Launch June 2005

11

Jodrell BankUK

DwingelooDWDM link

MedicinaItaly Torun

Poland

e-VLBI at the GÉANT2 Launch Jun 2005

12

e-VLBI at the GÉANT2 Launch Jun 2005

13

e-VLBI UDP Data Streams

14

UDP Performance: 3 Flows on GÉANT• Throughput: 5 Hour run 1500 byte MTU

• Jodrell: JIVE2.0 GHz dual Xeon – 2.4 GHz dual Xeon670-840 Mbit/s

• Medicina (Bologna): JIVE 800 MHz PIII – Mk5 (623) 1.2 GHz PIII 330 Mbit/s limited by sending PC

• Torun: JIVE 2.4 GHz dual Xeon – Mk5 (575) 1.2 GHz PIII

245-325 Mbit/s limited by security policing

(>400Mbit/s 20 Mbit/s) ?

• Throughput: 50 min period• Period is ~17 min

BW 14Jun05

0

200

400

600

800

1000

0 500 1000 1500 2000Time 10s steps

Rec

v w

ire ra

te M

bit/s

JodrellMedicinaTorun

BW 14Jun05

0

200

400

600

800

1000

200 250 300 350 400 450 500Time 10s steps

Rec

v w

ire ra

te M

bit/s

JodrellMedicinaTorun

15

UDP Performance: 3 Flows on GÉANT

• Packet Loss & Re-ordering• Each point 10 secs, 660k packets• Jodrell: 2.0 GHz Xeon

– Loss 0 – 12%– Reordering significant

• Medicina: 800 MHz PIII– Loss ~6%– Reordering in-significant

• Torun: 2.4 GHz Xeon– Loss 6 - 12%– Reordering in-significant

Torun 14Jun04

0

1

2

3

4

5

0 500 1000 1500 2000Time 10s

num

re-o

rder

ed

020000

400006000080000100000

120000140000

num

lost

re-ordered

num_lost

jbgig1-jivegig1_14Jun05

0

500

1000

1500

2000

0 500 1000 1500 2000Time 10s

num

re-o

rder

ed

0

50000

100000

150000

num

lost

re-ordered

num_lost

Medicina 14Jun05

0

1

2

3

4

5

0 500 1000 1500 2000Time 10s

num

re-o

rder

ed0

10000

20000

30000

40000

50000

60000

70000

num

lost

re-ordered

num_lost

16

18 Hour Flows on UKLightJodrell – JIVE, 26 June 2005

• Throughput:• Jodrell: JIVE

2.4 GHz dual Xeon – 2.4 GHz dual Xeon

960-980 Mbit/s

• Traffic through SURFnet

• Packet Loss– Only 3 groups with 10-150 lost

packets each– No packets lost the rest of the

time

• Packet re-ordering– None

man03-jivegig1_26Jun05

0

200

400

600

800

1000

0 1000 2000 3000 4000 5000 6000 7000

Time 10s steps

Rec

v w

ire r

ate

Mbi

t/s

w10


900910920930940950

960970980990

1000

5000 5050 5100 5150 5200

Time 10s

Recv w

ire r

ate

Mbit/s w10


1

10

100

1000

0 1000 2000 3000 4000 5000 6000 7000

Time 10s steps

Packet

Loss

w10

17

Conclusion

• Max data rates depends on the path:– Limited by end hosts? : lack of cpu power in end host jumbo

packets will help here– Local limits e.g. security : work with the network providers to

achieve the bandwidth we need– Networks have the capacity for >500 Mbps flows– Evidence for network bottlenecks somewhere : more evidence

being collected

• Packet loss will limit TCP flows – explains limits to data rates in EVN eVLBI tests: new protocols will help here

• More needs to be done before we can reliably get 512 Mbps eVLBI in EVN – especially study of end hosts.

18

Any Questions?

Documents

eVLBI Developments at Jodrell Bank Observatory