Networking for ATLAS Remote Farms

Meeting on ATLAS Remote Farms. Copenhagen 11 May 2004R. Hughes-Jones Manchester

Networking for ATLAS Remote Farms

Richard Hughes-JonesThe University of Manchester

DataGrid WP7 – Dante Tests on the GÉANT CoreEnd-2-End Measurements from the 4th Year VLBI Project at ManchesterNew TCP stacks – the effect on throughputSome Simple Network Tests CERN-Manchester


DataGrid WP7 – Dante Tests on the GÉANT Core

Set-up

Supermicro PC in: London GEANT PoP Amsterdam GEANT PoP

Smartbits in: London GEANT PoP Frankfurt GEANT PoP

Long link UK-SE-DE2-IT-CH-FR-BE-NL

Short Link UK-FR-BE-NL


Tests GÉANT Core: UDP throughput

UDP Throughput London-Amsterdam Available BW to packet on wire Then 1/t Wire rate 998 Mbit/s

for packets > 1400 bytes

Packet Loss None for large packets

Dips in BW lined to packet loss SysKonnect NIC int. per packet CPU load important

uk-nl_20tg4-hs-w100_01Oct03

0

100

200

300

400

500

600

700

800

900

1000

0 5 10 15 20 25 30 35 40Spacing between frames us

Rec

v W

ire r

ate

Mbi

ts/s

50 bytes 100 bytes 200 bytes 400 bytes 600 bytes 800 bytes 1000 bytes 1200 bytes 1400 bytes 1472 bytes

0102030405060708090

100


% P

acke

t los

s



Tests GÉANT Core: Packet re-ordering

Effect of Packet size London-Amsterdam Packets at 10 µs – line speed 10,000 sent Packet Loss ~ 0.1%

Re-order Distribution

Packet re-order uk-nl 10,000 BE sent wait 10 us 01 Oct 03

0

5

10

15

20

25

30

0 500 1000 1500Packet size bytes

Out

of o

rder

% 0

10

20

30

1400 1401 1402 1403 1404Packet size bytes

Ou

t o

f o

rde

r %

Packet re-order uk-nl 10,000 sent wait 10 us

0

100

200

300

400

500

0 1 2 3 4 5 6 7 8 9Length out-of-order

No.

Pac

kets 1400 bytes

1401 bytes

1402 bytes


Tests GÉANT Core: Packet re-ordering

Effect of LBE background Amsterdam-London BE Test flow Packets at 10 µs – line speed 10,000 sent Packet Loss ~ 0.1%

Re-order Distributions:

UDP 1472 bytes NL-UK-lbexxx_7nov03

02468

101214161820

2 2.2 2.4 2.6 2.8 3 3.2Total Offered Rate Gbit/s

% O

ut o

f ord

er

hstcpStandard TCP line speed90% line speed

Packet re-order 1472 bytes uk-nl 21 Oct 03 10,000 sent wait 10 us

020000400006000080000

100000120000140000160000180000200000

1 2 3 4 5 6 7 8 9Length out-of-order

No.

Pac

kets

0 % lbe

10 % lbe

20 % lbe

30 % lbe

40 % lbe

50 % lbe

60 % lbe

70 % lbe

80 % lbe

Packet re-order 1400 bytes uk-nl 21 Oct 03 10,000 sent wait 10 us

0500

100015002000250030003500400045005000

1 2 3 4 5 6 7 8 9Length out-of-order

No.

Pac

kets

0 % lbe

10 % lbe

20 % lbe

30 % lbe

40 % lbe

50 % lbe

60 % lbe

70 % lbe

80 % lbe


Tests GÉANT Core: Packet Jitter Amsterdam-London BE Test flow Packet spacing 80 µs

IPPremium Test flow

Flow: BE Background: none

0

10000

20000

30000

40000

50000

0 20 40 60 80 100 120 140

Latency us

Fre

qu

ency

Flow:IPP Background: none

0

50000

100000

150000

200000

250000

0 20 40 60 80 100 120 140Packet Jitter us

Fre

qu

ency

Flow:BE Background: 60% BE 1.4Gbit + 40% LBE 780Mbit

0

5000

10000

15000

20000

25000

30000

35000

40000

0 20 40 60 80 100 120 140

Packet Jitter us

Fre

qu

ency

BE Test flow + Background: 60% BE 1.4Gbit + 40% LBE 780Mbit

flow:IPP Background: 60% BE 1.4Gbit + 40% LBE 780Mbit

0

10000

20000

30000

40000

50000

60000

0 20 40 60 80 100 120 140Packet Jitter us

1-w

ay l

aten

cy u

s

IPPremium Test flow + Background


Tests GÉANT Core: 1-way Delay Amsterdam-London IPPremium Test flow Packet spacing 80 µs



Flow:IPP Background: none

11220

11240

11260

11280

11300

11320

11340

0 2000 4000 6000 8000 10000Packet No.

1-w

ay la

tenc

y us

Flow:BE Background: 60% BE 1.4Gbit + 40% LBE 780Mbit

11000112001140011600118001200012200124001260012800

0 2000 4000 6000 8000 10000Packet No.

1-w

ay la

tenc

y us

Flow:IPP Background: 60% BE 1.4Gbit + 40% LBE 780Mbit

1120011250113001135011400114501150011550116001165011700

0 2000 4000 6000 8000 10000Packet No.

1-w

ay la

tenc

y us


VLBI Project: Test Topology

SuperJANET4

Jodrell

Manchester

SURFnet

JIVEDwingaloo

Adam MathewsSteve O’TooleUniv of Manchester


Gnt5-DwMk5 11Nov03-1472 bytes

0

2

4

6

8

10

12


% P

acket

loss

Gnt5-DwMk5

DwMk5-Gnt5

Manchester to Dwingeloo 2.0G Hz Xeon 1.2 GHz PIII

Re-ordering vs Offered Load

VLBI Project: Throughput

Gnt5-DwMk5 11Nov03/DwMk5-Gnt5 13Nov03-1472bytes

0

200

400

600

800

1000

1200


Recv W

ire r

ate

Mbits/s

Gnt5-DwMk5

DwMk5-Gnt5

Gnt5-DwMk5 11Nov03 1472 bytes

020406080

100


% K

erne

l S

ende

r

Gnt5-DwMk5 11Nov03 1472 bytes

020406080

100


% K

erne

l R

ecei

ver


1472 byte Packets man -> JIVE FWHM 22 µs (B2B 3 µs )

VLBI Project: Jitter & 1-way Delay

1472 bytes w=50 jitter Gnt5-DwMk5 28Oct03

0

2000

4000

6000

8000

10000

0 20 40 60 80 100 120 140

Jitter us

N(t

)

1472 bytes w=50 jitter Gnt5-DwMk5 28Oct03

1

10

100

1000

10000

0 20 40 60 80 100 120 140

Jitter us

N(t

)

1472 bytes w12 Gnt5-DwMk5 21Oct03

0

2000

4000

6000

8000

10000

12000

2000 2100 2200 2300 2400 2500 2600 2700 2800 2900 3000Packet No.

1-w

ay

de

lay

us

1472 bytes w12 Gnt5-DwMk5 21Oct03

0

2000

4000

6000

8000

10000

12000

0 1000 2000 3000 4000 5000Packet No.

1-w

ay d

elay

us

1-way Delay – note the packet loss (points with 0 –way delay)


Aggregated Variance Method Divide time series length N into

blocks of size m Calc mean of each section Xm(k)

k= 1 … N/m Calc variance VXm of these Xm(k) Vary m size of the blocks

Plot on log-log & fit slope β Hurst parameter H

β = 2H -2 Measure:

β = -0.355 which gives H 0.822 H =1 no long range dependence

VLBI Project: Packet Loss – Long Range Effects?

y = -0.355x + 2.8826

0

0.5

1

1.5

2

2.5

3

3.5

4

0 0.5 1 1.5 2 2.5 3sub-sample size Log10( m )

Ag

gri

ga

te-v

ari

an

ce L

og

10

( X

(m)

)


Traffic Flows Manchester – NetNorthWest - SuperJANET Access links

Two 1 Gbit/s

Access links:SJ4 to GÉANT GÉANT to SurfNet


High Performance TCP – DataTAG Different TCP stacks tested on the DataTAG Network 128 ms round trip time Drop 1 in 106

High-SpeedRapid recovery

ScalableVery fast recovery

StandardRecovery would

take ~ 10 mins


Drop 1 in 25,000 Rtt 6.2 ms Recover in 1.6 s

High Performance TCP – MB-NG

Standard HighSpeed Scalable


Some Network Tests TCP Request – Response

Zero stats OK done

Send statistics:CPU load & no. int1-way delay

Send event data

Request-Response time (Histogram)

Request event

Get remote statistics

●●● ●●●


Lab Test: TCP Request-Response Histograms PC – router – PC BE Test flow Request spacing 0 µs

Request spacing 10 ms

0.5M bytes man02-3_7may04

0

200

400

600

800

1000

4280 4300 4320 4340 4360 4380 4400 4420 4440 4460

Latency us

N(t

)

0.5M bytes w 10ms man02-3_7may04

0

200

400

600

800

4280 4300 4320 4340 4360 4380 4400 4420 4440 4460Latency us

N(t

)

1.0M bytes man02-3_7may04

0

200

400

600

800

1000

8580 8600 8620 8640 8660 8680 8700 8720 8740 8760

Latency us

N(t

)

2.0 M bytes man02-3_7may04

0

200

400

600

800

1000

17080 17100 17120 17140 17160 17180 17200 17220 17240 17260

Latency us

N(t

)

1.0 M bytes w 10ms man02-3_7may04

0

200

400

600

800

1000

8580 8600 8620 8640 8660 8680 8700 8720 8740 8760

Latency us

N(t

)2.0 M bytes w 10ms man02-3_7may04

0

200

400

600

800

1000

17080 17100 17120 17140 17160 17180 17200 17220 17240 17260

Latency us

N(t

)


Man-CERN: TCP Request-Response Latency DataTAG PC – backup link

BE Tests Request spacing 0 µs Win size 2.5Mbytes

Compare with UDP latency Large differences

Rtt of 20 msdelay*bw = 2.5 Mbytes

1Mbyte data = 690 pkts interesting bursts !

w05gva-gnt5_7May04_TCP

0

50000

100000

150000

200000

250000

300000

0 20000 40000 60000 80000 100000 120000 140000 160000

Message length bytes

Lat

ency

us

req-resp UDP latency us

ave time

w05gva-gnt5_7May04_TCP

0

100000

200000

300000

400000

500000

600000

0 20000 40000 60000 80000 100000 120000 140000 160000

Message length bytes

La

ten

cy

us

ave time

min time

max time


Man-CERN: UDP Throughput & Packet Loss DataTAG PC – backup link BE Tests Throughput

Packet loss

w05gva-gnt5_7May04_UDP

0

100

200

300

400

500

600

700

800

900

1000


Rec

v W

ire

rate

Mb

its/

s


w05gva-gnt5_7May04_UDP

0

2

4

6

8

10

12

14

16

18


% P

acke

t lo

ss



Traffic Flows Manchester – NetNorthWest - SuperJANET Access links

Link to PC in M/c Access links: 1 GE Man to NNW

Total Man to NNW

NNW to SuperJANET4



Divide time series of packets into 1000 slices of 50 packets

Total lost packets 1410 Average number / slice = 1.4

Calc Poisson Probability P(n, µ) = µ n e -µ

n!

Curves close but not exact Could be more than 1 process

VLBI Project: Packet Loss – Is it Poisson?

0

50

100

150

200

250

300

350

400

0 5 10 15n num lost in sub-sample

N(n

)

run12b

1

1.3

1.4

1.8


Traffic QoS Classes on GÉANT Backbone

Normal Traffic

Normal Traffic +

Less Than Best Effort 2.0 Gbit/s

Normal Traffic +

Radio Astronomy Data 500 Mbit/s

Normal Traffic +

Radio Astronomy Data +

Less Than Best Effort 2.0 Gbit/s

Max Throughput on 2.5 G PoS


Some Measurements made during ER2002

No LBE

0

2

4

6

8

10

12

14

16

18

20

0 20 40 60 80 100 120 140 160 180 200Transfer number

No.

Out

of

ord

er

0

5000

10000

15000

20000

25000

No

. L

ost

num_badorder

num_lost

With 1.8Gbit LBE

0

5000

10000

15000

20000

25000

30000

35000

40000

45000

0 20 40 60 80 100 120 140 160Transfer number

No.

Out

of

ord

er

0

5000

10000

15000

20000

25000

No

. L

ost

num_badorder

num_lost

Documents

Networking for ATLAS Remote Farms