Data Transfers in the Grid: Workload Analysis of Globus GridFTP Nicolas Kourtellis, Lydia Prieto, Gustavo Zarrate, Adriana Iamnitchi University of South

Data Transfers in the Grid: Data Transfers in the Grid: Workload Analysis of Globus GridFTPWorkload Analysis of Globus GridFTP

Nicolas Kourtellis, Lydia Prieto, Gustavo Zarrate, Nicolas Kourtellis, Lydia Prieto, Gustavo Zarrate, Adriana IamnitchiAdriana IamnitchiUniversity of South FloridaUniversity of South Florida

Dan FraserDan FraserArgonne National LaboratoryArgonne National Laboratory

22

33

Objective 1Objective 1: : Quantify volume of transfersQuantify volume of transfersWhat is the transfer size distribution?What is the transfer size distribution?What is the volume of activity for the most active hosts?What is the volume of activity for the most active hosts?

Objective 2Objective 2: : Understand how tuning capabilities are Understand how tuning capabilities are usedused

What are the buffer sizes used during the transfers?What are the buffer sizes used during the transfers?What is the average bandwidth?What is the average bandwidth?What is the utilization of functionalities like streams and What is the utilization of functionalities like streams and stripes?stripes?

Objective 3Objective 3: : Quantify user base and predict usage Quantify user base and predict usage trendstrends

How does the user base evolve over time?How does the user base evolve over time?What are the geographical characteristics of the GridFTP What are the geographical characteristics of the GridFTP data transfers?data transfers?

44

OutlineOutline

Metrics datasetMetrics dataset

Surprises and …Surprises and …

… … zoom in (TeraGrid)zoom in (TeraGrid)

Lessons and discussionsLessons and discussions

55

GridFTP Metrics DatasetGridFTP Metrics DatasetField Range of Values Comment

Source hostname/host IP String/IPnet Anonymized

Start time of the transfer Timestamp Accuracy: ms

End time of the transfer Timestamp Accuracy: ms

TCP Buffer Size Integer (Bytes) ≥0

Total Number of Bytes Integer (Bytes) ≥0

Number of Streams Integer ≥1

Number of Stripes Integer ≥1

Store or Retrieve Integer (0, 1,2)STOR, RETR,

LIST

66

Started with ~137.5 million records (Jul’05 - Mar’07)Started with ~137.5 million records (Jul’05 - Mar’07)Cleaning:Cleaning:

– transfer size transfer size ≤≤0: 0: ~22.8 million records ~22.8 million records – buffer size <0: buffer size <0: ~1000 records~1000 records– directory listings: directory listings: ~3.9 million records~3.9 million records– invalid hostnames (e.g., /[B@89712e): invalid hostnames (e.g., /[B@89712e): ~4,600 records~4,600 records– ANL-TeraGrid testing: ANL-TeraGrid testing: ~11.4 million records~11.4 million records– duplicate reports: duplicate reports: ~16.8 million records~16.8 million records– self transfers (source=destination): self transfers (source=destination): ~5.75 million records~5.75 million records

Clean database: ~77.2 million records (~56.2%)Clean database: ~77.2 million records (~56.2%)

Metrics DatasetMetrics Dataset

77

Surprise #1: Transfer Size DistributionSurprise #1: Transfer Size Distribution

0

5

10

0 2 4 8

16

32

64

12

8

25

6

51

2 1 2 4 8

16

32

64

12

8

25

6

51

2 1 2 4 8

16

32

64

12

8

25

6

51

2 1

Bytes KB MB Transfer size

% o

f to

tal

Objective 1Objective 1: : Quantify volume of transfersQuantify volume of transfers

88

Zoom-in: TeraGridZoom-in: TeraGrid

Are these results representative for Are these results representative for production grids?production grids?– GridFTP testing for deployment and learningGridFTP testing for deployment and learning

Identify transfers from TeraGRid and Identify transfers from TeraGRid and analyze dataset.analyze dataset.

99

Transfer Size Distribution (TG)Transfer Size Distribution (TG)

0

5

10

15

20

25

0 2 4 8

16

32

64

12

8

25

6

51

2 1 2 4 8

16

32

64

12

8

25

6

51

2 1 2 4 8

16

32

64

12

8

25

6

51

2 1

Bytes KΒ MΒTransfer Size

% o

f to

tal


1010

0

5

10

0 2 4 8

16

32

64

12

8

25

6

51

2 1 2 4 8

16

32

64

12

8

25

6

51

2 1 2 4 8

16

32

64

12

8

25

6

51

2 1

Bytes KB MB Transfer size

% o

f to

tal

0

5

10

15

20

25

0 2 4 8

16

32

64

12

8

25

6

51

2 1 2 4 8

16

32

64

12

8

25

6

51

2 1 2 4 8

16

32

64

12

8

25

6

51

2 1

Bytes KΒ MΒTransfer Size

% o

f to

tal

All

TG

1111

Why So Small Transfers?Why So Small Transfers?

There are still many old versions (i.e., before There are still many old versions (i.e., before v3.9.5) of GridFTP in use. These versions do v3.9.5) of GridFTP in use. These versions do not include trace reporting capabilities.not include trace reporting capabilities.

Other data transfer protocols and Other data transfer protocols and implementations are usedimplementations are used

Users have turned off the reporting capabilityUsers have turned off the reporting capability Some of the logs are inevitably lost due to the Some of the logs are inevitably lost due to the

UDP-based reporting mechanismUDP-based reporting mechanism The low transfer volumes could suggest a shift The low transfer volumes could suggest a shift

towards data-aware job scheduling (?)towards data-aware job scheduling (?)

1212

Server to Server TransfersServer to Server Transfers

– High reporting of Self Transfers (more than 1/3)High reporting of Self Transfers (more than 1/3)

21.7%

39.5% 38.8%

72.2%

19.7%

8.2%

0%

10%

20%

30%

40%

50%

60%

70%

80%

InterDomain InterIP SelfTransfers

# Transfers Volume


1313

Top 6 Active Hosts (all)Top 6 Active Hosts (all)

Top 6 hosts traffic adds up to ~28% of total volumeTop 6 hosts traffic adds up to ~28% of total volume Next 48 hosts (IPs) transferred 10s of TBNext 48 hosts (IPs) transferred 10s of TB

0

50

100

150

200

250

1 2 3 4 5 6Host

Vo

lum

e T

ran

sfe

rre

d (

TB

)

1.0E+00

1.0E+01

1.0E+02

1.0E+03

1.0E+04

1.0E+05

1.0E+06

1.0E+07

1.0E+08

Nu

mb

er

of

Tra

ns

fers

Volume Transferred Number of Transfers


1414

Number of Transfers & Volume Number of Transfers & Volume (TG)(TG)

0

500

1000

1500

2000

2500

3000

Au

g-0

5

Se

p-0

5

Oct-

05

No

v-0

5

De

c-0

5

Ja

n-0

6

Fe

b-0

6

Ma

r-0

6

Ap

r-0

6

Ma

y-0

6

Ju

n-0

6

Ju

l-0

6

Au

g-0

6

Se

p-0

6

Oct-

06

No

v-0

6

De

c-0

6

Ja

n-0

7

Fe

b-0

7

Ma

r-0

7

Month-Year

Nu

mb

er

of

Tra

ns

fers

(th

ou

sa

nd

s)

0

10

20

30

40

50

60

70

80

Vo

lum

e (

TB

)

Number of Transfers per Month

Total Volume per Month


1515

Average Transfer SizeAverage Transfer Size& Total Volume (TG)& Total Volume (TG)

0

20

40

60

80

100

120

140

160

1 2 3 4 5 6 7 8

TERAGRID SITES

To

tal V

olu

me

(T

B)

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

Av

era

ge

Tra

ns

fer

Siz

e (

GB

)

Total Volume (TB) Average Transfer Size (GB)


1616

Daily Workload (TG)Daily Workload (TG)

Average volume transferred per day: ~ 0.6TBAverage volume transferred per day: ~ 0.6TB GridFTP doesn’t get weekends free!GridFTP doesn’t get weekends free!

0.00

0.20

0.40

0.60

0.80

1.00

1.20

1.40

Monday TuesdayWednesdayThursday Friday Saturday Sunday

Vo

lum

e (T

B)

0

5000

10000

15000

20000

25000

30000

35000

40000

45000

50000

Nu

mb

er o

f T

ran

sfer

s

Average Volume per Day Average Number of Transfers per Day


1717

Monthly Workload (TG)Monthly Workload (TG)

~50,000 transfers per day~50,000 transfers per day ~1TB per day of total volume~1TB per day of total volume Lowest around 0.5TB per dayLowest around 0.5TB per day Peaks due to particular daysPeaks due to particular days

0.0

0.5

1.0

1.5

2.0

2.5

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31

Day of the month

Vo

lum

e (

TB

)

0

20000

40000

60000

80000

100000

120000

140000

Nu

mb

er

of

Tra

ns

fers

Average Volume Transferred per day Average Number of Transfers per day


1818

Objective 1: Quantify volume of transfersObjective 1: Quantify volume of transfersWhat is the transfer size distribution?What is the transfer size distribution?What is the volume of activity for the most active hosts?What is the volume of activity for the most active hosts?

Objective 2Objective 2: : Understand how tuning capabilities are Understand how tuning capabilities are usedused




1919

Surprise #2: Usage of Streams and StripesSurprise #2: Usage of Streams and Stripes

Unreliably reported (from Globus team).Unreliably reported (from Globus team). Reliable observations:Reliable observations:

• At least ~20% of the transfers used 4 streams At least ~20% of the transfers used 4 streams (suggested number by ANL’s website)(suggested number by ANL’s website)

• At least ~10% of the transfers used a different At least ~10% of the transfers used a different value, larger than one.value, larger than one.

• Maximum number of streams reported: 1010 (!!)Maximum number of streams reported: 1010 (!!)

Objective 2Objective 2: : Understand how tuning capabilities are usedUnderstand how tuning capabilities are used

2020

Buffer Size DistributionBuffer Size Distribution

• 60% from the original table60% from the original table: OS-controlled (0 bytes): OS-controlled (0 bytes)• Most commonly used: 16–128KBMost commonly used: 16–128KB• Largest buffer size: 1-2GB (92 records)Largest buffer size: 1-2GB (92 records)

0

1

2

3

4

5

6

7

8

9

4 8 16 32 64 128 256 512 1 2 4 8 16 32

KB MBBuffer Size

% o

f to

tal

Objective 2Objective 2: : Understand how tuning capabilities are usedUnderstand how tuning capabilities are used

2121

Average Bandwidth DistributionAverage Bandwidth Distribution

• Peak: 128-256Mbps, ~7.7 million recordsPeak: 128-256Mbps, ~7.7 million records• Most common (58%): 4Mbps—1GbpsMost common (58%): 4Mbps—1Gbps

0

2

4

6

8

10

0 2 4 81

63

26

41

28

25

65

12 1 2 4 8

16

32

64

12

82

56

51

2 1 2 4 81

63

26

41

28

25

65

12 1 2 4

bps Kbps Mbps Average Bandwidth

% o

f to

tal

2222

Average Bandwidth Distribution (TG)Average Bandwidth Distribution (TG)

Compared to the total dataset:Compared to the total dataset:=> => The region of 4Mbps to 1Gbps includes more than 85% of the The region of 4Mbps to 1Gbps includes more than 85% of the

transfers (58% for the whole dataset)transfers (58% for the whole dataset)

0

2

4

6

8

10

12

14

16

18

201 2 4 8 16 32 64 128

256

512 1 2 4 8 16 32 64 128

256

512 1 2 4

Kbps Mbps GbpsAverage Bandwidth

% o

f to

tal

2323

Objective 1: Quantify volume of transfersObjective 1: Quantify volume of transfersWhat is the transfer size distribution?What is the transfer size distribution?What is the volume of activity for the most active hosts?What is the volume of activity for the most active hosts?

Objective 2: Understand how tuning capabilities are Objective 2: Understand how tuning capabilities are usedused




2424

Surprise #3: Geographic DistributionSurprise #3: Geographic Distribution

USA: 78.4% or ~50.8 million transfers and 82.9% or ~1.7 PBUSA: 78.4% or ~50.8 million transfers and 82.9% or ~1.7 PB Canada+Taiwan+Japan+Spain:~14M transfers and 346TBCanada+Taiwan+Japan+Spain:~14M transfers and 346TB 49 different countries and 446 different cities (178 cities from USA)49 different countries and 446 different cities (178 cities from USA)

0% 1% 2% 3% 4% 5% 6% 7% 8% 9% 10%

CA

TW

JP

IT

DE

AU

CH

ES

GB

AT

# transfers volume

Objective 3Objective 3: : Quantify user base and predict usage trendsQuantify user base and predict usage trends

2525

User and Domain Evolution (all)User and Domain Evolution (all)

Continuing increase of user and organization populationContinuing increase of user and organization population Forecasts: 67 new IPs and 14 new domains per monthForecasts: 67 new IPs and 14 new domains per month

y = 14.132x + 67.078

R2 = 0.9107

0

200

400

600

800

1000

1200

Ju

l-0

5A

ug

-05

Se

p-0

5O

ct-

05

No

v-0

5D

ec

-05

Ja

n-0

6F

eb

-06

Ma

r-0

6A

pr-

06

Ma

y-0

6J

un

-06

Ju

l-0

6A

ug

-06

Se

p-0

6O

ct-

06

No

v-0

6D

ec

-06

Ja

n-0

7F

eb

-07

Ma

r-0

7

IPs

0

50

100

150

200

250

300

350

400

Do

ma

ins

# IPs # Domains Linear Fit (# Domains)

Objective 3Objective 3: : Quantify user base and predict usage trendsQuantify user base and predict usage trends

2626

Summary of ResultsSummary of Results

Many transfers in the range of KBs to 10s MB Many transfers in the range of KBs to 10s MB (peak in 16MB-32MB).(peak in 16MB-32MB).

– relevant for setting up realistic simulations.relevant for setting up realistic simulations.– previous work assume different, larger file sizes.previous work assume different, larger file sizes.

Bandwidth measured in previous work is Bandwidth measured in previous work is confirmed by our workload analysis.confirmed by our workload analysis.

Tuning parameters:Tuning parameters:– Users tend not to set the buffer size explicitly (60%), leaving it to Users tend not to set the buffer size explicitly (60%), leaving it to

the OSthe OS– The unexpectedly small transfers do not justify tuning GridFTP The unexpectedly small transfers do not justify tuning GridFTP

parameters (stripes and streams)parameters (stripes and streams) The usage of Globus GridFTP is growing over The usage of Globus GridFTP is growing over

time in terms of IPs (users), domains time in terms of IPs (users), domains (organizations), and volume transferred.(organizations), and volume transferred.

Missing some of the big playersMissing some of the big players

??

Documents

Data Transfers in the Grid: Workload Analysis of Globus GridFTP Nicolas Kourtellis, Lydia Prieto, Gustavo Zarrate, Adriana Iamnitchi University of South