Upload
dokhanh
View
213
Download
0
Embed Size (px)
Citation preview
Supporting Advanced Scientific Computing Research • Basic Energy Sciences • Biological and Environmental Research • Fusion Energy
Sciences • High Energy Physics • Nuclear Physics
Utilization of Transatlantic Circuits by European LHC Tier 2 Accesses of US LHC
Tier 1 Centers
February, 2010
William E JohnstonSenior Scientist, Energy Sciences Network
Lawrence Berkeley National [email protected], www.es.net
Managing Transatlantic Capacity Used by non-OPN LHC
• The Tier 2 accesses across the Atlantic (in both directions) utilize a mix of planned and ad hoc capacity as opposed to the carefully managed capacity of the OPN
• GÉANT provides transatlantic capacity for general IP traffic, as does NSF / IRNC (?), and several European countries
• ESnet partnered with USHLCnet some time ago to provide some capacity specifically to support European Tier 2 accesses of US Tier 1 centers
• What ESnet is seeing is LHC traffic coming from Europe on almost every available path, including once or twice over the commercial peerings
Managing Transatlantic Capacity Used by non-OPN LHC
• An analysis of the traffic from ESnet’s point of view seems essential in order to understand
- whether the current planned capacity for this traffic is adequate and - how it will need to evolve
• The current exercise (deliberately a bottom up view) is to- determine the traffic patterns,- see if they are optimal, and- establish a baseline for capacity prediction when the LHC starts producing
physics data
• Most of the European Tier 2 to US Tier 1 traffic is currently Layer 3 traffic, so focus on that
- What data is available to determine traffic patterns and how can it be used?
• What is presented here is definitely a work in progress
Layer 3 Paths Related to LHC traffic to ESnet
Clev.
Wash. DC
n
Atlanta
Nashville
USLHCUSLHC
ORNL
FNAL
BNLPhil
MAX
MANLAN
LHC OPNTo CERN
ESnet IP coreESnet Science Data Network core (N X 10G)ESnet SDN core, NLR links (backup paths)Other internationalLHC OPN (USLHCNet)Metro ring linkInternational GÉANT
5G
xGCH
FR
UK
AT
DE
FNAL
Notes:• All paths are 10Gb/s unless noted• The two ESnet transatlantic paths are carried on virtual circuits provided by USLHCnet to Amsterdam where
oGÉANT carries the 3G circuit to Vienna where it advertises ESnet routesoUSLHCNet carries the 2G circuit to CERN where CERN advertises ESnet routes internally
NL
Starlight
Logical Connectivity for ESnet, GÉANT, and USLHCnet
ESnet IP
32 AofA, New YorkMAN LAN
ESnet GÉANTVienna
Frankfurt
ESnetGÉANT
ESnet CERN
CERN (internal)
London
ESnetGÉANT
Amsterdam
ESnetGÉANT
ESnet IP
WashingtonMAX
Paris
ESnetGÉANT
Fermilab
5G
xG
What are Top Level Traffic Patterns?
ESnet IP
32 AofA, New YorkMAN LAN
ESnet GÉANTVienna
Frankfurt
ESnetGÉANT
ESnet CERN
CERN (internal)
London
ESnetGÉANT
Amsterdam
ESnetGÉANT
ESnet IP
WashingtonMAX
Paris
ESnetGÉANT
Fermilab
5G
xG
Who comes to me?
How do I get to
Fermilab?
Who Comes to Fermilab?
020406080
100120140160
CSTN
ET-A
S-AP
(749
7)VA
NDE
RBIL
T (7
212)
UN
L-AS
(789
6)JA
NET
(786
)RE
DIRI
S (7
66)
MIT
-GAT
EWAY
S (3
)DF
N-IP
(680
)Ru
ssia
n (6
801)
BELN
ET (2
611)
SON
NET
-AS
(159
)PR
INCE
TON
-AS
(88)
SWIT
CH (5
59)
NER
DCN
ET (6
356)
UCD
AVIS
-CO
RE (6
192)
DESY
-HAM
BURG
(175
4)IN
2P3
(789
)U
LTRA
LIGH
T (3
2361
)IT
EP (2
148)
UN
IV-O
F-O
KLAH
OM
A (2
5776
)RU
TGER
S (4
6)N
DGF
(395
90)
JHU
(572
3)TI
SK (2
501)
HEPN
ET-J
(250
5)CO
RNEL
L (2
6)RC
CN (1
930)
UVI
C-AS
(164
62)
NSC
KIPT
-AS
(352
96)
SIN
ET-A
S (2
907)
Fund
acao
(125
1)SI
NP-
MSU
(129
25)
KNU
-AS
(100
52)
IHEP
-SU
(264
3)ER
X-ER
NET
-AS
(269
7)BR
OW
N (1
1078
)GR
IDPN
PI (2
9493
)JIN
R/HE
PNET
(287
5)RA
DIO
-MSU
(268
3)YA
HOO
-GQ
1 (3
6647
)FR
-REN
ATER
(220
0)PI
ON
IER-
AS (8
501)
MBL
-AS-
AP (2
3674
)U
MN
-AGS
-NET
-AS
(217
)N
CREN
(81)
UCR
-EDU
(610
6)U
IUC
(38)
BNL-
AS (4
3)Re
de (1
916)
UM
DNET
(27)
EEN
et (3
221)
Mb/
s, a
vera
ge
FNAL as source, top 75 flows, past monthBase Date = 2009-12-28
Series1
020406080
100120140160
VAN
DERB
ILT
(721
2)AS
GARR
(137
)JA
NET
(786
)HE
CPER
N-A
S-PK
(457
73)
JHU
(572
3)LE
VEL3
(335
6)FU
NET
AS (1
741)
UN
L-AS
(789
6)U
CDAV
IS-C
ORE
(619
2)EE
Net
(322
1)U
NIV
-OF-
OKL
AHO
MA
(257
76)
REDI
RIS
(766
)CE
A-Sa
clay
(777
)BE
LNET
(261
1)IN
2P3
(789
)DF
N-IP
(680
)SU
RFN
ET-N
L (1
103)
UM
N-A
GS-N
ET-A
S (2
17)
ACO
NET
(185
3)RA
DIO
-MSU
(268
3)JIN
R/HE
PNET
(287
5)KF
KI-A
S (3
314)
PRIN
CETO
N-A
S (8
8)SO
NN
ET-A
S (1
59)
MIT
-GAT
EWAY
S (3
)SI
NET
-AS
(290
7)U
-CHI
CAGO
-AS
(160
)U
CSB-
NET
-AS
(131
)VI
RGIN
IA-A
S (2
25)
DESY
-HAM
BURG
(175
4)YA
HOO
-GQ
1 (3
6647
)N
ERDC
NET
(635
6)SW
ITCH
(559
)RC
CN (1
930)
KNU
-AS
(100
52)
WAS
HIN
GTO
N-A
S (7
3)TT
UN
ET (1
0421
)U
OI (
8581
)N
SCKI
PT-A
S (3
5296
)CS
TNET
-AS-
AP (7
497)
TISK
(250
1)Ru
ssia
n (6
801)
INDI
ANAG
IGAP
OP
(197
82)
BNL-
AS (4
3)N
CU-T
W (1
8420
)Re
de (1
916)
ERX-
CERN
ET-B
KB (4
538)
ULA
KNET
(851
7)HE
PNET
-J (2
505)
EEN
et (3
221)
Mb/
s, a
vera
ge
FNAL as source, top 75 flows, past monthBase Date = 2010-02-28
Series1
LHC Known European T2 sites per Ian Bird
LHC Unknown European sites
Nor
dune
t
Kha
rkov
, Ukr
aine
Ioan
nina
, Gre
ece
Kha
rkov
, Ukr
aine
How do they get to Fermilab on the US side?JANET (786)(UK)
aofa-sdn1 JINR/HEPNET (2875) star-cr1
REDIRIS (766)(Spain)
aofa-sdn1 RADIO-MSU (2683) star-cr1
DFN (680)(Germany)
aofa-sdn1 FR-RENATER (2200) aofa-sdn1
Russian (6801) star-cr1 PIONIER-AS (8501) aofa-sdn1
BELNET (2611)(Belgian)
aofa-sdn1 EENet (3221) aofa-sdn1
SWITCH (559)(Switzerland)
aofa-sdn1 ASGARR (137) aofa-sdn1
FUNETAS (1741) aofa-sdn1
DESY-HAMBURG (1754) (Germany)
aofa-sdn1 CEA-Saclay (777) aofa-sdn1
IN2P3 (789) (France)
aofa-sdn1 SURFNET-NL (1103) aofa-sdn1
ITEP (2148)(Russia)
star-cr1 ACONET (1853) aofa-sdn1
NDGF (Nordunet) 39590 (Nordic)
aofa-sdn1 KFKI-AS (3314) aofa-sdn1
RCCN (1930)(Portugal)
aofa-sdn1 NERDCNET (6356) star-cr1
NSCKIPT-AS (35296)(Ukraine)
eqx-chi-rt1(ouch)
UOI (8581) aofa-sdn1
IHEP-SU (2643) star-cr1 Rede (1916) aofa-sdn1
GRIDPNPI (29493) (Russia)
aofa-sdn1 ULAKNET (8517) aofa-sdn1
How do they get to Fermilab on the US side?
• The traffic through star-cr1 is on an IRNC (?) or country provided path or GLORIAD path
• Not all of FNAL’s traffic is LHC, though most of it is (e.g. FNAL is also the data center for the Sloan Digital Sky Survey)
Top Level Traffic Patterns
ESnet IP
32 AofA, New YorkMAN LAN
ESnet GÉANTVienna
Frankfurt
ESnetGÉANT
ESnet CERN
CERN (internal)
London
ESnetGÉANT
Amsterdam
ESnetGÉANT
ESnet IP
WashingtonMAX
Paris
ESnetGÉANT
Fermilab
5G
xG
Who comes to me?
How do I get to
Fermilab?
Austria
Belgian
UK
Switzerland
Estonia
Spain
Germany
Looking Glass to traceroute from both ends
By-Site Traffic: FNAL-> EuropeThe pattern of substantially more Tier 1 (FNAL) to Tier 2 (Europe) traffic compared to T2 to T1 is at least approximately consistent with theEuropean Tier 2s getting data from the US Tier 1s
Next Steps
• Sort by-site traffic by path• Disambiguate LHC site traffic in European ASNs• Decide what is useful to publish monthly
- Engage Internet2 to publish similar data for their network
• Would expect to see the complimentary pattern –US Tier 2s pulling data from European Tier 1s
- Probably need to also involve NLR and the new IRNC links as some of the US Tier2 connect directly to the New York exchange point (MAN LAN) and if GEANT peers with them, Internet2 would not see the traffic
YUCCA MT
PNNL
SNLAAlliedSignal
ARM
KCP
NOAA
OSTI
ORAU
SRS
JLAB
PPPLLab DCOffices
MIT/PSFC
BNL
NREL
GA
DOE GTNNNSA
NNSA Sponsored (13+)Joint Sponsored (4)Other Sponsored (NSF LIGO, NOAA)Laboratory Sponsored (6)
~45 end user sites
SINet (Japan)Russia (BINP)CA*net4
FranceGLORIAD(Russia, China)
Korea (Kreonet2
Japan (SINet)Australia (AARNet)Canada (CA*net4Taiwan (TANet2)SingarenTranspac2CUDI
commercial peering points
PAIX-PAEquinix, etc.
ESnet
ESnet core hubs
CERN/LHCOPN(USLHCnet:
DOE+CERN funded)
GÉANT- France, Germany,
Italy, UK, etc
JGILBNLSLAC
NERSC
Equinix
MRENStarTapTaiwan (TANet2,
ASCGNet)
NASA
Ames
AU
AU
Specific R&E network peers
AMPATHCLARA
(S. America)CUDI(S. America)
R&Enetworks
Office Of Science Sponsored (22)
NSF/IRNCfunded
KAREN/REANNZODN Japan Telecom AmericaNLR-PacketnetInternet2Korea (Kreonet2)
KAREN / REANNZ Transpac2Internet2 Korea (kreonet2)SINGAREN Japan (SINet)ODN Japan Telecom America
NETL
International peers (10 Gb/s)10-20-30 Gb/s
SDN core (I2, NLR)10Gb/s IP coreMAN rings (Nx10 Gb/s)Lab supplied linksOC12 / GigEthernetOC3 (155 Mb/s)45 Mb/s and less
Salt Lake
DOE
Geography isonly representationalOther R&E peering points
USH
LCN
etto
GÉA
NT
CA*net4
GFDLPU Physics
UCSD Physics
NSTEC
An accurate map is located at http://www.es.net/pub/maps/current.pdf
ESnet Vienna peering with GÉANT (via USLHCNet circuit)