1

Status Report on US networks at the Turn of the Century

Les Cottrell – SLAC & Stanford U.www.slac.stanford.edu/grp/scs/net/talk/us-net-status-2000.htm

Presented at CHEP00, Padua Italy, February 9, 2000

Partially funded by DOE/MICS Field Work Proposal on Internet End-to-end Performance Monitoring (IEPM), also supported by IUPAP

2

Overview• U.S. Networks:

– Internet 2

– Federal networks, in particular ESnet

– Commercial

• Performance seen from U.S.: – Compare Internet2 vs. ESnet vs Commercial

– Performance to Europe and rest of world

– Trends

• Summary

3

US National Networks• Internet 2 - universities

– Abilene & vBNS backbones

• Federal backbone networks– ESnet (DoE), DREN (DoD), NREN (NSF), NSI

(NASA) ...

• Commercial Internet Service Providers (ISPs)

• Interconnection points (MAEs, NAPs, NGIXs, & colocation points …)

4

Internet 2• A project by consortium of universities (UCAID) to:

– foster development of advanced internet applications;

– foster development internet technology itself;

– provide a high performance network for general research.

• Not a Government project; no direct Federal subsidy

• Not a network itself:

– The NSF-funded vBNS evolved into a ‘pre’ Internet-2 backbone;

– Abilene is the UCAID-sponsored Internet-2 backbone.

5

Internet 2 Membership• Internet 2

– 170 universities– ~ 10-12 non-university members including CERN– other networks can connect as affiliates;– U.S. National Labs are not members of UCAID:

• National Lab Internet-2 participation assumed thru ESnet

• vBNS– 101 institutions connected– 22 peer networks

• Abilene– 96 participants, 75 connected– 15 peer networks

6

Internet 2 AUP• Internet 2 Acceptable Use Policy (AUP)

prohibits transit traffic between affiliates:– ESnet site with Internet-2 member site is ‘OK’ to use I2;– ESnet site to other affiliated network site via Internet-2

backbone is NOT permitted.

• Internet 2 AUP also prohibits “commodity” internet traffic!– Traffic to another Internet-2 site routed via Internet-2

backbone;– Traffic to a non-Internet-2 site must be routed via

commercial ISP;– Requires universities to have a separate Internet

connection for commodity traffic.

7

• 5500 miles deployed, backbone operates at speeds up 2.4Gbps. Interconnections at 155 & 622Mbps

Abilene

Seattle San Fran.

CalRen UCB

Stanford

UCSFUCDavis

UCSC

• Universities connect to GigaPoPs, GigaPoPs connect to backbone

• Peer with: APAN/Transpac, CA*net-2, DANTE, DFN, DREN, ESnet, ILAN, INFN, JANET, NACSIS, NORDunet, NISN, NREN, RENATER, SingAREN, SURFnet, vBNS …

Red=N.America, Blue=Europe, Green=Asia

ESnet

8

Abilene Network, Jan-2000

9

vBNS• ATM-based OC12 (622Mbps) backbone, with 2.4 Gbps IP-

over-SONET segments on parts of the backbone:– 45Mbps is the minimum connection line speed;– 98 connections as of 6/15, with 7 more pending;

• Run by MCI; subsidized out of NSF NGI funds;– NSF vBNS contract expires April 1st, 2000;

– June 99 MCI announces vBNS+ a 5 year agreement with EDUCAUSE

• 1750 higher education institutions members• vBNS+ no restrictive NSF imposed AUP, • Connection speeds from 1.5Mbps - 2.5Gbps• peering: with vBNS requires NSF authorization, none at

moment with Abilene, will offer to FEDnet and International nets

10

vBNS (101 institutions)

11

ESnet• DOE Energy Sciences Research Network

• Connects DOE ER labs and universities with major DoE funded projects– ~ 50 sites– Mainly 155 Mbps backbone with some 622Mbps links

• Peers with other major networks– 13 Internet Interconnect points– Peering exchange at MAE-West, MAE-East, Sprint NAP,

Ameritech NAP, PacBell NAP

• International connections– CERN, DFN, INFN, JAERI, KEK, Moscow, NIFS

12

ESnet• 100% growth/year since 1990

• New contract, Sprint did not bid– New contract with Qwest announced January 4, 2000 – 1 year transition, 2 concurrent contracts for coming year

means funding tight– Same supplier as US-CERN link (KPN-Qwest)– Initial deployment ATM based

• ESnet3 backbone to be Tbit/sec by 2003-2005– 5 major hubs see next transparency ...

13

ESnet-3 Initial Configuration

SNV

ALBORN

NYC

CHI

LANL

SNLA

BNLTELEHOUSE

OC48-ATM

OC12-ATM

OC3-ATM

OC48-SONET

OC3-SONET

T3-ATM

T3

MIT

CHI-NAP PPPL

ORNL

ATL

SRS

ANL

FNAL

AMES

PANTEX

JLAB

GTN

ASIG

60-HUD

DCOffices

DC

SNLL

LLNL

LBNL

NERSC

FIX-W

PB-NAP

MAE-W

OC12-SONET

SLAC

MAE-E

GA

JGI

PNNL

GA

(SDSC)

YUCCA-MT

(BECHTEL)

SEA

(SAIC)

FULL-MESHEDATM CORE

INEEL

JAnetSURFnet

NORDUnetAbilene

DFNINFN

DANTE

OC3?

CanadaFranceCERNKEK/China

Japan/Russia

Courtesy of Jim Leighton/ESnet

14

Testbeds for U.S. NRENs• “Foster Development of Internet Technology”

– IPv6 = Next Generation Internet• volunteers see warrenm@slac.stanford.edu

– QoS - VoIP, multimedia and data transfer– Computing & Data grids– Collaboratories - video, virtual reality, electronic

notebooks, multicast ..– Middleware - PKI, directories ...

15

Commercial Internet• Important to HENP

– ~ 18% of US HEP universities still rely solely on commercial ISPs for internet access:

• Internet-2 participation, even subsidized, isn’t cheap…

– Critical information needed from commercial sites

• Quality of US commercial Internet Service is improved:– Commercial ISPs have been keeping their backbone

capacity in line with (or ahead of…) demand;– Network Access Point (NAP) congestion is down.

• ISPs match research networks technologically:– ISPs are ahead in rollout of high bandwidth links;– ISPs are pursuing Quality of Service solutions

16

Commercial traffic: Ames Internet Exchange (AIX)

https://anala.caida.org/AIX/• TCP 90%, UDP 10%, Web ~ 55%, FTP ~ 5%, mail

~ 3%• Game traffic represents noticeable portion

– Quake & Starcraft account for 5% in summer, 2-3% term-time

– Fairly constant (0.5%)

• Real audio declining (factor 2 in 6 months, now 1%)• IPSEC traffic, small (< 0.2%) but growing (factor 3

in 6 months)• Spikes in ICMP (security scans?)

17

Performance• Measurements from

– 28 monitors in 15 countries

– Over 500 remote hosts– 72 countries (covers all 56 PDG booklet countries)

– Over 1200 monitor-remote site pairs

• Over 50% of HENP collaborator sites are explicitly monitored as remote sites by PingER project– Atlas (37%), BaBar (68%), Belle (23%), CDF (73%),

CMS (31%), D0 (60%), LEP (44%), Zeus (35%), PPDG (100%), RHIC(64%)

18

How are the U.S. Nets

doing?

In general performance is good (i.e. <= 1%).

Edu (vBNS/Abilene) is catching up with ESnet

XIWT (70% .com) 3-5 times worse than ESnet | I2

19

Europe seen from U.S.

650ms

200 ms

7% loss10% loss

1% loss

Monitor siteBeacon site (~10% sites)HENP countryNot HENPNot HENP & not monitored

20

Asia seen from U.S.

3.6% loss

10% loss

0.1% loss

640 ms

450 ms

250ms

21

Latin America, Africa & Australasia4% Loss

2% Loss

350 ms

700ms

170 ms

220 ms

22

Bulk transfer - Performance TrendsBandwidth TCP < 1460/(RTT * sqrt(loss))

Note: E. Europe NOT catching up

23

Summary• US HEP research network research environment is

improving…

• The Internet-2 project is already having positive results for collaborative research:– More research universities have high bandwidth, low

latency access to major U.S. research facilities;– Mission-specific research networks have something to

direct improving university access efforts at…;– Testbed projects emerging for new network technologies.

• International performance from US to sites outside W. Europe, Japan, Korea is generally poor to bad

24

More Information• ESnet home page

– http://www.es.net/

• Internet 2 home page– http://www.internet2.edu/

• vBNS+ home page– http://www.vbns.net/vBNS+/index.html

• IEPM/PingER home site– http://www-iepm.slac.stanford.edu/

• ICFA-SCIC Homepage– http://www.hep.net/ICFA/index.html

25

Peering• Not always optimal

– paths may go through congested exchange points -increased loss

– paths may be very indirect (e.g. KEK to SLAC was via NY) - adds 80 msec to RTT

Performance28 monitors in 15 countriesOver 500 remote hostsOver 1200 pairs72 countriesOver 50% HENP sites are monitored directly