Internet2: A Tutorial Part 2 of 4 17 th Brazilian Symposium on Computer Networks Paul Love, Internet2 Chair, I2 Topology WG [email protected]

Internet2: A TutorialPart 2 of 4

17th Brazilian Symposiumon

Computer Networks

Paul Love, Internet2Chair, I2 Topology WG

[email protected]

Abilene

25-28 May 99SBRC99 Salvador, Bahia, Brazil

Overview

• Abilene Background

• Review of Abilene Architecture

• 1999 Schedule

• Launched Network

• First Half 1999 Expectations

• Engineering Issues

• Additional informationhttp://www.internet2.edu/abilene/


Abilene: Objectives

• Provide high-quality, widely available Interconnect among participating gigaPoPs/universities

• IP as the bearer service

• Goals• Reliability• Functionality• Latency• Capacity


Abilene: Objectives cont

• Additional, separate networks to • Test advanced network capabilities

• Final proposals being evaluated this month• For network research - still TBD

• Support advanced functionality as it evolves• Multicast, QoS, IPv6


Abilene Team

• Overall direction by Internet2

• Qwest

• Cisco

• Nortel (Northern Telecom)

• Indiana University

• National Center for Network Engineering

• Open to other contributors • Collaborate with related efforts in

network or applications research


Key Attributes

• IP over Sonet initially

• Qwest OC-48 Sonet capacity and collocation sites

• Nortel OC-192 Sonet equipment

• Cisco GSR 12000 routers including support for • QoS• Native multicast


Abilene and Other Networks• Important for Abilene to interconnect with other high

performance networks• First I2 backbone: vBNS• Federal Agency NGI networks (ESnet, NREN, DREN, ...)• Non-US Research & Education Networks

• CANARIE, NORDUnet, SURFnet, TransPac, etc• Peering in hand or plans in process

• vBNS, ESnet, CANARIE, DREN, NREN, NORDUnet, SURFnet, etc

• Advanced functionality across I2 & our peers• Native multicast• QoS across multiple interconnected networks

*RSN


Broader Internet2, NGI, and International Advanced Nets

Initial NGIX sitesPossible CA*net3 peering sitesStarTapStarTap Plus


Notes on I2, NGI, and International Interconnections• NGIXes are the foundation for peering I2

backbones & with NGI nets• Abilene will connect with OC12

• StarTap is the foundation of international interconnections - universal solution

• New York is first example of StarTap Plus• Minimize latency• Provide redundancy• Abilene doesn’t provide transit


Example: New York City

• SURFnet has OC-3 circuits and routers at the Teleglobe POP at 60 Hudson Street

• NORDUnet the same

• And each connects from New York to the StarTap with DS3

• Details on the next slides


SURFnet

NORDUnet

Chicago

New York City Europe

Teleglobe

Qwest

Abilene

ISP


SURFnet

NORDUnet

Chicago

New York City Europe

Teleglobe

Qwest

Abilene

ISP


CA*net3

Charlottetown

Vancouver

Calgary ReginaWinnipeg

Ottawa

MontrealHalifax

Fredericton

Chicago Toronto

St. John’s


Abilene NOC

• Located at Indiana University

• Publish statistics & online information resources

• Coverage can be looked at in three broad areas• Administration• Engineering• Operations


Administration

• Security Management

• Communications

• Overall Management and Coordination


Engineering

• Configuration

• Testing

• Data collection

• Traffic Analysis


Operations

• 24x7 Monitoring

• Reporting

• Change and Problem management

• Information base

• Single point of contact• Network problems• Abilene connection process

SBRC99 Salvador, Bahia, Brazil25-28 May 99

Online Traffic Map


For more informationon the Abilene NOC

• www.abilene.iu.edu

• Abilene traffic map athttp://hydra.uits.iu.edu/~abilene/traffic


Couple of Notes on Costs

• Cost recovery on UCAID’s part• Set each year for following year

• Participant institution • Connector - based on BW

(can be connector and not participant)• nb There is also a small Qwest access fee

• Long-distance part of access line is considered part of the ‘backbone’

• Thus, number/location of core nodes does not affect costs borne by connecting sites


Qwest Fiber Networkhttp//www.qwest.com/network/netmaps.html


Schedule - part 1

• Design work: Mar-98 and ongoing

• Rack design: May-98 to Jul-98

• Initial assembly / testing: Jul/Aug-98

• Router Nodes / Interior Lines: Jul/Aug-98

• Demo network installed: Sep-98

• Remainder installed: Nov/Dec-98

• Production begins: 29-Jan-99

• Launch Event in DC: 24-Feb-99


Seattle

Kansas City

Denver

Cleveland

New York

Atlanta

Houston

Pittsburgh

Minneapolis

ColumbusWashington

Phoenix

Raleigh

TrentonSalt Lake City

Wilmington

Dallas

New Orleans

Lincoln

New Haven

Detroit

Miami

Westfield

Nashville

Philadelphia

Indianapolis

Newark

AlbuquerqueOklahoma City

Access Node

Router Node

Star Tap

Sacramento

Oakland

Eugene

Los Angeles

Anaheim

Boston

Abilene Sep 98 Demo Network


Abilene Architecture: Core• Router Nodes located at Qwest PoPs

• Cisco 12008 GSR• ICS Unix PC: IPPM and Network Mgmt• Cisco 3640 Remote Access for NOC• 100BaseT LAN and ‘console port’ access• Remote 48v DC Power Controllers

• Initially, ten Router Nodes• Seattle, Sacramento, Los Angeles, Denver,

Kansas City, Houston, Indianapolis, Cleveland, New York City, Atlanta

• And, coming this Fall - Washington, DC

• Interior lines connect core nodes• OC-12 and OC-48 Sonet• IP-over-Sonet interfaces



Abilene Architecture: Access• Access Nodes

• Located at Qwest PoPs• Sonet: Connects Local to Long-distance

• Currently about 135 Access Nodes:• This list grows as the Qwest Sonet plant grows

• Access lines connect from core node to gigaPoP or member• IP-over-Sonet or IP-over-ATM possible• OC-3 and OC-12 typical• Others as possible & appropriate

• GigE, ...


Seattle

Kansas City

Denver

Cleveland

New York

Atlanta

Houston

Pittsburgh

Minneapolis

ColumbusWashington

Phoenix

Raleigh

TrentonSalt Lake City

Wilmington

Dallas

New Orleans

Lincoln

New Haven

Detroit

Miami

Westfield

Nashville

Philadelphia

Indianapolis

Newark

AlbuquerqueOklahoma City

Access Node

Router Node

Sacramento

Oakland

Eugene

Los Angeles

Anaheim

Boston

Chicago

Abilene + some Access Nodes


Launch Network• Some Sonet Rings not completed

• Mix of OC-12 and OC-48 interior Lines

• NOC fully operational

• Some Access Lines not completed

• Connections at two NGIXes• NGIX/Central (OC3)• NGIX/West (OC12)

• OC12 into Union Station for demos


Schedule, part 2

• Production began: 29-Jan-99

• Launch Event in DC: 24-Feb-99

• Member Meeting: 28-Apr-99

• Native Multicast deployed in core: Jun-99

• Completion of OC-48 Core: TBD

• Addition of DC Router Node: 3Q99

• Seattle Member Meeting: 11/15-Oct-99


First Half 1999 Expectations

• Measurements of Delay, Loss, Utilization

• Gradual• Increase in OC-48 Interior Lines• Completion of (Qwest) Sonet Rings• Accesss to measurements and other

engineering data

• Participation in QBone

• Upgrade of NGIX/Central to OC12

• Native Multicast


Second Half 1999 Expectations

• Gigabit Ethernet connection(s)?

• OC48 connection?

• Connection to NGIX/East• Somewhere in Washington, DC, area• OC12


Abilene Core: CurrentSeattle

Kansas City

Denver

Cleveland

New York

Atlanta

Houston

IndianapolisSacramento

Los Angeles


Abilene: Current plus On OrderSeattle

Kansas City

Denver

ClevelandNew York

Atlanta

Houston

IndianapolisSacramento

Los Angeles

OC12

OC3

OC12 still being installed

OC12

OC48


Outline of Engineering Issues

• Routing:• OSPF, BGP4, Routing Arbiter Database

• Multicast• PIM-Sparse Mode, MBGP, MSDP

• Measurements• Surveyor: One-way delay and loss• Traffic utilization• End to end flows with gigaPoP help• OC3MON -- passive measurements

• QoS - QBone

GigaPoP Architecture


Acknowledgments

• Steve Corbató - University of Washington & Pacific/Northwest Gigapop

• Ron Hutchins - Georgia Institute of Technology & Southern CrossRoads Gigapop

• Ken Lindahl - University of California, Berkeley• Rick Summerhill - Executive Director, Great Plains

Network• David Wasley - CENIC Director of Projects/CalREN-2

Gigapop


GigaPoP Idea

• From George Strawn of NSF - Predates Internet2

• Idea was one of the catalysts for Internet2• Aggregation point• Economies of scale

• Member sites• Value added services shared by members

• Not limited to university/I2 members• (High-speed) local traffic stays local

• Attached networks


GigaPoPs

• Variety of services and styles• Technical and organizational differences• Mixture of technologies

• Some things must be the same• IP as a common bearer service• Inter-GigaPoP routing policy and design• Measurement• Trouble tickets among network operation

centers


Agency Networks

Customer links

Backbone nodes

Aggregation points/GigaPoPs

Peering connection

Robust Model

With thanks to Ron Hutchins


Peachnet

GSU

GT

SoXGigaPoP

NCSU

UNC

Duke

MCNC

vBNS

GCATT

Vanderbilt

FSU, UF,Miami

GigaPoPs can be Hierarchical

UA

UAH

UAB



Denotes GigaPoP

Internet2 GigaPoPsAs of November 1997


Denotes GigaPoP

Internet2 GigaPoPsAs of 19 May 1999


GigaPoPs - What Flavors Do They Come In

• Two broad sets of choices• Level• Topology

• Level• Layer 1 (optical)• Layer 2 (usually ATM)• Layer 3

• Topology• Star - aka Gigapoint• Distributed• Combination - figure “8”


Star or Gigapoint

• Advantages• No internal transit

issues

• Disadvantages• Longer backhauls

for both members & service providers

Member linksISP/HPNSP links


Distributed

• Advantages• Optimize backhauls• Take advantage

existing connections• Politics

• Disadvantages• BW between nodes

can be bottleneck Member linksIntra-GP linksISP/HPNSP links


Combination

• Advantages• Optimize backhauls• Take advantage

existing connections• Politics• Minimize internal transit

• Disadvantages• BW between hubs

can be bottleneck • Longer backhauls

Member linksIntra-GP linkISP/HPNSP links

Example - Non-ATMLevel 2 GigaPoP


P/NW GigaPoP Design

With thanks to Steve Corbató

Example - Combination GigaPoP


Great Plains Network

• Participating States• N & S Dakota• Nebraska• Kansas• Oklahoma• Missouri• Arkansas

• Topology

With thanks to Rick Summerhill

Example - Distributed GigaPoP


CalREN-2 Topology Plan (5/98)

Fresno

Internet2(vBNS )

Internet2(vBNS )

USC, ISI, UCLA, UCI, Caltech,CSU, JPL, UCR, UCSB

UCB, UCD, UCSF,UCSC, Stanford, UCOP

UCOP, Oakland

UC Berkeley

Monterey

Stanford

UC Davis

UCSD & SDSC

UC Riverside

Corporation for Education Network Initiatives in California

CalREN-2 Network

UC Santa Barbara

UC Irvine

UCLA

ISI

USC

UC San Francisco

UC Santa Cruz

JPL

Merced

Northern CaliforniaGigaPOP

Southern CaliforniaGigaPOP

San Diego GigaPOP (Future)

Sacramento

(Planned but not funded)

OC-12c

OC-48

CSU

UC San Diego

Caltech

Planned N-SInterconnect

With thanks to David Wasley


CalREN-2 TopologyStanford

UC Davis

UC SantaCruz

UC SanFrancisco UC OP

UCBerkeley

ESnet

Abilene

vBNS

UC SantaBarbara

JPL

CaltechUC

RiversideUC Irvine

UC LosAngeles

USC

ISI

UC SDSDSC

CSU

vBNS

ESnet

Abilene

NREN

CalREN2 North

CalREN2 South

OC-48

OC-12c

(future)

(future)

With thanks to Ken Lindahl


CalREN-2 North: Physical Topology

UC Berkeley

UC OPUC SF

Stanford

UCSC UCD

ESNet

Abilene

SS

S S

SSCBR

CBRCBR

CBR

vBNS

RR

OC-12 PoS

OC-12 ATM CBRcampusborder router

CBR

R R

CBR



CalREN-2 North: Logical Topology

UC Berkeley

UC OPUC SF

Stanford

UCSC UCD

ESNet

Abilene

CBR

CBRCBR

CBR

vBNS

RR

OC-12 PoS

OC-12 ATM

OC-3 VC

CBRcampusborder router

CBR

R R

CBR


Example - Star/Gigapoint


RR

RR

RR

RR

RR

RR

RR

vBNS

ISPISP

RR

Georgia GigaPoP

Peering using BGP

Logical Layer 3 Diagram


SoX: Layer 3 Example


vBNS

Router

Router

Router

Router

Router

GigaPOP Switch

Georgia GigaPOP

B

“External Gigapop” APNNI

PNNI

PNNI

PNNI

PNNI

GT Network

Dept. Net

ISP

vBNSLayer 2 diagram


SoX: Layer 2 Example


ISP 3 *

Campus Network

*

*

**

*ISP 1

HPNSP

ISP 2

Campus 1

Campus 2 3 4

SoX Gigapop

vBNSISPATM “Pipe”

ABR/UBR PVCBurst to Full OC-3

VBR PVCPreempt to 10Mb/s

Combined Services



The End

Documents

Internet2: A Tutorial Part 2 of 4 17 th Brazilian Symposium on Computer Networks Paul Love, Internet2 Chair, I2 Topology WG [email protected]