Network Update

Steve CotterDirector, Network Services

TNC2007, CopenhagenMay 22, 2007

Slide 2

Internet2 Network

• Layer 1: Managed wavelengths from Level(3) Communications • Level(3) owns and manages Infinera optical gear: responsible for software

upgrades, equipment maintenance, remote hands, sparing, NOC services• Internet2 NOC has total provisioning control

• Layer 2: Internet2 owned and managed Ciena CoreDirectors• Using DRAGON GMPLS control plane

• Layer 3: Internet2 owned and managed Juniper T640s • Expanded Observatory

• Platform for layer 1/3 network performance data collection, collocation, experimentation

• perfSONAR integration for intra- & inter-network performance analysis

• International connectivity• Layer 1 network extended to international exchange points in Seattle, Chicago

and New York City• Peering points in Seattle, PAIX, Equinix Chicago

Infrastructure Overview

Slide 3

DenverFront Range GP

1850 PearlLevel 3

Kansas CityGPN1100 WalnutLevel 3

Seattle1000 Denny WayLevel 3

Portland1335 NW NorthropLevel 3

SunnyvaleCENIC

1380 KiferLevel 3

Pacific Northwest GP2001 6th AveWestin Bldg

San Diego

PortlandOregon GP

707 SW WashingtonQwest

San Francisco

Oakland

Sacramento

Los AngelesCENIC

600 W 7thEquinix

Santa Barbara Los Angeles818 W 7th

Level 3

Salt LakeInter-Mountain GP

572 S DeLongLevel 3

AlbuquerqueNew Mexico GP

104 Gold Ave SELevel 3

TulsaOneNet

18 W ArcherLevel 3

HoustonLEARN1201 N I-45 Level 3

Dallas

San AntonioBaton RougeLONI9987 BurbankLevel 3

New Orleans

Chicago600 W Chicago

Level 3 MC

Detroit

ChicagoCIC/MREN

MERITBOREASInternet2

710 N LakeshoreStarlight

Atlanta180 Peachtree St NWLevel 3 MC

Birmingham

CharlotteRaleighNCREN5301 Departure DrLevel 3

PittsburghPittsburgh GP143 S 25thLevel 3 Washington

MAX1755 Old MeadowLevel 3

PhiladelphiaMAGPI401 N BroadLevel 3

New YorkNYSERNET32 Ave of the Americas

New York111 8th

Level 3

CambridgeNOX300 Bent StLevel 3

Tampa

MiamiSouth Florida GP

45 NW 5thLevel 3

Orlando

JacksonvilleFLR4814 Phillips HwyLevel 3

Indianapolis1902 S East StLevel 3

Hartford

Louisville848 S 8th StLevel 3

via 1005 N B St

via 1075 Triangle Ct

El Paso501 W Overland

Level 3

35mioverlap

NashvilleTennessee GP2990 Sidco Dr

Level 3

ClevelandTFN4000 ChesterLevel 3

Albany316 N PearlLevel 3

Ogden

Rieth

Boise

Tionesta

Rawlins Omaha

St. Louis

EurekaReno

Valentine

Sanderson

Raton

Phoenix

San Luis Obispo

Mobile

Tallahassee

SyracuseRochester

Cincinnati

Austin

Tucson

Buffalo

Rancho De La Fe(tentative)

Edison

Atlanta SLR345 Courtland

Optical Regeneration Node

Full Optical Add/ Drop Node

Internet2 Connector

Other Level 3 (Glass Through)

Internet2 Network - Optical

DenverFront Range GP

1850 PearlLevel 3

Kansas CityGPN1100 WalnutLevel 3

Seattle1000 Denny WayLevel 3

Portland1335 NW NorthropLevel 3

SunnyvaleCENIC

1380 KiferLevel 3

Pacific Northwest GP2001 6th AveWestin Bldg

San Diego

PortlandOregon GP

707 SW WashingtonQwest

San Francisco

Oakland

Sacramento

Los AngelesCENIC

600 W 7thEquinix

Santa Barbara Los Angeles818 W 7th

Level 3

Salt LakeInter-Mountain GP

572 S DeLongLevel 3

AlbuquerqueNew Mexico GP

104 Gold Ave SELevel 3

TulsaOneNet

18 W ArcherLevel 3

HoustonLEARN1201 N I-45 Level 3

Dallas

San AntonioBaton RougeLONI9987 BurbankLevel 3

New Orleans

Chicago600 W Chicago

Level 3 MC

Detroit

ChicagoCIC/MREN

MERITBOREASInternet2

710 N LakeshoreStarlight

Atlanta180 Peachtree St NWLevel 3 MC

Birmingham

CharlotteRaleighNCREN5301 Departure DrLevel 3

PittsburghPittsburgh GP143 S 25thLevel 3 Washington

MAX1755 Old MeadowLevel 3

PhiladelphiaMAGPI401 N BroadLevel 3

New YorkNYSERNET32 Ave of the Americas

New York111 8th

Level 3

CambridgeNOX300 Bent StLevel 3

Tampa

MiamiSouth Florida GP

45 NW 5thLevel 3

Orlando

JacksonvilleFLR4814 Phillips HwyLevel 3

Indianapolis1902 S East StLevel 3

Hartford

Louisville848 S 8th StLevel 3

via 1005 N B St

via 1075 Triangle Ct

El Paso501 W Overland

Level 3

35mioverlap

NashvilleTennessee GP2990 Sidco Dr

Level 3

ClevelandTFN4000 ChesterLevel 3

Albany316 N PearlLevel 3

Ogden

Rieth

Boise

Tionesta

Rawlins Omaha

St. Louis

EurekaReno

Valentine

Sanderson

Raton

Phoenix

San Luis Obispo

Mobile

Tallahassee

SyracuseRochester

Cincinnati

Austin

Tucson

Buffalo

Rancho De La Fe(tentative)

Edison

Atlanta SLR345 Courtland

Optical Regeneration Node

Full Optical Add/ Drop Node

Internet2 Connector

Other Level 3 (Glass Through)

Internet2 Network - Optical

Phase 1Phase 1CompletComplet

eePhase 2Phase 2March 2March 2

CompleteCompletePhase 4BPhase 4BJune 12June 12

Phase 3APhase 3AApril 19April 19

CompleteComplete

Phase 3BPhase 3BApril 27April 27

CompleteComplete

Phase 4APhase 4AMay 18May 18

CompleteComplete

Page 4

Page 5

Page 6

Slide 7

Internet2 Network

• Dedicated point-to-point wavelength services• Dynamic Circuit Service

• Point-to-point Ethernet (VLAN) Framed SONET Circuit• Point-to-point SONET Circuit• Bandwidth provisioning available in 50 Mbps increments

• Best-effort high-speed IP Service• Enables delivery of advanced content, commodity services,

etc.• Physical Connection

• 1 or 10 Gigabit Ethernet• OC-192 SONET

Flexible Infrastructure Supporting e-Science, Network Research & Education

Slide 8

Internet2 Network

• Connection Oriented Services provide for:• Guaranteed bandwidth (predictable, repeatable, dependable

performance between collaborating sites)• Traffic segregation (support specific policy or traffic engineering

requirements)• Router bypass: Express links created for high-bandwidth, limited

duration long-haul traffic reducing the need for mid-path L3 interfaces• Cost efficiency: L3 router blades cost > L2 ports > L1 or L0 interfaces• Capability tradeoff but could possibly improve performance

• Automated dynamic circuit services enable rapid provisioning and efficient utilization of capital investment• Establishing end-to-end lightpaths is a non-trivial task: it is resource

intensive and error prone • Automated reservation, allocation, and provisioning enables co-

scheduling of network and non-network resources.

Dynamic Circuit Services

Slide 9

Multi-Service/Domain/Layer/Vendor Provisioning

Regional Network Regional

Network

Internet2 Network

ESNet

Dynamic Ethernet Dynamic EthernetTDM

GEANT

IP Network (MPLS, L2VPN)

Ethernet

Router

SONET Switch

Ctrl Element

Domain Controller

LSP

Data PlaneControl Plane Adjacency

• Multi-Domain Provisioning• Interdomain ENNI (Web Service and OIF/GMPLS)• Multi-domain, multi-stage path computation process• AAA• Scheduling TDM

Slide from Tom Lehman, ISI-East

GUI

XML

AST

Slide 10

Internet2 Network

• Create private networks between key collaborating sites within a science community or on a project-by-project basis. Examples:• Radio Astronomy community sharing telescopes and

correlation facilities• High Energy Physics community distributing large data sets• Smaller projects may link data repositories and computational

resources among key participants – e.g. grid portals• Network engineering and operations community can use

these capabilities to support traffic engineering objectives• We expect applications will use these circuit services

to carry high-bandwidth, limited duration flows and will retain use of IP links for generic Internet access.

Dynamic Circuit Services Applications

Slide 11

Example: E-VLBI Application Specific Network

Correlator/Compute ClusterMark 5

Mark 5 VLSR

VLSR

Analysis station

VLSR

VLSR Global R&E Hybrid Infrastructure

Globally Distributed, Unified Storage Clusters

Sensors

Slide from Jerry Sobieski, MAX

Slide 12

Collaborations

• The DICE group - Dante (GEANT2), Internet2, CANARIE, and ESnet• Working closely with ESnet on interfacing OSCARS and

HOPI - involves AAA work, using OSCARS interface• “Stitching” project to describe data layer interconnections

between segments of a PTP path• Reporting back progress to the GLIF and other

organizations• For example, Phosphorus, in coordination with the SURFnet

and University of Amsterdam participants• Also having discussions with JGN2

• Coordinating with OGF on various schema - topology, path computation, signaling

• Working with the appropriate standards bodies - ITU, IETF, and OIF

Slide 13

For more information:

http://www.internet2.edu/network/

http://i2net.blogspot.com scotter@internet2.edu

Thank you!

Slide 14

• Layer 2: Cisco 6513 Ethernet switch for IPv4/v6 peering with 1GigE & 10 GigE interfaces

• Layer 1: TDM based optical equipment (SONET / Ethernet interfaces)• Cisco 15454• Nortel OME 6500• Nortel HDX

• MAN LAN website: http://networks.internet2.edu/manlan/

MAN LAN Exchange Point32 Avenue of the Americas, NYC

Slide 15

Trans-Atlantic ConnectivityConnection to Link Operator (Funder) BW (Gbps) N.A. Interconnect

CERN LHCnet (CalTech) (DOE) 2x10, 1x10 MAN LAN, StarLight

CESnet (CZ) CESnet (CESnet) 10 StarLight

GEANT2 (Europe) DANTE (GEANT2) 10, 10 NGIX-East, MAN LAN

GEANT2 (Europe) TransLight/StarLight (NSF) 10 MAN LAN

GEANT2 (Europe) Internet2/HOPI (Internet2) 10 MAN LAN

SURFNET (NL) SURFnet 10, 10 StarLight, MAN LAN

SURFNET (NL) Internet2&SURFnet (IEEAF) 622Mbps Abilene-NYC

NetherLight TransLight-StarLight (NSF) 10 StarLight

NetherLight Internet2&SURFnet (IEEAF) 10 MAN LAN

RBnet (Russia) Gloriad (NSF/Russia) 3x1 (on TL/SL link to NetherLight)

StarLight

ENERGI (Egypt) ENERGI (MCIT) 155Mbps MAN LAN

UKLight (UK) UKERNA (JISC) 10 StarLight

Qatar Foundation Network

Qatar Foundation (Qatar Foundation)

155Mbps MAN LAN

TENET (S. Africa) Telkom SA (TENET) GRE tunnel MAN LAN

Slide 16

Trans-Pacific ConnectivityConnection to Link Operator (Funder) BW (Gbps) N.A. Interconnect

APAN TokyoXP TransPAC2 (NSF/NICT) 10 Pacific Wave-LA

JGN2 (Japan) NICT (NICT) 10 StarLight

AARNET (Australia) AARNET (AARNET) 2x10 Hawaii/Pacific Wave

CERNET (China) CERNET (CERNET) 155Mbps StarLight

CSTNET (China) GLORIAD (CSTNET) 2.5 PacificWave-Seattle

KREONET (Korea) GLORIAD (KREONET2) 10 PacificWave-Seattle

Hong Kong HARNET 45Mbps StarLight

SINET (Japan) NII (NII) 10, 2.5 MAN LAN, PW-LA

GEMNET (NTT-Japan) NTT (NTT) 622Mbps Pacific Wave

SingAREN (Singapore) SingAREN (SingAREN) 155Mbps Pacific Wave-LA

NUS (Singapore) NUS (NUS) 155Mbps Pacific Wave - Seattle

TWAREN (Taiwan) NCHC (NCHC) 2, 2.5, 1.2 Pacific Wave, MAN LAN

ASNET (Taiwan) ASNET (Academica Sinica) 2.5 StarLight

UNINET (Thailand) UNINET (UNINET) 155Mbps LA

WIDE WIDE&PNWGP (IEEAF) 10, 622Mbps Pacific Wave - Seattle

Qatar FN Qatar FN (Qatar Foundation) 155Mbps Pacific Wave - LA

Slide 17

Americas ConnectivityConnection to Link Operator (Funder) BW (Gbps) US Interconnect

CA*net (Canada) CANARIE (CANARIE) ~50 StarLight/PWave/MAN LAN

redCLARA (Latin America), ANSP (Brazil), RNP (Brazil)

WHREN-LILA (NSF, FAPESP)

1.2 AMPATH

redCUDI (Mexico) WHREN-LILA (NSF) 1 CALREN2

redCUDI (Mexico) UTEP, CUDI (UTEP, CUDI)

1 UTEP

REACCIUN-2 (Venezuela) AMPATH (AMPATH) 45Mbps AMPATH

Gemini/NOAO, Chile (NSF funding) 55Mbps AMPATH

Slide 18

RON Dynamic Infrastructure Ethernet VLAN

RON Dynamic Infrastructure Ethernet VLAN

Internet2 DCS Ethernet Mapped SONET

Domain Controller

Domain Controller

Domain Controller

GUI

XML

AST

1. Service Request2. Path Computation Request3. Recursive Per Domain Path Computation/Scheduling Processing4. Path Computation/Scheduling Response (loose hop route object returned)5. Service Instantiation (Signaling)

A. Abstracted topology exchange

AA

AA

13

3

2 NARB

VLSR

4

AAA AAAAAANeed more work on AAA, Scheduling

5

Flexible Edge Mappings

(port(s), tag, untag)

Provisioning Flow