The HOPI Project

Rick SummerhillAssociate Director, Backbone Network Infrastructure, Internet2

Internet2 Spring Member MeetingArlington, VA20 April, 2004

4/21/2004 2

Outline

Resources• Abilene• NLR• Experimental MAN LAN Facility• RONs

The HOPI Project – Hybrid Optical and Packet Infrastructure

• Architectures based on availability of optical infrastructure

4/21/2004 3

Abilene

4/21/2004 4

Abilene Particulars

Performance• 6.2 Gbps single flows across Abilene • Consistent 9.5 gbps traffic patterns during SC2003 from Phoenix

• The performance is good, but we need to look to the future

Agreement with Qwest ends in 2.5 years

• How should we go forward?

4/21/2004 5

NLR Summary

Largest higher-ed owned/managed optical networking & research facility in the world• Over 10,000 route-miles of dark fiber• Four 10-Gbps λ’s provisioned at outset

– One allocated to Internet2

Primarily an experimental platform for research, but may be used for other purposes

4/21/2004 6

NLR footprint and physical layer topology – Phase 1

ATL

POR

RAL

CHICLE

KAN

OGD

WDCDEN

LAX

SVL

SEA

SAN

PIT

JAC

BOI

15808 Terminal15808 OADM15808 RegenFiber routeLeased waves

Note: California (SAN-LAX-SVL) routes shown are part of CalREN; NLR is adding waves to CalREN systems. Also the CENIC SVL-Sacramento (SAC) ELH route will become part of NLR SVL-SEA in exchange for a SVL-SAC LH route NLR is building (not shown here).

11/03

2/043/04

1/04

6/04

7/04

8/04

8/04

4/04

4/21/2004 7

MAN LAN

Ethernet Switch• Layer2 Interconnectivity – Classic exchange point• VLANs between connectors

ONS Cisco 15454• TYCO/IEEAF Circuit moved to experimental facility

–Circuit was router to router, now is ONS to ONS–Ability to map circuits to Abilene or for other

experimental reasons

• OC-192s: CANARIE, Surfnet, Abilene• Experiments with the international community

4/21/2004 8

Leading & Emerging Regional Optical Initiatives

California (CALREN)Colorado (FRGP/BRAN)Connecticut (Connecticut Education Network)Florida (Florida LambdaRail)Georgia (Southern Light Rail)Indiana (I-LIGHT)Illinois (I-WIRE)Maryland, D.C. & Northern Virginia (MAX)MichiganMinnesota

New York + New England region (NEREN)North Carolina (NC LambdaRail)Ohio (Third Frontier Network)OregonPacific Northwest (Lariat – support by NIH) Rhode Island (OSHEAN)SURA Crossroads (southeastern U.S.)Texas (LEARN)UtahVirgina (MATP)Wisconsin

4/21/2004 9

Gauging community-wide progress with dark fiber

Aggregate dark fiber held for and assigned by U.S. R&E optical initiatives (segment-miles):

• CENIC (for NLR and CalREN via L3, WilTel) 6,200• FiberCo (for NLR and RONs via Level 3) 4,900• SURA (via AT&T) 6,000

– (plus 2,000 route-miles for research usage)• Oak Ridge National Lab. (via Qwest) 900• Ohio 1,600• Other state projects (IN,IL,MI,OR, etc.) 1,500+

Total (conservative estimate) 21,000+Pending procurements (TX, NY, NE) 1,700+

4/21/2004 10

Architectural Issues

Some specific disciplines have enormous bandwidth requirements

• High Energy Physics and the Large Hadron Collider• The Square Kilometer Area (SKA) Community• Medical Imaging, Real Time and File Transfer

Questions concerning packet infrastructures• The shared packet infrastructure itself – the ability to support

multiple large flows on the order of 6 gbps.• Unlikely to have 40 gbps or 100 gbps in near future• Increasing demands by some for deterministic paths• Ability to have resources that don’t exhibit congestion• Demand for more dynamic control of bandwidth and topology

Availability of dark fiber at the national, regional and campus levels has changed the landscapeWhere are we going?

4/21/2004 11

HOPI Project - SummaryIn the near future we will see a richer set of

capabilities available to network designers and end users

• Core IP packet switched networks• A set of optically switched waves available for dynamic

provisioning

Fundamental Question: How will the core Internet architecture evolve?Examine a hybrid of shared IP packet switching and

dynamically provisioned optical lambdasHOPI Project – Hybrid Optical and Packet

Infrastructure• Immediate Goals

– Create a white paper describing a testbed to model the above infrastructure

– Implement testbed over the next year– Coordinate and experiment with other similar projects

• Design Team

4/21/2004 12

HOPI Project Design Team

Linda Winkler, Argonne (CoChair)Rick Summerhill, Internet2 (CoChair)Cees de Laat, U of AmsterdamRene Hatem, CANARIEMark Johnson, MCNCTom Lehman, USC/ISIPeter O’Neil, NCARBill Owens, NYSERnetPhilip Papadopoulos, UCSDSylvain Ravot, Caltech/CERNDavid Richardson, U Washington Chris Robb, Indiana UJerry Sobieski, U MarylandSteven Wallace, Indiana UBill Wing, Oak RidgeInternet2 Staff – Guy Almes, Heather Boyles, Steve Corbato, Chris Heermann, Christian Todorov, Matt Zekauskas

4/21/2004 13

HOPI Resources

The Abilene Network – MPLS tunnelsThe Internet2 Wave on the NLR footprintMAN LAN Experimental Facility• TYCO/IEEAF 10 Gbps lambda NYC - Amsterdam

Collaborations with Regional Optical Networks (RONs) and other related efforts (GLIF, DRAGON, etc.)

4/21/2004 14

Abilene/NLR Map

4/21/2004 15

HOPI Project

Problems to understand• Goal is to look at architecture• Temporal degree of dynamic provisioning• Temporal duration of dynamic paths and requirement for scheduling

• Topological extent of deterministic provisioning• Examine backbone, RON, campus hierarchy –how will a RON interface with the core network?

• Understand connectivity to other infrastructures –for example, international or federal networks?

• Network operations, management and measurement across administrative domains?

4/21/2004 16

HOPI Basic Service

Given the available resources, we cannot use multiple waves to study new architectures – have only a limited number of wavesInstead we’ll model waves using lower

bandwidth “deterministic” paths – paths that resemble circuits – “lightpaths”Basic service – A 1 or 10 GigE unidirectional

point-to-point path with reasonable jitter, latency, and loss characteristicsAccess – Direct to HOPI node or an MPLS

L2VPN tunnel through Abilene

4/21/2004 17

HOPI Node

A fiber cross-connect switch• Ability to switch the entire NLR wave to Abilene, to a RON,

or to pass through the wave

An Ethernet switch or TDM device to partition the wave into 1 GigE paths when necessaryControl devices

• Ad hoc control plane computer• Measurement computer• Experimental computer

Out of band access

4/21/2004 18

HOPI Node

4/21/2004 19

Connector Interface

A 1 or 10 GigE connection to the FXC, either dark fiber or a provisioned service, including NLRA 1 or 10 GigE connection to the Switch or TDM device, either dark fiber or a provisioned service, including NLRAn MPLS L2VPN service through Abilene to the Ethernet switch or TDM device

• Provides immediate connection to the Internet2 NLR wave from Abilene

4/21/2004 20

Control Plane

Basic Ideas• Routing – selection of paths with certain characteristics

• Signaling – the process by which the network allocates and maps the switching elements

• User interface – how does the user access a path? Web interface? Automated?

• Policy – who is allowed to request paths and how are they authenticated and how is accounting accomplished?

4/21/2004 21

Control Plane

Phase 1 – Manual Configuration• Ad hoc control of devices that don’t support control plane

protocols• Understand control plane ideas

Phase 2 – Intra-domain Configuration• Automate software control of setup• Investigate temporal and topological extent of paths

Phase 3 – Inter-domain Configuration• May require extensions to standards based protocols

4/21/2004 22

Experiments

Planned Experiments – 15 to 20 initial experiments

• Dynamic Provisioning• Deterministic Paths• Applications Based• Miscellaneous

Encourage use by the community for experimentation – both operational and research communitiesCan start in near future by using MPLS tunnels from Abilene

4/21/2004 23

Lightpath

Lightpath from LA to CERN• Internet2, CANARIE, GEANT, Starlight, Surfnet

Crossed multiple administrative domains at a variety of layers

• Abilene/Internet2 (MPLS layer2 VPN)• Starlight (Ethernet switch, layer2)• CANARIE (TDM, layer1)• MAN LAN / Internet2 (TDM, layer1)• Surfnet (TDM, layer1)• GEANT (PIP, layer3)

4/21/2004 24

4/21/2004 25

Control PlaneWas bandwidth available, how much?

• 1 Gbps

What path to take?• Described above• Note that we knew bandwidth was available along the path! If not,

then scheduling would have been needed

What framing to use?• 1 GigE and IP

Were interfaces available?• Some needed to be installed

Configuration of TYCO/IEEAF circuit needed to be changed

• Was router to router• Changed to TDM to TDM

4/21/2004 26

Control Plane

What layer2 tagging was involved?• Range requirements for the MPLS L2VPN across Abilene• Configuration of Starlight switch• Host in LA set VLAN tagging for outgoing traffic

What layer3 addressing was needed?• Whose address space?

How was routing to be done across the path?How was it all put together?

• Manually• Roughly 8 hours of conference calls• Roughly 500 email messages

Will provide report in the near future

4/21/2004 27

References

Request for Comment – We would like your feedback on the HOPI testbed

• hopi@internet2.edu

More Information• http://abilene.internet2.edu• http://www.nationallambdarail.org• http://hopi.internet2.edu• abilene@internet2.edu

4/21/2004 28