Internet2 Network Design

Rick Summerhill, DirectorNetwork Research, Architecture, and Technologies

Internet2

Internet2 Staff Retreat22 June 2006

2 Rick Summerhill

Contents• Summary• Strategic Objectives• Background• Requirements• Architecture• Services• Transition• Network Usage Policy• Engineering• Network Research• Advisory Groups• Partnerships• Agreement Information• Timeline

3 Rick Summerhill

Abilene Backbone

4 Rick Summerhill

Summary

• Hybrid IP and Dedicated Wave System utilizing Level3 optical fiber platform; equipment and fiber dedicated to Internet2, sparing and equipment maintenance by Level3, including carrier class service for the wave system

• Community control at layer 1• Platform support for both highly experimental projects

and production services; will enable corporate and network research partnerships

• Initially provisioned with ten 10-Gbps wavelengths, with capacity up to 80 wavelengths; scalability potential for 40 and 100 Gbps interfaces

Summary

5 Rick Summerhill

Summary

• Flexible add-drop wave capabilities for RON’s and other networks

• Simplified and rapid wave provisioning, within hours for new waves

• Grooming capabilities through a highly flexible layer 1 system that enables sub channel provisioning at the lowest level SONET granularity

• Dynamic provisioning of sub channels and waves across the network within seconds

Summary

6 Rick Summerhill

Strategic Objectives

• Ensure community control of the underlying network infrastructure• Control those elements that allow development of new network

capabilities

• Leverage the capabilities of a global telecommunications leader• Providing carrier class reliability and expanded breadth of services,

along with a broad set of partnership options

• Capitalize on the latest technological advancements in networking

• Create an asset that benefits the entire community - researchers, universities, regional optical networks, industry, government, K-12, and the international community

Strategic Objectives

7 Rick Summerhill

Requirements • Throughout the process, the requirements of the

community were considered - the process did not occur in a vacuum

• Requirements documents produced during the spring of 2005 were considered• The Group A report, produced by a joint engineering

committee from NLR and describing a wide ranging set of network attributes

• The Abilene Technical Advisory Committee (TAC) Report outlining a need to provide for new experimental services along with production services like IP

• The MORPHnet document (1997) - Multi-Modal Organizational Research and Production Heterogeneous Network

Requirements

8 Rick Summerhill

Requirements, continued • Input from the Quilt Optical workshops and discussions

with the RONs• Research examining a wide variety of different types of

capabilities• DRAGON and CHEETAH - NSF funded projects examining

dynamic provisioning• Science related projects such as UltraLight and eVLBI• OptIPuter and TeraGrid• HOPI Testbed

• Input from the International community• GEANT2 providing hybrid types of capabilities• GLIF and lightpath capabilities for scientific work

Requirements

9 Rick Summerhill

Architecture Design Goals

• Develop an innovative optical system on a national footprint to serve the broad research and education community• The goal is to support all Internet2 members and to enhance

the ability to serve a wider community• Develop a hybrid network capable of providing point-

to-point services together with an IP network• Every connector (RON) connects a NewNet backbone

ring across a metro location not requiring extensive backhaul

• The community retains complete control of the layer 1 optical system including provisioning and switching of wavelengths

Architecture

10 Rick Summerhill

Architecture Design Goals, continued

• Internet2 does not have to concentrate on reliability and sparing - Level 3 is responsible for an SLA.• The community focus is on networking, not on

managing devices like amplifiers

• The system is capable of supporting network research in wide variety ways

• Minimal Conditions of Use (CoU), allowing full participation from the entire community in providing new services and capabilities

Architecture

11 Rick Summerhill

The System

• System uses dedicated fiber from the carrier• Built on Infinera platform providing innovative

optical technology• Simple and convenient add/drop technology• Simple and convenient wave setup, requiring only

additional endpoint interfaces• Demonstrated high reliability in initial period of

operation on the Level3 network• Economics of Infinera system are disruptive in the

market place

Architecture

12 Rick Summerhill

The System, continued

• Architecture has maximum flexibility. Every RON can access every wave on the system if needed

• System includes grooming capabilities - lightpaths can be built over Ethernet or SONET• Can take advantage of advanced SONET

cpabilities like GFP, VCAT, and LCAS• Capable of lightpath provisioning to the campus• Lighpath capabilities illustrated in the following

diagram

Architecture

13 Rick SummerhillRick Summerhill

Lightpath Capabilities

Architecture

14 Rick Summerhill

System, continued

• RON connectivity expected to evolve to 2 x 10 Gbps connections• 10 Gbps IP connection• 10 Gbps point-to-point connection (capable of STS-

1 granularity lightpaths provisioned in seconds), most likely provisioned using Ethernet (GFP based)• Hybrid capabilities

• Expect 20 - 24 connectors• Simple and consistent connection scheme• Promotes aggregation

Architecture

15 Rick Summerhill

System, continued

• The following diagrams represent preliminary thoughts on topology• Exact locations may vary, but fiber will be

on the Level3 footprint• Convenient drop/add capabilities of the

Infinera system provides flexibility for locations of the optical nodes

Architecture

16 Rick SummerhillRick Summerhill

Layer 1 Topology

PROVISIONAL TOPOLOGY – SUBJECT TO DISCUSSION

Architecture

17 Rick Summerhill

IP Network

• IP network built on top of optical system• High reliability - architecture provides a variety of

protection options• Commodity service offering - commodity services

will likely be a service offering• Current plan is to continue to use highly reliable

Juniper routers, but open to new technologies• Will likely use fewer routers, emphasizing point-to-

point capabilities and hybrid networking• Potential near term option of 40 Gbps

Architecture

18 Rick SummerhillRick Summerhill

Layer 1 Topology with IP Network

PROVISIONAL TOPOLOGY – SUBJECT TO DISCUSSION

Architecture

19 Rick Summerhill

Architecture Summary

• Initially provisioned with ten 10-Gbps wavelengths (100 Gbps)

• Capacity: up to 80 wavelengths (0.8 Tbps)• Scalability: Potential for 40 & 100 Gbps

wavelength support• Reliability: Carrier provides standard service

assurances for these wavelengths• Provide a variety of different types of reliability

• Flexibility: Support for dynamic provisioning and wavelength switching

Architecture

20 Rick Summerhill

Node Architecture•Advanced optical DWDM equipment•Grooming capabilities in ADM to provide sub channels•Support for other networks having a national footprint•Simplified and standardized interface to connectors, exchange points, and other global research and education networks

•2 x 10 Gbps interfaces•Example displayed is the Denver optical node, showing connections required by the collocated IP router node and an additional network

Architecture

21 Rick Summerhill

Circuit Services

• Short Term Dynamically Configured STS-1 Granularity deterministic circuits using two waves on the full footprint, with framing either SONET or GFP mapped Ethernet. There is no additional cost to use these capabilities. They are included in the base connectivity Costs. There may be call blocking in the use of these capabilities.

• Long Term Circuits configured as above. There is an extra fee for these circuits, proportional to bandwidth, and charged on a day-by-day basis. The shortest term is one day, and there is no call blocking for these capabilities.

• Long Term full waves configured and provisioned for a minimum of a year.

Architecture

22 Rick Summerhill

Projects on NewNet

• Dynamic services• Creating lightpaths across multiple administrative

domains in seconds• Development of “Domain Controllers”• Support for applications - setup of “Application

Specific Topologies” on the network• Dynamic inclusion of optical paths into IP

networks for backup or expansion• TCP data flows• File transfers over long distance segments not

requiring congestion control

Architecture

23 Rick Summerhill

Other Projects

• Network research projects deploying programmable Ethernet capable chipsets at each optical node using “NetFPGA” cards from Stanford• Programmable chipsets for experimenting with new protocols

• Service trial with GEANT2 on dynamic provisioning of 1 GigE circuits across Internet2 and GEANT2• Develop inter-domain 1 GigE services between hosts or

clusters of hosts in Europe and the US - technology, policies, and cost models

• Participation by RONs and campuses in the trial• The following diagram illustrates the ideas

Architecture

24 Rick Summerhill

Internet2/GEANT2 Service Trial

25 Rick Summerhill

Network Usage Policy

• The use of the network facility shall be consistent with the broad set of non-profit objectives of the U.S. research and education community and its affiliates

• Internet2 will seek to maximize the ability of the network to support the broad collaboration requirements of the Internet2 membership and their affiliates to interact with the broadest scope of domestic and international networks and collaborating organizations

• Each organization utilizing the facility will be responsible for complying with all technical and operational standards developed for the network, and will avoid actions that adversely impact the performance or stability of the facility itself or those experienced by other users

Network Usage Policy

26 Rick Summerhill

Network Usage Policy, continued

• The facility shall not be used for illegal, classified, life-safety, or unrelated commercial activities

• Additionally, Internet2 will seek to maximize to its members the value of IP transit by expanding connectivity through peering with commercial networks including content providers (e.g., Google, Akamai) and other regional and national service providers (i.e., tier 2 ISPs)

Network Usage Policy

27 Rick Summerhill

Engineering

• Building on the experience from projects such as Abilene, HOPI, and NLR, operational support is expected to fall into the three broad categories of control plane development, application support, and advanced engineering.

• The HOPI testbed support center, through a solicitation in 2005, has been supported by the Mid-Atlantic Crossroads (MAX), the North Carolina Research and Education Community (NCREN), and the Global NOC at Indiana University.

• These relationships are expected to provide engineering support for the new Internet2 network.

• The engineering support is roughly defined by the following:

Engineering

28 Rick Summerhill

Engineering Capabilities

• Control plane development, deployment, and support: The Mid-Atlantic crossroads through the DRAGON project has focused on dynamic resource allocation.

• Application support: Enabled by targeting key applications for the research community, applications such as eVLBI, the North Carolina Research and Education Network has focused on this area for the past year

• Advanced engineering, monitoring, and management has been provided by the Global Network Operations Center at Indiana University

Engineering

29 Rick Summerhill

Network Research

• The Internet2 network is an ideal platform for network research - the ability to support both highly experimental projects along with production based services is a key objective

• The Abilene observatory will be expanded to include the new capabilities of the network• Data collection at all layers of the network, with

datasets made available to network researchers • Support for collocation of equipment in optical

nodes

Network Research

30 Rick Summerhill

Network Research, continued

• Internet2 has supported individual network research projects whenever possible. Examples include• The router clamp• MPLS tunnels for researchers - these can now be supported

at layer 1 on the new Internet2 network• The 100x100 project

• Flexibility is key feature of the new network• Smaller projects at lower bandwidths can be supported on

variable footprints• Projects can also be supported for variable lengths of time

• Projects like VINI and GENI can be supported by the new Internet2 network

Network Research

31 Rick Summerhill

Additional Agreement Information

• Continuation of FiberCo

• Introduction of WaveCo• Essentially, a “circuit” version of FiberCo• Waves or lower bandwidth circuits can be

provisioned across Level3 and other networks, providing an additional network service offering

• More information will be presented later

Agreement Summary

32 Rick Summerhill

Timeline

• Community Design Workshop• 15, 16 June in Indianapolis

• ESCC/Internet2 Joint Techs Meeting• BoF and further discussions

• Workshops as the network is rolled out to enhance capabilities of RONS and campuses

• Roll-out• Over the next 15 months, between now and

Summer 2007

Timeline

33 Rick Summerhill

Community Design Workshop Goals

• Provide greater detail about the new Internet2 network

• Obtain feedback from the community about the network

• Excellent engagement from the community about the new network!

Community Design Workshop

34 Rick Summerhill

Main Issues from Workshop

• Circuit availability and provisioning

• Redundancy Model

• Cost Recovery

• Commodity Internet Services

• Network Usage Policy

• Internet2 Focus

• Advisory Groups

Community Design Workshop

35 Rick Summerhill