Internet2 Engineering and Abilene Update Westnet Meetings :: Boulder Guy Almes 29 June 2001

Internet2 Engineering and Abilene Update

Westnet Meetings :: BoulderGuy Almes <[email protected]>

29 June 2001

Outline of Talk

Internet2 Engineering Objectives

Abilene History and Status

Engineering Update Multicast, IPv6, QoS

Issues in End-to-End Performance

Internet2 Engineering Objectives

Provide our universities with superlative networking: Performance

Functionality

Understanding

Make superlative networking strategic for university research and education

The End to End Challenge

Support advanced networking end to end Performance

100 Mb/s across the country normative

several multiples possible in some cases

Functionality Multicast

Quality of Service

IPv6

Measurements

Abilene Update

Original Abilene Objectives -- 15-Apr-98

Provide high-quality, widely available Interconnect among participating gigaPoPs/universities

Connect to Internet2 members via the vBNS and to other key research/ education sites via Internet2/NGI-class federal and non-US nets

Abilene Objectives, cont.

[Support] advanced functionality

Maximize Robustness Minimize Latency Provide Capacity to Avoid Congestion

Key Attributes

12 Router Nodes Cisco 12008 Routers

Qwest collocation

OC48 Interior Circuits connect them Packet over Sonet in all cases

Access: 52 total OC3, OC12, and some OC48

via any Qwest Sonet PoPs (Access Nodes)

ATM and POS both supported

Abilene Partners

Qwest: Sonet and Collocation

Nortel: OC192 Sonet ADMs

Cisco: 12008 Routers

Indiana University: NOC

Abilene core

Seattle

Kansas City

Denver

Cleveland

New York

Atlanta

Houston

Sunnyvale

Los Angeles

Indianapolis

Washington

The Qwest plow laying:- two conduits- one with 96 fibers- one empty

Abilene Router Nodesbeing assembled

Summer 1998

Qwest'right of way'PoP

60 Hudson

an early carrier hotel

Abilene Connectionsby (roughly) summer 2001

Current Status

About 50 Connections Growing number of OC3-to-OC12 POS

upgrades e.g., NYSERnet Buffalo

but also NoX, Univ Pennsylvania, Florida

185 Participants in 50 States and DC Puerto Rico soon

Sponsored Education Group Participants – Other Possible

Increasing Routing Complexity

Abilene International Transit Network with CA*net3 and StarTap

Abilene policy on Federal Labs differentiate Federal Labs from BB agencies

Abilene Traffic Matrix

NNNNYYFederal

Network

NNNYYYNon-AITN

Peer

NNYYYYAbilene

ITN PeerSource

NYYNYYFederal Lab/BB

YYYYNYCorporate

Participant

YYYYYYRegular

Participant

Federal

Network

Non-AITN

Peer

Abilene

ITN Peer

Federal

Lab/BB

Corporate

Participant

Regular

Participant

eringPeransitT

inationDest

International Peering

STAR TAPAPAN/TransPAC, CA*net3, IUCC, RENATER, REUNA, SURFnet, SINET, TAnet2 CERnet, (HARnet)

OC12 New YorkDANTE*, JANET, NORDUnet, SURFnet CA*net3

SeattleCA*net3, (AARnet)

Sunnyvale(SINET)

Los AngelesSingAREN, SINET

Miami(REUNA, RNP2, RETINA)

OC3-12El Paso(CUDI)

San DiegoCUDI

Early NoF Planning

Commitment to ongoing needs of the Internet2 infrastructure beyond 2003

Leverage growing DWDM/fiber provisioning with many 10-Gb/s ls

Needs: Leverage Backbone/GigaPoP/Campus structure

Serious attention to international/federal peering

Current advanced services now normative

The Houston Flood

Tropical storm Allison hit Houston hard 26 inches in 24 hours

Abilene was effected by this Houston Router Node went down Saturday morning

No news until mid-day Monday

"Technicians could not access this facility until late this afternoon when it was determined that the Abilene equipment there had been damaged beyond repair."

The Houston Flood, cont.

Then (!), on Tuesday morning, it came up First, just a few circuits

Operational for most Texas/Louisiana connectors

Last circuit came up Thursday

Networking, like baseball, is a "game of inches"

Advanced Functionality

Multicast

IPv6

QoS

Internet2 Multicast

Multicast Working Group Kevin Almeroth, Univ California Santa Barbara, chair

Encouraging more pervasive high-quality deployment of native IP multicast throughout the Internet2 infrastructure

Fighting fires Keeping an eye on SSM Clarifying the application story

Internet2 Multicast Architecture

PIM-SparseMode multicast routing within an Autonomous System

quite scalable

notion of rendezvous points

MBGP between Autonomous Systems

MSDP Source Discovery

Longer-term WG Issues

Scalability (what happens if it does catch on?)

Exploring the role of Source-Specific Multicast

Could SSM be Enough?

'Classic' Multicast Group <g> has global significance

A user creates, joins, sends to g

Others can join, then send to and/or listen to g

MBGP, PIM-SM, MSDP triad

Source Specific Multicast Group <g> has local significance

A user 's' creates, sends to <s,g>

Others can subscribe to, then list to <s,g>

No need for MSDP (or allocation of <g> values)

Implications of SSM

Simplify Multicast Routing / Addressing No need for global class-D address allocation

No need for source discovery

Complicates 'few-to-few' applications Define all the members of the application-level group

Both a burden and an opportunity

Allows better Security, Scalability

Requires new version of IGMP

Multicast Summary

Full functionality supported now Deployment steadily increasing Some international peering, e.g., CA*net3 Performance excellent

Scalability? Applications?

Internet2 IPv6

IPv6 Working Group Dale Finkelson, Univ Nebraska, chair

Build the Internet2 IPv6 infrastructure Educate campus network engineers to

support IPv6 Explore the Motivation for IPv6 within

the Internet2 community

IPv6 Infrastructure

vBNS and Abilene both support IPv6

Abilene IPv6 with IPv6/IPv4 Four 'backbone' nodes: Cisco 7200

Atlanta, Pittsburgh, Denver, and Indianapolis

Managed by the Abilene NOC

IPv6 WG: address allocation and engineering coordination

Education / Training Goals

IPv6 hands-on workshop Lincoln, Nebraska; 17 May 2001

starting from scratch, build an IPv6 network, including routers, hosts, DNS tools and various transition tools, ending up with a functional IPv6 network fully interconnected to the global Internet.

Materials from this workshop will be available to enable gigaPoPs and others to use in their own workshops.

Explore IPv6 Motivation

Why should our users, campus decision-makers, and community generally care about IPv6? we like Steve Deering

IPv6 preserves the classic end-to-end transparency of the Internet architecture

improved support for mobility

key for IPsec

key for the scalability of the Internet

The answers must be pragmatic.

Internet2 QoS

Quality of Service Working Group Ben Teitelbaum, Internet2 staff, chair

QBone Premium Service Scavenger Service Architectural and ad-hoc projects

QBone Premium Service

For a given bit/second rate, minimize: Delay and variation in delay, and

Loss

And support Interoperability of separately designed/managed IP

networks (e.g., Abilene, gigaPoP, ESnet, campus)

Interoperability of different (compliant) equipment

This is hard and very important

Abilene Premium Service

Goal Make APS a reference implementation of QBone

Status CAR policing + uncongested ABES

Seven participating connectors Summer 2001:

Turn on PQ and stochastically detect illegal EF traffic via NetFlow monitoring

Fall 2001: Engine-3-based real policing

Scavenger Service

Suppose there were a less-than-best-efforts IP service within Internet2? users can mark their packets LBE

best-efforts traffic generally routed before LBE traffic

what bottom-feeding applications would emerge?

much easier than Premium Service

Other Abilene items

DDoS Detection Technique Development collaboration of Asta Networks and the NOC

reflects open measurement/management stance

similar relation anticipated with Arbor Networks

High Performance Demos Fall Member Meeting; Austin in October

SC'2001; Denver in November

Issues inEnd-to-End Performance

The Current Situation

Our universities have access to an infrastructure of considerable capacity examples of 240 Mb/s flows

End-to-end performance varies widely but 40 Mb/s flows not always predictable

users don't know what their expectations should be

Note the mismatch

What are our Aspirations?

Candidate Answer #1:Switched 100BaseT + Well-provisioned Internet2 networking ® 80 Mb/s

But user expectations and experiences vary widely


Candidate Answer #2:Lower user expectations and minimize complaining phone calls

There is a certain appeal I suppose...


Candidate Answer #3:Raise expectations, encourage aggressive use, deliver on performance/functionality to key constituencies.

Not the easy way, but necessary for success

Why should we Care?

"We" as the university community. "We" as campus networking specialists. "We" as networking professionals. "We" as the (broad) Internet2 project.

Low aspirations are dangerous to us.

End to End Performance Initiative

Goal: To create a ubiquitous, predictable, and well-

supported environment in which Internet2 campus network users have routinely successful experiences in their development and use of advanced Internet applications, by focusing resources and efforts on improving performance problem detection and resolution throughout campus, regional, and national networking infrastructures.

Threats toEnd to End Performance

BW = C x packet-size / ( delay x sqrt(packet-loss ))(Mathis, Semke, Mahdavi, and Ott, CCR, July 1997)

Context: Network capacity Geographical distance Aggressive application


Fiber problems dirty fiber

dim lighting

'not quite right' connectors


Fiber problems

Switches horsepower

full vs half-duplex

head-of-line blocking


Fiber problems

Switches

Inadvertently stingy provisioning mostly communication

happens also in international settings


Fiber problems

Switches

Inadvertently stingy provisioning

Wrong Routing asymmetric

best use of Internet2

distance


Fiber problems

Switches

Inadvertently stingy provisioning

Wrong Routing

Host issues NIC

OS / TCP stack

CPU

Perverse Result

'Users' think the network is congested or that the Internet2 infrastructure cannot help them

'Planners' think the network is underutilized, no further investment needed, or that users don't need high performance networks

Promising Approaches

Work with key motivated users 'Shining a flashlight' on the problem Measurements Divide-and-Conquer Understanding Application Behavior Getting it right the first time

Internet2 End-to-End Performance Initiative

Very recently hired / deployed staff Cheryl Munn-Fremon, initiative director

Russ Hobby, chief technical architect

George Brett, chief information architect

$1.5M budgeted by Internet2


Distributed measurement infrastructure Enable rapid effective understanding of why an

instance of end-to-end performance is limited

Make the work of PERT members rewarding

Enable initiation of tests by PERT members

Teams of performance analysis specialists (PERTs)

Dissemination of best practices


Distributed measurement infrastructure

Teams of performance analysis specialists (PERTs) members at campuses, gigaPoPs, backbones

socially and technically coordinated

committed to effecting radical change

Dissemination of best practices


Distributed measurement infrastructure

Teams of performance analysis specialists (PERTs)

Dissemination of best practices Identify key techniques, tools, and 'best practices'

Make them common

Work toward widespread / routine excellent user experiences

Improve the reputation / status of network engineers

Anticipated Partners

NLANR: DAST, MOAT, and NCNE

Web100 Project Abilene partners Leading campuses and gigaPoPs Internet2 corporate members

Internet2 Measurements

Measurement Working Group Matt Zekauskas, Internet2 Staff

Define architecture: Usage

Active Measurements of Performance

Passive Measurements

Uniform Access to Results Contributing to Measurement

Infrastructure for the E2EPerf

Applications for Measurements

End-to-end Performance Debugging Verification of QoS Performance

Characteristics Support for Operations Forward engineering of new

infrastructure Supporting research, e.g., by university

computer scientists

Active Measurements within Abilene

Surveyors with:Active delay/loss measurementsAd hoc throughput tests

Application to Performance Debugging

Application to Performance Debugging

Divide and Conquer

Systematically identify/isolate the network segment at fault

Can we make this systematic and (eventually) automated?

Access to Key Resources

Optical telescopes in Hawaii

CRAFT Project

PACI Supercomputer Facilities

CERN

Working Groups as Opportunities

We intend the WGs to be effective as: means for interested engineers to 'sink their teeth

into' hard Internet2 engineering problems

means for disseminating best practices etc to the Internet2 membership

New Engineering Area of Internet2 web site due up by 14-Feb-01

Internet2 and Stephen F Austin

Can we defeat distance as a barrier to: human collaboration?

effective access to key instruments / data sources?

For very large research universities, this is somewhat important, but it is key for smaller ones!

Applications Communities

General notion: distributed sets of researchers who collaborate at a distance High Energy Physics (CERN, MIT, Caltech)

Space Physics & Aeronomy Research Collaboratory

Geospatial Information Systems community

These groups explore why advanced Internet2 infrastructure is important

Documents

Internet2 Engineering and Abilene Update Westnet Meetings :: Boulder Guy Almes 29 June 2001