OmniPoP: GigaPoP-of-GigaPoPs Design and Future Patrick Christian, OmniPoP TAC (chair) University of Wisconsin - Madison April 25, 2007

OmniPoP:OmniPoP:GigaPoP-of-GigaPoPsGigaPoP-of-GigaPoPs

Design and FutureDesign and Future

Patrick Christian, OmniPoP TAC (chair)Patrick Christian, OmniPoP TAC (chair)University of Wisconsin - MadisonUniversity of Wisconsin - Madison

April 25, 2007April 25, 2007

OutlineOutline

Formation driversFormation drivers

Administrative organizationAdministrative organization

Design goalsDesign goals

ArchitectureArchitecture

QuestionsQuestions

Formation DriversFormation Drivers



Formation DriversFormation DriversOmni-Omni-– A prefix meaning “all”A prefix meaning “all”

PoPPoP– Three letter acronym Three letter acronym

popular in network popular in network engineering meaning engineering meaning “Point of Presence”“Point of Presence”

– A place where an A place where an organization’s (or organization’s (or several) networking several) networking equipment is presentequipment is present

OmniPoPOmniPoP– A place where all A place where all

types of networking types of networking equipment is presentequipment is present

– A place where all A place where all members are welcome members are welcome to place equipmentto place equipment

– A place to develop a A place to develop a rich mesh of all types rich mesh of all types of services between of services between membersmembers

– A facility to provide A facility to provide common and small common and small group servicesgroup services


RegionalOptical

Networks

NetworkOperations

Center

24/7/365Production

Support

CommodityInternetService

DirectConnection

Between CICMembers

Colocation Space

For NetworkEquipment

NationalLambda

Rail

Internet2Connection

For CICMembers

CIC FiberIn

Chicago

FormationDrivers


OmniPoP

Colocation Space Network Hardware NOCTechnical Supportand Engineering

Services

Governance

Multiple fiber-rich, friendly locations

Optical switchesEthernet switchesRouters, and more

7/24/365Hands and Eyes

Production operations

Administrative OrganizationAdministrative Organization



Administrative OrganizationAdministrative OrganizationCIC OmniPoP participants (4/07 connection status)CIC OmniPoP participants (4/07 connection status)– University of Chicago (soon)University of Chicago (soon)– University of Illinois (UIUC & UIC) (soon)University of Illinois (UIUC & UIC) (soon)– Indiana University (active)Indiana University (active)– University of Iowa (active)University of Iowa (active)– University of Michigan (active)University of Michigan (active)– Michigan State University (active)Michigan State University (active)– University of Minnesota (active)University of Minnesota (active)– Northwestern University (soon)Northwestern University (soon)– Ohio State University (soon)Ohio State University (soon)– Purdue University (active)Purdue University (active)– University of Wisconsin – Madison (active)University of Wisconsin – Madison (active)

Administrative OrganizationAdministrative Organization

CIC CIOs(MSU CIO Dave Gift TAC liaison)

Technical Advisory Council (2 reps per member)

U-WI: Patrick Christian (chair)

Technical Tactical Team(5 reps, subset of TAC)

Karen PartlowCIC Assistant DirectorTech Collaborations

All Techs(unlimited, interested parties)

Subcommittees

Administrative GovernanceAdministrative GovernanceGoverning BoardGoverning Board– One representative from each Partner (initially One representative from each Partner (initially

the participating CIC schools)the participating CIC schools)– Responsible forResponsible for

PoliciesPolicies

Finances, including major expendituresFinances, including major expenditures

Approving new Partners (not limited to CIC schools)Approving new Partners (not limited to CIC schools)

Selecting the OmniPoP operatorSelecting the OmniPoP operator

Requires two-thirds vote to approve all actionsRequires two-thirds vote to approve all actions

– Meets at least annuallyMeets at least annually

Administrative GovernanceAdministrative GovernanceTechnical Advisory Council (TAC)Technical Advisory Council (TAC)– Up to two representatives from each Partner plus Up to two representatives from each Partner plus

representation from the OmniPoP operator & CICrepresentation from the OmniPoP operator & CIC– Responsible forResponsible for

Preparing an annual strategic technical planPreparing an annual strategic technical plan

Recommending an annual budgetRecommending an annual budget

Recommending changes to services offeredRecommending changes to services offered

Approving routine purchases and service contractsApproving routine purchases and service contracts

Tactical Technical TeamTactical Technical TeamNot more than six membersNot more than six members

To work with the OmniPoP operator and CIC on day-To work with the OmniPoP operator and CIC on day-to-day technical coordination of operational issuesto-day technical coordination of operational issues

Design GoalsDesign Goals



Design GoalsDesign GoalsPlatform capabilitiesPlatform capabilities– Robust, flexible and highly available infrastructureRobust, flexible and highly available infrastructure– OmniPoP facility capable/supports leading edge OmniPoP facility capable/supports leading edge

interfacesinterfaces– High-speed interface density (needed 20x10GEs day 1)High-speed interface density (needed 20x10GEs day 1)– Line rate AND/OR oversubscription architecture Line rate AND/OR oversubscription architecture

capabilitiescapabilities– Provision high-performance, full-featured switching & Provision high-performance, full-featured switching &

routing capability to reduce CIC duplication of routing capability to reduce CIC duplication of infrastructure in Chicagoinfrastructure in Chicago

– Transmit data at low latency and low jitter rates as well as Transmit data at low latency and low jitter rates as well as minimize bottlenecks between devicesminimize bottlenecks between devices

– Separation of control/data planeSeparation of control/data plane

Design GoalsDesign Goals(select) Technical features(select) Technical features

Nx10GE (802.3ad) link aggregation support (e.g. 2x10GE to I2)Nx10GE (802.3ad) link aggregation support (e.g. 2x10GE to I2)

Significant pluggable optics supportSignificant pluggable optics support

VLAN tagging (802.1q) & tag swappingVLAN tagging (802.1q) & tag swapping

Queue in Queue capabilitiesQueue in Queue capabilities

RADIUS supportRADIUS support

Strong QoS capabilitiesStrong QoS capabilities

Multicast (PIM, MLD, and IGMP snooping)Multicast (PIM, MLD, and IGMP snooping)

Broadcast/multicast suppressionBroadcast/multicast suppression

(rapid) Per VLAN spanning tree(rapid) Per VLAN spanning tree

Spanning tree controls (root guard, etc.)Spanning tree controls (root guard, etc.)

BFD (bidirectional forwarding detection)BFD (bidirectional forwarding detection)

L2 filteringL2 filtering

Broadcast/multicast suppressionBroadcast/multicast suppression

Design GoalsDesign GoalsMisc. non-technicalMisc. non-technical– Give up a little control to enable member-Give up a little control to enable member-

institutions to save $$$ & timeinstitutions to save $$$ & time– # & location(s) of facilities# & location(s) of facilities– How to buy issues (RFP v. existing contract)How to buy issues (RFP v. existing contract)– Reduced operating system software complexity Reduced operating system software complexity

(single train to reduce “feature fights” within CIC)(single train to reduce “feature fights” within CIC)– Operator has experience managing platform Operator has experience managing platform

(familiar CLI & tool extensions) – reduces OpEx(familiar CLI & tool extensions) – reduces OpEx

ArchitectureArchitecture



ArchitectureArchitectureSelected Force10 E1200 switchSelected Force10 E1200 switch

Equipment collocated at “StarLight” (710 NLSD)Equipment collocated at “StarLight” (710 NLSD)

Each institution receives a 1 & 10GE “line rate port”Each institution receives a 1 & 10GE “line rate port”– Required 20x10GE ports Required 20x10GE ports – Purchased 3 line rate & 2 (4:1) 16 port cardsPurchased 3 line rate & 2 (4:1) 16 port cards

Shared links use line rate portsShared links use line rate ports

Best effort across shared links (for now) with no shaping/policingBest effort across shared links (for now) with no shaping/policing

Aggregation of CIC NLR accessAggregation of CIC NLR access

Aggregation of connectivity to national/international Aggregation of connectivity to national/international R&E networksR&E networks

Interconnection of CIC institutions to each otherInterconnection of CIC institutions to each other

(pending) Aggregation of Internet2 access(pending) Aggregation of Internet2 access

Architecture (future)Architecture (future)

Commercial peering (non-transit/transit)Commercial peering (non-transit/transit)

Collocation space in other Chicago Collocation space in other Chicago locationslocations

Layer 3 infrastructureLayer 3 infrastructure

Layer 1 infrastructure???Layer 1 infrastructure???

DRAGON/OSCARs RD&D???DRAGON/OSCARs RD&D???

Architecture – Phase IArchitecture – Phase I

Mod

em

MREN

Participant Network #1

Participant Network #N1G Backup

NLR L3 Service

710 N Lakeshore Dr

International R&E Peers

10G NLR L3 Service

Fiber provided by CICRing participants

1G NLR L2 Natonal Peering Fabric

MgmtSwitch

ConsoleServer


National R&E Peers

Internet2 Network

National Lambda Rail

StarLight

OmniPoPSwitch

OmniPoP: Phase IIOmniPoP: Phase II

Mod

em

MREN


StarLight


NLR L3 Service

710 N Lakeshore DrAbilene

Futur

e

Transit

Peering


10G NLR L3 Service

Fiber provided by CIC Ring participants

CommercialInternet

Futur

e Peerin

g

Transit

Peering


MgmtSwitch

ConsoleServer


National R&E Peers

if possible

OmniPoPSwitch

OmniPoPRouter


Architecture - CurrentArchitecture - Current

Mod

em

MREN



NLR L3 Service

710 N Lakeshore Dr


10G NLR L3 Service

Fiber provided by CICRing participants


MgmtSwitch

ConsoleServer


National R&E Peers

Internet2 Network

StarLight

OmniPoPSwitch


QuestionsQuestions

[email protected]@doit.wisc.edu

OmniPoP CollaborationOmniPoP CollaborationLessons learnedLessons learned

Need collaboration idea & drivers to maintain group cohesionNeed collaboration idea & drivers to maintain group cohesionRecognize and balance constant tension between Recognize and balance constant tension between collaboration and self-control exists & requires managementcollaboration and self-control exists & requires managementCIC an important group to facilitate “people networking” to CIC an important group to facilitate “people networking” to identify other collaborators and common problemsidentify other collaborators and common problemsGood ideas grow – Chicago fiber and OmniPoP collaboration Good ideas grow – Chicago fiber and OmniPoP collaboration groups started small (2 members) and grew quicklygroups started small (2 members) and grew quicklyNeeds assessment to determine sharing opportunities difficult Needs assessment to determine sharing opportunities difficult and time consuming but vital to successand time consuming but vital to successSuccessful collaborations yield substantial rewards after Successful collaborations yield substantial rewards after accounting for additional time, communication and effort accounting for additional time, communication and effort required to sustain collaborationrequired to sustain collaboration

Documents

OmniPoP: GigaPoP-of-GigaPoPs Design and Future Patrick Christian, OmniPoP TAC (chair) University of Wisconsin - Madison April 25, 2007