1

UCB External ConnectivityUCB External Connectivity

ken lindahlcommunication and network services

uc berkeleylindahl@berkeley.edu

EE122, 7 May 2004

EE122, 7 May 2004 2

CalRENCalREN--2: 1998 to 20032: 1998 to 2003• conceived as a state-wide high-performance, advanced

services network serving California’s research and higher education communities.

• charter members: UC, CSU, Stanford, CalTech, USC

• CalREN-2 network was supposed to OC-48 (2.5Gbps), but what got delivered was two OC-12 (622 Mbps) SONET rings– should have asked for “OC-12c.” oops.

• implemented as one POS (Packet Over SONET) ring and one ATM ring

EE122, 7 May 2004 3

CalRENCalREN--2 northern 2 northern gigaPOPgigaPOP, ca. 11/98, ca. 11/98• “distributed gigaPOP,” with backbone equipment

(SONET MUXes, ATM switches, IP routers) located in campus facilities.

• commercial SONET service for connections between campuses.

EE122, 7 May 2004 4

CalRENCalREN--2 southern 2 southern gigaPOPgigaPOP, ca. 11/98, ca. 11/98

EE122, 7 May 2004 5

CalRENCalREN--2 backbone, circa 11/982 backbone, circa 11/98

EE122, 7 May 2004 6

CalRENCalREN--2: 1998 to 20032: 1998 to 2003• UCB initially connected to CalREN-2 with a single

OC-12 POS link

• subsequently added one, then two, OC-3 (155 Mbps) ATM links for ISP traffic– separate path required in order to control campus ISP costs– diverse routing provided a measure of redundancy, for protection

against failures (but Evans Hall Machine Room was still single point of failure)

2

EE122, 7 May 2004 7

UCB external connectivity, ca. 5/2003 (1)UCB external connectivity, ca. 5/2003 (1)

inr-205

1000

1000

10001000

inr-220inr-230 inr-240 inr-260 inr-270

inr-281

inr-280

inr-120 inr-130 inr-140 inr-160 inr-170 inr-100

100

inr-150

100 100 100 100 100 100

inr-101 inr-102

inr-299

ResHall

EECS Network100

155

1000 1000

Internet2

622

CalREN-2

inr-000

10001000

1000

100

155

CENIC ISP(Level3, Qwest)

100

100

inr-667inr-666

100

100

inr-668

Cogent(SETI@home ISP)

PAIX

100

AffiliatesAffiliates

Affiliates

inr-201 inr-202

100

rate limitingdevices

inr-203

100

Tier 1Backbone

Tier 2Backbone

Seti@Home

BorderRouters

EE122, 7 May 2004 8

shortcomings of the original CalRENshortcomings of the original CalREN--2 network2 network• campus facilities often not hardened against power

outages, flooding, rats, … (read: Evans Hall)

• very little redundancy in network; failures tended to have wide-spread, significant effects

• campus support staff often not funded for 24x7 support

• commercial SONET circuits very expensive, especially after initial 3-year contract expired

• mixing research and ISP traffic on a single backbone had an unexpected result: reliability and availability (for ISP) became more important than flexibility (for research), it became very difficult to deploy new features on the backbone.

EE122, 7 May 2004 9

CENIC next generation backbone considerations (1)CENIC next generation backbone considerations (1)• running out of bandwidth on OC-12 backbone ciruits

• eliminate dependence on flakey campus facilities

• provide redundant connections to campuses

• need to separate production and research networks

• need for true separation of ISP traffic for rate-limiting

capacity planning based on large amounts of headroom needed for occasional use

capacity planning based on expected long-term utilization

outages are undesirable, but acceptable

very low tolerance for outages

run new code as soon as it becomes available

run well-tested code, let somebody else find the bugs

constantly changingavoid change whenever possible

Advanced Services NetworksProduction Networks

EE122, 7 May 2004 10

CENIC next generation backbone considerations (2)CENIC next generation backbone considerations (2)• over-investment in fiber plants causing many carriers to

sell or lease fiber at low cost– vs. recurring high costs for commercial circuits

• development of wave division multiplexing technology– providing multiple high-speed (2.5 Gbps, 10 Gbps, 40 Gbps, …)

channels over a single fiber pair (single fiber strand in some cases)

• researchers requesting “private pipes” between campuses– experiment with non-IP (unroutable) packets– experiment with routing protocols– experiment with “dangerous” packets

• small number of researchers requesting dark fiber between campuses

EE122, 7 May 2004 11

the pyramid!the pyramid!

• plus, ISP “sub-layer” within CalREN-DCEE122, 7 May 2004 12

CalREN Optical Backbone, ca. 3/2004CalREN Optical Backbone, ca. 3/2004

3

EE122, 7 May 2004 13

Wave Division Multiplexing (sort of) demystifiedWave Division Multiplexing (sort of) demystified• multiple channels of data carried over single fiber pair

between WDM terminals

• each channel has its own wavelength (“lambda” or “wave”)

• optical multiplexers combine multiple wavelengths into single composite signal; optical demultiplexers split composite signal into individual wavelengths

• amplification and regeneration needed for longer distances

EE122, 7 May 2004 14

CalREN backbone wavesCalREN backbone waves

San Diego

Tustin

Los Angeles

Santa Barbara

SunnyvaleOakland

San LuisObispo

Soledad

HPRTeragridHPR & DC

CalREN DC

Fergus

Fresno

Bakersfield

Sacramento

UC Davis

Triangle Court

EE122, 7 May 2004 15

Bay Area Metro RingBay Area Metro Ring

EE122, 7 May 2004 16

CalRENCalREN--DC: 10/2003 to presentDC: 10/2003 to present• OC-48c (2.5 Gbps) backbone over private DWDM fiber

• Six GE links to the commodity Internet (Qwest, Level3, Cogent)

• “no-cost” private peerings at PAIX, LAAP, SDNAP

• Serving:– California State University (all campuses)– California Community Colleges (all campuses)– University of California (all campuses)– ~80% of K-12 schools in California

• Estimated user base is 10 million faculty, staff and students!

AOL: 26 million usersCalREN-DC: 10 million usersMSN: 9 million usersEarthlink: 5 million usersNetzero/Juno: 5 million usersComcast/ATT: 3.6 million usersSBC Internet Services: 2.2 million users

EE122, 7 May 2004 17

CalRENCalREN--HPR: 2/2004 to presentHPR: 2/2004 to present• 10Gigabit Ethernet backbone over private DWDM fiber

• Cisco 12400 routers

• 10GE connection to Abilene

• connection to CUDI (Mexico’s version of Internet2)

• Jumbo frames (9180) and IPV6 enabled

• Serving:– University of California (all campuses)– Stanford– University of Southern California – CalTech & Jet Propulsion Laboratory (JPL)

• (future) 10GE to campuses, 12-18 months from now

• (future) PacificWave: distributed exchange point with POPs between LA and Seattle, ideal for landing trans-Pacific fiber links

EE122, 7 May 2004 18

UCB connectivity to CalREN (1)UCB connectivity to CalREN (1)• Total of six GE links:

– GE to CalREN-HPR in CEV near Recreational Sports Facility**– GE to CalREN-DC in CEV near Recreational Sports Facility– GE to CalREN-ISP in CEV near Recreational Sports Facility– GE to CalREN-HPR in Evans Hall**– GE to CalREN-DC in Evans Hall*– GE to CalREN-ISP in Evans Hall*

* share a single 1 Gbps channel over SBC EGM** not yet in service, expect service June 2004

• Redundancy: two campus locations, connected to different CalREN POPs via diverse fiber paths– OAK POP (Emeryville) via dark fiber leased from MFN– SFO POP via SBC Extended GigaMan (EGM) service

4

EE122, 7 May 2004 19

new UCB border routersnew UCB border routers• existing border routers were not up to the task

– inr-666, a cisco 7507, can handle GE interfaces but with only 300-400 Mbps onto the backplane.

– inr-000, a cisco GSR 12008, can handle multiple GE interfaces, but at great expense, and the 12008 is already being “sunsetted” by cisco.

• requirement for redundancy means at least two routers

• need to maintain separation of ISP from non-ISP traffic would have require (at least) four border routers, rather than just two…

• … or use “Policy Constrained Routing” developed in Internet2 Routing WG.

• RFP for new campus border routers led to selection and purchase of two Juniper M40e routers– being installed and configured now, in full production Summer

2004EE122, 7 May 2004 20

Rec Sports CEV

Evans Hall

HPR

DC

ISP

HPR

DC

ISP

HPR+DC

ISP (pri)

ISP (sec)

ISP (pri)

ISP (sec)

mirrorport

mirrorport

shaper 1

shaper 2

resnetshaper

HPR+DC

HPR+DC

HPR+DC

CalREN

affiliate

cogent INR-668

affiliatesborderrouter

borderrouter

#2to

CalREN

campusnetwork

resnetINR-205

6500

INR-201

6500

INR-202

6500

akamaicluster

borderrouter

#1to

CalREN

3

1

2

04

567

8

3

1

2

0

4

56

7

8

MFN fiber toEmeryville

9

9

Figure 2. UC Berkeley Border Topology showing connections to next-generation CalREN network.

CENICcatalyst

4003

SBC GEservice to SF

UCB connectivity to CalREN (2)UCB connectivity to CalREN (2)

EE122, 7 May 2004 21

Policy Constrained RoutingPolicy Constrained Routing• developed by Cisco and Juniper, working with CENIC

and Internet2 Routing Working Group to address limitations of traditional destination-based routing decisions

• uses “virtual routing and forwarding” tables (VRFs) to make forwarding decisions based on router input interface and destination IP address– a specific VRF is applied to all packets entering a particular

interface– different VRFs, expressing different routing policies, applied to

different interfaces

• needed at UCB to maintain separation of ISP vs non-ISP traffic

EE122, 7 May 2004 22

Policy Constrained Routing in UCB border routersPolicy Constrained Routing in UCB border routers

HPR

DC

ISP

5

6

4

3

1

0

HPR+DCVRF

ISPVRF

2

8.0 8.1 8.2

ResNetPacketShaper

CampusPacketShaper

802.1Q subinterfaceson Gigabit Ethernet

AffiliateConnections

"ALL"VRF

INR-201

INR-202

INR-201

INR-202

INR-2057

UCBVRF

CalREN

A-1A-2

A-3

CalREN

UCB-ONLYVRF

EE122, 7 May 2004 23

National National LamdaRailLamdaRail• coalition of E&R entities, including CENIC, PNWGP, I2, Cisco

• uses DWDM (40 10GE lamdas) for multiple testbed networks– waves (L1) or switched ethernet (L2) or routed IP (L3

• first transcontinental Ethernet network

• additional dark fiber pairs