Optical Networks for the Rest of Us“Customer Empowered Networking”

NANOG 17 – Montreal 1999

http://www.canarie.ca

Background Papers on Gigabit toThe Home and Optical InternetArchitecture Design AvailableOptical Internet News list:Send e-mail to Bill@Canarie.ca

Bill.St.Arnaud@canarie.cahttp://Tweetie.canarie.ca/~bstarnTel: +1.613.785.0426

Mission: To facilitate the development of Canada’s communications infrastructure and stimulate next generation products, applications and services

Canadian equivalent to Internet 2 and NGI private-sector led, not-for-profit consortium consortium formed 1993 federal funding of $220m (1993-99) total project costs estimated over $500 M currently over 140 members; 21 Board members Phase III funding to be announced 1998-2001

$55 million announced for Optical Internet -March 1998

CANARIE Inc

GigaPOP

CA*net 3 National Optical Internet

Vancouver

Calgary ReginaWinnipeg

Ottawa

Montreal

Toronto

Halifax

St. John’s

FrederictonCharlottetown

ORAN

BCnet

Netera SRnet MRnet

ONet RISQ

ACORN

ChicagoSTAR TAP

CA*net 3 Primary Route

Seattle

New YorkLos Angeles

CA*net 3 Diverse Route

Deploying a 4 channel CWDM Gigabit Ethernet

network – 700 km

Deploying a 4 channel Gigabit

Ethernet transparent optical DWDM–

1500 km

Multiple Customer Owned Dark Fiber

Networks connecting

universities and schools

16 channel DWDM-8 wavelengths @OC-192 reserved for CANARIE-8 wavelengths for carrier and other customers

Consortium Partners:Bell Nexxia

NortelCisco

JDS UniphaseNewbridge

Condo Dark Fiber Networks

connecting universities and

schools

What is an Optical Internet? WDM fibers where individual wavelengths are the link layer

interconnect directly connected to routers via Optical ADM (Add Drop Mux) or WDM coupler

High Performance Router acts as the main switching routing device Bypass or cut-thru connections via dedicated wavelengths SONET or Gigabit Ethernet framing (also 10xGbE or SDL) Use intrinsic self healing nature of Internet for redundancy and

protection (don’t require SONET/SDH layer) Traffic engineering and network management done via MPLS Network design optimized for unique characteristics of Internet traffic

– fractal traffic, asymmetric traffic and congestion at the edge

Lessons Learned Carrier transport people now must learn to deal with customers directly

Require network management tools that give customer a view of “their” wavelengths

A whole new set of operating procedures required OAM&P issues between router vendors and DWDM remain a challenge

SONET management systems expect to see a contiguous network CA*net 3 required DCC work arounds

Need network tools to measure end to end performance and throughput at OC-48 or greater speeds – HP is about to release a couple of beta products

MPLS is proving a lot more difficult in practice to implement Need tools for management of tunnels Need Inter-domain MPLS-TE Mythology of 50msec “fast restoral” still not understood OSPF with very short hold down timers and GRE tunnels or policy routing may be an

adequate alternative

10xGbE & CWDM Several companies have announced long haul GbE and CWDM with

transceivers at 50km spacing 10GbE coming shortly

Costs are as little as $12K US per node (or transceiver) Future versions will allow rate adaptive clocking for use with “gopher

bait” fiber, auto discovery, CPE self manage Excellent jitter specification Most network management and signaling done at IP layer Anybody with LAN experience can build a long haul WAN – all you need

is dark fiber With CWDM, no EDFA power disbursement

Repeater distance independent number of wavelengths

Many ISPs, regional networks, municipalities, school districts are purchasing dark fiber or building dark fiber networks up to 1000 km rather than managed bandwidth

With dark fiber increased bandwidth only entails upgraded equipment costs and no additional monthly charges

Significant savings in relocating servers to central site and using VoIP Also many carriers willing to sell “gopher bait” fiber (fiber that does not meet stringent

SONET/DWDM requirements) at a discount As such, cost of transmission equipment is becoming a significant factor versus cost of fiber SONET and ATM networks require specialized engineering knowledge and skills Customers want a technology in the WAN they are familiar with and that is easily

extensible from the LAN e.g. Ethernet Don’t require the same reliability as telco systems

Market drivers for GbE in the WAN

Major Long Haul Componentsfor IP/DWDM network

SONETRegen$250k per Tx/Rx

Approximate Distances for OC-192 systemTypical Cost $6000 per km (not counting cost of fiberrouter, and transponder) for one OC-192 channelAdvantage – can support multi-services and well known technology

WDM Coupler$20K

250 km

50 km

WidebandOptical Repeater$250K

SONET TransportTerminal

Transponder

For transponder currentlyusing regen box$125K

Terabit Router$400K

Major Long Haul Componentsfor 10xGbE CWDM network

Approximate Distances for 10xGbE systemTypical Cost $400 per km (not counting cost of fiberOr 10xGbE switches) for 10 GbpsAdvantage – very low cost 1/10 cost of SONET & DWDMDisadvantage – requires 2 fibers and can only carry IP traffic

CWDM Coupler$5K

50 km

10x Transceiver$20K

10xGbE Switch$20K

G G

GG

Optical Networks for the Rest of Us With customer owned dark fiber, 10GbE and 4 channel CWDM

anybody can build a 40Gbps network up to 1000km or greater at a fraction of cost of traditional telco network

May not be suitable for mission critical traffic (at least not yet) But ideal for high bandwidth Internet to the school, small business

and home Ring structures are a customer option – not a mandatory requirement The driver is NOT new applications, but cost savings –1 year

payback Typical cost is one time $20K US per school for a 20 year IRU In Ottawa we are deploying a 60km- 96 strand network connecting

22 institutions – cost $500K US

Where are we going? Today the Internet is “virtual” network riding on top of a traditional

“connection oriented” network of cooper and fiber With optical technology such as customer owned dark fiber, customer

owned wavelengths, 10GbE etc we can extend the model of the Internet as tool to empower the user to build networks in the physical domain as well as the virtual domain

The future telecommunication’s world may be dominated by thousands of customer owned networks that peer at the physical as well as at the virtual level, “Optical Networks for the Rest of Us” A national or provincial K-12 network with its own wavelengths and dark fiber A national banking network with its own wavelengths and dark fiber A national auto network with its own wavelengths and dark fiber

A radical departure from the “carrier centric” view of the universe

2 Different Views

Telco Network“Ring of Rings”99.999% reliability only in the SONETRing for the telco, no guarantees for thecustomer

ISP A

ISP B

ISP C

CO

Customer Empowered Network“Warp and Weave”Customer responsible for building rings99.999% reliability to the customer butno guarantees for the ISP

SONET ring

3 separate dark fiber builds

Customer Empowered Networks

ISP A

ISP B

ISP C

ISP A

ISP B

ISP C

Dark fiber or CWDM NetworkCity A

First Dark fiber NetworkCity B

Dim Wavelength

Long Haul DWDM

Second Dark Fiber Network

Customer achieves 99.999reliability by multi-hometo different ISPs

ISP D

Optical LabelSwitched Router

Dark fiber NetworkCity C

New Challenges and Opportunities “Customer empowered networks” present a whole new set of

research challenges: Peering and topology protocols in the optical domain – what

will be the equivalent to BGP and OSPF in the optical domain Multi Protocol Lambda Switching? Defining LSP attributes such as power level, wavelength, encoding,

etc? Interdomain optical MPLS?

Customer controlled establishment of wavelengths, routing and service delivery Auto discovery of wavelengths?

Management and interface systems, etc, etc