CANARIE

“Critical Role Universities Will Play in the Future of the Internet”

The Customer Empowered Networking Revolution

http://www.canarie.ca

http://www.canet3.net

Bill.St.Arnaud@canarie.caTel: +1.613.785.0426

Outline The Message CANARIE CA*net 3 Customer Empowered Networking CA*net 4 and GigaPOP eScience

The Message In mid 1990s the prevailing wisdom was that commercial sector would drive

design of Internet infrastructure R&E networks would focus on applications or specialized services

As a result in North America R&E networks were commercialized or discontinued e.g NSFnet & CA*net

However new network architectures and most importantly dark fiber is allowing R&E networks to once again redefine telecommunications and the future of the Internet

LAN architectures, technologies and most importantly LAN economics are invading the WAN

Carrier neutral IXs (in essence GigaPOPs) are the essence of this network Control and management of the optics and wavelengths will increasingly be under

the domain of the LAN customer at the GigaPOP, as opposed to the traditional carrier in the center

These new concepts in customer empowered networking are starting in the same place as the Internet started – the university and research community.

Internet 2, SURFnet5 and CA*net 3 & 4

Mission: To facilitate the development of Canada’s communications infrastructure and stimulate next generation products, applications and services

Canadian equivalent to Internet 2 and NGI private-sector led, not-for-profit consortium consortium formed 1993 federal funding of $300m (1993-99) total project costs estimated over $600 M currently over 140 members; 21 Board members

CANARIE Inc

GigaPOP

CA*net 3 National Optical Internet

Vancouver

Calgary ReginaWinnipeg

Ottawa

Montreal

Toronto

Halifax

St. John’s

FrederictonCharlottetown

ORAN

BCnet

Netera SRnet MRnet

ONet RISQ

ACORN

ChicagoSTAR TAP

CA*net 3 Primary Route

Seattle

New York

CA*net 3 Diverse Route

Deploying a 4 channel CWDM Gigabit Ethernet

network – 400 km

Deploying a 4 channel Gigabit

Ethernet transparent optical DWDM–

1500 km

Multiple Customer Owned Dark Fiber

Networks connecting

universities and schools

16 channel DWDM-8 wavelengths @OC-192 reserved for CANARIE-8 wavelengths for carrier and other customers

Consortium Partners:Bell Nexxia

NortelCisco

JDS UniphaseNewbridge

Condo Dark Fiber Networks

connecting universities and

schools

Condo Fiber Network linking all

universities and hospital

Customer Empowered Networks Universities in Quebec are building their own 3500km fiber condo network in

partnership with number of carriers Universities in Alberta are deploying their own 700 km 4xGbe dark fiber network

Soon will be extended to all communities in Alberta School boards and municipalities throughout North America are deploying their

condo open access, dark fiber networks Illinois Iwire, Indiana Gwire, California DCP, Georgia SURAnet, Stockholm

condo fiber builds Chicago CivicNET, Alberta SuperNET, Canadian NBTF Carrier are selling “dim wavelengths” managed by customer to interconnect dark

fiber networks Williams, Level 3, Hermes

Typical cost is one time $20K US per school for a 20 year IRU for condominium fiber connection

What is condominium fiber? A number of organizations such as schools, hospitals, businesses and universities get together

to fund and build a fiber network Carrier partners are also invited to be part of condominium project

Several next generation carriers and fiber brokers are now arranging condominium fiber builds IMS, QuebecTel, Videotron, Cogeco, Dixon Cable, GT Telecom, etc etc

Fiber is installed, owned and maintained by 3rd party professional fiber contractors – usually the same contractors used by the carriers for their fiber builds

Each institution gets its own set of fibers, at cost, on a 20 year IRU (Indefeasible Right of Use) One time up front cost, plus annual maintenance and right of way cost approx 5% of the

capital cost Institution lights up their own strands with whatever technology they want – Gigabit Ethernet,

ATM, PBX, etc New long range laser will reach 120 km

Ideal solution for point to point links for large fixed institutions Payback is usually less than 18 months

Why Condo Fiber? First - low cost

Up to 1000% reduction over current telecom prices. 6-12 month payback Second - LAN invades the WAN – no complex SONET or ATM required in

network Network Restoral & Protection can be done by customer using a variety of

techniques such as wireless backup, or relocating servers to a multi-homed site, etc

Third - Enables new applications and services not possible with traditional telecom service providers Relocation of servers and extending LAN to central site Out sourcing LAN and web servers to a 3rd party because no performance impact IP telephony in the wide area (Spokane) HDTV video

Fourth – Allows access to new competitive low cost telecom and IT companies at carrier neutral meet me points Much easier to out source servers, e-commerce etc to a 3rd party at a carrier

neutral collocation facility

À venir

Bande passante louée

Projet démarré

Construit

Observatoire Mont-Mégantic

Val d’Or/Rouyn

MAN de Montréal

MAN de Québec

MAN de Sherbrooke

MAN d’Ottawa/Hull

Quebec University Condo Network

St-Laurent/Vanier

Lanaudière

Maisonneuve

Marie-Victorin

Champlain

Rosemont

Sorel-Tracy

Montmorency

Édouard-Montpetit

Vieux-Montréal

Bois-de-Boulogne

Ahuntsic

Lionel-Groulx

Vers Québec

Gérald-Godin

John-Abbott

André-Laurendeau

Dawson

À venir

Bande passante louée

Projet démarré

Construit

Montreal Public Sector Condominium Networks

School Board

Stud

y

Eng

inee

ring

Con

stru

ctio

n

PROJECT

Capitale x 140 km of fibre optics 80 schoolsRégion-de-Sherbrooke x 180 km of fibre optics 66 schoolsRivière-du-Nord x 175 km of fibre optics 52 schoolsSeigneurie-des-Mille-Iles x x x 200 km of fibre optics 80 schools 4 partnersAmiante x x x 12 km of fibre optics 9 schoolsLaval x x x 170 km of fibre optics 111 schools 3 partnersSaint-Hyacinthe x 250 km of fibre optics 51 schoolsAffluents x x x 170 km of fibre optics 70 schools 4 partnersBois-Francs x x x 60 km of fibre optics 12 schools 4 partnersDraveurs x x 90 km of fibre optics 40 schoolsGrandes-Seigneuries x 210 km of fibre optics 58 schoolsHautes-Rivières x 250 km of fibre optics 54 schoolsLaurentides x 200 km of fibre optics 35 schoolsPatriotes x 2 km of fibre optics 3 schoolsPremières-Seigneurie x 190 km of fibre optics 73 schoolsSamares x 460 km of fibre optics 72 schoolsTrois-Lacs x x 45 km of fibre optics 15 schoolsChemin-du-Roy x 29 km of fibre optics 11 sitesMarie-Victorin x 6 km of fibre optics 5 schoolsSir-Wilfrid-Laurier x x x 92 km of fibre optics 20 schools

List of Schoolboard Fiber Builds

• Province wide network of condominium fiber to 420 communities in Alberta• Guaranteed cost of bandwidth to all public sector institutions

• $500/mo for 10 Mbps, $700/mo for 100 Mbps• Network a mix of fibre builds and existing supplier infrastructure

(swap/buy/lease)• Condominium approach: All suppliers can

• Buy (or swap) a share of the fibre (during build or after)• Lease bandwidth at competitive rates

• GOA has perpetual right to use (IRU) • Ownership will be held at arms length• GOA/stakeholder rates are costs to run divided over users• Because of fibre capacity, bandwidth can be made available to businesses

at urban competitive rate• Total cost $193m• Bell Intrigna prime contractor

Alberta SUPERnet

National Broadband Task Force

Mandate:To map out a strategy and advise the Government on best approaches to make high-speed broadband Internet services available to businesses and residents in all Canadian communities by the year 2004.

To ensure Canada’s competitiveness in a global economy To address the Digital Divide To create opportunities for all Canadians 35 members including carriers, educators, librarians, communities,

equipment manufacturers, etc

Chair – David Johnston

Fredericton Fiber Build Started as Economic Development tool MUSH, Govt., Research - ISP, carriers invited to participate Build partners emerged quickly, $50,000 “donated” by three firms Contracting now for 8 km phase 1, $110,000, complete Sept 2001 48 fiber min. Unique experiment to extend off campus Internet access through

802.11 wireless

Ottawa Fiber Condominium

Consortium consists of 16 members from various sectors including businesses, hospitals, schools, universities, research institutes

26 sites Point-to-point topology 144 fibre pairs Route diversity requirement for one member 85 km run $11k - $50K per site Total project cost $CDN 1.25 million Cost per strand less than $.50 per strand per meter 80% aerial Due to overwhelming response to first build – planning for second

build under way

Condo Fiber Costs - Examples Des affluents: Total cost $1,500,00 ($750,00 for schools)

70 schools 12 municipal buildings 204 km fiber $1,500,000 total cost average cost per building - $18,000 per building

Mille-Isles: Total cost $2,100,000 ($1,500,000 for schools) 80 schools 18 municipal buildings 223km $21,428 per building

Laval: Total cost $1,800,000 ($1,000,000 for schools) 111 schools 45 municipal buildings 165 km $11,500 per building

Peel county: Total cost $5m – 100 buildings Cost per building $50,000

Typical Payback for school(Real example – des affluents – north of Montreal)

Over 3 years total expenditure of $1,440,000 for DSL service Total cost of dark fiber network for 75 schools $1,350,000 Additional condominium participants were brought in to

lower cost to school board to $750,000 School board can now centralize routers and network servers

at each school Estimated savings in travel and software upgrades

$800,000 Payback typically 8 –16 months Independent Study by Group Secor available upon request

Before

After fiber

fiberAntennas 780Novell Servers 82 1SQL Servers 13 3Lotus Notes Servers 21Tape Backup Servers 12 4Ethernet switches/hubs 10 98Routers 1083Cache/proxy (Linux) 120Fire walls (Linux) 11

Reduction in the number of servers

Community Fiber Architecture A community consortia would put together a plan to fiber up all public sector

buildings in their community A community can be a province, a municipality, village, etc

A fiber splice box that terminates the fiber at the street side nearby each public sector building such as school, hospital, library is called a “Node”

Community should must insure that potential facilities exist near the for private sector equipment to connect up future home owners – colo facility

Colo facility allows private sector to extend wireless, VDSL or HFC services to the neighbourhood around the school

Public sector buildings will have dedicated fiber strands that connect to a “Supernode” which is a fiber splice box on the street beside outside of major public sector central facility such as school board office, city hall, university, etc

Community should insure that facilities exist nearby the Supernode for the private sector to install equipment to service home owners and businesses – colo facility

Additional fibers are made available from the Supernode to all Nodes such that competitive service providers can purchase fiber to the node at some future date

Possible architecture for large town

School

School board office

School

Telco Central Office

Central OfficeFor Wireless

Company

VDSL, HFC or FiberProvisioned by service provider

Condominium Fiber with separate strands owned by school and by service providers

Carrier Owned Fiber

Cable head end

Average Fiber Penetration to 250-500

homes

ColoFacility 802.11b

Benefits to Industry For cablecos and telcos it help them accelerate the deployment of high speed

internet services into the community Currently deployment of DSL and cable modem deployment is hampered

by high cost of deploying fiber into the neighbourhoods Cable companies need fiber to every 250 homes for cable modem service,

but currently only have fiber on average to every 5000 homes Telephone companies need to get fiber to every 250 homes to support

VDSL or FSAN technologies Wireless companies need to get fiber to every 250 homes for new high

bandwidth wireless services and mobile Internet It will provide opportunities for small innovative service providers to offer

service to public institutions as well as homes For e-commerce and web hosting companies it will generate new business in out

sourcing and web hosting For Canadian optical manufacturing companies it will provide new opportunities

for sales of optical technology and components

CA*net 4 Overall Objective

To deploy a novel new optical network that gives GigaPOPs at the edge of the network (and ultimately their participating institutions) to setup and manage their own wavelengths across the network and thus allow direct peering between GigaPOPs on dedicated wavelengths and optical cross connects that they control and manage

To allow the establishment of wavelengths by the GigaPOPs and their participating institutions in support of QoS and eScience applications

To allow connected regional and community networks to setup transit wavelength peering relationships with similar like minded networks to reduce the cost of Internet transit

To offer an “optional” layer 3 aggregation service for those networks that require or want such a facility

CA*net 4 Possible Architecture

Vancouver

Calgary ReginaWinnipeg

Ottawa

Montreal

Toronto

Halifax

St. John’s

Fredericton

Charlottetown

Chicago

Seattle

New York

Europe

Customer controlledoptical switches

Layer 3 aggregation serviceOptional Service Available to any GigaPOP

Large channel WDM system

eScience The ultimate goal of e-science is to allow students and eventually members

of the general public to be full participants in basic research. We have seen in other fields like bird census, comet watching, SETI@home,

public are interested in participating in basic research Using advanced high speed networks like CA*net 4 and novel new concepts

in distributed peer to peer computing, called “Grids” many research experiments that used to require high end super computers can now use the computer capabilities of thousands of PCs located at our schools and in our homes.

High performance computers that are part of C3.ca can be seamlessly integrated with eScience distributed computers using CANARIE Wavelength Disk Drive over CA*net 4

Allows researcher access to the significant computational capabilities of all these distributed computers at our schools and homes

Will also allow students and individuals to be a full participant in the analysis and basic research.

With e-science it might be possible that the next big scientific discovery could be by a student at your local school.

SETI@home Demonstrated that PC Internet Computing

Could Grow to Megacomputers

Running on 500,000 PCs, ~1000 CPU Years per Day Over Half a Million CPU Years so far! 22 Teraflops sustained 24x7

Sophisticated Data & Signal Processing Analysis Distributes Datasets from Arecibo Radio Telescope

Next Step-Allen Telescope Array

AreciboRadio Telescope

Forest Grid– on CA*Net 4

BC

Alberta Sask. Man.Ont.

Quebec.

Maritimes

ORAN

Univ.

Labs/Ministries.

PFC

UVic

UBC

U of A

CFSNorthern

York

U. Ottawa

Waterloo

EDC

ChicagoSTARTAP.

CCRS

CFSGLFC

CFS-HQ

CFSLaurentian

UNB

CFSAtlantic

Cornerbrook

Min. of For.

Min. of For.

Min. of For.Min. of For.

Min. of For.

Min. of For.

Min. of For.

Quebec.

EuropeanNSFVBNS

UCAIDAbilene

JPLSeattle

DRENESnet

CFS

New York

NISN

NREN

CSARSI

Neptune eScience Grid Joint US-Canadian project to build large undersea dark fiber network off

west coast of USA and Canada Undersea network will connect instrumentation devices, robotic submarines,

sensors, under sea cameras, etc All devices available to students and researchers connected to CA*net 4

and Internet 2 networks Neptune will be used to gather research data in a variety of fields –

seismology, sea vents, fish migrations and population, deep sea aquatic life, etc

Distributed computing and data storage devices on CA*net 4 and Internet 2 will be used to analyze and store data

Neptune – Undersea Grid

Neptune eScience

Wavelength Disk Drives

Vancouver

Calgary

Regina

Winnipeg

Ottawa

Montreal

Toronto

Halifax

St. John’s

Fredericton

Charlottetown

CA*net 3/4

WDD Node

Wavelength Disk Drives

CA*net 3 and CA*net will be “nation wide” virtual disk drive for grid applications

Big challenges with grids or distributed computers is performance of sending data over the Internet TCP performance problems Congestion

Rather than networks being used for “communications” they will be a temporary storage device

Conclusion

Many governments have recognized the importance of access to low cost dark fiber as fundamental economic enabler

It will be the 21st century equivalent to the roads and railways that were built in the 20th century