CA*net 4 Optical Networking and Third Wave

of the Internet

Bill St. Arnaud

CANARIE Inc – www.canarie.ca

Bill.st.arnaud@canarie.ca

Overview

> 1. CA*net 4 Network Architecture> 2. Applications that may drive require a new

network architecture

CA*net 4 Drivers-1Reduce cost of Internet

> Set up lightpaths to no cost peering exchanges– Most lambda sales in Canada and USA are for “Remote peering” to

no cost peering points– Allows for considerable savings in Internet transit costs– Each lightpath is directly connected to a high volume peer and

bypasses peering router– Good example is “STAR LIGHT” where high volume peers have

direct connect and small volume peers use a router– CA*net 4 “customer controlled patch panel” allows peers to change

peering relationship remotely without contacting technical staff at peering exchange• Very similar in concept to WorldCom “Peermaker” at MAEs which use

e-mail to setup peers

CA*net 4 Drivers-2Reduce cost of routers

> Eliminate expensive high end routers and replace them with lower cost optical switches

– But circuits are NOT intended to replace packet networks– Use rich mesh of circuits between edge routers to eliminate high

cost of 10GbE core routers• 10Gbe routers ~ $500K with interfaces at ~$200k each

• 10Gbe switches ~$25K with interfaces at ~$20k each

– Trade off between cost of multiple lightpaths and loss of multiplexing versus cost of high end core routers

– 10Gbe wavelengths ~$1000/km for 5 years (lifetime of router)– Assume 1 GbE lightpaths per edge institution then

• One 5000km Gbe lightpath (or 8 x 600km GbE) lightpaths per institution is cheaper than routers

CA*net 4 Drivers-3A VPN alternative to MPLS

> Allows customer to create “customer owned and managed” networks with resource heterogeneity– Integration of wavelengths and dark fiber from different carriers– Create customer controlled VPNs for downstream users and

overlay networks across multiple suppliers– Customers can manage their own restoral and protection schemes– Allows for inter-domain end to end setup of VPNs– End users do not need to to signal carrier for VPN management

• Create VPNs

• Cross connect VPNs from independent users

• Partition or spawn VPNs

• Establish VPNs across multiple management domains

CA*net 4 Drivers-4Application Specific Networks

> Lambda Grids - “Underlay” networks to support Grids and overlay projects like PlanetLab and Oceanstore– A lot of exciting research into overlay networks – At some point in time when traffic volume is sufficient in overlay network

to setup its own direct path> Soon high end grid applications will have sufficient traffic volume to

require their own underlay networks ”Complementing” routed networks – Not a replacement for routed networks – only increasing the direct

peering mesh of the routed network for specific applications– But peering may be more dynamic (and not globally advertised) than

traditional IP BGP peering> Discipline or applications specific networks

– VBLI grids like European EVN – High energy physics grid – Ultralight– NEES grid, Bio-informatics Grid, etc

Example – EVN traffic flows over GEANT

UK

SE

FR

NLBE

DE1DE2

CZ

PL

CH

IT

AT

SURFnet

JANET

GARR

PSNC

DFN

NORDUnet

2.5G10G

JIVE

Provided courtesy of Dai Davies

Issues

> How do you charge for bandwidth and usage when single application traffic dwarfs all other IP traffic?

> Who pays for the traffic volume when it sinks into one NREN?> Possible solutions:

1. GMPLS (with QoS)• Requires expensive routers and complex coordinated central

management to setup and tear down tunnels

• Does not address issue of traffic charging

• Interdomain still unproven

2. Optical overlay/underlay –ASON – same problems as GMPLS

3. Application specific optical BGP networks– Increase BGP mesh for specific applications or disciplines

OBGP applied to EVN

UK

SE

FR

NLBE

DE1DE2

CZ

PL

CH

IT

AT

SURFnet

JANET

GARR

PSNC

DFN

NORDUnet

JIVE

EVN sites will see 2 BGP routes to SURFnet:-the normal IP route over GEANT-Express route using dedicated lightpaths (in green)

Express route

CA*net 4 Drivers-5QoS

> Spatial QoS – TCP throughput over long fat pipes very susceptible to packet

loss, MTU, TCP kernel, Buffer memory, AQM optimized for commodity Internet, Auto negotiating Ethernet, etc

– May also require consistent and similar TCP throughput for multiple sites to maintain coherency for grids and SANs

– Some exciting new TCP protocols like FAST, XCP, etc• Mice and Elephant problem• Without careful design may look like a DOS attack on a router network

– Many commercial SAN/Grid products will only work with QoS network

– Some users want to have super jumbo MTU (64K) or protocols other than IP

Spatial QoS

Application or end user controls peering of BGP optical paths for transfer of elephants!!!

Normal BGP pathx.x.x.1 y.y.y.1

OBGP pathOnly y.y.y.1 advertised to x.x.x.1 via OBGP path

Only x.x.x.1 advertised to y.y.y.1 via OBGP path

Optical “Peermaker”

CA*net 4 Drivers-6Extend E2E principle to circuits

> Extend the Internet end to end principle to circuit based networks– The success of the Internet is largely attributable to the classic e2e principle– Allowed development of exciting new applications or services

> E2E principle presumes all users behave honorably– Otherwise you get spam and DDoS attacks– This ungentlemanly behavior can be reasonably contained on commercial

Internet but pose big challenge on research networks> What happens if you have a malicious implementation of FAST or XCP on a

big pipe network?> Can the Internet e2e principle be applied to circuit based networks?

– Will it engender the same creativity in new applications and services?> MPLS and ASON are classic network state based solutions for VPNs

– CA*net 4 architecture is an alternate approach– All VPNs are BGP direct static routes using lightpaths that are setup and

controlled by end user

CA*net 4 is NOT a network

> It is an aggregation of point to point 10 Gbps wavelengths from a number of carriers

> CA*net 4 is made up of may parallel application or discipline specific networks that may (or may not) BGP peer with each other

> The wavelengths and switches are partitioned into smaller lightpaths with user control of the switch partition which are used for a variety of applications particularly grids– International Grid Testbed – 10 Gbe server to server to CERN– WESTgrid – 1 Gbe lightpaths for distributed backplane– Numerous lightpaths to support direct peering between regional networks

and universities– Lightpaths to support TransLight projects between North America, Europe

and Asia– Many, many more coming – Virtual Astronomy, HDTV video walls, etc

Applications for E2E Lightpaths

The three waves

– The first wave of the Internet consisted primarily of text and data services such as e-mail and FTP.

– The second wave was the web which improved ease of use and facilitated the transfer of images, sound and video.

– The third wave is the integration of applications, p2p networking, open source, distributed computing enabled by next generation web services, semantic web and high speed networks

What is the Third Wave?

> Before the Web on-line information was only available through a small number of information providers who charged high fees

– Compuserve, Dialogic, etc> The Web allowed millions of creators of information to make it

easily accessible to all others at very low cost, bypassing the information middleman

> The Third Wave proposes to extend the WEB paradigm to processes, applications and content

> Third Wave is about creation of tools and applications (i.e. services) in variety of fields such as eLearning, eBusiness, eScience, eHealth, etc that can make these services easily available to all others

– At there are millions of web sites, there will be millions of Third wave services

Today’s Network

Application

OS

Data

Application

OS

Data

Network

User User

The application is tightly bound to the OS

The network is a mechanism for applications to communicate with each other

The network is subservient to the computer

Third Wave Network

Network

Application and Data

Third Wave

OS

Data

Third Wave

OS

Data

Third Wave

OS

Data

Third Wave

OS

Data

Third Wave

OS

Third Wave

OS

Application and data exist on the network and are uncoupled from any specific machine or location

The computer is subservient to the network

A new way of doing science

> Science used to about test tubes, wet labs and big instruments

> But increasingly science is moving to networks and computers

> Science is now longer bound by bricks and mortar or geography

> NSF has announced “Cyber Infrastructure” initiative

– https://worktools.si.umich.edu/workspaces/datkins/001.nsf

> DOE SciDAC “Scientific Discovery through Advanced Computing”

– http://www.er.doe.gov/feature_articles_2001/august/SCIAC/SciDAC_announcement.htm

> Recognition that more and more science is network and computationally based

> Grids using web services will be foundation of this new research methodology

Many e-Research Projects Coming

ATLAS

Sloan Digital Sky Survey

LHC

ALMA

International Grid Testbed

Tier2 Centre ~1 TIPS

Online System

Offline Processor Farm

~20 TIPS

CERN Computer Centre

FermiLab ~4 TIPSFrance Regional Centre

Italy Regional Centre

Germany Regional Centre

InstituteInstituteInstituteInstitute ~0.25TIPS

Physicist workstations

~100 MBytes/sec

~100 MBytes/sec

~622 Mbits/sec

~1 MBytes/sec

Physicists work on analysis “channels”.

Each institute will have ~10 physicists working on one or more channels; data for these channels should be cached by the institute server

Physics data cache

~PBytes/sec

~622 Mbits/sec per channel

Tier2 Centre ~1 TIPS

Tier2 Centre ~1 TIPS

Tier2 Centre ~1 TIPS

Caltech ~1 TIPS

~622 Mbits/sec per channel

Tier 0Tier 0

Tier 1Tier 1

Tier 2Tier 2

Tier 4Tier 4

1 TIPS is approximately 25,000

SpecInt95 equivalents

ATLAS

International Grid Testbed

> First production use of international e2e lightpaths to transfer Forward Calorimeter data from CERN to Canada

> Experiments planned to directly transfer low level trigger data from CERN to Canadian computers

> 10 GbE server to server experiments> RDMA, TCP/IP offload, etc

Virtual Observatory

> http://www.us-vo.org/ > Discovery process will rely

on advanced visualization and data mining tools

> Not tied to a single brick and mortar location

> Will cross correlate existing multi-spectral databases petabytes in size

> Web services will integrate data and applications

No new telescopes or radio dishes. Just big networks interconnecting large databases

Canada Virtual Observatory Data Flows

> Flow to Terapix : 1TB/month> Flow to science centres: 0.5 TB/month> Return from Terapix: 2 TB/month> Over 5 years users will download full dataset (100

TB) ~ 30 times: 50 TB per month> Average flow 154 Mbps for five years> User network load is the largest, least predictable,

and least manageable component of the network traffic

> Accessible to students at schools

Canadian Forestry Grid

> SAFORAH (System of Agents for Forest Observation Research with Automation Hierarchies).

> SAFORAH connects five locations across the country to support the monitoring of Canada's forests

> Together, all five locations will generate data equaling 40 terabytes (TB) per month

Grids for Kids

> The ultimate goal of Grids for Kids is to allow students and eventually members of the general public to be full participants in scientific discovery and innovation.

> Will allow increasing number of computationally or networked research experiments to be seamlessly integrated with the computer capabilities of thousands of PCs located at our schools

> Some early primitive examples…

FightAIDS@Home

•Scientists at The Scripps Research Institute (TSRI) are using computational methods to identify drugs that have the right shape and interaction characteristics to fight diseases such as AIDS.

•Once such candidates are identified, they can be synthesized in a laboratory, tested according to FDA guidelines, and released as prescription drugs to benefit the public.

•Such computations require a vast number of trial dockings, testing variations in the target protein and the trial drug molecules

Folding@home

> This "virtual supercomputer" uses peer-to-peer technology to make unprecedented amounts of processing power available to medical researchers to accelerate the development of improved treatments and drugs that could potentially cure diseases.

> Rapid new discoveries in cancer research

> Two projects in Canada:– Smallpox cure at UWO– Leukemia research at Mt Sinai

Climate Prediction

> Predict future climate due to greenhouse affect> Distribute climate model to thousands of PCs worldwide> www.climateprediction.com

ALTA Cosmic Ray eScience

> Collaborative scientific research project involving the University of Alberta Center for Subatomic Research and over 50 high schools across Canada in the area of cosmic ray detection.

> Teachers and students actively contribute to the physics research while learning about an exciting area of modern science. 

> Distributed computing at schools required to analyze data from sensors in near real time

Neptune/Venus Grid

> Joint US-Canadian project for undersea dark fiber network off west coast of USA and Canada

> Undersea network will connect instrumentation devices, robotic submarines, sensors, under sea cameras, etc

> All devices available to students and researchers connected to CA*net 4 and Internet 2 networks

Distributed computing and data storage devices on CA*net 4 and Internet 2 will be used to analyze and store data

Faulkes Telescope

> Provide UK schools with access to a research class telescope in Hawaii

> Provides an exciting resource for teachers to use via the Web

> To provide a real-time experience of astronomy, through live use of a telescope

> To allow students to participate in real research programs, mentored by professional astronomers

> Provides other public interest groups, such as amateurs, access to high quality astronomical data

> http://www.faulkes-telescope.com/

More Information

> http://www.canarie.ca/canet4/library/canet4design.html

> http://www.canarie.ca/canet4/obgp/index.html> http://www.canarie.ca/canet4/library/customer.html

> Thanks to the design teams at Carleton U, Ottawa U, CRC, UQAM, UoWaterloo, Montague, etc

Background Slides

RMI

Resource Management Layer

User Access Layer

Grid Application

Create service Access service

Web Server

DB

GT3 Hosting Environment

LPO FactoryService

Grid Service Interface

LPO Delegate Service J2EE Application Server

LPO Service

EJB Remote

LPO Service

EJB Home

JDBC

LPO Service

Implemen-tation

RMI

RMI

LPO Grid Service

User Controlled Service Provisioning Layer

Using Third Wave Service to manage a network

OGSI-conformant services LPO advertisement LPO query LPO termination LPO access LPO reconfiguration LPO spawning LPO concatenation End-to-end LPO establishment

OGSI-conformant services

Resource Agent

RMIService Provisioning Layer

LPO ControllerLPO Controller

RMI

TL1

Request Controller

Switch Interface