Hybrid Packet-Optical Infrastructure

Tom DeFanti, Maxine Brown, Joe Mambretti & Linda Winkler

The Hard Problems

• New protocols needed: the Internet is not designed for single large-scale users

• Circuits are not scalable, but neither are router$• All intelligence has to be on the edge• Tuning compute, data, visualization, networking

using clusters to get even a simple order of magnitude improvement is non-trivial

• Security at 10Gb line speed

Knowing the User’s Bandwidth Requirements

DSL GigE LAN

C

A

B

A -> Need full Internet routing

B -> Need VPN services on/and full Internet routing

C -> Need very fat pipes, limited multiple Virtual Organizations

Source: Cees de Laat, UvA

Number of users

Bandwidth consumed

We Can Now Build Lambda Grids!

• Importance for Applications– To create high-performance trials of new technologies

that support application-dictated software toolkits, middleware, computing and networking

– To provide known and knowable characteristics with deterministic and repeatable behavior on a persistent basis, while encouraging experimentation with innovative concepts

– It isn’t science if you can’t repeat it!

Performance Monitoring & Control

IIIIIII

IIIIIII

IIIIIII

IIIIIII

IIIIIII

IIIIIIIIIIIIII

IIIIIII

IIIIIII

OMNInet Testbed Experiments - MEMS-Based Dynamic Lambda Switching (MREN Used as Out-of-Band Control Channel)

Metro Lambda Grid (I-WIRE and OMNInet – An Advanced Photonic Metro Testbed, Joint Project with iCAIR et al)

UIC

Chicago

Illinois I-WIRE State US$7.5M Lambda Grid

Source: Charlie Catlett, ANL

Illinois’ I-WIRE:Distributed Cluster Computing

Research Areas•Displays/VR•Collaboration•Rendering•Applications•Data Mining

NCSA

Argonne UIC/EVL

Research Areas• Latency-Tolerant

Algorithms• Interaction of

SAN/LAN/WAN technologies

• Clusters

UIUC

CS

StarLight

Source: Charlie Catlett

TeraGrid @ StarLight

TeraGrid, an NSF-funded Major Research Equipment initiative, has its Illinois hub located at StarLight.

20 TF Linux TeraGrid

32

32

5

32

32

5

Router or Switch/Router

32 quad-processor McKinley Servers(128p @ 4GF, 8GB memory/server)

Fibre Channel Switch

HPSS

HPSS

ESnetHSCCMREN/AbileneStarlight

10 GbE

16 quad-processor McKinley Servers(64p @ 4GF, 8GB memory/server)

NCSA500 Nodes

8 TF, 4 TB Memory240 TB disk

SDSC256 Nodes

4.1 TF, 2 TB Memory225 TB disk

Caltech32 Nodes

0.5 TF 0.4 TB Memory

86 TB disk

Argonne64 Nodes

1 TF0.25 TB Memory

25 TB disk

IA-32 nodes

4

Juniper M160

OC-12

OC-48

OC-12

574p IA-32 Chiba City

128p Origin

HR Display & VR Facilities

= 32x 1GbE

= 64x Myrinet

= 32x FibreChannel

Myrinet Clos SpineMyrinet Clos Spine Myrinet Clos SpineMyrinet Clos Spine

= 8x FibreChannel

OC-12

OC-12

OC-3

vBNSAbileneMREN

Juniper M40

1176p IBM SPBlue Horizon

OC-48

NTON

32

24

8

32

24

8

4

4

Sun E10K

4

1500p Origin

UniTree

1024p IA-32 320p IA-64

2

14

8

Juniper M40vBNS

AbileneCalrenESnet

OC-12

OC-12

OC-12

OC-3

8

SunStarcat

16

GbE

= 32x Myrinet

HPSS

256p HP X-Class

128p HP V2500

92p IA-32

24Extreme

Black Diamond

32 quad-processor McKinley Servers(128p @ 4GF, 12GB memory/server)

OC-12 ATM

Calren

2 2

Source: Rick Stevens 12/2001

The US National Lambda Rail The Cost of a Couple of University Buildings

Source: John Silvester, Dave Reese, Tom West, CENIC

USAWaves Over AT&T’s Next Generation Network

CA*net 4 Physical Architecture

VancouverCalgary

Regina Winnipeg

Ottawa

Montreal

Toronto

Halifax

St. John’s

Fredericton

Charlottetown

Chicago

Seattle

New YorkLos Angeles

Miami

Europe

Dedicated Wavelength or

SONET channel

OBGP switches

Optional Layer 3 aggregation service

Large channel WDM system

SURFnet5

• Partners BT and Cisco

• 15 PoPs connected by thirty 10 Gbit/s lambdas

• Dual stack IPv4 and IPv6

• 500,000 users

• 84 institutes connected at Gbit/s level

Source: Kees Neggars

StarLight in Chicago:A 1GigE and 10GigE Exchange

Abbott Hall, Northwestern University’sChicago downtown campus

Operational since summer 2001, StarLight is a 1GigE and 10GigE switch/router facility for high-performance access to participating networks.

StarLight is equipped for optical switching facility for wavelengths.

www.startap.net/starlight/NETWORKS/

StarLight US and International Networks as of August 2003

• Abilene 10Gb • ESnet (DOE) • DREN (DOD)• NREN (NASA) • AMPATH (South America)• CA*net4 (Canada) • SURFnet (Netherlands)• CERN/DataTAG• TransPAC/APAN (Asia)• NaukaNET (Russia)• ASnet (Taiwan)• Others via STAR TAP OC-12 and Abilene transit• See http://loadrunner.uits.iu.edu/mrtg-monitors/starlight/ for

statistics on usage

StarLight is

• StarLight is a Gigabit Ethernet and 10 Gigabit Ethernet exchange for R&E Production Networks (Force 10)

• And a GigE lambda exchange for US, Canada, Europe, Asia and South America for Experimental Networks

• And 1&10Gb MEMS-switched Research Network hub• And the Chicago host to the NSF DTFnet, a 4x10Gb

Network for the TeraGrid and DTF/ETF links to Abilene; NLR, USAWaves, others coming.

• A colo space: 66 racks for networking and computing, data management and visualization support equipment

• Using fiber and circuits from SBC, Qwest, AT&T, Global Crossing, T-Systems, Looking Glass, RCN, and I-WIRE

CANARIE2xGigE circuitsSURFnet 2xGigE

circuits

NetherLightStarLight

NLNLSURFnet

CERN

UKUKSuperJANET4

AbileneAbilene

ESNETESNET

MRENMREN

ITITGARR-B

GEANT

NewYork

STAR-TAP

STAR-LIGHT

DataTAG project

Major 2.5 Gbps circuits between Europe & USA

FRFRINRIA ATRIUM/

VTHD

ATRIUM/VTHD

What is TransLight?

• TransLight is a global-scale experimental networking initiative to support prototypes of the most aggressive e-science applications.

• TransLight consists of dozens of provisioned Gigabit Ethernet (GigE) circuits among North America, Europe and Asia via StarLight in Chicago, NetherLight in Amsterdam.

• Some 10Gb circuits are also available and schedulable.

European lambdas to US–8 GigEs Amsterdam—Chicago–8 GigEs London—Chicago

Canadian lambdas to US–8 GigEs Chicago-Canada-NYC–8 GigEs Chicago-Canada-Seattle

US lambdas to Europe–4 GigEs Chicago—Amsterdam–3 GigEs Chicago—CERN

European lambdas–8 GigEs Amsterdam—CERN –2 GigEs Prague—Amsterdam–2 GigEs Stockholm—Amsterdam–8 GigEs London—Amsterdam

TransPAC lambda (yellow)–1 GigE Chicago—Tokyo

IEEAF lambdas (blue)–8 GigEs NYC—Amsterdam–8 GigEs Seattle—Tokyo

TransLight Lambdas

TransLight Optical/Electronic Switches at StarLight and NetherLight

16-processor cluster

Data plane Data plane

N E T H E R L I G H T

8-processor cluster

Router

8 GigE16 GigE

8 GigE16 GigE

Control plane

15454 at StarLight

10 GigE

2 GigE

128x128Calient MEMSOptical Switch

N-processor cluster

10 GigE

N GigE

15454 at NetherLight

Control plane

2 GigE

2xOC-192

N GigE

64x64Calient MEMSOptical Switch

Router

An OptIPuter Prototype

What is the Global Lambda Integrated Facility (GLIF)?

• GLIF is a global-scale experimental facility being designed to support advanced applications and to develop new technologies.

• The GLIF is a multi-organizational international partnership

• The GLIF will be based on leading edge optical technologies

• Last Meeting – August, Iceland after NORDUnet conference, 3rd year of Global Lamda Workshops

Bring Us Your Lambdas!

• Please bring your lambdas to StarLight, NetherLight, CERN, UKLight, CzechLight, NorthernLight, …

• Build a hub like StarLight• Propose an application or

network experiment• See

www.startap.net/starlight and www.startap.net/translight

Thank You!

• StarLight planning, research, collaborations, and outreach efforts are made possible, in major part, by funding from: – National Science Foundation (NSF) awards ANI-9980480, ANI-9730202, EIA-

9802090, EIA-9871058, ANI-0225642, and EIA-0115809

– NSF Partnerships for Advanced Computational Infrastructure (PACI) cooperative agreement ACI-9619019 to NCSA

– State of Illinois I-WIRE Program, and major UIC cost sharing

– Northwestern University for providing space, engineering and management

• NSF/CISE/ANIR and DoE/Argonne National Laboratory for StarLight and I-WIRE network engineering and design

• NSF/CISE/ACIR and NCSA/SDSC for DTF/TeraGrid/ETF opportunities

• UCAID/Abilene for Internet2 and ITN/GTRN transit; IU for the GlobalNOC

• CA*net4 for North American transport

• Bill St. Arnaud of CANARIE, Kees Neggers of SURFnet, Olivier Martin of CERN and Harvey Newman of CalTech for networking leadership

• Larry Smarr of Cal-(IT)2 for I-WIRE and OptIPuter leadership