The OptIPuter as a Prototype for CalREN-XD

The OptIPuter as a Prototype for CalREN-XD

Briefing to the CalREN-XD Subcommittee

CENIC Board

July 29, 2005

Dr. Larry Smarr

Director, California Institute for Telecommunications and Information Technologies

Harry E. Gruber Professor,

Dept. of Computer Science and Engineering

Jacobs School of Engineering, UCSD

From “Supercomputer–Centric” to “Supernetwork-Centric” Cyberinfrastructure

1.E+00

1.E+01

1.E+02

1.E+03

1.E+04

1.E+05

1.E+06

1985 1990 1995 2000 2005

Ba

nd

wid

th (

Mb

ps

)

Megabit/s

Gigabit/s

Terabit/s

Network Data Source: Timothy Lance, President, NYSERNet

32x10Gb “Lambdas”

1 GFLOP Cray2

60 TFLOP Altix

Bandwidth of NYSERNet Research Network Backbones

T1

Optical WAN Research Bandwidth Has Grown Much Faster Than

Supercomputer Speed!

Co

mp

utin

g S

peed

(G

FL

OP

S)

Challenge: Average Throughput of NASA Data Products to End User is Only < 50 Megabits/s

Tested from GSFC-ICESATJanuary 2005

http://ensight.eos.nasa.gov/Missions/icesat/index.shtml

San Francisco Pittsburgh

Cleveland

National Lambda Rail (NLR) and TeraGrid Provides Researchers a Cyberinfrastructure Backbone

San Diego

Los Angeles

Portland

Seattle

Pensacola

Baton Rouge

HoustonSan Antonio

Las Cruces /El Paso

Phoenix

New York City

Washington, DC

Raleigh

Jacksonville

Dallas

Tulsa

Atlanta

Kansas City

Denver

Ogden/Salt Lake City

Boise

Albuquerque

UC-TeraGridUIC/NW-Starlight

Chicago

International Collaborators

NLR 4 x 10Gb Lambdas Initially Capable of 40 x 10Gb wavelengths at Buildout

NSF’s TeraGrid Has 4 x 10Gb Lambda Backbone

Links Two Dozen State and Regional Optical

Networks

DOE, NSF, & NASA

Using NLR

The OptIPuter Project – A Model of Cyberinfrastructure Partnerships

• NSF Large Information Technology Research Proposal– Calit2 (UCSD, UCI) and UIC Lead Campuses—Larry Smarr PI– Partnering Campuses: USC, SDSU, NW, TA&M, UvA, SARA, NASA

• Industrial Partners– IBM, Sun, Telcordia, Chiaro, Calient, Glimmerglass, Lucent

• $13.5 Million Over Five Years• Linking User’s Linux Clusters to Remote Science ResourcesNIH Biomedical Informatics NSF EarthScope

and ORION

http://ncmir.ucsd.edu/gallery.html

siovizcenter.ucsd.edu/library/gallery/shoot1/index.shtml

Research Network

What is the OptIPuter?

• Optical networking, Internet Protocol, Computer Storage, Processing and Visualization Technologies– Dedicated Light-pipe (One or More 1-10 Gbps WAN Lambdas)– Links Linux Cluster End Points With 1-10 Gbps per Node– Clusters Optimized for Storage, Visualization, and Computing– Does NOT Require TCP Transport Layer Protocol – Exploring Both Intelligent Routers and Passive Switches

• Applications Drivers: – Interactive Collaborative Visualization of Large Remote Data Objects

– Earth and Ocean Sciences– Biomedical Imaging

• The OptIPuter Exploits a New World in Which the Central Architectural Element is Optical Networking, NOT Computers - Creating "SuperNetworks"

OptIPuter Middleware Architecture-- The Challenge of Transforming Grids into LambdaGrids

Distributed Applications/ Web Services

Telescience

GTP XCP UDT

LambdaStreamCEP RBUDP

Vol-a-Tile

SAGE JuxtaView

Visualization

DVC ConfigurationDVC API

DVC Runtime Library

Data Services

LambdaRAM

Globus

XIOPIN/PDC

DVC Services

DVC Core Services

DVC Job Scheduling

DVCCommunication

Resource Identify/Acquire

NamespaceManagement

Security Management

High SpeedCommunication

Storage Services

GRAM GSI RobuStore

Photonic Infrastructure

Source: Andrew Chien, UCSD

UCSD

StarLight Chicago

UIC EVL

NU

CENIC San Diego GigaPOP

CalREN-XD

6

6

The OptIPuter LambdaGrid is Rapidly Expanding

NetherLight Amsterdam

U Amsterdam

SDSU

CICESE

via CUDI

CENIC/Abilene shared network

1 GE Lambda

10 GE Lambda

PNWGP Seattle

CaveWave/NLR

NASA Goddard

NASA Ames

NLRNASA JPL

UCI

CENIC Los Angeles

GigaPOP

22USC/ISI

CineGrid Circuit

Source: Greg Hidley, Aaron Chin, Calit2

UCSD Packet Test BedOptIPuter Year 2 – Ring Configuration

10

1

10ChiaroEnstara

SDSC JSOE

CSE

410

1

9-nodeviz cluster

IBM 9-node viz cluster

SIO

SOM

IBM 48-node storage cluster

IBM 128-node compute cluster

Sun 128-node computecluster

Sun 17-nodecompute storagecluster

CRCA

6th College

3-node viz cluster

1

IBM 9 mpixeldisplay pairs

Geowall 2 Tiled Display

10

Sun 22-node

viz cluster

10

1

To UCI and ISI via CalREN-HPR

DellGeowall

Preuss


Dell VizDell 5224

Dell 5224

Dell 5224

Dell 5224

Dell 5224

Extreme 400

Juniper T320

Cisco 6509

HP 96-nodecompute cluster

HP4-nodecontrol


Infiniband4 nodes

Infiniband64 nodes

Sun 5-node

viz cluster

Sun17-nodecomputecluster

Fujitsu

Promicro 16-node compute cluster

To StarLight via NLR

Extreme 400

Extreme 400Extreme 400


UCSD Packet Test BedOptIPuter Year 3 – Star Configuration

10

10

10

10

1

10 ChiaroEnstara

SDSC JSOE

CSE

10

1

9-nodeviz cluster

IBM 9-node viz cluster

SIO

SOM

IBM 48-node storage cluster

IBM 192-node compute cluster


Sun 17-nodecompute storagecluster

CRCA

6th College

3-node viz cluster

1


Geowall 2 Tiled Display

Sun 22-node

viz cluster

To UCI and ISI via CalREN-HPR

DellGeowall

Preuss


Dell Viz

Dell 5224

Dell 5224

Dell 5224

Extreme 400

Juniper T320

Cisco 6509


HP4-nodecontrol


Infiniband4 nodes

Infiniband64 nodes

Sun 5-node

viz cluster

Sun17-nodecomputecluster

Fujitsu

Promicro 16-node compute cluster

To StarLight via NLR CAVEwave

Extreme 400

Extreme 400


Extreme 400

10

10

Cisco 6509

Dell 5224

UCSD Campus LambdaStore ArchitectureDedicated Lambdas to Labs Creates Campus LambdaGrid

SIO Ocean SupercomputerIBM Storage Cluster

Extreme Switch with 2 Ten Gbps Uplinks

Streaming Microscope

Source: Phil Papadopoulos, SDSC, Calit2

The Calit2@UCSD Building is Designed for Extremely High Bandwidth

1.8 Million Feet of Cat6 Ethernet Cabling

150 Fiber Strands to BuildingExperimental Roof Radio Antenna Farm

Building Radio Transparent Ubiquitous WiFiPhoto: Tim Beach,

Calit2

Over 9,000 Individual

10/100/1000 Mbps

Drops in the Building

Calit2 Partnering with CENIC and Campuses

• OptIPuter– Campus Donated Multiple Single Mode Fiber Pairs Between Major OptIPuter

Labs For Research– Campus Provided Routable IP Space for OptIPuter, Allowing for Easier

Network Expansion– Campus Agreed to House and Monitor the Core Networking Gear While the

New Calit2 Building is Being Built• CalREN-HPR

– Campus Provided Connectivity to the CalREN-HPR– Calit2 Provided Funding to Upgrade CalREN-HPR Access to 10GE– Other UC Campuses are Now Following Suit with UCLA Expected to Upgrade

this Summer• CalREN-XD

– Campus Funded About 50% of the Dedicated 1GE Connections to UC Irvine and the University of Southern California (ISI)

– It was the First XD Deployment for CENIC – Planning Underway for XD to be Extended UC Wide

UCSD OptIPuter Network Discovery

Picture Below Displays ~500 Hosts (Including ~300 Shared)80 Gbps Cisco 6509 backbone in the Core

> 20 Switches Including 7 with 10Gbps Uplinks

Year 3 Plans:Enhance Campus OptIPuter

• A Substantial Portion of the Physical Build Completes in Year 2– Endpoints, Cross-campus Fiber, Commodity Endpoints

• Increase Campus Bandwidth– Work Towards More Extensive 10GigE Integration

– Optiputer HW Budget Limited In Year 3, Focus is on Network Extension

– Connect Two Campus Sites with 32-node Clusters At 10GigE– 3:1 Campus Bisection Ratio

• Add/Expand a Moderate Number of new Campus Endpoints– Add New Endpoints Into The Chiaro Network

– UCSD Sixth College

– JSOE (Engineering) Collaborative Visualization Center

– New Calit2 Research Facility

– Add 3 General-purpose Sun Opteron Clusters at Key Campus Sites (Compute and Storage); Clusters Will All Have PCI-X (100 Mhz, 1Gbps)

– Deploy Infiniband on Our IBM Storage Cluster and on a Previously-Donated Sun 128-node Compute Cluster

• Complete Financial Acquisition of the Chiaro Router

Year Three GoalsIntegrate New NSF Quartzite MRI

• Goal -- integration of Packet-based (SoCal) and Circuit-based (Illinois) Approaches a Hybrid System– Add Additional O-O-O Switching Capabilities Through a Commercial

(Glimmerglass) All-optical Switch and the Lucent (Pre-commercial) Wavelength Selective Switch

– Begin CWDM or DWDM Deployment to Extend Optical Paths Around UCSD and Provide Additional Bandwidth

– Add Additional 10GigE in Switches and Cluster Node NICs

• MRI Proposal (Quartzite, Recommended for Funding) Allows Us to Match the Network to the Number of Existing Endpoints

• This is a New Kind of Distributed Instrument – 300+ Components Distributed Over the Campus– Simple and Centralized Control for Other Optiputer Users

UCSD Quartzite Core Year 3

QuartziteCore

CalREN-HPRResearch

Cloud

Campus ResearchCloud

GigE Switch withDual 10GigE Upliks

.....To cluster nodes





GigE

10GigE

...Toothernodes

Quartzite CommunicationsCore Year 3

ProductionOOO

Switch

Juniper T320

4 GigE4 pair fiber

Wavelength Selective

Switch

To 10GigE clusternode interfaces

..... To 10GigE clusternode interfaces and

other switches

Chiaro Enstara

32 10GigE

UCSD Glimmerglass

SDSC

JSOE

SIO

SOM (NCMIR)Extreme 400

Fujitsu

Extreme 400

10 GigE Physical Connection

Extreme 400

CSE

1

2

3

4

5

6

7

8

Glimmerglass

ChiaroEnstara

0

9

101

102

103

104

105

106

107

108

100

109

Extreme 400

10

111

110

11

ChicagoCaveWave

CISCO6509

76

5

4

3

21

8

Lustre vs. PVFS2 Comparison

•9 servers•8 data, 1 meta-data

•Connected via dedicated GigE network•Iozone tests with multiple clients accessing the same file (size 10/30 GB)•Default setup for PVFS-2 and Lustre•Optimal record size selected for PVFS2 comparison

Data Throughput

0

50

100

150

200

250

300

350

1 2 3 4

Number of Clients

Ag

gre

gat

e T

hro

ug

hp

ut

(MB

/s)

Lustre

PVFS2

Brainywall – Predecessor to the Biowall• Powered by a 5 node cluster

– Four render nodes and one front-end– Each node drives one half of one display

(1920x2400)• Each display accepts 1-4 DVI inputs

– Refresh rate is bound by number of DVI inputs.

– Full resolution of the display at 60hz, exceeds the maximum bandwidth of the DVI specification.

– Each additional DVI connection increases the refresh rate (2xDVI=20.1hz)/display

• 18 million pixels (9Mpixels/display)• Single user station

Electron Microscope Datasets: 2D

• High resolution 2D image acquired from the 4k x4k camera• Displayed on an IBM T221 9 million pixel display, 3840x2400 QUXGAW resolution

GeoWall2: OptIPuter JuxtaView Software for Viewing High Resolution Images on Tiled Displays

This 150 Mpixel Rat Cerebellum Image is a Montage of 43,200 Smaller Images

Source: Mark Ellisman, Jason Leigh -

OptIPuter co-PIs

40 MPixel Display Driven By a 20-Node Sun Opteron Visualization Cluster

Currently Developing OptIPuter Software to Coherently Drive 100 MegaPixel Displays

• 55-Panel Display – 100 Megapixel

• Driven by 30 Dual-Opterons (64-bit)

• 60 TB Disk

• 30 10GE interfaces– 1/3 Tera bit/sec!

• Linked to OptIPuter

• We are Working with NASA ARC Hyperwall Team to Unify Software

Source: Jason Leigh, Tom DeFanti, EVL@UICOptIPuter Co-PIs

iCluster – ANFwall (Array Network Facility)

Source: Mark Ellisman, Jason Leigh -

OptIPuter co-PIs

16 MPixel Display (30” Apple Cinema) Driven by a 3-Node Dual G5 Visualization Cluster

High Resolution Portals to Global Science Data --200 Million Pixels of Viewing Real Estate!

Calit2@UCI Apple Tiled Display WallDriven by 25 Dual-Processor G5s

50 Apple 30” Cinema Displays

Source: Falko Kuester, Calit2@UCINSF Infrastructure Grant

Data—One Foot Resolution USGS Images of La Jolla, CA

LambdaRAM: Clustered Memory To ProvideLow Latency Access To Large Remote Data Sets

• Giant Pool of Cluster Memory Provides Low-Latency Access to Large Remote Data Sets – Data Is Prefetched Dynamically– LambdaStream Protocol Integrated into

JuxtaView Montage Viewer

• 3 Gbps Experiments from Chicago to Amsterdam to UIC – LambdaRAM Accessed Data From

Amsterdam Faster Than From Local Disk

all

8-14

none

all

8-14

1-7

Displayed region

Visualization of the Pre-Fetch Algorithm

none

Data on Disk in Amsterdam

Local Wall

Source: David Lee, Jason Leigh

Multiple HD Streams Over Lambdas Will Radically Transform Network Collaboration

U. Washington

JGN II WorkshopOsaka, Japan

Jan 2005

Prof. OsakaProf. Aoyama

Prof. Smarr

Source: U Washington Research Channel

Telepresence Using Uncompressed 1.5 Gbps HDTV Streaming Over IP on Fiber

Optics

Establishing TelePresence Between AIST (Japan) and KISTI (Korea) and PRAGMA in Calit2@UCSD Building in 2006

Two New Calit2 Buildings Will Provide a Persistent Collaboration “Living Laboratory”

• Over 1000 Researchers in Two Buildings– Linked via Dedicated Optical Networks– International Conferences and Testbeds

• New Laboratory Facilities– Virtual Reality, Digital Cinema, HDTV– Nanotech, BioMEMS, Chips, Radio, Photonics

Bioengineering

UC San Diego

UC Irvine

Calit2 Collaboration Rooms Testbed UCI to UCSD

In 2005 Calit2 will Link Its Two Buildings

via CENIC-XD Dedicated Fiber over 75 Miles Using OptIPuter Architecture to Create a

Distributed Collaboration Laboratory

UC Irvine UC San Diego

UCI VizClass

UCSD NCMIR

Source: Falko Kuester, UCI & Mark Ellisman, UCSD

SDSC/Calit2 Synthesis CenterWill Be Moving from SDSC to Calit2 Building

Collaboration to RunExperiments

Collaboration to Set Up Experiments

Collaboration to StudyExperimental Results

Cyberinfrastructure for the

Geosciences

www.geongrid.org

Southern California CalREN-XD Build Out

1GigE

WDM

ONS15808CENIC

at UCSD

ONS15540CENICat USC

ONS15540CENICat UCI

CiscoUCI

FoundryISI

OptIPuterLambdaGrid

FoundryUSC

10GigE

ONS15808CENIC

at Tustin

ONS15808CENICat LA

ONS15540CENIC

at TustinONS15808

CENICat UCSD

ONS15540CENICat LA

ONS15530CENIC

at UCSD

ONS15530CENICat LA

ONS15530CENIC

at Tustin

ChiaroUCSD

10GE

10GE

2 1GE

2 1GE

2 1GE

2 1GE CENIC proposed solution for UCSD to UCI, UCSD to ISI,

UCSD to StarLight and UCSD to NASA Goddard

2 1GE transponders for 15540s10GE transponders for 15808sONS15530s at UCSD and LA

1GE to StarLight1GE to NASA Goddard

Funded May 2004 by UCSD and OptIPuter

UC Irvine

1

EGWbuilding

1

Charlie ZenderESMF cluster

Padhraic Smythcluster

Falko Kuestertest cluster

Hyperwall 55-node

viz cluster

Calit2building

Hyperwall

2To UCSD via CalREN-XD

Cisco 3550-24Cisco 3508

Steve Jenkscluster

1 GigE

Cisco 15540

Cisco 3550-24

Cisco 3550-24

Applying OptIPuter Technologies to Support Global Change Research

• UCI Earth System Science Modeling Facility (ESMF)– Calit2 is Adding ESMF to the OptIPuter Testbed

• ESMF Challenge:– Improve Distributed Data Reduction and Analysis– Extending the NCO netCDF Operators

– Exploit MPI-Grid and OPeNDAP– Link IBM Computing Facility at UCI over OptIPuter to:

– Remote Storage

– at UCSD– Earth System Grid (LBNL, NCAR, ONRL) over NLR

• The Resulting Scientific Data Operator LambdaGrid Toolkit will Support the Next Intergovernmental Panel on Climate Change (IPCC) Assessment Report

Source: Charlie Zender, UCI

Variations of the Earth Surface TemperatureOver One Thousand Years

Source: Charlie Zender, UCI

NLR CAVEwave

WDM

CiscoPNWGP

at Seattle

OptIPuterUCSD, SoCal

10GigE

Extreme S400SDSC

at San Diego

Cisco 15808NLR/CENIC

at Los Angles

ONS15808NLR/CENIC

at San Diego

Force10StarLight

at Chicago

Cisco 15808NLR/CENIC

at Sunnyvale

Cisco 15808NLR

at Portland

Cisco 15808NLR

at Seattle

Cisco 15808NLR

at Seattle

Cisco 15808NLR

at Boise

Cisco 15808NLR

at Ogden

Cisco 15808NLR

at Kansas City

Cisco 15808NLR

at Denver

Cisco 15808NLR

at Chicago

Cisco 15454NLR

at Chicago

OptIPuterUCI, NU,

Amsterdam

EVL

10GE OptIPuter CAVEWAVEHelped Launch the National LambdaRail

Next Step: Coupling NASA Centers

to NSF OptIPuter Source: Tom DeFanti, OptIPuter co-PI

SL Force10

SL OPSW

NL

CA*net4

UKLight

JGN ii

OC-48

10GE MREN

Abilene

2xOC-192to Amsterdam

4xOC-192to Canada, Seattle,

Korea, Taiwan, NYC, Ireland, Amsterdam

OC-192 to London

OC-192 to Tokyo

16x10GE96x1GE

16-processor cluster

ESnet, NREN,NISN, DREN, etc.

GE ElectronicallySwitched

GE OpticallySwitched

10GE ElectronicallySwitched/Routed

128x128Calient

Optical Switch

6

4

Clusters at EVL, LAC

N

N

StarLightGLIF

ExchangeJan. 2005

10GEInt’l 2

TeraGrid Juniper T640

Nx10GENxOC-192

To NCSA/SDSCANL/ETF

10GE

SL Cisco 6509

3x10GE80x1GE

10GE

MidWestMREN

N

GLORIADGE to Russia and

China

10GE

CERNOC-192 to

CERN

2Fermilab DWDM

10GE soon

1

CalTech Juniper T320

10GEN

OC-192

10GE NCSA

1x10GE1xOC-192

10GE

Calit2

CAVEwave/NLR

10GE UIC

10GEOC-192

N

8x1GE UIC

Clusters at NCSA

Clusters at StarLight/NU

Nx10GE

Nx1GE

The International Lambda Fabric Being Assembled to Support iGrid Experiments

Source: Tom DeFanti, UIC & Calit2

September 26-30, 2005Calit2 @ University of California, San Diego

California Institute for Telecommunications and Information Technology

The Networking Double Header of the Century Will Be Driven by LambdaGrid Applications

iGrid

2oo5T H E G L O B A L L A M B D A I N T E G R A T E D F A C I L I T Y

Maxine Brown, Tom DeFanti, Co-Organizers

www.startap.net/igrid2005/

http://sc05.supercomp.org

LOOKING: (Laboratory for the Ocean Observatory

Knowledge Integration Grid)

Adding Web and Grid Services to Lambdas to Provide Real Time Control of Ocean Observatories

• Goal: – Prototype Cyberinfrastructure for NSF’s Ocean

Research Interactive Observatory Networks (ORION) Building on OptIPuter

• LOOKING NSF ITR with PIs:– John Orcutt & Larry Smarr - UCSD– John Delaney & Ed Lazowska –UW– Mark Abbott – OSU

• Collaborators at:– MBARI, WHOI, NCSA, UIC, CalPoly, UVic,

CANARIE, Microsoft, NEPTUNE-Canarie

www.neptune.washington.edu

http://lookingtosea.ucsd.edu/

Goal – From Expedition to Cable Observatories with Streaming Stereo HDTV Robotic Cameras

Scenes from The Aliens of the Deep, Directed by James Cameron &

Steven Quale

http://disney.go.com/disneypictures/aliensofthedeep/alienseduguide.pdf

Proposed UW/Calit2 Experiment for iGrid 2005 –Remote Interactive HD Imaging of Deep Sea Vent

Source John Delaney & Deborah Kelley, UWash

To Starlight, TRECC,

and ACCESS

Canadian-U.S. Collaboration

Monterey Bay Aquarium Research Institute (MBARI) Cable Observatory Testbed – LOOKING Living Lab

Tele-Operated Crawlers

Central Lander

Monterey Accelerated Research System

(MARS) Installation Oct 2005 -Jan 2006

Source: Jim

Bellingham, MBARI