The OptIPuter –From SuperComputers to SuperNetworks

Vanguard NextGens Conference

Coronado, CA

November 19, 2002

Dr. Larry Smarr

Director, California Institute for Telecommunications and Information Technologies

Professor, Dept. of Computer Science and Engineering

Jacobs School of Engineering, UCSD

California Has Initiated Four New Institutes for Science and Innovation

UCSBUCLA

California NanoSystems Institute

UCSF UCB

California Institute for Bioengineering, Biotechnology,

and Quantitative Biomedical Research

UCI

UCSD

California Institute for Telecommunications and Information Technology

Center for Information Technology Research

in the Interest of Society

UCSC

UCDUCM

www.ucop.edu/california-institutes

Cal-(IT)2

An Integrated Approach to the Future Internet

www.calit2.net

220 UC San Diego & UC Irvine FacultyWorking in Multidisciplinary Teams

With Students, Industry, and the Community

The State’s $100 M Creates Unique Buildings, Equipment, and Laboratories

Over Fifty Industrial Sponsors From a Broad Range of Industries

Akamai Technologies Inc.AMCCAmpersand VenturesArch VenturesAvalon VenturesThe Boeing CompanyBroadcom CorporationCAIMIS, Inc.Chiaro NetworksConexant Systems, Inc.Connexion by BoeingCox CommunicationsDiamondhead VenturesDupontEmulex Corporation

Network SystemsEnosys MarketsEnterprise PartnersEntropia, Inc.Ericsson ESRIExtreme NetworksGlobal Photon SystemsGravitonIBM

ComputersCommunications

SoftwareSensors

BiomedicalStartups

Venture Capital

Newport CorporationOracleOrincon IndustriesPanoram TechnologiesPrintronixQUALCOMMQuantumThe R.W. Johnson Pharmaceutical Research InstituteSAICSamueli, Henry (Broadcom)SciFrame, Inc.Seagate Storage ProductsSGISilicon WaveSonySTMicroelectronics, Inc.Sun MicrosystemsTeraBurst NetworksTexas InstrumentsTime DomainUCSD HealthcareWebEx

IdeaEdge VenturesThe Irvine CompanyIntersil CorporationIrvine Sensors CorporationJMI, Inc.Leap Wireless InternationalLink, William J. (Versant Ventures)Litton Industries, Inc.MedExpert InternationalMerckMicrosoft CorporationMission VenturesNCR

$140 Million in Industrial Matching

Closing in on the Dream

“Using satellite technology…to demowhat It might be like to have high-speed fiber-optic links between advanced computers in two different geographic locations.”

― Al Gore, SenatorChair, US Senate Subcommittee on Science, Technology and Space

“What we really have to do is eliminate distance between individuals who want to interact with other people and with other computers.”

― Larry Smarr, DirectorNational Center for Supercomputing Applications, UIUC

SIGGRAPH 89Science by Satellite

Source: Maxine Brown, EVL, UIC

Boston

Illinois

The Move to Data-Intensive Science & Engineering-e-Science Community Resources

ATLAS

Sloan Digital Sky Survey

LHC

ALMA

Four Large Hadron Collider Experiments: The Petabyte to Exabyte Challenge

ATLAS, CMS, ALICE, LHCBHiggs + New particles; Quark-Gluon Plasma; CP Violation

Data stored ~40,000 Terabytes/Year and UPData stored ~40,000 Terabytes/Year and UP

Source: Harvey Newman, Caltech

NIH is Funding a Brain Imaging Federated Repository

National Partnership for Advanced Computational Infrastructure

Part of the UCSD CRBS Center for Research on Biological Structure

Biomedical Informatics Research Network

(BIRN)NIH Plans to Expand

to Other Organs and Many Laboratories

NSF’s EarthScopeRollout Over 14 Years Starting

With Existing Broadband Stations

“The Grid” is the EmergingDistributed Cyberinfrastructure Middleware

Applications and Capabilities

Toolkits/Workbenches and PortalsDistributed Computing

Collaboration Visualization

Data Mining

Instruments /

Sensors

Problem Solving

Environments

GRID Services (Middleware)Authenticatio

n / Authorization

Resource Discovery

Scheduling /

Allocation

Fault Detection / Recovery

Event Services

Information Infrastructur

e

GRID Fabric (Resources)QoS /

Diffserv

Directory Services

Information Mgmt / Storage

Public Key Infrastructure

Site accounting / scheduling

Operating Systems

Open Grid Services Architecture (OGSA)Integrates Globus with Web Services

Grid Computing is Becoming Mainstream

Why Optical Networks Are Emerging as the 21st Century Driver for the Grid

Scientific American, January 2001

The Crossing Exponentials Requires a Rethinking of Grid Architecture

“A global economy designed to waste transistors, power, and silicon area

-and conserve bandwidth above all- is breaking apart and reorganizing itself

to waste bandwidth and conserve power, silicon area, and transistors."

George Gilder Telecosm (2000)

The Rapid Increase in Bandwidth is Driven by Parallel Lambdas On Single Optical Fibers

fc *

(WDM)

Parallel Lambdas Will Drive This DecadeThe Way Parallel Processors Drove the 1990s

CONTROL

PLANE

Clusters

DynamicallyAllocatedLightpaths

Switch Fabrics

PhysicalMonitoring

Apps Middleware

A LambdaGrid Will Be the Backbone for an e-Science Network

Source: Joe Mambretti, NU

The Next S-Curves of NetworkingExponential Technology Growth

0%

100%

Research

Experimental/Early Adopters

Production/Mass Market

Time

Technology S-Curve

Gigabit Testbeds

Connections Program

Internet2 Abilene

DWDM

Experimental Networks

Lambda Grids

~1990s 2000 2010

Networking Technology S-Curves

Tec

hn

olog

y P

enet

rati

on

Data Intensive Scientific Applications Require Experimental Optical Networks

• Large Data Challenges in Neuro and Earth Sciences– Each Data Object is 3D and Gigabytes– Data are Generated and Stored in Distributed Archives– Research is Carried Out on Federated Repository

• Requirements– Computing Requirements PC Clusters– Communications Dedicated Lambdas Over Fiber– Data Large Peer-to-Peer Lambda Attached Storage – Visualization Collaborative Volume Algorithms

• Response– OptIPuter Research Project

UIUC/NCSA

Starlight(NU-Chicago)Argonne

UChicagoIIT

UIC

Illinois Century NetworkJames R. Thompson CtrCity HallState of IL Bldg

4 pair

12 pair

4 pair

2 pair 2 pair

4 pair

18 pair

4 10 pair

12 pair

2 pair

Level(3)111 N. Canal

McLeodUSA151/155 N. MichiganDoral Plaza

Qwest455 N. Cityfront

UC Gleacher450 N. Cityfront

Illinois’ I-WIREThe First State Dark Fiber Experimental Network

Source: Charlie Catlett 12/2001

• Fifteen Countries/Locations Proposing 28 Demonstrations: Canada, CERN, France, Germany, Greece, Italy, Japan, The Netherlands, Singapore, Spain, Sweden, Taiwan, United Kingdom, United States

• Applications Demonstrated: Art, Bioinformatics, Chemistry, Cosmology, Cultural Heritage, Education, High-Definition Media Streaming, Manufacturing, Medicine, Neuroscience, Physics, Tele-science

• Grid Technologies: Grid Middleware, Data Management/ Replication Grids, Visualization Grids, Computational Grids, Access Grids, Grid Portal

iGrid 2002 September 24-26, 2002, Amsterdam, The Netherlands

www.startap.net/igrid2002UIC

Sponsors: HP, IBM, Cisco, Philips, Level (3), Glimmerglass, etc.

iGrid 2002 Was Sustaining 1-3 Gigabits/s

Total Available Bandwidth Between Chicago and Amsterdam

Was 30 Gigabit/s

The NSF TeraGridA LambdaGrid of Linux SuperClusters

NCSA8 TF

4 TB Memory240 TB disk

Caltech0.5 TF

0.4 TB Memory86 TB disk

Argonne1 TF

0.25 TB Memory25 TB disk

TeraGrid Backbone (40 Gbps)

SDSC4.1 TF

2 TB Memory250 TB disk

This will Become the National Backbone to Support Multiple Large Scale Science and Engineering Projects

DataCompute

VisualizationApplicationsIntel, IBM, QwestMyricom, Sun, Oracle

$53Million from NSF

From SuperComputers to SuperNetworks--Changing the Grid Design Point

• The TeraGrid is Optimized for Computing– 1024 IA-64 Nodes Linux Cluster– Assume 1 GigE per Node = 1 Terabit/s I/O– Grid Optical Connection 4x10Gig Lambdas = 40 Gigabit/s– Optical Connections are Only 4% Bisection Bandwidth

• The OptIPuter is Optimized for Bandwidth– 32 IA-64 Node Linux Cluster– Assume 1 GigE per Processor = 32 gigabit/s I/O– Grid Optical Connection 4x10GigE = 40 Gigabit/s– Optical Connections are Over 100% Bisection Bandwidth

The OptIPuter is an Experimental Network Research Project

• Driven by Large Neuroscience and Earth Science Data• Multiple Lambdas Linking Clusters and Storage

– LambdaGrid Software Stack– Integration with PC Clusters– Interactive Collaborative Volume Visualization– Lambda Peer to Peer Storage With Optimized Storewidth– Enhance Security Mechanisms– Rethink TCP/IP Protocols

• NSF Large Information Technology Research Proposal– UCSD and UIC Lead Campuses—Larry Smarr PI– USC, UCI, SDSU, NW Partnering Campuses– Industrial Partners: IBM, Telcordia/SAIC, Chiaro Networks– $13.5 Million Over Five Years

The OptIPuter is an Experimental Network Research Project

• Driven by Large Neuroscience and Earth Science Data• Multiple Lambdas Linking Clusters and Storage

– LambdaGrid Software Stack– Integration with PC Clusters– Interactive Collaborative Volume Visualization– Lambda Peer to Peer Storage With Optimized Storewidth– Enhance Security Mechanisms– Rethink TCP/IP Protocols

• NSF Large Information Technology Research Proposal– UCSD and UIC Lead Campuses—Larry Smarr PI– USC, UCI, SDSU, NW Partnering Campuses– Industrial Partners: IBM, Telcordia/SAIC, Chiaro Networks– $13.5 Million Over Five Years

Metro Optically Linked Visualization Wallswith Industrial Partners Set Stage for Federal Grant

• Driven by SensorNets Data– Real Time Seismic– Environmental Monitoring – Distributed Collaboration– Emergency Response

• Linked UCSD and SDSU– Dedication March 4, 2002

Linking Control Rooms

Cox, Panoram,SAIC, SGI, IBM,

TeraBurst NetworksSD Telecom Council

UCSD SDSU44 Miles of Cox Fiber

OptIPuter NSF Proposal Partnered with National Experts and Infrastructure

Vancouver

Seattle

Portland

San Francisco

Los Angeles

San Diego(SDSC)

NCSA

SURFnet CERNCA*net4

AsiaPacific

AsiaPacific

AMPATH

PSC

Atlanta

CA*net4

Source: Tom DeFanti and Maxine Brown, UIC

NYC

TeraGrid DTFnet

CENIC

Pacific LightRail

Chicago

UICNU

USC

UCSD, SDSUUCI

switch switch

switchswitch

• Cluster – Disk

• Disk – Disk

• Viz – Disk

• DB – Cluster

• Cluster – Cluster

Medical Imaging and Microscopy

Chemistry, Engineering, Arts

San Diego Supercomputer Center

Scripps Institution of Oceanography

ChiaroEnstara

OptIPuter LambdaGridEnabled by Chiaro Networking Router

www.calit2.net/news/2002/11-18-chiaro.html

½ Mile

The UCSD OptIPuter Deployment

SIO

SDSC

CRCA

Phys. Sci -Keck

SOM

JSOE Preuss

6th College

Phase I, Fall 02

Phase II, 2003

SDSCAnnex

To Other OptIPuter Sites

Collocation point

Node M

The UCSD OptIPuter LambdaGrid Testbed

Earth Sciences

SDSC

Arts

Chemistry

Medicine

Engineering

High School

UndergradCollege

Phase I, Fall 02

Phase II, 2003

SDSCAnnex

To Other OptIPuter Sites

Collocation point

Collocation

Fast polygon and volume rendering with stereographics

GeoWall

Earth Science

GeoFusion GeoMatrix Toolkit

Underground Earth Science

Rob Mellors and Eric Frost, SDSUSDSC Volume Explorer

Dave Nadeau, SDSC, BIRNSDSC Volume Explorer

NeuroscienceAnatomy

Visible Human ProjectNLM, Brooks AFB,

SDSC Volume Explorer

3D APPLICATIONS:

+

=

OptIPuter Transforms Individual Laboratory Visualization, Computation, & Analysis Facilities

The Preuss School UCSD OptIPuter Facility