FY2003 Annual Progress Report and FY2004 Program Plan

The OptIPuter FY2003 Annual Progress Report and FY2004 Program Plan 1

FY2003 Annual Progress Report and FY2004 Program Plan

NSF ITR Cooperative Agreement ANI-0225642

October 1, 2002 – September 30, 2003

Submitted September 1, 2003

Larry Smarr, Principal Investigator California Institute for Telecommunications and Information Technology [Cal-(IT)2]

University of California, San Diego [email protected] www.optiputer.net


Table of Contents

1. Participants 4 1.A. Primary Personnel 4 1.B. Other Senior Personnel 4 1.C. Other Partner Organizations 5 1.D. Other Collaborators and Contacts 5

2. Activities and Findings 6 2.A. Research Activities 6

2.A.1. Year 1 Milestones 6 2.A.2. Network and Hardware Infrastructure Activities 6 2.A.3. Software Architecture Activities 11 2.A.4. Data, Visualization and Collaboration Activities 14 2.A.5. Applications and Education Activities 17 2.A.6. Meetings, Presentations, Conference Participation 18

2.B. Research Findings 24 2.B.1. Network and Hardware Infrastructure Findings 24 2.B.2. Software Architecture Findings 24 2.B.3. Data, Visualization and Collaboration Findings 25 2.B.4. Applications and Education Findings 26

2.C. Research Training 27 2.D. Education/Outreach 27

3. Publications and Products 28 3.A. Journals/Papers 28 3.B. Books/Publications 30 3.C. Internet Dissemination 30 3.D. Other Specific Products 30

4. Contributions 31 4.A. Contributions within Discipline 31 4.B. Contributions to Other Disciplines 31 4.C. Contributions to Education and Human Resources 31 4.D. Contributions to Resources for Science and Technology 31 4.E. Contributions Beyond Science and Engineering 31

5. Special Requirements 32 5.A. Objectives and Scope 32 5.B. Special Reporting Requirements 32 5.C. Unobligated Funds 32 5.D. Animals, Biohazards, Human Subjects 32

6. OptIPuter FY2004 Program Plan (October 1, 2003-September 30, 2004) 33 6.A. Requested Modifications to Cooperative Agreement 33 6.B. Year 2 Milestones 33 6.C. Network and Hardware Infrastructure Activities 33 6.D. Software Architecture Activities 34 6.E. Data, Visualization and Collaboration Activities 35 6.F. Applications and Education Activities 36

7. OptIPuter FY2003 Expenses (Year 1) 39 7.A. FY2003 Expense Justification 39

7.A.1. Introduction 39 7.A.2. UCSD FY2003 Expense Justification 39 7.A.3. NU FY2003 Expense Justification 40 7.A.4. SDSU FY2003 Expense Justification 40 7.A.5. UCI FY2003 Expense Justification 40 7.A.6. UIC FY2003 Expense Justification 40 7.A.7. USC FY2003 Expense Justification 40

7.B. UCSD FY2003 Expenses 41


7.C. NU FY2003 Expenses 41 7.D. SDSU FY2003 Expenses 41 7.E. UCI FY2003 Expenses 41 7.F. UIC FY2003 Expenses 41 7.G. USC FY2003 Expenses 41

8. OptIPuter FY2004 Budgets (Year 2) 42 8.A. FY2004 Budget Justification 42

8.A.1. Introduction 42 8.A.2. UCSD FY2004 Budget Justification 42 8.A.3. NU FY2004 Budget Justification 42 8.A.4. SDSU FY2004 Budget Justification 42 8.A.5. TAMU FY2004 Budget Justification 42 8.A.6. UCI FY2004 Budget Justification 43 8.A.7. UIC FY2004 Budget Justification 43 8.A.8. USC FY2004 Budget Justification 43

8.B. UCSD FY2004 Budget 44 8.C. NU FY2004 Budget 44 8.D. SDSU FY2004 Budget 44 8.E. TAMU FY2004 Budget 44 8.F. UCI FY2004 Budget 44 8.G. UIC FY2004 Budget 44 8.H. USC FY2004 Budget 44

9. Cumulative Budgets 45 9.A. OptIPuter Expenditures Cumulative Summary 45 9.B. UCSD OptIPuter Expenditures Cumulative 45 9.C. NU OptIPuter Expenditures Cumulative 45 9.D. SDSU OptIPuter Expenditures Cumulative 45 9.E. UCI OptIPuter Expenditures Cumulative 45 9.F. UIC OptIPuter Expenditures Cumulative 45 9.G. USC OptIPuter Expenditures Cumulative 45

10. Appendix: “Mod-1” OptIPuter Visualization Cluster Configuration (Draft) 46 11. Appendix: Report on Workshop on OptIPuter Networking/Backplane Architecture 48 12. Appendix: Software Architecture of the OptIPuter System 52


1. Participants

1.A. Primary Personnel Name Project Role(s) >160 Hours/Yr Larry Smarr Principal Investigator Yes Thomas A. DeFanti Co-Principal Investigator Yes Mark Ellisman Co-Principal Investigator Yes Jason Leigh Co-Principal Investigator Yes Philip Papadopoulos Co-Principal Investigator Yes 1.B. Other Senior Personnel Additional people who contributed to project, and received a salary, wage, stipend or other support from this grant:

Northwestern University (NU) Name Project Role(s) >160 Hours/Yr Valerie Taylor* Senior Personnel Yes Joe Mambretti Senior Personnel Yes * This year Valerie Taylor left Northwestern University for Texas A&M University and will continue to work on the OptIPuter grant. San Diego State University (SDSU) Name Project Role(s) >160 Hours/Yr Eric Frost Senior Personnel Yes University of California Irvine (UCI) Name Project Role(s) >160 Hours/Yr Michael Goodrich Senior Personnel Yes Stephen Jenks Senior Personnel Yes Kane Kim Senior Personnel Yes Padhraic Smyth Senior Personnel Yes University of California San Diego (UCSD) Name Project Role(s) >160 Hours/Yr Andrew Chien Senior Personnel Yes Greg Hidley Senior Personnel Yes Sid Karin Senior Personnel Yes John Orcutt Senior Personnel Yes Rozeanne Steckler Senior Personnel Yes Michael Bailey Senior Personnel Yes University of Illinois at Chicago (UIC) Name Project Role(s) >160 Hours/Yr Maxine Brown Senior Personnel Yes Robert Grossman Senior Personnel Yes Tom Moher Senior Personnel Yes Oliver Yu Senior Personnel Yes Alan Verlo Senior Personnel Yes University of Southern California (USC) Name Project Role(s) >160 Hours/Yr Joe Bannister Senior Personnel Yes Robert Braden Senior Personnel Yes


Ted Faber Senior Personnel Yes Aaron Falk Senior Personnel Yes Carl Kesselman Senior Personnel Yes Marcus Thiébaux Senior Personnel Yes 1.C. Other Partner Organizations Chiaro Networks <www.chiaro.com> is an OptIPuter industrial partner. Steve Wallach, Vice President of Technology, is a member of the OptIPuter Frontier Advisory Board. Chiaro Networks is developing a unique optical packet switching technology for a scalable, highly reliable system architecture. The OptIPuter project at UCSD is centered on a Chiaro Enstara router. Cisco <www.cisco.com> is collaborating with OptIPuter partner USC to implement XCP on a Cisco router. HP/Compaq <www.hp.com> works closely with the UCSD National Center for Microscopy and Imaging Research (NCMIR) on evaluating its new Sepia technology in the medical imaging graphics cluster used by the OptIPuter project. IBM <www.ibm.com> is an OptIPuter industrial partner. Alan Benner, a senior member of the IBM Systems Architecture and Performance Team within the IBM eServer group, participates in the OptIPuter project and a member of the OptIPuter Frontier Advisory Board. IBM also works closely with the UCSD National Center for Microscopy and Imaging Research (NCMIR) on utilizing the T221 nine megapixel array display technology for interactively visualizing large montage brain microscopy images. In addition, the OptIPuter project acquired a ten-node graphics-intensive cluster, plus an experimental IBM Scalable Graphics Engine, and two more T221s for the earth sciences application work at UCSD Scripps Institution of Oceanography. Sun Microsystems <www.sun.com> is working closely with UCSD to develop an OptIPuter compute cluster. Sun has donated a 128-node compute-intensive cluster for the UCSD OptIPuter testbed. Telcordia Technologies, Inc. <www.telcordia.com> is an OptIPuter industrial partner. George Clapp, a senior member of the Telcordia Applied Research Team and an expert in optical control plane and networking for lambda networks, is the SAIC technical project manager for the OptIPuter and a member of the OptIPuter Frontier Advisory Board. He spends 50% time on the OptIPuter project. University of Amsterdam <www.science.uva.nl/~delaat/> is the OptIPuter’s first international affiliate partner, working with colleagues at UIC to develop an optically-switched OptIPuter node, connecting through StarLight. US Geological Survey (USGS) Earth Resources Observation Systems (EROS) Data Center (EDC) <http://edc.usgs.gov/> archives data from many optical land remote sensing satellite missions and conducts research in applications of this data as well. As an affiliate partner of the OptIPuter project, USGS EDC works with team members on application, technology transfer and outreach activities. Brian Davis and Dan Steinwand are USGS points of contact to the OptIPuter team. 1.D. Other Collaborators and Contacts CENIC <www.cenic.org>, the Corporation for Education Network Initiatives in California, hopes to provide the OptIPuter project team with either CalREN-HPR or National Lambda Rail (NLR) networking, to enable participating universities in southern California to connect to one another, as well as team sites in Chicago. San Diego Telecom Council <www.sdtelecom.org>, a 300-member southern California telecom council, strongly endorses the OptIPuter efforts. Co-founder Franz Birkner is a member of the OptIPuter Frontier Advisory Board.


2. Activities and Findings 2.A. Research Activities The OptIPuter Team’s mission is to enable scientists to explore very large remote data objects in a novel interactive and collaborative fashion, which would be impossible on today’s shared internet. We do this by developing a radical LambdaGrid architecture which shows great promise for enabling a number of this decade’s e-science shared information technology facilities. The research agenda involves the design, development and implementation of the OptIPuter – a tightly-integrated cluster of computational, storage and visualization resources, linked over parallel dedicated optical networks across campus, metro, national, and international scales. The OptIPuter runs over 1-10 Gbps lambdas, with advanced middleware and network management tools and techniques to optimize transmissions so distance-dependent delays are the only major variable.

2.A.1. Year 1 Milestones Our proposed Year 1 Milestone, as described in the OptIPuter Cooperative Agreement, has been met:

Two sites will have operational first (UCSD) and second (UIC) generation OptIPuter systems – module version (“Mod”) 0 and 1, respectively. The hardware and software specifications for a generic OptIPuter node will be defined, and all partner sites will make detailed plans to connect to either the Southern California or the Chicago regional optical network (using CENIC or I-WIRE, respectively).

2.A.2. Network and Hardware Infrastructure Activities Year 1 tasks: UCSD created a “Mod 0” OptIPuter by connecting a few already-capable UCSD campus sites and clusters using a (multi) gigabit backbone. This optical LambdaGrid was integrated with the optically-linked Panoram visualization theaters at UCSD and SDSU. UIC and NU have been focusing on next-generation (“Mod 1”) OptIPuter development, exploring novel networking technologies among a small number (~2) of clustered development endpoints for high-throughput data transfer and intradomain control of lightpaths.

Network Infrastructure

Internet 2Abilene(today)

SDSC90 Nodes

JSOE24 Nodes

Keck Chem39 Nodes

SIO IGPP Viz

BIRN SOM32 Nodes + EM

SDSC Annex16 Nodes

Preuss Viz

SDSU Viz

CRCA Viz

6th College Viz

4 Bonded GigELinks each

Cluster[s]

EVL 16Nodes

EVL16 Nodes

Cisco 6509

Foundry

Juniper

GlimmerGlass ONI DWDM ONI DWDM

Calient

Cisco 6509

Star LightChicagoEVL ChicagoUCSD

NWUChicagoLAC UIC

Chicago

UCI

USC

ISI

U of Amsterdam

Amsterdamn Nodes

Cisco 6509

Calient

12 GigE

LAC 16Nodes

Foundry

FoundryLA POPI Wilshire

ISI1152 Nodes

ISI64 SMP

ISI32 Nodes

ISIxTB

CalREN-HPRNLR?

(tomorrow)

OptIPuter Network

7/28/03 -grh

Chiaro

10GigEGigE (multiple)

4 GigE

GigE

SDSU

Teraburst

Teraburst

OC-48/192Future

San Diego

Los Angeles

ChicagoEurope


The above diagram shows how the various OptIPuter sites are being connected; completion is planned by September 30, 2003. These fiber paths will be upgraded over the next few years as part of the OptIPuter project; upgrades will include both increases in bandwidth and enhanced fiber efficiency due to DWDM deployments.

Southern California OptIPuter Network

• The UCSD Campus-Scale OptIPuter is quite complex, so further described below. The UCSD administration provided additional support for OptIPuter activities, beyond the matching funds described in the original proposal, by repurposing existing dark fiber and purchasing new fiber and networking gear to create a Campus-Scale LambdaGrid.

• The SoCal Metro-Scale OptIPuter was formed by optically linking the UCSD/SIO Visualization Center to the SDSU Visualization Center using Teraburst Networks lambda-management boxes that control the ends of the 44-miles of Cox Communications optical fiber. This integrates with existing optically-linked Panoram visualization facilities at UCSD/SIO and SDSU. A SGI visualization system runs the Panoram multi-megapixel display at SIO. SIO and SDSU earth scientists use this environment to collaboratively explore high-resolution stereo visualizations of complex datasets.

• The SoCal Regional-Scale OptIPuter is formed by connecting two Metro-Scale OptIPuters: the San Diego system with the Metro-Scale system formed by linking USC/ISI and UCI in Los Angeles/Orange County. USC/ISI and UCI are connected via Abilene.

• UCI, starting in August 2003, began receiving its off-campus network connection via CENIC ONI (Optical Network Initiative): two Gigabit links, one via dedicated fiber, and one via a SBC Gigaman connection. The immediate benefits include more off-campus bandwidth (to as much as 2000Mbps up from 622Mbps), a redundant off-campus network path in case one fails, and available dark fiber for high-end research activities, such as the OptIPuter. [In Year 2, we will extend the OptIPuter by building out the UCI Campus-Scale OptIPuter, by connecting to the clusters and visualization facilities in labs run by Steve Jenks and Falko Kuester. With a GigE path to the Chiaro on the UCSD campus, we will have a very nice testbed for UCI OptIPuter-funded faculty (Goodrich, Kim, and Smyth) to work on locally.]

Chicago OptIPuter Network

• The UIC Campus-Scale OptIPuter involves the Electronic Visualization Laboratory (EVL) and the Laboratory for Advanced Computing (LAC) using fiber provided by the UIC Academic Computing and Communications Center (ACCC) worth an estimated $280,000. Nortel Networks, Inc. has provided two Optera DWDM devices worth approximately $1,000,000 to light up the fiber between EVL and LAC with 16 GigE channels. Clusters bought through other NSF grants to EVL and LAC (CISE RI, MRI, RR) will continue to exploit these channels this coming year.

• The Chicago Metro-Scale OptIPuter was formed by optically linking EVL to StarLight by I-WIRE and OMNInet. OMNInet is a 10GigE metro-scale testbed using SBC fiber and Nortel Networks equipment, with access provided to NU as well as UIC. The value of OMNInet is not easily assessable, but exceeds several million dollars were it to be a permanent asset. I-WIRE is a $7,500,000 State of Illinois initiative, of which $300,000 was used to connect EVL to the State of Illinois building, and another $500,000 to get to StarLight. In addition, UIC/ACCC spent $140,000 to build a lateral into I-WIRE fiber with State of Illinois funds. Clusters bought using other NSF grants to UIC and NU are used for OptIPuter experiments at both campuses and StarLight.

• The Midwest Regional-Scale OptIPuter will be constructed with partners at Argonne/University of Chicago Computational Institute via I-WIRE, UIUC/NCSA via I-WIRE, and Indiana University/Purdue via I-Light. The University of Wisconsin-Madison and the University of Michigan also are bringing fiber into Chicago and StarLight.

• The National-Scale OptIPuter will be constructed with lambdas from the National Lambda Rail (NLR) and the CIC, with planning going on this year, and implementation in subsequent years.

• The International-Scale OptIPuter is being tested now, using 15Gb of NSF and Dutch-provided lambdas between StarLight and NetherLight (in Amsterdam), connecting clusters at EVL, LAC, StarLight and the University of Amsterdam. Near future extensions to London and CERN are planned. Optical switches and 10Gb electronic switches have been purchased by other NSF grants to EVL (see diagram).


SL Force10

Abilene T640

SL OPSW

NL 15454

CA*net4 15454

DataTAG/CERN Router

UK 15454/equiv

OC-48GC

10GEMREN

10

Abilene

2xOC-192to Amsterdam

2xOC-192to Canada

OC-192to CERN

OC-192to London

6x10GE24x1GE

2x10GE

4

16-processor cluster

Chicago/Qwest

ESnet, NREN,NISN, DREN, etc.

N

GE ElectronicallySwitched

GE OpticallySwitched

10GE ElectronicallySwitched/Routed

128x128Calient

Optical Switch6T

4

2T

4

Many Clusters

N

N

Draft Plans for StarLight

September 2003 Yellow is Now Installed

Green is ComingSeveral Links in Progress

10GEFed/Int’l

SL M10 OC-12 toSTAR TAP

1

OC-192T

4

TeraGrid T640

Nx10GENxOC-192

To DTF/ETF

10GE

SL 6509

2x10GE80x1GE

10GE

MREN

N

OC-48GC

SURFnetCisco 12008

Chicago3

2GC

2GC

T=T-SystemsGC=Global Crossing

TransPAC M320OC-48to Asia

3

4

NaukaNet RouterOC-3 toRussia

1

Fermilab DWDM

4

The UCSD Optical Network LambdaGrid…UCSD, SDSC and the OptIPuter project have invested jointly in the last six months to greatly improve the experimental networks on campus that the OptIPuter researchers are using. The UCSD Vice Chancellor for Research provided the promised institutional cost sharing for the OptIPuter project; in Year 1, the cash amount was $45,749, which provided partial support to establish or upgrade IA-32 PC Linux clusters for nine key OptIPuter sites on campus. In addition, UCSD provided additional cost sharing by allocating dark fiber – repurposing existing fiber (valued at $125,000), and purchasing new fiber (valued at $136,187) – resulting in the deployment of four-pair of dedicated single-mode fiber to each of these nine key campus research facilities:

• UCSD School of Medicine (SOM) National Center for Microscopy and Imaging Research (NCMIR) • UCSD Scripps Institution of Oceanography (SIO) Visualization Center • UCSD Computer Science & Engineering (CSE) department • UCSD Jacobs School of Engineering (JSOE) • UCSD Preuss School • UCSD Sixth Undergraduate College • UCSD Center for Research in Computing & the Arts (CRCA) • UCSD San Diego Supercomputer Center (SDSC) • UCSD SDSC Annex

Five of the above locations (SOM, SIO, CSE, JSOE and SDSC) now have Dell 5224 GigE switches connected to the Chiaro router. As of August 2003, four locations have OptIPuter computers connected: SOM, CSE, JSOE and SDSC. By September 30, CSE, Preuss and SIO will have clusters connected.


The UCSD campus and UCSD/SDSC also provided Juniper router equipment (valued at $690,000). A Juniper T320 router connects the Chiaro next-generation optical core router, purchased for the OptIPuter’s optical network LambdaGrid testbed, to CENIC’s High Performance Research (HPR, Tier-2) network. UCSD is part of the CENIC ONI effort and will be served by CENIC’s HPR network at 10Gb speeds and CENIC’s Experimental/Development (XD, Tier-1) network with multiple 10Gb services, which will use a combination of switched lambdas and dark fiber. SDSC is an ONI hub site for these services. The NLR, once viable, will link Southern California to Chicago, as well as nationally and internationally. Note: SDSU has been working with Carlos Casasus of Mexico’s CUDI Research & Education network on joint US/Mexico activities requiring optical networking. CUDI and CENIC recently established a new dark fiber connecting the CUDI Internet2 network with the CENIC state network. This sets the stage for potentially extending the OptIPuter to include Mexican researchers under other funding.

Network Hardware Infrastructure UCSD and UIC are exploring different switching/routing technologies.

MEMS Switching (Layer 2) Routing (Layer 3) 1x 10x 100x OOO OEO OEOEO Calient, Glimmerglass Cisco Chiaro Data Plane Control Plane Routing

The UCSD OptIPuter team selected the Chiaro Enstara as the routing platform for its LambdaGrid testbed because it provides the advantages of traditional routing along with leading-edge optical switching technologies. The Enstara is programmable with 24/7 reliability and virtual partitions. Chiaro’s optical core can reconfigure lightpaths very quickly because it has no moving parts and instead uses interference patterns to “bend” light to desired destinations. Because of its solid state design, the OptIPuter team can experimentally explore where small-sized packet switching

SDSC90 Nodes

JSOE4 Nodes

CSE APM

SIO IGPP Viz9 node cluster

BIRN SOM32 Nodes + EM

SDSC Annex16 Nodes Preuss Viz

GeoWall

SDSU Viz

CRCA Viz

6th College Viz

4

4

4

4 BondedGigE Linksover fiberpairs

4

4

4

4

4

UCI

CalREN-HPR

National Lambda Rail

OptIPuter UCSD / San Diego Network

As of 07/25/03

Chiaro

4 GigE

Teraburst

Teraburst

1

OC-48/192Future

4

USC

NWUUIC

ISI Amsterdam

Dell 5224

Dell 5224

Dell 5224

Dell 5224Dell 5224

Dell 5224

Dell 5224

Dell 5224

Dell 5224

Viz Node

ClusterNode


enabled by Chiaro’s Optical Phased Array (OPA) technology fits with lambda circuit switching in the overall OptIPuter project. This experimental duality will be a key enabler for exploring the many deep research issues in OptIPuter. The Enstara routing platform employs several innovative technologies such as tens of nanosecond optical packet switching, centralized switch fabric scheduling, and the use of programmable network processors. Access to these network processors allows OptIPuter researchers to test and explore new types of protocols while the network is running applications. Enstara can seamlessly integrate 10 Gigabit Ethernet (GigE), OC-192 and future OC-768 interfaces.

UIC has two 16-CPU clusters for metro-distance OptIPuter research, one located at UIC/EVL and one at the StarLight facility. DWDM gear from UIC to StarLight, a 10Gb link fanning out to 8GigEs, was purchased with I-WIRE funds and NSF infrastructure grants. An additional 8GigEs between UIC and StarLight, provided by the OMNInet project, connects these two clusters, providing total connectivity of 16Gb.

Both electronic (Cisco 6509) and optical Microelectromechanical System (MEMS) switches (Calient DiamondWave) are now operational to support the OptIPuter project; a 128-port Calient is at UIC, but will be moved to StarLight by late September 2003, and a 64-port Calient is at the University of Amsterdam. UIC has also purchased a Glimmerglass Reflexion MEMS switch with a photonic multicasting option. By using different switch technologies over two different DWDM paths, UIC shall test and compare these competitive strategies for lambda-switching.

With existing funds, UIC upgraded the StarLight Cisco 6509 switch/router and also purchased a Force10 E1200 to handle up to 20 10GigEs and 128 1GigEs, so that the UIC and StarLight clusters can be coupled to incoming wavelengths of 10Gb each from Canada, the Netherlands, CERN and the UK, as well as 10GigE routed circuits from the TeraGrid, Abilene, ESnet, and other National R&E networks (such as USAWaves and the NLR), all of which have (or will soon have) presence at StarLight. Many of these 1GigEs and 10GigEs will also be switched with the Calient MEMS optical switch.

Cluster Hardware Infrastructure The UCSD/NCMIR lab directed by OptIPuter co-PI Mark Ellisman developed an OptIPuter testbed in Year 1 that connects its high-speed laser microscope, used for part of its brain mapping efforts, to a large-field high-resolution 3D display (2 x 9Megapixels) using a 64-processor cluster to handle some tasks on the streams of data coming from the imager. UCSD/NCMIR is now working with HP to prototype a visualization cluster with its new Sepia graphics card, which is expected to help get pixel pipes to the large pair of displays. NCMIR is working with OptIPuter co-PI Phil Papadopoulos to design a data server, which will be deployed in Year 2 to accompany each of its high-data-rate instruments (multi-photon microscope, energy filtering intermediate voltage electron microscope) for shuttling massive data into the OptIPuter.

In addition, in Year 1 UCSD worked with IBM and Sun to design visualization-, compute- and storage-intensive clusters which could be linked by the dedicated campus fiber through the Chiaro router described above. For Year 2, the focus is on deploying a powerful set of these clusters at three of the nine OptIPuter sites: SIO, JSOE, and SDSC.

• SIO: 10-node IBM visualization-intensive cluster driving an Scalable Graphics Engine with two T221 9-MegaPixel displays [this has been purchased and will arrive in early September 2003]

• JSOE: 48-node 21TB IBM storage-intensive cluster [Proposal for donation submitted to IBM August 29, 2003]

• SDSC: 128-node Sun compute-intensive cluster: Sun Microsystems has joined the OptIPuter project by donating to UCSD a 128-node Linux Intel IA-32 cluster using Sun V60x nodes in Sun’s so-called “large” configuration (1U Dual 2.8 GHz XEONs, 1GB memory, one 36GB SCSI drive). The motherboard has two built-in PCI-X interfaces, and two copper GigE ports. The cluster will be interconnected with SMC switches.

UIC is focusing on next-generation (“Mod 1”) OptIPuter development. All development clusters are currently based on IA-32 architecture. UIC has already built and evaluated a 16-node cluster, and then revised and re-spec’ed a 9-node cluster. UIC is evaluating Itanium2s. Most recently, UIC came up with initial specs for small, medium and large visualization OptIPuter clusters for other OptIPuter sites, which will be refined in the coming months to track new hardware developments. (See Section 10 for the current specification.)


2.A.3. Software Architecture Activities Year 1 tasks: UCSD, UCI and USC defined a first-generation OptIPuter Systems Architecture (previously called the LambdaGrid Middleware Architecture toolkit), experimented with new software stacks to bandwidth-match PCs to lambdas, and are examining Globus’ GridFTP and GSI modules from an OptIPuter viewpoint. UCSD, USC, UIC and NU established a base-line design for network management, lambda assignment and signaling. UCSD, UCI and USC are conducting a digital rights management study and beginning to design a security plan. TAMU is using Prophesy to instrument applications and develop analytical models for performance analysis.

Optical Architecture NU (Mambretti) held a workshop with UIC (Yu) and UCSD (Chien) to define the OptIPuter optical backplane architecture – for optical network signaling, resource utilization management, lambda assignment (lightpath provisioning), network survivability and monitoring (see meeting notes, Section 11.) This architecture:

• Creates a framework and protocol implementation model for the OptIPuter distributed optical backplane. • Examines adaptation of GMPLS protocols to support intradomain optical backplane signaling. • Determines general application requirements for OptIPuter communication services, as a generic class. • Examines options for policy driven resource access and determines that, in the first instantiation, (I)AAA

services would be used as an interface architecture • Determines middleware requirements and components, and examines adaptation of Globus v.3 in the

OptIPuter environment. • Defines key tasks and schedules for the five-year OptIPuter program

Note: A more complete architecture requires additional detail, in particular, the interrelationships among components. Compatibility of the signaling and resource management architecture within the OGSA/OGSI context is just now being examined in detail. Initially, one objective was to support optical switching of labeled switched paths that transport InfiniBand messages. In the storage-intensive cluster proposal to IBM mentioned above, we asked for hardware that will enable detailed study of InfiniBand, led by IBM’s Alan Benner. We will pursue this topic further in Year 2, depending on the outcome of the IBM grant.

NU (Mambretti) and UIC (Yu) conducted exploratory experiments in simulation and physical testbeds related to optical transport and enhanced L3 protocols.

• Examined and experimented with high-performance L4/L3 protocols mechanisms, began to define protocol-independent metrics

• Examined potential designs for intelligent application signaling for lightpaths, and related protocols • Examined options for an architecture that includes enhanced interaction with state information; i.e.,

physical elements and their attributes, resources (e.g., lightpaths), and fault conditions • Defined a set of required resource objects, such as signaling (lightpaths are defined in part by source and

destination addresses, which are key signaling elements)

UIC 16-node OptIPuter visualization cluster displaying UCSD/SIO “glyphs.”


• Explored novel networking technologies among a small number (~2) of clustered development endpoints for high-throughput data transfer and intradomain control of lightpaths

• Created a preliminary architecture for interdomain signaling • Began designing methods allowing lightpath channels to be extended to edge devices • Began creating enhanced interfaces; e.g., to GMPLS libraries, TL1 APIs • Formulated strategies to enable secure optical VPN across the OptIPuter backplane • Created concepts for the first-phase architecture for large flow cut through • Specified concepts for robust, survivable optical data communications, including multi-layer protection and

restoration • Examined Management Information Base (MIB) to support distributed OptIPuter backplane management. • Examined SNMP-type concepts to create management architecture

Standards activities…During Year 1, IETF produced a Draft RFC, “IP over Optical Networks: A Framework” <ftp://ftp.ietf.org/internet-drafts/draft-ietf-ipo-framework-03.txt>, which some OptIPuter team members, led by Joe Mambretti and George Clapp, are reviewing for its relationship to the OptIPuter basic research program. IETF is a standards body which translates promising research done elsewhere into specifications that can be used by product developers for commercialization. We look forward in Year 2 to working with IETF on their “IP over Optical” draft, which we believe describes an excellent conceptual framework of current industry concepts; however, few, if any, of the concepts are novel.

Given that the OptIPuter by necessity is leveraging existing commercial products, it is important to clearly distinguish OptIPuter innovations from more general architectures and concepts being examined and developed by standards bodies. NU (Mambretti) is working with other team members to develop a white paper, in which emerging OptIPuter concepts are compared and contrasted with developing IETF architectures. The OptIPuter team will pursue good relations with IETF, as well as other standards bodies, whose efforts are related to our activities.

LambdaRAM: Optically Connected Wide-Area Network Memory LambdaRAM is an application being developed by UIC/EVL, supported by other NSF funding, to address long-haul latency in optical networks. This technique collects memory in a compute cluster and then allocates it as a cache to minimize the effects of latency over long-distance, high-speed networks. LambdaRAM takes advantage of multiple-gigabit networks (available on the StarLight and OMNInet testbeds) to pre-fetch information before an application is likely to need it (similar to how RAM caches work in computers today). LambdaRAM extends this concept over high-speed networks. EVL has been testing LambdaRAM (UDP blasting) between clusters at UIC/EVL and UIC/LAC. For more information, see <http://www.evl.uic.edu/cavern/teranode/gridram>

EVL is modifying LambdaRAM to be generic middleware that can support OptIPuter applications.

Security UCI (Goodrich) published algorithms for secure authentication of geometric and network data at the RSA Conference – Cryptographer’s Track. Goodrich also published algorithms for collecting and distributing authenticatable data in the Secure Transaction Management System (STMS), at iTrust, the International Conference on Trust Management. (See Section 3: Publications.)

UCSD Sid Karin is carrying out a “best-of-breed” study of existing research by other groups on security over fiber-optic networks. He plans to deliver an internal OptIPuter white paper on this subject by the end of summer 2003, from which the security research plan for Year 2 will be derived.

Also see the System Software Architecture paragraphs below.

Performance Analysis TAMU (Taylor) completed the model builder component of Prophesy, such that users can go from instrumentation to developing performance models using Prophesy. Prophesy now includes three techniques for developing models: curve fitting, parameterization and coupling. Each has distinct advantages and disadvantages. The models can be used to predict the performance of an application under different scenarios. Further, the detailed performance information can be used to fine-tune codes. Taylor developed an interface that allows performance data from SvPablo as well as Prophesy to be automatically sent to the Prophesy database.


Real-Time Execution Environment For Year 1, no deliverables were required. However, UCI (Kim) designed a real-time Time-Triggered Message-Triggered Object (TMO) support middleware subsystem model that can be easily implemented on both Windows and Linux platforms. Kim developed a global time-based coordination approach for use in fair and efficient distributed on-line game systems and a feasibility demo based on LANs and the TMO subsystem, as a step toward realizing an expanded demo in the OptIPuter environment.

High-Performance Transport Protocols: XCP Note: In Cooperative Agreement, this section was called “Unicast and Multicast Lambda Overlay Protocol” and “Dependable Service and Recovery.” The USC XCP development team is working with Dina Katabi and John Wroclawsi of MIT.

For Year 1, no deliverables were required. However, USC (Bannister, et. al.) have been developing XCP, and exploring ways to tailor it to the OptIPuter environment. They are doing simple testbed functionality testing and benchmarking of XCP, performing simulations to explore the dynamics of XCP during traffic discontinuities, developing a preliminary congestion header format, investigating representation issues to balance scalability versus fast processing, and identifying some engineering issues; e.g., what is the best location and numerical representation of the fields in the congestion header. Platform testing will help determine the maximum throughput that a two-port BSD router can deliver at various packet sizes. They developed preliminary end-system and router code for the BSD stack, which will be shared with Cisco as part of an engineering collaboration.

Some Year 2 research issues have been identified; e.g., link layer congestion (i.e., non-router queues), links with varying rates, and non-XCP queues.

High-Performance Transport Protocols: Quanta/RBUDP and SABUL The OptIPuter project made considerable progress in Year 1 on alternate protocols based on UDP, with UIC/EVL focusing on Reliable Blast UDP (RBUDP) and UIC/LAC focusing on SABUL. RBUDP, developed in part with NSF PACI/Alliance funding, is built into the Quanta applications-centric communications middleware software. It has been tested over LAN and WAN links (including Chicago to Amsterdam, and Urbana to San Diego via TeraGrid). In August 2003, Leigh and his students used RBUDP to blast data from NCSA to SDSC over the TeraGrid DTFnet, achieving18Gbps file transfer out of the available 20Gbps (one of the 10Gb links at San Diego was down). (Note: The TeraGrid network can be scheduled for experiments.) Leigh plans to use the link for future OptIPuter experimentation.

SABUL, a technique for high-performance data transport, combines a UDP data channel and a TCP control channel. Over the past year, UIC/LAC improved the rate control mechanism for SABUL, and added a congestion control mechanism in order to make the protocol friendly. (Note: a standard data-mining application operating between Chicago and Amsterdam can process data at about 5Mbps due to limitations with current TCP implementations.)

UIC provided NCMIR with Quanta, and NCMIR is investigating the use of Quanta for a number of medical imaging applications.

System Software Architecture Note: This section encompasses the efforts listed as “LambdaGrid Middleware Globus Toolkit Extensions Architecture Development” and “Software Systems Architecture: Bundled Endpoint and Computing (E&C) Systems” in the Cooperative Agreement.

The OptIPuter System Software team, headed by UCSD (Chien), consists of UCSD (Phil Papadopoulos, Chaitan Baru, Sid Karin), UCI (Mike Goodrich, Kane Kim), USC (Carl Kesselman), and UIC (Bob Grossman, Jason Leigh). This year the group designed an initial software architecture; version 0.9 of the OptIPuter Software Architecture document is in Section 12. The group discovered that traditional layered system architectures are not appropriate – many attributes and capabilities cut across layers. Theirs allows integration of functionality that was traditionally layered as services (horizontal integration) and supports flexible integration of OptIPuter technologies and outside (emerging Grid and Web Services) technologies.

For Year 2, the group will simulate several OptIPuter network configurations and systems using GT2.2 grid middleware and the MicroGrid grid emulation system (from UCSD) to gain insights into impact of massive


bandwidth on scheduling and resource management policies and techniques.

2.A.4. Data, Visualization and Collaboration Activities Year 1 tasks: UCSD and SDSU are working with UIC visualization experts to investigate new collaboration and computer-graphics tools and techniques for volume visualization, information visualization, streaming video and high-definition tiled displays tailored to huge datasets. UIC, UCSD and UCI data experts are developing specialized protocols to provide end-to-end performance for data-intensive applications over long-haul networks, a peer-to-peer block-level storage system for the OptIPuter, and high-level abstractions for working with remote and distributed data. A variety of data and visualization experiments are being carried out using large multi-scale datasets from seismic and brain mapping projects that currently challenge data exploration environments. Tractable application challenges are being used to make informed decisions about future software development, to be deployed in Year 2.

Data For Year 1 of the OptIPuter, no deliverables were required.

UIC (Grossman) has been exploring Data Webs and Data Web Services. Data webs are web-based infrastructures for data. Unlike grid middleware which is built over authentication, authorization and access (AAA) control mechanisms for rationing and scheduling presumably scarce high-performance computing resources, data webs are built using W3C standards and emerging standards for web services and packaging (SOAP and XML). Data webs, in contrast to data grids, are designed to encourage the open sharing of data resources without AAA controls, in the same way that the web today encourages the sharing of document resources without AAA controls. UIC has previously developed a protocol for moving data between data web servers and data web clients called the Data Space Transfer Protocol (DSTP). DSTP includes functions for retrieving data, metadata, and keys, for selecting rows and columns of data, and for sampling. During the report period, UIC improved the DSTP’s server ability to import data in a variety of formats, and improved the integration of DSTP, SABUL, and local input/output so that the end-to-end, disk-to-disk performance of DSTP improved. Work in this area is continuing. Data Web applications have been developed for astronomical, earth sciences, bioinformatics, and social science data.

UCI (Smyth) developed a general theoretical framework and a set of algorithms for statistical modeling and clustering of sets of curves and trajectories. This methodology performs curve clustering and curve alignment simultaneously and optimally. Prior work in this area relied on separate (and suboptimal) steps of alignment and clustering. The methodology has been successfully applied to clustering of two different large scientific datasets: extra-tropical cyclones and time-course gene expression data. Smyth has also begun implementation of a data wall at UCI (in collaboration with UCI’s graphics and visualization research group), consisting of a 3x3 LCD tiled display with a Linux machine coordinating each LCD panel. The data wall will be used to provide a testbed environment at UCI for visualization, exploration, and data mining of high-throughput scientific datasets.

Multicast Dedicated optical networking impacts classical assumptions about the ways applications operate. MEMS-switched networking to support visualization and data exploration is motivated by an observation that the predominant model for large-scale data-intensive applications employs the following Data Exploration Pipeline:

Data Sources Data Correlation/Filtering System Visualization System Display System

Large collections of real or simulation data are fed into a data correlation or filtering system that generates sub-sampled or summarized results from which a visual representation can be created and displayed on single screens, tiled displays or virtual-reality systems. In the context of collaborative data exploration, the results of the pipeline may need to be multicast to several end points. Unlike web browsing on the Internet (which tends to involve users jumping from website to website), the connectivity between the computing components in a large-scale Data Exploration Pipeline typically is static once connections are established. Hence, costly packet-by-packet multi-gigabit routing is unnecessary.

UIC/EVL is working with GlimmerGlass Networks to develop photonic multicast switches that can be combined to create very-high-performance mesh networks to support fully dedicated high-speed data transmissions between network elements (see diagrams below). When combined with proper scheduling of cluster computing resources and lightpaths, this architecture will allow applications to create multiple, simultaneous, distributed computing pipelines.


Top diagram: In the OptIPuter, a number of distributed compute clusters are interconnected via a photonic switch rather than with traditional routers, creating a Photonic Data Exploration Pipeline. The optical transport platform performs the task of multiplexing and demultiplexing multiple lightpaths over a DWDM network. Bottom diagram: This shows the integration of photonic multicasting into the Photonic Data Exploration Pipeline.

Motivated by the emergence of photonic networking techniques, the Chicago OptIPuter team is developing the Continuum, a collaborative project room that fuses together a broad range of contemporary display technologies, such as passive stereo displays (e.g., GeoWalls), high-resolution tiled displays such as AccessGrids, and large-format shared digital whiteboards. The Continuum brings together prototype visualization and collaboration tools that are designed specifically to take advantage of the OptIPuter, namely TeraScope, JuxtaView and TeraVision (described below).

Visualization/Collaboration Tools (General) UIC’s (Leigh) TeraVision graphics streaming hardware, developed with other NSF funding and tested from Chicago to Amsterdam, Urbana and Greece, was also tested over a fully photonic testbed using two Calient switches at EVL controlled by EVL’s Photonic Domain Controller (PDC) software.

SpaceGlider tiled display interaction software was developed.

SIO did visualizations of earthquake point data using an existing software package <www.geol.binghamton.edu/faculty/jones/seisvole.readme>. Earthquake distribution (global, regional, local)

ComputeCluster 1

(Data source)

ComputeCluster 2

(Data Correlation/ Filterning)

ComputeCluster 3

(Visualization)

ComputeCluster 4(Display)



Photonic Switch withPhotonic Multicast

Service

DWDM

Mux/DeMux

ComputeCluster 1

(Data Source)

ComputeCluster 2

(Data Correlation/ Filterning)

ComputeCluster 3

(Visualization /Rendering)


DWDM

Mux/DeMux

PhotonicSwitch

Phot

onic

Dat

a Ex

plor

atio

n Pi

pelin

e


included exchanging data from data centers in (1) Memphis, TN, (2) Alaska, (3) Stanford University, and (4) ANZA seismic network in Southern California. Using OptIPuter funds, SIO hired Atul Nayak, who recently received his Master of Science degree from UIC/EVL, their first dedicated visualization specialist. SIO deployed a GeoWall for passive stereo visualizations.

UIC (Leigh) worked with UCSD/NCMIR (Ellisman, Lee) to modify EVL’s JuxtaView and ImmersaView software for biological data. JuxtaView is used to display large-scale brain maps of animal models of disease; NCMIR provided EVL with large mosaics and large-format tomography data to be visualized on tiled display systems. ImmersaView, passive stereo software, is currently running in the NCMIR conference room on their GeoWall.

Over the past year, SDSU has evaluated oil company software packages, including VoxelGeo by Paradigm Geophysical and software by GeoViz <www.geoviz.com>, GeoFusion <www.geofusion.com>, Muse <www.musecorp.com>, Pacific Northwest National Labs Starlight program <http://availabletechnologies. pnl.gov/infotechenergy/sta.stm>, and MindTel’s Neat Tools <www.mindtel.com>, <www.pulsar.org/2k/images/neatimages>

Volume Visualization Tools Groups are benchmarking potential OptIPuter hardware as well as developing software. UIC has been looking at throughput from memory to graphics hardware and vice versa on various processors, including the HP Itanium systems. NCMIR is evaluating and pursuing SGE (IBM) which can be used for high-resolution image streaming. Also, NCMIR signed an MOU with KDDI Labs for HDTV (and beyond) over IPv4/IPv6 networks.

UIC (Leigh) and USC (Thiébaux) are defining an OptIPuter adaptive visualization pipeline to support both high-resolution digital montages and volume visualization. The Grid Data Transport (GDT) is a robust API for constructing tiered pipelines with MxN connectivity for parallel data filtering and transport, and the Grid Visualization Utility (GVU) defines an extensible dataset and filter-command class hierarchy for VTK-style datasets and rendering/analysis algorithms. GDT and GVU can be applied to 2D and/or 3D visualizations; however, they are currently being used to generate volumetric visualizations. The goal is to use the concept of adaptive networking in Quanta, but extend it to the entire visualization pipeline. The system would monitor different resource availabilities, including the network, and would monitor the performance of different visualization algorithms. This information would be used to help high-level applications make smart choices on how to deliver final services to the user. If it works as planned, a user would have one universal volume visualization software package that could run on any computer regardless of capability and it would know how to adapt itself appropriately for the volume of data it has to access and the resources it has available to accomplish its goals.

UIC (Leigh) and SIO (graduate student Christine Reif) are working on volume visualization techniques. With EVL assistance, Reif experimented with isosurfaces in MATLAB to gain a better understanding of her data, as illustrated here. Prior to involvement with the OptIPuter project, Reif examined her 3D data using 2D cross-sections, which proved to be non-optimal. Subsequently, multiple techniques were explored.

UIC’s (Leigh) WiggleView, real-time seismic visualization software for Geowall, was integrated with Antelope and IRIS’ BUD seismic data servers. A WiggleView seismoglyph information visualization technique developed, and the software tailored to display results across a tiled display.

SIO scientists converted a seismic volume dataset from the ARAD 3D experiment into a more user-friendly format, and have distributed copies to the UCSD/SDSC and UIC/EVL visualization groups. Data is based on sub-volume pieces, given constraints in computer memory. SIO is working with UIC (Leigh) to obtain an initial image by September, 2003. This large volume (800x800x1000 at 32 bits) highlights the first 3D reflection image of a crustal magma chamber, which was located beneath the East Pacific Rise, off-shore Mexico. Initial images were published by Kent et al. (2000) in the journal Nature.

UCSD/NCMIR (Lee) implemented VolView and applied it to biological data. VolView can be used to display stereo on IBM T221 displays with a modified Wheatstone Stereoscope placed in front of the displays. UIC and NCMIR collaborated on the development of the physical design of the 9MegaPixel active 3D display.

UIC and NCMIR are looking into adopting concepts/code from Utah’s transfer function GUIs for displaying voxel visualization on display walls.


UCSD/NCMIR is working with the IBM display group to receive new firmware updates for its T221 displays; a recent update will hopefully solve the input genlock problem. NCMIR is also working with HP to integrate 3D display with its Sepia cluster.

2.A.5. Applications and Education Activities Year 1 tasks: UCSD application teams are distributing and integrating simulation codes on clusters located at several campus sites. UCSD and SDSU application teams are providing sample datasets to the technology teams and are working alongside them to prototype possible solutions, helping define the architecture of the OptIPuter. Training of UCSD Preuss School teachers on visualization of earth sciences datasets has begun. In addition, an annual OptIPuter meeting/workshop was held, with some days dedicated to OptIPuter partners and some days open to the wider e-Science and high-performance computing and communications communities. (Note: Work with the UCSD Sixth College’s Curriculum Support Group has not started.)

Application Codes Running on OptIPuter Clusters See Section 2.A.4, which describes a number of activities between the application scientists at UCSD/NCMIR and UCSD/SIO involving the use of their datasets on clusters running visualization and collaboration codes. More specifically, UIC/EVL is working with UCSD/SIO to create high-resolution images of bathymetry, and with UCSD/NCMIR to create high-resolution images of a rat’s brain (the node of ranvier). SDSU participated in several projects, all involving students:

• SDSU acquired very large databases of regions such as the Caspian Sea, Iraq, Aral Sea, and southern California, to be used as building blocks for educational products. Much of the data was satellite imagery (Landsat 7, ASTER, MODIS, and SeaWiFS). Datasets were used to test the usefulness of the data and possible best-processing techniques. Several datasets (e.g., Iraq), were used for humanitarian and medical help in support of US efforts in the southern portion of the country before, during and after the war.

• SDSU acquired a large amount of air photo data and multi-spectral data taken from a helicopter by SDSU staff members. These datasets are used to demonstrate major volume visualization techniques and transport protocols across optical networks.

• SDSU worked with the NSF-sponsored High-Performance Wireless Research and Educational Network (HPWREN) to demonstrate applications such as videoconferencing and application sharing via wireless networks off the end of the optical networks to remote sites in the California desert.

UCSD/Preuss School In Year 1, the OptIPuter funded a GeoWall at the UCSD Preuss School, as scheduled. UCSD/SDSC (Steckler, Bailey) worked with UCSD/SIO (Kilb) and the UCSD/Preuss School to develop a six-week curriculum as well as compelling images, movies and 3D interactive graphics packages for teaching earth science to 8th graders using this facility. They are now using an earth science hands-on-learning lesson book (developed primarily by IRIS <www.iris.edu>), which has over 30 hands-on-learning lessons that can be immediately used in the classroom and can be further developed through the OptIPuter project to include 3D graphical components. SIO developed educational teaching modules for the GeoWall, and worked with the company IVS to have its software package Fledermaus produce stereo images for the GeoWall, which greatly extends the number of 3D stereo visualizations available.

Lincoln Elementary School in Oak Park, IL Although the UIC Education group had no explicit goals for Year 1, Tom Moher (UIC) and Debi Kilb (SIO) designed RoomQuake, to introduce Lincoln Elementary School students to concepts of plate tectonics, geologic faults, and the determination of earthquake epicenter, focus, and magnitude through interpretation of seismographs and mathematical triangulation. In RoomQuake, multiple PocketPCs are positioned in fixed locations within a classroom and used as simulated seismometers, with a series of 30-40 simulated earthquakes scheduled sporadically over a one-month period. When a “quake” occurs (to the accompaniment of rumbling provided by a subwoofer installed in the classroom), students read the seismographs, and determine the distance of the epicenter – within the room – from the PocketPCs. Using tape measures anchored at the site of the “seismometers,” three students sweep out arcs, with their collision point defining the quake’s epicenter. Styrofoam balls of varying sizes (reflecting magnitude) are then hung from the classroom ceiling on strings of different lengths (reflecting depth, computed


using the Pythagorean Theorem); over the course of a month, the classroom “fault lines” emerge. RoomQuake, embedded within a larger curriculum unit on earth structures currently in development with teachers, will be introduced in fifth grade classrooms during the fall semester. Quantitative and qualitative assessments will be employed to characterize student learning of earth science and mathematics content.

UCSD/SIO Outreach Activities

SIO Teacher Workshop (May 24, 2003)…On Saturday, May 24, 2003, the SIO Institute of Geophysics and Planetary Physics (IGPP), US Geological Survey (USGS), Southern California Earthquake Center (SCEC), and the Birch Aquarium at Scripps (BAS) held an earthquake education workshop at SIO’s Visualization Center <http://siovizcenter.ucsd.edu/workshop>. Within 2 days, 40 people had signed up for this workshop but SIO could only accommodate 25. SIO hopes to hold another workshop to accommodate the needs of those on its waiting list.

SIO/Regional Workbench Consortium (RWBC)…On May 29, 2003, the RWBC met at SIO. A demonstration, illustrated above, included an introduction to the 3D Regional Canvas of the Californias – a digital elevation model of Southern California/Northern Baja California region, including land and sea floor terrain features. This unique dataset was assembled with high-resolution regional elevation data developed by members of the RWBC Consortium, from institutions of both sides of the US/Mexico border, and SIO (Kent, Kilb).

SIO/GeoWall Consortium…SIO (Rob Newman) works closely with members in the GeoWall consortium <www.geowall.org> to port SIO-generated visualizations to GeoWalls nationwide. The SIO GeoWall is used extensively in education and outreach presentations, both at the SIO Visualization Center and at off-campus locations.

Annual OptIPuter Meeting During the week of February 3, 2003, two major workshops took place: the ON*VECTOR Photonics Workshop (February 3-4) and the OptIPuter All Hands Meeting (February 6-7). The former meeting brought together photonics experts from the USA, Canada, Europe and Asia, including many of the people involved with the OptIPuter project. Similarly, those not involved got a chance to learn more about this project. The agenda, attendee list and PPTs from the ON*VECTOR meeting are available upon request. The OptIPuter All Hands Meeting is documented on the web <http://www.optiputer.net/events/2-6-03.html>.

In addition, in October 2002, the OptIPuter sponsored an Optical Switch Workshop, bringing together interested parties from academia and industry, to discuss available hardware. The agenda, attendee list and PPTs can be found on the web <http://www.optiputer.net/events/10-25-02.html>.

2.A.6. Meetings, Presentations, Conference Participation August, 24-27, 2003. Padhraic Smith presented the paper “Translation-invariant mixture models for curve-clustering,” at the Ninth ACM International Conference on Knowledge Discovery and Data Mining (KDD-2003),


the 9th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Washington DC.

August 27, 2003. 3rd annual Lambda Workshop, to be held after NORDUnet 2003 in Reykjavik, Iceland. The objectives of this year’s meeting are to (1) identify technical research needed for development of LambdaGrids, (2) how to deal with Lambda switching, (3) hosting policies and agreements, and (4) testbeds and deployments on 1, 2, 5 year scale. Larry Smarr (UCSD), Tom DeFanti and Maxine Brown (UIC) and Joe Mambretti (NU) attended.

August 24-26, 2003. NORDUnet 2003 in Reykjavik, Iceland <http://www.nordunet2003.is/>. Larry Smarr (UCSD), Tom DeFanti and Maxine Brown (UIC) and Joe Mambretti (NU) attended. Smarr gave the presentation “OptIPuter and ENDfusion,” and Tom DeFanti gave the presentation “StarLight and TransLight.”

August, 8-10, 2003. Padhraic Smith presented the paper “Probabilistic models for joint clustering and time-warping of multidimensional curves,” at the 19th Conference on Uncertainty in Artificial Intelligence (UAI-2003), Acapulco, Mexico.

July 27-31, 2003. SIGGRAPH 2003, San Diego, CA <www.siggraph.org/s2003>. Larry Smarr, Greg Hidley, Sheldon Brown and Mike Bailey (UCSD), Eric Frost (SDSU) and Maxine Brown (UIC) are attending. Carol Hobson (UCSD) is on the SIGGRAPH 2003 Art Gallery subcommittee and is the “unofficial” local outreach coordinator. Hobson and Sheldon Brown (UCSD) are meeting with the SD Museum of Contemporary Art’s downtown curator to plan an exhibition, installation and/or related event at their venue by the train station.

July 7, 2003. ESRI International User Conference at the San Diego Convention Center. Brian Davis of USGS gave GeoWall demonstrations. Eric Frost of SDSU also attended. ESRI makes ARCGis/Info which is the main GIS tool the Geoscience community uses.

June 24-27, 2003. Cal-(IT)2 and SDSU hosted the North American Summit for IPv6 <www.usipv6.com>. SDSU demonstrated the use of IPv6 and IPv4 networks in a dual stack connecting information from sensors (cameras) to the SDSU command-and-control center and to the US-Mexico border using application sharing software (Wave3 Sessions software). Virtual-reality scenes were built in real time and transported back to computers in the SDSU command-and-control center. Follow-on demonstrations of this technology are being planned for a conference on Sensor Networks (March 1-3, 2004) to show off IPv6 sensor network capabilities and a conference for Pentagon leaders (Nov. 2003) doing data fusion of situational awareness over wireless and optical networks.

June 23-27, 2003. Global Grid Forum 8 (GGF 8), Sheraton Seattle Hotel & Towers, Seattle, WA. Tom DeFanti, Bob Grossman and Jason Leigh (UIC), and Joe Mambretti (NU), attended. DeFanti gave the plenary talk “LambdaGrids and the OptIPuter,” and chaired the panel “iGrid Applications;” Joe Mambretti, Bob Grossman, Jason Leigh and Cees de Laat participated.

June 22, 2003. Third Annual Workshop on Advanced Collaborative Environments (WACE), co-located at the 12th IEEE International Symposium on High Performance Distributed Computing (HPDC 12) and Global Grid Forum 8 (GGF), Seattle, WA. Jason Leigh (UIC) was a member of the program committee and participated in WACE <www.mcs.anl.gov/fl/events/wace2003/>.

June 5, 2003. An exploratory meeting with SIO researchers (Debi Kilb, Rob Newman) and specialists in museum displays from the Natural History Museum and the Birch Aquarium at Scripps gathered to discuss incorporation of seismic displays and OptIPuter-generated 3D interactive visual objects in museum settings in San Diego <www.siovizcenter.ucsd.edu/news_events.html>.

May 29, 2003. SIO (Debi Kilb, Graham Kent) did a demonstration for the Regional Workbench Consortium (RWBC), introducing attendees to the 3D Regional Canvas of the Californias – a digital elevation model of Southern California/Northern Baja California region, including land and sea floor terrain features. This unique dataset was assembled with high-resolution regional elevation data developed by members of the Consortium from institutions of both sides of the US/Mexico border. The audience included those with a common interest in linking science and policy for a better regional planning. Kilb constructed many of the 3D visualizations under the auspices of the OptIPuter project. Kent presented his work on Lake Tahoe.

May 24, 2003. The SIO Institute of Geophysics and Planetary Physics (IGPP), US Geological Survey (USGS), Southern California Earthquake Center (SCEC), and Birch Aquarium at Scripps (BAS) held an earthquake education workshop at SIO’s Visualization Center <www.siovizcenter.ucsd.edu/workshop>. Many of the 3D visual objects developed for the OptIPuter outreach program to the Preuss School were presented to the ~40 San Diego K-12 teachers. Many teachers expressed significant interest in supporting the continual development of such OptIPuter-


funded 3D visualizations that, with minimal computer equipment, could be used in the classroom.

May 22, 2003. OptIPuter BackPlane (a.k.a. Network) Architecture Workshop, held at UIC in Chicago. Joe Mambretti (NU) organized the meeting, attended by Andrew Chien (UCSD), Oliver Yu (UIC), Tom DeFanti (UIC), Jason Leigh (UIC), George Clapp (Telcordia) and others. The goal of the Workshop was to determine an architectural framework for the OptIPuter “backplane,” which is different than a traditional network, in part, because of its close integration among other components. (Workshop information to be posted to OptIPuter website.)

May 14, 2003. Tom Garrison’s honor class from Orange Coast College/USC visited the SIO Visualization Center to participate in a lively discussion about earthquakes, data artifacts and plate tectonics led by Debi Kilb (SIO). Garrison is the author of the best-selling oceanography textbook, Oceanography: An Invitation to Marine Science, and is currently incorporating supplemental online 3D visualizations. Garrison expressed interest in incorporating some visual objects developed by SIO researchers into his online book supplement.

May 8, 2003. Larry Smarr (UCSD), Tom DeFanti (UIC), Maxine Brown (UIC) and Steve Wallach (Chiaro) visited HP headquarters in Palo Alto, CA, to meet with Steve Squires, Chief Science Officer, and staff. One major topic of conversation was the OptIPuter project and HP’s involvement. Specifically, we discussed connecting UCSD’s Chiaro router to one being installed at NASA Ames, and jointly developing a distributed next-generation computer.

May 8, 2003. CENIC 2003 conference, Santa Barbara, CA. Greg Hidley gave a presentation on the OptIPuter <http://www.cenic.org/CENIC2003/agenda.htm>.

May 5, 2003. Andrew Chien presented, “OptIPuter: The Project and System Software Research Challenges,” a Distinguished Lecture at the University of California, Santa Barbara.

April 9-10, 2003. Andrew Chien gave the talk, “OptIPuter: Project and Research Issues,” at Hewlett-Packard Laboratories, Palo Alto, California.

April, 2-6, 2003. Debi Kilb (SIO) visited Atul Nayak (UIC/EVL) to collaborate on visual 3D displays pertaining to seismology. She also met with Tom Moher (UIC/EVL) about his Lincoln Elementary School OptIPuter earthquake science project, and numerous other UIC/EVL scientists about volume visualization.

March 20, 2003. Graham Kent (SIO) participated in Career Day at Deportola Middle School in Tierrasanta. He showed over 300 students the latest 3D visualization techniques in Earth Science. Many of the demonstrations were developed for the OptIPuter project. Shown were images from Axial Seamount, Juan de Fuca Ridge and Lake Tahoe.

March 19, 2003. Tom DeFanti, Maxine Brown, Jason Leigh and Luc Renambot of EVL/UIC and Paul Morin of University of Minnesota visited the US Geological Survey (USGS) EROS Center to learn more about their large-scale mapping activities and need for high-resolution visualization displays and high-speed networks. USGS EROS is an OptIPuter affiliate partner.

March 18-21, 2003. NPACI All-hands meeting, San Diego, CA. Rozeanne Steckler and Mike Bailey gave a presentation of the earth science education effort taking place at the UCSD Preuss School as part of the OptIPuter project.

March 10. 2003. Cees de Laat of University of Amsterdam (UvA), an OptIPuter affiliate partner, visited Calient Networks in the Bay Area, California, to discuss their optical switch. UIC subsequently purchased two Calient switches, one for Chicago and one for UvA, for OptIPuter tests.

February 27, 2003. High school mentors and middle school youth from Aquatic Adventures Science Education Foundation, a program that targets under-represented youth who otherwise are not afforded such opportunities, visited the SIO Visualization Center to learn more about plate tectonics and seismology. Debi Kilb (SIO) and Cheryl Peach (BAS) led the students on a virtual tour of the sea floor, demonstrated the effects of geologic activity, and showed students how organisms adapt to a deep sea environment. Shara Fisler, Executive Director of Aquatic Adventures noted, “The tour was so phenomenal! It turned out better than I could have possibly imagined. We all had an amazing time and the level of information was ideal.” On the pre-evaluation students received 11/21. On the post evaluation students received 20/21.

February 26, 2003. The SIO Visualization Center hosted a meeting of the Regional Workbench Consortium [RWBC] <www.regionalworkbench.org>, a collaborative network of university and community-based partners dedicated to enabling sustainable city-region development. Participants included researchers based at UCSD/SIO,


UCSD/SDSC (SALT lab) and UCSD Medical School; researchers from other universities (including Mexico); local think tanks; non-profit agencies; UCSD-TV; urban and regional planners, community activists, and industry representatives.

February 25-27, 2003. NSF Workshop on Cyberinfrastructure Research for Homeland Security, UCSD, San Diego, CA. This Workshop was organized by Ramesh Rao, Larry Smarr and Frieder Sieble (UCSD), with participation from Maxine Brown and Tom DeFanti (UIC), Joe Bannister (USC/ISI), and Dan Blumenthal (UCSB).

February 21, 2003. Steve Briggs, HP business development manager for Sepia, a clustered visualization project, and Steve Hirschfeld, HP salesman, visited the Electronic Visualization Laboratory to meet with Tom DeFanti, Jason Leigh, and other technical people regarding hardware platforms for the OptIPuter.

February 12, 2003. An art exhibit highlighting the powerful synergy between science and art opened at the UCSD Faculty Club. The exhibit, titled Æsthetics of Science, was curated by Alain Cohen and featured visualizations from diverse UCSD science scholars, including the SIO Visualization Center and SDSC. Many of the SIO visualizations were created through OptIPuter funds <www.siovizcenter.ucsd.edu/news_events/2003/02/13th.shtml>.

February 6-7, 2003. OptIPuter All Hands Meeting, UCSD. Agenda and presentations are documented on the OptIPuter website <www.optiputer.net>. This meeting was followed by a meeting of the OptIPuter’s Frontier Advisory Board (FAB), to reflect on what they heard from participants and give the PI and co-PIs feedback on future directions.

February 3-4, 2003. The ON*VECTOR Photonics Workshop was hosted by Larry Smarr of the UCSD California Institute for Telecommunications and Information Technology [Cal-(IT)2] and PI of the NSF-funded OptIPuter project. ON*VECTOR (Optical Networked Virtual Environments for Collaborative Trans-Oceanic Research) is a joint project of NTT Network Innovation Laboratories, University of Tokyo and University of Illinois at Chicago, and managed by Pacific Interface Inc. (PII). OptIPuter principals interested in optical architectures, and guests from the USA, Canada, Europe and Asia, convened to share information on photonics networks.

February 3, 2003. Preuss School UCSD Visualization Center dedication, with presentations by Rozeanne Steckler and Larry Smarr. This is an advanced visualization facility for teaching earth sciences and eventually biomedical imaging, and is the result of OptIPuter funding.

January 26, 2003. Staged in conjunction with the SuperBowl in San Diego ShadowBowl <www.shadowbowl.org> was an community readiness and medical response drill to prepare local San Diego emergency medicine providers with a short burst of “what-ifs” in case the unthinkable occurred. ShadowBowl allowed participating organizations to demonstrate rapid response, on-demand services and expertise for a mass casualty event. SDSU organized the event. SDSU used this venue to transport 2D and 3D volume visualizations from SDSC (Mike Bailey) to SDSU (Frost) using the TeraBurst Networks optical switches at SDSC and SDSU. SDSU displayed the information on the GeoWall to many dignitaries, including local Congressional Representative Susan Davis and most of the leadership of the university. SDSU also looped the connection to SDSC (88 miles round trip) and performed continuous 24-hour/day Quality of Service experiments from January, 2003 to the present. SDSU also sent volume visualization information from its computers through the network back to its display devices across the 88 miles of fiber for numerous demonstrations.

SDSU has done numerous projects to photograph field images and build virtual realities for transport across the network, especially using the Wave3 Sessions software. Images from many of these software products were sent out over an OC-48 link between SDSU and SDSC and then displayed on screens at SDSU during multiple events, such as ShadowBowl and other demonstrations of Homeland Security and “Smart” campus efforts SDSU. In both these types of demonstrations, sensor data were fused into data fusion products and then transmitted to other sites in the US or through wireless networks to local and regional sites in urban and remote centers.

January 29, 2003. Larry Smarr gave a presentation to the EECS Joint Colloquium Distinguished Lecture Series, University of California, Berkeley, on “The OptIPuter Project – Removing Bandwidth as an Obstacle in Data Intensive Sciences.”

January 25, 2003. CENIC Board Meeting, Sacramento, California. Rozeanne Steckler gave an invited presentation on the OptIPuter education project, describing how educational material being developed for the Preuss School will be made available on the web for all schools. (Quicktime movies will be used in lieu of the GeoWall for classrooms that do not have state-of-the-art equipment.)

January 9-10, 2003. NSF ANIR Principal Investigator meeting, Hyatt Regency in Reston, Virginia. Tom DeFanti,


Jason Leigh, Oliver Yu, Maxine Brown (UIC) and Joe Mambretti (NU) attended, representing grants to UIC and NU respectively, but also gave a poster presentation on the OptIPuter. Mambretti participated in the Panel on Optical Networking Research Topics and Directions, and gave a presentation entitled “Advanced Optical Networking Research: The Global Lambda Grid, OMNInet, and StarLight.”

December 13, 2002. Larry Smarr visited the DARPA Microsystems Technology Office and gave the presentation “The OptIPuter Project–Eliminating Bandwidth as a Barrier to Collaboration and Analysis.” (PPT presentation)

December 12, 2002. Larry Smarr visited NSF ANIR to meet with Aubrey Bush, Mari Maeda, Ty Zynati and Tom Greene, and presented the “Year End Report of the NSF OptIPuter ITR Project.” (PPT presentation)

December 10, 2002. American Geophysical Union (AGU) annual conference. UIC/EVL, SIO and USGS had research booths. Orcutt, president of the AGU, reported that SIO’s geosciences and booth were very well received because of the ability to show significant visualizations. Orcutt chaired a session on Visualization with Larry Mayer at the U of New Hampshire. EVL demonstrated WiggleView on the GeoWall (a program for visualizing real-time data from seismometers) at a poster session, and at the IRIS booth. Tim Ahern, program manager for IRIS’ data management group, has agreed to let EVL host a clone of their seismic data servers at Starlight; IRIS is in charge of Earthscope.

December 3-6, 2002. ICAT 2002, University of Tokyo, Japan. Tom DeFanti participated in the Telecommunication, Telimmersion and Telexistence Symposium, and presented the paper “Telecommunication, Teleimmersion and Telexistence with the OptIPuter,” which was subsequently published in the proceedings.

November 18-22, 2002. SC 2002 in Baltimore, MD. Tom DeFanti, Jason Leigh, Bob Grossman, Maxine Brown (UIC) and Eric Frost (SDSU) attended.

• SDSU (Frost) assisted Sun Microsystems demonstrate optical technologies. SDSU continues to work with Sun to acquire one of their new Zulu graphics boxes to connect to the TeraBurst Networks optical system and do distributed collaboration over optical fiber. SDSU receives the Zulu box in August 2003, to be readied for use in a Sun Microsystems’ collaborative event on Sept. 8, 2003. During this same year, SDSU was named as the Sun Microsystems Center for Excellence in Collaborative Visualization.

• Leigh and Grossman conducted high-speed networking demonstrations as part of the research exhibition. • Grossman demonstrated a collaborative project with researchers from Chicago, Ottawa and Amsterdam,

and was awarded the SC’02 High Performance Bandwidth Challenge Award for Best Use of Emerging Infrastructure. Researchers from NCDM, CANARIE (Canada) and SARA (Amsterdam) have been working together over the past year to perform real-time data correlation over lambda networks.

October 6-10, 2002. SDSU (Frost) participated in the Society of Exploration Geophysicists meeting in Salt Lake City, working with the BP Visualization Center from the University of Colorado, Boulder <www.bpvizcenter.com> to show major volume visualization software that they had developed being transported across 80 km of fiber in the convention center between the SGI booth and University of Colorado/Cal-(IT)2 booth. SDSU demonstrated the collaborative interaction of volume visualization of oil fields, seismic cubes and real-time reservoir management software using both an OC-48 connection provided by TeraBurst Networks and the VizServer technology provided by SGI. These two techniques offer a high-end, complete data-transfer capability (TeraBurst Networks) and a low-end image transfer (VizServer) capability. Both technologies are used at SDSU for a variety of applications, using both wired (optical fiber) connectivity to SDSC (TeraBurst) and wireless for field applications (VizServer). SDSU worked with the University of Colorado to demonstrate regional connectivity between visualization centers using oil company software, for which they are the world’s major repository.

October 31, 2002. Joe Mambretti (NU) gave the presentation “Creating a Global Lambda GRID: International Advanced Networking, StarLight, OMINet and CivicNet,” at the GE Alliance Conference, SDSC, UCSD, San Diego, California.

October 25, 2002. Larry Smarr and Greg Hidley (UCSD) and Tom DeFanti (UIC) hosted an OptIPuter Optical Switch Workshop, with presentations by Jamie Munroe (Calient Networks), Tom Myers (TeraBurst Networks),

Faults in blue, person kneeling down to look at oil in red under the layer at waist height–manipulated in stereoscopic 3D across the fiber for first time with Cal-(IT)2/SDSU co-sponorship and help.


Steve Wallach (Chiaro), Kevin Conklin and Mark Housley (Glimmerglass), and Denny Miu (Integrated Micro Machines Inc/IMMI). Agenda and presentations are documented on the OptIPuter website <www.optiputer.net>.

October 18, 2002. Cees de Laat and Leon Gommans (University of Amsterdam) visited UIC/EVL for discussions with Tom DeFanti, Jason Leigh, Oliver Yu, Bob Grossman, Maxine Brown (UIC) and Joe Mambretti (NU) and students about collaborative research, notably the OptIPuter. Peter Clarke of University College London also attended, interested in collaborating with us when UKLight (an experimental 10Gb network between London and Chicago and London and Amsterdam) comes online.

October 14, 2002. UIC/EVL conducted a tour and demonstrations during the Global Grid Forum 6 (GGF6) event. Demonstrations from iGrid 2002 were conducted by Joe Mambretti (NU) and Jason Leigh and Bob Grossman (UIC).

October 11, 2002. UIC/EVL met with Level 3 to discuss long-term plans and collaborations, including OptIPuter. Attending were Tom DeFanti, Jason Leigh, Maxine Brown, Oliver Yu (UIC); Joe Mambretti (NU); and, Jack Waters, Paul Fernes, John Verduzco and Sarah Bleau (Level 3).

October 10, 2002. Greg Hidley (UCSD) and Dan Blumenthal (UCSB) visited Calient Networks.

October 8, 2002. HP in Fort Collins, CO, hosted UCSD/UIC people to discuss the OptIPuter and visualization clusters (Sepia). Attending were Tom DeFanti, Jason Leigh, Alan Verlo, Eric He, Javier Girado (UIC); Greg Hidley, Phil Papadopoulos, Mike Bailey (UCSD); and, Eric Frost (SDSU). Meeting was hosted by Philippe Lindheimer (HP Scaleable Visualization Business Manager) and Steve Hirschfeld (HP Sales Representative), with participation from Dan Nordhues, Kevin Spooner and Jim Kapadia.

August 20, 2002. Phil Papadopoulos (UCSD), Greg Hidley (UCSD) and Linda Winkler (StarLight/Argonne National Laboratory) did an NDA visit to Chiaro Networks in Richardson, TX, to understand what Chiaro is doing, how it relates to OptIPuter, and how UCSD should proceed with them.


2.B. Research Findings 2.B.1. Network and Hardware Infrastructure Findings [Note that Section 2.A covers many research findings but given the NSF reporting structure we split these results between Sections 2.A and 2.B.]

Network Infrastructure All partner sites are either connected or have plans to be connected to the Southern California or Chicago regional optical network (CENIC or I-WIRE).

Network Hardware Infrastructure UCSD acquired a Chiaro router, and UIC has both a Cisco 6509 switch and MEMS switches operational. UIC developed the Photonic Domain Controller (PDC) to control the Calient and Glimmerglass photonic switches. An initial conceptual model for building a photonic computing pipeline has been devised.

Cluster Hardware Infrastructure UCSD is striving to achieve a 10:1 campus bisection bandwidth ratio by November 15, 2003. As of August 2003, UCSD has four “Mod-0” 32-node computing and storage clusters at OptIPuter sites, achieving 8:1 bisection bandwidth, and an 8-node visualization cluster at SIO (2:1 bisection). UIC “Mod 1” evaluations show that the performance of the Itanium2 in bus bandwidth over the IA-32 is almost triple, and able to sustain full rate transmission of 2 GigE simultaneously. However, the system is unable to handle 3 GigE (the target for Mod 2).

Prototype/Evaluate High-Speed NIC Hardware and Driver/Software Stack Evaluations are ongoing. AAA will need to be incorporated into PDC when the Calient is deployed at StarLight. A higher-level resource addressing scheme is needed for accessing systems connected to photonic switches.

2.B.2. Software Architecture Findings Optical Architecture Preliminary indications are that data-intensive applications can significantly benefit by providing them with new methods of dynamic digital communication provisioning, based on advanced optical technologies; e.g., which allow for instantaneous on-demand lightpath provisioning and optimized L2 switched paths. In addition to dynamic provisioning, it is clear that wavelength-based services can be optimized through various methods for advanced scheduling.

LambdaRAM LambdaRAM performance is confirmed to be better than from local disk. LambdaRAM needs to be redesigned as a remote service rather than as daemons spawned on a per application basis.

Security UCI (Goodrich) is developing techniques to efficiently authenticate data structures that represent networks, such as path and connectivity queries, and represent collections of geometric objects, such as intersection and containment queries. This work has application to the authentication of network management systems and geographic information systems. In addition, Goodrich has several schemes for efficiently populating an authenticated dictionary with fresh credentials, allowing many data authors (i.e., sources) to collectively publish information to a common repository, which is then distributed throughout a network to allow for authenticated queries on this information. While the goal of this research is the dissemination of credential status data from multiple credential issuers, applications of this technology also include time stamping of documents, document version integrity control, and multiple-CA certificate revocation management, to name just a few.

Performance Analysis Prophesy will soon be applied to applications in order to evaluate.

Real-Time Execution Environment


UCI (Kim) developed a feasibility demo based on LANs and a TMO support middleware subsystem, for which a paper is currently being written for the IDPT 2003 conference, to take place in December 2003.

High-Performance Transport Protocols: XCP Several engineering and research issues have been identified, which will be investigated in Year 2.

High-Performance Transport Protocols: Quanta/RBUDP and SABUL UIC (Leigh) has found the performance of Quanta/RBUDP to be excellent for large payloads. UIC will continue to use the TeraGrid DTFnet for experiments. Quanta is able to predict throughput for a variety of network conditions and payload sizes.

UIC (Grossman) needs to improve the friendliness of SABUL so that it can be deployed on general use grids and shared LambdaGrids, and needs to improve the interface between network transport and local input/output.

System Software Architecture Traditional layered architecture is not appropriate for LambdaGrids and the OptIPuter project. Open services architecture enables open composition of project research technologies and emerging external technologies. The performance of OptIPuter networks has strong influence on appropriate scheduling and resource management policies and techniques.

2.B.3. Data, Visualization and Collaboration Findings Data UCI’s (Smyth) algorithms for statistical modeling and curve clustering were successfully applied to two different large scientific datasets: extra-tropical cyclones and time-course gene expression data.

UIC (Grossman) achieved a land-speed record using a striped version of SABUL to move earth science data at 2.7 Gbps at iGrid 2002 between two 3-node clusters, one at StarLight and one in Amsterdam. At SC 2002, UIC (Grossman) set a record for merging and integrated distributed data by integrating data at over 5 Gbps from three distributed DSTP sources (a DSTP server in Amsterdam, a DSTP server in Ottawa, and a DSTP server in Chicago).

Multicast UIC/EVL will be testing GlimmerGlass Networks’ photonic multicast option in year 2.

Visualization/Collaboration Tools (General) Optiputer-based collaborative applications will require photonic multicasting to support. Extensible dataset and filter models are necessary for adaptation to custom scientific codes and experimental visual analysis.

SIO (Kilb) used Fledermaus <www.ivs.unb.ca/products/fledermaus/> visualization software, which comes with a freeware viewer that runs on multiple platforms (Mac, IBM, Sun, SGI, Linux). This makes it easy to port the end “scene” files to many locations for many purposes (e.g., research, education, or outreach). These scene files were used to explore global and local seismicity and identify mainshock/aftershock sequences, easily spot artifacts in the data (e.g., erroneous depths), and explore the correlations between seismicity and topography in a new and innovative way. An interesting void of seismicity marking an “X” pattern in the aftershocks from a magnitude 5.1 earthquake in Southern California is being investigated. The visualization also helped determine locations for future seismic instruments, a major cost savings as the previous method involved renting helicopters to fly to the remote locations!

Display-rich environments do improve group awareness in collaborative problem solving. Collaborators tend to treat large tiled displays as single desktops. Existing operating systems are not equipped to deal with this. There are complex usability issues that result from a tradeoff between display size, resolution and proximity to the display.

Note: The tools we are developing now can be extended for ENDfusion – a proposal submitted to the NSF EIN program in May 2003 – by fusing real-time remote sensing data into the pan/zoom of high-resolution maps. OptIPuter partner USGS EROS Data Center is providing UIC/EVL with high-resolution (1/3-meter) data of Chicago in time to do something with it for the OptIPuter Site Visit, September 29, 2003. Ideally, USGS wants to able to be able to zoom and pan over the city and then drop to one-foot elevation on a distinctive architectural


feature, such as the Picasso sculpture in downtown Chicago.

Volume Visualization Tools UIC (Leigh) realized that a scalable GDT/GVU architecture is necessary to channel arbitrarily large data visualization tasks over high-bandwidth connections. Stochastic volume sampling offers viable reduction in transport requirements and viewpoint latency for volumetric time-series browsing.

Memory-to-graphics RAM transfers are improving over time but are lagging too far behind visualization requirements of domain science applications. An adaptive visualization system is needed to allow OptIPuter visualizations to work flexibly on a variety of rendering systems and displays.

SIO (Reif) found it an easy task to analyze the Earth’s convection by imaging slow/fast velocities in the Earth’s mantle using 3D visualizations instead of 2D cross-sections.

The HP Itaniums performed well for disk-to-memory and network-to-memory transfers but extremely poorly for memory-to-graphics transfers.

2.B.4. Applications and Education Findings Application Codes Running on OptIPuter Clusters UCSD/NCMIR, because of participation in the OptIPuter project with access to experts in networking, storage and visualization, has:

• Made data-rate throughput improvements on a two-photon confocal scope • Using 2 IBM T221 9-MegaPixel displays to explore high-resolution datasets; doing stereo to view volumes

using a modified Wheatstone stereoscope with the two T221s • Using a Geowall to collaboratively view volumes in 3D • Is connected to the Chiaro router on the OptIPuter infrastructure, including systems in the lab (3100

electron microscope, two Photon confocal microscopes

UCSD/Preuss School Real-time 3D visualization and ready, high-speed access to data can have a significant impact on learning in Earth science. Teacher education is essential for the success of the overall project.

UCSD/SIO maintains and operates a GeoWall, which makes it feasible to develop products that can then be directly ported to the GeoWall system at the UCSD/Preuss School.

SIO researchers published the paper “The Visualization Center at Scripps Institution of Oceanography: Education & Outreach” in a special Education and Outreach issue of the journal Seismological Research Letters describing the OptIPuter project.

Lincoln Elementary School in Oak Park, IL RoomQuake, embedded within a larger curriculum unit on earth structures currently in development with teachers, will be introduced in fifth grade classrooms during the fall semester. Quantitative and qualitative assessments will be employed to characterize student learning of earth science and mathematics content.

UCSD/SIO Outreach Activities At the SIO Teacher Workshop, teachers found 3D interactive visualizations to be an outstanding teaching tool. At the Regional Workbench Consortium (RWBC) meeting, the audience included those with a common interest in linking science and policy for better regional planning, and was extremely receptive to using compelling 3D visualizations.

Being portable, the GeoWall is an excellent teaching tool that can be used by a single educator in multiple locations.

Annual OptIPuter Meeting There was considerable interest in OptIPuter-sponsored meetings, and several interested parties have been asked to be on mailing lists so they can attend future meetings/workshops.


2.C. Research Training There is clearly a critical mass of professors and students at nine institutions (UCSD, NU, SDSU, TAMU, UCI, UIC, and USC as well as USGS EDC and University of Amsterdam), involved with the OptIPuter, as indicated by this annual report, facilitating greater advances than a single-investigator effort would afford. Moreover, the project is local, regional, national and international in scope. As noted in Section 2.B (Research Findings), all the people working on OptIPuter-related projects are involved in furthering the research, taking a “systems-wide” view of the project, which is clearly interdisciplinary in nature. It is our hope that our students will benefit most, and be in high demand by the commercial sector for R&D jobs when they graduate. After only one year of support, the OptIPuter has gained international recognition as a major driving force for the development of LambdaGrids. Interest comes from not only computer scientists and network engineers, but also discipline scientists who are facing unprecedented challenges dealing with large datasets in the 21st century. The OptIPuter involves academicians, graduate students, undergraduates, K-12 teachers and students, and industry. Already several research papers have published and presentations are being given professional conferences. 2.D. Education/Outreach The OptIPuter’s primary education and outreach activities include web documentation, journal articles, and conference presentations and demonstrations. In addition to participation at major computer conferences, such as IEEE ACM/IEEE Supercomputing, team members are active at networking conferences, including Internet2 meetings, CENIC meetings, and international conferences and workshops (NORDUnet3, 3rd annual international LambdaGrid Workshop). The OptIPuter is receiving a great deal of media attention, and there have been a number of news articles describing it, which can be found on our website <http://www.optiputer.net/news/index.html>. We also provide PowerPoint slides and other promotional material to collaborators to give presentations at education conferences, government briefings, etc.


3. Publications and Products

3.A. Journals/Papers Maxine Brown (guest editor), “Blueprint for the Future of High-Performance Networking,” Special issue, Communications of the ACM, November 2003, to appear. Tom DeFanti, Cees de Laat, Joe Mambretti, Kees Neggers, and Bill St. Arnaud, “TransLight: A Global-Scale LambdaGrid for e-Science,” Communications of the ACM, November 2003, to appear. Ted Faber, Aaron Falk, Joe Bannister, Andrew Chien, Robert Grossman, Jason Leigh, “Transport Protocols for High Performance: Whither TCP?,” Communications of the ACM, November 2003, to appear. Larry Smarr, Andrew A. Chien, Tom DeFanti, Jason Leigh, Philip M. Papadopoulos, “The OptIPuter,” Communications of the ACM, November 2003, to appear. Ian Foster and Robert Grossman, “Data Integration in a Bandwidth-Rich World,” Communications of the ACM, November 2003, to appear. Harvey Newman, Mark Ellisman, John Orcutt, “Data-Intensive e-Science Frontier Research in the Coming Decade,” Communications of the ACM, November 2003, to appear. Xin Liu and Andrew Chien, “Traffic-based Load Balance for Scalable Network Emulation,” SC 2003, Phoenix, Arizona, November 2003, to appear. D. Chudova, S. Gaffney, E. Mjolsness and P. Smyth, “Translation-invariant mixture models for curve-clustering,” Proceedings of the Ninth ACM International Conference on Knowledge Discovery and Data Mining (SIGKDD), Washington DC, August, to appear. Robert Grossman, Donald Hamelberg, Pavan Kasturi, and Bing Liu, “Experimental Studies of the Universal Chemical Key (UCK) Algorithm on the NCI Database of Chemical Compounds,” Proceedings of the IEEE Computer Society Bioinformatics Conference, CSB 2003, to appear. D. Chudova, S. Gaffney and P. Smyth, “Probabilistic models for joint clustering and time-warping of multidimensional curves,” Proceedings of the 19th Conference on Uncertainty in Artificial Intelligence, Acapulco, Mexico, August 2003; Morgan Kaufmann, San Francisco, CA, to appear. Cees de Laat, Maxine Brown and Tom DeFanti, (guest editors), Journal of Future Generation Computer Systems (FGCS), Elsevier Science Press, Volume 19, Issue 6, August 2003, to appear. Rajvikram Singh, Jason Leigh, Thomas A. DeFanti, “TeraVision: A High Resolution Graphics Streaming Device for Amplified Collaboration Environments,” Journal of Future Generation Computer Systems (FGCS), Elsevier Science Press, Volume 19, Issue 6, August 2003, to appear. E. He, J. Alimohideen, J. Eliason, N. Krishnaprasad, J. Leigh, O. Yu, T. A. DeFanti, “QUANTA: A Tooklit for High Performance Data Delivery over Photonic Networks,” Journal of Future Generation Computer Systems (FGCS), Elsevier Science Press, Volume 19, Issue 6, August 2003, to appear. Shalini Venkataraman, Jason Leigh, Tom Coffin, “Kites Flying In and Out of Space - Distributed Physically-based Art on the GRID,” Journal of Future Generation Computer Systems (FGCS), Elsevier Science Press, Volume 19, Issue 6, August 2003, to appear. Charles Zhang, Jason Leigh, Thomas A. DeFanti, Marco Mazzucco, Robert Grossman, “TeraScope: Distributed Visual Data Mining of Terascale Data Sets Over Photonic Networks,” Journal of Future Generation Computer Systems (FGCS), Elsevier Science Press, Volume 19, Issue 6, August 2003, to appear.


Robert L. Grossman, Yunhong Gu, Dave Hanley, Xinwei Hong, Dave Lillethun, Jorge Levera, Joe Mambretti, Marco Mazzucco, and Jeremy Weinberger, “Experimental Studies Using Photonic Data Services at IGrid 2002,” Journal of Future Generation Computer Systems (FGCS), Elsevier Science Press, Volume 19, Issue 6, August 2003, to appear. J. Mambretti, J. Weinberger, J. Chen, E. Bacon, F. Yeh, D. Lillethun, R. Grossman, Y. Gu, M. Mazzuco, “The Photonic TeraStream: Enabling Next Generation Applications Through Intelligent Optical Networking at iGrid 2002,” Journal of Future Generation Computer Systems (FGCS), Elsevier Science Press, Volume 19, Issue 6, August 2003, to appear. T. Moher, X. Ding, J. Wiley, D. Conmy, S. Hussain, P. Singh and V.l Srinivasan, “Combining Handhelds with a Whole-Class Display to Support the Learning of Scientific Control,” ACM Conference on Human Factors in Computing Systems Extended Abstracts, Ft. Lauderdale, FL, April 2003, pp. 882-883. D. Kilb, C.S. Keen, R.L. Newman, G.M. Kent, D.T. Sandwell, F.L. Vernon, C.L. Johnson, J.A. Orcutt, “The Visualization Center at Scripps Institution of Oceanography: Education & Outreach,” Seis. Res. Lett., 2003, in press. M.T. Goodrich, M. Shin, R. Tamassia and W.H. Winsborough, “Authenticated Dictionaries for Fresh Attribute Credentials,” 1st Int. Conference on Trust Management (iTrust), LNCS 2692, Springer-Verlag, 2003, pp. 332-347. M.T. Goodrich, R. Tamassia, N. Triandopoulos and R. Cohen, “Authenticated Data Structures for Graph and Geometric Searching,” Proc. RSA Conference – Cryptographers’ Track (CT-RSA), LNCS 2612, Springer-Verlag, 2003, pp. 295-313. Tom DeFanti, Maxine Brown, Jason Leigh, Oliver Yu, Eric He, J. Mambretti, D. Lillethun and J. Weinberger, “Optical Switching Middleware for the OptIPuter,” Institute of Electronics, Information and Communication Engineers (IEICE) Transactions on Communications (special issue on Photonic IP Network Technologies for Next Generation Broadband Access), Japan, to appear Marco Mazzucco, Asvin Ananthanarayan, Robert L. Grossman, Jorge Levera, and Gokulnath Bhagavantha Rao, Merging Multiple Data Streams on Common Keys over High Performance Networks, Proceedings of Supercomputing 02. Jason Leigh, Luc Renambot, Thomas A. DeFanti, Maxine Brown, Eric He, Naveen Krishnaprasad, Javid M Alimohideen Meerasa, Atul Nayak, Kyoung Park, Rajvikram Singh, Shalini Venkataraman, Chong (Charles) Zhang, Drake Livingston, Michael McLaughlin, “An Experimental OptIPuter Architecture for Data-Intensive Collaboration Visualization,” Third Annual Workshop on Advanced Collaborative Environments, co-located at the Twelfth IEEE International Symposium on High Performance Distributed Computing (HPDC 12) and Global Grid Forum 8, Seattle, Washington, June 22, 2003 <http://www.mcs.anl.gov/fl/events/wace2003/index.html> Kyoung Park, Luc Renambot, Jason Leigh, Andrew Johnson, “The Impact of Display-rich Environments for Enhancing Task Parallelism and Group Awareness in Advanced Collaboration Environments,” Third Annual Workshop on Advanced Collaborative Environments, co-located at the Twelfth IEEE International Symposium on High Performance Distributed Computing (HPDC 12) and Global Grid Forum 8, Seattle, Washington, June 22, 2003 <http://www.mcs.anl.gov/fl/events/wace2003/index.html> David Lee, Enhancing Electron and Light Microscopy Functionality with the IBM T221 Monitor: Current Applications and Future Directions, IBM T221-NCMIR White Paper, 03-ILS-090, V1.0, May 9, 2003. Thomas A. DeFanti, Jason Leigh, Maxine D. Brown, Daniel J. Sandin, Oliver Yu, Chong Zhang, Rajvikram Singh, Eric He, Javid Alimohideen, Naveen K. Krishnaprasad, Robert Grossman, Marco Mazzucco, Larry Smarr, Mark Ellisman, Phil Papadopoulos, Andrew Chien, John Orcutt, “Teleimmersion and Visualization with the OptIPuter,” Proceedings of the 12th International Conference on Artificial Reality and Telexistence (ICAT 2002), The University of Tokyo, Japan, December 3-6, 2002 <www.ic-at.org>.


3.B. Books/Publications Thomas A. DeFanti, Jason Leigh, Maxine D. Brown, Daniel J. Sandin, Oliver Yu, Chong Zhang, Rajvikram Singh, Eric He, Javid Alimohideen, Naveen K. Krishnaprasad, Robert Grossman, Marco Mazzucco, Larry Smarr, Mark Ellisman, Phil Papadopoulos, Andrew Chien, John Orcutt, “Teleimmersion and Visualization with the OptIPuter,” Telecommunication, Teleimmersion and Telexistence, (Susumu Tachi, editor), Ohmsha/IOS Press, 2003, pp. 25-71. 3.C. Internet Dissemination www.optiputer.net 3.D. Other Specific Products None at this time.


4. Contributions

4.A. Contributions within Discipline The OptIPuter team’s mission is to enable scientists to explore very large remote data objects in a novel interactive and collaborative fashion, which is impossible on today’s shared internet. This involves the design, development and implementation of the OptIPuter – a tightly-integrated cluster of computational, storage and visualization resources – linked over LambdaGrids, parallel dedicated optical networks across campus, metro, national, and international scales. The OptIPuter project aims at the re-optimization of the entire Grid stack of software abstractions, learning how to “waste” bandwidth and storage in order to conserve “scarce” computing in this new world of inverted values. A major outcome of this research will be the development of advanced middleware and network management tools and techniques to optimize transmissions so distance-dependent delays are the only major variable. The group of computer scientists and network engineers assembled represent many of this nation’s high-performance computing and communications leaders. Potential collaborators (e.g., NASA, DOE) are seeking out the group’s expertise in order to jointly develop a common framework for optimizing optically linked clusters over LambdaGrids. 4.B. Contributions to Other Disciplines The OptIPuter's mission is to enable collaborating scientists to interactively explore massive amounts of previously uncorrelated data by developing a radical new architecture for a number of this decade’s e-science shared information technology facilities. The OptIPuter’s broad multidisciplinary team is conducting large-scale, application-driven system experiments with two data-intensive e-science efforts to ensure a useful and usable OptIPuter design: EarthScope, funded by the National Science Foundation (NSF), and the Biomedical Informatics Research Network (BIRN) funded by the National Institutes of Health (NIH). These application drivers have many multi-gigabyte-sized individual data objects -- gigazone seismic images of the East Pacific Rise Magma chamber and 100 megapixel montages of rat cerebellum microscopy images -- which are very large volumetric data objects with visualizations so big they exceed the capacity of the current shared Internet and laptop displays. There is interest from other user communities (such as High Energy and Nuclear Physics) to take advantage of the architectures developed for the OptIPuter. 4.C. Contributions to Education and Human Resources The OptIPuter supports 25 senior faculty and staff, some part-time staff, and 14 graduate students, spanning 7 institutions. Non-funded faculty, staff and students from other university departments, two affiliate institutions, and several industrial partners also work tirelessly on OptIPuter research. Our initial efforts in the K-12 public schools (Preuss School in San Diego and Lincoln School in Oak Park, IL), are also engaging teachers and school children. We are building a worldwide community eager for new methodologies for the real-time exploration of e-science. 4.D. Contributions to Resources for Science and Technology The OptIPuter exploits a new world in which the central architectural element is optical networking, not computers - creating “supernetworks.” This paradigm shift requires large-scale applications-driven, system experiments and a broad multidisciplinary team to understand and develop innovative solutions for a “LambdaGrid” world. The goal of this new architecture is to enable scientists who are generating terabytes and petabytes of data to interactively visualize, analyze, and correlate their data from multiple storage sites connected to optical networks. 4.E. Contributions Beyond Science and Engineering Beyond servicing the scientific and engineering research communities, the OptIPuter can be an enabling technology for broader societal needs, including emergency response, homeland security, health services, and science education.


5. Special Requirements

5.A. Objectives and Scope Our scope of work has not changed (see Section 6: OptIPuter FY2004 Program Plan). 5.B. Special Reporting Requirements UCSD is honoring its commitment of matching funds as originally proposed and budgeted; no deviation is reported. 5.C. Unobligated Funds N/A. 5.D. Animals, Biohazards, Human Subjects No.


6. OptIPuter FY2004 Program Plan (October 1, 2003-September 30, 2004)

6.A. Requested Modifications to Cooperative Agreement We wish to modify the list of Key Personnel listed in Section G of the original OptIPuter Cooperative Agreement, dated September 4, 2002.

• We wish to add the names of the co-PIs, software architect and project manager to the Key Personnel list.

• We wish to update the list of PIs of the subaward institutions, making two changes. First, General Atomics (GA) is no longer a subcontractor; GA previously managed the UCSD San Diego Supercomputer Center (SDSC), but that role has now been assumed by UCSD, and GA PI Rozeanne Steckler has become a UCSD employee. Second, Valerie Taylor left NU to go to TAMU; Joe Mambretti has become PI of the NU subaward and Taylor is PI of a TAMU award, starting year 2, so she can continue to work on the OptIPuter.

We therefore wish to amend the Cooperative Agreement to contain the following Key Personnel. These people are essential to the work on the project:

Larry Smarr, PI, University of California San Diego (UCSD) Mark Ellisman, OptIPuter co-PI, UCSD Philip Papadopoulos, OptIPuter co-PI, UCSD Tom DeFanti, OptIPuter co-PI and PI, University of Illinois at Chicago (UIC) Jason Leigh, OptIPuter co-PI, UIC Eric Frost, PI, San Diego State University Michael Goodrich PI, University of California–Irvine Joe Bannister, PI, University of Southern California Valerie Taylor, PI, Texas A&M University Joe Mambretti, PI, Northwestern University Andrew Chien, OptIPuter System Software Architect, UCSD Maxine Brown, OptIPuter project manager, UIC

6.B. Year 2 Milestones Our Year 2 Milestone, as described in the OptIPuter Cooperative Agreement:

All Southern California OptIPuter partners (UCSD, UCI, USC and SDSU) will have at least one fully functioning OptIPuter node(s) connected to California’s CENIC experimental network. Chicago partners (UIC and NU) will have OptIPuter nodes connected to I-WIRE. The nodes will all be based on UIC/NU’s “Mod 1” work in Year 1.

6.C. Network and Hardware Infrastructure Activities Year 2 tasks: UCSD had hoped to install a DWDM core system on campus and interconnecting the Southern California OptIPuter sites; however, this activity may overlap with year 3, given the limited capital money in the budget. USC, UCI and SDSU, in addition to UCSD, will install OptIPuter nodes and link them to the CENIC dark fiber experimental network. UIC, however, will deploy OptIPuter nodes connected via I-WIRE DWDM and other dedicated lambdas; the nodes will all be based on UIC’s “Mod 1” work in Year 1.

UIC, with input from the application teams, will begin design work on OptIPuter “Mod 2” using 64-bit processors and upgraded storage, visualization and optical networking components. UIC and NU will continue work with metro/national/international partners through StarLight to implement connections for the wide area. StarLight plans to install a large aggregation switch to connect national and international research networks. UIC will work with NU and UCSD to measure if a 4:1 local bisection bandwidth can be achieved.

Network Infrastructure UCSD will continue to work with its campus and CENIC to connect all OptIPuter sites in southern California. UIC will continue to build up the facilities at StarLight to accept several new 10Gb links, notably from Asia, the UK and CERN.

IPv6: Tom Hutton (UCSD/SDSC) is being given one month of funding to develop an IPv6 OptIPuter testbed


between UCSD and NASA Ames.

Network Hardware Infrastructure Based on anticipated equipment acquisition goals, the following Southern California activities need to be accomplished:

• Purchase and install Chiaro GigE upgrades • Purchase and install campus fiber upgrades to key locations • Integrate 8 additional OptIPuter endpoints into Chiaro network • Extend the Chiaro network to SDSU, UCI and USC • Work towards 10GigE integration (whether point-to-point, switch, or routed, needs to be determined) • Work toward DWDM integration

At the StarLight facility in Chicago, UIC will install a Force10 electronic switch and a 128x128 optical switch to connect GigE and 10GigE metro, regional, national and international research networks and lambdas.

Cluster Hardware Infrastructure Based on anticipated equipment acquisition goals, the following Southern California activities need to be accomplished:

• Complete installation of a visualization cluster at UCSD/SIO • Purchase and install four general-purpose IA-32 clusters at key campus sites (compute and storage);

clusters will all have PCI-X (100 MHz, 1Gbps). Note: Use of InfiniBand is dependent on emerging company positions/products and on result of IBM equipment proposal.

• Expand an IA-32 system to 16- or 32-way. • Connect one or two campus sites with 32-node clusters at 10GigE (3:1 campus bisection ratio) • Integrate the SIO/SDSU Panoram systems into the OptIPuter network • Install and configure a variety of OptIPuter management workstations (work has already begun): Dell IA32

and HP IA64 workstations, integrated with the Chiaro network, and equipped with National Middleware Initiative (NMI) software capabilities, OptIPuter accounts, shared storage, revision control and other general support services

UCSD (Papadopoulos) is developing a storewidth evaluation platform. Some clusters will have 16 nodes with 4-6 drives each. UCSD is planning on 25% of cluster platforms to be high-bandwidth storage nodes.

UIC will evaluate a broad range of new 64-bit architectures (AMD and IBM).

Prototype/Evaluate High-Speed NIC Hardware and Driver/Software Stack UCSD’s (Papadopoulos) dynamic credit scheme design will be dependent on the availability of InfiniBand hardware.

6.D. Software Architecture Activities Year 2 tasks: UIC will work with NU to develop optical network control technologies, incorporating network performance and protocol parameters. The first-generation LambdaRAM will be designed and implemented. USC will begin designing Lambda Overlay Protocols and the interface between the cluster communication fabric and the wide-area optical network. Design of a transparent (MAN/WAN) lambda ↔ OS-bypass (local cluster communication) messaging transport framework will be undertaken. UCSD will collaborate with UCI to begin exploring security models using packet-level redundancy and multi-pathing. The first-generation OptIPuter System Software will be released to partner sites for application development and evaluation. UCI will begin design of the Time-Triggered Message-Triggered Object (TMO) programming framework for the OptIPuter. TAMU will include network protocols and middleware in its performance analysis work.

Optical Architecture The OptIPuter Optical Architecture team will refine their optical backplane architecture. They will evaluate techniques for optical control and switching. AAA will be developed by University of Amsterdam (an OptIPuter partner).


LambdaRAM UIC will design a second prototype of LambdaRAM.

Security The OptIPuter Security team will develop techniques to increase security after intrusion insider attacks, and develop techniques for packet-level redundancy and multi-pathing.

Performance Analysis TAMU (Taylor) will instrument applications and refine Prophesy analytical models.

Real-Time Execution Environment UCI (Kim) will develop a prototype implementation of a real-time object (TMO) support middleware subsystem on Linux platforms, including cluster types such as the OptIPuter visualization cluster being designed at UIC and the UCI cluster (Jenks), which were connected to the OptIPuter network in Year One. He will refine and integrate the TMO support middleware subsystem model into the overall OptIPuter middleware system software architecture being developed.

High-Performance Transport Protocols: XCP USC (Bannister, et. al.) will document XCP’s design, develop a protocol specification, and publish XCP header format and other protocol parameters as Internet Drafts.

They will benchmark XCP performance using BSD end-systems and PC routers. They will compare XCP to TCP to determine how fast XCP will go on a PC router. They will consider the effects of XCP on TCP and other background traffic.

They will use a prototype implementation to explore research issues, such as: How does XCP behave when there are L2 queues? How does XCP behave over variable bandwidth links?

They will continue exploration of router implementation constraints in protocol design; e.g., header format and number representation.

They will disseminate XCP into OptIPuter. The XCP PC-router design will be made available to OptIPuter collaborators. USC will help OptIPuter applications migrate to XCP through creating instructions on how to modify code and (possibly) writing wrapper/shim software to adapt TCP binaries to use XCP.

They will collaborate with router vendors to bring about XCP router implementations.

High-Performance Transport Protocols: Quanta/RBUDP and SABUL Work on Quanta/RBUDP and SABUL will continue.

System Software Architecture UCSD (Chien) plans to:

• Prototype Distributed Virtual Computer alternatives on OptIPuter prototypes • Prototype novel Group Transport Protocols (GTP) and demonstrate on OptIPuter prototypes • Do detailed studies of resource management in OptIPuter systems using the MicroGrid.

6.E. Data, Visualization and Collaboration Activities Year 2 tasks: UIC, UCSD and SDSU will deploy data management, visualization and collaboration software systems developed in Year 1 to OptIPuter application teams, both for testing and for feedback. UIC will continue to enhance computing and visualization cluster hardware/software to achieve very-large-field real-time volume rendering. UIC, UCSD and UCI will integrate the data middleware and high-level data services into an OptIPuter application and begin using the data middleware to develop data-mining algorithms for clustering data streaming over the OptIPuter. In addition, high-level software abstractions for remote data retrieval and for accessing bundled computing resources as high-level computational units or peripherals will be studied. Multi-site synchronous and near-synchronous visualization will be demonstrated.


Data UCSD will deploy a large storage-intensive cluster if the IBM SURS grant is funded in December 2003. This was not anticipated in the original OptIPuter proposal and if funded will open up a major new Storewidth research front for the OptIPuter team.

UIC (Grossman) will achieve data transport at rates of 5 Gbps, improve the local input/output interfaces of his DSTP servers, and begin development of wide-area storage and database services. He will investigate streaming algorithms for clustering dimensions 2-3.

UCI (Smyth) will develop new statistical modeling and clustering algorithms capable of handling high-bandwidth, spatio-temporal scientific data streams, which allow interactive visualization and detection of clusters, outliers, and other structural properties of streaming datasets.

Multicast UIC/EVL will be testing GlimmerGlass Networks’ photonic multicast option.

Visualization/Collaboration Tools (General) Further evaluation by UIC/EVL and UCSD/SIO of the effectiveness of seismoglyphs is needed.

UCSD/SIO needs to continue development of visualization techniques for point/line/topography, similar to Fledermaus-based “scenes,” to address other geophysical problems. Atul Nayak, funded by the OptIPuter will develop a set of visualization tools using the IBM visualization cluster purchased late in Year One for SIO.

Collaborative user interface design for multi-site synchronous and near-synchronous visualizations needs to be conducted between EVL and UCSD.

UCSD/CRCA (Brown) \ is upgrading his multi-user, interactive graphic environment authoring toolkit (SceneEditor) with OptIPuter-based applications in mind

SDSU will help university and corporate partners connect visualization centers over high-speed networks and will help data fusion and applications, using both wireless and optical communication, especially with IPv6 components.

Volume Visualization Tools UIC will continue to benchmark potential OptIPuter visualization hardware, notably Itanium, Opteron and IBM G5.

UIC (Leigh) and USC (Thiébaux) will begin development of a prototype adaptive visualization pipeline to support both high-resolution digital montages and volume visualization. They will develop a performance monitoring capability into every aspect of the OptIPuter visualization pipeline model to allow adaptability. With respect to GDT/GVU, they will implement basic performance instrumentation as part of the GDTfoundation layer; will implement GVU instrumentation for visualization performance evaluation; will complete deployable pipeline utility with basic filters and GUI control/monitoring; will implement example VTK extension (isosurfaces); will implement remote frustum culling for tiled display optimization; and, will apply utility to multivariate earthquake simulation volumes.

UCSD/SIO will map out the pros/cons of each visualization package and refine data transfer methods.

UCSD/SIO will extend its suite of 3D seismic volume visualization techniques to include multi-attribute imaging (e.g., two volumes) to investigate amplitude variation with offset (AVO) properties of the melt sill reflector, including an additional layer representing the multi-beam derived seafloor. SIO hopes to have this software also run on Mac OS X to enable sharing of datasets with the Ocean Sciences community.

6.F. Applications and Education Activities Year 2 tasks: UCSD and SDSU will conduct full-scale visualization and collaboration trials using sample geoscience and bioscience datasets over updated OptIPuter testbeds in order to identify key system bottlenecks and usability problems with the software developed in Year 1. Education efforts will continue with the Preuss School and the Lincoln Elementary School in Chicago, testing first-draft modules in the experimental section of its core curriculum, involving grade-school children in observational science activities. The Sixth College will select a group of students, from its distinctive Honors Students, to join OptIPuter application teams and work on


the OptIPuter node located within the College. In addition, an annual OptIPuter meeting/workshop will be held, with some days dedicated to OptIPuter partners and some days open to the wider e-Science and high-performance computing and communications communities.

Application Codes Running on OptIPuter Clusters SDSU will continue to work with UCSD/SDSC (Mike Bailey) and others at UCSD/SIO to develop content and applications across the optical connectivity between the two sites as well as further develop connections to other universities both in US and internationally. This will involve large datasets from both satellite imagery and seismic cubes. Seismic data cubes, such as the Beta field in the offshore Long Beach area, which SDSU provided to Mike Bailey, will be acquired where possible and satellite imagery over the top of the fields will be incorporated into the visualizations. SDSU will also work to develop a high-resolution image of the San Diego county region for use by first responders during crisis management situations.

UCSD application teams will continue to provide sample datasets to technologists.

UCSD/NCMIR goals are to expand its testbed to include additional UCSD cluster computing sites, and to introduce Infiniband interconnect technologies (upon evaluation). NCMIR wants to enhance its computing and visualization cluster hardware/software to achieve very-large-field real-time volume rendering.

UCSD/Preuss School UCSD/SDSC (Steckler) will harden the visualization facility; i.e., it will go from a pilot facility to one easily used by all teachers in multiple classrooms. Emphasis will be on using the OptIPuter network to bring data to Preuss for analysis and visualization by the classes. Steckler will start exploring the use of the OptIPuter resources for other areas outside of earth science.

UCSD/SIO (Kilb) will continue to provide Preuss School with educational material.

UCSD/Sixth College The Sixth College will select a group of students, from its distinctive Honors Students, to join OptIPuter application teams and work on the OptIPuter node located within the College.

UCSD/SIO (Kilb) will work with UCSD (Steckler, Wienhausen) to bring the education programs similar to what was done in year 1 at the Preuss School to the UCSD Sixth College.

Lincoln Elementary School in Oak Park, IL During Year 2, UIC (Moher) will (a) finalize, deploy and assess the effectiveness of the RoomQuake system that was developed during Year 1 in collaboration with UCSD/SIO (Kilb), and (b) develop of a new application, RoomBrain. These applications are part of a series of RoomWare systems designed to map scientific phenomena into/onto the physical space of the classroom. Moher’s working hypothesis is that the classroom holds a special place in the minds of students and teachers – apart from their bedroom, it is likely the place that a child spends most of his or her time. By using the physical space of the classroom as a “stage” populated with props and a “thin layer of technology”, he hope to provide settings in which elementary students can “play-act” their roles as scientists in “participatory simulations.” In these settings, thin clients (e.g., PDAs, tablet displays) and large shared displays serve as “activity points” for spontaneously aggregated groups of students.

RoomQuake will be introduced in fifth grade classrooms during the fall semester as part of a planned district-mandated curriculum unit on earth structures, plate tectonics, and dynamics (esp. earthquakes). Midwestern students rarely have direct experience with earthquakes; the goal of RoomQuake is to provide a more visceral experience than simply working from maps. A number of interesting learning issues will be explored through quantitative and qualitative assessment, including (a) transfer of classroom-centered learning to broader domain knowledge applicable at a global level, (b) effectiveness of asynchronous (with respect to conventional schedule) activity structure, (c) understanding of need for, and process of, mathematical triangulation and applied use of Pythagorean theorem, among others. Moher will employ a quasi-experimental research design, using two classes of fifth graders at Lincoln school.

RoomBrain will be developed while working with OptIPuter bio scientists to identify accessible phenomena; ideas currently under consideration include neuron-firing rules (simple Boolean algebras), perception (e.g., olfactory sensing), or drug education (impact of drug, tobacco, and alcohol usage on brain centers). The Statement of Work


for Year 2 calls for UIC to focus on “observational sciences,” and RoomQuake directly supports this goal. With the RoomBrain system, Moher expects to introduce control affordances that will permit students to actively manipulate system variables and conduct experiments; recent studies he has conducted appear to indicate that manipulative affordances induce a more active stance toward scientific investigation among elementary school children.

UCSD/SIO Outreach Activities SIO hopes to hold another teacher workshop at the SIO Visualization Center. SIO will work in tandem with the Regional Workbench Consortium (RWBC), as needed. SIO will continue development of 3D visualizations for use by members of the GeoWall Consortium in educational venues.

Annual OptIPuter meeting Our second year annual OptIPuter meeting is scheduled for January 14-16. All partners and interested parties will be invited.


7. OptIPuter FY2003 Expenses (Year 1)

7.A. FY2003 Expense Justification

7.A.1. Introduction The OptIPuter award was 90% of the original requested amount; however, annual funding was not consistently allocated throughout the five years of the grant, and was below the norm in Year 1 and above the norm in Year 2. While we believe overall goals can be achieved over the lifetime of the award, some of the proposed deliverables were pushed out to later years. The Year 1 budget of $1,910,000 was 64% of the original first-year request. Allocations to participating sites were as follows:

UCSD* $1,015,000 UIC $575,000 NU** $60,000 UCI $100,000 USC $120,000 SDSU $40,000

* In our ITR proposal, Rozeanne Steckler was listed as PI of the General Atomics (GA) subaward. Steckler works at the UCSD San Diego Supercomputer Center (SDSC) and was, until recently, an employee of General Atomics. As of November 2002, General Atomics’ employees who worked at SDSC became UCSD employees. Because OptIPuter funding to GA was only for a portion of Steckler’s salary, and since she is now a UCSD/SDSC employee, the subaward funds have been added to the UCSD budget.

** In January 2003, Valerie Taylor left NU to become head of the Computer Science Department at Texas A&M University (TAMU). Joe Mambretti replaced Taylor as PI of the NU subaward. Because Larry Smarr approved Taylor’s continuing to work on the OptIPuter, NU subcontracted funds to TAMU for Year 1. In Year 2, UCSD will make a separate subaward directly with TAMU for OptIPuter research. (Note: For the duration of the OptIPuter award, NU and TAMU will divide amongst themselves the amount originally slated to go to NU.)

7.A.2. UCSD FY2003 Expense Justification UCSD repurposed some first-year services, graduate student research positions (GSRs) and staff funds to address project goals. Due to start-up timing issues related to academic year constraints, and difficulty locating and hiring specialized technical talent required by the project, the UCSD OptIPuter team was unable to hire the proposed number of graduate students and academic staff professionals as quickly as initially hoped. For Year 1, UCSD focused on OptIPuter networking infrastructure deployment on campus and accelerated applications and middleware development efforts, by:

• Providing Andrew Chien with increased GSR support, and campus fiber connections to help establish a CSE OptIPuter node in support of OptIPuter middleware and applications activities

• Providing Mark Ellisman with staff support get the BIRN electronic microscopy OptIPuter node on-line and to support biomedical application efforts.

• Building up GIS and Earth Sciences applications support by adding student support for John Orcutt and summer salary for Mike Bailey.

• Building up the OptIPuter team’s collection of experimental end nodes by purchasing additional cluster systems, some additional router GigE network interfaces, and increasing the summer salary of Phil Papadopoulos, who oversees the infrastructure architecture.

• Supporting security research activities by providing Sid Karin with one-month of summer salary.

Smarr, Papaopoulos, Chien and Steckler received salaries. Salaries initially requested for Ellisman, Orcutt and Hidley were instead used to pay for members of their staff working on the OptIPuter project. UCSD hired two part-time professional staff people and a total of 6 graduate students. Equipment expenses totaled ~$336,000; clusters, networking components, and partial payment for a Chiaro router (two installments out of four) were made.

To set up subaward accounts, UCSD charged overhead on the first $25,000 each to UIC, USC, NU and SDSU (first year only).

The UCSD campus provided the following in support of OptIPuter:


• Campus cost-share funds, used to purchase cluster equipment ($45,746) in Year 1 • Campus-allocated dark fiber and repurposed existing fiber for OptIPuter ($125,000 ) • New fiber to help build out the OptIPuter dark-fiber network on campus ($49,000 from campus networking

funds and $87,187 from the VC Academic Affairs) • Juniper router equipment from campus funds ($240,000) • Juniper router equipment from SDSC funds ($450,000)

The campus is also providing cost-share in the form of graduate student fellowships (~2/yr) to increase diversity; currently we have no recipients, and are trying to identify potential candidates.

7.A.3. NU FY2003 Expense Justification Note: In Year 1, Valerie Taylor was at NU and subsequently moved to TAMU. NU subcontracted a portion of Year 1 funds to her for OptIPuter research. In future years, UCSD will subcontract directly to TAMU for OptIPuter research. Taylor received summer salary and graduate student support. Mambretti received travel support.

7.A.4. SDSU FY2003 Expense Justification No changes to original budgeted amounts. Frost received summer salary, travel support, and funds for materials and supplies and audio/visual consulting services.

7.A.5. UCI FY2003 Expense Justification Goodrich, Kane and Smyth received summer salary. There was a delay in hiring a graduate student due to the academic year cycle and grant start date. There was a decision to accelerate the purchase of a visualization cluster in Year 1; components for this will continue to be purchased in Year 2.

7.A.6. UIC FY2003 Expense Justification No changes to original budgeted amounts. DeFanti, Leigh, Brown, Grossman, Moher, Yu and Verlo received support, and 6 graduate students were funded. Funds for travel to San Diego for OptIPuter meetings, as well as material and supplies and computer services, were used.

7.A.7. USC FY2003 Expense Justification Because of the delay in hiring graduate students due to the academic year cycle and grant start date, salary funds were increased for Bannister and Kesselman in year 1.


7.B. UCSD FY2003 Expenses Submitted to NSF.

7.C. NU FY2003 Expenses Submitted to NSF.

7.D. SDSU FY2003 Expenses Submitted to NSF.

7.E. UCI FY2003 Expenses Submitted to NSF.

7.F. UIC FY2003 Expenses Submitted to NSF.

7.G. USC FY2003 Expenses Submitted to NSF.


8. OptIPuter FY2004 Budgets (Year 2)

8.A. FY2004 Budget Justification

8.A.1. Introduction Year 2 OptIPuter funding of $3,460,000 is above the five-year norm. Participating sites have been encouraged to use some of the additional funds to purchase an OptIPuter Mod-0 or Mod-1 “node.” Allocations to participating sites are as follows:

UCSD* 1,655,000 UIC $955,000 NU** $106,420 UCI $260,000 USC $240,000 SDSU $140,000 TAMU** $103,580

* The UCSD budget allocation includes funding originally allocated to General Atomics (GA), which previously managed the UCSD San Diego Supercomputer Center (SDSC). SDSC employees who previously worked for GA now work for UCSD.

** NU and TAMU share the $210,000 originally allocated for NU.

8.A.2. UCSD FY2004 Budget Justification Summer salaries as proposed for Smarr, Ellisman, Papadopoulos, Karin, Chien, Hidley and Steckler.

UCSD is requesting four part-time Academic Professionals (for Orcutt, Ellisman, Papadopoulos and Hidley) and nine graduate students.

Equipment in the amount of $255,000 (as originally requested), will be used to purchase compute, visualization and storage clusters and networking components.

Tech services of $38,373 (down from $69,090) is requested, to help defray the costs of CENIC connectivity to UCI, USC and SDSU.

Travel expenses are in the amount of $30,000. Participant costs of $10,000 will be used to defray the costs of OptIPuter All Hands Meetings.

8.A.3. NU FY2004 Budget Justification NU will use funds for undergraduate student support, travel and a medium-sized Mod-1 OptIPuter visualization cluster.

8.A.4. SDSU FY2004 Budget Justification There are no changes to the Year 2 budget; funds will be used for Frost’s summer salary, graduate student support, some equipment (display projectors, video controllers and networking components), travel and audio/visual consulting services.

SDSU will develop an OptIPuter node from other funds and deploy to groups such as first responders who do not have a large hardware budget. SDSU’s focus will be on using commodity projectors linked to a heterogeneous mix of computers (Windows, Mac, Unix) as would typically exist in a government (federal, state, local) department. SDSU will purchase projectors and video mixer equipment to display 8 to 10 screens within its visualization center in a linked array of information that will be aggregated and sent along the OptIPuter fiber to SDSC/UCSD and then on to UIC and other OptIPuter partners.

8.A.5. TAMU FY2004 Budget Justification Taylor’s summer salary is not included since she is on a 12-month appointment now. Funds will be used for graduate student support, travel and a small-sized Mod-1 OptIPuter visualization cluster.


8.A.6. UCI FY2004 Budget Justification There are no changes to the Year 2 budget. UCI will develop an OptIPuter node by using the clusters and visualization facilities in labs run by UCI professors Steve Jenks and Falko Kuester.

8.A.7. UIC FY2004 Budget Justification UIC is requesting funding for 12 academic months divided among DeFanti, Grossman, Leigh, Moher and Yu to conduct OptIPuter research and supervise the 10 graduate students, whose stipends and tuition waivers are also requested. Maxine Brown, Project Manager, is allocated 6 months salary to manage the entire OptIPuter project (not just UIC’s part). Alan Verlo is allocated 6 months to provide technical supervision of OptIPuter clusters and networks at UIC. UIC asks for $25,000 for equipment to add to its OptIPuter cluster, and requests $9,108 for materials and supplies to build OptIPuter clusters, networks, and related parts, as well as $50,000 for network router/switch maintenance contracts and other computer services anticipated. Overhead is also requested as allowed.

The year 2 budget for UIC is the same as originally submitted and approved by NSF as part of the cooperative agreement except that UIC would like to shift $75,000 from equipment into and additional $35,000 (plus overhead) into services, mainly to cover maintenance contracts, software licenses and the cost of shipping and installing/de-installing equipment for demonstrations and testbed activities, and to cover the salary change of Dr. Jason Leigh from academic staff to faculty.

8.A.8. USC FY2004 Budget Justification USC is decreasing salary support for Bannister and Kesselman in order to purchase a medium-sized Mod-1 OptIPuter visualization cluster. Also, no support for graduate students is requested. Funds for two professional staff, who are working on OptIPuter network protocols and volume visualization software, is requested, as is support for computer services and travel.


8.B. UCSD FY2004 Budget Submitted to NSF.

8.C. NU FY2004 Budget Submitted to NSF.

8.D. SDSU FY2004 Budget Submitted to NSF.

8.E. TAMU FY2004 Budget Submitted to NSF.

8.F. UCI FY2004 Budget Submitted to NSF.

8.G. UIC FY2004 Budget Submitted to NSF.

8.H. USC FY2004 Budget Submitted to NSF.


9. Cumulative Budgets

9.A. OptIPuter Expenditures Cumulative Summary Submitted to NSF.

9.B. UCSD OptIPuter Expenditures Cumulative Submitted to NSF.

9.C. NU OptIPuter Expenditures Cumulative Submitted to NSF.

9.D. SDSU OptIPuter Expenditures Cumulative Submitted to NSF.

9.E. UCI OptIPuter Expenditures Cumulative Submitted to NSF.

9.F. UIC OptIPuter Expenditures Cumulative Submitted to NSF.

9.G. USC OptIPuter Expenditures Cumulative Submitted to NSF.


10. Appendix: “Mod-1” OptIPuter Visualization Cluster Configuration (Draft) The following document was created by the OptIPuter Visualization Team <[email protected]>.

OptIPuter Visualiztion Cluster Specifications 7/21/03

Cluster nodes described below have the following configuration:Dual Intel Pentium4 XEON 2.4GHz 533MHz FSB 512K Cache CPU2GB RAMNvidia Quadro FX 2000 256MB with dual DVI (note: Quadros are $1,400 each)360GB hard disk drive Quadro FX 3000G (with genlock)Intel Dual Copper Integrated Gigabit NIC Just announced July 22

Spares. We recommend (and have budgeted for) spare PCs and LCDs, should one break or get damaged.

Prices include educational discount.

LARGE OptIPuter Viz Cluster Mod 1 (16-node) – with 3x5 LCD tiled display<UIC/EVL OptIPuter Viz Cluster>The 16-node cluster = 15 nodes to drive 15 displays + 1 master console node.(17) PCs (16 nodes + 1 spare) 4,320 17 73,440(1) KVM Switch (Avocent Autoview 416 16 Port KVM) 1,400 1 1,400(16) KVM Cables (Avocent AV400 PS/2 Cables) * 50 16 800(1) On-cluster Network Switch (Cisco WS-C3750G-24TS-S) 5,000 1 5,000(18) LCD panels at 1600x1200 resolution (15 + 3 spare) 1,300 18 23,400(1) LCD panel frame 5,000 1 5,000(16) DVI fiber-optic cables to remote cluster/display (per 33') 440 16 7,040Subtotal 116,080EVL ONLY: (16) Intel PRO/1000 SMF Server Adapter LX 800 16 12,800TOTAL 128,880* Note: The video portion of KVM (Keyboard-Video-Mouse) is not used.

MEDIUM OptIPuter Viz Cluster Mod 1 (9-node) – with 3x5 LCD tiled display<UIC/EVL "Road Warrior" (travel version)>The 9-node cluster = 8 nodes to drive 15 displays (2 outputs per PC) + 1 master console node.(10) PCs (9 nodes + 1 spare) 4,320 10 43,200(1) KVM Switch (Avocent Autoview 416 16 Port KVM) 1,400 1 1,400(9) KVM Cables (Avocent AV400 PS/2 Cables) * 50 9 450(1) On-cluster Network Switch (Cisco WS-C3750G-24TS-S) 5,000 1 5,000(18) LCD panels at 1600x1200 resolution (15 + 3 spare) 1,300 18 23,400(1) LCD panel frame 5,000 1 5,000(16) DVI fiber-optic cables to remote cluster/display (per 33') 440 16 7,040TOTAL 85,490* Note: The video portion of KVM (Keyboard-Video-Mouse) is not used.

Prices for racks built into road cases (for traveling) are given below. If you do not plan to move your cluster, you still need to budget for racks.


MEDIUM OptIPuter Viz Cluster Mod 1 (9-node) – with (2) IBM Big Bertha displays<UCSD NCMIR and SIO clusters>

(10) PCs (9 nodes + 1 spare) 4,320 10 43,200(1) LCD panel at 1600x1200 resolution (for master console) 1,300 1 1,300(1) KVM Switch (Avocent Autoview 416 16 Port KVM) 1,400 1 1,400(9) KVM Cables (Avocent AV400 PS/2 Cables) * 50 9 450(1) On-cluster Network Switch (Cisco WS-C3750G-24TS-S) 5,000 1 5,000(2) IBM Big Berthas 6,700 2 13,400(9) DVI fiber-optic cables to remote cluster/display (per 33') 440 9 3,960TOTAL 68,710* Note: The video portion of KVM (Keyboard-Video-Mouse) is not used.

SMALL OptIPuter Viz Cluster Mod 1 (3-node) – with (1) IBM Big Bertha display

(4) PCs (3 nodes + 1 spare) 4,320 4 17,280(1) LCD panel at 1600x1200 resolution (for master console) 1,300 1 1,300(1) KVM Switch (Avocent Autoview 416 16 Port KVM) 1,400 1 1,400(3) KVM Cables (Avocent AV400 PS/2 Cables) * 50 3 150(1) On-cluster Network Switch 2,500 1 2,500(1) IBM Big Berthas 6,700 1 6,700(5) DVI fiber-optic cables to remote cluster/display (per 33') (4+1 spare) 440 5 2,200TOTAL 31,530* Note: The video portion of KVM (Keyboard-Video-Mouse) is not used.

OPTIONAL Traveling Road Cases with Built-In Racks **Traveling Case(s) for Cluster (maximum for 1-2 racks) 6,000 1 6,000– AND/OR – Traveling Case for LCD Tiled Display *** 5,000 1 5,000TOTAL 11,000** If you don't want traveling road cases, budget approx. $1000 (3-node stationary rack) or $2,000 (9-node rack).*** For Big Berthas, keep shipping boxes; travel case not spec'ed.

The 9-node cluster = 8 nodes to drive 2 Big Berthas (effectively a total of 8 screens in quad packages) + 1 master console node.

The 3-node cluster = 2 nodes to drive 1 Big Bertha (2 outputs per PC, to 4 screens in a quad package) + 1 master console node.


11. Appendix: Report on Workshop on OptIPuter Networking/Backplane Architecture Date: May 22, 2003 Time: 9:00 - 3:00 Place: Electronic Visualization Lab, University of Illinois at Chicago Attendees: Joe Mambretti (NU), Tom DeFanti (UIC), Jason Leigh (UIC), Luc Renambot (UIC), Eric He (UIC), Oliver Yu (UIC), Andrew Chien (UCSD), George Clapp (Telcordia), Narayanan Natarajan (Telcordia) Objectives This meeting was planned not as a discussion forum, but rather as a productive workshop to create outlines and initial drafts of several key documents required to guide the development of OptIPuter data communications components and services. In general, a key objective of this workshop is to create an architecture for the distributed OptIPuter “backplane.” Various outlines/lists that could become parts of the required documents were developed during this workshop, and individual OptIPuter researchers were identified as the principals for identified topics. A number of presentations were given at the start of the meeting, presenting work-in-progress related to these topics. In subsequent months, this working group will produce documents that will specify the following information:

1. An architectural framework for the OptIPuter backplane and related components and services. Much of this workshop focused on identifying the components of this architecture, the definition of those components, the relationship among those components, and the individuals responsible for each area.

2. A preliminary definition of required components and services was discussed as well as potential sources for both, within the context of import versus build versus acquire-and-improve discussions.

3. The topics discussed included requirements of the specific large-scale applications that are targeted by this initiative. These requirements should be translatable into a series of quantifiable measurements that can be communicated as resource requests to lower-level infrastructure. Requests will be governed by an access control method that has yet to be determined (See item 4). Some consideration was given to “packaging” sets of requirements for specific applications or classes of applications. Although there is much existing material on the application requirements, further definitions of these requirements will be developed.

4. The general OptIPuter software architecture group has produced a concept of a “Dynamic Virtual Computer,” which would be created dynamically within a single administrative domain in response to an application request. In other words, instead of being restricted to a static set of pre-determined resources, this architecture will allow for the dynamic creation of specific types of virtual infrastructure as required. One of the objectives of the OptIPuter network research group is creating the mechanisms to address this goal as it relates to the communications processes of such a Dynamic Virtual Computer.

5. The OptIPuter represents a new infrastructure architectural approach, which migrates high-performance infrastructure from one that maximizes computing and minimizes bandwidth to one that maximizes bandwidth, based on new capabilities of advanced optical networking. Consequently, creating a new type of digital communications infrastructure is a key issue, which can be viewed, in part, by a close examination of existing and emerging protocols and technologies within various infrastructure layers. In general, this architecture will have to define (a) an overall framework, (b) the components within that framework, (c) the definitions of those components and their elements, (d) related processes, (e) types of interconnections, including for edge devices, and (f) methods for management, monitoring, restoration/ recovery, and survivability.

6. Today, it is commonly known that traditional digital communication infrastructure does not manage large-scale data flows efficiently. For example, one key area of research activity focuses on providing for optimization within the context of the limitations of traditional Internet protocols, and the current limitations of L3 protocols (e.g., UDP, TCP) were discussed briefly. As a result of these limitations, multiple projects have been established to mitigate various types of restrictions, some at L4, some at L3, and some using mixed approaches. These approaches variously address rate control, window control, fairness, RTT usage and other variables.

7. There was some discussion of new types of “Layer 4” protocols, including RBUDP, Quanta, SABUL, FAST, Tsunami, etc. There will be a survey undertaken of the current state of these protocols. XCP was noted as a protocol to be investigated; it is directed, in part, at mitigating Internet congestion control using


explicit and precise feedback mechanisms that are complementary to current Internet architecture.

8. There was a general consensus that this initiative should not concentrate on Layer 3 technologies (an already crowded field) but to focus on new potentials for L1 and L2 technologies. IP as a data protocol and options for framing, using existing and new methods, were also discussed. It was determined that the OptIPuter backplane would provide for both scheduled and on-demand communication services – one of the implications of a Dynamic Virtual Computer. Another implication is that this initiative must find a scheduler (possibly imported into the project), perhaps, the project will implement a simple scheduler as a first step. One issue here is that most of today’s high-performance schedulers are directed at computational, not communication, resources.

9. There was an extended discussion on signaling for the set-up of communication services, e.g., lightpaths with extensions, which may include L2 electronic paths at the edge and some degree of routing. A related topic discussed was control plane architecture as well as various functions and interprocess communication for lightpath provisioning, discovery, protection paths, and restoration. The role of management planes was also discussed, at the end-to-end level, the device level, and the element level. Traditionally, these functions are management of performance, faults, configurations, accounting and security. The relative positioning of management, control and data planes was described. These concepts were discussed with the context of both intradomain and interdomain lightpath service provisioning. It was determined that this project would pursue a federated model of interdomain provisioning.

10. Overviews of existing research projects related to these components and services were discussed (within and outside the project, nationally and internationally).

11. There was some discussion of acquisition paths for identified required components and services (e.g., create, import into the project, build on existing research efforts, etc). A number of required components may be brought into the project from external sources, particularly with regard to Globus and access control modules, (e.g., IETF AAA vs Globus security components). A near-term task will be a conversation with appropriate members of the Globus community on the state of the Globus security modules. Also, a determination will be made on the availability of resources to develop those modules for this project. Given that AAA components have been developed by a community of researchers at the University of Amsterdam, who have committed significant resources toward this project, those components will be given primary consideration for incorporation into this architecture. It may be worthwhile to pursue an implementation of directory-enabled networking (DEN) in conjunction with either approach.

12. A future (near-term) task will be to produce timeline for research and development (and/or acquisition) of components and services that currently either do not exist or exist only in prototype. This document will be distributed for review soon, based on workshop discussions.

13. The workshop specified a general on-going development and communications process for information, software, and documents related to this project. It was determined that the OptIPuter initiative would manage its own software libraries and technical document and use, for now, existing mailing lists.

14. This workshop discussed related standardization processes and organizations and determined that for the near term, no single standards organization can serve as an appropriate or obvious context for this research.

15. This workshop developed an understanding of areas of responsibility for deliverables as well as individuals charged with producing those deliverables. This information will be formalized in a document that will be distributed with the task timeline noted in item 5.

16. In general, this responsibility distribution will be as follows, with revisions to follow based on near term activities:

a. Overall OptIPuter Architectural Framework – Andrew Chien, Philip Papadopoulos, Carl Kesselman, Tom DeFanti, et. al.

b. Overall OptIPuter Backplane Architecture – Joe Mambretti, Oliver Yu c. Specifications of Application Requirements, L7-L4 issues – Jason Leigh, Luc Renambot, Eric He,

Tom DeFanti d. (I)AAA – Oliver Yu, Jason Leigh, Luc Renambot, Eric He, Andrew Chien e. Middleware Issues – Andrew Chien, Philip Papadopoulos, Jason Leigh, Luc Eric He, Carl

Kesselman


f. L3 Issues – Jason Leigh, Eric He, Luc Renambot, Joe Bannister, Oliver Yu g. Signaling – Joe Mambretti, Oliver Yu, Aaron Johnson, Yunhong Gu, Jeremy Weinberger h. Optical Service Layer – Joe Mambretti, Oliver Yu, Fei Yeh, Jeremy Weinberger i. Interdomain Signaling – Oliver Yu, Joe Mambretti, Tom DeFanti j. L2 Transport (& Data Plane) – Joe Mambretti, Oliver Yu, Fei Yeh k. Control Plane – Joe Mambretti, Oliver Yu, Fei Yeh, Jeremy Weinberger l. Control Switching – Oliver Yu, Jason Leigh, Eric He m. Recovery, Restoration – Oliver Yu, George Clapp, Narayanan Natarajan n. Management Plane – George Clapp, Narayanan Natarajan, Oliver Yu, Joe Mambretti

Agenda 9:00-9:10 Introductions 9:10-9:20 Objectives/Agenda Review – All 9:20-9:30 OptIPuter Architectural Framework – Andrew Chien 9:30-9:40 OptIPuter Communications Arch. Framework/Context – Joe Mambretti 9:40-9:50 Interdomain Considerations – Oliver Yu 9: 50-10:00 Quanta - Eric He 10:00-10:10 Management System Considerations – George Clapp/Narayanan Natarajan 10:10-10:15 Summary/Review of Key Issues Identified – All 10:15-10:30 Break 10:30-10:45 Key Layer 3 Technologies – All 10:45-11:15 Key Layer 2 Technologies, Planned Implementations – All 11:15-11:40 Layer 1 Topics, Existing and Emerging Infrastructure – All 11:40-12:00 Optical Networking Service Layers - All 12:00-1:30 Lunch/ Continuing Discussions 1:30-2:00 Examination of potentials for integration of Globus modules – Led by Andrew Chien 2:00-2:30 Definition of Architecture for Access (AAA, DEN, etc.) – All 2:30-2:45 Determination of need for scheduling mechanism(s), potential sources – All 2:45-3:00 Definition of process for on-going design, development, implementation of OptIPuter

communication technologies – All OptIPuter BackPlane Tasks 5-Year Plan Year 1 Year 2 Year 3 Year 4 Year 5

Specifications of Applications by Data Communica-tion Requirements – e.g., those with Very-Large-Scale Flows

Define specific quantified application requirements for OptIPuter com-munication services as a generic class.

Define specific quantified application requirements for OptIPuter targeted applications, initial testing

Create metrics; test for general and targeted application performance characteristics

Test, demonstrate applications on OptIPuter in large-scale scenarios, Create, implement VLS tests

Create monitoring and reporting tool for application performance end-2-end, Undertake scalability testing

(I)AAA, Resource discovery, State Information (“I” for Identification) Policy DEN

Design and/or select (I)AAA services and create prototype implementation, link to resource access, define interface architecture

Design, prototype policy mechanism for optical backplane service, integrate with access method for interfaces app, admin, process

Optimize database and signaling functions for policy controls including interfaces

Test and demonstrate functions in large-scale scenarios, expand interface signaling methods

Create monitoring and reporting tool for access control systems, including across various interfaces

Define Required Data Communications (vs Computational) Middleware Requirements

Determine middle-ware requirements, required compo-nents, (decisions on import, revise, build …), Def integration of middleware and backplane

Gather, integrate initial required key middleware components, implement, test, measure

Select components for removal, enhancement, replacement based on testing

Design, Revise middleware – 2nd phase implementation of middleware components, test, measure

Create monitoring and reporting tool for middleware components


L4/L3 (Transit and Transport) Protocols (e.g., SABUL, GridFTP, DiffServ, Striped TCP, XCP, FAST, Quanta, etc)

Examine and select high-performance L4/L3 protocols (or L3 bypass) mechanisms, define protocol independent metrics

Determine efficient mechanism for macro traffic flow focus, e.g., L4/L3 by-pass, large buffers etc

Design and implement TE mechanisms for optimized L4/L3 processes

Test and demonstrate in large-scale scenarios, extend TE mechanisms

Design and implement analysis and reporting mechanisms, including for TE

Message Signaling Protocols (API or application-layer protocol, or Clients)

Design extended architecture for intelligent application signaling for lightpaths, and protocols, e.g., XML, SOAP, etc

Create and implement object library OptIPuter digital communications messaging

Create extended set of message signaling modalities and feedback mechanisms

Create and implement secondary (tertiary) messaging paths, for each interface, app, admin, process etc

Create monitoring and reporting tool for messaging systems

Optical Service Layer Layer for Lightpath Management, Path Resources & Attributes

Create an architecture that would include enhanced interaction with state information, Define resource objects

Define path management for optical service layers and control information

Optimize for internode communications of state information

Extend object libraries to include additional devices, including edge devices

Design and implement analysis and reporting mechanisms

Interdomain Signaling

Create architecture for interdomain signaling, create prototype libraries

Integrate methods with multiple domains, test

Design & implement secondary path mechanisms

Optimize for interdomain signaling and messaging

Test and demonstrate in large-scale scenarios

L2 Provisioning and Transport Protocols

Extend prototype software that allows lightpath channels to be extended to edge devices

Integrate L2 switching with (I)AAA, ensure appropriate bi-directional methods

Integrate with additional L2 methods, e.g., MPLS, vLANs, Static circuits

Integrate with protection mechanisms


Control plane, and Related Tools, e.g., GMPLS

Create enhanced interfaces, e.g., to GMPLS libraries, TL1 APIs

Extend CP functions to include peering concepts, e.g., with GMPLS

Extend CP mechanisms to include add. optical devices

Extend integration of CP and service layer functions

Extend CP functions to include integrated device concepts

Control Switching (e.g., OBS)

Create and implement 1st phase architecture for large flow cut-through

Create optimal path methods, e.g., based on statistical optimizations

Enhance with linear optimization techniques

Design, implement protection mechanisms for CS


Recovery/ Restoration Survivability

Design architecture for robust, survivable optical data com, including for multi-layer protection & restoration.

Implement prototype failover lightpath restoration mechanism, diverse path options

Create mechanism for interlinking to restoration/ recovery lightpaths

Design and implement multi layer, multiple path protection

Create, implement mechanism for linking all key resources to these mechanisms

Management Plane Integrating management at all layers (perhaps based on SNMP)

Start with SNMP-type concepts and create extensions, create management arch for levels: end-2-end, node (device), element

Design and implement MIB optical domain equivalents

Design and implement SNMP- type traps for specific events

Design and implement extended MP for additional types of paths and events



12. Appendix: Software Architecture of the OptIPuter System Version 1.0 July 2003 Andrew Chien, UCSD for the OptIPuter System Software Team (Chaitan Baru, Mike Goodrich, Bob Grossman, Sid Karin, Kane Kim, Carl Kesselman, Jason Leigh, Phil Papadopoulos) Abstract The objective of this document is to outline the major software research areas, research objectives, and system software architecture for the OptIPuter project. The description is intended to be comprehensive, but will inevitably fall short of capturing all of the activities. The key elements of the research agenda that emerge include elements of communication, scheduling/resource management, security, network management, visualization and data. This document does not focus on issues of the experimental network infrastructure or the driving applications which are a key part of the OptIPuter project.

Context The world is undergoing rapid technological change with the capabilities of processors, disks, memories and networks all increasing rapidly. The rapid exponential improvement of nearly all of these capabilities means that the rate of exponential increase becomes important in a short period of time altering the balance of resources – and thereby fundamentally altering appropriate system balance and system organization. The OptIPuter focuses on the rapid improvement of network bandwidth, doubling every nine months, which is outpacing even the increase in disk capacity as well as processing power and memory capacity. In summary, the key underlying revolutionary changes which underpin the OptIPuter project include:

• advent of massive bandwidth; orders of magnitude increases both in the local-area and wide-area for wired systems,

• lambda programmed “end to end” connections which can be used as private networks and can provide guaranteed bandwidth,

• endpoint machines which cannot terminate more than a single lambda, due to performance scaling, • large-scale network-attached storage, instruments, displays, and other peripherals, and • Grids and flexible wide-area sharing.

While network hardware technologies like DWDM, optical fibers, and high-speed packet and circuit switches provide raw capabilities, the middleware and system software must render these capabilities in a form usable by applications. The OptIPuter LambdaGrid software architecture will allow applications to dynamically provision dedicated lambdas in seconds; these optical paths may even span the fiber plant and management domains of metro, national and international network service providers. Such dynamic connections raise questions of routing, security, cross-vendor signaling, management, and presentation but potentially offer dramatic new levels of performance, quality of service, and distributed resource abstractions that simplify the construction of distributed applications.

A key element of our approach is to develop a resource abstraction (Distributed Virtual Computer or DVC) which can be used by the application as a simple model for the decidedly complex distributed environment. These DVCs are collections of resources with fixed capabilities, trust relationships, network provisioning, and other forms of performance quality of service. We expect these metacomputers to span multiple administrative domains and be coupled by high-speed dedicated optical connections. Typical useful DVCs include: a virtual computer (a processor, storage device, and display device), a virtual cluster (a homogeneous collection of processors, storage, and display devices), a heterogeneous collection (various processors, storage devices, displays, and other peripherals), or any collection of network devices.

One important use for DVCs is to encapsulate the difficult problem of configuration management, easing the secure configuration of resources from multiple administrative domains to perform a useful scientific or commercial application. A DVC is an entity that can be configured, named, and instantiated on a variety of time scales.

Usage Scenarios We motivate research questions by describing a set of scenarios involving the use of the OptIPuter network, compute, data, and other networked devices to support innovative science and engineering activities. In the pictures


below, the colored boxes depicted correspond to distinct administrative sites on the OptIPuter network.

In all of the scenarios, applications are presumed to use varied resources spanning multiple sites, and including network-attached storage, computing elements, and special instruments and displays. The middleware supports configuration of the optical network, high speed striped communication, flexible access and use of resources, and proactive data layout, replication, and reorganization to support high-speed visualization.

1. Dynamic Distributed Virtual Computer A user allocates network resources dynamically to form a DVC including high-speed peripherals and computing resources from multiple sites; optical network connections are configured; within the DVC resources can access each other directly. Within the DVC, applications flexibly share resources directly and with high performance. This virtual computer serves as a single administrative domain, and users can be granted access to resources within this domain by a single administrator.

2. Pseudo-static Distributed Virtual Computer. A user uses a pseudo-statically defined Distributed Virtual Computer which pools and shares high-speed peripherals and computing resources from multiple sites;

Dynamic Distributed Virtual Computer (DVC)

Pseudo-static Network Configuration


preprogrammed optical network configurations are set up; within the DVC resources can access each other directly. This distributed virtual computer serves as a single administrative domain, and users can be granted access to resources within this domain by a single administrator.

3. Grid with Direct Resource Access. A Globus Grid which federates resources from multiple administrative

domains using virtual organization and a full GSI security infrastructure to coordinate access to resources; after access granted, resources access each other directly.

Viewing collections of distributed resources in DVCs enables the construction of simplified distributed resource abstractions for applications. First, in the network, communication attributes, such as guaranteed performance, physically secured, high-speed, in-order, and low-loss, can be ensured. Second, for displays and storage devices without significant Grid middleware capability, they can be tightly coupled into a DVC and used in fashion similar to a physically secure, localized LAN/SAN environment. Finally, within a DVC, peer device performance, security, and resource management can be done on a simplified basis. In short, DVCs can enable applications to view their world as a safe local cluster environment rather than a hostile, best effort open Internet.

In short, the DVC enables use of a simpler model, a layered single administrative domain with a collection of resources at one’s disposal, but with predictable performance.

High-Level Research Questions The above use scenarios motivate a series of research questions across many aspects of the OptIPuter system. For example:

• Does the increase in network speed relative to other system capabilities give us “just more bandwidth” or do they enable qualitatively different systems?

• What abstractions and resource administrative structures make sense over long periods of time? During the course of a program execution?

• What implications does the abstraction of system have for security and management? • How do we communicate at lambda speeds? What are the parallel abstractions for communication and

data movement? • How do we proactively organize data and compute results? • How do we integrate network devices at high speed? (particularly, data-storage devices, high-resolution

displays, sensor networks, and computing) • How do we share and aggregate them with appropriate security/access control in a high-speed network

environment? • OptIPuter posits removal of communication as a bottleneck, moving resource contention to the endpoints

– how does this change how we manage/allocate/share resources?


• How do we manage lambda networks, switch control and resource management? • What novel opportunities for new system architectures does the changing balance in capabilities provide?

Our list cannot hope to be complete, but forms a motivation for a range of activities.

OptIPuter Software Areas Communication, Compute, and Data Scheduling (Chien, Kim, Bannister, Kesselman, Papadopoulos)

1. Communication o Lightweight and High-speed protocols (speed and simple devices and security)

Development and evaluation of high speed IP protocols (SABUL, RBUDP, GridFTP, Psockets, etc. and proposed TCP extensions/replacements (hsTCP, Fast start, XCP, etc.) and new things

Explore use of high-speed cluster protocols over IP (FM, AM, Hamlyn, PM, BIP, etc.) which do cooperative explicit buffer management for fast start, low overhead transfer, interaction with parallel buffer presentation

Metrics: time to bandwidth, sustained bandwidth, latency o Parallel striping abstractions and data presentation to the application

Striped endpoints for performance, management issues Parallel buffers and presentation to protocols and application layers Coordinated, balanced transfers and achievable robustness and improved performance in

differential behavior MxN transfers and corner turns; composition into an application Time Triggered Message Triggered Object Model (TMO)

o Efficient communication with networked storage, displays, or other simple devices (direct access at high speed)

Termination in simple devices (protocol and management issues above) o High speed secure communications to support Virtual Computer

Setup, access, sharing, data hiding and integrity 2. Integrating Sensors and Instruments

o Device access and integration model (system level) o Data – sensor and stored data integration and access model

3. Resource Allocation/Scheduling o Resource access and reservation in high speed environment

What contention emerges What resource control is needed to ensure good performance What factors (load, data location, devices, etc.) are critical

o High speed and aggregates of low speed devices ¼ and ½ speed devices forming collections of devices into faster/highBW virtual devices

o High performance and opportunistic resource use Techniques for expending resources for better / more robust performance Replication, resource speculation, discarding, and waste

4. Data Placement and Scheduling o Proactive data distribution and Automated placement and optimization for maximum performance

Data replication/dispersion techniques Model-driven, pattern-driven, and application-driven placement Adaptive, intelligent, aggressive fetching techniques Predictive optimization and speculative resource use Geographic data placement and use Redundant encoding, bundling, group prefetching Throughput guarantees for a single high performance job Throughput guarantees for individual jobs in a shared workload and resource

environment All security aware and policy compliant

o Security and High performance access


High speed, direct access to devices Low overhead aggregation/coordination and direct access to devices Trans-organization device sharing and aggregation Integrated data security

5. Distributed System management and configuration control (Papadopoulos, Kesselman) o Rocks-based configuration management for OptIPuter sites o System management based on iVDGL standards?

Security (Karin, Goodrich)

– To be determined

Visualization (DeFanti, Thiébaux, Leigh) (see other documents)

Optical Network (Bannister, Clapp, Hidley, Papadopoulos, Mambretti, Defanti)

– Network switch configuration and management (see other documents)

Data (Baru, Grossman, Smyth)

(see other documents) OptIPuter Software Architecture The OptIPuter software posits a technology change which alters the balance of systems, with communication becoming cheap relative to computing, data access, and virtually all other operations in the system. Examination of the consequences of the availability of private lambdas, high bandwidth, widespread network-attached storage, and acceptance of flexible resource sharing requires reexamination of a number of elements of the emerging de facto standard Grid middleware, the Globus Toolkit (including versions 2.2, 2.4, and the recent release of Globus 3.0).

The OptIPuter project is exploring changes (or additions) needed in several areas: communication, resource management, data movement, and security which underpins all of these protocols. Our intent is to leverage Globus and existing tools to the degree possible, but innovate where necessary to extract the maximum benefit from the opportunity of lambda-based grids. Because of the basically sound architecture of the Globus toolkit, these changes are likely to be wholly or in part integrable with the current software architecture of the Globus system. More specifically, considering the core of the Globus protocols, we describe a range of places where innovation may be incorporated – in some cases requiring extension/change and in others fitting the existing modular structure of Globus and simply instantiating new capabilities.

Communication…The innovative parallel communication abstractions described in software issues would most naturally fall under the capabilities provided by Globus I/O communication libraries or the notions of transports supported by the GT3/Grid Services/Web Services approach. Whether the emerging parallel communication abstractions, novel quality of service, and private secure channels are compatible with these existing notions of communications abstractions is the subject of active research.

Resource Management…The resource management research involves exploration of how to reserve and provision resources in the face of high-bandwidth lambda networks, and the increasing relative expense of storage and computation. We anticipate that new abstractions, algorithms and protocols for resource management (and reservation) will be needed. At present, it appears that integration of these new functionalities under the general GRAM/GARA protocols will be possible. However the relationship of resource reservations, dynamic use, and distributed virtual computers is unclear at this point, and a topic of active research.

Data Movement…OptIPuter communication and direct storage access capabilities present compelling opportunities for next generation proactive data movement and aggressive data replication. The current Globus framework uses GASS and GridFTP to request, move, and cache data. While these APIs and protocols are useful, this area is one of major research and innovation in the OptIPuter software project. We anticipate that significant new capabilities, including proactive mechanisms, direct access to storage, parallel transfers, and optimized storage and network access will require new protocols, interfaces, and functionality.

Security…The opportunities presented by the OptIPuter in high-bandwidth coupling as well as secure private


networks are significant foci of the software research in the project. While the elements of the distributed security infrastructure and protocols supported by GSI are clearly useful in the context of the OptIPuter, we will explore novel security structures enabled by the communication capabilities of emerging optical networks, including the AAA framework from the IETF.

Relation To Emerging Grid Middleware The development of OptIPuter system software is properly viewed against a background of emerging rich and complex Grid middleware (see “Middleware and Data” in this issue), which provides high-performance distributed applications with basic elements of security, resource access, and coordination all built on an IP network fabric. The OptIPuter software builds on this Grid middleware infrastructure, providing unique capabilities derived from dedicated optical paths.

Layer 4: XCPNode Operating Systems

λ-configuration, Net Management

Existing Grid Middleware – (Globus/OGSA)

Physical Resources

DVC #1

OptIPuter Applications

DVC #2 DVC #3

Layer 5: SABUL, RBUDP, Fast, Quanta, GTP

Real-TimeObjects

Security Models

Data Protocols

Higher Level Grid Services

Visualization

Layer 4: XCPNode Operating Systems

λ-configuration, Net Management

Existing Grid Middleware – (Globus/OGSA)

Physical Resources

DVC #1

OptIPuter Applications

DVC #2 DVC #3

Layer 5: SABUL, RBUDP, Fast, Quanta, GTP

Real-TimeObjects

Security Models

Data Protocols

Higher Level Grid Services

Visualization

Figure 1. OptIPuter Software Architecture. The green boxes indicate OptIPuter research areas. In Figure 1, the gray areas represent existing technologies, such as node operating systems, Globus Toolkits and the Open Grid Services Architecture (OGSA), which is evolving from low-level to higher-level grid services. OptIPuter research exploits variations of Grid services that capitalize on new capabilities provided by dedicated lambdas: large, guaranteed bandwidth, controlled jitter and latency, and private channels.

The OptIPuter software architecture includes a variety of models of integration – traditional layered approaches where APIs are presented and used by higher level layers and an unstructured “all services” approach where services are flexibly composed of other network services without strict layering. This latter approach is reflective of increasing momentum behind Grid Services and Web Services, and enables significantly greater flexibility in system design, modularity, and openness. In particular, we expect most of the new services and modules in the OptIPuter software architecture to be presented as OGSA services <http://www.globus.org/ogsa/>, using OGSI <http://www.gridforum.org/ogsi-wg/>.

Documents

FY2003 Annual Progress Report and FY2004 Program Plan