Southern California Infrastructure

Philip Papadopoulos

Greg Hidley

UCSD Packet Test BedOptIPuter Year 2

4 1

OptIPuter UCSD / San Diego Network and Nodes -- 2004

2

ChiaroEnstara

SDSC JSOE

CSE

4 4

Bonded GigE 1

8-nodecluster

(shared)

IBM 9-node Viz cluster

SIO

SOM

IBM 48-node storage cluster

IBM 128-node CPU cluster

Sun 128-node computecluster

Sun 24-32-node

compute cluster

CRCA

6th College

3-node Viz cluster

1

IBM 9 mpixeldisplay pairs

Geowall 2Tiled Display

10

Sun 22-node

Viz cluster

10

Extreme 400Extreme 400

1

1To UCI and ISI via

CalREN-HPR

DellGeowall

Preuss

IBM 9 mpixeldisplay pairs

Dell Viz

Dell 5224

Dell 5224

Dell 5224

Dell 5224

Dell 5224

Dell 5224

Juniper T320 Cisco 6509

HP28-nodecluster

(shared)

HP4-nodecontrol

Sun 24-32-node

computecluster

Infiniband4 nodes

Infiniband64 nodes

Sun 5-node cluster

Sun24-32-node

cluster

9-nodecluster

(shared)

Extreme 400

7-node cluster(shared)

Goals

• Expand our Campus-wide Research Instrument– Support of Researcher Needs – Focus on Application

Needs– Deployment of Scalable Endpoints– Continued Evolution of the Packet-Based OptIPuter– Begin Deployment of 10 GigE Technologies– Drive Towards Goal of 1:1 Bisection Bandwidth

– 5:1 For This Year

• Evaluation of Parallel Storage Systems with Remote Access

• Expand the UCSD-based OptIPuter from Campus-only to Southern California and Eventually to Chicago

Year 2 Accomplishments• Staffing

– Hired UCSD Optiputer Project Manager Aaron Chin– Additional 1.5 FTE Effort Towards Infrastructure Deployment:

– Max Okumoto, Sean O’Connell, David Hutches, Mason Katz

• Node Buildout – 48 Node (IA-32), 21 TB (300 Spindles) IBM Storage Cluster (JSOE)– 128 Node (IA-32) Sun Compute Cluster (SDSC)– 22 Node (Opteron) Sun Viz Cluster (SOM) Geowall2– 3 Node (IA-32) Shuttle Viz Cluster (CRCA) Display Wall– Expected By 10/1/04

– 3 Sun (Opteron) Clusters, Size to be Determined

• Network Buildout– 1 Gigabit Transport

– Connectivity To Campus VLAN Infrastructure And Border Router (T320)– 10 Gigabit Transport

– Single Interface for Chiaro– Extreme 400 (48 Port Gige with two 10gige Uplinks) Layer 2 Fabric

– 1 Each at JSOE, CSE and SDSC– 5:1 Bisection for the Storage Cluster

Making The Campus-Wide OptIPuterUsable

• Solving Network Issues– Converted to Public IP Address Space to Facilitate Off-campus Connectivity– 4 GigE Channel Bonding – Worked With Chiaro to Improve Performance

– Load-balancing Algorithm Among Links is Handled Differently at Layer 2 (Dell) and Layer 3 (Chiaro)

– 35% of Bonded Link Capacity Before “Fix”, 60% Afterwards. L2-L2 (Dell-to-Dell) is 95% For Comparison

– PVFS – 8 IBM Storage Nodes Running as PVFS Servers

– Clients Running at SIO, JSOE and SOM

– Performance Reasonable but not Significantly Larger than 1 Gigabit

• Developing Support for Users [ HTTP://web.optiputer.net ]– Account Request Tools– Network Monitoring Facilities– Technical Network Configuration Information– Node Configuration Information– IP Allocation Map

1GigE

WDM

ONS15808CENIC

at UCSD

ONS15540CENICat USC

ONS15540CENICat UCI

CiscoUCI

FoundryISI

OptIPuterLambdaGrid

FoundryUSC

10GigE

ONS15808CENIC

at Tustin

ONS15808CENICat LA

ONS15540CENIC

at TustinONS15808

CENICat UCSD

ONS15540CENICat LA

1GigE proposed

ONS15530CENIC

at UCSD

ONS15530CENICat LA

ONS15530CENIC

at Tustin

ChiaroUCSD

10GE

10GE

2 1GE

2 1GE

2 1GE

2 1GE CENIC proposed solution for UCSD to UCI, UCSD to ISI,

UCSD to StarLight and UCSD to NASA Goddard

2 1GE transponders for 15540s10GE transponders for 15808sONS15530s at UCSD and LA

1GE to StarLight1GE to NASA Goddard

SoCal CalREN-XD OptIPuter Build-OutExpected Completion July 2004

Year 3 Plans:Enhance Campus OptIPuter

• A Substantial Portion of the Physical Build Completes in Year 2– Endpoints, Cross-campus Fiber, Commodity Endpoints

• Increase Campus Bandwidth– Work Towards More Extensive 10GigE Integration

– Optiputer HW Budget Limited In Year 3, Focus is on Network Extension

– Connect Two Campus Sites with 32-node Clusters At 10GigE– 3:1 Campus Bisection Ratio

• Add/Expand a Moderate Number of new Campus Endpoints– Add New Endpoints Into The Chiaro Network

– UCSD Sixth College

– JSOE (Engineering) Collaborative Visualization Center

– New Calit2 Research Facility

– Add 3 General-purpose Sun Opteron Clusters at Key Campus Sites (Compute and Storage); Clusters Will All Have PCI-X (100 Mhz, 1Gbps)

– Deploy Infiniband on Our IBM Storage Cluster and on a Previously-Donated Sun 128-node Compute Cluster

• Complete Financial Acquisition of the Chiaro Router

Year Three GoalsIntegrate New NSF Quartzite MRI

• Goal -- integration of Packet-based (SoCal) and Circuit-based (Illinois) Approaches a Hybrid System– Add Additional O-O-O Switching Capabilities Through a Commercial

(Calient Or Glimmerglass) All-optical Switch and the Lucent (Pre-commercial) Wavelength Selective Switch

– Begin CWDM Deployment to Extend Optical Paths Around UCSD and Provide Additional Bandwidth

– Add Additional 10GigE in Switches and Cluster Node NICs

• MRI Proposal (Quartzite, Recommended for Funding) Allows Us to Match the Network to the Number of Existing Endpoints

• This is a New Kind of Distributed Instrument – 300+ Components Distributed Over the Campus– Simple and Centralized Control for Other Optiputer Users

UCSD Quartzite Core at Completion (Year 5 of OptIPuter)

QuartziteCore

CalREN-HPRResearch

Cloud

Campus ResearchCloud

GigE Switch withDual 10GigE Upliks

.....To cluster nodes

GigE Switch withDual 10GigE Upliks

.....To cluster nodes

GigE Switch withDual 10GigE Upliks

.....To cluster nodes

GigE

10GigE

...Toothernodes

Quartzite CommunicationsCore Year 3

ProductionOOO

Switch

Juniper T320

4 GigE4 pair fiber

Wavelength Selective

Switch

To 10GigE clusternode interfaces

..... To 10GigE clusternode interfaces and

other switches

Chiaro Enstara

32 10GigE

• Recommended for Funding

• Physical HW to Enable Optiputer and Other Campus Networking Research

• Hybrid Network Instrument