Grid Computing @ Sun€¦ · Grid Computing @ Sun Carlo Nardone Technical Systems Ambassador GSO...

Grid Computing@ SunCarlo NardoneTechnical Systems AmbassadorGSO – Client Solutions

2cmn 2004-11

Phases of Grid Computing

Campus Grids• Multiple user communities• Single organization

Global Grids• Multiple user communities• Multiple organizations

Cluster Grids• Single user community• Single organization

3cmn 2004-11

Convergence of Grid and Web Services World● Sun is actively participating

– Platinum Sponsor of Global Grid Forum from its start– Enterprise Grid Alliance– OASIS

● Other standardization efforts– OGSA/OGSI

– WS – RF (Web Services Resource Framework)

– WS – GAF (Web Services Grid Application Framework)

– OMII (Open Middleware Infrastructure, UK)

– NMI (NSF Middleware Initiative)

4cmn 2004-11

Sun Grid Services Environment Small to Large Grid Computing Solutions

Sun QFS/SAM-FSSolaris ZFS, CacheFS

Solaris ContainersSolaris Resource Manager

Sun N1 Grid Engine 6

Cluster Grid Cluster Grid Infrastructure Infrastructure

Global Grid Global Grid Infrastructure Infrastructure

Enterprise Grid Enterprise Grid Infrastructure Infrastructure

Sun Management CenterSun Control Station

Service Service Discovery Discovery

Authentication/Authentication/Authorization Authorization

Data Data Management Management

Policy Policy Management Management

Resource Resource Management Management

System System Management Management

Data Data Access Access

Industry Standards and Industry Standards and partner technologies partner technologies

OGSA, WS-RF ... OGSA, WS-RF ... Globus ToolkitGlobus ToolkitAvaki, SRB ...Avaki, SRB ...

Sun N1 Provisioning System

Java Enterprise System, J2EEJava Enterprise System, J2EE

5cmn 2004-11

Java Enterprise System for EDU

6cmn 2004-11

JES Components

7cmn 2004-11

N1 Grid Engine Distributed Resource Management in Cluster & Enterprise Grids

● Multi–platform, open source– +10,000 downloads since 2000– Queueing, prioritizing, scheduling– Policy–based sharing btw groups & departments– Community project at gridengine.sunsource.net

● New features in N1GE 6– Improved throughput and scalability– New scheduler: resource reservation with backfilling– Standard DRMAA compliant– More supported platforms, Mac OS/X included– Optional BerkeleyDB–based logs– ARCo: Accounting & Reporting Console (Sun product only)– Future Windows exec client (Sun product only)

8cmn 2004-11

Grid Engine Portal An Open Source project based on Java Portal Server

9cmn 2004-11

Campus Grid Computing

10cmn 2004-11

A Successful Example● White Rose University Consortium

– Collaboration btw Univ. of York, Sheffield and Leeds– Critical mass for R&D amongst University departments

● White Rose Grid– In production since 2002– Globus Toolkit 2.0 middleware– GridEngine used for fair sharing of resources btw sites

GridProjects

12cmn 2004-11

Some Interesting Grid Project Using Sun Technologies● JGrid● JxGrid● GrIDE● Many others:

– EPCC TOG (Transfer-queue Over Globus), see gridengine.sunsource.net/project/gridengine/tog.html

– EPCC JOSH (JOb Scheduling Hierarchically) , see www.epcc.ed.ac.uk/sungrid/WP5/ and gridengine.sunsource.net/project/gridengine/josh.html

– JiniGrid, see http://pds.irt.vein.hu/jgrid/jgrid.html

13cmn 2004-11

JGrid

● Provides a useful remote API for submitting objects to the grid

● Lets Grid Engine do the scheduling and accounting

● Allows remote class loading● Minimizes overhead, enables scalability● Alpha available at – http://gridengine.sunsource.net/project/

gridengine/howto/JGrid/jgrid.html

A Java Interface to Grid Engine

14cmn 2004-11

JGrid Architecture

GEMaster

GE execdaemon

Gridlet

Gridlet

Native Peer

Native Peer

Com

pute

Pro

xyResultChannel

ResultChannel

Com

pute

Age

nt

Master Exec Host

Serialized Gridlet

Shared Storage

RMI interface

15cmn 2004-11

J2EE–JGrid Environment

App ServerN1GEMaster HostServlet Container

EJB Container

JGridAgent

execd

EJB

Servlet

Execution Host

EJBEJB

qmas

ter

JGri

d Se

rver

JGridAgent

execd

JGridAgent

execd

Execution Host

Execution Host

16cmn 2004-11

Project JxTA Peer–to–peer Networking via XML Protocols

17cmn 2004-11

JxTA Peer Groups

18cmn 2004-11

JxGrid A JXTA Approach to Resource Discovery for Grid Engine

19cmn 2004-11

GrIDE

● Developed at APSTC– Asia-Pacific Science & Technology Centers, a network

of R&D Centers in partnership btw Sun Microsystems EDU LOB and Academia

● GrIDE is an integrated development environment for Grids– Workflow Editor– Crosscompiler– Grid Debugger– Performance Tuning– Data Grid Access– Project Collaboration

20cmn 2004-11

GriDE Architecture

Middleware Layer

Programming languages/ Libraries Layer

GriDE Layer

NetBeans Tools Platform

Griddebugger

Data GridAccess

Java MPI-G2

SGE

ProjectCollaboration

Globus NimRod

Computing Resources

PerformanceTurning

Grid SimulationC/C++ Fortran

Flow Editor

AppsTemplates

Cross Compiler

Apps API

CoGKit

21cmn 2004-11

22cmn 2004-11

GrIDE Workflow Editor

DataGrids

24cmn 2004-11

Data Grids● Federated, transparent services, independent of

– Location– Name– Distribution– Replication– Ownership– Heterogeneity– Change of schema

● Support for domain–specific and general data access layer

● Replication management and consistency● Consistency services appropriate to nature of the Grid● Globus GridFTP is not enough!

25cmn 2004-11

Data Grids: EPCC OGSA–DAI● Main Grid effort at EPCC● Open Grid Services Architecture – Data Access

and Integration● Main tenet: while compute on demand is

important, it is a niche market● Distributed data on demand is much more

interesting prospect● ELDAS – J2EE version of OGSA–DAI● Check–out www.ogsadai.org.uk

26cmn 2004-11

Data Grids: Avaki

● Commercial result of Legion research project● Enterprise Information Integration● Unified data catalogue● Single data service layer● Access via Web services, ODBC, JDBC, file I/O● Caching for performance● Data integration accessible as a web service –

regardless of location● Available via Sun Client Solutions

27cmn 2004-11

Data Grids: SRB● Storage Resource Broker● Research project from SDSC, San Diego, see

www.npaci.edu/dice/srb

● Client–server middleware● Uniform interface for connecting to

heterogeneous data over networks to enable data grids, digital libraries, etc.

● Works with MCAT (Metadata Catalog) to:– Access data sets based on attributes and logical names– Various interfaces: inQ, MySRB, Jargon, Matrix

● Commercial version from Nirvana Storage

Thanks!carlo.nardone@sun.comcell: 335 5828197

The Network is the Computer