Grid Computing

Grid Computing, B. Wilkinson, 2004 1.1

Grid Computing


Grid Computing

• Using distributed computers and resources collectively.

• Usually associated with geographically distributed computers and resources on a high speed network.

• Often about teams sharing resources.


• For some people, grid computing is just cluster computing in the “large”


LocalCluster

Inter PlanetGrid

2100

2100 2100 2100 2100

2100 2100 2100 2100

Personal Device SMPs or SuperComputers

GlobalGrid

PERFORMANCE

+

Q

o

S

•Individual•Group•Department•Campus•State•National•Globe•Inter Planet•Universe

Administrative Barriers

EnterpriseCluster/Grid

Scalable Computing

Figure due to Rajkumar Buyya, University of Melbourne, Australia, www.gridbus.org


But grid computing can be more than this.

It offer the potential of virtual organizations – groups of people both geographically and

organizationally distributed working together on problems, sharing computers AND other resources such as databases and experimental equipment.


G R I D I N F R A S T R U C T U R E

G r id - e n a b le d

R e a l F ie ld E x p e r im e n t

G r id - e n a b le dV ir t u a l L a b o r a to r y

G r id - e n a b le dA lg o r it h m s

( i. e . , d a t a m i n i n g a n d m o d e l s )


D a ta C o l le c tio n

G R I D I N F R A S T R U C T U R E


R e a l F ie ld E x p e r im e n t

G r id - e n a b le dV ir t u a l L a b o r a to r y

G r id - e n a b le dA lg o r it h m s

( i. e . , d a t a m i n i n g a n d m o d e l s )


D a ta C o l le c tio n

Distributed Collaborative Experiment

Figure from M. Faramawi and B. Ramamurthy, SUNY- Buffalo


Some “Computational” Grid Projects

• Large Hadron Collider experimental facility for complex particle experiments at CERN (European Center for Nuclear Research, near Geneva Switzerland).

• DOE Particle Physics Data grid

• DOE Science grid

• AstroGrid Project

• Comb-e-Chem project


CERN grid


Key aspects

• Using distributed computers and resources collectively.

• Often crossing organizational boundaries

• Fueled by the Internet providing communication network.


Background

• Emergence and immense success of the Internet and the world-wide web, with agreed upon Internet standards for communication and access.

• Continual improvement on computer and network technology and speeds.


History of the Internet

• Internet grew from academic research projects to interconnect of high performance computers

• Started in late 1960’s with the US Defense Department Advanced Research project ARPANET.

• During 1980’s, National Science Foundation expanded ARPANET into NSFNET.

• In 1990’s, privatized and expanded into Internet.


Need to harness computers

• Original driving force behind Internet same as grid computing!

– the need for high performance computing by connecting computers at distributed sites.


Grid computing networks

• Numerous very high performance computing projects developed in late 1990’s and 2000’s.

• Examples: USA TeraGrid (next slide), UK e-Science Grid, etc., etc.


TeraGrid


TeraGrid


UK e-Science Grid


EU grid


Computational Grid Applications

• Biomedical research

• Industrial research

• Engineering research

• Studies in Physics and Chemistry


Key aspects of these grids

• State-of-the-art interconnection networks.

• Sharing resources.

• Community of scientists.


Shared Resources

Can be much more than just computers:

• Storage

• Sensors for experiments at particular sites in the grid

• Application Software

• Databases, ...


Resource sharing and collaborative computing

• Grid computing is about collaborating and resource sharing as much as it is about high performance computing.

• Many projects


Some Grid Projects & Initiatives• Australia

– Nimrod-G– Gridbus– GridSim– Virtual Lab– DISCWorld– GrangeNet.– ..etc

• Europe– UK eScience– EU Data Grid– Cactus– XtremeWeb– ..etc.

• India– I-Grid

Japan– Ninf– DataFarm

• Korea...N*Grid

• SingaporeNGP

• USA– AppLeS– Globus– Legion– Sun Grid Engine– NASA IPG– Condor-G– Jxta– NetSolve– AccessGrid– and many more...

• Cycle Stealing & .com Initiatives– Distributed.net– SETI@Home, ….– Entropia, UD, SCS,….

• Public Forums– Global Grid Forum– Australian Grid Forum

– IEEE TFCC– CCGrid conference– P2P conference

http://www.gridcomputing.comFigures due to Rajkumar Buyya,

University of Melbourne, Australia, www.gridbus.org


SDSCs Grid Port Toolkit generalises the HotPage infrastructure to develop a reusable portal toolkit –gridport.npaci.edu/

Grid Port

Gateway offers a programming paradigm implemented over a virtual Web of accessible resources - www.npac.syr.edu/users/haupt/WebFlow/demo.html

Gateway

NetSolve is a project that aims to bring together disparate computational resources connected by computer networks. It is a RPC based client/agent/server system that allows one to remotely access both hardware and software components – www.cs.utk.edu/netsolve/

NetSolve

Harness builds on the concept of the virtual machine and explores dynamic capabilities beyond what PVM can supply. It focused on developing three key capabilities: Parallel plug-ins, Peer-to-peer distributed control, and multiple virtual machines – www.epm.ornl.org/harness

Harness

This project aims is to develop, deploy, and evaluate mechanisms and policies that support high throughput computing (HTC) on large collections of distributed computing resources – www.cs.wisc.edu/condor/

Condor

The Information Power Grid is a testbed that provides access to a Grid – a widely distributed network of high performance computers, stored data, instruments, and collaboration environments – www.ipg.nasa.gov

NASA IPG

This is an application-specific approach to scheduling individual parallel applications on production heterogeneous systems – apples.ucsd.edu

AppLes

Javelin: Internet-based parallel computing using Java – www.cs.ucsb.edu/research/javelin/Javelin

Legion is an object-based metasystem. Legion supports transparent scheduling, data management, fault tolerance, site autonomy, and a wide range of security options – www.legion.virginia.edu

Legion

This project is developing basic software infrastructure for computations that integrate geographically distributed computational and information resources – www.globus.org

Globus

Focus and Technologies DevelopedInitiative

Some American Grid Projects

Figures due to Rajkumar Buyya, University of Melbourne,

Australia, www.gridbus.org

http://www.gridport.npaci.edu/






http://www.npac.syr.edu/users/haupt/WebFlow/demo.html











http://www.cs.utk.edu/netsolve/









http://www.epm.ornl.org/harness






http://www.cs.wisc.edu/condor/







JaWS is an economy-based computing model where both resource owners and programs using these resources place bids to a central marketplace that generates leases of use – roadrunner.ics.forth.gr

JaWs

This is a wide-area distributed cluster, used for research on parallel and distributed computing by five Dutch universities – www.cs.vu.nl/das

DAS

MetaMPI supports the coupling of heterogeneous MPI systems, thus allowing parallel applications developed using MPI to be run on Grids without alteration – www.lfbs.rwth-aachen.de/~martin/MetaMPICH/

MetaMPI

This project aims to develop middleware and tools necessary for the data-intensive applications of high-energy physics – .www.eu-datagrid.org

Date Grid

Poznan Centre works on development of tools and methods for metacomputing - www.man.poznan.pl/metacomputing/

Pozan

Globe is a research project aiming to study and implement a powerful unifying paradigm for the construction of large-scale wide area distributed systems: distributed shared objects – www.cs.vu.nl/~steen/globe

Globe

The use of Grid for constructing Science applications– www.nesc.ac.ukeScience

Metacomputer OnLine is a toolbox for the coordinated use of WAN/LAN connected systems. MOL aims at utilizing multiple WAN-connected high performance systems for solving large-scale problems that are intractable on a single supercomputer – www.uni-paderborn.de/pc2/projects/mol

MOL

The UNiform Interface to Computer Resources aims to deliver software that allows users to submit jobs to remote high performance computing resources – www.fz-juelich.de/unicore

UNICORE


Some European Grid ProjectsFigures due to Rajkumar Buyya, University of Melbourne, Australia, www.gridbus.org

http://www.roadrunner.ics.forth.gr/

http://www.cs.vu.nl/das

http://www.lfbs.rwth-aachen.de/~martin/MetaMPICH/

http://www.lfbs.rwth-aachen.de/~martin/MetaMPICH/

http://www.grid.web.cern.ch/grid

http://www.grid.web.cern.ch/grid

http://www.man.poznan.pl/metacomputing/

http://www.man.poznan.pl/metacomputing/

http://www.cs.vu.nl/~steen/globe

http://www.n/

http://www.uni-paderborn.de/pc2/projects/mol

http://www.fz-juelich.de/unicore


Bricks is a performance evaluation system that allows analysis and comparison of various scheduling schemes on a typical high-performance global computing setting – matsu-www.is.titech.ac.jp/~takefusa/bricks

Bricks

Ninf allows users to access computational resources including hardware, software and scientific data distributed across a wide area network with an easy-to-use interface – ninf.etl.go.jp

Ninf


Some Japanese Grid Projects

Figures due to Rajkumar Buyya, University of Melbourne, Australia, www.gridbus.org

http://www.matsu-www.is.titech.ac.jp/~takefusa/bricks

http://www.ninf.etl.go.jp/


A resource broker for parametric computing on computational grids. It supports computational economy paradigm for grid computing and deadline and budget constraints based scheduling. www.csse.monash.edu.au/~davida/nimrod/

Nimrod-G

A toolkit for service-oriented computing. It provides services for (a) management of resources based on distributed computational economy at co-operative and competitive levels and (b) deployment of compute and data Grid applications on them. www.gridbus.org

Gridbus

An infrastructure for service-based metacomputing across LAN and WAN clusters. It allows remote users to login to this environment over the Web and request access to data and operations on the available data – dhpc.adelaide.edu.au/Projects/DISCWorld/

DISCWorld


Some Australian Grid Projects

Figures due to Rajkumar Buyya, University of Melbourne,

Australia, www.gridbus.org


Evolution of grid computing

• Started as a form of distributed computing.

• Early distributed computing systems:– 1980’s - Remote Procedure calls (RPC) client -

server model with a service registry.– Later - Distributed objects systems:

• CORBA (Common Request Broker Architecture• Java RMI (Remote Method Invocation)


Grid computing

• With the use of the Internet interconnection technology, implementation now based upon Internet technologies.

• Now uses a form of web services.

• Enables using existing protocols, security mechanisms, etc.


Applications

• Original e-Science applications– Computational intensive, not necessarily

one big problem but a problem that has to be solved repeatedly.

– Data intensive.– Experimental collaborative projects

• e-Business applications to improve business models and practices.


ComputationalEconomy

Grid Challenges and Technologies

Security

Resource Allocation & Scheduling

Data locality

Network Management

System Management

Resource Discovery

Uniform Access

Application ConstructionFigures due to Rajkumar Buyya,

University of Melbourne, Australia, www.gridbus.org

Documents

Grid Computing