Globus: A Core Grid Middleware

Source: The Globus ProjectArgonne National Laboratory,

University of Southern California / ISI

www.globus.orgLocalised by: Rajkumar Buyya

The Globus Project

Basic research in grid-related technologies Resource & data management, security, QoS,

policy, communication, adaptation, etc. Development of Globus Toolkit

Core services for grid-enabled tools & apps Construction of production grids & testbeds

Multiple deployments to distributed organizations for production & prototyping

Application experiments Distributed applications, tele-immersion, etc.

Globus Approach

A toolkit and collection of services addressing key technical problems Modular “bag of services” model Not a vertically integrated solution General infrastructure tools (aka middleware) that

can be applied to many application domains Inter-domain issues, rather than clustering

Integration of intra-domain solutions Distinguish between local and global services

Globus Hourglass

Focus on architecture issues Propose set of core services

as basic infrastructure Use to construct high-level,

domain-specific solutions Design principles

Keep participation cost low Enable local control Support for adaptation “IP hourglass” model

Diverse global services

Core Globusservices

Local OS

A p p l i c a t i o n s

Technical Focus & Approach

Enable incremental development of grid-enabled tools and applications Model neutral: Support many programming

models, languages, tools, and applications Evolve in response to user requirements

Deploy toolkit on international-scale production grids and testbeds Large-scale application development & testing

Information-rich environment Basis for configuration and adaptation

Globus Toolkit Services

Security (GSI) PKI-based Security (Authentication) Service

Job submission and management (GRAM) Uniform Job Submission

Information services (MDS) LDAP-based Information Service

Remote file management (GASS) Remote Storage Access Service

Remote Data Catalogue and Management Tools Support by Globus 2.0 released in 2002

Grid Services Architecture

Grid FabricLayer

Applications

Transport Multicast

Instrumentation Control interfaces QoS mechanisms

Grid ServicesLayer

Information Resource mgmt

Security Data access Fault detection

. . .

. . .

High-energyphysics data

analysis Regionalclimate studies

Collaborativeengineering

Parameterstudies

On-lineinstrumentation

ApplicationToolkit Layer

Highthroughput

Data-intensive

Collab.design

Remoteviz

Remote control

Layered Architecture

Applications

Grid ServicesGRAM

GSI HBM

Nexus

I/O

Grid Fabric

LSF

Condor MPI

NQEPBS

TCP

NTLinux

UDP

Application Toolkits

DUROC globusrunMPI Nimrod/GCondor-G HPC++

GlobusView Testbed Status

GASS

Solaris DiffServ

GSI-FTPMDS

Sample of High-Level Services

Resource brokers and co-allocators DUROC, Nimrod/G, Condor-G, GridbusBroker

Communication & I/O libraries MPICH-G, PAWS, RIO (MPI-IO), PPFS, MOL

Parallel languages HPC++, CC++, Nimrod Parameter Specification

Collaborative environments CAVERNsoft, ManyWorlds

Others MetaNEOS, NetSolve, LSA, AutoPilot, WebFlow

A resource broker for managing, steering, and executing task farming (parameter sweep/SPMD model) applications on the Grid based on deadline and computational economy.

Based on users’ QoS requirements, our Broker dynamically leases services at runtime depending on their quality, cost, and availability.

Key Features A single window to manage & control experiment Persistent and Programmable Task Farming Engine Resource Discovery Resource Trading Scheduling & Predications Generic Dispatcher & Grid Agents Transportation of data & results Steering & data management Accounting

Uses Globus – MDS, GRAM, GSI, GASS

The Nimrod-G Grid Resource Broker

Condor-G: Condor for the Grid

Condor is a high-throughput scheduler Condor-G uses Globus Toolkit libraries for:

Security (GSI) Managing remote jobs on Grid (GRAM) File staging & remote I/O (GSI-FTP)

Grid job management interface & scheduling Robust replacement for Globus Toolkit programs

Globus Toolkit focus is on libraries and services, not end user vertical solutions

Supports single or high-throughput apps on Grid Personal job manager which can exploit Grid resources

Production Grids & Testbeds

Production deployments underway at: NSF PACIs National Technology Grid NASA Information Power Grid DOE ASCI European Grid

Research testbeds EMERGE: Advance reservation & QoS GUSTO: Globus Ubiquitous Supercomputing

Testbed Organization Particle Physics Data Grid World-Wide Grid (WWG)

Production Grids & Testbeds

NASA’s Information Power Grid The Alliance National Technology Grid

GUSTO Testbed

World Wide Grid (WWG)

GMonitor

@ SC 2002/Baltimore

Grid MarketDirectory

Australia

Melbourne+Monash U:

VPAC, Physics

Solaris WS

Gridbus+Nimrod-G

Europe

ZIB: T3E/OnyxAEI: Onyx CNR: ClusterCUNI/CZ: OnyxPozman: SGI/SP2Vrije U: ClusterCardiff: Sun E6500Portsmouth: Linux PCManchester: O3KCambridge: SGIMany others

Asia

AIST, Japan: Solaris ClusterOsaka University: ClusterDoshia: Linux clusterKorea: Linux cluster

North America

ANL: SGI/Sun/SP2NCSA: ClusterWisc: PC/clusterNRC, CanadaMany others

InternetWW Grid

MEG Visualisation

Example Applications Projects (via Nimrod-G or Gridbus)

Molecular Docking for Drug Discovery Docking molecules from chemical databases

with target protein Neuro Science

Brain Activity Analysis High Energy Physics

Belle Detector Data Analysis Natural Language Engineering

Analyzing audio data (e.g., to identify emotional state of a person!)

Example Application Projects

Computed microtomography (ANL, ISI) Real-time, collaborative analysis of data from X-

Ray source (and electron microscope) Hydrology (ISI, UMD, UT; also NCSA, Wisc.)

Interactive modeling and data analysis Collaborative engineering (“tele-immersion”)

CAVERNsoft @ EVL OVERFLOW (NASA)

Large CFD simulations for aerospace vehicles

Example Application Experiments

Distributed interactive simulation (CIT, ISI) Record-setting SF-Express simulation

Cactus Astrophysics simulation, viz, and steering Including trans-Atlantic experiments

Particle Physics Data Grid High Energy Physics distributed data analysis

Earth Systems Grid Climate modeling data management

The Globus Advantage

Flexible Resource Specification Language which provides the necessary power to express the required constraints

Services for resource co-allocation, executable staging, remote data access and I/O streaming

Integration of these services into high-level tools MPICH-G: grid-enabled MPI globus-job-*: flexible remote execution commands Nimrod-G Grid Resource broker Gridbus: Grid Business Infrastructure Condor-G: high-throughput broker PBS, GRD: meta-schedulers

Resource Management

Resource Specification Language (RSL) is used to communicate requirements

The Globus Resource Allocation Manager (GRAM) API allows programs to be started on remote resources, despite local heterogeneity

A layered architecture allows application-specific resource brokers and co-allocators to be defined in terms of GRAM services

GRAM GRAM GRAM

LSF EASY-LL NQE

Application

RSL

Simple ground RSL

Information Service

Localresourcemanagers

RSLspecialization

Broker

Ground RSL

Co-allocator

Queries& Info

Resource Management Architecture

GRAM Components

Globus SecurityInfrastructure

Job Manager

GRAM client API calls to request resource allocation

and process creation.

MDS client API callsto locate resources

Query current statusof resource

Create

RSL Library

Parse

RequestAllocate &

create processes

Process

Process

Process

Monitor &control

Site boundary

Client MDS: Grid Index Info Server

Gatekeeper

MDS: Grid Resource Info Server

Local Resource Manager

MDS client API callsto get resource info

GRAM client API statechange callbacks

A simple run

[raj@belle raj]$ globus-job-run belle.anu.edu.au /bin/date

Mon May 3 15:05:42 EST 2004

Resource Specification Language (RSL)

Common notation for exchange of information between components Syntax similar to MDS/LDAP filters

RSL provides two types of information: Resource requirements: Machine type,

number of nodes, memory, etc. Job configuration: Directory, executable,

args, environment API provided for manipulating RSL

RSL Syntax

Elementary form: parenthesis clauses (attribute op value [ value … ] )

Operators Supported: <, <=, =, >=, > , !=

Some supported attributes: executable, arguments, environment, stdin,

stdout, stderr, resourceManagerContact,resourceManagerName

Unknown attributes are passed through May be handled by subsequent tools

Constraints: “&”

globusrun -o -r belle.anu.edu.au "&(executable=/bin/date)"

For example:& (count>=5) (count<=10) (max_time=240) (memory>=64) (executable=myprog)

“Create 5-10 instances of myprog, each on a machine with at least 64 MB memory that is available to me for 4 hours”

Disjunction: “|”

For example: & (executable=myprog) ( | (&(count=5)(memory>=64))

(&(count=10)(memory>=32))) Create 5 instances of myprog on a

machine that has at least 64MB of memory, or 10 instances on a machine with at least 32MB of memory

Multirequest: “+”

A multi-request allows us to specify multiple resource needs, for example+ (& (count=5)(memory>=64) (executable=p1)) (&(network=atm) (executable=p2)) Execute 5 instances of p1 on a machine with

at least 64M of memory Execute p2 on a machine with an ATM

connection Multirequests are central to co-allocation

Co-allocation

Simultaneous allocation of a resource set Handled via optimistic co-allocation based on

free nodes or queue prediction In the future, advance reservations will also

be supported

globusrun and globus-job-* will co-allocate specific multi-requests Uses a Globus component called the

Dynamically Updated Request Online Co-allocator (DUROC)

DUROC Functions

Submit a multi-request Edit a pending request

Add new nodes, edit out failed nodes Commit to configuration

Delay to last possible minute Barrier synchronization

Initialize computation Bootstrap library

Monitor and control collection

DUROC Architecture

ControllingApplication

ControlledJobs

RSL multi-request

Job 1

RM1

Job 4

Job 5

RM4

Job 2

RM2

Job 3

RM3

Edit request

Subjob status

Barrier

RSL Creation Using globus-job-run

globus-job-run can be used to generate RSL from command-line args:globus-job-run –dumprsl \

-: host1 -np N1 [-s] executable1 args1 \ -: host2 -np N2 [-s] executable2 args2 \ ... > rslfile

-np: number of processors -s: stage file argument options for all RSL keywords -help: description of all options

Job Submission Interfaces

Globus Toolkit includes several command line programs for job submission globus-job-run: Interactive jobs globus-job-submit: Batch/offline jobs globusrun: Flexible scripting infrastructure

Other High Level Interfaces General purpose

Nimrod-G, Condor-G, PBS, GRD, etc Application specific

ECCE’, Cactus, Web portals

globus-job-run

For running of interactive jobs Additional functionality beyond rsh

Ex: Run 2 process job w/ executable stagingglobus-job-run -: host –np 2 –s myprog arg1 arg2

Ex: Run 5 processes across 2 hostsglobus-job-run \

-: host1 –np 2 –s myprog.linux arg1 \-: host2 –np 3 –s myprog.aix arg2

For list of arguments run:globus-job-run -help

globus-job-submit

For running of batch/offline jobs globus-job-submit Submit job

Same interface as globus-job-run Returns immediately

globus-job-status Check job status globus-job-cancel Cancel job globus-job-get-output Get job stdout/err globus-job-clean Cleanup after job

globusrun

Flexible job submission for scripting Uses an RSL string to specify job request Contains an embedded globus-gass-server

Defines GASS URL prefix in RSL substitution variable:(stdout=$(GLOBUSRUN_GASS_URL)/stdout)

Supports both interactive and offline jobs Complex to use

Must write RSL by hand Must understand its esoteric features Generally you should use globus-job-* commands

instead

Simultaneous start

co-allocator

InformationService

“Run SF-Expresson 300 nodes”

"Run SF-Expresson 256 nodes”

“Run adistributed interactive

simulation involving100,000 entities”

“80 nodes on Argonne SP,256 nodes on CIT Exemplar300 nodes on NCSA O2000”

“Supercomputers providing 100 GFLOPS, 100 GB, < 100 msec latency”DIS-Specific

Broker

" . . ."

“Performa parameter studyinvolving 10,000separate trials”

Parameter studyspecific broker

Supercomputerresource broker

NCSAResource Manager

ArgonneResource Manager

CITResource Manager

Resource Brokers

" . . ."

“Create ashared virtual space

with participantsX, Y, and Z”

Collaborativeenvironment-specific

resource broker

"Run SF-Expresson 80 nodes”

Brokering via Lowering

Resource location by refining a RSL expression (RSL lowering):

(MFLOPS=1000) (& (arch=sp2)(count=200)) (+ (& (arch=sp2) (count=120)

(resourceManagerContact=anlsp2)) (& (arch=sp2) (count=80) (resourceManagerContact=uhsp2)))

Remote I/O and Staging

Tell GRAM to pull executable from remote location

Access files from a remote location stdin/stdout/stderr from a remote

location

What is GASS?

(a) GASS file access API Replace open/close with globus_gass_open/close;

read/write calls can then proceed directly

(b) RSL extensions URLs used to name executables, stdout, stderr

(c) Remote cache management utility(d) Low-level APIs for specialized behaviors

GASS Architecture

CacheCache

GASS Server

HTTP Server

FTP Server

% globus-gass-cache

(c) Remote cache management

GRAM

(a) GASS file access API

&(executable=https://…)

(b) RSL extensions

(d) Low-level APIs for customizing cache & GASS server

main( ) { fd = globus_gass_open(…) … read(fd,…) … globus_gass_close(fd)}

GASS File Naming

URL encoding of resource nameshttps://quad.mcs.anl.gov:9991/~bester/myjob

protocol server address file name Other examples

https://pitcairn.mcs.anl.gov/tmp/input_dataset.1https://pitcairn.mcs.anl.gov:2222/./output_datahttp://www.globus.org/~bester/input_dataset.2

Supports http & https Support ftp & gsiftp.

GASS RSL Extensions

executable, stdin, stdout, stderr can be local files or URLs

executable and stdin loaded into local cache before job begins (on front-end node)

stdout, stderr handled via GASS append mode

Cache cleaned after job completes

GASS/RSL Example

&(executable=https://quad:1234/~/myexe) (stdin=https://quad:1234/~/myin) (stdout=/home/bester/output) (stderr=https://quad:1234/dev/stdout)

Example GASS Applications

On-demand, transparent loading of data sets

Caching of data sets Automatic staging of code and data

to remote supercomputers (Near) real-time logging of

application output to remote server

GASS File Access API

Minimum changes to application globus_gass_open(), globus_gass_close()

Same as open(), close() but use URLs instead of filenames

Caches URL in case of multiple opens Return descriptors to files in local cache or

sockets to remote server globus_gass_fopen(),

globus_gass_fclose()

GASS File Access API (cont)

Support for different access patterns Read-only (from local cache) Write-only (to local cache) Read-write (to/from local cache) Write-only, append (to remote server)

Remove cachereference

Upload changes

Modified no

yes

globus_gass_open()/close()

Download Fileinto cache

open cached file,add cachereference

URL in cache? no

yes

globus_gass_open()

globus_gass_close()

GASS File API Semantics

Copy-on-open to cache if not truncate or write-only append and not already in cache

Copy on close from cache if not read only and not other copies open

Multiple globus_gass_open() calls share local copy of file

Append to remote file if write only append: e.g., for stdout and stderr

Reference counting keeps track of open files

globus-gass-server

Simple file server Run by user wherever necessary Secure https protocol, using GSI APIs for embedding server into other programs

Exampleglobus-gass-server –r –w -t

-r: Allow files to be read from this server -w: Allow files to be written to this server -t: Tilde expand (~/… $(HOME)/…) -help: For list of all options

1. Derive Contact String2. Build RSL string3. Startup GASS server4. Submit to request5. Return output

jobmanager

gatekeeper

program

GRAM & GASS: Putting It Together

stdout

GASS server

3

4

globus-job-run

Host name

Contactstring

1

RSLstring

2CommandLine Args

4

4

55

55

Example: A Simple Broker

Select machines based on availability Use MDS queries to get current host loads Look at output and figure out what

machines to use Generate RSL based on selection

globus-job-run -dumprsl can assist Execute globusrun, feeding it the RSL

generated in previous step

GRAM & GASS

Using RSL with globusrun Running globus-gass-server Modifying a program to use

globus_gass_open() to read files remotely from a GASS server

Globus Components In ActionLocal Machine

mpirun

globusrun

GRAM

ClientGSI

GRAM

ClientGSI

Remote Machine

AppNexus

AIX

PBS

MPI

grid-proxy-initX509UserCert

UserProxyCert

Machines

GRAM Gatekeeper

GSI

GRAM Job Manager

GASS Client

Remote Machine

AppNexus

Solaris

Unix Fork

MPI

GRAM Gatekeeper

GSI

GRAM Job Manager

GASS Client

RSL string

RSL multi-request

RSL single requestDUROC

GASS Server

RSL parser

GRAM Components

Globus SecurityInfrastructure

Job Manager

GRAM client API calls to request resource allocation

and process creation.

MDS client API callsto locate resources

Query current statusof resource

Create

RSL Library

Parse

RequestAllocate &

create processes

Process

Process

Process

Monitor &control

Site boundary

Client MDS: Grid Index Info Server

Gatekeeper

MDS: Grid Resource Info Server

Local Resource Manager

MDS client API callsto get resource info

GRAM client API statechange callbacks

MDS: Monitoring and Discovery Service

Learn how to use the MDS to locate and determine characteristics of resources

Locate resources Where are resources with required

architecture, installed software, available capacity, network bandwidth, etc.?

Determine resource characteristics What are the physical characteristics,

connectivity, capabilities of a resource?

The Need for Information

System information is critical to operation of the grid and construction of applications How does an application determine what

resources are available? What is the “state” of the computational grid? How can we optimize an application based on

configuration of the underlying system? We need a general information

infrastructure to answer these questions

Using Information forResource Brokering

“10 GFlops, EOS data,20 Mb/sec -- for 20 mins”

MetacomputingDirectoryService

GRAMGRAMGRAM

ResourceBroker

Info service:location + selection

Globus ResourceAllocation Managers

GRAM

ForkLSFEASYLLCondoretc.

“What computers?”“What speed?”“When available?”

“50 processors + storage from 10:20 to 10:40 pm”

“20 Mb/sec”

Examples of Useful Information

Characteristics of a compute resource IP address, software available, system

administrator, networks connected to, OS version, load

Characteristics of a network Bandwidth and latency, protocols, logical

topology Characteristics of the Globus

infrastructure Hosts, resource managers

Grid Information Service

Provide access to static and dynamic information regarding system components

A basis for configuration and adaptation in heterogeneous, dynamic environments

Requirements and characteristics Uniform, flexible access to information Scalable, efficient access to dynamic data Access to multiple information sources Decentralized maintenance

MDS

Store information in a distributed directories Directory stored in collection of LDAP servers Each server optimized for particular function

Directory can be updated by Information providers and tools Applications (i.e., users) Backend tools which generate info on demand

Information dynamically available to Tools Applications

Directory Service Functions

White Pages Look up the IP number, amount of memory, etc.,

associated with a particular machine Yellow Pages

Find all the computers of a particular class or with a particular property

Temporary inconsistencies are often considered okay In a distributed system, you often do not know the

state of a resource until you actually use it Information is often used as “hints” Information itself can contain ttl, etc.

MDS Approach

Based on LDAP Lightweight Directory Access Protocol v3

(LDAPv3) Standard data model Standard query protocol

Globus specific schema Host-centric representation

Globus specific tools GRIS, GIIS Data discovery, publication,…

SNMP

GRIS

NIS

NWS

LDAP

LDAP API

Middleware

…

Application

GIIS…

MDS Components

Uses standard LDAP servers OpenLDAP, Netscape, Oracle, etc

Tools for populating & maintaining MDS Integrated with Globus Toolkit server release, not

of concern to most Globus users Discover/update static and dynamic info

APIs for accessing & updating MDS contents C, Java, PERL (LDAP API, JNDI)

Various tools for manipulating MDS contents Command line tools, Shell scripts & GUIs

Anonymous Grid info search

grid-info-search -x -h belle.anu.edu.au….Mds-Computer-isa: i686Mds-Computer-platform: i686Mds-Computer-Total-nodeCount: 1Mds-Cpu-Cache-l2kB: 512Mds-Cpu-features: fpu vme de pse tsc msr pae mce cx8

apic sep mtrr pge mca cmo v pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tmMds-Cpu-Free-15minX100: 400Mds-Cpu-Free-1minX100: 400Mds-Cpu-Free-5minX100: 400Mds-Cpu-model: Intel(R) Xeon(TM) CPU 2…

Summary

MDS provides the information needed to perform dynamic resource discovery and configuration Critical component of resource brokers

MDS is base on existing directory service standards (LDAPv3)