JSAGA 2

OverviewOverview

jobdesc.

gLiteplug-ins

Globusplug-ins

JSAGAJSAGA

hid

eh

ide

mid

dle

war

em

idd

lew

are

het

ero

gen

eity

het

ero

gen

eity

(e.g

. gLi

te, G

lobu

s, U

nico

re)

(e.g

. gLi

te, G

lobu

s, U

nico

re)

JDL RSL

JSAGA 3

WMSWMS

inputdata

SRMSRM

GridFTPGridFTP

WS-GRAMWS-GRAMLCG-CELCG-CELCG-CELCG-CE WS-GRAMWS-GRAMfi

rew

all

OverviewOverview

jobdesc.

gLiteplug-ins

Globusplug-ins

JSAGAJSAGA

jobjob

staginggraph

dele

gate

sel

ectio

n

& fil

es s

tagi

ng

jobjob

OPlastEGEE

hid

e g

rid

hid

e g

rid

infr

astr

uct

ure

s

infr

astr

uct

ure

sh

eter

og

enei

ty

het

ero

gen

eity

(e.g

. EG

EE

, OS

G, D

EIS

A)

(e.g

. EG

EE

, OS

G, D

EIS

A)

hid

eh

ide

mid

dle

war

em

idd

lew

are

het

ero

gen

eity

het

ero

gen

eity

(e.g

. gLi

te, G

lobu

s, U

nico

re)

(e.g

. gLi

te, G

lobu

s, U

nico

re)

JDL RSL

JSAGA 4

What is JSAGA ?What is JSAGA ?h

ide

gri

dh

ide

gri

din

fras

tru

ctu

res

infr

astr

uct

ure

sh

eter

og

enei

ty

het

ero

gen

eity

(e.g

. EG

EE

, OS

G, D

EIS

A)

(e.g

. EG

EE

, OS

G, D

EIS

A)

hid

eh

ide

mid

dle

war

em

idd

lew

are

het

ero

gen

eity

het

ero

gen

eity

(e.g

. gLi

te, G

lobu

s, U

nico

re)

(e.g

. gLi

te, G

lobu

s, U

nico

re)

An API for submitting a collectionAn API for submitting a collectionof jobs to several grid infrastructuresof jobs to several grid infrastructureswith a single job descriptionwith a single job description

A Java implementation of theA Java implementation of theSAGA specificationSAGA specification

JSAGA is…JSAGA is…

JSAGA 5

Global architectureGlobal architecture

For end-users: ready-to-use software adapted to their needs

For application developers: hide infrastructure heterogeneity

For application developers: hide middleware heterogeneity

For plug-in developers: As many interfaces as ways to implement the functionalities

ApplicationsApplications

Plug-ins interfacesPlug-ins interfaces

Job CollectionJob Collection

SAGASAGA

JSAGA 6planned

construction

done

Provided plug-insProvided plug-ins

SecuritySecurity X5

09

SS

H

InM

em

Cre

d

Glo

bu

s

G. L

eg

acy

G. R

FC

82

0

MyP

roxy

VO

MS

Lo

gin

/ pw

d

DataData

Exec.Exec.(control)(control)

ExpressionExpression

cata

log

rns

lfn

srb / iro

ds

http

http

s

sftp

rbyte

io

file

zip

gsiftp

tar

ftp

ma

il

cach

e

srm

Job controlJob control

ga

teke

ep

er

gL

ite-W

MS

wsg

ram

un

icore

6

ssh

loca

l

crea

m

grid

vm

PB

S

rem

ote

Exec.Exec.(monitor)(monitor)

ga

teke

ep

er

gL

ite-L

B

wsg

ram

un

icore

6

ssh

loca

l

crea

m

…

ba

sic

de

fau

lt

JEP

Be

an

Sh

ell

LanguageLanguage

JS

DL

+e

xt.

SA

GA

JDL

RS

L-2

RS

L-4

JSAGA 7

Hide middleware heterogeneity– Uniform interface (SAGA)

Hide infrastructure heterogeneity– Selection of the right security

context

– Selection of the right client API configuration

– Transport of the job input/output data to/from worker nodes

Describe the capabilities of the infrastructures you want to use

requirements

WorldWorld

Grid Grid

EGEEEGEE

localhostlocalhost

OpenPlastOpenPlast

CC-IN2P3CC-IN2P3

Describe grid infrastructuresDescribe grid infrastructures

wsgramwsgram

WMSWMS

VOMSVOMS

VOMSVOMS

GlobusGlobus

gsiftp://

JSAGA 8

Describe grid infrastructuresDescribe grid infrastructures

Hide middleware heterogeneity– Job management technologies

• e.g. CREAM, WMS, SSH, GK

Hide infrastructure heterogeneity– Computing Elements (e.g. GK)

• Different grid or site policies– e.g. network filtering, shared FS

• Different environment variables– e.g. $VO_?_SW_DIR, /usr/local

• Different configuration attributes– e.g. monitor service URL, shell

path on cygwin, default SE URL

• Commands available on worker– e.g. globus-url-copy, srmcp,

Scp, wget, tar

WorldWorld

Grid Grid

EGEEEGEE

OpenPlastOpenPlast

localhostlocalhost

CC-IN2P3CC-IN2P3

http://

tar://

srb://

gsiftp://

srm://

lfn://

gatekeepergatekeeper

gatekeepergatekeeper

wsgramwsgram

WMSWMS

gatekeepergatekeeper

VOMSVOMS

VOMSVOMS

GlobusGlobus

examples

JSAGA 9

url://

Transfer path depends on…Transfer path depends on…

Using a single grid– all files can be transferred to the worker through a single node

Using several grids– need to dynamically build a transfer graph, according to…

WorldWorld

Grid Grid

EGEEEGEE

OpenPlastOpenPlast

localhostlocalhost

CC-IN2P3CC-IN2P3

http://

tar://

srb://

gsiftp://

srm://

lfn://

gatekeepergatekeeper

gatekeepergatekeeper

wsgramwsgram

WMSWMS

gatekeepergatekeeper

VOMSVOMS

VOMSVOMS

GlobusGlobus

jobjobdesc.desc.

plug-insplug-ins

JSAGAJSAGA

JSAGA 10

Transfer path depends on…Transfer path depends on…

grid or site– network filtering policy

– commands available on workers

– services available from workers (close Storage Element, shared FS)

– supported context instances

data to stage– shared by several jobs– installed on some worker nodes– file size– required data protection level

execution service– protocols supported for staging

transfer protocol– access mode (RO, WO, RW)– third-party transfer– supported data protection level

jobjobdesc.desc.

WorldWorld

Grid Grid

EGEEEGEE

OpenPlastOpenPlast

localhostlocalhost

CC-IN2P3CC-IN2P3

http://tar://

srb://

gsiftp://

srm://lfn://

gatekeepergatekeeper

gatekeepergatekeeper

wsgramwsgram

WMSWMS

gatekeepergatekeeper

VOMSVOMS

VOMSVOMS

GlobusGlobus

url://plug-insplug-ins

JSAGAJSAGA

JSAGA 11

OpenPlastOpenPlast

OpenPlastOpenPlast

WorldWorld

Grid Grid

EGEEEGEE

OpenPlastOpenPlast

localhostlocalhost

CC-IN2P3CC-IN2P3

http://tar://

srb://

gsiftp://

srm://lfn://

gatekeepergatekeeper

gatekeepergatekeeper

wsgramwsgram

WMSWMS

gatekeepergatekeeper

VOMSVOMS

VOMSVOMS

GlobusGlobus

Transfer path depends on…Transfer path depends on…

grid or site– network filtering policy

– commands available on workers

– services available from workers (close Storage Element, shared FS)

– supported context instances

data to stage– shared by several jobs– installed on some worker nodes– file size– required data protection level

execution service– protocols supported for staging

transfer protocol– access mode (RO, WO, RW)– third-party transfer– supported data protection level

OPlast

OPlast

OPlast

OPlastGSIFTPGSIFTP

OPlastEGEECA

SMTPSMTP GSIFTPGSIFTP GSIFTPGSIFTP HTTPHTTPOPlastSRBSRB

jobjob

jobjob

jobjob

CC

RR11

EE11

C'C'

EE11

RR11

CC C'C' common

result

std-error

JSAGA 12

EE11

RR11

CC C'C'C''C'' common

EE EEsrcsrc executable

DD11input data

result

std-error

OPlast

OPlast

OPlast

OPlast

OPlastEGEECA

SMTPSMTP GSIFTPGSIFTP GSIFTPGSIFTP HTTPHTTP

GSIFTPGSIFTP

OPlastSRBSRB

Transfer path depends on…Transfer path depends on…

grid or site– network filtering policy

– commands available on workers

– services available from workers (close Storage Element, shared FS)

– supported context instances

data to stage– shared by several jobs– installed on some worker nodes– file size– required data protection level

jobjob

jobjob

jobjob

C"C"EEsrcsrc DD11

EE

CC

RR11

EE11

C'C'

TARTARTARTAR

execution service– protocols supported for staging

transfer protocol– access mode (RO, WO, RW)– third-party transfer– supported data protection level

iGetiGet

JSAGA 13

Example of generated graphExample of generated graph

Data flowData flowData flowData flowEE11

RR11

CC C'C'C''C'' common

EE EEsrcsrc executable

DD11input data

result

std-error

OPlast

OPlast

several protocols used,but

only 3 jobs submittedon 1 grid…

JSAGA 14

ConclusionConclusion

Assets of JSAGA– does not sacrifice scalability and efficiency for abstraction

• thanks to design (in particular the definition of plug-in interfaces)

• thanks to caching mechanisms

– does not only hide heterogeneity of middleware, but also hides heterogeneity of grid infrastructures

• thanks to the description of used grid infrastructures (configuration file)

• thanks to the information provided by the plug-ins

• thanks to the information provided by the user in his job description

http://grid.in2p3.fr/jsaga/

JSAGA 15

Related projectsRelated projects

JSAGA is used by…– Elis@

• a web portal for submitting jobs to industrial and research grid infrastructures

– JJS (Java Job Submission)• a tool for submitting job collections to EGEE

• optimized for short-life jobs (resource selection based on QoS observed while submitting jobs)

– JUX (Java Universal eXplorer)• a multi-protocol file browser

ApplicationsApplications

//