Introduction to UNICORE - twiki.cern.ch€¦ · TSI 16/09/2010Rebecca BreuFolie 6. When a job is being submitted ... Client Gateway unicorex XUUDB TSI Client establishes SSL-Connection

Mit

glie

dd

erH

elm

hol

tz-G

emei

nsc

haf

t

Introduction to UNICORE

16/09/2010 Rebecca Breu

Some Facts About UNICORE

First version has been developed in 1997

UNICORE has been worked on in several German andEuropean projects since

Integrated, complete middleware stack, including user clients

Written in Java, thus many supported operating systems(*nix, Mac, Windows)

Many supported batch systems (LoadLeveler, Torque, SunGrid Engine, . . . )

Web services for communication (WS-RF)

SSL (Secure Sockets Layer) for transport

16/09/2010 Rebecca Breu Folie 2


UNICORE Architecture

Global Registry:Central point of a UNICORE gridKeeps track of all available services

Gateway:”Door to outside world” in firewallmay serve several resources behind one firewall

UNICORE/X:Central point for job processing and managingChecks user certificate with XUUDB

XUUDB (UNICORE user database):Mapping between user certificates, user logins, roles

TSI (Target System Interface):Submits jobs to batch system

Components use SSL connections


The Global Registry

The Global Registry:

Provide clients with information about services

Two kinds: global / local

Global or central registry:

Serves as a ‘Grid’Knows all target systems and workflow servicesServices dynamically register with (one or more) registries

Local registry per service container (e.g. UNICORE/X)

For registering service instances

Full WS-RF Service


When a job is being submitted . . .

Client

Gateway

unicorex

XUUDB

TSI

Client establishes SSL-Connection to Gateway

Client contacts UNICORE/X via Gateway

Client sends signed abstract job to UNICORE/X

UNICORE/X asks XUUDB if the user belongingto the certificate is allowed job execution

UNICORE/X gets login from XUUDB

UNICORE/X translates abstract job intomachine dependent script

UNICORE/X sends machine dependent script toTSI



Client

Gateway

unicorex

XUUDB

TSI










Client

Gateway

unicorex

XUUDB

TSI










Client

Gateway

unicorex

XUUDB

TSI










Client

Gateway

unicorex

XUUDB

TSI










Client

Gateway

unicorex

XUUDB

TSI










Client

Gateway

unicorex

XUUDB

TSI









The XUUDB

XUUDB:

Maps user certificates to logins on that machine

Assigns roles (user, admin, . . . )

Nr|GcID |Xlogin |Role |Projects |DN

---------------------------------------------------------------------------

1 |OMII_EI |rbreu |user | |CN=Rebecca Breu , OU=JSC , OU=Forschungszentrum Juelich GmbH , O=GridGermany , C=DE

2 |OMII_EI |sandra |user | |EMAILADDRESS=s.bergmann@fz -juelich.de , CN=Sandra Bergmann , OU=ZAM , O=Research Center Juelich , L=Juelich , ST=NRW , C=DE


The TSI

The TSI . . .

forks a process which runs with the user’s ID

creates a temporary directory on the target system (uspace)

changes current working directory to uspace

submits job to local batch system

Input and ouput:

all input needed for job has to be copied into the uspace

all output that is to survive the end of job execution has to becopied elsewhere

Terms used:

File import: File tranfer from somewhere into uspace

File export: File tranfer from uspace to somewhere


The UNICORE/X

Authorises requests using the authorisation service XUUDB

Translates abstract job into concrete job for target system viathe IDB

Provides UNICORE Atomic Services (UAS):

Job Management ServiceStorage Management ServiceFile Transfer ServiceTarget System Sevice. . .


UAS: Target System Factory Service


UAS: Target System Service

Abstract web service interface to target system

List of available applicationsDescription of available resources (e.g. CPUs, RAM)Job submissionLinks to jobs and storages (e.g. user home)

Security & Privacy

Authorisation: Users’ target system instances and jobs protectedby configurable XACML policySecure job submission through message signing


Jobs

Abstract job definitions:

Given in JSDL (Job Submission Description Language)

Job name, description

Resource requirements (RAM, numer of CPUs needed, . . . )

Information about file transfers

An application name and version

Job instances:

Web service interface to submitted jobs

Job status (queued, running, finished, failed, ...)

Exit code of the application

Link to storage that contains input and output files

Have a life time


UAS: Jobs and Storages


UAS: Filetransfers


UAS: File Transfer Protocols

Pluggable mechanism

Both for client-server and server-server transfers

Supported protocols:

Baseline File Transfer (BFT)OGSA ByteIOGridFTPUDT


IDB: Incarnation Database

The IDB is a file with rules for translating abstract jobs intoexecutable scripts. (Belongs to the UNICORE/X configuration.)

<idb:IDBApplication >

<idb:ApplicationName >Bash shell</idb:ApplicationName >

<idb:ApplicationVersion >3.1.16 </idb:ApplicationVersion >

<jsdl:POSIXApplication xmlns:jsdl="http: // schemas.ggf.org/jsdl /2005/11/ jsdl -posix">

<jsdl:Executable >/bin/bash</jsdl:Executable >

<jsdl:Argument >--debugger$DEBUG?</jsdl:Argument >

<jsdl:Argument >-v$VERBOSE?</jsdl:Argument >

<jsdl:Argument >$ARGUMENTS?</jsdl:Argument >

<jsdl:Argument >$SOURCE?</jsdl:Argument >

</jsdl:POSIXApplication >

</idb:IDBApplication >


Define New Applications

To bring new applications into UNICORE:

Define Application in IDB

Access it via UCC or GenericGridBean

For a more customised graphical interface: Write a clientplugin (GridBean)

high-level Java-API



Workflows

Workflow:

Several jobs that depend on each other.

Job excecution after another job has finished

File transfers between jobs

Repeated job execution (loops)

Job execution depending on other job status (if-then-else)


Workflow Services

Client

Workflow Service

Service Orchestrator

unicorexunicorex

Client submits workflow to WorkflowService

Workflow Service creates n normal jobsand passes them to Service Orchestrator

Service Orchestrator submits jobs toUNICORE/X

Service Orchestrator keeps track ofcontstraints (if-conditions, jobs beingexecuted after one another . . . )


Workflow Services

Client

Workflow Service


unicorexunicorex






Workflow Services

Client

Workflow Service


unicorexunicorex






Workflow Services

Client

Workflow Service


unicorexunicorex






UNICORE Clients

Several clients for different purposes:

Rich Client: Eclipse-based graphical client for complex tasks

UCC: Command line client

(Portal Client: Web based client to use with a browser)


Basic Client Configuration

Create a password-protected keystore that holds all keysneededAdd keys to keystore:

User’s private and public keyPublic keys of trusted CAs (must trust signer of involvedgateway keys!)

Add URI of Global Registry


More information

UNICORE Website: www.unicore.eu/

Documentation: www.unicore.eu/documentation

Wiki: sourceforge.net/apps/mediawiki/unicore/


http://www.unicore.eu/

http://www.unicore.eu/documentation

http://sourceforge.net/apps/mediawiki/unicore/

Documents

Introduction to UNICORE - twiki.cern.ch€¦ · TSI 16/09/2010Rebecca BreuFolie 6. When a job is being submitted ... Client Gateway unicorex XUUDB TSI Client establishes SSL-Connection