gLite

A.GhiselliINFN-CNAF

Bologna

tf-progettazioneCASPUR, 26-27 marzo 2009

What is a Grid

2

In 1998, I. Foster and C. Kesselman wrote in The Grid:

Blueprint for a New Computing Infrastructure:

“A computational grid is a hardware and software infrastructure that provides dependable, consistent, pervasive access to highend computational capabilities.”

In 2002, I. Foster wrote in What is the Grid? A Three

Point Checklist:

“... The essence of the [definition] can be captured in a simple checklist,... a Grid is a system that:

1. coordinates resources that are not subject to centralized control…

2. … using standard, open, general purpose protocols and interfaces...

3. … to deliver nontrivial qualities of service.

3

4

Architettura e servizi di Grid e confronto con l’architettura di Internet

ip

linkfabric

application

collective

Resource

Connectivity transport

application

Inte

rnet

pro

toco

l arc

hit

ect

ure

Gri

d p

roto

col

arc

hit

ect

ure

GRAM

GSI

neck

Resource BrokerGIS, RLS..

CE, SE,..

Computer, storage, rete, apparati sperimentali…

5

OGSA capability gLite UNICORE ARC

Security.AttributeAuthority VOMS (AC+SAML) WS-UUDB, SAML-VOMS

VOMS+SAM

Security.Accounting DGAS, APEL RUS  SGAS, APEL

Data.Management.Storage StoRM, DPM SMS Smart-SE, dCache

ARC Gridftp

Data.Management.Transfer FTS JMS  FTS, gridftp2

Data.Access.Relational      

Data.Access.FlatFiles GFAL TSI  ARC Caching

Information.Model GLUE GLUE (future) GLUE, arcschema

Information.Discovery OpenLDAP   OpenLDAP

Information.Monitoring GridICE, LLview, CIS, RSS NG-Monitor

(R-GMA)

ExecMan.ExecService GT2 (4) Gram, TSS, OGSA-BES Grid-Manager+AREX (BES), gridftp interface

CREAM+BES

ExecMan.JobManager WMS XNJS  ARC Client

ExecMan.CandidateSetGen  

ExecMan.ExecPlannService    

EGI staks

6

UserInterface

VOMSatlas

Grid services

CE SE

Computing Element

Storage Element

GIIS GRIS1

GRIS

Information SystemResource Broker

UserInterface

CREAM

VOMS cms

WorkerNodeWorkerNode

...WorkerNode

CE SE

Computing Element

Storage Element

GIIS GRIS1

GRIS

GRAM

WorkerNodeWorkerNode

...WorkerNode

Grid Monitoring(GridICE)

LFCLogical File Catalogue

VO GPboxcms

Site GPboxcms

VO GPboxatlas

account system

AuthorisationPolicies

AuthenticationCredential manager

Computers and storage

Job Submission / WMS

Workload Management System (WMS)• Essential functionality:

– manage grid jobs from submission to completion– match and rank suitable resources– deal with failures (resubmit jobs transparently)

• Highlights:– job rate up to 100Kjob/day (single job)

– job collections, bulk matchmaking– parallel matchmaking– prioritized queue

Job Management / LB

Logging and Bookkeeping (L&B)

• Essential functionality:– gather information on job execution on grid

components– provide users with digested consistent view on job

state

• Highlights:– knowledge on job processing concentrated in state

machine– reliable information delivery

• Short term plans:– integration with monitoring tools (RTM, Dashboard,

GridView)– advanced authorization

Job Submission/cream CE• The CREAM (Computing Resource Execution And Management) Service is a

simple, lightweight service that implements all the operations at the Computing Element (CE) level; its well-defined WebService-based interface and its implementation as an extension of the Java-Axis servlet (running inside the Apache Tomcat container) provide interoperability with clients written in any programming language and running on any computer platform.

• The CREAM interface is well-defined using the Web Service Description Language (WSDL); anyone can generate his/her CREAM client by simply filling in the stub code generated by WSDL parser (gSOAP for C/C++, Axis for Java, Perl module for perl).

• CREAM Main functionality: – Job Submission

• Possibility of direct staging of input sandbox files GLITE WMS JDL compliance (with CREAM-specific extensions)

• Support for batch and MPI jobs • Support for bulk jobs being integrated

– Manual and automatic proxy delegation – Job Cancellation – Job Info with configurable level of verbosity and filtering based on submission time and/or

job status – Job List – Job Suspension and Resume – GSI based authentication – VOMS based authorization – Job Purge for terminated jobs – Possibility (for admin) to disable new submissions

CREAM can be used

• by the Workload Management System (WMS), via the ICE (Interface to CREAM Environment) service

• by a generic client, e.g. an end-user willing to directly submit jobs to a CREAM CE. A C++ command line interface and Java clients are available.

Thanks to ICE (a gSOAP/C++ intermediate layer) CREAM can receive job operations

directly from a Grid WMS (Resource Broker). The ICE layer subscribes to the CEMon

service in order to asynchronously receive notifications about job status changes. In

case some notifications are lost, ICE performs synchronous status polling for jobs for

which it hasn't received status for some time. To maintain its subscriptions ICE

periodically checks them and renews the expiring ones. CEMon gets informations about

job status from the CREAM data persistency backend through JNDI APIs. ICE basically

has the role that the Job Controller (LC) and Log Monitor (LM) play when dealing with

non-CREAM CEs

Job Submission / cream

Supported Job Types

• Batch-like• Dag workflow

• Collection• Parametric

• MPI

• Interactive

11

Compound

K

Storage Element for files

• SE definition– The Storage Element (SE) is responsible for saving/retrieving files to/from the

local storage that can be a disk or a mass storage system. It manages disk space for files and maintains the cache for temporary files.

• SE Interfaces: there are two interfaces into the SE– The first interface is the Storage Resource Management (SRM) interface. This

interface allows the client to manage the storage space - to allocate space for jobs, to prepare data to be retrieved through a certain protocol, etc.

– posix-like File I/O.

• Files and Replica catalogs– In the Grid the user identifies files by logical file names (LFNs). The LFN is the

key by which the users locate the actual locations of their files. We refer to file replicas if the file has several managed copies, being tracked by the Replica Catalogue (Globus-RLS (Replica Location Service), LFC (LCG File Catalogue)).

12

13

SE – SRM implentations

An SRM standard interface hides storage characteristics.

From client point of view no difference appears!

SRM Interface

SRM-dCache

Access

StoRMSRM-Castor

Storage System

GPFS

Storage System

DCACHE

Storage System

CASTOR

Access Protocol

rfio

Access Protocol

dcap

Access Protocol

posix

Data Access

ManagementAccess

Access Protocol

posix

rfioand

da R.Zappi

StoRM overview

• StoRM is a storage resource manager for disk based

storage systems, implementing the SRM interface v2.2.• It is designed to take advantage from high performing

cluster file system, as GPFS or Lustre, but it supports also every standard POSIX FS (anche Amazon S2).

• It allows direct access (through the protocol le:// ) to the storage resource, as well as other standard grid protocol as gsiftp and rfio.

• Authentication and authorization are based on the VOMS• credential.• Permission enforcing are based on setting physical ACLs

on files and directories.

14

StoRM role in a site

Data Management

Grid file access library (GFAL)– POSIX-like access to SE– SRM-compliant back-ends– unified access for LFN, GUID, SURL, and TURL

LCG file catalog (LFC)– map LFN to GUID and SURL– authorization on logical namespace

File transfer service (FTS)– manage transfers between storage elements– asynchronous bulk requests– parform transfers reliably, monitor progress

17

Collaborating e-Infrastructures

Potential for linking ~80 countries by 2008

18

EGI: The future evolution

Testbeds Utility ServiceRoutine Usage

National

Global

European e-Infrastructure

19

EGI_DS Schedule

Duration 27 months:

5 6 7 8 9 10 11 12 1 2 3 4 5 6 7 8 9 10 11 12 1 2 3 4 5 6 7 8 9 10 11 12 1 2 3 4 5 6 7 8 9 10

Develop EGI Proposal NGIs signing Proposal

Start of EGEE-III

Final Draft of EGI Blueprint Proposal

EGI Blueprint Proposal

EGEE-III transition to EGI-like structure

EGI Entity in place

EU Call Deadline for EGI Proposal

Submission of EGEE-III

Start of EGI Design Study

2008 2009 2010

EGEE-II (2YEARS) EGEE-III (2YEARS) EGI operational