General Intro to GridLab Jarek Nabrzyski et al. [email protected] [email protected] Poznań Supercomputing and Networking Center

General Intro to GridLab

www.gridlab.org

Jarek Nabrzyski et al.

[email protected]

[email protected]ń Supercomputing and

Networking Center

GridLab Conference, Eger, 31 March-1 April 2003

GridLab Project

Funded by the EU (5+ M€), January 2002 – December 2004Application and Testbed oriented

Cactus Code, Triana Workflow, all the other applications that want to be Grid-enabled

Main goal: to develop a Grid Application Toolkit (GAT) and set of grid services and tools...:

resource management (GRMS), data management,monitoring, adaptive components, mobile user support, security services,portals,

... and test them on a real testbed with real applications


GridLab Members

PSNC (Poznan) - coordination AEI (Potsdam) ZIB (Berlin) Univ. of Lecce Cardiff University Vrije Univ. (Amsterdam) SZTAKI (Budapest) Masaryk Univ. (Brno) NTUA (Athens)

Sun MicrosystemsCompaq (HP)

ANL (Chicago, I. Foster) ISI (LA, C.Kesselman) UoWisconsin (M. Livny)

collaborating with:Users!

EU Astrophysics Network,

DFN TiKSL/GriKSL

NSF ASC Project

other Grid projectsGlobus, Condor,

GrADS,

PROGRESS,

GriPhyn/iVDGL,

CrossGrid and all the other European Grid Projects (GRIDSTART)

other...


GridLab Aims

Get Computational Scientists using the “Grid” and Grid services for real, everyday, production work (AEI Relativists, EU Network, Grav Wave Data Analysis, Cactus User Community), all the other potential grid apps

Make it easier for applications to make flexible, efficient, robust, use of the resources available to their virtual organizations

Dream up, prototype, and test new application scenarios which make adaptive, dynamic, wild, and futuristic uses of resources.


What GridLab isn’t

We are not developing low level Grid infrastructure,

We do not want to repeat work which has already been done (want to incorporate and assimilate it …)

Globus APIs,

OGSA,

ASC Portal (GridSphere/Orbiter),

GPDK,

GridPort,

DataGrid,

GriPhyn,

...


GridLab end user requirements

Application oriented environment,

Applications on one or more virtual organisations,

Flexible, easy-to-use, simple interfacesresources, jobs, and data (including compiling, tracking jobs, cataloguing data),

Means to make efficient and effective use of resources,

Robustnesssmart adaptivity, complete control and fail safety are available on all levels,

The ability to work in a disconnected environment,

Mobile working,


GridLab end user requirements

From laptops to fully deployed Virtual Organisations,

Complexity hidden as much as possible,

Collaborative infrastructure,

The infrastructure for all classes of applications

The infrastructure must provide capabilities to customise choice of service implementation (e.g. using efficiency, reliability, first succeeding, all)


Solution...

GAT – a layer between apps and emerging grid technologies

GridLab testbed/VO

Close cooperation between developers and deployers

End Users

GAT Tool Developers

Grid Infrastructure Developers

GAT-APIDevelopers


The Grid is complex …

Monitoring

Resource Management

InformationSecurity

DataManagement

GLOBUS

ApplicationManager

Logging

NotificationMigration

Profiling

SOAP WSDL Corba OGSA Other

Other GridInfrastructure?

Application

“Is there a better resource I could be using?”


…need to make it easier to use

GAT

Application

“Is there a better resource I could be using?”

GAT_FindResource( )

The Grid


GridLab Architecture


GridLab Architecture


Grid Application Toolkit

The GAT provides functionality through a carefully constructed set of generic high-level APIs, through which an application will be able to call the underlying grid services,

Set of application developer APIs for Grid tools, services and software libraries, (and example implementations) that support the development of grid-enabled applications (open source!)

Usable from any high level “application” (any generic code, Cactus, Triana, Portals, Scripts, …)


GAT: AIM

Abstract Grid capabilities (services) from the application developer.

Application developer concentrates on the functionality as needed by the application.

Hide complexity.

Provides a layer (buffer zone) between applications and the Grid.


GAT

More or less …Set of calls GAT_ToolOrService(arguments)

Your chosen tools/services: resource broker, information server, application manager, grid monitoring, data manager, notification, etc.

Set of APIs for dealing with the GAT (registration, information, errors, fault tolerance)


GAT Engine

When an application makes a GAT-API call, the engine searches through an internal database of adaptors for the requested capability and calls it


GAT Adaptor

Interface between GAT Engine and one or more capabilitiesTranslates user requests to appropriate interface syntax for a capability providerActive adaptors change dynamicallyIncludes “security context”Return appropriate error codes

ExamplesOGSA adaptor (provides many capabilities)Globus adaptor (directly talk to gatekeepers)Adaptors for each GridLab service provider“Local” adaptors (GAT_MoveFile => “cp”, GATFindResource => “localhost”)


The Same Application …

Application

GAT

Application

GAT

Application

GAT

Laptop The GridSuper Computer

No network! Firewall issues!


GridLab services

Software environment for Grid-enabling scientific applications

GridLab services, third party services and various core-grid services will be supported by GAT

In the advent of the Open Grid Service Architecture (OGSA), GridLab's architecture will revolve around the notion of services,

all the GridLab services will be OGSA compliant

currently all the services are Web Services based

roadmap for Web Services to OGSA transformation is being prepared (6-8 months from now)


GridLab services

A primary aim of this project is to produce a GridLab GAT containing a set of high quality services which provide a complete environment for Grid- enabling generic applications

GridLab services:implement common (strict) security

use common service conneciton protocols (WSDL/OGSA)

are built primarily for Globus infrastructure


What are the GL services?

Authorisation Service (WP6)

Adaptive Services (WP7)

Data Management Services (WP8)

Resource Management System (GRMS) (WP9)

Information Services (WP10)

Monitoring Services (WP11)

Mobile User Support (WP12)


Security (WP6)

Security WP focuses right now on the Authorization Service (AS)

The main requirement is flexibility

The AS is about to provide universal way of defining security policy for the whole Grid, independent of technologies used at lower levels

It should be able to implement most security models for Grids and use many different scenarios at the same time


Adaptive Components (WP7)

Adaptive Components Service (ACS) and the Local Adaptive Components (LAC). ACS provides an interface to query the adaptive system. It currently supports calls to:

rank resourcesestimate transfer timeestimate usage (of some given metric)

LAC uses the monitoring system (shown in blue), to continuously collect data about the resource and applications running on it (load information, queue lengths, network bandwidth to other machines, etc.).


Data Management Services (WP8)

replica catalog prototype was ready at Zakopane meeting

data movement/copy service also since Zakopane meetingsupports reliable gridftp file transfer

is gsi enabled with authentication and delegation

scalable and fault-tolerant replica catalog in M12, based on ongoing research at ZIB

SoapSoap

SoapSoap

Host A

Host B

Host C

Host D

automatic load-balance, fail-over between replica catalogs

external access via SOAP and OGSA

internal communication via more efficient protocol (Corba)


Resource Management (WP9)

Job Receiver

Jobs Queue

BrokerExecution

Unit

ResourceDiscovery

Scheduler

ResourceReservation

PredictionUnit

File TransferUnit

MDSData

Management

ExecutableRepository

GRAM

AuthorizationSystem

Adaptive

WorkflowManager

QoS/SLANegotiation

GRMS

Third partysystems

Chimera DataGrid Crossgrid ...


Information Services (WP10)

Client

Information Service(OGSA)

SOAP over GSI

Users

Software

Services

Firewall

V.O.

C. A.

Cluster

Job Queues

MDS

GSI-SASL


Monitoring System (WP11)

• Implements GGF’s GMA architecture• Fast and robust• Small resource usage• Can monitor hosts and jobs• Can deliver event notifications


Mobile User Support (WP12)

Grid Services

Network Environment / Grid

Portal

Applications

User

Mobile device


Services - Summary

GridLab Services are the primary capability providers for GAT

They can be used by other projects and production environments

GridLab can also accomodate services from other projects, when necessary

First pilot implementations are ready and have been used for several demos including SC02, where GridLab Team working with GGTC won or co-won 2 Challenge Competitions


GridLab Testbed

Managed by Masaryk University, Czech RepublicOperational since March 2002Most of the machines are not dedicated, except one cluster at PSNCBuilt on existing middleware:

Globus Toolkit 2.0 -> now migrating to 2.2MDS 2PKI/GSI for secure access

Provide functional Grid environment for development and testing of GATs, services and their componentsProduction stability a goalNo long production jobsA core team of the Global Grid Testbed Collaboration


Highlights

The project is up and running!All partners are now contributing at contractual level

Project administrative and managerial structure established with minimum resources

First testbed delivered with real production demos

First (very successful) demos have been given

2 all-hands meeting a year


Project Schedule

Project started on 1st of January 2001

Requirements documents: Month 3-4

Design documents: Month 6-12 (continously updated)

Testbed running: Month 3

First prototypes: Month 12 (actual: 9)

First release: Month 18-24

Second release: Month 28-36


More info / summary

www.GridLab.org

[email protected], [email protected]

[email protected]

Bring your application and test it with the GAT and our services.


The goal of this meeting

„Let’s make the most advanced grid in the world” Michael Russell,

AEI

Documents

General Intro to GridLab Jarek Nabrzyski et al. [email protected] [email protected] Poznań Supercomputing and Networking Center