GridLab: A Grid Application Toolkit and Testbed Jarosław Nabrzyski GridLab Project Manager Poznań Supercomputing and Networking Center, Poland [email protected]

GridLab:GridLab:A A Grid Application Grid Application ToolkitToolkit and Testbed and Testbed

Jarosław NabrzyskiGridLab Project ManagerPoznań Supercomputing and Networking Center, Poland

[email protected]@gridlab.org

www.GridLab.org

mailto:[email protected]

mailto:[email protected]

MANs Meeting, Warsaw, NASK, September 2002

GridLab ProjectGridLab Project

EU Funded (5+M€) 2002-2004 Application and Testbed

orientated Applications and grid

application developers are in the center of the project


GridLab MembersGridLab Members PSNC (Poznań) -

coordination AEI (Potsdam) ZIB (Berlin) Univ. of Lecce Cardiff University Vrije Univ.

(Amsterdam) SZTAKI (Budapest) Masaryk Univ. (Brno) NTUA (Athens) Sun Compaq (HP) ANL (Chicago, I. Foster) ISI (LA, C.Kesselman) UoWisconsin (M. Livny)

Also collaborating with: Users!!!

(EU Astrophysics Network,High Performance Computing Group, Ottawa)

DFN TiKSL/GriKSL NSF ASC Project GrADS Project GriPhyn/iVDGL


GridLab on a mapGridLab on a map


GridLab AimsGridLab Aims

Get Computational Scientists using the “Grid” and Grid services for real, everyday, production work (AEI Relativists, EU Network, Grav Wave Data Analysis, Cactus User Community).

Make it easier for applications to make flexible, efficient, robust, use of the resources available to their virtual organizations.

Dream up, prototype, and test new application scenarios which make adaptive, dynamic, wild, and futuristic uses of resources.


What GridLab Isn’t …What GridLab Isn’t …

Don’t want to develop low level Grid Infrastructure

Don’t want to repeat work which has already been done (want to incorporate and assimilate it … Globus APIs, OGSA, ASC Portal (GridSphere/Orbiter), GPDK, GridPort, DataGrid, GriPhyn)


GridLab end user GridLab end user requirementsrequirements

Give us an application oriented environment, Applications running on resources of one or

more virtual organisations, Flexible, easy-to-use, simple interfaces to

resources, jobs, and data (including compiling, tracking jobs, cataloguing data)

Give us means to make efficient and effective use of resources,

Fault tolerance is a key requirement, The ability to work in a disconnected

environment, Mobile working,


GridLab end user requirements GridLab end user requirements (cont)(cont)

The ability to run in environments as minimalistic as laptops with no grid infrastructure to fully deployed Virtual Organisations

Complexity should be hidden as much as possible,

Provide a collaborative infrastructure, The infrastructure must cater for all classes of

applications, from lightweight to largescale, The infrastructure must provide capabilities to

customise choice of service implementation (e.g. using efficiency, reliability, first succeeding, all).


SolutionSolution

“Grid Application Toolkit” or GATProvides a layer between applications and emerging grid technologies. Provides an application developer orientated API, allowing the flexible use of different tools and services, as well as providing protection from developing software.

“GridLab Testbed/VO”Diverse controllable environment for developing and testing applications and tools, software maintained by people who know it.

End Users

GAT Tool Developers

Grid Infrastructure

Developers

GAT-APIDevelopers


General ArchitectureGeneral Architecture


What is the GAT ?What is the GAT ?

Set of application developer APIs for Grid tools, services and software libraries, (and example implementations) that support the development of grid-enabled aplications (open source!)

Usable from any high level “application” (any generic code, Cactus, Triana, Portals, Scripts, …)

More or less … Set of calls GAT_ToolOrService(arguments) Your chosen tools/services: Resource broker,

information server, application manager, grid monitoring, data manager, notification, etc, etc

Set of APIs for dealing with the GAT (registration, information, errors, fault tolerance)


GAT ArchitectureGAT Architecture

Applications

GAT-API

GridLab Services

Grid Core Services

Cactus, Triana, Generic Codes, Portals, Scripts

High level calls based on functionality

GAT_FileCopy, GAT_ResourceFind

Resource Broker, Monitoring System,

Data ManagerGRAM, GridFTP, GridSSH, MDS

but not limited to Globus!

(Triana and JXTA)


Proposed GAT Proposed GAT ImplementationImplementation

Applications

GAT Engine

GAT Adaptors

Grid Infrastructure(GAT/GridLab+base/core grid services)

Cactus, Triana, Generic Codes, Portals, Scripts

Layer providing access to services

Resource brokers, basic grid services (GRAM,

GSIFTP, MDS)

Library containing registration routines and empty implementations

GAT-API


GAT ExampleGAT Example

Machine A

GridftpGridscp

Machine B

Gridscp

“Application” (Cactus Migration Tool, User Portal, Grid Script)

wants to move a file between two other

machines

How to move the file?

Available software

User authentification

Disk properties (disksize,

user quota, inodes)

Network bandwidth


GAT ExampleGAT Example

GAT_FileCopy(GAT, “ThisMachine:ThisFile”,”ThatMachine:ThatFile”)

GAT_GFTPAdaptorIGAT_FileCopyIGAT_FileMove

IGAT_MkDir

GAT API call from Cactus Application, or Portal, or

Python script

GridFTP Adaptor says it knows how to both Copy and Move files between machines

GridSSH Adaptor only knows how to Copy files

GAT_GSCPAdaptorIGAT_FileCopy

GAT EngineEngine decides (by different means) how to respect this

request


ImplementationImplementation

GridLab is an early OGSA adopter, however We start with the web services

implementation (all the GridLab services will be web services based and in the next step OGSA compliant)

Several implementations of GAT will be developed


Workpackages Workpackages

WP1: Grid Application Toolkit (AEI) This is a key component of GridLab - link between Grid

middleware and applications, usable by any conforming application or middleware component. Requiring input from, and connecting to, most other workpackages and components.

WP2: Cactus Grid Application Toolkit (AEI) Provides an extended GAT interface for Cactus, a very

general toolkit framework supporting different Grid applications, from astrophysics to chemical engineering. Cactus will be one of the primary application drivers for the GAT, and the project generally.

WP3: Work-flow Application Toolkit (CARDIFF) Will develop Grid capabilities for a widely used dataflow

programming environment, Triana, used in gravitational wave and other data analysis areas.


Workpackages (cont.)Workpackages (cont.)

WP4: Grid Portals (AEI) Will be highly application driven, aimed at providing uniform,

flexible and intuitive user access to Grid resources from anywhere, as well as administration tools for maintaining a Grid environment.

WP5: Testbed management (MU) Will administrate and maintain an active development testbed

across roughly a dozen EU sites (leveraging the work of the EGrid), deploying technologies as they are developed by the project. This workpackage will also coordinate with sites in the USA-based NCSA Alliance and others to test and develop interoperability.

WP6: Security (PSNC) Will develop the required security mechanisms and will ensure

the integration of all the technologies developed under other WPs, taking into account the various local security requirements and state of the art solutions.



WP7: Adaptive Application Components (VU) develops a set of components and APIs to be plugged into the

toolkit, for example to take monitoring information and implement basic techniques for short-term forecasting and behavior adaptation/optimization.

WP8: Data Handling and Visualization (ZIB) will provide Grid aware techniques for data management, analysis,

and visualization, needed especially for applications that make use of multiple sites in a dynamic, time dependent manner, leaving data unpredictably scattered across the Grid.

WP9: Resource Management (PSNC) will develop resource need estimators, resource brokers, and other

tools, for both Grid users and the applications themselves to make intelligent decisions about which Grid resources should be used at any instant in the lifetime of a simulation.



WP10: Information Services (ISUFI) will extend existing Grid middleware toolkits with

dynamic features needed by applications to select appropriate Grid resources and to provide simulation information to collaborative user groups.

WP11: Monitoring (SZTAKI) will develop new components that will fit in the

general Grid monitoring architecture to support application steering, adaptive monitoring, and automatic analysis and prediction of performance data.

WP12: Access for mobile users (PSNC) will develop and test Grid access and monitoring

technologies through a variety of mobile devices,



WP13: Information Dissemination and Exploitation (PSNC) will ensure the active dissemination of the project results

through a variety of channels, including active participation in international organizations (e.g. GGF), co-development with other Grid projects in the USA and EU, participation in international conferences, training programs, instruction of GridLab technologies into various communities, and introduction into the commercial vendor world.

WP14: Project Management (PSNC) day-to-day scientific, financial and administrative

management of the project, including careful orchestration and monitoring of work across groups, major project decisions, liaisons with external projects and with the international advisory board, reporting


More info / summaryMore info / summary

www.GridLab.org [email protected],

[email protected] [email protected]’re welcome to join our testbed We are open for collaborations!

Documents

GridLab: A Grid Application Toolkit and Testbed Jarosław Nabrzyski GridLab Project Manager Poznań Supercomputing and Networking Center, Poland [email protected]