19
INFSO-RI-508833 Enabling Grids for E-sciencE www.eu-egee.org Ganga 4 – The Ganga Evolution Andrew Maier

INFSO-RI-508833 Enabling Grids for E-sciencE Ganga 4 – The Ganga Evolution Andrew Maier

Embed Size (px)

Citation preview

Page 1: INFSO-RI-508833 Enabling Grids for E-sciencE  Ganga 4 – The Ganga Evolution Andrew Maier

INFSO-RI-508833

Enabling Grids for E-sciencE

www.eu-egee.org

Ganga 4 – The Ganga Evolution

Andrew Maier

Page 2: INFSO-RI-508833 Enabling Grids for E-sciencE  Ganga 4 – The Ganga Evolution Andrew Maier

Ganga Tutorial - A. Maier 2

Enabling Grids for E-sciencE

INFSO-RI-508833

Overview

• Problems with Ganga 3

• Ganga 4 development– Based on the Ganga scripting interface

– Functional decomposition Client Application Manager Job Manager Remote Registry

– Handler Plugin Example

• Next Steps

Page 3: INFSO-RI-508833 Enabling Grids for E-sciencE  Ganga 4 – The Ganga Evolution Andrew Maier

Ganga Tutorial - A. Maier 3

Enabling Grids for E-sciencE

INFSO-RI-508833

Problems with Ganga 3

• Ganga’s current version is 3.0.1– The 3.x.x series is now in maintenance mode

No new features Only bug fixes

• Needs a simple user interface• Lacks a clear job structure• Needs better internal decomposition

– To enhance maintainability

• Would like to integrate new features– Remote registry– Job templates– Merging in addition to splitting– Better support for user plugins

Page 4: INFSO-RI-508833 Enabling Grids for E-sciencE  Ganga 4 – The Ganga Evolution Andrew Maier

Ganga Tutorial - A. Maier 4

Enabling Grids for E-sciencE

INFSO-RI-508833

Ganga 4 development

• Ganga’s next version will be Ganga 4– Discussion on Ganga development started at

the beginning of the year Complete re-implementation decided

– End of main design phase in March

– Design principles have been tested with a set of mini prototypes

Page 5: INFSO-RI-508833 Enabling Grids for E-sciencE  Ganga 4 – The Ganga Evolution Andrew Maier

Ganga Tutorial - A. Maier 5

Enabling Grids for E-sciencE

INFSO-RI-508833

• Ganga 4 is based on the Ganga scripting interface, the GUI will follow later

• Ganga 4 will overcome some of the shortcomings of the current release

– By using a modular design

– Applying functional decomposition of modules

– Having extended functionality

Ganga 4 - Introduction

Page 6: INFSO-RI-508833 Enabling Grids for E-sciencE  Ganga 4 – The Ganga Evolution Andrew Maier

Ganga Tutorial - A. Maier 6

Enabling Grids for E-sciencE

INFSO-RI-508833

Ganga 4 - Introduction

• Ganga 4 – Software Reengineering– Starting point:

recode Ganga 4 from scratch where needed add GUI later

– GPI – Ganga Public Interface 95% based on Ganga3 CLIP

– Ganga Core: fast and reliable self-contained (NO GUI) clear architecture and module dependency

• allow remote registry for sharing jobs (near future)

• allow Client-Server split (far future)

– Functionality reuse (at least at the prototype level) all existing Ganga 3 handlers

Page 7: INFSO-RI-508833 Enabling Grids for E-sciencE  Ganga 4 – The Ganga Evolution Andrew Maier

Ganga Tutorial - A. Maier 7

Enabling Grids for E-sciencE

INFSO-RI-508833

Ganga Public Interface

• The Ganga Public Interface (GPI) represents the logical model of Ganga.

• It describes the objects (jobs, applications, backends…) and their properties.

• For the user the GPI is accessible (to a large extent) via the CLIP (Command Line Interface in Python)

• It is extensible in the sense that each application or backend may define their own set of properties visible to the user.

Page 8: INFSO-RI-508833 Enabling Grids for E-sciencE  Ganga 4 – The Ganga Evolution Andrew Maier

Ganga Tutorial - A. Maier 8

Enabling Grids for E-sciencE

INFSO-RI-508833

Internal architecture

ApplicationManager

Job ManagerRemoteRegistry

Client

• Ganga 4 is decomposed into 4 functional components

• These components also describe the components in a distributed model.

• Strategy: Design each component so that it could be a separate service.

• But allow to combine two or more components into a single service

Page 9: INFSO-RI-508833 Enabling Grids for E-sciencE  Ganga 4 – The Ganga Evolution Andrew Maier

Ganga Tutorial - A. Maier 9

Enabling Grids for E-sciencE

INFSO-RI-508833

Client

ApplicationManager

Job ManagerRemoteRegistry

Client

• Runs the Ganga interface (CLIP, GPI, GUI)

• The user interacts exclusively through the client

• With the client, the user creates, modifies, submits and monitors jobs

• Job configuration is kept in a registry which can be local (within the client) or remote.

Page 10: INFSO-RI-508833 Enabling Grids for E-sciencE  Ganga 4 – The Ganga Evolution Andrew Maier

Ganga Tutorial - A. Maier 10

Enabling Grids for E-sciencE

INFSO-RI-508833

Client

ApplicationManager

Job ManagerRemoteRegistry

Client

• The client should be a thin client (pure python)

• The client can be a command line client or a GUI

• The client interacts with the application manager to configure applications

• It submits and monitors jobs via the job manager

• It keeps state by storing persistent information in the registry

Page 11: INFSO-RI-508833 Enabling Grids for E-sciencE  Ganga 4 – The Ganga Evolution Andrew Maier

Ganga Tutorial - A. Maier 11

Enabling Grids for E-sciencE

INFSO-RI-508833

Application Manager

ApplicationManager

Job ManagerRemoteRegistry

Client

• Prepares and configures the application

• Compiles user code• Sets-up the necessary

environment• Provides information to the

client on available applications, versions, platforms, etc.

Page 12: INFSO-RI-508833 Enabling Grids for E-sciencE  Ganga 4 – The Ganga Evolution Andrew Maier

Ganga Tutorial - A. Maier 12

Enabling Grids for E-sciencE

INFSO-RI-508833

Job Manager

ApplicationManager

Job ManagerRemoteRegistry

Client

• Submits the configured job to the submission backend

• A submission handler submits a job to a backend

– Creates the starter script and the JDL

– performs the monitoring

• The application runtime handler

– prepares the application dependent wrapper script, depending on the backend.

– E.g., DIRAC knows how to run LHCb applications with a different setup as LSF.

Page 13: INFSO-RI-508833 Enabling Grids for E-sciencE  Ganga 4 – The Ganga Evolution Andrew Maier

Ganga Tutorial - A. Maier 13

Enabling Grids for E-sciencE

INFSO-RI-508833

Remote Registry

ApplicationManager

Job ManagerRemoteRegistry

Client

• Keeps track of jobs

• Is a “passive” data store, typically using a database backend

• Keeps a roaming profile of the user jobs

• Keeps track of the job status

• May cache the output of a job in a local SE.

• This component is the most likely to be run on a separate host.

Page 14: INFSO-RI-508833 Enabling Grids for E-sciencE  Ganga 4 – The Ganga Evolution Andrew Maier

Ganga Tutorial - A. Maier 14

Enabling Grids for E-sciencE

INFSO-RI-508833

Ganga 4 - Plans

• Ganga 4 Reengineering– Planned functionality ( -> order of

implementation): Applications: Generic Executable -> Gaudi

(DaVinci, Brunel, Boole,...) Backends: Local -> LSF -> DIRAC -> Glite Registry: Local -> Remote

– Design help: quick series of mini-prototypes to explore new areas e.g. remote registry tests (~ create/update10

jobs/s) using ARDA MD

Page 15: INFSO-RI-508833 Enabling Grids for E-sciencE  Ganga 4 – The Ganga Evolution Andrew Maier

Ganga Tutorial - A. Maier 15

Enabling Grids for E-sciencE

INFSO-RI-508833

Handler Plugin Example

• A practical example for experts:

• Creating a Ganga 4 handler will be easy to write

• As an example: the prototype Ganga 4 application handler for Gaudi:– Create a class derived from GaudiObject

– Define a schema of data the handler requires

– Implement the interface for the handler

Page 16: INFSO-RI-508833 Enabling Grids for E-sciencE  Ganga 4 – The Ganga Evolution Andrew Maier

Ganga Tutorial - A. Maier 16

Enabling Grids for E-sciencE

INFSO-RI-508833

Handler Plugin Example

## Import the GPIfrom Ganga.GPIDev.Base import GangaObject from Ganga.GPIDev.Schema import *[…]class Gaudi(GangaObject):# Set up the schema for this application _schema = Schema(Version(1,0),{'optsfile':FileItem(), 'version':SimpleItem(None), 'platform':SimpleItem(None), 'package': SimpleItem(None), 'appname':SimpleItem(None)})

_category='applications‘

Describe the data needed to configure and run an application

Set the type of your plugin

•Create a class based on GangaObject.•Provide a Schema for you data•Fulfill the interface for the type of handler

Page 17: INFSO-RI-508833 Enabling Grids for E-sciencE  Ganga 4 – The Ganga Evolution Andrew Maier

Ganga Tutorial - A. Maier 17

Enabling Grids for E-sciencE

INFSO-RI-508833

Handler Plugin Example

def configure(self):[…] extra=GaudiExtras()[…]

extra.flatopts=FileParser.writeString(gaudiopts,"expand")

return (None,extra)

def list_choices(self,property):

class GaudiExtras(GangaObject): _schema = Schema(Version(1,0),{"flatopts":

SimpleItem(None)}) _name="GaudiExtras" _category="extras“ _name='Gaudi'

Use a GaudiExtras object to return information added by configure

Create a Schema with the data of the configured app

Set the category of this class to “extras”

Page 18: INFSO-RI-508833 Enabling Grids for E-sciencE  Ganga 4 – The Ganga Evolution Andrew Maier

Ganga Tutorial - A. Maier 18

Enabling Grids for E-sciencE

INFSO-RI-508833

Next Steps

• The first alpha release of Ganga 4 is expected for tomorrow (15/4/05)

• This will be a first integration release to integrate the four components

• This alpha release will be the code base for future development

• A first beta release will contain the functionality of Ganga 3 (without GUI)

• Followed by a user release (possibly with GUI)

Page 19: INFSO-RI-508833 Enabling Grids for E-sciencE  Ganga 4 – The Ganga Evolution Andrew Maier

Ganga Tutorial - A. Maier 19

Enabling Grids for E-sciencE

INFSO-RI-508833

Ganga 4 - Plans

• Ganga 4 Reengineering – General strategy

code for now but design for the future• start with all-in-one-Client but gradually put more

functionality on the Server side limit the number of binary dependencies ability to import Ganga GPI into another python

application

– Goal: by mid-April: reproduce Ganga 3 functionality (NO GUI) First alpha release on Friday (First integration

release)