P-GRADE and WS-PGRADE portals supporting desktop grids and clouds

Peter KacsukMTA SZTAKIkacsuk@sztaki.hu

2

Parameter study workflow Parameter study workflow execution support in P-GRADE portalexecution support in P-GRADE portal

GEN

SEQ

COLL

SEQSEQSEQ

Parameter sweep jobs:

To be executed in

Grids, Desktop Grids

and Clouds

Generates input

parameter space

Evaluates the results of the

simulation

P-GRADE Portal

3

Motivation: Local DG in Univ. of WestminsterMotivation: Local DG in Univ. of Westminsterbased on BOINC (SZTAKI Desktop Grid)based on BOINC (SZTAKI Desktop Grid)

3

1

2

34

5

6

1. New Cavendish Street 576 nodes2. Marylebone Campus 559 nodes3. Regent Street 395 nodes4. Wells Street 31 nodes5. Little Tichfield Street 66 nodes6. Harrow Campus 254 nodesTotal:

1881 nodes

Lifecycle of a node:1. PCs basically used by students/staff2. If unused, switch to Desktop Grid

mode3. No more work from DG server ->

shutdown (green solution)

4

Generating DG applications

• To port an application to a BOINC system requires three steps:1. Registering the application at the BOINC server2. Creating the master code running on the server3. Creating the client code

• In BOINC Step 2 and 3 require the modification of the original application and this is not trivial

• Using the gUSE/WS-PGRADE environment:– Step 1 by the DG system administrator– Step 2 and 3 are done nearly automatically without any

code modification

5

Automatic generation of master and client code

• To facilitate Step 2 and 3 SZTAKI developed the DC-API (Distributed Computing API) as part of the SZTAKI Desktop Grid (SZDG) package

• DC-API can automatically generate WUs for PS jobs arriving from the workflow

• However, DC-API still requires the modification of the original application to create the client code

• SZTAKI has also developed Genwrapper a generic wrapper that can– eliminate the boincification of the code– automatically generates the client code without the

modification of the original code

6

gUSE architecture and usage

WS-PGRADEWS-PGRADE

Workflow EngineWorkflow Engine

Workflow storageWorkflow storage File storageFile storage

EGEESubmitter

EGEESubmitter

Dedicatedsite

Dedicatedsite

gUSEServices

Meta-brokerMeta-broker

Desktop GridSubmitter

Desktop GridSubmitter

LocalSubmitter

LocalSubmitter

Web ServiceClient

Web ServiceClient

DatabaseClient

DatabaseClient

WMSWMS DG serverDG server WebService

WebService DBMSDBMS

User action, external event or time triggering

7

Connecting gUSE with DGs in CancerGrid

DG ServerBOINCServer

Components

BOINCTaskDB

Sche-duler

Dataserver

QueueManager

DC

-AP

I m

aste

rWU

Job Database(Description of Jobs:Apps, Args, I/O files)

Jobdescr.

gUSE(Workflowenactor)

gUSE DesktopGridgUSE-DG integration

Schedulingpolicy

Batchcreation

gUSEStorage

gUSEWS

Submitter

WS-PGRADE

(User IF)

(WF repre-sentation)

BOINC client

GenWrapper forbatch execution

DC-API cli

LegacyApplication

BOINC client

GenWrapper forbatch execution

DC-API cli

LegacyApplication

gUSE 3G

Bridge SubmitterJob

descr.

gUSELocal

Submitter

8

UoW Local DG

Protein Molecule Simulation using AutoDock

WS-PGRADE portal

9

PS job execution in Grids by P-GRADE

P-GRADE Portal

•3G Bridge

Target Grid

Plugins

•gLite Grid

•ARC Grid

•BOINC Grid

EC2, Eucalyptus

EC2, Eucalyptus

10

P-GRADE portal → 3G Bridge

P-GRADE PortalP-GRADE Portal

FilesFiles

WSClientWSClient

TomcatTomcat

RuntimeRuntime3G

BridgeJobDB

3GBridge

JobDB

Qu

eu

e M

an

ag

er

Grid

Han

dler

Inte

rfac

e

GridPlugin

2

BOINCPlugin

3

CloudPlugin

n

WS

Sub

mitt

er

WS

Sub

mitt

er

GridPlugin

1

DownloadManager

DownloadManager

HT

TP

DHT

TP

D

Submit jobCheck status

GetOutput

11

Host A

3G-Bridge

DC-API – Condor Plugin

Cloud Plugin

Queue 1

……

Queue 2

……

Condor Submitter/ Master

Cloud Resource 2

(Condor Worker)

Cloud Resource N

(Condor Worker)

…

Scheduler

P-Grade Portal

Amazon/ Eucalyptus Cloud Interface

Cloud Resource 1

(Condor Worker) Legend

InformationCommandJob

1. Job is submitted from P-Grade Portal to the 3G-Bridge.

2. 3G-Bridge submits the job to a Condor Cluster using the DC-API Condor Plugin (Queue 1).

• The cluster consists of workers running in the cloud.

3. The Scheduler keeps track of the number of jobs in the Condor cluster and of the number of the running Cloud Resources (workers).

4. If the cluster is overcommitted, the Scheduler starts new workers by submitting a job to the 3G-Bridge queue of the Cloud Plugin (Queue 2).

5. If the cluster under utilized, the Scheduler stops some workers (cloud resources) by cancelling some jobs in the Cloud Plugin Queue (Queue 2).

P-GRADE portal supporting Clouds

12

Conclusions

• By the P-GRADE portal you can submit PS jobs to– Grids– Local clusters– BOINC desktop grids– Clouds

• By writing new target DG plugins you can easily add new type of DG resources

• By writing new target Cloud plugins you can easily add new type of Cloud resources

Thank you