WS-PGRADE portal and its usage in the CancerGrid project

M. Kozlovszky, P. KacsukComputer and Automation Research Institute of the Hungarian Academy of Sciences

PUCOWO,Zurich, Switzerland10-11/06/2010

2

Motivations of creating gUSE

•To overcome (most of) the limitations of P-GRADE portal:

•To provide better modularity to replace any service•To improve scalability to millions of jobs•To enable advanced dataflow patterns•To interface with wider range of resources•To separate Application Developer view from Application User view

WS-PGRADE(Web Services Parallel Grid Runtime and Developer Environment)

and gUSE

(Grid User Support Environment) architecture

3

WS-PGRADE/gUSE

•Creating complex workflow and parameter sweeps•Seamless access to various types of resources

•clusters, •service grids, •desktop grids, •databases.

•Scalable architecture•Advanced data-flows•Creating complex applications using embedded workflows, legacy codes•Comfort features

•Separated views•Community components from workflow repository

www.wspgrade.hu

4

WS-PGRADE architecture

Graphical User Interface: WS-PGRADEGraphical User Interface: WS-PGRADE

WorkflowEngine

WorkflowEngine

Workflowstorage

Workflowstorage File

storage

Filestorage

Applicationrepository

Applicationrepository

LoggingLogging

gUSEinformation

system

gUSEinformation

system

SubmittersSubmitters

Gridsphere portlets

Autonomous Services: high level

middleware service layer

Resources: middleware service layer

Local resources, service grid VOs, Desktop Grid resources, Web services, Databases

Local resources, service grid VOs, Desktop Grid resources, Web services, Databases

gUSE

Meta-brokerMeta-broker SubmittersSubmittersSubmittersSubmitters

Filestorage

Filestorage

SubmittersSubmitters

5

Application lifecycle in WS-PGRADE

•Define workflow structure•Configure workflow

•Define content for tasks

•Run a test•Use local resources, Web services, Databases

•Scale workflow for large simulations•Use batch systems, use cluster grids, use desktop grids

•Fix some parameters, leave some open•Result: An application specific science gateway for end users

6

WS-PGRADE application: Acyclic dataflow

•Job to run on dedicated machine

•Job to run in a gLite VO

•Job to run in a Globus 2/4 VO

•Task to run in a BOINC Grid

•Web service invocation

•Database operation (R / W)

•File from the client host

•File from a GridFTP site

•File from an LFC catalog

•Input string from a task or service

•Result of a Database query

7

Dataflow programming with gUSE

•Separate application logic from data•Cross & dot product data-pairing

•Concept from Taverna

•All-to-all vs. one-to-one pairing of data items

•Generator components: to produce many output files from 1 input file•Collector components: to produce 1 output file from many input files•Any component can be generator or collector•Conditional execution based on equality of data•Nesting, cycle, recursion

40

401000

50 20

5000

1

5000 1

7042 tasks

8

Ergonomics

•Users can be grid application developers or end-users. •Application developers design sophisticated dataflow graphs•Embedding into any depth, recursive invocations, conditional structures, generators and collectors at any position•Publish applications in the repository at certain stages of work

•Applications•Projects•Concrete workflows•Templates•Graphs

•End-users see WS-PGRADE & gUSE as a science gateway •List of ready to use applications in repository•Import and execute application without knowledge of programming, dataflow or grid

9

Current users of gUSE•EDGeS project (Enabling Desktop Grids for e-Science)

•Integrating EGEE with BOINC and XtremWeb technologies•User interfaces and tools

•ProSim project•In silico simulation of intermolecular recognition •See next presentation

•University of Westminster Desktop Grid

•Using AutoDock on institutional PCs•CancerGrid project

•Predicting various properties of molecules to find anti-cancer leads•Creating science gateway for chemists

10

Motivation to use gUSE and WS-PGRADE for CancerGrid

• Arbitrary number of generators (and collectors) within one workflow (at arbitrary locations).• Scalability: Number of jobs within one workflow is at range: 100K…1M !• Import of the existing EndUser configuration GUI (Easy-to-use, web based, user specific) application specific portlet for end users was not needed.

11

moleculedatabase

Executingworkflows

Browsing molecules

DG clients from all partners Molecule database server

Portal and DesktopGrid

server

BOINCserver

3GBridge

Portal

DG jobs

WU 1WU 2WU N

Job 1Job 2Job N

GenWrapper forbatch execution

BOINC client

LegacyApplication

PortalStorage

LocalResource

Local jobs

LegacyApplication

WU X

WU Y

12

CancerGrid Workflows

Descriptor Calculation Property Prediction

Model building

Screening

13

Working on the CancerGrid Portal – step-by-step

•Initial state: molecules/structures stored in DB, organised into lists•User selects list of molecules/structures•User selects/downloads a workflow from repository•User configures the workflow to take the list as input•User optionally updates parameters of the modules•Submits workflow•Optionally monitors the status•When workflow finished, results are stored in the DB

14

Molecule lists

15

Molecule viewer

16

Downloading workflow from repository – End user view

17

Workflow configuration

18

List of workflows (Novice user view)

19

Status monitor – End user view

20

Job statuses of a workflow - Developer view

21

Conclusions

•WS-PGRADE: Implemented on top of scalable, WS based gUSE architecture •More expressive dataflow patterns•Transparent access to

•Local resources •Service Grids•Desktop Grids•Databases•Web services

•Application repository•Service for collaboration of developers and end-users

22

Next steps at www.guse.hu

User manual

Request a user account

23

Thank you for your attention!

Questions?www.wspgrade.hu

Acknowledgement: CancerGrid EU FP6 project (FP6-2005-LIFESCTHTALTH-7)

http://www.cancergrid.eu

24

Applications in CancerGrid•Flexmol is an XML-based molecular language •Molecule 2D/3D converter (Cmol3D)•Molecule 3D conformation generator (Cmol3D)•MOPAC (Molecular Orbital PACkage) is a semiempirical quantum chemistry program based on Dewar and Thiel's NDDO approximation•Codessa Pro (Comprehensive Descriptors for Structural and Statistical Analysis) is a software suite for developing quantitative structure-activity/property relationships •Matrix former•QSAR Model builder Quantitative structure-activity relationship (QSAR) is the process by which chemical structure is quantitatively correlated with a well defined process, such as biological activity or chemical reactivity. •(Chemical) Property Predictor•File format converters (to integrate the previous tools into a workflow)

25

Cmol3D property settings

26

The CancerGrid infrastructure• PRODUCTION system

• gUSE portal•https://grid.cancergrid.eu/gridsphere/gridsphere

• BOINC server (private desktopgrid with firewall and controlled donor access)•https://grid.cancergrid.eu/cancergrid

• Monitoring info•https://grid.cancergrid.eu/cancergrid/hostinfo.php•69 machines (AMRI 10, SZTAKI 56, UPF 2, UoJ 1)

•TEST system• gUSE portal

•https://cancergrid.lpds.sztaki.hu/gridsphere• BOINC server (private desktopgrid with firewall and controlled donor access)

•https://cancergrid.lpds.sztaki.hu/cgrid/ops/• Monitoring info

•https://cancergrid.lpds.sztaki.hu/cgrid/hostinfo.php

Performance measurements:

30.000mols 6 days ~70 machines 10 confs