Coarse-Grained workflow interoperability in the SHIWA … · MOTEUR WE Kepler WE Taverna WE Triana WE local cluster ASKALON WE SHIWA VO ASKALON WE ... –Get an account on SSP –Join

Coarse-Grained workflow interoperability in the SHIWA platform

P. Kacsuk, G. Kecskemeti, T. Kiss, V. Korkhov, D. Krefting, T. Kukla, T. Glatard, K. Maheshwari, J. Montagnat, S. Olabarriaga,

G. Terstyanszky, N. Weingarten

MTA SZTAKI, University of Westminster, CNRS, Charité UniversitätMedizin, Academic Medical Centre

http://www.shiwa-workflow.eu

2

SHIWA

• Project objectives– Sharing workflows from different environments– Interoperability between workflow engines

• Consortium (2010-2012)– MTA SZTAKI, Budapest, PI– University of Insbruck– University of Westminster– Cardiff University– Academic Medical Center, Amsterdam– Charité hospital, Berlin– CNRS, Lyon and Sophia-Antipolis– USC, USA


3

Workflow enginesand languages

• In the consortium– P-Grade– ASKALON (AGWL)– MOTEUR (Gwendia)– Triana– Pegasus (DAX)

• External, supported by the platform– Kepler (MoML)– Taverna (Scufl)– GWES (GWDL)


4

Coarse-Grained Interoperability

• Principle– Share workflows in their native environments

• Method– Provide cross-language workflow repository– Wrap workflow engines as legacy codes (black boxes)

• Motivating use-cases– Share applications as ported to grids– Build meta-workflows from existing workflows– Access multiple computing infrastructures


5

CGI concept

• Workflows are black boxes– Not interpreted by SHIWA– Wrapped as GEMLCA services– Only in/outputs are known to the platform

• Meta-workflow composition– Build workflow from GEMLCA services– Can wrap heterogeneous workflows

• WFs are executed by their native engines– Engines embedded in the platform, or– Engines remotely invoked

NativeJob

0

2

1

3

NativeJob

0 1

2

0 21

3

0 21

3

MOTEURWF

AskalonWF


6

Core engine

Mast

er

work

flow

sy

stem

Foreach d in D { A(d);}

Workflowlanguage

Front-end

Back

-end

DCI 1

Distributed data

Application services

Computing resources

Em

bedded

work

flow

sy

stem

Workflowlanguage

Front-end

Core engine B

ack

-end

DCI 2

Distributed data

Application services

Computing resources

CGI implementation


7

Overall architecture ofSHIWA Simulation Platform v1

SHIWA RepositorySHIWA Portal

WF1

GEMLCA admin

SHIWA Science Gateway

gLite DCIWFn

WE1 WEp

GEMLCA Repository

WF1 WFm

MOTEUR WE

PGRADEWorkflow

engine

PGRADE Workflow

editor GWES WE

Globus DCI

pre-deployed-WEs

MOTEUR WE

Kepler WE

Taverna WE

Triana WE

local cluster

ASKALON WE

SHIWA VO

ASKALON WE

GEMLCA Service

GEMLCA with GIB


8


SHIWA RepositorySHIWA Portal

WF1

GEMLCA admin


WFn

WE1 WEp

GEMLCA Repository

WF1 WFm

PGRADEWorkflow

engine

PGRADE Workflow

editor

GEMLCA Service

GEMLCA with GIB

• SHIWA Portal • P-GRADE 2.4 portal technology• certificate/proxy and DCI resource management • access to different DCI information systems• integrated with the P-GRADE Workflow System

(used as native workflow engine)• administration of GEMLCA services

• GEMLCA Service• converts legacy applications such as workflows

and workflow engines into Grid services• invokes locally or remotely pre-deployed workflow

engines or submits workflow engines to local or remote resources to execute workflows

• GEMLCA Repository • workflow engine (WE) and workflow (WF) data

supporting execution

• SHIWA Repository• create, add, edit and delete workflow metadata• upload and download workflows with their

implementations and configurations.


9

SHIWA repository

• Workflow retrieval

• Workflow implementation – Concrete description– Executed in SHIWA portal


10

SHIWA VO

gLite DCI

MOTEUR WE

GWES WE

Globus DCI

pre-deployed-WEs

MOTEUR WE

Kepler WE

Taverna WE

Triana WE

local cluster

ASKALON WE

SHIWA VO

ASKALON WE

• VO services• Logical File Catalog

• lfc-egee.in2p3.fr

• Computing elements: • karwendel.dps.uibk.ac.at – Austrian NGI (Globus)• othello.zih.tu-dresden.de – German NGI (Globus)• ngs.wmin.ac.uk – UK NGI (Globus)• phoebe.deimos.htc.biggrid.nl – Dutch NGI (gLite)

• Storage resources: • carme.htc.biggrid.nl (9.9TB)

• Workflow engines: • Moteur engine to submit to gLite resources• Askalon and GWES engines to submit to Globus

resources• Triana, Taverna, Moteur, Kepler and Askalon engines

deployed locally at UoW


11

SHIWA Use Cases

• Use case A: Running non-native workflows• Use case B: Creating and running meta-workflows• Use case C: Running workflows on multiple DCIs


12

Use case 1: FSL BespostX

• Neuroscience, brain imaging

• Based on FMRIB Software Library (FSL) - a toolbox of programs to analyze brain images.

• BedpostX is a tool to process DiffusionTensor Imaging (DTI) data acquired with Magnetic Resonance Imaging (MRI). It reconstructs the fibers in each voxel using an advanced method that supports crossing fibers

• CGI scenario: Use different implementations of the same workflow in different WF languages, run on different DCIs in parallel to get more resources


13

FSL BedpostX implementation

MOTEUR/DutchGrid

GWES/D-Grid


14

Use case 2: DTI Atlas• Neuroscience, brain imaging• Magnetic Resonance Imaging (MRI) is a non-invasive in-vivo

imaging modality.

• Diffusion Tensor MRI (DTI) is a specific MR-modality enabling the identification of the orientation of human tissue. DTI is used in comparative studies of brain diseases that are thought to cause local damage to brain tissue, possibly only in specific nerve bundles related to a given brain function.

• DTI Atlas: single average tensor field is computed for a group of subjects, which can then be used for further analysis.

• CGI scenario: Combine a pipeline workflow of sub-workflows performing separate analysis steps


15

DTI Atlas implementation

MOTEUR/DutchGrid

MOTEUR/DutchGrid

MOTEUR/DutchGrid


16

Use case 3: GATE

• Medical simulation

• GATE is a simulation software developed by the OpenGate Collaboration (http://opengatecollaboration.healthgrid.org). It is used here for radio- and hadrontherapy simulations, but it can also simulate image acquisition, e.g., Positron Emission Tomography.

• CGI scenario: Integrating heterogeneous workflows into the native workflow to achieve better performance by accessing different DCIs via workflow engines.

http://opengatecollaboration.healthgrid.org/


17

GATE implementation

Simulation splitting

Execution on EGI

Execution on UK NGS

Merge

Execution on D-Grid


18

Summary

• Achievements– Workflow repository– Meta-workflow composition – Meta-workflow execution on different DCIs

• Challenges in crossing DCIs– Credential management: proxy formats– Data transfers: heterogeneous clients– Code porting: software dependencies


19

Conclusions

• How to test Coarse-Grained workflow interop. ?– Get an account on SSP– Join the SHIWA VO

• SHIWA User Forum– http://shiwa-workflow.eu

• Coming soon: Fine-Grained interoperability– Workflows migrated accross engines– Based on an Intermediate Workflow Interoperability

Representation (IWIR)

http://shiwa-workflow.eu/

Documents

Coarse-Grained workflow interoperability in the SHIWA … · MOTEUR WE Kepler WE Taverna WE Triana WE local cluster ASKALON WE SHIWA VO ASKALON WE ... –Get an account on SSP –Join