Embed Size (px)
DESCRIPTION
: Sharing Workflows through Coarse-Grained Workflow Interoperability G . Terstyanszky , T. Kukla , T. Kiss, S. Winter, J.: Centre for Parallel Computing University of Westminster, London , United Kingdom P . Kacsuk , A. Balasko MTA-SZTAKI, Budapest Hungary. - PowerPoint PPT Presentation
Citation preview
:
Sharing Workflows through Coarse-Grained Sharing Workflows through Coarse-Grained Workflow InteroperabilityWorkflow Interoperability
G. Terstyanszky, T. Kukla, T. Kiss, S. Winter, J.: Centre for Parallel Computing
University of Westminster, London, United Kingdom
P. Kacsuk, A. BalaskoMTA-SZTAKI, Budapest Hungary
SHIWA is supported by the FP7 Capacities Programme under contract No. RI-261585
22
Computer and Automation Research Institute, Hungarian Academy of Sciences
MTA SZTAKI Hungary
University of Innsbruck UIBK Austria
Charité - Universitätsmedizin Berlin C-UB Germany
French National Centre for Scientific Research CNRS France
University of Westminster UOW United Kingdom
Cardiff University CU United Kingdom
Academic Medical Centre of the University of Amsterdam AMC Netherlands
University of Southern California USC USA
ShSharing aring IInteroperable nteroperable WWorkflows through orkflows through Large-Scale Scientific Simulation on Large-Scale Scientific Simulation on AAvailable vailable
DCIs (SHIWA)DCIs (SHIWA)Partners:
Duration:July 2010 – June 2012 (extended to September 2012)
3
Workflow Interoperability ChallengeWorkflow Interoperability Challenge
SHIWA Objectives and ServicesSHIWA Objectives and Services
Objectives:• To create an environment which enables seamless execution of
workflows of different workflow systems through workflow interoperability
Services:• To support the whole workflow lifecycle: editing, uploading, browsing
downloading and executing workflows
• To provide coarse- and fine-grained workflow interoperability solution
• To offer Distributed Computing Infrastructure interoperability solution
• To provide desktop computer and portal interfaces to manage workflows
Key actors:• researchers workflow engine developers
workflow developers
4
555
SHIWA WF Repository
SHIWA WF Repository
SHIWA WERepository
SHIWA WERepository
E-ScientistE-ScientistWF DeveloperWF Developer WE DeveloperWE Developer
fMRI fMRI
MRI simulatorMRI simulatorSequencingSequencing MOTEURMOTEUR
….….
TrianaTriana
PegasusPegasus
ASKALONASKALON
P-GRADEP-GRADE
……
Upload Upload
End User PortalEnd User PortalDeveloper PortalDeveloper Portal Admin PortalAdmin Portal
BrowseRun
IWIR front,back endWF engine XML description
WF engine binary
ImportIWIR
TargetWF SystemTargetWF System
Master WF SystemMaster WF System
RunDCI a
SHIWA Science Gateway
WF descriptionWF descriptionWF executablesWF executables
Embedded WF System 2Embedded WF System 2RunDCI c
SourceWF SystemSourceWF System
ExportIWIR
Embedded WF System 1Embedded WF System 1RunDCI b
SourceWF SystemSourceWF System
ExportIWIR
Embedded WF System 1Embedded WF System 1RunDCI b
CGI FGI
NonLinSyst simulatorNonLinSyst simulator
SHIWA WF enactorSHIWA
WF enactor
IWIR
SHIWA SolutionSHIWA Solution
6
Workflow description abstract workflow•plain text which describes the inputs and outputs and explains what the workflow doesWorkflow implementations concrete workflow
• It contains all implementation files or references to these files (via eg. URLs) and also holds other data/metadata necessary to run the workflow on its associated workflow engine.
Workflow configurations• It contains pre-defined input files and parameters or references to these files and parameters and other
data/metadata of a workflow.Workflow engines• It contains files and other data/metadata necessary to execute a workflow engine on a grid site or references
to them
SHIWA Data StructureSHIWA Data Structure
7
WFSubmission
clientSubmission
ServiceWorkflow Engine B
Workflow of Workflow Engine B
DCI
Workflow Engine A
Coarse-Grained Interoperability: Coarse-Grained Interoperability: Single Non-native WorkflowSingle Non-native Workflow
8
J2 WF4
J1
J3
GEMLCA Client
GEMLCA Submission
Service
Workflow Engine B
Workflow of Workflow Engine B
Workflow of Workflow Engine A
Coarse-Grained Interoperability: Coarse-Grained Interoperability: Meta-WorkflowMeta-Workflow
DCI Resources
• native workflows: J1, J2, J3• non-native workflows: WF4
- black boxes which are managed as legacy code applications
Workflow Engine A
9
gLite DCI Globus DCI
DCIs
ARC DCI Unicore DCI
Kepler WEGWES WE
Taverna WE
Triana WE
MOTEUR WE
ASKALON WE
PGRADE WE
SHIWA Portal
GEMLCA admin
SHIWA Science Gateway
WE1 WEp
GEMLCA Repository
WF1 WFm
SHIWA Repository
WF1 WFn
WS-PGRADE Workflow
engine
WS-PGRADE Workflow
editor
GEMLCA Service
GEMLCA with GIB
SHIWA Proxy Server
Proxy Server
Galaxy WE
Pegasus WE
ProActive WE
Workflow Engines
SSP: CGI InfrastructureSSP: CGI Infrastructure
SHIWA Science Gateway Resourcesnative WE WS-PGRADE local resources: invocation of locally deployed WEs portal WS-PGRADE v3.4.1WE submission to local cluster repository GEMLCA + SHIWA repo remote resources: through remotely pre-deployedsubmitter GEMLCA with GIB WEs to ARC, gLite, Globus Unicore DCIsproxy management SHIWA Proxy Server
1010
SHIWA Portal: Abstract WorkflowSHIWA Portal: Abstract Workflow
Jobs
Ports
11
SHIWA Portal: Configuring WorkflowSHIWA Portal: Configuring Workflow
12
SHIWA Portal: Executing WorkflowSHIWA Portal: Executing Workflow
13
SHIWA Repository: Browse ViewSHIWA Repository: Browse View
domain
search
14
inputsoutputs
data sets
implementation
SHIWA Repository: Browse ViewSHIWA Repository: Browse View
1515
Browse Workflows
Browse Implementations
Search
Advanced Search
Sort
SHIWA Repository: Table ViewSHIWA Repository: Table View
1616
SHIWA Repository: Execution SupportSHIWA Repository: Execution Support
Execution data
17
Taverna workflow
SHIWA Portal: Legacy Code ExplorerSHIWA Portal: Legacy Code Explorer
18
SHIWA Portal: Repository PortletSHIWA Portal: Repository Portlet
Workflow Repository
19
CGI Developer Scenario: Specifying WE CGI Developer Scenario: Specifying WE
20
CGI Developer Scenario: CGI Developer Scenario: Specifying WF through portalSpecifying WF through portal
21
CGI Developer Scenario:CGI Developer Scenario: Specifying WF through desktopSpecifying WF through desktop
2222
CGI User Scenario: Native WE CGI User Scenario: Native WE
SHIWA Repository
SHIWA Portal
WF1
SHIWA Science Gateway
GEMLCA Service
WFn
WE1 WEp
GEMLCA Repository
WE + WF
WF1 WFm
GEMLCA with GIB
WF list
WS-PGRADE Workflow
engine
WS-PGRADE Workflow
editor
user
SHIWA Proxy Server
Proxy Server
step 1search WF
step 2 edit WF
Kepler WEGWES WE
Taverna WE
Triana WE
MOTEUR WE
ASKALON WE
PGRADE WE
Galaxy WE
Pegasus WE
ProActive WE
Workflow Engines
gLite DCI Globus DCI
DCIs
ARC DCI Unicore DCI
step 7 run WFstep 6
retrieve proxy
step 5retrieve WF
step 4submit WF
step 3retrieve WF data
SHIWA Repository
SHIWA Portal
WF1
SHIWA Science Gateway
GEMLCA Service
WFn
WE1 WEp
GEMLCA Repository
WF1 WFm
GEMLCA with GIB
WS-PGRADE Workflow
Engine
WS-PGRADE Workflow
Editor WE + WF
user
GEMLCA Client
MOTEURWorkflow
Engine
MOTEUR Workflow
editor
GEMLCA UI
SHIWA Proxy Server
Proxy Server
CGI User Scenario: Non-native WECGI User Scenario: Non-native WE
step 1 search WF
step 2 edit WF
step 3 submit WFstep 6
retrieve proxy
step 5 retrieve WF
step 7 run WF
Kepler WEGWES WE
Taverna WE
Triana WE
MOTEUR WE
ASKALON WE
PGRADE WE
Galaxy WE
Pegasus WE
ProActive WE
Workflow Engines
gLite DCI Globus DCI
DCIs
ARC DCI Unicore DCI
step 4 forward WF
24
SHIWA Repository
SHIWA Portal
WF1
SHIWA Science Gateway
GEMLCA Service
WFn
WE1 WEp
GEMLCA Repository
WF1 WFm
GEMLCA with GIB
WS-PGRADE Workflow
Engine
WS-PGRADE Workflow
Editor
SHIWA Proxy Server
Proxy Server
gLite DCI Globus DCI
DCIs
ARC DCI Unicore DCI
SHUWA
Desktop
WE
user
step 1 search WF
step 2 retrieve WF
step 3forward
WF bundle
step 4edit WF
step 6submit WF
step 5ask run WF
CGI User Scenario: CGI User Scenario: WF Execution via Desktop WF Execution via Desktop
2525
SHIWA Simulation Platform:Access
SHIWA Science Gateway (SHIWA Portal + SHIWA Repository)
• joint portal and repository account given by the science gateway administrator
DCI resources
• certificate given by the relevant DCI administrator, for example the VO sysadmin
Access to the SHIWA Portal
ssp.shiwa-workflow.eu
Access to the SHIWA Repository
repo.shiwa-workflow.eu
26
Conclusions
• the simulation platform supports the whole lifecycle of workflows, i.e. creating, testing, uploading, browsing, downloading and running workflows
• users can use workflows of their own and of other workflow systems through the same user interface, i.e. they can re-cycle or share workflows created by other research teams or even other research communities
• workflow developers (or e-scientists) can create, run, upload workflows through the simulation platform
• researchers (astro physicists, earth scientists, bio scientists, etc.) can browse the workflow repository, select and run workflows through the simulation platform
• the simulation platform provides access to major European e-infrastructure, the European Grid Initiative (EGI) infrastructure which offers large pool of compute and storage resources
