Upload
others
View
3
Download
0
Embed Size (px)
Citation preview
Executing Multi-workflow simulations on a mixed grid/cloud infrastructure using the SHIWA
TechnologyPeter Kacsuk
MTA SZTAKI and Univ.of [email protected]
What is a multi-workflow simulation?
• A simulation workflow where nodes of the simulation are themselves workflows potentially based on different workflow languages
• A practical example: LINGA application (outGRID project)– Combining several workflows (2 CIVETs+FreeSurfer+STAT)
– Heterogeneous workflow systems (LONI/MOTEUR)
• It also should enable the execution in different DCIs. LINGA application:– LONI Cluster (USA)– gLite-based neuGRID infrastructure (Europe)– CBRAIN HPC infrastructure (Canada)
2
Experiment setup
3
CIVET @ CBRAIN
LONI Pipeline151 input data items
CIVET @ neuGRID
LONI Pipeline146 input data items
FreeSurfer @ CRANIUM
LONI Pipeline
STATS @ EGI
MOTEUROutputs of both CIVETs
SHIWA solution for LINGA
Sub-WorkflowsManagement
4
Multi-Workflow
5
Start date: 01/07/2010
Duration: 27 months
Total budget: 2,101,980 €
Funding from the EC: 1,800,000 €
Total funded effort in person-months: 231
Web site: www.shiwa-workflow.eu
Coordinator: Prof. Peter Kacsuk, email: [email protected]
SHIWA (SHaring Interoperable Workflows for Large-Scale Scientific Simulations on
Available DCIs) project
6
Motivations 1• In many cases large simulations are organized
as scientific workflows that run on DCIs• However, there are too many different
• WF formalism• WF languages• WF engines
• If a community selected a WF system it is locked into this system:• They can not share their WFs with other communities
(even in the same scientific field)• They can not utilize WFs developed by other
communities
WF Ecosystem
7
Who are the members of an e-science community from WF applications point of view?
End-users (e-scientists) (5000-50000)• Execute the published WF applications with custom
input parameters by creating application instances using the published WF applications as templates
WF Application Developers (500-1000)• Develop WF applications
• Publish the completed WF applications for end-users
WF System Developers (50-100)• Develop WF systems
•Writes technical, user and installation manuals
accessing a large set of various DCIs to make these WF
applications run
Clouds
Local clusters
Supercomputers
Desktop grids (DGs)(BOINC, Condor, etc.)
Cluster based service grids (SGs)(EGEE, OSG, etc.)
Supercomputer based SGs
(DEISA, TeraGrid)
Grid systemsE-science infrastructure
What does a WF developer need?WF App.
RepositoryAccess to a large
set of ready-to-run scientific WF applications
Portal
Using a portal/desktop to parameterize and run these applications, and to further
develop them
accessing a single DCI to make these WF applications run
Cluster based service grids (SG)
(e.g. ARC)
Grid system
In the past: WF developers worked in an isolated way, on a single DCI
Portal/desktop
Using a portal/desktop to develop WF applications
As a result if a community selected a WF system it is locked into this DCI• Porting the WF to another
DCI required large effort• Parallel execution of the
same WF in several DCIs is usually not possible
After SHIWA: Collaboration between WF application developers
SHIWA App.
Repository
SSP Portal
Local clusters
Supercomputers
Desktop grids (DGs)(BOINC, Condor, etc.)
Cluster based service grids (SGs)(EGEE, OSG, etc.)
Supercomputer based SGs
(DEISA, TeraGrid)
Grid systems
Application developer
• Publish WF applications in the repository to be continued by other appl. developers
•Application developers use the portal/desktop to develop complex
applications (executable on various DCIs) for various end-user communities
Project objectives• Enable user communities to share their WFs
– Publish the developed WFs– Access and re-use the published WFs– Build multi-workflows from the published WFs
• Toolset:– SHIWA Simulation Platform
• WF Repository (production)• SHIWA Portal (production)• SHIWA Desktop (prototype)
12
Coarse-grained interoperability (CGI)
• CGI = Nesting of different workflow systems to achieve interoperability of WF execution frameworks
Multi-workflow
Export to IWIR
Import from IWIR
WFBWFA
Interoperable Workflow Intermediate Representation IWIR
Fine-grained interoperability (FGI)
Tools for CGI SHIWA services
• SHIWA repository to:– Describe workflows– Share workflows
• SHIWA portal to:– Access and enact registered workflows– Compose and enact multi-workflows – Monitor workflows and multi-workflows
execution in various DCIs– Retrieve results of the execution
SHIWA Repository facilitates publishing and sharing workflowsSupports:• Abstract workflows with multiple implementations of over 10 workflow systems• Storing execution specific dataAvailable:• from the SHIWA Portal• standalone service at: repo.shiwa-workflow.eu
EGI Community Forum, Munich, March 28, 2012
Scenario: Find and test WFs• SHIWA Repository: Analyze description, inputs and outputs of published WFs• SHIWA Portal: Instantiate WF from repo, execute with given sample data
(inside WS-PGRADE workflow used as the Master WF system)
17
18
Title: Work Package
SA1...Author:. G
18
SHIWA Portal: Workflow Editor
19
Title: Work Package
SA1...Author:. G
19
SHIWA Portal: Configuring Workflow
20
Title: Work Package
SA1...Author:. G
20
SHIWA Portal: Executing Workflow
21
SHIWA RepositorySHIWA Portal
WF1
SHIWA Science Gateway
GEMLCA Service
WFn
WE1 WEp
GEMLCA Repository
WE + WF
WF1 WFm
GEMLCA with GIB
WF listWS-
PGRADE Workflow
engine
WS-PGRADE Workflow
editor
edit WF
s2
search WF
s1
s5
s4
gLite DCI
MOTEUR WE
GWES WE
Globus DCI
pre-deployed- WEs
MOTEUR WE
Kepler WE
Taverna WE
Triana WE
local cluster
ASKALON WE
SHIWA VO
ASKALON WE
researcher
invoke WEs6
CGI User Scenario with WS-PGRADE as master
SHIWA Proxy Server
Proxy Server
s3
s6submit WE
22
2222
SHIWA RepositorySHIWA Portal
WF1
SHIWA Science Gateway
GEMLCA Service
WFn
WE1 WEp
GEMLCA Repository
WF1 WFm
GEMLCA with GIBWS-PGRADE
Workflow Engine
WS-PGRADE Workflow
Editor
search WF
s1
WE + WF s5
gLite DCI
MOTEUR WE
GWES WE
Globus DCI
MOTEUR WE
Kepler WE
Taverna WE
Triana WE
local cluster
ASKALON WE
SHIWA VO
ASKALON WE
user
invoke WE s7
GEMLCA admin
GEMLCA Client
MOTEUR Workflow
Engine
MOTEUR Workflow
editor
s2
s3
GEMLCA UI
SHIWA Proxy Server
Proxy Server
s6
s4
CGI User Scenario with MOTEUR as master
23
Integration of SCI-BUS and SHIWA gateway with clouds
CloudBroker Platform
SCI‐BUSScienceGateways
API
App Storefor Cloud
Platformto accessapps inClouds
CloudProvider
AraGrid (Spain), Armenian Grid, Baltic Grid, Belgian GridBIFI Desktop Grid, Bulgarian Grid, ClGrid (Chile), COMPCHEM VO of EGEE
Croatian Grid, EGRID (Economy Grid, Italy)GILDA training grid,Grid Ireland, HunGrid (Hungarian National Grid)
IberGrid (Portugal and Spain), MathGrid (Spain)MoSGrid (Molecule Simulation Grid of D‐Grid), KnowledgeGrid Malaysia
PireGrid (Spain and France), See‐Grid (South‐East European Grid)SwissGrid, Turkish Grid, UK NGS, UK White Rose Grid
VOCE (Central European Grid), Westminster Desktop Grid
Clusters: PBS, LSF, Condor, SGE
...
Liferay and P‐Grade
DesktopGrids:
BOINC, Xtrem‐Web, OurGrid
Super‐computers
WebService / API
Eucalyptus OpenNebula
gLiteMiddleware
UNICOREMiddleware
ARCMiddleware
App 1 App 2 App 3
SGIAmazo
nEC2
IBM
...
... Private Clouds
API API API API
Commercial Components
SCI‐BUSgenericgateway
German
MosGrid
Commun
ity
Statistical Seism
ology
Commun
ity
Blender R
endering
Co
mmun
ity
Amsterdam Medical
Center Com
mun
ity
Swiss P
roteom
ics
Commun
ity
Astroph
ysics
Commun
ity
Helio physics
Commun
ity
Business Process
Commun
ity
Software bu
ild and
test Com
mun
ity
Citizen
Web
Co
mmun
ity
PireGrid Co
mmercial
Commun
ity
24
University of Westminster UoW United KingdomMagyar Tudomanyos Akademia Szamitastechnikai es Automatizalasi Kutato Intezete
MTA-SZTAKI Hungary
Centre National de la Recherche Scientifique CNRS FranceStichting European Grid Initiative EGI.eu The NetherlandsAcademic Medical Center of the University of Amsterdam
AMC The Netherlands
Technische Universität Dresden TUD GermanyLudwig-Maximilians-Universität München LMU GermanyUniversity College London UCL United KIngdomTrinity College Dublin TCD IrelandIstituto Nazionale di Astrofisica INAF Italy
Partners:
Technology providers:CNRS, EGI.eu, MTA-SZTAKI, UoW
Research Communities:Astro-Physics INAFComputational Chemistry LMU + TUDHelio-Physics TCD + UCLLife Science AMC
Duration:September 2012 – August 2014
ER-flow Project
Advantages for the various types of user communities using SHIWA
• WF system developers– Better visibility: much more WF developers can access and use their
WF system than before (through the applications stored in the SHIWA repo)
– The joint impact is much bigger than the individual WF systems can achieve
• WF developers– They can collaborate: share and re-use existing WF applications– WF application development can be accelerated– More complex WFs can be created in shorter time– They will access many different DCIs (their WF will be more popular)
• End-users– much bigger set of usable and more sophisticated WF applications– These applications can run on various DCIs
25
Conclusions
• SHIWA brings advantage for all the 3 kinds of user communities:– WF system developers– WF developers– End-users
• With relatively little effort– WF systems can join the SSP – WF system developers can adapt SHIWA technology
• Further information: www.shiwa-workflow.eu
26
Summer school on workflows and gateways
27