Enabling Grids for E-sciencE Astronomical data processing workflows on a service-oriented Grid architecture Valeria Manna INAF - SI The

Enabling Grids for E-sciencE

www.eu-egee.org

Astronomical data processing workflows on a service-oriented Grid architecture

Valeria Manna INAF - SI

The 17th Global Grid Forum - GGF17Tokyo, JapanMay 10-12, 2006

The 17th Global Grid Forum - GGF17 Tokyo, May 10-12, 2006 2


Motivation and Objectives

Working Group:

Claudio Vuerli, Giuliano Taffoni, Valeria Manna, Andrea Barisani, Fabio Pasian

• Problem analysis related to the integration of astronomical applications in a Grid environment

• Description of astronomical tasks sequences with workflow languages to process complex compositions of astronomical applications



Astrophysics and distributed computing

Computational problems

• Theoretical Codes

• Computation and analysis of many TBs of images

• Database of catalogues of the detected elements

• data-mining tools

- Data source and organization interaction- Geographically distributed resources (database, telescope, storage, etc)

Request of distributed computing to provide:•User virtualisation

•Resource virtualisation

GRID



Astrophysics Application

TasksTasks::

•Database AccessDatabase Access

•Data ReductionData Reduction

•Pure Computational Pure Computational

•Instrument MonitoringInstrument Monitoring



Workload Management

The task sequence of the astronomical application is managed by a Workload Management System that makes a task- job mapping

User :– Submit a job

– Create some script for the monitoring of job status

– Create script to create some task connecting link

– Recover I/O data

Need to a Workflow

Management System



Workflow Management

A workflow language allows to describe the execution flow of pipeline tasks :

–Query astronomical catalogues–Data transfer–Synchronize job –Analyse the resulting data

Mapping task → service



Web Services CompositionWeb Services Composition

The composition of WS consists of providing logic around a set of interactions between the composition and the WS that participate in it. These interactions are simply invocations to the operations offered by the services in play. One approach to providing the control and data logic is the use of workflow specifying:

– the execution order of operations from a collection of WS – the data shared among composed WS– partner involved– how the partner are involved in the flow process

Service orchestrationweb services can interact with each other at the message level, including the flow logic and execution order of the interactions. For orchestration, the process is always controlled by the perspective of one of the parties.

Service choreographytypically associated with the public message exchanges that occur between multiple web services, rather than a specific business process that is executed by a single party. Each part involved in the process describes the part it plays in the interaction.



A workflow language: Bpel

Bpel (Business Process Execution Language) is a workflow-based composition language for Web Services

• Standard• Open Source• Support the Service Orchestration

Feature:– Interaction: define WSDL port types for each interface that is

used in the workflow definition– Basic Activity: to allow for interaction with the applications

being composed invoke, reply and receive – Structured Activity : conditional execution and iteration /

recursion of a sequence of activities to manage process flow

Likely to be adopted as EGEE standard



Bpel: process flow



Hypothetical Scenario using Bpel

BPEL ENGINE

CREAMEGEE

SERVICE

WEBSERVICES

GRID/WEBSERVICES

ASTROGRIDSERVICES

WF



WSIF

ProblemCurrently the LCG middleware is not Web Service oriented and in GLite (middleware of EGEE project) the WM and the other services are expected to offer a Web Service interface.In this scenario the use of BPEL is not feasible

In a BPEL workflow it is very useful to provide a set of services that are always available.

SolutionTo use BPEL partner abstraction that are services that workflow needs to use and typically are mapped to Web (or Grid) services. It is possible to map some BPEL partners to locally implemented services as long as a WSDL port type for a partner is provided.

WSIF (Web Service Invocation Framework) is a simple Java API for invoking Web services, enables to interact with abstract representations of Web services through their WSDL descriptions



Current Scenario with WSIF

WSIF

BPEL ENGINE

WSIF

LCG

WEBSERVICES

GLOBUS 2x

WEBSERVICES

WF



Conclusions

• Astronomical applications described using workflow language

• A solution to integrate astronomical task sequences in gLite middleware

• Use of Service Oriented technology• Use of Grid Service composition

Integrated solution in the current middleware and in its future development



The End

Thank you for your attention

Documents

Enabling Grids for E-sciencE Astronomical data processing workflows on a service-oriented Grid architecture Valeria Manna INAF - SI The