Transcript
Page 1: A Practical Approach for a Workflow Management Systemcppwfms.sourceforge.net/papers/slides-CGW07.pdf · Workflow Management Systems Overview A lot of interest around WfMSs exist –

A Practical Approach for a Workflow Management System

Simone Pellegrini, Francesco Giacomini, Antonia GhiselliINFN Cnaf 

Viale B. Pichat, 6/2 40127 Bologna{simone.pellegrini | francesco.giacomini | antonia.ghiselli}@cnaf.infn.it

Page 2: A Practical Approach for a Workflow Management Systemcppwfms.sourceforge.net/papers/slides-CGW07.pdf · Workflow Management Systems Overview A lot of interest around WfMSs exist –

Outline

● Workflow Management Systems overview● A practical approach for real workflows● Implementation issues● A case study: JDL to GWorkflowDL conversion

Page 3: A Practical Approach for a Workflow Management Systemcppwfms.sourceforge.net/papers/slides-CGW07.pdf · Workflow Management Systems Overview A lot of interest around WfMSs exist –

Outline

● Workflow Management Systems overview● A practical approach for real workflows● Implementation issues● A case study: JDL to GWorkflowDL conversion

Page 4: A Practical Approach for a Workflow Management Systemcppwfms.sourceforge.net/papers/slides-CGW07.pdf · Workflow Management Systems Overview A lot of interest around WfMSs exist –

Workflow Management Systems Overview

● A lot of interest around WfMSs exist– Thanks to workflows, the processes' business logic 

can be easily expressed using graphs● Appealing for users with limited programming skills

● However, the lack of a recognized standard causes incompatibility among WfMSs:– several languages for workflow description exist:

● Based on different modeling formalisms (DAGs, Petri­Nets, Pi­Calculus...);

Page 5: A Practical Approach for a Workflow Management Systemcppwfms.sourceforge.net/papers/slides-CGW07.pdf · Workflow Management Systems Overview A lot of interest around WfMSs exist –

Problems in Workflow Management

● The choice of a WfMS usually binds users to a specific workflow language– Change of the WfMS has a high cost:

● Legacy workflows must be rewritten.

● Interoperability among WfMSs is still an open issue

● Difficulty to manage large, complex and real life scientific processes

Page 6: A Practical Approach for a Workflow Management Systemcppwfms.sourceforge.net/papers/slides-CGW07.pdf · Workflow Management Systems Overview A lot of interest around WfMSs exist –

Outline

● Workflow Management Systems overview● A practical approach for real workflows● Implementation issues● A case study: JDL to GWorkflowDL conversion

Page 7: A Practical Approach for a Workflow Management Systemcppwfms.sourceforge.net/papers/slides-CGW07.pdf · Workflow Management Systems Overview A lot of interest around WfMSs exist –

A practical approach for real workflows

● Introducing a WfMS:– Petri­Nets based

● Formal semantics● Turing­complete (deals with Workflow Patterns)● Build time analysis tools (reachability, boundedness...)

– Independent from the underlying Grid middleware– Multi­language:

● GWorkflowDL used as internal representation

– Deals with interoperability

Page 8: A Practical Approach for a Workflow Management Systemcppwfms.sourceforge.net/papers/slides-CGW07.pdf · Workflow Management Systems Overview A lot of interest around WfMSs exist –

Workflow Management Workflow Management SystemSystem

The WfMS Architecture

Workflow GatewayWorkflow Gateway

Grid Abstraction LayerGrid Abstraction Layer

Grid Middleware/sGrid Middleware/s

Workflow EngineWorkflow Engine

Layered architecture

Language Interoperability

Engine Interoperability

Execution nodes Storage nodes

Aims at Grid middleware 

independence

Aims at language independence

JDL, GworkflowDL, BPEL

Page 9: A Practical Approach for a Workflow Management Systemcppwfms.sourceforge.net/papers/slides-CGW07.pdf · Workflow Management Systems Overview A lot of interest around WfMSs exist –

Grid Abstraction Layer

● Abstracts the basic functions of a Grid providing portability of the Workflow engine over different Grid middlewares

Grid Abstraction LayerGrid Abstraction Layer

Dispatcher

ObserverData Transfer

Reservation

– Dispatcher: Job submission/cancellation– Data Transfer: Move data between Grid nodes– Observer: Monitor submitted job status– Reservation: Resource reservation

Page 10: A Practical Approach for a Workflow Management Systemcppwfms.sourceforge.net/papers/slides-CGW07.pdf · Workflow Management Systems Overview A lot of interest around WfMSs exist –

Workflow Gateway

● Provides language and model converters in order to achieve compatibility with legacy workflows and legacy WfMSs 

Workflow GatewayWorkflow Gateway

Parser

Model Translator

DagBPELgLite (classAd)JSDLCompilerPi­Calculus

– Parser: Extracts the model from a workflow description

– Compiler: Produces a workflow representation using a target language (JDL, GWorkflowDL, BPEL, …)

Page 11: A Practical Approach for a Workflow Management Systemcppwfms.sourceforge.net/papers/slides-CGW07.pdf · Workflow Management Systems Overview A lot of interest around WfMSs exist –

Language Interoperability

● The Gateway solves some interoperability issues in workflow management:– Part of a workflow (or a sub­workflow) can be 

translated and successively delegated to a third­party WfMS

– Legacy workflows can be executed on our WfMS without being rewritten

● Every process can be expressed in terms of a Petri Net (Turing­complete)

Page 12: A Practical Approach for a Workflow Management Systemcppwfms.sourceforge.net/papers/slides-CGW07.pdf · Workflow Management Systems Overview A lot of interest around WfMSs exist –

The Workflow Engine

● Our goal is to keep the engine as simple as possible and concentrate on interoperability issues

● Main characteristics– Petri­Net base– Micro­Kernel architecture: 

● aims at modularity and extendibility

– Distributed

Page 13: A Practical Approach for a Workflow Management Systemcppwfms.sourceforge.net/papers/slides-CGW07.pdf · Workflow Management Systems Overview A lot of interest around WfMSs exist –

Outline

● Workflow Management Systems overview● A practical approach for real workflows● Implementation issues● A case study: JDL to GWorkflowDL conversion

Page 14: A Practical Approach for a Workflow Management Systemcppwfms.sourceforge.net/papers/slides-CGW07.pdf · Workflow Management Systems Overview A lot of interest around WfMSs exist –

EGEE/gLite as a Grid Middleware

● Choice of EGEE/gLite middleware because of:– Services maturity– Reliability– Large adoption

● Job management is done by the Workload Management System (WMS)

● Job monitoring is done by the Logging and Bookkeeping Service (LB)

Page 15: A Practical Approach for a Workflow Management Systemcppwfms.sourceforge.net/papers/slides-CGW07.pdf · Workflow Management Systems Overview A lot of interest around WfMSs exist –

Project Goal: A WfMS over the WMS

● The practical outcome of our work is to build a WfMS relying on the WMS + LB

● Both WMS and LB provide a Web Service interface– simplify interaction with Grid services

Page 16: A Practical Approach for a Workflow Management Systemcppwfms.sourceforge.net/papers/slides-CGW07.pdf · Workflow Management Systems Overview A lot of interest around WfMSs exist –

WfMS Deploy (1/2)

● WfMS running on a dedicated server:– Client sends the workflow description to the server;– WfMS server manages workflows execution; 

GridWorkflow Description

WFMS Server

Monitoring

Client

SubmittedJobs

Interface to the Grid is provided by the WMS API.

Workflow running instance

Page 17: A Practical Approach for a Workflow Management Systemcppwfms.sourceforge.net/papers/slides-CGW07.pdf · Workflow Management Systems Overview A lot of interest around WfMSs exist –

WfMS Deploy (2/2)

● WfMS running as a Grid job:– Client submits a workflow to the Grid via the WMS and 

monitors it via the LB– The WfMS ends up running on a Grid node

Grid

Workflow Descriptionsubmit

WFMSrunning 

instances

MonitoringClient

– The WfMS instance submits workflow tasks to the Grid using the WMS and monitors them via the LB

● Taking full advantage from the Grid resources and facilities (e.g. checkpointing)

Page 18: A Practical Approach for a Workflow Management Systemcppwfms.sourceforge.net/papers/slides-CGW07.pdf · Workflow Management Systems Overview A lot of interest around WfMSs exist –

Outline

● Workflow Management Systems overview● A practical approach for real workflows● Implementation issues● A case study: JDL to GWorkflowDL conversion

Page 19: A Practical Approach for a Workflow Management Systemcppwfms.sourceforge.net/papers/slides-CGW07.pdf · Workflow Management Systems Overview A lot of interest around WfMSs exist –

The Job Description Language (JDL)

● The Job Description Language describes a job to be execution on the Grid

● The JDL adopted within the gLite middleware is based upon Condor's Classified Advertisements (ClassAds)– record­like structure composed of a finite number of 

attributes separated by semi­colon (;)

Attribute = Value;

Page 20: A Practical Approach for a Workflow Management Systemcppwfms.sourceforge.net/papers/slides-CGW07.pdf · Workflow Management Systems Overview A lot of interest around WfMSs exist –

DAG Model

A JDL DAG Workflow

[  Type = “dag”;   [ ... ]

nodes = [father = [ ... ];

   son1   = [ ... ];    son2   = [ ... ];   final  = [ ... ];    dependencies =  {      {father, {son1, son2}}, 

{son1, final}, {son2, final}

};  ];]

father

son1 son2

final

● JDL allows workflow (DAG based) definition:

Job

Dependencies 

JDL parser +

Model extractor

Page 21: A Practical Approach for a Workflow Management Systemcppwfms.sourceforge.net/papers/slides-CGW07.pdf · Workflow Management Systems Overview A lot of interest around WfMSs exist –

DAGs in gLite

● Many legacy workflows expressed in JDL exist;● JDL workflows are managed by the Condor 

DAG Manager (DAGMan):– Acts as a Meta­Scheduler for Condor jobs– Submits job respecting their inter­dependencies– In case of job failure, DAGMan continues until it can 

no longer make progress.

Page 22: A Practical Approach for a Workflow Management Systemcppwfms.sourceforge.net/papers/slides-CGW07.pdf · Workflow Management Systems Overview A lot of interest around WfMSs exist –

DAG -> Petri Net

● The current DAG model is very limited, e.g.– lack of error handling– lack of task types other than computation

● DAGs can be easily described using Petri Nets– a DAG node can be represented by a Petri Net 

transition– the flow of data among DAG nodes is modeled 

using data tokens

Page 23: A Practical Approach for a Workflow Management Systemcppwfms.sourceforge.net/papers/slides-CGW07.pdf · Workflow Management Systems Overview A lot of interest around WfMSs exist –

DAG -> Petri Net

DAG Model

father

son1 son2

final

Petri­Net Model

P4

init

father

P1 P2

P3

son1 son2

final

end

DAG to Petri Net Model Converter

Page 24: A Practical Approach for a Workflow Management Systemcppwfms.sourceforge.net/papers/slides-CGW07.pdf · Workflow Management Systems Overview A lot of interest around WfMSs exist –

Petri Net -> GWorkflowDL

Petri­Net Model

P4

init

father

P1 P2

P3

son1 son2

final

end

<workflow [...]><place ID=”init” /><place ID=”P1” /><place ID=”P2” />[...]<place ID=”end” /><transition ID=”father”>

<inputPlace placeID=”init”/><outputPlace placeID=”P1”/><outputPlace placeID=”P2”/><operation />

</transition>[...]

</workflow>

Compiler

DEMODEMO

Page 25: A Practical Approach for a Workflow Management Systemcppwfms.sourceforge.net/papers/slides-CGW07.pdf · Workflow Management Systems Overview A lot of interest around WfMSs exist –

From abstract to concrete workflow (1/4)

● The workflow description needs a refinement process in order to perform concrete task operations

● In the case of the gLite middleware task execution is asynchronous– The WMS serves job submission request returning 

an ID that identifies the job in its task queue– The ID is used to query (or register for notifications 

by) the LB service until task termination (or failure)

Page 26: A Practical Approach for a Workflow Management Systemcppwfms.sourceforge.net/papers/slides-CGW07.pdf · Workflow Management Systems Overview A lot of interest around WfMSs exist –

From abstract to concrete workflow (2/4)

JobExecuteJobExecute

P1 P1­1

JobRegister

jobIDjdl

InputSandbox

P1­2

data movement

JobStart

P1­3

wait_for_termination

P1­4

P1­5P2

if(result.FAIL)

if(result.SUCCESS)data movement

RecoveryStrategy

result

Task Task ExecutionExecution

P1

JobExecute

P2

Page 27: A Practical Approach for a Workflow Management Systemcppwfms.sourceforge.net/papers/slides-CGW07.pdf · Workflow Management Systems Overview A lot of interest around WfMSs exist –

From abstract to concrete workflow (3/4)

wait_for_termination (polling)wait_for_termination (polling)

P1­3­1

getJobState

P1­3

N sec

jobID

if(s.job_running)

if(s.job_done || s.job_fail)

P1­4

s

● Waits until the task is done (or failed) using polling:–  getJobState() operation of the LB service.

Page 28: A Practical Approach for a Workflow Management Systemcppwfms.sourceforge.net/papers/slides-CGW07.pdf · Workflow Management Systems Overview A lot of interest around WfMSs exist –

From abstract to concrete workflow (4/4)

wait_for_termination (notify)wait_for_termination (notify)

jobIDP1­4P1­3

P1­3

N sec do_polling

result

● Same sub­workflow implementation using notifications;

Page 29: A Practical Approach for a Workflow Management Systemcppwfms.sourceforge.net/papers/slides-CGW07.pdf · Workflow Management Systems Overview A lot of interest around WfMSs exist –

Conclusions and Future Work

● Workflow Gateway as a standalone component– Other WfMS can easily take advantage from 

languages conversions– customizable target language depending on the 

underlying WfMS● A lightweight WfMS with basic functionalities

– solves low level aspects of workflow management● Investigate WfMSs engine interoperability

Page 30: A Practical Approach for a Workflow Management Systemcppwfms.sourceforge.net/papers/slides-CGW07.pdf · Workflow Management Systems Overview A lot of interest around WfMSs exist –

Thank You!


Recommended