Holding slide prior to starting show

Portals for Workflow Composition and Service

Management

David W. Walker

School of Computer Science

Cardiff University

http://www.cs.cf.ac.uk/User/David.W.Walker/

System Portals

• We need portals to be able to handle tasks such as: :– Problem specification, resource discovery and job

composition– Service deployment and lifecycle management– Submission, execution, monitoring, and (maybe)

interaction

• These tasks are quite generic, rather than application-specific.

• Portal can behave rather like a “Grid Shell”

Web Services

• Everything is a (Web/Grid) service.

• This includes:– Computation routines– Access to files and databases– Components of the Grid infrastructure,

such as workflow enactment engines, resource monitors, etc.

A Common Approach to Workflow

• Visual service composition.

• Service interfaces and other metadata expressed in an XML-based service description document.

• Services registered with, and discovered through, a registry.

Visual Service Composition Portlet

• Applications and high-level services are composed from interacting services.

• Composition can be done graphically using a “drag and drop” style of interface to draw a workflow.

• In a workflow graph the nodes represent services and the edges represent data flow or control flow.

• Need special nodes to represent loop and conditional constructs.

• Need query mechanism to discover services

Workflow Description

• A workflow can be represented with an XML-based workflow description language such as WSFL, SWFL, SCUFL, BPEL4WS, etc.

• A workflow description document can be passed to a workflow enactment engine, which provides an execution environment for the workflow.

Two Hard Problems

• Semantic specification of applications• Scheduling of workflow nodes on distributed

resources.– Early binding model: bind to specific service/platform

at composition time (“validation”).– Intermediate binding model: bind at “compile” time

(when converting from XML to executable form).– Late binding model: bind dynamically at runtime.

• Later binding allows the use of more up-to-date information to make scheduling decisions.

Mobility

• Should (some) services be mobile and/or clonable?

• Allows the automatic creation of new services that are heavily used.

• May be important for autonomic grids.

• Generalises current data migration and replication concepts.

Communication and Notification

• Another important issue is how to carry out the data flow and control flow in a workflow.

• Need an asynchronous messaging system to support interoperability and to handle connection problems.

Portlets

• Portlet technology endorsed by GCE working group of GGF, e.g., JetSpeed

• Web services have user-facing ports that provide content to users.

• Portlets can be regarded as interfaces to that content, specified via an extension to WSDL.

• Portals can be built from collections of portlets, e.g., as a set of panels for job submission, job status, visualisation, etc.

• Portlets provide as component model for user interfaces in the same way that web services provide a component model to middleware.

Grid-Oriented Portal Work at Cardiff

• Visual Service Composition Environment (VSCE) for building applications based on service workflow.

• Service Workflow Language (SWFL) for describing composite applications.

• Tools for transforming between workflow graph SWFL executable code.

• Jini Service-oriented Grid Architecture (JISGA). An execution environment taking SWFL as input.

• Java-C Automatic Wrapper (JACAW). Wraps C code as Java using JNI for deployment as Web service in JISGA.

• Grid-Service-in-a-Box (GSiB). Supports service creation, deployment, management, and monitoring.

Motivation and Targets

• Easy-to-use tools are needed to allow scientists to take full advantage of the new Grid without them needing detailed knowledge of the underlying infrastructure.

• GSiB project is developing a visual problem-solving environment/portal for all service users (clients and providers).

Service Deployment

• A web service– Its public interfaces and bindings are described in

an XML document.– It is registered in a service registry to allow service

discovery and query.

• Usually additional code or wrapper is needed to make existing software available as a Web service – a lot of repeated work.

• Legacy problems exists.

Service Deployment

• JACAW (Java-C Automatic Wrapper) automatically generates the JNI wrapping code for a legacy code.

• Axis Java2WSDL utility generates WSDL document from the Java code.

• UDDI registry server.• UDDI4J provides a Java API for

interacting with a UDDI registry.

Service Deployment

JNI Wrapper

Legacy code

WSDL Document

UDDI Registry

JACAW

JAVA2WSDL

UDDI4J

Service Composition

• Graphical symbols for constructing a flow model.

Data input port

Data output port

Control input port

Control output port

Data link

Control link

Normal and assignment activity

Control activity

An Example

A

B C

For

retA = activityA(..);

for( i=0; i<100; i++){

activityB(retA, i);

}

retC = activityC();

retA

YES

NO

retA i

Service Workflow Language (SWFL)

<activity name="for"> <for setParallel="no"> <input message=“ForInput”/> <expression><![CDATA[index=0;index<100;index++]]></expression> </for></activity>

<dataLink name=“A_For” source=“A” target=“for”> <swflMap sourceMessage=“AOutput” targetMessage=“ForInput”>

<part> <sourcePart name=“retA”/><targetPart name=“retA”/> </part> </swflMap></dataLink>

To be continued

Service Workflow Language (SWFL)

<dataLink name=“For_B” source=“for” target=“B”> <swflMap sourceMessage=“ForOutput” targetMessage=“BInput”> <part> <sourcePart name=“retA”/><targetPart name=“retA”/> <sourcePart name=“i”/><targetPart name=“i”/> </part> </swflMap></dataLink

<controlLink name=“A_For” source=“A” target=“for” /><controlLink name=“For_B” source=“for” target=“B” controlPort=“YES”/><controlLink name=“For_C” source=“for” target=“C” controlPort=“NO”/><controlLink name=“B_For” source=“B” target=“for” />

Service Workflow Language (SWFL)

• SWFL is XML based for describing interacting Web services.

• Extends Web Service Flow Language (WSFL) by – supporting the application of all the

conditional and loop control constructs of the Java language to the composition of Web services.

– allowing more general data mappings.

Another Picture

ValidatorValidator

Create SWFLCreate SWFLDrawing ScreenDrawing Screen

swflJobswflJob

WorkflowEngine

WorkflowEngine

Visual Service Composition Environment

Execution Environment

SWFL2Java

• It converts the description of a job in SWFL into executable Java code.

SWFLDescriptionDocument

Java Executable

Code

DataflowGraph

ControlFlowGraph

FlowModel

Visual Service Composition Environment

• Collaborative code development environment using a visual programming tool for seamlessly integrating services into higher-level services or complete applications.

• Applications are created by plugging together services to form a workflow graph. Service metadata is used to check if interfaces are compatible.

• Different developers can place their components in a shared repository, e.g, UDDI or Jini lookup service.

• Output is in the form of SWFL.

JISGA

• JISGA consists of two main parts:– A WorkflowEngine service– A JobProcessor service

• Grid application is submitted to a WorkflowEngine service as SWFL.

• Sequential jobs are handled directly by the WorkflowEngine service.

• Parallel jobs involve multiple JobProcessor services.

What the Workflow Engine Is

• Processes service-based applications described in an XML-based workflow language.

• Determines the order of execution and generates the harness code.

• Executes the code which includes discovering services, invoking services, receiving results and sending results to clients.

What the Workflow Engine Does

• Converts a SWFL description of a composite service-based job into an executable Java code, and executes it.

Java Executable Code

<?xml…><JFlowModel……………………….……….….

SWFL description

IntermediateFlowModel

object

SWFL2Graph Graph2Java

SWFL2Java

To be continued…

Future Research• Further development of GSiB for creating,

deploying, managing and monitoring Grid applications.

• More experiments in integration of Jini services and Web services in JISGA.

• Demo applications to illustrate use of JISGA on UK e-Science Grid.

• Integration of better scheduling in JISGA.

Final Comments

• Service-oriented approach to Grid computing is generally agreed on

• But there are many options for how such Grids might be build (GT3, Jini, JXTA, etc)

• Ability to interoperate between different Grids is important.

• Need to get more experience with actually using services for e-Science.