Gsoc airavata

Apache Airavata

GSOC 2013

Knowledge and Expertise

Computational Resources

Scientific Instruments

Algorithms and Models

Archived Data and Metadata

Advanced Science Tools

Target Community: Science Gateways Enabling & Democratizing Scientific Research

What does Apache Airavata do?• Compose, manage, execute,

and monitor distributed, computational workflows.

• Wrap legacy command line scientific applications with Web services.

• Run jobs on computational resources ranging from local resources to computational grids and clouds.

• Manage provenance data.

Workflow Interpreter

Application Factory

Message Box

Registry

Apache Airavata

API

Lorem ipsum

insolens

p1m5

duo x

End

Use

rsG

atew

ay

Dev

elop

er

Scientific Applicati

on

Core Developer

Computational Resources

Apache Airavata

Apache Airavata ComponentsComponent Description

XBaya Workflow graphical composition tool.

Registry Service Insert and access application, host machine, workflow, and provenance data.

Workflow Interpreter Service

Execute the workflow on one or more resources.

Application Factory Service (GFAC)

Manages the execution and management of an application in a workflow

Messaging System WS-Notification and WS-Eventing compliant publish/subscribe messaging system for workflow events

Airavata API Single wrapping client to provide higher level programming interfaces.

Hi, I’m Nolram. I’m a computational

physicist. I run computational

experiments everyday

This is how typically I run my experiments

Scientific Application

Another ScientificApplication

First I collect my observed data

And then pass data to my applications & get

the result

This is starting to become a very tiring

task

How can I make this much simpler…?

Logically, this is how my life would be

made easier…

Is it possible to automate this flow

sequence without my guidance?

Scientists from many different fields face this

problem everyday.

The solution is to use a workflow-powered science gateway to

manage the experiment online.

What is a workflow you ask? Well, you just saw one in

our previous animation…

We introduce Apache Airavata, a system capable of composing, managing, executing, and monitoring

small to large scale applications and workflows

Want to see how it works?

A Typical Workflow

Apache Airavata

I will handover my data & my experiment details (the workflow)

to the Airavata server

The Gateway

Airavata will complete the experiment & return me the results

Results

Progress of the experiment

… and while I wait for results, Airavata will notify me with

progress updates of my experiment

Let’s look closely how Airavata manages workflows.

The GatewayResults

Experiment progress

Apache Airavata

Let’s look closely how Airavata manages workflows.

The GatewayResults

Experiment progress

Airavata main has 4 components…

The Gateway

1. Workflow InterpreterSteer the workflow execution

2. The GFacSteer science app executions & data

transfers

Workflow Interpreter

GFac

Message Box

Registry

3. The RegistryDefines the available applications &

records all results of experiments

4. The Message BoxRecords the progress of the workflow

execution

A Stable API for Airavata

Apache Airavata

Lorem ipsum

insolens

p1

m5

duo x

End

Use

rsG

atew

ay D

evel

oper

Scientific Application

Computational Resources

Application Registration

UIApplication Developer

A1

Airavata Service Interface

(wraps client API)

Service Map XML

Airavata Server

Service Map to AWSDL

Web Based workflow composerWorkflow

Developer

Get AWSDL

Put XWF

A2

A3

W1W2

W3

Web Based Experiment

BuilderExperimentBuilder

Get WI’s

E1

E2

E3

W4

Shred Workflow

Inputs

Launch Workflow

Web Based Workflow MonitorWatch Progress

M1M2

Get Workflow Graph

M3 Monitor Workflow

Goal of the project

• Design Web-Based interfaces for Airavata:– Application Registration– Workflow Construction– Workflow Execution– Workflow Monitoring

• Provide an opportunity for GSoC to understand Distributed System in action

• Scope for Research and Software Engineering papers

Data Model

• Application Description – User describes inputs and outputs of the

application. – Currently this information is captured in Service

Map Schema. – This schema is stored in Airavata Registry as XML.

Also the schema utility generates a application service WSDL from this schema using the Airavata WSDL Generator.

Application Registration

UI

Application Developer

A1

Service Map XML

Service Map to AWSDL

Web Based workflow composer

WorkflowDeveloper

Get AWSDL

A2

W1

W2

XML

Aira

vata

Ser

ver A

PI

Launch & Manage Jobs

Notify progress of job or workflow execution

Real-Time Monitoring

Messaging Subsystem

Registry

Execute & Manage Computations

Workflow Interpreter

Application Factory (Gfac)

Application Desc

Application Desc

Workflow

A peek at one of the cluster

InterconnectNodes

Scheduling ‘qsub’ batch jobs on the cluster worker node

worker node

worker node

worker node

worker node

SGE MASTER node

Queue-A Queue-B Queue-C

A S

lot 1

A S

lot 2

B S

lot 1

C S

lot 1

C S

lot 2

C S

lot 3

B S

lot 1

B S

lot 2

B S

lot 3

B S

lot 1

C S

lot 1

C S

lot 2

A S

lot 1

B S

lot 1

C S

lot 1

Queues

Policies

Priorities

Share/Tickets

Resources

Users/Projects

JOB Y JOB ZJOB X

JOB U

JOB OJOB N

System characteristics

System statusResources

Job policiesResources

Resource Matching

JOB

User

User policiesGroupsRolesDepartmentsProjects

Selection Scheduling

Simplified Gateway Architecture

One time Gateway Community Setup

Community Account Grid Certificate username, password

Gateway InterfaceGateway Server

Compute Servers

Gateway Authentication

Fetch

Community C

reden

tialGrid

Prox

y

Job Submit or File Transfer request

Output

Proxy, Job Request

Job Status, Output

Step 0

Step 1

Step 2,3,,

XSEDE User Portal

ParamChem

BioVLab

GridChem

DES

VLAB

UltraScan

Apache Airavata 1.0

NSG

ParamChem

BioVLab

CIPRES

POPLAR

GridChem

DES UltraScan

VLAB

…

…

…

Apache Airavata 2.0

Apache Airavata 1.0

Apache Airavata 1.0

Apache Airavata 1.0

Apache Airavata 1.0

Apache Airavata 1.0