Managed by

P. Kacsuk, G. Sipos, A. Toth, Z. Farkas, G. Kecskemeti P. Kacsuk, G. Sipos, A. Toth, Z. Farkas, G. Kecskemeti and G. Hermannand G. Hermann

MTA SZTAKIwww.lpds.sztaki.hu/pgportal

pgportal@lpds.sztaki.hu

Defining and Running Parametric Study Workflow Applications by the P-GRADE

Portal

European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies 2

Outline of the talk

1. Introduction to P-GRADE portal

2. PS-extension motivations

3. Principles of the PS-portal

4. Usage of the PS-portal

5. Distributing workflows among different Grid resources

6. Conclusions

European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies 3

Proxy management

P-GRADE portal in a nutshell

Definition of Grid resources

Workflow creation

Job mapping to Grid resources

Workflow management and

execution visualization

European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies 4

References

P-GRADE Portal service is available for

– Central European VO of EGEE– GILDA: Training VO of EGEE– HUNGRID VO of EGEE– SEE-GRID infrastructure– GIN VO– National Grids (UK National Grid Service, HunGrid, etc.)

Under preparation for

– US Open Science Grid, TeraGrid– Economy-Grid, Swiss BioGrid, Bio and Biomed EGEE

VOs, BioInfoGrid, BalticGrid – Many national Grids

European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies 5

User’s feedback

Users want parameter study (PS) support at workflow level

It means:– If the user has an existing workflow in a

repository, he would like to run • the same workflow (without any change) • with many different parameters

European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies 6

Related research

1. PS at job level – main subject is scheduling

2. PS at workflow level – The workflow description supports the creation of PS runs

3. What our users want: executing workflows stored in workflow repositories with many inputs as PS

European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies 7

Principles of the PS-portal

1. Take the existing workflow as a black box

2. Modify only the external input ports to accept many input files

3. Create the cross-product of the input files and run the workflow with all combinations

4. Because of the large number of files both PS inputs and outputs are stored in the Grid!

Storage elements

Storage elements

Storage elements

4 input files

3 input files

12 outputfiles

European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies 8

Generating cross products and executable workflows (e-WFs)

PS port: 4 instances of the

input file

PS port: 3 instances of the input file =

4 x 3 normal workflow execution1 PS workflow execution

European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies 9

Steps of creating a PS-workflow1. Start from a tested Workflow

European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies 10

2. Define the PS Port(s)

Any external input port can be defined as

PS-port

European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies 11

3.Define Subdirectory of Inputs

PS port is represented by a directory and not

by a single file

European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies 12

4. Define target Grid and subdirectory for the results

European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies 13

5. PS Workflow ready to Submit

European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies 14

Progress of submissions in PS detailed view

Total = Init + Submitted + Rescue + Error + Finished

European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies 15

How to distribute workflows among different Grid resources?

PS port: 4 instances of the

input file

PS port: 3 instances of the input file =

Goal: Assign the 24 jobs to 24 different Grid resources1 PS workflow execution

European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies 16

Single-Grid, Static Assignment

PS port: 4 instances of the

input file

PS port: 3 instances of the input file =

Assigns the 24 jobs to 2 Grid Resources within 1 Grid1 PS workflow execution

Assign resource A in Grid X

Assign resource Bin Grid X

A A A A

A A A A

A A A A

B B B B

B B B B

B B B B

European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies 17

Single-Grid, Dynamic Assignment via Broker

PS port: 4 instances of the

input file

PS port: 3 instances of the input file =

1 PS workflow execution

Assign to broker of Grid X

Assign to brokerof Grid X

A B C D

I J K L

R S T V

E F G H

M N O P

U X Y Z

Assigns the 24 jobs to 24 Grid Resources within 1 Grid

European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies 18

P-GRADE-Portal

London Rome

Athens

TeraGrid (GT2/GT4)

SEE-Grid (glite)

P-GRADE portal is a multi-Grid portal

European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies 19

Static Multi-Grid Assignment

PS port: 4 instances of the

input file

PS port: 3 instances of the input file =

1 PS workflow execution

Assign to broker of Grid X

Assign to brokerof Grid Y

XA XB XC XD

XI XJ XK XL

XR XS XT XV

YE YF YG YH

YM YN YO YP

YU YX YY YZ

Assigns the 24 jobs to 24 Grid Resources within 2 Grids

European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies 20

Dynamic Multi-Grid Assignment

PS port: 4 instances of the

input file

PS port: 3 instances of the input file =

1 PS workflow execution

Assign to meta-broker

Assign to meta-broker

XA XB YC XD

YI XJ XK XL

XR XS XT YV

YE XF YG XH

YM XN YO YP

YU YX YY XZ

Assigns the 24 jobs to 24 Grid Resources within 2 Grids

European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies 21

Conclusions

Our solution:1. Keeps both the workflow GUI and the parameter

study support concept as simple as possible. 2. Enables run any existing workflow with different

parameter sets without modifying the structure of the workflow.

3. Manages the execution of the workflows on as many Grid resources as possible.

4. Enables the collection of Grid resources from several Grids even if they are based on different Grid technology.

European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies 22

Thanks for your attention

Thanks for your attention

?

Further information: www.lpds.sztaki.hu

European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies 23

Distributing workflows among different Grid resources

1. Single-Grid, Static Assignment: Statically assigning each node of the workflow to different resources in the same Grid (done)

2. Single-Grid, Dynamic Assignment: Dynamically assigning each node of the workflow to different resources in the same Grid using the broker of the Grid (done)

3. Static multi-Grid Assignment: Statically assigning different Grids to different nodes (done)

4. Dynamic multi-Grid Assignment: Dynamically assigning the nodes to different Grids using a meta-broker (under investigation)

European Research Network on Foundations, Software Infrastructures and Applications for large scale distributed, GRID and Peer-to-Peer Technologies 24

Multi-Grid portals with meta-broker

EGEE Grid

UK NGS

BrokerBroker

BrokerBroker

Meta-Broker

Selects Grid for each node of the PS workflow

Provides the largest achievable parallelism for PS workflows