Grid Application Support by the P-GRADE Portal

1CEOS WGISS Meeting, May 8-12, 2006, Budapest

MTA SZTAKIComputer and Automation Research Institute

Hungarian Academy of Sciences

www.lpds.sztaki.hu/[email protected]

Grid Application SupportGrid Application Support by the P-GRADE Portal by the P-GRADE Portal

Gergely SiposGergely Sipossipossipos@@sztaki.husztaki.hu


OutlineOutline

• Motivation of our work

• P-GRADE Portal in a nutshell

• Grid application development

• Grid application execution and monitoring

• Support for collaborative work

• Conclusions


Layered view of Grid systemsLayered view of Grid systems

Basic Grid services:security, job submission, info, …

Higher-level grid services: brokering, data replica, …

Application toolkits, portals

Applications

Command line grid tools

Graphical grid tools:P-GRADE Portal

Resources to be shared

Grid middleware:e.g. Globus, EGEE, …

E-Scientists


Motivation to theMotivation to theP-GRADE PortalP-GRADE Portal

• Fast evolution of Grid middleware technologies and tools:– GT2, OGSA, GT3 (OGSI), GT4 (WSRF), LCG-2, gLite, …

• Many production Grids are now freely available for e-Scientists– EGEE (LCG-2 gLite), UK NGS (GT2),

US Open Science Grid (GT2 GT4), NorduGrid (ARC), …• The same set of services are available everywhere, but

implemented in different ways– Computation services, data services, security services, (brokers)

Let’s provide a technology-neutral, intuitive graphical interface for the most common Grid middleware services!


P-GRADE Portal in a nutshellP-GRADE Portal in a nutshell• General purpose, workflow-oriented computational Grid

portal. Supports the development and execution of workflow-based Grid applications.

• Based on standard portlet framework (Gridsphere)– Easy to expand with new portlets (e.g. application-specific portlets)– Easy to tailor to community needs

• Grid services supported by the portal:

Service EGEE grids Globus grids

Job execution Computing Element GRAM

File storage Storage Element GridFTP server

Certificate management MyProxy

Information system BDII MDS-2

Brokering Workload Management System ---

Job monitoring Mercury

Workflow & job visualization PROVE

The P-GRADE Portal hides middleware technologies and solves Grid interoperability problem at the workflow level


What is a P-GRADE Portal workflow?What is a P-GRADE Portal workflow?

• a directed acyclic graph where– Nodes represent jobs (batch

programs to be executed on a computing resource)

– Ports represent input/output files the jobs expect/produce

– Arcs represent file transfer operations

• semantics of the workflow:– A job can be executed if all

of its input files are available


Two levels of parallelismTwo levels of parallelism

• The workflow concept of the P-GRADE Portal enables the efficient parallelization of complex problems

• Semantics of the workflow enables two levels of parallelism:

Each job can be a parallel program

– Parallel execution inside a workflow node– Parallel execution among workflow nodes

Multiple jobs run parallel


25 x

10 x25 x 5 x

Forecasting dangerous weather situations (storms, fog, etc.), crucial task in the protection of life and property

Processed information:surface level measurements, high-altitude measurements, radar, satellite, lightning, results of previous computed models

Requirements:•Execution time < 10 min•High resolution (1km)

Ultra-short range weather forecast Ultra-short range weather forecast (Hungarian Meteorology Service)(Hungarian Meteorology Service)


Multi-Grid portal: Multi-Grid portal: Demonstration @ EGEE User ForumDemonstration @ EGEE User Forum

Job Repository Job Repository (GEMLCA)(GEMLCA)

ManchesterPortal Client in Geneva

Leeds

P-GRADE Portalserver

Portal Server in London

Executable

NGS GT2 (UK)

WestFocus GT4 Grid (UK)

UoW

Brunel

Poznan

Budapest

EGEE (CE)

Executable

Executable

ServiceInvocation

Executable


The typical user scenarioThe typical user scenarioPart 1 - development phasePart 1 - development phase

Certificate servers

Portalserver

Gridservices

START EDITOR

OPEN & EDIT or DEVELOP WORKFLOW

SAVE WORKFLOW


Certificate servers

Portalserver

Gridservices

TRANSFER FILES, SUBMIT JOBS

DOWNLOAD RESULTS

DOWNLOAD RESULTS

The typical user scenarioThe typical user scenarioPart 2 - execution phasePart 2 - execution phase

VISUALIZE JOBS and

WORKFLOW PROGRESS

MONITOR JOBS

DOWNLOAD PROXY CERTIFICATES

SUBMIT WORKFLOW


Opening the workflow editorOpening the workflow editor

The editor is a Java Webstart applicationdownload and installation is only a click!


Workflow Workflow EditorEditor

Define DAG of batch jobs:1. Drag & drop components:

jobs and ports

2. Connect ports by channels (no cycles, no loops)

3. Define job and port properties


Job property windowJob property window

Properties of a job:• Binary executable• (Type of executable)• (Number of processors)• Command line

parameters• The resource to be used

for the execution:•Grid•Broker / resource


Support for manual resource selection:Support for manual resource selection:information system browserinformation system browser

The information system portlet queries EGEE

and Globus information systems


Support for broker-based Support for broker-based resource selectionresource selection

1. Select a broker Grid/VO for the job

2. (Specify extra ranks & requirements for the job in Job Description Language)

3. The broker will find the best resource for your job!


Defining input/output dataDefining input/output data

File type: input: required by the job output: produced by the jobFile location: local: my desktop remote: grid storage siteFile name: unique name of the fileFile storage type: Permanent: final result of the WF Volatile: only used for inter-job data transfer


The typical user scenarioThe typical user scenarioExecution phase – step 1:Execution phase – step 1:

Certificate servers

Portalserver

Gridservices

DOWNLOAD PROXY CERTIFICATES


Prerequisite of workflow execution: Prerequisite of workflow execution: Download user certificatesDownload user certificates

1. Certificate server details:• Hostname• Port number• User name• Password

2. Certificate parameters:• Lifetime


Result of certificate downloadResult of certificate download

Multiple certificates for multiple Grids/VOs can be available on the portal server at the same time!

SEE-GRID resourcesHUNGRID resources

21CEOS WGISS Meeting, May 8-12, 2006, BudapestWhite/Red/Green color means the job is initial/running/finished state

Workflow ExecutionWorkflow Execution(observation by the workflow portlet)(observation by the workflow portlet)









White/Red/Green color means the job is initialised/running/finished


On-Line application monitoringOn-Line application monitoring

- The portal monitors and visualizes parallel jobs

- The portal monitors and visualizes workflows:

-Input data transfer

-Job execution

-Output data transfer


Downloading the results…Downloading the results…

- Download small files in a single ZIP file

- Keep large files in the Grid for future analysis!


Collaborative work Collaborative work by the P-GRADE Portalby the P-GRADE Portal

• Export and import workflows and execution results from/into the portal

• Share legacy applications as services with other portal users• Share successfully executed jobs as services with other

portal users• Integrate other users’ legacy applications and jobs into your

workflow


• P-GRADE Portal service is available for– SEE-GRID infrastructure– HUNGRID VO of EGEE– Central European VO of EGEE– GILDA: Training VO of EGEE– UK National Grid Service

• Under preparation for– US Open Science Grid, Economy-Grid,

Swiss BioGrid, BioInfoGrid, Turkish Grid

• P-GRADE portal can be installed for any public/private EGEE or Globus Grid/VO

Please contact us!

RReferenceseferences


Main conclusion:Main conclusion:Easy-to-use, technology-neutral Easy-to-use, technology-neutral

Grid portal for e-ScientistsGrid portal for e-Scientists• The P-GRADE Portal hides differences of Grids

• Globus X – LCG2 – gLite Grid interoperability• Transparent switching between Grid technologies

• Graphical tools for application development, execution and monitoring

• Seq & parallel components can be integrated into large Grid applications

• Your code does not have to contain grid specific calls• Support for collaborative research

• Share workflows• Share legacy applications and jobs as application services

• Built by standard portlet API• customizable to specific application areas and user groups

(such as earth scientists…)


Further information…Further information…

– Visit the www.lpds.sztaki.hu/pgportal site(manuals, slide shows, installation procedure, etc.)

– Visit or request a training event!(see list of events on P-GRADE Portal homepage)

• Lectures, demos, hands-on tutorials, application development support

– Get an account for a portal service:• VOCE portal - SZTAKI• SEEGRID portal – SZTAKI• HUNGrid portal – SZTAKI• NGS portal – University of Westminster

– Install a portal for your community: If you are the administrator of a Grid/VO SZTAKI is pleased to help you install a portal for your community!


• P-GRADE Portal tutorial in Paris (free event!)22-23, June – lectures, hands-on, application

supportSee details on P-GRADE Portal homepage!

• Joint Regional CE EGEE and SEE-GRIDSummer School on Grid Application Support

3-8 July, Budapest, SZTAKI

www.egee.hu/grid06

Thank you!Thank you!Questions?Questions?

Documents

Grid Application Support by the P-GRADE Portal