Grid Tools Jai Dayal & Casey Rathbone. Outline Web Services Unicore BOINC Globus Condor

Grid Tools

Jai Dayal & Casey Rathbone

Outline Web Services Unicore BOINC Globus Condor

Web Services“A Web service is an abstract notion that must be implemented by a concrete agent. … The agent is the concrete piece of software or hardware that sends and receives messages, while the service is the resource characterized by the abstract set of functionality that is provided.”

Web Services

Web Services

Remote Procedure Calls (RPC) - distributed function call interface

Service Oriented Architecture (SOA) - utilizes message passing to communicate to services also known as "message-oriented" services

Representational State Transfer (REST) - emulates HTTP (or similar protocols) to communicate stateful resources

Unicore - Overview Single point of entry – security

Uses SSL

Uses web services SOAP Messages WSDL

Reputation for simple deployment

Unicore - Architecture

Most basic resourceA cluster, a PC, web service, etc

Vsite

Unicore - Architecture The gateway is the point of entry to the

Unicore grid

Vsites

Gateway

Unicore - Architecture Gateway + Vsites = Usite

Vsites

Gateway

Usite

Unicore - ArchitectureUNICORE - Architecture

Vsites

Gateway

Usite

Client

Unicore - Architecture

Vsites

Gateway

Usite

Client

Client communicates using SOAP SOAP/HTTP binding

SOAP Message

SOAP contains: Address of

gateway Port number Vsite name task descriptions

Unicore - Security• Gateway handles authentication

– SSL used to authenticate the client– SSL also used by client to authenticate

gateway– Both present digital certificates

• Vsite handles authorization– Gateway passes along certificate in SOAP header– If client has privileges, Vsite authorizes

•This model allows end to end security

• Removes some strain from gateway

• Trust between gateway and Vsite

BOINC - Overview

Grid computing vs. public resource computing

Grid Public Resource Computing

Owned by institutions/orgs Individuals own pc's

Managed/used by pro's Used by commoners

High level of trust Low (no?) level of trust

BOINC - Overview

Grid computing vs. public resource computing

Grid Public Resource Computing

Owned by institutions/orgs Individuals own pc's

Managed/used by pro's Used by commoners

High level of trust Low (no?) level of trust

BOINC GoalsCheap and easyProject varietyPublic participation → reward points

BOINC - Overview Grid computing vs. public resource computing

Grid Public Resource Computing

Owned by institutions/orgs Individuals own pc's

Managed/used by pro's Used by commoners

High level of trust Low (no?) level of trust BOINC Goals

Cheap and easyProject varietyPublic participation → reward points

BOINC projectsSETI at homeFolding at homePredictor at home

BOINC - ArchitectureBOINC

BOINC - ArchitectureBOINCProjects

BOINC - ArchitectureBOINCProjects

Servers

Scheduling Data

BOINC - ArchitectureBOINCProjects

Servers

Scheduling Data

Tools

BOINC - ArchitectureBOINCProjects

Servers

Scheduling Data

Tools

Client

BOINC - Workflow Client contacts Scheduling Server

Server returns XML Document

Workunit: inputs for a computation/jobApplicationsInput filesEnvironment variables

Client runs job and places results in output files

Result:Reference to workunitReference to output files

BOINC – Fault tolerance

TransitionerIdentifies bad results

ValidatorDetermines canonical result

Public Resource Computing is volatileUsers leave before job completingOver-clocking can lead to bad resultsMalicious users

AssimilatorHandles canonical result

These daemons can run on different hosts

Homogenous RedundancyComputer architecture specific jobs

BOINC – Client settings Client implements “local scheduling policy”

Utilizes maximum resourcesMeet job deadlinesRespect clients resource allocation wishesOffers some variety to the client

Anonymous platformUsed for open source projectsClients can modify apps to fit their resourcesCommunicate with scheduling server using XML config file.

Globus Toolkit 4

• GT4 is a open-source toolkit developed to build grids • It provides full capabilities for sharing computing power

and databases • Usage of Globus is extensive throughout the scientific

community within NSF, DOE, DARPA, IBM, NASA, and Microsoft projects

• The Large Hadron Collider project utilizes a Globus based grid for the European Data Grid.

Globus Toolkit 4

Globus Toolkit is a suite of Web-Services and Unix that controls provides an interface to submit, monitor, and cancel jobs on the Grid resources.

Globus Toolkit is an interface that allows for jobs to be submitted and executed at the appropriate resources within the grid

Globus Toolkit 4

Globus Toolkit 4 Security• GT4 uses (Grid Security Infrastructure) GSI with an public

key cryptography • GSI Functional Layers - Authorization, Authentications,

Delegation, Message Protection, and Message Format

Globus Toolkit 4 Data Management

• Data Management tools within the toolkit fall into either of two categories data replication and data movement

• Data Replication consists of Replica Location Service (RLS) • Data Movement consists of GridFTP and Reliable File

Transfer (RFT)

GT4: Data Replication

Replica Location Service provides the capability to track and maintain multiple locations of data across the grid.  It is a distributed registry system that allows users and applications to register the locations of data.

GT4: Data Movement

• GridFTP - GT4 provides a server implementation, scriptable command-line interface and several GUI client interfaces

 • Reliable File Transfer (RFT) - A Web Services Resource

Framework that schedules file transfers based on a set of criteria of when specific resources and bandwidth is available

Globus Toolkit 4 Execution Management

• Community Scheduler Framework - Grid level meta-scheduling framework

• Grid Teleoperation Control Protocol - A service interface that has been used to

• Dynamic Accounts - This service provides functionality to create groups and accounts for specific jobs that need to be submitted

• Grid Resource Allocation & Management - Used to locate, submit, monitor and cancel jobs on Grid computing resources.  It is the fabric that ties all the schedulers and job queues together to utilize the resources.

GT4: Community Scheduler Framework

Open-Source implementation of OSGA-based meta scheduler

Provides an interface to existing grid platforms making integration more efficient

GT4: Community Scheduler Framework

The diagram below shows the basic layout of the scheduler framework

http://www.globus.org/grid_software/computation/csf.php

GT4: Grid Teleoperation Control Protocol

The GTCP is used to control heterogenous physical and computational simulations.It exposes the WSRF-compliant service interface used to control remote instruments and simulations and a interface to the backend GTCP to allow for additional control abstraction.

Globus Toolkit 4 Information Services

Monitoring and Discovery Service is a suite of web services used to discover resources and services on a grid

This service is dedicated to handling the complex task of organizing resources into Virtual Organizations (VO)

http://www.globus.org/toolkit/docs/4.0/info/key-index.html

Globus Toolkit 4 Common Runtime

GT4 contains a multitude of implementations of libraries allowing the suite to be platform independent.

• C Common Libraries• C WebService Libraries• Java WebService Libraries• Python WebService Libraries• XIO

Condor - Overview• University of Wisconsin – Madison• Open source• Linux, Solaris, Mac OS, Windows• No program modification

• Jobs tend to be architecture specific– Designed on x86 → job run on x86– Java applications possible exception

• Overall concept is simple– Install it on each desired machine– Designate one as central manager

• Practice is much harder– Installation is tricky– Lots of commands to learn

Condor• Central Manager

– Detects idle machines– Matches job with available resources– Considers each processor as separate resource

• Jobs– Submitted via “submit file”– Universe, executable name, inputs, outputs, error, log file

• Sequential Jobs– Standard universe

• Used for long jobs (weeks, months)• Allows partial execution, pausing• Can relocate job to another machine

– Vanilla universe• For short term jobs

– Local universe• Submitter executes job

• Parallel Jobs– Supports MPI and PVM

Condor - variety• Flocking

– Connects multiple pools– If current pool has no resources, job “flocks”– Set “flock_to” variable in config file

• Condor-G– Distributed flocking– Utilizes Globus– Universe = Globus– Also specify “globusscheduler”

• Points to the globus job manager

References[1] D. P. Anderson, BOINC: A System for Public-Resource Computing and Storage.  In Proceedings of the 5th IEEE/ACM International Workshop on Grid Computing. November, 2004.

[2] S. Bagchi, E. Hung, A. Iyengar, N. Vogl, and N. Wadia, Capacity Planning Tools for Web and Grid Environments.  In Proceedings of the 1st international conference on Performance evaluation methodolgies and tools. October, 2006.

[3] F. Hernández, P. Bangalore,  K. Reilly, End-User Tools for Grid Computing. In Proceedings of the first workshop on End-user software engineering. July, 2005.

[4]I. Habib, Getting Started with Condor.  Linux Journal, Volume 2006, Issue 149, September 2006.

[5]Menday, R., "The Web Services Architecture and the UNICORE Gateway," Telecommunications, 2006. AICT-ICIW '06. International Conference on Internet and Web Applications and Services/Advanced International Conference on , vol., no., pp. 134-134, 19-25 Feb. 2006

[6]Hellinckx, P.; Stuer, G.; Hendrickx, W.; Arickx, F.; Broeckhove, J., "Grid-user driven grid research, the CoBRA grid," Cluster Computing and the Grid Workshops, 2006. Sixth IEEE International Symposium on , vol.2, no., pp. 8 pp.-, 16-19 May 2006