A Framework for Dynamic Resource Management on the Grid Jyotishman Pathak, Jem Treadwell, Raj Kumar, Philip Vitale, Fernando Fraticelli HP Laboratories Palo Alto HPL-2005-153 August 22, 2005* Grid resource management, Dynamic adaptation, resource provisioning, SLA management

A grid computing environment enables sharing of loosely coupledresources and services required by various applications in a large-scale. In such an environment, one of the key challenges is to develop a flexible, scalable, and self-adaptive resource management system which would allow users to carry out their jobs by transparently accessingautonomous, distributed, and heterogeneous resources. In this paper, wediscuss the research issues and conceptual architectural design of such a dynamic resource management framework, which leverages the open-source Globus Toolkit and commercially available HP OpenViewConfiguration Management Solutions software (Radia). Our approachprovides adaptive and scalable middleware for static and dynamic resource provisioning, resource monitoring, virtual organization-wide authorization, and business policy management. The framework is basedon automated, policy-driven change and configuration management functionality that can dynamically adjust the size, configuration andallocation of various resources that will be consumed in the environment.

* Internal Accession Date Only Approved for External Publication © Copyright 2005 Hewlett-Packard Development Company, L.P.

A Framework for Dynamic Resource Management on the Grid

Jyotishman Pathak1,3,∗ Jem Treadwell3 Raj Kumar2 Philip Vitale3 Fernando Fraticelli31Artificial Intelligence Lab, Department of Computer Science, Iowa State University, Ames, IA 50011, USA2Hewlett-Packard Laboratories & Office of Corporate Strategy and Technology, Palo Alto, CA 94304, USA

3Hewlett-Packard Company, Mount Laurel, NJ 08054, USA{firstname.lastname}@hp.com

Abstract

A grid computing environment enables sharing ofloosely coupled resources and services required by var-ious applications in a large-scale. In such an envi-ronment, one of the key challenges is to develop aflexible, scalable, and self-adaptive resource manage-ment system which would allow users to carry outtheir jobs by transparently accessing autonomous, dis-tributed, and heterogeneous resources. In this pa-per, we discuss the research issues and conceptual ar-chitectural design of such a dynamic resource man-agement framework, which leverages the open-sourceGlobus Toolkit and commercially available HP Open-View Configuration Management Solutions software(Radia). Our approach provides adaptive and scal-able middleware for static and dynamic resource provi-sioning, resource monitoring, virtual organization-wideauthorization, and business policy management. Theframework is based on automated, policy-driven changeand configuration management functionality that candynamically adjust the size, configuration and alloca-tion of various resources that will be consumed in theenvironment.

Keywords: Grid resource management, Dynamicadaptation, Resource provisioning, SLA management.

1 Introduction

The development of grid computing technologies[20]over the past several years has provided us a meansof using and sharing heterogeneous resources over lo-cal/wide area networks, and geographically dispersedlocations. This has resulted in the ability to formloosely coupled, high-performance computational envi-ronment comprising numerous scalable, fault tolerant,and platform-independent services across the entire In-ternet. The grid infrastructure provides a way to ex-

∗This work was done while Jyotishman Pathak was an internat Hewlett-Packard. Additional email: jpathak@cs.iastate.edu

ecute applications over autonomous, distributed andheterogeneous nodes by secure resource sharing amongindividuals and institutions. Typically, a user can sub-mit jobs to a grid without necessarily knowing (or evencaring) where it will be executed. It is the responsibil-ity of the grid resource management system to distrib-ute such jobs among a heterogeneous pool of servers,trying to optimize the resource usage and provide thebest possible quality of service. However, the designand implementation of such systems where clients canseamlessly execute their applications pose several chal-lenges. Firstly, the grid is a dynamic framework whereresources are subjected to changes due to system per-formance degradation, node failure, allocation of newnodes in the infrastructure, etc. As a result, a gridresource management system should have the capa-bility to adapt to these changes and take appropriateactions to improve performance of various computingapplications. For example, if a node running an appli-cation crashes, then the resource management systemshould migrate the jobs running in that machine to analternate one. Secondly, typical grid resource require-ments for an application vary during the course of itsexecution. Similarly, user demand patterns may alsofluctuate, as a result of which, some resources maybe over-utilized while others under-utilized. In suchcases, it might be beneficial to provision additional re-sources (if all resources are over-utilized) or migratejobs from an under-utilized resource and de-allocate it,hence ensuring optimal resource usage1. At the sametime, the decisions for resource provisioning and shar-ing should be made while maintaining the autonomy oftheir environments and geographical locations. Thus,the resource management framework should providea highly scalable and configurable approach for pro-visioning and securely accessing the resources. Also,

1In some situations allowing jobs to execute in an under-utilized node (resulting in faster execution) might be optimal,as opposed to migrating them. Such decisions could be madeusing various optimization heuristics[35, 37] or policy-basedmodeling[31].

1

lately there has been interest in using grid computingfor e-Business and e-Commerce applications which de-mand stricter service guarantees. Typically, such guar-antees are specified as part of agreements, which needto be monitored and assured by the grid resource man-agement system.

Keeping these issues in mind, in this paper we inves-tigate a framework for dynamic grid resource manage-ment using the open-source Globus Toolkit[19]—a soft-ware toolkit for building grid infrastructure—and com-mercially available HP OpenView Configuration Man-agement Solutions software (Radia)[3]—an automated,policy-driven change and configuration managementinfrastructure for dynamic allocation of physical sys-tem and data resources. More specifically, we introducethe architecture of Globus-Radia Resource Man-agement System (GRRMS)—a framework for sta-tic and dynamic resource provisioning, resource moni-toring, virtual organization-wide resource sharing, SLAmanagement and resource brokerage. The main goalof this system is to provide seamless access to users forsubmitting jobs to a pool of heterogeneous resources,and at the same time, dynamically monitoring the re-source requirements for execution of applications andadapting to those needs in accordance with the chang-ing load characteristics of the underlying resources.The architecture has been designed to be a pluggablecore component for resource management processeswhich leverage various services provided by the existingtechnologies, namely the Globus Toolkit (hereafterGlobus) and HP OpenView Configuration Man-agement Solutions software (also known as Radia).GRRMS implements various WSRF-compliant[14] ser-vices to provide the dynamic resource managementfunctionalities required in a typical grid computing en-vironment.

The rest of the paper is structured as follows: inSection 2, we give a brief introduction to Globus andRadia software. Section 3 describes the architecture ofGRRMS in detail. Related work is presented in Section4, while Section 5 concludes with a summary of ourwork.

2 Existing Technologies

In what follows, we provide an overview about themain functionalities of Globus and Radia software.

2.1 Globus

The development of Globus began in the early 90’sto support the building of distributed computing appli-cations and infrastructures[19]. Globus has been used

to build computational Grids and Grid-based appli-cations by enabling sharing of computer power, datastorage, and various other facilities online, spanningthe Internet across institutional and geographic bound-aries in a secured manner. The latest release of thesoftware, Globus Toolkit version 4.0, provides signifi-cant Web services implementation advancements overits previous counterparts in terms of features, compli-ance to various emerging standards (e.g., WSRF[14]),and usability. The following is a brief description ofsome of its core components:

• Security: Globus security tools build on the Grid Se-curity Infrastructure (GSI)[21] for authenticating usersand services, secure communications, and authoriza-tion. These tools support functions such as managinguser credentials, delegation of credentials and main-taining group membership information. Examples ofthe tools include Community Authorization Service,Delegation Service etc.

• Execution Management: The execution manage-ment tools enable initiation, monitoring, management,scheduling, and/or coordination of remote computa-tions. Globus implements the Grid Resource Alloca-tion and Management (GRAM) service for providingthese functionalities. In order to address issues such asdata staging, delegation of proxy credentials, and jobmonitoring and management, the GRAM server is de-ployed along with Delegation and Reliable File Transfer(RFT) servers.

GRAM typically depends on a local mechanismfor starting and controlling the jobs. To achievethis, GRAM provides various interfaces/adapters tocommunicate with local resource schedulers (e.g.,Condor[10], PBS[8], LSF[7]) in their native messagingformats. The job details to GRAM are specified us-ing an XML-based job description language, known asResource Specification Language (RSL). RSL providessyntax consisting of attribute-value pairs for describingresources required for a job, including memory require-ments, number of CPU’s needed etc.

• Information Services: The information servicesprovide static and dynamic data about the various Gridresources. These services are mainly concerned withgathering, indexing, archiving, processing, and distrib-uting information about the configuration and stateof various resources and services over the Grid. TheGlobus component within this package is called Mon-itoring and Discovery Service (MDS). Globus uses itsWSRF core and WS-Notification[13] interfaces to sim-plify the tasks of registering information sources andlocating and accessing the required information. OtherGlobus components, such as GRAM and RFT, also de-

2

fine various resource properties, providing a basis formonitoring and discovery.

• Data Management: The data management pack-age provides many utilities and tools for transmitting,storing and managing large amounts of data requiredfor various Grid-based applications. The elements ofthis package include GridFTP service, Reliable FileTransfer (RFT) service, Replication Location Service(RLS) etc.

2.2 Radia

Radia[3] implements an automated, policy-drivenchange and configuration management system for dy-namically adjusting the size, configuration and alloca-tion of physical system and data resources. It providesa highly adaptable, flexible, and automated approachto resource provisioning for an infrastructure. Radiaadopts an object-oriented technology for transformingsoftware and content from file-based media into self-aware, platform-independent, intelligent objects thatcan automatically assess the environment into whichthey are deployed, and have the capability to install,update and repair themselves accordingly. This deploy-ment of resources is governed by a distribution model ordesired state which records the identities and intendedconfigurations of the various nodes (including deploy-ment destinations e.g., PDAs, desktops) that are partof the Radia-managed infrastructure. Figure 1 showsthe elements for such a distribution model, which wecategorize and explain in detail in the remainder of thissection.

The various components of the Radia DistributionModel can be divided into four main categories:

• Radia Management Applications: The RadiaManagement Applications are the client-side compo-nents that allow for automated deployment (of soft-ware), updates, repairs, and deletion activities. Theyalso have the ability to inspect the hardware and soft-ware configurations of the client machine.

There are three types of Radia Management Appli-cations: Radia Application Manager allows the admin-istrator to control the distribution of applications. Us-ing this software, the administrator can select, install,uninstall, and update a subscriber’s software and con-tent automatically in a seamless way. Radia SoftwareManager allows subscribers to install, delete, verify,and update their own elective software and content viaa user interface. However, the administrator still de-cides which software and content each subscriber uses.The Radia Inventory Manager Client gathers informa-tion about software and hardware configurations auto-

matically and stores them in a centralized server loca-tion. This information can be viewed as reports. Typi-cal information gathered might be the amount of RAMin a computer, hard disk capacity, processor type, ver-sions of OS running etc.

• Radia Management Infrastructure: This mod-ule is the heart of all Radia activities. It is usedto maintain the desired state or distribution model,store software and content packages, automate soft-ware management activities, and perform various ad-ministrative functions. The core of this infrastructureconsists of the Radia Configuration Server and RadiaAdministrator Workstation.

The Radia Configuration Server dynamically gener-ates the distribution model based on situation-specificdata, creating a software environment that automati-cally adapts to changes in a user or machine environ-ment. The server synchronizes distribution of objectssuch as application packages, computer configurations,and policy relationships across the network automat-ically. Typically, a policy defines which services sub-scribers, client computers or managed devices are en-titled to. These policies are stored in the Radia Con-figuration Server Database. It also stores the varioussoftware components (e.g., application packages) thatRadia distributes and the security and access rules forRadia administrators.

The Radia Administrator Workstation provides var-ious tools, namely Radia Publisher, Radia SystemExplorer, Radia Client Explorer and Radia ScreenPainter, for centralized control of Radia objects andentitlements. These tools carry out basic Radia func-tions such as manage Radia database, prepare appli-cations for deployment, view Radia client objects etc.The Radia Publisher provides an interface for packag-ing all software for deployment and post-deploymentmanagement. It determines what packages to installby scanning the destination node before and after in-stalling the software. The Radia System Explorer al-lows the administrator to view and configure policy andapplication services stored in the Radia database. Us-ing the explorer, the administrator can modify applica-tion packages after the initial publishing process or de-fine new application service prerequisites and policiesfor application entitlements. The administrator alsouses the Radia Client Explorer to manipulate (createnew, view/edit existing) objects, and the Radia ScreenPainter to create new customized dialog boxes.

• Radia Extended Infrastructure: The Radia Ex-tended Infrastructure provides a scalable solution tosoftware management services across an enterprise.Through distributed administrative capabilities, repli-

3

Figure 1. Elements in Radia Distribution Model[ 3]

cation services, and bandwidth conservation and me-tering, it gives an end-to-end management solution.This infrastructure comprises the following compo-nents:

Radia Distributed Configuration Server allows shar-ing of information about policies and management con-tent in an enterprise-wide network comprising morethan one Radia Configuration Server. It can automat-ically synchronize distributed Radia databases allow-ing managed applications and policy information to beshared across the enterprise.

The Radia Multicast Server enhances and simplifiesdata-transmission technology by reducing the numberof transmissions necessary, optimizing the use of net-work bandwidth. Such an approach transmits the samedata stream to many receivers simultaneously as op-posed to transmitting individually. However, the servermakes sure that only the participants eligible for re-ception receive the data and only the data requestedis transmitted. The information required to determinewhether a client is eligible is stored in the Radia data-base.

The Radia Integration Server integrates indepen-dent modules, giving them access to all the functionsand resources under its control. This server com-prises the following modules: Radia Management Por-tal, provides a web-based single access point from whichRadia administrators can deploy Radia client compo-nents, detect the current status of some Radia com-ponents, manage the Radia database, and track the

completion status of many tasks. It also exposes thefollowing Web services - GroupManagement service formanaging device groups, their attributes and member-ships, ServiceManagement service for managing vari-ous Radia services, their availability and deployment,PolicyManagement service for managing the definitionof policies in devices, groups etc., JobManagement ser-vice for managing jobs that are created and monitoredand EntityManagement service for managing entitiesthat can access the various Radia objects. The Ra-dia Inventory Manager Server stores the hardware andsoftware information collected from various machinesby the inventory manager client. This server enablescentralized reporting and administration based uponthe discovery results. The Radia Mobility Server actsas an interface between the mobile device and the Ra-dia Configuration Server for mobile devices distribu-tion activities. The Radia Proxy Server localizes digi-tal content enabling Radia-managed devices to receiveapplication data over LANs as opposed to WANs, as aresult reducing the overall network traffic. Such serversare beneficial in environments where many clients re-quest the same resource to be deployed from the samelocation. Placement of proxy servers at strategic loca-tions in a network improves its efficiency. Radia Stag-ing Server also helps in reducing the network traffic.Typically, when a client first connects to the RadiaConfiguration Server to retrieve an application pack-age, a copy of the software is sent from the Radia Con-figuration Server to the Radia Staging Server. The

4

next client that connects to the Configuration Server isre-directed (based on network locality-aware heuristicssuch as[36]) to obtain its application from the StagingServer. This also reduces the workload on the Config-uration Server.

• Radia Management Extensions: These exten-sions provide enterprise integration and extended func-tionality. The following are the various extensions:Radia Extensions for Windows Installer is a manage-ment system which gives complete control over resourcegathering and analysis of Windows installer packaging.It provides the ability to build, test, maintain, deployand troubleshoot applications and installation pack-ages. The Radia Policy Server is used for administra-tion purposes. It integrates with LDAP servers, NovellNDS etc. to enable single source points of control foruser authentication, access policies and subscriber en-titlement. The Radia Configuration Server can be con-figured to query the Policy Server to determine whatapplications and software components should be dis-tributed and managed for a particular subscriber. TheRadia Adapter for SSL utilizes the latest encryptionand security protocols for secured network and soft-ware distribution. The Radia Publishing Adapter is atool that allows automated, unattended updates to ap-plication packages. It identifies a set of files and compo-nents, and publishes them, in a controlled, automated,and repeatable manner, to the Radia database wherethey are stored as objects (to be deployed).

3 Globus-Radia Resource ManagementSystem

This section describes the conceptual architecturaldesign of Globus-Radia Resource Management System(GRRMS). It provides an automated, scalable, highlyconfigurable and adaptable solution to dynamic re-source management for a grid environment. One of thekey features of this framework is the ability to automat-ically provision (and de-provision) resources for execu-tion of applications based on the changing load charac-teristics of the environment, thereby ensuring optimalresource usage. The GRRMS architecture, shown inFigure 2, implements a set of WSRF-compliant ser-vices to provide various functionalities for authoriza-tion, resource discovery, job monitoring and manage-ment, SLA management, and resource provisioning aspart of the Globus-Radia Broker (GRB). In what fol-lows, we describe these components in detail.

3.1 Globus-Radia Broker

The Globus-Radia Broker (GRB) provides a gate-way to clients for interacting with GRRMS. The aimof this module is to efficiently use various componentsof GRRMS and control the whole process of job andresource management. Typically, a client interactswith the broker by providing its own proxy creden-tials, which are authenticated and authorized by theAuthorization Manager (Section 3.3). Once the clientgains access to GRRMS, it can perform various actions(e.g., submit a job request for execution, query for jobstatus, request for advance reservation of multiple re-sources). GRB provides generic interfaces for achievingthese functionalities by communicating with differentmodules within GRRMS.

Figure 2. GRRMS Architecture

3.2 Job Queue

This module provides a secure (GSI-enabled[21])JobQueue interface for GRRMS, using which clientscan submit job requests. The module validates eachincoming job description and puts the job in the job

5

queue. The job queue stores the jobs which areready for execution and implements First In FirstOut (FIFO) strategy, although in principle, it canimplement any alternative queue management algo-rithm (e.g., Shortest Job First, Negotiated Priorities).JobQueue also communicates with the JobManager(Section 3.5) via GRB, as a result of which clients canmonitor the status of their jobs (either by querying orsubscribing to notifications). The job descriptions inour framework are specified using Resource Specifica-tion Language (RSL), an XML-based job descriptionlanguage for Globus. However, the grid communityis currently working on an (abstract) Job SubmissionDescription Language[4] which is independent of lan-guage bindings and provides a rich set of features andattribute semantics. We are following these develop-ments and will adhere to the standards as and whenavailable.

3.3 Authorization Manager

Globus provides a secure framework for submittingand executing jobs and applications on various grid re-sources. It uses the GSI[21] infrastructure for authenti-cating users and services, secure communications andauthorization. This infrastructure is based on a sta-tic mapping from the user’s Distinguished Name (DN)to a local user-id using a grid-mapfile. While this ap-proach provides the benefits of secure single-signon au-thentication, for our purposes we need a frameworkwhich is more scalable (support for several thousandsof users), extensible (support for detailed authoriza-tion rules) and expressive (support for authorizationpolicies between Virtual Organizations and communi-ties).

GRRMS provides these features with theAuthorizationManager service. This service acts asa mediator between the clients and resource providersfor authentication and authorization. It provides aregistry (based on My-Proxy[19]) to which all the po-tential grid users and services/sites are registered. Thevarious grid resources and sites belonging to a partic-ular Virtual Organization (VO) or community registervia a local identity. Typically, the registry storesinformation about user profiles, proxy credentialsauthorized by a Certificate Authority (CA), and useraccess control policies. Such policies describe ‘who,what and when’ issues related to user access, essen-tially specifying resources, users, and when users havepermission to access the resources. These policies arerepresented using Extensible Access Control MarkupLanguage (XACML)[2]. The AuthorizationManagerservice implements an authorization server based on

Sun’s open-source XACML engine[9]. The servergrants or denies authorization to users based ontheir associated access policies (specified by theadministrator) and generates a temporary proxycredential for resource access, valid for a particulartime period. However, in a multi-VO configuration,it is possible that the requesting user does not haveaccess to the referenced VO. In such situations,the AuthorizationManager service maps the user’scredentials to a guest/template user credential toprovide authorization. These guest accounts are alsospecified as part of the XACML policy and, in general,have limited access and authorization capabilities.

3.4 Resource Manager

The Resource Manager is responsible for findingand monitoring appropriate resources for execution ofjobs and applications over the grid and provides aResourceManager interface. It relies on static anddynamic data provided by a set of information andmonitoring services. Static data is the informationthat does not vary during run-time (e.g., OS type),whereas dynamic data refers to variables affecting theactual resource availability (e.g., CPU usage). In ourcurrent framework, the ResourceManager communi-cates with Globus’s Monitoring and Discovery Service(MDS) and Radia’s Inventory Manager Server (IMS)for providing such information. Both these servicesprovide centralized access to the various hardware andrun-time configuration-related information about thegrid resources. Typically in a grid environment onecan envision multiple independent projects or applica-tion suites making use of many heterogeneous clustersand services. Similar to [32], we assume a hierarchicallystructured set of clustered resources, spanning multipleVOs, such that they deploy information providers (e.g.,Ganglia[24], Hawkeye[10]) for local/host-level monitor-ing and allow MDS & IMS access to scheduler andcluster information. The ResourceManager filters theresources by comparing the dynamic resource informa-tion received from MDS & IMS and static data with thejob requirements received from the client. Once a re-source (or a set of candidate resources) is selected, themanager interacts with the AuthorizationManager in-terface to determine if the requested user has accessand authorization to the resource.

However, in certain cases it might be that there areinsufficient resources available for execution of a par-ticular job (based on its requirements). In such situ-ations, the ResourceManager communicates with theProvisioning Manager (Section 3.7) via GRB, for pro-visioning of new resources.

6

3.5 Job Manager

The Job Manager is responsible for job submis-sion, monitoring, logging and rescheduling. The man-ager comprises several other subsystems which workcooperatively to provide these functionalities. It ex-poses a JobManager interface via which the Globus-Radia Broker (GRB) can interact. A particular jobexecution request from the client is forwarded to theJobExecution interface which provides the job launch-ing facility. It implements customized methods forsubmitting jobs to various local resource managers ofthe grid (e.g., GRAM[19], Condor-G[10]). These lo-cal resource managers also provide job status changeinformation, which can be either queried for or sub-scribed (for notifications[13]) via the interface. TheJob Manager module is also responsible for loggingjob status-related information in a database togetherwith some additional data (e.g., user id, job id, re-source requirement, submission time, total executiontime, wait time etc.). Access to such historical in-formation not only helps clients in decision-making,but also allows implementation of advanced (meta)scheduling algorithms (e.g., prediction based schedul-ing). In addition, such information can be used forimplementing a usage-based ‘chargeback model’ (e.g.,GridBank[11]). Finally, the manager module im-plements a ReScheduler interface for providing dy-namic adaptability to changes in the load character-istics of the resources on the grid. Specifically, it en-ables job/application migration between resources. Toachieve this, ReScheduler leverages the checkpoint-ing and job migration functionalities provided by thelocal resource management systems (e.g., Condor[10],LSF[7]). During the migration process, it communi-cates with the AuthorizationManager interface to en-sure that the user has access and authorization to usethe resource. ReScheduler implements three modesof migration: on-demand, timeserving, and failure-based2. During the on-demand mode, the client (viaGRB) may send a request to migrate a job fromone resource to another (e.g., because of pre-emptionby a higher-priority job). In the timeserving mode,the ReScheduler interacts with JobExecution andResourceManager (Section 3.4) interfaces to gatherreal-time data about job and available resource status.Based on this information, it may make a greedy choiceto migrate selected job(s) such that its total executiontime is reduced. Finally, in the case of failure-basedmigration, the ReScheduler migrates a job to an al-ternate resource due to a resource failure (e.g., server

2Failures solely correspond to the malfunctioning of resources,and not the various components of GRRMS.

crash). Note that in each of these three migratingmodes it might be possible that there are not adequateresources available to migrate a particular application.In such situations, the ReScheduler interacts with theProvisioning Manager (Section 3.7) to attempt to allo-cate additional resources.

3.6 SLA Manager

The increasing adoption of grid computing in thecommercial space has shifted its principal of operationof what used to be a ‘best effort approach’[30]. Withthis shifting focus, there is an emerging need to con-sider and incorporate agreements as part of the soft-ware building process. These agreements represent ex-pectations and responsibilities of different parties in-volved about the functionality and performance of sys-tems and services. They also help support automatedadaptation and management in the presence of mul-tiple interests[27]. Such contracts are called ServiceLevel Agreements (SLAs). SLAs encompass both thefunctional and non-functional requirements of the re-sources. In addition, they cover the cost of consuming aresource, as well as a penalty for breaking the contract.In our framework, these SLA-related issues are handledby the SLA Manager. It exposes an SLAManager inter-face by which the clients can interact (via GRB). Theclients formalize their requirements unambiguously us-ing SLAs, which are published and stored using theSLAPublishing interface. The SLANegotiator engineacts as a broker between such demands of the client andthe capabilities of the resources. This engine interactswith the ResourceManager interface (Section 3.4) todiscover resource(s) which meet the SLA requirements.It also maintains an ordered list of resources based ontheir (historical) performance rating. Upon successfulcompletion of a job, it requests client feedback (aboutthe resources utilized for that particular job), which isused for rating the performance of the resources. Sucha rating is used for assigning best of breed rankingsamong alternative resources for execution of a partic-ular application.

However, simply having an SLA-based negotiationmechanism is not sufficient to guarantee that the re-quirements will be fulfilled. To assure correct behav-ior of the system and ensure that SLAs are not vi-olated, we need to monitor the resources involved inorder to detect failures and, if needed, migrate jobsto other available resources. This is done by theSLAMonitoring module. The module communicateswith the ResourceManager and JobManager interfacesto determine the state of the jobs as well as the re-sources executing those jobs. Based on the status, it

7

identifies and takes action if there is any violation of thenegotiated SLA between the client and the provider.For example, if there is a hardware resource failure,then the SLAMonitoring engine activates the ‘failure-based’ re-scheduling mode of the ReScheduler for mi-grating jobs to alternate resources. At the same time,the engine penalizes the failed resource by reducing itsperformance rating. In addition, the SLA Managerprovides an SLAProvisioning interface using whichthe clients can determine appropriate resources (e.g.,based on functional/non-functional capabilities, perfor-mance ratings etc.) for executing their application andmake advance reservations (also called static provision-ing). This interface can also be used for dynamicallyprovisioning additional resources (based on the client’srequirements) during run-time, if such resources arenot available for migrating (and further executing) theclient application. Similarly, if a particular job has ex-ceeded its wait time (as specified in the SLA) in thejob queue, the monitoring engine might invoke ‘on-demand’ provisioning of new resources for executionof the job. Such active monitoring capabilities are vi-tal to ensure SLA compliance. To provide these func-tionalities, the interface interacts with the ProvisioningManager (Section 3.7).

The agreements in our framework are defined us-ing the WS-Agreement specification[12] of Global GridForum. This specification defines a protocol for estab-lishing agreement between two parties and provides anXML-based representation for specifying them.

3.7 Provisioning Manager

The Provisioning Manager is responsible for alloca-tion and deployment of resources needed to execute aclient application. It provides a ResourceProvision

interface via which the various modules of GRRMScan interact. In our framework, we use Radia[3] asa tool for provisioning of hardware and software re-sources. Radia provides a Web services-based API(called Radia Management Portal) for basic manage-ment of resources and their deployment. As mentionedin Section 2.2, the provisioning of resources in Radiais determined by a distribution model. These modelsare created dynamically by the Radia ConfigurationServer (based on situation-specific data) and are spec-ified (by the Radia administrator) using XML-based‘policies’, which broadly defines what application needsto be installed/deployed where, when and how. Theyalso specify software dependency, i.e., the deploymentrelationship with other software components, operat-ing systems and hardware. The policies are stored inthe Radia Database, and queried by the Configuration

Server during run-time. Similar policies can also bedefined for resource de-allocation, wherein applicationsare terminated and/or uninstalled.

Typically, the ResourceProvision interface re-ceives resource allocation requests from various com-ponents of GRRMS—namely Resource Manager, JobManager and SLA Manager. An example of such a re-quest could be ‘Allocate an HP-UX 11i node with 20GBof free hard disk space and 2Gigabit network connectiv-ity into a local Condor pool.’ Upon receiving such re-quests, the interface interacts with the Radia InventoryManager Server to determine an appropriate destina-tion node with appropriate hardware requirements (inthis case, 20GB free hard disk space and 2Gigabit net-work connection). Assuming that such a node is avail-able (and hence selected), ResourceProvision sub-mits an allocation request to the JobManagement Webservice of the Radia Management Portal (Section 2.2)by providing destination node details, and the action toperform (in this case, installing Condor[10] daemons,setting the appropriate CONDOR HOST, and startingthe CONDOR MASTER). The JobManagement servicesubmits this request to the Radia Configuration Server,which creates an appropriate distribution model for re-source allocation. The Configuration Server interactswith various other Radia components (e.g., Radia Pub-lisher for packaging all the software for deployment,Radia Policy Server for securing access to the nodeetc.) during this process. The JobManagement ser-vice monitors the progress of the actions and updatesthe ResourceProvision interface. On successful pro-visioning, the Radia Inventory Manager Server is up-dated, as a result of which the Resource Manager (Sec-tion 3.4) becomes aware of the newly provisioned re-source. In addition, the Provisioning Manager providesan interface for ActiveMonitoring. This service con-stantly communicates with the Resource Manager andJob Manager (Section 3.5) to gather up-to-date statusinformation about the various jobs that are running aswell as the health of the resources as a whole. Basedon this run-time status and requirements of the in-frastructure, the service determines whether to allocateor de-allocate additional resources. Such active moni-toring results in optimal resource usage. For example,if the infrastructure is over-subscribed (e.g., CPU uti-lization exceeds a certain threshold) and still more jobsare waiting in the job queue, then ActiveMonitoringwill direct the ResourceProvision service to allocatemore resources.

GRRMS provides two basic modes of provisioning:static and dynamic. During the static mode, clientscan request provisioning of resources before executingtheir application. To achieve this, ResourceProvision

8

provides various generic methods for creation, modi-fication, monitoring, and cancellation of a provision-ing request. In our framework, these methods im-plement the interfaces provided by the Maui ClusterScheduler[5], but in principle any reservation managerframework [34] can be adopted. However, if a partic-ular reservation cannot be fulfilled (e.g., due to non-availability of resources at the requested time), thenResourceProvision submits a request to Radia for al-location of additional resources based on the client’srequirements. On the other hand, in the dynamic pro-visioning mode, the client is not aware apriori of the re-source where its application will be executed. Instead,provisioning requests are coordinated and formulatedat run-time by the various components of GRRMS.

4 Related Work

Over the past several years, there has been a lotof work towards the development of grid middlewaretechnologies. A good survey of such approaches canbe found in [23]. Due to space limitation, we discussonly a few other projects (which were not introducedin [23]).

UNICORE[16] is a vertically integrated grid com-puting environment that facilitates seamless, secureand intuitive access to resources in a distributed envi-ronment. It adopts a 3-tier architecture and provides aclient-side graphical interface allowing them to createAbstract Job Object (AJO) represented as a serializedJava object. The UNICORE Network Job Supervisor(NJS) incarnates these AJOs into target system specificactions. NJS also manages the submitted jobs and per-forms user authorization (single sign-on through X.509certificates). The UNICORE Target System Interface(TSI) accepts incarnated job components from NJS,and passes them to the local batch schedulers for exe-cution.

Gridbus[11] is an open-source middleware toolkitfor computational grids. It provides a service-orientedcluster and grid middleware for e-Business and e-Science applications. Its design and development isbased around the notion of ‘utility computing’ and usesvarious economic models for efficient management andusage of shared resources. Gridbus provides an arrayof software for: Grid Economy and Scheduling (Econ-omy Grid), Data Grid Brokering and Scheduling (Grid-bus Broker), Cluster Economy and Scheduling (Libra),.NET-based Grid computing (Alchemi), Grid MarketDirectory and Service Publication (GMD), Grid Simu-lation (GridSim toolkit), Resource Usage and Account-ing (GridBank) and Grid Portals (GridScape, GMoni-tor).

Another on-going project which provides a suite ofsoftware for various grid middleware functionalities isGridLab[15]. The GridLab framework is based arounda Grid Application Toolkit (GAT), which implementsa set of high-level APIs, using which clients and appli-cations are able to call the underlying grid services.

NAREGI[25] aims to research and develop high-performance, scalable grid middleware for the Japanesenational scientific computational infrastructure. Theimplementation of the NAREGI framework is dividedinto six R&D groups, also called ‘Work Packages’(WPs): WP-1 focuses on resource management andmanaging information services; WP-2 works on basicparallel programming tools; WP-3 develops grid toolsfor end users; WP-4 deals with packaging and configu-ration management of the software products from theproject; WP-5 investigates issues related to network-ing, security, user management; and WP-6 is in chargeof grid-enabling various nanoscience applications.

The ASKALON[18] project provides a tool setfor service-based performance-oriented development ofgrid applications. The tools implemented can bebroadly categorized as: resource brokerage and moni-toring, scalable discovery and organization of resources,workflow-based dynamic and fault-tolerant executionof activities, meta scheduling, performance predictionfor estimation of job execution time, and performancemonitoring, instrumentation and analysis of grid appli-cations. The GLARE system[33], developed as part ofthe ASKALON framework, allows automatic deploy-ment and on-demand provisioning of components forgrid applications. In this work, the authors adopta peer-to-peer architecture for resource managementand consider ‘activities’ as generalized abstractions ofgrid tasks/jobs, which can be deployed during run-timeon different computers and executed in a coordinatedmanner to accomplish a particular application goal.

GrADS[35] (Grid Application Development Soft-ware) is a framework designed to solve numerical ap-plications over the grid. An important aspect of thisframework is the ability to decide when to migrate jobsbased on actual and predicted execution times. If thesystem detects minimal differences between the actualand predicted performance of the application, it avoidsjob migration. Such an approach makes those jobswhich are about to finish execution in a short periodof time less susceptible to suspension and migration.

Othman et al.[28] implemented an adaptive grid re-source broker using reflection technology to facilitateadaptation during run-time to variations in system re-sources, such as load or CPU utilization of a particularjob. They also implement prediction based modelingthat is used as a basis for a decision as to whether

9

job migration is required to satisfy a pre-specified timeconstraint.

SmartFrog[22] is a framework for service configu-ration, description and lifecycle management. Theframework adopts a component-based architecture andprovides a declarative language for describing serviceconfiguration and provisioning. This language is usedto design code templates which are executed by Smart-Frog engines running on remote nodes. The SmartFrogcomponent model enforces lifecycle management bytransitioning components through various stages (in-stalled, initiated, started, terminated, failed), which inturn allows the SmartFrog engines to redeploy compo-nents during a failure.

Apart from the above mentioned approaches, theGlobal Grid Forum CDDLM working group[1] is alsoworking towards development of specifications for de-scribing configuration of services, their deployment onthe grid, and management of their life cycle. In addi-tion, there is on-going work in the Open Grid ServicesArchitecture (OGSA) Basic Execution Service WorkingGroup[6] which focuses on many aspects of job execu-tion and management. We are closely following thesedevelopments and will adhere to the specifications infuture.

There has also been an increasing amount of workon grid resource management based on Service agree-ments. An overview of such approaches can be foundin[27]. Sahai et al.[29, 30] proposed an architecture forSLA monitoring to ensure stricter service guarantees.Their architecture provides a language for specifyingSLAs and relies on a network of communicating prox-ies, each maintaining SLAs committed within the ad-ministrative domain of the proxy. SLAs are either ne-gotiated between or specified to management proxies,which are responsible for automated monitoring andtriggering of appropriate actions for the agreements.The problem of resource allocation for SLA-constrainedgrid applications is discussed in [26]. In this system,an application is decomposed into simple tasks whichexhibit precedence relationships. Then, a heuristic isused to optimally allocate resources for running thetasks by minimizing total cost and preserving executiontime SLAs. Burchard et al.[17] introduced the VirtualResource Manager which builds on existing resourcemanager systems and provides various components forQoS management. The granularity for the deploymentof these components (decided by the local administra-tor) effects the overall QoS management offered by thesystem.

Our framework is also motivated by the approachesmentioned above. However, one of the key facetsof our approach is the ability to dynamically moni-

tor and adapt to the run-time resource requirementsof the various applications that are executing. TheResource Manager, Job Manager, and SLA Managerconstantly monitor the overall functioning of the in-frastructure to ensure optimal resource usage as wellas strict guarantee of service agreements. For exam-ple, if additional resources are needed for executionof a particular job, the Provisioning Manager givesGRRMS the ability to provision brand-new resourcesbased on such requirements. To the best of our knowl-edge, none of the proposed approaches deals with thisproblem of requirements-based resource provisioning.At the same time, GRRMS implements an XACML-policy based authorization framework. Such an ap-proach is not only scalable and flexible in a grid en-vironment, but in a way realizes the true meaning of‘Virtual Organizations’[20].

5 Conclusion

In this paper, we introduced the conceptual ar-chitectural design of Globus-Radia Resource Manage-ment System (GRRMS)—a modular and extensible ap-proach for dynamic resource allocation and manage-ment in a grid environment. The framework providesa pluggable component by leveraging a set of core ser-vices provided by two existing technologies: Globusand Radia. The novelty of this approach is the abilityto provision resources based on the job requirements.Such automated and highly adaptive systems will playan important role in realizing next-generation grid ap-plications. Apart from this, the framework also pro-vides a flexible approach for SLA-based service man-agement and an XACML-based VO-wide authoriza-tion mechanism, which are vital for realizing commer-cial grid applications. Our future work is targeted to-wards a prototype implementation of various compo-nents of GRRMS. We also plan to include additionalfeatures (e.g., prediction-based scheduling and migra-tion, chargeback model for utility computing applica-tions) into our framework.

Acknowledgment. The authors would like to thankGreg Astfalk from Hewlett-Packard for his encourage-ment and continued support in pursuing this research.

References

[1] Configuration, Deployment and Life CycleManagement of Grid Services,https://forge.gridforum.org/projects/cddlm-wg.

[2] Extensible Access Control Markup Language,http://www.oasis-open.org/committees/xacml.

10

[3] HP OpenView Configuration Management SolutionsSoftware,http://managementsoftware.hp.com/solutions/ascm.

[4] Job Submission Description Language,https://forge.gridforum.org/projects/jsdl-wg.

[5] Maui Cluster Scheduler,http://www.clusterresources.com/products/maui/.

[6] OGSA Basic Execution Service Working Group,https://forge.gridforum.org/projects/OGSA-BES-wg.

[7] Platform’s Load Sharing Facility,http://www.platform.com/products/LSF.

[8] Portable Batch Scheduler, http://www.openpbs.org.

[9] Sun’s XACML Implementation,http://sunxacml.sourceforge.net.

[10] The Condor Project,http://www.cs.wisc.edu/condor.

[11] The Gridbus Project, http://www.gridbus.org.

[12] Web Services Agreement,https://forge.gridforum.org/projects/graap-wg.

[13] Web Services Notification,http://ws.apache.org/pubscribe.

[14] Web Services Resource Framework,http://ws.apache.org/wsrf.

[15] G. Allen, K. Davis, and et al. Enabling Applicationson the Grid: A GridLab Overview. Intl. Journal ofHigh Performance Computing Applications,17(4):449–466, 2003.

[16] J. Almond and D. Snelling. UNICORE: UniformAccess to Supercomputing as an Element ofElectronic Commerce. Future Generation ComputerSystems, 15(5-6):539–548, 1999.

[17] L.-O. Burchard, B. Linnert, F. Heine, M. Hovestadt,O. Kao, and A. Keller. A Quality-of-ServiceArchitecture for Future Grid ComputingApplications. In IEEE Parallel and DistributedProcessing Symposium, 2005.

[18] T. Fahringer, A. Jugravu, S. Pllana, R. Prodan, C. S.Junior, and H.-L. Truong. ASKALON: A Tool Setfor Cluster and Grid Computing. Concurrency andComputation: Practice and Experience,17(2-4):143–169, 2005.

[19] I. Foster and C. Kesselman. Globus: AMetacomputing Infrastructure Toolkit. Intl. Journalof Supercomputer Applications and HighPerformance Computing, 11(2):115–128, 1997.

[20] I. Foster and C. Kesselman. The GRID: Blueprintfor a New Computing Infrastructure, 2nd Edition.Morgan-Kaufmann, San Mateo, CA, 2004.

[21] I. Foster, C. Kesselman, G. Tsudik, and S. Tuecke. ASecurity Architecture for Computational Grids. In5th ACM Conference on Computer andCommunications Security, 1998.

[22] P. Goldsack, J. Guijarro, A. Lain, G. Mecheneau,P. Murray, and P. Toft. SmartFrog: Configurationand Automatic Ignition of Distributed Applications.In HP OpenView University Association Workshop,Hewlett-Packard Labs, 2003.

[23] K. Krauter, R. Buyya, and M. Maheswaran. ATaxonomy and Survey of Grid Resource ManagementSystems for Distributed Computing. Software -Practice and Experience, 32(2):135–164, 2002.

[24] M. Massie, B. Chun, and D. Culler. The GangliaDistributed Monitoring System: Design,Implementation and Experience. Parallel Computing,30(7):817–840, 2004.

[25] S. Matsuoka, S. Shimojo, M. Aoyagi, S. Sekiguchi,H. Usami, and K. Miura. Japanese ComputationalGrid Research Project: NAREGI. Proceedings of theIEEE, 93(3):522–533, 2005.

[26] D. Menasce and E. Casalicchio. A Framework forResource Allocation in Grid Computing. In Intl.Symposium on Modeling, Analysis, and Simulation ofComputer and Telecommunications Systems, 2004.

[27] C. Molina-Jimenez, J. Pruyne, and A. van Moorsel.The Role of Agreements in IT Management Software.In Architecting Dependable Systems III, LNCS 3549,pages 36–58, 2005.

[28] A. Othman, P. Dew, K. Djemame, and I. Gourlay.Adaptive Grid Resource Brokering. In IEEE Intl.Conference on Cluster Computing, 2003.

[29] A. Sahai, A. Durante, and V. Machiraju. TowardsAutomated SLA Management for Web Services. InHP Labs Technical Report, HPL-2001-310R1, 2002.

[30] A. Sahai, S. Graupner, V. Machiraju, and A. vanMoorsel. Specifying and Monitoring Guarantees inCommercial Grids through SLA. In HP LabsTechnical Report, HPL-2002-324, 2002.

[31] C. A. Santos, A. Sahai, X. Zhu, D. Beyer,V. Machiraju, and S. Singhal. Policy-Based ResourceAssignment in Utility Computing Environments. InHP Labs Technical Report, HPL-2004-142, 2004.

[32] J. Schopf, M. D’Arcy, N. Miller, L. Pearlman,I. Foster, and C. Kesselman. Monitoring andDiscovery in a Web Services Framework: Funtionalityand Performance of the Globus Toolkit’s MDS4. InSubmitted to SuperComputing Conference, 2005.

[33] M. Siddiqui, A. Villazon, J. Hofer, and T. Fahringer.GLARE: A Grid Activity Registration, Deploymentand Provisioning Framework. In Intl. Conference onHigh Performance Computing and Communications,2005.

[34] W. Smith, I. Foster, and V. Taylor. Scheduling withAdvanced Reservations. In Intl. Parallel andDistributed Processing Symposium, 2000.

[35] S. Vadhiyar and J. Dongarra. Self Adaptivity in GridComputing. Concurrency and Computation: Practiceand Experience, 17(2):235–257, 2005.

[36] B. Zhao, A. Joseph, and J. Kubiatowicz.Locality-aware Mechanisms for Large-scale Networks.In Workshop on Future Directions in DistributedComputing, 2002.

[37] S. Zhuk, A. Chernykh, A. Avetisyan, S. Gaissaryan,N. Kuzjurin, and A. Pospelov. Comparison ofScheduling Heuristics for Grid Resource Broker. In3rd Intl. Conference on Parallel Computing Systems,2004.

11