39

CHAPTER 4

PROPOSED GRID NETWORK MONITORING

ARCHITECTURE AND SYSTEM DESIGN

This chapter discusses about the proposed Grid network monitoring

architecture and details of the layered architecture. This chapter describes the

design of the proposed system, the design of an automated deployment of the

proposed system as a service in Grid, the network metrics used for

performance evaluation and Network Cost Function. This chapter also

describes about the computation of Resource Cost Value using resource

metric, and Network Cost Value using network metrics. The Network Aware

Resource Selection strategy is also explored in this chapter.

4.1 PROPOSED ARCHITECTURE

The complex system like grid, monitoring is essential for

understanding its operation, debugging, and failure detection and also for

performance optimization. Due to the heterogeneity and constantly varying

nature of grid, estimation of network performance is indispensable. If the

status of the network path can be predicted, it is possible to use that

information in grid applications. For example, the network status can be used

to adapt the traffic load in order to avoid congestion on a network path.

Scheduling of large data flows for data intensive applications is highly

dependent on network path characteristics. For computationally intensive

applications, resource broker or scheduler needs to have comprehensive and

40

accurate knowledge of network properties to fulfill service level agreements,

ensure QoS, and to make fit choices for advance reservation.

The four layered Grid network monitoring architecture is proposed

and modeled with the grid scheduler in the collective layer. The proposed

architecture is based on OGSA compliant layered architecture which is shown

in the Figure 4.1. In the proposed approach, the CARE Resource Broker

(CRB) is used for job submission (Thamarai Selvi et al 2009).

Figure 4.1 Grid Network Monitoring Architecture

Fabric Layer

The Grid fabric layer defines protocols for the publication,

discovery, negotiation, monitoring, accounting and payment of the operations

41

on individual resources. The resources may be computational resources,

storage systems, catalogues, network resources and sensors or may be a

logical entity, such as a distributed file system, computer cluster, or

distributed computer pool. The Grid Resource Access and Management

(GRAM) protocol is used for allocation of computational resources and for

monitoring and control of computation on those resources, and Grid File

Transfer.

Resource and connectivity layer

This layer consists of low-level middleware that provides secure

and unified access to remote resources. Depending on the type of resources,

different middleware can be chosen such as Globus, Unicore, Alchemi, and

Storage Resource Broker. Using services of such low-level middleware layer,

one can create high-level middleware services that support rapid creation and

deployment of applications on global Grids.

Collective Layer

The proposed architecture is modeled in this layer with a grid

scheduler. The Request handler which resides in CRB receives job requests

from the users. The controller is in CRB which controls the scheduling,

selection of the suitable resource for job submission from the matched

resource list, the monitoring the execution of jobs in Grid, and also maintains

the status of the submitted jobs. CRB selects the suitable resource using

resource metrics. The network monitoring is fit in this layer to retrieve the

network metrics which have influence on the resource selection.

More sensors are deployed in grid resources to provide more

network metrics so that the measurement of the network performance

becomes more reliable. The sensors are the network monitoring tools and

42

utilities which are started through migration of mobile agents from resource

broker to all resource sites when there is a need of unplanned monitoring in

the Grid environment. The planned network monitoring gathers the network

metrics from information repository, because the sensors are the network

monitoring tools which are initiated on all the Grid resources to retrieve the

network metrics periodically and update the information repository. The

network performance measurement and prediction utilizes the information

repository to measure and predict the performance of the Grid.

Network Monitor monitors the network and collects the network

metrics such as bandwidth, RTT, packet loss, jitter and stores it into

information repository using agent based information aggregator. The

Resource Monitor monitors the grid resources and collects the resource

metrics and stores the collected information to information repository with the

aid of agent based information aggregator. The agent based information

aggregator aggregates the resource and network information from the Grid

resources and periodically updates the information repository. It maintains the

information about every physical resources and its performance of the

network through end to end network monitoring across the grid infrastructure.

The data manager uses the network cost function to measure the network

performance which is described in the section 4.4.

CRB catalogs the matched resources depends on the job

requirements submitted by the user. Resource selector queries the information

repository to select the suitable resource with network aware resource

selection strategy and sending that information to the scheduler. The job

monitor maintains its current status of the job execution and reports the

progress to the user through Resource Broker. The network predictor predicts

the future network performance using History Based (HB) approach and

stores the predicted values into the information repository. It is also

43

responsible for sending the predicted value to the scheduler to take complex

decisions.

Application layer

The application layer facilitates the use of resources in a grid

environment through resource access protocols. The portal present at this

layer allows the grid user to submit resource requirements to find out suitable

resources for the execution of the submitted applications. It also includes

software and tools to support application workflow and composition.

4.2 DESIGN OF THE GRID NETWORK MONITORING SYSTEM

The proposed Grid Network Monitoring System design is based on

the architecture described above. The Figure 4.2 provides the view of the Grid

Network and Resource monitoring at resource level.

Figure 4.2 Grid Network and Resource Monitoring at Resource Level

44

Grid Network Monitor

The Grid network monitor initiates measurements or predictions on

demand. The sensors are deployed to provide more network metrics which

provides reliable network performance. The sensors are network monitoring

tools and utilities like UDPmon, TCPmon, IPerf and Ping. These sensors

provide network metrics like bandwidth, RTT, packet loss, and jitter which in

turn facilitates the network status monitoring.

Mobile Agent Generator

In Grid, whenever the user submits the job through the resource

broker, the mobile agent is created from mobile agent generator and it is

cloned and migrated to all grid resources and starts the sensors. The sensors

are the network monitoring tools which are used to retrieve the network

metrics between the end-to-end node in all grid resources.

Resource Monitor

The mobile agent migrates from the resource broker to all resources

and collects the resource metrics. The Resource Monitor monitors the resource

metric and sends the monitored information to data collector. The data

collector aggregates the collected information and periodically updates the

local archive maintained in every CE.

Data Accumulator

The Grid network monitor initiates data accumulator to collect the

metrics. In the Resource Broker and Grid Resources (i.e. head node), the

server for all the sensors running, from that the network metrics for the link

between the head to each compute node is sent to data accumulator. The

accumulator extracts the necessary data from multiple compute nodes and

45

stores it in global archive, also called as information repository present in the

head node. The resource metrics are collected from each node along with IP

address of the corresponding node and a time stamp to maintain validity for

the data. The agent performs this task for all computed nodes in the grid

cluster and updates the global archive.

Data Processor

The data processor process the network data collected through

deployed numerous sensors to measure the network performance using

network cost function. The network cost value and resource cost value is

calculated for all Grid Resources, through which the compound cost value is

calculated to help the scheduler for selecting the suitable resource for job

submission.

Global Archive

It contains aggregated information about network metrics and

resource metrics of all Grid Resource. The Global archive, also called as

Information Repository resides in the head node. This also stores the results of

the cost function computations which are used by the data processor to

measure the network performance and also for prediction.

Predictor

The predictor is linear and Historic-Based. This model uses

standard time series forecasting techniques to predict the performance based

on a history of measurements from previous behaviors on the same path. In

the proposed system, Holt-Winters (HW) model is used for predicting

network performance in the near future.

46

Visualizer

This part deals with displaying the monitored network

characteristics and the predicted network performance. The deliverables from

this component may be a graph or a chart providing a clear vision about the

network status of all grid resources.

Sensors

This component deals with the actual data collection in the Grid

Resources. The numerous sensors are deployed using network monitoring

tools and utilities like UDPMon, TCPMon, IPerf and Ping. These sensors

provide metrics like bandwidth, latency, packet loss rate, jitter, round trip

time (RTT) and one-way delay which in turn facilitates for network status

monitoring. The communication among the model components is depicted

in Figure 4.3.

Figure 4.3 Work Flow of the Proposed Monitoring System

47

4.3 NETWORK METRICS FOR PERFORMANCE

EVALUATION

Monitoring the Grid network performance requires the analysis of

various parameters like bandwidth, RTT, packet loss, jitter, etc which varies

frequently depending upon the real time network conditions across the links.

These parameters individually won’t determine the network performance

accurately. Thus combinations of the parameters are required.

4.3.1 Bandwidth

The maximum amount of data per time unit that a hop or path can

provide given the current utilization. The available bandwidth of a link relates

to the unused, or “spare”, capacity of the link during a certain time period. At

any specific instant in time, a link is either transmitting a packet at the full

link capacity or it is idle so the instantaneous utilization of a link can only be

either 0 or 1. Thus any meaningful definition of available bandwidth requires

time averaging of the instantaneous utilization over the time interval of

interest. The average utilization, u(T-t , T) for a time period (T-t, T) is given

by u(T-t , T) = (u(x) ) over the limit T-t and T. where u(x) is the

instantaneous available bandwidth of the link at time x. IPerf deals with TCP

bandwidth and UDP bandwidth.

Bandwidth = S / ( T – Latency) (4.1)

where, S is the message size, T is the message transfer time, and Latency is

measured from RTT.

4.3.2 RTT

Round Trip Time (RTT) is the time at which the last packet byte

departs from the source t(D), and the time at which the last packet byte arrives

48

at the packet destination t(A). The Ping utility finds an application to measure

this metric.

RTT = t (A) - t (D) (4.2)

4.3.3 Jitter

Jitter is the "instantaneous packet delay variation" (IPDV) and it

denotes the difference experienced by subsequent packets, I and I+1, on a

one-way transit from source to destination. Iperf is used to measure it.

4.3.4 Packet Loss

Packet Loss indicates the percentage of loss of data packets when

the packets are transmitted between the end hosts. Packet loss may take place

due to hardware fault, congestion in the channel, corruption in the data packet

sent. The data packets that are discarded by the routers when the load

becomes heavy also accounts for the packet loss percentage. Iperf and

UDPmon Tools are used to measure packet loss rate.

Packet loss percent = {1 - (Received Acks / Sent Packets)} * 100 (4.3)

4.4 NETWORK COST FUNCTION

The Measurement of the network characteristics like latency,

throughputs, packet loss rate, jitter, etc, is a repeated operation in any network

management system and also in Grid Environment. A single network

characteristic does not provide the significant information about the network

performance through network resources. So an aggregation of multiple

network metrics known as Network Cost Function (NCF) is needed to

measure the performance of the network. The metrics considered are

bandwidth, RTT, packet loss, and jitter between any nodes in the grid cluster.

49

The location of nodes could be inferred from the list of nodes available in

Grid.

The bandwidth measure is taken with respect to the average and the

maximum value. BWmax value is considered because of considering the

maximum possible bandwidth in the network channel. The variation of the

RTT values is very wide; hence there is a need of normalization. The half

normal form is used here, because the RTT values contain only the positive

values and this distribution is very specific version of normal distribution. The

packet loss, p and jitter are raised to the powers of the arbitrary values which

are decided based on the current Grid set up. The values of the arbitrary varies

[0, 1], hence the resulting values also reflect the values in [0, 1]. Packet loss

rate is significant in network cost functions because it provides an estimate of

both short and long-term congestion on a given data path due to packet drop

which depends on the performance of the transfer protocols. IPDV is an

important quality of service factor in assessment of network performance. If

there is no packet loss and jitter, the NCF influenced with bandwidth, and

RTT. The NCF (Network Cost Function) varies in the range [0, 1], where 0

indicates that a given node is not reachable and 1 denotes the maximum

degree of usage of the link if the network is congestion free.

Let BWmax denotes the maximum available bandwidth between

the corresponding pair of nodes.

Let BWavg denotes the mean available bandwidth between the

corresponding pair of nodes, i.e., BW = BW , where n is the

number of values taken for calculating the mean bandwidth for a period of

time.

50

Let RTTmax denotes the maximum RTT between the corresponding

pair of nodes, RTTavg denotes the mean RRT values measured by individual

probes between the corresponding pair of nodes, i.e., RTT = RTT ,

where k denotes the number of values taken for calculating the mean RTT for

the same period of time. And denotes how the RTT have influence on NCF.

= RTTavg RTTmax (4.4)

Let p denotes the packet loss, varies in interval [0, 1] and tuned to

balance the dependency of packet loss.

Let jitter denotes the measure of the variability over time of the

packet delay across a network, called as IPDV. varies in interval [0, 1] and

tuned to balance the dependency of jitter. The setting of , and are depend

on the Grid cluster which influences the maximum value.

The NCF of end-to-end node is measured by analysing the network

parameter values such as bandwidth, RTT, packet loss and jitter between that

pair of nodes. The NCF of the individual links are calculated using the

following expression.

NCF = × e × × (4.5)

4.5 RESOURCE COST VALUE

The computation of the Resource Cost Value (RCV) of the Grid

Site (GS) is based on the available Free Memory of the grid resources. The

average Free Memory, FreeMemavg is calculated for each Grid Resource in a

Grid environment, i.e., FreeMem = FreeMem , where, n is the

51

number of resources in a GS. The maximum available memory for a

Grid Site, i.e., FreeMemmax = max n (FreeMemk) is also identified to

evaluate the RCV, where n is the number of resources or Computing

Elements (CE) available in a Grid Site. The RCV is computed by the

following expression.

RCVGS=avg

max

FreeMem

FreeMem (4.6)

According to the Equation (4.6), the RCV of Grid resource, RCVGS

is varies in the range [0,1].

4.6 NETWORK AWARE RESOURCE SELECTION STRATEGY

The integration of network information with resource information

has very much influence in the decision making process of a Grid Resource

Broker. One of the major functions of the resource broker is to select the

suitable resource from the list of Grid resources which are geographically

distributed. The components of the network aware resource selection are

shown in Figure 4.4.

Figure 4.4 Network Aware Resource Selection Component

52

The primary selection is based on the requirements to be needed for

the execution of job. In the proposed approach, the CRB is used as a primary

selection to identify the matched grid resources for the specific job. The

primary selection rules are defined by the job when it is submitted from CRB,

which are the requirements needed to execute the job, such as software

needed to execute the job, CPU Speed, and Memory, etc. The primary

selection rules are used by the CRB broker to list the suitable resources to

execute the job which are available in Grid environment. The execution of the

job needs of transferring multiple input files and output data which lead to

produce the traffic across the Grid. The amount of traffic can be reduced

while selecting the node with better network connectivity. So there is a need

of measuring the performance of the network using network metrics such as

network cost value. The resource cost value is also computed by considering

the available free memory. The Resource Discovery uses the secondary

selection rule which is defined by the combination of network cost value and

resource cost value called as compound cost value to select a suitable node

from the CRB matched list of resources.

4.7 JOB MONITORING

The Job Monitor is responsible for maintaining the status of the

execution of the job to track its progress which is shown in the Figure 4.5.

Grid Resource Allocation Management (GRAM) is implemented as a Web

Services Resource Framework (WSRF) service in GT4 (Feller et al 2007).

GRAM provides an API that allows for submitting and canceling a

job request as well as checking the status of a submitted job. The job file is

written using JSDL. After the job file is given as the input, it is passed to

53

JobMonitor. The JobMonitor uses the security provided by the Grid Security

Infrastructure (GSI) of the Globus toolkit. WS-GRAM supports signature and

XML encryption. It uses digital certificates to send secure XML SOAP

messages between the Resource Broker and the Grid Resource.

Figure 4.5 Function of Job Monitor

Grid Resources have the local resource management system

(LRMS) which controls jobs running on CEs. It allocates CEs to jobs, starts

and stops jobs on user request and possibly restarts jobs if an error occurs.

The LRMS identi es the job it manages using local job identi er (LJID). The

jobmanager is the Globus GRAM which allows Grid users to start jobs on a

Grid resource.

Store

Credent ials

using RFT

Authent icate

Schedule

Job M onitor

Store

Report StatusReport StatusJob State

M onitor

Job Event

Daemon

Executes jobUser Job

Fork

Scheduler

Delegat ion

GRAM

Job Manager

User

Reports job status

Submits job file

54

The user submits a job manifest, a document which contains the job

description and the specification of the requested local resources to the

jobmanager. After successful authentication and authorization, the

jobmanager translates the job manifest into a form understood by the local

resource management system and starts the job under a local user account i.e,

user ID. A different user account is assigned to identify processes belonging

to a job is to start. Then the GRAM ensures that each job is started under a

user account that is distinct from accounts used by other presently running

Grid jobs.

First user delegation is performed, and then the job execution starts

at the CEs. Then the job status and progress is updated periodically at the

Resource Broker where the user submits the job. Once job execution

completes, the results are reported to the user along with any possible errors

like the job contact string does not match any which the job manager is

handling which are reported by GRAM. The JobMonitor also identifies the

current directory of the job, files in that directory, the permission set up of

those files and its resource consumption.

The Figure 4.6 depicts the process of job submission and

monitoring. The jobmonitor represents the Grid service, which allows Grid

users to start jobs on Grid resource.

Figure 4.6 Sequence diagram of job submission and monitoring process 55

56

4.8 NETWORK PERFORMANCE PREDICTION

There are three simple linear predictors namely Moving Average,

Exponential Weighted Moving Average (EWMA), and non-seasonal Holt-

Winters. These predictions have some order for prediction say ‘n’ based on

the number of previous values taken into consideration for the prediction. A

first order prediction is simpler but may not be accurate; higher the order then

the predicted value is more accurate. There are also more complex linear

predictors but selecting their order and linear coefficients requires a large

number of past measurements. So the simple predictor Holt-Winters (HW)

method is considered for predicting network performance rather than complex

ones.

4.8.1 History-Based Prediction

The History-Based (HB) prediction method is similar to traditional

time series forecasting, where past samples of an unknown random process

are used to predict the value of the process in the future.

4.8.1.1 Moving Average

Given a time series Y, the one-step n-order Moving Average (MA)

(n-MA) predictor is:

i+1 = (n)-1

{ Y(i-n+1)+Y(i-n+2)+..+Y(i-1)+Y(i) } (4.7)

where, i is the predicted value and Yi is the actual (observed) value at time i.

If n is too small, the predictor cannot smooth out the noise in the underlying

measurements. On the other hand, if n is too large the predictor cannot aptly

adapt to non-stationarities.

57

4.8.1.2 Exponentially Weighted Moving Average

The one-step Exponentially Weighted Moving Average (EWMA)

predictor is

X = X + ( )X (4.8)

where, is the weight of the last measurement (0< <1). Similar to the MA

predictor, a higher cannot smooth out the measurement noise, while a lower

is slow in adapting to changes in the time series.

4.8.1.3 Level Shifters and Outliers

While experimenting with various predictors, it was found that the

largest prediction errors are often caused by level shifts and outliers in the

observed time series. Furthermore, if there is a need of manage to avoid these

two characteristics in the time series forecasting, the exact choice of the

predictor, or of its parameters, does not make a significant difference.

A level shift is a type of non-stationarity, and it causes a significant

and typically sudden change in the mean of the observed time series. An

outlier is a measurement that is significantly different, beyond the typical

level of statistical variations, relative to nearby measurements. One way to

deal with level shifts, after they are detected, is to restart the predictor,

ignoring all previous history. Outliers, on the other hand, can be just ignored.

4.9 ARCHITECTURE OF THE AUTOMATED DEPLOYMENT

OF NETWORK AWARE RESOURCE MONITORING

SERVICE

The proposed architecture for an agent based Automated

Deployment of Network aware Resource Monitoring service is shown in the

Figure 4.7. The proposed mobile agent based automated deployment avoids

58

the maintenance costs and human errors occurring during deployment. Since

mobile agents are capable of operating even without active connections

between nodes, they are not affected by network failures. Further mobile

agents reduces network load. Mobile agent technology is very much flexible

to support the rum-time mobility through push and pull interaction models.

And the characteristics persistence, cloning and migration of the mobile

agents improve the reliability through replication.

Figure 4.7 Automated Deployment of Network aware Resource

Monitoring service

A mobile agent is composed of code and data which migrates to other

nodes and executes there in the node to which it migrates. The mobile agents

exploit the basic communication protocols defined within IBM Aglets

Workbench (Aglets 2004) for agent migration and to dispatch messages from

one node to another node. Deployment Agent is a mobile agent which contains

code for deployment of the services. It resides in the Resource Broker. On

request from a newly arrived Grid resource, it migrates to the Grid resource and

executes. The node where the Resource Broker is running is act as a Registration

Node which maintains a database of the Grid resources that arrive and also it

sends the IP address of the Resource Broker to the newly arrived resource.

The Resource Broker contains the services to be deployed and the

deployment agent. The registration node maintains a repository of IP

59

addresses of the resources in which the monitoring service is deployed and

also the IP address of the Resource Broker. Mobile agents are used to get the

IP address of the Resource Broker from the registration node and for

deploying the services. For an automated deployment of service, the mobile

agent migrates from the new resource to the registration node, collects the IP

address of the Resource Broker and migrates back. Using the IP address of

the Resource Broker, the newly arrived resource requests the Resource Broker

for deployment of the services. Then the deployment agent migrates from the

Resource Broker containing the services to the newly arrived resource and

executes thereby deploying the services.

4.10 SUMMARY

The monitoring of Grid resources is a momentous task because of the

diversity of the computing resources and applications in Grid environments. The

existing resource brokers are not considering the factors such as location of data,

bandwidth availability and data transfer time while scheduling data-intensive

applications on Grid Resources. This chapter presented a four layered

architecture for Grid network monitoring system which is modeled with Grid

scheduler. The Grid network performance is measured using Network Cost

Function by analyzing the network metrics such as bandwidth, RTT, packet loss

and jitter between the pair of nodes in Grid. The resource cost value is computed

using resource metrics and the network cost value is computed using network

metrics. The proposed system is integrated with CARE Resource Broker (CRB)

which is used for job submission. The Network Aware Resource Selection

Strategy is proposed for resource selection by computing the compound cost

value using network cost value and resource cost value for the selection of a

suitable node from the CRB matched list of resources. Once the job execution is

completed, the results or errors are reported to the user which is provided by

GRAM (Grid Resource Allocation and Management). The agent based

automated deployment of Network aware Resource Monitoring service also

explored in this chapter.