A Framework for Divisible Load E-science Applications in ... ... Mohamed Moustafa Mohamed Ahmed Abouelela, candidate for the degree of Doctor of Philosophy in Electronic Systems Engineering,

• View
0

0

Embed Size (px)

Text of A Framework for Divisible Load E-science Applications in ... ... Mohamed Moustafa Mohamed Ahmed...

• A Framework for Divisible Load E-science Applications in Optical Grids

A thesis Submitted to the Faculty of Graduate Studies and Research

In Partial Fulfillment of the requirements For the degree of

Doctor of Philosophy in

Engineering University of Regina

By

Mohamed Moustafa Mohamed Ahmed Abouelela

• UNIVERSITY OF REGINA

FACULTY OF GRADUATE STUDIES AND RESEARCH

SUPERVISORY AND EXAMINING COMMITTEE

Mohamed Moustafa Mohamed Ahmed Abouelela, candidate for the degree of Doctor of Philosophy in Electronic Systems Engineering, has presented a thesis titled, A Framework for Divisible Load E-Science Applications for Optical Grids, in an oral examination held on April 18, 2013. The following committee members have found the thesis acceptable in form and content, and that the candidate demonstrated satisfactory knowledge of the subject material. External Examiner: *Dr. Diwakar Krishnamurthy, University of Calgary

Supervisor: Dr. Mohamed El-Darieby, Software Systems Engineering

Committee Member: Dr. Craig M. Gelowitz, Software Systems Engineering

Committee Member: Dr. Malek Mouhoub, Department of Computer Science

Committee Member: **Dr. Mehran Mehrandezh, Industrial Systems Engineering

Chair of Defense: Dr. Yuchao Zhu, Department of Political Science *teleconference **Not present at defense

• A Framework for Divisible Load E-science Applications in Optical Grids

Mohamed Abouelela

Submitted for the Degree of Doctor of Philosophy

May 2013

Abstract

E-science applications require discovering, collecting, transferring and processing large

volumes of scientific data. In divisible load e-science applications, data is generated

and stored in geographically distributed repositories (e.g., instruments, sensors, cam-

eras, satellites and storage facilities). The generated data can be divided into in-

dependent subsets to be analysed distributed at many computing locations. Such

applications usually require optical networking for fast and reliable data transfer. In

this thesis, we propose a framework for divisible load applications in optical grids.

Within this framework, schedulers are developed to co-schedule computational and

optical network resources. Moreover, processes to handle divisible load applications

in different architectures including centralized, hierarchical, peer-to-peer and super-

peer are proposed. Fault management techniques are introduced to handle network

faults by considering distinctive characteristics of e-science applications and optical

grids. This research conducts a comprehensive study of different algorithms, tech-

niques and processes that have been introduced within the framework. The results

provide guidelines to build more efficient, scalable and reliable optical grid systems.

The results can serve as a guide to the best choices in selecting the scheduling algo-

rithms, fault management techniques and architectures according to different network

and application parameters.

ii

• Acknowledgements

It gives me great pleasure in expressing my gratitude to all those people who have

supported me and had their contributions in making this thesis possible. First and

foremost, I must acknowledge and thank the Almighty God for blessing, protecting

and guiding me throughout this period. I could never have accomplished this without

the faith I have in the Almighty.

I wish to express my deepest and sincere gratitude and appreciation to my super-

visor, Prof. Mohamed El-Darieby, Associate Professor of Software Systems Engineer-

ing department, for his support, guidance and helpful suggestions. His guidance has

served me well and I owe him my heartfelt appreciation.

I am very grateful to Faculty of Graduate Studies and Research, University of

Regina for an FGSR Research Award and Graduate Scholarship.

My family has played always an important role in my studies, so I would like to

express my deep gratitude and appreciation to my parents, my wife and my sister for

their continuous support.

iii

1 Introduction 1

1.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

1.2 Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

1.3 Problem Statement . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

1.4 Assumptions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

1.5 Proposed Solution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

1.6 Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

2 Related Work 18

2.1 Optical Grid Projects . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

2.2 Inter Domain Operability in Optical Grids . . . . . . . . . . . . . . . 21

2.3 Grid Network Architecture . . . . . . . . . . . . . . . . . . . . . . . . 23

2.4 Multi-source Divisible Load Job Scheduling . . . . . . . . . . . . . . . 26

2.5 Network Recovery Mechanisms . . . . . . . . . . . . . . . . . . . . . 28

iv

• 2.5.1 Recovery Mechanisms in Optical Networks . . . . . . . . . . . 28

2.5.2 Recovery Mechanisms in Optical Grids . . . . . . . . . . . . . 29

3 Framework Overview and Methodology 31

3.1 High Level Framework . . . . . . . . . . . . . . . . . . . . . . . . . . 31

3.1.1 Optical Grid System Architecture . . . . . . . . . . . . . . . . 32

3.1.2 Core Components . . . . . . . . . . . . . . . . . . . . . . . . . 37

3.1.3 Multi-source Divisible Load Job Scheduling . . . . . . . . . . 39

3.1.4 Fault Management . . . . . . . . . . . . . . . . . . . . . . . . 41

3.2 Performance Evaluation Methodology . . . . . . . . . . . . . . . . . . 44

3.2.1 Input Parameters . . . . . . . . . . . . . . . . . . . . . . . . . 46

3.2.2 Performance Metrics . . . . . . . . . . . . . . . . . . . . . . . 48

4 Core Components 51

4.1 Resource Information Aggregation and Abstraction . . . . . . . . . . 51

4.1.1 Scalar Abstraction Techniques . . . . . . . . . . . . . . . . . . 52

4.1.2 Time Availability Aggregation Algorithms . . . . . . . . . . . 59

4.2 Load Balancing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68

4.2.1 DLA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69

4.2.2 NADLA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70

4.2.3 GA-LD . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72

v

• 4.3 Path Computing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74

4.4 Performance Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . 75

4.4.1 Performance of Aggregation and Abstraction Techniques . . . 75

4.4.2 Performance of Load Balancing Algorithms . . . . . . . . . . . 79

4.4.3 Performance of Path Computing Algorithms . . . . . . . . . . 80

5 Scheduling Multi-Source Divisible Load Applications 86

5.1 Problem Formulation . . . . . . . . . . . . . . . . . . . . . . . . . . . 87

5.2 GA Based Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . 91

5.2.1 GA Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91

5.2.2 Fitness Function . . . . . . . . . . . . . . . . . . . . . . . . . 91

5.2.3 Initialization . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93

5.3 DLT Based Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . 94

5.4 Iterative Scheduling Algorithm . . . . . . . . . . . . . . . . . . . . . . 96

5.5 Performance Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . 99

5.5.1 Performance of GA Based Algorithm . . . . . . . . . . . . . . 99

5.5.2 DLT Based Algorithm vs. Iterative Scheduling Algorithm . . . 104

6 Scheduling Divisible Loads in Different Architectures 112

6.1 Hierarchical Scheduling: Top Down Approach . . . . . . . . . . . . . 114

6.2 Hierarchical Scheduling: Bottom Up Approach . . . . . . . . . . . . . 116

vi

• 6.3 P2P Scheduling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120

6.3.1 P2P Resource Discovery . . . . . . . . . . . . . . . . . . . . . 120

6.3.2 P2P Resource Allocation . . . . . . . . . . . . . . . . . . . . . 123

6.4 Superpeer Scheduling . . . . . . . . . . . . . . . . . . . . . . . . . . . 127

6.5 Performance Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . 130

6.5.1 Comparison Between Different Architectures . . . . . . . . . . 130

6.5.2 Performance of P2P Scheduling Process . . . . . . . . . . . . . 137

6.5.3 Performance of Superpeer Scheduling Process . . . . . . . . . 158

6.5.4 Performance of Hierarchical Scheduling Process: Top Down Ap-

proach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164

7 Fault Management 172

7.1 Grid Based Network Restoration Techniques . . . . . . . . . . . . . . 173

7.1.1 Re-routing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 174

7.1.2 Relocation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175

7.1.3 Divisible Load Relocation . . . . . . . . . .

Documents
Documents
Science
Documents
Documents
Documents
Documents
Documents
Documents
Documents
Documents
Documents
Documents
Documents
Documents
Documents
Social Media
Documents
Documents
Documents
Documents
Documents
Documents
Documents
Documents
Documents
Documents
Documents
Divisible Profit
Economy & Finance
Documents