Click here to load reader

A Framework for Divisible Load E-science Applications in ... ... Mohamed Moustafa Mohamed Ahmed Abouelela, candidate for the degree of Doctor of Philosophy in Electronic Systems Engineering,

  • View
    0

  • Download
    0

Embed Size (px)

Text of A Framework for Divisible Load E-science Applications in ... ... Mohamed Moustafa Mohamed Ahmed...

  • A Framework for Divisible Load E-science Applications in Optical Grids

    A thesis Submitted to the Faculty of Graduate Studies and Research

    In Partial Fulfillment of the requirements For the degree of

    Doctor of Philosophy in

    Engineering University of Regina

    By

    Mohamed Moustafa Mohamed Ahmed Abouelela

    Regina, Saskatchewan May, 2013

    c©Copyright 2013: M. Abouelela

  • UNIVERSITY OF REGINA

    FACULTY OF GRADUATE STUDIES AND RESEARCH

    SUPERVISORY AND EXAMINING COMMITTEE

    Mohamed Moustafa Mohamed Ahmed Abouelela, candidate for the degree of Doctor of Philosophy in Electronic Systems Engineering, has presented a thesis titled, A Framework for Divisible Load E-Science Applications for Optical Grids, in an oral examination held on April 18, 2013. The following committee members have found the thesis acceptable in form and content, and that the candidate demonstrated satisfactory knowledge of the subject material. External Examiner: *Dr. Diwakar Krishnamurthy, University of Calgary

    Supervisor: Dr. Mohamed El-Darieby, Software Systems Engineering

    Committee Member: Dr. Craig M. Gelowitz, Software Systems Engineering

    Committee Member: Dr. Malek Mouhoub, Department of Computer Science

    Committee Member: **Dr. Mehran Mehrandezh, Industrial Systems Engineering

    Chair of Defense: Dr. Yuchao Zhu, Department of Political Science *teleconference **Not present at defense

  • A Framework for Divisible Load E-science Applications in Optical Grids

    Mohamed Abouelela

    Submitted for the Degree of Doctor of Philosophy

    May 2013

    Abstract

    E-science applications require discovering, collecting, transferring and processing large

    volumes of scientific data. In divisible load e-science applications, data is generated

    and stored in geographically distributed repositories (e.g., instruments, sensors, cam-

    eras, satellites and storage facilities). The generated data can be divided into in-

    dependent subsets to be analysed distributed at many computing locations. Such

    applications usually require optical networking for fast and reliable data transfer. In

    this thesis, we propose a framework for divisible load applications in optical grids.

    Within this framework, schedulers are developed to co-schedule computational and

    optical network resources. Moreover, processes to handle divisible load applications

    in different architectures including centralized, hierarchical, peer-to-peer and super-

    peer are proposed. Fault management techniques are introduced to handle network

    faults by considering distinctive characteristics of e-science applications and optical

    grids. This research conducts a comprehensive study of different algorithms, tech-

    niques and processes that have been introduced within the framework. The results

    provide guidelines to build more efficient, scalable and reliable optical grid systems.

    The results can serve as a guide to the best choices in selecting the scheduling algo-

    rithms, fault management techniques and architectures according to different network

    and application parameters.

    ii

  • Acknowledgements

    It gives me great pleasure in expressing my gratitude to all those people who have

    supported me and had their contributions in making this thesis possible. First and

    foremost, I must acknowledge and thank the Almighty God for blessing, protecting

    and guiding me throughout this period. I could never have accomplished this without

    the faith I have in the Almighty.

    I wish to express my deepest and sincere gratitude and appreciation to my super-

    visor, Prof. Mohamed El-Darieby, Associate Professor of Software Systems Engineer-

    ing department, for his support, guidance and helpful suggestions. His guidance has

    served me well and I owe him my heartfelt appreciation.

    I am very grateful to Faculty of Graduate Studies and Research, University of

    Regina for an FGSR Research Award and Graduate Scholarship.

    My family has played always an important role in my studies, so I would like to

    express my deep gratitude and appreciation to my parents, my wife and my sister for

    their continuous support.

    iii

  • Table of Contents

    1 Introduction 1

    1.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

    1.2 Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

    1.3 Problem Statement . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

    1.4 Assumptions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

    1.5 Proposed Solution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

    1.6 Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

    2 Related Work 18

    2.1 Optical Grid Projects . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

    2.2 Inter Domain Operability in Optical Grids . . . . . . . . . . . . . . . 21

    2.3 Grid Network Architecture . . . . . . . . . . . . . . . . . . . . . . . . 23

    2.4 Multi-source Divisible Load Job Scheduling . . . . . . . . . . . . . . . 26

    2.5 Network Recovery Mechanisms . . . . . . . . . . . . . . . . . . . . . 28

    iv

  • 2.5.1 Recovery Mechanisms in Optical Networks . . . . . . . . . . . 28

    2.5.2 Recovery Mechanisms in Optical Grids . . . . . . . . . . . . . 29

    3 Framework Overview and Methodology 31

    3.1 High Level Framework . . . . . . . . . . . . . . . . . . . . . . . . . . 31

    3.1.1 Optical Grid System Architecture . . . . . . . . . . . . . . . . 32

    3.1.2 Core Components . . . . . . . . . . . . . . . . . . . . . . . . . 37

    3.1.3 Multi-source Divisible Load Job Scheduling . . . . . . . . . . 39

    3.1.4 Fault Management . . . . . . . . . . . . . . . . . . . . . . . . 41

    3.2 Performance Evaluation Methodology . . . . . . . . . . . . . . . . . . 44

    3.2.1 Input Parameters . . . . . . . . . . . . . . . . . . . . . . . . . 46

    3.2.2 Performance Metrics . . . . . . . . . . . . . . . . . . . . . . . 48

    4 Core Components 51

    4.1 Resource Information Aggregation and Abstraction . . . . . . . . . . 51

    4.1.1 Scalar Abstraction Techniques . . . . . . . . . . . . . . . . . . 52

    4.1.2 Time Availability Aggregation Algorithms . . . . . . . . . . . 59

    4.2 Load Balancing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68

    4.2.1 DLA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69

    4.2.2 NADLA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70

    4.2.3 GA-LD . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72

    v

  • 4.3 Path Computing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74

    4.4 Performance Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . 75

    4.4.1 Performance of Aggregation and Abstraction Techniques . . . 75

    4.4.2 Performance of Load Balancing Algorithms . . . . . . . . . . . 79

    4.4.3 Performance of Path Computing Algorithms . . . . . . . . . . 80

    5 Scheduling Multi-Source Divisible Load Applications 86

    5.1 Problem Formulation . . . . . . . . . . . . . . . . . . . . . . . . . . . 87

    5.2 GA Based Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . 91

    5.2.1 GA Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91

    5.2.2 Fitness Function . . . . . . . . . . . . . . . . . . . . . . . . . 91

    5.2.3 Initialization . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93

    5.3 DLT Based Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . 94

    5.4 Iterative Scheduling Algorithm . . . . . . . . . . . . . . . . . . . . . . 96

    5.5 Performance Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . 99

    5.5.1 Performance of GA Based Algorithm . . . . . . . . . . . . . . 99

    5.5.2 DLT Based Algorithm vs. Iterative Scheduling Algorithm . . . 104

    6 Scheduling Divisible Loads in Different Architectures 112

    6.1 Hierarchical Scheduling: Top Down Approach . . . . . . . . . . . . . 114

    6.2 Hierarchical Scheduling: Bottom Up Approach . . . . . . . . . . . . . 116

    vi

  • 6.3 P2P Scheduling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120

    6.3.1 P2P Resource Discovery . . . . . . . . . . . . . . . . . . . . . 120

    6.3.2 P2P Resource Allocation . . . . . . . . . . . . . . . . . . . . . 123

    6.4 Superpeer Scheduling . . . . . . . . . . . . . . . . . . . . . . . . . . . 127

    6.5 Performance Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . 130

    6.5.1 Comparison Between Different Architectures . . . . . . . . . . 130

    6.5.2 Performance of P2P Scheduling Process . . . . . . . . . . . . . 137

    6.5.3 Performance of Superpeer Scheduling Process . . . . . . . . . 158

    6.5.4 Performance of Hierarchical Scheduling Process: Top Down Ap-

    proach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164

    7 Fault Management 172

    7.1 Grid Based Network Restoration Techniques . . . . . . . . . . . . . . 173

    7.1.1 Re-routing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 174

    7.1.2 Relocation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175

    7.1.3 Divisible Load Relocation . . . . . . . . . .