View
25
Download
0
Category
Preview:
DESCRIPTION
Resource Allocation for Distributed Streaming Applications. Qian Zhu and Gagan Agrawal Department of Computer Science and Engineering The Ohio State University. ICPP 2008 Conference. Sept. 10 th , 2008 Portland, Oregon. ICPP 2008. Data Streaming Applications. Computational Steering - PowerPoint PPT Presentation
Citation preview
Euro-Par, 2006 1
Resource Allocation for Distributed Streaming Applications
ICPP 2008 Conference
Sept. 10th, 2008 Portland, Oregon
Qian Zhu and Gagan Agrawal
Department of Computer Science and Engineering
The Ohio State University
ICPP 2008
Euro-Par, 2006 2
Data Streaming Applications
• Computational Steering– Interactively control scientific simulations
• Computer Vision Based Surveillance– Track people and monitor critical infrastructure– Images captured by multiple cameras
• Online Network Intrusion Detection– Analyze connection request logs – Identify unusual patterns
ICPP 2008
Euro-Par, 2006
Streaming Applications in Wide Area Environments
• Distributed high-volume data sources • Increasing WAN bandwidths
– Better than secondary storage bandwidths
• Geographically distributed users / consumers of data
• Exploit flexibility in resource usage in Grid Environments
3
Euro-Par, 2006
Our Previous Work
• A middleware system GATES – Grid-based AdapTive Executions on Streams
• Integration with Grid Standards• Support for self-adaptation• Dynamic allocation and fault-tolerance
4
Euro-Par, 2006 5
Resource Allocation in Streaming Grid Applications
• Challenges– Pipeline of processing stages
• Computation and communication requirements
– Long running nature• Dynamic grid resources
• Current Approach– Ad Hoc and Heuristics-based
– Not considering both bandwidth and computing power
ICPP 2008
Euro-Par, 2006
Overview of Our Research
• Static Resource Allocation– Subgraph isomorphism based
– Handle Network bandwidth and Computing power
– Effectiveness value
• Goal– To minimize the execution time of the data
streaming applications
ICPP 2008
Euro-Par, 2006 7
Outline
• Motivation and Introduction
• Resource Allocation in Data Stream Processing
• Resource Allocation Algorithm
• Experimental Evaluation
• Related Work
• Conclusion
ICPP 2008
Euro-Par, 2006
Data Stream Processing Model
• Directed Acyclic Graph (DAG) – Gp(Vp, Ep)
1800
2300
3200
41000
100
100
120
200
GP
sink
source
Processing nodes
Bandwidth
Requirement
Computing Power
Requirement
ICPP 2008
Euro-Par, 2006
Resource Model
• Directed Acyclic Graph (DAG) – GR(VR, ER)
A1000
B400
C200
D100
E2000 F
600G
800
120400
150 200
30
200
100
100
200
G2
Bandwidth
Computing power
ICPP 2008
Euro-Par, 2006
Problem Description
• To Allocate Resources to the Data Stream Application– A mapping from Gp(Vp, Ep) to GR(VR, ER)
• Modified Subgraph Isomorphism Based– To choose an isomorphic subgraph of GR
– Transporters
• Optimal Mapping– Effectiveness value
– To minimize the execution time
ICPP 2008
Euro-Par, 2006
Example
transporter
A1000
B400
C200
E2000
D100
ICPP 2008
Euro-Par, 2006
Effectiveness Value
• Bandwidth only
• Including Computing Power
A sigmoid function
Number of transporters
Overhead of adding transporters
Computing power match
ICPP 2008
Euro-Par, 2006 13
Outline
• Motivation and Introduction
• Resource Allocation in Data Stream Processing
• Resource Allocation Algorithm
• Experimental Evaluation
• Related Work
• Conclusion
ICPP 2008
Euro-Par, 2006
Proposed Algorithm
• Background: VF algorithm (L.P.Cordella et al.)
– State Space Representation (SSR)– Feasibility rules– Depth-First Search
• Pros and Cons– Efficient with small graphs (<200 nodes)– A large number of candidate partial mappings
ICPP 2008
Euro-Par, 2006
Proposed Algorithm – Step 1• Prune Candidate Partial Mappings
– Candidate node list– Reduce potential matches– Multiple Partial Mapping set
1 2 3 4
B
C
D
E
F
G
C
D
G
A
E
F
Cand(3)={C,D,G}
3200
A1000
B400
C200
E2000
D100
ICPP 2008
Euro-Par, 2006
Proposed Algorithm – Step 2
• Modified Subgraph Isomorphism Mapping– Transporters
A1000
B400
C200
E2000
D100
3200
1 2 3 4
B
C
D
E
F
G
C
D
G
A
E
F
Candidate pair: (3,C)
Candidate pair: (3,D)
transporter
ICPP 2008
Euro-Par, 2006
Handle Computing Power
• Computing Node Network Link– Computing power Network bandwidth
• Effectiveness Value Calculation • Possible Issues: high bandwidth and low
computing power– Map one node onto a cluster of network nodes
ICPP 2008
Euro-Par, 2006 18
Outline
• Motivation and Introduction
• Resource Allocation in Data Stream Processing
• Resource Allocation Algorithm
• Experimental Evaluation
• Related Work
• Conclusion
ICPP 2008
Euro-Par, 2006
Goals for the Experiments
• Demonstrate the Scalability of Our Resource Allocation Algorithm
• Demonstrate the High Performance of the Applications
ICPP 2008
Euro-Par, 2006
Experiment Setup
• Algorithms Compared– Optimal– Streamline
• Streaming Applications– Volume Rendering Application– A Synthetic Application
ICPP 2008
Euro-Par, 2006
Scalability of the Resource Allocation Algorithm
ICPP 2008
Euro-Par, 2006
Application Performance
• Volume Rendering
Within 4%
33%
29%
27%
ICPP 2008
Euro-Par, 2006
Application Performance
• A Synthetic Application
Within 3%
40%
36%
34%
ICPP 2008
Euro-Par, 2006 24
Outline
• Motivation and Introduction
• Resource Allocation in Data Stream Processing
• Resource Allocation Algorithm
• Experimental Evaluation
• Related Work
• Conclusion
ICPP 2008
Euro-Par, 2006
Related Work
• Resource Allocation for Stream Processing– Tang et al. (HPCC 06), Ali et al. (PDPTA 02)
• Resource Allocation for Grid Computing– Abdu et al. (IPDPS 01), Bhat et al. (Grid 07),
Hong et al. (ICPP 03)• Subgraph Isomorphism Algorithms and
Applications– Bioinformatics (Online Information 90), VLSI design
(ISCAS 95), Mobile robot design (JPR 95)
ICPP 2008
Euro-Par, 2006
Conclusion
• Modified Subgraph Isomorphism Algorithm for Resource Allocation in Grid Streaming Applications
• Handling Network Bandwidth and Computing Power
• Comparable Overhead with Streamline• Improved Application Performance
ICPP 2008
Euro-Par, 2006 27
Thank you!
ICPP 2008
Recommended