52
Powerpoint Templates 1 Powerpoint Templates An Adaptive Distributed Simulator for An Adaptive Distributed Simulator for Cloud and MapReduce Cloud and MapReduce Algorithms and Architectures Algorithms and Architectures Pradeeban Kathiravelu Luis Veiga INESC-ID Lisboa Instituto Superior Técnico, Universidade de Lisboa IEEE/ACM 7th International Conference on Utility and Cloud Computing – UCC 2014. Dec 8 th – 11 th , 2014.

An adaptive distributed simulator for cloud andmap reduce algorithms and architectures

Embed Size (px)

DESCRIPTION

The presentation I did, when presenting my work at UCC 2014 in London on the 8th of December, 2014. http://kkpradeeban.blogspot.com/2014/09/ucc-2014-adaptive-distributed-simulator.html

Citation preview

Page 1: An adaptive distributed simulator for cloud andmap reduce algorithms and architectures

Powerpoint Templates 1Powerpoint Templates

An Adaptive Distributed Simulator for An Adaptive Distributed Simulator for Cloud and MapReduce Cloud and MapReduce Algorithms and ArchitecturesAlgorithms and Architectures

Pradeeban KathiraveluLuis VeigaINESC-ID Lisboa Instituto Superior Técnico, Universidade de Lisboa

IEEE/ACM 7th International Conference on Utility and Cloud Computing – UCC 2014. Dec 8th – 11th, 2014.

Page 2: An adaptive distributed simulator for cloud andmap reduce algorithms and architectures

Powerpoint Templates 2

Agenda

•Introduction•Background•Solution Architecture•Implementation•Evaluation•Conclusion

Page 3: An adaptive distributed simulator for cloud andmap reduce algorithms and architectures

Powerpoint Templates 3

Introduction•Computing systems becoming

increasingly larger. •Simulations empower researches.•Cloud simulators are mostly

sequential and executed from a single computer.–CloudSim (Calheiros et al. 2009; Buyya et al. 2009; Calheiros et al. 2011)

–SimGrid (Casanova 2001; Legrand et al. 2003; Casanova et al. 2008)

–GreenCloud (Kliazovich et al. 2012)

Page 4: An adaptive distributed simulator for cloud andmap reduce algorithms and architectures

Powerpoint Templates 4

Motivation

•Large and complex simulations.•Distributed Execution Frameworks.– Illusion of a single large system.

•Clusters in the research labs.

•What if..?

Page 5: An adaptive distributed simulator for cloud andmap reduce algorithms and architectures

Powerpoint Templates 5

Goals

•An adaptive distributed cloud and MapReduce simulator.

•Extending CloudSim Cloud Simulator –Leveraging in-memory data grids.•Hazelcast (Johns 2013)

• Infinispan (Marchioni 2012)

• ...

Page 6: An adaptive distributed simulator for cloud andmap reduce algorithms and architectures

Powerpoint Templates 6

Contributions•An adaptive distributed architecture– for cloud and MapReduce simulations.

•A generic adaptive scaling algorithm.•A scalable middleware platform–elastic–multi-tenanted

•Evaluation of MapReduce implementations.–Hazelcast vs Infinispan.

Page 7: An adaptive distributed simulator for cloud andmap reduce algorithms and architectures

Powerpoint Templates 7

Major Features of the Work•Simulations → Actual Technology.•Loosely coupled.•Fault-Tolerant.• Internal cycle-sharing.•Deployable over real clouds.

Page 8: An adaptive distributed simulator for cloud andmap reduce algorithms and architectures

Powerpoint Templates 8

Cloud2Sim

Page 9: An adaptive distributed simulator for cloud andmap reduce algorithms and architectures

Powerpoint Templates 9

Design and DeploymentStorage, Execution, and Data Locality

• Simulator–Initiator based Approach

• Simulator–SimulatorSub based Approach

• Multiple Simulator Instances Approach

Page 10: An adaptive distributed simulator for cloud andmap reduce algorithms and architectures

Powerpoint Templates 10

Cloud2SimExecution

Flow

Page 11: An adaptive distributed simulator for cloud andmap reduce algorithms and architectures

Powerpoint Templates 11

1. Objects Initialization & Scheduling

Page 12: An adaptive distributed simulator for cloud andmap reduce algorithms and architectures

Powerpoint Templates 12

2. Final Execution

Page 13: An adaptive distributed simulator for cloud andmap reduce algorithms and architectures

Powerpoint Templates 13

Cloud2SimExecution

Flow

Page 14: An adaptive distributed simulator for cloud andmap reduce algorithms and architectures

Powerpoint Templates 14

Cloud2SimSoftware

Architecture

Page 15: An adaptive distributed simulator for cloud andmap reduce algorithms and architectures

Powerpoint Templates 15

Algorithms:Dynamic Scaling and Elasticity

Page 16: An adaptive distributed simulator for cloud andmap reduce algorithms and architectures

Powerpoint Templates 16

Algorithms:Dynamic Scaling and Elasticity

•Auto Scaling•Adaptive Scaling

Page 17: An adaptive distributed simulator for cloud andmap reduce algorithms and architectures

Powerpoint Templates 17

Auto Scaling

Page 18: An adaptive distributed simulator for cloud andmap reduce algorithms and architectures

Powerpoint Templates 18

Adaptive Scaling

Page 19: An adaptive distributed simulator for cloud andmap reduce algorithms and architectures

Powerpoint Templates 19

IntelligentAdaptiveScaler

Page 20: An adaptive distributed simulator for cloud andmap reduce algorithms and architectures

Powerpoint Templates 20

Subscribing for Scaling

Page 21: An adaptive distributed simulator for cloud andmap reduce algorithms and architectures

Powerpoint Templates 21

High Load

Page 22: An adaptive distributed simulator for cloud andmap reduce algorithms and architectures

Powerpoint Templates 22

Updating the flag

Page 23: An adaptive distributed simulator for cloud andmap reduce algorithms and architectures

Powerpoint Templates 23

Open Access

Page 24: An adaptive distributed simulator for cloud andmap reduce algorithms and architectures

Powerpoint Templates 24

Scaling Out

Page 25: An adaptive distributed simulator for cloud andmap reduce algorithms and architectures

Powerpoint Templates 25

Spawning an Initiator Instance

Page 26: An adaptive distributed simulator for cloud andmap reduce algorithms and architectures

Powerpoint Templates 26

Waiting Period..

Page 27: An adaptive distributed simulator for cloud andmap reduce algorithms and architectures

Powerpoint Templates 27

Waiting Period..

Page 28: An adaptive distributed simulator for cloud andmap reduce algorithms and architectures

Powerpoint Templates 28

Monitor for Scale Ins Too..

Page 29: An adaptive distributed simulator for cloud andmap reduce algorithms and architectures

Powerpoint Templates 29

After some time..

Page 30: An adaptive distributed simulator for cloud andmap reduce algorithms and architectures

Powerpoint Templates 30

Scale Out Again..

Page 31: An adaptive distributed simulator for cloud andmap reduce algorithms and architectures

Powerpoint Templates 31

One more Initiator..

Page 32: An adaptive distributed simulator for cloud andmap reduce algorithms and architectures

Powerpoint Templates 32

After more scalings..

Page 33: An adaptive distributed simulator for cloud andmap reduce algorithms and architectures

Powerpoint Templates 33

Scale In..

Page 34: An adaptive distributed simulator for cloud andmap reduce algorithms and architectures

Powerpoint Templates 34

Shut down an Initiator Instance

Page 35: An adaptive distributed simulator for cloud andmap reduce algorithms and architectures

Powerpoint Templates 35

Finally..

Page 36: An adaptive distributed simulator for cloud andmap reduce algorithms and architectures

Powerpoint Templates 36

Parallel Simulations

Page 37: An adaptive distributed simulator for cloud andmap reduce algorithms and architectures

Powerpoint Templates 37

Multi-tenanted Deployments

Page 38: An adaptive distributed simulator for cloud andmap reduce algorithms and architectures

Powerpoint Templates 38

MapReduceExecutions

Page 39: An adaptive distributed simulator for cloud andmap reduce algorithms and architectures

Powerpoint Templates 39

Implementation

•CloudSim trunk forked•Hazelcast version 3.2 and Infinispan

version 6.0.2.•Dependencies abstracted away.

Page 40: An adaptive distributed simulator for cloud andmap reduce algorithms and architectures

Powerpoint Templates 40

Evaluation

•Setup: Cluster with 6 identical nodes–Intel® Core™ i7-2600K CPU @ 3.40GHz and 12 GB memory.

•Varying number of parameters–Cloudlets: 100 → 400. –VMs: 100 → 200.–Nodes: 1 → 6.

Page 41: An adaptive distributed simulator for cloud andmap reduce algorithms and architectures

Powerpoint Templates 41

Simulation 1: CloudSim and Cloud2Sim

•Round robin application scheduling with 200 VMs and 400 cloudlets.

Execution Time

Page 42: An adaptive distributed simulator for cloud andmap reduce algorithms and architectures

Powerpoint Templates 42

Varying number of Cloudlets

Page 43: An adaptive distributed simulator for cloud andmap reduce algorithms and architectures

Powerpoint Templates 43

With Adaptive Scaling

Page 44: An adaptive distributed simulator for cloud andmap reduce algorithms and architectures

Powerpoint Templates 44

Simulation 2: Matchmaking-based Application Scheduling

Execution Time

Page 45: An adaptive distributed simulator for cloud andmap reduce algorithms and architectures

Powerpoint Templates 45

Speed up

Page 46: An adaptive distributed simulator for cloud andmap reduce algorithms and architectures

Powerpoint Templates 46

Simulation 3: MapReduce Implementations

Page 47: An adaptive distributed simulator for cloud andmap reduce algorithms and architectures

Powerpoint Templates 47

Scalability

Hazelcast Implementation

Map() invocations = 3

Infinispan Implementation

Reduce() invocations = 159,069

Page 48: An adaptive distributed simulator for cloud andmap reduce algorithms and architectures

Powerpoint Templates 48

Conclusion•Summary–Distribution strategies and algorithms for

cloud and MapReduce simulations.– Implementation of an Elastic Middleware

platform.– Scale and perform with multiple nodes and

larger simulations.

Page 49: An adaptive distributed simulator for cloud andmap reduce algorithms and architectures

Powerpoint Templates 49

Conclusion• Conclusions–Distributed architecture facilitates larger

simulations.– Faster execution of time-consuming

applications.

Page 50: An adaptive distributed simulator for cloud andmap reduce algorithms and architectures

Powerpoint Templates 50

Conclusion• Conclusions– Distributed architecture facilitates larger

simulations.– Faster execution of time-consuming

applications.

• Future Work– State-aware Adaptive Scaling– Infinispan based Cloud Simulations.– Lighter objects.– Generic Elastic Middleware Platform-as-a-

Service.

Page 51: An adaptive distributed simulator for cloud andmap reduce algorithms and architectures

Powerpoint Templates 51

Conclusion• Conclusions– Distributed architecture facilitates larger

simulations.– Faster execution of time-consuming

applications.

• Future Work– State-aware Adaptive Scaling– Infinispan based Cloud Simulations.– Lighter objects.– Generic Elastic Middleware Platform-as-a-

Service.

Thank you! Questions?Thank you! Questions?

Page 52: An adaptive distributed simulator for cloud andmap reduce algorithms and architectures

Powerpoint Templates 52

References Buyya, R., R. Ranjan, & R. N. Calheiros (2009). Modeling and simulation of scalable cloud computing

environments and the cloudsim toolkit: Challenges and opportunities. In High Performance Computing & Simulation, 2009. HPCS’09. International Conference on, pp. 1–11. IEEE.

Calheiros, R. N., R. Ranjan, C. A. De Rose, & R. Buyya (2009). Cloudsim: A novel framework for modeling and simulation of cloud computing infrastructures and services. arXiv preprint arXiv:0903.2525

Calheiros, R. N., R. Ranjan, A. Beloglazov, C. A. De Rose, & R. Buyya (2011). Cloudsim: a toolkit for modeling and simulation of cloud computing environments and evaluation of resource provisioning algorithms. Software: Practice and Experience 41 (1), 23–50.

Casanova, H. (2001). Simgrid: A toolkit for the simulation of application scheduling. In Cluster Computing and the Grid, 2001. Proceedings. First IEEE/ACM International Symposium on, pp. 430–437. IEEE.

Casanova, H., A. Legrand, & M. Quinson (2008). Simgrid: A generic framework for large-scale distributed experiments. In Computer Modeling and Simulation, 2008. UKSIM 2008. Tenth International Conference on, pp. 126–131. IEEE.

Johns, M. (2013). Getting Started with Hazelcast. Packt Publishing Ltd.

Kliazovich, D., P. Bouvry, & S. U. Khan (2012). Greencloud: a packet-level simulator of energy-aware cloud computing data centers. The Journal of Supercomputing 62 (3), 1263–1283.

Legrand, A., L. Marchal, & H. Casanova (2003). Scheduling distributed applications: the simgrid simulation framework. In Cluster Computing and the Grid, 2003. Proceedings. CCGrid 2003. 3rd IEEE/ACM International Symposium on, pp. 138–145. IEEE.

Marchioni, F. (2012). Infinispan Data Grid Platform. Packt Publishing Ltd.