TOWARDS MIGRATABLE ELASTIC VIRTUAL CLUSTERS ON HYBRID CLOUDS8th IEEE International Conference on

Cloud Computing (IEEE Cloud 2015), June 28, New York, USA

This work has been supported by the Ministerio de Economía y Competitividad via the CLUVIEM project (TIN2013-44390-R)

Instituto de Instrumentación para Imagen Molecular

Universitat Politècnica de València, Spain

INTRODUCTION

• Clusters of PCs are a widely used well known computing facility for many types of workloads• High Performance Computing• High Throughput Computing

• Virtual Elastic Clusters on a Cloud introduce significant advantages• No upfront investments in hardware• Ability to dynamically resize the clusters

depending on the workload• Provision resources from multi-Clouds

to create hybrid clusters

WN

WN

WN

WN

FE

Cluster: Computing nodes connected by a network and managed by a Local Resource Management System (LRMS)

ELASTIC CLOUD COMPUTING CLUSTER (EC3)

• http://www.grycap.upv.es/ec3• Cost-aware elastic hybrid

clusters on multi-Clouds • Open-source tool

HARNESSING SPOT INSTANCES FOR COST-EFFECTIVE VIRTUAL CLUSTERS• Automatic checkpointing (based on BLCR) as

a countermeasure for the termination of EC2 instances

• Dynamic threshold management to try to perform the checkpoint before the instance is terminated

MIGRATION SCHEMES FOR VIRTUAL CLUSTERS

• Different approaches to migration• Migration of applications

• Hybrid clusters across providers (A,B) connected via VPN where nodes from A are deallocated and allocated in B.

• Migration of clusters across multi-Clouds• Virtual cluster cloning & application

state transfer via checkpointing.• Using containers and checkpointing

via Docker + CRIU.

CASE STUDIES

• Execution of scientific applications on hybrid elastic virtual clusters.• Nonlinear analysis

of structural buildings.

• Sequential and parallel jobs executed in a single cluster across :• An OpenNebula on-

premises Cloud• Amazon Web

Services

ELASTIC CLUSTERS AS A SERVICE (I)

• SaaS offered for free to deploy virtual clusters• No need to register, deploy on the Clouds

you can access

http://servproject.i3m.upv.es/ec3

ELASTIC CLUSTERS AS A SERVICE (II)

• Support for multiple Clouds (even OpenStack via the CLI)

• Advantages• Provide cluster-

based computing access to the scientific community

• Deploy customized virtual clusters across Clouds

• Avoid hardware maintenance

• Pay-as-you-go, only for the resources used

ELASTIC CLUSTERS AS A SERVICE (III)

• Use the CLI or the SaaS

ECOSYSTEM OF TOOLS

IM CloudVAMP

EC3 VMCA

VMRC CLUES

• A Vertical Elasticity IaaS Memory Overprovisioning Manager for On-premises Cloudshttps://github.com/grycap/cloudvamp

• A Virtual Machine Consolidation & Migration Agentwww.grycap.upv.es/

vmca

• An Energy/Elasticity Management System for Clusters and Cloud infrastructures.

www.grycap.upv.es/clues

• Deployment & Configuration of Complex Infrastructures on Multi-Cloudswww.grycap.upv.es/im

• Deployment of Elastic Virtual Hybrid Clusters across Public and On-premises IaaS Clouds

www.grycap.upv.es/ec3

• A Rich Metadata Catalog of Virtual Machine Images.www.grycap.upv.es/vmrc

• Open-source developments available at: https://github.com/grycap

CONTACT

Germán Moltó, Amanda CalatravaInstituto de Instrumentación para Imagen

MolecularUniversitat Politècnica de ValènciaCamino de Vera s/n46022, ValenciaSPAIN

gmolto@dsic.upv.es, amcaar@i3m.upv.es

http://www.grycap.upv.es/gmolto@gmolto