How to Manage Your Cloud by Drupal (DrupalCon CPH 2010)

PROPEOPLE26. aug 16:00

How to ManageYour Cloud by Drupal

Yas Naoi

The Purpose of This Presentation

•Learn about Cloud Computing

•Introduce Virtual Infra Manager

•Share Business Value

•Feedback from YOU!

Biz ValueDemoVirtual InfraCloud

Agenda

•Cloud Computing Overview

•Virtual Infra Manager

•Demo

•Business Value

•Q&A


Cloud Computing Overview


Centralized to the Internet

UsersUsers Service Service ProvidersProviders

Server

Networks

AuthN

DevelopmentEnvironment

UIContents

Database

Application

Operations

Develop

Services

ProvideServices

DevelopersCommunity



UsersUsers Service Service ProvidersProviders

Server

Networks

AuthN

DevelopmentEnvironment

UIContents

Database

Application

Operations

Low PricePCs

ServerVirtualizationBroadband

Network-centricServices

Develop

Services

ProvideServices

DevelopersCommunity



UsersUsers Cloud Service Service ProvidersProviders

Low PricePCs

ServerVirtualizationBroadband

Network-centricServices

Develop

Services

ProvideServices


Cloud Computing Features

•On-Demand Resources•Cost Reduction by

Pay-as-you-go Billing / Server Consolidation

•Easy Provisioning•Auto-scaling•Higher Productivity


Cloud Computing Architecture

Cloud Application ： Software as a Service

Photo VirtualDesktopBackupMailSearch

Cloud Infrastructure ： Infrastructure as a Service

Cloud Management Infrastructure ： Platform as a ServicePaaSPaaS

SaaSSaaS

IaaSIaaS

Log AnalysisMovie …Maps Chat

Virtual InfraManagement Security Auto-scalingLoad

Balancing …

PersistentStorage

MessageQueue

DistributedFile System

VirtualServer …


Virtual Infra Manager


Why Cloud for Service Providers?

•Problems

•Manage Tons of Servers

•Manage Peta-byte Class Data

•Requirements

•Downsize and Reduce the Cost

•Flexible Capacity Planning



•Problems

•Manage Tons of Servers

•Manage Peta-byte Class Data

•Requirements

•Cost Reduction

•Flexible Resource Management



•Cost Reduction

•Open Source

•Browser-based Easy Operations

•Flexible Resource Management

•Multiple Cloud Access

•Private

•Public


Private Cloud Requirements

•Browser-based UI

•Unified Access for Multiple Cloud



•Browser-based UI


•Management

•Linux & Windows

•Virtual Machine Image

•Server Template

•Network

•Audit Log and Reports

•Backup



•Security

•User Authentication

•Permissions

•Browser-based UI


•Management

•Linux & Windows


•Server Template

•Network


•Backup



•Security


•Permissions

•Monitoring

•Physical Servers / Cloud Nodes

•Virtual Machines

•Applications

•Browser-based UI


•Management

•Linux & Windows


•Server Template

•Network


•Backup



•Security


•Permissions

•Monitoring

•Physical Servers / Cloud Nodes

•Virtual Machines

•Applications

•Open Source

•Browser-based UI


•Management

•Linux & Windows


•Server Template

•Network


•Backup


The Goal ofVirtual Infra Manager

Multiple Public Cloud

Users

FWC6

C8C7

C5

C2

C4C3

C1- Amazon- Google-Windows

Azure- etc.Multiple Private Cloud

C2

C4C3

C1 C6

C8C7

C5

• Unified Management Console

• Flexible Resource Management


The Goal ofVirtual Infra Manager

Multiple Public Cloud

Users

FWC6

C8C7

C5

HybridHybridCloud PortalCloud Portal C2

C4C3

C1- Amazon- Google-Windows

Azure- etc.Multiple Private Cloud

C2

C4C3

C1 C6

C8C7

C5

• Unified Management Console

• Flexible Resource Management

• Monitoring• Security• Backup• Failover, etc.


Architecture OverviewHome

Control Life Logs DataAnalysisHealthcare Location

Service…

Private Cloud

EucalyptusXCP

Public Cloud

Amazon

UserAuthN Permissions User

Management Access LogPortal UI JobQueue

Virtual InfraManagement

ServerCloning

LoadBalancing DatastoreFailover

StorageSnapshot

User ActivityAuditing

Batch JobExecution

Provisioning

Monitoring ServerImage Backup

Billing

PersistentStorage

MessageQueue

DistributedFile System memcached


SaaSPaaSIaaS

Why Drupal?Biz ValueDemoVirtual InfraCloud


Drupal + Custom Modules

Enomaly(Until Feb. 2010)

oVirt(As of Mar. 2010)

OpenXenManager

Developer - Enomaly Red Hat CitrixLicense GPL2 AGPL3 GPL2 GPL2Browser-based Yes Yes Yes Yes (Recently)Programming Language PHP Python Ruby on Rails

Python Python

Database MySQL MySQL PostgreSQL N/AHypervisor Support (Any) KVM / Xen / etc

(thru libvirt)KVM / Xen /etc

(thru libvirt) Xen Only

Install & Setup Easy Difficult Difficult EasyFunctionality (Yes) Yes w/ Limitations (Yes) Yes

API Drupal / REST REST REST(Not Exposed) XAPI / XML-RPC

Perspective Users (and Admin) Admin (and Users) Admin and Users Admin

Stability D6 Core Stable SometimesDatabase Corrupt

ImplementationImmature

SometimesHang-up





OpenXenManager


Python Python









SometimesHang-up





OpenXenManager


Python Python









SometimesHang-up

Why Drupal?

•Web Application Framework

• Highly-Modularized Architecture

• User Authentication & Management

• Permissions

• Key-value Pair Programming Style (Array)

• Queuing System Support (D7)

• NoSQL Support (D7)


Why Drupal?




• Permissions





Why Drupal?




• Permissions







Service…

Private Cloud

EucalyptusXCP

Public Cloud

Amazon




ServerCloning


StorageSnapshot


Batch JobExecution

Provisioning


Billing

PersistentStorage

MessageQueue



SaaSPaaSIaaS



Service…

Private Cloud

EucalyptusXCP

Public Cloud

Amazon




ServerCloning


StorageSnapshot


Batch JobExecution

Provisioning


Billing

PersistentStorage

MessageQueue



SaaSPaaSIaaS Drupal Core

Virtual Infra Manager•Works with Multiple Cloud

•Amazon EC2

•Eucalyptus

•XCP (Xen Cloud Platform)


Architecture


Development Module

Drupal Core Drupal Modules

IaaS

Eucalyptus

IaaS

XCP

IaaS

Amazon EC2

Drupal

• Drupal Core as Web Application Framework

Hybrid Cloud Portal

Public

Private

Private

UI Module


Existing Component

Architecture


Development Module


IaaS

Eucalyptus

IaaS

XCP

IaaS

Amazon EC2AWS

Drupal

• Drupal Core as Web Application Framework• Drupal Custom Modules for Each Cloud, Templates, etc.

Hybrid Cloud Portal

Public

Private

Private

Eucalyptus

XCPUI Module


Existing Component

Architecture


Development Module


IaaS

Eucalyptus

IaaS

XCP

IaaS

Amazon EC2AWS

Drupal

Existing Component

• Drupal Core as Web Application Framework• Drupal Custom Modules for Each Cloud, Templates, etc.• Failover by Monitoring Virtual Machines

Hybrid Cloud Portal

Public

Private

Private

Monitoring

Backup

Failover

Eucalyptus

XCPUI Module


DeploymentPrivate Cloud

• User AuthN• Permissions• User Activity Logs• VM Image Management• Provisioning• Server Template• Snapshot / Backup• Monitoring

• User AuthN• Permissions• User Activity Logs• VM Image Management• Provisioning• Server Template• Snapshot / Backup• Monitoring

Virtual InfraManager

RESTREST

XML-RPCXML-RPC

ActiveDirectory

LDAPLDAP

HTTPSHTTPS

Eucalyptus Cloud

Cloud Controller + Cluster Controller

Node Controller VM VM VM VMNode Controller VM VM VM VM

XCP Cloud

Pool Master VM VM VM

VM VM VM VM

Slave VM VM VM

VM VM VM VMSlave VM VM VM

VM VM VM VM

Storage

VMImage

• Provisioning• Monitoring• Alert

• Provisioning• Monitoring• Alert

PhysicalServer

MaintenanceSystem


Systems Requirements

•Drupal 6.19

•Ubuntu 10.04

•Apache 2.2

•MySQL 5.1

•PHP 5.3

•Collectd 4.8

•RRDTool 1.3

• Nagios 3.2


ModulesCategory Module Name Description

Cloud Management

Common •System Common LibraryAmazon •Manage Amazon EC2Eucalyptus •Manage Eucalyptus CloudXCP •Manage XCP Cloud



Cloud Management


Provisioning

Registration •Manage Virtual Server ImagesServer Templates •Manage Server TemplatesCluster •Manage Server Template CollectionScripting •Manage Scripts for AutomationInput Parameters •Input Parameters for Running Virtual Server



Cloud Management


Provisioning


ReportingAudit •User Activity LogUsage Report •Cost Estimation Depending on Hours of UsageAlerts •Provides Alert Functionality



Cloud Management


Provisioning


ReportingAudit •User Activity LogUsage Report •Cost Estimation Depending on Hours of UsageAlerts •Provides Alert Functionality

ReliabilityFailover •Provide Failover FunctionalityBackup •Provide Backup Functionality


Problem in-the-CloudBiz ValueDemoVirtual InfraCloud

Problem in-the-Cloud•Async Operations for VMs on Virtual Infra


Problem in-the-Cloud•Async Operations for VMs on Virtual Infra

•Boot

•Shutdown

•Suspend / Resume

•VM Backup


XCP Cloud

Master VM VM VM

VM VM VM VM

Slave-1 VM VM VM

VM VM VM VM


Virtual Infra Operation

BootXCP Module

Drupal Core

Browser

Slave-2 VM VM VM

VM VM VM VM


XCP Cloud

Master VM VM VM

VM VM VM VM

Slave-1 VM VM VM

VM VM VM VM



Boot

Shutdown

XCP Module

Drupal Core

Browser

Slave-2 VM VM VM

VM VM VM VM


XCP Cloud

Master VM VM VM

VM VM VM VM

Slave-1 VM VM VM

VM VM VM VMSlave-2 VM VM VM

VM VM VM VM



Boot

Shutdown

Backup

XCP Module

Drupal Core

Browser


XCP Cloud

Master VM VM VM

VM VM VM VM

Slave-1 VM VM VM

VM VM VM VM



XCP Module

Drupal Core Boot

Shutdown

Backup

Browser

Slave-2 VM VM VM

VM VM VM VM

Shutdown


XCP Cloud

Master VM VM VM

VM VM VM VM

Slave-1 VM VM VM

VM VM VM VM



VM VM VM

VM VM VM VM

Bottleneck

XCP Module

Drupal Core Boot

Shutdown

Backup

Browser

Slave-2 VM VM VM

VM VM VM VM

Shutdown



Virtual Infra OperationUsing Drupal Queue

XCP Module

Browser

Drupal Core

XCP Cloud

Master VM VM VM

VM VM VM VM

Slave-1 VM VM VM


VM VM VM VM

Dequeue

Drupal QueueCron

Enqueue

Drupal QueueBootBoot

ShutdownShutdown

SuspendSuspend

BootBoot



Virtual Infra OperationUsing Drupal Queue

XCP Module

Browser

Drupal Core

XCP Cloud

Master VM VM VM

VM VM VM VM

Slave-1 VM VM VM


VM VM VM VM

Dequeue

Drupal QueueCron

Enqueue

Drupal QueueBootBoot

ShutdownShutdown

SuspendSuspend

BootBoot



VM Image BackupUsing Drupal Queue

XCP Module

Drupal Core

Enqueue

Dequeue

Drupal Queue

Drupal QueueCron

W1

Persistent Storage

Image RepositoryUpload Im

age

3 – 20 min+ 3 – 10 min+

1. Shutdown VM2. Bundle Image3. Upload Image

Browser

Wn …Worker Process




XCP Module

Drupal Core

Enqueue

Dequeue

Drupal Queue

Drupal QueueCron

W1

W2

Persistent Storage


age

3 – 20 min+ 3 – 10 min+


Browser





XCP Module

Drupal Core

Enqueue

Dequeue

Drupal Queue

Drupal QueueCron

W3

W1

W2

Persistent Storage


age

3 – 20 min+ 3 – 10 min+


Browser



Demo


Business Value byVirtual Infra Manager


Cost ReductionBiz ValueDemoVirtual InfraCloud

Cost ReductionTraditional Traditional


Cost Reduction

Install Windows

Traditional Traditional


Cost Reduction

Windows Update

Install Windows



Cost Reduction

Software Installation

Windows Update

Install Windows



Cost Reduction

Printer Setup


Windows Update

Install Windows



Cost Reduction

Printer Setup


Network Setup

Windows Update

Install Windows



Cost Reduction

Network Setup

Just Copya Pre-configured VM Image

Virtual Infra Virtual Infra


Printer Setup


Network Setup

Windows Update

Install Windows


Business Value (1/2)•Virtual Infra Manager•No Promising OSS•Reference Model and Implementation for

Cloud Management•Multiple Cloud Support •Security•Reliability•Scalability


Business Value (2/2)•Virtual Infra Manager•Effective Cost Reduction of Cloud

Management

•Unified Access to Multiple Cloud•Easy Provisioning•Easy Operations•Open Source


Future DirectionMail Maps DocsSearch Location

Service…

Drupal 7 Core

User & AuthNManagement Permissions Report Cache

Custom Modules

Queue


ServerCloning

NoSQL

Provisioning

StorageSnapshot


Automation

Failover


Billing

Portal UI

Batch JobExecution

Private Cloud

EucalyptusXCP MySQL Mem

cachedVarnish

Public Cloud

Amazon Google MongoDBPersistentStorage


SaaSPaaSIaaS Custom ExistingD7 Core

Future DirectionMail Maps DocsSearch Location

Service…

Drupal 7 Core

User & AuthNManagement Permissions Report Cache

Custom Modules

Queue


ServerCloning

NoSQL

Provisioning

StorageSnapshot


Automation

Failover


Billing

Portal UI

Batch JobExecution

Private Cloud

EucalyptusXCP MySQL Mem

cachedVarnish

Public Cloud

Amazon Google MongoDBPersistentStorage


SaaSPaaSIaaS

Virtual Infra ManagerMore Scalable and Reliable

Cloud Management Platformon D7

Custom ExistingD7 Core

http://cph2010.drupal.org/node/14728

Q&A

Biz ValueVirtual InfraCloudBackground

Yas Naoi

[email protected]

Thank You!“Coming together is a beginning, keeping together is a progress; working together is a success.”- Henry Ford

Biz ValueVirtual InfraCloudBackground

Yas Naoi

[email protected]

Backup


Hybrid (Public / Private) Cloud ManagementHybrid (Public / Private) Cloud Management

Instance MonitoringInstance Monitoring

Supports Windows and Linux VMs


Report – Usage Estimate


Technology

How to Manage Your Cloud by Drupal (DrupalCon CPH 2010)