Apache CloudStack Architecture by Alex Huang

Apache CloudStackArchitecture

Alex HuangSoftware Architect, Citrix Systems

Deployment Architecture

• Hosts• Servers onto which services will be provisioned

• Primary Storage• VM disk storage

• Cluster• A grouping of hosts and their associated storage

• Pod• Collection of clusters in the same failure boundary

• Network• Logical network associated with service offerings

• Secondary Storage• Template, snapshot and ISO storage

• Zone• Collection of pods, network offerings and

secondary storage

• Management Server Farm• Management and provisioning tasks

Components

Zone

CloudStack Pod

Cluster

Host

HostNetwork

PrimaryStorage

VM

VM

CloudStack Pod

ClusterSecondary

Storage

Pod 1

Host 2

Cluster 1

Host 1

PrimaryStorage

L3 switch

SecondaryStorage

L2 switch

Two Types of Storage

• Stores disk volumes for VMs in a cluster• Configured at Cluster-level. • Close to hosts for better performance• Cluster have at least one primary storage• Requires high IOPs (can be expensive)

Primary Storage

• Stores all Templates, ISOs and Snapshots• Configured at Zone-level• Zone can have one or more secondary

storages• High capacity, low cost commodity

storage

Secondary Storage

Pod 1

….

Cluster N

L2

Host 2

Cluster 1

Deployment Architecture

Host 1

Hypervisor is the basic unit of scale.

Cluster consists of one ore more hosts of same hypervisor

All hosts in cluster have access to shared (primary) storage

Pod is one or more clusters, usually with L2 switches.

Availability Zone has one or more pods, has access to secondary storage.

One or more zones represent cloud

PrimaryStorage

Zone 1

….

L3

SecondaryStorage

Pod N

Management Server Cluster

Internet

Management Server Cluster

Replica

InfrastructureResources

User API

Admin API

Load Balancer

Management Server

Management Server

MySQL

MS is stateless. MS can be deployed as physical server or VM

Single MS node can manage up to 10K hosts. Multiple nodes can be deployed for scale or redundancy

RHEL 5.4+, Ubuntu 10.0.4, Fedora 16

Replication

Managing Complexity

The Three C’s of Complexity

• Control• Choice• Compliance

Compute

Giving Control Brings Complexity

Network Storage

Admin

Users

Org A

Admin

Users

Org BUsers

End User

AdminVM

Ware

XenServer

KVM

NFS

iSCSI

FC

NetScaler

F5

Jun. SRX

Local DiskCisco ASA

Swift

HDFS

Hyper-V

• ACL• Limits• Governance

OracleVM

BareMetal

Guest Virtual Layer-2 NetworkGuest 1

VM 1

Guest 1 VM 2

Guest 1 VM 3

Guest 1 Virtual Network 10.1.1.0/24

Gateway 10.1.1.1

Guest 10.1.1.2

Guest 10.1.1.3

Guest 10.1.1.4

Guest 1 Virtual Router

Guest 2 VM 1

Guest 2 VM 2

Guest 2 VM 3

Guest 2 Virtual Network 10.1.1.0/24

Gateway 10.1.1.1

Guest 10.1.1.2

Guest 10.1.1.3

Guest 10.1.1.4

Guest 2 Virtual Router

Public IP 65.37.141.2465.37.141.80

Public IP 65.37.141.1165.37.141.36

Internet

Multi-tier Network

Private IP10.1.1.112

DHCP, DNSUser-data

Public IP 65.37.141.112

10.1.1.1Web VM

1

10.1.1.3Web VM

2

10.1.1.4Web VM

3

10.1.1.5Web VM

4

NetscalerLoad

Balancer

Private IP10.1.1.111

Public IP 65.37.141.111

Juniper SRX

Firewall

Virtual Router

Virtual Network 10.1.1.0/24VLAN 100


10.1.2.21

10.1.2.18

10.1.2.38

10.1.2.39

10.1.2.31App VM

1 10.1.3.21


10.1.2.24App VM

2 10.1.3.45

10.1.3.24 DB VM 1

DHCP, DNS, User-data

DHCP, DNSUser-data,Source-NAT, VPN Public IP

65.37.141.115

Virtual Router

Virtual Router

Unified Multi-tier Network

10.1.1.1Web VM 1

10.1.1.3Web VM 2

10.1.1.4Web VM 3

10.1.1.5Web VM 4



10.1.2.31App

VM 1


10.1.2.24App

VM 2

10.1.3.24DB VM

1

Virtual Router Customer

Premises

IPSec or SSL site-to-site VPN

Internet

Monitoring VLAN

Virtual Router Services• IPAM• DNS• LB [intra]• S-2-S VPN• Static Routes• ACLs• NAT, PF• FW [ingress & egress]• BGP

Load Balancer

Other Topologies

Guest Virtual Network 10.1.1.0/24VLAN 100

Gateway address 10.1.1.1

10.1.1.1Guest VM 1

10.1.1.3Guest VM 2

10.1.1.4Guest VM 3

10.1.1.5Guest VM 4


DHCP, DNSUser-data

10.1.1.1Guest VM 1

10.1.1.3Guest VM 2

10.1.1.4Guest VM 3

10.1.1.5Guest VM 4

No services [Static IPs] Dedicated VLAN with DHCP and DNSUser can request specific IP[s] for NIC

Core switch


Core switch

Virtual Router

Other Topologies



10.1.1.100Guest VM 1

10.1.1.200Guest VM 2

10.1.1.101Guest VM 3

10.1.1.115

Guest VM 4


DHCP, DNSUser-data

10.1.1.1Guest VM 1

10.1.1.3Guest VM 2

10.1.1.4Guest VM 3

10.1.1.5Guest VM 4

MPLS Use Case Shared VLAN with DHCP and DNS

CSVirtual Router

Core switch


Core switch

MPLS VLAN 100

DHCP, DNSUser-data

CSVirtual Router

…

DB Security Group

WebSecurity Group

Layer 3 Networking (Amazon Style)

… …

Web VM

Web VM

Web VM

Web VM

DB VM

Web VM

DB VM

Web VM

Software Architecture

Management Server

Orchestration Engine- Drives long running VM

operations- Syncs between resources

managed and DB- Generates events

Resource Management

Cluster Management

JobManagement

DB

UI Cloud Portal CLI

Other Clients

Deployment Planning

Network Gurus

Network Elements

Hypervisor Gurus

DatabaseAccess

Alert & EventManagement

Plug

in A

PI

Resource APIHypervisor Resources

Network Resources

Storage Resources

ImageResources

SnapshotResources

REST API

OAM&P API End User API EC2 API Pluggable Service API EngineOther APIs

Security Adapters

Account Management Connectors

ACL & Authentication- Accounts, Domains, and Projects- ACL, limits checking

Services API

Serv

ices

API

Console Proxy Management

Template Access

HA

Usage Calculations

Additional Services

Event BusMessage Bus

Usage Server

Orchestration Engine

• Understands how to orchestrate long running processes (i.e. VM starts, Snapshot copies, Template propagation)

• Well defined process steps• Calls Plugin API to execute functionalities that

it needs

Plugins

• Various ways to add more capability to CloudStack

• Implements clearly defined interfaces• All operations must be idempotent• All calls are at transaction boundaries• Compiles only against the Plugin API module

Anatomy of a Plugin

ServerResource- Optional. Required if Plugin needs to be co-

located with the resource- Implements translation layer to talk to

resource- Communicates with server component via

JSON

Rest API- Optional. Required only if needs to expose

configuration API to admin.

Plug

in A

PI

Data Access Layer

Implementation

• Can be two jars: server component to be deployed on management server and an optional ServerResource component to be deployed co-located with the resource

• Server component can implement multiple Plugin APIs to add its feature

• Can expose its own API through Pluggable Service so administrators can configure the plugin

• As an example, OVS plugin actually implements both NetworkGuru and NetworkElement

Plugin Interfaces Available• NetworkGuru – Implements various network isolation and ip address

technologies• NetworkElement – Facilitate network services on network elements

to support a VM (i.e. DNS, DHCP, LB, VPN, Port Forwarding, etc)• DeploymentPlanner – Different algorithms to place a VM and

volumes.• Investigator – Ways to find out if a host is down or VM is down.• Fencer – Ways to fence off a VM if the state is unknown• UserAuthenticator – Methods of authenticating a user• SecurityChecker – ACL access• HostAllocator – Provides different ways to allocate host• StoragePoolAllocator – Provides different ways to allocate volumes

Separating Data and Control

Data Center 1

Cloud

Data Center 2

Data Center 3

Management

Server

Management Servers control all resources, both virtual and physical

SSVMs deployed to transfer data between zones

CPVMs deployed to transfer VNC console traffic

VR deployed for traffic into public internet

Management Server is never in the data path

SSVM

SSVM

SSVMTransfer of Templates,

ISOs, Snapshots

CPVMCPVM

CPVM

VR

VR

VR

Internet

Kernel

Sequence Flow for VM CreationEnd User Rest API

SecurityCheckers

User VM Mgr

Network Mgr

Storage MgrJob

SchedulingVirtualMachine Mgr

Network Guru

Deploy VM

ACL Checks

Allocate Entity in CS

Allocate VM

Allocate NIC

Allocate Volume

Allocate IP

Schedules Deploy Job

Returns with job id, VM id

Query Job Result

Returns with job status

Sequence Flow for VM CreationJob Threads

Network Element

User VM Mgr

Network Mgr

Storage Mgr

VirtualMachine Mgr

Network Guru

Start VM

Start VM

Prepare Nics

Notify that Nic is about to be started in network

Reserve resources for Nic

Services APIServer

Resources

Start User VM

Agent Calls

Prepare Volumes

Template Mgr

Deployment

Planner

Get a Deployment Plan (Host and StoragePool)

Prepare template on Primary Storage

Agent Calls

Agent Start VM Call

Stores job result

Conclusion

Design Goals for CloudStack

• Design for complexity– Clear interfaces

• Design for scalability– Separate out data path and control paths– Design to maximize the use of database connections

• Design against failure– Provide clear boundaries (process and compilation)– Utilize cloud administrator to give guidance

More Information

27

• http://cloudstack.org

• Apache mailing lists–[email protected] –[email protected]

• Thank you

http://cloudstack.org/

mailto:[email protected]

mailto:[email protected]

Technology

Apache CloudStack Architecture by Alex Huang