vCloud Automation Center 6.0

Notes on vCAC for VMware Architects

By Preetam Zare

vZare.com

Disclaimer

• Based on N+2 failure• Focuses on large scale deployment of vCAC infrastructure• Targets RTO of less than 1 min• Components like ITBM, vCenter Orchestrator are not discussed• These slides are aimed to be read rather presented. Therefore lot of texts is

seen• This is first Draft of the PoV

vCAC Appliance

• A pre-configured virtual appliance• Provides single portal for

– self-service provisioning– Management of cloud services– Authoring– Administration

• Comes with embedded PostgreSQL database• Recommended Hardware Configurations

Server Role CPU Memory (GB) Storage (GB)

vCAC Appliance 2 8 30

vCAC - High Availability & Load Balancing

• Scaling of vCAC web component is achieved using multiple instance of vCAC appliance.

• The traffic entering vCAC appliance will be shared using load balancer• Since Database is heart of vCAC appliance, PostgreSQL will be installed on

separate server• To make database highly available, PostgreSQL DB will be installed on

multiple servers and DB can be clustered• Recommended Hardware Configurations of PostgreSQL Server

Server Role vCPU RAM (GB) Storage (GB)

PostgreSQL 2 4 20

IaaS Server

Website

Manager Service

Model Manager

DEMorchestrator

WorkerAgent

IaaS Component

IaaS Website

• A Windows Instance, provides access to infrastructure capabilities in the vCAC web console

• IaaS Website uses a MS SQL Server DB to maintain information about the machines it manages and its own elements and policies.

• The machine that hosts first IaaS website node must also host the model manager data component.

• Recommended Hardware Configurations (without Database)

Server Role CPU Memory (GB) Storage (GB)

IaaS Website 2 4 40

IaaS Website - High Availability & Load Balancing

• Scaling of IaaS website will be achieved using 3 instance of IaaS server.• These 3 instance will be load balanced using 3rd party load balancer• Since Database is heart of IaaS website, install MS SQL Server on separate

server• Database will be made highly available (near zero downtime), By installing

MS SQL Server on two nodes and Failover cluster will be configured to protect DB

• Recommended Hardware Configurations

Server Role vCPU RAM (GB) Storage (GB)

MS SQL Server 8 16 80

NB: Model Manager Data Component (MMDC) is needed to be installed only once and to put MMDC behind load balancer you must disable loop backup check

Manager Service

• Co-ordinates communication between agents (including proxy agents), the Database and SMTP

• To enable High availability, three VMs will host 3 instances of manager services. Two manager service cannot be active at same time. On Node2 (passive node02) and Node3 (passive node03) manager service will be stopped & disabled. Failover is manual process anywhere between 1-10 minutes are needed to bring service back online.

• Concurrent Data collection per agent by default is limited to two. Data collection time will be observed for initial month and later either CPU or agent limit will be increased.

• Data collection is CPU intensive activity. If High CPU utilization is observed it will be compensated by increasing cores for the VM

Distributed Execution Manager

• DEM executes business logic of custom models by interacting with vCAC DB and external databases and systems as required

• DEMs are used for the provisioning and management of virtual, cloud, and physical machines on:– Red Hat Enterprise Virtualization (RHEV) Manager– Microsoft System Center Virtual Machine Manager (SCVMM)– Amazon and VMware vCloud Director– Dell, HP, and Cisco

• DEMs and agents enable vCAC to communicate with external systems for provisioning and managing machines

Distributed Execution Manager Orchestrator

• Each DEM instance performs two roles– Orchestrator– Worker: Worker is responsible for execution of workflows

• DEM Orchestrator Monitors status of DEM worker.• If DEM worker loses connectivity with model manager or stops, DEM

orchestrator puts workflows in the queue of another DEM worker’s to execute.

• DEM Orchestrator manages scheduled workflow by creating new workflow instances at the scheduled time.

• DEM Orchestrator also ensures only one instance of scheduled workflow is running at any given time.

• Pre-processes workflow before executing, including checking if preconditions for workflows are met.

• Creates workflow execution history.

DEM Orchestrator High Availability Considerations

• Orchestrator DEMs support active-active (semi-active) high availability. • When a DEM Orchestrator starts, it searches for another running DEM

Orchestrator. If none is found, it starts executing as the primary DEM Orchestrator.

• If it does find another running DEM Orchestrator, it goes into a passive mode and monitors the other primary DEM Orchestrator to detect an outage.

• If it detects an outage, it takes over as the primary. When the previous primary comes back online, it detects that another DEM Orchestrator has taken over its role as primary and goes into a passive state.

• Hardware Recommendations DEM Orchestrator & Manager Service

High availability behavior is little bit similar to vSphere HA master behavior especially the failure detection

Server Role vCPU RAM (GB) Storage (GB)

Manager Service & DEM Orchestrator

4 6 80

DEM Worker High Availability Considerations

• If a DEM Worker instance fails, the DEM Orchestrator detects the failure and cancels any workflows being executed by the DEM Worker instance.

• When the DEM Worker instance comes back online, it detects that the DEM Orchestrator has canceled the workflows of the instance and stops executing them.

• To prevent workflows from being canceled prematurely, a DEM Worker instance must be offline for several minutes before its workflows can be cancelled.

Agents

• vCAC uses agents to integrate with external systems• Types of Agents

– Hypervisor Proxy Agents for• vSphere• Citrix Xenservers• Hyper-v Servers• WMI Agents

– External Provisioning EPI Agents allows you to run VB script as extra step during Guest OS provisioning

– WMI Agents• The vCAC WMI agent collects data from windows VMs managed by

vCAC• vCAC uses these agents to send command and collect data from ESX

server, XenServer and Hyper-V and Provisioned VMs.

Agent (cont.)

• Requires administrative access to hypervisors• Communicates with manager service• Is installed separately with its configuration file• During installation you have option to install Only vSphere agent• You can scale the solution by installing as many agents as you want for

additional throughput & concurrency

Server Role vCPU RAM (GB) Storage (GB)

Agents & DEM Worker 2 4 80

Hardware Recommendations DEM Worker & Agents

Design Decisions

• DD01 –DEM Orchestrator and Manager services will be installed on same host. As these two components needs to have reliable network connectivity. Option to host them on dedicate VM is ruled out to avoid spread of server estate and potential increase management cost.

• DD02 – 3 VMs hosting both Manager Service & DEM Orchestrator will be placed across three different hosts. This will allow protection against ESX node failures. In worst case when more than one node fails, we still have at least one DEM orchestrator running. 2 instances of manager service and DEM Orchestrator will be idle under any given time.

• DD03: Virtual Machines to Hosts should rule be applied to keep each instance of DEM Orchestrator & Manager and Worker & Agent VM on same host. This will allow all three VMs to stick to individual host and in case of failure of more than 2 ESX host, it will ensure at least 3 VMs are available at ESX host at any given time.

• DD03 – DEM worker and Agent will be placed on same VM. Both these components can be scaled up (DEM worker) and scale out (using multiple VMs). Primary reason to keep both the components same VM is that these two components can be scale out and scale up as required

• DD04 – DEM Orchestrator, DEM Worker, Agent, Manager services, vCAC Appliance, Identity Appliance, ITBM, vCenter Orchestrator will be using single VLAN for better network connectivity among the components

High Availability and ScalabilityComponent High Availability Downtime VM

InstancesvCAC Web Component vSphere HA + Load Balancer Zero 3

IaaS Web Component vSphere HA + Load Balancer Zero 3

Manager Service vSphere HA + Load Balancer 1-10 Min

3DEM Orchestrator vSphere HA + In-built Application Clustering

Zero

DEM Worker vSphere HA + In-built Application Clustering

Zero

3Agents vSphere HA + In-built Application

ClusteringZero

Identity Appliance vSphere HA 1-10 Min 1

MS SQL Server vSphere HA + Failover Clustering Near Zero 2

PostgresSQL vSphere HA + Clustering Near Zero 2

vCloud Application Director (vAppD)

vSphere HA 1-10 Min 1

ITBM vSphere HA 1-10 Min 1

Notes on High Availability and Scalability

• vSphere HA provides protection against ESX failover• Load balancer provides protection against VM and application failover• Since Manager service acts in Active-Passive configuration only, therefore

there is manual downtime between 1-10 mins involved• Identity Appliance, vCloud AppD and ITBM are stand-alone application and

will be protected using vSphere HA• MS SQL Server and PostgreSQL server will be protected using application

based clustering• Near Zero time represents the time needed to failover Database from one

node to another• Except for Manager services, all components of vCAC and IaaS can be

scale out or scaled up. Scaled out decision will be based on CPU utilization.

Additional Information

• Each DEM Worker instance can process 15 concurrent workflows. Excess workflows are queued for execution.

• Increase in processing time will directly impact the number of concurrent workflows that can be executed

• Review the number of pending workflows, or if workflows execution time, If either of the thing is observed, more DEM Worker instances can be added to pick up the workflows. CPU utilization should also be observed to make decision either to add more DEM workers or add additional DEM machines

• Some workflows, particularly certain custom workflows, can be very CPU-intensive. If the CPU load on the DEM Worker machines is high, consider increasing the processing power of the DEM machine or adding more DEM machines to your environment.

• You can use the Workflow History page to determine how long it takes to execute a given workflow.

• The average workflow processing time (from when the DEM Orchestrator starts preprocessing the workflow to when the workflow finishes executing) increases with the number of concurrent workflows.

Additional Information (cont.)

• Do not deploy vCAC Appliance, vAPPD or ITBM over a WAN as performance will be degraded.

• DEM Worker and Agent can be deployed in WAN• Best performance VMware recommends installing DEMs and agents on

different machine• VMware recommends using external vCenter Orchestrator for each tenant

to enforce multi tenancy• When deploying vCloud Application Director you can use the vCloud

Automation Center Single Sign On (SSO) capability to manage users in one place

vCAC Scalability –VMware Recommendation

Components Large Deployment VM QTY

Medium Deployment VM QTY

Identity Virtual Appliance 1 1

vPostgreSQL Appliance 1 1

vCAC Appliance (web component)* 2 2

IaaS Server (web component)* 2

2#IaaS Manager Service* 2

IaaS DEM Server 2 2

IaaS Agent Server 2 2

MS SQL Server (2 node failover cluster) 2 2

Items Large Deployment Medium Deployment

Managed Machines 50,000 10,000

Catalog Items 2,500 2,500

Concurrent Deployments 100 50

• Must be placed behind the load balancer• # Both roles (Manager service and Web components) are placed on same VMs

References

• Reference Architecture –vCloud Automation Center 6.0• Hany

Michael -Reference Architecture – vCloud Automation Center (vCAC) 6.0 Distributed Environment – Part 1 of 2