OpenStack HA

1

HA in OpenStack

September 24th 2013

RACKSPACE® HOSTING | WWW.RACKSPACE.COM

Agenda

• What is HA

• Compute HA

• Controller HA• Corosync, Pacemaker and DRBD

• Galera

• HAProxy, keepalived, VRRP

• Resources and Summary

2


What is HA?

3


High Availability

4

Minimize data loss

Minimize system downtime


High Availability Concepts

• Stateless services– There is no dependency between requests

– For example: Nova API, Nova Scheduler, etc.

• Stateful services– An action typically comprises multiple requests

– For example: MySQL, RabbitMQ, etc.

5


High Availability Concepts

• Active/Passive– Redundant instances of stateless services are load balanced

– For Stateful services a replacement resource can be brought online.

• Active/Active– Redundant instances of stateless services are load balanced

– Stateful services are managed in such a way that services are redundant, and that all instances have an identical state.

– Updates to one instance of a database would also update all other instances.

6


Compute HA

7


No vSphere Style HA with KVM

8


Server Evacuation

9

Evacuation

Evacuation

• Without Shared Storage– The instance will be booted from a

new disk, but will preserve the configuration, e.g. id, name, uid, ip...etc.

• With Shared Storage– The instance will be booted from

same disk and data will be preserved


Instance Migration

10


11

Virtualization vs. Cloud

• Virtualization needs care and feeding– Name the VM

– Tune and groom regularly

– Feed it with good food and supplements

– Take to the vet when sick

• Cloud servers are disposable– VMs are not unique– Tune and groom apps not the cows– Keep the cow upright– Shoot the cow when it is sick


Scale Up vs. Scale Out

12

Traditional Cloud


Application HA

13


Controller HA Options

14


Pacemaker, Corosync and DRBD

15

• Pacemaker– high availability and load balancing stack for

the Linux platform.

– Interacts with applications through Resource Agents (RA)

• Corosync– Totem single-ring ordering and membership

protocol

– UDP and InfiniBand based messaging, quorum, and cluster membership to Pacemaker.

• DRBD (Distributed Replication Block Device)– Synchronizes Data at the block device

– Uses a journaling system (such as ext3 or ext4)


Galera

• Synchronous multi-master cluster technology for MySQL/InnoDB– MySQL patched for wsrep (Write Set

REPlication)

– Active/active multi-master topology

– Read and write to any cluster node

– True parallel replication, in row level

– No slave lag or integrity issues

16


HA with Keepalived and HAProxy

17


Keepalived, HAProxy and VRRP

• HAProxy– Load Balancing and Proxying for HTTP and TCP Applications

– Works over multiple connections

– Used to load balance API services

• VRRP (Virtual Router Redundancy Protocol)– Eliminates SPOF in a static default routed environment

• Keepalived– Based on Linux Virtual Server (IPVS) kernel module to provide layer 4 Load Balancing

– Implements a set of checkers to check service status and to maintain health

– Leverage the VRRP Protocol to remap VIPS in event of failure

18


Sample OpenStack HA Architecture

19

.… Availability

Zone 1

Dedicated Firewalls

BON

D0

BON

D1

BON

D0

BON

D1

ControllerAPI ServicesAPI & Horizon

Cinder APINova Scheduler

KeystoneGlance

RabbitMQMYSQLChef

ServerRecipes

Load Balancers

Redundant Network Switches

StorageEMC, NetApp, or

SolidfireVols

BON

D2

Redundant Network Switches

Inside LB VLANStorage Network (private)

Fixed Network (private)

Compute 1

KVM

G2

G1

G4

G3

Compute N

KVM

G6

G5

G7

BON

D0

BON

D1

BON

D2

.… Availability

Zone 2

BON

D0

BON

D1

BON

D2

Compute 1

KVM

Compute N

KVM

G16

G15

G17

BON

D0

BON

D1

BON

D2

BON

D0

BON

D1

BON

D2

G12

G11

G14

G13

BON

D2

ControllerAPI ServicesAPI & Horizon

Cinder APINova Scheduler

KeystoneGlance

RabbitMQMYSQLChef

ServerRecipes


HA on OpenStack

20


HA on OpenStack

21


22

Comparison

Database Replication method

Strengths Weakness/Limitations

Keepalived/HAProxy/VRRP

Works on MySQL master-master replication

Simple to implement and understand. Works for any storage system.

Master-master replication does not work beyond 2 nodes.

Pacemaker/Corosync/DRBD

Mirroring on Block Devices

Well tested More complex to setup. Split Brain possibility

Galera Based on write-set Replication (wsrep)

No Slave lag Needs at least 3 nodes. Relatively new.

Others MySQL Cluster, RHCS with DAS/SAN storage

Well tested More complex setup.


Resources and Contact

23


Resources

• OpenStack– openstack.org

– launchpad.net/openstack

– #openstack

– #openstack on webchat.freenode.net

• OpenStack HA– http://docs.openstack.org/trunk/openstack-ha/openstack-ha-guide-trunk.pdf

– https://github.com/rcbops-cookbooks/

• MySQL HA– http://www.mysql.com/why-mysql/white-papers/mysql-high-availability-drbd-configuration-de

ployment-guide/

– http://dev.mysql.com/doc/refman/5.7/en/ha-overview.html

– https://www.hastexo.com/

– http://www.drbd.org/

http://www.openstack.org/

https://launchpad.net/openstack

https://launchpad.net/openstack

http://docs.openstack.org/trunk/openstack-ha/openstack-ha-guide-trunk.pdf

http://docs.openstack.org/trunk/openstack-ha/openstack-ha-guide-trunk.pdf

http://www.mysql.com/why-mysql/white-papers/mysql-high-availability-drbd-configuration-deployment-guide/



http://dev.mysql.com/doc/refman/5.7/en/ha-overview.html

http://dev.mysql.com/doc/refman/5.7/en/ha-overview.html

https://www.hastexo.com/

https://www.hastexo.com/

RACKSPACE® HOSTING | WWW.RACKSPACE.COM 25


For More Information

You can reach me at:

Kenneth Hui

Open Cloud Architect

Rackspace

E-mail: [email protected]

Twitter: @hui_kenneth

Blog: http://cloudarchitectmusings.com

mailto:[email protected]

http://cloudarchitectmusings.com/

26

RACKSPACE® HOSTING | 5000 WALZEM ROAD | SAN ANTONIO, TX 78218

US SALES: 1-800-961-2888 | US SUPPORT: 1-800-961-4454 | WWW.RACKSPACE.COM

RACKSPACE® HOSTING | © RACKSPACE US, INC. | RACKSPACE® AND FANATICAL SUPPORT® ARE SERVICE MARKS OF RACKSPACE US, INC. REGISTERED IN THE UNITED STATES AND OTHER COUNTRIES. | WWW.RACKSPACE.COM

Technology

OpenStack HA