Openstackha 130925132534-phpapp02

1

HA in OpenStack

September 24th 2013

RACKSPACE® HOSTING | WWW.RACKSPACE.COM

Agenda

• What is HA

• Compute HA

• Controller HA

• Corosync, Pacemaker and DRBD

• Galera

• HAProxy, keepalived, VRRP

• Resources and Summary

2


What is HA?

3


High Availability

4

Minimize data loss

Minimize system downtime


High Availability Concepts

• Stateless services

– There is no dependency between requests

– For example: Nova API, Nova Scheduler, etc.

• Stateful services

– An action typically comprises multiple requests

– For example: MySQL, RabbitMQ, etc.

5


High Availability Concepts

• Active/Passive

– Redundant instances of stateless services are load balanced

– For Stateful services a replacement resource can be brought online.

• Active/Active

– Redundant instances of stateless services are load balanced

– Stateful services are managed in such a way that services are redundant, and that all instances have an identical state.

– Updates to one instance of a database would also update all other instances.

6


Compute HA

7


No vSphere Style HA with KVM

8


Server Evacuation

9

Evacuation

Evacuation

• Without Shared Storage

– The instance will be booted from a new disk, but will preserve the configuration, e.g. id, name, uid, ip...etc.

• With Shared Storage

– The instance will be booted from same disk and data will be preserved


Instance Migration

10


11

Virtualization vs. Cloud

• Virtualization needs care and feeding

– Name the VM

– Tune and groom regularly

– Feed it with good food and supplements

– Take to the vet when sick

• Cloud servers are disposable

– VMs are not unique

– Tune and groom apps not the cows

– Keep the cow upright

– Shoot the cow when it is sick


Scale Up vs. Scale Out

12

Traditional Cloud


Application HA

13


Controller HA Options

14


Pacemaker, Corosync and DRBD

15

• Pacemaker

– high availability and load balancing stack for the Linux platform.

– Interacts with applications through Resource Agents (RA)

• Corosync

– Totem single-ring ordering and membership protocol

– UDP and InfiniBand based messaging, quorum, and cluster membership to Pacemaker.

• DRBD (Distributed Replication Block Device)

– Synchronizes Data at the block device

– Uses a journaling system (such as ext3 or ext4)


Galera

• Synchronous multi-master cluster technology for MySQL/InnoDB

– MySQL patched for wsrep (Write Set REPlication)

– Active/active multi-master topology

– Read and write to any cluster node

– True parallel replication, in row level

– No slave lag or integrity issues

16


HA with Keepalived and HAProxy

17


Keepalived, HAProxy and VRRP

• HAProxy

– Load Balancing and Proxying for HTTP and TCP Applications

– Works over multiple connections

– Used to load balance API services

• VRRP (Virtual Router Redundancy Protocol)

– Eliminates SPOF in a static default routed environment

• Keepalived

– Based on Linux Virtual Server (IPVS) kernel module to provide layer 4 Load Balancing

– Implements a set of checkers to check service status and to maintain health

– Leverage the VRRP Protocol to remap VIPS in event of failure

18


Sample OpenStack HA Architecture

19

.… Availability

Zone 1

Dedicated Firewalls

BO

ND

0 BO

ND

1BO

ND

0 BO

ND

1

Controller

API Services

API & HorizonCinder API

Nova SchedulerKeystone

GlanceRabbitMQ

MYSQL

Chef ServerRecipes

Load Balancers

Redundant Network Switches

StorageEMC, NetApp, or

SolidfireVols

BO

ND

2

Redundant Network Switches

Inside LB VLAN

Storage Network (private)

Fixed Network (private)

Compute 1

KVM

G2

G1

G4

G3

Compute N

KVM

G6

G5

G7

BO

ND

0 BO

ND

1

BO

ND

2

.… Availability

Zone 2

BO

ND

0 BO

ND

1

BO

ND

2

Compute 1

KVM

Compute N

KVM

G16

G15

G17

BO

ND

0 BO

ND

1

BO

ND

2

BO

ND

0 BO

ND

1

BO

ND

2

G12

G11

G14

G13

BO

ND

2

Controller

API Services

API & HorizonCinder API

Nova SchedulerKeystone

GlanceRabbitMQ

MYSQL

Chef ServerRecipes


HA on OpenStack

20


HA on OpenStack

21


22

Comparison

Database Replication

method

Strengths Weakness/Limita

tions

Keepalived/HAPro

xy/VRRP

Works on MySQL

master-master

replication

Simple to

implement and

understand.

Works for any

storage system.

Master-master

replication does

not work beyond 2

nodes.

Pacemaker/Coros

ync/DRBD

Mirroring on Block

Devices

Well tested More complex to

setup. Split Brain

possibility

Galera Based on write-

set Replication

(wsrep)

No Slave lag Needs at least 3

nodes. Relatively

new.

Others MySQL Cluster,

RHCS with

DAS/SAN storage

Well tested More complex

setup.


Resources and Contact

23


Resources

• OpenStack

– openstack.org

– launchpad.net/openstack

– #openstack

– #openstack on webchat.freenode.net

• OpenStack HA

– http://docs.openstack.org/trunk/openstack-ha/openstack-ha-guide-trunk.pdf

– https://github.com/rcbops-cookbooks/

• MySQL HA

– http://www.mysql.com/why-mysql/white-papers/mysql-high-availability-drbd-configuration-deployment-guide/

– http://dev.mysql.com/doc/refman/5.7/en/ha-overview.html

– https://www.hastexo.com/

– http://www.drbd.org/

http://www.openstack.org

https://launchpad.net/openstack

http://docs.openstack.org/trunk/openstack-ha/openstack-ha-guide-trunk.pdf

http://www.mysql.com/why-mysql/white-papers/mysql-high-availability-drbd-configuration-deployment-guide/

http://dev.mysql.com/doc/refman/5.7/en/ha-overview.html

https://www.hastexo.com/

RACKSPACE® HOSTING | WWW.RACKSPACE.COM 2

5RACKSPACE® HOSTING | WWW.RACKSPACE.COM

For More Information

You can reach me at:

Kenneth Hui

Open Cloud Architect

Rackspace

E-mail: [email protected]

Twitter: @hui_kenneth

Blog: http://cloudarchitectmusings.com

mailto:[email protected]

http://cloudarchitectmusings.com

26

RACKSPACE® HOSTING | 5000 WALZEM ROAD | SAN ANTONIO, TX 78218

US SALES: 1-800-961-2888 | US SUPPORT: 1-800-961-4454 | WWW.RACKSPACE.COM

RACKSPACE® HOSTING | © RACKSPACE US, INC. | RACKSPACE® AND FANATICAL SUPPORT® ARE SERVICE MARKS OF RACKSPACE US, INC. REGISTERED IN TH E UNITED STATES AND OTHER COUNTRIES. | WWW.RACKSPACE.COM

Technology

Openstackha 130925132534-phpapp02