HA in OpenStack service - meetup #9

High Available controller core servicesVIETOPENSTACK 9th MEETUP

Ha Noi, 07/05/2016

Click to edit Master title styleOpenStack release: LibertyOS version: Ubuntu 14.04.3 LTS, Trusty TahrVMware ESXi 5.5

Lab information

2High Available controller core services

Click to edit Master title styleHigh availability concepts


System downtimeOccurs when a user-facing service is unavailable beyond a specified maximum amount of time.Data lossAccidental deletion or destruction of data.

Stateless servicenova-api, nova-conductor, glance-api, keystone-api, neutron-api and nova-scheduler,…Stateful serviceOpenStack database and message queue. Making stateful services highly available can depend on whether you choose an active/passive or active/active configuration.

One uses a cluster manager such as Pacemaker to co-ordinate the actions of the various services across a set of machinesHAProxy provides a fast and reliable HTTP reverse proxy and load balancer for TCP or HTTP applications

Click to edit Master title style

KeystoneGlanceNovaNeutronL3HAMySQLRabbitMQ

Controller core services to be HA



Each instance of HAProxy configures its front end to accept connections only from the virtual IP (VIP) address and to terminate them as a list of all instances of the corresponding service under load balancing, such as any OpenStack API service.

This makes the instances of HAProxy act independently and fail over transparently together with the network endpoints (VIP addresses) failover and, therefore, shares the same SLA

HAproxy service


Load distribution API isolation Simplified process for adding/removing of nodes Enhanced failure detection

Click to edit Master title stylePacemaker to manage VIP & Haproxy service.

Pacemaker relies on the Corosync messaging layer for reliable cluster communications.

Corosync implements the Totem single-ring ordering and membership protocol. It also provides UDP and InfiniBand based messaging, quorum, and cluster membership to Pacemaker.

Pacemaker relies on resource agents (RAs), scripts that encapsulate the knowledge of how to start, stop, and check the health of each application managed by the cluster

Pacemaker cluster stack & corosync service


Click to edit Master title styleArchitecture


Click to edit Master title styleUse Galera Cluster to provide replication between instances of the database on controller nodes.

Galera Cluster is a synchronous multi-master database cluster, based on MySQL and the InnoDB storage engine.

It is a high-availability service that provides high system uptime, no data loss, and scalability for growth.

Mariadb Galera cluster


Click to edit Master title styleMemcached is a general-purpose distributed memory caching system. It is used to speed up dynamic database-driven websites by caching data and objects in RAM to reduce the number of times an external data source must be read.

Memcached is a memory cache demon that can be used by most OpenStack services to store ephemeral data, such as tokens.

Access to memcached is not handled by HAproxy because replicated access is currently only in an experimental state. Instead OpenStack services must be supplied with the full list of hosts running memcached.

Memcached


Click to edit Master title styleRabbitMQ is AMQP broker to coordinate the execution of jobs entered into the system (communication service)

Configure RabbitMQ for HA queuesBuild a cluster of RabbitMQ nodes to construct a RabbitMQ broker, which is a logical grouping of several Erlang nodes.Mirrored queues in RabbitMQ improve the availability of service since it is resilient to failures.

Configure OpenStack services to use Rabbit HA queuesWe have to configure the OpenStack components to use at least two RabbitMQ nodes.

RabbitMQ cluster



1, Stop Pacemaker or Corosync service

2, Stop 1 of 3 nodes

3, Stop 2 of 3 nodes

4, Other recommended

Failover & failback senarios



VPN to lab

Demo


Click to edit Master title styleThe end


Source: http://docs.openstack.org/ha-guide/Support team: VietOpenStack

Presentations & Public Speaking

HA in OpenStack service - meetup #9