CoreOS automated MySQL Cluster Failover using Galera Cluster

© Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

CoreOS automated MySQL Cluster Failover using Galera Cluster

Hewlett-Packard (Enterprise) Advanced Technology Group – CloudYazz Atlas

Patrick Galbraith Spring 2015


About the speaker ●  Yazz Atlas●  HP Advanced Technology Group●  HP Cloud Services PaaS/DBaaS/

LBaaS Environment ●  Has worked at Windermere Real

Estate, Keas, Slashdot, Cobalt Group, University of Iowa, Antioch College


About the speaker ●  Patrick Galbraith ●  HP Advanced Technology Group●  Has worked at Blue Gecko,

MySQL AB, Classmates, Slashdot, Cobalt Group, US Navy, K-mart

●  MySQL projects: memcached UDFs, DBD::mysql, federated storage engine

●  Family●  Outdoors

© Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.4

Purpose of this talk – why are you here?

Basic Pre-req•  Overview of the Demo Setup

Docker•  Containers vs. Virtulization

•  Simple Docker usage

•  Clustered Docker

CoreOS•  Container optimized, stripped down Linux

distribution

•  Overview of core components of CoreOS – fleet, etcd, systemd

Galera•  Synchronous replication – excellent solution to

clustering MySQL

•  HA Proxy for failover

Discussion•  What is next

•  What about Kubernetes


Laptop Host- VirtualBox- OpenStack- Vmware- Bare metal- et al.

Master-01

Basic Pre-reqs Setup

CoreOS + etcd + confdSome of the times you will need•  CoreOS

•  Go

–  https://golang.org/doc/install

•  Etcdctl–  go get github.com/coreos/etcd/etcdctl

•  Fleetctl–  go get github.com/coreos/fleet/fleetctl

Core-01

Core-02

Core-03


What are containers?

•  Operating-system-level virtualization•  Encapsulated, hermetically sealed applications

•  Relatively isolated•  Small footprint

•  Fast to launch!

•  Portable. And did I mention, portable?!•  Use of host OS and Kernel

•  Execution consists of time to startup application in question•  LXC, Docker, Solaris Zones, BSD Jails, Parallels Virtuozzo, OpenVZ, …

Photo credit: expo_2020 - http://goo.gl/OeSQhi License CC v2.0


Containers vs. VMs


DockerWhat is Docker?

•  Set of tools for managing containers

•  Command line tool that doubles as a daemon

•  Uses Linux Kernel Features:

–  Kernel namespaces – the core ingredient to containers working: PID, IPC ,uts (what will be seen by a group of processes), mount, network and user

–  Cgroups (control groups) -- limit, account and isolate resource usage (CPU, memory, disk I/O, etc.) of process groups

•  Originally used lxc, now defaults to Libcontainer but meant for any containerization mechanism

•  Much more light weight than VMs

•  Encapsulated application containers in a relatively isolated but lightweight operating environment

•  Written in Go


Clustered Docker•  CoreOS (https://coreos.com/)

•  Kubernetes (http://kubernetes.io)

•  Mesos + Marathon (http://mesos.apache.org/) Apache project, Zookeeper, etc

•  Project Atomic (http://www.projectatomic.io/) RH/Fedora/Centos designed for running Docker

•  Docker Openstack (https://wiki.openstack.org/wiki/Docker) Hypervisor Driver for Openstack Compute

•  Swarm/Compose/Machine

•  RancherOS (http://rancher.com/rancher-os) Minimalist Linux, Docker daemon runs as PID 1 first process the kernel starts known as “System Docker,”

•  Flocker (https://clusterhq.com)

•  Spotify Helios (https://github.com/spotify/helios) Zookeeper

•  Flynn (https://flynn.io/)

•  Deis (http://deis.io)

•  Maestro (https://github.com/toscanini/maestro)

•  Shipyard (http://shipyard-project.com)

•  … others to come!


What is CoreOS?

CoreOS

• Minimalist Linux

•  Optimized for containers•  Easy to run containers

•  Service discovery, container management,

•  Docker -- Container runtime and management, though Rocket long-term•  Etcd – distributed global key value store for config data on each node

•  Fleet – Rudimentary Scheduler interacts with systemd and etcd•  Systemd – Newer Linuxi – system and service manager for Linux

•  Flannel – Networking across nodes


Connecting it all together

CoreOS•  Etcd

•  Fleetd

•  Flanneld

Photo credit: Stéfan – http://goo.gl/ye8Icd License CC v2.0


Galera secret sauce•  Galera (Percona XtraDB Cluster) container

•  FLEETCTL_ENDPOINT set to ETCD host:port

•  etcd included

•  Entrypoint script

•  Generates SQL file to create SST (single state transfer) user and clustercheck user

•  Builds cluster.cnf

•  Wsrep_node_address set to its own IP

•  Wsrep_cluster_address set using entries from etcd

•  Server-id random number

•  clustercheck on each galera node container (port 8000) to determine health and which nodes available using clustercheck user to connect

•  Ports 3306, 4444, 4567, 4568, 8000 exposed provides access to Galera cluster node containers for external and HAProxy container


Galera secret sauce – haproxy sprinkling•  HAProxy container

•  FLEETCTL_ENDPOINT set to ETCD host:port

•  Confd included

•  Confd builds haproxy.cfg from template using etcd entries for each Galera node

•  Confd loop entered to start container

•  HAProxy uses clustercheck on each node (port 8000) to determine health and which nodes available

•  Port 33006 exposed provides access to Galera cluster node containers


Laptop

Virtualbox

Master-01

Core-01

Core-02

Core-03

etcd

flanneld

Galera-1

Galera-2

Galera-3

haproxy

Sidekick-1

Sidekick-2

Sidekick-2Flannel network

Gal

era

repl

icat

ion

HAP

roxy

etcd

com

mun

icat

ion

Connecting it all together.

$> vagrant up


Galera on CoreOS


Disaster Strikes!

Don’t put all your containers on one ship (CoreOS host).

•  HA Proxy is the middleman connecting to your Galera Cluster

–  Minimal impact to connecting agents

•  Adding a new CoreOS host and redeploy the Galera container

Photo credit Paul Townsend – http://goo.gl/ImL8hX License CC v2.0


Demo time…

DEMO•  Vagrant - 1.7.2

•  Virtualbox - 4.3.26r98988

–  coreos-alpha (virtualbox, 660.0.0)

•  Github Repos

–  https://github.com/EntropyWorks/coreos-vagrant

–  https://github.com/EntropyWorks/fleet-units-galera-cluster

–  https://github.com/EntropyWorks/docker_haproxy

–  https://github.com/CaptTofu/percona_xtradb_cluster_docker (coreos branch)

•  Asciinema recordings

–  https://asciinema.org/a/19559


Thank You

Hewlett-Packard, Advanced Technology Group

Yazz Atlas [email protected]

Patrick Galbaith [email protected]