Upload
roberto-polli
View
325
Download
5
Embed Size (px)
Citation preview
Scaling MySQL with pythonEuroPython 2015, Bilbao
Roberto Polli - [email protected]
Par-Tec Spa - Rome Operation UnitP.zza S. Benedetto da Norcia, 33
00040, Pomezia RM - www.par-tec.it
20 July 2015
Roberto Polli - [email protected]
Agenda
IntroMySQLPython UtilitiesConnectors
ReplicationFailoverFabricProvisioningFabric in the Cloud
Roberto Polli - [email protected]
Who? What? Why?
Roberto Polli - Solutions Architect @ par-tec.it. Loves writing in C, Java andPython. Red Hat Certified Engineer and Virtualization Administrator.Par-Tec – Proud sponsor of this talk ;) Contributes to various FLOSS.Provides:
• IT Infrastructure & Services expertise;• Business Intelligence solutions;• Vertical Applications for the financial market.
Manage, replicate, scale MySQL is easy with python (and Docker).
Intro Roberto Polli - [email protected]
MySQL
• Single-process, multi-thread• 2-tier architecture
• Connection + SQL• Storage Backend
• Greatly improved in 5.6• More to come in 5.7: Dynamic Buffers & Multi-source replication
MySQL Roberto Polli - [email protected]
MySQL
Many Backends aka Storage Engines:• InnoDB1: ACID, MVCC2, Redo|Undo logs;• Memory: fast & volatile;• NDB: MySQL Cluster, network distributed, auto-sharded;
Replication based on changelog files (binary logs).
1default2Multi Versioning Concurrency Control
MySQL Roberto Polli - [email protected]
Manage & Use a database
Monitor:• Database size: Tables, Indexes,
Binary Logs• Replication inconsistencies• Failover
Connect:• Native Drivers
• for Python 2.6, 2.7 and 3.3• SSL & compression
• OrchestrateDownload scripts from some blog, add random queries from docs.mysql.com,Shake.
MySQL Roberto Polli - [email protected]
MySQL Utilities
A python package with utilities, drivers, fabric orchestrator
$ wget http://bit.ly/1CxNuZe$ tar xf mysql-utilities-1.6.1.tar.gz$ cd mysql-utilities-1.6.1$ python setup.py install
mysql-utilities-1.6.1|-- mysql|-- connector| |-- django| ‘-- fabric|-- fabric| |-- protocols| |-- providers| ‘-- services‘-- utilities
Python Utilities Roberto Polli - [email protected]
MySQL Utilities:
Start with mysqluc for a single entrypoint• entrypoint for all utilities• contextual help• TAB completion
Or access each utility separately.Specify server URIs as user:pass@host[:port].
Python Utilities Roberto Polli - [email protected]
MySQL Utilities
mysqluc> help utilities
Utility Description----------------- --------------------------------------------------------mysqlauditadmin audit log maintenance utilitymysqlauditgrep audit log search utility...mysqldbcompare compare databases for consistency...mysqldiskusage show disk usage for databasesmysqlfailover automatic replication health monitoring and failover
Python Utilities Roberto Polli - [email protected]
Disk Usage
A single command to show all disk usage infos (excluded system logs)$ mysqldiskusage --all --server=$SERVER...| db_name | total || akonadi | 969150919 |...Total database disk usage = 971233529 bytes or 926,24 MB...| ib_logfile0 | 67108864 || ib_logfile1 | 67108864 || ibdata1 | 44040192 |Total size of InnoDB files = 178257920 bytes or 170,00 MB
Python Utilities Roberto Polli - [email protected]
Connectors
from django.db.backends import BaseDatabaseValidation
if django.VERSION < (1, 7):from django.db import models
else:from django.core import checksfrom django.db import connection
class DatabaseValidation(BaseDatabaseValidation):if django.VERSION < (1, 7):
Connectors Roberto Polli - [email protected]
Log-based Replication: Master
Replication is asynchronous or semi − synchronous.A Master with a Unique Server ID
• produces a changelog named binary log;• assign each transaction a Global Transaction ID;• grants access to a replica user;
Optionally:• stores replication infos in ACID tables;• may track slave-updates;• waits for one slave ack on semi − synchronous replication;
Replication Roberto Polli - [email protected]
Log-based Replication: Slave
Slave• connects to the master as the replica user;• retrieves the binlog;• applies the changes;• ∀ slave ∃! master;
Optionally• forbids local changes from non-root users;
If binlogs have been purged, you need to import the master database first!Replication Roberto Polli - [email protected]
Benefits of replication
• Availability.• Scaling reads with R/W split.• Scaling reads differencing indexes.• Backup and data warehouse
strategies.
# mysqlrplshow output## Replication Topology Graphs-1.docker:3306 (MASTER)
|+--- s-3.docker:3306 - (SLAVE)|+--- s-4.docker:3306 - (SLAVE)
Replication Roberto Polli - [email protected]
mysqlreplicate
Configuring replication with one command:mysqlreplicate -b --pedantic \
--master=$MASTER --slave=$SLAVE --rpl-user=repl:rpass
# master uuid = 2cda700...# slave uuid = 7520cf0...# Checking for binary logging on master...# Granting replication access to# replication user...# Connecting slave to master...# CHANGE MASTER TO MASTER_HOST = ’172.17.0.5’,...
• preliminary checks;• create replica user on the master;• point the slave to the master;• start loading the first available
transaction in binlogs;
Replication Roberto Polli - [email protected]
Initialize new slavesBeware!
1. Avoid replicating user provisioning and other host-related [email protected]_LOG_BIN=0;
2. Binlogs are usually retained for a fixed period - eg 15days.To provision new slave after that period you must:
• start from a backup or a snapshot;• apply the binlogs.
mysqldbexport >> data.sql --server=root:pass@master --rpl-user=repl:rpass \--export=both --rpl=master --all
mysqldbimport --server=root:pass@slave data.sql
Replication Roberto Polli - [email protected]
Failover BasicsFrom replication to high availability, automagically discovering slaves.
• promote the most updated slave• reconfigure the others• disable the master• eventually fix ip-address
# Read carefully the docs of# mysqlfailover and of all# the supported parameters!mysqlfailover --master=$MASTER \--discover-slaves-login=root:password \--candidates=$SLAVE1,$SLAVE2 \--exec-before=/pre-fail.sh \--exec-after=/post-fail.sh
Failover Roberto Polli - [email protected]
Fabric Orchestrator
You can try the content of those slides downloading code and baked dockerimages from here:
git clone https://github.com/ioggstream/mysql-communitycd mysql-community/fabricdocker-compose scale fabric=1 slave=4
Let me know after the talk!
Fabric Roberto Polli - [email protected]
Fabric - HLA
A python framework for managing, replicating and scaling mysql servers.• aggregate servers in high availability groups• configure single-master replication topologies• monitor and heal failures• director for rw/split• fabric connectors (caching db topology)
Fabric Roberto Polli - [email protected]
mysqlfabric
mysqlfabric commands:• manage - setup and manage fabric service• group - administer high availability groups• server - manage, clone, provision and decommission servers• provider - manage server providers (eg. openstack, ...)
Further functionalities:• statistics, dump, role, user,• threat, snapshot, event
Fabric Roberto Polli - [email protected]
ManageSetup and administration is done viamysqlfabric manage [setup|start|ping|stop|teardown]
Configure in /etc/fabric.cfg
• listening ports for xmlrpc service• admin user (eg. fabric to manage servers)• applicative user credentials• db to store fabric data
Setup and start the servicemysqlfabric manage setupmysqlfabric manage start --daemon
Fabric Roberto Polli - [email protected]
Replication Groups
Groups are managed viamysqlfabric group [create|add|promote|activate] $GROUP
Crete an High Availability group and add a bunch of servers.
mysqlfabric group create $GROUPmysqlfabric group add $GROUP my-host-1:3306mysqlfabric group add $GROUP my-host-2:3306mysqlfabric group add $GROUP my-host-3:3306
Fabric Roberto Polli - [email protected]
Replication Groups
Promote one server as a master, eventually picking one.mysqlfabric group promote $GROUP [--slave_id=UUID]
Show the server groupmysqlfabric group lookup_servers haFabric UUID: 5ca1ab1e-a007-feed-f00d-cab3fe13249eTime-To-Live: 1server_uuid address status mode weight------------ ----------- --------- --------- ------2cda..110015 172.17.0.21 PRIMARY READ_WRITE 1.02cda..110016 172.17.0.22 SECONDARY READ_ONLY 1.02cda..110017 172.17.0.23 SECONDARY READ_ONLY 1.0
Fabric Roberto Polli - [email protected]
Replication GroupsFabric support spare servers.mysqlfabric server set_status $UUID spare
Monitoring failover and deactivating the master.mysqlfabric group activate $GROUPmysqladmin -h $MASTER shutdown
Checking group health we’ll see that...mysqlfabric group health ha
uuid is_alive status ..error status..io_error sql_error----------- -------- --------- --- --- --- --- -------- ---------2cd..110015 0 FAULTY 0 0 0 0 False False2cd..110016 1 SECONDARY 0 0 0 0 False False2cd..110017 1 PRIMARY 0 0 0 0 False False
Fabric Roberto Polli - [email protected]
Connecting programmatically
from mysql.connector import connect, fabric# Ask for a connection to the fabric server.c = connect(fabric={host: .., port: .., user: ..},
autocommit=True,database=’sample’, **sql_user)
# Fabric will point you to a suitable host# - in this case the master - of the given# groupc.set_property(mode=fabric.MODE_READWRITE, group="my-cluster-1")
Fabric Roberto Polli - [email protected]
Provisioning a new slave
Fabric uses mysqldump + mysql internally to clone a new slave.# Always reset TARGET configuration# before reinitializingmysql -e ’SET @SESSION.SQL_LOG_BIN=0;STOP SLAVE;RESET MASTER;’
Cloning doesn’t attach the server to a group, nor starts the replica.mysqlfabric server clone $GROUP $TARGET
Provisioning Roberto Polli - [email protected]
Reingesting a failed master
Reingesting a failed master is not trivial.• non-replicated transactions;• corrupt data;
Always reset it!mysqlfabric server set_status f484...110039 sparemysqlfabric server set_status f484...110039 secondary
Provisioning Roberto Polli - [email protected]
Fabric in the CloudFabric can provision new machines via eg. Openstack API.We implemented a DockerProvider: deploy containers, not machines.
# mysql.fabric.providers.dockerproviderclass MachineManager(AbstractMachineManager):
"""Manage Docker Containers."""def create(self, parameters, wait_spawning):
...def search(self, generic_filters, meta_filters):
...def destroy(self, machine_uuid):
...
Fabric in the Cloud Roberto Polli - [email protected]
Docker Provisioning# Register a provider (requires docker in http)mysqlfabric provider register mydocker
user passwordhttp://172.17.42.1:2375--provider_type=DOCKER
# Create or remove Containersmysqlfabric server create mydocker
--image=name=ioggstream/mysql-community--flavor=name=v1--meta=command="mysqld --log-bin=foo"
# List serversmysqlfabric server list docker
Fabric in the Cloud Roberto Polli - [email protected]
Next steps
Openstack interface supports machine snapshots via NovaClient.Docker volumes doesn’t support snapshots, but something is moving (in theexperimental tree).VM Snapshot alternatives require root access to docker host and a FLUSHTABLES on the source container:
• rsync on volumes• implement snapshot via docker volume plugins
Fabric in the Cloud Roberto Polli - [email protected]
Wrap Up
• Replication is easier with Fabric and MySQL 5.6• You can clone servers• Failover is just one command ahead• Don’t re-ingest failed masters (luckily;)• Try Fabric with Docker!• Play with docker volumes
Fabric in the Cloud Roberto Polli - [email protected]
That’s all folks!
Thank you for the attention!Roberto Polli - [email protected]
Fabric in the Cloud Roberto Polli - [email protected]