Bootstrapping Made Easy Matt Ullmer · Puppet Clusters and Roles Snapshot Server Smartstack Bootstrapping Made Easy: Tying it all together. Cluster Main Monitoring Roles Master Reporting

Matt [email protected]

Bootstrapping Made Easy

Yelp’s MissionConnecting people with great

local businesses.

Four DBAs: Jenni, Susanne, Patrick, myself

Hundreds of serversFully managed, configured, and maintained by usPercona Server 5.6Ubuntu 14.04

Who We Are

Basic Architecture

Tying it all together

Global Expansion


Linux

Apache

MySQL

Python

Basic Architecture

Multiple clusters

Multiple services

Multiple datastores

Basic Architecture: Scaling Out

Time to Automate

Basic Architecture: When To Automate

Basic Architecture


Global Expansion


What is the goal of the system?Self ManagingSelf DeployingSelf TestingSelf Healing

Bootstrapping Made Easy: Tying it all together

New hosts automaticallyPick a source host in their clusterPull a backup in real timeReparent to an intermediate master databaseService Discovery




Pick a source host in their clusterPuppet Clusters and RolesSnapshot ServerSmartstack


ClusterMainMonitoring

RolesMasterReportingRead

Bootstrapping Made Easy: Puppet Clusters and Roles

profile_mysql::instance::valid_roles: - master - read - reporting

profile_mysql::instance::backup_roles: - master - read - reporting

profile_mysql::read_only::master: '0'profile_mysql::read_only::read: '1'profile_mysql::read_only::reporting: '1'

How we classify environments

EcosystemSuperregionRegionHabitat

Bootstrapping Made Easy: Puppet Clusters and Roles



Snapshot ServerListens on a portCompares incoming connections to expected clusterTriggers an xtrabackup_to_stream encrypted with KMS


class BackupStreamingRequestHandler(BaseRequestHandler): def handle(self): cluster_name = self.request.recv(4096).rstrip() if cluster_name != get_cluster(): return

os.nice(10)

self.request.setsockopt(socket.SOL_SOCKET, SO_PRIORITY, 1)

fileno = self.request.fileno()

os.dup2(fileno, sys.stdin.fileno()) os.dup2(fileno, sys.stdout.fileno())

try: os.execv('/usr/bin/xtrabackup_to_stream', ['xtrabackup_to_stream', '3306']) except ProcessExecutionError as e: if e.returncode == 255: print("This error typically occurs when qpress is called with a blank or corrupted stream") print("Try netcatting the snapshot server on the remote host to debug further") raise



SmartstackZookeeper, Nerve, SynapseAdvertise thing 'X' at port 'Y' across the infrastructure on a common IP

Susanne is speaking about smartstack later today @ 3:50 in Ballroom D


New hosts automatically Pick a source host in their clusterPull a backup in real timeReparent to an intermediate master databaseService Discovery


Pull a backup in real time BackupsBackup TestingStreaming


Xtrabackup

Low impact

Compression

Streaming

Backups

xtrabackup.cnf:port = 3306socket = /mysqld/mysqld.sockdatadir = /mysqld/data/innodb_log_group_home_dir = /mysqld/data/open_files_limit = 2048

Wrapping innobackupex:/usr/bin/innobackupex --defaults-extra-file=xtrabackup.cnf --stream=xbstream --parallel=1 /xtrabackup/tmp --slave-info | nice pigz -p 4 > $OUTFILE

Crontab:45 0 * * * root local_backup_wrapper_script45 10 2-30/2 * * root encryption_and_s3_offsite_script



Scan for backups, restore, and verify restorability

Backup Testing



ssh -A db1 "sudo -H /usr/bin/innobackupex --stream=xbstream --parallel=1 --no-version-check --slave-info /databases/tmp | pigz -p 10" 2> xtrabackup.err | pv --name input -c | pigz -d | pv --name output -c | sudo xbstream -C /databases/mysql -x

Xtrabackup_to_streamAutomated version of the previous streamingAdvertises itself with a snapshot server

Streaming Xtrabackup

compressed_contents() { set +e innobackupex \ --defaults-file='/nail/etc/xtrabackup.cnf' \ --user xtrabackup \ --parallel=8 \ --stream=xbstream \ --slave-info \ --no-version-check \ --tmpdir "$tempdir" \ "$tempdir" 2> innobackupex_stderr innobackupex_retcode="$?" set -e echo "$innobackupex_retcode" > innobackupex_retcode xbstream -c innobackupex_stderr innobackupex_retcode return "$innobackupex_retcode"}

2016-07-20 09:41:43,377 INFO:root:Connecting to snapshot server on 192.168.0.12016-07-20 09:41:43,378 INFO:root:Unpacking backup

New hosts automatically Pick a source host in their clusterPull a backup in real timeReparent to an intermediate master databaseService Discovery


Reparent to an intermediate master databaseIntermediate MasterReparenting


Intermediate MasterSmartstack DiscoverySaves BandwidthEases ReparentingDon't bootstrap from this host


Reparent to an intermediate master database Intermediate MasterReparenting


ReparentingDowntime sensu alertingCHANGE MASTER TO…


2016-12-15 11:23:27 33529 [Note] Event Scheduler: Loaded 0 events2016-12-15 11:23:27 33529 [Note] /usr/sbin/mysqld: ready for connections.Version: '5.6.32-78.1-log' socket: '/nail/databases/mysql/mysqld.sock' port: 3306 Percona Server (GPL), Yelp Release 12016-12-15 11:30:16 33529 [Note] 'CHANGE MASTER TO executed'.



Service DiscoveryHealthchecksAuto TopologyJoin the Load Balancer






We now have all the pieces


Basic Architecture


Global Expansion


Snapshotting in a new region

How do you launch databases

How do you do service discovery

Global Expansion




Global Expansion

main: extra_advertise: region:old_region: - region:new_region

Cross region advertising unlocks automatic bootstrapping

Global Expansion




Global Expansion

How do you getdatabases setup?

Hand rolled?

Named?

Global Expansion

Advice: no host is special

Global Expansion

Amazon ASG

Schedules

Terraform

Global Expansion




Global Expansion

Discovery Service

Global Expansion

Jenkins builds new topology

Jenkins checks the topology is valid

Jenkins pushes it to all services

Global Expansion

Four DBAsMatt UllmerSusanne Lehmann (25 April - 3:50 PM - 4:40 PM @ Ballroom D)Patrick King (26 April - 1:00 PM - 1:50 PM @ Ballroom G)Jenni Snyder

Yelp DBAs

www.yelp.com/careers/

We're Hiring!

Questions

@YelpEngineering

fb.com/YelpEngineers

engineeringblog.yelp.com

github.com/yelp

Documents

Bootstrapping Made Easy Matt Ullmer · Puppet Clusters and Roles Snapshot Server Smartstack Bootstrapping Made Easy: Tying it all together. Cluster Main Monitoring Roles Master Reporting