+

Database Change Management* as a service

* Also known as ‘patching’ or ‘refactoring’

+About me

Was …

an academic teaching Perl

Am now doing …

Perl Web Development at Net-A-Porter

(learning) Devops at Net-A-Porter

Teaching at Geekuni

(Andrew Solomon)

+Motivation

Next app release coming

App has new features

Needs different DB structure

Can’t just trash and install DB like an app

Data is inconvenient

+Principles

Backward compatibility

Old and new app versions are ok with the refactored DB

DB is versioned

You can quickly determine which patches have been applied

App won’t be run if the DB is not compatible

Write a rollback for every DB change

For safe refactoring

http://www.amazon.co.uk/Continuous-Delivery-Deployment-

Automation-Addison-Wesley/dp/0321601912

+

The N

et-

a-P

ort

er

Way

Old

sch

ool

+The Net-a-Porter Way

Put refactor code into My-App’s code base

Approach A: Run when installing the My-App RPM

Approach B: Run when starting My-App

Log the patches run in the database

Old school

+The Net-a-Porter Way

Put refactor.pl into My-App’s code base

PROBLEM: A slightly different refactor.pl for each app

PROBLEM: refactor.pl has extra CPAN dependencies

Approach A: Run when installing the My-App RPM

Approach B: Run when starting My-App

Log the patches run in the database

Old school

+The Net-a-Porter Way

Put refactor.pl into My-App’s code base

PROBLEM: A slightly different refactor.pl for each app

PROBLEM: refactor.pl has extra CPAN dependencies

Approach A: Run when installing the My-App RPM

Approach B: Run when starting My-App

PROBLEM: Multiple My-App servers per DB

Log the patches run in the database

Old school

+The Service Way

+Available Refactoring Tools

Flyway

Liquibase

DB Deploy

+Winner: Liquibase

Big user base

Bought by Datical

Has branching and

More features than Flyway

DB Deploy uses Apache ANT

Overview

Liquibase

+Liquibase

Input:

An XML file listing SQL files to run

Special comments in the SQL files for tagging

Output:

‘databasechangelog’ tables

Which patches have been run

‘databasechangeloglock’

Ensure only one refactoring process!

How it works

+Liquibase

<databaseChangeLog …>

<include file="1.0.0/000-create_schema.sql”

relativeToChangelogFile="true"/>

<include file="1.0.0/010-populate.sql”

relativeToChangelogFile="true"/>

<changeSet author="a.solomon" id="tag-1.0.0">

<tagDatabase tag="1.0.0"/>

</changeSet>

</databaseChangeLog>

Example XML

+Liquibase

--liquibase formatted sql

--changeset a.solomon:1.0.1-000

BEGIN;

CREATE TABLE public.foobar (

id INTEGER PRIMARY KEY,

name TEXT UNIQUE NOT NULL

);

ALTER table public.foobar OWNER to magpie;

--ROLLBACK DROP TABLE public.foobar;

COMMIT;

Example SQL

+Liquibase

/usr/bin/liquibase \

--changeLogFile=changeset.xml \

--url="jdbc:postgresql://db1.myapp.com:5432/myapp-db1" \

--username=liquibase –password=foobar update

Example Update

+Liquibase

/usr/bin/liquibase \

--changeLogFile=changeset.xml \

--url="jdbc:postgresql://db1.myapp.com:5432/myapp-db1" \

--username=liquibase –password=foobar rollback 1.0.0

Example Rollback

+Liquibase

author | filename | md5sum | tag

a.solomon | 1.0.1/000-create_schema.sql | 7:49c8 | 1.0.1

Example databasechangelog

+Liquibase::Git

Perl module and script (soon on CPAN?)

A perl script

Clones git repo of the app

Looks in a directory of patches

Calls liquibase

What is it?

Liquibase::Git

+Liquibase::GitExample

$ liquibase-git --username liquibase \

--password foobar \

--db mydb-db1 \

--hostname db1.myapp.com \--git-repo https://github.com/nap/myapp.git \

--git-changeset-dir db/db1 \

--git-identifier master \

--db-type postgresql \

--changeset-file changeset.xml

+NAP::Liquibase

Most of Liquibase::Git’s parameters are retrieved from the Puppet

DB

Just reads a new file from My-App – the list of DBs to refactor

Getting the Puppet to help

# in the puppet hieradata

$ cat myapp.com.yaml

liquibase:

databases:

- db: 'myapp-db1'

- db: 'myapp-db2’

# ask mco for myapp’s dbhost

$ nap-liquibase --db-host db1.myapp.com \

--app-git-repo https:://github.com/nap/myapp.git \

--app-git-identifier master

NAP::Liquibase

Jenkins Deploy-oid

mco ping … -F system_db=liquibase

+Appendix… managing DBs outside the Puppet estate

Cases where the box is being managed differently with puppet…

Jenkins Continuous Integration Test Server

Developer’s Box

Option 1: Emulate Liquibase on an empty DB – 5 lines

Option 2: Call liquibase-git to update a DB dump

+Conclusion

Don’t write new DB refactoring

code for each app

Don’t assume you’ll only have one

app server per DB

Don’t deploy DB refactoring code

on the app’s host

Treat it as a

service

+But wait!!! There’s more…

Ilmari told me about it

It’s a Perl project of David E. Wheeler / theory

Based on sane principles: http://sqitch.org/

Sqitch is to Liquibase

Is what

Git is to CVS

The switch to sqitch will be almost invisible at the architecture level

Only difference is that the developers get LOCAL application of the patches on the dev box with a simple command-line interface

+

Questions?

… or grab me at the pub.