View
421
Download
2
Category
Tags:
Preview:
DESCRIPTION
Net-a-Porter has embarked on the mission of separating database refactoring from code deployment. The solution we've come up with is "refactoring as a service" and a soon-to-be released Perl module which drives it. I'll explain how Liquibase, our Git repositories, Puppet and Jenkins all fit together to make database refactoring easy and deployment safe and roll-backable. I'll also tell you how I just discovered Sqitch as a possible replacement for Liquibase. ... and all within 20 minutes!
Citation preview
+
Database Change Management* as a service
* Also known as ‘patching’ or ‘refactoring’
+About me
Was …
an academic teaching Perl
Am now doing …
Perl Web Development at Net-A-Porter
(learning) Devops at Net-A-Porter
Teaching at Geekuni
(Andrew Solomon)
+Motivation
Next app release coming
App has new features
Needs different DB structure
Can’t just trash and install DB like an app
Data is inconvenient
+Principles
Backward compatibility
Old and new app versions are ok with the refactored DB
DB is versioned
You can quickly determine which patches have been applied
App won’t be run if the DB is not compatible
Write a rollback for every DB change
For safe refactoring
http://www.amazon.co.uk/Continuous-Delivery-Deployment-
Automation-Addison-Wesley/dp/0321601912
+
The N
et-
a-P
ort
er
Way
Old
sch
ool
+The Net-a-Porter Way
Put refactor code into My-App’s code base
Approach A: Run when installing the My-App RPM
Approach B: Run when starting My-App
Log the patches run in the database
Old school
+The Net-a-Porter Way
Put refactor.pl into My-App’s code base
PROBLEM: A slightly different refactor.pl for each app
PROBLEM: refactor.pl has extra CPAN dependencies
Approach A: Run when installing the My-App RPM
Approach B: Run when starting My-App
Log the patches run in the database
Old school
+The Net-a-Porter Way
Put refactor.pl into My-App’s code base
PROBLEM: A slightly different refactor.pl for each app
PROBLEM: refactor.pl has extra CPAN dependencies
Approach A: Run when installing the My-App RPM
Approach B: Run when starting My-App
PROBLEM: Multiple My-App servers per DB
Log the patches run in the database
Old school
+The Service Way
+Available Refactoring Tools
Flyway
Liquibase
DB Deploy
+Winner: Liquibase
Big user base
Bought by Datical
Has branching and
More features than Flyway
DB Deploy uses Apache ANT
Overview
Liquibase
+Liquibase
Input:
An XML file listing SQL files to run
Special comments in the SQL files for tagging
Output:
‘databasechangelog’ tables
Which patches have been run
‘databasechangeloglock’
Ensure only one refactoring process!
How it works
+Liquibase
<databaseChangeLog …>
<include file="1.0.0/000-create_schema.sql”
relativeToChangelogFile="true"/>
<include file="1.0.0/010-populate.sql”
relativeToChangelogFile="true"/>
<changeSet author="a.solomon" id="tag-1.0.0">
<tagDatabase tag="1.0.0"/>
</changeSet>
</databaseChangeLog>
Example XML
+Liquibase
--liquibase formatted sql
--changeset a.solomon:1.0.1-000
BEGIN;
CREATE TABLE public.foobar (
id INTEGER PRIMARY KEY,
name TEXT UNIQUE NOT NULL
);
ALTER table public.foobar OWNER to magpie;
--ROLLBACK DROP TABLE public.foobar;
COMMIT;
Example SQL
+Liquibase
/usr/bin/liquibase \
--changeLogFile=changeset.xml \
--url="jdbc:postgresql://db1.myapp.com:5432/myapp-db1" \
--username=liquibase –password=foobar update
Example Update
+Liquibase
/usr/bin/liquibase \
--changeLogFile=changeset.xml \
--url="jdbc:postgresql://db1.myapp.com:5432/myapp-db1" \
--username=liquibase –password=foobar rollback 1.0.0
Example Rollback
+Liquibase
author | filename | md5sum | tag
a.solomon | 1.0.1/000-create_schema.sql | 7:49c8 | 1.0.1
Example databasechangelog
+Liquibase::Git
Perl module and script (soon on CPAN?)
A perl script
Clones git repo of the app
Looks in a directory of patches
Calls liquibase
What is it?
Liquibase::Git
+Liquibase::GitExample
$ liquibase-git --username liquibase \
--password foobar \
--db mydb-db1 \
--hostname db1.myapp.com \--git-repo https://github.com/nap/myapp.git \
--git-changeset-dir db/db1 \
--git-identifier master \
--db-type postgresql \
--changeset-file changeset.xml
+NAP::Liquibase
Most of Liquibase::Git’s parameters are retrieved from the Puppet
DB
Just reads a new file from My-App – the list of DBs to refactor
Getting the Puppet to help
# in the puppet hieradata
$ cat myapp.com.yaml
liquibase:
databases:
- db: 'myapp-db1'
- db: 'myapp-db2’
# ask mco for myapp’s dbhost
$ nap-liquibase --db-host db1.myapp.com \
--app-git-repo https:://github.com/nap/myapp.git \
--app-git-identifier master
NAP::Liquibase
Jenkins Deploy-oid
mco ping … -F system_db=liquibase
+Appendix… managing DBs outside the Puppet estate
Cases where the box is being managed differently with puppet…
Jenkins Continuous Integration Test Server
Developer’s Box
Option 1: Emulate Liquibase on an empty DB – 5 lines
Option 2: Call liquibase-git to update a DB dump
+Conclusion
Don’t write new DB refactoring
code for each app
Don’t assume you’ll only have one
app server per DB
Don’t deploy DB refactoring code
on the app’s host
Treat it as a
service
+But wait!!! There’s more…
Ilmari told me about it
It’s a Perl project of David E. Wheeler / theory
Based on sane principles: http://sqitch.org/
Sqitch is to Liquibase
Is what
Git is to CVS
The switch to sqitch will be almost invisible at the architecture level
Only difference is that the developers get LOCAL application of the patches on the dev box with a simple command-line interface
+
Questions?
… or grab me at the pub.
Recommended