Continuous Delivery for DatabasesSQL Relay – Cardiff - 2014

Presented by: James SmithCo-Founder, DevOpsGuys

ABOUT ME

Co-founder @ DevOpsGuys

Two decades building & delivering enterprise web systems

Helped many, many companies implement Continuous Delivery practices

Found High Quality Belgian Beer in late nineties!

Who is this guy?

ABOUT DEVOPSGUYS

We provideweb application management

for customers, giving them access to the expertise necessary for building, launching, maintaining & optimising applications

allowing them to accelerate time-to-market

and to focus on adding value to their business

Who are these guys?

WHY?Why is this important?

WHY?Why is this important?

BUSINESS HAS CHANGED

SPEED MATTERS

8

9

DEPLOYMENT RATES

300 Deployments / Year

50-60 Deployments / Day10+ Deployments / Day

Every 11.6 seconds

10

HIGH PERFORMERS HIGH PERFORMING IT

ORGANISATIONS DEPLOY 30 TIMES MORE FREQUENTLY, WITH 50% FEWER FAILURES WITH 8000X

FASTER LEAD TIMES THAN THEIR PEERS.

11

HIGH PERFORMERS THEY ARE ALSO TWO TIMES MORE LIKELY TO

EXCEED PROFITABILITY, MARKET SHARE & PRODUCTIVITY GOALS

THEY EXPERIENCE 50% HIGHER MARKET CAPITALIZATION GROWTH OVER 3 YEARS

HOW?How have things changed?

12

20102012

2001 (1998)

2001 (and earlier)

DEV HAS CHANGED

WE KNOW AGILE WORKS

15

Plan

Code

Build

Test

Release

Deploy

Operate

CONTINUOUS X?

Continuous Integration

Continuous Delivery

Continuous Deployment

CONTINUOUS DELIVERYWhat is it?

REGULAR, RAPID, RELIABLE AND CONTROLLED DELIVERY OF WORKING SOFTWARE SYSTEMS INTO PRODUCTION

Matthew Skelton

REGULAR, RAPID, RELIABLE AND CONTROLLED DELIVERY OF WORKING SOFTWARE SYSTEMS INTO PRODUCTION

Matthew Skelton

Nhan Ngo, a QA engineer at Spotify

20

CONTINUOUS DELIVERY1. Build Management & CI2. Environments & Deployment3. Release Management & Compliance4. Testing5. Data Management

22

BUT DATABASE CHANGE IS SCARY

Photo Credit: http://cache.lego.com/r/www/r/movie/-/media/franchises/the%20lego%20movie/explore/downloads/wallpapers/lego_wps_1600_emmet.jpg?l.r=-1410608028

NO, THIS IS SCARY

Photo Credit: ?

SO WE MINIMISE RISK

Photo Credit: https://ideas.lego.com/projects/62456

WE DEFEND THE THREAT

Photo Credit: "Rumrunner" via Compfight cc

26

WE STOP CHANGE

Photo Credit: s.kosoris via Compfight cc

27

CUSTOMERS GET SAD

Photo Credit: Kalexanderson via Compfight cc

28

THE BUSINESS GETS ANGRY

Photo Credit: powerpig builds via Compfight cc

29

AND IT GET THE BLAME!

Photo Credit: bobsfever via Compfight cc

30

YES BUT…. Dev are shipping application changes – frequently

Ops are creating servers - automatically

WHY IS THERE SO MUCH LEGO IN YOUR SLIDE DECK?

Perfectly normal question

LEGO = DEVOPS

Rapid proto-typing & experimentationBuilding blocks – no right or wrong wayPromotes collaborationStrong cultural appealSmall batch sizesVisibly measureableWe’ve even automated it! Manufacturing Zenon Mindstorms

Danish phrase leg godt, which means "play well".

WHY?Why is database automation so hard?

!= CTRL C, CTRL V Database deployment is not copying and replacing.

It is the transformation from a previous version to the next version while preserving data integrity.

Deploying database change is hard

Deploying database change frequently is even harder

WHAT?What are the common challenges?

#1: VERSION CONTROL

Photo Credit: ntr23 via Compfight cc

VERSION CONTROL Database changes not under VCS Worse – Changes not “always” committed to VCSCommunication of changeLiving in “the sea of ‘branch/merge’ filth”

38

#2: AGAINST THE FLOW

Photo Credit: slang589 via Compfight cc

Photo Credit: slang589 via Compfight cc

39

AGAINST THE FLOW

DEV PRODUCTION

DEV PRODUCTION

CODE

DATA

40

#3: COMPLEXITY

Photo Credit: L@go via Compfight cc

COMPLEXITYDependenceCentralisation

#4: DATA VOLUME

Photo Credit: http://www.calgaryherald.com/news/calgary/Gallery+LEGO+KidsFest/9848406/story.html

DATA VOLUMETransit timeHistorical data – no archivingOne of the biggest impacts on Cycle time

#5: CONWAY'S LAW

Photo Credit: kirk_arts via Compfight cc

45

CONWAY'S LAW "Any organization that designs a system (defined more broadly

here than just information systems) will inevitably produce a design whose structure is a copy of the organization's communication

structure.“ - Conway, 1968

CONWAY'S LAWIs (was) a database really needed? 3 Tiers anyone?

Centralised vs Decentralised?

Formalised [change] control

HOW?So what are some of the solutions?

#1: GET UNDER CONTROL

Photo Credit: RHiNO NEAL via Compfight cc

VERSION CONTROLIt’s code – it should be in VCS!It’s code – it should be in VCS!It’s code – it should be in VCS!

Schema & Static/Reference Data

Reverse engineer existing schema & reference data

50

#2: INTEGRATE CONTINUOUSLY

Photo Credit: ....Tim via Compfight cc

51

CONTINUOUS INTEGRATION Automate your build steps (Full vs Incremental)

Build a library of manual tests

Automate your tests

52

#3: AUTOMATE TESTING

Photo Credit: s3aphotography via Compfight cc

53

TEST, TEST, TESTSelect the right tests for each stage;Unit testing Integration TestingDeployment ValidationBehaviour Validation

Determine the right data for testingDo you need it all?

54

#4: AUTOMATION

Photo Credit: pasukaru76 via Compfight cc

55

AUTOMATE ALL THE THINGSTestingDeploymentBack-up and more importantly restoreArchivingRollback

56

#5: MICROSERVICES

57

AS A SERVICE Treat your database as a service

Contracts – that you do not break (lightly)

58

RE-ARCHITECT Split data along sensible partitions

Polyglot persistence

59

POLYGLOT PERSISTENCE

60

#6: CHANGE YOUR WAYS

Photo Credit: Stéfan via Compfight cc

61

IT’S CULTURE

Technology is only half of the storyDev’s must work with DBA’s (no silo’s)Management must think of operations as part of developmentDeployment is part of developmentData retention is part of developmentFail faster, but fail safely

TOOLINGCan tools help?

62

THE PIPELINESource

Control

Integration

Test Deploy

64

IN REALITYDevelopment

Plan Code Build Test Release Deploy Operate

Operations

SOURCE CONTROL Source control is the foundation of Continuous Integration

Plus many, many more

66

CONTINUOUS INTEGRATION Continuous Integration is the common orchestration point

67

DEPLOYMENT Repeatable and consistent deployments every time.

MONITORING Continuous delivery elevates the need for monitoring in production

THE TRUTH Continuous Delivery relies on having 2 basic things;

1. Version Control

2. Automation

70

AND FINALLY…Customers see results and new features more quickly.Shorter feedback cycles increases our ability to learn.Improve the whole system.Reduce firefighting.Everyone wins!

71

EVERYBODY WINS!

Photo Credit: Kalexanderson via Compfight cc

72

THAT’S ALL FOLKS

Photo Credit: Walter Benson via Compfight cc