A Continuous Delivery Safety Net for Databases

Speaking Today

• Eric is a DevOps Evangelist with IBM where he helps customers get the most out of their build, deploy and release processes.

• Today he works with customers and industry leaders to find the best ways of adopting continuous delivery and DevOps.

• Eric is the author of Application Deploy and Release for Dummies.

• @EricMinick

Eric Minick

• Yaniv is the Co-Founder and CTO of DBmaestro, the leading provider of DevOps for Database solutions which enable control of database development and deployment.

• Yaniv has spent the last years raising awareness about the challenges around database development and deployment, and how to support database Continuous Delivery.

• @DBmaestro

Yehuda

Why Waterfall Fails

Continuous Delivery helps us learn faster

Automated flow from Build to “ready for prod”

Push button release to production

Lots of feedback Emphasis on always

shippable

Themes

“Classic” CD

build dev

system

test UAT sign-off staging prod

The Build Pipeline

build dev

system

for (env in testEnvironments) {

deploy( build, env );

runTests (env.testType, env);

Composite apps: many tiers & components

An Application might have

dozens of components

An Application might have

dozens of components

Delivered by Different Teams

Which build does “Login” test?

Build pipelines in composite applications

build dev

system

build dev

system

build dev

system

build dev

system

build dev

system

build dev

system

build dev

system

build dev

system

build dev

system

build dev

system

build dev

system

build dev

system

Combine Versioned Stuff into Versioned App

• Manage a set of tested component versions as a single unit

• Guarantee the validated versions of related components are

deployed together

• Ease of promoting applications between environments

UAT Stage PROD

Promote Promote

Mid. Code

Mid. Config

Snapshot

Expanding “All the Stuff”

Systems of Record

– z/OS

– iSeries

Mobile

– iOS / Android

– Mobile Testing

Infrastructure

– Treating environment as part of Application

Combine Versioned Stuff into Versioned App

• Manage a set of tested component versions as a single unit

• Guarantee the validated versions of related components are

deployed together

• Ease of promoting applications between environments

UAT Stage PROD

Promote Promote

Mid. Code

Mid. Config

Snapshot

• Old adage but true

• The database is often neglected and therefore can become the weakest link

• Manual processes

• Database/Code Silos exist…

• Don’t always communicate effectively

• Need to follow same procedures & best practices

• Essential from a compliance and business point of view

• Should be the strongest link

The Weakest Link In a Chain ???

Recently Conducted DBmaestro Survey – Over 150 Companies

– Over 200 participants

Why is that happening?

Continuous Delivery Moving Ahead!

Why Continuous Delivery?

What about the database?

Mistrust…

So why not move forward?

Mistrust…

What is the problem?

Root Causes for issues: – Challenging manual version control process

– Static deployments code overrides

– Dynamic deployments unaware of version control

– No release automation red-flags – don’t know when to stop the line…

Mistrust in version control…

Version Control Process

(file based)

Development Process

Check-Out Script

Modify Script

Get updated Script from

Check-In Script

Compile Script in DB

Debug Script in DB

(file based)

Development Process

Check-Out Script

Modify Script

Check-In Script

Debug Script in DB

(file based)

Development Process

Check-Out Script

Modify Script

Check-In Script

Debug Script in DB

No single source of truth…

Scripts & Version Control

Challenges…

– Code-overrides

– Working on the wrong revisions

– Scripts do not always find their way to the version control solution

– Out of process updates go unnoticed

– Hard to locate outdated update scripts

Playing safe? what we really need:

– The actual code of the object

– The upgrade script

– A roll-back script

Scripts

– Hard to test in their entirely (holistically)

– Hard to test due to colliding dependencies

– Need to run in a specific order…

– Much harder to deal with project scope changes

1.1 1.1.1 1.1 1.2 1.3 1.4 1.5 1.6 1.7

Mistrust in static scripting automation…

Int QA Stage Prod

Database Deploy Script

Environment

Re-Base (due to

defects)

1.1 1.2

1.2 1.3

1.3 1.4

1.1 1.1 1.4 1.7 ? 1.1.1

1.1 1.1.1 1.1 1.2 1.3 1.4 1.5 1.6 1.7

Int QA Stage Prod

Environment

Re-Base (due to

defects)

1.1 1.2

1.2 1.3

1.3 1.4

1.1 1.1 1.4 1.7

1.1 1.2

1.2 1.3

1.3 1.4

? 1.1.1

1.1 1.1.1 1.1 1.2 1.3 1.4 1.5 1.6 1.7

Int QA Stage Prod

Environment

Re-Base (due to

defects)

1.1 1.2

1.2 1.3

1.3 1.4

1.4 1.5

1.5 1.6

1.6 1.7

1.1 1.1 1.4 1.7

1.1 1.2

1.2 1.3

1.3 1.4

1.4 1.5

1.5 1.6

1.6 1.7

? 1.1.1

1.1 1.1.1 1.1 1.2 1.3 1.4 1.5 1.6 1.7

Int QA Stage Prod

Environment

Re-Base (due to

defects)

1.1 1.2

1.2 1.3

1.3 1.4

1.4 1.5

1.5 1.6

1.6 1.7

1.1 1.1 1.4 1.7

1.1 1.2

1.2 1.3

1.3 1.4

1.4 1.5

1.5 1.6

1.6 1.7

Out of

Process

Change

? 1.1.1

X 1.1 1.1.1 1.1 1.2 1.3 1.4 1.5 1.6 1.7

Int QA Stage Prod

Environment

Re-Base (due to

defects)

1.1 1.2

1.2 1.3

1.3 1.4

1.4 1.5

1.5 1.6

1.6 1.7

1.1 1.1 1.4 1.7

1.1 1.2

1.2 1.3

1.3 1.4

1.4 1.5

1.5 1.6

1.6 1.7

1.1 1.2

1.2 1.3

1.3 1.4

1.4 1.5

1.5 1.6

1.6 1.7

Out of

Process

Change

? 1.1.1

Scripts are static…

Scripts, unless super sophisticated:

– Unaware of changes made in the target environment

– Time passed from their coding to the time they are run

– Potentially overriding production hot-fixes or work done in parallel by another team

Content changes are very hard to manage

– Metadata & lookup content does not practically fit into the VC

– In most cases they are simply not managed

Gaining Trust!

Coordinated Process Traceability

Start in the Beginning

No Out-of-Process Changes

Impact Analysis

Automation

Task Based Development

Well Defined Processes

Version Control - One Enforced Process

Dealing with challenges…

Integrated Database Version Control process – Leverage proven version control best practices

• Forcing check in & out for changes

• Labels

• etc..

– No code-overrides

– Always working with the correct revision

– All changes are documented

– Always know who did what, when, why and from where

– No out-of-process changes

– Supporting structure, code and content

No time spent on manual coding of the change scripts

Bonus points – dealing with scope changes

Task based development… – Correlate each database change with a change request

• Task ID

• Work Item

• Trouble Ticket

• CR

• etc…

…enables task based deployments – Partial deployments (a feature, a collection of bugs, etc…)

– Scope changes easily synced between code and database

For deployment automation we need:

Repeatability & Safety

Using tools make sense …

Build & Deploy On Demand

1.1 1.2 1.3 1.4 1.5 1.6 1.7

Int QA Stage Prod

Environment

Re-Base (due to

defects)

1.1 1.2

1.2 1.3

1.3 1.4

1.1 1.4

1.1 1.1 1.1 1.4 1.7 1.1.1 1.7 1.1.1 1.7

1.1 1.2 1.3 1.4 1.5 1.6 1.7

Int QA Stage Prod

Environment

Re-Base (due to

defects)

1.1 1.2

1.2 1.3

1.3 1.4

1.4 1.5

1.5 1.6

1.6 1.7

1.1 1.4

1.4 1.7

1.1 1.1 1.1 1.4 1.7 1.1.1 1.7 1.1.1 1.7

1.1 1.2 1.3 1.4 1.5 1.6 1.7

Int QA Stage Prod

Environment

Re-Base (due to

defects)

1.1 1.2

1.2 1.3

1.3 1.4

1.4 1.5

1.5 1.6

1.6 1.7

1.1 1.4

1.4 1.7

1.1 1.1 1.1 1.4 1.7

Out of

Process

Change

1.1.1 1.7 1.1.1 1.7

1.1 1.2 1.3 1.4 1.5 1.6 1.7

Int QA Stage Prod

Environment

Re-Base (due to

defects)

1.1 1.2

1.2 1.3

1.3 1.4

1.4 1.5

1.5 1.6

1.6 1.7

1.1 1.4

1.4 1.7

1.1.1 1.7

1.1 1.1 1.1 1.4 1.7

File Based

Version

Control

Out of

Process

Change

1.1.1 1.7 1.1.1 1.7

1.1 1.2 1.3 1.4 1.5 1.6 1.7

Int QA Stage Prod

Environment

Re-Base (due to

defects)

1.1 1.2

1.2 1.3

1.3 1.4

1.4 1.5

1.5 1.6

1.6 1.7

1.1 1.4

1.4 1.7

1.1.1 1.7

1.1 1.1 1.1 1.4 1.7

File Based

Version

Control

Out of

Process

Change

1.1.1 1.7 1.1.1 1.7

Using tools

Test cases using compare & sync tools:

An index exists in source (QA) but not in target (Production)

What should we do? Add the index or not?

Compare & Sync tools

Safe to automate?

Sure… (?)

Compare & Sync tools

Safe to automate?

No. Requires manual inspection…

Simple, right? NO! we are going to BREAK production without even

knowing…

Why break production???

A compare & sync tool: Is unaware of any changes that occurred before the time it ran

Has no knowledge of changes that took place at the target environment

Does not leverage version control for more information

Unable to deal with conflicts & merges between different teams

Requires manual inspection

Requires detailed knowledge regarding each change as part of the process

Mistrust AGAIN…

So…no automation… as we fear automating problems into production and a major risk!!!

We need to leverage version control into deployment decisions…

Safety Net For Deployment Automation

Static script based deployment

Deploying Changes if Needed

Development Baseline

Previous Label /

Production Golden Copy

Production

If we had the index in the baseline =>

we should take it down from production…

(Deploy Change)

Or Protecting Target Environment…

Development Baseline

Previous Label /

Production Golden Copy

Production

BUT… If no index in baseline =>

we should protect the NEW index on production!!!

(Protect Target)

Deployment Automation

And Merge!!!

Conflict Resolving – Database Code

Impact Analysis – Code & Content

Impact Analysis! not Damage Control…

Safety Net For Deployment Automation

Database Safe Deployment Automation: Leverages version control (baselines & previous revisions)

Has a flexible scope (deploy multi schema to single task or work item)

Can be run as a batch process (repeatable & consistent)

Integrates to ALM (labels, CRs, Continuous Integration & Delivery)

Deals with conflicts & merges to match code agility

Can raise red flags to stop the line…

if requires human intervention

What is DBmaestro TeamWork?

Database Enforced Change Management solution + Database version control + Enforce best practices + Plugs into the ALM (change request, tickets & work items)

+ Database merge & change impact analysis + Know who can do what, where, when & why

DevOps Solution for databases + Baseline aware deployment automation, rollback &

recovery + Reduce database deployment issues + Plugs into release management & Continuous Delivery

IBM UrbanCode Deploy Visibility and automated control of your application deployment process

Mobile App

Middleware

team Backend Systems of

Record

• Manage application

components and versions

• Manage configurations across

all environments

• Offer secure ‘self-service’ capabilities

• Increase transparency

• Ensure governance and

compliancy

www.ibm.com/devops

© Copyright IBM Corporation 2014. All rights reserved. The information contained in these materials is provided for informational purposes only, and is provided AS IS without warranty of any kind, express or implied. IBM shall not be responsible for any damages arising out of the use of, or otherwise related to, these materials. Nothing contained in these materials is intended to, nor shall have the effect of, creating any warranties or representations from IBM or its suppliers or licensors, or altering the terms and conditions of the applicable license agreement governing the use of IBM software. References in these materials to IBM products, programs, or services do not imply that they will be available in all countries in which IBM operates. Product release dates and/or capabilities referenced in these materials may change at any time at IBM’s sole discretion based on market opportunities or other factors, and are not intended to be a commitment to future product or feature availability in any way. IBM, the IBM logo, Rational, the Rational logo, and other IBM products and services are trademarks of the International Business Machines Corporation, in the United States, other countries or both. Other company, product, or service names may be trademarks or service marks of others.

www.ibm.com/devops

A Continuous Delivery Safety Net for Databases

Software

Continuous Delivery requires Continuous Communication

Testing in a continuous delivery world - continuous delivery Amsterdam meetup

Continuous Delivery for databases - microservices, team structures, and Conway's Law - develop:BBC 2014

Practicing Continuous Integration and Continuous Delivery ... · Practicing Continuous Integration and Continuous Delivery on AWS Accelerating Software Delivery with DevOps June 2017

Appian Continuous Delivery Framework - Incessant …...Appian Continuous Delivery Framework iACD Incessant’s iACD solution provides a Continuous Delivery Framework enabling organizations

Continuous Integration & Continuous Delivery

Continuous Deployment: Beyond Continuous Delivery

Definitions Background/History Continuous Delivery › How to practice Continuous Delivery Continuous Integration Continuous Integration Tools

Drive Continuous Delivery With Continuous Testing

Continuous Delivery - Jfokus · Continuous Delivery from Zero to Hero AXEL FONTAINE @axelfontaine axelfontaine.com Entrepreneur - Public Speaker - Continuous Delivery Expert

Continuous Delivery of Continuous Delivery · 08/07/2016 · 7/8/2016 Continuous Delivery of Continuous Delivery ... JENKINS PUPPET ... Puppet Docker-Module erzeugt OS-Service

Continuous delivery for databases - Bristol DevOps Edition

Continuous testing for continuous delivery

Continuous delivery continuous integration 0.3

Flusso Continuous Integration & Continuous Delivery

Db2 12 Continuous Delivery New challenges for deployment · 16-03-2018 · Continuous Delivery and DevOps 7 Agile development requires near-time delivery • Continuous Delivery

Continuous Delivery Seminars...DevOps, Continuous Delivery and Automation DevOps (Philosophy) Continuous Delivery (Practice) Application Release Automation (ARA) Application Release

Continuous Delivery Vortrag @ Continuous Lifecycle #ConLi2013

Continuous integration for databases using Red Gate tools€¦ · 4 Continuous integration for databases using Red Gate tools source control test build continuous integration

Acceptance Testing for Continuous Delivery › events › 2015 › acceptance...self.store.createBook(“Continuous Delivery”); order = self.store.placeOrder(book=“Continuous Delivery")