A Continuous Delivery Safety Net for Databases

© 2013 IBM Corporation

A Continuous Delivery Safety Net for Databases

2 © 2015 IBM Corporation

Speaking Today

• Eric is a DevOps Evangelist with IBM where he helps customers get the most out of their build, deploy and release processes.

• Today he works with customers and industry leaders to find the best ways of adopting continuous delivery and DevOps.

• Eric is the author of Application Deploy and Release for Dummies.

• @EricMinick

Eric Minick

• Yaniv is the Co-Founder and CTO of DBmaestro, the leading provider of DevOps for Database solutions which enable control of database development and deployment.

• Yaniv has spent the last years raising awareness about the challenges around database development and deployment, and how to support database Continuous Delivery.

• @DBmaestro

Yaniv

Yehuda


Why Waterfall Fails


Continuous Delivery helps us learn faster

Automated flow from Build to “ready for prod”

Push button release to production

Lots of feedback Emphasis on always

shippable

Themes


“Classic” CD

build dev

test

system

test UAT sign-off staging prod


The Build Pipeline

build dev

test

system


for (env in testEnvironments) {

deploy( build, env );

runTests (env.testType, env);

}


Composite apps: many tiers & components



An Application might have

dozens of components



An Application might have

dozens of components

Delivered by Different Teams



Which build does “Login” test?


Build pipelines in composite applications

build dev

test

system


build dev

test

system


build dev

test

system


build dev

test

system


build dev

test

system


build dev

test

system


build dev

test

system


build dev

test

system


build dev

test

system


build dev

test

system


build dev

test

system


build dev

test

system



Combine Versioned Stuff into Versioned App

• Manage a set of tested component versions as a single unit

• Guarantee the validated versions of related components are

deployed together

• Ease of promoting applications between environments

1

1

2

3

UAT Stage PROD

Promote Promote

Web

Mid. Code

Mid. Config

DB

Snapshot


Expanding “All the Stuff”

Systems of Record

– z/OS

– iSeries

Mobile

– iOS / Android

– Mobile Testing

Infrastructure

– Treating environment as part of Application


Combine Versioned Stuff into Versioned App

• Manage a set of tested component versions as a single unit

• Guarantee the validated versions of related components are

deployed together

• Ease of promoting applications between environments

1

1

2

3

UAT Stage PROD

Promote Promote

Web

Mid. Code

Mid. Config

DB

Snapshot


• Old adage but true

• The database is often neglected and therefore can become the weakest link

• Manual processes

• Database/Code Silos exist…

• Don’t always communicate effectively

• Need to follow same procedures & best practices

• Essential from a compliance and business point of view

• Should be the strongest link

The Weakest Link In a Chain ???


Recently Conducted DBmaestro Survey – Over 150 Companies

– Over 200 participants

Why is that happening?

Continuous Delivery Moving Ahead!


Why Continuous Delivery?


Why?

What about the database?


Mistrust…

So why not move forward?


Mistrust…


What is the problem?

Root Causes for issues: – Challenging manual version control process

– Static deployments code overrides

– Dynamic deployments unaware of version control

– No release automation red-flags – don’t know when to stop the line…


Mistrust in version control…

Version Control Process

(file based)

Development Process

Check-Out Script

Modify Script

Get updated Script from

DB

Check-In Script

Compile Script in DB

Debug Script in DB



(file based)

Development Process

Check-Out Script

Modify Script


DB

Check-In Script


Debug Script in DB

A

A’




(file based)

Development Process

Check-Out Script

Modify Script


DB

Check-In Script


Debug Script in DB

?

? ?

?

A

A’


No single source of truth…


Scripts & Version Control

Challenges…

– Code-overrides

– Working on the wrong revisions

– Scripts do not always find their way to the version control solution

– Out of process updates go unnoticed

– Hard to locate outdated update scripts

Playing safe? what we really need:

– The actual code of the object

– The upgrade script

– A roll-back script

Scripts

– Hard to test in their entirely (holistically)

– Hard to test due to colliding dependencies

– Need to run in a specific order…

– Much harder to deal with project scope changes


1.1 1.1.1 1.1 1.2 1.3 1.4 1.5 1.6 1.7

Mistrust in static scripting automation…

Int QA Stage Prod

Database Deploy Script

Environment

Re-Base (due to

defects)

Dev

Dev

Dev

Model

1.1 1.2

1.2 1.3

1.3 1.4

1.1 1.1 1.4 1.7 ? 1.1.1


1.1 1.1.1 1.1 1.2 1.3 1.4 1.5 1.6 1.7

Int QA Stage Prod


Environment

Re-Base (due to

defects)

Dev

Dev

Dev

Model

1.1 1.2

1.2 1.3

1.3 1.4

1.1 1.1 1.4 1.7

1.1 1.2

1.2 1.3

1.3 1.4

? 1.1.1



1.1 1.1.1 1.1 1.2 1.3 1.4 1.5 1.6 1.7

Int QA Stage Prod


Environment

Re-Base (due to

defects)

Dev

Dev

Dev

Model

1.1 1.2

1.2 1.3

1.3 1.4

1.4 1.5

1.5 1.6

1.6 1.7

1.1 1.1 1.4 1.7

1.1 1.2

1.2 1.3

1.3 1.4

1.4 1.5

1.5 1.6

1.6 1.7

? 1.1.1



1.1 1.1.1 1.1 1.2 1.3 1.4 1.5 1.6 1.7

Int QA Stage Prod


Environment

Re-Base (due to

defects)

Dev

Dev

Dev

Model

1.1 1.2

1.2 1.3

1.3 1.4

1.4 1.5

1.5 1.6

1.6 1.7

1.1 1.1 1.4 1.7

1.1 1.2

1.2 1.3

1.3 1.4

1.4 1.5

1.5 1.6

1.6 1.7

Out of

Process

Change

? 1.1.1



X 1.1 1.1.1 1.1 1.2 1.3 1.4 1.5 1.6 1.7

Int QA Stage Prod


Environment

Re-Base (due to

defects)

Dev

Dev

Dev

Model

1.1 1.2

1.2 1.3

1.3 1.4

1.4 1.5

1.5 1.6

1.6 1.7

1.1 1.1 1.4 1.7

1.1 1.2

1.2 1.3

1.3 1.4

1.4 1.5

1.5 1.6

1.6 1.7

1.1 1.2

1.2 1.3

1.3 1.4

1.4 1.5

1.5 1.6

1.6 1.7

Out of

Process

Change

X

X

X

X

X

? 1.1.1

X



Scripts are static…

Scripts, unless super sophisticated:

– Unaware of changes made in the target environment

– Time passed from their coding to the time they are run

– Potentially overriding production hot-fixes or work done in parallel by another team

Content changes are very hard to manage

– Metadata & lookup content does not practically fit into the VC

– In most cases they are simply not managed


Gaining Trust!

Coordinated Process Traceability

Start in the Beginning

No Out-of-Process Changes

Impact Analysis

Automation

Task Based Development

Well Defined Processes


Version Control - One Enforced Process


Dealing with challenges…

Integrated Database Version Control process – Leverage proven version control best practices

• Forcing check in & out for changes

• Labels

• etc..

– No code-overrides

– Always working with the correct revision

– All changes are documented

– Always know who did what, when, why and from where

– No out-of-process changes

– Supporting structure, code and content

No time spent on manual coding of the change scripts


Bonus points – dealing with scope changes

Task based development… – Correlate each database change with a change request

• Task ID

• Work Item

• Trouble Ticket

• CR

• etc…

…enables task based deployments – Partial deployments (a feature, a collection of bugs, etc…)

– Scope changes easily synced between code and database


For deployment automation we need:

Repeatability & Safety

Using tools make sense …


Build & Deploy On Demand

1.1 1.2 1.3 1.4 1.5 1.6 1.7

Int QA Stage Prod


Environment

Re-Base (due to

defects)

Dev

Dev

Dev

Model

1.1 1.2

1.2 1.3

1.3 1.4

1.1 1.4

1.1 1.1 1.1 1.4 1.7 1.1.1 1.7 1.1.1 1.7



1.1 1.2 1.3 1.4 1.5 1.6 1.7

Int QA Stage Prod


Environment

Re-Base (due to

defects)

Dev

Dev

Dev

Model

1.1 1.2

1.2 1.3

1.3 1.4

1.4 1.5

1.5 1.6

1.6 1.7

1.1 1.4

1.4 1.7

1.1 1.1 1.1 1.4 1.7 1.1.1 1.7 1.1.1 1.7



1.1 1.2 1.3 1.4 1.5 1.6 1.7

Int QA Stage Prod


Environment

Re-Base (due to

defects)

Dev

Dev

Dev

Model

1.1 1.2

1.2 1.3

1.3 1.4

1.4 1.5

1.5 1.6

1.6 1.7

1.1 1.4

1.4 1.7

1.1 1.1 1.1 1.4 1.7

Out of

Process

Change

1.1.1 1.7 1.1.1 1.7



1.1 1.2 1.3 1.4 1.5 1.6 1.7

Int QA Stage Prod


Environment

Re-Base (due to

defects)

Dev

Dev

Dev

Model

1.1 1.2

1.2 1.3

1.3 1.4

1.4 1.5

1.5 1.6

1.6 1.7

1.1 1.4

1.4 1.7

1.1.1 1.7

1.1 1.1 1.1 1.4 1.7

File Based

Version

Control

Out of

Process

Change

1.1.1 1.7 1.1.1 1.7



1.1 1.2 1.3 1.4 1.5 1.6 1.7

*

Int QA Stage Prod


Environment

Re-Base (due to

defects)

Dev

Dev

Dev

Model

1.1 1.2

1.2 1.3

1.3 1.4

1.4 1.5

1.5 1.6

1.6 1.7

1.1 1.4

1.4 1.7

1.1.1 1.7

1.1 1.1 1.1 1.4 1.7

File Based

Version

Control

Out of

Process

Change

1.1.1 1.7 1.1.1 1.7


Using tools

Test cases using compare & sync tools:

An index exists in source (QA) but not in target (Production)

What should we do? Add the index or not?


Compare & Sync tools

Safe to automate?

Sure… (?)


Compare & Sync tools

Safe to automate?

No. Requires manual inspection…


Safe?

Simple, right? NO! we are going to BREAK production without even

knowing…


Why break production???

A compare & sync tool: Is unaware of any changes that occurred before the time it ran

Has no knowledge of changes that took place at the target environment

Does not leverage version control for more information

Unable to deal with conflicts & merges between different teams

Requires manual inspection

Requires detailed knowledge regarding each change as part of the process

Mistrust AGAIN…

So…no automation… as we fear automating problems into production and a major risk!!!


We need to leverage version control into deployment decisions…


Safety Net For Deployment Automation

Static script based deployment


Deploying Changes if Needed

Development Baseline

Previous Label /

Production Golden Copy

Production

If we had the index in the baseline =>

we should take it down from production…

(Deploy Change)


Or Protecting Target Environment…

Development Baseline

Previous Label /

Production Golden Copy

Production

BUT… If no index in baseline =>

we should protect the NEW index on production!!!

(Protect Target)


Deployment Automation

And Merge!!!


Conflict Resolving – Database Code


Impact Analysis – Code & Content


Impact Analysis! not Damage Control…


Safety Net For Deployment Automation

Database Safe Deployment Automation: Leverages version control (baselines & previous revisions)

Has a flexible scope (deploy multi schema to single task or work item)

Can be run as a batch process (repeatable & consistent)

Integrates to ALM (labels, CRs, Continuous Integration & Delivery)

Deals with conflicts & merges to match code agility

Can raise red flags to stop the line…

if requires human intervention


What is DBmaestro TeamWork?

Database Enforced Change Management solution + Database version control + Enforce best practices + Plugs into the ALM (change request, tickets & work items)

+ Database merge & change impact analysis + Know who can do what, where, when & why

DevOps Solution for databases + Baseline aware deployment automation, rollback &

recovery + Reduce database deployment issues + Plugs into release management & Continuous Delivery


IBM UrbanCode Deploy Visibility and automated control of your application deployment process

Mobile App

Team

Middleware

team Backend Systems of

Record

• Manage application

components and versions

• Manage configurations across

all environments

• Offer secure ‘self-service’ capabilities

• Increase transparency

• Ensure governance and

compliancy


www.ibm.com/devops

http://www.ibm.com/software/rational


© Copyright IBM Corporation 2014. All rights reserved. The information contained in these materials is provided for informational purposes only, and is provided AS IS without warranty of any kind, express or implied. IBM shall not be responsible for any damages arising out of the use of, or otherwise related to, these materials. Nothing contained in these materials is intended to, nor shall have the effect of, creating any warranties or representations from IBM or its suppliers or licensors, or altering the terms and conditions of the applicable license agreement governing the use of IBM software. References in these materials to IBM products, programs, or services do not imply that they will be available in all countries in which IBM operates. Product release dates and/or capabilities referenced in these materials may change at any time at IBM’s sole discretion based on market opportunities or other factors, and are not intended to be a commitment to future product or feature availability in any way. IBM, the IBM logo, Rational, the Rational logo, and other IBM products and services are trademarks of the International Business Machines Corporation, in the United States, other countries or both. Other company, product, or service names may be trademarks or service marks of others.

www.ibm.com/devops

http://www.ibm.com/software/rational

Software

A Continuous Delivery Safety Net for Databases