The Challenges & Pitfalls of Database Continuous Delivery

#

Uri MargalitDirector of Products

The Challenges and Pitfalls of Database Continuous Delivery

#

Uri MargalitDirector of ProductsDBmaestroUri is a product executive, who brings over 15 years of experience and a proven track record in taking new ideas to the market. With a passion for understanding and translating market requirements into working products and solutions, Uri is able to effectively lead Product teams in the enterprise software, systems management. Uri earned a BA in Sciences from the Open University.

In his spare time, Uri plays chess and enjoys gardening.

.

#

The Need

#

• Doing better with less• Reacting quickly to market needs• Getting ahead of competition• Just can’t wait 6 months for that next release…

– Agile Development– Process Automation– DevOps

Agile World…

#

Continuous Delivery

Image from Wikipedia

FailPass

#

• Rapid changes– Reacting quickly to market needs– Getting ahead of competition

• Fewer changes backed out• Better collaboration• Fewer defects

• Ultimately better service • Happy customers • Profitability

Why Continuous Delivery?

#

• Team and process• Version everything• Automate your tests• Fix it, properly (no out of process changes!)• Automate your deployments• Create the deployment pipeline

Focus points

#

But…

#

Database & CD

#

• The database holds your essential information• Changes can impact the entire system• Need to be synchronized with other changes• Often overlooked

Database is a Key Component

#

• There is more to a database than SQL scripts– Schema structure– Code– Content and meta-content– Internal dependencies– …

• Ensure that changes are not made without authorization

• Ensure no out-of-process changes

Reaching Inside the Database

#

• Old adage but true• The database is often neglected and therefore can

become the weakest link• Essential from a compliance point of view• Should be the strongest link

The Weakest Link In a Chain

#

• Silos exist…• Don’t always communicate effectively• Need to share knowledge• Need to follow same procedures & best practices

Developers and DBAs

#

Real-life Tales

#

We recently had a disaster - the script in the version control was not updated and when executed in production, ran the wrong revision. That cost tens of thousands of $(an out-of-process update to QA that was not properly tracked)

DBA @ Algo Trading company

We had multiple releases to production every day. That is one release a week with multiple follow up fixes, and yet more fixes(code overrides, partial versions, wrong versions – all pushed to production)

CTO @ Credit Card company

Real-life Tales…

“

“

#

We had an incident where a trigger was not correctly implemented during a code release. a trigger from a previous build was used instead which was only detected on Tuesday morning on the first business day after our code release. this was a customer-facing application and made our team look, and feel, bad about the release.

we realized that we needed to bring more discipline and rigor to our database changes.(manual process are hard to repeat over and over without errors)

Sr. DBA @ Large USA Bank

Real-life Tales…

“

#

Challenges

#

• Manual script based version control process• Deployment tools unaware of version control• No red-flags…

Root Causes

#

Version Control Process (file based)

Two Isolated Processes

Development Process

Check-Out Script

Modify Script

Get updated Script from

DB

Check-In Script

Compile Scriptin DB

Debug Scriptin DB

?

??

?

A

A’

#

Pit falls

Scripts

#

• Challenges… – Code-overrides– Working on the wrong revisions – Scripts do not always find their way to the version

control solution– Out of process updates go unnoticed– Hard to locate outdated update scripts

Scripts & Version Control

#

1.11.21.31.41.51.61.7 1.11.11.41.7 1.1.1

X1.11.1.1

Scripts… Build Once Deploy Many

Int QA Stage Prod

Database Deploy Script

Environment

Re-Base (due to defects)

DevDev

DevModel

1.1 1.2

1.2 1.3

1.3 1.4

1.4 1.5

1.5 1.61.6 1.7

1.1 1.2

1.2 1.3

1.3 1.4

1.4 1.5

1.5 1.61.6 1.7

1.1 1.2

1.2 1.3

1.3 1.4

1.4 1.5

1.5 1.6

1.6 1.7

Out of Process Change

X

X

X

X

X

?

X

#

• Scripts, unless super sophisticated:– Unaware of changes made in the target environment– Time passed from their coding to the time they are run– Potentially overriding production hot-fixes & work done

in parallel by another team

• Content changes are very hard to manage– Metadata & lookup content does not practically fit into

the VC– In most cases they are simply not managed

Scripts are Static…

#

Dealing With Challenges

#

Version Control – 1 Enforced Process

#

• Integrated Version Control process– Leverage proven version control best practices

• Forcing check in & out for changes• Labels• etc..

– No code-overrides– Always working with the correct revision– All changes are documented

Dealing With Challenges…

#

• Integrated Version Control process– Always know who did what, when, why and from where– No out-of-process changes

– Supporting structure, code and content

• No time spent on manual coding of the change scripts

Dealing With Challenges…

#

• Correlate each database change with a change request – Task ID– Work Item– Trouble Ticket– CR– etc…

Bonus Points – Task Base Development

#

• Partial deployments (a feature, a collection of bugs, etc…)

• Scope changes easily synced between code and database

Bonus Points – Task Base Deployments

#

Change Policy Enforcement

#

Change Policy Enforcement

#

For deployment automation we need:

Automatic, repeatable, safe process for deployment automation

Using tools make sense …

#

1.11.21.31.41.51.61.7 1.11.11.41.7 1.1.11.11.1.1

Build & Deploy On Demand

Int QA Stage Prod

Database Deploy Script

Environment

Re-Base (due to defects)

DevDev

DevModel

1.1 1.2

1.2 1.3

1.3 1.4

1.4 1.5

1.5 1.61.6 1.7

1.1 1.4

1.4 1.7

1.1.1 1.7

Out of Process Change

1.7

*

1.7

#

Automation Requirements

#

Test cases using compare & sync tools:

An index exists in source (QA) but not in target (Production)What should we do? Add the index or not?

Deployment Automation

#

Compare & Sync tools

Safe to automate?Sure…

#

• An index exists in Target (Production) but not in source (QA)

• What should we do? Drop the index or not?

Deployment Automation

#

Compare & Sync tools

Safe to automate?No. Requires manual inspection…

#

Safe?

Simple, right?

NO! we are going to BREAK production without even knowing…

#

• A compare & sync tool:– Is unaware of any changes that occurred before the

time it ran– Has no knowledge of changes that took place at the

target environment– Does not leverage version control for more information– Unable to deal with conflicts & merges between different

teams

Why Breaking Production???

#

• A compare & sync tool:– … Requires manual inspection – Requires detailed knowledge regarding each change as

part of the process

So… no automation… as automating problems into production is a major risk!!!

Why Breaking Production???

#

Behind the Scenes

#

We need to leverage version control into deployment decisions…

#

Source vs. Target

Action

= No Action

≠ ?

Source vs. Baseline

Target vs. Baseline

Action

= = No Action

≠ = Deploy Changes

= ≠ Protect Target

≠ ≠ Merge Changes

You do not have all of the information

With baselines aware analysis the unknown is now known

Simple Compare & Sync Baseline aware Analysis

Deployment Automation - Safety Net

# 45

Protecting Target Environment

Development BaselinePrevious Label /

Production Golden Copy

Production

No index in baseline =>

we should protect the NEW index on production!!!

(Protect Target)

# 46

Protecting Target Environment

Development BaselinePrevious Label /

Production Golden Copy

Production

BUT… If we had the index in the baseline =>

we should take it down from production…

(Deploy Change)

#

Deployment Automation

And Merge???

#

Conflict Resolving – Meta Data/Content

#

Impact Analysis

#

• Database Safe Automation:– Leverages version control– Has a flexible scope– Can be run as a batch process– Integrates to ALM (labels, CRs, Continuous Integration

& Delivery)– Deals with conflicts & merges to match code agility

Can raise red flags to stop the line…

if requires human intervention

Safety Net For Deployment Automation

#

• Database Enforced Change Management solution– Database version control– Enforce best practices– Plugs into the ALM (change request, tickets & work

items)– Database merge & change impact analysis– Know who can do what, where, when & why

Summary - DBmaestro TeamWork

#

• DevOps Solution for databases – Baseline aware deployment automation, rollback &

recovery– Reduce database deployment issues– Plugs into release management & Continuous Delivery

Summary - DBmaestro TeamWork

#

• Database Enforced Version Control – Enforced Check Out/In – Labels– Rollback/Undo – Audit trail reports

• Database Impact Analysis – Utilizes version control repository information – Baseline aware analysis

Summary - How?

#

• Database deployment automation– API for automation– Baselines – Conflict resolution – Customized business logic

Summary - How?

##

Thank you!Uri Margaliturim@DBmaestro.com@UriMargalit