Upload
perforce
View
1.435
Download
1
Tags:
Embed Size (px)
DESCRIPTION
Practicing database Continuous Delivery saves time and money and prevents downtime in production. However, dealing with automating database deployments is tricky. In this presentation, you will learn how to overcome the challenges and potential pitfalls of database Continuous Delivery and proven best practices for automating database changes.
Citation preview
#
Uri MargalitDirector of Products
The Challenges and Pitfalls of Database Continuous Delivery
#
Uri MargalitDirector of ProductsDBmaestroUri is a product executive, who brings over 15 years of experience and a proven track record in taking new ideas to the market. With a passion for understanding and translating market requirements into working products and solutions, Uri is able to effectively lead Product teams in the enterprise software, systems management. Uri earned a BA in Sciences from the Open University.
In his spare time, Uri plays chess and enjoys gardening.
.
#
The Need
#
• Doing better with less• Reacting quickly to market needs• Getting ahead of competition• Just can’t wait 6 months for that next release…
– Agile Development– Process Automation– DevOps
Agile World…
#
Continuous Delivery
Image from Wikipedia
FailPass
#
• Rapid changes– Reacting quickly to market needs– Getting ahead of competition
• Fewer changes backed out• Better collaboration• Fewer defects
• Ultimately better service • Happy customers • Profitability
Why Continuous Delivery?
#
• Team and process• Version everything• Automate your tests• Fix it, properly (no out of process changes!)• Automate your deployments• Create the deployment pipeline
Focus points
#
But…
#
Database & CD
#
• The database holds your essential information• Changes can impact the entire system• Need to be synchronized with other changes• Often overlooked
Database is a Key Component
#
• There is more to a database than SQL scripts– Schema structure– Code– Content and meta-content– Internal dependencies– …
• Ensure that changes are not made without authorization
• Ensure no out-of-process changes
Reaching Inside the Database
#
• Old adage but true• The database is often neglected and therefore can
become the weakest link• Essential from a compliance point of view• Should be the strongest link
The Weakest Link In a Chain
#
• Silos exist…• Don’t always communicate effectively• Need to share knowledge• Need to follow same procedures & best practices
Developers and DBAs
#
Real-life Tales
#
We recently had a disaster - the script in the version control was not updated and when executed in production, ran the wrong revision. That cost tens of thousands of $(an out-of-process update to QA that was not properly tracked)
DBA @ Algo Trading company
We had multiple releases to production every day. That is one release a week with multiple follow up fixes, and yet more fixes(code overrides, partial versions, wrong versions – all pushed to production)
CTO @ Credit Card company
Real-life Tales…
“
“
#
We had an incident where a trigger was not correctly implemented during a code release. a trigger from a previous build was used instead which was only detected on Tuesday morning on the first business day after our code release. this was a customer-facing application and made our team look, and feel, bad about the release.
we realized that we needed to bring more discipline and rigor to our database changes.(manual process are hard to repeat over and over without errors)
Sr. DBA @ Large USA Bank
Real-life Tales…
“
#
Challenges
#
• Manual script based version control process• Deployment tools unaware of version control• No red-flags…
Root Causes
#
Version Control Process (file based)
Two Isolated Processes
Development Process
Check-Out Script
Modify Script
Get updated Script from
DB
Check-In Script
Compile Scriptin DB
Debug Scriptin DB
?
??
?
A
A’
#
Pit falls
Scripts
#
• Challenges… – Code-overrides– Working on the wrong revisions – Scripts do not always find their way to the version
control solution– Out of process updates go unnoticed– Hard to locate outdated update scripts
Scripts & Version Control
#
1.11.21.31.41.51.61.7 1.11.11.41.7 1.1.1
X1.11.1.1
Scripts… Build Once Deploy Many
Int QA Stage Prod
Database Deploy Script
Environment
Re-Base (due to defects)
DevDev
DevModel
1.1 1.2
1.2 1.3
1.3 1.4
1.4 1.5
1.5 1.61.6 1.7
1.1 1.2
1.2 1.3
1.3 1.4
1.4 1.5
1.5 1.61.6 1.7
1.1 1.2
1.2 1.3
1.3 1.4
1.4 1.5
1.5 1.6
1.6 1.7
Out of Process Change
X
X
X
X
X
?
X
#
• Scripts, unless super sophisticated:– Unaware of changes made in the target environment– Time passed from their coding to the time they are run– Potentially overriding production hot-fixes & work done
in parallel by another team
• Content changes are very hard to manage– Metadata & lookup content does not practically fit into
the VC– In most cases they are simply not managed
Scripts are Static…
#
Dealing With Challenges
#
Version Control – 1 Enforced Process
#
• Integrated Version Control process– Leverage proven version control best practices
• Forcing check in & out for changes• Labels• etc..
– No code-overrides– Always working with the correct revision– All changes are documented
Dealing With Challenges…
#
• Integrated Version Control process– Always know who did what, when, why and from where– No out-of-process changes
– Supporting structure, code and content
• No time spent on manual coding of the change scripts
Dealing With Challenges…
#
• Correlate each database change with a change request – Task ID– Work Item– Trouble Ticket– CR– etc…
Bonus Points – Task Base Development
#
• Partial deployments (a feature, a collection of bugs, etc…)
• Scope changes easily synced between code and database
Bonus Points – Task Base Deployments
#
Change Policy Enforcement
#
Change Policy Enforcement
#
For deployment automation we need:
Automatic, repeatable, safe process for deployment automation
Using tools make sense …
#
1.11.21.31.41.51.61.7 1.11.11.41.7 1.1.11.11.1.1
Build & Deploy On Demand
Int QA Stage Prod
Database Deploy Script
Environment
Re-Base (due to defects)
DevDev
DevModel
1.1 1.2
1.2 1.3
1.3 1.4
1.4 1.5
1.5 1.61.6 1.7
1.1 1.4
1.4 1.7
1.1.1 1.7
Out of Process Change
1.7
*
1.7
#
Automation Requirements
#
Test cases using compare & sync tools:
An index exists in source (QA) but not in target (Production)What should we do? Add the index or not?
Deployment Automation
#
Compare & Sync tools
Safe to automate?Sure…
#
• An index exists in Target (Production) but not in source (QA)
• What should we do? Drop the index or not?
Deployment Automation
#
Compare & Sync tools
Safe to automate?No. Requires manual inspection…
#
Safe?
Simple, right?
NO! we are going to BREAK production without even knowing…
#
• A compare & sync tool:– Is unaware of any changes that occurred before the
time it ran– Has no knowledge of changes that took place at the
target environment– Does not leverage version control for more information– Unable to deal with conflicts & merges between different
teams
Why Breaking Production???
#
• A compare & sync tool:– … Requires manual inspection – Requires detailed knowledge regarding each change as
part of the process
So… no automation… as automating problems into production is a major risk!!!
Why Breaking Production???
#
Behind the Scenes
#
We need to leverage version control into deployment decisions…
#
Source vs. Target
Action
= No Action
≠ ?
Source vs. Baseline
Target vs. Baseline
Action
= = No Action
≠ = Deploy Changes
= ≠ Protect Target
≠ ≠ Merge Changes
You do not have all of the information
With baselines aware analysis the unknown is now known
Simple Compare & Sync Baseline aware Analysis
Deployment Automation - Safety Net
# 45
Protecting Target Environment
Development BaselinePrevious Label /
Production Golden Copy
Production
No index in baseline =>
we should protect the NEW index on production!!!
(Protect Target)
# 46
Protecting Target Environment
Development BaselinePrevious Label /
Production Golden Copy
Production
BUT… If we had the index in the baseline =>
we should take it down from production…
(Deploy Change)
#
Deployment Automation
And Merge???
#
Conflict Resolving – Meta Data/Content
#
Impact Analysis
#
• Database Safe Automation:– Leverages version control– Has a flexible scope– Can be run as a batch process– Integrates to ALM (labels, CRs, Continuous Integration
& Delivery)– Deals with conflicts & merges to match code agility
Can raise red flags to stop the line…
if requires human intervention
Safety Net For Deployment Automation
#
• Database Enforced Change Management solution– Database version control– Enforce best practices– Plugs into the ALM (change request, tickets & work
items)– Database merge & change impact analysis– Know who can do what, where, when & why
Summary - DBmaestro TeamWork
#
• DevOps Solution for databases – Baseline aware deployment automation, rollback &
recovery– Reduce database deployment issues– Plugs into release management & Continuous Delivery
Summary - DBmaestro TeamWork
#
• Database Enforced Version Control – Enforced Check Out/In – Labels– Rollback/Undo – Audit trail reports
• Database Impact Analysis – Utilizes version control repository information – Baseline aware analysis
Summary - How?
#
• Database deployment automation– API for automation– Baselines – Conflict resolution – Customized business logic
Summary - How?
##
Thank you!Uri [email protected]@UriMargalit