56
In Database Automation We Trust May 13 th , 2015

Geek Sync I In Database Automation We Trust

Embed Size (px)

Citation preview

Page 1: Geek Sync I In Database Automation We Trust

In Database Automation We Trust

May 13th , 2015

Page 2: Geek Sync I In Database Automation We Trust

2

About Yaniv Yehuda

Co-Founder and CTO of DBmaestro

Spent the last years raising awareness about the challenges around database development and deployment, and how to support database Continuous Delivery.

Page 3: Geek Sync I In Database Automation We Trust

3

About DBmaestro

• The leading provider of DevOps for Database• Database development and deployment automation

Page 4: Geek Sync I In Database Automation We Trust

4

Agenda

▪ Continuous Delivery is coming

▪ CD benefits

▪ But what about the database?

▪ Database version control

▪ Database deployment automation

▪ Database impact analysis

▪ Stopping the CD line

▪ Q&A

Page 5: Geek Sync I In Database Automation We Trust

5

DevOps & CD: a must for every company

▪Every business is an IT business

▪Customers demand that you deliver new features faster

− Agile Development

− Process Automation

− DevOps

▪Can’t wait 6 months for that next waterfall release…

▪ If you don’t, your competitor probably will

Page 6: Geek Sync I In Database Automation We Trust

6

Continuous *what*?

Continuous Integration

Continuous Delivery

Continuous Deployment

!

?

Page 7: Geek Sync I In Database Automation We Trust

7

What is Continuous Integration?

• Principles and practices

• Been around for a while

▪ Focus on streamlining development ■ Developers integrate code into shared repository■ Each check-in is verified■ Automated builds■ Automated tests■ High visibility – a feedback loop

▪ Easier & quicker to find problems, less back tracks => short integrations

Page 8: Geek Sync I In Database Automation We Trust

8

And Continuous Delivery?

▪ Next step after continuous integration

▪ Becoming lean, and even more Agile■ Make sure each change is releasable

• Develop-> build-> test-> move to staging-> acceptance test■ Build a process to release with a push of a button

• Deploy to production-> test production

▪ Actual deployment to production in manually actuated

=> Ensure risk mitigation and high efficiency

Page 9: Geek Sync I In Database Automation We Trust

9

Continuous Delivery

Image from Wikipedia

Fail

Pass

Page 10: Geek Sync I In Database Automation We Trust

10

Continuous Deployment

• Automating changes all the way to production■ Develop-> build-> test-> move to staging-> acceptance test->

deploy to production-> test production

• Makes sense for Facebook, Amazon, i.e. - for large SAAS solutions.■ A/B test new features■ Throttle traffic to new releases etc…

▪ Not for everyone…

Page 11: Geek Sync I In Database Automation We Trust

11

How Do I Measure Success?

• More rapid changes

• Fewer changes backed out

• Better collaboration

• Fewer defects

• Ultimately better service

• Happy customers

• Profitability

Page 12: Geek Sync I In Database Automation We Trust

12

Continuous Delivery moving ahead!

▪ Recently Conducted DBmaestro Survey■ Over 350 participants

Page 13: Geek Sync I In Database Automation We Trust

13

Why Continuous Delivery?

Page 14: Geek Sync I In Database Automation We Trust

14

But…what about the database?

Page 15: Geek Sync I In Database Automation We Trust

15

What about the database?

Page 16: Geek Sync I In Database Automation We Trust

16

Down to 14% !!!

• Based on the questions they answered, only 14% from the ones reported doing CD for DB, are actually performing basic CD practices!!!

• The rest are plugging the automated process with various manual steps…

• Why?• What is so special about the database?

Page 17: Geek Sync I In Database Automation We Trust

17

The Weakest Link In a Chain ???

•Old adage but true• The database is often neglected and therefore can

become the weakest link• Manual processes

•Database/Code Silos exist…• Don’t always communicate effectively• Need to follow same procedures & best practices

•Essential from a compliance and business point of view

•Should be the strongest link

Page 18: Geek Sync I In Database Automation We Trust

18

A quick poll

How often do you or others make changes to your development databases?

• Daily

• Few times a week

• Once every few weeks

• Almost never

Page 19: Geek Sync I In Database Automation We Trust

19

The Database is a constraint

Page 20: Geek Sync I In Database Automation We Trust

20

CD for the database

• In many organizations, changes to databases are not included in the CD picture

• Too much risk – seems to be very challenging

• Too little awareness of the packaging, deployment, promotion etc. concepts used for application code

• Insufficient tool support for these concepts

Page 21: Geek Sync I In Database Automation We Trust

21

Manual steps in an automated process…

• At first you think: It is easy! We can handle it!

• Then your start fighting a loosing game…

• And then you have to speed up!!!

Page 22: Geek Sync I In Database Automation We Trust

22

So why not move forward?

• Mistrust…

Page 23: Geek Sync I In Database Automation We Trust

23

What is the problem?

• Root Causes for issues:– Challenging manual version control process– Static deployments code overrides– Dynamic deployments unaware of version control– No release automation red-flags – don’t know when to stop the line…

Page 24: Geek Sync I In Database Automation We Trust

24

Another quick poll

How do you source control your database changes?

• We use an automated/dedicated database source control solution

• We manually save scripts to an application-code source control solution

• We keep track with excel sheets and other lists

• We don't

Page 25: Geek Sync I In Database Automation We Trust

25

Two isolated processes

•Version Control Process (file based)

Development Process

Check-Out Script

Modify Script

Get updated Script from DB

Check-In Script

Compile Scriptin DB

Debug Scriptin DB

?

??

?

A

A’

Version control repository is out of sync from the tested database and cannot act as a Single Source of Truth

Page 26: Geek Sync I In Database Automation We Trust

26

Scripts & version control

▪Challenges:− Code-overrides

− Working on the wrong revisions

− Scripts do not always find their way to the version control solution

− Out of process updates go unnoticed

− Hard to locate outdated update scripts

▪Playing safe? What we really need: − The actual code of the object

− The upgrade script

− A roll-back script

Page 27: Geek Sync I In Database Automation We Trust

27

86%Rate this as a risk Factor

64%Rate this as a HIGH risk Factor

Page 28: Geek Sync I In Database Automation We Trust

Scripts… Build Once Deploy Many

X1.11.1.11.11.21.31.41.51.61.7

Int QA Stage Prod

Database Deploy ScriptEnvironment

Re-Base (due to defects)

DevDev

DevModel

1.1 1.2

1.2 1.3

1.3 1.4

1.4 1.5

1.5 1.6

1.6 1.7

1.11.11.41.7

1.1 1.2

1.2 1.3

1.3 1.4

1.4 1.5

1.5 1.6

1.6 1.7

1.1 1.2

1.2 1.3

1.3 1.4

1.4 1.5

1.5 1.61.6 1.7

Out of Process Change

X

X

X

XX

? 1.1.1

X

a

a

28

Page 29: Geek Sync I In Database Automation We Trust

29

Scripts are static…

▪ Scripts

− Hard to test in their entirely (holistically)

− Hard to test due to colliding dependencies

− Need to run in a specific order…

− Much harder to deal with project scope changes

▪ Scripts, unless super sophisticated:

− Unaware of changes made in the target environment

− Time passed from their coding to the time they are run

− Potentially overriding production hot-fixes or work done in parallel by another team

Page 30: Geek Sync I In Database Automation We Trust

30

60%of those manually building scripts have

to fix or tweak them regularly as part of a deployment process

Page 31: Geek Sync I In Database Automation We Trust

31

89%Rate this as a risk Factor

60%Rate this as a HIGH risk Factor

Page 32: Geek Sync I In Database Automation We Trust

32

Gaining Trust!

Coordinated Process Traceability

Start in the Beginning

No Out-of-Process Changes

Impact Analysis

Automation

Task Based Development

Well Defined Processes

Page 33: Geek Sync I In Database Automation We Trust

33

Version control: Integrated & Enforced process

Page 34: Geek Sync I In Database Automation We Trust

34

Dealing with challenges…

▪Integrated Database Version Control process

− Leverage proven version control best practices • Forcing check in & out for changes• Labels• etc..

− No code-overrides

− Always working with the correct revision

− All changes are documented

− Always know who did what, when, why and from where

− No out-of-process changes

▪ No time spent on manual coding of the change scripts

Page 35: Geek Sync I In Database Automation We Trust

1.11.21.31.41.51.61.7

*

Int QA Stage Prod

Database Deploy Script

Environment* Execute the same script being executed at the Stage environment

Re-Base (due to defects)

DevDev

DevModel

1.1 1.2

1.2 1.3

1.3 1.4

1.4 1.5

1.5 1.6

1.6 1.7

1.1 1.4

1.4 1.7

1.1.1 1.7

1.11.1 1.1

1.41.7

File Based Version Control

Out of Process Change

1.1.11.7 1.1.11.7

Validate

Build & deploy on demand

35

Page 36: Geek Sync I In Database Automation We Trust

36

Using tools

Test cases using compare & sync tools:

An index exists in source (QA) but not in target (Production)

What should we do? Add the index or not?

Page 37: Geek Sync I In Database Automation We Trust

37

Compare & sync tools

Safe to automate?Sure… (?)

Page 38: Geek Sync I In Database Automation We Trust

38

Compare & sync tools

Safe to automate?No. Requires manual inspection…

Page 39: Geek Sync I In Database Automation We Trust

39

Challenges…

Compare & sync tools:

▪ Are great for finding out what is out of sync

▪ Not so great for automating deployments

▪ Requires manual inspection

▪ Requires detailed knowledge regarding each change as part of the process

▪ Is unaware of any changes that occurred before the time it ran

▪ Has no knowledge of changes that took place at the target environment

▪ Unable to deal with conflicts & merges between different teams

Mistrust AGAIN… So…no automation… We fear for automating problems into production and a major risk!!!

Page 40: Geek Sync I In Database Automation We Trust

40

Another poll? Last one!

How much automation do you have in your database deployments?

• None, scripts are written manually

• Writing scripts manually, executing them as part of an automated process

• Generating scripts automatically, reviewing them manually

• We have a fully automated CD process (databases updates are created and executed automatically)

Page 41: Geek Sync I In Database Automation We Trust

41

We need to leverage

knowledge from version control

Page 42: Geek Sync I In Database Automation We Trust

42

Safety net

Static script based deployment

Page 43: Geek Sync I In Database Automation We Trust

43

Deploying changes if needed

DevelopmentBaseline

Previous Label /

Production Golden Copy

Production

If we had the index in the baseline =>

we should take it down from production…

(Deploy Change)

Page 44: Geek Sync I In Database Automation We Trust

44

Or protecting target environment…

DevelopmentBaseline

Previous Label /

Production Golden Copy

Production

BUT… If no index in baseline =>

we should protect the NEW index on production!!!

(Protect Target)

Page 45: Geek Sync I In Database Automation We Trust

45

Safe?

Simple, right?NO! we are going to BREAK production without even knowing…

Page 46: Geek Sync I In Database Automation We Trust

46

Dealing with conflicts => merging changes

Page 47: Geek Sync I In Database Automation We Trust

47

Conflict Resolving – Meta Data/Content

Page 48: Geek Sync I In Database Automation We Trust

48

Safety Net Deployment Automation

■ Raise red flags on conflicts ■ Support out-of-process changes■ Utilize baseline aware analysis■ Understand the nature of the changes

Source vs. Baseline

Target vs. Baseline

Action

= = No Action

≠ = Deploy Changes

= ≠ Protect Target

≠ ≠ Merge Changes

Page 49: Geek Sync I In Database Automation We Trust

49

Impact Analysis! not Damage Control…

Raise red flags to stop the line…

if requires human intervention

Page 50: Geek Sync I In Database Automation We Trust

50

Safety Net For Deployment Automation

Database Safe Deployment Automation:• Leverage one source of truth (baselines & previous revisions)

• Flexible scope (deploy multi schema to single task or work item)

• Run as a batch process (repeatable & consistent)

• Integrates to ALM (labels, CRs, Continuous Integration & Delivery)

• Deal with conflicts & merges to match code agility

Can raise red flags to stop the line…

if requires human intervention

Page 51: Geek Sync I In Database Automation We Trust

51

To summarize CD…

•Automate “everything”• Package the deployment of database changes along with all your other application

components to give a unified picture

•Move the process upstream• Easily promote the same package (including database changes!) from one environment to

the next, handling environment-specific differences automatically

•Create the deployment pipeline

Page 52: Geek Sync I In Database Automation We Trust

52

What does DBmaestro offer?Database Enforced Change Management solution

+ Database version control+ Enforce best practices+ Plugs into the ALM (change request, tickets & work items)+ Database merge & change impact analysis+ Know who can do what, where, when & why

DevOps Solution for databases + Baseline aware deployment automation, rollback & recovery+ Reduce database deployment issues+ Plugs into release management & Continuous Delivery

Allows you to package, verify, deploy and promote database changes just as you would do with application code…

putting you in a position to build a full delivery pipeline…

Page 53: Geek Sync I In Database Automation We Trust

53

Change Policy Enforcement

Page 54: Geek Sync I In Database Automation We Trust

54

Impact Analysis

Page 55: Geek Sync I In Database Automation We Trust

Head over to our community to see how everyoneresponded to the poll questions.

Poll Questions

community.idera.com/forums

Page 56: Geek Sync I In Database Automation We Trust

56

Thank you!

Q & AYaniv [email protected]