liquibase migration 0.0 - Medisys › usermanual › liquibase › ... · í î n w p 'dwdedvh &uhdwlrq ,psruw h[hfxwh gdwdedvh fuhdwlrq vfulsw iru hpsw\ sdvvzrug p\vto xurrw vuf

$Page 1: liquibase migration 0.0 - Medisys › usermanual › liquibase › ... · í î n w p 'dwdedvh &uhdwlrq ,psruw h[hfxwh gdwdedvh fuhdwlrq vfulsw iru hpsw\ sdvvzrug p\vto xurrw vuf$
Medical Systems Co. Ltd

LIQUIBASEDBMS Migration Tools for Java

This is a controlled document. Unauthorized access, copying, replication or usage for a purpose other than for which it is intended, are prohibited. All tradthe document have been used for identification purposes only and belong to their respective companies.


IQUIBASE 3.5.3DBMS Migration Tools for Java

Version: 0.0.01 (Based on MariaDB 10.1.28)

This is a controlled document. Unauthorized access, copying, replication or usage for a purpose other than for which it is intended, are prohibited. All trademarks that appear in the document have been used for identification purposes only and belong to their

August, 2018


3.5.3 DBMS Migration Tools for Java

This is a controlled document. Unauthorized access, copying, replication or usage for a emarks that appear in

the document have been used for identification purposes only and belong to their

2 | P a g e

3 | P a g e

© Copyright 2018, Medical Systems Co. Ltd

4 | P a g e

5 | P a g e

Table of Contents I. Document Information 7 II. Document Release Note 8 III. Notations Used 9

Overview 10

Database Creation & Import 12

Generate the Change Log 13

1.1 Change Set for Regions Table 15 1.2 Change set to load data of Departments Table (dev, qac) 16 1.3 Change set to load data of Departments Table (uat, pro) 16 1.4 Change set for Primary Key 16 1.5 Change set for Unique Key 17 1.6 Change set for Index 17 1.7 Change set for Foreign Key Constraint 17

Fresher Empty Database Migration 18

Existent Database Fresher Migration 20

Existent Database Update Migration 21

Routine Migration for Development 22

Generate Differential Change Log 23

Exercise & Experiment of Migration 25

Precautions & Good Practice 26

6 | P a g e

Intentionally Blank

7 | P a g e

Document Information

Issued by: Medical Systems Co. Ltd Head Office: K.S.A - Riyadh FAX: +966-1-414 3211 TEL: +966-1-4141 330 Email: [email protected]

Service-related contact information worldwide

All service-related contact information is available on official

website. Please follow the link: http://medisys.com.my/ihe

8 | P a g e

Document Release Note

Document Details

Details Revision How to Implement Liquibase Migration for Java 0.0.01

Author

Revision Changes Documented Revised Updated 0.0.00 Initial Release Md. MizanurRahman Md Shahed Hossain Aug 16, 2018 0.0.01 Initial Release Md. MizanurRahman Md Shahed Hossain Aug 28, 2018

9 | P a g e

Notations Used

S/N Notation Elaboration 1 DBMS Database Management System 2 RDBMS Relational Database Management System 3 DDL Data Definition Language 4 DML Data Manipulation Language 5 TAB Table 6 NDX Index 7 PKC Primary Key Constraint 8 FKC Foreign Key Constraint 9 UKC Unique Key Constraint

10 SQN Sequence 11 PRO Procedure, Production 12 DEV Development 13 QAC Quality Assurance & Control 14 UAT User Acceptance Testing 15 VEW View 16 INI Initialization 17 TRX Transaction 18 INS Insert 19 UPD Update 20 COL Column

10 | P a g e

Overview Liquibase is an open source community project for refactoring Relational Database Management System. It’s also called VCS (Version Controlling System) for Database. Its platform independent migration tools regardless RDBMS. Only you have to maintain some good practice.

During the migration process running, be aware about contexts those are directly belongs to the deployment environments as following:

1. DEV (Development ) 2. QAC (Quality Assurance & Control) 3. UAT (User Acceptance Test) 4. PRO (Production)

There are some labels; those are usages for filtering the changes logs as following:

1. M00, DDL, TAB (Create/Drop Table) 2. M00, DDL, COL (Add/Drop Column) 3. M00, DDL, VEW (Create/Drop View) 4. M00, DML, INI (Initialization Data) 5. M00, DML, TRX (Transactional Data) 6. M00, DDL, PKC (Primary Key Constraint) 7. M00, DDL, UKC (Unique Key Constraint) 8. M00, DDL, FKC (Foreign Key Constraint) 9. M00, DDL, NDX (Create/Drop Index)

There are some tags those are usages for checkout specific Commit or Rollback. Although Liquibase is a database refactoring and version control system where its performance & success depends on some good practices. Its means order of change log id, dbms, context, labels, logicalFilePath and relativeToChangelogFile. Please find the example project in Github.

# https://github.com/medisysco/medisys-dbms-change

git clone https://github.com/medisysco/medisys-dbms-change.git

cd ./medisys-dbms-change

Snippet: Clone Human Resource Example Project

11 | P a g e

Find the example resources inMySQL/MariaDB Schema files as following

Figure: Liquibase Migration Resources

mkdir -p $HOME/.hmis/etc/dbms/liquibase

cp ./src/test/resources/liquibase/*.properties

cd $HOME/.hmis/etc/dbms/liquibase

explorer .

Snippet: Copy & Edit Properties Files

export MYSQL_HOME=/c/opt/xampp/mysql

export PATH=$PATH:$MYSQL_HOME

Snippet: MySQL Path Settings

inside the project folder. Please find the Liquibaseas following folders:

Resources

/.hmis/etc/dbms/liquibase

test/resources/liquibase/*.properties $HOME/.hmis/etc/dbms/liquibase


Edit Properties Files

/c/opt/xampp/mysql

$MYSQL_HOME/bin

iquibase properties and


12 | P a g e

Database Creation & Import

# execute database creation script for empty password

mysql -uroot < ./src/test/resources/schema/hr_all_ini.sql

# execute database creation script for prompt password in cli

mysql -uroot -p < ./src/test/resources/schema/hr_all_ini.sql

Snippet: Execute Database Creation Dump

# import hr_dev_bas_ini.sql dump into hr_dev from cli

mysql -uroot hr_dev < ./src/test/resources/schema/hr_dev_bas_ini.sql

# import hr_dev_bas_ini.sql dump into hr_dev & prompt password from cli

mysql -uroot -p hr_dev < ./src/test/resources/schema/hr_dev_bas_ini.sql

Snippet: Import Human Resources Development Dump After execution that scripts right now we have Four Databases in MySQL/MariaDB under Root Schema. Those are as following:

1. HR_DEV (Human Resources Development) 2. HR_QAC (Human Resources Quality Assurance & Control) 3. HR_UAT (Human Resources User Acceptance Test) 4. HR_PRO (Human Resources Production)

Where HR_DEV contains some Database Objects and Data, those are used to realizing different environment and context. Please find the Human Resources Example Database structure in the next page.

13 | P a g e

Generate the Change Log

Figure: Human Resources Development Database Considering that we have a Human Resources have to generate change logs then organized the change logs according to the good practise and the company policy. During the time ofbut need to add/overriderelativeToChangelogFile and etc. Where need to specify some initialization/setup data required for production perspectives of development Considering such situation we have defined four contextpro. There are some labels too; those were Considering that, we are going to implement dev.properties as following for respective database

vim $HOME/.hmis/etc/dbms/liquibase/

Snippet: Configure dev.properties

Change Log

Human Resources Development Database

uman Resources Database in development. Regarding this scenario we have to generate change logs then organized the change logs according to the good practise and the

During the time of change log generation we can specify the add/override some id, dbms, context, labels, file,

etc.

Where need to specify some DDL and DML for different context/environment.data required for production but no transactional data required. In

and quality assurance we need to add some Considering such situation we have defined four contexts those are as respectively

o; those were mentioned on page-10.

Considering that, we are going to implement Liquibase migration for fist time. Please for respective database.

/.hmis/etc/dbms/liquibase/dev.properties

properties

. Regarding this scenario we have to generate change logs then organized the change logs according to the good practise and the

we can specify the context and author file, logicalFilePath,

environment. For example data required. In the

some dummy data. those are as respectively dev, qac, uat and

migration for fist time. Please edit the

14 | P a g e

mvn clean install –Pdev,log

Snippet: Liquibase Change log Generation Here is the Change Log generated with exported data. Data may require for Development (dev), Quality Assurance & Control (qac), User Acceptance Test (uat) and Production (pro). Please find the change log in the following location including CSV files those contains data.

# target/db.changelog-V${version.medisys.version}_${change.on}.xml

vim target/db.changelog-V0.0.00_20181010_101010.xml

ls -la target/20181010_101010

Snippet: Change log with data This is the very raw change log those are not production ready. Need to split and amend it according to our organization business. According to the development to production we have four contexts and lots of labels. This is the right time to organize the change log and the data. Don’t mixing the change set and data with other context. Be aware about the good practice during the organization or reorder the change set and the data. Right now change log are ready for development and deployment environment. If you are aware most may have doubt why change log ready for development. No doubt on deployment. Those who have doubt, we would like to appreciate you for raising this doubt. Those who have not any doubt please concentrates the next scenario. Each time of Liquibase update or migration (deprecated) the Liquibase engine compare the change set with databasechangelog table. No such table found in your database, nothing to be wonder and it the reality. If databasechangelog & databasechangeloglock tables are not found, it create itself to perform update where its might be causes error. As Liquibase treated no update performed before. To overcome such type of situation we have to synchronize the change log change set with development and deployment environment for base version only. Since the next version development will be synchronize only and deployment will be updated.

15 | P a g e

Figure: Managed and Ordered Change log

<changeSet author="medisys"

id="1533121211000-7" logicalFilePath=

<createTable tableName=

<column name="region_id"

<constraints nullable=

</column>

<column name="region_name"

</createTable>

</changeSet>

Snippet: Change Set for Regions Table

Managed and Ordered Change log

" labels="m00,ddl,tab" context="dev,qac,uat,pro

logicalFilePath="db.changelog-V0.0.00_20180801_110011.xml

tableName="regions">

"region_id" type="DECIMAL(22)">

nullable="false"/>

"region_name" type="VARCHAR(25)"/>

Regions Table

dev,qac,uat,pro"

V0.0.00_20180801_110011.xml">

16 | P a g e

<changeSet author="medisys" labels="m00,dml,trx" context="dev,qac"

id="1533207611000-2" logicalFilePath="db.changelog-V0.0.00_20180802_110011.xml">

<loadData commentLineStartsWith="#" encoding="UTF-8"

file="20180802_110011/departments__dev_qac.csv" relativeToChangelogFile="true"

quotchar=""" separator="," tableName="departments">

<column header="department_id" name="department_id" type="NUMERIC"/>

<column header="department_name" name="department_name" type="STRING"/>

<column header="manager_id" name="manager_id" type="NUMERIC"/>

<column header="location_id" name="location_id" type="NUMERIC"/>

</loadData>

</changeSet>

Snippet: Change set to load data of Departments Table (dev, qac)

<changeSet author="medisys" labels="m00,dml,ini" context="uat,pro"


<loadData commentLineStartsWith="#" encoding="UTF-8"

file="20180802_110011/departments__uat_pro.csv" relativeToChangelogFile="true"

quotchar=""" separator="," tableName="departments">

<column header="department_id" name="department_id" type="NUMERIC"/>

<column header="department_name" name="department_name" type="STRING"/>

<column header="manager_id" name="manager_id" type="NUMERIC"/>

<column header="location_id" name="location_id" type="NUMERIC"/>

</loadData>

</changeSet>

Snippet: Change set to load data of Departments Table (uat, pro)

<changeSet author="medisys" labels="m00,ddl,pkc" context="dev,qac,uat,pro"


<addPrimaryKey columnNames="department_id" constraintName="PRIMARY"

tableName="departments"/>

</changeSet>

Snippet: Change set for Primary Key

17 | P a g e

<changeSet author="medisys" labels="m00,ddl,ukc" context="dev,qac,uat,pro"


<addUniqueConstraint columnNames="email" constraintName="emp_email_uk"

tableName="employees"/>

</changeSet>

Snippet: Change set for Unique Key

<changeSet author="medisys" labels="m00,ddl,ndx" context="dev,qac,uat,pro"


<createIndex indexName="dept_location_ix" tableName="departments">

<column name="location_id"/>

</createIndex>

</changeSet>

Snippet: Change set for Index

<changeSet author="medisys" labels="m00,ddl,fkc" context="dev,qac,uat,pro"


<addForeignKeyConstraint baseColumnNames="location_id"

baseTableName="departments" constraintName="dept_loc_fk" deferrable="false"

initiallyDeferred="false" onDelete="NO ACTION" onUpdate="NO ACTION"

referencedColumnNames="location_id" referencedTableName="locations"/>

</changeSet>

Snippet: Change set for Foreign Key Constraint In this perspective we have some change logs; those are organized in specific order. Each change set we have already amended some attribute like id, dbms, context, labels, logicalFilePath, relativeToChangelogFile and etc. Right now we need to synchronize the change logs with development Database.

mvn clean install –Pdev,syn

Snippet: Liquibase Change logs Synchronization After having synchronization we are going to update the change logs. If there is an error occurred it’s suppose to that your change logs not ready for deployment else it’s ready.

18 | P a g e

mvn clean install –Pdev,upd

Snippet: Liquibase Change logs Update There are some thumb rules for deployment change logs into Database. Need to be aware and good practise before deployment. We have to consider the following phase in perspective of our database. Those are as following:

1. Fresher Empty Database Migration 2. Existent Database Fresher Migration 3. Existent Database Update Migration

Before migration/update please detect your database phase. Then follow the procedure according to the detection of phase and context. We are going to describe each phase for respective context (Mentioned in Page-10).

Fresher Empty Database Migration

Development:

vim $HOME/.hmis/etc/dbms/liquibase/dev.properties

Snippet: Configure dev.properties

git pull origin master



Snippet: Migration to the Development

19 | P a g e

Quality Assurance & Control:

vim $HOME/.hmis/etc/dbms/liquibase/qac.properties

Snippet: Configure qac.properties


mvn clean install –Pqac,syn

mvn clean install –Pqac,upd

Snippet: Migration to the QAC User Acceptance Test:

vim $HOME/.hmis/etc/dbms/liquibase/uat.properties

Snippet: Configure uat.properties


mvn clean install –Puat,syn

mvn clean install –Puat,upd

Snippet: Migration to the UAT Production:

vim $HOME/.hmis/etc/dbms/liquibase/pro.properties

Snippet: Configure pro.properties


mvn clean install –Ppro,syn

mvn clean install –Ppro,upd

Snippet: Migration to the Production

20 | P a g e

Existent Database Fresher Migration

Development:




Snippet: Migration to the Development Quality Assurance & Control:




Snippet: Migration to the QAC User Acceptance Test:




Snippet: Migration to the UAT Production:





21 | P a g e

Existent Database Update Migration Development:




Snippet: Migration to the Development Quality Assurance & Control:




Snippet: Migration to the Quality Assurance & Control User Acceptance Test:




Snippet: Migration to the User Acceptance Test Production:





22 | P a g e

Routine Migration of Development

In this point we will focus only Development context/environment of respective developer in his own workstation as because here is the matter of aware, need to be concern and have some good practise. Else it might be wastage of time and money. Might be causes the reinvention of this circle what’s already practising by us? Rest of the contexts/environments will be following the thumb rules according to the detection of its phase as mentioned in Pages-18, 19, 20, 21. Detect your migration phase and context then execute the respective command. Let’s start our Daily/Routine Migration for Development in respect of our personal official workstation. Please follow the following step before adding any database object, generating, manipulating, commit and push change logs:

1. Communicate with your team if anybody has change logs to push 2. Before push and pull negotiate with team to overcome possible conflict 3. Always pull first before push, to avoid conflict in change log. 4. After pull resolve the possible conflict with change log.

a. In case of conflict backup, drop the conflicted objects & logs b. Then update your database with newly pulled change logs

mvn clean install -Pdev,upd c. Recreate your database object from backup

5. For conflict free environment please follow the instruction

a. Just backup and drop the generated & manipulated logs b. Update your database with newly pulled change logs

mvn clean install -Pdev,upd

6. Generate and manipulate the change logs using

a. mvn clean install –Pdif b. mvn clean install -Pdev,log

7. Synchronize the newly manipulated change logs

a. mvn clean install –Pdev,syn

8. After Synchronize please Update to cross check a. mvn clean install -Pdev,upd

9. Based on the permission of your team push to git

a. git push –u origin master

23 | P a g e

Generate

Figure: Differences between Development and Production

export MYSQL_HOME=/c/opt/xampp/mysql

export PATH=$PATH:$MYSQL_HOME

Snippet: MySQL Path Settings

# import hr_dev_dif_add.sql

mysql -uroot hr_dev < ./src/test/resources/schema/

# import hr_dev_dif_add.sql

mysql -uroot -p hr_dev <

Snippet: Simulate the Difference

Development:

vim $HOME/.hmis/etc/dbms/liquibase/

Snippet: Configure dif.properties

Generate Differential Change Log

Differences between Development and Production

/c/opt/xampp/mysql

$MYSQL_HOME/bin

.sql dump into hr_dev from cli

./src/test/resources/schema/hr_dev_dif_add

.sql dump into hr_dev & prompt password

< ./src/test/resources/schema/hr_dev_dif_add

ce of Human Resources Development Database

/.hmis/etc/dbms/liquibase/dif.properties

properties

Change Log

hr_dev_dif_add.sql

password from cli

hr_dev_dif_add.sql

24 | P a g e

mvn clean install –Pdif

Snippet: Generate the Differ Change log



Snippet: Generated differ change log

Please find the differential change set inside the change log. This is the very raw change log those are not production ready. Need to split and amend it according to our organization business. According to the development to production we have four contexts and lots of labels. This is the right time to organize the change log and the data. Don’t mixing the change set and data with other context. Be aware about the good practice during the organization or reorder the change set and the data. After reverting to the Database Objects Changes we can examine the generated change set inside the change log. Do you distinguish the different of change log and change set?

# import hr_dev_dif_rev.sql dump into hr_dev from cli

mysql -uroot hr_dev < ./src/test/resources/schema/hr_dev_dif_rev.sql

# import hr_dev_dif_rev.sql dump into hr_dev & prompt password from cli

mysql -uroot -p hr_dev < ./src/test/resources/schema/hr_dev_dif_rev.sql

Snippet: Revert the Human Resources Development Database

mvn clean install –Pdif

Snippet: Generate the Differ Change log



Snippet: Generated differ change log

25 | P a g e

Exercise & Experiment of Migration Base on the previous exercise right now you are suppose to be experience in Liquibased migration. Most of us naturally talent enough to predict the scenario what would be happened during the migration process mistaken.

1. During the Base Version Change logs Generation (DEV): Considering that there is no databasechangelog & databasechangeloglock tables exist in your Development Database. Change logs generated and manipulated as the organizational perspective for Base Version. Let Update Development Database with the change logs. Examine the update; find out the fault and solution? mvn clean install -Pdev,upd

2. Fresher Empty Database Migration (QAC): Considering this phase let execute the following command and examine what happened? Find out the fault and solution? mvn clean install -Pqac,syn mvn clean install -Pqac,upd

3. Existent Database Fresher Migration (UAT): Considering this phase let execute the following command and examine what happened? Find out the fault and solution? mvn clean install -Puat,upd

4. Existent Database Update Migration (PRO): Considering this phase let execute the following command and examine what happened? Find out the fault and solution? mvn clean install -Ppro,syn mvn clean install -Ppro,upd

26 | P a g e

Precautions & Good Practice

1. Do not add generated change log or change set directly in the change logs 2. Amend those attribute for deployment. Ex: id, dbms, context, labels, logicalFilePath,

relativeToChangelogFile and etc 3. Maintain the order of change log and change set. 4. Separate the context specific change set with Data 5. Be aware about context; do not mix the change log with others 6. Add change set for tag at end of each change log 7. Developer should aware about pull, update, differ and synchronization 8. Before aware to add change set into change log and its effect 9. Don’t use loadUpdateData on before Primary Key creation or implementation. Might raise

data duplication error! In such case use loadData only. Where it should be distinguish by respective context.

10. You should use one of four contexts during update; 11. Empty context Update might cause data duplication and violation of integrity. 12. Example Project: https://github.com/medisysco/medisys-dbms-change

Documents

liquibase migration 0.0 - Medisys › usermanual › liquibase › ... · í î n w p 'dwdedvh &uhdwlrq ,psruw h[hfxwh gdwdedvh fuhdwlrq vfulsw iru hpsw\ sdvvzrug p\vto xurrw vuf