Upload
kimberly-lloyd
View
230
Download
3
Tags:
Embed Size (px)
Citation preview
Database Agility
Pramod SadalageThoughtWorks
© 2002. ThoughtWorks, Inc.©
Agile.Database:// Introduction
The Traditional Database
Continuous changes to the data model or the database are not allowed
Production changes are rare or not allowed
Migration is a project There is one Development / QA / UAT /
Production database Change management is related to
database code
© 2002. ThoughtWorks, Inc.©
Agile.Database:// Introduction
The Agile Database
Slowly changing database. Functionality added in increments. Change management Facilitates automated testing Knowledge by developers of the
functionality to be developed Acknowledged importance of team
interaction
© 2002. ThoughtWorks, Inc.©
Agile.Database:// Foundation
Everyone Gets a Database
A database instance is the same as an application instance
Developers can work independent of other developers
Everyone has the liberty of trying various changes with the database as if it were a scratch pad
Scripts exist in order to provide a clean database for everyone
© 2002. ThoughtWorks, Inc.©
Agile.Database:// Foundation
Why Everyone Needs a Database
Databaseschema
Dev1
Dev2
Devn
Dev2
Devn
S1S2 Sn
.... ....Dev1
...
© 2002. ThoughtWorks, Inc.©
Agile.Database:// Foundation
The Database Instance
The database analog to the application instance
A data space containing, at the very least, the schema and setup data necessary to support an application instance
A data space that is independent not affected by and not able to affect the space of another database instance
© 2002. ThoughtWorks, Inc.©
Agile.Database:// The Database Instance
Examples
File
FileFile
Microsoft Access
MySQL Server
DB
DB
DB
UserUser
User
Oracle Database
Sybase Server
DB
DB
DB
© 2002. ThoughtWorks, Inc.©
Agile.Database:// The Database Instance
Why does the DBA need it?
Easier to maintain specialized datasets for development, automated testing,
demos, load testing, production copies, etc.
Ability to migrate older datasets to newer versions of the application
Allows us to think of the database as another component in the system, with a build number and a context
© 2002. ThoughtWorks, Inc.©
Agile.Database:// The Database Instance
Is State full and Locatable
E.g. iteration 2, regression test database
Time
Context
1 2 3
© 2002. ThoughtWorks, Inc.©
Agile.Database:// The Database Instance
Why Would the Team Want It?
To allow every developer the ability to code and test with his or her own, independent database Would you have an entire team develop all using
the same single application instance? To make the database portable and restorable
A developer can code and test against a specific set of data. If the developer breaks the dataset, no one else is affected and the developer simply loads up a fresh copy.
Promotes better team velocity
© 2002. ThoughtWorks, Inc.©
Agile.Database://The Database Instance
The Database as File System
Think of the database instance as if it were a file. Commands such as create, copy, move and delete can be made applicable to the database instance
Application
Database
Development
Automated QA
Development
Reporting
User Acceptance
Terri
Build
Ajey
QA
Master
JK
© 2002. ThoughtWorks, Inc.©
Agile.Database:// Lineage
Schema and Setup Data
Setup data; is non transactional, allows testing to occur.
Every alteration to the schema and setup data must be chronologically logged and source-controlled with the application code
A fully-functional database instance can be built at any time to any point in the development timeline
Old database instances can be rolled forward by applying the logged changes made to schema and setup data
© 2002. ThoughtWorks, Inc.©
Agile.Database:// Lineage
Master and Child Instances
MasterInstance(Base)
JK Ajey Build Terri QA
© 2002. ThoughtWorks, Inc.©
Agile.Database:// Lineage
New Lineages
Should be created when A portion of the team (QA, reporting, etc)
requires a unique and maintainable dataset The application is branched
Should not be created for non-essential reasons
example: to maintain a unit-test suite that fails to work on the latest load of converted data
© 2002. ThoughtWorks, Inc.©
Agile.Database:// Lineage
Every Lineage Has ...
A master instance A change log An update list A collection of child instances
© 2002. ThoughtWorks, Inc.©
Agile.Database:// Lineage
The Master Instance
Every lineage has a master upon which it isbasedThe master instance: Is the gold standard for that lineage Is updated every time an update is made to the
lineage Is the mold from which all new copies of that
instance are made Should be updated only by designated members
of the team
© 2002. ThoughtWorks, Inc.©
Agile.Database:// Lineage
The Change Log
A stored collection of all changes(Schema and Data) made to
the lineage master and its child instances.The change log: Will likely be written in SQL Should be source controlled Is the tool by which instances will be rolled
forward (backward if needed) Should be named by Build/Release/Day/Week/
etc
© 2002. ThoughtWorks, Inc.©
Agile.Database:// Lineage/ Change Log
Destructive Actions
Whenever possible, use a clean up log Destructive actions are entered into the clean-up
log with a run date that is several days in the future.
Actions such as Drop table, Drop column, Make Column Not Null are considered destructive.
Destructive changes are propagated only after all team members have been given a chance to move to the latest code.
Assumes, code is being integrated more than once a day. http://cruisecontrol.sourceforge.net
© 2002. ThoughtWorks, Inc.©
Agile.Database:// Lineage
The Update List
The means by which database instances are kept up to date ... While the data model evolves As developers and other team members actively
code and test against their individual database instances
Communicates changes to the lineage’s master and all its child instances, automatically.
Each master instance should have an update list Database instances can subscribe to the update
lists that belongs to their lineage
© 2002. ThoughtWorks, Inc.©
Agile.Database:// Lineage
The Update List [an example]
connect base/base@devdbrun c:\dbscripts\changes.sqlconnect jk/jk@devdbrun c:\dbscripts\changes.sqlconnect ajey/ajey@devdbrun c:\dbscripts\changes.sqlconnect build/machine@devdbrun c:\dbscripts\changes.sqlconnect terri/terri@testdbrun c:\dbscripts\changes.sqlrun c:\dba\lists\QA_Update_list.sql
© 2002. ThoughtWorks, Inc.©
Agile.Database.Framework://
The Fundamentals
Allow Developers to Experiment Function-Driven Database Design Clean Tests People Infrastructure Complexity
© 2002. ThoughtWorks, Inc.©
Agile.Database:// Fundamentals
Allow Developers To Experiment Developers can change the model in their
local schema so that they can rollback if the change they made does not work efficiently.
The DBA should get involved in the design sessions for the functionality being developed.
The DBA should help the developers to design the tables, Indexes, tune queries for performance.
© 2002. ThoughtWorks, Inc.©
Agile.Database:// Fundamentals
Function-Driven Database Design
Use functionality or change requests orstory cards to grow the database
A typical change should contain: The schema alterations to be done Comments for all new database objects
being added (two years down the line, someone will need to know; why a given attribute was added)
Default values and data migration details Define proper constraints on columns and
tables.
© 2002. ThoughtWorks, Inc.©
Agile.Database:// Fundamentals
Function-Driven Database Design
1. Developers design changes along with the DBA
When starting work on a new piece of functionality or when refactoring.
2. DBA reviews all change requests Speaks for other parties (e.g. conversion and
reporting) Delays destructive actions Keep a log of all change requests Writes data migration code.
© 2002. ThoughtWorks, Inc.©
Agile.Database:// Fundamentals
People
Team collaboration is essential Developers, QA, Analysts and DBAs need
to be on board Project management must understand
and think Agile Encourage experimentation and tolerate
failures Push for automated testing and demand
clean tests. Encourage team to use production data.
© 2002. ThoughtWorks, Inc.©
Agile.Database:// Fundamentals
Infrastructure
Hardware to support one or more databases per person is necessary
It should not take more than 30 minutes to copy a database instance
If necessary, work with slices of production data
Start converting data; day 1 of the project.
If feasible use light weight DBMS, while in development e.g. Hypersonic, MySQL etc.
© 2002. ThoughtWorks, Inc.©
Agile.Database:// Fundamentals
Complexity
One system, many data sources One data source, many systems Many systems, many data sources
© 2002. ThoughtWorks, Inc.©
Agile.Database://
Production
Handling the application in production issurprisingly simple. Because: The database is being developed, tracked
and sourced with the same rigor as code The Change Logs are your migration
scripts
© 2002. ThoughtWorks, Inc.©
Agile.Database:// Production
First Time Deployment
Should be handled almost exactly the same as deploying code
Archive a copy of the production instance Branch if necessaryThe production instance will be either A clean copy of the application database Full of converted data
© 2002. ThoughtWorks, Inc.©
Agile.Database:// Production
Additional Releases
All of the change logs should be applied following the last production migration. In chronological order. (Change Logs are stored by Build/Release/Day/Week)
Since data migration scripts are part of change logs, data is also migrated
It is wise to test the migration on a test database
© 2002. ThoughtWorks, Inc.©
Agile.Database:// Production
Additional Releases
(Release 2)
2 3 4 5 6 7 8
Release 1Production Data
(Release 1) Migration Script
34 36 38 40 42 44 46Week
Iteration
Initial Deployment Script
© 2002. ThoughtWorks, Inc.©
Agile.Database:// Production
Thinking Forward
Production database instances can be migrated to a future state of development.
New releases can be acceptance tested against the most recent data currently in production.
Migration is a snap, so why not deploy weekly or frequently.
© 2002. ThoughtWorks, Inc.©
Database Agility
Tools and Automation
© 2002. ThoughtWorks, Inc.©
Tools.and.Automation://
Process Automation
The DBA should attempt to automate any process he finds himself doing repeatedly
Whenever possible these tools should be pushed out to team members
It is better to give too much access than not enough
Process automation is absolutely, utterly essential if the DBA is to respond to a myriad of requests in a timely manner
© 2002. ThoughtWorks, Inc.©
Tools.and.Automation://
Tools
DB Manager A ThoughtWorks Tool
© 2002. ThoughtWorks, Inc.©
Tools.and.Automation://
Other Tools
On-Line Data Dictionary (Java)Data loading utilities
(Visual Basic in Excel)Reporting replication process
(UNIX shell scripts)DB backup and schema file transfer
(Perl)
© 2002. ThoughtWorks, Inc.©
More Resources.. http://groups.yahoo.com/group/agileDatabases/
http://martinfowler.com