Software Development using Production Data By Karen Ambrose Wellcome Trust Sanger Institute (WTSI, UK) Monday 28 th April 2008 – OLSUG, BOSTON, MA

Software Development using Production Data

By Karen AmbroseWellcome Trust Sanger Institute (WTSI, UK)

Monday 28th April 2008 – OLSUG, BOSTON, MA

Disclaimer

The information contained within this

presentation is based on systems at the

Wellcome Trust Sanger Institute.

Overview

About the Wellcome Trust Sanger Institute (WTSI)

What do we do? Case Study – Sequencing pipeline

Development requirements Smoke and Mirrors Logical intervention – Logical Standby

Questions?

About the WTSI

One of the leading genomics centres in the world. Founded in 1993 by the Wellcome Trust (major

funding provider) and the UK Medical Research Council (MRC).

Formerly the Sanger Centre. Named after the double Nobel prize-winning

biochemist Dr. Fred Sanger

What do we do?

Human Genetics - e.g. Cancer studies, SNP, WTCCC, Copy Number Variation (CNV)

Model Organism & Pathogen Genetics -Mouse, Zebrafish, Pathogens

http://images.wellcome.ac.uk/indexplus/result.html?_IXFIRST_=151&_IXSS_=_IXFIRST_%3D1%26_IXINITSR_%3Dy%26_IXACTION_%3Dquery%26_IXMAXHITS_%3D250%26%252asform%3Dwellcome%252dimages%26%2524%253dsort%3Dsort%2Bsortexpr%2Bimage_sort%26%3D%26_IXSESSION_%3D8AirKj3UPgT%26c%3D%2522contemporary%2Bclinical%2Bimages%2522%2BOR%2B%2522contemporary%2Bimages%2522%2BOR%2B%2522corporate%2Bimages%2522%26%2524%253ds%3Dhuman%2Bdna%26%2524%2Bnot%2B%2522Contemporary%2Bclinical%2Bimages%2522%2Bindex%2Bwi_collection%3D%252e%26_IXFPFX_%3Dtemplates%252ft%26%2524%253dsi%3Dtext&_IXACTION_=query&_IXMAXHITS_=1&_IXSR_=ZdxrUOqowE2&_IXSPFX_=templates%2Ft&_IXFPFX_=templates%2Ft

http://images.wellcome.ac.uk/indexplus/result.html?_IXFIRST_=149&_IXSS_=_IXFIRST_%3D1%26_IXINITSR_%3Dy%26_IXACTION_%3Dquery%26IXFROM%3D%26IXTO%3D%26_IXMAXHITS_%3D250%26%252asform%3Dwellcome%252dimages%26%2524%253dsort%3Dsort%2Bsortexpr%2Bimage_sort%26_IXSESSION_%3D8AirKj3UPgT%26c%3D%2522contemporary%2Bclinical%2Bimages%2522%2BOR%2B%2522contemporary%2Bimages%2522%2BOR%2B%2522corporate%2Bimages%2522%26i_num%3D%26i_pre%3D%26%2524%253ds%3Dmouse%26%2524%2Bnot%2B%2522Contemporary%2Bclinical%2Bimages%2522%2Bindex%2Bwi_collection%3D%252e%26_IXFPFX_%3Dtemplates%252ft%26%2524%253dsi%3Dtext%26t%3D%26w%3D&_IXACTION_=query&_IXMAXHITS_=1&_IXSR_=DewDsa0IUno&_IXSPFX_=templates%2Ft&_IXFPFX_=templates%2Ft

What do we do?

Model Organism Sequencing – e.g. Human, Mouse, Yeast, Worm

Bioinformatics –Analysis, Annotation, Data Storage, Data Processing etc.

http://images.wellcome.ac.uk/indexplus/result.html?_IXFIRST_=12&_IXSS_=_IXFIRST_%3D1%26_IXINITSR_%3Dy%26_IXACTION_%3Dquery%26IXFROM%3D%26IXTO%3D%26_IXMAXHITS_%3D250%26%252asform%3Dwellcome%252dimages%26%2524%253dsort%3Dsort%2Bsortexpr%2Bimage_sort%26_IXSESSION_%3D8AirKj3UPgT%26c%3D%2522contemporary%2Bclinical%2Bimages%2522%2BOR%2B%2522contemporary%2Bimages%2522%2BOR%2B%2522corporate%2Bimages%2522%26i_num%3D%26i_pre%3D%26%2524%253ds%3Dhuman%2Bbody%26%2524%2Bnot%2B%2522Contemporary%2Bclinical%2Bimages%2522%2Bindex%2Bwi_collection%3D%252e%26_IXFPFX_%3Dtemplates%252ft%26%2524%253dsi%3Dtext%26t%3D%26w%3D&_IXACTION_=query&_IXMAXHITS_=1&_IXSR_=f9pk1ztUEhD&_IXSPFX_=templates%2Ft&_IXFPFX_=templates%2Ft

http://images.wellcome.ac.uk/indexplus/result.html?_IXFIRST_=156&_IXSS_=_IXFIRST_%3D1%26_IXINITSR_%3Dy%26_IXACTION_%3Dquery%26_IXMAXHITS_%3D250%26%252asform%3Dwellcome%252dimages%26%2524%253dsort%3Dsort%2Bsortexpr%2Bimage_sort%26%3D%26_IXSESSION_%3D8AirKj3UPgT%26c%3D%2522contemporary%2Bclinical%2Bimages%2522%2BOR%2B%2522contemporary%2Bimages%2522%2BOR%2B%2522corporate%2Bimages%2522%26%2524%253ds%3Dhuman%2Bdna%26%2524%2Bnot%2B%2522Contemporary%2Bclinical%2Bimages%2522%2Bindex%2Bwi_collection%3D%252e%26_IXFPFX_%3Dtemplates%252ft%26%2524%253dsi%3Dtext&_IXACTION_=query&_IXMAXHITS_=1&_IXSR_=ZdxrUOqowE2&_IXSPFX_=templates%2Ft&_IXFPFX_=templates%2Ft

What do we do?

Current Finished Sequence

Total: 3,684,395,290 bases (25th April 2008)

Data produced is made freely available to

researchers worldwide.

Case Study – Sequencing Pipeline

High throughput sequencing application pipelines to support laboratory practices.

Mixture of new bespoke and legacy systems.

Multiple Oracle databases support a collection of co-operating production application systems.

Result of multi-developer effort over a period of 10 years.

Abstract sequencing pipeline map

Mapping

Dna_receptionSubcloning

PickingPreppingSequencing

Post sequence processing

Internal trace

External trace

Assembly

Finishing

Checking

Gull/kitETS Archives Chromoview EPS

Genetrap

Exoseq

Viralsequencing

EST Back end

Devmin

Corf

Epigenomics

Restriction digests

Development requirements

Availability of “up to date” datasets for development testing.

Developer flexibility and autonomy.

Stable and robust development environment.

Ability to test component parts in isolation.

Easily define, store, recreate and test use cases.

Smoke and Mirrors – 1st solution

Using the production database system with an

additional development database to support

read/write processes.

Smoke and Mirrors - Architecture

PRODUCTION/PRIMARY

DATABASE(100GB)

DEVELOPMENTDATABASE

(20GB)

PHYSICALSTANDBY

DATABASE

PHYSICALSTANDBY

DATABASE(100GB)

Archive Redo Logs

Manual copies of datasets

ORACLE 9i (9.2.0.5) on Compaq Tru64 UNIX V5.1

Smoke and Mirrors - Setup

Development DB

DS1

PrivatesynonymsAcross dblinks.

Publicsynonyms

Production/Primary DB

Production schema

SS1

● Setup new “special_” user on the production database i.e. SS1 ● Setup new user on the development database i.e. DS1



Publicsynonyms

Development schemaswith read/write Tables

Development DB


Production schema

SS1

DS1

Database linksT1

T

Triggers replace FK which reference R/O production tables

● Isolate read/write (r/w) and read only (r/o) table access for an application system.

● Foreign keys replaced with triggers (where applicable).



Publicsynonyms


Development DB


Production schema

SS1

DS1

Database links

Database linksS T1

T

T1@


● For r/w access - Setup private synonyms over database links to the new development user.

● For r/o access – Use public synonyms from production schemas or create private synonyms.


● Run the software application using the “special_” new user login.


Publicsynonyms


Development DB


Production schema

SS1

DS1

Database links

Database links

Application Login

S T1T

T1@



Run SQL check to ensure there are no leakages using the new special_ user.

select * from all_tab_privs where grantee in (select granted_role from user_role_privs where username = (SELECT USER FROM DUAL))UNIONselect * from all_tab_privs where GRANTEE = (SELECT USERFROM DUAL)UNIONselect * from all_tab_privs where GRANTEE = 'PUBLIC' and TABLE_SCHEMA not in ('SYS', 'SYSTEM', ‘WMSYS’, 'EXFSYS', 'DMSYS', 'XDB’) and PRIVILEGE != 'SELECT';

Smoke and Mirrors

Advantages

Current dataset available for R/O access.

Less disk space required for development database.

Quick setup.

Disadvantages

Possibility of leakage into production database.

Possible performance issues across database links.

Less developer autonomy.

Logical Standby – 2nd solution

Using the logical standby technology, the

development schemas and production

schemas are within the same database without

possibility of leakage onto the production

database.

Logical Standby - Architecture

ORACLE 10GR2 (10.2.0.2) on SUSE LINUX (x86_64) SLES9

PRODUCTION/PRIMARY

DATABASE(100GB)

LOGICALSTANDBY

DATABASE(120GB)

PHYSICALSTANDBY

DATABASE

PHYSICALSTANDBY

DATABASE(100GB)

Archive Redo Logs

Archive Redo Logs

SQLapply

Logical Standby - Setup

Prepare the Production DB to support a

logical standby.

Determine support for Datatypes and storage attributes.

Ensure table rows can be uniquely identified.

Logical Standby – Setup

Production DB - Dataguard parameters

db_name=‘DB1’db_unique_name=‘DB1’log_archive_config='DG_CONFIG=(DB1,DB3)' log_archive_dest_3='service=DB3 valid_for=(online_logfiles,primary_role) db_unique_name=DB3 optional reopen=15' log_archive_dest_state_3=ENABLE

Create physical standby using RMANDuplicate target database for standby dorecover;

Stop Redo apply.

Build redo dictionary on production db

execute DBMS_LOGSTDBY.BUILD


Physical Standby - Dataguard parameters

*.db_name = ‘DB1’*.db_unique_name=‘DB3‘*.fal_client=‘DB3'*.fal_server=‘DB1'*.standby_file_management='AUTO'


On the Standby

Transition the Physical to a Logical standby

Alter database recover to logical standby new_dbname;

Shutdown and amend the pfile parameters.*.db_name=‘DB3‘

*.standby_archive_dest='/oracle/lnnn/logstby/DB3'



●Create a new password file.

●Create new SPFILE.

●Open database resetlogs.

●Commence the SQL apply process.

Alter database start logical standby apply;

Logical Standby – Development Setup

Alter Dataguard level to STANDBY.

Create separate tablespaces for new development users and objects.

Assign space quotas for each new user’s development schema.

Create new development user accounts in a standard format. *

Logical Standby - Development Setup

Create tables in development schema for read write (r/w) access.

Use tables maintained by SQL apply process for read only (r/o) access.

Foreign keys are replaced with triggers (where applicable). **

Run application with development user login.

Logical Standby – Development Setup

Logical Standby DB

Archived redolog transfer


Production schema

Production schema

DS1

Public synonyms

Development schema areasPrivate synonyms.Local objects with R/W access.Triggers replace FK. Use public synonyms for R/O table access.

R/O

R/W

Application login

T1

T

S

SQLapply

Create additional development users in a standard format using the USER_ADMINISTRATION package

PROCEDURE CREATE_USER

Argument Name Type In/Out Default?

------------------------------ ---------------------- ------ --------

P_DEBUG BOOLEAN IN DEFAULT

P_USERNAME VARCHAR2(30) IN

Development Area - Setup


PROCEDURE DROP_USER

Argument Name Type In/Out Default?------------------------------ ----------------------- ------ --------P_DEBUG BOOLEAN IN DEFAULTP_USERNAME VARCHAR2(30) IN

PROCEDURE VALIDATE

Argument Name Type In/Out Default?------------------------------ ----------------------- ------ --------P_DEBUG BOOLEAN IN DEFAULTP_USERNAME VARCHAR2(30) INP_MODE VARCHAR2 IN

Development Area – Setup

Package Usage Examples:-

exec USER_ADMINSTRATION.CREATE_USER(TRUE,’DEV_user’);

Output as follows:-

CREATE USER DEV_user IDENTIFIED BY user DEFAULT TABLESPACE DEV_TB_01 TEMPORARY TABLESPACE DEV_TMP_01 QUOTA 1024M ON DEV_TB_01 GRANT RESOURCE, CONNECT, DEV_USER_ADMINISTRATION TO DEV_user CREATE SYNONYM DEV_user.QUICK_FK_TRIGGER_PACKAGE GRANT EXECUTE ON QUICK_FK_TRIGGER_PACKAGE TO DEV_user

NO foreign keys permitted from the development sandbox to SQL apply maintained schemas.

2 solutions:Copy the required table locally and create

the FK within the development area. Use a package which replaces FK's with

triggers.


Create triggers to replace foreign keys between developer sandboxes and schemas maintained by SQL apply process using the QUICK_FK_TRIGGER package.

PROCEDURE QUICK_CREATE_FK_TRIG

Argument Name Type In/Out Default?------------------------------ ----------------------- ------ --------LOCAL_TABLE VARCHAR2 INREFERENCE_TABLE VARCHAR2 INREFERENCE_DBLINK VARCHAR2 IN DEFAULTDEBUG NUMBER IN DEFAULT


Development Area – Setup

Package Usage Examples:-

exec QUICK_FK_TRIGGER_PACKAGE.

QUICK_CREATE_FK_TRIG('DEV_user.table_name.column_name',‘ref_user.table_name.column_name',[db.domain|NULL],1);

e.g.QUICK_FK_TRIGGER_PACKAGE.QUICK_CREATE_FK_TRIG('DEV_USER.FINISH_BATCH.PROJECTNAME',‘REF_USER.PROJECT.PROJECTNAME','testdb.world',1);


CREATE OR REPLACE TRIGGER AG_PROJECTNAME_FINISH_B_BR_IUBEFORE INSERT OR UPDATE ON FINISH_BATCHFOR EACH ROWDECLAREFK_ENTRY FINISH_BATCH.PROJECTNAME%TYPE;BEGIN------------------------------------------ AUTO GENERATED TRIGGER FROM THE-- QUICK_FK_TRIGGER_PACKAGE---- NAME: AG_PROJECTNAME_FINISH_B_BR_IU-- AUTH: kva-- DATE: 05-DEC-07----------------------------------------FK_ENTRY:= :NEW.PROJECTNAMEIF FK_ENTRY IS NOT NULL THENQUICK_FK_TRIGGER_PACKAGE.QUICK_FK_CHECKS(FK_ENTRY,’REF_USER',‘PROJECT',‘PROJECTNAME',‘TESTDB.WORLD');END IF;END AG_PROJECTNAME_FINISH_B_BR_IU;


PROCEDURE QUICK_FK_CHECKS

Argument Name Type In/Out Default?------------------------------ ----------------------- ------ --------VAL VARCHAR2 INREFOWNER VARCHAR2 INREFTABLE VARCHAR2 INREFCOL VARCHAR2 INREFDBLINK VARCHAR2 IN DEFAULTDEBUG NUMBER(38) IN DEFAULT


PROCEDURE QUICK_FK_CHECKS

If no DBLINK specified

SELECT projectname FROM ref_user.project WHERE projectname = ‘PROJECT1’;

If DB link specified

SELECT projectname FROM [email protected] WHERE projectname= ‘PROJECT1;

EXCEPTION RAISED FOR MISSING ENTRY

RAISE_APPLICATION_ERROR(-20000,'Entry does not exist for PROJECT1 - MISSING ENTRY IN ref_user.project.projectname');

Any attempt to alter the schema being maintained by the SQL apply process, raises the following error:

ORA-16224: DATABASE GUARD IS ENABLED


Logical StandbyAdvantages

Current dataset for RO access.

All development contained within one database.

More developer autonomy. No direct interaction with

the production DB. Full copy of production

data.

Disadvantages

More disk space required (1 database plus development area)

Security of sensitive production data.

Maintenance & Monitoring

Skipping sensitive schemas by executing the DBMS_LOGSTDBY.SKIP package.

exec dbms_logstdby.skip('SCHEMA_DDL',‘schema','%');

exec dbms_logstdby.skip('DML',‘schema','%');

Maintenance & Monitoring

Various views to monitor the processes which maintain the Logical Standby database.

DBA_LOGSTDBY_EVENTS Records SQL apply events

DBA_LOGSTDBY_LOG Details archived logs processed

V$LOGSTDBY_STATS SQL apply statistics

V$LOGSTDBY_PROCESS Processes involved with SQL apply process

V$LOGSTDBY_PROGRESS Progress made by SQL apply process

V$LOGSTDBY_STATE Current state of SQL apply process

Summary

Both solutions provide access to current datasets for application integration and development testing.

The Smoke and mirrors solution is relatively quick to setup.

Logical standby is a more stable solution and provides more development flexibility and scaling.

Conclusion

The Logical Standby has currently proved to be

a more stable and popular solution for our

immediate development requirements.

Future Plans

Investigation of Rapid Application Testing (RAT), new in Oracle 11g.

Web References

LOGICAL STANDBY CREATION -10gR2 (ORACLE)http://download.oracle.com/docs/cd/B19306_01/server.102/b14239/create_ls.htm#SBYDB00300

LOGICAL STANDBY BY Julian Dykehttp://64.233.183.104/search?q=cache:2VixdNn9HN0J:julian.dyke.users.btopenworld.com/com/Presentations/LogicalStandby.ppt+logical+standby+-+julian+Dyke&hl=en&ct=clnk&cd=2&gl=uk

LOGICAL STANDBY MANAGEMENT -10gR2 (ORACLE)http://download.oracle.com/docs/cd/B19306_01/server.102/b14239/manage_ls.htm

http://download.oracle.com/docs/cd/B19306_01/server.102/b14239/create_ls.htm#SBYDB00300

http://download.oracle.com/docs/cd/B19306_01/server.102/b14239/create_ls.htm#SBYDB00300

http://64.233.183.104/search?q=cache:2VixdNn9HN0J:julian.dyke.users.btopenworld.com/com/Presentations/LogicalStandby.ppt+logical+standby+-+julian+Dyke&hl=en&ct=clnk&cd=2&gl=uk



http://download.oracle.com/docs/cd/B19306_01/server.102/b14239/manage_ls.htm

http://download.oracle.com/docs/cd/B19306_01/server.102/b14239/manage_ls.htm

Acknowledgements

DBA group @Sanger Institute (UK) Production Development group @Sanger

Institute (UK) Wellcome Trust Sanger Institute (UK)

Questions?

Contact [email protected], if you require more information.

mailto:[email protected]

Documents

Software Development using Production Data By Karen Ambrose Wellcome Trust Sanger Institute (WTSI, UK) Monday 28 th April 2008 – OLSUG, BOSTON, MA