Prligence Empowering Intelligence A Multi-Source Time-Variant Datawarehouse Case Study Session#...

pr ligence Empowering Intelligence

AMulti-Source Time-Variant

DatawarehouseCase Study

Session# 36605

byArup Nanda

Proligence, Inc.Norwalk, CT

Objectives

• Exploring DW Techniques in Oracle• Case Study• Oracle 10G Additions

Datawarehouse

Cust10

Cust11 ?

Cust5 Cust6

DB3 DB6DB4 DB5

DB1 DB2

A Real Life Case

• Claims Datawarehouse• Several Customers/Sources• Several Quarters• Data Volume Was High• Irregular Frequency• Data Comes Often Late• Near Real Time Requirements

DetailTable

SummaryTable

DetailTable

Problem of Irregular Data

DBMS_MVIEW.REFRESH (…)

Problems

• Incoming Data Irregular• Summary Tables Need Refreshing• Quarters Added Continuously• Archival Requirements Vary Across Customers• Quick Retrieval of Archival Needed

Problems contd.

• Summary on Summary Tables as Materialized Views

• Need Refresh Whenever New Data Arrives• Or When Data is Purged/Reinstated• Customers Added and Deleted Frequently

Objective

• To Minimize Downtime for Refreshes– Incrementally Refresh– Partitioning Techniques

• To Add Customers Easily• To Add Quarters Easily• To Archive Off and Purge Easily and Atomically• To Restore Archives Quickly

Objective contd.

• To have an ETL Setup for Easy Addition of Objects Such As Tables, Indexes, Mat Views.

• Use Only Available Oracle and Unix Tools– PL/SQL– Unix Shell Scripts– SQL*Plus

Design

• Varying Dimensions – – Customer – Quarter

• Composite Partitioning– Range (for Quarters)– List (for Customers)

• Local Indexes

Partitioning

• Partitioned on CLAIM_DATE– RANGE– Partitioned named YyyQq– Storage Clauses Not Defined

• Supartitioned on CUST_NAME– LIST– Named YyyQq_CustName, e.g. Y03Q3_CUST1

Indexing

• All Indexes LocalCREATE INDEX IN_CLAIM_SUM_01

ON SUMTAB1 (COL1, COL2)…

• No Indexes UNIQUE and GLOBAL

Storage

Each Subpartition – of Index or Table is kept in separate tablespaces named in the format

Y<Year>Q<Qtr>_<CustName>_DATAe.g. Y02Q2_CUST1_DATA Y02Q2_CUST2_DATA

Y03Q3_CUST1_DATA

Quarter

s Cust3Y03 Q3

In Tablespace Y03Q3_CUST3_DATA

Cust3Y03 Q3

In Tablespace Y03Q3_CUST3_INDX

Tablespace

create tablespace y03q3_cust1_data

datafile ‘/oradata/y03q3_cust1_data_01.dbf’

size 500m

autoextend on next 500m

extent management local

segment space management auto

Table DDLCREATE TABLE TAB1( … )PARTITION BY RANGE (CLAIM_DATE)SUBPARTITION BY LIST (CUST_NAME) ( PARTITION Y03Q1 VALUES LESS THAN (TO_DATE(‘2003/04/01’,’YYYY/MM/DD’)), ( SUBPARTITION Y03Q1_CUST1 VALUES (‘CUST1’) TABLESPACE Y03Q1_CUST1_DATA, SUBPARTITION Y03Q1_CUST2 VALUES (‘CUST2’) TABLESPACE Y03Q1_CUST2_DATA, … and so on for all subpartitions … SUBPARTITION Y03Q1_DEF VALUES (DEFAULT) TABLESPACE USER_DATA ), PARTITION Y03Q2 VALUES LESS THAN (TO_DATE(‘2003/07/01’,’YYYY/MM/DD’)), ( SUBPARTITION Y03Q2_CUST1 VALUES (‘CUST1’) TABLESPACE Y03Q2_CUST1_DATA, SUBPARTITION Y03Q2_CUST2 VALUES (‘CUST2’) TABLESPACE Y03Q2_CUST2_DATA, … and so on for all subpartitions … SUBPARTITION Y03Q2_DEF VALUES (DEFAULT) TABLESPACE USER_DATA ), … and so on for all the partitions … PARTITION DEF VALUES LESS THAN (MAXVALUE), ( SUBPARTITION DEF_CUST1 VALUES (‘CUST1’) TABLESPACE USER_DATA, SUBPARTITION DEF_CUST2 VALUES (‘CUST2’) TABLESPACE USER_DATA, … and so on for all subpartitions … SUBPARTITION DEF_DEF VALUES (DEFAULT) TABLESPACE USER_DATA ))

Index DDLCREATE INDEX IN_TAB1_01 ON TAB1 (COL1)LOCAL NOLOGGING( PARTITION Y03Q1 ( SUBPARTITION Y03Q1_CUST1 TABLESPACE Y03Q1_CUST1_INDX, SUBPARTITION Y03Q1_CUST2 TABLESPACE Y03Q1_CUST2_INDX, … and so on for all subpartitions … SUBPARTITION Y03Q1_DEF TABLESPACE USER_DATA ), PARTITION Y03Q2 ( SUBPARTITION Y03Q2_CUST1 TABLESPACE Y03Q2_CUST1_INDX, SUBPARTITION Y03Q2_CUST2 TABLESPACE Y03Q2_CUST2_INDX, … and so on for all subpartitions … SUBPARTITION Y03Q2_DEF TABLESPACE USER_DATA ), … and so on for all the partitions … PARTITION DEF ( SUBPARTITION DEF_CUST1 TABLESPACE USER_DATA, SUBPARTITION DEF_CUST2 TABLESPACE USER_DATA, … and so on for all subpartitions … SUBPARTITION DEF_DEF TABLESPACE USER_DATA ))

Creating DDLs

StaticPart

VariablePart

create table tab1(………)

partition y03q1 (subpartition y03q1_cust1 tablespace …)

DDL toCreateTable

Constraints

Constraints defined asDISABLE NOVALIDATE RELY

ALTER TABLE … ADD CONSTRAINT …RELY DISABLE NOVALIDATE;

Constraint

• VALIDATE/NOVALIDATE– Table TAB1 (Column: STATUS)– Current Values A, I, F– Check Constraint: STATUS IN (‘A’,’I’)

• ENABLE/DISABLE– New Value ‘F’

• RELY

Reasons• To Include Relation Information to the

Metadata• To Enable Query Rewrite

Summary Tab and View

Summary Table

CUST_NAMECLAIM_DATEPROVIDER_IDNUM_CLAIMSNUM_LINES

ViewSELECT‘CUST1’ AS CUST_NAME,CLAIM_DATE,PROVIDER_ID,COUNT(DISTINCT CLAIM_ID) AS NUM_CLAIMS,COUNT(*) AS NUM_LINESFROM ….GROUP BY …

On Source

Casting

SELECT CAST (CUST_NAME AS VARCHAR2(20))AS CUST_NAMEFROM <viewname>

CAST (column_name AS datatype (precision))

SummaryTable

ViewView

Owned byCust Schema

TemporaryTable

Index ofTemporary

Massaging

Analyzing

Filter:Where CLAIM_DATE

is in that quarter

For Customer Cust1 and Quarter Q1

ViewView

Old Sub Partition

ALTER TABLE … EXCHANGE SUBPARTITION subpartnameWITH TEMPTABLEINCLUDING INDEXES

Technique

• Not Using DBMS_MVIEW.REFRESH• MV is always STALE

Temp TableCREATE TABLE T1_Y03Q1_CUST1TABLESPACE Y03Q1_CUST1_DATAPARALLEL 8 NOLOGGING ASSELECT …FROM CUST1.VIEW1@DB1WHERE CLAIM_DATE >=add_months(trunc(to_date(‘03','RR'),'YYYY'), 3*(to_number(‘1')-1))and batch_date < last_day(add_months(trunc( to_date(‘03','RR'),'YYYY'), 3*(to_number(‘1')) - 1 )) + 1

ScriptCREATE TABLE T1_Y&&YY.Q&&Q._&&CUSTTABLESPACE Y&&YY.Q&&Q._&&CUST._DATAPARALLEL 8 NOLOGGING ASSELECT …FROM &&CUST..VIEW1@&&DBLINKWHERE CLAIM_DATE >=ADD_MONTHS(TRUNC(TO_DATE('&&YY','RR'),'YYYY'), 3*(TO_NUMBER('&&Q')-1))AND BATCH_DATE < LAST_DAY(ADD_MONTHS(TRUNC( TO_DATE('&&YY','RR'),'YYYY'), 3*(TO_NUMBER('&&Q')) -1 )) + 1

External Table

ReasonSource is a non-Oracle DB, e.g. DB2Source is External, no DB Link Allowed

Fixed Format –vs- DelimitedFixed Format

Faster, EasierMore Space

DelimitedLess SpaceSlower, Slightly More Complex

Massaging

• Removing NOT NULL Constraints• Making Datatypes Consistent

– The CAST operation converts NUMBER(m,n) to NUMBER

– cast(col1 as number(10,2)) as col1_m

– COL1 NUMBER(5,2)– COL1_M NUMBER

Analyzing• Using DBMS_STATS.GATHER_TABLE_STATS• PARALLEL Degree

dbms_stats.gather_table_stats ( ownname => ‘DWOWNER', tabname => '&&TABNAME', estimate_percent => dbms_stats.auto_sample_size, method_opt => 'FOR ALL INDEXED COLUMNS SIZE AUTO', degree => dbms_stats.default_degree, cascade => TRUE );

Mat Views

MVs Created as TablesCREATE TABLE MV_SUMMTAB1

Storage clauses just like the underlying tableCREATE MATERIALIZED VIEW

MV_SUMMTAB1

ON PREBUILT TABLE

AS SELECT ……

http://www.proligence.com/painless_alter.pdf

Query Rewrite

Table SUM_CLAIMS PROVIDER_ID, STATE, TYPE, TOT_AMT

Table MV_SUM_CLAIMS PROVIDER_ID, STATE, SUM(TOT_AMT) TOT_AMTGROUP BY PROVIDER_ID, STATE

SELECT SUM(TOT_AMT) FROM SUM_CLAIMSSELECT SUM(TOT_AMT) FROM

MV_SUM_CLAIMS

Query Rewrite

Init.ora Parametersquery_rewrite_enabled='TRUE'query_rewrite_integrity='STALE_TOLERATED‘

ENFORCED – Rewrite only if guaranteedTRUSTED – Uses only if RELYSTALE_TOLERATED – Even if not RELY

Checking QR

dbms_mview.explain_rewrite (‘select cust_name, count(*) from summtab1 group by

cust_name’ );

select message from rewrite_table;

QSM-01033: query rewritten with materialized view, MV_SUMMTAB1

QSM-01101: rollup(s) took place on mv, MV_SUMMTAB1

Design …

MV_* subpartitions are on the same tablespace as the parents.

Subparts of MV_SUMMTAB1_0? are in the same TS as SUMMTAB1

Subparts of MV_SUMMTAB2_0? in SUMMTAB2

Custom

me Quarter

PARENT

TableSpace1 TableSpace2

MV and Parents

• Partition Pruning• Partition-wise Joins• Partition Independence

Adding Quarters/Customers

• Partition– Default Partition – VALUES LESS THAN

(MAXVALUE)• Subpartition

– Default Subpartition – VALUES (DEFAULT)

Qtr1 Qtr2 Qtr3 DEF

alter table … split subpartition

Qtr1 Qtr2 Qtr3 DEF

Qtr1 Qtr2 Qtr3 Qtr4

alter table … split partition

Backup/Restore

• Backup– ALTER TABLESPACE <TSName> READ ONLY– Copy the files to tape/CD.

• Restore– Copy the file back into the directory– ALTER TABLESPACE <TSName> RECOVER

Archival/Purge

SP1 SP2 SP3Table

Table4

SP1 SP2 SP3 SP4Table Table4

Table4

Archival/PurgeCREATE TABLE S1_Y<yy>Q<q>_<CustName>TABLESPACE Y<yy>Q<q>_<CustName>_<TSType> AS SELECT * FROM SUMMTAB1 WHERE 1=2/CREATE INDEXES, CONSTRAINTS, etc./ALTER TABLE SUMMTAB1 EXCHANGE SUBPARTITION Y<yy>Q<q>_<CustName> WITH TABLE Y<yy>Q<q>_<CustName> INCLUDING INDEXES/

Check TTS

ALTER TABLESPACE Y<yy>Q<q>_<CustName>_<TSType> READ ONLY;

DBMS_TTS.TRANSPORT_SET_CHECK ( <DataTS>,<IndexTS>) ;

SELECT * FROM TRANSPORT_SET_VIOLATIONS;

Transport TS

Export Parameter FileTRANSPORT_TABLESPACE=y TTS_FULLCHECK=YFILE=‘<FileLocation>/exp<TS>.dmp’TABLESPACES=(<DataTS>, <IndexTS>)

Copy the exp.dmp and Datafiles to tape/CD.

Drop SubpartitionDrop the Tablespace

DROP TABLESPACE <TSName> INCLUDING CONTENTS AND DATAFILES;

Restore

• ALTER TABLE SPLIT SUBPARTITION <DefaultSP>

• Copy Datafiles & Export Dump Files from CD/Tape

• Import Parameter FileTRANSPORT_TABLESPACES=YTABLESPACES=(<DataTS>,<IndexTS>)DATAFILES=(…)

Minimizing Refresh Unit

• Months – instead of quarters refreshed at a time.• Last Quarter Split into a Subpartition per Month• Naming Convention

– YyyQqMmm– Y03Q3M09

• Merge Subpartition

Merging Subpartitions

• Index Subpartitions Created in User’s Default Tablespace

• Subpartition TemplateALTER TABLE SUMTAB1 ADD SUBPARTITION

TEMPLATE

Resumable Statement• When?

– Running Large Report Jobs– Creating Large Indexes

• ALTER SESSION ENABLE RESUMABLE NAME ‘Job1’;• View DBA_RESUMABLE

– NAME – Name specified in ALTER SESSION– COORD_SESSION_ID – Coord Session in PQ– SQL_TEXT – The text of the SQL– STATUS - RUNNING, SUSPENDED, ABORTED, ABORTING,

TIMEOUT – ERROR_NUMBER/ERROR_MSG

Objectives Revisited

• To Minimize Downtime for Refreshes– Incrementally Refresh– Partitioning Techniques

• To Add Customers Easily• To Add Quarters Easily• To Archive Off and Purge Easily and Atomically• To Restore Archives Quickly

Oracle 10G

• Transportable Tablespaces Can Be Reinstated At a Different Operating System– Can be used for Restoring to a Different OS

• Tablespaces Can Be Renamed– Restoring Tablespace of the Same Name

• Multiple Temporary Tablespace– For Large Index Creation, Sorting, etc.

Oracle 10G contd.

• Partition Change Tracking Support for List Partitioning

• Query Rewrites Can Use Multiple MVs• OEM Shows All Partitioning Features• Data Pump

– Export/Import on Steroids– Parallel Operation

Oracle 10G contd.

• External Table Download– A Utility to Create File from Table Data

CREATE TABLE …

ORGANIZATION EXTERNAL

AS SELECT * FROM <a query>

– Platform Independent File– Can Be Used In External Tables

Thank You!

Please Submit ReviewsSession# 36605

www.proligence.cowww.proligence.comm

Prligence Empowering Intelligence A Multi-Source Time-Variant Datawarehouse Case Study Session#...

Documents

Microservices using Python and Oracle - Proligence › pres › ioug18 › microservices.pdf · Microservices with Python and Oracle 2 What’s the problem we are trying to solve?

Are Indexes Unnecessary in Exadata? - Proligence Home · Are Indexes Unnecessary in Exadata? Arup Nanda Longtime Oracle DBA (and now DMA)

Dorschel, B. , Hellmer, H. , Schröder, M. Summer …epic.awi.de/36605/1/Bornemann-etal_SCAR_2014.pdf · 3 MuseoArgentino de CienciasNaturales “Bernardino Rivadavia”, Buenos Aires,

Handbook of Food Chemistry - Springer978-3-642-36605-5/1.pdf · Handbook of Food Chemistry ... This Handbook is intended to be a comprehensive reference for ... chemistry in the areas

Plan Stability, Baselines, and SQL Plan - Proligence... 2009 Arup Nanda 1 Plan Stability, Baselines, and SQL Plan ... • e.g. is EMP a table or a synonym ... •Table: SQL> select

Partitioning What, When, Why & How - Proligence Home · 2008-09-28 · Partitioning – What, When, Why and How by Arup Nanda 3 About this Session zThis is not an introduction to

SUMMER 14 Hadoop - Proligence · predict in advance, at least accurately, the exact information you will expect to see in them. So, putting a structure around the data storage not

Case Study of Upgrade - Proligence · 2009. 12. 8. · Concocting Prod Work Workload generator tools such as Load Runner can simulate user actions, yCt likt bCapture clickstream on

Networking for the Cloud DBA - Proligence · 2018-04-24 · Networking for the Cloud DBA Arup Nanda Longtime Oracle DBA And Explorer of New Things

Art and Craft of Tracing - Proligence · Art and Craft of Tracing Arup Nanda Longtime Oracle DBA DocID91201 Date 161022 Agenda The Art and Craft of Tracing 2 My session or application

understanding oracle locking - Proligence Oracle Locking Internals 16. ... where rowid = dbms_rowid.rowid_create ... • A txn marks the rows locked in the block itself

Oracle 11g New Features for DBAs - Proligence Home

Oracle 11g New Features for DBAs - Proligence · PDF fileOracle 11g New Features for DBAs Arup Nanda. 2 ... • Companion material – "Oracle Database 11g: ... Compared to QA Tools

Oracle Database 12c - Proligence · PDF file1. Pluggable Database Oracle 12c: Gee Whiz Features 3 User SIEBEL Application 1 User SIEBEL Application 2 User SIEBEL Application 3 User

Prligence Empowering Intelligence All About Fine Grained Access Control by Arup Nanda

Beginning Performance Tuning - Proligence Home · Why Most Troubleshooting Fails • Not systematic or methodical • Not looking at the right places • Confusing Symptoms with Causes

Beginning Performance Tuning - Proligence

Secure Database in a Single Day - Proligence · sudo –u oracle sqlplus / as sysdba • This leaves an audit trail of actions 8 @ArupNanda Secure your Database in 1 Day 1. Listener

· 2014-08-26 · list of des/ sdes, transferred to other circles on dated -08-2014. their regular promotion as des. st-±ff no. 30692 31964 37675 37682 37704 36605 36625 36637 36692

Location Agnostic Data - Proligence · Datawarehouse Location Agnostic Data 6 Schema A SALES SALES_HISTORY Schema B SALES_HISTORY Datawarehouse. Location Agnostic Data 7 Schema A