1

<Insert Picture Here>

1TB/20minutes with Exadata

The Real World Performance Team

<Insert Picture Here>

Real World Performance DW/BI Agenda

• The DW/BI Death Spiral

• Data Loading and Movement

• ETL and DML

• Resource Management and Query Queuing

3

The DW/BI Death Spiral

4

The DW/BI Death Spiral

Data Warehouse Death Spiral

• The Wrong HW pathology

– Too few CPUs to drive multi million row queries 10X

– I/O not balanced with CPUs 10X

– Incorrect Networking strategies 10X

• Queries

5

• Queries

– Incorrect Query strategy 10X

• ETL and Loading Degraded

– Slowdowns and serialization

– Database size bloat

• Desire to make the systems look productive

– Overloaded query streams 10X

Data Warehouse Death Spiral

What we see

• HW procurement process wrong

• Serial query execution strategy based upon OLTP

experience

• Database bloat due to indexes and aggregates

• ETL and loads running too slow

6

• ETL and loads running too slow

• No ability to handle Ad Hoc queries

• Hard to get CPUs busy so high concurrent number of

query streams

Data Loading and Movement

7

Oracle Retail Data Warehouse Schema

8

Retail Demonstration

Table Sizes

TABLE SIZE OF SOURCE DATA NUMBER OF RECORDS

• Transactions • 52 GByte • 461M

• Payments • 54 GByte • 461M

9

• Line Items • 936 Gbyte • 6945M

• Total • 1042 GByte • 7867M

Retail Demonstration

Table Sizes (Default Compression)

TABLE SIZE OF TABLE COMPRESSION RATIO

• Transactions • 30 GByte • 1.77 : 1

• Payments • 30 GByte • 1.84 : 1

10

• Line Items • 268 Gbyte • 3.55 : 1

• Total • 327 GByte • 3.23 : 1

Retail Demonstration

Table Sizes (With HCC)

TABLE SIZE OF TABLE COMPRESSION RATIO

• Transactions • 5 GByte • 7.00 : 1

• Payments • 5 GByte • 7.60 : 1

11

• Line Items • 54 Gbyte • 12.85 : 1

• Total • 64 GByte • 11.98 : 1

NOTE: The compression ratio calculated using size w/o compression

Data Loading

Bulk Loading Challenges

• Problem: Moving data to the database host machine

– For high load rates the data staging machine and network

becomes the serialization point/bottleneck

– Increased network and staging area I/O bandwidth is an

expensive option

• Solution: Compress the source data files

12

• Solution: Compress the source data files

– Compression reduces the number of bytes copied from disk

and over the network

Data Loading

Consider the Data Transfer Rate

• What would it take to load 1 TByte in one Hour?

– 17 GByte/minute or 291 MByte/second

• This is higher than the specification of most networks

and any portable drive

• So compression of source data becomes crucial

13

• So compression of source data becomes crucial

– 1057 Gbyte � 136 Gbyte (7.7x compression)

– 2.3 GByte/minute or 40 MByte/s

• This eliminates the first challenge to migrating data

• Extraction of the data from legacy systems often

takes much longer than this!

Data Staging

Data Sources

SOURCE THROUGHPUT

• USB Drive • 20 MByte/s

14

• Local Disk • 30-40 MByte/s

• Scalable NFS Server• Potentially at Network

Speeds

• DBFS• Fastest ( assuming data

has been copied !)

Data Loading

Bulk Loading Challenges

• Problem: Data loading is CPU/Memory Constrained

– Data loads scale well over multiple CPUs, cores and hosts

(assuming no other form of contention)

– Memory usage for meta data associated with highly

partitioned objects can become significant at high DOP

• Solution: Use the correct tools and plan accordingly

15

• Solution: Use the correct tools and plan accordingly

– Use external tables with a parallel SQL statement (e.g. CTAS

or IAS) to minimize on-disk and in-memory meta data. Do

NOT use multiple copies of SQL*Loader

– Data types for columns have a huge impact on the CPU

required to load the data. Raw is the cheapest and

Timestamp is the most expensive.

Loading Data

Measure the Impact of Compression

• Compression incurs costs when loading

– Increased CPU time

– Increased elapsed time

• Compression provides benefits

– For scans

16

– For scans

– For backup and recovery

• Write-once and Read-many means that compression

is a net benefit, not a cost

Loading Data

Measure the Impact of Partitioning

• Partitioning incurs costs when loading

– Increased CPU time

– Increased elapsed time

• Partitioning provides benefits

– For queries

17

– For queries

– For manageability

• Write-once and Read-many means that partitioning is

a net benefit, not a cost

Data Loading

Anatomy of an External Table

create table FAST_LOAD(column definition list ... )organization external(type oracle_loaderdefault directory SPEEDY_FILESYSTEMpreprocessor exec_file_dir:’zcat.sh’characterset ‘ZHS16GBK’ badfile ERROR_DUMP:’FAST_LOAD.bad’

External Table Definition

Reference the Mount Point

Uncompress the data using a secure wrapper

The Characterset

18

badfile ERROR_DUMP:’FAST_LOAD.bad’logfile ERROR_DUMP:’FAST_LOAD.log’(file column mapping list ...)location(file_1.gz, file_2.gz, file_3.gz, file_4.gz ) reject limit 1000parallel 4/

The Characterset must match the Characterset of the Files

Note Compressed Files

Parallel should match or be less than the number of Files

Data Movement and Loading Summary

• Data Movement

– Compression is essential

– Device and Network Speeds need to be evaluated and understood.

– Need to strategize for large tables vs large number of tables ( Data

vs Meta data movement )

• Data Loading

19

• Data Loading

– External Tables

– Data Pump

– Restore from backups, transportable tablespaces

• Data Unloading

– Tools

• Home grown eg. Parallel Unloader

• Fastreader by WisdomForce

ETL and DML

20

ETL and DML

Validation and Transformation

• Apply referential integrity checks and business rules

by the use of SQL statements executed in parallel as

an alternative to declarative constraints

• A simple workflow may be:

– Stage the data into the database

• Basic data integrity, nulls, data types etc.

21

• Basic data integrity, nulls, data types etc.

– Check the data

• Uniqueness, foreign keys, business rules etc.

• Apply constraints with “RELY DISABLE NOVALIDATE”

– Transform the data

• Tax corrections, time zone corrections, etc.

– Always leverage parallel set-based processing

Validation Example

Set based processing vs. row by row

Time

22

Time

H:MI:SS

Validation and Transformation

Proof Points

• For the two validation processes we can now make

these claims

– Store Validation - Over 200 times faster

– Product Validation - Over 2500 times faster

• Exact same hardware

23

• Exact same hardware

– This is a case of using the wrong methodology

Set Based Processing

Move away from row by row processingdeclare

cur rec_cur;

rec type_table%rowtype;

begin

open cur for select * from table1;

loop

fetch cur into …

24

fetch cur into …

if condition(rec)

then

insert into table2 …

else

insert into table3 …

end if;

end loop;

end;

Set Based Processing

To set based processing

insert /*+ append */ into table2

select *

from table1

where condition …

25

insert /*+ append */ into table3

select *

from table1

where not condition …

Set Based Processing

There are usually multiple ways to get the same end

result

insert /*+ append */ first

when condition then

into table2 values …

else

into table3 values …

26

into table3 values …

select *

from table1

Set based processing

• Use SQL to tell the DB what to do

• Let the DB figure out how to do it

• Some SQL operations:

– create table as select

– insert /*+ append */ select

27

– insert /*+ append */ select

– intersect

– minus

– exists

– not exists

– window functions

– multi-table inserts

– outer joins

Data Validation SQL

• Elimination of duplicates

– Outer Join back to the table

– Window function

– Aggregate with HAVING clause

• Foreign Key References

28

• Foreign Key References

– Outer Joins between tables

• The choice of techniques will be dependent on the

following

– Good/Bad validation of of the data

– The desire to identify and locate bad rows e.g. find ROWIDS

– The desire to programmatically eliminate bad rows

Data Validation SQL

Uniqueness Check

Simply Check the

Data

Obtain one of the

ROWIDs of duplicates

to investigate

Query the rows you wish to

keep eliminating duplicates

based on the load time

select pk_id,count(*)from DIRTY_DATAgroup by pk_id

select pk_id,count(*),max(rowid)

select column_listfrom (select

29

group by pk_idhaving count(*)>1;

max(rowid)from DIRTY_DATAgroup by pk_idhaving count(*)>1;

select a.*,row_number() over(partition by pk_idorder by load_time desc) rowno

from DIRTY_DATA a)where rowno=1

Data Validation SQL

Orphaned Row Check

Look For Orphans Look for Parents with no Children

select C.rowidfrom CHILD C,PARENT P where C.pk_fk = P.pk_id(+)and P.pk_id is null;

select P.rowidfrom PARENT P,CHILD Cwhere P.pk_id = C.pk_fk(+) and C.pk_fk is null;

30

Data Transformation SQL

Transformation vs. Modification

Driver Transformation Modification

Compression No Impact Compression may be Lost and severely impact

performance

Fragmentation None Fragmentation, row chaining, and holes will almost

certainly take place

Logging and UNDO Possible to eliminate Will take place and may impact performance and

administration requirements

Indexes Indexes need to be rebuilt if used Indexes will be maintained in place. This may be a

31

Indexes Indexes need to be rebuilt if used Indexes will be maintained in place. This may be a

performance overhead and Bit Map indexes may

become fragments and require rebuilding

Meta Data Grants etc will require redefinition No impact

Space Overhead of maintaining 2 copies of the data Overhead of UNDO and Logging

Coding New code required writing and new techniques

need teaching

Old Code runs with performance challenges

3rd Party Issues May not be supported by Tool Vendors

Data Transformation SQL

• Use either

– INSERT /*+APPEND */ INTO … SELECT

– CREATE TABLE … AS SELECT

• It is recommend that you create the destination table

first and then use INSERT

32

first and then use INSERT

– Constraints such as NOT NULL can be correctly applied and

enforced

– Data type, column lengths and precision can be defined and

preserved

Rewriting DML

DELETE

alter session enable parallel dml/delete from tx_logwheresymbol = ‘JAVA’

/commit/

create table tx_log_newnologgingparallel compress for all operationsasselect * from tx_logwheresymbol != ‘JAVA’

/alter table tx_log rename to tx_log_old

33

alter table tx_log rename to tx_log_old/alter table tx_log_new to tx_log/

The predicate is the compliment of the DELETE, it selects the rows to keep

Rewriting DML

UPDATE

alter session enable parallel dml/update sales_ledgerset tax_rate = 9.9where tax_rate = 9.3and sales_date > ‘01-Jan-09’/commit/

create table tx_log_newnologgingparallelcompress for all operationsasselect., case

sales_date>‘01-Jan-09’ and

34

/ andtax_rate=9.3

then 9.9

else tax_rate

end,.from sales_ledger/

The UPDATE predicates are moved to the SELECT list in a CASE statement to transform the rows

Rewriting DML

MERGE

• This combines techniques from UPDATE and

DELETE and may also include an INSERT

• For bulk MERGES join the input and merged table as

a FULL OUTER JOIN and then case/decode the

NULL columns to apply DELETE, INSERT and

35

NULL columns to apply DELETE, INSERT and

UPDATE components

ETL and DML Challenges in the Real World Debate

• The performance gains are often 2-3 orders of

magnitude

• The challenge is getting customers to re-write code to

exploit these techniques

• You are often dealing with an OLTP vs. DW culture

36

• You are often dealing with an OLTP vs. DW culture

issue

• Your competitors Netezza and Teradata are very

good at rewriting serial, row-by-row ORACLE code to

parallel transformations. You may have to do it

anyway to compete.

ETL and DML Summary

• Data validation and modification

– Best executed in the database via SQL

– This presents big challenges to users who have committed to

classic ETL tools such as Informatica

– Changes of data are best made via transformation and

redefinition than via classic OLTP DML statements ( delete,

37

redefinition than via classic OLTP DML statements ( delete,

update, merge )

• Allows exploitation of HW and parallelism

• Minimizes fragmentation and maximizes compression

• Minimizes logging and minimizes recovery

– Set based techniques use efficient CPU and IO techniques

Ad Hoc Query

Question

“What were the most popular items in the baskets of

shoppers who visited stores in California in the first

week of May and didn't buy bananas?”

38

Ad Hoc Query

SQLwith qbuy as

( select rt.TRX_NBR

from DWR_ORG_BSNS_UNIT obu, DWB_RTL_TRX rt, DWB_RTL_SLS_RETRN_LINE_ITEM rsrli, DWR_SKU_ITEM sku

where obu.ORG_BSNS_UNIT_KEY = rt.BSNS_UNIT_KEY

and rt.TRX_NBR = rsrli.TRX_NBR

and rt.DAY_KEY = rsrli.DAY_KEY

and rsrli.SKU_ITEM_KEY = sku.SKU_ITEM_KEY

and rt.DAY_KEY between 20090501 and 20090507

and obu.STATE in 'CA’

and sku.SKU_ITEM_DESC = 'Bananas'),

qall as

4 Table join to select all transactions buying Bananas in California in the first week of May

2 Table join to select all transactions in

39

( select rt.TRX_NBR

from DWR_ORG_BSNS_UNIT obu, DWB_RTL_TRX rt

where obu.ORG_BSNS_UNIT_KEY = rt.BSNS_UNIT_KEY

and rt.DAY_KEY between 20090501 and 20090507

and obu.STATE in 'CA')

select sku.SKU_ITEM_DESC,q.SCANS

from

( select SKU_ITEM_KEY,count(*) as SCANS,rank() over (order by count(*) desc) as POP

from qall,qbuy, DWB_RTL_SLS_RETRN_LINE_ITEM rsrli

where qall.TRX_NBR = qbuy.TRX_NBR(+)

and qbuy.TRX_NBR IS NULL

and rsrli.TRX_NBR = qall.TRX_NBR

and rsrli.DAY_KEY between 20090501 and 20090507

group by SKU_ITEM_KEY) q, DWR_SKU_ITEM sku

where q.SKU_ITEM_KEY = sku.SKU_ITEM_KEY

order by q.POP asc;

all transactions in California in the first week of May

Join the results sets in an outer join to find the exclusions, then rank,group and sort the results

1 Terabyte Loaded and Ready To Go In 20 Minutes

Operation Time

Create Tablespaces and Initial Load 0:39

Initial 1TB Load 9:55

Gather Statistics 3:36

Daily Incremental Load 1:44

Referential Integrity Check 0:51

40

Referential Integrity Check 0:51

Transform Data 1:09

Exchange and Incremental Statistics 0:22

Query from Hell 0:32

Total 18:48

1 Terabyte Loaded and Ready To Go In 20 Minutes

Gather Statistics, 3:36

Daily Incremental Load, 1:44

Referential Integrity Check, 0:51

41

Create Tablespaces and run DDL, 0:39

Initial 1TB Load, 9:55

Transform Data, 1:09

Exchange and Incremental Statistics,

0:22

Query from Hell, 0:32

Resource Management

42

Resource Management

and Query Queuing

Resource Management for DW Debate

• Why do most Oracle Data Warehouses adopt no

resource management techniques ?

– Storage is the ultimate bottleneck, queuing is done in HW

– Not many users can fully exploit parallel query

• Exadata changes everything because I/O is not the

43

• Exadata changes everything because I/O is not the

bottleneck anymore and the challenge moves to CPU

and memory allocation

• It is a new job and needs to be done.

• We need to educate our customers on the need and

implications of what resource management means e.g.

system stability and predictability

• Resource Management

– Does not make systems run faster

– Should be used to ensure the system does not run out of

control

– Makes some job run slow so that other jobs can run fast

DW Resource Management Principals

44

• In Summary - Resource Management

– Is NOT a turbo

– Is a gatekeeper

• Database Services

– Course grain resource management

• jobs to RAC nodes mapping

• Database Resource Manager

– Instance caging

DW Resource Management Tools and Options

45

– Instance caging

– Active session limit

– Max DOP setting ( This is the best place to set it )

– IO resource manager

– Query queuing with Auto-DOP

• Non-Oracle techniques

– Middleware or tool queuing

Concurrent Query Testing

Out of the Box Settings (secs)

User #7

User #8

User #9

User #10

User #11

User #12

46© 2009 Oracle Corporation – Proprietary and Confidential

0 20 40 60 80 100 120

User #1

User #2

User #3

User #4

User #5

User #6

Concurrent Query Testing

DBA Restricting DOP (secs)

User #7

User #8

User #9

User #10

User #11

User #12

47

0 20 40 60 80 100 120

User #1

User #2

User #3

User #4

User #5

User #6

Concurrent Query Testing

Query Queuing(secs)

User #7

User #8

User #9

User #10

User #11

User #12

48

0 20 40 60 80 100 120

User #1

User #2

User #3

User #4

User #5

User #6

Concurrent Query Testing

Out of the Box Fixed DoP With Queuing

User 1 30 24 10

User 2 33 27 11

User 3 34 27 11

User 4 39 29 11

User 5 40 29 15

49

User 6 41 29 19

User 7 43 29 21

User 8 47 28 16

User 9 49 30 25

User 10 106 28 27

User 11 108 26 25

User 12 112 27 26

Average 57 28 18

50