Presentation - Three R’s of Data Warehouse Fault Tolerance

8/12/2019 Presentation - Three Rs of Data Warehouse Fault Tolerance

1/60

T : (888) 631 1410 or+44 (0) 8446 697 995 E : [email protected] W: www.rittmanmead.com

Stewart Bryson, Managing Director, Rittman Mead AmericaOracle Open World 2010, San Francisco, September 2010

Three Rs of Data Warehouse Fault Tolerance

Thursday, September 23, 2010


2/60


Who am I?

http://www.rittmanmead.com/http://www.rittmanmead.com/


3/60


Stewart Bryson, Managing Director, Rittman MeadAmerica

Oracle BI/DW Architect and Delivery Specialist

Oracle Database and BI Stack since 1996

Regular Community Speaker

Writer for Rittman Mead Blog:http://www.rittmanmead.com/blog

Developer of Transcend Framework

Email : [email protected]

Twitter : @stewartbryson

http://www.rittmanmead.com/bloghttp://www.rittmanmead.com/bloghttp://www.rittmanmead.com/http://www.rittmanmead.com/


4/60


Who is Rittman Mead?



5/60


Oracle BI&DW Experts

Consulting, Training and Support

Global Company, Offices in the US, UK and India

Clients in the UK, USA, Europe, Middle-East

Voted UKOUG BI Partner of the Year 2008

13 Sessions at Oracle Open World 2010

Twitter : @stewartbryson



6/60


Rittman Mead Blog



7/60


Longest-running Oracle (BI) Blog

OBIEE, OWB, BI, data warehousing, Hyperion, etc.Step-by-step demonstrations

Test cases

Product evaluations

Authors include

! Mark Rittman (ACE Director)

! Venkat Janakiraman (ACE)

! Stewart Bryson

! Borkur Steingrimsson

! Peter Scott



8/60


Rittman Mead Oracle BI Training Days 2010

Running in London, Atlanta and Bangalore, Fall 2010

Three days of intense OBIEE 11gR1 hands-on training

Led by the Rittman Mead 11g Beta Testing team

Limited places still available: http://www.rittmanmead.com/trainingdays2010



9/60


What will this session answer?



10/60


!What do I mean by fault tolerance?

!How is fault tolerance different for BI/DW systems?

!Are there standard features of the Oracle Databasethat can help with fault tolerance?

!What do I mean by Resuming, Restarting andRestoring?



11/60


How are BI/DW Systems Different?



12/60


Millions or billions of rows

Constraints are often disabled

!No referential integrity protecting errant ETL runs

! Long query processing to find issues

DDL operations included in ETL code

Large memory sizes, parallelism



13/60


What is Fault Tolerance for BI/DW?



14/60


Ability to recover from different kinds of errors

!Hardware and software

!

General System issues: space, network latency!Human mistakes

Elimination of aftermath scenarios

!Data correction scenarios

! Long rerun processes



15/60


What is available to us with the database?



16/60


The first thing we should do is exploit the features inthe database we already own

Threat a BI/DW environment as a production system

Simple errors should never cause downtime

Complex errors should cause minimal downtime

Understand that ETL processes will sometimes breakand plan for that



17/60


Resuming



18/60


Error is detected and processing is suspended

Once the error is corrected, processing continuesautomatically

Oracle Database offers Resumable Space Allocation

!Available since 9i

! Functionality for all space-related errors: targetdatafiles, tempfiles, UNDO space

Supported for SELECT, DML and DDL

Available in Oracle Utilities

!Datapump (automatic)

!SQL-Loader

!Export/Import



19/60


Enabling Resumable Space Allocation



20/60



Enabled at the system Level

!RESUMABLE_TIMEOUT parameter



21/60





Enabled at the session Level

!ENABLE RESUMABLE

!

TIMEOUT!NAME



22/60





Enabled at the session Level

!ENABLE RESUMABLE

!

TIMEOUT!NAME

Enabled with command-line parametersfor Oracle Utilities

!RESUMABLE

!RESUMABLE_NAME

!RESUMABLE_TIMEOUT



23/60


Monitoring Resumable Sessions



24/60





25/60





26/60





27/60


AFTER SUSPEND Trigger



28/60


AFTER SUSPEND Trigger

System-wide trigger fires whenoperations suspended

Can use the DBMS_RESUMABLEpackage

!SPACE_ERROR_INFO function

!SET_TIMEOUT

Seems unnecessary

!BI/DW environments should beproduction-ready

!Should already have monitoring andalerts

!AUTOEXTEND handles adding space



29/60


Resumable Space AllocationDemonstration



30/60


Restarting



31/60


Complex errors occur during ETL processing

!Hardware and software issues

!Unforeseen data issues

!Human error

!Resumable operations are uselessPlanning means eliminating aftermath scenarios

Investigate the current landscape for errors

Easy errors can sometimes be corrected with a simpleDML statement

What about the rest?



32/60


Planning For Complex Errors



33/60



Need a framework that makes it easy to identify majormilestones in the ETL process

Need to catch unexpected errorsNot the same as code-controlled restartability



34/60



Need a framework that makes it easy to identify majormilestones in the ETL process

Need to catch unexpected errors

Not the same as code-controlled restartability



35/60


Code-Controlled Restartability



36/60


Code-Controlled Restartability

Coding for exceptions is a best practice and should alwaysbe done, but we cant catch everything



37/60


Oracle Flashback Technology



38/60


Revert whole database, or just portions of it

Flashback Table or Flashback Database

Point-in-time is the System Change Number (SCN)

Can create restore points, which are named SCNs

Alternatively, log SCNs with messages in the log table



39/60


Completely an UNDO operation

Identify a particular SCN to flashback to

Can only flashback to SCNs still in the UNDO

This is not really new functionality

Doesnt require operations involvement

!

grant flashback on sales_fact to ETL;

Flashback Table



40/60


Flashback TableDemonstration



41/60


Flashback Database



42/60


Flashback Database Scenario: Aftermath

ODS and staging-area lookup components

!Deleting from history tables

!Correcting MDM scenarios

Star Schemas!SCD Type 2 or Hybrid Type 1/Type 2

!Deleting or updating fact tables

!Recalculating aggregates

Difficult to quantify exactly what went wrong

Need a complete do-over

Alternatives!Execution ID

!Audit dimension



43/60


Flashback Database

Not an UNDO operation like Flashback Table

Uses Flashback Logs

!Contains prior versions of changed blocks

!Physical not logical

Redo Logs (online or archived)

!Contain logically applied SQL statements

! Logical not physical

Uses a combination of Flashback and Redo Logs

! Find the data block from the Flashback Log just priorto the SCN

!Use the REDO Logs to roll forward to theappropriate SCN

Requires operations involvement



44/60


Preparing for Flashback Database

Set the DB_RECOVERY_FILE_DEST_SIZEparameter

!SQL> alter system setdb_recovery_file_dest_size=3G;

Set the DB_RECOVER_FILE_DEST parameter

!SQL> alter system set db_recovery_file_dest='/u01/app/oracle/flash_recovery_area';

Set the DB_FLASHBACK_RETENTION_TARGETparameter (in minutes)

!SQL> alter system set

db_flashback_retention_target=2880;

Turn on Flashback Database

!SQL> alter database flashback on;



45/60


Flashback DatabaseDemonstration



46/60


Should a BI/DW Database in in ARCHIVELOG Mode?

Explanations for why BI/DW databases are usually in NOARCHIVELOG Mode

!REDO generation is only important for transactional systems

!Generation of REDO causes performance overhead

!Putting a database in NOARCHIVELOG Mode eliminates REDO generation



47/60


Database in ARCHIVELOG Mode with a NOLOGGING Table

SQL> ALTER TABLE target.sales

2 nologging;

Table altered.

SQL> SET autotrace on statistics

SQL> INSERT INTO target.sales

2 SELECT * FROM sh.sales;

918843 rows created.

Statistics

-------------------------------------------------------

15401 recursive calls

33766 db block gets

13759 consistent gets

6534 physical reads

38250540 redo size

738 bytes sent via SQL*Net to client

942 bytes received via SQL*Net from client

4 SQL*Net roundtrips to/from client 123 sorts (memory)

0 sorts (disk)

918843 rows processed

SQL>


2 nologging;

Table altered.


SQL> INSERT /*+ APPEND */ INTO target.sales



Statistics

-------------------------------------------------------

1 recursive calls

4637 db block gets


1704 physical reads

8028 redo size




0 sorts (disk)


SQL>



48/60


Database in ARCHIVELOG Mode with a NOLOGGING Table


2 nologging;

Table altered.


SQL> INSERT INTO target.sales



Statistics

-------------------------------------------------------

15401 recursive calls

33766 db block gets


6534 physical reads

38250540 redo size




0 sorts (disk)


SQL>


2 nologging;

Table altered.


SQL> INSERT /*+ APPEND */ INTO target.sales



Statistics

-------------------------------------------------------

1 recursive calls

4637 db block gets


1704 physical reads

8028 redo size




0 sorts (disk)


SQL>



49/60


Database in NOARCHIVELOG Mode

SQL> set autotrace on stat

SQL> insert into target.sales

2 select * from sh.sales;


Statistics

-------------------------------------------------------

9 recursive calls

33251 db block gets 10533 consistent gets

879 physical reads

38282140 redo size



6 SQL*Net roundtrips to/from client

1 sorts (memory)

0 sorts (disk)


SQL>


SQL> insert /*+ APPEND */ into target.sales



Statistics

-------------------------------------------------------

0 recursive calls


62 physical reads

7984 redo size




1 sorts (memory)

0 sorts (disk)


SQL>



50/60


Database in NOARCHIVELOG Mode


SQL> insert into target.sales



Statistics

-------------------------------------------------------

9 recursive calls


879 physical reads

38282140 redo size




1 sorts (memory)

0 sorts (disk)


SQL>


SQL> insert /*+ APPEND */ into target.sales



Statistics

-------------------------------------------------------

0 recursive calls


62 physical reads

7984 redo size




1 sorts (memory)

0 sorts (disk)


SQL>



51/60


Real Meaning of NOARCHIVELOG



52/60


All REDO is not suppressed

! REDO is still needed for crash recovery

! Oracle wouldnt be able to open after a simple servercrash

NOARCHIVELOG Mode only means we are foregoing theability to do Media Recovery

Direct-path writes bypass the buffer cache

! Blocks inserted directly into the appropriate datafiles! INSERT /*+ APPEND */ statements

! SQL-Loader direct-path mode

These statements are not needed for crash recovery

! They already exist inside the datafile as they arecommitted

! No chance of losing them during server crashes Unless direct-path operations are occurring, then REDO is

being generated regardless of ARCHIVELOG mode



53/60


Restoring (Actually, Backing Up)



54/60



Put the database in ARCHIVELOG Mode

!Enables Online Backups

!Enables Flashback Database



55/60






Create a Block Change Tracking file

! Increases backup performance



56/60








Modify tables to be NOLOGGING



57/60









Perform INSERT /*+ APPEND */statements as part of the ETL



58/60









Perform INSERT /*+ APPEND */statements as part of the ETL

Perform an incremental backup uponcompletion of the ETL process

!

Incremental backups affect just thechanged blocks

!Can be run throughout the day

!Effective with mini-batch scenarios



59/60


Conclusions

All ETL mappings and flows, plus maintenance processes, should use ResumableSpace Allocation

Flashback provides data management restartability

Still need code-controlled restartability

Flashback Table doesnt require operations involvement, while Flashback Database

does

Keep your database in ARCHIVELOG Mode

Use ONLINE Backups for the BI/DW environment

Execute incremental backups at the completion of the ETL processing



60/60


Stewart Bryson, Managing Director, Rittman Mead AmericaOracle Open World 2010, San Francisco, September 2010

Three Rs of Data Warehouse Fault Tolerance

Documents

Presentation - Three R’s of Data Warehouse Fault Tolerance