BRSPACE Online ReorganizationsAdvanced Reorganizations

Martin Frauendorfer, SAP Active Global Support

martin.frauendorfer@sap.com

April 2009

© SAP 2009 / Page 2

1. Overview2. Advanced Reorganizations

Agenda

© SAP 2009 / Page 3

Overview (1)

Reorganizations in general

n A reorganization in a database context is the recreation or cleanup of one or several segments.

n Several types of reorganization exist:n Index reorganization (e.g. via REBUILD or COALESCE), mainly used to

reduce index fragmentationn Table reorganization with tools like:

– BRSPACE (online, offline, export / import)– Oracle tools (DBMS_REDEFINITION, MOVE, EXP / IMP, EXPDP /

IMPDP, …)– SAP functionality (ICNV, SE14, …)

n Tablespace reorganization (reorganization of all tables in the tablespaceand recreation of the tablespace)

n In this presentation we concentrate on the BRSPACE online reorganization of tables (which also includes tablespacereorganizations).

© SAP 2009 / Page 4

Overview (2)

In which situations can a table reorganization be useful?

n Reduction of table sizen Cleanup of table fragmentationn Change of table storage parameters (e.g. INITRANS)n Move of table to a different tablespacen Reduction of chained and migrated rowsn Reduction of number of allocated extentsn Transition from LONG to LOB columns (Oracle >= 10g)n Reduction of hot spots on disk leveln Activation of Transparent Data Encryptionn Change of table structure

© SAP 2009 / Page 5

Overview (3)

In which situations can a tablespace reorganization be useful?

n Activation of LMTSn Activation of ASSMn Cleanup of freespace block corruptionn Reduction of database sizen Reduction of backup sizen Reduction of filesystem sizen Reduction of fragmentationn Reduction of number of datafilesn Increase of number of datafilesn Transition to new tablespace layoutn Other change to tablespace layoutn Optimization of filesystem layout

© SAP 2009 / Page 6

Overview (4)

How does a BRSPACE online reorganization work?n Determination of CREATE commands for target table and indexes based

on DBMS_METADATA.GET_DDLn Creation of target table with naming convention <source_table>#$n Export of source table CBO statistics based on

DBMS_STATS.EXPORT_TABLE_STATSn Copy of source table data to target table based on

DBMS_REDEFINITION.START_REDEF_TABLE. This package will implicitlycreate a materialized view log MLOG$_<source_table> to track all changes that are performed on the source table while the reorganizationis running.

n Creation of target indexes with naming convention <source_index>#$n Import of source table CBO statistics to target table based on

DBMS_STATS.IMPORT_TABLE_STATSn Finalizing of the online reorganization using

DBMS_REDEFINITION.FINISH_REDEF_TABLE. In this step all changes in the materialized log are applied to the target table and the names of source and target table are exchanged.

n The source table is dropped.n The „#$“ suffix of the target table indexes is removed.

© SAP 2009 / Page 7

Overview (5)

What are the advantages of a BRSPACE Online Reorganization?

n It uses the advantages of DBMS_REDEFINITION:n The reorganization can be performed in parallel to production operation.

This means the table is accessible and records can be inserted / updated / deleted.

n If anything goes wrong during the reorganization the source table still exists unchanged and no restore / recovery is necessary.

n Compared to an offline reorganization via export / import all data has to becopied only once and not twice

n It provides possibilities for client and server side parallelism in order to speed up the reorganization if sufficient systemresources exist and the reorganization should finish within a specific time window.

n It has already proven to be a reliable reorganization tool at manycustomers.

n It performs a lot of useful activities implicitly that would require a lot of manual efforts and impose the risk of errors ifDBMS_REDEFINITION is used directly.

© SAP 2009 / Page 8

Overview (6)

What are the restrictions of a BRSPACE Online Reorganization?

n It is only available as of Oracle 9i.n It cannot handle tables with LONG or LONG RAW columns. As of

10g it is possible to convert them during an online reorganizationon the fly to LOB columns (if SAP >= 6.40 is used).

n Temporarily twice the space is needed because both source and target table and indexes exist in parallel.

n If a table has no primary key constraint (or no unique index withNOT NULL columns) a ROWID based online reorganization isnecessary that requires overhead (ROWID index, ROWID column).

n The setup of the materialized view is an overhead that cannegatively impact the performance of the reorganization of manyvery small tables.

n Structure changes on the source table and its indexes are notpossible during the reorganization.

© SAP 2009 / Page 9

Overview (7)

How is a normal BRSPACE Online Reorganization started?

n Table reorganization:brspace -f tbreorg -t <table_name>

n Tablespace reorganization:brspace -f tbreorg -s <source_tsp> -t "*" -n <target_tsp>

n Additional useful options exist, e.g.:n -p <degree>à BRSPACE parallelism degree

n -e <degree>à PX parallelism degree

n -l <category>à Initial extent size category

n -i <target_ind_tsp>à Definition of target index tablespace (if different from target tabletablespace)

n For more detailed information see note 646681 and the BRSPACE online documentation.

© SAP 2009 / Page 10

Advanced Reorganization Scenarios Overview

Example for advanced reorganization scenarios:

n Optimization of data clusteringà Changing the sort order of the table recordsà Transition to an Index Organized Table (IOT)

n Partitioningà Activation of partitioning for a formerly unpartitioned tableà Repartitioning of an already partitioned table

n Table structure changesà Additional columnsà Modified columns (e.g. datatype and default value change)à Removed columns

© SAP 2009 / Page 11

Scenario 1: Optimization of Data Clustering (1)

Motivation

n In many cases good table data clustering in central tables is a keyfactor for good performance.

n This factor is often ignored and instead optimizations are onlyperformed on index, CBO or ABAP side.

n Typical candidates for a better clustering:n RESB: Sorting by MATNRn DFKKOP: Sorting by GPART / VKONT / VTREFn VDBEKI / VDBEPI: Sorting by RANL

n If e.g. 100 records for one DFKKOP business partner (GPART) exist and they are scattered across the whole table, in the worstcase 100 table blocks have to be read to retrieve the records.

n If the table is sorted by GPART and there are e.g. 10 records in each block, only about 10 blocks need to be read.

n This can easily improve the total access performance by factor 5.n With BRSPACE the sorting can be done online.

© SAP 2009 / Page 12

Scenario 1: Optimization of Data Clustering (1)

Warning

n Only do this if you understand clearly how the critical and typicaltable accesses look like!

n With the wrong data clustering you can cause criticalperformance problems!

n It is a good idea to monitor the access predicates and the sharedcursor cache data of the different table accesses for a longer time before deciding to implement a different data clustering.

© SAP 2009 / Page 13

Scenario 1: Optimization of Data Clustering (2)

Approach 1: Sorting table records by an index

n Directly supported by BRSPACE reorganization option "-sortind<index_name>"

n Advantages:n No table structure change

n Disadvantages:n Only a temporary solution because future DML operations can reduce the

clustering more and more. So it may be necessary to perform the sortingon a regular basis.

© SAP 2009 / Page 14

Scenario 1: Optimization of Data Clustering (3)

Approach 2: Transition to an Index Organized Table (IOT)

n No automatic support by BRSPACE, manual adaptations neededn Steps:

n Start the BRSPACE online reorganization with the additional option "-d first".

n BRSPACE will stop after it had created the CREATE commands for thetarget table and indexes in $SAPDATA_HOME/sapreorg/<reorg_subdir>/ddl.sql

n Now the CREATE TABLE command can be adapted manually:– Remove ") PCTFREE" – Add instead ", CONSTRAINT "<table_name>~<iotindex_suffix>"

PRIMARY KEY (<iotindex_columns>)) ORGANIZATION INDEX" – Remove "PCTUSED <percent>"– Remove CREATE command for secondary index if used as IOT

primary key index.n Afterwards the BRSPACE reorganization can be continued with "c"

© SAP 2009 / Page 15

Scenario 1: Optimization of Data Clustering (3)

Approach 2: Transition to an Index Organized Table (IOT)

n Advantages:n Permanently guaranteed sort ordern Table and IOT index combined in one segment

n Disadvantages:n Errors due to manual adaptations possiblen Typical IOT disadvantages:

– Larger secondary indexes– Overhead of secondary index accesses– Additional maintenance tasks like logical ROWID recalculation– Table fragmentation can increase because of its index design– Primary key constraint required– Only supported by ABAP DDIC if IOT is based on SAP primary index

à Mainly useful for IOTs on primary index on tables with a smallnumber of secondary indexes

© SAP 2009 / Page 16

Scenario 2: Partitioning (1)

Motivation

n Partitioning can be useful for several reasons, e.g.:n Optimization of data clustering: Similar like the previous approaches

also partitioning can be used to optimize data clusteringà Partitioning based on clustering column

n Archiving / deletion support: Partitioning can also be used to make surethat the table can be easily cleaned after archiving or deletion activitiesusing DROP PARTITION or MERGE PARTITIONà Partitioning based on archiving criteria

n Administration support: Large segments that are difficult to administer(e.g. consistency check, statistics creation, reorganization) can be split intosmaller partitions that can be administered individuallyà E.g. hash partitioning based on any column

© SAP 2009 / Page 17

Scenario 2: Partitioning (2)

Procedure

n No automatic support by BRSPACE, manual adaptations neededn Steps as described for the IOT approachn PARTITION clauses have to be added to the CREATE TABLE and

CREATE INDEX commands as requiredn Advantages:

n Permanently guaranteed clustering and reduced segment sizes

n Disadvantages:n Errors due to manual adaptations possiblen Often continuous partitioning maintenance necessary

© SAP 2009 / Page 18

Scenario 3: Structure Changes (1)

Motivation

n Structure changes like added or modified columns often cause trouble on large tables:n Long conversion times because each record has to be modified individuallyn Sometimes even longer conversion times because of ASSM problemsn Danger of significant amount of chained and migrated rows if blocks don't

contain sufficient freespace for additional datan SAP standard tools like SE11 / SE14 / ICNV don't provide a possibility for

parallelizationn A conversion based on a BRSPACE online reorganization

provides the following advantages:n Shorter runtime due to parallelism possiblen No danger of chained and migrated rows as target table is created from

scratchn Conversion can be done fully online

© SAP 2009 / Page 19

Scenario 3: Structure Changes (2)

Proceduren No automatic support by BRSPACE, manual adaptations neededn Utilization of the undocumented BRSPACE column mapping

featuren Attention: Errors in the manual procedure can result in a wrong

table structure and a restore may be required!n Functionality available as of BRSPACE 7.00 (40) and 7.10 (20) n BRSPACE profile with parameter _reorg_col_map for column

mapping information has to be createdn BRSPACE has to be started with this profile ("-p <profile>"), the "-

d first" option and the "-UCM" switch in order to activate thecolumn mapping feature.

n When BRSPACE stops, the CREATE TABLE command has to beadapted as required (additional columns, removed columns, changed columns).

n Before doing the final switch BRSPACE will stop a second time. This allows you to control the time of the switch (e.g. duringupgrade downtime).

© SAP 2009 / Page 20

Scenario 3: Structure Changes (2)

Proceduren No automatic support by BRSPACE, manual adaptations neededn Utilization of the undocumented BRSPACE column mapping

featuren Attention: Errors in the manual procedure can result in a wrong

table structure and a restore may be required!n Functionality available as of BRSPACE 7.00 (40) and 7.10 (20) n BRSPACE profile with parameter _reorg_col_map for column

mapping information has to be createdn BRSPACE has to be started with this profile ("-p <profile>"), the "-

d first" option and the "-UCM" switch in order to activate thecolumn mapping feature.

n When BRSPACE stops, the CREATE TABLE command has to beadapted as required (additional columns, removed columns, changed columns).

n Before doing the final switch BRSPACE will stop a second time. This allows you to control the time of the switch (e.g. duringupgrade downtime).

© SAP 2009 / Page 21

Scenario 3: Structure Changes (3)

Case Studyn During a SAP upgrade the following changes are necessary for

table DBERCHZ4:n Add column GGVERTRAG (VARCHAR2(18), DEFAULT ' ') n Add column OLD_BELNR (VARCHAR2(12), DEFAULT ' ')n Increase length of column LAUFNR from 2 to 5 characters, fill up with

'0' on left siden The incremental conversion (ICNV) of DBERCHZ4 took days and

weeks because the table contained 4 billion records.n If it is possible to use BRSPACE we could do the conversion also

online and we could use advanced possibilities likeparallelization.

n Therefore it was decided to use BRSPACE for the conversion.

© SAP 2009 / Page 22

Scenario 3: Structure Changes (4)

Case Study (2) - Proceduren Implementation of BRSPACE 7.00 (40)n Definition of column mapping parameter in BRSPACE profile:

_reorg_col_map = "MANDT MANDT,BELNR BELNR,…CONCAT(''000'', LAUFNR) LAUFNR,…"

n Start of BRSPACE:brspace -p <profile> -f tbreorg -t DBERCHZ4 -e 12 -d first -n <target_tablespace> -l 2 -UCM

n -e 12: Parallel execution with degree 12n -d first: Stop after DDL commands are generatedn -l 2: Define initial extent with 64Kn -UCM: Use column mapping parameter (_reorg_col_map)

© SAP 2009 / Page 23

Scenario 3: Structure Changes (5)

Case Study (3) - Proceduren Wait until BRSPACE stops and modify the generated DDL

commands in ddl.sql:n Add: "GGVERTRAG" VARCHAR2(18) DEFAULT ' ' NOT NULL ENABLE,n Add: "OLD_BELNR" VARCHAR2(12) DEFAULT ' ' NOT NULL ENABLE,n Change LAUFNR:

– Old: "LAUFNR" VARCHAR2(2) DEFAULT '00' NOT NULL ENABLE,– New: "LAUFNR" VARCHAR2(5) DEFAULT '00000' NOT NULL ENABLE,

n Continue BRSPACE runn With the "-UCM" option BRSPACE stops before the final switch

happensn During the upgrade downtime BRSPACE was continued and the

structure switch was executed.n The total conversion runtime was 11:05 hours.

© SAP 2009 / Page 24

Thank you!

© SAP 2009 / Page 25

Copyright 2008 SAP AGAll rights reserved

No part of this publication may be reproduced or transmitted in any form or for any purpose without the express permission of SAP AG. The information contained herein may be changed without prior notice.Some software products marketed by SAP AG and its distributors contain proprietary software components of other software vendors.SAP, R/3, mySAP, mySAP.com, xApps, xApp, SAP NetWeaver, Duet, Business ByDesign, ByDesign, PartnerEdge and other SAP products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of SAP AG in Germany and in several other countries all over the world. All other product and service names mentioned and associated logos displayed are the trademarks of their respective companies. Data contained in this document serves informational purposes only. National product specifications may vary.

The information in this document is proprietary to SAP. This document is a preliminary version and not subject to your license agreement or any other agreement with SAP. This document contains only intended strategies, developments, and functionalities of the SAP® product and is not intended to be binding upon SAP to any particular course of business, product strategy, and/or development. SAP assumes no responsibility for errors or omissions in this document. SAP does not warrant the accuracy or completeness of the information, text, graphics, links, or other items contained within this material. This document is provided without a warranty of any kind, either express or implied, including but not limited to the implied warranties of merchantability, fitness for a particular purpose, or non-infringement.SAP shall have no liability for damages of any kind including without limitation direct, special, indirect, or consequential damages that may result from the use of these materials. This limitation shall not apply in cases of intent or gross negligence.The statutory liability for personal injury and defective products is not affected. SAP has no control over the information that you may access through the use of hot links contained in these materials and does not endorse your use of third-party Web pages nor provide any warranty whatsoever relating to third-party Web pages

Weitergabe und Vervielfältigung dieser Publikation oder von Teilen daraus sind, zu welchem Zweck und in welcher Form auch immer, ohne die ausdrückliche schriftliche Genehmigung durch SAP AG nicht gestattet. In dieser Publikation enthaltene Informationen können ohne vorherige Ankündigung geändert werden.Einige von der SAP AG und deren Vertriebspartnern vertriebene Softwareprodukte können Softwarekomponenten umfassen, die Eigentum anderer Softwarehersteller sind.SAP, R/3, mySAP, mySAP.com, xApps, xApp, SAP NetWeaver, Duet, Business ByDesign, ByDesign, PartnerEdge und andere in diesem Dokument erwähnte SAP-Produkte und Services sowie die dazugehörigen Logos sind Marken oder eingetragene Marken der SAP AG in Deutschland und in mehreren anderen Ländern weltweit. Alle anderen in diesem Dokument erwähnten Namen von Produkten und Services sowie die damit verbundenen Firmenlogos sind Marken der jeweiligen Unternehmen. Die Angaben im Text sind unverbindlich und dienen lediglich zu Informationszwecken. Produkte können länderspezifische Unterschiede aufweisen.

Die in diesem Dokument enthaltenen Informationen sind Eigentum von SAP. Dieses Dokument ist eine Vorabversion und unterliegt nicht Ihrer Lizenzvereinbarung oder einer anderen Vereinbarung mit SAP. Dieses Dokument enthält nur vorgesehene Strategien, Entwicklungen und Funktionen des SAP®-Produkts und ist für SAP nicht bindend, einen bestimmten Geschäftsweg, eine Produktstrategie bzw. -entwicklung einzuschlagen. SAP übernimmt keine Verantwortung für Fehler oder Auslassungen in diesen Materialien. SAP garantiert nicht die Richtigkeit oder Vollständigkeit der Informationen, Texte, Grafiken, Links oder anderer in diesen Materialien enthaltenen Elemente. Diese Publikation wird ohne jegliche Gewähr, weder ausdrücklich noch stillschweigend, bereitgestellt. Dies gilt u. a., aber nicht ausschließlich, hinsichtlich der Gewährleistung der Marktgängigkeit und der Eignung für einen bestimmten Zweck sowie für die Gewährleistung der Nichtverletzung geltenden Rechts.SAP übernimmt keine Haftung für Schäden jeglicher Art, einschließlich und ohne Einschränkung für direkte, spezielle, indirekte oder Folgeschäden im Zusammenhang mit der Verwendung dieser Unterlagen. Diese Einschränkung gilt nicht bei Vorsatz oder grober Fahrlässigkeit.Die gesetzliche Haftung bei Personenschäden oder die Produkthaftung bleibt unberührt. Die Informationen, auf die Sie möglicherweise über die in diesem Material enthaltenen Hotlinkszugreifen, unterliegen nicht dem Einfluss von SAP, und SAP unterstützt nicht die Nutzung von Internetseiten Dritter durch Sie und gibt keinerlei Gewährleistungen oder Zusagen über Internetseiten Dritter ab.Alle Rechte vorbehalten.