BI OptimizationLoads ABAP BIAnalyst

Page 1

Optimizationof SAP-BW Loads

JJöörg Brg Bööke & Raymond Busherke & Raymond Busher

(Performance & Architecture)

Page 2

In This Session …o You will learn:

o Techniques to improve your throughput in:o Extraction Processeso Transformationso Loading

o You will take away:o How to check for potential performance issueso What is coming in BW 7.3 to help resolve these

issueso Hints to improve your ETL Processing

o What I don't cover is: BWA and HANA

BW Workshop © BIAnalyst GmbH & Co. KG

Page 3

What We’ll Cover …o Better ABAP techniqueso SOMo Extraction in OLTP-Systemo User-Exitso Indexeso DELTA vs. Fullo Data Stagingo Efficient Transformationso Choosing the right settingso Building better process chainso Finding Problems and Bottleneckso Wrap up


Page 4

User Exits: Improving Stability

o Don't use INCLUDESo CASE …WHEN… INCLUDE ZBWI_....

Programming technique are often in place, especially in User-Exit for variables and extractors

o In large development teams the danger of overshooters is high

o Transported errors / non-compiling exits can for example effect all extractors crash all queries

o Use generated / dynamic calls so that bad programming techniques or overshooters effect only the implementation transported and not the complete systemo Store function, form or method names in customizing tables,

enables re-usable codeso Generate names of functions, methods or forms that include

the relevant variable or data source. Limits options for reusable code


Page 5

Better ABAP Techniques (BAdI – Business AddIns)

o Extraction enhancemento User Exit: RSAP0001o BAdI: RSU5_SAPI_BADI

o Variables Exito User Exit: RSR00001o BAdI:

o Virtual characteristics and key figureso User Exit: RSR00002o BAdI: RSR_OLAP_BADI

o Report to report interface field mappingo User Exit: RSR00004o BAdI: RS_BBS_BADI


Page 6

Better ABAP Techniques (SQL)


Source: SAP

Page 7

Better ABAP Techniques (SQL)SELECT <result>INTO TABLE <l_t_table>FROM TABLE <db-table>WHERE <condition>

<result> specifies only required fields<condition> specifies which rows

Bad coding:SELECT * INTO CORRESPONDING FIELDS OF TABLE <l_t_table> FROM ….

DB-Tables with many fields select much data only to return a few fields in the internal table.


Page 8


o Use ABAP SORT BY rather than SQL ORDER BY clauseso The database is a central non-scalable

resource so that sorting in ABAP makes better use of resources

o READ TABLE … BINARY SEARCH does not recognize that the data comes sorted from the database and may set SY-SUBRC unexpectedly


Page 9



Page 10


o Use SORT and DELETE ADJACENT DUPLICATES to reduce the number of background SQL statements

o Always check that the entries table is not empty, otherwise the complete table will be selected


Page 11

Better ABAP Techniques (SQL)o Use Cursors when dealing with large volumes of data

o OPEN CURSORo FOR defines the selecto WITH HOLD prevents automatic closure of cursor

o FETCH NEXT CURSORo If the target is not a table single rows are fetchedo PACKAGE SIZE can be used to manage resources better

o CLOSE CURSORo Always close cursor to free resources

o NOTESo The number of simultaneous database cursors is limitedo CURSORS will not be held across multiple RFC callso CLEAR does not re-initialize a cursor completelyo CURSORS are sometimes lost in the DEBUGGER


Page 12

Better ABAP Techniques (DB)o Using Primary and Secondary indiceso User members of the primary key when

possible in WHERE clauseso When not possible secondary indices can

be defined in ABAP Workbencho The Database-Optimizer decides when

and which indices are used

o Use Database Monitor to catch bad SQL-Statements and check execution plans.


Page 13

Better ABAP Techniques (Internal Tables)

o Choose the right Internal Table type

STANDARDoUse for indexed accessoData with multiple sort criterionoFOR ALL ENTRIES clauses SORTEDoBest for UNIQUE keys and a single SORT criterionoLOOP AT … WHERE is optimized (key used from left to right)HASHEDoOnly full unique key is used to find recordsoUnique Key must be available


Page 14

Better ABAP Techniques (Internal Tables)

o STANDARD tableso Use for INDEX operator wherever possibleMODIFY l_t_table FROM wa_rec INDEX l_tabix.

DELETE l_t_table INDEX l_tabix.

o SORT table and read with BINARY SEARCH

SORT l_t_table BY field1 field2

READ TABLE l_t_table INTO wa_rec

WITH KEY field1 = 'VALUE'

BINARY SEARCH.


Page 15

Even better ABAP Techniques (Field-Symbols)FIELD-SYMBOLS:

<lt_any_table> TYPE STANDARD TABLE,<ls_any_line> TYPE ANY,<lv_any_field> TYPE ANY,<source_fields> TYPE _t_s_sg_1.

oA Field-Symbol is a pointer to the data it containsoData is not copied to field-symbols, updates are direct to the memory pointed tooField-Symbols due through indirect referencing can be used for processing dynamic structures


Page 16

Even better ABAP Techniques (Field-Symbols)

o When to use Field-Symbolso For larger internal tableso For internal table with wide line typeso To avoid copy costs via work areaso For multidimensional internal tableso If single lines are expected to be updates

o Where do we have the above in BWo User-Exits for extractorso START-Routineso END-Routines


Page 17


o Without using Field-SymbolsDATA: l_s_src_fields TYPE _t_s_sc_1.

LOOP AT source_package INTO l_s_src_fields.

l_tabix = sy-tabix.

…

l_s_src_fields-field1 = l_new_value1.

l_s_src_fields-field2 = l_new_value2.

…

MODIFY source_package FROM l_s_src_fields INDEX l_tabix.

ENDLOOP.

o When using Field-SymbolsFIELD-SYMBOLS: <SOURCE_FIELDS>.

LOOP AT source_package ASSIGNING <SOURCE_FIELDS>.

…

<SOURCE_FIELDS>-field1 = l_new_value1.

<SOURCE_FIELDS>-field2 = l_new_value2.

…

ENDLOOP.


Page 18


o When to use Field-Symbolso For larger internal tableso For internal table with wide line typeso To avoid copy costs via work areaso For multidimensional internal tableso If single lines are expected to be updates

o Where do we have the above in BWo User-Exits for extractorso START-Routineso END-Routines


Page 19

Even better ABAP Techniques (Move To ABAP-OO)

o Move from FORMS, INCLUDES and FUNCTIONS to OO Class-Libraries, Methods because:


ABAP-OO Advantages are:

Page 20

If you have done all this, and…


… it is still not fast enough

Page 21

SOM – Shared Object Memory

o The Shared Memory Objects concept is available from release 6.40 onwards. It consists of storing data in SAP memory (much like the EXPORT/IMPORT statements before ECC6), for rapid data retrieval without the need for physically reading persistent DB tables. Unlike the “old” EXPORT/IMPORT technique, however, the Shared Memory Object concept also allows for business logic to be stored (for instance, sorting or calculations based on the stored data).

o In addition, it is, as the name implies, shared between user sessions, so that the data can be freely used and updated by any number of users or sessions. This makes shared memory objects more flexible than the session memory technique.


Page 22

SOM – Shared Object MemoryKey Points to consideroShared memory-enabled OO-classes (root classes) are defined in SE24, using the “shared memory-enabled” addition oEach shared memory area relates to one such specific global rootclass oShared memory areas can have several area instances oEach area instance can have several versions (if versioning is allowed) oAn area class is generated by defining an area in SHMA – this is used as the area handle (attach methods) oThe root class is used for manipulating the data of the shared object oShared memory objects can be accessed by any number of users/sessions oWrite/update locks are exclusive oAll locks are for complete area instance and version (no partiallocks possible on specific attributes of the object)oMemory is limited => consider sizes, efficiency and cleaning up


Page 23

SOM – How it improves Performance

Read large volumes of data from database once and subsequent access from memory.e.g. oMaster Data lookupsoDSO lookups


Reading MARAV with classical ABAP Methodology

Reading MARAV with SOM (Initial loading is not reflected)

Page 24

SOM – How Too Transaction SHMA used to defined SOM-

Objects.

o AREA identifies your area in Shared Memory and is generated as Class Library

o Root Class defined the OO-Class to access, load and manage the your SOM. Must be "Shared Memory Enabled"

o "With Versioning" enables new version to be created in background, if current version is being accesed.Subsequent accesses go to the new versionOlder versions are automatically removed.


Helic

opte

r Vie

w

Page 25

SOM – Monitoring

o Transaction SHMM is used to monitor the SOM-Objects.o SHMM shows all created instances currently in shared

memoryo Double Clicking drills through to the objects in the

instance.


Helic

opte

r Vie

w

Page 26

Putting data to SOM* Define the area and memory object references

data: lcl_som_area type ref to zcl_my_area.

data: lcl_som_root type ref to zcl_my_root.

* Call the area method attach_for_read to get an area handle

try.

lcl_som_area = zcl_tst_materials=>attach_for_read( ).

catch cx_shm_no_active_version

cx_shm_read_lock_active

cx_shm_change_lock_active

cx_shm_exclusive_lock_active

cx_shm_inconsistent

into lv_exception.

endtry.

If lcl_som_area IS NOT INITIAL.

* If successful, we invoke the GET method of the root class:

* GET THE DATA USING APPLICATION SPECIFIC GET-Methods

* Finally, detach the area after use to release read locks

lcl_som_area->detach( ).

endif.


Helic

opte

r Vie

w

Page 27

Getting data from SOMDATA: myShmHandle TYPE REF TO zcl_my_area. DATA: myRoot TYPE REF TO zcl_my_root.

myShmHandle = zcl_my_area=>attach_for_write( ).CREATE OBJECT myRoot AREA HANDLE myShmHandle. myRoot->name = `My first area instance`.

*Get data from database and copy to table(s) in SOM* APPEND … TO myRoot->itab_1. * APPEND … TO myRoot->itab_2.

* tell area handle who is going to work as root- don’t miss this, Root is always stored as attribute of area class myShmHandle->set_root( myRoot ).

IF …myShmHandle->detach_commit( ).

ELSE. myShmHandle->detach_rollback( ).

ENDIF.


Alternatively the generated method IF_SHM_BUILD_INSTANCE~BUILD in the class can be defined to fill the tables on first call

Helic

opte

r Vie

w

Page 28

Where to use the SOM

o User-Exits in OLTPo Get internal tables from memoryo Minimum Release 6.40

o START-Routines / Process Chainso Use Process Chain steps to fill and clean-up SOM-Objectso In START-Routines read from SOM rather than going back to

the DB.

o Unmanaged Varianto The first to adress the SOM fills it, this may result in more than

one instance filling the SOM when started parallel (e.g. DTP with several processes)

o Manages Varianto Extra ABAP step at the start to fill the SOM with current datao Extra ABAP step at the end of the Process Chain to release objects

and memory.OF course Hold time parameters etc. make this unnecessary but isfair to other applications


Page 29

SOM – Usage (Unmanaged Variant)


Source: SAP

Page 30

SOM – Usage (Managed Variant)


Source: SAP

Page 31

If you have done all this, and…


… it is still not fast enough, what then… HANA?

Page 32

AGENDAo Better ABAP techniqueso SOMo Extraction in OLTP-Systemo User-Exitso Indexeso DELTA vs. Fullo Data Stagingo Efficient Transformationso Choosing the right settingso Building better process chainso Finding Problems and Bottleneckso Wrap up


Page 33

How to improve the extraction process

o DB-Time to select the datao If you are using SAP-Business Content, check OSS for

performance relevant notes. Often there are notes recommending additional secondary indices

o Check your User Exito Check if it is your User Exit that is killing performance, here

you have more chances to get a better performance.o Check for select single, select … endselecto Use Static tables to reduce the number of database accesseso Move to FIELD-SYMBOLS if you have lots of table modifications

o Finding problemso Use DB-Monitoring options to catch the SQL statement and

analyze to check for required secondary indiceso Is a full upload more efficient and manage deltas in BW (DSO)


Page 34

AGENDAo Better ABAP techniqueso SOMo Extraction in OLTP-Systemo User Exitso Indexeso DELTA vs. Fullo Data Stagingo Efficient Transformationso Choosing the right settingso Building better process chainso Finding Problems and Bottleneckso Wrap up


Page 35

Migration Option

o Option is available, but do you want to use it?o Transformation Rules can be passed only the fields

they requireo START-Routine is replaced by a FORM, which does

not improve readability and not conform to the OO approach

o Are your update rules so complex that they can't be re-created?

o If the update rules contain routines, all fields are transferred, but this is not necessary in TRFNs.

o Migrate Option does not pick up what you do in the Transfer Rules. These have to be leveraged to the Transformation


Page 36

Building Better Transformations

o START Routineso Use START-Routines to process data

efficiently before single record processing begins.o Build LOOKUP-Tableso DELETE unrequired records before processing

begins (Better use DTP filters that don't pick up unrequired data)

o END Routineso Most efficient place to copy records.o Replaces functionality provided by

"RETURN TABLES".


Page 37

DTP checks

o Semantic Groups are good for ensuring data consistency, but add sorting overhead

o DELTA and/or FILTER criterion may not use indices, especially when non-key fields are used

o Sometimes secondary indices are necessary in PSA tables


Page 38

AGENDAo Better ABAP techniqueso SOMo Extraction in OLTP-Systemo User Exitso Indexeso DELTA vs. Fullo DataStagingo Efficient Transformationso Choosing the right settingso Building better process chainso Finding Problems and Bottleneckso Wrap up


Page 39

Optimizing the Data ModelsDimensions

o Use as many Dimensions as possibleo Use Line-Item Dimension when relevant (Document

Number). Consider other master data such as Customer, Material Plant especially if Navigational Attributes are used.

o Do not set High Cardinality flago Define dimensions in Basis Providers for high

performing queries. Check distribution after initial loads using SAP_INFOCUBE_DESIGNS.

o Define dimensions in MultiProvider for easy query definition.

o Add all relevant Time Fields (0CAL*) that can be determined from the most granular to provide the most flexible reporting options without reloading


Define Guidelines for your development team

Page 40

Choosing the Right Settings -Implement Number Range Buffers

o The number range tables (NRIV) are called for every new distinctrecord that is loaded to the BW system as either master data or dimension in an InfoCube.

o The NRIV-Table is accessed with a select for update statement which can be quite slow.

o Buffering should be done as follows:o Determine the large number ranges (Document Number, dimensions with

document numbers or many distinct values)o Use the SNRO transaction to adjust

buffers.

o Check relevant OSS-Noteso The Objects are generated on

transport and therefore not thesame in E, K and P systems.


Page 41

AGENDAo Better ABAP techniqueso SOMo Extraction in OLTP-Systemo User-Exitso Indexeso DELTA vs. Fullo Data Stagingo Efficient Transformationso Choosing the right settingso Building better process chainso Finding Problems and Bottleneckso Wrap up


Page 42

Finding the Bottleneckso 7.0 use ST13 go to the DTP, Load processes

time details available for all parts of the data staging process

o 7.3 allows a direct viewo DB-Logs checko Catch the SQL statemento Understand how to analyze SQLo Localize the problem with objective

criterion, not just a gut feelingo Keep an eye out for side-effects (all

process occupied, memory consumption, etc)


Page 43

Use ABAP Programso Use ABAP programs to keep an eye on

your system, e.g.o Overview of used and - more importantly -

unused aggregateso DB partitioning that is expired. Re-

partitioning is easier and quick if MAX-Partition is not reached

o List largest tables. If they show PSAs, then cleanup has been forgotten.

o Why not E-Mail results to BW Administrators? (E-Mail is on board)


Page 44

Things to Take Home (1)


Page 45

Things to Take Home (2)1. System Monitoring and performance

checking is an on-going process2. Fixing one bottle-neck may create side-

effects that impact other processes.


Page 46

Your Turn!

How to contact me:Raymond Busher

[email protected]

BIAnalyst GmbH & Co KGAltdorferstrasse 9132791 LageGERMANY


Page 47

DisclaimerSAP, R/3, mySAP, mySAP.com, SAP NetWeaver®, Duet®, PartnerEdge, and other SAP products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of SAP AG in Germany and in several other countries all over the world. All other product and service names mentioned are the trademarks of their respective companies.

BIAnalyst mentioned herein as well as their respective logos are trademarks or registered trademarks of BIAnalyst GmbH & Co.KG


Documents

BI OptimizationLoads ABAP BIAnalyst