Performance Optimization in BODS

SAP Business Objects Data Services delivers a powerful, enterprise-class data integration and data quality solution. It provides one development, run-time, and management environment that allows you to integrate, transform, improve, and deliver data anywhere, at any frequency.With SAP BusinessObjects Data Services, you can: Respond to information needs and accelerate time to market Deliver the most trusted and accurate information Maximize operational efficiency with a single platform Leverage a collaborative, complete, and open solution scalable for any project sizeFeatures and Functions:The comprehensive features and functions of SAP BusinessObjects Data Services can enable the following:Agility Leverage features that can accelerate time to market, encourage reuse and collaboration, increase developer productivity, and create unified skill sets.Trusted information Improve the reliability of your data with functions for data governance including transparency, change management, security, and controls global data support, and end-to-end metadata management.Operational excellence Gain a single data repository, platform, and environment so that costs associated with supporting multiple applications can be reduced; Leverage multi-platform support for 64bit, UNIX, Linux, and Windows.Enterprise class Benefit from highly scalable performance with support for parallelization and grid computing, enterprise connectivity, and support for service-oriented architecture (SOA).It is recommended that the BODS jobs should be tuned accordingly to be able to complete in less time and utilize less resource. This document deals with the different Performance Optimization techniques needed to fine tune the BODS jobs there by enabling the BODS jobs to complete in less time with less rates of failure.2 BODS Performance OptimizationBODS is an ETL (Extraction, Transformation and Load) tool that selects data from one data base, processes it in a job server by applying the transformation rules specified in the mapping and loads the data into the specified target database.There are 7 touch points that need to be considered for performance optimization. They are given below:1. Source Operating System.2. Source Data Base.3. Target Operating System.4. Target Database.5. Network.6. Job Server OS.7. BODS Repository database.The BODS Job Server and the BODS Repository Database can reside on the same server. If installed on different servers, Network throughput configured between Job Server and BODS Repository Database will play a key role. The performance tuning methods for each identified touch point are given below.2.1.1 Source Operating SystemThe Source Operating System should be tuned to quickly read data from disks. This can be done by the following methods:Set the read-ahead protocol to 64 KB to make the I/O operations fast.The size of read-ahead protocol is by default set to 4-8 KB.2.1.2 Source DatabaseThe source database should be tuned to make SELECTs as quickly as possible. This can be done by the following methods:Increase the size of Database I/O to match with the OS read-ahead protocol; otherwise it may cause bottle neck there by affecting the performance.Increase size of shared buffer to cache more data in the database server.Cache tables that are small enough to fit in the shared buffer by enabling cache at table level. Caching reduces the number of I/Os thereby improving the speed of access to the data.Turn off PARALLEL for small tables.Create indexes on appropriate columns by looking into the dataflow to identify the columns used in the join criteria.Create Bitmap indexes on the columns with low cardinality as the index does not take much space in the memory, there by resulting in faster SELECTs.2.1.3 Target Operating SystemThe Target Operating System should be tuned to quickly write data to disks. This can be done by the following methods:Turn on the asynchronous I/O to make the Input/output operations as fast as possible.2.1.4 Target DatabaseThe Target Database should be tuned to perform INSERTs and UPDATEs as quickly as possible. This can be done by the following methods:Turn off archive logging.Turn off redo logging for all tables.Tune rollback segments for better performance.Place redo log files and data files on a raw file if possible.Increase the size of the shared buffer.2.1.5 NetworkEven if the Source Database and Target Database are tuned if the network band width is small, there is possibility for the occurrence of bottle neck which hampers the performance.Adjust the size of the Network buffer in the database client so that each client request completely fills a small number of network packets, there by reducing the number of round trips across databases which in turn improves the performance.2.1.6 Job Server OSData Services jobs are multi-threaded applications. Typically a single data flow in a job initiates one al_engine process that in turn initiates four threads. For maximum performance benefits:Consider a design that runs one al_engine process per CPU at a time.Tune the Job Server OS so that the Data Services threads spread to all available CPUs.The above performance optimization methods need to be implemented during the environment and infrastructure preparation of BODS components.2.1.7 Data Services JobsThe following execution options can be used to improve the performance of BODS jobs:Monitor Sample Rate: If the job processes large amount of data set the Monitor Sample Rate to a higher value (maximum being 50,000, default is 1000) to reduce the number of I/O calls to the log file there by improving the performance.If virus scanner is configured on the BODS JobServer, exclude the Data Services log from the virus scan. Otherwise the virus scan scans the Data Services log repeatedly during the execution, which causes performance degradation.Collect Statistics for self-tuning: BODS has a self-tuning capability to determine the cache type by looking into the statistics of previous job executions. The Collect Statistics option needs to be selected during the first execution of the job. BODS collects the statistics for that job and stores in the metadata for that job. In the next execution select the Use Collected statistics option to allow BODS to decide the type of cache to be used to execute the job, there by improving the performance of the job.Set the data flow properties like Degree of Parallelism depending upon the number of CPUs available for processing and set the cache type to in-memory if the data volume is less.If source tables are from same schema and using same Data Store, identify the joins in the early phase so that the join can be pushed to the database. Create synonyms for the tables in other schemas to pushdown join conditions of tables belonging to different schemas in a database.Use data base links to connect different schemas in different databases to pushdown the joins of tables available in these schemas to database.Use data transfer (type = TABLE) to pushdown the complex logic in a dataflow to the database.Do not use advanced features like Run Distinct as Separate process etc available in Advanced tab in Query transform, as it starts multiple sub-processes which causes heavy traffic between the processes and there by lead to the termination of job.Do not use Data Transfer transform unless required. (Use table type if required as it is more reliable.). SAP suggests that Data Transform is not a reliable transform and hence recommends to not using it unless required.Turn off the Cache option for the tables with larger amounts of data. Cache is turned on by default for every source table. Make sure that there are indexes created on key columns for these tables on which cache is turned off.Do not use BODS functions like job_name(), instead initialize a variable in a script and use that variable for mapping in query transforms.Use Join where ever applicable in place of look up transform as the look up transform has to access the table for each and every record which increases the number of I/O operations to database.Use Query transforms to split the data in place of Case transforms as it is a costly transform in BODS.

SAP Business Objects Data Services Target Table Options:created by Ankit Sharma on Sep 18, 2013 11:37 AM, last modified by Ankit Sharma on Sep 23, 2013 12:48 PM Version 3

This will help you to understand how to make use of the different options available in the target table while loading data.Below is the descriptions of all the options with the sample data.Auto Correct Load: Auto correct loading ensures that the same row is not duplicated in a target tableInput Data Screen Short:

Target Option Window: Following steps need to be done

Uncheck the Delete sata from Table before LoadingBefore Updating in Update Control:

After Updating:

Now go to Source Data and add two Duplicate rowsBefore Updating:

After Updating:

After Creation of Jobs in Data Services Designer

Now Apply Auto Correct Load on Target Tables

Note: After apply Auto Correct Load Duplicate values are removed from Target TablesNow Execute the Job

Output:

Ignore Column Option: If a matching row exists, the row is updated depending on the values of Ignore columns with value, and Ignore columns with null:Ignore Columns Options are Worked with Auto Correct Load: When the Ignore columns with null option is set to Yes and the column data from the source is NULL, then the corresponding column in the target table is not updated. Otherwise, the corresponding target column is updated as NULL since the source column is NULL.Now Go To Target Table Properties:Before Update in the Target Table Properties:

After Update in the Target Table Properties:

Now Source Data Information:

Now create Jobs in Data Services Using Job-Work Flow->Data Flow-> Source & Target and make EMP_ID as a Primary key

View of Source table Before Loading:

View of Source table After Null Values Before Loading:

Now update in the Target Table before loading the data

Then Finally Execute the Job

Output after executing the job:

Ignore Column with Value Option: If a matching row exists, the row is updated depending on the values of Ignore columns with value, and Ignore columns with null: When the column data from the source matches the value in Ignore columns with value, the corresponding column in the target table is not updated. The value may be spaces. Otherwise, the corresponding column in the target is updated with the source data.Now Go To Target Table Properties:Before Update in the Target Table Properties:Before Update in the Target Table Properties:

After Update in the Target Table Properties:

Now Source Data Information:

Now create Jobs in Data Services Using Job-Work Flow->Data Flow-> Source & Target and make EMP_ID as a Primary key

View of Source table Before Loading:

Now update in the Target Table before loading the data

Then Finally Execute the Job

Output after executing the job:

Update Key Columns in Update Control: Now Go To Target Table Properties:Before Update in the Target Table Properties:

After Update in the Target Table Properties:

Now Source Data Information:Before Update:

After Update in Source Table:

Now create Jobs in Data Services Using Job-Work Flow->Data Flow-> Source & Target and make EMP_ID as a Primary key

View of Source table Before Loading:

Now update in the Target Table before loading the data

Then Finally Execute the Job

Output after executing the job:

Sending IDOC from BODS to SAPDocument PurposeThis document provides steps required to load data in SAP using one of the SAP interfaces. This document will be used at the time of proof of concept to demonstrate the ability of Business Objects Data Services (BODS) to invoke IDOC interface to load data in SAP.The main objectives are: Loading data in SAP using IDOC Demonstrate how BODS can be used to load data using IDOC Display required configurations step-by-step & invocation Demonstrate actual sample load using BODS jobOverview:This document provides systematic approach for the data load in three sections:1. Configuration Required at SAP Side:2. Configuration Required at BODS Side:3. Demonstration Sample Load using BODS Job:Each of above sections provide information as in how to configure the specific parameters required to get both SAP & BODS ready to communicate with each other. Screen shots provided for better understanding of the parameters & their configuration.Configuration Required On SAP (ECC 6.0) Side

Following is the required configuration on SAP side.1.Display IMG [ /nsale ]Connect to SAP System using SAP GUIUse Transaction Code (TCODE) /nsale to Display IMG

Screenshot 1: Display IMG screenExpand Logical Systems tree & click on the Clock icon in front of Define Logical System2. Defining Logical System

Screenshot 2: Defining Logical System screenClick on the New Entry Button For SAP Inbound IDOC One Logical system (FRMDICL100) is requiredCreate a Logical system as shown below

Screenshot 3: Overview of added entriesClick on the save button to save both the Logical System. Verify that the LS FRMDICL100 is present in the SAP system

Screenshot 4: Overview of Logical Systems3. Create Partner Profile [ /nwe20 ]Use TCODE /nwe20 to configure partner profile.

Screenshot 05: Partner Profile screenClick on Create button to create new partner profileCreate partner profile as shown below Note : 1. Provide partner no. FRMDICL100 i.e same as BODS Logical System.2. Agent 50010120 EDI Department is necessary for communication with non-SAP systems like BODS. Without this agent IDOC communication will not be successful.

Screenshot 06: Partner Profile deatil screenClick on the Save button to save the partner profileAfter saving; the Create Inbound parameter will be activated in Inbound paramtrs

Screenshot 07: Partner Profile create Inbound paramtrs screenClick on Create Inbound parameter Green Plus button in Inbound paramtrs section

Screenshot 08: Partner Profile Inbound paramtrs screenNote : In Inbound options tab, select appropriate process code relevant to message type.(In this case; for MATMAS message type, process code is MATM)

Screenshot 09 In "Post processing : permitted agent" tab provide details as shown above.Save "Partner profile: Inbound parameters" using Save buttonAfter saving verify Partner Profile & Inbound Parameter added.

Screenshot 10Configuration Required On BODS SideFollowing is the required configuration on BODS side1. Create SAP Datastores Create SAP Datastore by providing required SAP Application Server Information & creadentials

Screenshot 11: SAP Datastore in BODSRight click on IDOCs and select Import By Name

Screenshot 12: IDOC list under SAP datastore in BODS

Screenshot 13: Import IDOC screenProvide Type and Name as shown above and click on Import to import IDOC in datastore2. Generating input XML schemaTo create the input XML structure with the correct datatypes quickly drop the IDOC as message source onto the dataflow, go into this object and rightclick the schema. There, you can ask the Designer to generate the entire structure as XML schema, thus you get all columns, schemas including the datatypes.

Screenshot 14: MATMAS01 IDOC screen MATMAS01_OUT this XML file source is created using generated schema inFile Format tab.

Screenshot 15: XML Schema screen3. Build Batch JobBuild a Batch Job with a DataFlow insideDataFlow is as Follows

Screenshot 16: Batch Job screenDouble click on source XML File Source configure it as shown below.Note: For demonstration purpose we have used XML file that was output of the Real-Time Job designed in article "Receiving IDOC In BODS from SAP".

Important : Make sure that in XML file Sending Partner setting i.e. should be same as BODS logical system name. (FRMIDCL100) as shown below.

Screenshot 18: IDOC out XML file strucutre screenUnnest by right clicking on MATMAS01_OUT schema in "Query" as shown below

Screenshot 19: IDOC structre screenDrag the IDOC MATMAS01 from SAP DataStore as shown

Screenshot 20: IDOC in SAP DatastoreNote: Make sure to select option "Make IDOC Message Target"Configure IDOC Message Target

Screenshot 21: BODS side IDOC configuration

Receiving IDOC in Business Objects Data Services (BODS) from SAPcreated by ROSHAN CHHAJED on Jul 16, 2012 10:51 AM, last modified by ROSHAN CHHAJED on Jul 16, 2012 5:43 PM Version 1

Document PurposeThis document provides steps required to load data in BODS from SAP using IDOC interface. This document will be used at the time of proof of concept to demonstrate the ability of Business Objects Data Services (BODS) Real-Time job to receive data using IDOC interface from SAP.The main objectives are:Loading data in BODS from SAP using IDOCDemonstrate how BODS can be used to receive data using IDOCDisplay required configurations step-by-step & invocationDemonstrate actual sample load using BODS Real-Time jobOverview:This document provides systematic approach for the data load in three sections:1. Configuration Required at SAP Side:2. Configuration Required at BODS Side:3. Demonstration Sample Load using BODS Job:Each of above sections provide information as in how to configure the specific parameters required to get both SAP & BODS ready to communicate with each other. Screen shots provided for better understanding of the parameters & their configuration. SAP [ECC 6.0] Configuration for Outbound IDOC Communication

1. Display IMG [/nsale]Connect to SAP System using SAP [ECC 6.0] GUIUse Transaction Code (TCODE) /nsale to Display IMG

1.0 Display IMG2. Defining Logical System Expand Logical Systems tree & click on the Clock icon in front of Define Logical Systems

2.0 Logical Systems: overview Click on the New Entries ButtonFor SAP Outbound IDOC, two Logical Systems are requiredFirst Logical System for SAP (ID4CLNT100)Second Logical System for BODS (ID100CBODS)Create these two Logical System as shown below

2.1 New Entries for Logical System Click on the "SAVE" Button to save both the Logical Systems. Confirmation message Data was Saved will be displayed as shown below.

2.2 Data Saved ConformationVerify two (2) Logical Systems are present in the SAP

2.3 Verifying Logical System

3. Configure RFC Connection [/nsm59]Use TCODE "/nsm59" to configure RFC Connection

3.0 Configurations of RFC Connection

Select TCP/IP Connection as shown above & click on the Create Button to configure new RFC Connection.

3.1 Creation of New RFC Destination: Technical SettingsProvide required information as shown aboveNote: We are using registered Program ID as DI_RFC_Client. This could be any simple name, but the same name should be used while configuring RFC Clients using BODS Administrator. (Ref: 2.Client Interfaces, page no 21)Click on the MDMP & Unicode tab

3.2 Creation of New RFC Destination: MDMP & UnicodeNote: We have selected Unicode radio button because BODS Target RFC Client is Unicode.Important: There are two buttons Connection Test and Unicode Test we will test RFC Connection after Configuring RFC Client using BODS Administrator in Steps provided later in this document. (Ref: 3. Testing RFC Client from SAP, page no 23) 4. Create Transactional RFC Port [/nwe21]Use TCODE "/nwe21" to configure the RFC Port Expand Ports tree to go to Transactional RFC

4.0 Creation of New Transactional RFC Port Select Transactional RFC and click on Create Button.

4.1 Transactional RFC Port NameProvide Name "DI_PORT" as shown above.Click on Continue button Port Name "DI_PORT" will appear in the list as shown below.

4.2 Transactional RFC Ports OverviewClick on the DI_PORT to provide Description & RFC destination (DI_RFC_DESTINATION ) that we have created in earlier section (Ref: 3.Configure RFC Connection, page no 7)5. Create Partner Profile [/nwe20]Use TCODE "/nwe20" to configure Partner Profile.

5.0 Creation of New Partner Profile Click on Create button to create new partner profileCreate partner profile as shown below Note : 1. Provide partner no. ID100CBODS i.e same as BODS Logical System.2. Agent 50010120 EDI Department is necessary to communicate with non-SAP systems like BODS. Without this agent IDOC communication will not be successful.

5.1 Partner Profile Permitted AgentClick on the Save button to save the partner profileAfter saving the "Create outbound parameter" will be activated in Outbound paramtrs

5.2 Create outbound paramtrs.Click on Create outbound parameter button in Outbound paramtrs section.

5.3 Partner Profile: Outbound parametersProvide the Message Type "MATMAS" & IDoc Basic type "MATMAS01"Save "Partner profile: Outbound parameters" using Save buttonConfirm Partner Profile & Outbound Parameters added.

5.4 Outbound Parameters Overview

6. Change Distribution Model [/nbd64]

Use TCODE "/nbd64" to configure Change Distribution Model

Screen Shot 6.0 Change Distribution ModelClick on the Create Model View Note: If this button is inactive then select the Model View in left hand tree and click on Edit button to enable Create Model View button & active.

6.1 Create Model ViewProvide Short text & Technical name as follows:Tech name same as short text.Click continue button The DI_MODEL will be added to the list as shown below

6.2 Distribution Model Changed overviewSelect DI_MODEL and click on Add Message Type button

6.3 Add Message Type for Changed Distribution ModelSender is SAP Logical System & Receiver is BODS Logical System.Provide all information as shown above and click on continue button.The Distribution Model created is shown below.

6.4 over View of Distribution ModelCheck if IDOC distribution is turned on.Run transaction "/nbd61" and make sure the checkbox is set.

6.5 Activation StatusCreate Real Time Job in BODS

1. Create SAP Data storeGo to Data stores tab in BODS Designer, right click on blank area to create "NEW".Create SAP Datastore by providing required SAP Application Server Information & creadentials

1.0 Creation of SAP Data StoreExpand the "SAP_TTL_Client100_DS" datastoreSelect the IDOCs and Right click , click on Import By Name

1.1 IDOC'S importing

1.2 IDOC Import by NameProvide Type & Name as shown above & click on Import to import IDOC in data-store2. Build Real Time JobBuild a Real Time Job with a DataFlow inside

2.0 Real Time Job Overview.Data Flow is as Follows

2.1 Data Flow

Query is used to Unnest as follows

2.2 Query Unnest Note: You can unnest by right clicking on MATMAS01 as shown below

2.3 Unnest OptionThe Target XML file configuration is a follows :

2.4 Target XML file Configuration

Note: Make sure Make Port checkbox is checked. Configuring RT Services and RFC for IDOC through Web Administrator

1. Configuring Real Time Service & Service Provider Open BODS Web Administrator Expand Real-Time tree and select Real-Time Services to do Real-Time Service ConfigurationProvide required information as shown below:

1.0 Web Administrator: Real Time Services ConfigurationJob Name is Real Time Job name that you have created in earlier stage in designer (Ref: 2.Build Real Time Job, page no 18)Browse the Job and select Job nameIn service provider settings - Click on Add button to add Job Server and make it enabled. Once the Service is configured you can start the service by selecting the service & click on start button.

1.1 Real Time Services Status

2. Client Interfaces

Using BODS Web Administrator Expand Real-Time tree and select Client Interfaces to do RFC Client ConfigurationProvide required information as shown below

2.0 RFC Client Configurations Click on Apply buttonNote: RFC Program ID must be same as register Program ID given while configuring RFC Connection. (Ref: 3.Configure RFC Connection, page no 7) Go to List of Supported IDOC tab & then click on Add button to add IDOC.

2.1 Supported IDOC Provide IDOC Type which is BASIC IDOC TYPE in SAP. (Ref: Screen Shot 5.3 Partner Profile: Outbound parameters, page no 13)Service Name must be same as Real Time Service we have created & click on Apply button to get the IDoc Information registered.List of Supported IDOC will be displayed as shown below

2.2 Lists of Supported IDOC'sNote: You may add more than one iDOC type per RFC ClientOnce the RFC Client configured you can start RFC Client.Make sure RFC Client is started & status is Waiting for Idoc as shown below:

Shot 2.3 RFC Client Interface Status

3. Testing RFC Client from SAP Use TCODE "/nsm59" to go to Configuration of RFC ConnectionsOpen TCP/IP Connections & locate DI_RFC_DESTINATIONDouble click & open DI_RFC_DESTINATIONClick on Connection Test button

3.0 Testing RFC Client from SAPIf everything configured properly, you will see success message as shown below.

3.1 Connection Test Success

Note: If Connection test failed, make sure that the RFC client on BODS side started properly with green icon.Sending Material from SAP [/nbd10]1. Sending Material MATMAS from SAP Use TCODE "/nbd10" to "Send Material"

1.0 Sending Material from SAP

Click on the match code button

1.1 Material Numbers / Material Description

Click on Start Search button for finding the entries for material numbers.

1.2 Selections of Material and Material Description Select any Material number from the list displayed above and click on OK button.Selected material number will appear in front of "Material" on Send Material screen. Provide required fields as shown below & make sure the "Logical system" is BODS Logical System ID100CBODS

1.3 Send Material

Click on Execute button. We will get couple of following messages.

1.4 IDOC Information after material sent Click on OK

2. Check Received MATMAS IDOC in BODS Using BODS Web AdminGo to BODS Web Admin, click on Real Time Services to check that one IDOC request received as shown below

2.0 Received Requests in Web Administrator: Real Time Services 1. Checking the output data in XML FileGo to BODS Designer Open the Real-Time Job DataflowGo to the target MATMAS01_OUT.xml & click on view data to confirm the data receivedGenerated output XML file data as follows