SAP NetWeaver 7.0: ETL and EIIOverview

Product Management SAP NetWeaver BI

November 2007

© SAP 2007 / Page 2

1. Overview2. ETL

2.1. Extraction2.2. Transformation2.3. Data Distribution2.4. Real-Time Data Acquisition

3. Distributed Query / EII

Agenda

© SAP 2007 / Page 3

Architecture SAP NetWeaver BI

SAPOperational Data

Non-SAPOperational Data

SAP NetWeaverBI Data3rd-Party BI Data

Information Broadcasting

Analyzer

BI Consumer ServicesBI Consumer Services

Visual ComposerEmbedded BI

BI Kit

Services &B

APIS

Composite

ReportDesigner

Web ApplicationDesigner

WebAnalyzer

Business Explorer Suite (BEx)

SAP NetWeaver Portal

UIs can be embedded

Enterprise Search WorklistsKnowledge Management Collaboration

Data Sources

BI Layer

SAP NetWeaverBI Accelerator

Appliance

Meta D

ataR

epository

Master D

ata

Open HubService

OperationalData Store

Data Marts

Data Warehouse

Virtual-Provider

Query Designer

Planning Modeler

Analytic Engine

PSA

Downstream System

ODBO/XMLA

Web Services

Near-LineStorage

EnterpriseReportBI App MS ExcelAd Hoc

PlanningLayout

© SAP 2007 / Page 4

1. Overview2. ETL

2.1. Extraction2.2. Transformation2.3. Data Distribution2.4. Real-Time Data Acquisition

3. Distributed Query / EII

Agenda

© SAP 2007 / Page 5

Conceptual Layers of Data Warehousing

(Persistent) Staging Area

OperationalData Store

DataWarehouse

(Architected)Data Marts

Any Source

Information Access

DataSources

© SAP 2007 / Page 6

Data Acquisition Layer – DataSource Types

RDBMSDBSL File data XML dataSAP

Source

DataSource

UDConnect

DBConnect

WebService

FileInterface

BI ServiceAPI

e.g. Oracle,Informatix,

MSSQL

e.g. SAPBusiness

Suite

© SAP 2008, SAP NetWeaver BI 7.0/ Page 6

RDBMSJDBC

e.g. Teradata,

BusinessObjects

DataServices

StagingBAPI

any Source,e.g. Oracle

Applications

© SAP 2007 / Page 7

New BI DataSource concept withSAP NetWeaver 7.0

Highlightsunique look and feel for all of the DataSource TypesPSA is attached to DataSource

InfoPackage writes to PSAData Transfer Process writes from PSA to data targets

direct/remote access is optionalpreview feature is standardautomated conversions (e.g. date format detection)

© SAP 2007 / Page 8

Source System Tree

Source sytems categories:

SAP vs. non SAPFile vs. databaseRelational vs.Multidimensional DBABAP vs. JavaXML vs. Text/BinaryPull vs. PushRealtime vs. Batch

© SAP 2007 / Page 9

DataSource Example – One fits all approach

General InformationDescriptionsReconciliation flag (notfunctional)Opening Balance(inventory)Error handling (duprecs)

© SAP 2007 / Page 10

Data Flow Concept in SAP NetWeaver 7.0

SAP NetWeaver BI

Source

InfoProvider

Source System 1

DataSource / PSA

Transformation

Process Chain

InfoPackage

Data TransferProcess

© SAP 2007 / Page 11

Data Flow Concept in SAP NetWeaver 7.0Simplified

SAP NetWeaver BI

Source

InfoProvider

Source System 1

DataSource / PSA

Transformation

Process Chain(optional)

Data TransferProcess

XRestrictions:

Not optimized formass data transfer

No packaging of data

Full Mode Only

© SAP 2007 / Page 12

1. Overview2. ETL

2.1. Extraction2.2. Transformation2.3. Data Distribution2.4. Real-Time Data Acquisition

3. Distributed Query / EII

Agenda

© SAP 2007 / Page 13

Data Flow in SAP NetWeaver 7.0 BI

Any Source

SAP NetWeaver Business Intelligence

DataSource / PSA

Non-SAP SAPSAP NetWeaver BI

SAP NetWeaver PI

InfoProvider

Transformation

InfoPackage

Downstream Systems

Open Hub DestinationInfoProvider

DataTransferProcess

DTP DTP

InfoPackage

Transformation Transformation

© SAP 2007 / Page 14

Transformation

Source

Package 1

Transformation

Start Routine

ExpertRoutine

End Routine

Transformation Rule 1Transformation Rule n

Package 2Package m

Target

SemanticGroups

* optional

*

*

*

*

Universal transformation from sourceto target objects

Transformation types:Move, aggregate, constant, master data lookup, …Business rules, e.g. unit + currency translationFormula builder with rich predefined functionslibraryABAP routines incl. regular expressions

SAP NetWeaver 7.0 EnhancementsIntuitive UIUnit conversionUnified transfer + update rules into all-in-onecapabilityIntegration of Open Hub Service

© SAP 2007 / Page 15

Transformation – Definition

Access from the Data Warehousing WorkbenchNew transformation

Unification of transfer and update rulesInfoSource not mandatory anymore

Former concept of update rulesSmall square next to the transformation iconAccess from context menu via ‘additional functions’

Links sources and targetNew source: InfoSetOther sources: DataSource, InfoCube, DataStore object, InfoObject, InfoSourceTargets: InfoCube, DataStore object, InfoObject, InfoSource, Open Hub Destination

Transformation

Update rule

© SAP 2007 / Page 16

Transformation – Graphical UI

Sourcefields

Targetfields

Rules pergroup

Note: Key figures, characteristics and datefields are shown on the same level(transformation group)

© SAP 2007 / Page 17

Transformation Rules

Transformation rule detailsInformation on

Rule typeCurrency/UnitConversionSourcefieldsTarge fields

© SAP 2007 / Page 18

Enhanced Data Flow in SAP NetWeaver 7.0 BI

SAP NetWeaver Business Intelligence

DataSource / PSA

InfoProvider

DataTransferProcess

InfoProvider

TransformationTransformation

InfoSource (optional)

Transformation (optional)

InfoSource (optional)

Transformation (optional)

© SAP 2007 / Page 19

Transformations – InfoSource – 1 –

InfoSourceTransformation directly links from a source InfoProvider (or DataSource) to a targetInfoProviderAn InfoSource is usually not neededNew InfoSource architecture is used (flat InfoObject-based structure)Scenarios for (flexible) InfoSource

A flexible InfoSource is necessary in order to use currency or unit conversionfrom the source DataSource Define InfoSource as an intermediate structureYou can use a flexible InfoSource as a uniform source for several targets; theInfoSource can the be target from different sources (see next slide)

Note: for ‘direct’ InfoSources (for master data updates), there is no differencebetween ‘old’ and ‘new’ InfoSource, i.e. you can define a transformation as well astransfer rules

Pre-requisite: InfoObject is defined as InfoProvider

© SAP 2007 / Page 20

Transformations – InfoSource – 2 –

InfoSourceScenario: InfoSource as a uniform source for several targets and as target fromdifferent sources

SAP NetWeaver Business Intelligence

DataSource

InfoProvider

DataSource 1 DataSource 2 DataSource n…

InfoProvider 1 InfoProvider 2 InfoProvider m…

Transformation…

InfoSource

Transformation…

© SAP 2007 / Page 21

Transformation Groups – 1 –

Transformation GroupsSummarize key figures with the same characteristics assignments

All key figures of one transformation are updated based on the same characteristicvaluesIf other characteristic updates are necessary for particular key figures, a newtransformation is created

© SAP 2007 / Page 22

Transformation Groups – 2 –

Transformation GroupsUse / Example

Scenario: overview on bonus-relevant sales of all employeesAn employee generates a certain sales volume, which is the basis for his/her bonusThe manager of the employee will be assigned 10% of the employee’s bonus as manager’sbonus relevant

two transformation groups are generated (e.g. ‘employee’ and ‘manager’)

Giles1000Johnson

ManagerSales VolumeEmployee

Source

Transformation Group 1Employee Employee

Sales Volume Bonus-relevant Sales

100Giles

1000Johnson

Bonus-relevant SalesEmployee

Target Transformation Group 1Manager Employee

Sales Volume*0,1 Bonus-relevant Sales

© SAP 2007 / Page 23

1. Overview2. ETL

2.1. Extraction2.2. Transformation2.3. Data Distribution2.4. Real-Time Data Acquisition

3. Distributed Query / EII

Agenda

© SAP 2007 / Page 24

Data Transfer Process: Complex Example

DataStore Object 1

DTP

DataStore Object 2

DataStore Object 3

InfoSource

DTPDTP

DTPDTP

DTP

IP IP

TR

TR TR

TR

TRTR

DataSource (PSA)DataSource (PSA) DataSource (PSA)DataSource (PSA)

Source System 1 Source System 2

TRProcess Chain

Process Chain

SAP Netweaver BI

© SAP 2007 / Page 25

Benefits of New Data Transfer Process

Data Transfer Process (DTP): data ‘distribution’ withinSAP NetWeaver BI

Loading data from one layer to others except InfoSourcesSeparation of delta mechanism for different data targetsEnhanced filtering in dataflowImproved transparency of staging processes across datawarehouse layers(PSA, DWH layer, ODS layer, Architected Data Marts)Improved performance: optimized parallelizationEnhanced error handling for DataStore object (error stack)Enables real-time data acquisition

© SAP 2007 / Page 26

Process chaincan automatethe loadingprocess

Error Handling Overview

DTP SchedulerDTP Scheduler

Error StackError Stack

DataSource (PSA)DataSource (PSA)IP DTP

There is no error handlingavailable for an InfoPackage.In case of invalid records,data needs to be reloadedfrom the source system.

Invalid records can be corrected in the errorstack and updated into the data target

Source System

Error DTP

© SAP 2007 / Page 27

Error Handling Features

Possibility to choose in the scheduler to...abort process when errors occurprocess the correct records but do not allow reporting on themprocess the correct records and allow reporting on them

Number of wrong records which lead to a wrong request

Invalid records can be written into an error stack

Keys should be defined for error stack to enable the error handling ofDataStore object

Temporary data storage can be switched on/off for each substep of theloading process

Invalid records can be updated into data targets after their correction.

© SAP 2007 / Page 28

Error Stack

Error StackStores erroneous records

Automatic checks: Existence of master data, conversion exit (restricted, e.g.Alpha)Customer-defined checks in transformation routines (see appendix for moreinformation)

Keeps the right sequence of records for consistent DataStore handlingKey of error stack defines which data should be detained from the update after theerroneous data recordAfter correction, Error-DTP updates data from error stack to data targetNote: Once the request in the source object is deleted, the related data records inerror stack are automatically deleted

© SAP 2007 / Page 29

Temporary Data Storage

Temporary Data StorageHelp for tracing the erroneous records and transformationsData records from different steps within the data transfer process can be storedtemporarilyStores complete set of data (erroneous as well as valid records)Scenario:

If the debugging mode is switched onTrace the erroneous recordsTrace Transformation

© SAP 2007 / Page 30

Data Transfer Process Monitor

DTP MonitorIntegrated in InfoProvider management screenIntegrated in DTP maintenanceAdditional information: duration of each stepTemporary storage access – if activatedError Stack is displayed in DTP Monitor

Data display intemporary storage

Error Stack

© SAP 2007 / Page 31

1. Overview2. ETL

2.1. Extraction2.2. Transformation2.3. Data Distribution2.4. Real-Time Data Acquisition

3. Distributed Query / EII

Agenda

© SAP 2007 / Page 32

Motivation for real-time

Upload frequency for regular staging not sufficientNumber of requests can not be handled by BI systemReporting requirements of data with sub 1 hours actuality

Operational reportingThe clear distinction between tools for analytical applications and tools for operationalapplications is more and more difficult.See transactional data in reporting as it is created in source system („post & see“)

Availability of data in source system not knownUse „Push“ mechanism (e.g. SAP NW XI)Polling orTrigger data load process via event from source system

Remote access of data not feasible due to resource consumption insource system (and cross application reporting requirements)

Splitting of staging processes (EDW) – General business rulesare processed during the day whereas scenario specific rules (definedby application, region, time zone, etc.) are processed at customerdefined times

© SAP 2007 / Page 33

ExternalSystem

Web ServicePush

Real-time Data Acquisition (RDA)

SAP Source SystemService API

ApplicationDelta QueueReal-time

Update

SAP NetWeaver Business Intelligence

DataSource

Data Warehouse Layer

Operational DataStore

Architected Data Marts

Daemon

Pull~ 1/min

InfoPackage for Real-Time Data Acquisition

DataTransfer Process forReal-time Data Acquisition

Scheduled DataTransfer Process

Scheduled DataTransfer Process

Real-timeUpdate

PSA

DataStore Objects

© SAP 2007 / Page 34

RDA Scenarios

Real-time data acquisition can be used in two primary scenarios:via the Service API (SAPI)

Incorporates usage of InfoPackage for Real-time Data Acquisition (source to PSA)Then leverages Data Transfer Process for Real-time Data Acquisition (PSA toDataStore Object)

via a Web ServiceIncorporates usage of Web Services to populate the PSAThen leverages the Real-time DTP to transfer data to the DataStore Object

A system Daemon is used to initiate and control data transfer in both scenarios.

© SAP 2007 / Page 35

What is a Daemon ?

DaemonSystem process fulfils a specific task at regular intervalsThe SAP NetWeaver BI Daemon performs three steps (depending on the scenario chosen):

Initiate Service-API (SAPI) data pull via InfoPackage for Real-time DataAcquisition into PSATrack status of data transfer from source systemInitiate update of DataStore Object via DTP

Successful execution of each step is tracked in a control tableAllows restarting if necessaryRestart can be initiated so it starts at the next step after the last successfullyexecuted step.

© SAP 2007 / Page 36

Daemon Monitoring

Maintenance screen for demon control can be called from the Data Warehousing WorkbenchTransaction RSRDA

© SAP 2007 / Page 37

Real-time requests in DataStore Object

Latency between source system transaction and availability in DataStore Object is minimal due to “LeanStaging”.

“Lean Staging” means that the DataStore Object handles the data differently internally – less loggingand activation activity is allowed when the DTP for Real-time Data Acquisition is connected to theDataStore Object

From an administrative point of view, the DataStore Object is a standard one.

The DataStore Object also handles the availability of this real-time data (e.g. data whose request hasnot been closed) for reporting.

DataSource / PSA

ActivationQueue

ActiveData

ChangeLog

Per.1 Per. 2 Per.3

Req.ID, Pack ID, Rec. No.

Req.SID, Pack ID, Rec. No.

Document No.

© SAP 2007 / Page 38

DataSourcePSA

RegularDataStore Object

OLTP

BI

Delta Queue

2LIS_02_VAITM Application

real-timeUpdate

Using one DataSource

How to embed RDA in your data flow

Extraction done via RDA and InfoPackage scheduling(“Normal Delta”) can currently not be done in parallel

DataSource can only use a single extraction mechanism

DataSourcePSA

“RDA”DataStore Object

RegularDataStore Object

OLTP

BI

Daemon

Pull~5/min

Delta Queue

2LIS_02_VAHDR2LIS_02_VAITM

PeriodicScheduling

2LIS_02_VAITM

Application

real-timeUpdate

Using two DataSources

Daemon

Pull~5/min

Well established Data Flow cancan be usedSeparate (potentially) leanerDataStore Object can be usedReporting has to be enabled viaMultiProviderData has to be deleted frequentlyfrom „RDA“ DataStore Object toavoid redundant data

Old data flow has to be replacedby new real-time data flowNo issues with data alignmentfrom different DataSourcesCan facilitate upload of hugedata volumes by spreadingdata load over many smallerdata loads

PeriodicScheduling

2LIS_02_VAITM

© SAP 2007 / Page 39

1. Overview2. ETL

2.1. Extraction2.2. Transformation2.3. Data Distribution2.4. Real-Time Data Acquisition

3. Distributed Query / EII

Agenda

© SAP 2007 / Page 40

Scenarios

Direct/Remote Access ScenariosIntegration of external data into BI Analyses without storing data persistently into BI (Example:Teradata Integration)SAP NetWeaver BI is owning the data modelPersistency of the transient data might be an option for the future (integration can be the firststep into an migration scenario)Master data of transient transactional data has to be available in the BI environment

© SAP 2007 / Page 41

VirtualProvider

VirtualProviderScenarios

Data stored outside BI (other SAP or Non-SAP systems)Analysis requires complex algorithms which can be implemented via VirtualProvider

Support of InfoObjects for direct accessTypes:

BAPIFunction ModuleDTP

VirtualProviderw/o data persistency

Types

BAPI FunctionModule DTP

© SAP 2007 / Page 42

Direct Access: Simple Example

TRANSFORMATION

DataSource(direct access)DataSource

(direct access)

SourceSystem

BI

DataTransfer Processfor

Direct Access

VirtualProvider

Query

© SAP 2007 / Page 43

Direct Access: Complex Example II

InfoSourceDTP for

Direct AccessTR

TR

Transactional Data

TR

BIVirtualProvider

Query

DataSource(direct access)DataSource

(direct access)

TR

Master Data

DataSource(direct access)DataSource

(direct access)

Source System

Remote InfoObject

DTP forDirect Access

© SAP 2007 / Page 44

Remote Characteristic

© SAP 2007 / Page 45

DTP Type: Direct Access

© SAP 2007 / Page 46

Copyright 2007 SAP AGAll rights reserved

No part of this publication may be reproduced or transmitted in any form or for any purpose without the express permission of SAP AG. The information contained herein may be changedwithout prior notice.Some software products marketed by SAP AG and its distributors contain proprietary software components of other software vendors.SAP, R/3, mySAP, mySAP.com, xApps, xApp, SAP NetWeaver, Duet, Business ByDesign, ByDesign, PartnerEdge and other SAP products and services mentioned herein as well as theirrespective logos are trademarks or registered trademarks of SAP AG in Germany and in several other countries all over the world. All other product and service names mentioned andassociated logos displayed are the trademarks of their respective companies. Data contained in this document serves informational purposes only. National product specifications may vary.

The information in this document is proprietary to SAP. This document is a preliminary version and not subject to your license agreement or any other agreement with SAP. This documentcontains only intended strategies, developments, and functionalities of the SAP® product and is not intended to be binding upon SAP to any particular course of business, product strategy,and/or development. SAP assumes no responsibility for errors or omissions in this document. SAP does not warrant the accuracy or completeness of the information, text, graphics, links, orother items contained within this material. This document is provided without a warranty of any kind, either express or implied, including but not limited to the implied warranties ofmerchantability, fitness for a particular purpose, or non-infringement.

SAP shall have no liability for damages of any kind including without limitation direct, special, indirect, or consequential damages that may result from the use of these materials. This limitationshall not apply in cases of intent or gross negligence.The statutory liability for personal injury and defective products is not affected. SAP has no control over the information that you may access through the use of hot links contained in thesematerials and does not endorse your use of third-party Web pages nor provide any warranty whatsoever relating to third-party Web pages

Weitergabe und Vervielfältigung dieser Publikation oder von Teilen daraus sind, zu welchem Zweck und in welcher Form auch immer, ohne die ausdrückliche schriftliche Genehmigung durchSAP AG nicht gestattet. In dieser Publikation enthaltene Informationen können ohne vorherige Ankündigung geändert werden.Einige von der SAP AG und deren Vertriebspartnern vertriebene Softwareprodukte können Softwarekomponenten umfassen, die Eigentum anderer Softwarehersteller sind.SAP, R/3, mySAP, mySAP.com, xApps, xApp, SAP NetWeaver, Duet, Business ByDesign, ByDesign, PartnerEdge und andere in diesem Dokument erwähnte SAP-Produkte und Servicessowie die dazugehörigen Logos sind Marken oder eingetragene Marken der SAP AG in Deutschland und in mehreren anderen Ländern weltweit. Alle anderen in diesem Dokument erwähntenNamen von Produkten und Services sowie die damit verbundenen Firmenlogos sind Marken der jeweiligen Unternehmen. Die Angaben im Text sind unverbindlich und dienen lediglich zuInformationszwecken. Produkte können länderspezifische Unterschiede aufweisen.

Die in diesem Dokument enthaltenen Informationen sind Eigentum von SAP. Dieses Dokument ist eine Vorabversion und unterliegt nicht Ihrer Lizenzvereinbarung oder einer anderenVereinbarung mit SAP. Dieses Dokument enthält nur vorgesehene Strategien, Entwicklungen und Funktionen des SAP®-Produkts und ist für SAP nicht bindend, einen bestimmtenGeschäftsweg, eine Produktstrategie bzw. -entwicklung einzuschlagen. SAP übernimmt keine Verantwortung für Fehler oder Auslassungen in diesen Materialien. SAP garantiert nicht dieRichtigkeit oder Vollständigkeit der Informationen, Texte, Grafiken, Links oder anderer in diesen Materialien enthaltenen Elemente. Diese Publikation wird ohne jegliche Gewähr, wederausdrücklich noch stillschweigend, bereitgestellt. Dies gilt u. a., aber nicht ausschließlich, hinsichtlich der Gewährleistung der Marktgängigkeit und der Eignung für einen bestimmten Zwecksowie für die Gewährleistung der Nichtverletzung geltenden Rechts.

SAP übernimmt keine Haftung für Schäden jeglicher Art, einschließlich und ohne Einschränkung für direkte, spezielle, indirekte oder Folgeschäden im Zusammenhang mit der Verwendungdieser Unterlagen. Diese Einschränkung gilt nicht bei Vorsatz oder grober Fahrlässigkeit.Die gesetzliche Haftung bei Personenschäden oder die Produkthaftung bleibt unberührt. Die Informationen, auf die Sie möglicherweise über die in diesem Material enthaltenen Hotlinkszugreifen, unterliegen nicht dem Einfluss von SAP, und SAP unterstützt nicht die Nutzung von Internetseiten Dritter durch Sie und gibt keinerlei Gewährleistungen oder Zusagen überInternetseiten Dritter ab.

Alle Rechte vorbehalten.