“Good to Know” Topics for a Smooth SAP NetWeaver MDM 7.1 Implementation

SAP NetWeaver

How-To Guide

Good to know topics for a smooth

SAP NetWeaver MDM 7.1

implementation

Applicable Releases:

SAP NetWeaver MDM 7.1 (SP01-SP05)

Topic Area:

Information Management

Capability:

Master Data Management

Version 1.02

December 2010

Copyright 2010 SAP AG. All rights reserved.

No part of this publication may be reproduced or

transmitted in any form or for any purpose without the

express permission of SAP AG. The information contained

herein may be changed without prior notice.

Some software products marketed by SAP AG and its

distributors contain proprietary software components of

other software vendors.

Microsoft, Windows, Outlook, and PowerPoint are

registered trademarks of Microsoft Corporation.

IBM, DB2, DB2 Universal Database, OS/2, Parallel

Sysplex, MVS/ESA, AIX, S/390, AS/400, OS/390,

OS/400, iSeries, pSeries, xSeries, zSeries, z/OS, AFP,

Intelligent Miner, WebSphere, Netfinity, Tivoli, Informix,

i5/OS, POWER, POWER5, OpenPower and PowerPC are

trademarks or registered trademarks of IBM Corporation.

Adobe, the Adobe logo, Acrobat, PostScript, and Reader

are either trademarks or registered trademarks of Adobe

Systems Incorporated in the United States and/or other

countries.

Oracle is a registered trademark of Oracle Corporation.

UNIX, X/Open, OSF/1, and Motif are registered

trademarks of the Open Group.

Citrix, ICA, Program Neighborhood, MetaFrame,

WinFrame, VideoFrame, and MultiWin are trademarks or

registered trademarks of Citrix Systems, Inc.

HTML, XML, XHTML and W3C are trademarks or

registered trademarks of W3C, World Wide Web

Consortium, Massachusetts Institute of Technology.

Java is a registered trademark of Sun Microsystems, Inc.

JavaScript is a registered trademark of Sun Microsystems,

Inc., used under license for technology invented and

implemented by Netscape.

MaxDB is a trademark of MySQL AB, Sweden.

SAP, R/3, mySAP, mySAP.com, xApps, xApp, SAP

NetWeaver, and other SAP products and services

mentioned herein as well as their respective logos are

trademarks or registered trademarks of SAP AG in

Germany and in several other countries all over the world.

All other product and service names mentioned are the

trademarks of their respective companies. Data contained

in this document serves informational purposes only.

National product specifications may vary.

These materials are subject to change without notice.

These materials are provided by SAP AG and its affiliated

companies ("SAP Group") for informational purposes only,

without representation or warranty of any kind, and SAP

Group shall not be liable for errors or omissions with

respect to the materials. The only warranties for SAP

Group products and services are those that are set forth in

the express warranty statements accompanying such

products and services, if any. Nothing herein should be

construed as constituting an additional warranty.

These materials are provided as is without a warranty of

any kind, either express or implied, including but not

limited to, the implied warranties of merchantability,

fitness for a particular purpose, or non-infringement.

SAP shall not be liable for damages of any kind including

without limitation direct, special, indirect, or consequential

damages that may result from the use of these materials.

SAP does not warrant the accuracy or completeness of the

information, text, graphics, links or other items contained

within these materials. SAP has no control over the

information that you may access through the use of hot

links contained in these materials and does not endorse

your use of third party web pages nor provide any warranty

whatsoever relating to third party web pages.

SAP NetWeaver How-to Guides are intended to simplify

the product implementation. While specific product

features and procedures typically are explained in a

practical business context, it is not implied that those

features and procedures are the only approach in solving a

specific business problem using SAP NetWeaver. Should

you wish to receive additional information, clarification or

support, please refer to SAP Consulting.

Any software coding and/or code lines / strings (Code)

included in this documentation are only examples and are

not intended to be used in a productive system

environment. The Code is only intended better explain and

visualize the syntax and phrasing rules of certain coding.

SAP does not warrant the correctness and completeness of

the Code given herein, and SAP shall not be liable for

errors or damages caused by the usage of the Code, except

if such damages were caused by SAP intentionally or

grossly negligent.

Disclaimer

Some components of this product are based on Java. Any

code change in these components may cause unpredictable

and severe malfunctions and is therefore expressively

prohibited, as is any decompilation of these components.

Any Java Source Code delivered with this product is only

to be used by SAPs Support Services and may not be

modified or altered in any way.

Document History

Document Version Description

1.00 First official release of this guide for SAP NetWeaver MDM 7.1

1.01 Change chapter 4.4 System landscape for parallel and serial imports

and moved to chapter 6.11 Parallel Import

Change chapter 9.1.2 Disable stemming for matching to 9.1.2

Stemming for matching

1.02 Change chapter 5.10 Number of images

Typographic Conventions

Type Style Description

Example Text Words or characters quoted

from the screen. These

include field names, screen

titles, pushbuttons labels,

menu names, menu paths,

and menu options.

Cross-references to other

documentation

Example text Emphasized words or

phrases in body text, graphic

titles, and table titles

Example text File and directory names and

their paths, messages,

names of variables and

parameters, source text, and

names of installation,

upgrade and database tools.

Example text User entry texts. These are

words or characters that you

enter in the system exactly as

they appear in the

documentation.

Variable user entry. Angle

brackets indicate that you

replace these words and

characters with appropriate

entries to make entries in the

system.

EXAMPLE TEXT Keys on the keyboard, for

example, F2 or ENTER.

Icons

Icon Description

Caution

Note or Important

Example

Recommendation or Tip

Table of Contents

1. Business Scenario .......................................................................................................... 1

2. Related Information ......................................................................................................... 1

3. Prerequisites.................................................................................................................... 1

4. Architecture/ Landscape ................................................................................................. 2

4.1 Overview .................................................................................................................. 2

4.2 Network .................................................................................................................... 3

4.3 Number of servers .................................................................................................... 3

5. MDS .................................................................................................................................. 4

5.1 MDM Server Architecture .......................................................................................... 4

5.2 Concurrent users ...................................................................................................... 5

5.3 Memory usage .......................................................................................................... 5

5.4 Repository load time ................................................................................................. 6

5.5 Accelerator files ........................................................................................................ 7

5.6 MDM Server Client Requests .................................................................................... 8

5.6.1 Process Flow ................................................................................................ 8

5.6.2 Queue Handling Locking ............................................................................ 9

5.7 MDS.ini file parameter: CPU Count ......................................................................... 14

5.8 MDS.ini file parameter: "Max Threads Per Operation" ............................................. 14

5.9 MDS.ini file parameter: Extra DBConnection Validation ........................................... 15

5.10 Number of images .................................................................................................. 15

5.11 Performance for AIX ............................................................................................... 15

6. Import and MDIS ............................................................................................................ 16

6.1 Steps during update and import .............................................................................. 16

6.2 MDIS.ini File Parameters ........................................................................................ 17

6.2.1 Interval ....................................................................................................... 18

6.2.2 Chunk Size ................................................................................................. 18

6.2.3 Inter-Chunk Delay MS................................................................................. 19

6.2.4 No. Of Chunks Processed In Parallel .......................................................... 20

6.3 Slicing..................................................................................................................... 20

6.4 Registry parameters................................................................................................ 23

6.5 Bulk_Import_Silo..................................................................................................... 23

6.6 Cartesian Product ................................................................................................... 25

6.7 How to import lookup main ...................................................................................... 29

6.8 Importing single source field to multiple display fields .............................................. 32

6.9 Strategies for Attribute/Value Import........................................................................ 34

6.10 Excel Import............................................................................................................ 37

6.11 Parallel Import ........................................................................................................ 40

6.12 Transport of maps (value mapping) ......................................................................... 41

6.13 Import XML (XSD.exe) ............................................................................................ 42

6.14 How to avoid disappearing value mappings............................................................. 42

6.15 Difference between Value Exceptions and Import Exceptions.................................. 43

6.16 Limitation on number of source fields ...................................................................... 43

6.17 Limitation on import file size .................................................................................... 43

6.18 Port Sequence ........................................................................................................ 43

6.19 Access, Excel, complex XML not supported on Win64 ............................................ 45

6.20 Maximum multi-record value ................................................................................... 45

7. Syndicator and MDSS ................................................................................................... 46

7.1 Steps during Syndication ........................................................................................ 46

7.2 MDSS.ini Parameters ............................................................................................. 48

7.3 Syndication remote keys ......................................................................................... 50

7.4 Suppressing Initial Syndication................................................................................ 50

7.5 Field triggers for syndication ................................................................................... 51

7.6 Changed Records are not being syndicated .......................................................... 52

8. Data Model ..................................................................................................................... 53

8.1 Performance considerations.................................................................................... 53

8.2 Number of main table fields..................................................................................... 54

8.3 Lookup tables ......................................................................................................... 54

8.4 Nested lookups ....................................................................................................... 55

8.5 Qualified tables ....................................................................................................... 55

8.6 Lookup main uni-/ bidirectional relationship .......................................................... 57

8.7 Tuples .................................................................................................................... 61

8.8 Validation Tuples vs. Validation on qualified table ................................................... 62

8.9 Key mapping........................................................................................................... 66

8.10 Change tracking...................................................................................................... 67

8.11 Calculated fields ..................................................................................................... 67

8.12 Keyword index ........................................................................................................ 67

8.13 Multi-Lingual Fields ................................................................................................. 68

8.14 Sort index ............................................................................................................... 69

8.15 Display fields .......................................................................................................... 70

8.16 Taxonomy............................................................................................................... 70

8.17 Relationships .......................................................................................................... 71

8.18 Workflow................................................................................................................. 71

9. Core Features ................................................................................................................ 72

9.1 Matching ................................................................................................................. 72

9.1.1 Matching Performance ................................................................................ 72

9.1.2 Stemming for matching ............................................................................... 74

9.1.3 Matching Strategy: Match attribute values ................................................... 75

9.2 Stemming ............................................................................................................... 77

9.3 Workflow................................................................................................................. 78

9.3.1 Accessing the Workflows Table .................................................................. 78

9.3.2 Import and Workflow ................................................................................... 79

9.3.3 Completed Workflows ................................................................................. 79

9.3.4 Workflow Thread ........................................................................................ 79

10. JAVA-API ....................................................................................................................... 80

10.1 Notifications (Events) .............................................................................................. 80

11. Portal Integration ........................................................................................................... 81

11.1 Portal MDM connectivity ...................................................................................... 81

11.2 Garbage Connections ............................................................................................. 83

11.3 How to find MDM 7.1 Portal Content build version ................................................... 83

11.4 MDM Web Services ................................................................................................ 84

11.5 MDM Web Dynpro Components.............................................................................. 86

12. PI Adapter ...................................................................................................................... 88

12.1 Communication with MDS ....................................................................................... 88

12.1.1 Import - MDM PI Adapter to MDS................................................................ 88

Good to know topics for a smooth SAP NetWeaver MDM 7.1 implementation

December 2010 1

1. Business Scenario

This document describes challenges often experienced in SAP NetWeaver Master Data Management (SAP NetWeaver MDM) implementation projects. Besides the explanation of "technical" topics like Multithreading, Locking, and Scalability, there are also "application"-specific topics like Matching, Import Steps, and Syndication Steps discussed, and in addition data modeling recommendations are provided. All of this is given with the purpose of making your own implementation project smoother as well as providing an understanding of the internal architecture of SAP NetWeaver MDM.

As the way MDM handles some of the described topics will change in future releases, please

check for further versions of this document.

In addition please check the coming release notes for SAP NetWeaver MDM 7.1 SP06 for

planned innovations to optimize performance, stability and supportability.

2. Related Information

SAP MDM 7.1 Documentation Center: service.sap.com/installMDM

How to guides: www.sdn.sap.com/irj/sdn/howtoguides Information Management/ Data Unification

o Developing Applications on Top of SAP NetWeaver MDM - An Architect's Guide

http://www.sdn.sap.com/irj/scn/index?rid=/library/uuid/7012ea38-6069-2c10-0097-a95ad0b9fc21

o Best Practices Workflow for Master Data Management

http://www.sdn.sap.com/irj/scn/index?rid=/library/uuid/50f1c01b-972d-2c10-3d9d-90887014fafb

o Best Practices for Repository Migration from SAP NetWeaver MDM 5.5 to SAP NetWeaver MDM 7.1

http://www.sdn.sap.com/irj/sdn/index?rid=/library/uuid/80765a21-78f3-2b10-74a2-dc2ab57a1bd2

o How to Optimize an MDM Matching Process

http://www.sdn.sap.com/irj/sdn/go/portal/prtroot/docs/library/uuid/408b8031-6fc4-2b10-c18b-a77abefb75b9

o How to Activate Field Triggers for Syndication

http://www.sdn.sap.com/irj/sdn/go/portal/prtroot/docs/library/uuid/60a2b4e4-6993-2a10-0e9a-c6d720f1571b

o How to Avoid Problems with Your Data Model in SAP NetWeaver MDM Do's and Don'ts http://www.sdn.sap.com/irj/sdn/go/portal/prtroot/docs/library/uuid/d0d8aa53-b11d-2a10-aca2-ff2bd42a8692

3. Prerequisites SAP NetWeaver MDM 7.1


December 2010 2

4. Architecture/ Landscape

4.1 Overview

SAP NetWeaver MDM is based on a client-server architecture with several server and a variety of

client components. User interfaces are provided for administrative task, repository design and for

working with master data.

MDM Server

The MDM server is a standalone application implemented in C++ which is not running inside the SAP

NetWeaver Application Server. MDM Server (MDS) is the core component of SAP NetWeaver MDM.

MDM Repository

An MDM repository certainly includes a database of information consisting of text, images, PDFs, and

other data about each record.

MDM Client

Users can access the MDM functions using the MDM portal UI or a variety of client applications for the

different MDM activities.

MDM Import Server

The Master Data Import Server (MDIS) automates the task of importing master data into MDM

repositories. It allows you to import data automatically in conjunction with predefined inbound ports

and import maps.

MDM Syndication Server

The Master Data Syndication Server (MDSS) automates the task of syndicating master data from

MDM repositories. It allows you to export data automatically in conjunction with predefined outbound

ports and outbound maps.


December 2010 3

MDM Architecture Components:

4.2 Network

All MDM and DBMS servers should be installed in one LAN with a backbone of 1 Gbps. A network

connection between MDM Servers and MDM clients should be at least 100 Mbps.

All clients (console, data manager and so on) should reside inside the LAN.

For clients connected via slower networks, the recommendation is to install client software on

Windows Terminal Server, which should be in the fast network environment of the servers.

4.3 Number of servers

It is recommended you separate the DB server, MDS and MDIS/ MDSS. It is also recommended you

separate MDIS and MDSS, but this is not a precondition.

These recommendations are based on test experience as there will be a competition of the different

modules for RAM and CPU if they run on one physical server.

For more information see the sizing guide for SAP NetWeaver MDM 7.1. ...


December 2010 4

5. MDS

5.1 MDM Server Architecture

The following chapter gives an overview about the MDM Server Architecture:

The MDM server contains core MDM functionality like the engines for searching, matching and

validating. The MDM server is a standalone application implemented in C++ which is not running

inside the SAP NetWeaver Application Server.

In general, MDS is multithreaded. Multiple threads are used to listen for incoming requests, multiple

working threads are used for performing MDM tasks. Any given request is typically allocated one

thread by the server for its processing, allowing multiple requests to run in parallel on multiple

threads/CPUs. (The only exception here is write requests, which reserve the repository exclusively in

order to ensure data integrity)

The MDM repositories are stored in relational databases. One MDM server can access multiple MDM

repositories that may be persisted on one or on multiple database servers. It is also possible to store a

single MDM repository using multiple (predefined) physical partitions that are stored in separate

databases.

Inside the MDM server there are the core MDM services that provide the functionality and a database

abstraction layer that encapsulates the details of different relational database management systems.

To achieve high performance the MDM server uses an in-memory cache into which complete MDM

repositories (*means the indices) are loaded. Before a repository can be accessed by clients, it must

first be loaded into the cache of the MDM server by an administrator. The in-memory architecture is

required to enable the fast searching and matching provided by the MDM server. But it also has an

impact on hardware requirements and server operation: The server hardware needs enough main

memory to load completely all repositories assigned to that server. Loading of very big repositories

takes considerable time which must be added to the start-up time after a server restart or repository

maintenance.

Clients access the MDM server using a MDM specific network protocol based on TCP/IP.


December 2010 5

5.2 Concurrent users The number of concurrent users is an influencing parameter for sizing. It is the number of read and write connections to the repository, where the connection is every instance of an application that is connected to the repository. Client connections such as GUI Client, API Client, Portal, and so on have parallel connections to the MDM Server. The Portal uses a connection pool for simultaneous accesses to the MDM server. For further information about calculating the concurrent users see the sizing guide for SAP NetWeaver MDM 7.1.

5.3 Memory usage

For the calculation of memory, the used/ allocated memory is used, not the memory needed following

the field definition.

Example: Text field, width 255

For the memory usage, the defined width of 255 is not relevant, but the real-life fill rate is. This means

that fields with a large field width definition do not lead to a high memory usage if the real-life fill rate is

pretty low.


December 2010 6

Example: Example for MS SQL Database:

Tables are defined as nvarchar.

nchar and nvarchar:

Character data types that are either fixed-length (nchar) or variable-length (nvarchar) Unicode data

and use the UNICODE UCS-2 character set.

nvarchar(n):

Variable-length Unicode character data n can be a value from 1 through 4,000 (4*1024). The storage

size, in bytes, is two times the number of characters entered + 2 bytes. The data entered can be 0

characters in length.

nvarchar (2000) corresponds to a storage size of 4000 bytes (Unicode character data) for each record.

5.4 Repository load time

There is a significant difference between the loading time for loading with or without indices. If the

indices are created once, there is normally no need to load the repository with indices again. The

indices have to be recreated only if an error occurs. By loading with indices, the load time can

increase several times compared to a load time for loading without indices.


December 2010 7

What can be done if the load process crashes?

1) Load without creating indices Nevertheless, the missing indices will be created.

Only if step 1 fails:

1) Restart MDS 2) Verify, repair, and load with rebuild indices

In version MDM 7.1 load immediate and load with indices algorithms were reviewed and modified to enable shorter load time:

Load immediate is about 2-4 times faster (in average)

Load with indices is about 2-3 times faster (in average) The improvements to the load times are highly repository dependent. The greatest improvements will likely be seen if any of the following apply:

Usage of multiple CPUs and a large number of fields indexed

Usage of a large number of key mappings

Usage of a large number of small flat lookup tables This could result in even faster load times than mentioned above.

5.5 Accelerator files

The MDS in-memory data representation and indexes are updated and asynchronously flushed to

accelerators on disk. If the repository is loaded immediately, the data representation and indices are

loaded from accelerator files instead of being regenerated by reading them from the DB.

Not all data is loaded into memory. MDM needs continuously access to the DB. For example, PDF

documents or images are loaded from Database on request and not during repository loading.


December 2010 8

Load Repository Immediate

o Loads indexes from accelerator files instead of being regenerated by reading the data

them from the DB and creating the index.

o Automatically builds indexes stored in out-of-date accelerator files.

o Is usually fast.

Load Repository with Update Indices

o Rebuilds all MDM indexes files by reading and indexing data stored in the DB. The

results are flushed on disk as accelerator files.

o Necessary by repository schema changes or if any error occurs.

o Might take significant amount of time to complete.

5.6 MDM Server Client Requests

5.6.1 Process Flow

The following chapter shows the process of the MDM Server client requests:

Each client (MDM Data Manger, Portal client, MDIS, MDSS communicates via TCP /IP

sockets with the MDS server.

Listener thread continuously scans all sockets and assigns requests to worker threads.

Worker thread performs first de-serialization transforming byte stream in C++ classes

If the request requires read operations on one repository, worker thread tries to acquire a

shared lock on the repository

If the request requires write operations on one repository, worker thread tries to acquire an

exclusive lock on the repository

The request is being processed. If repository data were changed, notifications are being sent

Lock is released

Responses end and thread returned to worker thread


December 2010 9

5.6.2 Queue Handling Locking

In general, MDS is multithreaded. Multiple threads are used to listen for incoming requests, multiple

threads are used for reading, and multiple background threads are used for workflow.

A situation might occur that it is not possible to work with the Console, Import or Data Manager. What

is the reason for this behavior?

As described, MDS is multithreaded and will handle multiple read requests in parallel, taking

advantage of multiple CPUs on the system. Any given request is typically allocated one thread by the

server for its processing, allowing multiple requests to run in parallel on multiple threads/CPUs.

The only exception here is write requests, which reserve the repository exclusively in order to ensure

data integrity.

This means that all MDM Requests to a given repository are placed in that repository's Queue. Whereas read requests can be executed in parallel, a write request will require exclusive access to the MDM repository. If a write operation is the next task in the request queue, the execution will wait until previous read requests have been worked off. Then the write operation will be initiated and subsequent operations have to wait until it is finished.


December 2010 10

Multiple CPU is useful for read processes since reads occur in parallel. If one CPU is working at full

capacity, the other one is still available to service read-only requests.

Details of MDM locking:

All MDM operations place either a shared or exclusive lock on the repository. This results in the

following blocking behavior:

o Shared lock operations can occur in parallel with other Shared lock operations

o Shared lock operations block Exclusive lock operations.

o Exclusive lock operations block Shared lock operations

o Exclusive lock operations block Exclusive lock operations

MDM operations can be broken into short duration and long duration operations. The ones that are

important here are the long duration operations because these may block other operations.

Example

A short duration Exclusive lock operations will block all other operations. But this is a minor concern since the operation is short, and therefore the wait time for the other operations will also be short. What is important are the long duration operations, since they may block other operations. Whether they block depends on the above matrix of

Shared vs. Exclusive lock.


December 2010 11

The list of long duration MDM operations and their lock type follows:

o Import - Exclusive

o Syndicate - Exclusive

o Modify many records - Exclusive

o Delete many records - Exclusive

o Add many records to workflow - Exclusive

o Check out, Check in, Roll back many records - Exclusive

o Match many records - Share

All other operations are considered short duration operations. The ones that modify data require an

Exclusive lock, and the ones that just retrieve data require a Shared lock.

To determine if two operations can be run in parallel, check the duration and Lock Type (Shared vs.

Exclusive) of each operation and use the above matrix to determine if they will block each other.

Example

Match many records can occur simultaneously as Retrieve records since both are Shared lock operations.

Match many records cannot occur simultaneously as Import since one is Shared and one

is Exclusive. In this case one will occur first, and the other second.

Note

In a syndication process the read process ends up in a write process when the

syndication date is written back to the database. This leads to an exclusive lock of the

repository. (Syndication is actually multiple operations, most time is spent in Shared

operations, but since there are operations that require an Exclusive lock, the entire

Syndicate is just considered as an Exclusive operation).

Note

Check-in and Check-out operations performance was enhanced. Please see note

1415628.

Note

Record lock will be released when a user is trying to access the record, which was

locked by a dead session, e.g. user application cut the network line, user application

finished without destroying session, user session timed out.

This means invalidated locks are not actually released on disconnect/expiration. Instead,

they stay in place until another application wants to lock the affected record. At that time,

locking code checks if the existing lock has a valid owner. If yes, locking fails. If not, the

old lock is discarded and the record is locked by the new application.

Important

As the lock of a repository only takes a very short period of time, the user might notice

this behavior only during parallel import, syndication of high volumes or during checking

in/out of large volumes of records.


December 2010 12

The performance of import and syndication has been tremendously improved so the

probability of noticing this effect is decreasing.

Example 1: Single processing capability during import and syndication

The Import locks the repository exclusively during the import of each batch. Therefore you cannot work

on the repository during import and you cannot perform a parallel import.

MDM server:

CPU usage:

Data Manager on client server 1:


December 2010 13

Data Manager on client server 2:

Recommendation

Large imports, updates or syndications should be scheduled to times when it will not interrupt regular users.

Example 2:

The following picture shows the activity monitor of the Solution Manager Diagnostics.

1. Opening of huge maps by user Smith caused a read lock. Please see get matched numbers.

2. User C3VTOMDM tries to add an image at the same time. This is requesting a write lock. The write lock will be given when the task 1 is finished. The user C3VTOMDM waits.

3. All other read repository locks for user C2VTOMDM and C4VTOMDM are now waiting for the 2. request with the write lock to finish.

Smith


December 2010 14

Example 3: Request for a server/repository read lock timed out

Under the current locking model, when a user, for example, adds a field to a table, both the server and the repository get locked. When another user attempts to add a field to a table of another repository, the user is unable to lock the server. Until MDM 5.5 SP06 patch 3, users were waiting as long as necessary to acquire the locks. Starting with MDM 5.5 SP06 patch 3 and MDM 7.1, lock attempts time out in 2 minutes to prevent Consoles from locking up for long times. This is not an error and does not

require server to be restarted.

5.7 MDS.ini file parameter: CPU Count The parameter CPU count is replaced by the new parameter Max Threads Per Operation.

5.8 MDS.ini file parameter: "Max Threads Per

Operation"

In MDM 7.1 the old mds.ini option "CPU Count" has been replaced by "Max Threads Per Operation" The parameter defines the maximum number of threads that a single operation is allowed to use. Default value is Auto, which automatically limits the number of threads an operation can use to the number of logical cores on the MDM Server host (if the server has more than two cores, the maximum number of threads MDS will use per operation is equal to number of cores 1). Potential values for the "Max Threads Per Operation" in the MDS.INI file can be

An empty value ("Max Threads Per Operation="): nothing is set. MDS will set the value to auto.

The automatic option ("Max Threads Per Operation=Auto"): MDS will automatically determine the number of cores in the system. If the system has more than two cores, MDS will use up to the number of cores less one thread per operation. Otherwise, it will use one thread per operation.

A fixed number of threads option (an integer number greater than 0): MDS will use up to the specified number of threads per operation.

Any other values (text other than Auto, negative numbers, 0, etc) will be considered to be errors and MDS will switch to Auto.


December 2010 15

Note

This parameter does not limit the total number of threads used by MDS.

Example

A multi-core processor combines two or more independent cores into a single package. The individual core is normally a CPU. A dual-core processor contains two cores, and a

quad-core processor contains four cores.

Number of cores = 2 MDS will use max 1 thread per operation Number of cores = 4 MDS will use max 3 threads per operation

5.9 MDS.ini file parameter: Extra DBConnection

Validation

The parameter makes sure that the DBMS connection is live prior to every DBMS request and silently restores it if necessary. It is useful for the small minority of MDM installations where the network connection between the MDM Server and the DBMS is unreliable and frequently lost. It improves reliability but slows the MDM Server. Therefore the default setting should be False. The idea is that if the network is faulty to the degree that MDM cannot perform, then the network needs to be repaired. The setting (to true) exists for customers who need the reliability while they are waiting for the repair of their network.

..

5.10 Number of images

If images are not categorized under Data Groups then each CRUD operation needs to show the

thumbnail of all images and it can be slow.

But usually images are categorized so that each time you show only the relevant data group, then it is very fast even with overall high number of images.

5.11 Performance for AIX With MDM 7.1 SP03 SAP ships a build for AIX which is optimized for performance. This is based on an updated C++ run time library shipped by IBM, and the customer is encouraged to update his landscape to the relevant AIX C++ RTE release. This fix may significantly improve performance on AIX for single user operations. You can find more details at the following links: http://www-01.ibm.com/support/docview.wss?rs=2239&uid=swg21110831 Section: C++ Runtime Environment AIX 5.3 TL6 - AIX 6.1: XL C++ RTE for AIX, V10.1 July 2009 English international 64-bit versions of the released UNIX platforms. MDM on AIX requires AIX service pack 5.3.0.50, which can be downloaded from

the IBM Web site. Also, MDM requires the IBM C++ Runtime Environment Components for AIX version 8 or higher contained in the Runtime package. This package can be found at: http://www-306.ibm.com/software/awdtools/xlcpp/ Please see note 1342611. .


December 2010 16

6. Import and MDIS

6.1 Steps during update and import

Generally, the following interdependent steps occur during updates to fields of a record.

Automatic validations are performed and all constraints are checked, including rights, unique constraints, and data integrity

DBMS tables are updated

MDS in-memory data representation and indexes are updated (and asynchronously flushed to accelerators on disk)

Calculated fields are refreshed

Workflows are checked and triggered

Notifications are generated

Generally, the following interdependent steps occur during import. During import, this process

becomes more complex.

Import Server processes data to MDS in batches, which are configured in the MDIS.INI file The source file is parsed to fill a batch

The data is normalized, transformed, and mapped to MDM structures

The batch of records is matched against the repository and MDIS determines the import action for each record

Within each batch, Import Server may also include data not provided in the simple update process described above

New lookup records are created including remote keys New attributes and text attribute values including remote keys are created

The MDIS sends the batch of records to MDS and the Import Server continues its processing of source data in parallel. MDIS does not import records directly from import files. Instead, it imports a transformed version of the original record called the virtual extended record. This transformation is part of the overall process through which source data is imported into an MDM repository.

MDS locks the repository as described in the previous chapters and processes the data; it releases its locks between each batch in order to allow clients to refresh

MDIS processes import files in several steps: 1. Scanning Ports: MDIS regularly triggers a port scan. During the port scan it checks if files exist in

the port folder structure that need to be imported. If the port scan finds files, it triggers an import. The actual import starts by separating a file into several chunks if its row count exceeds the chunk size defined in mdis.ini. This step takes usually only a few ms.

2. Structural Transformation Stage: The import file or import chunk undergoes a structure and

value mapping where the source structure is mapped to the target structure and source values are replaced by target values. The number of executions corresponds to the number of chunks or files transformed. The time spent for the transformation depends strongly on the size of the import files. This means source tables are converted to a single virtual table following the rules of the import map. During the structural transformation stage, MDIS flattens all source tables in the import file into one virtual extended table. For text files, there is a 1:1 ratio of source table records to virtual extended table records (although the number of fields per record increases on the virtual extended table).


December 2010 17

For XML files, there can be a 1:many ratio of original records to virtual extended records, depending on the way record data is contained in the XML schema (Please see chapter Cartesian Product). Also during the structural transformation stage, MDIS applies any field transformation operations (cloning, splitting, pivoting, etc.) specified in the import map. If there are discrepancies between the structure of the tables in the import file and the structure in the import map (e.g. missing fields), MDIS will be unable to proceed. If this occurs, MDIS logs a structural error and moves the source file to a separate folder for manual processing in the Import Manager.

3. Value Transformation Stage. Source values on the virtual table are converted according to the

rules of the import map. 4. Trigger Import: The transformed virtual records are imported into the MDM repository. The

number of executions corresponds to the number of chunks or files imported.

6.2 MDIS.ini File Parameters The parameter Interval defines the number of seconds MDIS waits between scans of the MDM

Server.

The following parameters can be used to adjust the responsiveness of MDS during an import:

Chunk Size={number of records for each import chunk, default is 50000}

Inter-Chunk Delay MS= {delay between chunks, default is 0}

No. Of Chunks Processed In Parallel


December 2010 18

6.2.1 Interval

Interval= The number of seconds MDIS waits between scans of the MDM Server.

For MDM 7.1 the files are immediately processed once theyre put into a ready-folder. This means

MDIS directly picks up the file no special configuration is required.

The parameter Interval is left in the MDIS.ini file as a safeguard. This means if MDIS should miss a

notification, the port will be scanned at the latest following the interval definition.

6.2.2 Chunk Size

Chunk Size={number of records for each import chunk, default is 50000}

The Chunk Size parameter defines the number of virtual extended records to include in a chunk.

Typically, the smaller the chunk size the better, as smaller chunks can be processed faster within

MDIS and also require less wait time in transactions between MDIS and the MDM Server. However,

setting the chunk size too low can reduce overall performance because there is a fixed overhead per

chunk.

Example

The import task downloads its own source file as a whole and processes it. If the chunk size is set to 20000, a file containing 50000 records is split into three files

(20000/20000/10000).


December 2010 19

Recommendation

Chunk Size=50000 (default)

Tweak this value slowly, as the larger the chunk size, the more memory is

consumed by MDIS.

Setting this parameter smaller reduces the time it takes MDS to process each

chunk of records.

Setting this parameter larger reduces the total time it takes MDS to process all

the chunks. Since MDIS uses a multithreaded pipeline to process chunks, you

should set the chunk size so that large imports are broken up into multiple

chunks.

6.2.3 Inter-Chunk Delay MS

Inter-Chunk Delay MS= {delay between chunks, default is 0}

The number of milliseconds MDIS waits between processing chunks of records to allow other clients

access to MDS.

Note

Increasing this parameter gives MDS more time to process other client requests in

between chunks.


December 2010 20

6.2.4 No. Of Chunks Processed In Parallel No. Of Chunks Processed In Parallel= {Number. The number of chunks processed simultaneously during streaming import}

The chunks are created in parallel. Therefore multiple CPUs help create parallel chunks. The final

import step in MDS is done sequentially and cannot make use of multiple CPUs.

The No. of Chunks Processed in Parallel parameter optimizes the way chunks are processed within MDIS. Rather than wait for one chunk to pass through all three processing stages before starting work on the next chunk, MDIS instead moves chunks from stage to stage as each chunk is processed. This parameter determines how many chunks MDIS can manage simultaneously, counting one chunk per stage plus chunks queued between stages.

Recommendation No. Of Chunks Processed In Parallel= 5 -10

Optimal setting depends on the available memory and processing power of the machine running MDIS.

Minimum requirements = 4 GB available memory and dual-core processor. Increase the value from 5 to 10 in proportion to actual system specifications.

Example

A file with 10k records has to be imported. The chunk size is set to 2k and the number of parallel chunks to 5. When the structural transformation thread is finished with the first chunk, the chunk travels through the pipeline to be value transformed and imported. Meanwhile the structural transformation thread will break down the next 2k and prepares for travel into the pipeline. If the number of parallel chunks was set to 1, then the structural transformation thread has to wait until the

1st chunk was imported and is ready for re-use.

6.3 Slicing When importing records into an MDM repository, the MDM Server (MDS) locks the repository from all other client activities in order to safeguard the integrity of the repositorys data. To maintain the safe and timely import of data without sacrificing performance for other MDM clients, a balance must be struck between the amount of time MDS devotes to importing records and the time it allows for processing requests from other clients. To help you optimize this balance, MDM 7.1 introduces the concepts of import slices and import sleep time. Import slices divide a set of import records into smaller groups (slices) for MDS to process. By breaking up a large import job into multiple slices, other MDM clients no longer have to wait until the entire import job is complete before their requests can be answered by MDS. Now, MDS can set aside time between processing each import slice to respond to requests from other clients. During this period, called the import sleep time, MDS becomes available to all other client activities. Once the import sleep time ends, MDS starts processing the next import slice. MDS continues alternating between import slices and import sleep time until the entire import job is completed. Both the number of records to include in each import slice and the length of the import sleep time can

be customized by entering configuration parameters in the mds.ini file.


December 2010 21

Example

MDIS would send 50K records (the chuck size) to MDS for importing. On the MDS side, the 50K chunk would be sub-divided into 2K slices (the slice size). MDS is processing the import in small 2K slices, while the write lock is released in between the processing of the import slices during the entire 50K chunk import, and thus does not block other incoming requests for an extended period. This will allow other requests to be handled in

between the processing of the import slices.

The Import Slice Size Min and Import Slice Size Max parameters determine the average number of records to include in each slice. The higher the number of records in a slice, the more memory is required on the server hosting MDS and the longer the response time becomes for client requests to MDS. Too few records in a slice, however, can make the import process less efficient, as there is some overhead involved in each slice. Factors to consider when determining the optimal import slice size include:

Available memory and processing power on the server hosting MDS

Capabilities of the underlying DBMS

The complexity of the target repositorys data model. The Import Sleep Time Max parameter dictates the length of the pause which MDS takes in-between processing each import slice. The higher this value is set, the longer it will take to complete the import job. The lower this value is set, however, the more likely it is that client requests are delayed. These configuration parameters and their recommended values are summarized in the following


December 2010 22

Note

In future versions of MDM, import slice sizes will alternate between the Min and Max values based on MDS activity levels. Currently, MDS only and always uses the average

of these two values.

Note

Each of these parameters must be added manually to the server-level settings in the

mds.ini file. If these parameters are not added to mds.ini or have no values set, their default values will be used.

MDS Import Slice Size and MDIS Chunk Size

Do not confuse Import Slice parameters with the MDIS parameter, Chunk Size.

The Chunk Size value tells MDIS how many records from an aggregated file set to process at a time. Once MDIS processes a chunk of records, it immediately sends this chunk to MDS for import into the repository. When MDS receives the chunk, the Import Slice Size setting controls how many records from the chunk it should process at a time (several slices may be required to process an entire chunk). You will want to experiment with both chunk and slice settings to achieve optimal import performance for your environment.

Note

Please see note 1329424, section MDM Import Server:

Optimal MDIS Chunk and Slice size: The optimal MDIS chunk size to use in MDM is

2000 and the MDS slice size is 100.


December 2010 23

Import Slice Size and Record Checkouts

If you have configured MDM to check out records automatically upon import, MDS will check out all records in an import slice as part of processing that slice. Import Slice Size and Workflows

If you are importing records into a workflow, the workflow is not launched until all import slices from an import package have been processed. An import package refers to the complete set of records received:

From Import Manager or an Import Records API function; or From a set of import files aggregated by MDIS.

Note

The number of files which MDIS aggregates into an import package is controlled by a ports File Aggregation Count

6.4 Registry parameters

As youve just read, the import slice size and the import sleep time have an effect on how responsive

MDS will be to incoming requests from MDM clients e.g. Data Manager, Import Manager, Java API,

etc when MDS is importing data. While these settings do have a major impact on how quickly MDS

responds to client requests, Data Manager can still appear unresponsive. The reason is that Data

Manager constantly polls MDS for any changes that occur on the data. It does this in order to keep the

data being shown as up to date as possible. This is usually a wanted feature but it becomes

problematic during imports because large amounts of data are being modified. All these record

modifications cause MDS to create notifications that Data Manager retrieves and processes in order to

refresh the data. The problem is that since there are so many notifications during a mass import, Data

Manager doesnt have time to respond to user requests e.g. search, create, modify, etc unless the

import sleep time setting is set to a large value, which isnt recommended.

Instead, a registry setting was introduced in order to control how often Data Manager retrieves the

notifications. Retrieve Modifications Delay Seconds is the name of the setting and it has a default

value of 10. The smaller the value the quicker Data Manager is refreshed. The bigger the value, the

more responsive Data Manager will be during an import. The recommended value is 10 but it can be

changed if warranted during performance tests.

6.5 Bulk_Import_Silo Bulk Import Silo = True/False.

Inreases import performance by optimizing SQL access methods.

The mds.ini parameter BULK_IMPORT_SILO_LEVEL is deprecated.

Instead the following two mds.ini parameters are used:

Bulk Import Silo

Safe Silo Mode

Bulk Import Silo is a Boolean value with default value true.

During the bulk import, if this option is set to true, MDS will group multiple similar SQL statements into

one Silo and execute it later.


December 2010 24

Example

MDS could group multiple insert statements into one bulk-insert operation which will run much faster than individual ones.

When Bulk Import Silo is true, Safe Silo Mode (Boolean value) can be set to true/false and default

value is false.

If Safe Silo Mode is set to true, MDS will flush (execute) the silo whenever next SQL statement

change type (insert/update/delete).

If Safe Silo Mode is set to false, MDS will keep the Silos until the end of import and flush them

altogether. This is by far the most efficient way of bulk import.

So there're 3 level of optimization:

1. Bulk Import Silo = false: no optimization

2. Bulk Import Silo = true and Safe Silo Mode = true: partial optimization

3. Bulk Import Silo = true and Safe Silo Mode = false: full optimization

Recommendation

We recommend option 3.

Note

Please also see note 1329424.


December 2010 25

6.6 Cartesian Product

An import of a very small number of records lasts very long and consumes a lot of memory. What are

the reasons for this behavior?

Example

Two Main table records (e.g material master), several segments for each record

Sample XML Structure - and two sample records for your orientation:


December 2010 26

Based on the sample the following number of records will be imported:

In a real life example for the number of 2 records for Main Table and 5 nested structures with different

occurrences per record this will lead to the import of 1860 records!

The increase of numbers of records is caused by adding Joins & Lookups.

Due to better visibility of this topic in MDM 5.5 (in MDM 7.1 it occurs rather under the hood) well do

the explanation of the root cause based on screenshots in MDM 5.5.


December 2010 27

In MDM 5.5, the record data contained in source XML files was staged in MS Access before being

imported into MDM (right after connect to source). This staging process occurred in a virtual

workspace and consisted of decomposing the hierarchically structured data in the XML file into a

series of flat Access tables ("flatten) (users would later have to manually recompose the original data

structure in the Import Manager). From a point in time this happens during creating joins and lookups

in the Import Manager.

But right before import, we are "forced" to c onvert these hierarchy data into records ("flatten").

In MDM 7.1 XML is NOT converted into MS Access. Now the XML files are flattened in a MDM internal

data format during the import (means after the user pressed the Import button). XML is converted into

a staging area which preserves the hierarchy data. But right before import, these hierarchy data is

converted into records ("flatten"). This is necessary as any source fields regardless of how nested they

are can be mapped to any destination fields, and the only way to know the relationship between these

source fields is to convert them into records. In order to not convert XML into records, heavy

restrictions during field mapping would be needed, which allow import to directly translate the source

hierarchy into repository => staging still happens, but its no longer visible to the user.


December 2010 28

Recommendation

o Perform several imports with simplified Import Maps. Split a single complex import map into individual maps - one to load the main table, and additional maps for each qualified lookup table.

o Split Import Files into several files with less segments

One for Main table only

One for each Segment in XSD-Structure

o Whenever possible, load all the reference data (flat/ hierarchy/taxonomy tables) before loading the main table.

Recommendation applied to our real life example leads to import of 43 records

compared to 1860 records !


December 2010 29

6.7 How to import lookup main

Import Manager handles main table lookup fields (Lookup [Main]) differently than other MDM lookup

fields. Specifically, Import Manager does not display the complete set of display field values of the

records of the underlying lookup table. Instead, the values it displays for a main table lookup field are

limited by both the key mappings for the lookup table and the values in the source file.

Also, Import Manager does not automatically display the values of a Lookup [Main] destination field in

the Destination Values grid when you select the field in the Destination Fields grid. Instead, for a main

table lookup field value to appear in the Destination Values grid, all of the following conditions must be

met:

The lookup table must have key mapping enabled

The lookup field must be mapped to a source field

The source field must contain key values for the lookup table

The destination value must have a key on the current remote system

The destination values key must match a source field value

Important

Source values are added to text display fields only.

If the lookup table has no text display fields, the new record is entirely blank.

Note

At the moment lookup main cannot be used as matching field in Import Manager

Example Product lookup main Supplier

The lookup table must have key mapping enabled

The lookup field must be mapped to a source field


December 2010 30

The source field must contain key values for the lookup table

The destination value must have a key on the current remote system

The destination values key must match a source field value


December 2010 31

Example Product lookup main Supplier (multivalue)

The source field contains multi-value key values for the lookup table.

The destination values keys do not match a source field value.

Therefore set a Field Mapping Delimiter:


December 2010 32

The destination values keys are now matched to the source field values.

6.8 Importing single source field to multiple display

fields

A Main table-field with reference to Lookup Table containing two Display fields has to be imported.

Lookup table has key-mapping disabled, the initial population of lookup table using both display-fields

as matching key

During Import into Main table only incomplete information regarding lookup-values is provided (Only

ISO-Code, but no country name provided).

This means automapping of values doesnt work here.

How to perform an efficient automated import of the values?


December 2010 33

For better understanding the problem in detail:

Solution:

Enable Key-mapping for Lookup-table and use remote-key for matching during initial population.

For import in the main table mapping in Import Manager is much more comfortable.


December 2010 34

6.9 Strategies for Attribute/Value Import

From an ERP-system main table records including taxonomy and attribute information should be

included. How to perform Import Mapping for attributes as convenient as possible?

Solution 1:

o XSD-File Definition has attributes in own segment

Challenges

o Each attributes defined as an own element !!!! => XSD-Definition could get very large

o New attribute in Taxonomy leads to enhancement in XSD-Definition => Repository needs to be unloaded for updates

o The more attributes you have, the slower the Import works

o In Import Manager each attribute needs to be mapped


December 2010 35

Solution 2:

o XSD-File Definition has attributes in own segment

o Attributes are defined abstract

o Attributes are unbounded in definition


December 2010 36

Example for a XML-File with values:

What has to be done in Import Manager?


December 2010 37

Advantages of solution 2:

XSD structure is very simple structured

o Doesnt change when additional attributes need to be created

o This means also no repository maintenance required

o Import Mapping very simple

Only one field to be mapped to handle all attribute/value combinations !!!!

6.10 Excel Import

In the following business case Excel sheets are imported from different regions.

o What is the influence of the regional settings?

o How should the different decimal separators , and . be handled in the import maps?

o How are the different formats of Excel are handled in Import Manager?


December 2010 38

Example: Import Price information

Currency (General, Number)

Text - Different settings of decimal point


December 2010 39

In the current release it is not supported to parse two different settings of the decimal point

(/thousands) in source tables with the same map, since the current design is the ability to have only

one setting per map.

Recommendation

o Use only Numeric Excel formats for the prices as described in the previous slide

o If Text format should be kept, use two import maps if source is always known

Text - Text interpretated as Numeric

Why is the Price formatted as Text in Excel interpreted as NUMERIC in Import Manager?

For the interpretation of Import Manager not the format but the internal Data type Excel is using is

relevant

How to check the Excel Data Type Internal Data type:


December 2010 40

6.11 Parallel Import

You can place multiple repositories on the same server if sizing requirements are met.

Parallel imports to different ports of the same repository are not supported. However parallel imports

to different repositories on the same server are supported.

For parallel import/syndication a separate storage location for each repository should be used to

improve performance.

No additional hardware raid controller is needed. A redundant array of inexpensive (or independent)

drives (or disks) (RAID) is an umbrella term for data storage schemes that divide and/or replicate data

among multiple hard drives. They offer, depending on the scheme, increased data reliability and/or

throughput.

A RAID controller is a device which manages the physical storage units in a RAID system and

presents them to the computer as logical units.

External disk arrays are usually purchased as an integrated subsystem of RAID controllers, disk

drives, power supplies, and management software. RAID adapters for use internal to a PC or server

are sold either as embedded systems in the computer or as separate expansion cards.


December 2010 41

6.12 Transport of maps (value mapping)

Exporting an Import Map in DEV and Importing the map in QA might lead to wrong value mappings

during Import.

What is going on?

The problem exists only when Value-mapping is used. In this case MDM uses internal ids which can

be different from system to system to store the mapping info.

System behavior in MDM 5.5

If an import map was created the structural and the value mapping was based on the internal id of the

fields and the values. If the map was then transported from a Dev to a Quality system the internal ids

might differ. As a result the maps had to be reworked. In MDM 5.5 it could happen as well, that field

mapping and not only value mappings were lost.

System behavior in MDM 7.1

The structural mapping is based on the Field code now. As there are no codes for the values, the

value mapping is still based on the internal ids and not on the values itself. This means if the map was

transported from a Dev to a Quality system the internal ids might differ again and as a result the maps

have to be reworked.

Recommendation

Activate key-mapping for the respective table => the mapping will then not be performed

on values but on remote keys (The decrease of file size of the maps by switching to key

mapping is the positive side effect) or

Manually adjust value mapping


December 2010 42

6.13 Import XML (XSD.exe)

Starting with MDM 7.1 an additional file xsd.exe is required in order to process XML-Files with the

Import Manager. To enable the MDM Import Manager to generate XML schemas from XML files upon

import, the xsd.exe must reside in the same folder as the Import Manager executable.

The xsd.exe is part of the Microsoft .NET Framework SDK (Software Development Kit) 2.0, which can

be downloaded from the download center of the Microsoft web site.

6.14 How to avoid disappearing value mappings

During an Import with Import Manager value mappings are performed and saved in an already existing

map. Loading the map again shows that some value mappings disappeared from the map.

There are two ways to save a map

1) Save saves only the current mappings and disregards value mappings done in the past

2) Save update keeps previously made value mappings and adds the new ones to the map


December 2010 43

6.15 Difference between Value Exceptions and

Import Exceptions

Below are some examples in order to better understand the difference between Values Exceptions

and Import Exceptions during Imports using the Import Server.

Example for a Value exception

In an Import Map incoming text-values are mapped to an Integer-Field. The Values are ("A", "B", "C")

and they are mapped to values (1,2,3).

Incoming Value Value in Repository

A 1 B 2 C 3

A new value D is part of the import-File, but has no integer values mapped to it. In this case the

system doesnt know how to handle that value and as a consequence generates a value exception.

Example for an Import exception

In the MDM Console a field is defined as unique. There are three records stored in the repository. For

record 1 the field has the value "17"



During an import a new record has to be created, however for the unique field it has e.g. the value "18"

a value which already exists for that field in the repository. This would violate the uniqueness criteria

for the field and thus Import Server would generate an Import exception.

Import exceptions basically are raised for all other exceptions besides value exceptions and structural

exceptions.

6.16 Limitation on number of source fields

MDM 7.1 removed the limitation on number of source fields. It is now limitless where in MDM 5.5 SP6

it was limited to 255 fields because of the processing staging using MS Access (MDM 7.1 no longer

uses MS Access as a staging area).

6.17 Limitation on import file size

Please see note 1274858.

6.18 Port Sequence

Lookup [Main] fields provide the capability to relate two main table records together using a

unidirectional relationship from the origin record to the target record.


December 2010 44

Example: Material Lookup main [Suppliers]

It is necessary to import the Suppliers first so that references to them can be imported later on.

Therefore there is a need for controlling of which ports should be processed first.

Port Sequence:

Port 1: Suppliers

Port 2: Materials

Important

But now the following scenario could happen for which an (manual) exception handling has to be set up in your project.

MDIS is permanently polling the ports. At present the default behavior is to exhaust all import files in a given port prior to scanning of the next port. So MDIS starts with Port 1 but does not find any files and goes on to port 2. Before MDIS polls port 2 input files will

be placed in the corresponding ready folders:

Port 1: Suppliers

Port 2: Materials

Now MDIS would go on importing files from Port 2. But in this the import could fail

(corresponding to the map configuration) as the dependant supplier information is

missing. In this example file import of ZZ_Product_E.xml will fail as the file with the

supplier information SN239877523xxx.xml will not be imported in advance.


December 2010 45

6.19 Access, Excel, complex XML not supported on

Win64

When importing an Access, Excel, or complex (the XML includes external joins between elements )

the XML includes external joins between elements XML files through a Win64 Import Server (MDIS),

the import task fails with error message "Error: COM error 80040154 Class not registered" in the log

file.

The reason is that Microsoft has not released a 64-bit version of the JET Engine library, which the 64-

bit MDIS relies upon to process Access, Excel, and complex XML files. As a consequence, all

import tasks that require this library will fail with the error message "Error: COM error 80040154

Class not registered."

MDM version 5.5:

Customers who need to import Access, Excel, or complex XML files are suggested to switch to Import

Server Win32 version on Windows 64bit OS.

Another alternative is to eliminate the external join within the XML file / Import Map.

MDM version 7.1:

MDM Servers (MDS,MDIS and MDIS) for Windows 32bit platform are not released anymore in MDM

7.1, please see the MDM PAM (Product Availability Matrix) on the Service Market Place.

In order to import Excel/Access source files, you would need use the MDM Import Manager GUI or

alternatively, to extract/transform the data from the Excel/Access source files into XML files for

importing with the MDIS 64bit.

Please see note 1009016.

6.20 Maximum multi-record value

Performance in SAP NetWeaver MDM Data Manager can be dramatically changed by altering the

parameter Maximum multi-record value display in Record Detail pane in Data Manger configuration

options. Here low values like 100 or less should be set.


December 2010 46

7. Syndicator and MDSS

7.1 Steps during Syndication

The syndication is implemented as a separate worker thread in the Syndication Server. At the given interval, the main MDSS thread will get the list of currently running repositories and start or stop syndication tasks for each repository as necessary. For each port with a syndication map assigned, it executes the syndication query.

Task handling of MDSS:

The first step in the syndication process is the initialization of the table cache (this may take considerable amount of time depending on the number of lookup table records)

Remote keys for the main table records are generated in a separate protocol command before the first chunk is retrieved from MDS; Remote keys are generated in one step for the entire set of records selected by the query

Then the syndication port is locked; Port locks are held in MDS memory, not in the repository itself


December 2010 47

Updating record timestamps for successful syndication happens inside MDS as part of saving files to the distribution ready folder (after the file blobs are successfully transmitted, unwrapped and persisted in the ready folder, we update the timestamps)

Temp syndication files are then deleted

After the syndication is completed (succeeded or failed), the port is unlocked. The syndication chunk size is the number of source table records (and all the nested sub records

for mapped lookups) read from MDS in one transaction. Default is 100 records, but it can be configured for MDSS in its ini file (BLOCK_READ_SIZE entry), and for Syndicator in the registry.

Note

The smaller the chunk size, the more responsive MDS is for other clients' requests, especially those with data modifications. With bigger chunk sizes (such as 5000), MDS may block the repository while the chunk data is prepared inside the transaction,

although the delay highly depends on the syndication map and repository structure.

Note

The record timestamps for a particular syndication request will be updated only after all its result files (there may be many if split by records) have been successfully moved to

the ready folder.

If MDS ran out of disk space, the incomplete syndication results will be removed and the

record timestamps will not updated.


December 2010 48

The Log file is copied to MDSS memory, goes via TCP/IP to MDS and is then saved in the distribution port's log folder

After that the temp log. File is deleted

7.2 MDSS.ini Parameters MDSS has been redesigned in MDM 7.1 to improve performance by reducing port-processing delays. MDSS now assigns up to three port processing tasks to every remote system on an MDM Server. Each task is dedicated to a specific type of port on the remote system, as described below.


December 2010 49

Note

MDSS does not assign a task to a remote system if there are no corresponding ports for that task type on that remote system.

The Automated task syndicates only to Automatic|Continuous ports. It works in a continuous loop by syndicating to each continuous port on a remote system, on a repository-by-repository basis. After it finishes syndicating to all of a repositorys ports, the Automated task waits the number of seconds specified in that repositorys MDSS Auto Syndication Task Delay property before continuing. The settings can be specified in the global or repository specific section.

Auto Syndication Task Delay (seconds) = The number of seconds MDSSs automatic

syndication task waits after syndicating to all ports associated with the repository. Value copied from

the corresponding property in MDM Console.

The Scheduled task syndicates only to Automatic ports with Hourly; Daily; or Weekly values in their Port Processing Interval properties. Instead of looping, it sleeps until a syndication time is due for a port.

The Manual task syndicates only to Manual ports which have syndication job requests from a Workflow or API waiting on them. It works in a continuous loop, scanning each Manual port on a remote system for syndication requests to fulfill. Like the Automated task, it works on a repository-by-repository basis. After it finishes syndicating all of a repositorys manual jobs, the Manual task waits the number of seconds specified in that repositorys MDSS Manual Syndication Task Delay property before continuing. The settings can be specified in the global or repository specific section. Manual Syndication Task Delay (seconds) = The number of seconds MDSSs manual syndication task waits after syndicating all requests from ports associated with the repository. Value copied from the corresponding property in MDM Console.

Note

If more than one syndication request is waiting on a port, MDSS fulfills all requests on that port before it resumes scanning.

Recommendation

The number of ports for each repository should be limited due to performance reasons as far as possible.

The MDM Server Check Interval (sec) is used to monitor repositories start/stop events. At the given

interval, the main MDSS thread will get the list of currently running repositories and start or stop

syndication tasks for each repository as necessary. The default value is currently 20 seconds.

Note

The parameter Auto Syndication Task Enabled is obsolete, with MDM 7.1 auto syndication is always enabled


December 2010 50

The parameters can also be adjusted in the MDM Console on server level

or repository level

7.3 Syndication remote keys

Enabling the creation of remote keys can affect performance, for this requires an exclusive lock on the

repository.

7.4 Suppressing Initial Syndication

An Initial Syndication should be suppressed.

Important

Suppress Unchanged Records doesnt help

MDM memorizes updated systems by Remote system, not by Map or Port

Please see the following procedure:

o Use MDM Syndicator GUI and create a simple map, e.g. containing only one field

o Perform Syndication for remote-system and choose as destination your hard disk

o After syndication the remote-system is considered syndicated

o Now start Syndication server => from now on only Delta-changes will get syndicated for the remote-system.


December 2010 51

7.5 Field triggers for syndication

A record should be syndicated only, when certain fields, e.g. Description, Material Type have been

changed.

The motivation is to

o Perform Syndication only, when important fields are changed

o Avoid triggering unnecessary workflows approvals in Client system

A workflow does the trick:

o Workflow called immediately after Record Update, Record add, Record Import

o Validation checks whether specific fields have changed

One branch (called when validation returns true) triggers syndication (marks record for syndication)

One branch stops the workflow (no action in this case)


December 2010 52

For further information please see the How to guide How to Activate Field Triggers for Syndication

http://www.sdn.sap.com/irj/sdn/go/portal/prtroot/docs/library/uuid/60a2b4e4-6993-2a10-0e9a-

c6d720f1571b

7.6 Changed Records are not being syndicated MDM tracks record syndications on the remote system level, not by port or map. As such, the first time

a changed record is syndicated to a remote system, MDM considers that record to be unchanged for

all future syndications to that remote system. For this reason, if you have multiple ports or maps which

syndicate to the same remote system, only the first syndication will include the changed records.


December 2010 53

8. Data Model

8.1 Performance considerations

By design, access to tables and fields in MDM is optimized for the main table, which leads to

performance issues when other tables such as lookup tables or qualified lookup tables contain large

numbers of records.

Documents

“Good to Know” Topics for a Smooth SAP NetWeaver MDM 7.1 Implementation