IBM Tivoli Identity Manager Performance Tuning Guidepublib.boulder.ibm.com/tividd/td/ITIM/SC32-1496-06/en_US/PDF/... · 2.2 BEA WebLogic Server ... IBM DB2 monitoring information

IBM Tivoli Identity Manager Version 4.5.1 Performance Tuning Guide

Issue Date: 2007 November 30 – Seventh Edition

Publication Number: SC32-1496-06

Copyright Notice

Copyright IBM Corporation 2006. All rights reserved. May only be used pursuant to a Tivoli Systems Software License Agreement, an IBM Software License Agreement, or Addendum for Tivoli Products to IBM Customer or License Agreement. No part of this publication may be reproduced, transmitted, transcribed, stored in a retrieval system, or translated into any computer language, in any form or by any means, electronic, mechanical, magnetic, optical, chemical, manual, or otherwise, without prior written permission of IBM Corporation. IBM Corporation grants you limited permission to make hardcopy or other reproductions of any machine-readable documentation for your own use, provided that each such reproduction shall carry the IBM Corporation copyright notice. No other rights under copyright are granted without prior written permission of IBM Corporation. The document is not intended for production and is furnished “as is” without warranty of any kind. All warranties on this document are hereby disclaimed, including the warranties of merchantability and fitness for a particular purpose.

U.S. Government Users Restricted Rights -- Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corporation.

Trademarks

IBM, the IBM logo, Tivoli, the Tivoli logo, AIX, IBM DB2, IBM Tivoli Identity Manager and WebSphere Application Server are trademarks or registered trademarks of International Business Machines Corporation or Tivoli Systems Inc. in the United States, other countries, or both.

Microsoft, Windows, Windows NT, and the Windows logo are trademarks of Microsoft Corporation in the United States, other countries, or both.

UNIX is a registered trademark of The Open Group in the United States and other countries.

Java and all Java-based trademarks and logos are trademarks or registered trademarks of Sun Microsystems, Inc. in the United States and other countries.

Other company, product, and service names may be trademarks or service marks of others.

Notices

References in this publication to Tivoli Systems or IBM products, programs, or services do not imply that they will be available in all countries in which Tivoli Systems or IBM operates. Any reference to these products, programs, or services is not intended to imply that only Tivoli Systems or IBM products, programs, or services can be used. Subject to valid intellectual property or other legally protectable right of Tivoli Systems or IBM, any functionally equivalent product, program, or service can be used instead of the referenced product, program, or service. The evaluation and verification of operation in conjunction with other products, except those expressly designated by Tivoli Systems or IBM, are the responsibility of the user. Tivoli Systems or IBM may have patents or pending patent applications covering subject matter in this document. The furnishing of this document does not give you any license to these patents. You can send license inquiries, in writing, to the IBM Director of Licensing, IBM Corporation, North Castle Drive, Armonk, New York 10504-1785, U.S.A.

IBM Tivoli Identity Manager Performance Tuning Guide Page 1

Contents Contents ...................................................................................................................................................1 About this guide ........................................................................................................................................4

Who should use this guide ...................................................................................................................4 What’s new ...........................................................................................................................................4

Seventh edition ................................................................................................................................4 Sixth edition......................................................................................................................................4 Fifth edition.......................................................................................................................................4 Fourth edition ...................................................................................................................................4 Third edition .....................................................................................................................................5 Second edition .................................................................................................................................5

1 Introduction...........................................................................................................................................6 1.1 Vital tunings .................................................................................................................................6 1.2 Initial tunings................................................................................................................................6 1.3 Resource allocation .....................................................................................................................7

1.3.1 Memory....................................................................................................................................7 1.3.2 CPU .........................................................................................................................................7 1.3.3 Disk space...............................................................................................................................8

2 J2EE application servers ......................................................................................................................9 2.1 IBM WebSphere Application Server ............................................................................................9

2.1.1 Java virtual machine (JVM) size..............................................................................................9 2.1.2 Java virtual machine (JVM) options – Sun Solaris................................................................10 2.1.3 JDBC pool sizes and timeouts ..............................................................................................12

2.1.3.1 Pool sizes .....................................................................................................................12 2.1.3.2 Timeouts .......................................................................................................................13 2.1.3.3 Statement cache size ...................................................................................................13 2.1.3.4 Setting the JDBC values...............................................................................................13

2.1.4 JMS pool size and timeouts ..................................................................................................14 2.1.4.1 Pool size .......................................................................................................................14 2.1.4.2 Timeouts .......................................................................................................................14 2.1.4.3 Setting the JMS values.................................................................................................14

2.1.5 Client inactivity timeouts........................................................................................................15 2.1.6 IBM HTTP Server ..................................................................................................................15

2.1.6.4 Connections..................................................................................................................15 2.1.7 IBM WebSphere MQ .............................................................................................................16

2.2 BEA WebLogic Server...............................................................................................................16 3 IBM Tivoli Identity Manager application..............................................................................................17

3.1 Threads......................................................................................................................................17 3.2 Intel multi-processor systems ....................................................................................................18 3.3 Recycle bin ................................................................................................................................18

3.3.1 Disabling the recycle bin .......................................................................................................18 3.3.2 Emptying the recycle bin .......................................................................................................19

3.4 Reconciliations...........................................................................................................................19 3.4.1 Limiting attributes returned from the adapter ........................................................................19 3.4.2 Limiting the attributes evaluated............................................................................................19 3.4.3 Maximum duration.................................................................................................................20

4 IBM Tivoli Identity Manager adapters.................................................................................................21 4.1 Microsoft Active Directory..........................................................................................................21

5 Database servers ...............................................................................................................................22 5.1 IBM DB2 ....................................................................................................................................22

5.1.1 Quick guide for setting the IBM DB2 tuning parameters.......................................................22 5.1.2 Buffer pools ...........................................................................................................................22 5.1.3 Connections...........................................................................................................................23 5.1.4 Table spaces .........................................................................................................................23

5.1.4.5 Adding additional table space containers .....................................................................23

Page 2 IBM Tivoli Identity Manager Performance Tuning Guide

5.1.4.6 Enabling autoresize ......................................................................................................24 5.1.5 Transaction logs ....................................................................................................................24 5.1.6 Database application heaps ..................................................................................................25 5.1.7 Indexing .................................................................................................................................25 5.1.8 Runstats ................................................................................................................................26 5.1.9 Maximum open files ..............................................................................................................27 5.1.10 Lock tuning ............................................................................................................................27

5.1.10.7 Lock type ..................................................................................................................27 5.1.10.8 Lock list and maximum locks....................................................................................27 5.1.10.9 Lock timeout .............................................................................................................27

5.2 Microsoft SQL Server ................................................................................................................28 5.3 Oracle Database........................................................................................................................28

5.3.1 Server configuration ..............................................................................................................28 5.3.2 Table spaces .........................................................................................................................28

5.3.2.10 Spreading database data .........................................................................................28 5.3.2.11 Adding additional table space datafiles....................................................................29

5.3.3 Indexing .................................................................................................................................29 5.3.4 Updating statistics .................................................................................................................29

6 Directory servers ................................................................................................................................31 6.1 IBM Tivoli Directory Server........................................................................................................31

6.1.1 Quick guide for setting the IBM Tivoli Directory Server tuning parameters ..........................31 6.1.2 Cache sizes ...........................................................................................................................31 6.1.3 Buffer pools ...........................................................................................................................32 6.1.4 Connections...........................................................................................................................33 6.1.5 Transaction logs ....................................................................................................................34 6.1.6 Database statement heap .....................................................................................................34 6.1.7 Database application heaps ..................................................................................................35 6.1.8 Package cache size ..............................................................................................................35 6.1.9 Database sort heaps .............................................................................................................36 6.1.10 System limits .........................................................................................................................36 6.1.11 Indexing .................................................................................................................................37 6.1.12 Runstats ................................................................................................................................37

6.2 Sun ONE Directory Server ........................................................................................................38 6.2.1 All IDs Threshold Value.........................................................................................................38 6.2.2 Indexing .................................................................................................................................39 6.2.3 Cache sizes ...........................................................................................................................40

6.2.3.12 Sun ONE Directory Server V5.1...............................................................................40 6.2.3.13 Sun ONE Directory Server V5.2...............................................................................40

7 Operating systems..............................................................................................................................42 7.1 Red Hat......................................................................................................................................42

8 Best practices .....................................................................................................................................43 9 Regular maintenance .........................................................................................................................44 10 Troubleshooting..................................................................................................................................45

10.1 Log files .....................................................................................................................................45 10.2 Identifying bottlenecks ...............................................................................................................45 10.3 Specific problems ......................................................................................................................46

10.3.1 IBM Tivoli Identity Manager application ................................................................................46 10.3.2 IBM WebSphere MQ .............................................................................................................46 10.3.3 IBM Tivoli Directory Server....................................................................................................47 10.3.4 Sun ONE Directory Server ....................................................................................................48

11 IBM DB2 performance monitoring ......................................................................................................50 11.1 Enable monitoring......................................................................................................................50 11.2 Snapshots..................................................................................................................................50 11.3 Statement monitor .....................................................................................................................50 11.4 Buffer pool hit ratio.....................................................................................................................51

12 Other resources..................................................................................................................................52 13 Appendix A – scripts and files ............................................................................................................53


13.1 perftune_runstats.sh..................................................................................................................53 13.2 ids_indexes.ldif ..........................................................................................................................53


About this guide This guide identifies some ways to tune your IBM® Tivoli Identity Manager™ system to improve performance.

Who should use this guide Use this guide if you are responsible for installing or maintaining an IBM Tivoli Identity Manager system. The following competencies are recommended:

• Familiarity with basic database and directory design principles.

• General knowledge of system resource management (memory, disk, and processor) including your operating system’s memory model. Without knowledge of other resource requirements on the system it is possible to negatively impact performance by over tuning.

• Understanding of how to configure and administer your directory and database servers. You may need to have your local database administrator or directory administrator perform these tunings for you.

What’s new Seventh edition

New tuning: Disabling the recycle bin

Updated: DB2 lock timeout values

Sixth edition

New tuning: Minimum cardinality tunings for the IBM Tivoli Identity Manager IBM DB2 database Information on the IBM DB2 autoresize option available for tables paces Information about the IBM Tivoli Identity Manager recycle bin Information on improving IBM Tivoli Identity Manager reconciliations Section on IBM Tivoli Identity Manager adapters Section on regular maintenance activities

Updated: Clarified information regarding Java™ virtual machine (JVM) tuning parameters when using the Sun JVM Information on DB2_RR_TO_RS locking for IBM DB2 V8 Clarified cache sizings for Sun ONE Directory Server V5.2 IBM DB2 monitoring information Best practices section Oracle section with additional tunings and recommendations

Fifth edition

New tuning: Oracle indexes and table space distributions

Updated: Mechanism for adding IBM Tivoli Directory Server™ indexes IBM Tivoli Directory Server cache size recommendations Best practices section to include recommended provisioning policy sizes

Fourth edition

New tuning: IBM WebSphere Application Server client inactivity timeout values


IBM Tivoli Directory Server pckcachesz parameter tuning IBM HTTP Server ThreadsPerChild (Windows) MaxClients (Unix) setting

Updated: IBM Tivoli Directory Server indexing to include the O attribute IBM Tivoli Identity Manager database table space size and transaction tunings JVM minimum and maximum size recommendations IBM Tivoli Directory Server ACL and filter cache settings IBM Tivoli Directory Server STMTHEAP setting Sun ONE Directory Server All IDs Threshold value Added v3.user.at method of adding IBM Tivoli Directory Server indexes

New feature: Oracle database tuning

Third edition

New tuning: Additional IBM DB2® lock tuning information IBM DB2 maximum number of open files IBM WebSphere® Application Server on Sun Solaris™ IBM WebSphere MQ

Updated: Changed the IBM Tivoli Directory Server cache tunings

New feature: Best Practices section Troubleshooting section

Second edition

New tuning: Tunings for IBM HTTP Server included with the IBM WebSphere Application Server Tunings for Sun ONE Directory Server Information on the IBM Tivoli Identity Manager application threading IBM DB2 runstats tuning information for the IBM Tivoli Directory Server database

Updated: The perftune_runstats.sh script includes the new index tuning for the IBM Tivoli Directory Server database

Organization: Reorganized the IBM WebSphere Application Server sections to make applying tunings easier Changed the Quick guide sections to make them less verbose


1 Introduction

The IBM Tivoli Identity Manager product addresses the complex problem of identity management. Due to the complexity of the problem, it can be challenging to optimize the use of resources by IBM Tivoli Identity Manager – that is, to tune. This tuning guide provides a system administrator with the information needed to tune the application for your environment. Other individuals (such as IBM DB2 or IBM Tivoli Directory Server administrators) in your organization might offer differing advice. In our experience, your system administrators know your environment better, and their advice may be more accurate for your environment than this tuning document.

The Tivoli Identity Manager product can be divided into four major components: Java™ 2 Platform, Enterprise Edition (J2EE) application servers, the Tivoli Identity Manager application, database servers, and directory servers. Some of these components can be divided further such as workflow (WF) and user interface (UI) components for the Tivoli Identity Manager servers. We will address each of these separately in this document.

The Tivoli Identity Manager server can be installed in three different modes: single server, clustered servers, and functional cluster servers. For performance reasons single servers and clustered servers are the same – each server has both a WF and UI component on one physical machine. A clustered environment can be considered a group of single servers with regard to tuning. A functional cluster server is different than a regular cluster because in a functional cluster each component (WF/UI) is on a separate server. In a functional cluster environment you might have any multiple of servers.

This document is a working document, as more information is gathered settings may be added, removed or changed in future releases. It is recommended that you check the IBM web site for the most recent version. To find the most recent version, go to http://www.ibm.com/support/us. Type “ITIM Tuning Guide” in the search box under Search technical support, and click Search.

IBM Tivoli Identity Manager configurations vary significantly by platform, operating system, middleware applications, and hardware being used. You must tailor your configuration settings to your deployment. Configurations and settings used in this document include AIX®, Windows®, and Sun Solaris™ in single server and cluster configurations.

1.1 Vital tunings There are several thousand different parameters that you can modify to tune WebSphere Application Server, the IBM Tivoli Identity Manager product, directory servers, and database servers. This tuning guide discusses a small subset of these parameters that have proven effective during performance testing.

If you are setting up an acceptance or production environment, read each section and perform the applicable tunings for your systems. If you are setting up a test environment and want to get started as quickly as possible, focus on these areas:

• IBM WebSphere Application Server - Java virtual machine (JVM) size

• IBM DB2 - Buffer pools

• IBM Tivoli Directory Server – Indexing or Sun ONE Directory Server - Indexing

• IBM DB2 - Runstats or Oracle Database - Updating statistics Note: The database statistics tunings are a vital part of the IBM Tivoli Identity Manager product performance.

1.2 Initial tunings Most of these tunings can be implemented in a newly deployed environment or an environment that is already deployed. When tuning IBM DB2 in a newly deployed environment, it is important to prime your database statistics using the IBM DB2 runstats command and the corresponding manual cardinality

http://www.ibm.com/support/us


tunings. See the IBM DB2 - Runstats section for more information. Failing to prime the database in this manner can result in poor performance or transaction roll backs.

1.3 Resource allocation Tuning values are more complex to manage when more than one middleware component is running on a given system; for example, having the IBM Tivoli Identity Manager server, IBM DB2, and IBM Tivoli Directory Server all on the same server. Regardless of configuration, it is important to calibrate the following resources so that they are not over-allocated.

1.3.1 Memory All middleware components allow you to adjust how much memory they will use. When calculating how to allocate memory to middleware components, keep these considerations in mind:

• Configuring middleware memory settings too high can result in the operating system swapping memory out to disk if the physical memory is exceeded. This will result in extremely poor performance and should be avoided. After setting up or changing the memory values for the middleware, monitor the memory and swap space used to ensure that nothing is being swapped out to disk. If it is, adjust your memory settings to correct

• 32-bit processes can only allocate 2 GB (on AIX) to 4 GB (on Solaris) of RAM. If a 32-bit process is configured to allocate more than the OS-specific limit on process memory, the application (such as IBM Tivoli Directory Server) could halt or unexpectedly fail

• IBM Tivoli Directory Server has internal caches that contribute to the size of the IBM Tivoli Directory Server process. Ensure that the size of the IBM Tivoli Directory Server process does not exceed 2 GB on 32-bit platforms such as Windows. When this 2 GB limit is reached, the IBM Tivoli Directory Server will refuse new connections and fail. The entry cache size limit determines the number of entries in the cache, not the size of the cache. The size of each cache entry will vary based on the IBM Tivoli Identity Manager product configuration and any extensions to the base IBM Tivoli Directory Server schema. The default cache values may exceed the 2 GB limit in rare cases.

• Although the buffer pools account for a large amount of the memory used by IBM DB2, the application control heaps, the sort heaps, and the statement heaps also use memory. Do not overlook these sizes when computing how much memory to allocate to IBM DB2. See the IBM DB2 section.

• A large part of the WebSphere Application Server’s memory usage is the JVM size. However, the size of the JVM does not set an upper bound on the amount of memory that the WebSphere Application Server may use. See the IBM WebSphere Application Server section.

• Operating system limits may prohibit processes for accessing all the available memory. Confirm the appropriate ulimit values for your system to ensure they do not artificially limit the amount of memory available.

1.3.2 CPU All the components of the Tivoli Identity Manager product (Tivoli Identity Manager application, J2EE application server, database server, and directory server) are CPU intensive. This document groups the J2EE application server and the Tivoli Identity Manager application together because it is difficult to isolate their CPU usage. Keep in mind the following considerations:

• Tivoli Identity Manager product Version 4.4 and Version 4.5 workflow processes are limited to a single thread for performance reasons. These versions are best suited for machines with fast processors rather than machines with multiple, slower processors. The Tivoli Identity Manager product Version 4.5.1 can take better advantage of multi-processor machines.


• Some Tivoli Identity Manager server processes (such as some provisioning calculations) are single threaded and will be isolated to a single processor regardless of what version of Tivoli Identity Manager server you are using.

• Both IBM Tivoli Directory Server and IBM DB2 are multi-threaded (multi-process in DB2’s case) applications and will perform best on a multi-processor server.

• Even in a well tuned environment the system, bottleneck may vary between being CPU bound on the Tivoli Identity Manager server, on the directory server, and on the database server. For this reason, the Tivoli Identity Manager server, the directory server, and the database server should all be on separate machines. If that is not possible, put the database and directory server on the same server with a high performance disk configuration using multiple disks and a high performance RAID configuration to provide fast read and write capacity.

1.3.3 Disk space Each of the middleware components use different amounts of disk space for various purposes.

• The J2EE application server and the Tivoli Identity Manager application use disk space beyond their installation size because of log files (such as the itim.log file) and WebSphere MQ queues. Adjust the number of archives and size of the itim.log file in the enRoleLogging.properties file. Make sure that WebSphere MQ has enough disk space for its processing logs (not error logs) to grow. The Tivoli Identity Manager server pushes many entries onto the queues during large provisioning changes, causing the queues to grow.

• IBM Tivoli Directory Server uses disk space both from the IBM Tivoli Directory Server process (log files like ibmslapd.log) and the DB2 database. IBM Tivoli Directory Server uses system managed space (SMS) table spaces that allow the system to manage the amount of disk space used. SMS table spaces do not allow the specification of an upper bound so it is important to monitor the amount of disk space used to ensure that the drive does not become full.

• In addition to the table spaces for the database data, DB2 uses disk space for the transaction logs. When configuring the transaction logs (discussed in the IBM Tivoli Directory Server – Transaction logs and IBM DB2 – Transaction logs sections) ensure that you have enough disk space for the log files.

• The IBM Tivoli Identity Manager DB2 database uses directory-managed space (DMS) table spaces that require pre-allocating disk space for the database to use. If the table space becomes full, IBM DB2 will be unable to continue and will return an error in the itim.log file. If this occurs, add more table space containers or enable autoresize (available in later versions of IBM DB2). See the IBM DB2 – Table spaces section for additional information.

• The default maximum sizes of the IBM Tivoli Identity Manager Oracle datafiles are 1 GB. If the datafile becomes full Oracle will return an error and all workflow activities will return an error. If this occurs, add more datafiles or increase the maximum size of the existing datafiles. See the Oracle Database – Table spaces section for additional information.

• As a general rule, for every 160,000 transactions that occur, 1 GB of table space is needed to archive the historical transaction information. A policy change that makes modifications to 5,000 users is 5,000 transactions from an IBM Tivoli Identity Manager perspective.


2 J2EE application servers

Regardless of the installation type (single, regular cluster, or functional cluster), the Tivoli Identity Manager server can be thought of as two components: the J2EE application server running the application (WebSphere Application Server or BEA WebLogic® Server) and the Tivoli Identity Manager application itself. Both components need to be tuned for your environment.

2.1 IBM WebSphere Application Server WebSphere Application Server allows you to tune a variety of settings for your environment. This document discusses the Java virtual machine (JVM) size, the minimum and maximum Java database connectivity (JDBC) and Java Messaging Service (JMS) pool sizes, and the various timeouts.

2.1.1 Java virtual machine (JVM) size By default, WebSphere Application Server sets the maximum JVM size to 256 MB. This value is too small for the IBM Tivoli Identity Manager product to run beyond a basic concept test and should be increased to a minimum of 1 GB. If your server has adequate available RAM increase this value to as much as 1.5 GB. For large reconciliations or role and policy evaluations, the default values will not be enough memory to complete these tasks.

Setting the maximum memory value too high can cause memory allocation problems in Java when the 2 GB limit is reached for 32-bit processes. The maximum JVM size used in this value is not the actual maximum allocated size of the Java heap – as much as 15% is allocated to a portion of the heap for the system’s use. IBM recommends that you not use a value higher than 1.5 GB even if your system has the available memory.

Do not set the JVM heap size to be larger than the physical RAM. The WebSphere Application Server suffers significant performance degradation if the operating system swaps out the JVM to swap space. Consequences of this include very slow user interface (UI) performance, transaction roll backs, timeouts, and high disk utilization.

The WebSphere Application Server sets the initial JVM size to 0 MB and IBM recommends using a value of 256 MB. While previous JVM versions have worked well with the initial JVM size set the same as the maximum JVM size, this is not recommended with version 1.4.2 of the JVM (the version embedded in WebSphere Application Server V5.1) as it can cause memory fragmentation resulting in poor performance.

Determining the values

initial_jvm_heap_size – The initial size of the JVM heap in megabytes. Recommended value: 256 MB.

max_jvm_heap_size – The maximum size of the JVM heap in megabytes. Recommended value: 1 GB to 1.5 GB.

Setting the values

1) Open the Administration Console.

2) Expand the Servers list in the navigation pane.

3) Select Application Servers in the navigation pane.

4) Select the server to manage.

5) Select Process Definition from the Additional Properties pane at the bottom.

6) Select Java Virtual Machine from the Additional Properties pane at the bottom.

7) Set the Initial Heap Size to initial_jvm_heap_size.


8) Set the Maximum Heap Size to max_jvm_heap_size.

9) Repeat this procedure for each Tivoli Identity Manager server.

2.1.2 Java virtual machine (JVM) options – Sun Solaris Attention: This section applies specifically to WebSphere Application Server on Sun Solaris (including later fixpack releases) that use the Sun JVM versions 1.3.x and 1.4.x. This section does not apply to WebSphere Application Server for AIX, Windows, or Linux. WebSphere Application Server for Sun Solaris ships with Sun’s implementation of Java whereas WebSphere Application Server for other platforms ship with IBM’s implementation. IBM’s Java and Sun’s Java have two different garbage collection algorithms and need to be tuned differently. If you are using WebSphere Application Server on AIX, Windows, or Linux, skip this section.

Before proceeding, determine what JVM version and implementation you are using by using the following command:

<WebSphere Home Directory>/AppServer/java/bin/java –version

The output will look something like the following sample: java version "1.3.1_08" Java(TM) 2 Runtime Environment, Standard Edition (build 1.3.1_08-b03) Java HotSpot(TM) Client VM (build 1.3.1_08-b03, mixed mode)

If the version is not 1.3.x or 1.4.x, or if the output includes the string “IBM”, skip this section, because it does not apply to your JVM.

Sun’s 1.3.x and 1.4.x JVMs on Sun Solaris supports two different JVM types: client and server. The client JVM is designed to have a faster startup time and is tuned for shorter-running applications than the server JVM. The server JVM is designed for longer-running applications and therefore sets different initial values to some JVM tunings. By default the WebSphere Application Server uses the client JVM. Tivoli Identity Manager runs better under the server JVM.

These Sun JVMs use a generational garbage collection algorithm. How this algorithm works is outside the scope of this document, but we will give a short summary. See the Other resources section at the end of this document for links to more information.

The following is a simplified view of how the generational garbage collection algorithm works. The JVM divides the Java heap size into two major components: the young generation, which includes Eden and two Survivor pools, and the old generation (sometimes called the tenured generation). When objects are allocated inside Tivoli Identity Manager, they are created inside Eden. After Eden becomes full, a partial garbage collection occurs and any objects that are still being used (live objects) are moved to the first Survivor pool. Eden is cleared and object allocation begins again inside Eden. After Eden becomes full again, any live objects inside Eden and the first Survivor pool are moved to the second Survivor pool. Eden is cleared and object allocation begins again inside Eden. This process repeats with the objects being switched between the two survivor pools until objects die or are eventually tenured to the old generation. When there is no more space inside Eden or the Survivor pools, all remaining live objects are moved to the old generation. Eden is cleared and object allocation begins again. When the old generation becomes full, a full garbage collection occurs. Every object within the heap is analyzed and heap compaction occurs within the old generation.

When garbage collections, both partial and full, occur everything within the JVM stops. This causes small pauses within Tivoli Identity Manager during garbage collections. Partial garbage collections finish very quickly, usually in less than half a second. Full garbage collections take longer to compact the entire heap and can take 10 seconds or more to finish, depending on the size of your heap. Because of this, it is important to minimize the number of garbage collections, particularly full garbage collections. We do this by adjusting the size of the Java heap, the new generation, and the survivor pools.

A diagram illustrating the JVM heap layout:

Eden Tenured GenerationSurvivor pools

max_new_sizemax_jvm_size

new_ratiosurvivor_ratio


jvm_type – The type of JVM to use, either -client or -server. Recommended value: -server

max_new_size – The maximum size of the new generation within the JVM in megabytes. Set this to 1/3 of your maximum heap size.

new_ratio – The ratio of the new generation to the old generation. Set this to 2, thereby setting the size of your new generation to 1/3 of your heap size.

survivor_ratio – The ratio of a single survivor pool to Eden. Set this to 3, thereby setting the size of each survivor pool to 1/5 of your new generation (note this is not 1/4 because there are two survivor pools).

target_survivor_ratio – A target for the amount of survivor space to be used after a garbage collection. This value is used by the JVM to calculate for how many iterations an object will stay in the survivor pools before being tenured to the old generation. Set this to 75 to allow more time for objects to die in the survivor pool before being tenured.

With a total JVM heap size of 512 MB, the recommendations above would yield the following heap configuration:

Eden Tenured GenerationSurvivorpools

170 MB

512 MB

102 MB 34 MB

342 MB

Calculating the values

argument_string = jvm_type -XX:MaxNewSize=max_new_sizem -XX:NewRatio=new_ratio -XX:SurvivorRatio=survivor_ratio -XX:TargetSurvivorRatio=target_survivor_ratio

Tip: Ensure the MaxNewSize value has a trailing ‘m’ specifying that the value is in megabytes.

Example: For a JVM size of 512 MB, use the following argument string:

argument_string = -server -XX:MaxNewSize=170m -XX:NewRatio=2 -XX:SurvivorRatio=3 -XX:TargetSurvivorRatio=75

Setting the values







5) Select Process Definition from the Additional Properties pane at the bottom.

6) Select Java Virtual Machine from the Additional Properties pane at the bottom.

7) Append argument_string to Generic JVM arguments


Attention: Some older versions of the Sun 1.3.1 JVM will crash when using the –server option with WebSphere Application Server V4.0 and V5.0. See this InfoCenter topic for more information:

http://publib.boulder.ibm.com/infocenter/wasinfo/v4r0/index.jsp?topic=/com.ibm.support.was40.doc/html/Java_SDK/swg21194777.html

2.1.3 JDBC pool sizes and timeouts The Tivoli Identity Manager application uses JDBC connections to connect to the database backend during the processing of workflow items (such as changing a password, creating a new account, or loading a DSML feed). The JDBC pool size and timeouts need to be updated for your system.

2.1.3.1 Pool sizes

The minimum and maximum values for the JDBC connections might need to be increased depending on the function of the Tivoli Identity Manager installation.


Find the value in the following table that matches your Tivoli Identity Manager installation configuration.

Functional Cluster (per each component) Parameter Single Server or

Regular Cluster UI Component WF Component

min_pool_size 15 5 15

max_pool_size 60 10 50

The difference in the UI and WF component values is because most transactions in the UI (such as password changes) are delegated to a WF component, which makes the actual JDBC connection. If the Tivoli Identity Manager server is unable to keep your database server busy under heavy load, consider increasing the maximum pool size.

Your database needs to be able to accept the total number of connections plus 5 more for maintenance purposes.

Example: You have a functional cluster with 2 workflows (WF) and 2 user interfaces (UI). Your WF component will need 50 * 2 = 100 connections and your UI component will need 10 * 2 = 20 possible connections. You also need to include the additional 5 connections for administrative purposes, which means you need to set your database connections to 100 + 20 + 5 = 125.

Example: If you have a regular cluster with 3 nodes (each Tivoli Identity Manager node includes one WF and one UI) you need (50 + 10) * 3 + 5 = 185 database connections.

Refer to the database connections section for the appropriate database for more information on setting the maximum number of connections.

Setting the values

See the Setting the JDBC values section to set the pool size, timeouts, and statement cache size together.




2.1.3.2 Timeouts

The timeouts for the Tivoli Identity Manager application need to be adjusted from their initial settings. The following values work well for most environments. If you begin receiving a large number of timeouts, consider increasing the appropriate timeout value.


jdbc_connection_timeout = The number of seconds before the JDBC connection is closed due to inactivity. Recommended value: 60.

jdbc_idle_timeout = The number of seconds an allocated JDBC connection can remain in the pool before being freed. Recommended value: 90.

jdbc_orphan_timeout = The number of seconds an application can hold an unused JDBC connection before being freed. Recommended value: 60.

Setting the values


2.1.3.3 Statement cache size

The Tivoli Identity Manager application uses prepared statements when communicating with the database server. Adjust this value to yield a better tradeoff between performance and resource utilization.


jdbc_statement_cache_size = The maximum number of prepared statements to cache. Recommended value: 60.

Setting the values


2.1.3.4 Setting the JDBC values


2) Expand the Resources list in the navigation pane.

3) Select JDBC Providers in the navigation pane.

4) Single server: Select Server and click Apply. Cluster server: Select the Node and Server (using Browse Nodes and Browse Servers) and click Apply.

5) Select ITIM JDBC Provider (XA).

6) Select Data Sources (Version 4) from the Additional Properties pane at the bottom.

7) Select ITIM Data Source.

8) Select Connection Pool from the Additional Properties pane at the bottom.

1) Set the Minimum Pool Size to min_pool_size.

9) Set the Maximum Pool Size to max_pool_size.

10) Set the Connection Timeout to jdbc_connection_timeout (recommended value: 60).

11) Set the Idle Timeout to jdbc_idle_timeout (recommended value: 90).


12) Set the Orphan Timeout to jdbc_orphan_timeout (recommended value: 60).

13) Set the Statement Cache Size to jdbc_statement_cache_size (recommended value: 60).


2.1.4 JMS pool size and timeouts The Tivoli Identity Manager application uses JMS queues to manage workflow items. The JMS pool size and timeouts need to be updated for your system.

2.1.4.1 Pool size

The connection pool size for the Tivoli Identity Manager application may need to be adjusted for your system if you increased the maximum number of threads for the Tivoli Identity Manager messaging queues. See the IBM Tivoli Identity Manager application - Threads section for more information.


max_pool_size = The maximum number of connections available in the queue pool. Default value: 50.

Setting the values

See the Setting the JMS values section to set the pool size and timeouts together.

2.1.4.2 Timeouts

The timeouts for the Tivoli Identity Manager application need to be adjusted from their initial settings. The following values work well for most environments. If you begin receiving a large number of timeouts, consider increasing the appropriate timeout value.


queue_connection_timeout = The number of seconds before the queue connection is closed due to inactivity. Recommended value: 300.

queue_reap_time = The number of seconds between runs of the queue pool maintenance thread. Recommended value: 300.

Setting the values

See the Setting the JMS values section to set the pool size and timeouts together.

2.1.4.3 Setting the JMS values


2) Expand the Resources list in the navigation pane.

3) Select WebSphere JMS Providers in the navigation pane.

4) Single server: Select Server and click Apply. Cluster server: Select the Node and Server (using Browse Nodes and Browse Servers) and click Apply.

5) Select WebSphere Queue Connection Factories.

6) Select ITIM Queue Connection Factory.

7) Select Connection Pool from the Additional Properties pane at the bottom.


8) Set the Max Connections to max_pool_size.

9) Set the Connection Timeout to queue_connection_timeout.

10) Set the Reap Time to queue_reap_time.


2.1.5 Client inactivity timeouts IBM Tivoli Identity Manager activities that run for long periods can be timed out inside WebSphere and abort running operations. A primary example of this is a long running reconciliation process. For customers that have long running reconciliations and experience transaction rollback messages in the itim.log, increasing this value will allow these requests to complete.


inactivity_timeout = The number of seconds before a client transaction request is rolled back due to inactivity. Recommended value: 3600.

Setting the values





5) Select Transaction Service from the Additional Properties pane at the bottom.

6) Set the Client Inactivity Timeout to inactivity_timeout.


2.1.6 IBM HTTP Server The WebSphere Application Server uses the IBM HTTP Server as a front-end server in a single-server installation and as a load balancer between nodes in a cluster installation. The default configuration parameters for the IBM HTTP Server work well for small and medium configurations but may need to be increased in environments with a large number of concurrent users.

2.1.6.4 Connections

The IBM HTTP Server can be configured to limit the number of connections that it will accept at one time. This value may be too small if the servers experience a large number of concurrent users. Connections are maintained to the server until a user selects logout (which doesn’t always happen) or until after 10 minutes of inactivity when the connection is closed.

The IBM HTTP Server supports the HTTP/1.1 KeepAlive request that allows a client to make multiple HTTP requests through a single persistent connection. There is a limit on the number of KeepAlive requests that a single connection will accept. After this limit is reached, the connection is closed and another connection must be established. The default value may be too small for some external provisioning processes such as a Java Naming and Directory Interface (JNDI) feed.


ibmhttp_home = The home directory of the IBM HTTP Server, such as /usr/IBMHttpServer.

max_connections = The maximum number of connections that can be made to the HTTP server at one time. This should be set to the maximum number of concurrent users you expect on your


system. Recommended value: 200. This may be increased to as high as 600 for customers with a large number of concurrent users.

max_keepalive = The maximum number of requests allowed during a persistent connection. Recommended value: 0 (unlimited).

Setting the values

Tip: The MaxClients parameter on Windows is called ThreadsPerChild.

Edit the ibmhttp_home/conf/httpd.conf file and update the following entries: MaxClients max_connections MaxKeepAliveRequests max_keepalive

Stop and restart the IBM HTTP Server for these changes to take effect.

2.1.7 IBM WebSphere MQ The WebSphere Application Server uses an embedded copy of the IBM MQSeries messaging service to manage Java Message Service (JMS) messages. The IBM Tivoli Identity Manager server configures several persistent queues (the exact name and number can vary slightly from version to version). The IBM Tivoli Identity Manager server uses these queues during various provisioning actions. During large provisioning policies it is possible to reach the limit on the number of uncommitted messages, so this value should be increased.


queue_manager_name = The name of the queue manager, such as WAS_vanexel_jmsserver. You can find a list of valid queue manager names by running dspmq.

max_uncommitted_messages = The maximum number of uncommitted messages. Default value: 10000. Recommended value: 50000.

Setting the values

1) Start the MQ Script interpreter: runmqsc queue_manager_name

2) Alter the maximum uncommitted messages size: alter qmgr maxumsgs(max_uncommitted_messages)

3) Confirm the change: display qmgr maxumsgs

4) Exit the MQ Script interpreter: end

2.2 BEA WebLogic Server Steps to tune BEA WebLogic Server will be included here when those settings and their values are determined.


3 IBM Tivoli Identity Manager application

The IBM Tivoli Identity Manager application includes several configuration files that provide an area for tuning various parts of the application’s performance. These are in the data/ directory under the Tivoli Identity Manager product home directory.

3.1 Threads The IBM Tivoli Identity Manager product is a multi-threaded application and in most cases is able to take advantage of multiple processors. In the Tivoli Identity Manager product Version 4.4 and Version 4.5, the threads that process workflow items were set to one for performance reasons. The Tivoli Identity Manager product Version 4.5.1 is better able to take advantage of multiple processors and the number of workflow threads should be set to more than one.

In most cases the default number of threads are adequate. If during a large provisioning policy or other large workload run, you find your database server and the Tivoli Identity Manager server are not fully utilized, you might try increasing the maximum number of threads.

There are seven thread pools: adhocSyncQueue, workflowQueue, workflowPendingQueue, workflowAbortQueue, remoteServicesQueue, remotePendingQueue, and mailServicesQueue. Only two of these, workflowQueue and workflowPendingQueue, should need adjustment. The configuration file allows adjustments to both the maximum and the minimum number of threads. However, the minimum number of threads should not be changed as the system quickly increases the number of threads beyond this value.

The workflowQueue is used to process workflow items and should have the largest allocation of threads. The workflowPendingQueue is used to start certain provisioning activities and should have fewer threads than the workflowQueue. Use a 2:1 ratio between the number of workflowQueue threads and workflowPendingQueue threads.


itim_home = The Tivoli Identity Manager product home directory, such as /itim45.

max_workflow_threads = The maximum number of threads to use processing workflow items. Default value: 10.

max_workflow_pending_threads = The maximum number of threads to use when processing pending workflow items. Default value: 5.

Tip: When increasing the maximum number of threads, you might need to increase the size of the JMS connection pool within your J2EE application server. The size of the JMS connection pool should be greater than the sum of the maximum values for all messaging threads. Following this guideline ensures an adequate number of JMS connections in the event that the maximum number of threads for all Tivoli Identity Manager messaging queues are active at one time.

Setting the values

Tip: The lines below are split for readability purposes; they should be all in one line in the enRole.properties file.

Edit the itim_home/data/enRole.properties file and update the following lines: enrole.messaging.workflowQueue.attributes=TRANSACTED=true RECEIVE_TIMEOUT=60 MAX_THREADS=max_workflow_threads MIN_THREADS=5

enrole.messaging.workflowPendingQueue.attributes=TRANSACTED=true RECEIVE_TIMEOUT=60 WAIT_TIME=0 OVERCAPACITY_WAIT_TIME=10 MAX_THREADS=max_workflow_pending_threads MIN_THREADS=2


Restart the Tivoli Identity Manager server for the changes to take effect.

3.2 Intel multi-processor systems For multi-processor Intel systems it is necessary to update the enrole.messaging parameters in the enRole.properties file to eliminate Java context switching.


itim_home = The Tivoli Identity Manager product home directory, such as /itim45.

Setting the values

Tip: The lines below are split for readability purposes, they should be all in one line in the enRole.properties file.

Edit the itim_home/data/enRole.properties file and append PRIORITY=5 to the following lines as shown: enrole.messaging.adhocSyncQueue.attributes=TRANSACTED=true RECEIVE_TIMEOUT=60 MAX_THREADS=5 MIN_THREADS=1 PRIORITY=5

enrole.messaging.workflowQueue.attributes=TRANSACTED=true RECEIVE_TIMEOUT=60 MAX_THREADS=10 MIN_THREADS=5 PRIORITY=5

enrole.messaging.workflowPendingQueue.attributes=TRANSACTED=true RECEIVE_TIMEOUT=60 WAIT_TIME=0 OVERCAPACITY_WAIT_TIME=10 MAX_THREADS=5 MIN_THREADS=2 PRIORITY=5

enrole.messaging.workflowAbortQueue.attributes=TRANSACTED=true RECEIVE_TIMEOUT=60 WAIT_TIME=0 MAX_THREADS=5 MIN_THREADS=1 PRIORITY=5

enrole.messaging.remoteServicesQueue.attributes=TRANSACTED=false RECEIVE_TIMEOUT=60 WAIT_TIME=0 MAX_THREADS=7 MIN_THREADS=2 PRIORITY=5

enrole.messaging.mailServicesQueue.attributes=TRANSACTED=false RECEIVE_TIMEOUT=60 WAIT_TIME=0 MAX_THREADS=3 MIN_THREADS=1 PRIORITY=5

Restart the Tivoli Identity Manager server for the changes to take effect.

3.3 Recycle bin IBM Tivoli Identity Manager includes the ability to restrict newly generated IDs to being unique across all accounts, both current and deleted. This feature requires the use of the recycle bin to keep track of deleted accounts. If this feature is not required for your environment, disable the recycle bin to improve overall system performance.

When objects such as people, accounts, roles, and provisioning policies are deleted from the IBM Tivoli Identity Manager system using either the graphical user interface (GUI) or the application program interface (API), these objects are not removed from the underlying directory server but rather moved into the recycle bin. The recycle bin is implemented as the following LDAP container:

ou=recycleBin, ou=itim, ou=<tenant>, <suffix>

When LDAP entries are moved under this DN due to a deletion, the attribute erIsDeleted is set to the value Y to enable IBM Tivoli Identity Manager to identify these objects as entries it should neither display to the user nor act upon.

3.3.1 Disabling the recycle bin IBM Tivoli Identity Manager 4.5.1 IF83 allows disabling of the recycle bin for customers who do not need it. If your environment does not require restricting newly generated IDs to being unique across deleted accounts, disable the recycle bin to avoid the performance degradation it causes.

Setting the values

To disable the recycle bin, verify IF83 or later fixpack is installed on all nodes in your environment and follow these steps:


1) Edit enRole.properties

2) Locate the property enrole.recyclebin.enable

3) Set the property to false

4) Stop all IBM Tivoli Identity Manager nodes

5) Delete all entries out of the recycle bin using one of the methods discussed below

6) Restart all IBM Tivoli Identity Manager nodes

If the enrole.recyclebin.enable property does not exist, add it to the end of the file with the value of false.

3.3.2 Emptying the recycle bin Because of the filter that IBM Tivoli Identity Manager uses, having a large number of entries in the recycle bin can negatively impact performance. It is recommended that the size of the recycle bin be kept as small as possible for optimum performance.

There are several ways to remove entries from the recycle bin. IBM Tivoli Identity Manager includes a script that will delete entries in the recycle bin older than a specified age range. See the discussion of the recycle bin age limit in IBM Tivoli Identity Manager Server Installation and Configuration Guide for WebSphere Environments for more information.

An alternate method is to use an LDAP display tool to view the entries and delete them directly in the directory server. Be careful to delete only the entries themselves and not the ou=recycleBin container. Similarly, it is possible to use a combination of the IBM Tivoli Directory Server versions of ldapsearch and ldapdelete commands to delete entries. For example:

idsldapsearch -h <host> -p <port> -D <user> -w <password> \ -b "ou=recycleBin,ou=itim,ou=<tenant>,<suffix>" -s sub "erisdeleted=Y" dn | \ grep –i erglobalid | idsldapdelete -h <host> -p <port> -D <user> -w <password>

After deleting entries from the recycle bin when using an IBM Tivoli Directory Server, run runstats to make IBM DB2 pick up the changes. See the IBM Tivoli Directory Server – Runstats section for more information.

3.4 Reconciliations Reconciliations are resource-intensive operations and can take a while for services with a large account population. Limiting the number of attributes returned by the adapter and processed by IBM Tivoli Identity Manager can improve reconciliation performance. Large reconciliations may also exceed the default Max Duration and if so the value can be increased.

3.4.1 Limiting attributes returned from the adapter Some adapters (such as the adapter for Microsoft Active Directory) can limit the attributes that are returned to the IBM Tivoli Identity Manager server during reconciliations. This can reduce the amount of work required by the adapter to retrieve or calculate the values of the attributes as well as amount of data sent back to IBM Tivoli Identity Manager.

Consult the adapter documentation for information specific to that adapter. See also the IBM Tivoli Identity Manager adapters section.

3.4.2 Limiting the attributes evaluated During reconciliation, any attributes that were identified as having been changed are updated within the


IBM Tivoli Identity Manager directory server. Before this update takes place, the new value is evaluated against the provisioning policy that governs the account to ensure that the change is allowed within the policy, and if not, a policy enforcement is triggered. Any change to the account will trigger the policy evaluation for that account regardless if the change would invalidate the policy.

To reduce the number of policy evaluations, limit the attributes that are evaluated during reconciliation. Some endpoints (such as Microsoft Active Directory) contain attributes that change frequently but are seldom used to enforce policy, such as last logon time. If these attributes are required, consider setting up a second reconciliation to reconcile these attributes on a more infrequent schedule and remove them from the more frequently running reconciliations. If possible, reconcile only those attributes that are required for policy evaluation.


excluded_attributes = The list of attributes that are returned from the adapter to exclude from processing within IBM Tivoli Identity Manager. Ideally these would be all attributes except those that are required for policy evaluation.

Setting the values

1) Log into IBM Tivoli Identity Manager as a user with sufficient privileges to edit the service.

2) Select the Provisioning tab.

3) Browse to the service in the organizational tree.

4) Select the service to modify.

5) Select Reconciliation.

6) Select the reconciliation schedule to modify.

7) Select the Query tab.

8) Move all excluded_attributes from to the Excluded Attributes list.

9) Click Submit.

3.4.3 Maximum duration Large reconciliations can sometimes exceed the default maximum duration as specified in the reconciliation schedule. When this limit is reached the reconciliation is aborted. Increase the limit to allow longer-running reconciliations to complete.


max_duration = The maximum time in minutes that the reconciliation should run. To calculate this value, do an intial run with a very large duration and measure the time required. Consider setting the maximum duration to 10% above this time. Default value: 600.

Setting the values

1) Log into IBM Tivoli Identity Manager as a user with sufficient privileges to edit the service.

2) Select the Provisioning tab.

3) Browse to the service in the organizational tree.

4) Select the service to modify.

5) Select Reconciliation.

6) Select the reconciliation schedule to modify.

7) Set the Max Duration to max_duration.

8) Click Submit.


4 IBM Tivoli Identity Manager adapters

It is sometimes necessary to tune the IBM Tivoli Identity Manager adapters when doing large provisioning changes or reconciliations. This section should supplement, not supersede, the documentation included with the adapter.

4.1 Microsoft Active Directory The Microsoft Active Directory adapter returns attributes to IBM Tivoli Identity Manager that are not directly retrieved from Active Directory but rather calculated from other Windows sources. Querying these external sources can slow down Active Directory reconciliations and can be disabled if these attributes are not needed.

The Home Directory Security and Mailbox Permissions are two such attributes. Retrieving this information requires looking up the appropriate access control entry, which is a costly operation. Setting ReconHomeDirSecurity and ReconMailboxPermissions to FALSE in the adapter registry can disable this overhead.

Working with Windows Terminal Services (WTS) attributes can slow down provisioning and reconciliation as well. There are two adapter registry keys that control access to these attributes:

• WtsEnabled – This key controls the adapters access to WTS attributes. If this key is enabled (set to TRUE) the adapter will have access to provision and reconcile WTS attributes. If this key is disabled (set to FALSE) the adapter will not provision WTS attributes if requested, nor will it return them during reconciliation. The default value for this key is FALSE.

• WtsDisableSearch – This key controls whether the adapter will return WTS attributes during a reconciliation (a “search” from the adapter’s perspective). If this key is enabled (set to TRUE), WTS attributes will not be returned in a reconciliation but the attributes will still be updated in account provisions. If this key is disabled (set to FALSE), WTS attributes will be returned in a reconciliation. This key only applies if the WtsEnabled key is set to TRUE. The default value for this key is TRUE.


5 Database servers

The Tivoli Identity Manager product supports three different database systems: IBM DB2, Microsoft® SQL Server™, and Oracle® Database. Each database requires slightly different tunings.

Each database machine should have at least one processor and 1 GB of RAM. It can reside on a single processor machine by itself or share a multi-processor machine with other applications, but the requirements of 1 GB of RAM per one CPU is the minimum for the database server. Tuning the database is one of the most important tunings for the Tivoli Identity Manager product.

5.1 IBM DB2 Tuning IBM DB2 to run with the IBM Tivoli Identity Manager product involves adjusting the buffer pools, modifying the number of connections, modifying internal database values, adding table space, adjusting logs, indexing, and running runstats.

5.1.1 Quick guide for setting the IBM DB2 tuning parameters This section includes the commands needed to tune the IBM DB2 parameters. This section assumes you have been through the tunings before, know what the desired values are, and just need a reminder of how to set them.

As the database administrator, connect to the database and run the following commands: db2 alter bufferpool ibmdefaultbp size ibmdefaultbp_npages db2 alter bufferpool enrolebp size enrolebp_npages db2 update db cfg for itim_database_name using maxappls num_connections db2 "ALTER TABLESPACE ENROLE_DATA ADD ( \ FILE '/database_home/database_name/NODE0000/SQL00001/container_name' num_pages) \ PREFETCHSIZE 16 OVERHEAD 24.1 TRANSFERRATE 0.9 BUFFERPOOL ENROLEBP" db2 "ALTER TABLESPACE ENROLE_INDEXES ADD ( \ FILE '/database_home/database_name/NODE0000/SQL00001/container_name' num_pages) \ PREFETCHSIZE 16 OVERHEAD 24.1 TRANSFERRATE 0.9 BUFFERPOOL ENROLEBP" db2 "ALTER TABLESPACE TEMP_DATA ADD ( \ FILE '/database_home/database_name/NODE0000/SQL00001/container_name' num_pages) \ PREFETCHSIZE 16 OVERHEAD 24.1 TRANSFERRATE 0.9 BUFFERPOOL ENROLEBP" db2 update db cfg for itim_database using logsecond logs_secondary db2 update db cfg for itim_database using logfilsiz logs_size db2 update db cfg for itim_database using applheapsz applheap_size db2 update db cfg for itim_database using app_ctl_heap_sz appctl_size db2 update db cfg for itim_database using maxfilop max_files_open db2set DB2_RR_TO_RS=YES

5.1.2 Buffer pools IBM DB2 buffer pools are the secondary buffer for IBM Tivoli Directory Server. These buffer pools should be large enough that most table searches can be read directly from memory instead of using the disk. This value can be measured by looking at the hit ratio for the buffer pools. See the IBM DB2 performance monitoring section for information about calculating the buffer pool hit ratio.

The IBM Tivoli Identity Manager database has two buffer pools: IBMDEFAULTBP and ENROLEBP. IBMDEFAULTBP is used as a buffer for table spaces with small extent sizes (4 KB) and ENROLEBP is used as a buffer for table spaces with large extent sizes (32 KB). Most of the tables within the IBM Tivoli Identity Manager database use the table space with a large extent size and thus will use the ENROLEBP buffer pool. Use a 1:3 memory ratio between IBMDEFAULTBP and ENROLEBP.


First, determine the following values for your system:

total_mem_avail = The total amount of physical memory in bytes.

mem_for_itimdb = The amount of memory in bytes to allocate to the Tivoli Identity Manager


database. This value should be small enough that it will reside in physical memory and not be swapped out to disk. Recommended value: 500000000 (500 MB) or greater.

Next, calculate the following values:

ibmdefaultbp_npages = (mem_for_itimdb / 4096) * 0.25 enrolebp_npages = (mem_for_itimdb / 32768) * 0.75

Setting the values

As the database administrator, connect to the database and run the following commands: db2 alter bufferpool ibmdefaultbp size ibmdefaultbp_npages db2 alter bufferpool enrolebp size enrolebp_npages

5.1.3 Connections The Tivoli Identity Manager server uses JDBC connections from the J2EE application server to communicate with the database. The number of connections from the application server to the database depends on the needs of the application. Both application servers allow you to set maximum connection values. The DB2 variable MAXAPPLS needs to be set to a value greater than the maximum possible connections. We recommend setting MAXAPPLS to five more than the total maximum possible connections. The default value for MAXAPPLS with IBM DB2 V8 is AUTOMATIC, which is the recommended setting on that version.


itim_database_name = The name of your Tivoli Identity Manager database, such as itim.

num_connections = Maximum number of connections. See the J2EE application servers section to calculate the maximum number of connections and add five to that value.

Setting the values

As the database administrator, connect to the database and run the following command: db2 update db cfg for itim_database_name using maxappls num_connections

5.1.4 Table spaces The IBM Tivoli Identity Manager product uses a Database Managed Space (DMS) table space to store its data. This type of table space gives better performance than System Managed Space (SMS) table spaces but requires you to preallocate disk space for the database to use. The default size of the IBM Tivoli Identity Manager table space is 512 MB for ENROLE_DATA, 256 MB for ENROLE_INDEXES, and 256 MB for TEMP_DATA. This is frequently not large enough and should be increased by either adding more containers, enabling the autoresize option for the table space, or both.

It is possible to both add additional table space containers as well as enabling autoresize on the table spaces depending on your specific environment, disk restrictions, and table space layouts.

5.1.4.5 Adding additional table space containers

Add more table spaces by using the IBM DB2 alter tablespace command. If possible, add files that reside on another physical drive because IBM DB2 performs better if a table space has multiple containers on multiple drives. The creation and adoption of these altered table spaces is not immediate. Examine the output of the alter tablespace command as it is executing and rerun the command if the database is busy altering a table space.



tablespace_name – The name of the table space onto which you want to add containers. The IBM Tivoli Identity Manager table space names are ENROLE_DATA, ENROLE_INDEXES, and TEMP_DATA.

database_home – The home directory of your database administrator, such as /home/db2inst1.

database_name – The name of the database such as db2inst1. This is also a subdirectory under database_home.

container_name – The name of the file to create to hold the table space container, such as enrole_data2.

num_pages – The number of 32 KB pages to add to the table space. To calculate the number of pages from the amount of disk space, divide the size in megabytes by 0.032768. A 512 MB table space would be 15625 pages.

Setting the values

As the database administrator, connect to the database and run the following command: db2 "ALTER TABLESPACE tablespace_name ADD ( \ FILE '/database_home/database_name/NODE0000/SQL00001/container_name' num_pages) \ PREFETCHSIZE 16 OVERHEAD 24.1 TRANSFERRATE 0.9 BUFFERPOOL ENROLEBP"

5.1.4.6 Enabling autoresize

An alternative to adding another table space is to enable the current table space to grow larger. This feature is available in IBM DB2 V8.2.2 (IBM DB2 V8.2 FP2 or IBM DB2 V8.1 FP9) and later. If a table space has autoresize enabled, any containers for that table space will automatically grow if the containers become full. This can decrease the administrative overhead for these DMS table spaces; however, care must be taken to ensure that the disks the containers are on do not become full.

The autoresize option can be enabled on both new and existing table spaces using the IBM DB2 alter tablespace command.


tablespace_name – The name of the table space on which you want to enable autoresize. The IBM Tivoli Identity Manager table space names are ENROLE_DATA, ENROLE_INDEXES, and TEMP_DATA.

Setting the values

As the database administrator, connect to the database and run the following command for each table space you want to enable autoresize:

db2 "ALTER TABLESPACE tablespace_name AUTORESIZE YES”

5.1.5 Transaction logs IBM DB2 keeps logs during transaction processing. During large transactions the default log number and sizes might be too small and cause transaction roll backs. Increasing the size and number of log files available to IBM DB2 solves this issue.

IBM DB2 has two types of log files: primary and secondary. Primary logs are allocated when the database is started and remain allocated until the database is stopped. Secondary logs are allocated as needed after the primary logs are full and are released after they are no longer needed.


Increase the number of secondary logs to be prepared when large transactions occur. The default size of the log files is 1000 4 KB pages, or 4 MB. Increase this to 10000 4 KB pages, or 40 MB. Note that this will increase the size of both your primary and secondary log files.


itim_database = The name of the Tivoli Identity Manager database, such as itim.

logs_secondary = The number of secondary logs. Recommended value: 12.

logs_size = The size of the primary and secondary logs in 4k pages. Recommended value: 10000.

Setting the values

As the database administrator, connect to the database and run the following commands: db2 update db cfg for itim_database using logsecond logs_secondary db2 update db cfg for itim_database using logfilsiz logs_size

5.1.6 Database application heaps Some of the queries that the Tivoli Identity Manager application submits to the DB2 server result in complex SQL statements. The Tivoli Identity Manager installation program will increase the default application heap sizes to useable values. If you see transaction rollback errors in the itim.log file, increase the values of the heaps in increments of 256 until the errors stop.


itim_database = The name of the Tivoli Identity Manager database, such as itim.

applheap_size = The value of applheapsz in 4 KB pages. Default value: 256. Recommended value: 2048.

appctl_size = The value of app_ctl_heap_sz in 4 KB pages. Default value: 128. Recommended value: 1024.

Setting the values

As the database administrator, connect to the database and run the following commands: db2 update db cfg for itim_database using applheapsz applheap_size db2 update db cfg for itim_database using app_ctl_heap_sz appctl_size

Stop and restart the database instance for these changes to take effect.

5.1.7 Indexing Adding an index to a heavily used table can greatly increase performance. Without indexes, DB2 must scan every row of the table until it finds the specified data. With an index, it uses a more efficient search method. The following indexes have been identified as increasing database performance.

Attention: It is very important to run runstats on the system after adding new indexes. See the IBM DB2 - Runstats section for more information.


There are no values to be determined for this section.

Setting the values - Tivoli Identity Manager product Version 4.4 and Version 4.5

As the database administrator, connect to the database and run the following commands:

Tip: Some of these indexes have already been added to the Tivoli Identity Manager installation. If there are "unable to create index" errors, they can be ignored as duplicates.

db2 'CREATE UNIQUE INDEX ACTIVITY_ID_X ON ENROLE.ACTIVITY \ ("ID" ASC) INCLUDE ("SUBTYPE")' db2 'CREATE INDEX ACTIVITY_PID_X ON ENROLE.ACTIVITY ("PROCESS_ID" DESC)' db2 'CREATE INDEX ACTIVITY_STATE_X ON ENROLE.ACTIVITY ("STATE" DESC)' db2 'CREATE UNIQUE INDEX PROCESS_ID_X ON ENROLE.PROCESS ("ID" ASC)' db2 'CREATE INDEX PROCESSDATA_PID_X ON ENROLE.PROCESSDATA \ ("PROCESS_ID" DESC, "DEF_ID" DESC)' db2 'CREATE INDEX WORKITEM_PID_X ON ENROLE.WORKITEM ("PROCESS_ID" DESC)' db2 'CREATE INDEX PROCESS_SUB_X ON ENROLE.PROCESS ("SUBMITTED" DESC, "PARENT_ID" ASC)'


Setting the values - Tivoli Identity Manager product Version 4.5.1

No additional indexes are required for tuning the Tivoli Identity Manager product Version 4.5.1.

5.1.8 Runstats Statistics on the number of rows in the tables and what indexes are available are required for IBM DB2 to efficiently fulfill queries. It is important to update these table and index statistics after large Directory Server Markup Language (DSML) loads, HR feeds, and reconciliations. If you experience high CPU usage or poor IBM DB2 performance, run the runstats command on all the tables in the database. IBM DB2 reorgchk does not update index statistics and is not a replacement for runstats. To update index statistics, run the runstats command on each table individually. A script (perftune_runstats.sh) that detects the version of DB2 and runs the runstats command against all tables for a given schema in a database is provided in Appendix A – scripts and files.

If the runstats command is run in a working environment, allow the connected applications to continue to write to the database. Use the "shrlevel change" option on DB2 V7 and "allow write access" option on DB2 V8 to allow users to write to a database while runstats is running.

Use the runstats command on an idle or lightly-used database because it requires update locking on system statistics table to update the database statistics. A database with a heavy load might experience transaction rollbacks due to the system acquiring locks on the tables that are used by the database optimizer to fulfill queries.

In addition to running runstats on all tables within the database, we also manually update the statistics table for the ACTIVITY, PROCESS, PROCESSDATA, and SCHEDULED_MESSAGE tables to ensure a minimum cardinality. Setting a minimum cardinality on these tables helps the IBM DB2 query optimizer and can decrease locking issues in the database.


Use the runstats command on every table in the ENROLE schema. As the database administrator, connect to the database and use the following command to get a listing of all tables in the schema:

db2 list tables for all | grep ENROLE

Setting the values – DB2 V7

For each table in the ENROLE schema, run the following DB2 runstats command: db2 runstats on table ENROLE.table_name with distribution and \ detailed indexes all shrlevel change

Manually update the database statistics table for the LDAP_DESC and LDAP_ENTRY tables. db2 update sysstat.tables set card = 50000 where tabname = 'ACTIVITY' and card < 50000 db2 update sysstat.tables set card = 50000 where tabname = 'PROCESS' and card < 50000 db2 update sysstat.tables set card = 50000 where tabname = 'PROCESSDATA' and card < 50000 db2 update sysstat.tables set card = 50000 where tabname = 'SCHEDULED_MESSAGE' \ and card < 50000


For each table in the ENROLE schema, run the following DB2 runstats command: db2 runstats on table ENROLE.table_name with distribution and \ detailed indexes all allow write access

Manually update the database statistics table for the LDAP_DESC and LDAP_ENTRY tables. db2 update sysstat.tables set card = 50000 where tabname = 'ACTIVITY' and card < 50000 db2 update sysstat.tables set card = 50000 where tabname = 'PROCESS' and card < 50000 db2 update sysstat.tables set card = 50000 where tabname = 'PROCESSDATA' and card < 50000 db2 update sysstat.tables set card = 50000 where tabname = 'SCHEDULED_MESSAGE' \ and card < 50000


5.1.9 Maximum open files In an attempt to work well with other applications running on the system, DB2 sets a limit on the number of files it keeps open through the maxfilop setting. If it needs to open more files this value, DB2 will close a currently open file to open the new one. This can cause a performance loss on systems that do not need this restriction on the number of open files.

Increase this value in increments of 64 if a database snapshot shows that database files have closed.


max_files_open = The maximum number of files DB2 will have open at any one time. Default value: 64. Recommended value: 256.

Setting the values

As the database administrator, connect to the database and run the following command: db2 update db cfg for itim_database using maxfilop max_files_open

5.1.10 Lock tuning

5.1.10.7 Lock type

The IBM Tivoli Identity Manager application requires that RR_TO_RS locking be enabled. This setting allows select statements that involve rows that are deleted but not yet committed to be fulfilled without waiting for the transaction to complete.

Tip: This setting only applies to IBM DB2 V7 and earlier. The type 2 indexes introduced in IBM DB2 V8 make this setting unnecessary and will be ignored if set.



Setting the values

Run the following command as the IBM Tivoli Identity Manager database administrator: db2set DB2_RR_TO_RS=YES

Stop and restart the database instance for this change to take effect.

5.1.10.8 Lock list and maximum locks

The performance team has found that the default settings for the DB2 lock list (locklist) and maximum locks (maxlocks) are adequate for most installations. These may be increased if the local DB2 administrator recommends so for your environment.

5.1.10.9 Lock timeout

The default lock timeout (locktimeout) value in the IBM Tivoli Identity Manager database is infinity, represented by the value -1.

If locking problems are seen, it is possible to change this from infinity as long as the configured value is greater than or equal to the WebSphere total transaction timeout value. Setting this value to less than the WebSphere total transaction timeout is unsupported and will result in transaction rollback errors as not all components recover from a lock timeout.


5.2 Microsoft SQL Server Steps to tune Microsoft SQL Server will be included here when those settings and their values are determined.

5.3 Oracle Database

5.3.1 Server configuration The default Oracle configuration uses the “small” settings in the init.ora file for the database. Increasing the options in this file to use the “middle” or “large” values will provide faster performance. Consult with an Oracle DBA, Oracle administrator, or Oracle documentation for more information on tuning of the Oracle server.

5.3.2 Table spaces During database configuration, the IBM Tivoli Identity Manager application creates several small table spaces. The table spaces are large enough to demo the product and for small testing environments, but will need to be increased for a production environment. The initial size of the IBM Tivoli Identity Manager table space is 160 MB for ENROLE_DATA and 160 MB for ENROLE_INDEXES. Both table spaces are configured to autoextend up to 1 GB in size. Some versions also include an ITIML000_DATA table space with a maximum size of 80 MB. This is frequently not large enough and should be increased by either adding more containers, increasing the maximum size, or both.

Additionally, consider spreading the data across multiple physical disks either during the initial database configuration or afterwards by adding additional table space containers.

5.3.2.10 Spreading database data

Spreading the database files across multiple disks will decrease the I/O contention in the database and improve Oracle performance. When creating the Oracle database, as documented in Server Installation and Configuration Guide - Chapter 2, consider spreading the table space files across multiple disks. The following SQL statements illustrate the spreading for a Unix system that has 4 hard drives (disk1 through disk4 with Oracle on disk1):

Attention: The SQL statements below are for illustrative purposes only. It is environment specific – not only the file systems, but also the sizes that you can allocate to each table space file. Please consult your Oracle DBA and tailor them to your environment before you apply them.


disk# – The disk to spread the table spaces onto (in our example, disks 1 through 4).

tablespace_size – The size to make each table space (in our example between 1 GB and 3 GB).

Setting the values create tablespace RBS datafile '/disk2/oradata/rbs01.dbf' size 1000m; create temporary tablespace TEMP tempfile '/disk2/oradata/temp01.dbf' size 1000m reuse; create tablespace ENROLE_DATA datafile '/disk3/oradata/enrole_data_01.dbf' size 3000m; create tablespace ENROLE_INDEXES datafile '/disk4/oradata/enrole_indexes_01.dbf' size 1500m;

Tip: The itim_home/config/rdbms/oracle/create_rollbackSegment.sql script puts the roll back segment table space on ORACLE_HOME (because it only specifies the file name). Consider changing these to use a different disk if your environment allows.


5.3.2.11 Adding additional table space datafiles

The maximum size of the initial table spaces is unlikely to be large enough for a production system. Increase the amount of table space available by either increasing the maximum size of the existing datafile within a table space or add additional datafiles. When adding additional datafiles, consider adding them to different physical drives to decrease I/O contention.


tablespace_name – The name of the IBM Tivoli Identity Manager table space to alter, such as ENROLE_DATA, ENROLE_INDEXES, or ITIML000_DATA.

datafile_name – The name of the file to use when adding additional datafiles to a table space or modifying an existing datafile. Consider using a separate physical drive. Example value: /data/ou1/app/oracle/oradata/itimdb/enrole_data2.dbf.

initial_size – The initial size of the datafile. Example value: 512M.

maxsize_string – String used to set the maximum size of the datafile. Specify UNLIMITED to allow the datafile file to grow unbounded or maxsize <number> to limit it to a specific size. Example: maxsize 2048M.

Setting the values

To alter the maximum size of a table space, use the following SQL statement: alter database datafile 'datafile_name' autoextend on maxsize_string;

To add additional datafiles to a table space, use the following SQL statement: alter tablespace tablespace_name add datafile 'datafile_name' size initial_size autoextend on maxsize_string;

5.3.3 Indexing Adding an index to a heavily used table can greatly increase performance. Without indexes, Oracle must scan every row of the table until it finds the specified data. With an index, it uses a more efficient search method. The following indexes have been identified as increasing database performance.



Setting the values

As the database administrator, connect to the database and run the following SQL commands: create index PROCESS_SUB_X on ENROLE.PROCESS ("SUBMITTED" DESC, "PARENT_ID" ASC); create index PROCESS_ID_ST on ENROLE.PROCESS ("ID" ASC, "STATE" ASC); create index PROCESSDATA_ID_DEF on ENROLE.PROCESSDATA ("PROCESS_ID" ASC, "DEF_ID" ASC, "VALUE_LAST_MODIFIED" ASC); create index SCH_MSG_SVR on ENROLE.SCHEDULED_MESSAGE ("SERVER" ASC);

5.3.4 Updating statistics At regular intervals, from one week to one month on a production IBM Tivoli Identity Manager system, or after a large amount of processing has been completed, database statistics should be gathered and updated. These statistics are used by Oracle to make query decisions on locating information that have a dramatic impact on how fast Oracle can return requests. The DBMS_STAT package must be installed to issue the following DBMS_STAT commands.


database_instance = The name of the database instance such as enrole.


Setting the values

Create a file named Oracle_dbms.stat_cmds.txt containing the following lines: exec dbms_stats.gather_schema_stats(ownname => 'database_instance', cascade => true);

As the system user, execute sqlplus and run the following command: @ Oracle_dbms.stat_cmds.txt

Generating statistics may take from several minutes to several hours for a large database. Execution should be issued during off-peak times.


6 Directory servers

The Tivoli Identity Manager product supports two different directory servers: IBM Tivoli Directory Server and Sun ONE Directory Server (formerly known as Netscape iPlanet Directory Server).

6.1 IBM Tivoli Directory Server With respect to tuning, IBM Tivoli Directory Server can be divided into two parts: the IBM Tivoli Directory Server process and IBM DB2.

In a well-tuned environment, the IBM Tivoli Directory Server process and the IBM DB2 processes consume approximately the same amount of CPU. In a poorly tuned system, IBM DB2 can max out the CPU usage trying to fulfill queries in a poor manner.

Both IBM Tivoli Directory Server and IBM DB2 have caches that speed up data retrieval. Optimizing your available memory is the key to tuning IBM Tivoli Directory Server. When a read request comes in to IBM Tivoli Directory Server, it checks the filter cache to see if has seen that search filter before. If it has, the results are pulled from the cache, otherwise the query goes to IBM DB2. After the search filter is evaluated, IBM Tivoli Directory Server pulls the entries that match the search filter from the entry cache. If the values are not in the entry cache, it queries IBM DB2. For each request to IBM DB2, IBM DB2 checks to see if the data resides in a buffer pool. If not, it reads the value from the disk. Ideally, all requests to the directory server register an IBM Tivoli Directory Server cache hit or an IBM DB2 buffer pool hit for the quickest response. Queries that require disk access can be very slow.

If you encounter problems with the IBM Tivoli Directory Server, see the Troubleshooting – IBM Tivoli Directory Server section for help with common issues.

6.1.1 Quick guide for setting the IBM Tivoli Directory Server tuning parameters

This section includes the commands needed to tune the IBM Tivoli Directory Server parameters. This section assumes you have been through the tunings before, know what the desired values are, and just need a reminder of how to set them.

As the database administrator, connect to the database and run the following commands: db2 alter bufferpool ibmdefaultbp size ibmdefaultbp_npages db2 alter bufferpool ldapbp size ldapbp_npages db2 update db cfg for ldap_database using logsecond logs_secondary db2 update db cfg for ldap_database using logfilsiz logs_size db2 update db cfg for ldap_database using maxappls max_db_connections db2 update db cfg for ldap_database using stmtheap stmtheap_size db2 update db cfg for ldap_database using applheapsz applheap_size db2 update db cfg for ldap_database using app_ctl_heap_sz appctl_size db2 update db cfg for ldap_database using pckcachesz pckcache_size db2 update db cfg for ldap_database using sortheap sortheap_size db2 update dbm cfg using sheapthres sortheapthres_size

Stop IBM Tivoli Directory Server, edit /etc/slapd32.conf (IBM Tivoli Directory Server V4.1) or /etc/ibmslapd.conf (IBM Tivoli Directory Server V5.x) and update the following configuration options:

ibm-slapdDbConnections: ldap_db_connections ibm-slapdACLCache: acl_cache ibm-slapdACLCacheSize: acl_cache_size ibm-slapdFilterCacheSize: filter_cache_size ibm-slapdEntryCacheSize: entry_cache_size

Stop and restart DB2 and the IBM Tivoli Directory Server server for these changes to take effect.

6.1.2 Cache sizes IBM Tivoli Directory Server has four caches: the access control list (ACL) cache, the filter cache, the entry cache, and the attribute cache. Because the Tivoli Identity Manager server binds as an authoritative user,


the ACL cache is used only for internal processes and can be small to allow the memory to be used elsewhere. It has been found that a small value in ACL cache is beneficial for IBM Tivoli Directory Server performance.

The filter cache is most helpful for programs that issue more read requests than write or update requests because the entire filter cache is invalidated at every write. The Tivoli Identity Manager application frequently updates the directory server so it is not beneficial to allocate a large filter cache. However, because Tivoli Identity Manager frequently sends identical queries sequentially to the server, it is useful to have a small filter cache for those queries.

IBM Tivoli Directory Server allows you to control how many entries the entry cache can store but does not restrict the size of the cache. The size of each entry in the cache is based on the number and the size of attributes that a given LDAP entry has. Typically, many entries are users and their accounts, which have a fairly constant size. When setting the value for the entry cache, calculate the size of the average entry and divide that into the amount of memory used by the IBM Tivoli Directory Server process. Users with few attributes can generate entry sizes that are approximately 4k where users with more attributes can generate entry sizes around 9k. The IBM Tivoli Directory Server Performance Tuning Guide discusses the procedure for determining the size of the average entry.


acl_cache – Specifies whether the ACL cache is used. Recommended value: TRUE (enabled).

acl_cache_size – The size of the ACL cache. Recommended value: 100.

filter_cache_size – The size of the filter cache. Recommended value: 100.

entry_cache_size – The size of the entry cache. Recommended value: as large as the number of Tivoli Identity Manager users times the average number of accounts plus one per user. For example, if you have 25,000 users with two accounts each, you would want 25,000 * (2+1) = 75,000. This value is bounded by the amount of memory allocated to the IBM Tivoli Directory Server process minus the size of the process itself (about 128 MB).

Attention: Do not set the entry cache size larger than you have available physical memory. If the IBM Tivoli Directory Server process size exceeds the amount of available memory, significant performance degradation will occur because of page swapping. Care should be taken when increasing the size of the caches so the amount of memory required does not exceed the maximum amount a process can allocate such as the 2 GB for most 32-bit processes.

Tip: Performance metrics suggest that the Attribute cache available in the IBM Tivoli Directory Server version 5 and later does not provide a significant performance boost and the memory would be better allocated elsewhere.

Example: Setting the entry cache size to 75,000 and given a size of 9k for an entry in the entry cache would require 675 MB (75,000 * 9 KB = 675,000 KB = 675 MB) of physical RAM for the entry cache alone, not including the 128 MB needed for the server process itself.

Setting the values

Stop IBM Tivoli Directory Server, edit /etc/slapd32.conf (IBM Tivoli Directory Server V4.1) or /etc/ibmslapd.conf (IBM Tivoli Directory Server V5.x) and update the following configuration options:

ibm-slapdACLCache: acl_cache ibm-slapdACLCacheSize: acl_cache_size ibm-slapdFilterCacheSize: filter_cache_size ibm-slapdEntryCacheSize: entry_cache_size

Restart IBM Tivoli Directory Server for these changes to take effect.

6.1.3 Buffer pools IBM DB2 buffer pools are the secondary buffer for IBM Tivoli Directory Server. These buffer pools should be large enough that most table searches can be read directly from memory instead of using the disk. This value can be measured by looking at the hit ratio for the buffer pools. See the IBM DB2 performance monitoring section for information about calculating the buffer pool hit ratio.


The IBM Tivoli Directory Server database (ldapdb2 by default) has two buffer pools: IBMDEFAULTBP and LDAPBP. IBMDEFAULTBP is used as a buffer for table spaces with small extent sizes (4 KB) and LDAPBP is used as a buffer for table spaces with large extent sizes (32 KB). Most of the tables in the ldapdb2 database use the table space with a small extent size and use IBMDEFAULTBP. Use a 3:1 memory ratio between IBMDEFAULTBP and LDAPBP.


Allocate enough memory to the IBM DB2 buffer pools so the buffer pool hit ratio is greater than 95% and allocate the rest of the memory to the IBM Tivoli Directory Server process and the caches. The following formula can help in determining the buffer pool values for your server.

First, determine the values for your system:

ldap_database = The name of the IBM Tivoli Directory Server database, such as ldapdb2.

mem_for_ldapdb2_bps = The amount of memory in bytes to allocate to the ldapdb2 buffer pools. This value should be small enough that it will reside in physical memory and not be swapped out to disk. Recommended value: 500000000 (500 MB) or greater.

Next, calculate the following values:

ibmdefaultbp_npages = (mem_for_ldapdb2_bps / 4096) * 0.75 ldapbp_npages = (mem_for_ldapdb2_bps / 32768) * 0.25

Setting the values

As the database administrator, connect to the database and run the following commands: db2 alter bufferpool ibmdefaultbp size ibmdefaultbp_npages db2 alter bufferpool ldapbp size ldapbp_npages

6.1.4 Connections There are three connection values to modify for IBM Tivoli Directory Server. The first value is the number of connections that IBM Tivoli Directory Server should make to the DB2 backend. Set this to 30.

The second value is the number of connections that the DB2 server should allow to connect to it. This value should be greater than the number of connections that IBM Tivoli Directory Server will make to it. Set this value to 10 more than the number specified for the IBM Tivoli Directory Server server. These extra connections allow for administrative functions.

The third value only applies to Windows systems. Windows allows up to 64 sockets in a select call for each thread. This limits the number of inbound connections to 64. To overcome this problem set the SLAPD_OCHANDLERS environment variable to specify the number of connection threads, each one supporting 64 connections.


ldap_database_name = The name of your IBM Tivoli Directory Server database, such as ldapdb2.

num_ldap_connections = The number of connections from IBM Tivoli Directory Server to the DB2 backend. Recommended value: 30 (from the IBM Tivoli Directory Server Tuning Guide).

num_db_connections = The number of connections the DB2 backend server will accept. Recommended value: 40.

For Windows systems, also determine the following values:

max_num_LDAP_connections = The maximum number of IBM Tivoli Directory Server connections (enrole.connectionpool.maxpoolsize) from the enRole.properties file. The Tivoli Identity Manager application default is 100.


itim_nodes = The number of Tivoli Identity Manager nodes in your cluster. This is the sum of the WF and the UI components.

oc_handlers = (max_num_LDAP_connections * itim_nodes) / 64

Setting the values

As the database administrator, connect to the database and run the following commands: db2 update db cfg for ldap_database_name using maxappls num_db_connections

To set the number of connections the IBM Tivoli Directory Server server will make to the DB2 server, stop IBM Tivoli Directory Server, edit /etc/slapd32.conf (IBM Tivoli Directory Server V4.1) or /etc/ibmslapd.conf (IBM Tivoli Directory Server V5.x), and find the stanza:

dn: cn=Directory, cn=RDBM Backends, cn=IBM Directory, cn=Schemas, cn=Configuration

Change the following attribute: ibm-slapdDbConnections: num_ldap_connections

For Windows systems, also set the SLAPD_OCHANDLERS values. Stop IBM Tivoli Directory Server, edit slapd32.conf (IBM Tivoli Directory Server 4.1) or ibmslapd.conf (IBM Tivoli Directory Server 5.x) and find the stanza:

dn: cn=Front End, cn=Configuration

Add the following configuration option at the end of the stanza before the blank line: ibm-slapdsetenv: SLAPD_OCHANDLERS=oc_handlers

Stop and start the IBM Tivoli Directory Server database and restart IBM Tivoli Directory Server for these changes to take effect.

6.1.5 Transaction logs IBM DB2 keeps logs during transaction processing. During large transactions, the default log number and sizes might be too small and cause transaction roll backs. This is solved by increasing the size and number of log files available to IBM DB2.

IBM DB2 has two types of log files: primary and secondary. Primary logs are allocated when the database is started and remain allocated until the database is stopped. Secondary logs are allocated, as needed after the primary logs are full, and are released after they are no longer needed.


Increase the number of secondary logs to be prepared when large transactions occur. The default size of the log files is 1000 4 KB pages, or 4 MB. Increase this to 10000 4 KB pages, or 40 MB. Note that this will increase the size of both your primary and secondary log files.

ldap_database – The name of the IBM Tivoli Directory Server database, such as ldapdb2.

logs_secondary – The number of secondary logs. Recommended value: 12.

logs_size – The size of the primary and secondary logs in 4 KB pages. Recommended value: 10000.

Setting the values

As the database administrator, connect to the database and run the following commands: db2 update db cfg for ldap_database using logsecond logs_secondary db2 update db cfg for ldap_database using logfilsiz logs_size

6.1.6 Database statement heap The Tivoli Identity Manager application can submit very long LDAP queries to the IBM Tivoli Directory Server. If the query is particularly long, the query will not fit in the DB2 statement heap (stmtheap) and IBM Tivoli Directory Server will return an error to the Tivoli Identity Manager application.


Warning: The statement heap is allocated per agent (connection) therefore increasing this value can dramatically increase the memory used by the DB2 server.



stmtheap_size = The value of stmtheap in 4k pages. Default value: 2048. Recommended value: 4096.

Setting the values

As the database administrator, connect to the database and run the following commands: db2 update db cfg for ldap_database using stmtheap stmtheap_size

Stop and restart the IBM Tivoli Directory Server database and IBM Tivoli Directory Server for these changes to take effect.

6.1.7 Database application heaps Complex queries that the Tivoli Identity Manager application submits to IBM Tivoli Directory Server result in complex SQL statements to the DB2 database. The default sizes of the application heaps are not large enough to allow DB2 to fulfill these queries from IBM Tivoli Directory Server. Failing to increase these heaps might cause JNDI errors in the itim.log file. If you receive errors after increasing these values, continue to increase these values in increments of 256.



applheap_size = The value of applheapsz in 4 KB pages. Default value: 256. Recommended value: 2048.

appctl_size = The value of app_ctl_heap_sz in 4 KB pages. Default value: 128. Recommended value: 1024.

Setting the values

As the database administrator, connect to the database and run the following commands: db2 update db cfg for ldap_database using applheapsz applheap_size db2 update db cfg for ldap_database using app_ctl_heap_sz appctl_size


6.1.8 Package cache size Increasing the pckcachesz value size has been shown to decrease DB2 preparation time resulting in less CPU utilization when evaluating provisioning policies with many unique roles. The package cache stores evaluations of these complex queries and increasing this value prevents the DB2 server from repeatedly executing the same costly evaluations of the roles.



pckcache_size = The value of pckcachesz in 4 KB pages. Default value: 360. Recommended value: 3600.

Setting the values

As the database administrator, connect to the database and run the following commands: db2 update db cfg for ldap_database using pckcachesz pckcache_size



6.1.9 Database sort heaps Several of the SQL queries from IBM Tivoli Directory Server require sorts in the database. To increase performance for these queries, increase the size of the sort heaps in DB2.

There are two sort heap parameters inside DB2: sortheap and sheapthres. The sortheap parameter is a database-level parameter and is allocated as needed to perform sorts. Each sort is allocated its own sortheap. The sheapthres parameter is a database-manager level parameter and provides an upper bound on the total amount of sort heap available to sorts. When increasing the size of the sortheap, ensure that sheapthres is large enough for multiple sorts to allocate sort heap.

If you experience excessive sort overflows, as identified from a database snapshot, increase the sortheap parameter. You might need to increase the sheapthres parameter as well.



sortheap_size = The value of sortheap in 4 KB pages. Default value: 256. Recommended value: 1024.

sortheapthres_size = The value of sheapthres in 4 KB pages. Default value: 20,000 pages on UNIX® systems and 10,000 pages on Windows systems. Ideally this value would be sortheap * number_of_database_connections however this value may be too high for your system. The sortheapthres_size should be at least twice as large as sortheap_size.

Setting the values

As the database administrator, connect to the database and run the following commands: db2 update db cfg for ldap_database using sortheap sortheap_size db2 update dbm cfg using sheapthres sortheapthres_size


6.1.10 System limits System limits (also called ulimits) may prohibit the IBM Tivoli Directory Server process from accessing enough real or virtual memory. Failure to set system limits high enough can cause the IBM Tivoli Directory Server process to core or disappear.


process_data_size – The maximum data segment size for the process. Minimum value: 256000 (256 MB). Recommended value: unlimited.

virtual_mem_size – The maximum virtual memory size for the process. Minimum value: 256000 (256 MB). Recommended value: unlimited.

Setting the values

As the user who starts the IBM Tivoli Directory Server process, run the following commands prior to starting ibmslapd:

Solaris: ulimit –d process_data_size ulimit –v virtual_mem_size

AIX: ulimit –d process_data_size ulimit –m virtual_mem_size


You may wish to put these commands into the shell startup files for the user.

Alternatively, on AIX you can edit the /etc/system/limits file and change the data and rss limits in the default stanza to process_data_size and virtual_mem_size values respectively or -1 for unlimited. For these limits to take effect, the user must log out of the current session and log back in.

For more information on setting the ulimit correctly for IBM Tivoli Directory Server, search the IBM Support site (http://www.ibm.com/support) for reference 1206894.

6.1.11 Indexing Indexing the attributes that applications search on increases IBM Tivoli Directory Server performance. IBM Tivoli Directory Server indexes are automatically translated into IBM DB2 indexes when you update the IBM Tivoli Directory Server schema for those attributes.

If you extend the LDAP schema in IBM Tivoli Directory Server to include additional attributes, index those attributes that you will search for. Any filter in the Tivoli Identity Manager application (such as with dynamic roles) is translated into a search string for the IBM Tivoli Directory Server.

The Tivoli Identity Manager application frequently searches against the organization (o), organizational unit (ou), and owner attributes. Index these attributes either through the IBM Tivoli Directory Server Web Administration tool or by importing the ids_indexes.ldif file using the ldapmodify command

After updating the LDAP schema, run DB2 runstats on the database to update the statistics for the newly created indexes.


root_dn = The root DN of the IBM Tivoli Directory Server server.

root_password = The password for the root DN.

Setting the values

1) To use the LDIF, create the ids_indexes.ldif file from Appendix A – scripts and files and place it on the computer.

2) Run the following command to import the LDIF into IBM Tivoli Directory Server: ldapmodify –D root_dn –w root_password –f ids_indexes.ldif

6.1.12 Runstats The number of rows in the tables and what indexes are available are required for IBM DB2 to efficiently fulfill queries. Run the runstats command on all tables in the database before and after large DSML loads and reconciliations. If you experience high CPU usage or poor DB2 performance, run the runstats command on all the tables in the database. DB2 reorgchk does not update index statistics and is not a replacement for runstats. To update index statistics, run the runstats command on each table individually. A script (perftune_runstats.sh) that detects the version of DB2 and runs the runstats command against all tables for a given schema in a database is provided in Appendix A – scripts and files.

If the runstats command is run in a working environment, allow the connected applications to continue to write to the database. Use the "shrlevel change" option on DB2 V7 and "allow write access" option on DB2 V8 to allow users to write to a database while runstats is running.

Use the runstats command on an idle or lightly-used database because it requires update locking on system statistics table to update the database statistics. A database with a heavy load might experience transaction rollbacks due to the system acquiring locks on the tables that are used by the database optimizer to fulfill queries.

In addition to running runstats on all tables within the database, it is necessary to manually update the statistics table for the LDAP_DESC and LDAP_ENTRY tables. The choices IBM DB2 makes about when to use these tables for fulfilling queries for the IBM Tivoli Directory Server are not ideal for IBM Tivoli

http://www.ibm.com/support


Identity Manager. Manually adjusting the statistics table helps IBM DB2 make better choices and use these tables at the end of the access plan instead of the beginning.


As the database administrator, connect to the database and use the following command to get a listing of all tables in the schema:

db2 list tables for all | grep LDAPDB2


For each table in the LDAPDB2 schema, run the following DB2 runstats command: db2 runstats on table ENROLE.table_name on all columns shrlevel change

Manually update the database statistics table for the LDAP_DESC and LDAP_ENTRY tables. db2 update sysstat.tables set card = 9E18 where tabname = 'LDAP_DESC' db2 update sysstat.tables set card = 9E18 where tabname = 'LDAP_ENTRY'


For each table in the LDAPDB2 schema, run the following DB2 runstats command: db2 runstats on table ENROLE.table_name on all columns allow write access

Manually update the database statistics table for the LDAP_DESC and LDAP_ENTRY tables. db2 update sysstat.tables set card = 9E18 where tabname = 'LDAP_DESC' db2 update sysstat.tables set card = 9E18 where tabname = 'LDAP_ENTRY'

6.2 Sun ONE Directory Server The Sun ONE Directory Server (formerly known as Netscape iPlanet Directory Server) does not provide many parameters to adjust for better performance. The three areas that provide the best performance boost are indexing, the All IDs value, and the database cache size.

6.2.1 All IDs Threshold Value The Sun ONE Directory Server sets an upper bound on how many index entries should be allowed for each attribute indexed. If this value is exceeded, the index is invalidated and is not used during searches. This behavior prevents the server from including all entries in an index and consuming excessive amounts of memory.

The Sun ONE Directory Server Installation and Tuning Guide recommends setting this value to 5% of the size of your directory. The IBM Tivoli Identity Manager product uses the directory server in a dramatically different manner than the writers of Sun ONE Tuning Guide had in mind and this value should be increased to the largest number of users, groups, services, or accounts the IBM Tivoli Identity Manager server has stored in the directory.

Because the method to change this parameter is non-trivial, it is recommended that you set the value based on the estimated size of your directory in its final form. If the current size of your directory is 20,000 but you expect the final size of your directory to be 100,000 entries, use the larger value to calculate the All IDs Threshold value.

Please refer to the Sun ONE Directory Server Installation and Tuning Guide for more information on adjusting the All IDs Threshold value.


directory_home = The Sun ONE Directory Server home directory, such as /usr/iplanet/servers/slapd-gso-2DS.

number_of_entries = The number of entries in your directory. A good way to estimate this number is to multiply the number of Tivoli Identity Manager users by the average number of accounts for each user.


suffix_name = The name of the root suffix, such as dc=com.

database_name = The name of the directory server database, such as itim5145.

temp_directory = The name of a temporary directory, such as /tmp. This directory should have enough free space to store an LDIF of your entire directory. Allow 1KB per Tivoli Identity Manager user.

Setting the values

Warning: Setting this value requires the database to be rebuilt (the data exported and re-imported). If done incorrectly, this process has the potential to destroy information within your directory. It is highly recommended that you back up the information in your directory prior to setting this value.

Tip: Increasing the size of the All IDs Threashold will cause the size of the indexes to grow. Because the indexes are stored in the database cache, the cache size might need to be increased as well. See the See the Sun ONE Directory Server - Cache size section for information about increasing this value.

Edit directory_home/config/dse.ldif file and find the stanza: dn: cn=config,cn=ldbm database,cn=plugins,cn=config

Change the following attribute: nsslapd-allidsthreshold: number_of_entries

You will need to export and re-import all of your data for this parameter to take effect. This can be done with the db2ldif and ldif2db commands.

First, export your data: db2ldif -n database_name -a temp_directory/export.ldif -s "suffix_name"

Next, stop the directory server.

Finally, import your data: ldif2db -n database_name -i temp_directory/export.ldif

Start the directory server.

6.2.2 Indexing Indexing the attributes that applications search on increases Sun ONE Directory Server performance.

If you extend the LDAP schema to include additional attributes, index those attributes that you will search for. Any filter in the Tivoli Identity Manager application (such as with dynamic roles) is translated into a search string for the LDAP server.

The Tivoli Identity Manager application frequently searches against the O, Owner and OU. Owner is indexed by default in Sun ONE Directory Server but OU and O is not. Index the OU and O attribute from the directory server configuration console.



Setting the values

1) Open the Directory Server Console.

2) Select the Configuration tab.

3) Expand the Data node in the navigation pane.

4) Expand the suffix of your database in the navigation pane and select the database.

5) Select the Indexes tab.

6) Select the Add attribute… button.


7) Find OU in the list box and click OK.

8) If the Equality and Presence options are not already selected for the OU attribute, select them.

9) Click Save.

10) A progress dialog will appear while the index is added. Click Close after the index creation completes.

6.2.3 Cache sizes The caches used by Sun ONE Directory Server V5.1 and V5.2 are slightly different. In V5.2 there are two distinct caches that need to be tuned: the database cache that stores database-level entries and indexes, and the entry cache that stores formatted entries. In V5.1 there is only one cache, the database cache that stores index information, filter requests, and entries.

6.2.3.12 Sun ONE Directory Server V5.1

The directory server uses a database cache of 10 MB by default. This value is too small for almost all directories.


cache_size – The size of the database cache. Recommended value: 256000000 (256 MB). If the entry cache hit ratio on the Status tab for the database is below 95%, increase this value in increments of 128 MB until the hit ratio is above 95% or your system is out of physical memory.

Warning: Setting this value greater than the amount of physical RAM available will cause performance degradation as the system swaps the pages out to disk.

Setting the values





5) Select the Database Settings tab.

6) Set the Memory available for cache to cache_size.

7) Click Save.

6.2.3.13 Sun ONE Directory Server V5.2

As previously mentioned, V5.2 has two types of caches: database and entry. The entry cache is used only for base-level searches and thus provides no benefit for subtree or one-level searches. The database cache is used for both subtree and one-level searches as well as base-level searches if the results are not found in the entry cache.

A typical IBM Tivoli Identity Manager does about 50% base searches and 50% one-level and subtree searches. For this reason, we recommend allocating no more than 50% of your total cache memory to the entry cache. For larger directories, you may find better performance benefit by using significantly more memory in the database cache than in the entry cache.


db_cache_size – The size of the database cache. Recommended value: 512000000 (512 MB).

entry_cache_size – The size of the entry cache. Recommended value: 512000000 (512 MB).


Warning: Setting these values greater than the amount of physical RAM available will cause performance degradation as the system swaps the pages out to disk.

Warning: The nsslapd-dbcachesize (db_cache_size above) and nsslapd-cachememsize (entry_cache_size above) parameters have platform-specific limits. Consult the Sun ONE documentation on these parameters before increasing them above the recommended values.

Setting the values





5) Select the Database Settings tab.

6) Set the Maximum Cache size to db_cache_size.

7) Set the Memory Available for Cache to entry_cache_size.

8) Click Save.


7 Operating systems

Performance can be improved on some systems by making some operating system-specific changes.

7.1 Red Hat Update the msgmni kernel variable to 1026.


msgmni_value – 1026. The default is 16.

Setting the values

As root, run the following command: sysctl -w kernel.msgmni=msgmni_value

Update the open file limit to 8000.


ulimit_value – 8000. The default is 2000.

Setting the values

As root, run the following command: ulimit -n ulimit_value

and change the value of nofiles in /etc/security/limits to 8000.


8 Best practices

The IBM Tivoli Identity Manager product can be set up and configured in many ways. The following are some best practice suggestions to help guide you in setting up your environment.

• Each agent modifies the LDAP schema by adding new attributes to support a new service. These attributes are created without indexes, and for services that service thousand of users, a large benefit can be achieved by adding indexes to attributes with many members.

• Provisioning policies that touch a large number of users result in a single query to the directory server that returns a large number of users. Because the directory servers only process a query on a single thread, you will obtain faster provisioning with fewer, faster processors than more, slower processors. For example, a 2 x 1.4GHz machine is better suited than a 4 x 700MHz machine.

• In general, network latency is not a major performance bottleneck, but it is still a good idea to have as few hops as possible between components. This includes the IBM Tivoli Identity Manager server components, the directory server, database server, agents, and agent endpoints. Ideally all components would be on the same subnet or no more than one hop away and on 100-megabit or faster network.

• To minimize disk bottlenecks, it is frequently better to have several disks in the system rather than one large drive. IBM DB2 can make use of multiple disks in the system if it is configured to do so. The IBM Tivoli Identity Manager product itself can benefit from having the log files on a different disk than the messaging queues. Keep in mind that high-end I/O backplanes or other advanced storage systems can overcome this by balancing the load across multiple disks automatically.

• Database and directory activity can be very CPU- and memory-intensive. It is recommended that each application have at least one processor and 2 GB of RAM each. More processors are generally better. To obtain optimal performance, it is not recommended to have all IBM Tivoli Identity Manager components on the same system unless it is a very powerful unit such as an 8 x 1.4 GHz server with 12 GB RAM.

• Individual provisioning policies should not affect more than 20,000 users. In Tivoli Identity Manager version 4.5 and version 4.5.1 when an automatic provisioning policy is modified, the complete list of affected accounts is created in memory and can result in out-of-memory errors with more than 20,000 people. Tivoli Identity Manager version 4.6 does not have this limitation.

• Dynamic roles effect people in a given scope, either one-level or subtree. When a person object within that scope is modified or added, that role must be re-evaluated. This is true for every dynamic role in the system. For instance, if there are three dynamic roles with subtree scope and a person object within that scope is updated, all three dynamic roles will be re-evaluated. For this reason it is recommended that you limit the number of dynamic roles, either by number or by scope, that affect people that are modified frequently. It doesn’t matter if the dynamic role ends up enrolling the person or not, the evaluation itself is the performance-impacting overhead.

• Limiting the scope (via placement within the organizational tree) and number of ACIs will increase performance by requiring fewer evaluations. When doing a person search via the APIs, be sure to limit the scope of your search to be as narrow as possible to avoid the system evaluating more ACIs than are necessary.

• When enabling WebSphere global security, do not enable Java 2 security unless it is required for your environment. Enabling WebSphere global security automatically enables Java 2 security unless it is explicitly disabled. Having Java 2 security enabled can cause a significant performance degradation to IBM Tivoli Identity Manager.


9 Regular maintenance

To maintain optimal performance for your IBM Tivoli Identity Manager environment, regular maintenance is required.

• Regularly empty the IBM Tivoli Identity Manager recycle bin. As the number of objects in the recycle bin increase LDAP performance can degrade. The frequency at which the recycle bin is emptied depends on how frequently deletes occur in the system. Ideally, the recycle bin would contain fewer than 100 items. See IBM Tivoli Identity Manager application – Recycle bin for more information on how to empty the recycle bin.

• Update database statistics after a large number of updates. Updating database statistics in the underlying databases can significantly improve performance and should be done on a weekly basis for most environments. This applies to the IBM DB2 database if using IBM Tivoli Directory Server as well as the underlying IBM DB2 or Oracle Database of the IBM Tivoli Identity Manager application. See the appropriate database sections on how to update database statistics.


10 Troubleshooting

Finding performance problems with IBM Tivoli Identity Manager can be complicated by the large number of middleware dependencies. A problem such as a slow DSML feed with account provisioning could be the result of a slow directory server, database locking issues, or an inadequate number of Tivoli Identity Manager workflow threads. This section is designed to assist you in identifying problem areas and some pointers on fixing them.

Information in this section is provided with the assumption that you have read and applied the tunings recommended in the previous sections. If this is not the case, stop and review those sections now.

10.1 Log files When something seems to stop working, the first place to check is the itim.log file. It often contains useful information about what is currently being processed and any errors that may have occurred. If you suspect something isn’t working correctly and the itim.log files doesn’t contain enough information, increase the logging in the enRoleLogging.properties file. Edit the file and either uncomment or add one or more of the following lines: log4j.category.com.ibm.itim.workflow=DEBUG log4j.category.com.ibm.itim.workflowextensions=DEBUG log4j.category.com.ibm.itim.remoteservices=DEBUG

The workflow and workflowextensions classes are used during all workflow processes. Setting these to DEBUG can help track down workflow problems. The remoteservices class communicates with external agents (such as a Windows NT service and a Tivoli Access Manager service but not the ITIM Service) during provisioning. Set this to DEBUG to help diagnose problems with agents. Changes to the enRoleLogging.properties file are picked up every 10 minutes or whenever the Tivoli Identity Manager application is restarted.

Also, be sure to check the log file for the appropriate middleware components:

• IBM WebSphere Application Server - /usr/WebSphere/AppServer/logs/server_name/

• BEA WebLogic – weblogic_home/user_projects/itim/server_name/server_name.log

• IBM Tivoli Directory Server - /var/ldap/ibmslapd.log

• Sun ONE Directory Server – sun_one_home/logs/access

• IBM DB2 – instance_home/sqllib/db2dump/db2diag.log

• Oracle Database - *.trc files

• Microsoft SQL Server – mssql_home\LOG\

10.2 Identifying bottlenecks Start by monitoring the processor and disk usage of every server (IBM Tivoli Identity Manager nodes, directory, database) to see which server is most heavily utilized. Based on this information, review the monitoring and tuning steps specific to that component.

The IBM Tivoli Identity Manager application makes intense usage of both the database and the directory server it is configured into. The database is used as an information staging area and audit trail for provisioning actions. The directory server is used as a permanent storage location and can be heavily queried when evaluating provisioning policies. Because of this usage, it is possible for either the database or the directory server to be a bottleneck during heavy usage or large provisioning changes.


During an action that causes a large provisioning policy to be evaluated, such as adding a new policy or modifying an existing policy that impacts a large number of users, the affected users must be queried from the directory server. The directory server evaluates the query and returns the matching users. If the directory server is not tuned correctly it can become the bottleneck as the Tivoli Identity Manager server waits for the result set before starting the required provisioning action. Ensure the directory server is fulfilling the requested queries as quickly and efficiently as possible to minimize this behavior. Refer to the appropriate directory server tuning section for more information.

After the result set is returned, the Tivoli Identity Manager server will begin to enforce the provisioning policy for each user. This process is multi-threaded and benefits from multiple processors (the workflow cluster member in a functional cluster). If it appears that the processors on the server are not being fully utilized and no other server appears to be the bottleneck, consider increasing the maximum number of threads being used. See the IBM Tivoli Identity Manager application - Threads section for more information.

Any enforcement action required for a user (account addition, modification, or deletion) is stored as a workflow item in the database. When a thread becomes available, it queries the database for the next workflow item that needs processing and proceeds to act upon that item. This usage can result in access contention and thus locking in the database. Appropriate indexes and access-plan statistics will help minimize the number of locks needed to fulfill these requests. Refer to the appropriate database tuning section for more information.

10.3 Specific problems

10.3.1 IBM Tivoli Identity Manager application Problem: Transaction rollback errors in the itim.log file. Solution: Transaction rollbacks can occur for several different reasons, most of them database-related. There should be an error message within the itim.log file giving more information about what went wrong. Some areas to check when you get a transaction rollback:

• Database storage space. If the database runs out of storage space for the table paces, a transaction rollback error can occur. Increase the amount of disk space allocated to the table paces to resolve this problem.

• Database locking issues. If the database encounters extreme locking issues it is possible to get a transaction rollback error. Confirm that the locks are tuned appropriately and update the table statistics.

• Database memory issues. If there is not enough memory available to database structures to fulfill the requested query, a transaction rollback error may occur. The JNDI error in the itim.log file should indicate which database heap should be increased. Increase the appropriate heap using the recommendations provided in the section for that middleware.

Problem: Out of Memory errors in the itim.log file. Solution: The message is from the J2EE Application Server. The Java virtual machine (JVM) ran out of heap size. Increase the maximum heap size and restart the application server. If the heap size is already at the limit, then you may need to break up transactions; for example, use fewer services or roles in a provisioning policy.

10.3.2 IBM WebSphere MQ Problem: Nothing seems to be going through workflow and new items are piling up on the queue. Solution: A message on the message queue may be corrupt and blocking new transactions. Clearing the queues may help.


Caution: Clearing the queues can result in an inconsistent system state. It is recommended that you only resort to clearing your queues when instructed to do so by a support technician.

1. Stop the Tivoli Identity Manager application. You can’t clear the queues while the Tivoli Identity Manager application is running.

2. Find the name of the queue manager by listing all queues on the system: dspmq

3. Start the MQ Script interpreter: runmqsc WAS_machinename_jmsserver

4. List Tivoli Identity Manager queues: display queue('WQ_itim*')

5. For each queue, clear it: clear qlocal('WQ_itim_name')

Tip: If you receive a message that the queue is in use, stop the Tivoli Identity Manager server and try again.

6. Exit the MQ Script interpreter: end

10.3.3 IBM Tivoli Directory Server Problem: Poor search performance. Solution: Slow queries or high CPU utilization with the IBM Tivoli Directory Server can be caused by several factors. Some areas to consider:

1. Check for long-running queries that need indexes. The IBM Tivoli Directory Server uses IBM DB2 to process LDAP queries. By checking DB2 for long-running queries it is possible to find what attributes may need indexing. To find how long each query takes, turn on statement cache monitoring within DB2: db2 update dbm cfg using DFT_MON_STMT ON Stop the directory server, restart the database, and restart the directory server for this to take effect. After the monitoring is turned on, duplicate the suspect action within Tivoli Identity Manager. After the action has completed, get a snapshot of the statement cache: db2 get snapshot for dynamic sql on database_name The output will contain stanzas like the following: Number of executions = 12 Number of compilations = 1 Worst preparation time (ms) = 3 Best preparation time (ms) = 3 Internal rows deleted = 0 Internal rows inserted = 0 Rows read = 10024 Internal rows updated = 0 Rows written = 0 Statement sorts = 0 Total execution time (sec.ms) = 136.000663 Total user cpu time (sec.ms) = 62.010000 Total system cpu time (sec.ms) = 10.000000 Statement text = SELECT distinct E.EID FROM LDAPDB2.LDAP_ENTRY AS E, LDAPDB2.LDAP_ENTRY as pchild WHERE E.EID=pchild.EID AND pchild.PEID=? AND E.EID IN (SELECT EID FROM LDAPDB2.OU WHERE OU = ?)

Calculate the average execution time per query by dividing the total execution time by the number of executions: total execution time / number of executions. In the example above: 136 / 12 =


11.33 seconds per execution. Most queries should take a second or less to complete. Queries that take longer may be searching on columns that are not indexed by IBM DB2, and hence are not indexed within the IBM Tivoli Directory Server. Another symptom of this problem is a high average number of rows read, calculated by dividing the rows read by the number of executions. In the above example, the column OU is likely not indexed. Correct this by indexing the OU attribute within the IBM Tivoli Directory Server. See the IBM Tivoli Directory Server – Indexing section for more information. The Tivoli Identity Manager tuning scripts contain a script (perfanalyze_dynamicsql.pl) that will calculate the time per execution for all stanzas and sort the results. See Appendix A – scripts and files for information on how to get the tuning scripts.

2. Check for low buffer pool hit ratio. The IBM Tivoli Directory Server uses IBM DB2 to process LDAP queries. The IBM Tivoli Directory Server database should have a high (>95%) hit ratio. If the buffer pools are not large enough IBM DB2 will need to read more information from the disk resulting in high I/O wait. See the IBM DB2 performance monitoring - Buffer pool hit ratio section for information on calculating the buffer pool hit ratio. The Tivoli Identity Manager tuning scripts contain a script (perfanalyze_bufferpools.pl) that will calculate the hit ratio for all buffer pools for you. See Appendix A – scripts and files for information on how to get the tuning scripts.

Problem: The directory server crashes for no apparent reason. Solution: Check the size of your cache sizes. If your cache sizes cause the IBM Tivoli Directory Server process to grow beyond what is supported by your operating system’s memory model (in general around 2 GB on 32-bit operating systems) it can crash. See the IBM Tivoli Directory Server Tuning Guide (in Other resources) for more information. Alternatively the ibmslapd process may be hitting an artificial system limit, such as a ulimit. See IBM Tivoli Directory Server – System limits for more information on increasing system limits.

10.3.4 Sun ONE Directory Server Problem: Poor search performance. Solution: Slow queries or high CPU utilization with the Sun ONE Directory Server can be caused by several factors. Some areas to consider:

1. Check for long-running queries that need indexes. Most queries should take less than one second to complete. Queries taking longer than a second are generally searching on attributes that are not indexed. To find queries taking longer than a second, search the Sun ONE Directory server access logs for etime=X where X is an integer greater than 1, X representing the number of seconds the query took to complete. We are searching for queries running 2 seconds or longer. For example: [26/Mar/2004:12:54:25] conn=4236 op=1 msgId=2 - RESULT err=0 tag=101 nentries=1 etime=2 Now search for what query the search time is associated with by searching the same file for the matching connection and operation number, which in the above example is conn=4236 op=1. The query for the above example was: [26/Mar/2004:12:54:24] conn=4236 op=1 msgId=2 – SRCH base="ou=ibm,dc=com" scope=2 filter="(&(!(erIsDeleted=y))(&(erEnabled=true)(erPolicyMembership=2;*))(objectClass=erProvisioningPolicy))" attrs=ALL

Examine the LDAP query string and identify the attributes being searched on. In the above example they are erIsDeleted, erEnabled, erPolicyMembership, and objectClass. Ensure that all


attributes being searched on are correctly indexed. See the Sun ONE Directory Server - Indexing section for information on how to index attributes.

2. See if the All IDs Threshold value needs to be increased. If the All IDs Threshold value is too small, Sun ONE Directory Server may not be using the indexes for certain attributes. Determine if queries are searching on these attributes by searching for notes=U in the Sun ONE Directory Server access logs. The presence of this flag for a query indicates that one or more attributes being searched on is over the All IDs Threshold and should probably be increased. See the Sun ONE Directory Server - All IDs Threshold Value section for more information on how to increase this value.

3. See if the database cache hit ratio is low. If the Sun ONE Directory Server database cache is too small, the server must access the disk for the information and will result in a low database cache hit ratio. Identify the hit ratio for the database by viewing the Monitor tab for that database. The hit ratio should be 95% or higher. If it is lower than 95%, increase the size of your database cache. See the Sun ONE Directory Server - Cache sizes section for information on increasing the size.


11 IBM DB2 performance monitoring

IBM DB2 provides several tools for troubleshooting and analyzing performance problems. This section contains a primer for how to get started monitoring with IBM DB2. For more extensive information, refer to the IBM DB2 Administration Guide: Performance book.

11.1 Enable monitoring To gather performance information, turn on the IBM DB2 monitoring flags. To turn on monitoring, run the following commands as the database administrator. Note that this should be done for each database.

db2 update dbm cfg using DFT_MON_STMT ON db2 update dbm cfg using DFT_MON_BUFPOOL ON db2 update dbm cfg using DFT_MON_LOCK ON db2 update dbm cfg using DFT_MON_SORT ON db2 update dbm cfg using DFT_MON_TIMESTAMP ON db2 update dbm cfg using DFT_MON_UOW ON

Stop and restart the database instance for the monitoring to take effect.

Tip: We enable all monitors except the Table monitor as it has a slight performance impact when enabled..

11.2 Snapshots Snapshots allow you to view the internal state of various DB2 components. The following commands access important DB2 snapshots:

db2 get snapshot for database on database_name db2 get snapshot for dynamic sql on database_name db2 get snapshot for bufferpools on database_name db2 get snapshot for tables on database_name db2 get snapshot for locks on database_name

Alternatively you can gather all snapshots at one time using the following command: db2 get snapshot for all on database_name

11.3 Statement monitor The statement monitor is useful for examining what is occurring for each request sent to the database. While the statement monitor is turned on, each SQL request is recorded and the results of the queries can be examined looking for missing indexes, execution time, preparation time, database scans, and index scans. The monitor collects a great deal of information and should only be activated for a short time to gather requests.

Set up the monitor

If this is the first time you've done any monitoring or generated an explain plan on this database you'll need to do some one-time setup:

db2 -tf sqllib/misc/EXPLAIN.DDL db2 "create event monitor dstatement for statements write to file '/tmp/dstatements'" mkdir /tmp/dstatements

Use the monitor

To monitor the database statements, connect to the database, clear out any previous data, and turn on the monitor:

db2 connect to ldapdb2 rm -f /tmp/dtatements/* db2 "set event monitor dstatement state 1"

Run the query or the action that you wish to monitor and turn off the monitor:


db2 "set event monitor dstatement state 0"

Convert the data to a human-readable format: db2evmon -path /tmp/dstatements >/tmp/dstate.out

The do_statistics_monitoring.sh script in the tuning guide scripts package automates this process. You can use the explainSQL.sh script to then have IBM DB2 explain how the optimizer will process a particular query, including any use of indexes.

11.4 Buffer pool hit ratio The buffer pool hit ratio gives a good indication of how many data reads are coming from the buffer pool and how many are coming from the disk. The larger the hit ratio, the less disk I/O used. Calculate the buffer pool hit ratio by enabling buffer pool monitoring and taking a database snapshot. Calculate the buffer pool hit ratio with the following formula:

P = buffer pool data physical reads + buffer pool index physical reads L = buffer pool data logical reads + buffer pool index logical reads Hit ratio = (1-(P/L)) * 100%


12 Other resources

You will find the following resources useful for further tuning of the IBM Tivoli Identity Manager middleware:

DB2 Administration Guide: Performance

http://www.ibm.com/cgi-bin/db2www/data/db2/udb/winos2unix/support/v8pubs.d2w/en_main

IBM Tivoli Identity Manager Configuration Guide

http://publib.boulder.ibm.com/tividd/td/ITIM/SC32-1493-00/en_US/PDF/im451_config.pdf

IBM Tivoli Directory Server Tuning Guide

http://publib.boulder.ibm.com/tividd/td/IBMDirectoryServer5.2.html

WebSphere Application Server Monitoring and Tuning Performance

http://www.ibm.com/software/webservers/appserv/infocenter.html

Sun ONE Directory Server Documentation

http://docs.sun.com/app/docs/coll/S1_DirectoryServer_52 (Sun ONE Directory Server V5.2) http://developers.sun.com/prodtech/dirserver/reference/docs/ (Sun ONE Directory Server V5.1)

Sun Java Garbage Collection References

http://java.sun.com/docs/hotspot/gc/ http://java.sun.com/docs/hotspot/gc1.4.2/faq.html http://java.sun.com/docs/hotspot/VMOptions.html

http://www.ibm.com/cgi-bin/db2www/data/db2/udb/winos2unix/support/v8pubs.d2w/en_main

http://publib.boulder.ibm.com/tividd/td/ITIM/SC32-1493-00/en_US/PDF/im451_config.pdf

http://publib.boulder.ibm.com/tividd/td/IBMDirectoryServer5.2.html

http://www.ibm.com/software/webservers/appserv/infocenter.html

http://docs.sun.com/app/docs/coll/S1_DirectoryServer_52

http://developers.sun.com/prodtech/dirserver/reference/docs/

http://java.sun.com/docs/hotspot/gc/

http://java.sun.com/docs/hotspot/gc1.4.2/faq.html

http://java.sun.com/docs/hotspot/VMOptions.html


13 Appendix A – scripts and files

The most recent version of these scripts can be obtained at http://www.ibm.com/support/us. Type “ITIM Tuning Guide” in the search box under Search technical support, and click Search. Most of the scripts will need to be customized for your system.

13.1 perftune_runstats.sh #!/bin/sh # perftune_runstats.sh # Description: # Shell script to run the db2 runstats command on all tables for a given # database and schema. It autodetects the db2 version using db2level # and passes the appropriate string to runstats to allow writes to occur # during the runstats. # Usage: # This script should be run as the user for the database (ie: one that # has connect and runstats abilities and permissions). if [ $# -eq 0 ]; then DATABASE=ldapdb2 SCHEMA=LDAPDB2 elif [ $# -eq 1 ]; then DATABASE=$1 SCHEMA=LDAPDB2 else DATABASE=$1 SCHEMA=$2 fi

# Old way, for LDAP distribution and indexes are not needed # OPTIONS="with distribution and detailed indexes all"

OPTIONS="on all columns”

# Find out DB2 version for runstats syntax if db2level | grep "DB2 v7" > /dev/null; then ACCESS="shrlevel change" echo Detected DB2 major version 7 elif db2level | grep "DB2 v8" > /dev/null; then ACCESS="allow write access" echo Detected DB2 major version 8 fi # Connect to the database echo Connecting to $DATABASE db2 connect to $DATABASE # Execute runstats on all tables echo Performing runstats on all tables for schema $SCHEMA echo " with options: $OPTIONS $ACCESS" for i in `db2 connect to $DATABASE > /dev/null; db2 list tables for all | \ grep $SCHEMA | cut -d' ' -f1` do echo Table: $SCHEMA.$I db2 runstats on table $SCHEMA.$i $OPTIONS $ACCESS done # If the database we're tuning is an LDAP database, # update LDAP_DESC and LDAP_ENTRY stats in the statistics table if [ "$SCHEMA" = "LDAPDB2" ]; then echo Updating LDAP_DESC and LDAP_ENTRY stats in the statistics table db2 "update sysstat.tables set card = 9E18 where tabname = 'LDAP_DESC'" db2 "update sysstat.tables set card = 9E18 where tabname = 'LDAP_ENTRY'" fi

13.2 ids_indexes.ldif #---------- beginning of ids_indexes.ldif ---------

# ids_indexes.ldif # - Additional indexes for ITIM's use with IDS

http://www.ibm.com/support/us


# # Usage: # ldapmodify -D cn=root -w <password> -f ids_indexes.ldif #

dn: cn=schema changetype: modify replace: attributetypes attributetypes: ( 2.5.4.10 NAME ( 'o' 'organization' 'organizationName' ) DESC 'This attribute contains the name of an organization (organizationName).' SUP 2.5.4.41 EQUALITY 1.3.6.1.4.1.1466.109.114.2 SUBSTR 2.5.13.4 ) - replace: ibmattributetypes ibmattributetypes: ( 2.5.4.10 DBNAME ( 'o' 'o' ) ACCESS-CLASS NORMAL LENGTH 128 EQUALITY ORDERING )

dn: cn=schema changetype: modify replace: attributetypes attributetypes: ( 2.5.4.32 NAME 'owner' DESC 'Identifies the distinguished name (DN) of the person responsible for the entry.' SUP 2.5.4.49 EQUALITY 2.5.13.1 SYNTAX 1.3.6.1.4.1.1466.115.121.1.12{1000} ) - replace: ibmattributetypes ibmattributetypes: ( 2.5.4.32 DBNAME ( 'owner' 'owner' ) ACCESS-CLASS NORMAL LENGTH 1000 EQUALITY ORDERING ) dn: cn=schema changetype: modify replace: attributetypes attributetypes: ( 2.5.4.11 NAME ( 'ou' 'organizationalUnit' 'organizationalUnitName' ) DESC 'This attribute contains the name of an organization (organizationName).' SUP 2.5.4.41 EQUALITY 1.3.6.1.4.1.1466.109.114.2 SUBSTR 2.5.13.4) - replace: ibmattributetypes ibmattributetypes: ( 2.5.4.11 DBNAME ( 'ou' 'ou' ) ACCESS-CLASS NORMAL LENGTH 128 EQUALITY )

#------------------- end of LDIF ------------------

Documents

IBM Tivoli Identity Manager Performance Tuning Guidepublib.boulder.ibm.com/tividd/td/ITIM/SC32-1496-06/en_US/PDF/... · 2.2 BEA WebLogic Server ... IBM DB2 monitoring information