IBM Tivoli Identity Manager Performance Tuning Guide …publib.boulder.ibm.com/tividd/td/ITIM/SC32-5272-00/en_US/PDF/sc23... · IBM Tivoli Identity Manager Performance Tuning Guide

IBM Tivoli Identity Manager Performance Tuning Guide Version 4.6

Issue Date: 2005 June 30 – First Edition

Publication Number: SC23-5272-00

Copyright Notice

Copyright IBM Corporation 2005. All rights reserved. May only be used pursuant to a Tivoli Systems Software License Agreement, an IBM Software License Agreement, or Addendum for Tivoli Products to IBM Customer or License Agreement. No part of this publication may be reproduced, transmitted, transcribed, stored in a retrieval system, or translated into any computer language, in any form or by any means, electronic, mechanical, magnetic, optical, chemical, manual, or otherwise, without prior written permission of IBM Corporation. IBM Corporation grants you limited permission to make hardcopy or other reproductions of any machine-readable documentation for your own use, provided that each such reproduction shall carry the IBM Corporation copyright notice. No other rights under copyright are granted without prior written permission of IBM Corporation. The document is not intended for production and is furnished “as is” without warranty of any kind. All warranties on this document are hereby disclaimed, including the warranties of merchantability and fitness for a particular purpose.

U.S. Government Users Restricted Rights -- Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corporation.

Trademarks

IBM, the IBM logo, Tivoli, the Tivoli logo, AIX, IBM DB2, IBM Tivoli Identity Manager and WebSphere Application Server are trademarks or registered trademarks of International Business Machines Corporation or Tivoli Systems Inc. in the United States, other countries, or both.

Microsoft, Windows, Windows NT, and the Windows logo are trademarks of Microsoft Corporation in the United States, other countries, or both.

UNIX is a registered trademark of The Open Group in the United States and other countries.

Java and all Java-based trademarks and logos are trademarks or registered trademarks of Sun Microsystems, Inc. in the United States and other countries.

Other company, product, and service names may be trademarks or service marks of others.

Notices

References in this publication to Tivoli Systems or IBM products, programs, or services do not imply that they will be available in all countries in which Tivoli Systems or IBM operates. Any reference to these products, programs, or services is not intended to imply that only Tivoli Systems or IBM products, programs, or services can be used. Subject to valid intellectual property or other legally protectable right of Tivoli Systems or IBM, any functionally equivalent product, program, or service can be used instead of the referenced product, program, or service. The evaluation and verification of operation in conjunction with other products, except those expressly designated by Tivoli Systems or IBM, are the responsibility of the user. Tivoli Systems or IBM may have patents or pending patent applications covering subject matter in this document. The furnishing of this document does not give you any license to these patents. You can send license inquiries, in writing, to the IBM Director of Licensing, IBM Corporation, North Castle Drive, Armonk, New York 10504-1785, U.S.A.

IBM Tivoli Identity Manager Performance Tuning Guide Page 1

Table of contents Table of contents ......................................................................................................................................1 About this guide ........................................................................................................................................3

Who should use this guide ...................................................................................................................3 1 Introduction...........................................................................................................................................4

1.1 Vital tunings .................................................................................................................................4 1.2 Initial tunings................................................................................................................................4 1.3 Resource allocation .....................................................................................................................5

1.3.1 Memory....................................................................................................................................5 1.3.2 CPU .........................................................................................................................................5 1.3.3 Disk space...............................................................................................................................6

2 J2EE application servers ......................................................................................................................7 2.1 IBM WebSphere Application Server ............................................................................................7

2.1.1 Java Virtual Machine (JVM) size.............................................................................................7 2.1.2 Java virtual machine (JVM) options – Sun Solaris..................................................................8 2.1.3 IBM HTTP Server ....................................................................................................................9

2.1.3.1 Connections..................................................................................................................10 2.1.4 IBM WebSphere MQ .............................................................................................................10

3 IBM Tivoli Identity Manager application..............................................................................................11 4 Database servers ...............................................................................................................................12

4.1 IBM DB2 ....................................................................................................................................12 4.1.1 Quick guide for setting the IBM DB2 tuning parameters.......................................................12 4.1.2 Bufferpools ............................................................................................................................12 4.1.3 Connections...........................................................................................................................13 4.1.4 Tablespaces ..........................................................................................................................13 4.1.5 Transaction logs ....................................................................................................................14 4.1.6 Indexing .................................................................................................................................14 4.1.7 Runstats ................................................................................................................................15 4.1.8 Lock tuning ............................................................................................................................15

4.1.8.2 Lock type ......................................................................................................................15 4.1.8.3 Lock timeout .................................................................................................................16

4.2 Microsoft SQL Server ................................................................................................................16 4.3 Oracle Database........................................................................................................................16

4.3.1 Server Configuration .............................................................................................................16 4.3.2 Updating Statistics.................................................................................................................16

5 Directory servers ................................................................................................................................17 5.1 IBM Tivoli Directory Server........................................................................................................17

5.1.1 Quick guide for setting the IBM Tivoli Directory Server tuning parameters ..........................17 5.1.2 Bufferpools ............................................................................................................................17 5.1.3 Connections...........................................................................................................................18 5.1.4 Transaction logs ....................................................................................................................19 5.1.5 Indexing .................................................................................................................................19 5.1.6 Runstats ................................................................................................................................20

5.2 Sun ONE Directory Server ........................................................................................................20 5.2.1 All IDs Threshold Value.........................................................................................................20 5.2.2 Indexing .................................................................................................................................22 5.2.3 Database cache size .............................................................................................................22

6 Operating Systems .............................................................................................................................24 6.1 Red Hat......................................................................................................................................24

7 Best practices .....................................................................................................................................25 8 Troubleshooting..................................................................................................................................26

8.1 Log files .....................................................................................................................................26 8.2 Identifying bottlenecks ...............................................................................................................26 8.3 Specific problems ......................................................................................................................27

8.3.1 IBM Tivoli Identity Manager application ................................................................................27

Page 2 IBM Tivoli Identity Manager Performance Tuning Guide

8.3.2 IBM WebSphere MQ .............................................................................................................27 8.3.3 IBM Tivoli Directory Server....................................................................................................28 8.3.4 Sun ONE Directory Server ....................................................................................................29

9 IBM DB2 performance monitoring ......................................................................................................31 9.1 Enable monitoring......................................................................................................................31 9.2 Snapshots..................................................................................................................................31 9.3 Bufferpool hit ratio......................................................................................................................31 9.4 IBM DB2 Statement Monitor......................................................................................................31

10 Other resources..................................................................................................................................33 11 Appendix A – scripts and files ............................................................................................................34

11.1 perftune_runstats.sh..................................................................................................................34


About this guide This guide identifies some ways to tune your IBM® Tivoli Identity Manager™ system to improve performance.

Who should use this guide Use this guide if you are responsible for installing or maintaining an IBM Tivoli Identity Manager system. The following competencies are recommended:

Familiarity with basic database and directory design principles.

General knowledge of system resource management (memory, disk, and processor) including your operating system’s memory model. Without knowledge of other resource requirements on the system it is possible to negatively impact performance by over tuning.

Understanding of how to configure and administer your directory and database servers. You may need to have your local database administrator or directory administrator perform these tunings for you.


1 Introduction

The IBM Tivoli Identity Manager product solves the complex problem of identity management. Due to the complexity of the problem, itcan be challenging to optimize the use of resources by IBM Tivoli Identity Manager – or in other words, to tune. This tuning guide provides a system administrator with the information needed to tune the application for your environment. Other individuals (such as IBM DB2 or IBM Tivoli Directory Server administrators) in your organization might offer differing advice. In our experience, system administrators know your environment better and their advice may be more accurate for your environment than this tuning document.

The IBM Tivoli Identity Manager product can be divided into four major pieces: Java™ 2 Platform Enterprise Edition (J2EE) application servers, the IBM Tivoli Identity Manager application, database servers, and directory servers. We will address each of these separately in this document.

The IBM Tivoli Identity Manager server can be installed as either a single server or clustered servers. A clustered environment can be considered a group of single servers with regard to tuning.

This document is a working document. As more information is gathered settings may be added, removed or changed in future editions. It is recommended that you check the IBM web site for the most recent version. To find the most recent version, go to http://www.ibm.com/support/us. Select “Search technical support”, type “ITIM Tuning Guide” in the search box, and click submit.

IBM Tivoli Identity Manager configurations vary significantly by platform, operating system, middleware applications, and hardware being used. You must tailor your configuration settings for your deployment. Configurations and settings used in this document include AIX®, Windows®, and Solaris in single server and cluster configurations.

1.1 Vital tunings There are several thousand different parameters that you can modify to tune J2EE application servers, the IBM Tivoli Identity Manager product, directory servers, and database servers. This tuning guide discusses a small subset of these parameters that have proven effective during performance testing.

If you are setting up an acceptance or production environment, read each section and perform the applicable tunings for your systems. If you are setting up a test environment and want to get started as quickly as possible, focus on these areas:

IBM DB2 - Indexing

Indexing the following columns in the ITIM database was found during performance testing to significantly improve workflow performance. After adding these indexes you’ll need to run the runstats command on these tables to make sure the IBM DB2 optimizer knows to use them.

Determining the values

Check for the existence of these indexes using the following commands: db2 describe indexes for table ENROLE.PROCESS

db2 describe indexes for table ENROLE.PROCESSDATA

db2 describe indexes for table ENROLE.SCHEDULED_MESSAGE

Setting the values

Rrun the following IBM DB2 runstats commands: db2 create index PROCESS_SUB_X on ENROLE.PROCESS (“SUBMITTED” DESC, “PARENT_ID” ASC)

db2 create index PROCESS_ID_ST on ENROLE.PROCESS (“ID” ASC, “STATE” ASC)

db2 create index PROCESSDATA_ID_DEF on ENROLE.PROCESSDATA (“PROCESS_ID” ASC, “DEF_ID” ASC, “VALUE_LAST_MODIFIED” ASC)

db2 create index SCH_MSG_SVR on ENROLE.SCHEDULED_MESSAGE (“SERVER” ASC)


Runstats

IBM DB2 - Bufferpools

IBM WebSphere Application Server - Java Virtual Machine (JVM) size

The IBM DB2 - Indexing






Setting the values





Runstats tunings is a vital part of the IBM Tivoli Identity Manager product performance when IBM DB2 is used as the database server. See the Initial tunings section to tune this parameter in a newly deployed environment.

1.2 Initial tunings Most of these tunings can be implemented in a newly deployed environment or an environment that is already deployed. When tuning IBM DB2 in a newly deployed environment it is important to prime your database statistics using the IBM DB2 runstats command.

The runstats command creates information in the IBM DB2 statistics tables based on information currently in the data tables. However, in a pristine environment, those tables are empty and the statistics generated from runstats on these tables are useless to IBM DB2. Without adequate information on the distribution of data within these tables, IBM DB2 makes poor choices on how to fulfill the queries used by the IBM Tivoli Identity Manager application.

IBM Tivoli Identity Manager initializes the table statistics at configuration time so that IBM DB2 will make the best choices for a small startup configuration. As transactions are processed and the database grows, these initial statistics will no longer be enough to ensure good performance.

The solution to this problem is to prime your database by doing a small number of transactions, running the runstats command, and then running the full transaction. The IBM DB2 - Indexing



Check for the existence of these indexes using the following commands:


db2 describe indexes for table ENROLE.PROCESS



Setting the values





Runstats section discusses how to use the runstats command. Failing to prime the database in this manner can result in poor performance or transaction rollbacks.

Example: If the goal is to load 20,000 users into the IBM Tivoli Identity Manager product using a DSML feed, load the first 1000 people using the DSML file first. After the load completes, run the IBM DB2 runstats command and then load the remaining users with the DSML file.

1.3 Resource allocation Tuning values are more complex to manage when more than one piece of middleware is running on a given system. For example having the IBM Tivoli Identity Manager server, IBM DB2, and IBM Tivoli Directory Server all on the same machine. Regardless of configuration, it is important to calibrate the following resources so that they are not over-allocated.

1.3.1 Memory All middleware components allow you to adjust how much memory they will use. When calculating how to allocate memory to middleware components, keep these considerations in mind:

Configuring middleware memory settings too high can result in the operating system swapping memory out to disk if the physical memory is exceeded. This will result in extremely poor performance and should be avoided. After setting up or changing the memory values for the middleware, monitor the memory and swap space used to ensure that nothing is being swapped out to disk. If it is, adjust your memory settings to compensate.

32-bit processes can only allocate 2GB (or AIX) to 4GB (on Solaris) of RAM. If a 32-bit process is configured to allocate more than the OS-specific limit on process memory, the application (such as IBM Tivoli Directory Server) could halt or unexpectedly fail.

IBM Tivoli Directory Server has internal caches in addition to the IBM Tivoli Directory Server process. Ensure that the size of your caches plus the size of your IBM Tivoli Directory Server process does not exceed 2GB. When this 2GB limit is reached, the IDS server will refuse new connections and fail. The entry cache size limit determines the number of entries in the cache, not the size of the cache. The size of each cache entry will vary based on the IBM Tivoli Identity Manager product configuration and any extensions to the base IBM Tivoli Directory Server schema. The default cache values may exceed the 2GB limit in rare cases.

Although the bufferpools account for a large amount of the memory used by IBM DB2, the application control heaps, the sort heaps, and the statement heaps also use memory. Do not overlook these sizes when computing how much memory to allocate to IBM DB2. See the IBM DB2 section.

A large part of the WebSphere Application Server’s memory usage is the JVM size. However, the size of the JVM does not set an upper bound on the amount of memory that the WebSphere


Application Server may use. See the J2EE application servers - IBM WebSphere Application Server section.

1.3.2 CPU All the components of the IBM Tivoli Identity Manager product (IBM Tivoli Identity Manager application, J2EE application server, database server, and directory server) are CPU intensive. This document groups the J2EE application server and the IBM Tivoli Identity Manager application together because it is difficult to isolate their CPU usage. Keep in mind the following considerations:

In IBM Tivoli Identity Manager 4.6 role evaluation, policy enforcement and reconciliation have been enhanced by distributing work across multiple server threads and multiple TIM cluster members to complete these activities much faster than in previous TIM versions.

Both IBM Tivoli Directory Server and IBM DB2 are multi-threaded (multi-process in DB2’s case) applications and will perform best on a multi-processor machine.

Even in a well-tuned environment the system bottleneck(s) may vary between the CPU, memory and disk on the IBM Tivoli Identity Manager server, the directory server, and/or the database server. For this reason, the IBM Tivoli Identity Manager server, the directory server, and the database server can benefit by being deployed on separate machines. If that is not possible, put the database and directory server on the same machine with a high performance disk configuration using multiple disks and a high performance RAID configuration to provide fast read and write capacity.

1.3.3 Disk space Each of the middleware components uses different amounts of disk space for various purposes.

The J2EE application server and the IBM Tivoli Identity Manager application use disk space beyond their installation size because of log files (such as the msg.log and trace.log files) and WebSphere MQ queues. Adjust the number of archives and size of the msg.log and trace.log files in the enRoleLogging.properties file. Make sure that WebSphere MQ has enough disk space for its processing logs (not error logs) to grow. The IBM Tivoli Identity Manager server pushes many entries onto the queues during large provisioning changes, causing the queues to grow.

IBM Tivoli Directory Server uses disk space both from the IBM Tivoli Directory Server process (log files like ibmslapd.log) and the IBM DB2 database. IBM Tivoli Directory Server uses system managed space (SMS) tablespaces that allow the system to manage the amount of disk space used. SMS tablespaces do not allow the specification of an upper bound so it is important to monitor the amount of disk space used to ensure that the drive does not become full.

In addition to the tablespaces for the database data, IBM DB2 uses disk space for the transaction logs. When configuring the transaction logs (discussed in the IBM Tivoli Directory Server – Transaction logs and IBM DB2 – Transaction logs sections) ensure that you have enough disk space for the log files.

The IBM Tivoli Identity Manager DB2 database uses directory managed space (DMS) tablespaces that require pre-allocating disk space for the database to use. If the tablespace becomes full, IBM DB2 will be unable to continue and will return an error in the msg.log and trace.log files. If this occurs, add more tablespace containers. As a general rule, for every 160,000 transactions that occur, 1GB of tablespace is needed to archive the historical transaction information. A policy change that makes modifications to 5,000 users is 5,000 transactions from a IBM Tivoli Identity Manager perspective. See the IBM DB2 – Tablespaces section for additional information.


2 J2EE application servers

Regardless of the installation type (single machine or cluster), the IBM Tivoli Identity Manager server can be thought of as two components: the J2EE application server running the application (WebSphere Application Server or BEA WebLogic® Server) and the IBM Tivoli Identity Manager application itself. Both components need to be tuned.

2.1 IBM WebSphere Application Server WebSphere Application Server allows you to use a variety of settings to tune your environment. This document discusses the Java virtual machine (JVM) size, the maximum HTTP connections and Java Messaging Service queue size.

2.1.1 Java Virtual Machine (JVM) size By default, WebSphere Application Server sets the maximum JVM size to 256MB. This value is too small for the IBM Tivoli Identity Manager product to run beyond a basic concept test and should be increased to a minimum of 1GB. If your machine has adequate available RAM increase this value to as much as 1.5GB. For large reconciliations or role and policy evaluations the default values will not be enough memory to complete these tasks.

Setting the maximum memory value too high can cause memory allocation problems in JAVA when 2GB is reached. The maximum JVM size used in this value is not the actual maximum allocated size of the JAVA heap, and as much as 15% is allocated to a portion of the heap for the system’s use. IBM recommends that you not use a value higher than 1.5GB even if your system has the available memory.

Do not set the JVM heap size to be larger than the physical RAM. The WebSphere Application Server suffers significant performance degradation if the operating system swaps out the JVM to swap space. Consequences of this include very slow UI performance, transaction rollbacks, timeouts, and high disk utilization.

The WebSphere Application Server sets the initial JVM size to 0GB and IBM recommends using a value of 256MB. While previous JVM levels have worked well with the initial JVM size set the same as the maximum JVM size, setting these the same with version 1.4.2 of the JVM (the version embedded in WebSphere Application Server 5.1) can cause memory fragmentation resulting in poor performance.


initial_jvm_heap_size = The initial size of the JVM heap in megabytes. Recommended value: 256.

max_jvm_heap_size = The maximum size of the JVM heap in megabytes. Recommended value: 1GB to 1.5GB.

Setting the values

1) Open the WebSphere Administration Console.

2) Expand the Servers list in the navigation pane.

3) Select Application Servers in the navigation pane.

4) Select the server to manage.

5) Select Process Definition from the Additional Properties pane at the bottom.

6) Select Java Virtual Machine from the Additional Properties pane at the bottom.

7) Set the Initial Heap Size to initial_jvm_heap_size.

8) Set the Maximum Heap Size to max_jvm_heap_size.


9) Repeat this procedure for each IBM Tivoli Identity Manager server.

2.1.2 Java virtual machine (JVM) options – Sun Solaris Note: This section applies specifically to WebSphere Application Server V5.1 on Sun Solaris, not AIX or Windows. WebSphere Application Server V5.1 for AIX ships with IBM’s implementation of Java 1.4.2 whereas WebSphere Application Server V5.1 for Sun Solaris ships with Sun’s implementation of Java 1.4.2. IBM’s Java and Sun’s Java have two different garbage collection algorithms and need to be tuned differently. If you are using WebSphere Application Server on AIX or Windows, skip this section.

Sun’s 1.4.2 JVM on Sun Solaris supports two different JVM types: client and server. The client JVM is designed to have a faster startup time and is tuned for shorter-running applications than the server JVM. The server JVM is designed for longer-running applications and therefore sets different initial values to some JVM tunings. By default the WebSphere Application Server uses the client JVM. IBM Tivoli Identity Manager runs better under the server JVM than under the client JVM.

Sun’s 1.4.2 JVM uses a generational garbage collection algorithm. How this algorithm works is outside the scope of this document, but we will give a short summary. See the Other resources section at the end of this document for links to more information.

The following is a simplified view of how the generational garbage collection algorithm works. The JVM divides the Java heap size into two major components: the young generation, which includes Eden and two Survivor pools, and the old generation (sometimes called the tenured generation). When objects are allocated inside IBM Tivoli Identity Manager, they are created inside Eden. After Eden becomes full, a partial garbage collection occurs and any objects that are still being used (live objects) are moved to the first Survivor pool. Eden is cleared and object allocation begins again inside Eden. After Eden becomes full again, any live objects inside Eden and the first Survivor pool are moved to the second Survivor pool. Eden is cleared and object allocation begins again inside Eden. This process repeats with the objects being switched between the two survivor pools until objects die or are eventually tenured to the old generation. When there is no more space inside Eden or the Survivor pools, all remaining live objects are moved to the old generation. Eden is cleared and object allocation begins again. When the old generation becomes full, a full garbage collection occurs. Every object within the heap is analyzed and heap compaction occurs within the old generation.

When partial or full garbage collections occur everything within the JVM stops. This causes small pauses within IBM Tivoli Identity Manager during garbage collections. Partial garbage collections finish very quickly, usually in less than half a second. Full garbage collections take longer to compact the entire heap and can take 10 seconds or more to finish, depending on the size of the heap. Because of this, it is important to minimize the number of garbage collections, particularly full garbage collections. We do this by adjusting the size of the Java heap, the new generation, and the survivor pools.

A diagram illustrating the JVM heap layout:

Eden Tenured GenerationSurvivor pools

max_new_sizemax_jvm_size

new_ra tiosurvivor_ratio


jvm_type = The type of JVM to use, either -client or -server. Recommended value: -server

max_new_size = The maximum size of the new generation within the JVM in megabytes. Set this


to 1/3 of your maximum heap size.

new_ratio = The ratio of the new generation to the old generation. Set this to 2, thereby setting the size of your new generation to 1/3 of your heap size.

survivor_ratio = The ratio of a single survivor pool to Eden. Set this to 3, thereby setting the size of each survivor pool to 1/5 of your new generation (note this is not 1/4 because there are two survivor pools).

target_survivor_ratio = A target for the amount of survivor space to be used after a garbage collection. This value is used by the JVM to calculate for how many iterations an object will stay in the survivor pools before being tenured to the old generation. Set this to 75 to allow more time for objects to die in the survivor pool before being tenured.

With a total JVM heap size of 512MB, the recommendations above would yield the following heap configuration:

Eden Tenured GenerationSurvivorpools

170MB

512MB

102MB 34MB

342MB

Calculating the values

argument_string = jvm_type -XX:MaxNewSize=max_new_sizem -XX:NewRatio=new_ratio -XX:SurvivorRatio=survivor_ratio -XX:TargetSurvivorRatio=target_survivor_ratio

Note: Ensure the MaxNewSize value has a trailing ‘m’ specifying that the value is in megabytes.

Example: For a JVM size of 512MB, use the following argument string:

argument_string = -server -XX:MaxNewSize=170m -XX:NewRatio=2 -XX:SurvivorRatio=3 -XX:TargetSurvivorRatio=75

Setting the values

1) Open the WebSphere Administration Console.

2) Expand the Servers list in the navigation pane.

3) Select Application Servers in the navigation pane.

4) Select the server to manage.

5) Select Process Definition from the Additional Properties pane at the bottom.

6) Select Java Virtual Machine from the Additional Properties pane at the bottom.

7) Append argument_string to Generic JVM arguments

8) Repeat this procedure for each IBM Tivoli Identity Manager server.

2.1.3 IBM HTTP Server The WebSphere Application Server uses the IBM HTTP Server as a front-end server in a single-server installation and as a load balancer between nodes in a cluster installation. The default configuration parameters for the IBM HTTP Server work well for small and medium configurations but may need to be increased in environments with a large number of concurrent users.


2.1.3.1 Connections

The IBM HTTP Server can be configured to limit the number of connections that it will accept at one time. This value may be too small if the servers experience a large number of concurrent users. Connections are maintained to the TIM server until a user selects log-out (which doesn’t always happen) or until WebSphere clears the connection after 10 minutes.

The IBM HTTP Server supports the HTTP/1.1 KeepAlive request that allows a client to make multiple HTTP requests through a single persistent connection. There is a limit on the number of KeepAlive requests that a single connection will accept. After this limit is reached, the connection is closed and another connection must be established. The default value may be too small for some external provisioning processes such as a JNDI feed.


ibmhttp_home = The home directory of the IBM HTTP Server, such as /usr/IBMHttpServer.

max_connections (ThreadsPerChild on Windows) = The maximum number of connections that can be made to the HTTP server at one time. This should be set to the maximum number of concurrent users you expect on your system. Recommended value: 200. This may be increased to as high as 600 for customers with a large number of concurrent users.

Setting the values

Edit the ibmhttp_home/conf/httpd.conf file and update the following entries: MaxClients max_connections

Stop and restart the IBM HTTP Server for these changes to take effect.

2.1.4 IBM WebSphere MQ The WebSphere Application Server uses an embedded copy of the IBM MQSeries messaging service to manage Java Message Service (JMS) messages. The IBM Tivoli Identity Manager server configures several persistent queues (the exact name and number can vary slightly from version to version). The IBM Tivoli Identity Manager server uses these queues during various provisioning actions. During large provisioning policies it is possible to reach the limit on the number of uncommitted messages, so this value should be increased.


queue_manager_name = The name of the queue manager, such as WAS_vanexel_jmsserver. You can find a list of valid queue manager names by running dspmq.

max_uncommitted_messages = The maximum number of uncommitted messages. Default value: 10000. Recommended value: 50000.

Setting the values

1) Start the MQ Script interpreter: runmqsc queue_manager_name

2) Alter the maximum uncommitted messages size: alter qmgr maxumsgs(max_uncommitted_messages)

3) Confirm the change: display qmgr maxumsgs

4) Exit the MQ Script interpreter: end


3 IBM Tivoli Identity Manager application

The IBM Tivoli Identity Manager application includes several configuration files that provide an area for tuning various parts of the application’s performance. These are in the data/ directory under the IBM Tivoli Identity Manager product home directory.


4 Database servers

The IBM Tivoli Identity Manager product supports three different database systems: IBM DB2, Microsoft® SQL Server™, and Oracle® Database. Each database requires slightly different tunings.

Each database machine should have at least one processor and 1GB of RAM. It can reside on a single processor machine by itself or share a multi-processor machine with other applications, but the requirements of 1GB of RAM per one CPU is the minimum for the database server. Tuning the database is one of the most important tunings for the IBM Tivoli Identity Manager product.

4.1 IBM DB2 Tuning IBM DB2 to run with the IBM Tivoli Identity Manager product involves adjusting the bufferpools, modifying the number of connections, modifying internal database values, adding tablespace, adjusting logs, indexing, and running runstats.

4.1.1 Quick guide for setting the IBM DB2 tuning parameters This section includes the commands needed to tune the IBM DB2 parameters. This section assumes you have been through the tunings before, know what the desired values are, and just need a reminder of how to set them.

As the database administrator, connect to the database and run the following commands: db2 alter bufferpool ibmdefaultbp size ibmdefaultbp_npages db2 alter bufferpool enrolebp size enrolebp_npages db2 "ALTER TABLESPACE ENROLE_DATA ADD ( \ FILE '/database_home/database_name/NODE0000/SQL00001/container_name' num_pages) \ PREFETCHSIZE 16 OVERHEAD 24.1 TRANSFERRATE 0.9 BUFFERPOOL ENROLEBP"

db2 “ALTER TABLESPACE ENROLE_INDEXES ADD ( \ FILE '/database_home/database_name/NODE0000/SQL00001/container_name' num_pages) \ PREFETCHSIZE 16 OVERHEAD 24.1 TRANSFERRATE 0.9 BUFFERPOOL ENROLEBP”

db2 “ALTER TABLESPACE TEMP_DATA ADD ( \ FILE '/database_home/database_name/NODE0000/SQL00001/container_name' num_pages) \ PREFETCHSIZE 16 OVERHEAD 24.1 TRANSFERRATE 0.9 BUFFERPOOL ENROLEBP”

db2 update db cfg for itim_database using logsecond logs_secondary db2 update db cfg for itim_database using logfilsiz logs_size db2set DB2_RR_TO_RS=YES

4.1.2 Bufferpools IBM DB2 bufferpools are the secondary buffer for IBM Tivoli Directory Server. These bufferpools should be large enough that most table searches can be read directly from memory instead of using the disk. This value can be measured by looking at the hit ratio for the bufferpools. See the IBM DB2 performance monitoring section for information about calculating the bufferpool hit ratio.

The IBM Tivoli Identity Manager database has two bufferpools: IBMDEFAULTBP and ENROLEBP. IBMDEFAULTBP is used as a buffer for tablespaces with small extent sizes (4k) and ENROLEBP is used as a buffer for tablespaces with large extent sizes (32k). Most of the tables within the IBM Tivoli Identity Manager database use the tablespace with a large extent size and thus will use the ENROLEBP bufferpool. Use a 1:3 memory ratio between IBMDEFAULTBP and ENROLEBP.


First, determine the following values for your system:

total_mem_avail = The total amount of physical memory in bytes.

mem_for_itimdb = The amount of memory in bytes to allocate to the IBM Tivoli Identity Manager


database. This value should be small enough that it will reside in physical memory and not be swapped out to disk. Recommended value: 500000000 (500MB) or greater.

Next, calculate the following values:

ibmdefaultbp_npages = (mem_for_itimdb / 4096) * 0.25 enrolebp_npages = (mem_for_itimdb / 32768) * 0.75

Setting the values

As the database administrator, connect to the database and run the following commands: db2 alter bufferpool ibmdefaultbp size ibmdefaultbp_npages db2 alter bufferpool enrolebp size enrolebp_npages

4.1.3 Connections The IBM Tivoli Identity Manager server uses JDBC connections from the J2EE application server to communicate with the database. The number of connections from the application server to the database depends on the needs of the application. Both application servers allow you to set maximum connection values. Performance testing did not show any need to increase the maximum connection values from their default value of 50. The IBM DB2 variable MAXAPPLS must be set to a value greater than the maximum possible connections from all nodes. The default value for MAXAPPLS is AUTOMATIC, which is the recommended setting, however if you need to set it lower, we recommend setting MAXAPPLS to five more than the total maximum possible connections.


itim_database_name = The name of your IBM Tivoli Identity Manager database, such as itim.

num_connections = Maximum number of connections. See the J2EE application servers section to calculate the maximum number of connections and add five to that value.

Setting the values

As the database administrator, connect to the database and run the following command: db2 update db cfg for itim_database_name using maxappls num_connections

4.1.4 Tablespaces The IBM Tivoli Identity Manager product uses a Database Managed Space (DMS) tablespace to store its data. This type of tablespace gives better performance than System Managed Space (SMS) tablespaces but requires you to set aside disk space for the database to use. The default size of the IBM Tivoli Identity Manager tablespace is 512MB for ENROLE_DATA, 256MB for ENROLE_INDEXES, and 256MB for TEMP_DATA . This is frequently not large enough and should be increased by adding more containers to the tablespace.

Add more tablespaces by using the IBM DB2 alter tablespace command. If possible, add files that reside on another physical drive because IBM DB2 performs better if a tablespace has multiple containers on multiple drives. The creation and adoption of these altered tablespaces is not immediate. Examine the output of the alter tablespace command as it is executing and rerun the command if the database is busy altering a tablespace


database_home = the home directory of your database administrator, such as /home/db2inst1.

database_name = the name of the database such as db2inst1. This is also a subdirectory under database_home.

container_name = the name of the file to create to hold the tablespace container, such as enrole_data2.


num_pages = the number of 32k pages to add to the tablespace. To calculate the number of pages from the amount of disk space, divide the size in megabytes by 0.032768. A 512MB tablespace would be 15625 pages.

Setting the values

As the database administrator, connect to the database and run the following command: db2 "ALTER TABLESPACE ENROLE_DATA ADD ( \ FILE '/database_home/database_name/NODE0000/SQL00001/container_name' num_pages) \ PREFETCHSIZE 16 OVERHEAD 24.1 TRANSFERRATE 0.9 BUFFERPOOL ENROLEBP"

db2 "ALTER TABLESPACE ENROLE_INDEXES ADD ( \ FILE '/database_home/database_name/NODE0000/SQL00001/container_name' num_pages) \ PREFETCHSIZE 16 OVERHEAD 24.1 TRANSFERRATE 0.9 BUFFERPOOL ENROLEBP"

db2 "ALTER TABLESPACE TEMP_DATA ADD ( \ FILE '/database_home/database_name/NODE0000/SQL00001/container_name' num_pages) \ PREFETCHSIZE 16 OVERHEAD 24.1 TRANSFERRATE 0.9 BUFFERPOOL ENROLEBP"

4.1.5 Transaction logs IBM DB2 keeps logs during transaction processing. During large transactions the default log number and sizes might be too small and cause transaction rollbacks. Increasing the size and number of log files available to IBM DB2 solves this issue.

IBM DB2 has two types of log files: primary and secondary. Primary logs are allocated when the database is started and remain allocated until the database is stopped. Secondary logs are allocated as needed after the primary logs are full and are released after they are no longer needed.


Increase the number of secondary logs to be prepared when large transactions occur. The default size of the log files is 1000 4k pages, or 4MB. Increase this to 10000 4k pages, or 40MB. Note that this will increase the size of both your primary and secondary log files.

itim_database = The name of the IBM Tivoli Identity Manager database, such as itim.

logs_secondary = The number of secondary logs. Recommended value: 12.

logs_size = The size of the primary and secondary logs in 4k pages. Recommended value: 10000.

Setting the values

As the database administrator, connect to the database and run the following commands: db2 update db cfg for itim_database using logsecond logs_secondary db2 update db cfg for itim_database using logfilsiz logs_size

4.1.6 Indexing Indexing the following columns in the ITIM database was found during performance testing to significantly improve workflow performance. After adding these indexes you’ll need to run the runstats command on these tables to make sure the IBM DB2 optimizer knows to use them.





Setting the values

Rrun the following IBM DB2 runstats commands:


db2 create index PROCESS_SUB_X on ENROLE.PROCESS (“SUBMITTED” DESC, “PARENT_ID” ASC)




4.1.7 Runstats In order to efficiently execute queries, IBM DB2 requires information on the number of rows in the tables and what indexes are available. Run the runstats command on all tables in the database before and after large DSML loads and reconciliations. If you experience high CPU usage or poor IBM DB2 performance, run the runstats command on all the tables in the database. IBM DB2 reorgchk does not update index statistics and is not a replacement for runstats. To update index statistics, run the runstats command on each table individually. A script (perftune_runstats.sh) that detects the version of IBM DB2 and runs the runstats command against all tables for a given schema in a database is provided in Appendix A – scripts and files.

If the runstats command is run in a working environment, allow the connected applications to continue to write to the database. Use the "allow write access" option to allow users to write to a database while runstats is running.

Use the runstats command on an idle or lightly-used database because it requires update locking on the system statistics table to update the database statistics. A database with a heavy load might experience transaction rollbacks due to the system acquiring locks on the tables that are used by the database optimizer to fulfill queries.


Use the runstats command on every table in the ENROLE schema. As the database administrator, connect to the database and use the following command to get a listing of all tables in the schema:

db2 list tables for all | grep ENROLE

Setting the values

For each table in the ENROLE schema, run the following IBM DB2 runstats command: db2 runstats on table ENROLE.table_name with distribution and \ detailed indexes all allow write access

4.1.8 Lock tuning

4.1.8.2 Lock type

The IBM Tivoli Identity Manager application requires that RR_TO_RS locking be enabled. This setting allows select statements that involve rows that are deleted but not yet committed to be fulfilled without waiting for the transaction to complete.

More information on this environment setting is available at:

http://www.ibm.com/software/data/db2/udb/ad/db2irfp7/db2ir169.htm


There are no values to be determined for this section.

Setting the values

Run the following command as the IBM Tivoli Identity Manager database administrator: db2set DB2_RR_TO_RS=YES


Stop and restart the database instance for this change to take effect.

4.1.8.3 Lock timeout

The IBM Tivoli Identity Manager application can generally be classified an online transaction processing (OLTP) application. As such it is normally recommended that the lock timeout (locktimeout) value within IBM DB2 be decreased from the default value of -1, which represents infinity. However, it is strongly encouraged that you do not modify this from the -1 value. Not all components of the application recover from a lock timeout and making such a change can result in transaction rollback errors.

4.2 Microsoft SQL Server Steps to tune Microsoft SQL Server will be included here when those settings and their values are determined.

4.3 Oracle Database

4.3.1 Server Configuration The default Oracle configuration uses a low performance setting, increasing the configuration to the middle performance setting provides faster Oracle performance. Consult with an Oracle DBA, Oracle administrator, or Oracle documentation for more information on tuning of the Oracle server.

4.3.2 Updating Statistics At regular intervals from 1 week to 1 month on a production IBM Tivoli Identity Manager system, or after a large amount of processing has been completed, database statistics should be gathered and updated. These statistics are used by Oracle to make query decisions on locating information that have a dramatic impact on how fast Oracle can return requests. The DBMS_STAT package must be installed to issue the following DBMS_STAT commands.


database_instance = The name of the database instance such as enrole.

Setting the values

Create a file named Oracle_dbms.stat_cmds.txt containing the following lines: exec dbms_stats.gather_schema_stats(ownname => 'database_instance', cascade => true);

As the system user, execute sqlplus and run the following command: @ Oracle_dbms.stat_cmds.txt

Generating statistics may take from several minutes to several hours for a large database. Execution should be issued during off-peak times.


5 Directory servers

The IBM Tivoli Identity Manager product supports two different directory servers: IBM Tivoli Directory Server and Sun ONE Directory Server (formerly known as Netscape iPlanet Directory Server).

5.1 IBM Tivoli Directory Server With respect to tuning, IBM Tivoli Directory Server can be divided into two parts: the IBM Tivoli Directory Server process and IBM DB2.

In a well-tuned system, the IBM Tivoli Directory Server process and the IBM DB2 processes consume approximately the same amount of CPU. In a poorly tuned system, IBM DB2 can max out the CPU usage trying to fulfill queries in a poor manner.

Both IBM Tivoli Directory Server and IBM DB2 have caches that speed up data retrieval. Optimizing your available memory is the key to tuning IBM Tivoli Directory Server. When a read request comes in to IBM Tivoli Directory Server, it checks the filter cache to see if has seen that search filter before. If it has, the results are pulled from the cache, otherwise the query goes to IBM DB2. After the search filter is evaluated, IBM Tivoli Directory Server pulls the entries that match the search filter from the entry cache. If the values are not in the entry cache it queries IBM DB2. For each request to IBM DB2, IBM DB2 checks to see if the data resides in a bufferpool. If not, it reads the value from the disk. Ideally, all requests to the directory server register an IBM Tivoli Directory Server cache hit or a IBM DB2 bufferpool hit for the quickest response. Queries that require disk access can be very slow.

5.1.1 Quick guide for setting the IBM Tivoli Directory Server tuning parameters

This section includes the commands needed to tune the IBM Tivoli Directory Server parameters. This section assumes you have been through the tunings before, know what the desired values are, and just need a reminder of how to set them.

As the database administrator, connect to the database and run the following commands: db2 alter bufferpool ibmdefaultbp size ibmdefaultbp_npages db2 alter bufferpool ldapbp size ldapbp_npages db2 update db cfg for ldap_database using logsecond logs_secondary db2 update db cfg for ldap_database using logfilsiz logs_size

Stop IBM Tivoli Directory Server, edit ibmslapd.conf and update the following configuration options: ibm-slapdDbConnections: 30

Stop and restart IBM DB2 and the IBM Tivoli Directory Server server for these changes to take effect.

5.1.2 Bufferpools IBM DB2 bufferpools are the secondary buffer for IBM Tivoli Directory Server. These bufferpools should be large enough that most table searches can be read directly from memory instead of using the disk. This value can be measured by looking at the hit ratio for the bufferpools. See the IBM DB2 performance monitoring section for information about calculating the bufferpool hit ratio.

The IBM Tivoli Directory Server database (ldapdb2 by default) has two bufferpools: IBMDEFAULTBP and LDAPBP. IBMDEFAULTBP is used as a buffer for tablespaces with small extent sizes (4k) and LDAPBP is used as a buffer for tablespaces with large extent sizes (32k). Most of the tables in the ldapdb2 database use the tablespace with a small extent size and use IBMDEFAULTBP. Use a 3:1 memory ratio between IBMDEFAULTBP and LDAPBP.



Allocate enough memory to the IBM DB2 bufferpools so the bufferpool hit ratio is greater than 95% and allocate the rest of the memory to the IBM Tivoli Directory Server process and the caches. The following formula can help in determining the bufferpool values for your machine.

First, determine the values for your system:

ldap_database = the name of the IBM Tivoli Directory Server database, such as ldapdb2.

mem_for_ldapdb2_bps = the amount of memory in bytes to allocate to the ldapdb2 bufferpools. This value should be small enough that it will reside in physical memory and not be swapped out to disk. Recommended value: 500000000 (500MB) or greater.

Next, calculate the following values:

ibmdefaultbp_npages = (mem_for_ldapdb2_bps / 4096) * 0.75 ldapbp_npages = (mem_for_ldapdb2_bps / 32768) * 0.25

Setting the values

As the database administrator, connect to the database and run the following commands: db2 alter bufferpool ibmdefaultbp size ibmdefaultbp_npages db2 alter bufferpool ldapbp size ldapbp_npages

5.1.3 Connections There are two connection values to modify for IBM Tivoli Directory Server. The first value is the number of connections that IBM Tivoli Directory Server should make to the IBM DB2 backend. Set this to 30.

The second value only applies to Windows systems. Windows allows up to 64 sockets in a select call for each thread. This limits the number of inbound connections to 64. To overcome this problem set the SLAPD_OCHANDLERS environment variable to specify the number of connection threads, each one supporting 64 connections.


ldap_database_name = The name of your IBM Tivoli Directory Server database, such as ldapdb2.

num_ldap_connections = the number of connections from IBM Tivoli Directory Server to the IBM DB2 backend. Recommended value: 30 (from the IBM Tivoli Directory Server Tuning Guide).

For Windows systems, also determine the following values:

max_num_LDAP_connections = the maximum number of IBM Tivoli Directory Server connections (enrole.connectionpool.maxpoolsize) from the enRole.properties file. The IBM Tivoli Identity Manager application default is 100.

itim_nodes = the number of IBM Tivoli Identity Manager nodes in your cluster. This is the sum of the WF and the UI components.

oc_handlers = (max_num_LDAP_connections * itim_nodes) / 64

Setting the values

To set the number of connections the IBM Tivoli Directory Server server will make to the IBM DB2 server, stop IBM Tivoli Directory Server, edit ibmslapd.conf, and find the stanza:

dn: cn=Directory, cn=RDBM Backends, cn=IBM Directory, cn=Schemas, cn=Configuration

Change the following attribute: ibm-slapdDbConnections: num_ldap_connections

For Windows systems, also set the SLAPD_OCHANDLERS values. Stop IBM Tivoli Directory Server, edit ibmslapd.conf and find the stanza:


dn: cn=Front End, cn=Configuration

Add the following configuration option at the end of the stanza before the blank line: ibm-slapdsetenv: SLAPD_OCHANDLERS=oc_handlers

Stop and start the IBM Tivoli Directory Server database and restart IBM Tivoli Directory Server for these changes to take effect.

5.1.4 Transaction logs IBM DB2 keeps logs during transaction processing. During large transactions the default log number and sizes might be too small and cause transaction rollbacks. This is solved by increasing the size and number of log files available to IBM DB2.

IBM DB2 has two types of log files: primary and secondary. Primary logs are allocated when the database is started and remain allocated until the database is stopped. Secondary logs are allocated as needed after the primary logs are full and are released after they are no longer needed.


Increase the number of secondary logs to be prepared when large transactions occur. The default size of the log files is 1000 4k pages, or 4MB. Increase this to 10000 4k pages, or 40MB. Note that this will increase the size of both your primary and secondary log files.

ldap_database = the name of the IBM Tivoli Directory Server database, such as ldapdb2.

logs_secondary = the number of secondary logs. Recommended value: 12.

logs_size = the size of the primary and secondary logs in 4k pages. Recommended value: 10000.

Setting the values

As the database administrator, connect to the database and run the following commands: db2 update db cfg for ldap_database using logsecond logs_secondary db2 update db cfg for ldap_database using logfilsiz logs_size

5.1.5 Indexing Indexing the attributes that applications search on increases IBM Tivoli Directory Server performance. IBM Tivoli Directory Server indexes are automatically translated into IBM DB2 indexes by updating the IBM Tivoli Directory Server schema for those attributes.

If you extend the LDAP schema in IBM Tivoli Directory Server to include additional attributes, index those attributes that you will search for. Any filter in the IBM Tivoli Identity Manager application (such as with dynamic roles) is translated into a search string for the IBM Tivoli Directory Server.

The IBM Tivoli Identity Manager application frequently searches against the O, OWNER and OU attributes. After updating the LDAP schema, run IBM DB2 runstats on the database to update the statistics for the newly created indexes.

The “Indexes” sections of the IBM Tivoli Directory Server Tuning Guides (see reference) describe two procedures for indexing an attribute. If those do not work, the following is a third approach.


Check IBMAttributestypes = lines in the <IDS directory>/etc/v3.user.at file for the following three indexes: O, OU and OWNER. If the text ‘EQUALITY’ does not appear after the LENGTH value and before the ‘)’ than the attributes have not been indexed.

Setting the values

Shut down the IDS server. Modify the v3.user.at file for each of the attribute(s) that does not have the text ‘EQUALITY’ on the line between the number value and the ‘)’, and add ‘EQUALITY’ to each of the lines. Examples of the 3 modified lines as follows: IBMAttributetypes=( 2.5.4.32 DBNAME( 'owner' 'owner' ) ACCESS-CLASS normal LENGTH 1000 EQUALITY )


IBMAttributetypes=( 2.5.4.10 DBNAME( 'o' 'o' ) ACCESS-CLASS normal LENGTH 128 EQUALITY )

IBMAttributetypes=( 2.5.4.11 DBNAME( 'ou' 'ou' ) ACCESS-CLASS normal LENGTH 128 EQUALITY )

Save the new v3.user.at file, restart the IDS server and login to the IBM Tivoli Identity Manager login page. The first login may take several minutes to complete and may return an error, this is a result of the IDS server adding the new index(es).

5.1.6 Runstats The number of rows in the tables and what indexes are available are required for IBM DB2 to efficiently fulfill queries. Run the runstats command on all tables in the database before and after large DSML loads and reconciliations. If you experience high CPU usage or poor IBM DB2 performance, run the runstats command on all the tables in the database. IBM DB2 reorgchk does not update index statistics and is not a replacement for runstats. To update index statistics, run the runstats command on each table individually. A script (perftune_runstats.sh) that detects the version of IBM DB2 and runs the runstats command against all tables for a given schema in a database is provided in Appendix A – scripts and files.

If the runstats command is run in a working environment, allow the connected applications to continue to write to the database. Use the "allow write access" option to allow users to write to a database while runstats is running.

Use the runstats command on an idle or lightly-used database because it requires update locking on system statistics table to update the database statistics. A database with a heavy load might experience transaction rollbacks due to the system acquiring locks on the tables that are used by the database optimizer to fulfill queries.

In addition to running runstats on all tables within the database, we also manually update the statistics table for the LDAP_DESC and LDAP_ENTRY tables. The choices IBM DB2 makes about when to uses these tables for fulfilling queries for the IBM Tivoli Directory Server are not idea for IBM Tivoli Identity Manager. By manually adjusting the statistics table, we can help IBM DB2 make better choices and use these tables at the end of the access plan instead of the beginning.


As the database administrator, connect to the database and use the following command to get a listing of all tables in the schema:

db2 list tables for all | grep LDAPDB2

Setting the values

For each table in the LDAPDB2 schema, run the following DB2 runstats command: db2 runstats on table ENROLE.table_name allow write access

Manually update the database statistics table for the LDAP_DESC and LDAP_ENTRY tables. db2 update sysstat.tables set card = 9E18 where tabname = 'LDAP_DESC' db2 update sysstat.tables set card = 9E18 where tabname = 'LDAP_ENTRY'

5.2 Sun ONE Directory Server The Sun ONE Directory Server (formerly known as Netscape iPlanet Directory Server) does not provide many parameters to adjust for better performance. The three areas that provide the best performance boost are indexing, the All IDs value, and the database cache size.

5.2.1 All IDs Threshold Value The Sun ONE Directory Server sets an upper bound on how many index entries should be allowed for each attribute indexed. If this value is exceeded, the index is invalidated and is not used during searches. This behavior prevents the server from including all entries in an index and consuming excessive amounts of memory.


The Sun ONE Directory Server Version 5.2 Installation and Tuning Guide recommends setting this value to be 5% of the size of your directory. The IBM Tivoli Identity Manager product uses the directory server in a dramatically different manner than the writers of SunOne Tuning Guide had in mind and this value should be increased to the largest number of users, groups, services, or accounts the ITIM server has stored in the directory.

Because the method to change this parameter is non-trivial, it is recommended that you set the value based on the estimated size of your directory in its final form. If the current size of your directory is 20,000 but you expect the final size of your directory to be 100,000 entries, use the larger value to calculate the All IDs Threshold value.

Please refer to the Sun ONE Directory Server Version 5.2 Installation and Tuning Guide for more information on adjusting the All IDs Threshold value.


directory_home = The Sun ONE Directory Server home directory, such as /usr/iplanet/servers/slapd-gso-2DS.

number_of_entries = The number of entries in your directory. A good way to estimate this number is to multiply the number of IBM Tivoli Identity Manager users by the average number of accounts for each user.

suffix_name = The name of the root suffix, such as dc=com.

database_name = The name of the directory server database, such as itim5145.

temp_directory = The name of a temporary directory, such as /tmp. This directory should have enough free space to store an LDIF of your entire directory. Allow 1KB per IBM Tivoli Identity Manager user.

Calculating the values

all_ids = number_of_entries

Setting the values

Note: Setting this value requires the database to be rebuilt (the data exported and re-imported). If done incorrectly, this process has the potential to destroy information within your directory. It is highly recommended that you back up the information in your directory prior to setting this value. The db2ldif utility will export the data and the ldif2db utility will import your data and build your new indexes

Note: Increasing the size of the All IDs Threashold will cause the size of the indexes to grow. Because the indexes are stored in the database cache, the cache size might need to be increased as well. See the Sun ONE Directory Server - Database cache size section for information about increasing this value.

Edit directory_home/config/dse.ldif file and find the stanza: dn: cn=config,cn=ldbm database,cn=plugins,cn=config

Change the following attribute: nsslapd-allidsthreshold: all_ids

You will need to export and re-import all of your data for this parameter to take effect. This can be done with the db2ldif and ldif2db commands.

First, export your data: db2ldif -n database_name -a /full/path/to/ldif/export -s "suffix_name"

Next, stop the directory server.

Finally, import your data: ldif2db -n database_name -i /full/path/to/ldif/export

Start the directory server.


5.2.2 Indexing Indexing the attributes that applications search on increases Sun ONE Directory Server performance.

If you extend the LDAP schema to include additional attributes, index those attributes that you will search for. Any filter in the IBM Tivoli Identity Manager application (such as with dynamic roles) is translated into a search string for the LDAP server.

The IBM Tivoli Identity Manager application frequently searches against the O, Owner and OU. Owner is indexed by default in Sun ONE Directory Server but OU and O is not. Index the OU and O attribute from the directory server configuration console.


There are no values to be determined for this section.

Setting the values

1) Open the Directory Server Console.

2) Select the Configuration tab.

3) Expand the Data node in the navigation pane.

4) Expand the suffix of your database in the navigation pane and select the database.

5) Select the Indexes tab.

6) Select the Add attribute… button.

7) Find OU in the list box and click OK.

8) If the Equality and Presence options are not already selected for the OU attribute, select them.

9) Click Save.

10) A progress dialog will appear while the index is added. Click Close after the index creation completes.

5.2.3 Database cache size The Sun ONE Directory Server uses a database cache to store index information, filter requests, and entry requests. The directory server uses a database cache of 10MB by default. This value is too small for any but the smallest directory.

Setting this value greater than the amount of physical RAM available will cause performance degradation as the system swaps the pages out to disk.


cache_size = The size of the database cache. Recommended value: 256000000 (256MB). If the entry cache hit ratio on the Status tab for the database is below 95%, increase this value in increments of 128MB until the hit ratio is above 95% or your system is out of physical memory.

Setting the values

1) Open the Directory Server Console.

2) Select the Configuration tab.

3) Expand the Data node in the navigation pane.

4) Expand the suffix of your database in the navigation pane and select the database.

5) Select the Database Settings tab.

6) Set the Memory available for cache to cache_size.

7) Click Save.


6 Operating Systems

Performance can be improved on some systems by making some operating system specific changes.

6.1 Red Hat Update the msgmni kernel variable to 1026


msgmni_value = 1026 The default is 16.

Setting the values

As root, run the following command: sysctl -w kernel.msgmni=msgmni_value

Update the open file limit to 8000


ulimit_value = 8000 The default is 2000.

Setting the values

As root, run the following command: ulimit -n ulimit_value

and change the value of nofiles in /etc/security/limits to 8000


7 Best practices

The IBM Tivoli Identity Manager product can be set up and configured in many ways. The following are some best practice suggestions to help guide you in setting up your environment.

Each agent modifies the LDAP schema by adding new attributes to support the new service. These attributes are created without indexes and for services that service thousand of users a large benefit can be achieved by adding indexes to attributes with many members.

In general, network latency is not a major performance bottleneck, but it is still a good idea to have as few hops as possible between components. This includes the IBM Tivoli Identity Manager server components (WF/UI), the directory server, database server, agents, and agent endpoints. Ideally all components would be on the same subnet or no more than one hop away and on 100MBit or faster network.

To minimize disk bottlenecks, it is frequently better to have several hard drives in the system rather than one large drive. IBM DB2 can make use of multiple drives in the system if it is configured to do so. The IBM Tivoli Identity Manager product itself can benefit from having the log files on a different drive than the messaging queues. Keep in mind that high-end I/O backplanes or other advanced storage systems can overcome this by balancing the load across multiple drives automatically.

Database and directory activity can be very CPU and memory intensive. It is recommended that each application have at least one processor and 2GB of RAM each. More processors are generally better. To obtain optimal performance, it is not recommended to have all IBM Tivoli Identity Manager components on the same system unless it is a very powerful unit such as an 8 x 1.4GHz machine with 12GB RAM.

Complicated provisioning policies can result in complicated directory and database queries with poor performance. Policies with small numbers of roles and services will perform best.


8 Troubleshooting

The large number of middleware dependencies can complicate the task of finding performance problems with IBM Tivoli Identity Manager. A problem such as a slow DSML feed with account provisioning could be the result of a slow directory server, database locking issues, or an inadequate number of IBM Tivoli Identity Manager workflow threads. This section is designed to assist you in identifying problem areas and some pointers on fixing them.

Information in this section is provided with the assumption that you have read and applied the tunings recommended in the previous sections. If this is not the case, stop and review those sections now.

8.1 Log files When something seems to stop working, the first place to check is the msg.log and trace.log files. They often contain useful information about what is currently being processed and any errors that may have occurred. If you suspect something isn’t working correctly and the msg.log and trace.log files don’t contain enough information, increase the logging in the enRoleLogging.properties file. Edit the file and either uncomment or add one or more of the following lines: logger.trace.com.ibm.itim.workflow.level=DEBUG_MAX logger.trace.com.ibm.itim.remoteservices.level=DEBUG_MAX

The workflow class is used during all workflow processes. Setting this to DEBUG_MAX can help track down workflow problems. The remoteservices class communicates with external agents (such as a Windows NT service and a Tivoli Access Manager service but not the ITIM Service) during provisioning. Set this to DEBUG_MAX to help diagnose problems with agents. Changes to the enRoleLogging.properties file are picked up every 10 minutes or whenever the IBM Tivoli Identity Manager application is restarted.

Also, be sure to check the log file for the appropriate middleware components:

IBM WebSphere Application Server - /usr/WebSphere/AppServer/logs/server_name/

BEA WebLogic – weblogic_home/user_projects/itim/server_name/server_name.log

IBM Tivoli Directory Server - /var/ldap/ibmslapd.log

Sun ONE Directory Server – sun_one_home/logs/access

IBM DB2 – instance_home/sqllib/db2dump/db2diag.log

Oracle Database - *.trc files

Microsoft SQL Server – mssql_home\LOG\

8.2 Identifying bottlenecks Start by monitoring the processor and disk usage of every server (WAS/ITIM, directory, database) to see which server is most heavily utilized. Based on this information, review the monitoring and tuning steps specific to that component.

The IBM Tivoli Identity Manager application makes intense usage of both the database and the directory server it is configured into. The database is used as an information staging area and audit trail for provisioning actions. The directory server is used as a permanent storage location and can be heavily queried when evaluating provisioning policies. Because of this usage it is possible for either the database or the directory server to be a bottleneck during heavy usage or large provisioning changes.


During an action that causes a large provisioning policy to be evaluated, such as adding a new policy or modifying an existing policy that impacts a large number of users, the affected users must be queried from the directory server. The directory server evaluates the query and returns the matching users. If the directory server is not tuned correctly it can become the bottleneck as the IBM Tivoli Identity Manager server waits for the result set before starting the required provisioning action. Ensure the directory server is fulfilling the requested queries as quickly and efficiently as possible to minimize this behavior. Refer to the appropriate directory server tuning section for more information.

After the result set is returned, the IBM Tivoli Identity Manager server will begin to enforce the provisioning policy for each user. This process is multi-threaded and benefits from multiple processors (the workflow cluster member in a functional cluster). If it appears that the processors on the server are not being fully utilized, check to see if there is a bottleneck on the LDAP or database server.

Any enforcement action required for a user (account addition, modification, or deletion) is stored as a workflow item in the database. When a thread becomes available, it queries the database for the next workflow item that needs processing and proceeds to act upon that item. This usage can result in access contention and thus locking in the database. Appropriate indexes and access-plan statistics will help minimize the number of locks needed to fulfill these requests. Refer to the appropriate database tuning section for more information.

8.3 Specific problems

8.3.1 IBM Tivoli Identity Manager application Problem: Transaction rollback errors in the trace.log file. Solution: Transaction rollbacks can occur for several different reasons, most of them database-related. There should be an error message within the trace.log file giving more information about what went wrong. Some areas to check when you get a transaction rollback:

Database storage space. If the database runs out of storage space for the tablespaces, a transaction rollback error can occur. Increase the amount of disk space allocated to the tablespaces to resolve this problem.

Database locking issues. If the database encounters extreme locking issues it is possible to get a transaction rollback error. Confirm that the locks are tuned appropriately and update the table statistics.

Database memory issues. If there is not enough memory available to database structures to fulfill the requested query, a transaction rollback error may occur. The JNDI error in the trace.log file should indicate which database heap should be increased. Increase the appropriate heap using the recommendations provided in the section for that middleware.

Problem: Out of Memory errors in the trace.log file. Solution: The message is from the J2EE Application Server. The Java virtual machine (JVM) ran out of heap size. Increase the maximum heap size if possible and restart the application server. If the heap size is already at the limit, then you may need to break up transactions, e.g. use fewer services or roles in a provisioning policy.

8.3.2 IBM WebSphere MQ Problem: Nothing seems to be going through workflow, even after restarting the application servers, and new items are piling up on the queue. Solution: A message on the message queue may be corrupt and blocking new transactions. Clearing the queues may help.


Note: Clearing the queues can result in an inconsistent system state. It is recommended that you only resort to clearing your queues when instructed to do so by a support technician.

1. Stop the IBM Tivoli Identity Manager application. You can’t clear the queues while the IBM Tivoli Identity Manager application is running.

2. Find the name of the queue manager by listing all queues on the system: dspmq

3. Start the MQ Script interpreter: runmqsc WAS_machinename_jmsserver

4. List IBM Tivoli Identity Manager queues: display queue('WQ_itim*')

5. For each queue, clear it: clear qlocal('WQ_itim_name')

Note: If you receive a message that the queue is in use, stop the IBM Tivoli Identity Manager server and try again.

6. Exit the MQ Script interpreter: end

8.3.3 IBM Tivoli Directory Server Problem: Poor search performance. Solution: Slow queries or high CPU utilization with the IBM Tivoli Directory Server can be caused by several factors. Some areas to consider:

1. Check for long-running queries that need indexes. The IBM Tivoli Directory Server uses IBM DB2 to process LDAP queries. By checking IBM DB2 for long-running queries it is possible to find what attributes may need indexing. To find how long each query takes, turn on statement cache monitoring within IBM DB2: db2 update dbm cfg using DFT_MON_STMT ON Stop the directory server, restart the database, and restart the directory server for this to take effect. After the monitoring is turned on, duplicate the suspect action within IBM Tivoli Identity Manager. After the action has completed, get a snapshot of the statement cache: db2 get snapshot for dynamic sql on database_name The output will contain stanzas like the following: Number of executions = 12 Number of compilations = 1 Worst preparation time (ms) = 3 Best preparation time (ms) = 3 Internal rows deleted = 0 Internal rows inserted = 0 Rows read = 10024 Internal rows updated = 0 Rows written = 0 Statement sorts = 0 Total execution time (sec.ms) = 136.000663 Total user cpu time (sec.ms) = 62.010000 Total system cpu time (sec.ms) = 10.000000 Statement text = SELECT distinct E.EID FROM LDAPDB2.LDAP_ENTRY AS E, LDAPDB2.LDAP_ENTRY as pchild WHERE E.EID=pchild.EID AND pchild.PEID=? AND E.EID IN (SELECT EID FROM LDAPDB2.OU WHERE OU = ?)

Calculate the average execution time per query by dividing the total execution time by the number of executions: total execution time / number of executions. In the example above: 136 / 12 =


11.33 seconds per execution. Most queries should take a second or less to complete. Queries that take longer may be searching on columns that are not indexed by IBM DB2, and hence are not indexed within the IBM Tivoli Directory Server. Another symptom of this problem a high average number of rows read, calculated by dividing the rows read by the number of executions. In the above example the column OU is likely not indexed. Correct this by indexing the OU attribute within the IBM Tivoli Directory Server. See the IBM Tivoli Directory Server - Indexing section for more information. The IBM Tivoli Identity Manager tuning scripts contain a script (perfanalyze_dynamicsql.pl) that will calculate the time per execution for all stanzas and sort the results. See Appendix A – scripts and files for information on how to get the tuning scripts.

2. Check for low bufferpool hit ratio. The IBM Tivoli Directory Server uses IBM DB2 to process LDAP queries. The IBM Tivoli Directory Server database should have a high (>95%) hit ratio. If the bufferpools are not large enough IBM DB2 will need to read more information from the disk resulting in high I/O wait. See the IBM DB2 performance monitoring - Bufferpool hit ratio section for information on calculating the bufferpool hit ratio. The IBM Tivoli Identity Manager tuning scripts contain a script (perfanalyze_bufferpools.pl) that will calculate the hit ratio for all bufferpools for you. See Appendix A – scripts and files for information on how to get the tuning scripts.

Problem: The directory server crashes or hangs for no apparent reason. Solution: Check the size of your cache sizes. If your cache sizes cause the IBM Tivoli Directory Server process to grow beyond what is supported by your operating system’s memory model (in general around 2GB on 32-bit operating systems) it can crash. See the IBM Tivoli Directory Server Tuning Guide (in Other resources) for more information.

8.3.4 Sun ONE Directory Server Problem: Poor search performance. Solution: Slow queries or high CPU utilization with the Sun ONE Directory Server can be caused by several factors. Some areas to consider:

1. Check for long-running queries that need indexes. Most queries should take less than one second to complete. Queries taking longer than a second are generally searching on attributes that are not indexed. To find queries taking longer than a second, search the Sun ONE Directory server access logs for etime=X where X is an integer greater than 1, X representing the number of seconds the query took to complete. We are searching for queries running 2 seconds or longer. For example: [26/Mar/2004:12:54:25] conn=4236 op=1 msgId=2 - RESULT err=0 tag=101 nentries=1 etime=2

Now search for what query the search time is associated with by searching the same file for the matching connection and operation number, which in the above example is conn=4236 op=1. The query for the above example was: [26/Mar/2004:12:54:24] conn=4236 op=1 msgId=2 – SRCH base="ou=ibm,dc=com" scope=2 filter="(&(!(erIsDeleted=y))(&(erEnabled=true)(erPolicyMembership=2;*))(objectClass=erProvisioningPolicy))" attrs=ALL

Examine the LDAP query string and identify the attributes being searched on. In the above example they are erIsDeleted, erEnabled, erPolicyMembership, and objectClass. Ensure that all attributes being searched on are correctly indexed. See the Sun ONE Directory Server - Indexing section for information on how to index attributes.


2. See if the All IDs Threshold value needs to be increased. If the All IDs Threshold value is too small, Sun ONE Directory Server may not be using the indexes for certain attributes. Determine if queries are searching on these attributes by searching for notes=U in the Sun ONE Directory Server access logs. The presence of this flag for a query indicates that one or more attributes being searched on is over the All IDs Threshold and should probably be increased. See the Sun ONE Directory Server - All IDs Threshold Value section for more information on how to increase this value.

3. See if the database cache hit ratio is low. If the Sun ONE Directory Server database cache is too small, the server must access the disk for the information and will result in a low database cache hit ratio. Identify the hit ratio for the database by viewing the Monitor tab for that database. The hit ratio should be 95% or higher. If it is lower than 95%, increase the size of your database cache. See the Sun ONE Directory Server - Database cache size section for information on increasing the size.


9 IBM DB2 performance monitoring

IBM DB2 provides several tools for troubleshooting and analyzing performance problems. This section contains a primer for how to get started monitoring with IBM DB2. For more extensive information, refer to the IBM DB2 Administration Guide: Performance book.

9.1 Enable monitoring To gather performance information, turn on the IBM DB2 monitoring flags. To turn on monitoring, run the following commands as the database administrator. Note that this should be done for each database.

db2 update dbm cfg using DFT_MON_STMT ON db2 update dbm cfg using DFT_MON_BUFPOOL ON db2 update dbm cfg using DFT_MON_LOCK ON db2 update dbm cfg using DFT_MON_SORT ON db2 update dbm cfg using DFT_MON_TABLE ON db2 update dbm cfg using DFT_MON_TIMESTAMP ON db2 update dbm cfg using DFT_MON_UOW ON

Stop and restart the database instance for the monitoring to take effect.

9.2 Snapshots Snapshots allow you to view the internal state of various IBM DB2 components. The following commands access important IBM DB2 snapshots:

db2 get snapshot for database on database_name db2 get snapshot for dynamic sql on database_name db2 get snapshot for bufferpools on database_name db2 get snapshot for tables on database_name db2 get snapshot for locks on database_name

9.3 Bufferpool hit ratio The bufferpool hit ratio gives a good indication of how many data reads are coming from the bufferpool and how many are coming from the disk. The larger the hit ratio, the less disk I/O used. Calculate the bufferpool hit ratio by enabling bufferpool monitoring and taking a database snapshot. Calculate the bufferpool hit ratio with the following formula:

P = buffer pool data physical reads + buffer pool index physical reads L = buffer pool data logical reads + buffer pool index logical reads Hit ratio = (1-(P/L)) * 100%

9.4 IBM DB2 Statement Monitor The IBM DB2 statement monitor is useful for examining what is occurring for each request sent to the ITIM or IDS DB2 database. While the statement monitor is turned on, each SQL request is recorded and the results of the IBM DB2 queries can be examined looking for missing indexes, db2 execution time, db2 preparation time, database scans and index scans. The monitor collects a great deal of information and should only be activated for a short time to gather requests.

Steps:

First, become the instance user, e.g ldapdb2, and connect to the database: db2 connect to ldapdb2

If this is the first time you've done any monitoring or an explain on this database you'll need to do some one-time setup:

db2 -tf sqllib/misc/EXPLAIN.DDL


db2 "create event monitor dstatement for statements write to file '/tmp/dstatements'" mkdir /tmp/dstatements

Now to monitor the IBM DB2 statements that are run under the covers, first clear out any previous data and turn on the monitor:

rm -f /tmp/dtatements/* db2 "set event monitor dstatement state 1"

Now run the slow query and then turn off the monitor: db2 "set event monitor dstatement state 0"

Now run db2evmon to produce a report on the statements that were run: db2evmon -path /tmp/dstatements >/tmp/dstate.out

The do_statistics_monitoring.sh script in the tuning guide scripts package automates this process. You can use the do_explain_sql.sh script to then have IBM DB2 explain how the optimizer will process a particular query, including any use of indexes.


10 Other resources

You will find the following resources useful for further tuning of the IBM Tivoli Identity Manager middleware:

IBM DB2 Administration Guide: Performance

http://www.ibm.com/cgi-bin/db2www/data/db2/udb/winos2unix/support/v8pubs.d2w/en_main

IBM DB2 Linux Tuning

http://publib.boulder.ibm.com/infocenter/db2help/index.jsp?topic=/com.ibm.db2.udb.doc/start/t0008238.htm

IBM DB2 Solaris Tuning

http://publib.boulder.ibm.com/infocenter/db2help/index.jsp?topic=/com.ibm.db2.udb.doc/start/t0008238.htm

IBM Tivoli Directory Server Tuning Guide

http://publib.boulder.ibm.com/tividd/td/IBMDirectoryServer5.2.html (IBM Directory Server V5.2)

http://publib.boulder.ibm.com/tividd/td/IBMDirectoryServer6.0.html (IBM Directory Server V6.0)

WebSphere Application Server Monitoring and Tuning Performance

http://www.ibm.com/software/webservers/appserv/infocenter.html

WebSphere Application Server – Tuning Operating Systems

http://publib.boulder.ibm.com/infocenter/wasinfo/v5r1//index.jsp?topic=/com.ibm.websphere.base.doc/info/aes/ae/tprf_tuneopsys.html

Sun ONE Directory Server Documentation

http://docs.sun.com/db/coll/S1_DirectoryServer_52 (Sun ONE Directory Server V5.2) http://docs.sun.com/db/coll/S1_ipDirectoryServer_51 (iPlanet Directory Server V5.1)

Sun Java Garbage Collection References

http://java.sun.com/docs/hotspot/gc1.4.2/ http://java.sun.com/docs/hotspot/VMOptions.html

IBM Java Garbage Collection References

http://tesch.dfw.ibm.com/java/gc_faq.html http://www-106.ibm.com/developerworks/ibm/library/i-garbage1/


11 Appendix A – scripts and files

The most recent version of these scripts can be obtained at http://www.ibm.com/support/us. Select “Search technical support”, type “ITIM Tuning Guide” in the box, and click submit. Most of the scripts will need to be customized for your system.

11.1 perftune_runstats.sh #!/bin/sh # perftune_runstats.sh # Last Updated: 2004/03/04 14:24 CST # Desription: # Shell script to run the db2 runstats command on all tables for a given # database and schema. It autodetects the db2 version using db2level # and passes the appropriate string to runstats to allow writes to occur # during the runstats. # Usage: # This script should be run as the user for the database (ie: one that # has connect and runstats abilities and permissions). if [ $# -eq 0 ]; then DATABASE=ldapdb2 SCHEMA=LDAPDB2 elif [ $# -eq 1 ]; then DATABASE=$1 SCHEMA=LDAPDB2 else DATABASE=$1 SCHEMA=$2 fi

# Old way, for LDAP distribution and indexes are not needed # OPTIONS="with distribution and detailed indexes all"

OPTIONS="ON ALL COLUMNS”

# Find out DB2 version for runstats syntax if db2level | grep "DB2 v7" > /dev/null; then ACCESS="shrlevel change" echo Detected DB2 major version 7 elif db2level | grep "DB2 v8" > /dev/null; then ACCESS="allow write access" echo Detected DB2 major version 8 fi # Connect to the database echo Connecting to $DATABASE db2 connect to $DATABASE # Execute runstats on all tables echo Performing runstats on all tables for schema $SCHEMA echo " with options: $OPTIONS $ACCESS" for i in `db2 connect to $DATABASE > /dev/null; db2 list tables for all | \ grep $SCHEMA | cut -d' ' -f1` do echo Table: $SCHEMA.$I db2 runstats on table $SCHEMA.$i $OPTIONS $ACCESS done # If the database we're tuning is an LDAP database, # update LDAP_DESC and LDAP_ENTRY stats in the statistics table if [ "$SCHEMA" = "LDAPDB2" ]; then echo Updating LDAP_DESC and LDAP_ENTRY stats in the statistics table db2 "update sysstat.tables set card = 9E18 where tabname = 'LDAP_DESC'" db2 "update sysstat.tables set card = 9E18 where tabname = 'LDAP_ENTRY'" fi

Documents

IBM Tivoli Identity Manager Performance Tuning Guide …publib.boulder.ibm.com/tividd/td/ITIM/SC32-5272-00/en_US/PDF/sc23... · IBM Tivoli Identity Manager Performance Tuning Guide