Novell Identity Manager Troubleshooting

Novell® Identity Manager Troubleshooting

Reed Harrison

GTS Identity Services [email protected]

Rajiv Kumar

IDM [email protected]

© Novell, Inc. All rights reserved.2

Agenda

• IDM information sources

• IDM trace definition

• IDM trace capture

• IDM trace validation

• IDM trace reading

• Appendix

Information Sources


Where do I find product resources?

Where to find information?– Novell® Support Forums

http://forums.novell.com/– Novell Support Knowledgebase

http://support.novell.com– Novell Documentation

http://www.novell.com/documentation– Google

http://www.google.com/– 3rd Party Vendor website

> Microsoft, Oracle, IBM, SAP, MySQL, etc


What information do I need to troubleshoot my issue?

– Issue description as detailed as possible– Identify the environment - is it production? Lab?

include software versions and where each piece is installed> OS Type, version and patch level for all servers> Are those real machines or VMs? If VMs, which virtualization solution?

» Virtualization product name, version and patches

> eDirectory™, Security Services and IDM versions for all relevant servers> 3rd Party applications relevant to the issue, their versions and patch levels> eDirectory replicas present on the IDM server and their types> Location of the servers and connectivity between them

» Are the servers local, or across Wan links?

» Are there firewalls/routers/other network devices between them?


What information do I need to begin troubleshooting?What information should we gather for troubleshooting?

– Driver exports and/or Designer project exports (preferred)– OS-Related information

> supportconfig on Linux OShttp://www.novell.com/communities/node/2332

> config.txt on Netware® OStype LOAD CONFIG /ALL on the server's console

> for Windows/Solaris/AIX find out the version and if it is 32 or 64 bit. Also, on Windows, find out what domain functional level they are running. Note that 2008 and 2008 R2 are separate products.

– IDM traces, J2EE App server logs– (Optional)

DSTRACE & LAN trace files, ndsd.log (Linux/Unix), Event Viewer logs (Windows), logger.txt & console.log (Netware)

Novell® Identity Manager Traceor Now we have information

What do we look at first?


How IDM works review

Local Configuration:

eDirectory™

IDM Engine + Driver Shim

ConnectedApplication


How IDM works review

Remote Loader Configuration:

eDirectory™ConnectedApplication

IDM Engine + Driver Shim

RemoteLoader


Engine Flow Diagram - Subscriber

IDM Engine flow (simplified) – Subscriber only

TranslationProcessorPlacement

EventTransform

CreateMatching

MergeProcessor

Add?

YES

NO

NO

YES

Event

Sequencer

MatchFound?

CommandTransform

Notify &

Reset

Subscriber Filter

ADD Processor

Sync & Ignore

Subscriber Filter

Event Cache

TAOFile

Not part of the channel Thread

Association

Processor


Engine Flow Diagram - Publisher

IDM Engine flow (simplified) – Publisher only

Placement

CommandTransform

ADD Processor

EventTransform

Create Match

MergeProcessor

Sync &

Ignore Publisher Filter

Pre-filterAssociationProcessorAdd?

YES

NO

NO

Notify &

Reset

Publisher Filter

YES

Post-filterAssociationProcessor

MatchFound?

Modify?

OptimizeModify

NO

YES

TranslationProcessorEvent

Sequencer

IDM Trace Capture and Validation


What is the most effective way to troubleshoot? IDM traces• In IDM, traces are a way of following step by step how

the events are processed and executed• Reading an IDM trace is akin to debugging a program,

since most of what IDM does is execute DirXML-Script commands on an event's XML

• As with any programming language, you need to know the language well if you intend on debugging it

• DirXML-Script language is explained at:– http://www.novell.com/documentation/idm36/policy/data/policytypesoverview.html– http://www.novell.com/documentation/idm36/policy_dtd/data/dtddirxmloverview.html– http://www.novell.com/documentation/idm36/policy_designer/data/bookinfo.html


When to use IDM Traces

• Traces should be used only for troubleshooting, not for auditing events

• Tracing can have a huge impact on driver performance (tenfold or more, depending on trace level)

• IDM debug traces can be configured in iManager, Designer, or at the Remote Loader configuration file


IDM Trace Types and How to Capture

• There are 2 types of traces - Engine or Remote Loader

– IDM Engine trace: can be seen in 3 different ways> DSTRACE screen / DSTRACE file> iMonitor Trace Screen> IDM Trace file (also known as Java trace file)

– Remote Loader trace: can be traced only to file> On Windows there is a live trace screen that can be seen if

certain criteria is met (criteria varies per Windows version)


IDM Trace Levels

• Engine trace levels go from 0 to 4. Each trace level shows all the status messages from previous levels

– Level 0: Status Messages Only– Level 1: Current location in the Driver Logic flow– Level 2: Events (XML format)– Level 3: Driver Logic Execution Details– Level 4: Cache-related information about the event coming

from eDirectory™ (Subscriber channel)• Shim trace levels go from 3 to 10

– Information provided changes per driver, check driver docs for description of what each trace level provides for its shim


Capturing IDM Traces

• Step by Step instructions on setting IDM traces– http://www.novell.com/documentation/idm36/idm_common_

driver/data/b1rc1vm.html

• More information on how to read IDM traces– http://www.novell.com/communities/node/5681/capturing-

and-reading-novell-identity-manager-traces

• Best Information on trace reading– Trace reading cool solution:

http://www.novell.com/communities/node/9677/comprehending-idm-traces-part-1


Basic validation of IDM traces

Some things to check in the trace

– Does the test user show in the trace file? Look into the src-dn and dest-dn XML attributes of the operation

– Is the operation in the trace the same one performed during testing?

– If you are getting an error, is it in the trace.?

– Were the files taken with the proper trace level?


Basic validation of IDM Engine traces

Quick Trace Parsing– To find an event coming from eDirectory™, search for

> Start transaction

– To find an even coming from the Application, search for> Receiving DOM document from application

– Any actions performed in eDirectory are preceded by> Pumping XDS to eDirectory

– The result of all status messages shows after> DirXML Log Event

– Driver initialization starts with> Reading named passwords list


Basic validation of IDM traces

grep is your friend!– grep is a tool that allows to search several files quickly, and

returns one or more lines matching what you searched– grep accepts command line parameters like -A (after) and -B

(before) that can be extremely useful. Some examples:> Case-insensitive search

grep -i 'my text here in any case' trace.log> List all Status Log Messages in a trace

grep -B 1 -A 5 'DirXML Log Event' trace.log> List the first piece of all events coming from eDirectory (might need a bigger

number for the A parameter if the trace level is 4 or above)grep -A 9 'Start transaction' trace.log

> Counts how many times the driver got restarted in this trace filegrep 'Reading named passwords list' trace.log | wc -l

IDM Trace Reading BasicsThe Engine Side


Trace Reading - Basic

• Again, traces should be used only for troubleshooting, not for auditing events

• An IDM trace (level 3 and above) will show all the steps done by the engine while processing an Event

• We will trace Reed Harrison as he is added to OpenLdap from the Identity Vault



AddReed

Harrison



AddReed

Harrison



AddReed

Harrison



AddReed

Harrison



AddReed

Harrison



AddReed

Harrison



AddReed

Harrison



AddReed

Harrison



AddReed

Harrison



Summary

– Reading an IDM trace means following events from beginning to end, and seeing how the driver logic affected them before the event's XML is handed to the destination system

– An IDM engine trace level 3 or above will show all steps done while a driver processes an event

– Both iManager & Designer show simplified views of the logic processing, don't let them sidetrack you

Appendix

IDM Quick Reference Cards


Types of Cards

• Installation Troubleshooting• Engine does not load• Driver does not start• Password Synchronization Issues• Other driver issues

Installation Troubleshooting


Installation Troubleshooting

• Obtain OS name & patch level• Identify eDirectory version & patch level (if installing the

IDM engine)• Identify the IDM version being installed. Double-check if

the OS / eDir / IDM combination is supported in the Novell Documentation

• Obtain the Install logs following the Install troubleshooting steps in the docs.

• Use the cool solution “Identity Manager 3.6 Install Troubleshooting Tips” - This is the best reference for install issues.

Engine Does Not Load


IDM Engine Does Not Load

• Obtain OS name & patch level• Identify eDirectory version & patch level• Identify the IDM version• With the above information, see next page for Windows

Instructions, and the one after for Linux Instructions• The best TID for this is Troubleshooting errors -641

or -783 Starting an IDM driver. TID 7002449



• Windows:– IDM is installed in the same directory where eDirectory's dlms

are (by default, C:\Novell\NDS)– Stop the eDirectory service– Move the file “dirxml.dlm” from that directory– Start the eDirectory service– After eDirectory finishes loading, start DSTRACE.dlm, set the

flags 'DirXML', 'DirXML Drivers', 'Misc Other' and start tracing to file

– Move the file “dirxml.dlm” back to its original location– Close/reopen the eDirectory services console, select dirxml.dlm

and hit the start button



• Linux:– Stop ndsd ( /etc/init.d/ndsd stop )– Move the libvrdim.* files from their original directory to a

different directory> eDir 8.7.3.x: /usr/lib/nds-modules/> eDir 8.8.x: /opt/novell/eDirectory/lib/nds-modules/

– Start ndsd ( /etc/init.d/ndsd start )– Start ndstrace with only the flags 'time', 'tags',' misc', 'dxml', '

dvrs' and save the trace to a file. Leave it running on screen– Move the libvrdim.* files back to their original location– Back on the ndstrace screen, type 'load vrdim'– After you see the error, stop ndstrace and grab the file

Driver Does Not Start


Driver Does Not Start

• If you are receiving the following error codes, this is an engine problem, not driver problem:

> -783 VRDIM Not Initialized> -641 Invalid Request

• For all other errors starting a driver– (optional) Set Remote Loader trace level to 5 and make sure he

starts normally before attempting to start the driver– Set engine trace level to 3, and set trace to file– Try to start the driver again to capture the error in the trace file.

After the attempt to start fails, get the trace file

Password Synchronization Issues



•Obtain OS name & patch level•Identify eDirectory version & patch level•Obtain NMAS version & patch level•Identify the IDM version•Which drivers & connected applications are involved? Take note of their versions and where they are running•Check in the Matrix if that driver/application combination can sync passwords. IDM 3.6 docs:

http://www.novell.com/documentation/idm36/idm_password_management/data/bo1o7xz.html

http://www.novell.com/documentation/idm36/idm_password_management/data/bo1o7xz.html



•Check which direction passwords do not synchronize– If the problem is coming from eDirectory, make sure Universal

Password is configured properly and Tree keys are fine– If the problem is coming from the connected application, we

need to check different things based on the application> LDAP (SunONE only): Check the password plugin on SunONE> AD: Password Synchronization filters must be installed and running

http://www.novell.com/documentation/idm36drivers/ad/data/bow0k51.html

> Linux&Unix: Check the platform's PAM (or LAM) configuration

• Drivers have GCVs that control password flowhttp://www.novell.com/documentation/idm36/idm_password_management/data/bnwjt01.html

http://www.novell.com/documentation/idm36drivers/ad/data/bow0k51.html

http://www.novell.com/documentation/idm36/idm_password_management/data/bnwjt01.html

Other Driver Issues


For ALL Other Driver Issues

• ALWAYS obtain a current driver export OR designer project export

• Take note of IDM version, eDirectory version on the IDM server, OS (including version and patch level)

• Take note of 3rd party Application name, patch level and OS where it is running

• Identify if a Remote Loader is in use.– If there is, the reference to Shim trace levels will be applied in

the remote loader– If not, the Shim trace levels will be applied in the engine and the

recommendation for engine trace levels can be ignored


Active Directory Driver

• Users do not synchronize– Engine trace level 3, Shim trace level 3– Take note of the test user name, location and system where he

was created• Users synchronize in a single direction

– Check the driver filters– Check the placement policies in the appropriate channel– Engine trace level 3, Shim trace level 3

• Passwords are not synchronizing– See section on password sync on this document


Avaya PBX Driver

• Extensions are not created– Engine trace level 3, Shim trace level 3


Delimited Text Driver

• Users do not get created in eDirectory– Check if the input directory exists and is properly entered in the

driver configuration– Check filesystem rights and quotas on input directory&files– Engine trace level 3, Shim trace level 3– Input csv file used to create the users

• Driver does not write output files– Check if the output directory exists and is properly entered in

the driver configuration– Check filesystem rights and quotas on output directory– Engine trace level 3, Shim trace level 3


eDirectory™ Driver

• eDirectory drivers work in pairs– Engine trace level 3 on both trees being connected, on the

proper pair of eDirectory drivers– This driver does not support remote loader– For the Driver exports, make sure you get both eDirectory driver

exports (there is one driver per tree).– If you get a Designer project, make sure that both eDirectory

drivers are imported in the project


Entitlements Service Driver

• This driver enables/disables entitlements on objects– Engine trace level 5 for the entitlements driver itself– LDAP Export of the Entitlement Policies used in the Driverset

(they reside bellow the Driverset object)– Since this driver only changes the DirXML-EntitlementRef

attribute on a user, we need to get the appropriated traces on the other drivers being affected by that change


GroupWise® Driver

• Mail accounts are not created in GroupWise– Engine trace level 3, Shim trace level 5


ID Provider Driver

•This driver troubleshooting is unique in the sense it is also a service an can be accessed by external clients

– Traces can be enabled in the driver & client parameters, aside from the regular IDM tracing. The driver docs go into more details here:

– http://www.novell.com/documentation/idm36drivers/idprovider/data/bookinfo.html

– If a customer calls in with an ID provider call, do this:> document the issue in detail> get the ID driver export> get a LDAP export of their ID Policy objects> ask the customer to provide the XSLT / Java call made to the

ID Provider service

http://www.novell.com/documentation/idm36drivers/idprovider/data/bookinfo.html


JDBC Driver

•For ALL JDBC driver issues request– Database name, vendor and patch level– OS & patch level where the database in running at– Check if its a supported IDM/Database combination. Docs

http://www.novell.com/documentation/idm36drivers/jdbc/data/bw17kgf.html

– Driver connection mode> direct or indirect> triggered or triggerless

– Customer's database schema (SQL file for the tables/views that the driver connects to)

– Engine trace level 3, Shim trace level 3 (only request a higher trace level for this driver if oriented by Backline)

http://www.novell.com/documentation/idm36drivers/jdbc/data/bw17kgf.html


JMS Driver

• Messages are not being sent or received from the JMS Queue/application

– Engine trace level 3, Shim trace level 5


LDAP Driver

• Users are not synchronizing between systems– Engine trace level 3, Shim trace level 5– (Optional) LAN trace between the driver shim and the 3rd party

LDAP system• Passwords are not synchronizing from the LDAP

system into eDirectory– Password synchronization from the LDAP system is only

supported currently when the LDAP system is SunONE 5.2 on certain platforms. Check the LDAP driver documentation for steps on how to configure the password plugin for SunONE


Linux and Unix Settings Driver

• Attributes are not added to new users– Engine trace level 10


Linux and Unix Bi-directional Driver

• User is not created on the platform, or data is not synchronizing correctly after creation

– Engine trace level 3, Shim trace level 4– from the connected Linux/Unix platform, get the file:

/usr/local/nxdrv/logs/script-trace.log• Passwords are not syncing from the Linux/Unix

platform– Information above plus the platform's PAM (or LAM)

configuration files. Since those change per platform, there is no standard location to get them, but the customer's Linux/Unix admin should know where they are located


Linux and Unix Fan-out driver

Driver has 2 parts: core driver and platform agents• Core Driver

– IDM Driver connects to the Core Driver– Usually runs on the IDM server, but can run on a remote loader.

When running on a Remote Loader, the logs referenced bellow will be in the Remote Loader server

– Get the core driver Audit log and Operational log files> On Linux/Unix they are found at

/usr/local/ASAM/data/CoreDriver/logs> On Windows they are found at

C:\Novell\ASAM\data\CoreDriver\logs


Linux and Unix Fan-out driver

Driver has 2 parts: core driver and platform agents• Platform Agents

– Run on the connected system (1 platform agent per system)– Execute its action locally via shell scripts– Get the asamplat.conf file at

/usr/local/ASAM/data/asamplat.conf– Get the platform's log files

> On Linux/Unix the files reside at/usr/local/ASAM/data/PlatformServices/logs/

> On Midrange and Mainframe platforms, contact Novell Support for assistance with the call


Lotus Notes Driver

• For any issues, obtain– Engine trace level 3, Shim trace level 5

• Check the Documentation about a Notes driver issue. The troubleshooting section in the docs will solve most problems. Many of the problems can be traced to a rights issue.


Manual Task Service Driver

• For any issues– Engine trace level 5


PeopleSoft 5.2 Driver

•For connectivity issues with PeopleSoft– Output of the CITester application

http://www.novell.com/documentation/idm36drivers/peoplesoft_52/data/ah79lgj.html#ajn78pl

•For any other issues– Engine trace level 3, Shim trace level 5– Version of the PeopleTools (NOT the application, this is the API

we connect to) that the customer is using

http://www.novell.com/documentation/idm36drivers/peoplesoft_52/data/ah79lgj.html#ajn78pl


SAP HR Driver

• Cannot synchronize objects to SAP– Engine trace level 3, Shim trace level 5

• Cannot synchronize objects from SAP– Engine trace level 3, Shim trace level 5– Copy of the iDoc file processed by the driver

> iDoc file location can be seen in the driver's properties, as the value of the parameter “iDoc File Directory”


SAP User Management Driver

•For connectivity issues with SAP– Output of the SAP JCO test utility

http://www.novell.com/documentation/idm36drivers/sap_user/data/alvws18.html

•For any other issues– Engine trace level 3, Shim trace level 5

http://www.novell.com/documentation/idm36drivers/sap_user/data/alvws18.html


Scripting Driver

• NTS does not support customizations to the scripts of the scripting driver.

• We can help the customer with driver installation issues, but any custom code can only be reviewed by either Consulting or a Novell Partner (both cases for a fee, not included in any Novell Support contract)


SOAP Driver

• For connectivity issues with the SOAP system– LAN trace between the driver shim and the SOAP system– Engine trace level 3, Shim trace level 5

• For any other issues– Engine trace level 3


Workorder Driver

• For any issues– Engine trace level 3, Shim trace level 5


SIF Driver

• Only supported on IDM 3.5.1 and 3.0.1• NOT SUPPORTED on IDM 3.6• For any issues



Windows NT Driver




Microsoft Exchange 5.5 Driver




Loopback Driver

• Also known as “move-proxy driver” (old IDM 2.x nomenclature) or “Null” driver

• For any issues– Engine trace level 3


Issues With Jobs

• A driver export does not contain the Jobs information, so we absolutely need an Designer project export

• There are currently 4 types of pre-defined Jobs, take a note of the job being used and the issue description. What will be required to troubleshoot the Jobs varies per Job and issue.

Unpublished Work of Novell, Inc. All Rights Reserved.This work is an unpublished work and contains confidential, proprietary, and trade secret information of Novell, Inc. Access to this work is restricted to Novell employees who have a need to know to perform tasks within the scope of their assignments. No part of this work may be practiced, performed, copied, distributed, revised, modified, translated, abridged, condensed, expanded, collected, or adapted without the prior written consent of Novell, Inc. Any use or exploitation of this work without authorization could subject the perpetrator to criminal and civil liability.

General DisclaimerThis document is not to be construed as a promise by any participating company to develop, deliver, or market a product. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. Novell, Inc. makes no representations or warranties with respect to the contents of this document, and specifically disclaims any express or implied warranties of merchantability or fitness for any particular purpose. The development, release, and timing of features or functionality described for Novell products remains at the sole discretion of Novell. Further, Novell, Inc. reserves the right to revise this document and to make changes to its content, at any time, without obligation to notify any person or entity of such revisions or changes. All Novell marks referenced in this presentation are trademarks or registered trademarks of Novell, Inc. in the United States and other countries. All third-party trademarks are the property of their respective owners.

Documents

Novell Identity Manager Troubleshooting