Best Practices for Oracle Monitoring with Oracle Extended Agent. - IBM · PDF fileBest Practices for Oracle Monitoring with Oracle Extended Agent. Document version [0.3] [Byline for

[Title of paper] [Month Year]

Best Practices for Oracle Monitoring with Oracle Extended Agent.

Document version [0.3]

[Byline for author]

[Credential line or job title]

Page 2 of 30

� ��

��

�� !" ��#

iii

CONTENTS

List of Figures..................................................................................................................v

List of Tables.................................................................................................................. vi

Revision History ............................................................................................................ vii

1.1 Planning your enterprise monitoring infrastructure....................................1

1.2 Reference sample Oracle database environment .....................................1

2.1 Local Monitoring or Remote Monitoring ....................................................2

2.2 Oracle Instant Client or Oracle Home.......................................................2

2.3 Database Connection Number per Agent Instance...................................3

3.1 Remote installation...................................................................................3

3.2 Apply latest Interim Fix .............................................................................3

3.3 Oracle Instant Client.................................................................................4

3.4 Mount of Alert Log Directory.....................................................................4

3.5 Configure of TNSNAMES.ORA file in agent site. ......................................5

3.6 Disable of sqlnet.log file in agent site........................................................6

3.7 Agent configuration from command line ...................................................6

3.8 ASM database monitoring ........................................................................8

iv

3.9 Dataguard database monitoring ...............................................................8

3.10 Use non-root user to startup agent to monitor Oracle database..............10

3.11 Check the status for connections............................................................10

3.12 Summary................................................................................................11

4.1 Extend agent timeout .............................................................................12

4.2 Listener Monitoring.................................................................................12

4.2.1 Stop the Listener/Net Service Monitoring..............................................12

4.2.2 Avoid the SQLNET.LOG file generated in the ITM directory..................12

4.2.3 Selected Listener/Net Services to be monitored ...................................13

4.3 Alert Log Monitoring ...............................................................................13

4.4 Customized SQL ....................................................................................14

4.5 Product Predefined SQL Customization .................................................14

4.6 Cluster Support ......................................................................................14

Appendix ........................................................................................................................ vi

References......................................................................................................................8

v

LIST OF FIGURES

Figure 1: Blank Database Connection and Agent Event .......................................................... 10

Figure 2: Dataguard Database Connections ........................................................................... 11

Figure 3: ASM Database Connection ...................................................................................... 11

Figure 4: History Collection Configuration for TOPSQL ........................................................... 16

Figure 5: Selection of Oracle Instant Client Downloads .............................................................vi

vi

LIST OF TABLES

Table 1: Oracle Node, Instance Name, Server Name and the Net Service name in the Listener.................................................................................................................................. 1

Table 2: Compare of Local monitoring and Remote monitoring ............................................... 2

Table 3: Environment variable for agent timeout ................................................................... 12

Table 4: Environment variables for alert log monitoring ......................................................... 13

Table 5: Agent Affinities......................................................................................................... 14

vii

REVISION HISTORY

Date Version Revised By Comments

2011/4/27 0.1 [email protected] initials

2011/10/17 0.2 [email protected] Add more best practices items

2012-4-30 0.3 [email protected] Add ASM/Dataguard

1

1 Overview

Oracle Real Application Cluster Database (RAC) support users to access the same data from multiple instances on different servers. In RAC environment, the instance level information includes multiple instances information; the database level information has only one copy.

For example, you can get following information for each instance: Instance Name, SGA, Session, and Process. The database level information includes Tablespace, Table, Index and Datafile, such kind of database level information is consistent and shared by all instances in a RAC environment.

This document will introduce the best practices in using the Oracle Extended Agent to monitor RAC database, ASM database or Dataguard database.

1.1 Planning your enterprise monitoring infrastructure

The Oracle Extended Agent v6.3.1 requires IBM Tivoli Monitoring (ITM) Infrastructure version 622FP2 or higher.

1.2 Reference sample Oracle database environment

The following Oracle Database environment will be used as sample environment in this document.

The RAC database is installed into two Solaris Sparc servers as two nodes (RAC1 and RAC2). The listener is running on port 1521 in both two nodes.

The Oracle cluster ware (including ASM) is installed at /opt/oracle/grid, the Oracle

database software is installed at /opt/oracle/db. The Oracle database server’s alert log location is /opt/oracle/base/diag/rdbms/rac1/trace/alert_rac1.log. The Oracle software version is 11.2.0.1.

The Oracle RAC database myrac is created with two RAC instances, one instance name

is rac1 in the RAC1 node, and the other instance name is rac2 in the RAC2 node.

The Oracle ASM database +ASM is created and there are also two instance, one instance

is +asm1 in the RAC1 node, the other is +asm2 in the RAC2 node.

The Oracle Dataguard database is installed into two Solaris Sparc Sun OS 5.10 servers as Primary Node in the machine sunpri and standby node in the machine sunstd.

Table 1: Oracle Node, Instance Name, Server Name and the Net Service name in the Listener

Oracle Node Type Oracle Instance Name Server Name Net Service Name in Listener

RAC1 rac1 sol_rac1 myrac

RAC2 rac2 sol_rac2 myrac

ASM1 +ASM1 sol_rac1 +ASM

2

ASM2 +ASM2 sol_rac2 +ASM

Dataguard Primary oradb sol_dg1 dbpri

Dataguard Standby oradb sol_dg2 dbstd

2 Strategy of Monitoring

This chapter introduces the monitoring strategy with the Oracle Extended Agent about the agent capacity, limitation and scalability.

2.1 Local Monitoring or Remote Monitoring

Oracle extended agent support local monitoring and remote monitoring. Following table compares these two different monitoring ways.

Table 2: Compare of Local monitoring and Remote monitoring

Category Local Remote Installation in Oracle Server machine

Yes No

Oracle physical disk usage monitoring

Yes Not support

Oracle alert log monitoring

Yes Yes, need mount the remote machine’s log directory to local, and need to specify mount path at agent configuration

Other monitoring features

Yes Yes

Maintenance effort Big, need to apply agent patch in each Oracle server machines.

Little, only need to apply patch at agent installation machine once.

Note: For a RAC environment, we recommend installing the extended agent into one node of the RAC environment, and then the agent performs the remote monitoring. The disk usage for local node can still be monitored; the alert log for remote nodes can be monitored after mount the alert log directory at the remote node.

2.2 Oracle Instant Client or Oracle Home

The Oracle extended agent is built with Oracle Call Interface (OCI) technology, the Oracle extended agent can use the dynamic library provided as Oracle Instant Client or Oracle server installation (as Oracle home).

We recommend using the Oracle instant client, because the user can always use the latest version of Oracle Instant Client package which contains the latest Oracle client dynamic library code fixes. By using the latest Oracle Instant Client, the user does not need to upgrade the Oracle server in case there are some known bugs at Oracle Client dynamic library.

3

Oracle Instant Client can be downloaded from the oracle web site http://www.oracle.com/technetwork/database/features/instant-client/index-097480.html

2.3 Database Connection Number per Agent Instance

We suggest each agent instance monitor no more then 10 database connections in case there are no more then more then 5 situations or historical collections in one agent.

This will ensure the total request number in an agent instance is less then 50(10 connection * 5 situations/historical collections). If there are more situations or historical collections, then we suggest reducing the db connection number for each agent instance and adding more agent instances.

The agent is affected by ITM limitation also:

http://www-01.ibm.com/support/docview.wss?uid=swg21263550

Note: Assuming no OS agent, a maximum of 15 extended agent instances in ITM installation directory.

Note: This is a common ITM limitation and it can be extended by following technotes:

Technotes link:



3 Agent installation and configuration

3.1 Remote installation

Oracle Extended Agent V6.3.1 support remote installation and you can install agent into a separated remote machine with Oracle servers.

Example The agent installation directory is /opt/IBM/Tivoli.

3.2 Apply latest Interim Fix

There are two options to download and apply latest Interim Fix.

Optipon1:

Search with key word of 6.3.1-TIV-ITM_KRZ from the Fix Central below, then download the package.

4

Search for downloads and fixes

Option 2:

Download 6.3.1 IF0002 package from link below:


You can get all agents download link for latest fix pack or interim fix from below wiki.

Latest Fixpacks and IFixes of Tivoli Composite Application Manager for Applications

3.3 Oracle Instant Client

If there is no Oracle Home or Oracle Client installed in the agent machine by the Oracle OUI installer, then you need to download the Oracle Instant Client package and extract to one readable directory in the agent machine, for example, /opt/IBM/Tivoli/instantclient_11_2.

Note: The LITE version of Oracle Instant Client is not supported.

You need to create a soft link from libclntsh.so.11.1 to file libclntsh.so in the Oracle Instant Client directory.

Example #ln -s /opt/IBM/Tivoli/instantclient_11_2/libclntsh.so.11.1 /opt/IBM/Tivoli/instantclient_11_2/libclntsh.so

Note: There is a full step-by-step guide for the Oracle Instant Client downloading in the Appendix A. How to use Oracle Instant Client for Tivoli Extended Oracle Agent.

3.4 Mount of Oracle alert log directory

As the agent is installed in the remote server, so oracle alert log directory (such as /opt/oracle/base/diag) in the Oracle server need to be mounted to agent machine.

Following is sample steps:

Step 1) Check if the NFS server is started in the Oracle server, or run the following command as user ‘root’.

# /etc/init.d/nfs.server start

Step 2) Check alert log directory permission in the Oracle server and add the “rx” permission for directory recursively to ensure the mount client can access the mounted alert log files, run the following command as user ‘root’.

# chmod -R +rx /opt/oracle/base/diag

Step 3) Share the alert log directory with following command in the Oracle server rac1.

# share -F nfs -o ro=rac1 /opt/oracle/base/diag

Step 4) Create a local directory and mount the remote alert log directory into local directory.

5

# mkdir /mnt/rac1

# mount rac1:/opt/oracle/base/diag /mnt/rac1

3.5 Configure of TNSNAMES.ORA file in agent site.

A new file tnsnames.ora is created in the agent server. It will be used by agent to resolve the connection information from the connection string.

Following is sample steps:

Step 1) use editor like Vi to create a new file tnsnames.ora file in the server tivsun21 at /opt/IBM/Tivoli/config directory with following sample lines. The file permission should be set as rx-rx-rx.

Note: Connection name of TIVRAC must start the far left of the line; other lines should not start from left of the line.

Example # cat /opt/IBM/Tivoli/config/tnsnames.ora

#RAC database connection string

TIVRAC =

(DESCRIPTION=

(LOAD_BALANCE=ON) (FAILOVER=on)

(ADDRESS =

(PROTOCOL=tcp)(HOST=sol_rac1)(PORT=1521))

(ADDRESS =


(CONNECT_DATA=(SERVICE_NAME=myrac)

)

)

#ASM database connection string

TIVASM =

(DESCRIPTION=

(ADDRESS =


(ADDRESS =


(CONNECT_DATA=(SERVICE_NAME=+asm)

)

)

#Dataguard database primary node connection string

TIVDGPRI =

(DESCRIPTION=

(ADDRESS = (PROTOCOL=tcp)(HOST=sol_dg1)(PORT=1521))

6

(CONNECT_DATA=(SERVICE_NAME=dgpri)

)

)

#Dataguard database standby node connection string

TIVDGSTD =

(DESCRIPTION=

(ADDRESS = (PROTOCOL=tcp)(HOST=sol_dg2)(PORT=1521))

(CONNECT_DATA=(SERVICE_NAME=dgstd)

)

)

Step 2) Edit both file /opt/IBM/Tivoli/config/rz.ini and /opt/IBM/Tivoli/config/rz_<instance>.config to append one line below to ensure the variable TNS_ADMIN is set as the path directed to the new file tnsnames.ora.

Example export TNS_ADMIN=/opt/IBM/Tivoli/config

3.6 Disable of sqlnet.log file in agent site

A new file sqlnet.ora is created in the agent server tivsun21. It will be used by Oracle OCI library to get the Oracle Net Service settings; one of the settings will disable the sqlnet.log file in the Oracle client side, such as /opt/IBM/Tivoli/bin/sqlnet.log.

Fatal NI connect error 12514, connecting to:

(DESCRIPTION=(CONNECT_DATA=(SERVICE_NAME=orcl)(CID=(PROGRAM=krzstar

t)(HOST=tivagent)(USER=root)))(ADDRESS=(PROTOCOL=TCP)(HOST

=rac1.cn.ibm.com)(PORT=1521)))

Step 1) use editor like Vi to create a new file sqlnet.ora file in the agent server tivsun21 at /opt/IBM/Tivoli/config directory with following lines. The file permission should be set as rx-rx-rx.

LOG_DIRECTORY_CLIENT=/dev/null

LOG_FILE_CLIENT=/dev/null

Step 2) (Optional) Edit both file /opt/IBM/Tivoli/config/rz.ini and /opt/IBM/Tivoli/config/rz_<instance>.config to append one line below to ensure the variable TNS_ADMIN is set as the path directed to the new file sqlnet.ora.

3.7 Agent configuration from command line

Launch agent configuration from command line and input the specific value as following.

7

• Launch configuration tool from command line.

Example #/opt/IBM/Tivoli/bin/itmcmd config -A rz

• When ask for the input filed of “Oracle Home Directory”, input a blank space to make sure there is no valid value for “Oracle Home Directory”

Example Oracle home directory: (default is: ): [Input

space and press Enter]

• When ask for the input field of “Oracle Instant Client Installation Directory”, input the path where the Oracle Instant Client is extracted.

Example Oracle instant client installation directory:

(default is: ):

/opt/IBM/Tivoli/instantclient_11_2

• When ask for selection to edit the database connection settings, input 1

Example Edit 'Database connection' settings, [1=Add,

2=Edit, 3=Del, 4=Next, 5=Exit] (default is: 5): 1

• When ask for the input filed of “Database Connection Name”, input the database name “racnew”.

Example Database connection name: (default is: ): rac

• When ask for the input field of “Oracle Connection String”, input the tnsnames.ora connection identifier which is defined in the file /opt/IBM/Tivoli/config/tnsnames.ora, current value is “TIVRAC”. Or you can input the full connection string directly.

Example Oracle connection string: (default is: ): TIVRAC

Example Oracle connection string: (default is: ):

(DESCRIPTION=(LOAD_BALANCE=ON)(FAILOVER=ON)(ADDRE

SS=(PROTOCOL=tcp)(HOST=sol_rac1)(PORT=1521))(ADDR

ESS=(PROTOCOL=tcp)(HOST=sol_rac2)(PORT=1521))(CON

NECT_DATA=(SERVICE_NAME=myrac)))

• When ask for the input filed of Oracle alert log file paths, input the local alert log file full paths which is mounted from remote Oracle servers.

Example The absolute file path of mapped alert log files

of remote database instances in this database

connection. Multiple files are separated by ";"

on Windows systems, or ":" on UNIX systems. Each

file is matched to a database instance, by the

file name pattern alert_<instance>.log, or

ignored if unmatched. Local database instance

alert log files can be discovered automatically.

If this item was not configured, the alter logs

of remote database instances would be unable to

be collected. Oracle alert log file paths:

(default is: ):

/mnt/sol_rac1/rdbms/rac/rac1/trace/alert_rac1.log:

/mnt/sol_rac2/rdbms/rac/rac2/trace/alert_rac2.log

8

3.8 ASM database monitoring

You can use existing agent instance to add one new connection to monitor one ASM database or create a new agent instance with a new connection.

• When ask for selection to edit the database connection settings, input 1



• When ask for the input field of “Database Connection Name”, input the connection name like “asm”.

Example Database connection name: (default is: ): asm

Note: Only letters, Arabic number, underline and minus characters can be used in the connection name. Other connection name like “+asm” is invalid.

• When ask for the input field of “Oracle Connection String”, input the tnsnames.ora connection identifier which is defined in the file /opt/IBM/Tivoli/config/tnsnames.ora, current value is “TIVASM”. Or you can input the full connection string directly.

Example Oracle connection string: (default is: ): TIVASM


(DESCRIPTION=(LOAD_BALANCE=ON)(FAILOVER=ON)(ADDRE

SS=(PROTOCOL=tcp)(HOST=sol_rac1)(PORT=1521))(ADDR

ESS=(PROTOCOL=tcp)(HOST=sol_rac2)(PORT=1521))(CON

NECT_DATA=(SERVICE_NAME=+ASM)))

• When ask for the input field of “Username”, “Password” and “Role”, input the user information with SYSDBA role.

Example The set of privileges to be associated with the connection. For a user that was granted the SYSDBA system privilege, you can specify a connection that includes the privilege. If this item was not defined, 'DEFAULT' would be used as the role of the user. Role: [ 1=SYSDBA, 2=SYSOPER, 3=SYSASM, 4=DEFAULT ] (default is: 4): 1

Note: In Oracle 10g ASM database, a new Oracle user cannot be created in the sqlplus directly, you need to create the new Oracle user and grant as SYSDBA role in the RDBMS sqlplus and copy the RDB password file to the ASM password file, reference the technotes below:


3.9 Dataguard database monitoring

You can use existing agent instance to add two new connections for Dataguard Primary database and Standby database separately, or you can create a new agent instance with two new connections.

9

• When ask for selection to edit the database connection settings, input 1 for Primary Node connection.



• When ask for the input filed of “Database Connection Name”, input the connection name “dgpri” for the primary node connection.

Example Database connection name: (default is: ): dgpri

• When ask for the input field of “Oracle Connection String”, input the tnsnames.ora connection identifier which is defined in the file /opt/IBM/Tivoli/config/tnsnames.ora, current value is “TIVDGPRI”. Or you can input the full connection string directly.


TIVDGPRI


(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=sol_dg1

)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=dgpri)))

• When ask for the input field of “Username”, “Password” and “Role”, input the user with SYSDBA role. The user with SYSDBA role is not mandatory for Dataguard Primary Node,

• When ask for the selection to edit the database connection settings, input 1 for the Standby Node connection.

Example Edit 'Database connection' settings, [1=Add, 2=Edit, 3=Del, 4=Next, 5=Exit] (default is: 5): 1

• When ask for the input filed of “Database Connection Name”, input the connection name “dgpri” for the primary node connection.

Example Database connection name: (default is: ): dgstd

• When ask for the input field of “Oracle Connection String”, input the tnsnames.ora connection identifier which is defined in the file /opt/IBM/Tivoli/config/tnsnames.ora, current value is “TIVDGPRI”.


TIVDGSTD


(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=sol_dg1

)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=dgstd)))

• When ask for the input field of “Username”, “Password” and “Role”, input the user with SYSDBA role.

Example The set of privileges to be associated with the connection. For a user that was granted the SYSDBA system privilege, you can specify a connection that includes the privilege. If this item was not defined, 'DEFAULT' would be used as the role of the user.

10

Role: [ 1=SYSDBA, 2=SYSOPER, 3=SYSASM, 4=DEFAULT ] (default is: 4): 1

3.10 Use non-root user to startup agent to monitor Oracle database

Use ITM utility $ITMHOME/bin/itmcmd to startup agent appending with agent instance name in the -o option by non-root user.

Note: You need to grant the r+w permission of files “installdir/config/.Config/RunInfo*” and “installdir/logs/*rz*” to the non-root user before starting the agent.

Example #su - oracle -c “/opt/IBM/Tivoli/bin/itmcmd agent

-o inst1 start rz”

Example #su - tivmon -c “/opt/IBM/Tivoli/bin/itmcmd agent

-o inst1 start rz”

3.11 Check the status for connections.

Go to the Oracle Database Extended root node and check the “Database Status” column in the view of “Database Connection”.

If the “Database Status” is not “Active” or there is not data in the view of “Database Connection”, then check the “Error Message” column and “Suggestion” column in the Agent Event workspace.

Figure 1: Blank Database Connection and Agent Event

11

Figure 2: Dataguard Database Connections

Figure 3: ASM Database Connection

3.12 Summary

• Apply latest Interim Fix.

• Use latest Oracle Instant Client library instead of Oracle Home directory.

• Use remote monitoring

• Use file tnsnames.ora in the directory $ITMHOME/config, then specify the connection string using load balance and fail over, including all RAC instances listener address in one connection definition in the file tnsnames.ora.

• Mount the alert log directory from remote machine to local agent machine to monitor the alert log.

12

• Use non-root user to startup agent.

4 Customization

4.1 Extend agent timeout

You can set the environment variable below to extend the agent itmeout from default 60 seconds to more.

Table 3: Environment variable for agent timeout

Variable Description

CDP_COLLECTION_TIMEOUT The maximum time of data collection. If the execution time exceeded, the agent stops the execution, and reports timeout. The default value is 60 (in seconds). The minimum value is 5 (in seconds).

4.2 Listener Monitoring

4.2.1 Stop the Listener/Net Service Monitoring

Refer the readme file of 6.3.1-TIV-ITM_KRZ-IF0001.

https://www-304.ibm.com/support/docview.wss?uid=swg24029579

Enhancement: INTERNAL

Abstract: Providing the option to control whether monitoring the listener and Oracle net-service.

Additional Information: The listener monitoring and net-service monitoring are started by default. We provide the option that if you do not want this monitoring; you can turn off the related monitoring functions.

The environment variables KRZ_LISTENER_PING_INTERVAL and KRZ_TNS_PING_INTERVAL control the interval. If the value is less than 1, then it exits the thread and stops checking.

Example KRZ_LISTENER_PING_INTERVAL=0

Example KRZ_TNS_PING_INTERVAL=0

4.2.2 Avoid the SQLNET.LOG file generated in the ITM directory

When the listener monitoring is enabled and there is some ORA error, the Oracle OCI library file will generate the sqlnet.log file in both Oracle server directory and client directory.

13

There is such file in the ITM directory when agent is running and it will get grow more and more. User can avoid such sqlnet.log file by redirecting to another directory or set to /dev/null in the Oracle configuration file sqlnet.ora.

Example #cat $ORACLE_HOME/network/admin/sqlnet.ora | grep LOG

LOG_DIRECTORY_CLIENT=/dev/null

LOG_FILE_CLIENT=/dev/null

4.2.3 Selected Listener/Net Services to be monitored

If the Oracle home directory is selected during the agent configuration, then agent will load Listener definition from file $ORACLE_HOME/network/admin/listener.ora and load Net Service definition from file $ORACLE_HOME/network/admin/tnsnames.ora automatically.

If user do not want to monitor all Listener or Net Service from ORA files in the Oracle home directory and do not want to revise those files too, customer can specify the TNS_PATH during the agent configuration and create new listener.ora file and tnsnames.ora files in the new path and only the targeted Listener or Net Service is specified in the new files.

Example This is the directory that contains Oracle database net configuration file. This directory is defined by the TNS_ADMIN environment variable for each Oracle database instance. The default directory is $ORACLE_HOME/network/admin on UNIX or Linux systems, and %ORACLE_HOME%\NETWORK\ADMIN on Windows systems. If there are multiple net configuration file directories, use ";" on Windows systems, or ":" on UNIX systems, to separate the directories. If this item was not configured, the default directory would be used. Net configuration files directories: (default is: ): /opt/IBM/Tivoli/config/myrac

4.3 Alert Log Monitoring

To customize the severity of Oracle alert log messages, reference Info Center:

http://publib.boulder.ibm.com/infocenter/tivihelp/v24r1/topic/com.ibm.itcama.doc_6.2.4/oracleextagent631_user24.htm?path=2_11_5_1_5#customize_alert_log

You can set environment variable in the table below for the alert log monitoring.

Table 4: Environment variables for alert log monitoring

Variable Description

KRZ_LOG_MAXREAD Only the latest part of alert logs is scanned by the Oracle Database Extended agent while the agent starts. The size of the scanned part is specified by the KRZ_LOG_MAXREAD attribute. The default value is 5M. The value must be

14

positive. The valid data suffixes are M/m presenting megabyte, and K/k presenting kilobyte.

KRZ_LOG_INTERVAL This attribute defines the search interval (in seconds) for alert log files. The default value is 300 (seconds). Setting the value to 0 disables search. The valid range is from 60 to 3600.

KRZ_LOG_NOREPEAT If the value is true, the Oracle Database Extended agent does not re-send alert log message entries that were found in the last running of the Oracle Database Extended agent. The default value is FALSE. The valid values are TRUE and FALSE.

4.4 Customized SQL

To define a customized SQL statement to monitor in the Customized SQL workspace, reference Info Center:

http://publib.boulder.ibm.com/infocenter/tivihelp/v24r1/topic/com.ibm.itcama.doc_6.2.4/oracleextagent631_user25.htm?path=2_11_5_1_6#customize_sql

4.5 Product Predefined SQL Customization

You can use the environment variable KRZ_CUSTOM_SQLXML to specify a customized krzsql.xml, reference the technotes http://www.ibm.com/support/docview.wss?uid=swg21588392

4.6 Cluster Support

Reference to the technotes of “Agent clustering and positioning in the TEP Navigator”


Table 5: Agent Affinities

Affnities Node in the TEP

%IBM.OracleAgents Oracle Database Extended

%IBM.OracleAgentRDB Oracle RDBMS

%IBM.OracleAgentASM Oracle ASM

%IBM.OracleAgentDG Oracle Dataguard

15

5 Problem Determination

Troubleshooting: Slow Database Monitoring


Question:

How do you extended the timeout period of the oracle agent?

Cause:

The oracle database is slow on the system and the agent often times out during monitoring.

Answer:

Update the CDP_COLLECTION_TIMEOUT environment variable. The default is 60 seconds and the agent needs more time to verify database connection.

Troubleshooting: TOP SQL historical collection


Question:

How to configure the TOP SQL attribute group historical collection for the RZ agent

Answer:

To configure the historical collection for the TOP SQL attribute group, the user needs to click on the third tab ("Filters") and fill the requested filters: "begin hour" ,"end hour", "order by" and "row order". This setting is required to get the historical collection enabled for this attribute group.

Historical collection settings:

16

Figure 4: History Collection Configuration for TOPSQL

Troubleshooting: DB connection fails during Oracle Extended Agent configuration

Problem (Abstract):

Attempts to configure result in error "ORA-01722: invalid number" despite IF0001 is already applied for the RZ agent v6.3.1.

Symptom:

Error message "ORA-01722: invalid number"

Resolving the problem:

If the IF0001 has been applied, you can safely ignore the error message for test connection, as follows.

1. Set "NLS_NUMERIC_CHARACTERS" in <AGENT_INSTALL_HOME>/config/rz.ini

NLS_NUMERIC_CHARACTERS=.

Reconfigure agent and ignore the above error message.

Try to start this agent, if agent is not started, try next step (2).

2. Set "NLS_NUMERIC_CHARACTERS" in <AGENT_INSTALL_HOME>/config/rz.ini

NLS_NUMERIC_CHARACTERS=.,

Reconfigure agent and ignore the above error message.

17

Try to start this agent, if agent is not started, please collect the debugging data for the IBM support investigation as explained at next step (3).

3. Set "KBB_RAS1" in <AGENT_INSTALL_HOME>/config/rz.ini

KBB_RAS1=ERROR (UNIT:krz all)

Start agent and send to IBM Support the output of "digup -a" command.

Also, please use "sqlplus" to connect to database directly: issue following SQL and send the output to the IBM Support

select * from nls_database_parameters;

select * from nls_instance_parameters;

select * from nls_session_parameters;

select userenv('language') from dual;

select * from v$nls_valid_values;

Then quit "sqlplus" and setup the environment as follows:

NLS_LANG=AMERICAN_AMERICA.AL32UTF8 && export NLS_LANG

NLS_NUMERIC_CHARACTERS=. && export NLS_NUMERIC_CHARACTERS

Use again "sqlplus" to connect to oracle db and issue:

select userenv('language') from dual

select * from nls_session_parameters

Then quit "sqlplus" and setup system environment again as follows:

NLS_NUMERIC_CHARACTERS='.,' && export NLS_NUMERIC_CHARACTERS

Finally, issue the last SQL command:

Select * from nls_session_parameters

Please send the outputs from all the above SQLs statements to the IBM Support for more investigation.

Troubleshooting: Monitoring Oracle alert logs with remote RZ agent

Question:

How to remotely monitor the Oracle alert log on AIX with an RZ agent installed on Windows?

18

Answer:

For monitoring remote alert logs on AIX, UNIX or Linux systems, the file system containing the logs needs to be locally mounted on the RZ agent machine.

To mount AIX file systems in Windows, you need to install Samba on AIX.

Troubleshooting: Oracle Extended Agent (rz) fails loading Oracle OCI library

Problem (Abstract):

The Oracle Extended Agent is not running properly and produces the error: Loading Oracle OCI library failed!

Symptom:

The Oracle Extended Agent starts and connects to TEMS but the Agent does not display Oracle data.

Cause:

This error occurs because the Oracle Extended Agent(rz) can not use the Oracle Client Interface(OCI) to connect to the Oracle instance.

Diagnosing the problem:

The Oracle Extended Agent on unix/linux starts the krzagent and then launches the krzclient process. The krzclient process fails when it tries to connect to the Oracle database.

Example instance_rz_server_krzclient_timestamp.log reflects the following errors:

Fail to dynamically load library libclntsh.so.10.1, return code 8

Try to load another possible Oracle OCI library

Fail to dynamically load library libclntsh.so.11.1, return code 2

Loading Oracle OCI library failed!

Resolving the problem:

Reconfigure the agent and correct Oracle home or Oracle Instant Client Library. Reconfigure the agent and restart.

vi

APPENDIX

6 Appendix A. How to use Oracle Instant Client for Tivoli Extended Oracle Agent

Step 1) open Oracle Instant Client download page from Oracle Site below:

http://www.oracle.com/technetwork/database/features/instant-client/index-097480.html

Step 2) click one suitable platform from list, especially for the 32-bit and 64-bit platforms.

Figure 5: Selection of Oracle Instant Client Downloads

Step 3) select ‘Accept License Agreement’. For example:

vii

“Instant Client Downloads for Solaris Operating System(SPARC 64-bit)

Step 4) download Instant Client Package - Basic: All files requires to run OCI, OCCI, and JDBC-OCI applications.

Example basic-11.2.0.2.0-solaris-sparc64.zip (64,102,971 bytes) (cksum - 4103480643)

REFERENCES

[Reference List] (Use bibliography format. Style automatically creates a hanging indent when text runs over to the next line.)


错误错误错误错误！！！！文档中没有指定样式的文字文档中没有指定样式的文字文档中没有指定样式的文字文档中没有指定样式的文字。。。。

9

®

© Copyright IBM Corporation 2009 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp. IBM may not offer the products, services, or features discussed in this document in other countries. Consult your local IBM representative for information on the products and services currently available in your area. Any reference to an IBM product, program, or service is not intended to state or imply that only that IBM product, program, or service may be used. Any functionally equivalent product, program, or service that does not infringe any IBM intellectual property right may be used instead. However, it is the user's responsibility to evaluate and verify the operation of any non-IBM product, program, or service.

IBM may have patents or pending patent applications covering subject matter described in this document. The furnishing of this document does not grant you any license to these patents. You can send license inquiries, in writing, to:

IBM Director of Licensing IBM Corporation North Castle Drive Armonk, NY 10504-1785 U.S.A.

The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law: INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PAPER “AS IS” WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF NON-INFRINGEMENT, MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Some states do not allow disclaimer of express or implied warranties in certain transactions, therefore, this statement may not apply to you. This information could include technical inaccuracies or typographical errors. Changes may be made periodically to the information herein; these changes may be incorporated in subsequent versions of the paper. IBM may make improvements and/or changes in the product(s) and/or the program(s) described in this paper at any time without notice. Any references in this document to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites. The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk.

IBM may have patents or pending patent applications covering subject matter described in this document. The furnishing of this document does not give you any license to these patents. You can send license inquiries, in writing, to: IBM Director of Licensing IBM Corporation 4205 South Miami Boulevard Research Triangle Park, NC 27709 U.S.A. All statements regarding IBM's future direction or intent are subject to change or withdrawal without notice, and represent goals and objectives only. This information is for planning purposes only. The information herein is subject to change before the products described become available. If you are viewing this information softcopy, the photographs and color illustrations may not appear.

IBM, the IBM logo, and ibm.com are trademarks or registered trademarks of International Business Machines Corp., registered in many jurisdictions worldwide. Other product and service names might be trademarks of IBM or other companies. A current list of IBM trademarks is available on the Web at “Copyright and trademark information” at www.ibm.com/legal/copytrade.shtml.

==> The listed trademarks of the following companies require marking and attribution only if you mention any of them in your paper:


错误错误错误错误！！！！文档中没有指定样式的文字文档中没有指定样式的文字文档中没有指定样式的文字文档中没有指定样式的文字。。。。

10

Adobe, the Adobe logo, PostScript, and the PostScript logo are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States, and/or other countries.

IT Infrastructure Library is a registered trademark of the Central Computer and Telecommunications Agency which is now part of the Office of Government Commerce

Intel, Intel logo, Intel Inside, Intel Inside logo, Intel Centrino, Intel Centrino logo, Celeron, Intel Xeon, Intel SpeedStep, Itanium, and Pentium are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and other countries.

Linux is a registered trademark of Linus Torvalds in the United States, other countries, or both.

Microsoft, Windows, Windows NT, and the Windows logo are trademarks of Microsoft Corporation in the United States, other countries, or both.

ITIL is a registered trademark, and a registered community trademark of the Office of Government Commerce, and is registered in the U.S. Patent and Trademark Office

UNIX is a registered trademark of The Open Group in the United States and other countries.

Cell Broadband Engine is a trademark of Sony Computer Entertainment, Inc. in the United States, other countries, or both and is used under license therefrom.

Java and all Java-based trademarks and logos are trademarks of Sun Microsystems, Inc. in the United States, other countries, or both.

Other company, product, or service names may be trademarks or service marks of others.

Documents

Best Practices for Oracle Monitoring with Oracle Extended Agent. - IBM · PDF fileBest Practices for Oracle Monitoring with Oracle Extended Agent. Document version [0.3] [Byline for