36
CA Support Delivery T +1 800 225 5224 F +1 631 342 6800 ca.com CA Workload Automation DE (dSeries Edition) Troubleshooting Guide

CA Workload Automation DE Troubleshooting Guide CA Workload Automation DE R11.3 with Oracle RAC .....20 Cannot Restart the Server Due to RDBMS Oracle Error.....21 Oracle RAC Failover

Embed Size (px)

Citation preview

Page 1: CA Workload Automation DE Troubleshooting Guide CA Workload Automation DE R11.3 with Oracle RAC .....20 Cannot Restart the Server Due to RDBMS Oracle Error.....21 Oracle RAC Failover

CA Support Delivery

T +1 800 225 5224

F +1 631 342 6800 ca.com

CA Workload Automation DE (dSeries Edition)

Troubleshooting Guide

Page 2: CA Workload Automation DE Troubleshooting Guide CA Workload Automation DE R11.3 with Oracle RAC .....20 Cannot Restart the Server Due to RDBMS Oracle Error.....21 Oracle RAC Failover

Copyright © 2014 CA Technologies. All rights reserved. 2

Table of Contents

Introduction ............................................................................ 4

Verify that an Issue Has Already Been Fixed ........................................ 4

Compatibility Matrix for CA Workload Automation DE (dSeries Edition) ......... 6

Finding Logs or Additional Data for an Issue ........................................ 7

Troubleshooting the CA Workload Automation DE Server ....................... 10

Server Shuts Down When the Number of Active Application Generations Reaches 600 .........................................10

Server Shuts Down with Java Out of Memory Error .................................................................................11

Server Slows Down with Out.of.memory Exception Error. .........................................................................11

Unable to retrieve users from LDAP subdirectories .................................................................................12

Job Fails with ‘Command Requires a User ID’ Error .................................................................................12

How to Set Up a Disaster Recovery Environment for CA Workload Automation DE ..............................................12

Update the Server JRE to the Latest Update Version ...............................................................................13

Change the IP Address of the Server .................................................................................................13

Cannot Install the Server Due to Ports Conflict ......................................................................................13

Predefined (Canned) Report Generation Fails Due to Insufficient Java Heap Memory ...........................................14

Server Stops Sending Email Notifications Due to Connection Reset Error .........................................................15

Open a Support Ticket for Updating the CA Workload Automation DE License ...................................................15

Server Stops Triggering the Application When the Number of Application generations Reaches Maximum....................16

Troubleshooting CA WA Desktop Client ........................................... 17

Memory Issues Occur When Using Subscribe All ....................................................................................17

Unable to View Predefined Reports on the Reports Pane ..........................................................................17

Cannot Complete the Patch Installation in CA WA Desktop Client r11.3 SP1 ......................................................17

When CA WA High Availability Server Fails over, Will the CA WA Desktop Client Connection to Server Too Fail Over

Automatically? .........................................................................................................................18

CA WA Desktop Client Crashes When Opening an Application .....................................................................18

Troubleshooting the CA Workload Automation DE Database ................... 20

Configure CA Workload Automation DE R11.3 with Oracle RAC ...................................................................20

Cannot Restart the Server Due to RDBMS Oracle Error .............................................................................21

Oracle RAC Failover is Not Supported in CA Workload Automation DE r11.1 .....................................................21

Unable to Restart the Server Due to DB2 Database Exception Error ...............................................................22

Event Trigger Takes Long Time .......................................................................................................22

Server Shuts Down with the SQL Server Exception Error ...........................................................................23

Server Shuts Down When the Disk Space of DB2 Database is Full .................................................................23

Archiving or Purging CA Workload Automation DE Database ......................................................................24

Huge History Data in the Database Table Impacts Server Performance ...........................................................25

Page 3: CA Workload Automation DE Troubleshooting Guide CA Workload Automation DE R11.3 with Oracle RAC .....20 Cannot Restart the Server Due to RDBMS Oracle Error.....21 Oracle RAC Failover

Copyright © 2014 CA Technologies. All rights reserved. 3

Troubleshooting Agents and Jobs .................................................. 26

The agent may fail to start with the error message "Unexpected encryption type. None received. DES expected." ...........26

When there is huge number of agents in agent group configured with CPU load balancing, then lots of stale messages gets

exchanged between agent and server. This can result in decreased performance of the following: ............................26

dSeries agent fails to start after the owner of the dSeries installation directory was changed to a non-root user .............26

Jobs may go to 'Ready' state and won’t complete if they are run as non-root user. Following errors may show up in spool file:

..........................................................................................................................................27

Once the dseries is shut down and brought back after long time jobs which were missed during that time period will get

triggered. What is the way that these jobs should not be triggered. ..............................................................27

While a failover made by the changerole command some agents were not updated with active dSeries server and had

continuing trying the connection to the previous one ..............................................................................27

Understanding Agent communications ..............................................................................................28

Error creating stdout spool file .......................................................................................................28

A job failed with Status: WARNING: User <user_name> is not authorized to send AFM to Agent <user_name> ...............29

Common Questions ................................................................. 30

How to Upgrade the Server ...........................................................................................................30

How to Install CA License to CA Workload Automation DE r11.3? .................................................................31

Operating System might not be set for the Daylight Savings Time (DST) adjustment. If it is not set, would CA Workload

Automation DE still follow DST? ......................................................................................................31

When applying an upgrade patch should the CA Workload Automation DE dSeries 11.3 database instance be up and running?

..........................................................................................................................................32

Do desktop client and CAWA (dSeries) server of different version work together? ...............................................32

What are the Oracle 10.x/11.x certified character sets for dSeries server? .......................................................32

What to check if Agent is not communicating with CA (dSeries) Workload Automation Server? ................................32

Does CA Workload Automation DE (dSeries) server need to be down if we only purge the history staging tables (H_*)? .....32

Do I need to have all involved databases and CA Workload Automation dSeries servers running while migrating to version

11.3? ...................................................................................................................................33

How to cold start CA Workload Automation (CA WA) and what happens after cold start? ......................................33

CA Workload Automation dSeries server has five levels to implement the cold start. What are those levels meanings? ......34

Is it possible to retrieve users from two different LDAP servers at once? .........................................................34

What values should be assigned to the ldap.propertyNameForLogin= statement in the server.properties file? ...............34

What flavor of the JDK 6 should be chosen for the Web Services installation - 32 or 64-bit? ....................................35

How to disable CLI log in dSeries 11.1? ..............................................................................................35

How can I check if the port is available, the firewall does not block the traffic, and the service is running on it? ..............35

How to find the version and build number of the CA Workload Automation DE server? .........................................36

How to find the version and build number of CA Workload Automation Agent? .................................................36

How to find the version and build number of CA WA Desktop Client? .............................................................36

Page 4: CA Workload Automation DE Troubleshooting Guide CA Workload Automation DE R11.3 with Oracle RAC .....20 Cannot Restart the Server Due to RDBMS Oracle Error.....21 Oracle RAC Failover

Copyright © 2014 CA Technologies. All rights reserved. 4

Introduction

This guide contains troubleshooting information for CA Workload Automation DE Server, CA WA

Desktop Client, CA Workload Automation Agents, and other common issues.

Verify that an Issue Has Already Been Fixed

Before you raise a CA Support ticket for an issue, verify that the issue has already been fixed in one of

the latest patches.

Follow these steps:

1. Log in to CA Support.

2. Enter “CA Workload Automation Solutions & Patches” in the “Find It” field and click the Right

arrow .

3. Click “CA Workload Automation DE” under Product in the left pane.

4. Click “CA Workload Automation Solutions & Patches” from the search result to view the screen

as follows:

5. Click the patch number to verify the issues that are fixed as part of the published patches.

Page 5: CA Workload Automation DE Troubleshooting Guide CA Workload Automation DE R11.3 with Oracle RAC .....20 Cannot Restart the Server Due to RDBMS Oracle Error.....21 Oracle RAC Failover

Copyright © 2014 CA Technologies. All rights reserved. 5

Note: Alternatively, you can click the below link to view all the patches of CA Workload Automation DE.

Before you click the link, ensure that you are already logged in to CA Support:

https://support.ca.com/irj/portal/anonymous/phpsupcontent?contentID=7e52789e-ab05-4dea-b58b-

a4925a7f4beb&productID=7833

Page 6: CA Workload Automation DE Troubleshooting Guide CA Workload Automation DE R11.3 with Oracle RAC .....20 Cannot Restart the Server Due to RDBMS Oracle Error.....21 Oracle RAC Failover

Copyright © 2014 CA Technologies. All rights reserved. 6

Compatibility Matrix for CA Workload

Automation DE (dSeries Edition)

You can find the compatibility matrix for CA Workload Automation DE from CA Support Online.

Follow these steps:

1. Log in to CA Support.

2. Click Support.

3. Click Support By Product from the pop-up menu.

4. Under "Select by Product page:", enter or select "CA Workload Automation DE (CA dSeries) in

the drop-down.

5. Click on "CA Workload Automation Compatibility Information" under "Product Status" from the

search result.

6. Click “CA Workload Automation DE (dSeries Edition)” under “Scheduling Engines” from the

search result.

The compatibility matrix for CA Workload Automation DE opens.

Page 7: CA Workload Automation DE Troubleshooting Guide CA Workload Automation DE R11.3 with Oracle RAC .....20 Cannot Restart the Server Due to RDBMS Oracle Error.....21 Oracle RAC Failover

Copyright © 2014 CA Technologies. All rights reserved. 7

Finding Logs or Additional Data for an Issue

Before you raise a Support ticket for an issue, you must provide logs that are generated when the issue

occurs or additional information about the issue. The logs and additional information helps CA Support

analyze and resolve the issue quickly.

1. Server Trace Logs: Stores communication messages between the CA Workload Automation DE

components and maintains debugging information. By default, the trace log is located in the

following directory:

server_install_dir\logs\tracelog.timestamp.txt

Note: Trace logs are archived (rolls over) every day. Each trace log is created with a timestamp, for

example, trace log.<timestamp>.txt. You can find the older trace logs based on the date and time

when the trace log was created.

2. Server Buffer Logs (applicable only for R11.3.x server): Stores all messages between the server and

the database regardless of category or severity. By default, the buffer log is located in the following

directory:

server_install_dir/logs/buffer/buffer.timestamp.txt

Note: After the buffer log file reaches its maximum size, it is archived and a new buffer log file is

created with a new timestamp. By default, only the latest buffer log is archived and saved. So, we

recommend that you back up the buffer log as soon as the issue occurs.

(Applicable only for r11.1 server) If the problem is reproduced consistently, enable debug logs on

server side and collect the log using the setlogid CLI command. The following is the snippet of the

setlogid command:

[root@ajjra01-I112002 bin]# ./cli ajjra01-I112002 7500 ADMIN admin

Connected

ADMIN@ajjra01-I112002:7500(PRIMARY)> setlogid log("trace log") filterid("1-1100")

The filter ID for file C:\Program Files (x86)\CA\Workload Automation\LogFiles\tracelog.txt has been

set to 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23..............

3. Thread Dumps: The thread dumps shows the current threads stacktrace used by the server. Issue

the threaddump CLI command to view the stacktrace. Back up the stacktrace when the issue occurs.

Example: This example displays a partial response to the THREADDUMP command. It displays the

current threads stacktrace used by the server.

Page 8: CA Workload Automation DE Troubleshooting Guide CA Workload Automation DE R11.3 with Oracle RAC .....20 Cannot Restart the Server Due to RDBMS Oracle Error.....21 Oracle RAC Failover

Copyright © 2014 CA Technologies. All rights reserved. 8

[root@ajjra01-I112002 bin]# ./cli ajjra01-I112002 7500 ADMIN admin

Connected

ADMIN@ajjra01-I112002:7500(PRIMARY)> threaddump

[Thread group=system;active threads=57]

Reference Handler

java.lang.Object.wait(Native Method)

java.lang.Object.wait(Unknown Source)

java.lang.ref.Reference$ReferenceHandler.run(Unknown Source)

Finalizer

java.lang.Object.wait(Native Method)

java.lang.ref.ReferenceQueue.remove(Unknown Source)

java.lang.ref.ReferenceQueue.remove(Unknown Source)

java.lang.ref.Finalizer$FinalizerThread.run(Unknown Source)

Note: Threaddump can also be taken from the CA WA Desktop Client CLI perspective.

4. Version of the Server, Agent, and CA WA Desktop Client:

a. How to find the version and build number of the CA Workload Automation DE server?

b. How to find the version and build number of CA Workload Automation Agent?

c. How to find the version and build number of CA WA Desktop Client?

5. Agent Logs: Collect the agent logs if the agent was involved in an event monitoring or job related

process when the issue occurs. By default, agent logs are located in the agent_install_dir\log

directory. You can zip the entire log folder when the issue occurs and provide it to CA Support.

You can also provide the log details of the issue by following these steps:

a. Stop the agent.

b. Change the log level of the agent to 8 by modifying the log.level parameter in the

agentparam.txt file in the agent installation directory.

Note: The log level determines the type and number of logs the agent generates and the

amount of information the log contains. Setting the log level to 8, generates the tracing

information that is useful for troubleshooting communication issues.

c. Delete the contents of the log folder.

d. Start the agent.

e. Reproduce the issue.

f. Zip the entire log folder.

Page 9: CA Workload Automation DE Troubleshooting Guide CA Workload Automation DE R11.3 with Oracle RAC .....20 Cannot Restart the Server Due to RDBMS Oracle Error.....21 Oracle RAC Failover

Copyright © 2014 CA Technologies. All rights reserved. 9

6. Desktop Client Logs: If the issue is related to CA WA Desktop Client, collect the following desktop

client logs and provide the zipped file to CA Support:

Internal trace log: By default, the internal trace log is located in the

C:\Users\<User>\workspace-CAWA-11.x directory.

Trace log set by operator: The trace log that is set by the operator. You can set the trace log

location in CA WA Desktop Client by following these steps:

a. Click Trace, Set Trace File from the CA WA Desktop Client menu.

b. Specify the location of the trace file.

7. Database Dumps: If the issue is related to database, provide the database dump.

Note: For more information about collecting database dumps for a database, contact your database

administrator.

8. Heap Dumps: If the issue is related to memory, collect the heap dumps. To collect the heap dumps

from the server, do the following:

a. (On UNIX) If jdk is installed on your server machine, use the "jmap" tool by issuing the

following command:

jmap -dump:format=b,file=heapdump.dmp <dSeries_server_pid>

You can retrieve the server pid (process ID) from the server installation directory by running

the following command:

cat serverPID

Note: If jdk is not installed on the server, download it from Oracle.com.

b. (On Windows) If jdk (1.6) is installed on your machine, use the "jvisualvm" tool as follows:

a. Change to the following directory on the command prompt:

<jdk home>\bin

b. Enter the following command:

visualvm.exe --jdkhome "C:\Software\Java\jdk1.6.0" --userdir

"C:\Temp\visualvm_userdir"

The main window of VisualVM opens. By default, the Applications section is

displayed on the left pane of the main window. The Applications section shows the

Java applications running on the local and remote JVMs.

c. Right-click an application node and select Main application from the popup menu to

collect heap dumps.

Page 10: CA Workload Automation DE Troubleshooting Guide CA Workload Automation DE R11.3 with Oracle RAC .....20 Cannot Restart the Server Due to RDBMS Oracle Error.....21 Oracle RAC Failover

Copyright © 2014 CA Technologies. All rights reserved. 10

Troubleshooting the CA Workload Automation DE

Server

This section describes scenarios for troubleshooting the CA Workload Automation DE server issues.

Server Shuts Down When the Number of Active Application Generations

Reaches 600

Symptom:

When the number of active application generations reaches 600, the server shuts down.

Solution:

By default, the warning threshold for the monitoring of active application generations is 500 and the

shutdown threshold is 600. To fix this issue, you can increase the default shutdown threshold value.

Note: To prevent the number of active application generations from reaching shutdown threshold, we

recommend that you periodically force complete active applications that are not completed or not

needed.

To change the default shutdown threshold value, follow these steps:

1. Stop the server.

2. Do one of the following:

a. On Windows, open the following file in a text editor:

install_dir\conf\server.properties

b. On UNIX, open the following file in a text editor:

install_dir/conf/server.properties

install_dir

Specifies the directory where the server is installed.

3. Edit the following property in the file:

active.appl.gen.stop=shutdownthreshold

Note: If you increase the shutdown threshold value, you must also increase the heap memory size.

4. Save and close the file.

5. Restart the server.

Page 11: CA Workload Automation DE Troubleshooting Guide CA Workload Automation DE R11.3 with Oracle RAC .....20 Cannot Restart the Server Due to RDBMS Oracle Error.....21 Oracle RAC Failover

Copyright © 2014 CA Technologies. All rights reserved. 11

Server Shuts Down with Java Out of Memory Error

Symptom:

The CA Workload Automation DE server shuts down with the following error message:

The cause of the servershutdown: - java.lang.OutOfMemoryError: unable to create new native thread

Solution:

If the operating system has memory issues, the server might shuts down. To fix this issue, you must

increase the heap size.

To increase the heap size, follow these steps:

1. Restart the OS 2. Do one of the following to increase the heap size:

On Windows:

Edit the following property in the startServer.lax file in the server_install/bin directory:

lax.nl.java.option.java.heap.size.max=value_in_bytes

On UNIX:

Edit the following property in the startServer in the server_install/bin directory:

MAX_HEAP_SIZE=value_in_megabytes

Server Slows Down with Out.of.memory Exception Error.

Symptom:

The CA Workload Automation DE server slows down with the out.of.memory exception error. The trace

log shows java heap size usage spike over the threshold limit suddenly and the java heap size usage

reported in trace log periodically as follows:

20091117 09:05:06.352 [ID: 0] Thread-7: Esp Server State:

Memory:

Using 7 MB of Currently Allocated 25 MB. Total Free Memory is 1009 MB of the Max Heap Size 1016 MB

Solution:

If the java heap size spikes over the threshold limit suddenly instead of a gradual increase, one of the

most common causes is the jobs with "attach spool file" option in email notification settings run across a

very big spool file.

To fix this issue, either avoid using the "attach spool file" option in the job email notification settings or

limit the size of email attachments. You can limit the size of email attachments by setting the maximum

size of spool file attachments in email notifications.

Page 12: CA Workload Automation DE Troubleshooting Guide CA Workload Automation DE R11.3 with Oracle RAC .....20 Cannot Restart the Server Due to RDBMS Oracle Error.....21 Oracle RAC Failover

Copyright © 2014 CA Technologies. All rights reserved. 12

Unable to retrieve users from LDAP subdirectories

Symptom:

When I retrieve all the LDAP users from the configured LDAP directory, the users in the LDAP

subdirectories cannot be retrieved.

Solution:

To retrieve all users from the configured LDAP directory and its subdirectories, set the

"ldap.pullUsersFromSubdirectories" parameter in the server.properites file to true and restart the

server.

Job Fails with ‘Command Requires a User ID’ Error

Symptom:

When I run a job, the job fails with the following error message:

Command requires a User ID

Solution:

When the user name defined in the job and the name of the agent added to the agent topology are

same, the job fails with this error message.

To fix this issue, follow these steps:

1. Open the server.properties file from the following directory in a text editor:

server_install_dir/conf

2. Uncomment the following parameter and restart the server:

allow_agentname_as_username=true

How to Set Up a Disaster Recovery Environment for CA Workload Automation

DE

The disaster recovery environment lets you protect the data in your production environment from

system failure, natural disaster, or infrastructure failure. The jobs can be run continuously in the

sandbox environment when a disaster affects your production environment.

For more information about setting up a disaster recovery environment for CA Workload Automation

DE, refer to the PIB RI72688 on CA Support Online.

Page 13: CA Workload Automation DE Troubleshooting Guide CA Workload Automation DE R11.3 with Oracle RAC .....20 Cannot Restart the Server Due to RDBMS Oracle Error.....21 Oracle RAC Failover

Copyright © 2014 CA Technologies. All rights reserved. 13

Update the Server JRE to the Latest Update Version

CA Workload Automation DE server includes a specific release of JRE. To address security vulnerabilities,

you can update the JRE of the server to the latest update version within the same release. For example,

if the server includes JRE 1.7.0_67, you can update the JRE to the latest available update version of JRE

1.7.0_72.

For more information about updating the server JRE to the latest update version, refer to the PIB

RI69979 on CA Support Online.

Change the IP Address of the Server

1. Make sure your Oracle database is accessible from dSeries server after moving

2. Modify the database connection properties using the setdbparm utility, if applicable:

Here is an example of the setdbparm utility:

[root@ajjra01-I112002 bin]# ./setdbparm jdbc.URL

Changing JDBC url (jdbc.URL)

Enter new JDBC url:jdbc:oracle:thin:@[ajjra01-I112002]:1521:orcl

Verify new JDBC url:jdbc:oracle:thin:@[ajjra01-I112002]:1521:orcl

Changed. The Server must be restarted for the change to take effect.

3. Change the ip address of dSeries server:

a. Shutdown server

b. Modify host ip address and the DNS entries and /etc/hosts file

c. Start server

4. Change the ip address of agents. Check and modify agentparm.txt to reflect dSeries server address

change.

Cannot Install the Server Due to Ports Conflict

Symptom:

When I define TCP/IP ports and install the server, the server installation stopped indicating that none of

the TCP/IP ports are available.

Solution:

When you install the server, the server reads the system hostname and IP address from the system's

'hosts' file. To fix this issue, add the system IP address and the hostname in the hosts file. The hosts file

is located in the following directory:

(On Windows) <System root>\system32\drivers\etc

(On UNIX) <System root>/system32/drivers/tc

Page 14: CA Workload Automation DE Troubleshooting Guide CA Workload Automation DE R11.3 with Oracle RAC .....20 Cannot Restart the Server Due to RDBMS Oracle Error.....21 Oracle RAC Failover

Copyright © 2014 CA Technologies. All rights reserved. 14

Predefined (Canned) Report Generation Fails Due to Insufficient Java Heap

Memory

Symptom:

When I run the server predefined (canned) report, the report generation failed. The sever trace log and

the report log contain the following Java error message:

GC overhead limit exceeded

Solution:

The report generation failed due to out of memory error as the garbage collection process for retrieving

report data took long time. By default, the server allocates 512 MB to the Java process that generates

reports on the server. To fix this issue, allocate more memory to the report process. For example, you

can increase the maximum memory available for generating reports to 1024 MB.

To increase maximum memory, follow these steps:

1. Log in to CA WA Desktop Client as an administrator.

2. Open the "Admin" perspective

3. Right-click "Topology"

4. Right-click "Server connections"

5. Select "Configure Shared Parameters" --> "Internal"

6. Click the icon at the right side of "Internal" section to show the "Advanced Parameters"

7. Set "Maximum memory available to a reporting process (in megabytes)" value to a higher

memory (the default value is 512); increase it to 1024 or more.

Page 15: CA Workload Automation DE Troubleshooting Guide CA Workload Automation DE R11.3 with Oracle RAC .....20 Cannot Restart the Server Due to RDBMS Oracle Error.....21 Oracle RAC Failover

Copyright © 2014 CA Technologies. All rights reserved. 15

Server Stops Sending Email Notifications Due to Connection Reset Error

Symptom:

The CA Workload Automation DE server stopped sending email notifications. The server trace log

contains the following connection reset error message:

HAC.emailNotification: javax.mail.MessagingException:

Exception reading response; nested exception is:

java.net.SocketException: Connection reset

Solution:

The server connects to the email server with no timeout setting and relies on the email server to reset

the connection. If this connection gets disrupted and is not fully reset due to networking glitches, server

cannot send email notifications.

To fix this issue, restart the server in the standalone configuration or issue the CHANGEROLE command

in the CA WA High Availability configuration.

Open a Support Ticket for Updating the CA Workload Automation DE License

To request or update the license of CA Workload Automation DE, open a Support ticket.

1. Log in to CA Support.

2. Hover the mouse over ‘Case Management’

3. Click ‘Open a New Case’

4. Click ‘Licensing’ link on the left navigation pane:

The 'CA Licensing' page appears.

Page 16: CA Workload Automation DE Troubleshooting Guide CA Workload Automation DE R11.3 with Oracle RAC .....20 Cannot Restart the Server Due to RDBMS Oracle Error.....21 Oracle RAC Failover

Copyright © 2014 CA Technologies. All rights reserved. 16

5. Click 'Open an issue for CA Licensing' under the 'Support Requests' on right pane:

Note: For further information or clarification about your license, click Update an Existing Issue or

Contact CA Licensing.

6. Enter the appropriate information in the ‘Customer Care Case Request’ form and click 'Submit'.

Server Stops Triggering the Application When the Number of Application

generations Reaches Maximum

When the number of Application generations reaches a maximum value, the server stop triggering the

application and sends a warning email to the administrator.

To avoid this issue, we recommend that you issue the RESETGEN CLI command and reset the

application’s generation number periodically.

Page 17: CA Workload Automation DE Troubleshooting Guide CA Workload Automation DE R11.3 with Oracle RAC .....20 Cannot Restart the Server Due to RDBMS Oracle Error.....21 Oracle RAC Failover

Copyright © 2014 CA Technologies. All rights reserved. 17

Troubleshooting CA WA Desktop Client

This section describes scenarios for troubleshooting CA WA Desktop Client issues.

Memory Issues Occur When Using Subscribe All

Symptom:

When the workload is large and many users are using Subscribe All from CA WA Desktop Client, memory

issues occur.

Solution:

The Subscribe with filter and Subscribe All options are resource intensive and might cause degradation

of service when running multiple reports by multiple business units. Hence, we recommend that you

limit the use of these options:

Use "Subscribe Active" to monitor workload

Use "Subscribe with Filter" to view information about specific completed/active generation(s) at

specific times

Use "Subscribe All" to view at all the completed/active applications. This option retrieves huge

information, hence we recommend operators to not use 'Subscribe All' for day-to-day workload

monitoring.

Unable to View Predefined Reports on the Reports Pane

Symptom:

When I try to open predefined reports, the reports did not appear on the Reports pane in the Services

perspective.

Solution:

Verify that you have REPORT_DESIGN permission for opening predefined reports.

Cannot Complete the Patch Installation in CA WA Desktop Client r11.3 SP1

Symptom:

When I apply a patch on top of CA WA Desktop Client r11.3 SP1, the following error message is

displayed:

"Cannot complete the install because of a conflicting dependency. Software being installed: CA Workload Automation Desktop Client

11.3.0.201306061122_1069 (com.ca.wa.desktop.workbench.feature.feature.group 11.3.0.201306061122_1069) Software currently installed: Shared profile 1.0.0.1360093034868 (SharedProfile_SDKProfile 1.0.0.1360093034868) Only one of the following can be installed at once: CA WA WSDL Parser 11.3.0.201209210305_505 (com.ca.wa.desktop.wsdl 11.3.0.201209210305_505)

Page 18: CA Workload Automation DE Troubleshooting Guide CA Workload Automation DE R11.3 with Oracle RAC .....20 Cannot Restart the Server Due to RDBMS Oracle Error.....21 Oracle RAC Failover

Copyright © 2014 CA Technologies. All rights reserved. 18

CA WA WSDL Parser 11.3.0.201306061122_1069 (com.ca.wa.desktop.wsdl 11.3.0.201306061122_1069) ..."

Solution:

Stop the Desktop Client and then start it again by right-clicking and selecting "Run as administrator".

When CA WA High Availability Server Fails over, Will the CA WA Desktop Client

Connection to Server Too Fail Over Automatically?

When the preferred (primary) server goes down, CA WA Desktop Client will point to the standby server automatically and this connection will be shown as an active instance.

When the preferred server is back up and running, CA WA Desktop Client will automatically establish connection with the preferred server and this connection will be shown as an active instance.

CA WA Desktop Client Crashes When Opening an Application

Symptom:

CA WA Desktop Client frequently crashes when I open or update an Application. The CA WA Desktop Client reinstallation was not useful.

Solution:

Sometimes, CA WA Desktop Client crashes when you try to open or create a new Application in the Define perspective. If the crashes occur frequently, we recommend that you update the nVidia video card drivers to latest (2009 or newer) version. If you use laptops with extended display, disable extended monitors and start the Desktop Client.

If the issue still persists, follow this workaround:

1. Add the following properties in the config.ini file in the '<drive>:\Program Files (x86)\CA\WA Desktop Client\configuration' directory:

Dsun.java2d.d3d=false

Dj3d.displaylist=false

2. Disable threaded optimization on NVIDIA driver:

a. Open NVIDIA Control Panel.

b. Turn off Threaded Optimization under In Advanced 3D Setting.

3. Rename the following file as nvoglnt.dll.bak: <drive>:\WINDOWS\system32\nvoglnt.dll

Page 19: CA Workload Automation DE Troubleshooting Guide CA Workload Automation DE R11.3 with Oracle RAC .....20 Cannot Restart the Server Due to RDBMS Oracle Error.....21 Oracle RAC Failover

Copyright © 2014 CA Technologies. All rights reserved. 19

4. Rename or delete the .log in <drive>:\Users\Administrator\workspace-CAWA-11.x\.metadata

5. Restart the system and test.

Page 20: CA Workload Automation DE Troubleshooting Guide CA Workload Automation DE R11.3 with Oracle RAC .....20 Cannot Restart the Server Due to RDBMS Oracle Error.....21 Oracle RAC Failover

Copyright © 2014 CA Technologies. All rights reserved. 20

Troubleshooting the CA Workload Automation DE

Database

This section describes scenarios for troubleshooting database issues.

Validate if the Server Can Connect to the Database You can run the validatedbconnection utility to verify if the database is active. The utility validates the database connection parameters in server configuration. To validate if the server can connect to the database, follow these steps:

1. Do one of the following:

On Windows, change to the following directory at the command prompt: install_dir\bin\

On UNIX, change to the following directory at the command prompt:

install_dir/bin/

install_dir

Specifies the directory where the server is installed.

2. Enter the following command:

On Windows:

validateDbConnection.bat

On UNIX:

./validateDbConnection.sh

The validatedbconnection utility runs in interactive mode. You receive a success or a failure message.

Note: For more information on the failure message, see the Database_Connection.log located in the server installation directory.

Configure CA Workload Automation DE R11.3 with Oracle RAC

To configure CA Workload Automation DE R11.3 with Oracle RAC, refer to the PIB RI59256 on CA

Support Online.

Page 21: CA Workload Automation DE Troubleshooting Guide CA Workload Automation DE R11.3 with Oracle RAC .....20 Cannot Restart the Server Due to RDBMS Oracle Error.....21 Oracle RAC Failover

Copyright © 2014 CA Technologies. All rights reserved. 21

Cannot Restart the Server Due to RDBMS Oracle Error

Symptom:

The server does not restart when the ORA-28000 error occurs. The trace log contains the following error

message:

20091009 11:06:20.074 [ID: 0] CybEspressoRdbFailOverHandler: SQLCode: 28000 SQLState: 99999

Message: ORA-28000: the account is locked.

Solution:

The ORA-28000 error occurs when the Oracle user specified a wrong password consequently for

maximum number of times, which is specified in the user's profile parameter FAILED_LOGIN_ATTEMPTS.

Hence, the DBA user account gets locked.

To fix this issue, wait for PASSWORD_LOCK_TIME or contact your DBA administrator to verify your

password. To change the password that is specified in the server configuration, use the setdbparm

utility.

Oracle RAC Failover is Not Supported in CA Workload Automation DE r11.1

Description:

How to configure the database for CA WA High Availability (failover) in an Oracle RAC Cluster

Environment in r11.1?

Solution:

CA Workload Automation DE r11.1 can be configured with Oracle RAC, but RAC failover feature is not

supported. If a database failover to another node in an Oracle RAC database configuration, the server is

forced to shut down. The server has to be manually re-started to establish connection to the new Oracle

RAC database instance. To configure CA Workload Automation DE database for failover in an Oracle RAC

cluster environment, the required JDBC connection string is given below:

For example, in a 2 node Oracle RAC database instances:

jdbc:oracle:thin:@(DESCRIPTION=(load_balance=on)(FAILOVER=on)(ADDRESS_LIST=(ADDRESS=(PROTOC

OL=TCP)

(HOST= node1-name)(PORT=1521))(ADDRESS=(PROTOCOL=TCP)(HOST= node2-name)(PORT=1521)))

(CONNECT_DATA=(SERVER=DEDICATED)(SERVICE_NAME= service-name)))

where,

node1-name is the Node name of node1

node2-name is the Node name of node2

Page 22: CA Workload Automation DE Troubleshooting Guide CA Workload Automation DE R11.3 with Oracle RAC .....20 Cannot Restart the Server Due to RDBMS Oracle Error.....21 Oracle RAC Failover

Copyright © 2014 CA Technologies. All rights reserved. 22

service-name is the Service name of the database

You can modify the database connection jdbc.URL parameter using the setdbparm utility.

Unable to Restart the Server Due to DB2 Database Exception Error

Symptom:

The server cannot be restarted when the server to DB2 connection failed due to DB2 database exception

error. The server trace log contains the following error message:

20100209 10:25:33.821 [ID: 0] CybEspressoRdbFailOverHandler: com.ibm.db2.jcc.b.mm:

[jcc][t4][2012][11248][3.50.152] Connection authorization failure occurred.

Reason: Password expired. ERRORCODE=-4214, SQLSTATE=28000

Solution:

To fix this issue, contact your DBA administrator and verify the following:

1. The database is running and the server is connected to the database.

2. The database user ID and password are valid and functional.

If the database user's password is reported as expired, do one of the following:

1. Change the password in the DB2 database configuration. Change the rdbms.password parameter in the CA Workload Automation DE database connection properties using the 'setdbparm' utility.

2. Extend the expiration of the database user's password.

Event Trigger Takes Long Time

Symptom:

When I trigger an event for triggering an application generation manually, the server takes long time to

display the application workflow in the Monitor perspective.

Solution:

The server takes long time to display the application workflow in the Monitor perspective, when the

server waits for an existing database connection.

By default, the server can establish up to fifty database connections. You can increase the default value.

If the trace log file contains a message similar to the following, the server is waiting for an existing

connection to complete the task:

20130126 02:23:05.274 [ID: 0] DM.Appl.MY_APPLICATIONS.100: Out of DB connections, waiting: 0

To fix this issue, increase the number of database connections by modifying the database connection

properties.

Page 23: CA Workload Automation DE Troubleshooting Guide CA Workload Automation DE R11.3 with Oracle RAC .....20 Cannot Restart the Server Due to RDBMS Oracle Error.....21 Oracle RAC Failover

Copyright © 2014 CA Technologies. All rights reserved. 23

Server Shuts Down with the SQL Server Exception Error

Description:

The server shuts down with the following error message in the trace log:

com.microsoft.sqlserver.jdbc.SQLServerException: The transaction log for database 'ESP' is full

To find out why space in the log cannot be reused, see the log_reuse_wait_desc column in

sys.databases"

Solution:

The SQL Server database exception error indicates that the CA Workload Automation database

transaction log file is full and has run out of disk space, forcing the server to shut down.

To fix this issue, contact your DBA administrator and review the log_reuse_wait_desc column in

sys.databases catalog view to find root cause.

Server Shuts Down When the Disk Space of DB2 Database is Full

Symptom:

When the disk space of DB2 database is full, the server shuts down with the following error message in

the trace log:

20141106 08:57:23.608 [pubsub] [INFO] DM.Appl.BENCH_100_PERF_VM3_17.169: DELETE publisher:

[key:1069849] [group:BENCH_100_PERF_VM3_17.169]

[criteria:BENCH_100_PERF_VM3_17.169.WINDOWS97@@NORMAL@@] [subscriptions:none]

20141106 08:57:23.610 [relationaldatabase] [ERROR] DM.Appl.BENCH_100_PERF_VM3_24.62:

SQLCode: -289 SQLState: 57011 Message: DB2 SQL Error: SQLCODE=-289, SQLSTATE=57011,

SQLERRMC=TS4CQ, DRIVER=3.50.152

20141106 08:57:23.610 [relationaldatabase] [ERROR] DM.Appl.BENCH_100_PERF_VM3_24.62: SQL

Exception for query: INSERT INTO ESP_WSS_JOB (JOB_ID, JOB_NAME, APPL_ID, WOBGEN_AFM,

STATE_AFM) VALUES (?, ?, ?, ?, ?); the exception is: DB2 SQL Error: SQLCODE=-289, SQLSTATE=57011,

SQLERRMC=TS4CQ, DRIVER=3.50.152

com.ibm.db2.jcc.b.SqlException: DB2 SQL Error: SQLCODE=-289, SQLSTATE=57011, SQLERRMC=TS4CQ,

DRIVER=3.50.152

at com.ibm.db2.jcc.b.wc.a(wc.java:55)

at com.ibm.db2.jcc.b.wc.a(wc.java:126)

at com.ibm.db2.jcc.b.tk.b(tk.java:1593)

at com.ibm.db2.jcc.b.tk.c(tk.java:1576)

at com.ibm.db2.jcc.t4.db.k(db.java:353)

at com.ibm.db2.jcc.t4.db.a(db.java:59)

at com.ibm.db2.jcc.t4.t.a(t.java:50)

at com.ibm.db2.jcc.t4.tb.b(tb.java:200)

at com.ibm.db2.jcc.b.uk.Gb(uk.java:2355)

at com.ibm.db2.jcc.b.uk.e(uk.java:3129)

at com.ibm.db2.jcc.b.uk.zb(uk.java:568)

Page 24: CA Workload Automation DE Troubleshooting Guide CA Workload Automation DE R11.3 with Oracle RAC .....20 Cannot Restart the Server Due to RDBMS Oracle Error.....21 Oracle RAC Failover

Copyright © 2014 CA Technologies. All rights reserved. 24

at com.ibm.db2.jcc.b.uk.executeUpdate(uk.java:551)

at

com.cybermation.espresso.rdbms.PreparedStatementWrapper.executeUpdate(PreparedStatementWra

pper.java:323)

Solution:

To fix this issue, contact your administrator and increase the disk space on the database server.

Archiving or Purging CA Workload Automation DE Database

The best practices for archiving and purging CA Workload Automation DE database and maintaining the

server performance are as follows:

1. Remove active and completed jobs information from the server repository as follows:

Force complete active application generations that are in trouble or having failed/suberror jobs

for long time to ensure the following:

a. CAWA application server is not caching unwanted data.

b. Maximum number of active application generation threshold "active.appl.gen.stop=" in

the espresso.properties or server.properties file does not exceeded. If the threshold

exceeds, the server shuts down.

Issue the 'purgecompletedjobs' CLI command periodically or schedule the command to run once

a day for all applications (for example, over 7 days old or less) depending on the run frequency.

Example:

[root@ajjra01-I112002 bin]# ./cli ajjra01-I112002 7500 ADMIN admin

Connected

ADMIN@ajjra01-I112002:7500(PRIMARY)> purgecompletedjobs olderthan("now less 7 days")

application("*")

2. Scale down job historical data periodically by deleting 'excessive' records from ESP_APPLICATION

and ESP_GENERIC_JOB tables in server database:

a. Stop the server.

b. Issue single SQL statement to clear data from both database ESP_APPLICATION and

ESP_GENERIC_JOB tables.

Page 25: CA Workload Automation DE Troubleshooting Guide CA Workload Automation DE R11.3 with Oracle RAC .....20 Cannot Restart the Server Due to RDBMS Oracle Error.....21 Oracle RAC Failover

Copyright © 2014 CA Technologies. All rights reserved. 25

Example:

DELETE from ESP_APPLICATION where END_DATE_TIME < TO_DATE('2010-09-01','YYYY-

MM-DD') and STATE like 'COMPLETE';

The SQL statement will delete, using the END_DATE_TIME date job historical data from

both ESP_APPLICATION and ESP_GENERIC_JOB tables due to CAWA database schema

uses cascade delete and therefore sets up dependencies between rows in the tables. It

is recommended to keep few months’ old data and DO NOT specify a date that will

impact active workflow in the SQL statement.

Huge History Data in the Database Table Impacts Server Performance

Symptom:

The server performance gets impacted due to huge history data in the database table.

Solution:

The server collects and stores the information about completed Applications in the relational database.

The server performance gets impacted when the volume of history data increases over a certain period

of time.

To improve the server performance, issue the MOVEHISTORYDATA CLI command to move the history

data in the database tables to stage tables. This command has the following format:

movehistorydata [olderthan("olderthan")]

olderthan("olderthan")

Moves history data older than the specified time to stage tables.

Example: This example moves all the history data that is older than a month to stage tables:

movehistorydata olderthan("now less 1 month")

Note: When the data has been moved, a message is added to the tracelog.txt file indicating the number

of Applications and rows that were moved.

Page 26: CA Workload Automation DE Troubleshooting Guide CA Workload Automation DE R11.3 with Oracle RAC .....20 Cannot Restart the Server Due to RDBMS Oracle Error.....21 Oracle RAC Failover

Copyright © 2014 CA Technologies. All rights reserved. 26

Troubleshooting Agents and Jobs

This section describes scenarios for troubleshooting the agents and job failures.

The agent may fail to start with the error message "Unexpected encryption

type. None received. DES expected."

This problem can come when encryption type and key defined for the agent in the Topology does not

match the one specified during agent installation.

Please make sure that the encryption key defined for the agent in the Topology is of same type (DES)

and matches the one defined for the agent during installation.

When there is huge number of agents in agent group configured with CPU load

balancing, then lots of stale messages gets exchanged between agent and

server. This can result in decreased performance of the following:

Change role

Startup of server

Workload processing of server

For additional information on troubleshooting this problem, refer to the PIB Number RI73864

dSeries agent fails to start after the owner of the dSeries installation directory

was changed to a non-root user

Description:

ESP dSeries was installed on a Linux machine as root. After the installation the owner of the ESP_dSeries

directory was changed with the 'chown' command:

chown -R dseries:dseries ESP_dSeries

where, dseries is a normal local user on the server.

On subsequent start up the dSeries agent fails with the following message:

/opt/ESP_dSeries/startServer: WARNING: Could not increase file descriptor limit to 4096.

Solution:

1. Increase the nofile parameter by updating /etc/security/limits.conf:

soft nofile 4096

hard nofile 65535

Page 27: CA Workload Automation DE Troubleshooting Guide CA Workload Automation DE R11.3 with Oracle RAC .....20 Cannot Restart the Server Due to RDBMS Oracle Error.....21 Oracle RAC Failover

Copyright © 2014 CA Technologies. All rights reserved. 27

2. Edit /etc/pam.d/login, and add this line:

session required /lib/security/pam_limits.so

3. The system file descriptor limit is set in /proc/sys/fs/file-max. If its value is >= 65535, no action is

required. Otherwise update the file with this command:

echo 65535 > /proc/sys/fs/file-max to 65535:

4. Logoff/logon with the dSeries userid to verify the new setting with these commands:

ulimit -aS (to display the current limit)

ulimit -aH (to display the hard limit)

5. Verify the 'ulimit -aS' command shows 'open files 4096'

Jobs may go to 'Ready' state and won’t complete if they are run as non-root

user. Following errors may show up in spool file:

---------------------------------------------------------------------------------------------

Output of messages for workload object SECONDJOB/GEGU.10/MAIN

Start date Tue Jan 7 15:30:32 2014

---------------------------------------------------------------------------------------------

-bash: /data/CA/WorkloadAutomation/ESPSystemAgent/cybspawn: Permission denied

cybspawn: unable to obtain the message queue ID. errno:13 Reason:Permission

denied

cybspawn: unable to obtain the message queue ID. errno:13 Reason:Permission

denied

Solution:

Change the permission to 755 on the Directory: /data/CA/WorkloadAutomation/ESPSystemAgent.

Then, restart the agent and see if this solves the issue.

Once the dseries is shut down and brought back after long time jobs which were

missed during that time period will get triggered. What is the way that these

jobs should not be triggered.

If you don't want any missed scheduled executions of events during the outage to trigger after the

server is re-start, we would suggest you suspend the events before the server outage and resume the

events later.

While a failover made by the changerole command some agents were not

updated with active dSeries server and had continuing trying the connection to

the previous one

Most likely, the root cause was the 2nd execution of "changerole" command before the server

completed processing all the "Control Active" messaged received as part of the previous execution

changerole command.

Page 28: CA Workload Automation DE Troubleshooting Guide CA Workload Automation DE R11.3 with Oracle RAC .....20 Cannot Restart the Server Due to RDBMS Oracle Error.....21 Oracle RAC Failover

Copyright © 2014 CA Technologies. All rights reserved. 28

In order to avoid this situation make sure to wait till all the Agents marked as Active by the manager

before issuing the "changerole" command again.

Understanding Agent communications

Communication between ESP dSeries and Agents uses TCP/IP.

The ESP dSeries connects to the Agent, sends a message, waits for an acknowledgment of receipt, and

then disconnects.

When an Agent has something to send to ESP dSeries Manager, it does the same thing: connects, sends,

gets an acknowledgment, and disconnects.

ESP dSeries and Agents have sender and receiver components. The receiver listens on a predefined

TCP/IP port.

When the sender has one or more messages to transmit, it connects to the receiver's port, sends the

messages, and then closes the connection.

Each Agent has a dynamic sender port (i.e. the sender component grabs any free port, when it wants to

send a message to the Manager) and one receiver port (i.e. the communication.inputport ).

The ESP dSeries Manager has a receiver port (i.e. the communication.managerport ) and has a dynamic

sender port (i.e. the sender component grabs any free port, when it wants to send a message to the

Agent) for each connected Agent.

For these reasons, it's important that your firewall rules (were present) should be such that it

accommodates the Agent Server communication method.

Error creating stdout spool file

Symptom:

dSeries job has been failing due to "Error creating stdout spool file.

Solution:

If a dSeries job has failed due to "Error creating stdout spool file" you need to investigate the situation on the related Agent environment, i.e. the Unix/Linux operating system where this Agent is installed. The following are some of the common reasons:

1. The file system ran out of the available disk space. You could check it with the 'df' command (refer to your OS Reference guide for the command usage), for example as it's shown for the IBM AIX:

# df –k Filesystem 1024-blocks Free %Used Iused %Iused Mounted on /dev/hd4 65536 46488 30% 1690 14% / /dev/hd2 5898240 1087404 82% 29116 11% /usr /dev/hd9var 589824 493344 17% 463 1% /var /dev/hd3 2031616 2028036 1% 91 1% /tmp /dev/hd1 65536 63972 3% 51 1% /home

Page 29: CA Workload Automation DE Troubleshooting Guide CA Workload Automation DE R11.3 with Oracle RAC .....20 Cannot Restart the Server Due to RDBMS Oracle Error.....21 Oracle RAC Failover

Copyright © 2014 CA Technologies. All rights reserved. 29

/proc - - - - - /proc /dev/hd10opt 16842752 9292976 45% 22613 2% /opt

2. The permission on the Agent's spool directory was changed; use the 'ls -al' command against the spool directory and its subdirectories to display the permission.

3. The maximum value for a file reference has been exceeded. This value is defined by the operating system standard (LINK_MAX). It is usually 32,000 or 32,768 (depending on the OS). Most likely you haven't cleared the spool files for a long period of time so it might have reached this maximum value (you also might have too many files in the Agent's log directory so it would be useful to check this as well). You would need to move as many spool file as possible from their directories and then repeat this procedure regularly.

For all three possible cases above, stop the Agent first, and then perform necessary actions - if needed

please get help from your UNIX administrator.

A job failed with Status: WARNING: User <user_name> is not authorized to send

AFM to Agent <user_name>

Symptom:

A job failed with Status: WARNING: User dba is not authorized to send AFM to Agent <agent_name>.

Solution:

The WA user that triggered the application doesn't have permissions AGENT.* and AGENTUSR.* for the

agent on which the job is executed. WA administrator needs to add that permission to the WA user in

question.

Page 30: CA Workload Automation DE Troubleshooting Guide CA Workload Automation DE R11.3 with Oracle RAC .....20 Cannot Restart the Server Due to RDBMS Oracle Error.....21 Oracle RAC Failover

Copyright © 2014 CA Technologies. All rights reserved. 30

Common Questions

This section contains common questions related to the server, CA WA Desktop Client, Agents, and

database.

How to Upgrade the Server

To upgrade a previous release of the server, you install the current release and migrate artifacts, global

variables, and history data. Migration preserves the previous server while letting you use the r11.3

server to run your previously defined workload.

Migrating to CAWA DE 11.3, you need to have a new database instance created and install the new

CAWA DE 11.3 release and artifacts, global variables, and history data from the current CAWA DE11.1 to

CAWA DE 11.3 (you should not install on top of the current CAWA DE 11.1). Refer to the Implementation

Guide on upgrading the server.

You can upgrade to r11.3 from the following versions of the server:

r5.0.3

r11.1

Follow these steps:

1. Install the r11.3 server and default agent on Windows or UNIX with the latest service packs applied.

Note: If you install the r11.3 server on the same computer as your existing installation, you must use

different ports.

2. Assess the artifacts for migration impact.

3. Migrate artifacts, global variables, and history data.

4. Add permissions to r11.3 users and groups.

5. Reconfigure agent definitions in the r11.3 Topology, if necessary.

6. Modify agents to work with the r11.3 server.

7. Start workload processing on the r11.3 server.

For Agents, both CAWA DE 11.1 and 11.3 (on same server) can use same Agent R7 installed on other

nodes. There is a note in (1) above.

Note: If you install the r11.3 server on the same computer as your existing installation, you must use

different ports.

Page 31: CA Workload Automation DE Troubleshooting Guide CA Workload Automation DE R11.3 with Oracle RAC .....20 Cannot Restart the Server Due to RDBMS Oracle Error.....21 Oracle RAC Failover

Copyright © 2014 CA Technologies. All rights reserved. 31

How to Install CA License to CA Workload Automation DE r11.3?

Description:

CA Workload Automation DE r11.3 now requires a valid license. If a license has not been applied, you will receive emails with the subject "CA Workload Automation License Failure" from the server.

Solution:

If the CA Workload Automation DE server is the first CA product to be installed on the host, you must initialize the CA licensing program.

1. Run the following script on the Windows command prompt or Linux/UNIX shell:

On UNIX/Linux:

server_install_dir/ca_license_bins/ca_lic_ia_install.sh

On Windows:

server_install_dir\ca_license_bins\ca_lic_ia_install.bat

The script installs CA licensing program in the following directory.

On UNIX/Linux:

/opt/CA/SharedComponents/ca_lic

On Windows:

C:\Program Files\CA\SharedComponents\CA_LIC

2. Copy the ca.olf file in the above CA_LIC directory. A newly created CA_LIC folder will not have any

ca.olf file in it. NB. The ca.olf file should be provided by the CA Licensing.

3. Verify the license from CA WA Desktop Client:

a. Connect to the server from CA WA Desktop Client as an administrator.

b. Click CLI Perspective, enter 'LICENSESTATUS' in the command bar, and press Enter.

The "License has been enforced…." message appears.

Note: If ca.olf file already exists, backup the existing files and merge the new ca.olf to an existing file using the 'mergeolf' utility:

mergeolf <new_olf> [-c <current_olf>] [-o <output_olf>] [-b <backup_olf>] [-d <debug_log>]

Operating System might not be set for the Daylight Savings Time (DST)

adjustment. If it is not set, would CA Workload Automation DE still follow DST?

Yes. CA Workload Automation DE retrieves the DST information from the server JRE and does not use the DST setting at the operating system level. If any recent changes were made at your local time zone, please check the following site:

http://java.sun.com/javase/downloads/index.jsp#timezone

For the latest TZUpdater and if needed apply one to both Desktop Client and the dSeries server JREs.

The TZUpdater installation instructions could be found on aforementioned site.

Page 32: CA Workload Automation DE Troubleshooting Guide CA Workload Automation DE R11.3 with Oracle RAC .....20 Cannot Restart the Server Due to RDBMS Oracle Error.....21 Oracle RAC Failover

Copyright © 2014 CA Technologies. All rights reserved. 32

When applying an upgrade patch should the CA Workload Automation DE

dSeries 11.3 database instance be up and running?

Yes, when applying a patch the dSeries 11.3 database instance should be up and running.

Do desktop client and CAWA (dSeries) server of different version work

together?

No. Desktop client R11.3 only work with CAWA server R11.3; Desktop client R11.1 only work with CAWA server R11.1; Desktop client R5 only work with dSeries R5; -> Desktop Client 2.0.* only work with dSeries R5 (End of Support – January 31, 2013).

What are the Oracle 10.x/11.x certified character sets for dSeries server?

The certified character sets for Oracle 10.x/11.x are WE8ISO8859P1/WE8MSWIN1252 respectively. Oracle Unicode is not supported.

What to check if Agent is not communicating with CA (dSeries) Workload

Automation Server?

If the System Agent could not communicate with CA (dSeries) Workload Automation Server, you can issue 'listagent' command from the Command Line Interface (CLI) on the Desktop Client to check the Agent status or check the content of 'status' and 'status.os' files (located in the Agent installation directory). The latter would provide additional information about the Agent, including the last Agent start information. To diagnose the problem, you can check the following;

1. Do you have any communication/network problem? 2. Did Manager connection parameters change in the agentparm.txt file (located in Agent

installation directory) 3. Did Agent connection parameters change in the server Topology in Admin perspective on the

Desktop Client? 4. Did Manager and/or Agent IP addresses change? 5. Did Agent not start properly? 6. Is there Firewall between CA (dSeries) Server and Agent? 7. Did the Firewall rules change? 8. Does the encryption key is correct? 9. Is the agent name correct? Check the agent name in agentparm.txt file. 10. Is the agent port correct. The port info can be located inside the agentparm.txt file.

Does CA Workload Automation DE (dSeries) server need to be down if we only

purge the history staging tables (H_*)?

No, the CA Workload Automation DE (dSeries) server does not need to be stopped if the purging was intended for the staging tables only. These tables are not updated either for data or index by dSeries.

Page 33: CA Workload Automation DE Troubleshooting Guide CA Workload Automation DE R11.3 with Oracle RAC .....20 Cannot Restart the Server Due to RDBMS Oracle Error.....21 Oracle RAC Failover

Copyright © 2014 CA Technologies. All rights reserved. 33

Do I need to have all involved databases and CA Workload Automation dSeries

servers running while migrating to version 11.3?

When migrating to CA Workload Automation dSeries version 11.3 there are two flows that need to take place:

The migration utility obtains all artifacts, except for the history and the global variables, via the source server and stores them into the "migration\tmp\<migration_dir>" directory; then the destination server copies these artifacts into its database during the start routine;

The migration utility obtains the history data and the global variables and places them into the destination server's database directly by itself, i.e. without dSeries server’s involvement.

Therefore, in order to migrate the artifacts into the server 11.3 you would need to start up the source server (either the 5.0.3 or the 11.1 version) and the destination (the 11.3) server's database; then start the destination (the 11.3) server.

How to cold start CA Workload Automation (CA WA) and what happens after

cold start?

Symptom:

If CA Workload Automation shuts down improperly, it can result in inconsistencies in the database. This

can result in several issues, such as events not triggering properly to server shutdown. Cold start is then

required to clear these inconsistencies and errors in the database.

Solution:

To do a cold start of CA Workload Automation, rename the runonce.properties.bak to

runonce.properties in the install directory and start the CA WA.

Note: For high availability configuration, do this on both servers.

Cold start will do the following:

Clear out current and active workflow only.

The Monitor perspective will not show any application, since active workflow has been cleared

out which includes FAILED and COMPLETED generations.

No new event will be triggered after cold start. User can manually trigger them or issue

SCHEDULEALLEVENTS command from CLI.

All the historical data (completed jobs) is still kept intact, they are just not visible in Monitor

perspective.

All the events data is still intact. Users can navigate to Services -> Events and see all the events.

Users can query ESP_APPLICATION table in the CA WA database to see all the historical data. Here is a

query that will select all the rows from the defined date:

Page 34: CA Workload Automation DE Troubleshooting Guide CA Workload Automation DE R11.3 with Oracle RAC .....20 Cannot Restart the Server Due to RDBMS Oracle Error.....21 Oracle RAC Failover

Copyright © 2014 CA Technologies. All rights reserved. 34

SELECT APPL_NAME, JOB_NAME FROM ESP_APPLICATION WHERE END_DATE_TIME >='06-NOV-10';

(Note: Change '06-NOV-10' to date that is more suitable.)

All events can be triggered with SCHEDULEALLEVENTS from CLI. This will only schedule all the events, and NOT trigger them. Events will trigger as per schedule. If a user executes SCHEDULEALLEVENTS at 1PM, then an event that is supposed to trigger at 02:00 p.m., will get scheduled to run at 02:00 p.m. It will not execute at 01:00 p.m.

CA Workload Automation dSeries server has five levels to implement the cold

start. What are those levels meanings?

The cold start is purposed to clear out active workload and delete the events' schedules. However there are five levels of the cold start which allow to vary the behavior:

Start.type.level = -2 The server starts with normal cold start, but scheduled events that have not been processed at the time of the shutdown will be re-scheduled. Events that require a manual trigger are not preserved.

Start.type.level = -1

The server starts with a normal cold start, the generations count for all applications is set to zero - these applications will be purged also.

Start.type.level = 0 The server cold starts its scheduler and active workload (nothing is preserved).

Start.type.level = 1 The server starts with the cold start, but all scheduled events are preserved. Events that were not yet processed at the time of the shutdown continue to be triggered after the server is started. Events that require a manual trigger are also preserved.

Start.type.level = 2 The server starts with the cold start, but active workload is preserved. Workload that was running at the time of shutdown continues to run after the server is started.

Is it possible to retrieve users from two different LDAP servers at once?

dSeries 11.3 release can connect only to a single LDAP server at one time to retrieve users from configured LDAP directory location.

Even in case of multiple LDAPs defined in dSeries for a load balancing purpose, we expect them to be identical (i.e., use the same directory location to pull users from first LDAP server - if it is not available check with second LDAP server with the same directory/user location and so forth).

What values should be assigned to the ldap.propertyNameForLogin= statement

in the server.properties file?

The values assigned to the ldap.propertyNameForLogin= statement depend on how the LDAP user's login names were configured in the LDAP server by the LDAP administrator.

Page 35: CA Workload Automation DE Troubleshooting Guide CA Workload Automation DE R11.3 with Oracle RAC .....20 Cannot Restart the Server Due to RDBMS Oracle Error.....21 Oracle RAC Failover

Copyright © 2014 CA Technologies. All rights reserved. 35

If the user's login names were configured in the "cn" property of the Microsoft AD or Novel-eDirectory LDAP servers, then the ldap.propertyNameForLogin=cn

For the SUNONE LDAP it is ldap.propertyNameForLogin=uid.

If the user's login names were configured in other than either the "cn" or the "uid" properties then the ldap.propertyNameForLogin= needs to be set so that the dSeries server could get specified property values and shows them as user's login names, for instance:

ldap.propertyNameForLogin=sAMAccountName

What flavor of the JDK 6 should be chosen for the Web Services installation - 32

or 64-bit?

Currently the dSeries Web Services installer exists in the 32-bit compilation only therefore the JDK 6 should be chosen of 32-bit as well.

How to disable CLI log in dSeries 11.1?

Description:

By default CLI has log turned on, generating log files every time it starts.

Solution:

To turn off the CLI log, edit the CLI startup script as followed:

On Windows, edit the <dSeries>\Utilities\cli.bat comment out line containing LOG_FILE and add another line so it looks like: REM set LOG_FILE=C:\Program Files\CA\Workload Automation\clilog.txt

On Unix, edit the <dSeries>/Utilities/cli comment out line containing LOG_FILE and add another line so it looks like: #LOG_FILE=/opt/sunje01/dseries11.1/clilog.txt LOG_FILE=/dev/null

How can I check if the port is available, the firewall does not block the traffic,

and the service is running on it?

Telnet to <IP address>: <port number>.

For example:

C:\>telnet host1:80 C:\>Connecting to host1:80...Could not open a connection to host: Connect failed

Page 36: CA Workload Automation DE Troubleshooting Guide CA Workload Automation DE R11.3 with Oracle RAC .....20 Cannot Restart the Server Due to RDBMS Oracle Error.....21 Oracle RAC Failover

Copyright © 2014 CA Technologies. All rights reserved. 36

In this example, the port (port 80) is currently unavailable for the service. The web server is not running or is not authorized through the firewall.

How to find the version and build number of the CA Workload Automation DE

server?

You can run the version utility to verify the version and build number of the server. To verify the server version and build number, follow these steps:

1. Change to the following directory at the command prompt:

On Windows: install_dir\bin

On UNIX: install_dir/bin install_dir Specifies the server installation directory.

2. Enter the following command:

On Windows: version.bat

On UNIX: ./version.sh

Note: Alternatively you can also connect to desktop client as ADMIN user run the cli command “about” from the CLI perspective.

How to find the version and build number of CA Workload Automation Agent?

To find the version and build number of an agent, follow these steps:

1. Change to the agent installation directory on the command prompt. 2. Enter the following command:

On Windows: cybAgent –v

On UNIX: ./ cybAgent –v

How to find the version and build number of CA WA Desktop Client?

To find the version and build number of CA WA Desktop Client, follow these steps:

1. Launch the desktop client. 2. Click on the menu Help and then the sub menu option “About CA Workload Automation Desktop

Client”.