CA AutoSys WA-CAAutoSys WCC Workload Automation r11 Troubleshooting

CA AutoSys Workload Automation r11 Automation r11 TroubleshootingPresenters: Dan Shannon Elizabeth DexterPresenters: Dan Shannon, Elizabeth Dexter

Terms of This Presentation

This presentation was based on current information and resource allocations as of October 2009 and is subject to change or withdrawal by CA at any time without notice. Notwithstanding anything in this presentation to the contrary, this presentation shall not serve g y g p y, pto (i) affect the rights and/or obligations of CA or its licensees under any existing or future written license agreement or services agreement relating to any CA software product; or (ii) amend any product documentation or specifications for any CA software product. The development release and timing of any features or functionality described in this presentation development, release and timing of any features or functionality described in this presentation remain at CA’s sole discretion. Notwithstanding anything in this presentation to the contrary, upon the general availability of any future CA product release referenced in this presentation, CA will make such release available (i) for sale to new licensees of such product; and (ii) to existing licensees of such product on a when and if-available basis as part of CA maintenance and support, and in the form of a regularly scheduled major product release. Such releases may be made available to current licensees of such product who are current subscribers to CA maintenance and support on a when and if-available basis. In the event of a conflict between ppthe terms of this paragraph and any other information contained in this presentation, the terms of this paragraph shall govern.

2 October 2009 [CA AutoSys Workload Automation r11 Troubleshooting] Copyright © 2009 CA

For Informational Purposes Only

Certain information in this presentation may outline CA’s general product direction. All information in this presentation is for your informational purposes only and may not be incorporated into any contract. CA assumes no responsibility for the accuracy or completeness p y p y y pof the information. To the extent permitted by applicable law, CA provides this document “as is” without warranty of any kind, including without limitation, any implied warranties or merchantability, fitness for a particular purpose, or non-infringement. In no event will CA be liable for any loss or damage direct or indirect from the use of this document including liable for any loss or damage, direct or indirect, from the use of this document, including, without limitation, lost profits, lost investment, business interruption, goodwill, or lost data, even if CA is expressly advised of the possibility of such damages.


Agenda:

>Preparing for the Installation

>Installation and Initial Configurationg

>Customisation and Tuning

>Common Issues


Preparing for Installation

> ArchitectureWhat are you going to install, and where?

What are you planning to support?

> SystemM hi i iMachine sizing

> OS requirementsPatches, librariesPatches, libraries

> NetworkingCan the machines communicate?


Installation

> Verify that the common component installs were successful

> Verify that you can communicate between machines


Architecture


WCC PortsServices Related Applications Start Stop SSL

Launcher Server: WCC MainWCC Main LoginWeb ServicesWeb Services Samples ApplicationCredential Application

8080 8005 8443

C fi ti S C fi ti 10132 10133 10134Configuration Server: Configuration 10132 10133 10134

Event Server: Event ConsoleQuick Edit

10138 10139 10140

Job Status Console Servant: 10143 10144 10145

Job Status Console Server: Job Status Console 10122Job Status Views

Monitoring Server: Job Flow DesignJob Flow MonitoringCritical Path Monitoring

10135 10136 10137

h d lScheduling Server: Job EditorJob Editor Plus (BLOB editor)Application Editor

10129 10130 10131

UI Framework Server: Quick ViewQuick StartEnterprise Command Line

10146 10147 10148


HA Server: High AvailabilityReporting Collectors

10149 10150 10151

Installation and Initial Configuration

> Installing the Common ComponentsSSA

EEM

CCS/Event Management

> C ti it> ConnectivityEEM

AutoSys/WCC/SSAy / /

> Multicast (WCC)


Common Issues

> Collecting Information for SupportRelease, maintenance level, logs, system information

> SSA Issues

> EEM Issues

> AutoSys Issues

> WCC Issues

> Multiple NIC cards > Multiple NIC cards

> Command Sponsor is not installed


Logging and Debugging

> CA Workload Automation AutoSys Logging

> EEM Logginggg g

> WCC Logging

> SSA Logging


CA Workload Automation AutoSys Logging

> ISDBGACTIV

> Log Filesgunivagent.out

as_server.INS

t d INSevent_demon.INS

auto_rem.joid.runnum


EEM Logging (SDK)

> New With SDK 8.4 SR01

> EIAMCONFIG Environment

> Configuration Fileseiam.config

logger.config


WCC Logging

> WCC Logging

> Location: <WCC Install Root> Logs folderUses log4jUses log4j

– Access (not currently used)

– ApplicationAdmin, config, event, haserver, jobcontrolservant, jobcontrolserver, launcher, monitoring, reporting, uiframework

Log4j propertiesLog4j.properties

– Install

– Services

T t L> Tomcat Logs<WCC Install Root>/xxx/logs


Determining the Version

> AutoSysautoflags

> WCCuejmver.bat (Windows), uejmver.sh (*NIX)


SSA Logging

> Environment VariablesCSAM_SSL_DEBUG

CSAM_DEBUG

CSAM_LOG


Directory Layout/ o p t / C A / U n i c e n t e r A u t o S y s J M

a r c h i v e b i n

$ A U T O S Y S

a g e n t

o u t$ A U T O U S E R

o u t l i b

d b o b j O R A

I N G

t x

r e q u e s t

j o b s t

s n m p

d a t ai n s t a l l

S Y B

d o c m a n 8

r e s p o n s e

m e s s a g e s

t e s t j i l

b i n

n i t e l y

C o n f i g u r a t i o n F i l e s :

* c o n f i g . $ A U T O S E R V* a u t o s y s . s h . h o s t* a u t o s y s c s h h o s t b i b


d e m oO u t T h e D o o r

a u t o s y s . c s h . h o s t* a u t o s y s . k s h . h o s t* a u t o s y s . b a s h . h o s te t c .

b i n _ ba k . . .

l i b _ ba k . .

SSA Related Files


SSA Configuration Global


SSA Configuration Port (Cont)


SSA Application Logic


SSA Configuration Port


AutoSys Communication Problems

> CAUAJM_E_10029 Communication attempt with the Unicenter AutoSys JM Application Server has failed! [host:port]. [host:port].

Can you ping the App Server host?

Is the App Server running?

Are AutoServer/AutoServerPort values correct?

Is port 7163 (SSA) blocked by firewall?

Are SSA port configurations the same on client & App Are SSA port configurations the same on client & App Server machines the same?

– EnablePmux

E bl SSL– EnableSSL


AutoSys Communication Problems

> CAUAJM_E_10527 Timed out waiting for response from the Unicenter AutoSys JM Application Server [host:port]

Is the App Server running?Is the App Server running?

Are there messages in the App Server log file?– ISDBGACTIV=LIGHT,LOG

Is the SSA PortRange for 49152-50176 configured correctly?

– EnablePmux

– EnableSSL

Do the App Server and RDBMS machines have adequate resources?resources?

Does increasing DB_CONNECTIONS reduce problem?


Troubleshooting the Scheduler

Job stuck in STARTING

> Did Scheduler complete the Agent handshake?> Did Scheduler complete the Agent handshake?CAUAJM_I_10082 [POLAL02-GX280 connected for job1 626.225.1]

> What does corresponding Job Agent log file show?CAUAJM_E_10527 Timed out waiting for response from the Unicenter AutoSys JM Application Serverthe Unicenter AutoSys JM Application Server.

CAUAJM_E_10221 Exhausted list of application servers. Failing request.


Troubleshooting the Agent

CAUAJM_E_50026 ERROR: AutoPing WAS NOT SUCCESSFUL!

> Does univagent log file show activity under ISDBGACTIV=LIGHT,JOB traces?

> Do OS network tools reach Agent machine? (ping, nslookup, tracert/traceroute)

> Is firewall blocking SSA physical port 7163?> Is firewall blocking SSA physical port 7163?> Do SSA settings for AutoServerPort match (autoping –D)? > Do SSA settings for PortRange=49152-50176 match?

EnablePmuxEnableSSL


EEM Troubleshooting

> Performance IssuesCheck logging levels

Active Directory connectivity

Linux kernel

Cache settingsCache settings

> Unexpected permissions errorsCheck for mixed case directory entries y

> Active Directory Group RetrievalFilters


EEM Troubleshooting

CAUAJM_E_10448 FATAL SECURITY ERROR: Tampering or corruption of the security-related database keys detected

> Did someone install a new instance using an MDB from a previous instance installation?previous instance installation?

> Did someone directly modify the ujo_alamode or ujo_keymaster keys?j _ y y

> What is the state of the database tables? Did some unknown corruption of keys take place?


EEM Troubleshooting

CAUAJM_E_10436 Security server unreachable or invalid authentication certificate file

> Is the EEM backend running?> Do OS network tools reach EEM backend machine? (ping > Do OS network tools reach EEM backend machine? (ping,

nslookup, tracert/traceroute)> Is firewall blocking EEM physical port 5250?> Was the proper EEM backend host name entered?> Are you able to connect to the EEM backend via the

as safetool?as_safetool?> Does the AUTOUSER folder contain a valid certificate file?29 October 2009 [CA AutoSys Workload Automation r11 Troubleshooting] Copyright © 2009 CA

EEM Troubleshooting

Policy problems

> Does the EEM web UI Permission Check work as expected?

> Does autosec test work as expected?> Does autosec_test work as expected?> Does as_safetool work?


WCC - Other

> Job Flow MonitoringMonitor ID

Effective limit on the number of jobs in a flow

> Job Status ConsoleGl b l S i /Gl b l IDGlobal Session/Global ID

Prior to WCC r11.1 SP1, check alarms/alerts filters

> Unexpected login prompts> Unexpected login promptsIAM Security setting in server definition


Command Sponsor Troubleshooting

> Required for Quick View and ECLI in WCC

> Requires a credential user that is authorised on the OS to qexecute the command

> Restricted to the commands in the AutoSysCommandISponsorFilters txt fileAutoSysCommandISponsorFilters.txt file

> Uses iGateway


Questions?


Documents

CA AutoSys WA-CAAutoSys WCC Workload Automation r11 Troubleshooting