CA Spectrum Infrastructure Manager Troubleshooting · PDF fileCA Spectrum Troubleshooting Guide for CA Support Partners ... at CA Support and Customer Care for Businesses and Partners

2

This publication is based on current information and resource allocations as of its date of publication and is subject to change or withdrawal by CA at any time without notice. The information in this publication could include typographical errors or technical inaccuracies. CA may make modifications to any CA product, software program, method or procedure described in this publication at any time without notice.

Any reference in this publication to non-CA products and non-CA websites are provided for convenience only and shall not serve as CA’s endorsement of such products or websites. Your use of such products, websites, and any information regarding such products or any materials provided with such products or at such websites shall be at your own risk.

Notwithstanding anything in this publication to the contrary, this publication shall not (i) constitute product documentation or specifications under any existing or future written license agreement or services agreement relating to any CA software product, or be subject to any warranty set forth in any such written agreement; (ii) serve to affect the rights and/or obligations of CA or its licensees under any existing or future written license agreement or services agreement relating to any CA software product; or (iii) serve to amend any product documentation or specifications for any CA software product. The development, release and timing of any features or functionality described in this publication remain at CA’s sole discretion.

The information in this publication is based upon CA’s experiences with the referenced software products in a variety of development and customer environments. Past performance of the software products in such development and customer environments is not indicative of the future performance of such software products in identical, similar or different environments. CA does not warrant that the software products will operate as specifically set forth in this publication. CA will support only the referenced products in accordance with (i) the documentation and specifications provided with the referenced product, and (ii) CA’s then-current maintenance and support policy for the referenced product.

Certain information in this publication may outline CA’s general product direction. All information in this publication is for your informational purposes only and may not be incorporated into any contract. CA assumes no responsibility for the accuracy or completeness of the information. To the extent permitted by applicable law, CA provides this document “AS IS” without warranty of any kind, including, without limitation, any implied warranties of merchantability, fitness for a particular purpose, or non-infringement. In no event will CA be liable for any loss or damage, direct or indirect, from the use of this document, including, without limitation, lost profits, lost investment, business interruption, goodwill or lost data, even if CA is expressly advised of the possibility of such damages.

COPYRIGHT LICENSE AND NOTICE:

This publication may contain sample application programming code and/or language which illustrate programming techniques on various operating systems. Notwithstanding anything to the contrary contained in this publication, such sample code does not constitute licensed products or software under any CA license or services agreement. You may copy, modify and use this sample code for the purposes of performing the installation methods and routines described in this document. These samples have not been tested. CA does not make, and you may not rely on, any promise, express or implied, of reliability, serviceability or function of the sample code.

Copyright © 2012 CA. All rights reserved. All trademarks, trade names, service marks and logos referenced herein belong to their respective companies. Microsoft product screen shots reprinted with permission from Microsoft Corporation.

TITLE AND PUBLICATION DATE: CA Spectrum Troubleshooting Guide for CA Support Partners

Publication Date: September 25, 2012

CA Support Partner Troubleshooting Guide 3

CA would like to thank the following contributors:

CA Support

4

This problem determination guide is provided for use to CA Partners to assist with troubleshooting and problem determination as related to CA solutions and integrated services. Use these guidelines to help expedite the case handling process.

If additional assistance is required to resolve the customer inquiry or reported issue, please contact CA Support at CA Support and Customer Care for Businesses and Partners to locate the relevant contact information for the product or solution in question.

http://www.ca.com/us/customer-care.aspx


Spectrum Component List ........................................................................................................................................... 6

Integration Issues ........................................................................................................................................................ 6

Applications ............................................................................................................................................................... 13

Certification ............................................................................................................................................................... 15

Core / SpectroSERVER ............................................................................................................................................... 16

Discovery and Modeling ............................................................................................................................................ 18

Licensing .................................................................................................................................................................... 20

OneClick ..................................................................................................................................................................... 21

Reporting ................................................................................................................................................................... 21

Contacting CA Support .............................................................................................................................................. 25

The information below is routinely asked for by our CA Support Engineers during the first call with a customer in an effort to begin analysis and triage of an issue.

When opening a new CA Spectrum issue online, please include the general information requested below as well as any component specific details. This then will expedite the analysis for your particular issue as soon as it is assigned to the appropriate engineer.

In some instances, the same information may be requested multiple times for each scenario that is reproduced or each occurrence of a scenario.

The following component group codes should be used when opening a new issue online.

Below each component group code is a list of subcategories. Subcategories are not exposed to customers through the CA Support Online web site, but are used internally by CA Support.

■ Integrations

■ Alarm & Event Management

■ Applications

■ Certification

■ Core / SpectroSERVER

■ Discovery and Modeling

■ Licensing

■ OneClick

■ Reporting

Issues being reported under the Integrations component will be for one of these subcategories:

CA CMDB CA NSM

CA EEM CA SAM Spectrum Connector

CA eHealth CA Service Desk

CA Event Integration CA Wily Introscope

CA Insight Integrations Other

CA IT PAM/BMC Gateway Microsoft SCOM


The following type of information will be necessary to help troubleshoot your issue under the Integrations component.

■ What versions of the integrating components are running?

■ What part of the integration is not working?

To assist with eHealth integration issues, review the following known problems and follow the debugging instructions.

1. Can an eHealth report be run from a browser on SPECTRUM system?

a. If a report does not run from the eHealth web interface, then the problem is on the eHealth side, and the issue should be transferred accordingly.

2. Has the SPECTRUM-eHealth mapping been done correctly?

a. First, check the SPECTRUM administration page for eHealth Configuration. Confirm that the eHealth information for the eHealth topology is correct.

b. There may an issue with the mapping of eHealth elements to Spectrum models.

– Turn on the two eHealth mapping debug options in the OneClick Web Server Debug page.

– Run a “Map By IP” on one of the devices in question.

– Look for the words “Match summary” in the OneClick server logs.

– Find the device or port in question to see what is mapped to the model.

There may be more than one mapping for an interface or the wrong mapping for a device, so the user can set up a manual mapping in the mappings-override.xml file in tomcat\webapps\spectrum\WEB-INF\ehlth\config directory.

c. If the mapping is not being created you can enable eHealth Detailed Mapping Logging to record the web calls between Spectrum and eHealth.

Note: The issue may turn out to be a communication issue between the servers.

1. Check the Trap destination configuration in Live Exceptions to make sure SpectroSERVER name, IP and port are setup correctly. If the customer is using Secure Domain Manager with Spectrum they must specify the port of 4748 here.

2. If this is setup correctly, next check for any errors on the EventAdmin container representing the eHealth server and the VNM.

– Also, keep in mind that the customer can’t have the eHealth server modeled as a host and an EventAdmin in Spectrum 8.1, as this is not supported and the traps won’t be routed correctly.

8

– In Spectrum 9.0 H03 we added support for modeling the eHealth server as a SystemEdge and using this model type to represent the eHealth server instead of the EventAdmin model.

3. If there are no errors about processing any Live Health events, set up a sniffer trace on the Spectrum computer to see if the trap is arriving in Spectrum.

– Also look to see how many traps we are receiving on the Spectrum server

– Could a possible trap storm be the cause of missing certain Live Health alarms?

– In the sniffer trace, filter for any trap OIDs that contain 1.3.6.1.4.1.149, which is the OID for Live Exception traps.

4. If Spectrum is not receiving any traps from eHealth then there is a problem on the eHealth side with Live Exceptions not sending the traps.

– A sniffer trace should also be set up on the eHealth server to see if traps are being sent.

5. What is the setting for NH_TRAP_GOVENOR_SIZE? If this set too low it may be a cause for the problem.

a. This is the definition from the r6.1 eHealth Command and Environment Variables Reference Guide:

NH_TRAP_GOVERNOR_SIZE specifies the maximum number of traps to queue to the Live Exceptions notifier server at one time. Excess traps are discarded. Set this variable to a number that characterizes an abnormal amount of traps sent on your network.

b. The purpose of this variable is to prevent the notifier server from being overwhelmed by a sudden burst of alarms triggering it to send large numbers of traps.

– Alarms are sent in batches to the notifier, usually right after a poll period.

– The notifier queues traps to be sent out until the limit specified by this variable is reached (1,000 by default).

– At that point, the notifier stops adding new traps to the queue and proceeds to send them.

– After the whole batch of traps has been sent, the queue is empty and ready for the next batch.

– If there is a sudden burst of 1000+ alarms, it is clear that something serious has happened to a major portion of your network, and it is unnecessary to continue accumulating sympathy traps, so they are discarded. You want to be sure your clear alarm is not one of the alarms being discarded.

c. Here is an explanation on how to turn on eHealth debug logging in Spectrum OneClick and what each logging option details. The logging pages are located on the Administration link of the Spectrum OneClick web page.

– Click the Debugging link in the horizontal grey bar.

– Click the Web Server Debug Page (Runtime) link on the left grey bar to find the settings. The eHealth logging is at the bottom of the page.

– You can also view the logs by clicking the Web Server Log link on the left grey bar.


eHealth Discovery debugging information

Provides detailed logs on eHealth discovery.

eHealth Mapping Detailed Logging

Provides detailed logs about the element mapping, including the raw information that ALL of the eHealth web services send to SPECTRUM (including the discovery web service). These logs can be verbose and should only be enabled if Information logging does not provide enough data to accurately assess the issue.

eHealth Mapping Information

Provides informative set of logs, when this setting is turned on, for tracking the work that the element mapping does.

eHealth Synchronized Discovery Logging

Provides more information about the activities surrounding synchronized discovery. If you are experiencing problems or alarms with synchronized discovery, you can turn this setting on to see what is going wrong. Only this log setting supports Min/Max settings at the bottom of the page to control Info/Debug level logging.

To assist with Spectrum-NSM integration issues, review the following known problems and follow the debugging instructions.

1. Verify Alarmnotifier.exe (or UnicenterNotifier.exe) is running in Task Manager on the Spectrum server. If it is not running, open a command prompt then navigate and run <SPECROOT>\UnicenterNotifier\Alarmnotifier.exe (or UnicenterNotifier.exe in 9.0).

2. If is it running, check the Notifier.OUT file in <SPECROOT>\UnicenterNotifier directory for any errors. There could be a problem with the awtrap or cawto commands in the SetScript that are called to forward the alarms over to NSM.

3. If you have issues starting Alarmnotifier.exe, see the AlarmNotifier troubleshooting section.

4. If AlarmNotifier seems to be running fine, there may be a problem with the alarm getting to NSM. Add ‘echo’ in front of the cawto or awtrap command in the SetScript in the UnicenterNotifier directory. This will output what we’re trying to send to NSM in the Notifier.OUT and indicate if there are any errors.

5. If this looks good, setup a sniffer on the NSM server to see if we’re getting the traps (if they’re using awtrap).

Follow the steps documented in Techdoc TEC472065 to resolve this problem.

10

Techdoc TEC472065 has been created to address this problem.

For version 3.3 of the integration there is a troubleshooting section, Chapter 5: “Verifications and Troubleshooting” in the Inside the Spectrum with Unicenter NSM Integration Guide.

1. Verify that Microsoft MOM Connector (an add-on for the Foundation and Standard Suites) has been purchased and installed. Get the license number, a copy of the license, or install information. The integration will not be available if they did not purchase the product.

2. Verify the Microsoft .NET Framework v3.0 is installed. This is a requirement of the integration.

3. Does the Spectrum administrative user have the Admin User Role within SCOM? This is a requirement of the integration.

4. Get a copy of the <SPECROOT>/SCOMConnector/.scomrc file verify configuration parameters.

5. To enable debug

a. Add the following line to the <SPECROOT>/SCOMConnector/.scomrc file: debugLevel=3

b. Restart the connector process to have the change take effect.

c. Output is written to the <SPECROOT>/SCOMConnector/CONNECTOR_DEBUG.OUT

Send the output file to CA Support for review.

Many issues with the Service Desk integration with Spectrum have been due to incorrect parameters being used during the initial setup.

1. Compare the parameters being set: Get screenshots of the Service Desk configuration window and of the OneClick setup window (navigate to the OneClick home page (http://<machinename>), click on Administration, Service Desk Configuration).

2. Make sure you have created a Spectrum contact in the Service Desk.

3. Make sure your Web Services Policy is associated to the SPECTRUM problem type (SPECTRUM_PT) and that this is the default.

4. Verify the Web Services Policy is associated to the Spectrum_Template.


a. Turn on debug by running the OneClickIntegrationSetup utility located in the /<ServiceDesk install are>/bin.

When running this utility there is a prompt asking, “Enable logging? [yes|no].” When replying “yes” to this, it will output an “oc-notification.log” file on the Service Desk machine under the directory /<service_desk_install_area>/bin/. It will output data on submits, updates and clears. Sample output:

(Thu Mar 29, 2007, 12:50:45) - Received notification (Thu Mar 29, 2007, 12:50:45) - Close notification (Thu Mar 29, 2007, 12:50:46) - Operation succeeded.

b. Request a copy of the oc-integration.cfg file located in the /<service_desk_install_area>/bin/ directory

View the configuration settings the customer specified when setting up the integration.

c. If there are communication problems, navigate to the OneClick home page (http://<machinename>), click Administration, Service Desk Configuration and click the Test button on the page.

You can view the error if there are communication issues.

d. Enable debug from the OneClick debug page. There is a “Service Desk Integration” setting that can be turned on. Reproduce the issue, and then get the log file.

See if there are any exceptions or information messages pertaining to the issue.

Issues being reported under the Alarm & Event Management component will involve one of the following subcategories:

■ Alarms

■ Alarm Event Management

■ Archive Manager DDM

■ Event Configuration Editor

■ Other Events

■ Southbound Gateway

The following type of information will be necessary to help troubleshoot your issue under the Alarm & Event Management component.

■ Alarm probable cause code ID is useful, along with event ID information.

12

Problem: Trap, Event, or Alarm not being asserted on properly. Mapped events are not showing up properly in OneClick

1. These files are located on the SpectroSERVER computers in the $SPECROOT/SS/CsVendor/ and $SPECROOT/custom/Events directories.

2. These files are used to verify mapping through the Event Management System.

3. These files are used to see what the customer has done.

4. These files are used to attempt to duplicate the issue.

Use this trace to see how the traps are being sent into SPECTRUM. On Windows systems, use wireshark; on UNIX systems, use snoop.

Using this trace allows us to cross reference the traps with the entries made in the Alertmap files.

This allows us to see what SPECTRUM is doing with the event.

If the event seen in OneClick Event tab is different than what is expected, you need to check for and verify that the CsEvFormat file in the $SPECROOT/SG-Support/CsEvFormat directory is correct.

Check for the same filename in the $SPECROOT/custom/Events/SG-Support/CsEvFormat directory.

Files in this directory take precedence over files in the directory $SPECROOT/SG-Support/CsEvFormat. The same applies to AlertMap, EventDisp, EventTables and CsPCause files.

OneClick caches this info at startup and needs to be reloaded like the SpectroSERVER.


To check that the Apache Tomcat cache has been updated:

1. Navigate to the Start Console page for OneClick (http://<machinename>:port/) and

2. Click on the Administration link.

3. Click on the CsEvFormat/CsPCause entry and click Reload in the window on the right.

To update, do ONE of the following:

■ Click the Update button in ECEditor.

■ Issue the Update Action command in CLI.

■ Press the Update Event Files button from the SpectroSERVER Control windows on the VNM model.

Issues being reported under the Applications component will be for one of the following subcategories:

■ Alarm Notifier ■ Performance Managers

■ Applications Other ■ QOS Manager

■ ATM Manager ■ SANM

■ CLI ■ Service Manager

■ MPLS Manager ■ TL1 Gateway

■ Multicast Manager ■ Toolkits

■ Network Configuration Manager ■ VPN Manager

The following type of information will be necessary to help troubleshoot your issue under the Applications component.

14

■ Get a copy of the $SPECROOT/Install-Tools/.history file.

This data can be pasted into the support ticket or attached to the ticket.

■ Identify the version of the running AlarmNotifier executable, seen in the output file or screen as:

./AlarmNotifier

– AlarmNotifier 9.0.1.003

– Multiple Landscapes Mode

– SPECTRUM Alarm Notification Manager Enabled

■ Getting the Alarm probable cause code ID is useful, along with event ID information.

■ For SANM issues, policy information will be necessary.

■ For Alarm Notifier and CLI issues that involve custom scripts, a copy of the scripts may be necessary. It also may be necessary to run the scripts with set –x added to the scripts.

The set –x entry displays additional output for each command being executed in the script.

■ Also for Alarm Notifier, two other debugging flags may be useful:

1. Add the following entry to the .alarmrc file: debug_gas=true

2. Run alarmnotifier from command line with the following command:

./alarmnotifier –tl details > Notifer_debug.OUT 2&>1

All of these debug steps write to the output file (in this example, it is Notifier_debug.OUT).

Review the Spectrum Performance Manager (SPM) Test Events. Look at timeout event. Review the additional information provided by running the action code above on the SPM Test model.

■ Review the sniffer trace.

■ For Cisco devices we use the ciscoRTTMonMIB to create tests and read test results.

■ For Cisco Jitter Tests:

1. Review the read (get response) of the rttMonJitterStatsNumOfRTT object.

2. This should equal the sample count of the test if all of the tests ran successfully. We look for a non-zero value.

3. If there is a non-zero value we should generate a “SPM Test Result” event instead of a “SPM Timeout” event.

4. If it is non-zero and we are still generating a timeout event, look for SNMP errors in the trace and review the values returned back from the device.

5. If we receive a NULL value from a device, that could cause a “SPM Timeout” event to occur.


Gather the following information:

■ stdout.log or catalina.out file – ALWAYS get this from the customer

– NCM is a OneClick application - always get this document when dealing with OneClick / Tomcat applications.

■ Sniffer trace if the problem is capture related. Sniffer traces are beneficial to see the actual traffic. NCM is a network-based tool.

■ If the issue is capture-related, turn debugging to maximum in $SPECROOT/NCM/config.xml file. This will create log files in the NCM/logs/debug directory.

■ Screenshots of each configuration area (local, device family, and NCM model)

– Verify that that things are configured correctly

■ Screenshots of any errors received.

■ Event report of the device containing the NCM events

– NCM has very device-specific events, this will always be needed when captures are failing.

■ SysOID and firmware version of the device (or a simulation using sapwalk) to identify if the device is supported.

Issues being reported under the Certification component will be for one of the following subcategories:

■ Certification Other ■ Host Management

■ Device Type Identifier ■ MibTools

■ Generic SNMP ■ Vendor Support

The following type of information will be necessary to help troubleshoot your issue under the Certification component:

■ A simulation or mibwalk for reproductions and enhancement requests.

■ Copies of vendor MIBs.

16

Many times MIBs will not compile properly due to the import statements. Keep in mind that the MIB you are compiling needs to be in the same directory as the parent MIBs it relies on. You can view the MIB to see what other MIBs are needed for it to compile properly.

To try to reproduce the customer issue:

■ Run the MIB through an external validation site to determine if there are problems with it: http://wwwsnmp.cs.utwente.nl/ietf/mibs/validate//

■ Get a screenshot or the text of the error that is shown in MIB Tools. This error will tell you the line that is having the problem and point you in the general direction of where to look.

Issues being reported under the Core / SpectroSERVER component will be for one of the following subcategories:

■ Control Panel ■ Policy Manager

■ Distributed SpectroSERVER (DSS) ■ Processed

■ Fault Isolation ■ SpectroSERVER Database

■ Fault Tolerance ■ SpectroSERVER Intelligence

■ Location Server ■ SpectroSERVER Other

■ Maintenance Mode ■ Watch Management

■ Model Security

The following type of information will be necessary to help troubleshoot your issue under the Core / SpectroSERVER component:

■ For most SpectroSERVER type problems, a copy of the Spectrum modeling database may be necessary.

■ Step-by-step reproduction instructions to understand what is being done by the user.

■ Events leading up to an incident may be requested. These can easily be gathered from the OneClick Events tab. Please include the Event Type column in the Events table before exporting the Events list to a CSV file.

http://wwwsnmp.cs.utwente.nl/ietf/mibs/validate/


MapUpdate output (used to determine if landscape handles and precedence are set correctly) obtained by navigating to the $SPECROOT/SS-Tools directory and running:

./MapUpdate –v > MapUpdate.OUT

1. Osfind output in 8.1 and nsutil output in 9.0 (used to determine if all necessary SPECTRUM components can see each other). Osfind is obtained by navigating to the $SPECROOT/bin directory and running:

. /osfind.bat > osfind.OUT

Nsutil is obtained by navigating to the $SPECROOT/bin/VBNS directory and running:

./nsutil.bat > nsutil.OUT

2. CLI output of “show models mth=0x10004” from each SpectroSERVER is obtained by navigating to the $SPECROOT/vnmsh directory and running:

./connect

After being connected, run the following command to determine if user models are consistent across databases:

./show models mth=0x10004

3. Run the getinfo.sh scripts on the MLS and the remote SpectroSERVER to check the configuration of each server.

Note: Please review legacy tech solution TS29737 for information on the script.

4. Put location server in debug on both SpectroSERVER computers to verify communication between the SpectroSERVERs.

To enable the debug:

– Add the following three lines to the .locrc file in the <SPECROOT>/LS directory on each SpectroSERVER

ctrace=true

strace=true

debug=true

– Kill the location server on both machines (the process name is “locserver”)

– Get the LOCSERV.OUT file from the <SPECROOT>/LS directory.

The most important output that needs to be collected is the STACK TRACE. Debugging the cause of the crash is nearly impossible without the stack trace.

The following lists the information that needs to be gathered. Please be sure to find out which operating system your customer is running SPECTRUM on and send them the relevant information for their OS.

18

For all SpectroSERVER crashes, independent of the operating system, please provide us with the following information:

1. An event report for all models in the SSdb for an hour leading up to the crash.

This can easily be obtained through the OneClick events tab:

– After focusing the events tab on the landscape, add the Event Type column if it is not already included in the table of events

– Then select all events leading up to the crash

– Then select Export to CSV.

2. What is the landscape handle of the SpectroSERVER database?

3. Is the SpectroSERVER in a distributed SpectroSERVER environment?

4. Did the crash happen around the time of an Online Backup?

5. Is the SpectroSERVER the only application on the box? Virus scans and system backups should be configured to bypass all Spectrum related directories and databases.

6. Are there any scripts outside of SPECTRUM that are running or any actions that users were performing at the time of the crash?

7. How many OneClick servers are connected to the SpectroSERVER?

8. What add on IP Service Modules do you have? For example, eVPN, VPN, QOS, Multicast, SDM, or NCM?

9. OS specific data gathering instructions?

Issues being reported under the Discovery and Modeling component will be for one of these subcategories:

■ AutoDiscovery ■ RFC2790 Process Monitor

■ Live Pipes ■ Secure Domain Manager

■ Modeling ■ SNMPv3

■ Modeling Gateway ■ Virtual Host Manager (VHM)

■ Modeling Other ■ Vlan

The following type of information will be necessary to help troubleshoot your issue under the Discovery and Modeling component.

■ A combination of simulations, mibwalks, MIBs and the SpectroSERVER database may be needed.


1. Create simulations of the affected devices (use SAPWALK):

■ To determine what device information is being returned to the discovery requests made by SPECTRUM

■ To be able to use the simulations to try and reproduce the issue.

2. Enable Discovery in debug:

a. Navigate to the VNM model in OneClick.

b. Select Discovery.

c. Set the Debug Autodiscovery to On.

d. Set the IP address to 255.255.255.255.

e. Navigate to the Start Console page for OneClick (http://<machinename>:port/)

f. Click Administration.

g. Click on Web Server Debug (Runtime).

h. Find the Autodiscovery selection and turn it on.

i. Scroll to the bottom of the screen, make sure MAX is selected and click Apply.

j. Have the customer reproduce the issue. Try to have them only discover the problematic devices. You can do this by specifying the IPaddresses in the IPAddress list in Discovery.

k. The previous debug writes to the ADiscDebug_<timestamp> file in the directory $SPECROOT/SS and the stdout.log (on Windows) or the catalina.out (on UNIX), which is located in the directory $SPECROOT/tomcat/logs. You will need both files.

3. Find out from the customer what interfaces that the customer is expecting to see connected. For example, port 1 on device A is physically connected to port 1 on device B.

This allows you to pinpoint the data on the device and in the debug output.

1. Add “-aptrace –ertrace” to the front of the command line in the SDM.idb file located in the <SPECROOT>/lib/SDPM, like this:

ARGV;$WORKPATH/SDManager<CSEXE> -aptrace -ertrace -c64 -d –secpack …

2. Then re-import the SDM.idb file by stopping and restarting the processd.

Note: This action will stop the SpectroSERVER.

3. On Windows OS, cycle the SPECTRUM processd service.

4. On UNIX OS, navigate as root to the directory $SPECROOT/lib/SDPM and type the command:

./processd.pl stop

5. When processd has stopped, type the command:

20

./processd.pl start

6. Reproduce the issue and get the SDM.OUT file located in the directory <SPECROOT>/SDM.

Issues being reported under the Licensing component will be for one of these sub-categories;

■ Install

■ Keys

■ Licensing Other

The following type of information will be necessary to help troubleshoot your issue under the Licensing component.

■ A copy of the license certificate with the extraction key will be necessary.

■ A zipped or tarred copy of the $SPECROOT/Install-Tools/LOGS directory will give Tech Support a complete picture of the installation details.

To gather exceptions or errors during an install hang:

1. On Windows, double-click the executable, and as soon as it launches, hold down the Control key. A debug window will open. When the install hangs, see if there are any exceptions or errors in the window.

2. On Solaris, set the LAX_DEBUG variable to 1 and export it:

set LAX_DEBUG=1

export LAX_DEBUG

3. Launch the installer again and look for exceptions or errors.


Issues being reported under the OneClick component will be for one of these sub-categories:

■ Customization ■ OneClick Other

■ Global Collections ■ Security

■ Locator / Search Manager ■ User Editor

■ Login / User Authentication

The following type of information will be necessary to help troubleshoot your issue under the OneClick component:

■ Copies of any customized ($SPECROOT/custom/*) files may be necessary.

■ Information about how user authentication setup is being done will be necessary, along with version information for EEM if being used.

Issues being reported under the Reporting component will be for one of these subcategories:

■ BOXi ■ Report Manager

■ Report Gateway ■ Reporting Other

The following type of information will be necessary to help troubleshoot your issue under the Reporting component.

Information needed (these are the default file names and locations):

1. From the directory <SPECROOT>/SSLOGGER:

a. The .ssloggerrc file

Used to verify models file name and sslogger.dat file name.

b. The models file

Used to verify which models are being reported on and how many.

c. The sslogger.dat file

Used to verify which attributes are being reported on and which groups modeltypes belong to.

d. Five of the raw SSLOGGER output files

Used to verify that values are being reported.

22

2. From the directory SPECROOT>/Webapps/Report_Gateway:

a. The SRG_Configure.conf file

Used to verify configuration parameters. Make sure the rrd location and models file are correctly specified.

b. The Template file

Used to verify that the attributes and functions have been defined so that SRG can plot the data for the modeltypes the customer is reporting on.

3. From the <SPECROOT>/Webapps/Report_Gateway/SRG directory:

a. The SRG.conf file

Used to verify:

– the location of the SSLOGGER log files

– the SSLOGGER log files reside on a remote machine

– the models and .dat file are correct

– the attributes listed match the attributes in the sslogger.dat file.


Information needed:

1. Tomcat log file

stdout.log for windows and Catalina.out for *nix.

Report Manager outputs all of its errors to this log file. This is VERY important information to get.

2. Screenshots of the SPECTRUM Status Screen under Admin Tools.

Verify that the server is actually being monitored.

3. Screenshots of any errors.

4. Output from command that verifies the last update time of the Report Manager database:

a. In a shell, navigate to the following directory: <SPECROOT>/mysql/bin

b. Type the following: ./mysql -uroot -proot reporting

c. When connected, type: ./select * from landscape\G

d. Have the customer send in the output.

5. Confirmation if the reporting database is corrupt:

a. Run the commands in a shell window on UNIX or in a bash shell if on Windows:

– Start->Run, and type: bash –login

– Navigate to the <SPECROOT>/mysql/bin directory.

b. For the commands below, the user name is root and the password is also root.

c. The database_name is: reporting.

d. You can specify a table name if you find that one is corrupt.

e. You do not need to cycle Tomcat if you repair the database.

The mysqlcheck is a MySQL client utility that must be run while the database server is up and running. Using mysqlcheck is the recommended first method of detecting and repairing database corruption.

The suggested method to utilize mysqlcheck:

1. Execute mysqlcheck without any options to determine if any table corruption exists:

$> mysqlcheck -u<username> -p<password> <database_name>

– If any errors are reported, proceed to step 2.

– Otherwise, no further processing is required.

24

For example:

From: spect01%/c/win32app/SPECTRUM/mysql/bin

> ./mysqlcheck -uroot -proot reporting

2. Execute mysqlcheck with both '--repair' and '--quick' options to attempt a quick repair:

$> mysqlcheck -u<username> -p<password> --repair --quick <database_name> [<table_name>]

– If any errors are reported proceed to step 3.

– Otherwise no further processing is required.

For example:

From spec01%/c/win32app/SPECTRUM/mysql/bin:

> ./mysqlcheck -uroot -proot --repair --quick reporting

3. Execute mysqlcheck with just the '--repair' option to perform a standard repair operation:

$>mysqlcheck -u<username> -p<password> --repair <database_name> [<table_name>]

– If any errors are reported proceed to step 4.

– Otherwise no further processing is required.

4. Execute mysqlcheck with the '--repair' and '--force' options:

$>mysqlcheck -u<username> -p<password> --repair --force <database_name> [<table_name>]


https://support.ca.com/irj/portal/prddtlshome?prdhmpgform=p&productID=7832

https://support.ca.com/irj/portal/anonymous/pam

Please ensure the following information is provided when contacting CA Support/Customer Solution Support for assistance. If additional information was provided during the First Call interview, please have that available.

End Customer Name

Business Impact

Environment

Full Description of Problem

Full Description of Knowledge Base Searches

Problem recreated

Documentation provided

Potential Solutions or Work around

https://support.ca.com/irj/portal/prddtlshome?prdhmpgform=p&productID=7832

https://support.ca.com/irj/portal/anonymous/pam

https://support.ca.com/irj/portal/anonymous

Documents

CA Spectrum Infrastructure Manager Troubleshooting · PDF fileCA Spectrum Troubleshooting Guide for CA Support Partners ... at CA Support and Customer Care for Businesses and Partners