TS-RNC-SW-038-I26

TS-RNC-SW-038 Nokia Siemens Networks CONFIDENTIAL APPROVED 26.0 Page 1

Technical Support Note TS-RNC-SW-038

RNC PROBLEM REPORT INSTRUCTIONS Radio Control lers I PA- RNC: RN5. 0 , RN6. 0

mc RNC: mc RNC2. 0 , mc RNC3.0

F le x i D i rec t RNC: I - Ad a4 . 0

RNC O MS ( I n teg ra te d ) : RN5.0 , RN6. 0

RNC O MS ( S t and a lo ne) : RNC O MS 5 .0 , RNC O MS 1 . 0

Approval date: 08-Feb-2013

This document contains following type of information

Informative X Preventive Corrective Additional categorization Urgent Security Release Upgrade SW Update Information is classified as Internal Public X Customer Specific


Table of Contents 1. Validity ................................................................................................................................... 4 2. Compatibility / Dependencies to other products ...................................................................... 4 3. Keywords ............................................................................................................................... 4 4. Summary ................................................................................................................................ 4 5. Detailed description ................................................................................................................ 4

5.1 Hardware failure ..................................................................................................................... 7 5.2 System failure or problem ....................................................................................................... 7 5.3 Documentation failure ............................................................................................................ 7 5.4 Improvement proposal ............................................................................................................ 7

6. IPA-RNC problems ................................................................................................................. 8 6.1 Log collection with rnclogcol.exe ............................................................................................ 8

6.1.1 Emergency log collection ................................................................. 8 6.1.2 Normal log collection ........................................................................ 8 6.1.3 Running rnclogcol.exe ...................................................................... 9

6.2 Customized log collection with rnclogcol.exe ........................................................................ 10 6.2.1 Running without GUI in batch mode ............................................... 10 6.2.2 Remote message monitoring ......................................................... 10 6.2.3 Selecting the logs to be collected ................................................... 10 6.2.4 Adding own commands .................................................................. 10

7. mcRNC problems ................................................................................................................. 12 7.1 Collecting Symptom report ................................................................................................... 12 7.2 Collecting a full alarm history ................................................................................................ 12 7.3 Collecting a syslogs .............................................................................................................. 13 7.4 Collecting coredumps ........................................................................................................... 14 7.5 Samples of USCP-monitoring ............................................................................................... 14

8. OMS problems ..................................................................................................................... 15 8.1 General case ........................................................................................................................ 15

9. Flexi Direct RNC problems ................................................................................................... 16 9.1 Basic SW information of Flexi Direct RNC ............................................................................ 16 9.2 Preparation for monitoring .................................................................................................... 16

9.2.1 Logging to the local storage ........................................................... 16 9.3 Monitoring ............................................................................................................................ 17

9.3.1 Monster monitoring ........................................................................ 17 9.3.2 TCPdump monitoring ..................................................................... 18

9.4 Collecting of system logs ...................................................................................................... 18 9.5 Radio Network configuration ................................................................................................ 19 9.6 WBTS snapshot collection .................................................................................................... 19

10. RNW Download/Upload Problems from NetAct .................................................................... 20 11. Note ..................................................................................................................................... 20 12. References ........................................................................................................................... 20

Contact:

Contact your local Nokia Siemens Networks support.


Summary of changes:

25.02.2008 1.1 A new template and OMS log instructions added. 09.04.2008 1.2 Rnclogcol macro improvement. Amount of MML sessions are limited. 30.05.2008 1.3 Rnclogcol macro improvement and chapter 7.3 updated. 25.06.2008 1.4 Rnclogcol macro improvement. RNC name and time stamp added to

log directory name. Chapter 9 and chapter 10 ddesk command added. 23.09.2008 2.0 Chapter 7.3 updated 29.01.2009 3.0 RN4.0 validity added 25.02.2009 4.0 Chapters 13.1 and 13.2 updated. RN21 info removed. Added chapter

Collect NPGE and NPGEP units related information 06.03.2009 5.0 Updated new macro versions 26.03.2009 6.0 Chapters 5.1, 5.3, 7.2, 8.12 and 13.1 updated 09.06.2009 7.0 Chapter 13 updated and zcollectlogs.sh added 02.11.2009 8.0 Chapters 9 and 10 updated, RN2.2 removed, RN5.0 added 01.12.2009 9.0 Chapter 8.12 title updated 20.01.2010 10 Chapter 13.1 updated 22.04.2010 11 Chapter 14 updated, a new rnclogcol.exe and default_config.ini added,

other *.ini files removed. 29.04.2010 12 New rnclogcol.exe. 07.05.2010 13 New rnclogcol.exe. 21.05.2010 14 New rnclogcol.exe. 27.05.2010 15 Missing default_config.ini and rnc.txt added to rnclogcol.zip. 10.08.2010 16 New rnclogcol.exe version 1.4.8.15. 04.10.2010 17 Rnclogcol.exe 1_4_8_18 and chapter 5.1 updated, dsp_blackbox.ini

file added. Chapters 4.1 4.5 and 15 added. 29.12.2010 18 RN6.0 valid checked and RN3.0 removed

Updated rnclogcol.zip. Added Chapter 16. Added instructions to Chapter 9. Added info of new ini-files to Chapter 5.1. Removed RRMU from Chapter 7.2. Removed Chapter 4.5

23.12.2011 19 Rnclogcol.exe macro has been optimized and re-written, manual commands removed from document.

16.01.2012 20 RNCLogcol.zip updated (rnclogcol.exe macro version 1_5_1_70). 10.02.2012 21 RNCLogcol.zip updated (rnclogcol.exe macro version 1_5_1_75). 04.04.2012 22 RNCLogcol.zip updated (rnclogcol.exe macro version 1_5_1_79).

Template updated. Validity -chapter added.

30.05.2012 23 RNCLogcol.zip updated (rnclogcol.exe macro version 1_5_1_83). 28.09.2012 24 I-HSPA Adapter technical support note 22 has been integrated to this

technical support note starting ADA Release 4. 18.1.2013 25 Modified RNC problems part of the document.

Clarified usage instructions for log collection with rnclogcol.exe. Changed I-HSPA/ADA name to Flexi Direct RNC. Corrected /var/logs/Logs/ to be /var/log/ on Preparation for monitoring, Monster monitoring and TCPdump monitoring chapters for Flexi Direct RNC.

21.1.2013 26 Added mcRNC part RNCLogcol.zip updated (rnclogcol.exe macro version 1_5_1_92).


Purpose

This document contains generic information about products. These can be instructions that explain problem situations in the field, instructions on how to prevent or how to recover from problem situations, announcements about changes or preliminary information as requirements for new features or releases.

1. VALIDITY

Product RU20 RU30 RU40 IPA-RNC X RN5.0 X RN6.0 RN7.0 RNC OMS for IPA-RNC (Integrated)

X RN5.0 X RN6.0 Not Supported

RNC OMS (Standalone)

X RNC OMS 5.0 X RNC OMS 1.0 RNC OMS 2.0

mcRNC X mcRNC2.0 X mcRNC3.0 Flexi Direct RNC I-Ada3.0 X I-Ada4.0 I-Ada5.0 Flexi Direct OMS I-OMS3.0 I-OMS4.0 I-OMS5.0

Note! For RU40 only mcRNC3.0 part validated and updated!

2. COMPATIBILITY / DEPENDENCIES TO OTHER PRODUCTS

No affect.

3. KEYWORDS

Problem report, Resolve, RNC outage, System restart, Unit restart.

4. SUMMARY

This document contains basic instructions for making a problem report in such a way that all the necessary information is included in it. Filling in the problem report carefully and attaching required information to it makes the problem investigation process faster.

5. DETAILED DESCRIPTION

This document contains instructions for collecting basic data from the network elements RNC and OMS for fault analyzing purposes. This data shall always be collected if some abnormal situation has occurred in the system, and Nokia Siemens Networks (NSN) will be informed of it. An abnormal situation may be, for example, a decreased traffic handling capacity, or a spontaneous restart of a computer unit. In the worst case the abnormal situation may lead to a system outage.

The data to be collected is essential for the further analysis of the problem for trying to find out the root cause of the fault. The data shall be collected as soon as possible after the abnormal situation has taken place. This is important because the information stored about the problem (e.g. the black box of a certain unit, or alarm history) may get overwritten in the process of time.


The log collection macro (rnclogcol.exe) and ini file are available via NOLS. The macro makes possible to collect automatically the required logs for the following situations: NE spontaneous restart, unit spontaneous restart, RNC configuration, call success problems and OAM (BTS) problems. See chapter RNC problems for more details.

Relate to OMS cases see chapter OMS problems for more details.

In addition to collecting the data, fill in the problem report by taking into account the following issues:

Title of the problem report should be a very short description of the fault situation

Description of the problem in the problem report should be an exact and a clear description of the fault situation. Note that only one fault shall be reported in one problem report and at least the following information should be provided in addition to the actual problem description:

Situation in the beginning, e.g. the first symptoms of the failure

Operations made, which possibly caused the failure

Situation after the failure

Recovery actions made

Name and version of the possibly installed new software modules

Make sure that necessary attachments are included to the problem report to avoid unnecessary information requests

In a multivendor environment, collect detailed information of other products to the Description field of the problem report

Write down the SW Release used in the NE.

Write down the SW build of the SW Release and the implemented CD level in RNC. This information can be obtained with the MML command ZWQO:CR;

For filling in the severity of the problem, see the definition of the different severities in Table 1, NSNs Problem Classification


NSN Problem Class

Definition of problem report severity as defined in TL 9000 Quality Management System, Measurement Handbook

Examples

Major Only total or major outages that are not avoidable with a workaround solution.

Problems severely affect service, capacity/traffic, billing, and maintenance capabilities and require immediate corrective action, regardless of time of day or day of the week as viewed by a customer upon discussion with the collaborator.

System restart, all links down Simultaneous restarts of active computer

units More than 50% traffic handling capacity

out of use Subscriber related RNC functionality is

not working RNC cannot be accessed or monitored

from NetAct Medium The fault affects traffic randomly or problem leads only to degradation of network performance or the fault makes it difficult for the customer to operate the RNC.

Problems cause conditions that seriously affect system operation, maintenance and administration, etc. and require immediate attention as viewed by the customer upon discussion with the collaborator. The urgency is less than in critical situations because of a lesser immediate or impending effect on system performance, customers, and the customers operation and revenue

Single restart of computer units Problems with back-up Configuration changes (RNW, HW and

SW) are not working Problems seriously affecting end user

service, but avoidable with workaround solution

Capacity/quality related functionality is not working

Performance measurement or alarm management is not working

Activation of a new feature fails A single performance measurement is not

working completely Configuration changes (RNW, HW and

SW) are not working completely Subscriber related functions are not

working completely Alarm management of objects (BTS,

functional units) is not working completely Capacity/quality related functionality is not

working completely Major errors in documentation, for

example, an alarm or description is missing from documentation

Vital documents are missing from the documentation library


NSN Problem Class

Definition of problem report severity as defined in TL 9000 Quality Management System, Measurement Handbook

Examples

Minor Minor fault not affecting operation or service quality.

Minor fault not affecting operation or service quality Other problems that a customer does not view as major or medium are considered minor. Minor problems do not significantly impair the functioning of the system and do not significantly affect service to customers. These problems are tolerable during system use. Engineering complaints are classified as minor unless otherwise negotiated between the customer and the supplier.

Failures not seriously affecting traffic Errors in MML syntax Minor errors in MML/Statistic output Minor errors in documentation

Table 1. NSNs Problem Classification

5.1 Hardware failure

When you suspect that a failure or problem is caused by the hardware of the switch, but you cannot locate the faulty unit, you can report the fault using the problem reporting practice. If you find a faulty unit, fill in the hardware failure report, attach it to the faulty unit, and send them to your local technical support. Set the case type of the problem report to value Hardware.

5.2 System failure or problem

The most common use of the problem report is reporting a system defect or problems related to the software or data configuration of the switch. Set the case type of the failure report to value Software.

5.3 Documentation failure

If you notice deficiencies in the documentation of the network element, raise a problem report. Identify where in documentation the failure occurs and which documentation set or library the document belongs to, and, of course, the actual failure. Set the case type of the failure report to value Documentation.

5.4 Improvement proposal

When there is no actual fault but you want to suggest some improvement to the way the network element functions, you can do this using problem reporting. New feature proposals should be directed directly to Product Marketing. Set the case type of the problem report to value General.


6. IPA-RNC PROBLEMS

6.1 Log collection with rnclogcol.exe

In the rnclogcol.exe user selects one of four pre-defined sets of logs to be collected. Rnclogcol.exe GUI shows the following options:

(1) Emergency log collection

- This is needed for Emergency cases, if RNC needs to be restarted soon.

(2) Normal log collection that includes message monitoring - This is needed for all Resolve cases.

(3) Normal log collection without message monitoring - This is needed only when requested.

(4) Normal log collection and DSP black box collection, no message monitoring - This is needed only when requested.

Logging can be run with the GUI interactively or non-interactively without the GUI. For running of log collection even without GUI, see chapter 6.2 Customized log collection with rnclogcol.exe. In RU20 before MP5 logging macro can be even slower than previous version because it collects so much more data (30%). In RU20 after MP5 and in RU30 even more this version is considerably faster than its predecessor.

6.1.1 Emergency log collection

Fast emergency log collection (GUI option 1) should only be used if the RNC needs to be restarted to recover from an emergency. In Emergency log collection mode, the log collection is divided into two phases in order to allow collecting volatile information before system restart, and collecting non-volatile information after system restart. There is a prompt to allow for RNC restart between the phases:

Press Any Key when log collection can continue

In other cases where RNC restart is not imminent, normal log collection (GUI option 2 or 3) is recommended.

6.1.2 Normal log collection

Most of the RNC configuration data and logs for the call cases can be collected with the rnclogcol macro. Option 2: Normal log collection, with message monitoring and HPL logs included. Suitable for most Resolve cases. Option 3: Normal log collection, without message monitoring or HPL logs. Suitable when someone just needs a snapshot of RNC configuration. Option 4: Normal log and DSP black box collection, without message monitoring and HPL logs. Required for DSP supervision failure (alarm 1239) related troubleshooting.


6.1.3 Running rnclogcol.exe

Download the rnclogcol macro from NOLS and extract rnclogcol.zip file into the PC folder named e.g. C:\Temp\rnclog\. The rnclogcol.zip includes an one default_config.ini file for different purposes The same configuration ini file is suitable for all RU20 and RU30 releases.

The tool can be run either from the command line, or by double-clicking it. After the tool starts up, there is a small GUI. Some log collection tasks have already started at this point. The username given to the tool must have full access to RNC commands.

Command line usage examples:

rnclogcol.exe a -n - Macro will ask for username and password before continuing to log collection GUI. rnclogcol.exe a -n - u - Macro will ask for password before continuing to log collection GUI. rnclogcol.exe i [-u ] [-p ] - Macro will directly go to log collection GUI.

Example ip list file format. One line per RNC

10.11.12.13 10.12.13.14 RNCNAME 10.12.13.15,RNCNAME

Example execution printout:

D:\healtcheck\linux\RNCLogcol>rnclogcol.exe -a 10.16.30.130 -n yellow -u SYSTEM -p SYSTEM -t 0 Version: 1_5_1_69 RNC config The Benevolent Acquirer of All Logs D:\healtcheck\linux\RNCLogcol\rnclogcol.pl -v shows usage instructions file: default_config.ini basedir for 10.16.30.130 is yellow_logs Mon 2011-12-19 12:33:31 Version: 1_5_1_69 Mon 2011-12-19 12:33:32 Using high thread count 40 for yellow Mon 2011-12-19 12:33:34 10.16.30.130 SW level is RN60 RNC SW Technical Note 38 log collector- Please select the logs to be collected (1) Emergency log collection - This is needed for Emergency cases, if RNC needs to be restarted soon (2) Normal log collection that includes message monitoring


- This is needed for all Resolve cases (3) Normal log collection without message monitoring - This is needed only when requested (4) Normal log collection and DSP black box collection, no message monitoring - This is needed only when requested

6.2 Customized log collection with rnclogcol.exe

Command line example for customized ini files:

rnclogcol.exe i [-u ] [-p ] [-c ] - Macro uses the ip list file for RNC address and name, and custom config file.

6.2.1 Running without GUI in batch mode

For those who would like to run the tool as batch job, that is also possible.

Edit default_config.ini file (or make copy of default_config.ini and use the c option to give your new configuration file that is edited like explained below). show_gui = 1 or show_gui = 0 This way the GUI will not be visible, and all functions are run without the need for user.

6.2.2 Remote message monitoring

Remote monitoring can be started in either WO-EX or SP-EX on OMU. Parameter omu_state define which OMU is used. It is good idea to use SP-EX OMU, in order to have WO-EX OMU service terminals is available for other use.

6.2.3 Selecting the logs to be collected

Define function start order priority: the lower the value, the higher the priority. However, value 0 means that the function is not in use.

6.2.4 Adding own commands

Own commands can be created in [own_commands] section. For example unit = MML is for specific MML commands. The commands are created into the text file. Path and a name of text file are defined as follows: file = C:\Users\commands.txt. Own commands can be activated by get_own_commands parameter set to 1. Unit = ICSU means commands for ICSU functional units. The commands are again created into the text file. Path and a name of text file are defined as follows: file = C:\Users\commands_2.txt. Also other RNC functional units can be use on own command list.


Example own commands set up:

[own_commands] unit = MML file = C:\Temp\rnclog\commands.txt

Example content of commands.txt file

ZWQV:OMU,0; ZWQV:OMU,1;


7. MCRNC PROBLEMS

7.1 Collecting Symptom report

Valid in mcRNC2.0 and mcRNC3.0 releases.

Most of the Multicontroller RNC configuration data and logs essential for investigation of the problem can be collected with the standard symptom pluging report group-RNC. This data shall always be collected if Problem Report will be submitted to Nokia Siemens Networks. The data shall be collected as soon as possible after the abnormal situation has taken place and before any recovery action are performed, such as RNC and unit restarts, replacing HW, etc. This is important because the information stored about the problem (e.g. the black box of a certain unit) may get overwritten in the process of time or lost due to recovery actions. To save or collect the standard symptom report group-RNC, enter the following command: # save symptom-report name include group-RNC

Example:

root@CFPU-0 [RNC-311] > save symptom-report name symptoms-suzuki-RNC include group-RNC Timeout set to 30 minutes Collecting Symptom Report: ReportName Subreport Store FileName ---------- --------- ----- -------- symptoms-suzuki-RNC rncss7config CFPU-0 symptoms-suzuki-RNC_rncss7config_201208231439.tar.gz symptoms-suzuki-RNC rncrnw CFPU-0 symptoms-suzuki-RNC_rncrnw_201208231439.tar.gz symptoms-suzuki-RNC rncalarm CFPU-0 symptoms-suzuki-RNC_rncalarm_201208231439.tar.gz symptoms-suzuki-RNC rnchas CFPU-0 symptoms-suzuki-RNC_rnchas_201208231439.tar.gz symptoms-suzuki-RNC rncinfo CFPU-0 symptoms-suzuki-RNC_rncinfo_201208231439.tar.gz symptoms-suzuki-RNC rncuserplane CFPU-0 symptoms-suzuki-RNC_rncuserplane_201208231439.tar.gz symptoms-suzuki-RNC rncipconfig CFPU-0 symptoms-suzuki-RNC_rncipconfig_201208231439.tar.gz symptoms-suzuki-RNC rnchw CFPU-0 symptoms-suzuki-RNC_rnchw_201208231439.tar.gz ======================================================================== symptoms-suzuki-RNC_0.tar is available at /var/log/stdsymp ======================================================================== Constituent files of generated symptom report symptoms-suzuki-RNC: symptoms-suzuki-RNC_0.tar

The standard symptom report group-RNC is expected to be completed in around 15 minutes depending on Multicontroller RNC configuration. Recovery actions can be started, if needed, as soon as symptom report generation is completed. Copy of the symptom report can be transferred to local machine later on at the convenient time.

7.2 Collecting a full alarm history

Valid before mcRNC2.0 2.0.

After mcRNC2.0 2.0 full alarm history is collected as part of Symptom Report, before that it needs to be collected manually.

Full alarm history files(alarms, alarms1) can be found from /srv/Log/log/fsaudit - folder.


7.3 Collecting a syslogs

Valid before mcRNC3.0.

Syslog files can be found from /srv/Log/log - folder

1. Open the ssh connection.

2. Copy active syslog E.g.: # cp syslog syslog0502_1517

3. Transfer all syslogs (above copied syslog + syslog.1 syslog.9)

[root@CFPU-0(MCRNC-314) /root] # cd /srv/Log/log/ [root@CFPU-0(MCRNC-314) /srv/Log/log] # ll total 3552008 -rw-r--r-- 1 root root 26780 Feb 5 13:47 HPIMonitorAlarmdata.txt lrwxrwxrwx 1 root root 27 May 24 2012 alarms -> /srv/Log/log/fsaudit/alarms lrwxrwxrwx 1 root root 29 May 24 2012 auth.log -> /srv/Log/log/fsaudit/auth.log -rw-r--r-- 1 root root 858462 Feb 5 15:17 debug -rw-r--r-- 1 root root 209715232 Feb 5 15:03 debug.1 -rw-r--r-- 1 root root 19380801 Feb 5 11:12 debug.2.gz -rw-r--r-- 1 root root 209715330 Jan 25 13:05 debug.3 -rw-r--r-- 1 root root 18079165 Feb 4 14:10 debug.3.gz -rw-r--r-- 1 root root 209715224 Jan 24 17:11 debug.4 -rw-r--r-- 1 root root 20983463 Feb 4 00:21 debug.4.gz -rw-r--r-- 1 root root 209715256 Jan 24 05:55 debug.5 -rw-r--r-- 1 root root 19718943 Feb 1 11:02 debug.5.gz -rw-r--r-- 1 root root 209715227 Jan 22 07:59 debug.6 -rw-r--r-- 1 root root 21295673 Jan 31 03:49 debug.6.gz -rw-r--r-- 1 root root 209715202 Jan 19 20:05 debug.7 -rw-r--r-- 1 root root 19726098 Jan 29 13:25 debug.7.gz -rw-r--r-- 1 root root 209715301 Jan 18 03:15 debug.8 -rw-r--r-- 1 root root 20487836 Jan 28 12:12 debug.8.gz -rw-r--r-- 1 root root 209715248 Jan 17 10:11 debug.9 -rw-r--r-- 1 root root 22566760 Jan 27 19:48 debug.9.gz drwxr-x---+ 2 root _nokfssysseclog 4096 Jan 16 08:08 fsaudit -rw-r--r-- 1 root root 3538736 Feb 5 14:17 fsdistribute.log -rw-r--r-- 1 root root 5819318 Nov 26 11:52 fsdistribute.log.1 -rw-r--r-- 1 root root 261577 Oct 29 15:16 fsdistribute.log.2.gz -rw-r--r-- 1 root root 163019 Feb 4 15:08 ilselog -rw-r--r-- 1 root root 45499560 Feb 5 15:17 syslog -rw-r--r-- 1 root root 209715230 Feb 5 10:30 syslog.1 -rw-r--r-- 1 root root 21506046 Jan 28 12:03 syslog.2.gz -rw-r--r-- 1 root root 209715261 Dec 17 09:47 syslog.3 -rw-r--r-- 1 root root 23223437 Jan 25 12:21 syslog.3.gz -rw-r--r-- 1 root root 209715354 Nov 30 22:23 syslog.4 -rw------- 1 root root 14876672 Jan 28 12:04 syslog.4.gz -rw-r--r-- 1 root root 209715201 Nov 14 14:10 syslog.5 -rw-r--r-- 1 root root 209715315 Nov 12 12:10 syslog.6 -rw-r--r-- 1 root root 209715270 Nov 7 15:23 syslog.7 -rw-r--r-- 1 root root 209715299 Nov 5 10:13 syslog.8 -rw-r--r-- 1 root root 209715240 Nov 2 01:51 syslog.9 [root@CFPU-0(MCRNC-314) /srv/Log/log] # cp syslog syslog0502_1517 [root@CFPU-0(MCRNC-314) /srv/Log/log] # gzip syslog0502_1517 [root@CFPU-0(MCRNC-314) /srv/Log/log]


# ll total 3556944 -rw-r--r-- 1 root root 26780 Feb 5 13:47 HPIMonitorAlarmdata.txt lrwxrwxrwx 1 root root 27 May 24 2012 alarms -> /srv/Log/log/fsaudit/alarms lrwxrwxrwx 1 root root 29 May 24 2012 auth.log -> /srv/Log/log/fsaudit/auth.log -rw-r--r-- 1 root root 914361 Feb 5 15:18 debug -rw-r--r-- 1 root root 209715232 Feb 5 15:03 debug.1 -rw-r--r-- 1 root root 19380801 Feb 5 11:12 debug.2.gz -rw-r--r-- 1 root root 209715330 Jan 25 13:05 debug.3 -rw-r--r-- 1 root root 18079165 Feb 4 14:10 debug.3.gz -rw-r--r-- 1 root root 209715224 Jan 24 17:11 debug.4 -rw-r--r-- 1 root root 20983463 Feb 4 00:21 debug.4.gz -rw-r--r-- 1 root root 209715256 Jan 24 05:55 debug.5 -rw-r--r-- 1 root root 19718943 Feb 1 11:02 debug.5.gz -rw-r--r-- 1 root root 209715227 Jan 22 07:59 debug.6 -rw-r--r-- 1 root root 21295673 Jan 31 03:49 debug.6.gz -rw-r--r-- 1 root root 209715202 Jan 19 20:05 debug.7 -rw-r--r-- 1 root root 19726098 Jan 29 13:25 debug.7.gz -rw-r--r-- 1 root root 209715301 Jan 18 03:15 debug.8 -rw-r--r-- 1 root root 20487836 Jan 28 12:12 debug.8.gz -rw-r--r-- 1 root root 209715248 Jan 17 10:11 debug.9 -rw-r--r-- 1 root root 22566760 Jan 27 19:48 debug.9.gz drwxr-x---+ 2 root _nokfssysseclog 4096 Jan 16 08:08 fsaudit -rw-r--r-- 1 root root 3538736 Feb 5 14:17 fsdistribute.log -rw-r--r-- 1 root root 5819318 Nov 26 11:52 fsdistribute.log.1 -rw-r--r-- 1 root root 261577 Oct 29 15:16 fsdistribute.log.2.gz -rw-r--r-- 1 root root 163019 Feb 4 15:08 ilselog -rw-r--r-- 1 root root 45500815 Feb 5 15:18 syslog -rw-r--r-- 1 root root 209715230 Feb 5 10:30 syslog.1 -rw-r--r-- 1 root root 21506046 Jan 28 12:03 syslog.2.gz -rw-r--r-- 1 root root 209715261 Dec 17 09:47 syslog.3 -rw-r--r-- 1 root root 23223437 Jan 25 12:21 syslog.3.gz -rw-r--r-- 1 root root 209715354 Nov 30 22:23 syslog.4 -rw------- 1 root root 14876672 Jan 28 12:04 syslog.4.gz -rw-r--r-- 1 root root 209715201 Nov 14 14:10 syslog.5 -rw-r--r-- 1 root root 209715315 Nov 12 12:10 syslog.6 -rw-r--r-- 1 root root 209715270 Nov 7 15:23 syslog.7 -rw-r--r-- 1 root root 209715299 Nov 5 10:13 syslog.8 -rw-r--r-- 1 root root 209715240 Nov 2 01:51 syslog.9 -rw-r--r-- 1 root root 4984074 Feb 5 15:17 syslog0502_1517.gz

7.4 Collecting coredumps


Core dumps can be found from /srv/log/crash -folder.

7.5 Samples of USCP-monitoring


Collect USCP-monitoring for 20-30s from any of USPUs for getting samples from call scenarios. monster -f 4fd,509,0a49,9f6,997 -b USPU0.bin -c "SR:NOT(ATT=1,2)" E.g. USCP monitoring: [root@CFPU-0(MCRNC-314) /srv/Log/log] # ssh uspu-0 [root@USPU-0(MCRNC-314) /root] # monster -f 4fd,509,0a49,9f6,997 -b USPU0.bin -c "SR:NOT(ATT=1,2)" ---------------------------------------------------- Please wait, outputting the remaining frames in buffer... CTRL-C to cancel


8. OMS PROBLEMS

8.1 General case

If the problem concerns OMS, collect the following logs and transfer logs to local PC.

Note Never change any files or directories rights which are used by the system. For example /root, /var/log/ and /var/log/oms directories and all files in them.

Always copy the file first to other directory e.g. /home/Nemuadmin and then modify the file rights in that directory if needed for transfer. Activate OMS traces with commands: # ztracecli --on \\Turn on # ztracecli --p \\Check status # ztracecli --off \\Turn off Re-procedure a case and run log collect script zcollectlogs as root user (su -). Output log will create into a directory where the script is run. zcollectlogs o .tgz --full Example # zcollectlogs o rnclogs.tgz full Windows environment: Log in to OMS as Nemuadmin user using puTTY tool. Change to the root user by entering su command. Copy files first into the /home/Nemuadmin directory and then transfer files to local PC by WinSCP tool as Nemuadmin user. cp //.tgz /home/Nemuadmin/ E.g. cp /root/rnclogs.tgz /home/Nemuadmin Linux environment: Log in to OMS as Nemuadmin user. Transfer files to local PC with the following command. scp //.tgz @:/ E.g. scp /root/rnclogs.tgz @:/


9. FLEXI DIRECT RNC PROBLEMS

9.1 Basic SW information of Flexi Direct RNC

1. Current active SW level will be checked first. fsclish root@CLA-0 [ftlb] > show sw-manage app-sw-mgmt builds 2. Print out installed SW builds. root@CLA-0 [ftlb-1] > show sw-manage list root@CLA-0 [ftlb-1] > show sw-manage current all

3. Verify volume group on disk. # vgs 4. Print out active alarms. # zaho -a

9.2 Preparation for monitoring

Flexi Direct RNC monitoring will be made using monitoring tool and monitoring should be made to an external device by mounting it over the network via SSHFS. This requires that the external device is reachable over the network by the Flexi Direct RNC and that the Flexi Direct RNC is able to establish a regular SSH session to the device.

1. Connect Flexi Direct RNC Octeon to external device on the network

# sshfs -o idmap=user @:/ /var/log If the same networked storage will be used to collect logs from multiple Flexi Direct RNCs, it is advised to use a folder name that reflects the specific Flexi Direct RNC that is being monitored. This folder needs to be created first on the storage device before it can be mounted on the Flexi Direct RNC. Eg: sshfs -o idmap=user [email protected]:/home/sshuser/IADA-10/Logs /var/log

9.2.1 Logging to the local storage

In the case that it is not possible to mount a network drive directly, it is possible to collect a small amount of logs on the local storage. 1. Check the local free space with the commands: # df -h /var/log If the free space is less than 100MB, try to remove any unnecessary temporary files or old logs before continuing. 2. Limit the maximum size of the file that can be created from the logging process: # ulimit -f


Eg: ulimit -f 10240 This limits all files created by commands running in the current login session to 50MB. The limit applies until the login session is ended. If less free space is available, reduce the limit appropriately.

9.3 Monitoring

Monitoring online will be done with selected monitoring tool.

When mounting is done the monitoring can be started. Monitoring can be done with monster or TCPdump.

9.3.1 Monster monitoring

Message monitoring will be made to certain families RRC (04FD), NBA (04FE), BRM (04FB), HA3 (507), IUR (508), IUV (509), RM2 (4B4), NRM (04B7) and L3C (0A49).

1. Monitoring will be started in Flexi Direct RNC with command: # monster -S 50 -f 4FD,4FE,4FB,507,508,509,4B4,4B7,A49 -c

"SR:NOT(NUM=1,2,0D5BA,0C5DF,0D7A6,0D7A8,08037,08038,09904,09835,09836,0D9DA,0CEC9,0CECA)" -o /var/log/monster.bin &

2. Check the monster process id:

# ps -ef|grep monster

3. Stop the monitoring with command: # kill

4. (optional) Copy locally-stored logs If the logs have been collected on the local storage, copy them to external storage. The log file should be given a name that indicates which Flexi Direct RNC it was created on:

# scp /var/log/monster.bin @:/_monster.bin

Eg:

# scp /var/log/monster.bin [email protected]/home/testuser/IADA-10_monster.bin

5. Finish using the logs directory. If logs were stored locally, remove the unused files from the local storage:

# rm -f /var/log/monster.bin

Or,

If external storage was mounted to /var/log, unmount it #fusermount -u /var/log


9.3.2 TCPdump monitoring

In next examples Wireshark tool is used.

1. Locally connect the PC where the monitoring tool is active to ETH2 into FTLB. ETH2 configuration instructions can be found from Integrating I-HSPA document, DN0931138.

ADA Wireshark [Iub & Iu] - FTLB Marwell prompt [local connection telnet 192.168.255.129]

# ml2monitor -E set=EIF2 -I set=EIF2 -e add=OCT0 -i add=OCT0 -e add=OCT3 -i add=OCT3 2. Remote online monitoring

Start TCP dump in Octeon with command: # tcpdump -i eth3 -s 65535 -w /var/log/log_name.cap Monitoring can be stopped with command:

# Ctrl + C

3. (optional) Copy locally-stored logs If the logs have been collected on the local storage, copy them to external storage. The log file should be given a name that indicates which Flexi Direct RNC it was created on:

# scp /var/log/log_name.cap @:/_ log_name.cap

Eg:

# scp /var/log/log_name.cap [email protected]/home/testuser/IADA-10_ log_name.cap

4. Finish using the logs directory. If logs were stored locally, remove the unused files from the local storage:

# rm -f /var/log/log_name.cap

Or,

If external storage was mounted to /var/log, unmount it #fusermount -u /var/log

9.4 Collecting of system logs

4. Open the ssh connection. 5. Use SCP to copy syslog and crash log

/var/log/local/syslog /var/log/local/debug /srv/Log/crash/ /var/log/swm.log /var/log/fsconfigure.log /var/log/ilsw_upgrade.log /var/log/ss_ ilswman.log


/var/log/fsaudit/alarms

9.5 Radio Network configuration

The Radio Network configuration will be checked from RNW plan which will be uploaded via OMS. 1. Log into OMS 2. Select Configuration Management / CM Plan Management / Plan Management 3. Select the NE 4. Select Plan Type: RNW 5. Upload Plan 6. After success plan upload the file can be saved

9.6 WBTS snapshot collection

BTS Site Element Manager collects the WBTS data. Snapshot is needed when WBTS is suspected to cause the problem.

1. Log into BTS Site Element Manager and collect snapshot by File/Save/Snapshot

2. Collect FTM module data by logging into FTM with internet browser using BTS TRS IP address, both local and remote addresses can be used.

3. Collect FTM logs by selecting the Download FTM Log file and choose Enhanced Log

Collection.


10. RNW DOWNLOAD/UPLOAD PROBLEMS FROM NETACT

Following files and monitoring are needed when phasing Net-Act RNW download/upload Plan problems:

1. Repeat the problem NetAct RNW Download (pre-activation) /Activation scenario or provide some modifications to RNW (e.g. RNC, WBTS, COCO/IPNB, WCELL) with Object Browser/Parameter Editor.

2. Provided xml-file RNWPLAN placed at the following path can be used to check the RNW Configuration set by operator. /RUNNING/PLATMP/RNWPLAND.XML

3. Run rnclogcol.exe.

11. NOTE

-

12. REFERENCES

-


Disclaimer

The information in this document is subject to change without notice and describes only the product defined in the introduction of this documentation. This documentation is intended for the use of Nokia Siemens Networks customers only for the purposes of the agreement under which the document is submitted, and no part of it may be used, reproduced, modified or transmitted in any form or means without the prior written permission of Nokia Siemens Networks. The documentation has been prepared to be used by professional and properly trained personnel, and the customer assumes full responsibility when using it. Nokia Siemens Networks welcomes customer comments as part of the process of continuous development and improvement of the documentation.

The information or statements given in this documentation concerning the suitability, capacity, or performance of the mentioned hardware or software products are given as is and all liability arising in connection with such hardware or software products shall be defined conclusively and finally in a separate agreement between Nokia Siemens Networks and the customer. However, Nokia Siemens Networks has made all reasonable efforts to ensure that the instructions contained in the document are adequate and free of material errors and omissions. Nokia Siemens Networks will, if deemed necessary by Nokia Siemens Networks, explain issues which may not be covered by the document.

Nokia Siemens Networks will correct errors in this documentation as soon as possible. IN NO EVENT WILL NOKIA SIEMENS NETWORKS BE LIABLE FOR ERRORS IN THIS DOCUMENTATION OR FOR ANY DAMAGES, INCLUDING BUT NOT LIMITED TO SPECIAL, DIRECT, INDIRECT, INCIDENTAL OR CONSEQUENTIAL OR ANY LOSSES, SUCH AS BUT NOT LIMITED TO LOSS OF PROFIT, REVENUE, BUSINESS INTERRUPTION, BUSINESS OPPORTUNITY OR DATA, THAT MAY ARISE FROM THE USE OF THIS DOCUMENT OR THE INFORMATION IN IT.

This documentation and the product it describes are considered protected by copyrights and other intellectual property rights according to the applicable laws.

The wave logo is a trademark of Nokia Siemens Networks Oy. Nokia is a registered trademark of Nokia Corporation. Siemens is a registered trademark of Siemens AG.

Other product names mentioned in this document may be trademarks of their respective owners, and they are mentioned for identification purposes only.

Copyright Nokia Siemens Networks. All rights reserved.

1. VALIDITY2. COMPATIBILITY / DEPENDENCIES TO OTHER PRODUCTS3. KEYWORDS4. SUMMARY5. DETAILED DESCRIPTION5.1 Hardware failure5.2 System failure or problem5.3 Documentation failure5.4 Improvement proposal

6. IPA-RNC PROBLEMS6.1 Log collection with rnclogcol.exe 6.1.1 Emergency log collection 6.1.2 Normal log collection 6.1.3 Running rnclogcol.exe

6.2 Customized log collection with rnclogcol.exe6.2.1 Running without GUI in batch mode6.2.2 Remote message monitoring6.2.3 Selecting the logs to be collected 6.2.4 Adding own commands

7. MCRNC PROBLEMS7.1 Collecting Symptom report7.2 Collecting a full alarm history7.3 Collecting a syslogs7.4 Collecting coredumps7.5 Samples of USCP-monitoring

8. OMS PROBLEMS8.1 General case

9. FLEXI DIRECT RNC PROBLEMS9.1 Basic SW information of Flexi Direct RNC9.2 Preparation for monitoring9.2.1 Logging to the local storage

9.3 Monitoring9.3.1 Monster monitoring9.3.2 TCPdump monitoring

9.4 Collecting of system logs9.5 Radio Network configuration9.6 WBTS snapshot collection

10. RNW DOWNLOAD/UPLOAD PROBLEMS FROM NETACT11. NOTE12. REFERENCES

Documents

TS-RNC-SW-038-I26