NOKIA RNC Start-up Troubleshooting

1 © Nokia Siemens Networks RN33035EN40GLA0

RNC Start-up TroubleshootingRN3303-40A


Objectives

After completing this module the participant shall be able to:

• Understand the principles of a system start-up

• Understand the principles of a unit start-up in DMX and Chorus

• Investigate failures in the start-up

• Investigate spontaneous restart cases


Start-up Troubleshooting

When problems occur, they usually occur during the start-up phase of the network element.

Therefore, it is very important to understand this process and be able to make the required adjustments for the element to start-up correctly.

System or unit restart may be caused by:

• User (operator) command,

• Spontaneous event (for example power failure), or

• Fault management


Start-up Phases

DMX MessagingHMS Messaging

Initialize HW port for FU

Initialize ATM Interface for FU

Create ATM virtual connections

Load code Restart any FU

RESTART


Order of Unit Start-up

CACU

SFU

MXU

MXU

OMU

MXU

NIP1Iu

Iu / ATM Core NISx

Ethernet

2. Start active SYM MXU and SFU.· To allow ATM connection setup for OMU and

other SYM MXU tributary units· SFU is started only at flash boot level (Note:

includes only switch port initialization, but no message connectivity to the SFU unit.

1. Start the active SYM computer unit (OMU).· To start the system Maintenance· Only OMU has a direct

connection to disks

Iu / ATM Core NPS1x

IP Core NPGEx

RSMU

SFU

MXU

MXU

OMU

MXU

NIP1Iu

Iu / ATM Core NISxIu / ATM Core NISx

Ethernet

disks

3. Start the rest of the active MXUs, and SYM-MXU tributary units.

5. Start the interface units.· Starting in the last phase to

ensure that the network element is ready to respond immediately when network interfaces are opened.

4. Start the tributary units of the non SYM MXUs.

Iu / ATM Core NPS1xIu / ATM Core NPS1x

IP Core NPGExIP Core NPGEx

DMCU

A2SU

ICSU


System Start-up Principles

The starting order at high level is: 1. Active units up and running first (for user traffic)2. Redundant units next3. Rest of the units

The IPA2800 is a multiprocessor and multi-unit platform, which starts in a parallel and concurrent way:

• Parallel - units genuinely execute independently; high level control from centralized units• Concurrent - units perform internally concurrent operations during unit start-up

For units with multiple processors (A2SU, DMCU) • the master processor will be started first, • and then the rest of the processors will be started independently, and in parallel.

For those units with DSP resources (DMCU)• DSP-0 in each DMPG will be started first, • and then the rest of the DSPs under DMPG will be started in parallel


Parallel Start-up

1. Starting the active SYM unit (OMU)

2. Starting the MXU and SFU

3. Starting of the tributary units

4. Starting of the interface units


Unit Start-up

Informing centralized part for start-up

1. Initialising ATM hardware

2. Creating one or more non-guaranteed virtual connections

3. Opening virtual channels for communications

4. Loading the boot archive (Chorus) or load modules (DMX)

Starting the newly loaded boot archive (warm restart) or load modules and continue using it (this phase includes partially the same phases as described in 1-3 above)


DMX Start-up

BOLERO

Program blocksFrom disk

STQPRO starts program blocks needed to start the file loading

File loadingStart DMX loaded

from flash

EPISOD loads load modules from the external source

STQPRO starts program blocks loaded from Flash

Boot programsfrom FLASH

Start DMX loaded from the external source

TWarm restart?

Warm Restart?

F

STQPRO starts program blocks

Initial loading

RP0 -> RP1RP1 RP2

RP2 RP3

HMS ->DMX

Load boot programs from flash

HW reset


Phase 0Boot

• BOLERO starts the RAM tests

• BOLERO loads initial loading software from flash

• DMXRTE is started

• DMXRTE starts STQPRO

• STQPRO starts rest of initial loading software

• EPISOD connects to SYM unit and initial phase begins

SELF TEST RESULT = 00000000 PROCESSOR ID & REV = 000006D6

CPU ID = 63C0 CPIO FEATURES = D11D CPU RESET STATUS = 06D6

BOPCCP18.PAC 1.22-0 07/03/20

BOLEROGX.PAC 23.10-0 07/03/20 PLAENVC3.PAC 5.11-2

RESET TYPE = WARM, 00 RESET CAUSE = 35

NTC PCI Software version 0.2-0 created 24.10.1997

Copyright (c) 1993 1995 Nokia Telecommunications Oy Finland

LOADING MODULES FROM PROM

AIHLIB AIMPRB APPARA BBOLIB BIBLIO BLNAME BLSLIB BLTEXT BOSLIB BOXANA

BURNER CEPRXY CLUCIF DEBUTI DMXRTE DXPARA EKLIPS ENALIB EPISOD ETHLIB

ETOMAN FISLIB FIZSLM HAPERO HMEXTE HMILIB HMRESE INVOKE ISRLIB JUPINA

KERMIT LIBGEN LOGUTI NASEVA OSITUS PAGLIB PBHEXT PBMLIB PGEUTI PO2LIB

PODISK POFFIC POHEXT POMOXI POXLIB POZMAN PRSLIB PULINA RA2TST RAUDAT

RUBBER SEKLIB SHIFTY SQPAPL SQPCPL SQPDPL SQQEXT STQPRO SYHLIX TOELIB

TOHEXT TOMPRB UXDLIB UXMANA WYNONA

MODULES LOADED FROM PROM

DEBUG MODE = 00000000

STARTING DMX


Phase 99Boot and Normal

Phase 99 consists of two parts;

• Boot

• Normal loading

When the unit executes a total code loading from disk or the existing boot archive is corrupted,

• The unit will first start the code which was loaded from its PROM.

• This enables it to:– Load directly from the disk (OMU)

– Or to initialize and create its ATM connection


Phase 99Boot

Steps1. Process family starting

2. Obtaining loading info

3. Loading the Disk Boot Archive

4. Starting the DMX_RTE

START - PHASE 99 (BOOT) 15:51:04.53

STARTING FAMILIES BIB WYN HMR POZ NAS SYH UXM AIM TOM POF EKL FIZ PUL EPI

WAITING CONNECTION PERMISSION TO LOAD MANAGER... RUB

PROCESSING FAMILIES NAS

LOAD MANAGER INFORMATION RECEIVED

LOADING PACKET................ CB 8.8-0 0

READY - PHASE 99

LOADING MODULES FROM C000 DISK

LOADING LOAD LIST............. BOOLIS02.IMG

ZLI YYT ZAT ZAU YAN YEM XVA ZSK XPL XIP WUT WUP WYN XML WFA WSW VIM VTE UXD

URM UXM ULT ULA VID UBS UBE UAF TZI TUT TUK TSA UAM TOP TOH TLY TLX THE TOE

...........................................................................

LGR LAN KDD KDD IOL FSY FLL FLL FLL FLL FLL FLL DUC DOF DIP CPV BSP BLN ULT

LOADING DONE


Phase 99Normal

Unit Starting Program Block starts some special process families including

• Process families related to message transfer and recovery

• File System Library Manager FIZSLM which initializes the memory file system

START - PHASE 99 (NORMAL) 15:51:22.52

STARTING FAMILIES SYB WYN ALA HMR SYH NAS UXM AIM POZ TOM POF EKL HMA RUM

RUB PAX FIZ PUL RXS RXF

DEFINING UNIT TYPE

UBE OLH

PROCESSING FAMILIES NAS

READY - PHASE 99


Restart Phase 1

• Work files created by loading system initialized

• File loading• Two kinds of file loadings:

– Loading group loadings

– Loadings performed by application programs (mainly database managers)

START - PHASE 1 15:51:31.55

START - PHASE 1 SUBPHASE 1 15:51:31.55

STARTING FAMILIES PRT FUZ EDU RME TUK CEP OAZ IOM RDI FEM FED RD4 SOW MOP

MM2 RUX ST0 DUM HWO RM4 LSE

PROCESSING FAMILIES FUZ

READY - PHASE 1 SUBPHASE 1 15:51:32.09


INITIALIZING WORK FILES

FILE LOADING ACTIVATED

LOADING STARTED FROM 0000 DISK LOAD GROUP=3005

LOADING COMPLETED LOAD GROUP=3005





PROCESSING FAMILIES FUZ SYH TOM POF CPU RD4 RDI SOW FED HWO

STARTING FAMILIES WUP ODO NDV SAV REQ RIZ JAG VTE LPQ TZI RAA CNY FDM PWG

DYT IDM RUF OCL

PROCESSING FAMILIES TOM


READY - PHASE 1


Start-up Phase 2

• Unit can function as a file loading source for other units.

• Databases and other files started and loaded

• Process families started• Mainly database managers

START - PHASE 2 15:51:33.07


PROCESSING FAMILIES HMR NAS POF

STARTING FAMILIES DD1 SYA RXL EFO RIE RXU RIA RXE RXB TLX BUS EQB ARD



FILE LOADING ACTIVATED

LOADING STARTED FROM 0000 DISK APPLICATION=BUS

NO SOURCE UNIT AVAILABLE

FAILED SOURCE UNIT: 0000 ERROR CODE: 0000030A

NO SOURCE UNIT AVAILABLE

FAILED SOURCE UNIT: 544C ERROR CODE: 00000309

FAILED SOURCE UNIT: 5002 ERROR CODE: 0000030A

LOADING COMPLETED APPLICATION= EQB

LOADING STARTED FROM 0000 DISK APPLICATION=ARD






PROCESSING FAMILIES SYB PRT ALA HMR TOM EDU TUK HWO RUX ST0

STARTING FAMILIES DUD CPY DXH DPA ULA IKE ZAU PHL JAK XVA SEE NSM EQC LOS

PROCESSING FAMILIES HMR TOM


READY - PHASE 2


Start-up Phase 3

• Rest of process families started

• No more files loaded• Unit goes to WO-EX

(hopefully!)• POHEXT shows ”open”

START - PHASE 3 15:51:46.45


PROCESSING FAMILIES ALA NAS EKL DPA HMA WUP RD4 RDI FEM FED RXS RXB

STARTING FAMILIES JAP JAN HUT IPD JAS DIA DIA DIA DIA DIA DIA DIA DIA DIA

DIA DIA DIA DIA DIA DIA CBY RFG

PROCESSING FAMILIES ALA DPA





PROCESSING FAMILIES HMR UXM AIM TOM POF RUM ULA JAG

STARTING FAMILIES CPQ ERS PSE ERR AMN AVA HOK SW2 SW5 RBR JAF EXW DEE EHE

EHF FMP LMN ERD KAN PVM ARM XPL URM LOM SHO MMI SYP DBV

SUP THE VID HZC ATY DOC SVD EQS ENP DUK MTF CPS IFM SWU

COF PIG SCM CM6 LMP OLW IPQ SKO TSA ENN WSW SOH EEF ALV

PBL ZSK FLM SIK ROI DIQ FCM BJC DIS MXU JIG COK YEM FCP

RUU FC1 LMA NTX EID KAR ATX IPB JOR UAM DGT GIV LDA MNM

SLY OFM H2O ULT

PROCESSING FAMILIES TOM LOM THE


READY - PHASE 3

PROCESSING FAMILIES NAS RXL CPU RXS RXB IDM

READY - WO 2008-08-28 15:52:02


Warming Phase

• No new process families, only essential data of processes with corresponding processes in active unit

READY - PHASE 3

PROCESSING FAMILIES NAS RXS RXL HMA RXB CPU IDM

WARMING FAMILIES PIG MNM PHL ULA AMN AVA PMI ODO OCL EFO CPY NSM EQC

FMP URM PVM ARM RBR RAY REU RAK OBH REZ REK SYP JIG

EEF ENP KAR DGT OFM IFM CPS ROI ATY SLY OLW JOR VEX

CLW PI5 LZP JKR PAB

READY - SP


Interrupting DMX Start-up

At the serial connection console, press the following keys:• Flash Boot debugger state:

– STQPRO stops loading of process families; – Pressing 'B' key during starting.

• Minidebugger state: – STQPRO stops starting of process families after code loading phase– Pressing 'M' key during starting.

• Disk Boot debugger state: – STQPRO stops loading of process families from source unit– Press 'G' key during starting.

• Debugger state: – STQPRO stops starting of process families after system starting phase– Pressing 'D' key during starting.


CCP18D Restart

• Support reset types– Power-up reset (not possible via MML)

– Warm reset

– Cold reset (only possible in units whose restart will not lead to system restart)

• BIOS– Duplicated. One default, one as backup.

– User can define which one is default. BIOS itself can take back-up into use if the default one is broken.

– With DMX-specific functionality like warm reset counter.

• HDS-C– SAS is used instead of SCSI.


Chorus Two-Phase Boot

The main components of this boot sequence are the following:• preboot - a program that is started first

on each reset, stored on FLASH memory of unit, starts proper Chorus OS boot archive

• boot archive (CBOOTA, system image, BA) - a Chorus proprietary binary that contains Chorus OS and some drivers and actors, one BA is stored on FLASH memory, another loaded by LEADER from disk

• LEADER - a loader in FLASH boot archive, loads a new boot archive into system RAM

• DAQTOR - a loader and starter in disk boot archive (loaded by LEADER), loads and starts most of actors

HW reset

Warmrestart?

Reset counterOK?

Is the disk SIvalid in RAM?

Start SI from FLASH

Zero reset counter

Start disk SI from RAM,increase reset counter

Pre-boot

F

Boot actorsfrom FLASH

Boot actorsfrom disk

F

F

F

T

T

T

T

LEADER loads new SI and actorsfrom external source to RAM

LEADER: warm restartto the unit

Optimalloading

possible

FS init andboot actor starting

DAQTOR starts

DAQTOR starts actors

DAQTORloads from

external source

Zero reset counter

HW reset

Warmrestart?

Reset counterOK?

Is the disk SIvalid in RAM?

Start SI from FLASH

Zero reset counter

Start disk SI from RAM,increase reset counter

Pre-boot

F

Boot actorsfrom FLASH

Boot actorsfrom disk

F

F

F

T

T

T

T

LEADER loads new SI and actorsfrom external source to RAM

LEADER: warm restartto the unit

Optimalloading

possible

FS init andboot actor starting

DAQTOR starts

DAQTOR starts actors

DAQTORloads from

external source

Zero reset counter


Chorus Sub-phases

• Pre-boot

• Flash boot

• Disk boot

• File loading

• Starting of actors


PBOOT

• Processor and memory banks initialized

• Second phase boot loader started from flash

• Loads actors needed for HMS communication

• OMU transfers physical and ATM config info

• ATM interface initialized

• DMX messaging system opens channels

NOKIA MPC8560 BOOT LOADER

BUILD VERSION: @(#)MID: P3BMXG01.PAC 3.1-7 08/03/10

CONFIGURATION: mx1g6_v01.conf MX1G6_1_7

PREBOOT: pbootc.img P3BOOT_1_18

PHASE 2: cboota PBO_NEWHW1_A7_1.9-2_12

CHECKSUM: 03efa8c7

VERSION CHECKSUM: dd3cf3c6

RESET TYPE: COLD

RESET CAUSE: 00000000

MCPSUMR: 00000000

GPPORCR: 12180000

CPU INDEX: 0

MPC8560 VERSION: 80700020

CPU VERSION: 80200020

PIU TYPE: 00010a22

SYSCLK MHZ: 66

CCB MHZ: 333

CPU MHZ: 833

SDRAM SIZE: 10000000

MAC 0: 00:40:43:eb:34:3c

MAC 1: 00:40:43:eb:34:3d

TESTING MEMORY...10000000 BYTES OK

TESTING MEMORY...00800000 BYTES ON LOCAL BUS OK

INITIALIZING

CPM READY

LOADING PHASE 2

COMPRESSED IMAGE FOUND, SIZE 2681439 BYTES

DECOMPRESSING............................................DONE

UNCOMPRESSED SIZE 9383936 BYTES

STARTING PHASE 2


Disk Boot

• LEADER loads list of actors to be loaded

• Unit is restarted

• Loads ChorusOS and some actors

• OS started

RXFPRBRXSPRB

LEADER

Starting computer

FIMCHO

Source DMX computer

1. Request of the loading source list

3. Reserve the loading source

SYM unit

POSIX file system

Read files

Possible if the load source is SYM unit

Read files

Ack of 1. List of loading source list

2. Request disk boot list name and possible microcode

4. load modules

PULINARDISFS/RMEMFS

PUHINA

Source Chorus master unit

RAM disk

4

2

3

source is Chorus master unit

source is cluster center DMX unit

Legend


File Loading

Load dynamically started actors not in boot archive, service terminal commands and other data to RAM disk

Steps1. DAQTOR contacts RXFPRB located in active sym computer

2. RXFPRB gives DAQTOR loading mode to use and source unit address for file loading

3. DAQTOR contacts loading manager in source unit (PUHINA in Chorus/PULINA in DMX)

4. Loading manager gives permission to start loading

5. DAQTOR loads the AQLIST which tells the files to be loaded

6. DAQTOR loads to RAM disk

DAQTOR

FIMCHO

Starting Chorus unit

3. Read actor load module

Provides loading services for DAQTOR

1. Request of theloading source list

RXFPRB RXSPRB

PULINA

RD4FIM

RM4FIM

SYM / Source DMX computer

SYM / Source DMX computer

2. Request a load list from DMX

PUHINA

RAM diskRAM disk

2.Request a load list from master

unit

Master Chorus unit

3. Read actor load module


Starting of Actors

• Starts actors loaded to RAM disk not belonging to boot archive

• DAQTOR starts system starting phase actors (QXFPRB)

• QXFPRB starts defining unit characteristics

• DAQTOR starts actors in application starting phase

• Unit goes to WO-EX (POHEXT shows ”open”)

SYSTEM STARTING PHASE

STARTING ACTORS

puzage bonsaiBONSAI started - API v.AG_NP-4.0.0.52.APP6

reyska svator quzman ch1sup qalarm qxfprb uxwarm

qxuprb qieser qunman

DEFINING UNIT TYPE

swkonf qeffer fellow cpodrv cpovld oljman

SYSTEM STARTING PHASE COMPLETED

APPLICATION STARTING PHASE

STARTING ACTORS

cukyky mm0cho sboprb jungle naomic lmcprb mm1cho ivbeer rs3prbnokiasiemensnetw

orks:pci-app6: APP reseted and released from reset (0xff790d50=0xf8f0ff68, was 0

xf8f0ff68)

nokiasiemensnetworks:pci-app6: APP configured.

diecho bilman

APPLICATION STARTING PHASE COMPLETED

READY - WO 2008-08-29 17:40:28


Comparison of DMX and Chorus Start-up

Phase Description DMX equivalent

flash Boot Archive stored into the flash. There is no DAQTOR in this Boot Archive.

flash boot

boot Chorus boot loads and starts new Boot Archive. This Boot Archive contains DAQTOR.

booting

minidebugger state

code loading DAQTOR loads codes of dynamically started actors, service terminal extensions and data files to the local RAM disk

phase 1 (initial loading)

actor starting, system phase

DAQTOR starts actors that are needed in unit start -up (for example QXFPRB)

phase 1 (phase 99 of STQPRO)

debugger state

actor starting, application phase

DAQTOR starts application actors. QXFPRB controls execution of this phase.

phase 3


Investigating Start-up Failures: SQQ

• SQQ extension is a DMX service terminal extension used for handling unit start-up related operations.

• It commands the starting-, warming-and supervision services of process families.

• It is also effective aid for testing purposes.

0001-SQQ>? SQQEXT - STARTER EXTENSION AND INTERFACE ? ..... HELP D ..... DISPLAY START/WARM PARAMETER DATA F ..... FAMILY SERVICES: START/DESTROY FAMILY I ..... INTERRUPTED UNIT RESTART P ..... OUTPUT R ..... FAMILY RESTART RELATED COMMANDS S ..... SUPERVISION RELATED COMMANDS T ..... STATISTICS U ..... FAMILY_INFO Z ..... EXIT


SQQ Commands

Useful SQQ commands:P

• Prints out the start-up phases of the unit

SD/SC• Deactivates / Continues the supervision

for a process family Ix

• Handles break-points for an interrupted unit start-up

0001-SQQ>SD:00FFFAMILY FF SUPERVISION DEACTIVATED

0001-SQQ>SC:00FFFAMILY FF SUPERVISION CONTINUED

003F-SQQ>?IIC; Continue the interrupted start upIP:<phase>,<subphase>; Set breakpoint phase and subphaseIPS:<phase>,<subphase>; Set breakpoint in next restart! * subphase of phase 0x99 is to select which build to stop at! * (0 = Normal, 1 = Boot, 2 = both)IF:<fam_id>; Set breakpoints to certain familyIQ; Query:list of breakpoints and phase informationIR; Reset all breakpointsIS; Proceed one start task


Phase Output in Chorus

Kernel printout

• Prints everything from the preboot and flash boot phases

• sysdump -p

DAQTOR printout

• Prints everything from the disk boot phase

• cat /var/daqtor/phaseoutput.txt


Start-up Failure during Warming

The most common warming problems are:

• Warming is not started for the SP application at all

• Warming the SP application fails for some reason

• Warming the SP application times out


Warming in DMX

When warming is not started or times out for the SP application, the following will occur:

• Warming the unit takes a long time (warming sticks at the process that doesn’t warm up, until time-out occurs)

• If the warmed application is a critical one the unit will be restarted after three failed warming attempts

• If the application on the other hand is non-critical warming will continue after this one with next warmable application


Warming in DMX

If warming the SP application fails for some reason the following will occur:

• An error message will be written on the service terminal as soon as the error occurs

• Warming is then re-tried three times, and if it doesn’t succeed it is either skipped and warming continues with next warmable application, or

• The unit is restarted if the application was a critical one


Investigating Spontaneous Restart

Three tools are quite useful for determining the cause of a restart. These are:

• DMX unit Black Box

• Chorus unit Black Box

• Chorus unit System Dump

This information can be output for the previous 10 restarts


DMX Black Box

• Use BOXANA extension

• Contains data on unit states before restart


Chorus Black Box (BBLOG)

0010-$ bblog -?

Usage: bblog [args]

Possible arguments are:

-c <number> <file id>... Show given number computer log,

number=0 show log headers.

-c -a <file id>......... Show details of computer log.

-c -l <file id>......... Show headers of computer log.

-c -f <dmx_id> <file id> Show computer log by family id.

-o <file id>............ Show operating system log.

-o -f <dmx_id> <file id> Show operating system log by family id.

-o -t <type> <file id>.. Show operating system log by entry type.

-h ..................... Show this help.

Only one argument is allowed in time.

Select Black Box File <file id>, default is 2(previous restart).


BBLOG Computer Logs

0010-$ bblog -c -a 3

CALLER: 3CB TYPE: I-8 DATE: 01.01.1970 TIME: 00:00:33.01

USER TEXT: UXCPRB: NOTICE -- UX1 SYNC FOUND interrupt occur

USER DATA: port = 0

utopia = 6C

hw_block = 1

CALLER: 3CB TYPE: E-8 DATE: 01.01.1970 TIME: 00:00:35.91

USER TEXT: UXCPRB: hms_get_own_sym_hms_addr_r failed

USER DATA: Can not send fig_service_req_msg_s

ec = 45

CALLER: 3CB TYPE: E-8 DATE: 01.01.1970 TIME: 00:00:39.66

USER TEXT: UXCPRB: hms_get_own_sym_hms_addr_r failed

USER DATA: Can not send res_pqii_int_if_param_req_s

ec = 45


BBLOG Operating Logs

0010-$ bblog -o 4

NUMBER: 01 TYPE: 02 DATE:01.01.1970 TIME: 00:00:22

FAMILY: reyska BUILD. DMX ID IS: BE


FAMILY: qalarm BUILD. DMX ID IS: 143


FAMILY: svator BUILD. DMX ID IS: 448


FAMILY: fellow BUILD. DMX ID IS: 54C


Chorus System Dump

The Chorus System Dump provides following information

• General logs. These logs include exception notifications and logs made by Chorus application

• Failed system calls information

• Running actors and threads

• Last 16 context switching

• Kernel console printouts

It is possible to read kernel log information from

1. Running kernel,

2. Kernel from before previous reset

3. Kernel information from before previous two resets (useful in case of two-phase boot).

Documents

NOKIA RNC Start-up Troubleshooting