39

Click here to load reader

CUCM Traces

Embed Size (px)

Citation preview

Page 1: CUCM Traces

Understanding CUCM TracesCUCM Traces

AKHIL BEHLNCE, GDC UCCCIE (Voice, Security)

© 2010 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialAKHIL BEHL CUCM Traces 1

Page 2: CUCM Traces

Session Objective

CUCM Traces – IntroductionSDI t SDI traces

SDL traces Explain types of tracesp yp Trace configuration Collecting trace files from CUCM Common trace file locations and collection Trace file reading and analysis Best practices for collecting and configuring Best practices for collecting and configuring

traces

2© 2010 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialAKHIL BEHL CUCM Traces

Page 3: CUCM Traces

CUCM Traces –

Introduction

3© 2010 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialAKHIL BEHL CUCM Traces

Page 4: CUCM Traces

CUCM Trace Introduction

CUCM has its own logging system. These are in form of txt files

These txt files are called Traces. They help in troubleshooting and are a very effective way to solve an issue

Th t fil id i d th d t il f i t th The trace files can provide in depth details of various events as they occur with time stamp, event id, trace handles (TCP handle) etc

4© 2010 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialAKHIL BEHL CUCM Traces

Page 5: CUCM Traces

System Diagnostic Interface (SDI) Traces

CUCM (also known as System Diagnostic Interface or SDI) traces are the user friendly call processing trace files we have available

Traces can be set at different levels to capture different activities

D t il d P id d t il d d b i f ti d hi hl Detailed : Provides detailed debug information and highly repetitive messages that are primarily used for debugging, including Keepalives and responses

Arbitrary : Provides low-level debug traces. This level is best suited for debugging difficult problems. This level includes nearly everything that is included in Detailed with the exception ofeverything that is included in Detailed with the exception of KeepAlives

5© 2010 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialAKHIL BEHL CUCM Traces

Page 6: CUCM Traces

Error : Provides traces generated in abnormal conditions, such as coding errors or other errors that normally should not occur

S i l P id t f ll i f ti l titi Special : Provides traces for all informational, non-repetitive messages such as process startup messages, registration messages, and so on. All system and device initialization traces are at this levelare at this level

State Transition : Provides traces for call processing events or normal events traced for the subsystem (signaling layers)normal events traced for the subsystem (signaling layers)

Significant : Provides trace for Media Layer events

SCCPSignaling

RTP

6© 2010 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialAKHIL BEHL CUCM Traces

RTPVoice Stream

Page 7: CUCM Traces

System Distribution Layer : SDL Traces An SDL Trace log file contains call-processing information from services such as Cisco CallManager, Cisco CTIManager, and Cisco TFTPTFTP

Its important to understand how the SDL files are named –

SDL trace filenames will be in the format: SDLxxx_yyy_zzzzzz.txt

x is the node id , y is the application id, z is the trace file number

The node id is unique to each node in the cluster and is assigned at the time the server is added to the System->Communications Manager page in CCMAdmin

100 - CallManager (ccm), 200 - CTI Manager (ctimanager) etc

SDL

7© 2010 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialAKHIL BEHL CUCM Traces

SDL

Page 8: CUCM Traces

SDL Trace ContentsSDL trace contents

Line Number | date + timestamp | node id | type of signal | signal name | state ofLine Number | date + timestamp | node id | type of signal | signal name | state of receiving process | receiving process | sending process | tag field | signal queue depth | additional info

Line Number starts at 000000000 when the service starts It keeps incrementingLine Number starts at 000000000 when the service starts. It keeps incrementing until the service is restarted and the value is preserved across trace files

The date/timestamp is the date and time when the trace file was written

Node id is simply a reference within each SDL trace line indicating the id of the node that wrote the trace line

The type of signal has multiple valuesThe type of signal has multiple values

Signal - the name of the signal

8© 2010 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialAKHIL BEHL CUCM Traces

Page 9: CUCM Traces

SDL Contents (Contd.)SDL trace file contents

Line Number | date + timestamp | node id | type of signal | signal name | state ofLine Number | date + timestamp | node id | type of signal | signal name | state of receiving process | receiving process | sending process | tag field | signal queue depth | additional info

The state of the receiving process is the state before the signal is processedThe state of the receiving process is the state before the signal is processed

The receiving process is the PID of the process recieving the SDL signal

The sending process is the PID of the process sending the SDL signalThe sending process is the PID of the process sending the SDL signal

The tag field is miscellaneous information that is not commonly used in troubleshooting

The signal queue depth outlines the number of signals waiting to be processed in each queue

The additional info field is typically the data associated with the signal being sent

9© 2010 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialAKHIL BEHL CUCM Traces

The additional info field is typically the data associated with the signal being sent to the receiving process. For an error it will contain details about the error. It is valid for this field to be empty

Page 10: CUCM Traces

T f SDI TTypes of SDI Traces

10© 2010 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialAKHIL BEHL CUCM Traces

Page 11: CUCM Traces

Callmanager : Types of Traces and their meaningg

Cisco CallManager Trace Fields Enable H245 Message Trace : This is used for H 245 signaling forEnable H245 Message Trace : This is used for H.245 signaling for

H.323 calls, including the media processing messages. Enable PRI Trace : This activates a trace of Primary Rate Interface

(PRI) devices. Enable ISDN Translation Trace : This is used for a Layer 3 trace of

Q.931 (ISDN messages). Enable H225 & Gatekeeper Trace : This is used to trace the H.225

signaling messaging for H 323 callssignaling messaging for H.323 calls. Enable Miscellaneous Trace : Activates a trace of miscellaneous

devices.E bl C f B id T U d t f t f th Enable Conference Bridge Trace : Used to perform trace of the conference bridges.

Enable CM Real-Time Information Server Trace : This is used for CallManager RIS traces - the real-time information server.

11© 2010 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialAKHIL BEHL CUCM Traces

CallManager RIS traces the real time information server. Enable CDR Trace : This enables tracing of call detail record (CDR)

processing.

Page 12: CUCM Traces

Callmanager : Types of Traces and their meaning (Contd.)g ( ) Enable Analog Trunk Trace : This is used for a trace of all MGCP-based

devices using an analog interface.

Enable All Phone Device Trace : To get a trace of phone devices, including Cisco IP SoftPhones, and shows events such as on-hook, off-hook, key presses, and so on.

Enable MTP Trace : This trace if for media termination point devices and transcoders.

Enable All Gateway Trace : This activates a trace of all analog and digitalEnable All Gateway Trace : This activates a trace of all analog and digital gateways.

Enable Forward and Miscellaneous Trace : Used for trace of call forwarding and all subsystems not covered by another checkboxand all subsystems not covered by another checkbox.

Enable MGCP Trace : For MGCP messages for MGCP-based devices.

Enable Media Resource Manager Trace : This is trace for media resource

12© 2010 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialAKHIL BEHL CUCM Traces

gmanager.

Page 13: CUCM Traces

Other Traces There are other service traces as well in

addition to CUCM traces CTI traces to debug and troubleshoot CTI CTI traces – to debug and troubleshoot CTI related issues CDR – To troubleshoot CDR related issues DB and Admin To troubleshoot DB sync DB and Admin – To troubleshoot DB sync issues, MLA logging etc Performance and Monitoring – Same as windows Perfmon Security services – CAPF related traces Directory services – DirSync (LDAP trace) traces Backup and Restore – Related to DRS traces System Services – CCMWeb service, common interface etc traces

13© 2010 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialAKHIL BEHL CUCM Traces

SOAP Services – AXL SOAP related traces Platform services – OS Admin interface traces

Page 14: CUCM Traces

CUCM Trace

Configuration

14© 2010 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialAKHIL BEHL CUCM Traces

Page 15: CUCM Traces

CUCM Trace Configurationg

CUCM Trace – Configuration ->

Ci U ifi d i bilit T T fi tiCisco Unified serviceability -> Trace -> Trace configuration

15© 2010 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialAKHIL BEHL CUCM Traces

Page 16: CUCM Traces

CUCM Trace Configuration (Contd.)

16© 2010 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialAKHIL BEHL CUCM Traces

Page 17: CUCM Traces

Trace Collection

17© 2010 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialAKHIL BEHL CUCM Traces

Page 18: CUCM Traces

CUCM Trace Collection

CUCM Traces from CUCM 5.x onwards are collected using RTMT

RTMT is a plugin available in CUCM plugins

Download RTMT for windows and install it on your pc

Login with CUCM admin credentials with Pub’s IP

Login with desired profile (default is CM_Default)

G t > T l > t d l t l Go to -> Tools -> trace and log central

18© 2010 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialAKHIL BEHL CUCM Traces

Page 19: CUCM Traces

CUCM Trace Collection Procedure (Step by Step)by Step) Step 1: Turn on Traces in CallManager

Open up CallManager Admin page and navigate to Serviceability Then go toOpen up CallManager Admin page and navigate to Serviceability. Then go to Trace Configuration, select publisher and choose the CallManager Service. Make sure Trace is on, and that debug trace level is set to Detailed. We will also see a lot of options with checkboxes. Make sure that the “apply to all

d ” i l h k d d li knodes” is also checked and click save

Step 2: Download and Install RTMT (Real Time Monitoring Tool)( g )

This tool is accessed from the normal CallManager Administration page. It can be found under Application Plugins. Download and install this tool. Make sure we can run the RTMT tool and log in using the admin credentials you

t l i t th b d i Ch T l T T & Luse to log into the web admin page. Choose Tools ->Trace ->Trace & Log Central (ensure that is there’s a firewall between management PC and CUCM, the ports for RTMT are opened – 1090 and 1099 TCP )

19© 2010 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialAKHIL BEHL CUCM Traces

Page 20: CUCM Traces

CUCM Trace Collection Procedure (Step by Step) Contd.by Step) Contd. Step 3. Place Your Test Call

Once we have RTMT loaded up we need to place your test call or create theOnce we have RTMT loaded up, we need to place your test call or create the condition we are trying to capture. Try to repeat the problem re creation twice so that, we can be sure that we can find it

Step 4. Collect Trace Files

Now go to RTMT window that we opened in step 2, and double click “Collect Files” Find CallManager in the list and click the checkbox for “All Servers”Files . Find CallManager in the list, and click the checkbox for All Servers since we may have a situation where a phone is registered to one server and the gateway is registered to another. This is good info to have beforehand so the search for the trace is a bit easier. Now for the file options, choose relative

d b k 5 10 i t M k h t h irange and go back 5 or 10 minutes. Make sure whatever we choose is sufficient to capture the test call or error condition

20© 2010 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialAKHIL BEHL CUCM Traces

Page 21: CUCM Traces

CUCM Trace Collection

Click Collect files

21© 2010 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialAKHIL BEHL CUCM Traces

Page 22: CUCM Traces

CUCM Trace Collection (Contd.) Choose CUCM traces (and any other traces desired)

22© 2010 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialAKHIL BEHL CUCM Traces

Page 23: CUCM Traces

CUCM Trace Collection (Contd.) Accept the defaults for save (except delete collected log files from

server and also select Zip Files)

S l t th d t d ti f hi h t t b l d Select the date and time from which traces are to be analyzed

23© 2010 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialAKHIL BEHL CUCM Traces

Page 24: CUCM Traces

CUCM Trace Collection (Contd.) The RTMT tool will collect the files for the selected time period

The files will be saved in selected location (default is C:\Program Fil \Ci \C llM S i bilit \JRt t\Files\Cisco\CallManager Serviceability\JRtmt\<CUCMname_CiscoCallManager_sdi_2009-06-15_12-46-57.zip>

24© 2010 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialAKHIL BEHL CUCM Traces

Page 25: CUCM Traces

Common Trace locations and collection

25© 2010 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialAKHIL BEHL CUCM Traces

Page 26: CUCM Traces

CUCM frequently used traces and file locationlocation SDI traces - /var/log/active/cm/trace/ccm/sdi

SDL traces - /var/log/active/cm/trace/ccm/sdl

Perfmon logs - /var/log/active/cm/log/ris/csv and /var/log/active/cm/log/amc/PerfMonLog

Callmanager app log /var/log/active/syslog/CallManager* * Callmanager app log - /var/log/active/syslog/CallManager*.*

Note – to get the files use the syntax - file get activelog cm/trace/ccm/sd*/*

Equivalent to CLI trace collection, in RTMT this can be done as follows –

For UCM sdl and sdi traces, check the "Cisco CallManager" box for each server

For RIS Perfon logs, check the box "Cisco RIS Data Collector PerfMonLog" for each server you are interested in

For the System Event Logs and Application logs, check the box for "Event Viewer -

26© 2010 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialAKHIL BEHL CUCM Traces

For the System Event Logs and Application logs, check the box for Event Viewer System Log and Application log" for each server you are interested in

Page 27: CUCM Traces

Reading CUCM Traces

27© 2010 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialAKHIL BEHL CUCM Traces

Page 28: CUCM Traces

Reading CallManager SDI Tracesg gHow to Read a Trace ?

Stationinit is a message from the end point to the CallManager.

StationD is the message from CallManager to endpoint.

Start finding dd=“xxx, where dd is Dialed Digits and xxx is number dialled

When you get the Dialed String, make sure to note the TCP Handle because it doesn’t changes for a Phone unless its reset.

28© 2010 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialAKHIL BEHL CUCM Traces

Page 29: CUCM Traces

Reading CallManager Traces (Contd.)Reading CallManager Traces (Contd.)

Now follow the tcp handle to go with flow of callNow, follow the tcp handle to go with flow of call

Alternatively (less preferred method) you can follow the Device ID y ( p ) y(Mac address) or the IP of the endpoint

If f i i l d t h dl i t d S If a conference is involved, a new tcp handle is generated. So, one now has to follow this new tcp handle till call lasts or conference lasts

The dialed string space also gives a clear picture of CSS, PSS, translation pattern hit, Route Pattern hit etc

29© 2010 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialAKHIL BEHL CUCM Traces

Page 30: CUCM Traces

Example Trace Filep

Trace File

30© 2010 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialAKHIL BEHL CUCM Traces

Page 31: CUCM Traces

An Example of CFB TraceAn Example of CFB Trace

In this trace we can see how the CFB Resources are configured on the CallManager Also we can see the error of resource unavailablethe CallManager. Also we can see the error of resource unavailable.

CallManager|UnicastBridgeControl::StationRegister MaxStream=0, ActiveStream=0|<CLID::CALLMANAGER01-Cluster><NID::10.7.0.25><CT::1,100,95,1.7687829><IP::10.7.0.25><DEV::CFB CALLMANAGER>_04/25/2002 16:41:58.948 Cisco CallManager|UnicastBridgeControl::StationRegister CurrentAvail= 0 MaxAvail=0|<CLID::CALLMANAGER01-Cluster><NID::10.7.0.25><CT::1,100,95,1.7687829><IP::10.7.0.25><DEV::CFB_CALLMANAGER>

04/25/2002 16:41:58.838 Cisco IP Voice Media Streaming App|kRequestedCFBStreamsFailed - CFB requested streams failure. Requestedfailure. Requestedstreams:48 Allocated streams:0 App ID:Cisco IP Voice Media Streaming App Cluster ID:CALLMANAGER01-Cluster NodeID:10.7.0.25|<CLID::CALLMANAGER01-Cluster><NID::10.7.0.25><CT::Alarm>

04/25/2002 16:41:34.306 Cisco IP Voice Media Streaming App|CIpVMSMgr::ReadConfiguration CFB DeviceName =CFB_CALLMANAGER|<CLID::CALLMANAGER01-Cluster><NID::10.7.0.25>04/25/2002 16:41:34.322 Cisco IP Voice Media Streaming App|CIpVMSMgr::ReadConfiguration CFB ServerName[0] =04/25/2002 16:41:34.322 Cisco IP Voice Media Streaming App|CIpVMSMgr::ReadConfiguration CFB ServerName[0] 10.7.0.25, Port =2000|<CLID::CALLMANAGER01-Cluster><NID::10.7.0.25>04/25/2002 16:41:34.338 Cisco IP Voice Media Streaming App|CIpVMSMgr::ReadConfiguration CFB Number of Parties =48|<CLID::CALLMANAGER01-Cluster><NID::10.7.0.25>04/25/2002 16:41:34.338 Cisco IP Voice Media Streaming App|CIpVMSMgr::ReadConfiguration CFB Run Flag =1|<CLID::CALLMANAGER01-Cluster><NID::10 7 0 25>

31© 2010 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialAKHIL BEHL CUCM Traces

1|<CLID::CALLMANAGER01 Cluster><NID::10.7.0.25>

Page 32: CUCM Traces

An Example of AC Client tracep

Call control initialization failed error message comes on ac.

1 09/07 07:55:25.953 DEBUG New trace level: DEBUG

2 09/07 07:55:25.953 INFO

3 09/07 07:55:25.953 INFO **************************************

4 09/07 07:55:25.953 INFO ****** Attendant Console Client ******

5 09/07 07:55:25.953 INFO * Version : 1.2(1 ES7)

6 09/07 07:55:25.953 INFO * Built on : Thu Nov 6 13:17:48 PST 2003

7 09/07 07:55:25.953 INFO **************************************

8 09/07 07:55:25.953 INFO

9 09/07 07:55:25.953 DEBUG Command Argument 0: C:\Program Files\Cisco\Call Manager Attendant Console\bin\ACClient

12 09/07 07:55:26.140 WARN Line state socket creation failed on /199.42.66.144:1105: java.net.BindException: Cannot assign requested address: Cannot bind

The issue was related to the XP ghost and on reinstalling afresh, th ll t l

32© 2010 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialAKHIL BEHL CUCM Traces

the call control came up

Page 33: CUCM Traces

An Example of TFTP Tracep

TFTP trace showing an issue with file not being found in TFTP

The phone is unable to register and keeps on rebootingThe phone is unable to register and keeps on rebooting

02/20/2008 17:34:15.144

TFTP|CTftpConnect::checkAndSetIsStatic()[0268F1F8][3976:132 143 64 20:25000]TFTP|CTftpConnect::checkAndSetIsStatic()[0268F1F8][3976:132.143.64.20:25000],

[sep00506003606c.cnf.xml] is [not a] Static File|<CLID::StandAloneCluster><NID::CM-1>

02/20/2008 17:34:15.144

TFTP|-->CTftpConnect::tftp[3976:132.143.64.20:25000] |<CLID::StandAloneCluster><NID::CM-1>

02/20/2008 17:34:15.144

TFTP|CTftpConnect::checkAndHandleDiagReq[0268F1F8],[3976:132.143.64.20:25000],

File[SEP00506003606C.cnf.xml]|<CLID::StandAloneCluster><NID::CM-1>

02/20/2008 17:34:15.144 TFTP|

CTftpConnect::tftp[3976:132.143.64.20:25000] [SEP00506003606C.cnf.xml] not found in bin/config cache Looking at disk|<CLID::StandAloneCluster><NID::CM-1>

33© 2010 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialAKHIL BEHL CUCM Traces

This issue was resolved by rebuilding the CNF files from TFTP service parameters

Page 34: CUCM Traces

An Example of PRI Tracep

PRI Trace showing issue with an incoming call on MGCP gateway

03/09/2010 11:26:46.940 CCM|MGCPpn9d - S0/SU1/DS1-1@FFX-BW-7-VoIP-GW - Collision Detected in PriNi2 Interface, callphase= 9|<CLID::StandAloneCluster><NID::162.70.58.2><CT::2,100,135,1.2928815> <IP::1 62.70.58.4><DEV::><LVL::State Transition><MASK::2000>

03/09/2010 11:26:46.940 CCM|Entering default,restarting | g , gchannel|<CLID::StandAloneCluster><NID::162.70.58.2><CT::2,100,135,1.2928815 > <IP::162.70.58.4><DEV::><LVL::State Transition><MASK::2000> 03/09/2010 11:26:46.941 CCM|MGCPpn9d::conflictingChannelHandler - Don't Restart this channel since it is marked OOS_NE under service parameter:S0/SU1/DS1-1@FFX-BW-7-VoIP-GW= or due to MGCP gateway hardware failure|<CLID::StandAloneCluster><NID::162.70.58.2><CT::2,100,135,1.2928815 >

This was a Hardware issue as all channels were in service

This issue was resolved by upgrading DSP firmware

34© 2010 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialAKHIL BEHL CUCM Traces

Page 35: CUCM Traces

Best Practices for est act ces oTrace reading

35© 2010 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialAKHIL BEHL CUCM Traces

Page 36: CUCM Traces

Best Practices for Trace Readingg

Always, have an Idea about the time period when issue occurs. It becomes very very difficult to go through hundreds of Trace filesbecomes very very difficult to go through hundreds of Trace files

Always Understand the Call Flow before troubleshootingy g

Information about network setup and topology always helps d t d th fl f i li d diunderstand the flow of signaling and media

Always try to save older trace files and begin afresh if debugging aAlways try to save older trace files and begin afresh if debugging a new issue

36© 2010 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialAKHIL BEHL CUCM Traces

Page 37: CUCM Traces

Q & A ?

37© 2010 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialAKHIL BEHL CUCM Traces

Page 38: CUCM Traces

Th k YThank You

38© 2010 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialAKHIL BEHL CUCM Traces

Page 39: CUCM Traces

39© 2010 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialAKHIL BEHL CUCM Traces