Troubleshooting Guide for Cisco Unified Customer Voice Portal

Troubleshooting Guidefor Cisco Unified Customer Voice Portal7.0(2)

January 2011

Americas HeadquartersCisco Systems, Inc.170 West Tasman DriveSan Jose, CA 95134-1706USAhttp://www.cisco.comTel: 408 526-4000800 553-NETS (6387)Fax: 408 527-0833

THE SPECIFICATIONS AND INFORMATION REGARDING THE PRODUCTS IN THIS MANUAL ARE SUBJECT TO CHANGE WITHOUT NOTICE.ALL STATEMENTS, INFORMATION, AND RECOMMENDATIONS IN THIS MANUAL ARE BELIEVED TO BE ACCURATE BUT ARE PRESENTEDWITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED. USERS MUST TAKE FULL RESPONSIBILITY FOR THEIR APPLICATION OFANY PRODUCTS.THE SOFTWARE LICENSE AND LIMITED WARRANTY FOR THE ACCOMPANYING PRODUCT ARE SET FORTH IN THE INFORMATION PACKETTHAT SHIPPED WITH THE PRODUCT AND ARE INCORPORATED HEREIN BY THIS REFERENCE. IF YOU ARE UNABLE TO LOCATE THESOFTWARE LICENSE OR LIMITED WARRANTY, CONTACT YOUR CISCO REPRESENTATIVE FOR A COPY.The Cisco implementation of TCP header compression is an adaptation of a program developed by the University of California, Berkeley (UCB) aspart of UCBs public domain version of the UNIX operating system. All rights reserved. Copyright 1981, Regents of the University of California.NOTWITHSTANDING ANY OTHER WARRANTY HEREIN, ALL DOCUMENT FILES AND SOFTWARE OF THESE SUPPLIERS ARE PROVIDED"AS IS" WITH ALL FAULTS. CISCO AND THE ABOVE-NAMED SUPPLIERS DISCLAIM ALL WARRANTIES, EXPRESSED OR IMPLIED, INCLUDING,WITHOUT LIMITATION, THOSE OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OR ARISINGFROM A COURSE OF DEALING, USAGE, OR TRADE PRACTICE.IN NO EVENT SHALL CISCO OR ITS SUPPLIERS BE LIABLE FOR ANY INDIRECT, SPECIAL, CONSEQUENTIAL, OR INCIDENTAL DAMAGES,INCLUDING, WITHOUT LIMITATION, LOST PROFITS OR LOSS OR DAMAGE TO DATA ARISING OUT OF THE USE OR INABILITY TO USETHIS MANUAL, EVEN IF CISCO OR ITS SUPPLIERS HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.Cisco and the Cisco Logo are trademarks of Cisco Systems, Inc. and/or its affiliates in the U.S. and other countries. A listing of Cisco's trademarkscan be found at http://www.cisco.com/go/trademarks. Third party trademarks mentioned are the property of their respective owners. The use ofthe word partner does not imply a partnership relationship between Cisco and any other company. (1005R)Any Internet Protocol (IP) addresses used in this document are not intended to be actual addresses. Any examples, command display output, andfigures included in the document are shown for illustrative purposes only. Any use of actual IP addresses in illustrative content is unintentional andcoincidental.Copyright 2011 Cisco Systems, Inc. All rights reserved.

http://www.cisco.com/go/trademarks

Table of Contents

Preface ...........................................................................................................................................................1Purpose .....................................................................................................................................................1Audience ....................................................................................................................................................1Organization ..............................................................................................................................................1Related Documentation .............................................................................................................................2Conventions................................................................................................................................................3Obtaining Documentation and Submitting a Service Request...................................................................4Documentation Feedback...........................................................................................................................4

1. Troubleshooting Overview...........................................................................................................................5Problem Solving Process...........................................................................................................................5

Guidelines to Assist in Isolating Problems.............................................................................................5Problem Solving Tasks...............................................................................................................................6Troubleshooting Checklist.........................................................................................................................11

2. Symptoms and Solutions..........................................................................................................................13Overview...................................................................................................................................................13Call Server................................................................................................................................................13VXML Server............................................................................................................................................38Reporting Server......................................................................................................................................54Audio........................................................................................................................................................60Transfer and Connect...............................................................................................................................77Unified CM................................................................................................................................................91Unified ICME............................................................................................................................................91Content Services Switch..........................................................................................................................98General System Issues............................................................................................................................98H.323 Troubleshooting............................................................................................................................105SIP Troubleshooting................................................................................................................................109SNMP Troubleshooting...........................................................................................................................117Installation Troubleshooting....................................................................................................................118Gateway Troubleshooting.......................................................................................................................124Cisco Unified Presence Server..............................................................................................................128Operations Console (OAMP)..................................................................................................................129Cisco Security Agent (CSA)...................................................................................................................130Support Tools..........................................................................................................................................131Backup and Restore...............................................................................................................................131

Appendix A. Error Messages......................................................................................................................135Infrastructure Error Messages................................................................................................................135IVR Error Messages...............................................................................................................................141VMS Error Messages.............................................................................................................................145Reporting Server Error Messages..........................................................................................................149ICM Error Messages...............................................................................................................................151SIP Error Messages...............................................................................................................................154

Appendix B. Event and Cause Codes.........................................................................................................157IVR Service Error Codes........................................................................................................................157Unified ICME Error Codes......................................................................................................................159SIP Codes..............................................................................................................................................161

Appendix C. Troubleshooting Radvision Errors...........................................................................................163

Troubleshooting Guide for Cisco Unified Customer Voice Portal 7.0(2)

i

Appendix D. Accessing and Interpreting SNMP Data.................................................................................165

Appendix E. Internal Interfaces Description................................................................................................169

Appendix F. Trace Bits Description..............................................................................................................171

Appendix G. Debugging..............................................................................................................................175SIP IOS Debugging................................................................................................................................175

SIP IOS Troubleshooting Commands................................................................................................176H.323 IOS Debugging............................................................................................................................176

Index ...........................................................................................................................................................179


ii

Preface

Purpose

This manual describes how to troubleshoot common issues for the Cisco Unified CustomerVoice Portal (Unified CVP) product. The document also explains the tools and techniques thatcan assist in the troubleshooting process.

Audience

This document is intended for Contact Center managers, Unified CVP system managers, VoIPtechnical experts, and IVR Service application developers.

Organization

The document is divided into the following chapters.

DescriptionChapter

Describes various methods of approaching troubleshooting tasks.Chapter 1, TroubleshootingOverview

Describes particular issues, any associated error messages, the cause of the issue, aswell as the solution.

Chapter 2, Symptoms andSolutions

Provides information on Unified CVP error messages.Appendix A, Error Messages

Provides information on Unified CVP error codes.Appendix B, Event and CauseCodes


1

DescriptionChapter

Describes how to isolate Unified CVP errors from those of Radvision. This appendixaims at distinguishing these errors, along with an explanation of each error code,and the appropriate action to take before involving Radvision TAC.

Appendix C, TroubleshootingRadvision Issues

Provides information on SNMP severity levels and their descriptions.Appendix D, Accessing andInterpreting SNMP Data

Summarizes the major interfaces between the Unified CVP product components andother Cisco products.

Appendix E, Internal InterfacesDescription

Lists the common trace bits, their corresponding values and the description for eachUnified CVP subsystem type. Each value is unique within each Unified CVPsubsystem.

Appendix F, Trace BitsDescription

Describes SIP troubleshooting for some problems that may occur, and describessome useful command line commands for H.323 IOS troubleshooting.

Appendix G, Debugging

Related Documentation

Note: Planning your Unified CVP solution is an important part of the process in setting upUnified CVP. Cisco recommends that you read the Cisco Unified Customer Voice Portal Release7.x Solution Reference Network Design (SRND) guide before configuring your Unified CVPsolution. With Unified 7.x, the Planning Guide for Cisco Unified Customer Voice Portal hasbeen incorporated into the SRND guide.

Unified CVP provides the following documentation:

Cisco Security Agent Installation/Deployment for Cisco Unified Customer Voice Portalprovides installation instructions and information about Cisco Security Agent for the UnifiedCVP deployment.We strongly urge you to read this document in its entirety.

Cisco Unified Customer Voice Portal Release 7.x Solution Reference Network Design (SRND)provides design considerations and guidelines for deploying contact center voice responsesolutions based on Cisco Unified Customer Voice Portal (Unified CVP) 7.x releases.

Configuration and Administration Guide for Cisco Unified Customer Voice Portal describeshow to set up, run, and administer the Cisco Unified CVP product, including associatedconfiguration.

Element Specifications for Cisco Unified CVP VXML Server and Unified Call Studio describesthe settings, element data, exit states, and configuration options for Elements.

Installation and Upgrade Guide for Cisco Unified Customer Voice Portal describes how toinstall Unified CVP software, perform initial configuration, and upgrade.

Operations Console Online Help for Cisco Unified Customer Voice Portal describes how touse the Operations Console to configure Unified CVP solution components.

Port Utilization Guide for Cisco Unified Customer Voice Portal describes the ports used ina Unified CVP deployment.


2

Preface

Related Documentation

Programming Guide for Cisco Unified CVP VXML Server and Unified Call Studio describeshow to build components that run on the Cisco Unified VXML Server.

Reporting Guide for Cisco Unified Customer Voice Portal describes the Reporting Server,including how to configure and manage it, and discusses the hosted database.

Say It Smart Specifications for Cisco Unified CVP VXML Server and Unified Call Studiodescribes in detail the functionality and configuration options for all Say It Smart pluginsincluded with the software.

User Guide for Cisco Unified CVP VXML Server and Unified Call Studio describes thefunctionality of Call Studio including creating projects, using the Call Studio environment,and deploying applications to the Unified CVP VXML Server.

For additional information about Unified ICME, see the Cisco web site Cisco web site (http://www.cisco.com/en/US/products/sw/custcosw/ps1001/tsd_products_support_series_home.html)listing Unified ICME documentation.

Conventions

This manual uses the following conventions:

DescriptionConvention

Boldface font is used to indicate commands, such as user entries,keys, buttons, and folder and submenu names. For example:

boldface font

Choose Edit > Find.

Click Finish.

Italic font is used to indicate the following:italic font

To introduce a new term. Example: A skill group is acollection of agents who share similar skills.

For emphasis. Example: Do not use the numerical namingconvention.

A syntax value that the user must replace. Example: IF(condition, true-value, false-value)

A book title. Example: See the Cisco CRS Installation Guide.

Window font, such as Courier, is used for the following:window font

Text as it appears in code or that the window displays.Example: Cisco Systems,Inc.


3

Preface

Conventions

http://www.cisco.com/en/US/products/sw/custcosw/ps1001/tsd_products_support_series_home.html

DescriptionConvention

Angle brackets are used to indicate the following:< >

For arguments where the context does not allow italic, suchas ASCII output.

A character string that the user enters but that does not appearon the window such as a password.

Obtaining Documentation and Submitting a Service Request

For information on obtaining documentation, submitting a service request, and gatheringadditional information, see the monthly What's New in Cisco Product Documentation, whichalso lists all new and revised Cisco technical documentation, at:

http://www.cisco.com/en/US/docs/general/whatsnew/whatsnew.html

Subscribe to the What's New in Cisco Product Documentation as a Really Simple Syndication(RSS) feed and set content to be delivered directly to your desktop using a reader application.The RSS feeds are a free service and Cisco currently supports RSS version 2.0.

Documentation Feedback

You can provide comments about this document by sending email to the following address:

mailto:[email protected]

We appreciate your comments.


4

Preface

Obtaining Documentation and Submitting a Service Request

http://www.cisco.com/en/US/docs/general/whatsnew/whatsnew.htmlmailto:[email protected]

Troubleshooting OverviewWhen troubleshooting a telephony or IP network environment, it is important to define thespecific symptoms, identify all potential problems that could be causing the symptoms, andthen systematically eliminate each potential problem from most likely to least likely until thesymptoms disappear.

This chapter contains the following topics:

Problem Solving Process, page 5 Problem Solving Tasks, page 6 Troubleshooting Checklist, page 11

Problem Solving Process

The following steps provide some guidelines to assist in isolating a particular issue.

Guidelines to Assist in Isolating Problems

Step 1 Analyze the problem and create a clear problem statement. Define symptoms and potentialcauses.

Step 2 Gather the facts that you need to help isolate possible causes.

Step 3 Consider possible causes based on the facts that you gathered.

Was anything recently added, removed, or modified?

Is it a reproducible event?

Does it occur at a particular time of day, or day of week?


5

Chapter 1

Have there been any changes made to the domain, network, or security policies?

Step 4 Create an action plan. Begin with the most likely problem and devise a plan in which youmanipulate only one variable at a time.

Step 5 Implement the action plan, performing each step while testing to see whether the symptomdisappears.

Step 6 Analyze the results to determine whether the problem has been resolved. If it has, the processis complete.

Step 7 If the problem has not been resolved, create an action plan based on the next most probablecause on your list, or contact the Cisco Technical Assistance Center (TAC), or your CiscoPartner.

Note: Only change one variable at a time. If that does not resolve the issue, undo that changeand move on to the next step of your plan.

Problem Solving Tasks

In addition to the process discussed above, there are other areas to consider if a problem ariseswithin the Unified CVP environment. The information below provides some additionalinformation to assist in isolating or resolving a particular issue.

Verify that the software versions comply with the Hardware and System Software Specificationfor Unified Customer Voice Portal document. A version incompatibility might not necessarilybe the cause of the issue, but can be investigated and considered as a possibility. This documentis available at:

http://www.Cisco.com/en/US/products/sw/custcosw/ps1006/prod_technical_reference_list.html

Review the Unified CVP logs for possible error or warning events. Use the Support Toolsutility since it has the ability to retrieve and process trace logs from most components, andto set or reset trace levels on those components. To access the Support Tools, open theOperations Console, and click Support Tools from the Tools menu. The Configuration andAdministration Guide for Cisco Unified Customer Voice Portal and Cisco Support ToolsUser Guide for Cisco Unified Software provide additional information about Support Tools.

Note: Users can execute some UNIX equivalent commands, such as more, grep, diff, tail,tee on Unified CVP servers during troubleshooting. These commands are part of the SupportTools installation. The PATH environment variable allows these commands to be accessiblevia command line.

Unified CVP captures two different types of logs: events and errors. When a Unified CVPcomponent generates an event, it is logged in the Unified CVP log. When a log statement iswritten to the log file and the level is at ERROR or above, it is also written to error log file.


6

Chapter 1: Troubleshooting Overview


http://www.Cisco.com/en/US/products/sw/custcosw/ps1006/prod_technical_reference_list.htmlhttp://www.Cisco.com/en/US/products/sw/custcosw/ps1006/prod_technical_reference_list.html

Note: This file is created on demand. If there are no ERROR level (or above) logs, you donot see this file.

Below are the locations of the log files for Unified CVP. In the listings below, CVP_HOMEis the location that was selected for the installation of your Unified CVP software, such asC:\Cisco\CVP.

CVP_HOME\logs\

Call Server and/or Reporting Server logs.

CVP_HOME\logs\OAMP\

Operations Console logs.

CVP_HOME\logs\VXML\

VXML Server logs.

CVP_HOME\logs\SNMP\

SNMP Agent logs.

CVP_HOME\logs\ORM\

Unified CVP Resource Manager logs.

Note: The Operations Console Resource Manager (ORM) is a service that is installedduring the installation of the Operations Console. When installing the Operations Console,the two Windows services that are installed are Unified CVP OpsConsoleServer andUnified CVP Resource Manager. The Resource Manager resides on each Unified CVPserver and it allows the Operations Console to communicate with that server and performfunctions on that server, such as starting, stopping, or monitoring.

Virtually all of the Unified CVP logs start a new log file at 12:00AM every night, with thedate as part of the filename. The VXML Server logs do not necessarily start a new file at12:00AM. The format of the date is yyyy-mm-dd. These logs also start a new log file whenthey reach the predefined size limit of 10 MB and have a number as part of the filenameextension. The number indicates which log it was for that day. When the log directory sizereaches the maximum number of megabytes (20,000MB) for disk storage for log files, theold files are purged as necessary. Both the Log File Size and Log Directory Size areconfigurable in the device Infrastructure tab of the Operations Console.

Check that the network is functioning properly. For the initial connectivity tests, the followingcommand line tools may be helpful:

ping - tests whether a host is reachable across an IP network. The ping command offersseveral parameters or flags that can be set to produce various outputs. To view all of theavailable parameters, type ping /? at the command prompt.

C:\> ping 10.86.132.150


7



Pinging 10.86.132.150 with 32 bytes of data:

Reply from 10.86.132.150: bytes=32 time=5ms TTL=122




Ping statistics for 10.86.132.150: Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),Approximate round trip times in milliseconds: Minimum = 4ms, Maximum = 5ms, Average= 4ms

C:\>

traceroute - determines the packet route that is taken to the target host across an IP network.The command in a Windows environment is: tracert. The tracert command offers severalparameters or flags that can be set to produce various outputs. To view all of the availableparameters, type tracert /? at the command prompt.

C:\> tracert 10.1.1.6

Tracing route to 10.1.1.6 over a maximum of 30 hops

---------------------------------------------------

1 2 ms 3 ms 2 ms 10.1.2.1

2 25 ms 83 ms 88 ms 192.168.11.1

3 25 ms 79 ms 93 ms 10.1.1.6

Trace complete.

netstat - lists the active incoming and outgoing network connections on a system. Thenetstat command offers several parameters or flags that can be set to produce variousoutputs. To view all of the available parameters, type netstat /? at the command prompt.

C:\>netstat -a -n

Active Connections

Proto Local Address Foreign Address State

TCP 0.0.0.0:42 0.0.0.0:0 LISTENING

TCP 0.0.0.0:88 0.0.0.0:0 LISTENING

TCP 0.0.0.0:135 0.0.0.0:0 LISTENING

TCP 0.0.0.0:389 0.0.0.0:0 LISTENING


8



TCP 0.0.0.0:445 0.0.0.0:0 LISTENING

TCP 0.0.0.0:593 0.0.0.0:0 LISTENING

TCP 0.0.0.0:1038 0.0.0.0:0 LISTENING

TCP 0.0.0.0:1041 0.0.0.0:0 LISTENING

TCP 0.0.0.0:1048 0.0.0.0:0 LISTENING

TCP 0.0.0.0:1723 0.0.0.0:0 LISTENING

TCP 0.0.0.0:3268 0.0.0.0:0 LISTENING

TCP 10.99.99.1:53 0.0.0.0:0 LISTENING

TCP 10.99.99.1:139 0.0.0.0:0 LISTENING

TCP 10.99.99.1:389 10.99.99.1:1092 ESTABLISHED

TCP 10.99.99.1:1092 10.99.99.1:389 ESTABLISHED

TCP 10.99.99.1:3604 10.99.99.1:135 TIME_WAIT

TCP 10.99.99.1:3605 10.99.99.1:1077 TIME_WAIT

UDP 0.0.0.0:135 *:*

UDP 0.0.0.0:445 *:*

UDP 0.0.0.0:1087 *:*

UDP 10.99.99.1:53 *:*

UDP 10.99.99.1:137 *:*

UDP 10.99.99.1:138 *:*

Using the Windows Services control panel, verify that all appropriate services are runningfor that server. To view the Service control panel, go to Start and select Run. In the Runwindow, type services.msc and click OK.

Check the Application and System logs in the Windows Event Viewer for warning or errorevents. To start the Event Viewer, click Start and select Run. In the Run window, typeeventvwr.exe and click OK. This launches the Event Viewer console, which displays thethree logging options (Application, Security, and System).

A device may be in some state other than Up, such as Down, or Not Reachable. Check theOperations Console to verify the status of a particular device. To check the status:

From the Operations Console, select System > Control Center.

Click the Device Pool tab in the left frame of the page.


9



In the right frame of the page, click the General tab if it is not already selected.

Locate the device you are interested in and check the status of that device in the Statuscolumn.

Check the call statistics from the Operations Console to try to determine any abnormalities.The reported values from the various components might provide some information that willassist you in isolating the issue.

Note: The Operations Console online help describes the device status fields in detail. Fromthe console, choose System > Control Center and then select the Device Pool tab and choosea device pool. You can also view a particular type of device by selecting the Device Typetab and choosing a device type. The status of some devices such as, IOS devices, UnifiedCM, ICM servers, and SIP proxy servers display as N/A (Not Applicable) because theOperations Console does not monitor these device types.

You can access call statistics either by:

Choosing System > Control Center, selecting a Call Server, clicking Statistics icon inthe toolbar, and then selecting the various tabs (Unified ICME, SIP, IVR, or Infrastructure).

Choosing Device Management > CVP Call Server, selecting a Call Server, clicking theStatistics icon in the toolbar, and then selecting the various tabs (Unified ICME, SIP,IVR, or Infrastructure).

The examples above list the procedure for viewing the statistics for a Call Server. The statisticsfor other devices, such as the Reporting Server, VXML Server, and so forth can be viewedin the same manner.

Below are some of the initial areas to check in the Infrastructure tab for abnormal values orthresholds.

Check for licensing issues in the Infrastructure tab. The Licensing Statistics column isbroken up into Realtime, Interval, and Aggregate sections. Check the value being reportedfor Port Licenses Available against the value of Maximum Port Licenses Used.

Check the level of the Current Port Licenses State. This reports on the threshold level ofport license usage. There are four levels: safe, warning, critical and failure. An administratormay set the required percentage of port licenses in use needed to reach a given thresholdlevel, with the exception of the failure level which is reached when the number of portschecked out is equal to the total number of licenses ports.

Check the Uptime statistic, as well as the memory and thread pool statistics for possibleissues. If the Uptime statistic is low, then that indicates that the component was recentlyunavailable for some period of time. High memory or thread pool statistics may indicatea low available resource issue.

Below are some of the initial areas to check in the SIP tab for abnormal values or thresholds.


10



In the Interval Statistics section, check the value of the Failed SIP Transfers (Pre-Dialog).This is the total number of failed transfers on the first Unified CVP transfer in the laststatistics aggregation interval. A SIP dialog is established after the first Unified CVPtransfer is completed. The metric does not include rejections due to the SIP service beingout of service. The metric includes failed transfers that were made after a label was returnedfrom the Unified ICME in a CONNECT message.

Check the value of the Failed SIP Transfers (Post-Dialog). The total number of failedtransfers on second and subsequent Unified CVP transfers in the last statistics aggregationinterval. After SIP dialog is established, Unified CVP performs SIP transfers usingre-INVITE messages. The metric does not include rejections of SIP REFER transfers. Themetric includes failed transfers that were made after a label was returned from the UnifiedICME in a CONNECT message.

Below are some of the initial areas to check in the IVR tab for abnormal values or thresholds.

Check the Active Calls statistic. This is the number of active calls being serviced by theIVR service currently. Refresh the page to determine if the calls are increasing.

Check the Average, Maximum, and Minimum Call Latency statistics. These report theamount of time in milliseconds it took for the IVR Service to complete the processing ofa New Call Request or a Request Instruction Request during the time interval. Longdurations may indicate a network or load problem.

Also, verify that your network environment has been correctly configured. The table belowshows the speed and duplex settings for Unified CVP.

Server/Gateway NICSpeed/Duplex Setting

Switch Port Speed/DuplexSetting

Server/Gateway NIC SpeedCapability

Ethernet Switch SpeedCapability

Auto / AutoAuto / Auto1000 Mb1000 Mb

100 Mb / Full100 Mb / Full100 Mb1000 Mb



If you have exhausted all the common causes and actions including those outlined in thisdocument or others that you have identified in your environment, contact the Cisco TechnicalAssistance Center (TAC) or your Cisco Partner.

Troubleshooting Checklist

Complete this checklist to assist in isolating the issue, or to provide information to your supportpartner or Cisco Technical Support.

1. What is the version of Unified CVP that is currently running? Include any Hotfix or ServiceRelease (SR) information.

2. Is this a new installation or an upgrade?


11



3. If this is an upgrade, what version was previously installed?

4. When did the problem occur?

5. What are the observed symptoms, and the conditions under which these symptoms occur?

6. Was anything changed or updated in hardware, software, or network components prior tothe first occurrence of the observed symptoms?

7. Describe the related call flow. Some examples include: Public Switched Telephone Network(PSTN) originated or IP Phone originated. If it is a transfer, is it Unified CVP dequeue oragent originated? Is the call inbound ACD, inbound IVR Service treatment, inbound queuetreatment, outpulse transfer, or agent transfer?

8. Is the problem reproducible?

9. What Unified CVP deployment model used for deployment?

10. What is the call transfer method used?

11. Are you able to capture a screen shot of the error or failure? If Yes, save it to a file andattach to a case.

Network Topology

Complete this checklist to assist in isolating the issue, or to provide information to CiscoTechnical Support.

1. Has auto-negotiate been disabled on all PCs, routers, and switch ports?

Note: Duplex/speed mismatch between a device and its corresponding port on the switchis the single most common problem for network latency.

2. Is a network topology diagram available?

3. Which type of IP Gateway is being used in this Unified CVP solution?

4. Save the output from a show tech, show stack, and a show Gatekeeper endpoint fromthe Gateway.

5. Which type of Gatekeeper is being used in this Unified CVP solution?

6. On which server are the recorded media files located, and what is the path to those files?

7. Are Content Service Switches (CSS) used for failover?

8. Collect and provide versions of IOS, Digital Signal Processor (DSP) , applications, andEngineering Special (ES)/patch levels in the environment.


12



Symptoms and Solutions

Overview

This chapter contains problem descriptions and steps to diagnose and resolve each problem.Problem descriptions are grouped by Unified CVP component.

Call Server

This section contains problems associated with the Call Server.

Device Becomes Corrupted

Symptom:

A device has become corrupted. The user receives a specific error message that indicates whatdevice has been corrupted.

Message:

"Device with IP Address: {1} and Hostname: {2} is in an inconsistent state. Please Save andDeploy the properties of this Server again to fix the problem. If problem persists, please contactyour administrator for more details."

"Device with IP Address: {1} and Hostname: {2} is not ready at this time. Please retry theoperation at a later time."

"Device with IP Address: {1} and Hostname: {2} is in an inconsistent state and cannot beproperly deleted at this time. Please retry the operation later or try deleting again for forcefuldelete."


13

Chapter 2

Cause:

The most likely cause is that the Unified CVP software was reinstalled.

Action:

The following steps outline the process to resolve the issue of software being reinstalled on aserver.

1. Identify the affected devices from the Control Center of the Operations Console. Theaffected devices are:

- devices with the same IP addresses, and

- devices associated to the devices identified with the same IP address.

2. Go to the Edit page of each device and click Save&Deploy.

3. Restart all affected devices from Control Center in the Operations Console.

If the procedure above does not resolve the issue, then you must clean up the device byperforming the following steps.

Device Clean Up

1. Delete the device from the Operations Console.

2. Reinstall the software on that device.

3. Recreate the device from the Operations Console.

Each Operations Console Server will be identified by a unique identifier. The identifier isgenerated at the time of creating a first device through that Operations Console Server. Oncegenerated, the identifier will then be stored in the Operations Console Server database and inthe Resource Manager Configuration file on every device that includes the Operations ConsoleServer. The purpose of this unique identifier is to prevent management of a device from multipleOperations Consoles.

The unique identifier gets imported whenever the import functionality of an Operations Consoledatabase is invoked. Importing a database which was exported from a different OperationsConsole Server is not a supported feature unless the Operations Console is newly installed orhas never been used before.

If a user uses an Operations Console Server that is already managing at least one device toimport a database (from a different Operations Console Server or an empty database which wasnever used to create a device) the following side effects will occur:

User may not be able to manage devices which were managed by the same Operations Consoleserver prior to the import. This is because the unique identifiers will now be different betweenthe Operations Console Server and the previously managed device. User may not be able to


14

Chapter 2: Symptoms and Solutions

Call Server

create an IOS device such as Gatekeeper(s), Gateway(s), and CSS(s) from the OperationsConsole.

An Operations Console Server that is managing at least one device must not import an emptydatabase that was previously exported from any Operations Console Server. This will cause amismatch in the unique identifiers used since the Operations Console database and the OperationsConsole Server have a unique identifier and the empty database has no identifier. If a customerrequires replacing the existing database with an empty database, the recommended way is toreinstall the Operations Console Server software. If this step is not followed, the customer canno longer manage devices that were previously managed by the same Operations Console Server.

SSL Recovery

Symptom:

The Operations Console is unable to communicate with the Operations Console ResourceManager (ORM) service on a device.

Message:

Call Server device with IP Address: and Hostname: operation failed. Devicecould not be reached because of a failure in negotiating security certificates. Please refer todocumentation for configuring security before trying the operation again.

Cause:

If the user checks the secure communications checkbox in Operations Console deviceconfiguration without first configuring security certificates as outlined in Chapter 6 of theConfiguration and Administration Guide for Cisco Unified Customer Voice Portal, the OperationsConsole will not be able to communicate with the Operations Console Resource Manager onthe device.

Action:

The solution is to edit the orm_jmx.properties in the conf directory of the device. Change theproperty com.sun.management.jmxremote.ssl to false to put the device back intonon-secure mode. Next, uncheck the checkbox in the Operations Console and restart theOperations Console Resource Manager service on the device. This will put both the OperationsConsole and the device Resource Manager back into a non-secure mode. The user can thenconfigure security certificates before checking the secure communications checkbox.

Note: Changes in security settings require that the Operations Console service and the ResourceManager service be restarted.

Resource Manager Service Fails to Start

Symptom:


15


Call Server

Since the default log setting is ERROR for the Resource Manager Wrapper, the log file shouldprovide exception details of the failure. The wrapper log file does indicate there is a failure inResource Manager service startup, but the details of the event are not logged.

Message:

None.

Cause:

The wrapper log was set to the wrong level.

Action:

The change is unable to be made from Support Tools, so the change requires manual steps tochange this setting. The log file to look for the exception is%CVP_HOME%\logs\orm_wrapper.log.

1. Stop the 'Cisco CVP Resource Manager' service.

2. Edit %CVP_HOME%\conf\wrapper.conf file to change the property'wrapper.logfile.loglevel' value from 'ERROR' to 'INFO'.

3. Save and close the file.

4. Start the 'Cisco CVP Resource Manager' service.

If the Resource Manager service fails to start, the ORM logs (orm_wrapper.log) collected fromSupport Tools will show the detailed information of the event.

Call Server Is Not Reachable

Symptom:

After a forced delete of the Reporting Server, the Call Server state did not change from theDown state to the Partial or Up state.

Message:

Opsconsole Control Center: Call Server status is down.

Cause:

The Call Server messageAdapter.properties file is corrupt.

Action:

1. From the command line, run %CVP_HOME%\bin\tac\reimage.bat on the Call Server.

2. Restart the Operations Console Resource Manager (ORM) service.


16


Call Server

3. Log in to the Operations Console, and choose Device Management > CVP Call Server.

4. Click Save & Deploy to save the changes and apply them to the Call Server.

Note: All co-located devicesincluding VXML Server, Reporting Server, and VideoMedia Servermust be deployed after reimage.bat is run. You must repeat these stepsfor each co-located device.

Note: Optionally, you can reinstall the CVP device.

Call Server Cannot Communicate with Co-located Reporting Server

Symptom:

The environment:

Call Server 1 (primary) & Reporting Server 1.

Call Server 2 (secondary)

In the above deployment, when the Operations Console Resource Manager (ORM) on CallServer 1 is down, the ORM on Call Server 2 is unable to communicate with the ReportingServer.

Message:

None.

Cause:

This is not a supported model for deployment.

Action:

A co-located Reporting Server can never scale to support multiple Call Servers and must notbe configured in that manner.

Call Server or VoiceXML Service Cannot Be Restarted

Symptom:

After applying license files, the Call Server service or the VoiceXML service are unable to berestarted. Both services can be stopped, but receive a warning message when trying to restartthem.

Message:


17


Call Server

Windows could not start the Unified CVP Call Server/VXML Server on Local Computer. Formore information, review the System Event Log. If this is a non-Microsoft service, contact theservice vendor, and refer to service-specific error code 0.

Cause:

The problem is due to the side effect of the first version (1.0) of the MS06-040 patch (KB921883):

Action:

Apply the new version of the Microsoft security patch KB921883.

See Also

Refer to the Microsoft Security Bulletin KB921883 that is available on the Microsoft web site.Go to: http://support.microsoft.com/

Connection with PIM Dropped, Call Server Going Out of Service

Symptom:

The TCP connection between ICM VRU PG and CVP CallServer ICM subsystem was dropped.

Calls are dropped during load conditions.

Wireshark can not capture H.323 messages when a call processing load is running on theCall Server.

Message:

The following errors are written to the CVP error log file when either the connection betweenthe VRU PIM and CVP drops for an unknown reason, or the VRU PIM is intentionally shutdown for maintenance:

LOGMSG_ICM_SS_PIM_SHUTDOWN: Lost socket connection to VRU PIM. Transitioningto partial service.

LOGMSG_ICM_SS_STATE: Waiting for VRU PIM Connection.

Cause:

The TCP Chimney feature is an offload engine that was introduced in Windows Server 2003Scalable networking Pack.

Action:

Disable the TCP Chimney feature.


18


Call Server

http://support.microsoft.com/

1. Open Registry Editor, and locateHKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters

2. Set the EnableTCPChimney parameter to 0 .

3. Reboot the machine (required).

H.323 Service Cannot Access Call Server

Symptom:

Some HTTP requests from the Unified CVP Voice Browser to the Call Server are being blocked.

Message:

Blocked by port blocking rule VoiceBrowser.exe Prevent IRC communication.

Blocked by port blocking rule tomcat5.exe Prevent IRC communication.

Cause:

VirusScan is blocking ports in the range of 6666-6669, which are used in IRC communications.The application does not have to be listening on these ports for it to be blocked. If the applicationis assigned a client port in that range, the attempt to connect to a server using that port is blockedby VirusScan on the server machine. The connect attempt fails.

Action:

Check the VirusScan logs for error messages. access the logs,VirusScan Console -> Task ->View Log You might find entries like these: 10/10/2006 3:15:01 AM Blocked by port blockingrule VoiceBrowser.exe Prevent IRC communication 10.86.129.138 10/10/2006 8:49:01 PMBlocked by port blocking rule tomcat5.exe Prevent IRC communication 10.86.129.211.

To view VirusScan logs

1. Go to Start > Programs > Network Associates > VirusScan Console.

2. Click on the Task menu and select View Log.

You might find entries such as:

Blocked by port blocking rule VoiceBrowser.exe Prevent IRC communication.

Blocked by port blocking rule tomcat5.exe Prevent IRC communication.

INFO: Retrying communication with Call Server localhost:8000

INFO: Retrying communication with Call Server localhost:8000

Voice Browser to Call Server localhost:8000 connection unsuccessful.


19


Call Server

Voice Browser cannot access any Call Servers and is no longer accepting calls.

Voice Browser is out of service.

Exclude the Unified CVP processes from port blocking when using McAfee VirusScan on asystem running Unified CVP Call Server. If a different anti-virus product is used, do theequivalent exclusions for the port blocking rules for that product.

Instructions for McAfee VirusScan on Unified CVP Call Server.

1. Open the VirusScan Console.

2. Click Access Protection.

3. Click Properties under the Task menu.

4. Click the Port Blocking tab.

5. Check both Prevent IRC Communication rules.

6. Click the first Prevent IRC Communication rule and click Edit.

7. Enter VoiceBrowser.exe,tomcat5.exe to the Excluded Process field.

8. Click OK.

9. Click the second Prevent IRC Communication rule and click Edit.

10. Enter VoiceBrowser.exe,tomcat5.exe to the Excluded Process field.

11. Click OK.

Resetting System Clock Stops IVR Service Requests

Symptom:

Resetting the system clock on the Call Server causes the IVR Service to stop functioning.

Message:

None

Cause:

Changing the time of the system clock on the Unified CVP Call Server causes the IVR Serviceto stop accepting calls.

Action:


20


Call Server

Do not reset the Windows system clock on a machine running Unified CVP. Resetting theWindows system clock is not supported on a Call Server.

IVR Service CALL_RESULT Returns Error Code Other than 0

Symptom:

The IVR Service sends a CALL_RESULT error code other than a zero (0) to the Call Serverlog.

Message:

Below is an example of one of the messages:

4765560: 10.86.129.211: Aug 18 2007 06:09:59:484 -0400:%TEMP:CVP7.0_IVR-7-TRACE_CALL: {Thrd=http-8000-3} VBServlet:service: Requestfrom 10.86.129.20: {CALL_ID=979679B9-2DD811DB-B72D0014-6944B762,MSG_TYPE=CALL_RESULT, CALL_SEQ_NUM=2, ERROR_CODE=32}

Cause:

The problem is due to an error with the Run Script Request from Unified ICME.

Action:

Check the Error Code that was received in the Event Codes and Reasons appendix of thisdocument, and take the appropriate action. For additional information and a complete list of thecause codes, refer to the Cause Codes and Debug Values section of the Cisco IOS VoiceTroubleshooting and Monitoring Guide.

Call Server Is Unreachable

Symptom:

The Call Server is unreachable.

Message:

UNREACHABLE

Cause:

A configured component is unreachable in any one of the following three conditions:

Operations Console is unable to connect to the Operations Console Resource Manager thatis co-located with the Call Server. For example, the Operations Console Resource Manageris down.

The Operations Console Resource Manager on the Call Server has not received a state eventfrom the controller of that component.


21


Call Server

The Operations Console Resource Manager is unable to connect to the Call Server so it isnot receiving state events from the central controller. For example, the Call Server is down.

Action:

In this example, all three services (Unified ICME, IVR, and SIP) have been configured. Thecentral controller reports that IVR Service and SIP are IN_SERVICE, but it does not report thestatus of the Unified ICME Service to the Operations Console for some reason. When thisoccurs, Operations Console reports the status of Unified ICME as UNREACHABLE. OperationsConsole aggregates the status of the various components of the device to arrive at device status.It sees that two of the components (IVR Service and SIP) are in SERVICE, but Unified ICMEis UNREACHABLE. Operations Console shows the status of the Call Server asUNREACHABLE even though IVR Service and SIP are actually in service.

Check the network environment for connectivity, as well as the actual state of the server.

Received ERROR_SCRIPT_NAME = 6 on IVR Service

Symptom:

The ERROR_SCRIPT_NAME = 6 error code is received on the IVR Service.

Message:

ERROR_SCRIPT_NAME = 6

Cause:

The VRU Script Name data that is passed from the Unified ICME to the IVR Service does notcontain the expected components. These components are micro-application name, media filename, and media file type.

Action:

In the Unified ICME Network VRU Script List tool, locate the appropriate script and resolvethe configuration error.

See Also

For more information, refer to the Micro-Application configuration section of the Unified CVPConfiguration and Administration Guide.

Invalid Path or File Name

Symptom:

Note: This is a Call Server issue that is found within the Video Media Server (VMS), which isa part of the Call Server.


22


Call Server

Sometimes, a file is read from the VMS movies directory with a name that does not validate tothe standard accepted character set. These files are marked with the "File could not beread" status.

Message:

File could not properly be read from the file system.

Cause 1:

An invalid status for a media file stating that the "file could not be read", generallyimplies that the file has an invalid path or file name.

Action 1:

Verify and correct the path or file name.

Cause 2:

An invalid status for a video media file (that is a 3gp or an XML file) stating that the "filecould not be properly read", generally implies that the file has an invalid file name orpresents within an invalid path.

Action 2:

Verify and correct the path or file name.

The file names can contain alphanumeric characters, underscore, spaces, and the followingspecial characters: !$&().,'-;@[]^_`{}~+=. File names cannot lead with a period (.) or a space.

Only path names with alphabetical, numeric, spaces, and !$&().,'-;@[]^_`{}~+= characters arepermitted; path names cannot lead or end with a period (.) or a space.

Dialogue Failure Event Error in the Call Server Log Files

Symptom:

Callers hear critical error media and Dialogue Failure Event errors appear in the Call Serverlog files.

Message:

None.

Cause:

If a call reaches the end of ICM script processing without being queued or released, the UnifiedICME sends a Dialogue Failure Event message to the Call Server. The Call Server, in turn,sends an error message to the H.323 Service.

Action:


23


Call Server

The author of the script must ensure that each path in the script ends with one of the followingScript Editor nodes: Release, Label, Skill Group, or Queue to Skill Group.

Call Server Variable Data Is Invalid Error

Symptom:

The Call Server is reporting an error.

Message:

Variable Data is Invalid.

Cause:

The ICM script might truncate values with leading zeros or zeros after decimals or roundsvalues.

Action:

Put quotes around numbers in a Script Editor Set node so it is processed as a string. This isespecially important if:

Leading zeros are present (example: dates).

Trailing zeros are present after a decimal point (example: currency).

The number is very large (example: a number normally expressed through exponentialnotation).

Unable Add IOS Device Because Device Is Already Managed

Symptom:

When attempting to add a device to the Operations Console, an error message is received.

Message:

Gateway device with IP Address: and Hostname: cannot be created.The device is already managed by another Operations Console.

Cause:

This occurred because a database was imported from a different Operations Console. EachOperations Console Server is identified by a unique identifier. The identifier is generated at thetime of creating a first device through that Operations Console Server. Once generated, theidentifier will then be stored in the Operations Console Server database and in the ResourceManager Configuration file in every device that includes the Operations Console Server. The


24


Call Server

purpose of this unique identifier is to prevent multiple Operations Console Servers fromconfiguring the same device.

The side effects of importing a database to an Operations Console Server that is managing atleast one device, if the database to be imported is an exported database from a different OperationsConsole Server or is an empty database which was never used to create a device are:

1. User may not be able to manage devices which were managed by the same OperationsConsole server prior to the import.

2. User may not be able to create an IOS device such as Gatekeeper(s), Gateway(s), andCSS(s) from the Operations Console.

Importing a database which was exported from a different Operations Console Server is not asupported feature. In this case the import may succeed but will have various side effects includingthe above two side effects.

Action:

Importing a database which was exported from a different Operations Console Server is not asupported feature. The solution is to never import an empty database to an Operations ConsoleServer, if the server is managing at least one device prior to import. If customers require replacingthe existing database with an empty database, the customer is to reinstall the Operations ConsoleServer software. Doing so, the customer can no longer manage devices that were previouslymanaged by the same Operations Console Server.

VRU Application Error in Call Server Log

Symptom:

The call might not go through the Call Server, so a prompt is not played. An error or warningmessage might appear in the Call Server log.

Message:

VRU APPLICATION ERROR: Assigning the tag 7 to the non-existing ECC variableuser.microapp.error_code.

Cause:

The ECC Variable is not configured on the Unified ICME and/or NAM software, or the definedlength is not the same on both devices.

Action:

Add the identical ECC variable definition to the Unified ICME and/or NAM.


25


Call Server

ECC Variable Contents Truncated after Passing through Call Server

Symptom:

The contents of an ECC Variable configured in Script Editor are truncated after passing throughthe Call Server. For example, the Text-To-Speech (TTS) text that is spoken to the caller is notthe complete text that was configured. Another example of this behavior is where there aremedia fetch failures and the URL to the media file is only a subset of the expected URL.

Message:

None.

Cause:

The cause of this problem may be that the length of the ECC Variable value that was set inScript Editor is longer than the maximum length of the ECC Variable configured at setup time.

Action:

The solution is to make the maximum length of the ECC Variables longer. This is done usingthe Unified ICME Configuration Manager. If you do alter the maximum length of ECC Variables,you need to restart the Call Server after making the change.

Note: In a NAM/ICM environment, the length needs to be identical on all NAM and UnifiedICME environments or the variable will not pass.

"Have You Configured Your Call Server?" Error

Symptom:

The Call Server is running, but the an error message is logged in the log file.

Message:

Have you configured your Call Server?

Cause:

The Call Server was started from the Services Control Panel, but it was not configured fromthe Operations Console.

Action:

Configure the Call Server from the Operations Console. After completing the configuration,restart the Call Server from the Services Control Panel.


26


Call Server

Erratic Behavior When Everything Is Configured Properly

Symptom:

General erratic behavior when everything is configured properly. Sometimes TTS plays,sometimes it does not, or the call stops when the VXML Server is running in script node inUnified ICME. Other behaviors include calls succeed in Unified ICME, but no TTS plays, orthe caller hears silence, but no errors are reported in the log. Unavailable TTS errors are reported.

Message:

None.

Cause:

If everything appears to be configured properly, and behavior is generally erratic, check theconfiguration of your Ethernet link.

Action:

Do the following:

Check the Gateway log files.

Check the VXML Server application logs.

Verify the network settings. The chart below shows the speed and duplex settings for UnifiedCVP.

Table 1: Speed and Duplex Settings

Server/Gateway NICSpeed/Duplex Setting

Switch Port Speed/DuplexSetting

Server/Gateway NIC SpeedCapability

Ethernet Switch SpeedCapability

Auto / AutoAuto / Auto1000 Mb1000 Mb




IVR Subsystem Goes into Disabled State When Call Server Service Starts

Symptom:

When the Call Server service is started, the IVR subsystem goes into a Disabled state.

Message:

Cause 1:


27


Call Server

The Unified CVP license in use is a non-expiring license (no expiration date).

Action 1:

Use a Unified CVP license that has an expiration date

Cause 2:

If the Unified CVP license in use has an expiration date, the Unified CVP Call Server serviceneed to be recycled.

Action 2:

Recycle the Unified CVP Call Server Service. After recycling, check to see if the IVR subsystemis not in a Disabled state. If it is disabled, the process needs to be repeated a few times.

IVR Subsystem Returns a 500 Rejection on VXML Fetch from Gateway

Symptom:

IVR Subsystem returns a 500 rejection on VXML fetch from the gateway as seen in the gatewayor CVP logs.

Message:

Cause:

Incorrect setting of parameter "Max Length of DNIS" on ICM service configuration page ofOperations Console.

Action:

Set parameter to the same length as the DNIS part of the label returned by ICM.

Full Video Call Disconnected When Routed Through IVP

Symptom:

Full video call is disconnected when it is routed through IVP.

Message:

Cause:

Access number DN prefixes are not added to the iCONTACT application and/or the iCONTACTxml configuration.

Action:


28


Call Server

See Chapter 17 in the Configuration and Administration Guide for Cisco Unified CustomerVoice Portalfor information on modifying the Access Number in the iCONTACT xml file.

SIP Code 488 or 603 Returned on Full Video Call Rejection

Symptom:

SIP code 488 or 603 is returned on a full video call rejection from IVP. A Fast Busy is heard.

Message:

Cause:

MCU or MVP/EMP unit on IVP is not registering or is in service.

Action:

Follow the steps in the Radvision documentation to place the MCU or MVP in service.

Full Video Warm Transfer with Queuing Not Working

Symptom:

SIP trunk on outgoing call to VRU label is not getting routed to IVP.

Message:

Cause:

Action:

Set the SIP trunk on the route pattern for the ICME VRU label to point to IVP instead of CVP.

CVP Warm Transfer With Queuing Get Disconnected

Symptom:

CVP warm transfer with queuing gets disconnected after a few seconds when the transfer iscomplete. CUCM logs and/or sniffer trace show a reinvitation collision scenario. This onlyoccurs with IP originated callers, not TDM callers.

Message:

Cause:

This problem is visible when CVP and CUPS are using proxy and UDP on the outgoing direction,and the CUCM SIP trunk is using TCP on the outgoing direction.

Action:


29


Call Server

Set UDP on the CUCM SIP trunk security profile or set CVP and CUPS to use TCP.

Hold Movie not Visible to Caller on Hold

Symptom:

Note: This tip applies to full video only.

The caller does not see a hold movie when agent places the caller on hold.

Message:

Cause 1:

Mis-configuration of ECC variables that control the hold movie to be played to caller.

Action 1:

Make sure the user.microapp.media_server and user.cvpmovies_bg_media ECC variable areconfigured correctly via the "Set Variable" node in the ICM script: for example,user.microapp.media_server = "rtsp://10.86.129.233" user.cvpmovies_bg_media="app/hold.3gp"

Cause 2:

The default hold movie is not configured in the/home/rv/ivp/applications/iCONTACT/conf/iCONTACTConfiguration.xml file.

Action 2:

Verify if a default hold movie is configured in the/home/rv/ivp/applications/iCONTACT/conf/iCONTACTConfiguration.xml file as follows:

agent

rtsp:///en-us/app/hold.3gp

Specify the RTSP URL (as shown above) or as a file:// URL (if file stored locally on IVP

box). (e.g. file:///home/rv/movies/local/hold.3gp). 1

Full Video Calls Drop upon Dialing the DNIS

Symptom:

1) A Linux server in which the IVP software from Radvision is loaded is refered to as an IVP system. IVP systems are used as part of the Unified CVP Release7.x solution.


30


Call Server

Upon dialing the Dialed Number Identification Service (DNIS), full video calls are immediatelydropped.

Message:

none

Cause 1:

The connection between Unified CVP and the Interactive Video Platform (IVP) system fromRadvision (used as part of the Unified CVP solution) may not be established.

Action 1:

Verify the following requirements:

The Call Server status is functioning (up) in the OAMP control center.

A valid connection exists between Unified CVP and IVP via Diag Servlet. To verify, clickon Dump IVR subsystem and look for the following ContactClient statements:

ContactClientManager: video enabled: true

ContactClient count: 1

ContactClient [192.168.150.170]:9981, connectionId:iCONTACT-Connection-1

Look for following log message in CVP log: 2472: 192.168.150.173: Sep 24 200711:35:25.500 -0400: %CVP_7_0_IVR-1-VIDEO_CONNECTION_UP: Connection toiContact server [192.168.150.170]:9981 is UP [id:3100]

Cause 2:

The access numbers and/or prefix numbers have not been configured for iCONTACT in IVPManager and/or iCONTACT configuration file.

Action 2:

1. Configure the access numbers and/or prefix numbers for iCONTACT using IVP Manager.

2. Configure the same access numbers and/or prefix numbers in the iCONTACT configurationfile.

3. Restart IVP after making these changes.

Undecipherable Message Is Displayed When the VMS Subsystem Is Down

Symptom:


31


Call Server

When playing a video to a caller, the DarwinStreamingServer.exe is terminated. The video stopsplaying and an error message is recorded in the VMS Error Log.

Message:

340: 10.30.30.97: Oct 25 2007 16:59:46.267 -0400:%CVP_7_0_VMS-3-VMS_SUBSYSTEM_STREAMING_SERVER_DOWN: The StreamingServer State: UNKNOWN. [id:11512]

Cause:

Note: There are three internal states outputted by the VMS Darwin Streaming Server keep-alivethread which help drive the overall state of the VMS. These are internal states, not necessarilyrepresentative of the state of the VMS as a whole.

The appropriate error message for an agent to see when VMS detects Darwin is down is "VMSSubsystem is down". You may see a different error message if the streaming server polleris in between cycles when the VMS Subsystem goes down. This error message is accurate asit is the output from the keep-alive thread, which tracks the VMS Subsystem through threestates: IN_SERVICE, PARTIAL_SERVICE and UNKNOWN. These are not the states that theVMS Subsystem uses.

Action:

To see the appropriate error message, wait for one full polling cycle (default: 30 sec) to ensurethat the proper error message appears.

Connection between Unified CVP and IVP cannot be established when setting up TLS

Symptom:


When the Transport Layer Security (TLS) is being set up, the connection cannot be establishedbetween Unified CVP and IVP

Message:

SSL session handshake failed (is the server SSL enabled?).

Log messages constantly show connection being made and then broken.

Cause 1:

The TLS setting is enabled on the Call Server but not in iContact. Enable TLS setting on CallServer and IVP are not in sync.

Action 1:


32


Call Server

Verify that both Call Server and iContact are configured for TLS. Call Server configuration isenabled via OAMP. iContact configuration is via iContactConfiguration.xml. Both Call Serverand IVP must be restarted if their settings for enabling/disabling TLS are changed.

Cause 2:

The required procedures to import/export keys between Call Server and IVP has not completedor is not accurate.

Action 2:

Verify that the proper keys are imported and exported. Refer to the TLS configurationdocumentation for procedural Call Server/IVP information.

Only One Call Server Is Receiving Calls

Symptom:


When multiple Call Servers are connected to a single iCONTACT, only one of the Call Serversis receiving calls.

Message:

None

Cause:

iCONTACT configuration may not be set to distribute calls to Call Servers via round-robinmechanism.

Action:

Open the iCONTACT configuration file and verify that the notification type is set to roundrobin i.e Round-robin.

Calls Are Disconnecting

Symptom:

There are a number of abnormally disconnected calls.

Message:

Calls are disconnecting.

Cause:


33


Call Server

The Radvision IVP Linux interface, as well as Radvision MVP/MCU, does not autonegotiateinterface link speed/duplex mode properly.

Action:

On the Radvision IVP Linux server, use the ethtool utility to manually set the current interfacelink speed (MB) and duplex mode.

1. At the command prompt type, ethtool -s eth0 speed 100 duplex full

2. To permanently change the interface link speed/duplex mode settings (that is, to survivea reboot), in the /etc/sysconfig/network-scripts directory add the following line to theifcfg-eth0 file:

ETHTOOL_OPTS="speed 100 duplex full autoneg off "

Non-English Video Text Overlay is Garbled in Full Video

Symptom:

Message:

Cause 1:

The "user.micropp.charset" ECC variable is not set correctly for the target language.

Action 1:

Verify that the correct value is set in the "user.microapp.charset" ECC variable for the targetlanguage in the ICM script. The supported values values are specified in the Configuration andAdministration Guide for Cisco Unified Customer Voice Portal.

Cause 2:

The Radvision MCU is not configured for the target language.

Action 2:

Upgrade the Radvision MCU to the target language so that the proper fonts are loaded. If theMCU is already configured for the target language, restart the MCU and EMP

Movie Media of Video Micro-Application Cannot be Interrupted

Symptom:


Movie media of video micro-app cannot be interrupted by a DTMF barge-in or when the agentbecomes available.


34


Call Server

Message:

None.

Cause:

This can happen if the micro-application movie media only has an audio-stream and does nothave any video stream.

Action:

In the Network VRU script micro-app configuration, change the Show/Display option in theConfiguration to A for audio-only.

Media Poller timing issues

Symptom:

Metadata files are modified from within the administrative web UI or from within the file systemwhile the media poller is polling in the background.

Message:

none

Cause:

The media poller feature is used to automatically add new video resources from the file system,but can result in some rare timing issues.

The media poller begins its polling and cycles through the c:\foo directory. At this time, if anadministrator adds and saves a new file c:\foo\bar.xml using the administrative web UI, theadministrator (and possibly agents) see bar.xml displaying in their web UI. At this point themedia poller cycle ends as c:\foo\bar.xml was not present when that directory was polled. Thisis because c:\foo\bar.xml is considered to be deleted by the media poller and is removed frommemory. Subsequent refreshes of the administrative or agent web UIs show c:\foo\bar.xml asmissing. The following media poller cycle picks up c:\foo\bar.xml more permanently.

Action:

This is a subtle, timing issue. In the end, the media poller performs its function and gets backto synchronizing with the file system after another polling cycle. The VMS media poller processwill never delete or modify existing files on the system. It will only poll for their existence andcurrent status.

Invalid Thumbnail Images on VMS Server

Symptom:

Invalid images in the VMS movie library causing unintended consequences on the VMS system.


35


Call Server

Message:

none

Cause:

none

Action:

Be sure to adhere to the following guidelines for thumbnail images on the VMS system. Validthumbnail image pixel size is 120 x 80 (width x height).Thumbnail images must have the samename as the XML file, with the exception of the file extension. Supported file extensions are:.jpg, .gif, and .png.

Note: Large images use more network bandwidth and will impact Video Media Serverperformance when they are displayed to agents. Images that are larger or smaller than120 x 80will be resized for display only. The content on the image itself does not change on the filesystem.

CALL Server Cannot Connect to IVP Server

Symptom:


Call Server cannot get a connection to the IVP server.

Message:

The following is seen in the iCONTACT log: 2007-11-16 18:41:09,320 (iContactStartupHandler)- Can not start iContact application server. The following is seen in the Call Server logCVP_7_0_Infrastructure-3-SOCKET_SETUP: Failed getting socket for host: 10.86.129.104port: 9981 Exception: java.net.ConnectException: Connection refused: connect

Cause:

The most likely cause is that the iCONTACT XML configuration file has an XML syntax error.

Action:

Fix any XML syntax errors in the iCONTACT configuration file and restart the IVP serverusing the IVP Manager.

VMS Administrator Replaces Video Content While Agent is Previewing

Symptom:

Note: An administrator should be aware that deleting/replacing/moving video files while callsare being served can result in unpredictable behavior. VMS is limited in that it cannot control


36


Call Server

how the Darwin Streaming Server or the IVP server will react to file changes. However, theresults are very minor. At worst, the currently playing video stops, which is not fatal to thesystem.

VMS Administrator replaces a .3gp video file with new content, but retains the same filenamewhile at the same time a VMS Agent is Previewing the initial video. The video displaying inthe VMS Agent QuickTime window freezes.

Message:

none

Cause:

none

Action:

VMS Agent must close the QuickTime video window, reselect the video, and then Preview.The new video content will now be available.

VMS Administrator Replaces Video Content While Agent is Pushing Initial Video to Caller

Symptom:

Note: An administrator should be aware that deleting/replacing/moving video files while callsare being served can result in unpredictable behavior. VMS is limited in that it cannot controlhow the Darwin Streaming Server or the IVP server will react to file changes. However, theresults are very minor. At worst, the currently playing video stops, which is not fatal to thesystem.

VMS Administrator replaces a .3gp video file w\new content, but retains the same filenamewhile a VMS Agent is Pushing the initial video to a Caller. The video displaying in the Callerswindow stops.

Message:

none

Cause:

none

Action:

VMS Agent must reselect the video and then Push it to the Caller. The new video content willdisplay to the Caller.


37


Call Server

Unable to Process Playback Request

Symptom:

An agent may see this error in the Cisco Unified CVP Video Desktop if the agent attempts aplayback request after a call has ended. However, if the agent sees this during an active call, itmeans that the Video Desktop is still trying to send requests for a previous call.

Message:

Unable to process playback request. The call has been lost.

Cause:

Action:

To be able to send playback requests for the current call:

1. Close the Video Desktop browser window.

2. Click the Video Desktop button in CTIOS to open the Video Desktop with the new callinformation.

3. Playback requests should now work correctly.

VXML Server

Ended Session Request Error

Symptom:

This error message appears when an HTTP request initiated by the root document is made bythe browser for a session that the VXML Server has already ended. The way the VXML Serverknows what session the request is asking for is by looking at the cookie included in the requestfor the sessions ID. A request from the root document indicates either a VXML error occurredor the caller hung up. In a typical call, all requests made for a call session come in series andso there is never a request that comes after the VXML Server believes the call has ended. Thiserror message would appear if a request telling the VXML Server to end the session is followedby a new request initiated by the root document referencing the same cookie.

Message:

SERVER ERROR: There has been an error with the http session: a request was made due to anevent caught in the root document that refers to a session that has already ended.

Cause 1:


38


VXML Server

This is a very rare situation as normal operation would not cause any further requests to be madeafter the request that ended the session (it is expected that the session-ending request is the lastone). Note that for this error to appear, the new request must come from the root document, inmany cases indicating an error encountered by the H.323 Service. There could be several raresituations where this could occur. The first begins with the end of call event activated by thefinal request to VXML Server taking a long time to run. This would cause the browser to makea new request timing out the session. This next request would have to be made before the initialrequest returned, but after the end of call event completed, causing the VXML Server to endthe session before the new request was received. In order for this situation to occur, the timingwould have to be very precise.

Action 1:

For the first situation, the developer must ensure that their end of call events do not have thepossibility of taking longer to execute than the H.323 Services fetch timeout VoiceXMLproperty. Another possible solution if the end of call event is a Java class would be to executethe class in a separately spawned thread. The problem with this would be that the spawnedthread should not attempt to access content in the session due to the possibility that it could beinvalidated while the thread was executing.

Cause 2:

Another situation that could cause this would be some network overhead or slowdown causingthe final response for a call to fail to arrive at the browser, precipitating a time out to occur.Since VXML Server functioned without problem and ended the session, this new request wouldcause the error to occur. Note that some Call Servers and operating systems under high HTTPload can queue HTTP requests and potentially even lose requests if the queue is too large. Theadministrator should investigate this queue if their system is expected to support high load (forexample, on Apache Tomcat, this is known as acceptcount).

Action 2:

For the second situation, it may be an indication of excessive network, operating system, orCall Server load. For the network, perform diagnostics to ensure proper operation. For theoperating system, check the CPU and memory utilization to ensure they are not maxed out. Andfor the Call Server, check the configuration to ensure the HTTP request queue is not too small(for example, on Apache Tomcat, the acceptcount setting can be configured in the conf/server.xmlconfiguration file). Finally, the administrator may decide to direct less load on the system.

The administrator may also choose to ignore these issues if they occur rarely since there are nobad consequences for the call session that encounters this error.

Severity: Due to the fact that this error message appears only in very rare situations, as well asthe fact that even if it does occur, the caller does not encounter any issues with their call, thiserror message is benign.

Ending Session Request Error

Symptom:


39


VXML Server

This error message appears when an HTTP request initiated by the root document is made bythe browser for a session that VXML Server has started the process of ending but that processis not complete. This is similar to the previous error, but where in that situation the request wasmade after the session had completed, this situation involves the request occurring while theVXML Server was ending the session but before it completed.

Message:

SERVER ERROR: An HTTP request was made due to an event caught in the root documentto a session that has been slated for invalidation.

Cause:

One very rare situation that could cause this error to occur under standard call volume involvesa perfect timing of events. This error would not be produced if the extra request arrives whilean end of call event was executing. However if it did take a long time to execute, promptingthe browser to time out the request by having the root document make a new request, and thatrequest came at exactly the time the end of call event completed but before VXML Server wasable to finish the session ending process, this error would occur. Since this process normallyoccurs rapidly, the chances that this would occur during this period is very small.

This error would also be encountered if the network, operating system, or Call Server on whichVXML Server is installed is sufficiently overloaded such that either HTTP requests are beingqueued up too much, HTTP requests are being dropped or the Call Server is so overloaded thatthe normally trivial process of ending a session takes a non-trivial amount of time.

Severity: The severity for this error is very low because there is no effect on the caller sincethis occurs after the call is complete. Additionally, the situations in which this can occur duringnormal operation are extremely rare.

Action:

For the first situation, the developer must ensure that their end of call events do not have thepossibility of taking longer to execute than the H.323 Services fetch timeout VoiceXMLproperty. Another possible solution if the end of call event is a Java class would be to executethe class in a separately spawned thread. As long as the end of call class completed its executionbefore the session invalidation delay specified in the conf/global_conf.xml of the installationdirectory, there would be no risk of the class referring to data within the session after it wasinvalidated.

For the second situation, it may be an indication of excessive network, operating system, orCall Server load. For the network, perform diagnostics to ensure proper operation. For theoperating system, check the CPU and memory utilization to ensure they are not maxed out. Andfor the Call Server, check the configuration to ensure the HTTP request queue is not too small(for example, on Apache Tomcat, the acceptcount setting can be configured in the conf/server.xmlconfiguration file). Finally, the administrator may decide to direct less load on the system.

The administrator may also choose to ignore these issues if they occur rarely since there are nobad consequences for the call session that encounters this error.


40


VXML Server

Session Does Not Contain Call Information

Symptom:

This is a rare error message that is displayed after end session request error. In this situation,VXML Server has completed its process of ending a session, however the Call Server has notcompleted its process for ending the session when a new HTTP request is received referencingthat session. This would yield a legitimate session from the Call Servers standpoint but aninvalid session by VXML Servers standpoint.

Message:

SERVER ERROR: The session is not associated with a Server Configuration.

Cause:

The situations where this would be encountered are identical to those listed in the previous errorcondition except the timing of the second request would occur after the standard end of callprocess taken by VXML Server completed but before the Call Server actually removed thesession. The timing for this situation is actually tighter than the previous situation and thereforewould occur even more rarely. While there is a chance that this could occur under standard callvolume, it would more likely occur only under extreme load to the network, operating system,or network.

Severity: The severity for this error is very low because there is no effect on the caller since

Documents

Troubleshooting Guide for Cisco Unified Customer Voice Portal