Upload
truongdien
View
444
Download
32
Embed Size (px)
Citation preview
Troubleshooting Cisco Unified Contact Center Enterprise BRKCCT-3005
Gino D’Addario
Technical Solutions Manager
Customer Solutions Success
2
‟Have you tried turning it off and on again?” • “The IT Crowd”
• Graham Linehan
‟We cannot solve our problems with the same thinking we used when we created them.”
• Albert Einstein
Troubleshooting
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
Agenda
Serviceability
– Performance Monitors
– SNMP
– Logging Improvements
– OPC Capture
– The Truth About Minidumps
– Network Optimization
– Updated Platform Support
– Diagnostic Framework
4
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
Agenda
Troubleshooting Flagship Features in UCCE 9.0
– CCE Web Administration – Tomcat 101
– Deployment Types
– Congestion Control
– Precision Routing
– Virtualization
Q&A
– Also available for Meet the Engineer
BONUS! Multimedia Integration
BONUS! Solution Approach to Troubleshooting
5
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
Performance Monitors
Performance counters are measurements of system state or activity. – Microsoft TechNet
Reactive Usage: UCCE Serviceability Guide section 8.3 recommends which PerfMon counters to enable when diagnosing issues.
Proactive Benefits: Counters can also be enabled for trending and predicting upcoming capacity/performance requirements.
Counters added in UCCE 8.0 provide richer visibility into system activity and health.
7
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
Performance Monitors
Platform Diagnostic Counters still (7.0) collected automatically by Node Manager. High-level system resources.
– 1 Minute Intervals / 1MB Per File / 45 Days Max
– Saved to \icm\log\Perf_MACHINENAME_YYYYMMDDHHMMSS.CSV
8
Component Counter Name Units Threshold
Processor %Processor Time (_Total) Percentage <50% | 50% - 60% | >60%
(sustained)
Process Handle Count (_Total) # Handles <=2 x [# CPUs] (continuous)
Memory Page Faults / sec # Faults <10 | >=10 | >10 (sustained)
Memory Committed Bytes # Bytes Relative to paging, demand for
virtual memory
Memory Pages / sec # Pages / sec <10 | >10 (continuous)
System Threads # Threads Reference
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
Performance Monitors
Component-specific Counters for Further Diagnosis
* = Useful for Capacity Planning. Available Prior to UCCE 8.0.
9
Cisco Router Object Counter Default State
Agents Logged On* On
Calls In Progress* On
Calls/sec* On
Calls In Router On
Router State Size Off
Messages Processed/sec Off
Bytes Processed/sec Off
Avg Process Time/Message (ms) Off
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
Performance Monitors
Component-specific Counters for Further Diagnosis
* = Useful for Capacity Planning. Available Prior to UCCE 8.0.
10
Cisco Router Object Counter Default State
Agents Logged On* On
Calls In Progress* On
Calls/sec* On
Calls In Router On
Router State Size Off
Messages Processed/sec Off
Bytes Processed/sec Off
Avg Process Time/Message (ms) Off
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
Performance Monitors
Other new objects available for
– PIM – EA & VRU
– CTI Server, CTI OS
– OO Campaign Mgr, Import Process
Reference the UCCE Serviceability Guide for more information http://www.cisco.com/en/US/products/sw/custcosw/ps1844/products_installation_and_configuration_guides_list.html
11
Administration & Data Server New Counters
Default State
CallType DB Write Records Processed On
Route Queue Depth On
Route DB Write Average Time On
Route DB Write Records Processed On
Service Queue Depth On
Service DB Write Average Time On
Agent Queue Depth On
…and many more!
OPC New Counters
Default State
Call Count On
Agent Count On
Skill Group Count On
Services Count On
Logged-In Agent Count On
Ready Agent Count On
Not-Ready Agent Count On
…and many more!
Logger New Counters
Default State
Number of DB Write Records On
DB Write Average Time On
DB Write Records Processed On
Outbound Option Dialer New Counters
Default State
Number of DB Write Records On
DB Write Average Time On
DB Write Records Processed On
…and many more!
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
Simple Network Management Protocol
SNMP is a family of specifications that provide a means for collecting network management data from the devices residing in a network. – Microsoft TechNet
SNMP Agent installed on “managed devices” can report problems via “traps” to the Network Management System – and you(r team)!
Subagent Management Information Bases (MIBs) Installed:
– CISCO-CONTACT-CENTER-APPS-MIB
– CISCO-CVP-MIB
Reference the UCCE Serviceability Guide for Install, Config, Notifications: http://www.cisco.com/en/US/products/sw/custcosw/ps1844/products_installation_and_configuration_guides_list.html
12
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
Simple Network Management Protocol
UCCE SNMP Event Flow – High Level
13
SNMP Network Management System
SNMP
syslog
EMS
EMS
EMS
Router
ADS
PG
Logger
Syslog Collector
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
Simple Network Management Protocol
UCCE SNMP Event Flow – High Level
14
SNMP Network Management System
SNMP
syslog
EMS
EMS
EMS
Router
ADS
PG
Logger
Syslog Collector
Cisco Prime Collaboration
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
Cisco Prime Collaboration
Simple Network Management Protocol
UCCE SNMP Event Flow – High Level
»For more information about Cisco Prime Collaboration: http://www.cisco.com/en/US/products/ps12363/index.html
15
SNMP
syslog
EMS
EMS
EMS
Router
ADS
PG
Logger
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
t_Event
Simple Network Management Protocol
UCCE SNMP Event Flow – Digging Deeper
16
ProTip: Cisco SNMP Service starts SNMPDM, which starts
CCCAAgent Logger
CSFS
CCCA
Agent
SNMP
DM
MDS
Router
Syslog Collector
SNMP NMS
EMS EMS
CW2K
Feed
Other Agents (Generic traps,
platform-specific subagents)
EMS
EMS
SNMP
syslog
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
Simple Network Management Protocol
Logging Differences for SNMP vs Syslog Events
– Certain events hard-coded as “Objects”
– SNMP looks for this flag and will only process “Objects”
Example: Syslog-only “Simple” Event (no Object)
CW2KFeed picks the event from the pipe and pushes it out to Syslog collector
17
Logger
CSFS
CCCA
Agent
CW2K
Feed
03:37:06:096 la-csfs Trace: ProcName:mds SysName:85PG2B StatusCode:The remote process has exited. Severity:Error Category:Message Delivery
03:37:06:096 la-csfs Trace: ID: e1048013 Msg: Unexpected disconnect from client pim1.
03:37:06:096 la-csfs Trace: EMS ID [e1048013] in Primary Positive Table
03:37:06:096 la-csfs Trace: Using [e104801385PG2Bpim1] as String for Suppression Event
03:37:06:096 la-cw2kfeed Trace: CSysMsg::PutQMsg at Tail -- Q size = 1.
03:37:06:096 la-cw2kfeed Trace: CSyslog::GetCSFSMsg Reading data from CSFS pipe feed.
03:37:06:096 la-cw2kfeed Trace: CSysMsg::GetQMsg -- event for data in Q has been signalled.
03:37:06:096 la-cw2kfeed Trace: CSysMsg::GetQMsg -- Retrieving Q item # 1.
03:37:06:096 la-cw2kfeed Trace: <131>101: 85pg2b: Mar 8 2012 00:37:03.679 +0000: ICM_PG_MessageDelivery-3-1048013: [comp=PG2-
B][pname=mds][iid=ucce][mid=1048013][sev=error]: Unexpected disconnect from client pim1.
03:37:06:096 la-cw2kfeed Trace: CSysMsg::GetQMsg -- Q is empty. Waiting for data.
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
Simple Network Management Protocol
Logging Differences for SNMP vs Syslog Events
– Certain events hard-coded as “Objects”
– SNMP looks for this flag and will only process “Objects”
Example: SNMP Event (Object!)
CW2KFeed still picks the event from the pipe and sends to Syslog collector
18
Logger
CSFS
CCCA
Agent
CW2K
Feed
03:37:06:096 la-cw2kfeed Trace: <132>102: 85pg2b: Mar 8 2012 00:37:03.695 +0000: ICM_PG_NodeManager-4-102C113: [comp=PG2-
B][pname=nm][iid=ucce][mid=102C113][sev=warning]: ICM\ucce\PG2B node process pim1 exited from Control-C or window close.
03:37:06:096 la-csfs Trace: ProcName:nm SysName:85PG2B StatusCode: Severity:Warning Category:Node Manager
03:37:06:096 la-csfs Trace: ID: a102c113 Msg: ICM\ucce\PG2B node process pim1 exited from Control-C or window close.
03:37:06:096 la-csfs Trace: Using [3_2_85PG2B_ICM\ucce\PG2B_pim1] as Unique Object String
03:37:06:096 la-csfs Trace: Created New Base Object '3_2_85PG2B_ICM\ucce\PG2B_pim1'
03:37:06:096 la-csfs Trace: Using [a102c113_3_2_85PG2B_ICM\ucce\PG2B_pim1] as String for Object Suppression Event
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
Simple Network Management Protocol
Logging Differences for SNMP vs Syslog Events
– Certain events hard-coded as “Objects”
– SNMP looks for this flag and will only process “Objects”
Example: SNMP Event (Object!)
Events for Host\Instance\Process use same Object created since last CSFS restart
19
Logger
CSFS
CCCA
Agent
CW2K
Feed
03:37:06:096 la-csfs Trace: ProcName:nm SysName:85PG2B StatusCode: Severity:Warning Category:Node Manager
03:37:06:096 la-csfs Trace: ID: a102c113 Msg: ICM\ucce\PG2B node process pim1 exited from Control-C or window close.
03:37:06:096 la-csfs Trace: Using [3_2_85PG2B_ICM\ucce\PG2B_pim1] as Unique Object String
03:37:06:096 la-csfs Trace: Created New Base Object '3_2_85PG2B_ICM\ucce\PG2B_pim1'
03:37:06:096 la-csfs Trace: Using [a102c113_3_2_85PG2B_ICM\ucce\PG2B_pim1] as String for Object Suppression Event`
03:37:15:596 la-csfs Trace: ProcName:nm SysName:85PG2B StatusCode: Severity:Warning Category:Node Manager
03:37:15:596 la-csfs Trace: ID: a102c110 Msg: ICM\ucce\PG2B node process pim1 successfully reinitialized after restart.
03:37:15:596 la-csfs Trace: Using [3_2_85PG2B_ICM\ucce\PG2B_pim1] as Unique Object String
03:37:15:596 la-csfs Trace: Event for Existing Base Object '3_2_85PG2B_ICM\ucce\PG2B_pim1'
03:37:15:596 la-csfs Trace: Using [a102c110_3_2_85PG2B_ICM\ucce\PG2B_pim1] as String for Object Suppression Event
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
Logging Improvements
Call Router RTRTrace
Check boxes for:
– Config Changes
– Route Request
– Network VRU
– Translation Route
– Call Queueing
– CallType RealTime
– INRC (RC Dialogue)
20
Peripheral Gateway CTI OS
– Registry EMSTraceMask 0x60A0F
Peripheral Gateway EA PIM
Enable in Procmon:
– trace tp* /on
– trace precall /on
– trace *event /on
– trace csta* /on
– trace periph* /on
– trace opc* /on
Peripheral Gateway OPC:
‒ OPCTEST debug /agent /routing
/closedcalls /cstacer /ctimsg
/inrcmsg /calls
‒ Registry EMSTraceMask 0x40
Peripheral Gateway CTI Gateway
‒ Registry EMSTraceMask 0xF8
Peripheral Gateway JGW Enable in Procmon:
‒ trace JT_TPREQUESTS /on
‒ trace JT_JTAPI_EVENT_USED* /on
‒ trace JT_PIM_EVENT /on
‒ trace JT_ROUTE_MESSAGE /on
‒ trace *CONF* /on
Peripheral Gateway VRU PIM
‒ Registry EMSUserData 7F F7 C0 Peripheral Gateway MR PIM
‒ Registry EMSTraceMask 0xF8
‒ Registry EMSUserData DF
Logger Configlogger
‒ Registry EMSTraceMask 0xFFFF
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
Logging Improvements
Simplified Tracing Introduced in UCCE 8.0(1)
– Level 0 – Default / Error :: Zero to minimal performance impact
– Level 1 – Warning :: Small performance impact, Cumulative tracing
– Level 2 – Informational :: Medium performance impact, Cumulative tracing
– Level 3 – Debug :: High performance impact, Cumulative tracing
How do they translate?
21
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
Logging Improvements
Simplified Tracing Introduced in UCCE 8.0(1)
– Level 0 – Default / Error
– Level 1 – Warning
– Level 2 – Informational
– Level 3 – Debug
How do they translate?
Goal: Only ask for logs
ONCE
Reference the UCCE Serviceability Guide for more information! http://www.cisco.com/en/US/products/sw/custcosw/ps1844/products_installation_and_co
nfiguration_guides_list.html 22
PG Process Level 0
(Default/Error) Level 1
(Warning) Level 2
(Informational) Level 3 (Debug)
JTAPIGW
JT_JTAPI_EVENT_ USED
JT_TPREQUESTS JT_PIM_EVENT
JT_ROUTE_ MESSAGE
JT_CONNECTION *CONF*
JT_JTAPI*
JT_HEX
JT_ROUTE*
JT_TERM*
JT_LOW*
JT*
OPC default, cstacer
0x40
agent, inrcmsg, closedcalls,
tpmsg, routing 0x40
calls, NCT, simplified
0x40
Missingdata, halfhour
0x40
PGAGENT 0x00 0x03 0x0F 0xFF
CTI SVR 0xF0 0xF6 0xFE 0xFF
CTI OS 0x60A0F 0x240A2F 0x260A2F 0x2E0A2F
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
Logging Improvements
EMS Log Compression = Increased Retention (x10!) for:
– CTI OS Server
– OPC-CCE
OPC Capture Binary Retention also increased x4
~10% increase in CPU and Disk I/O during 150% load testing.
Introduced with 7.5(10), 8.0(3), 8.5(1)
Corresponding Dumplog enhancements made on Peripheral Gateways
– Uses \icm\bin\gzip.exe to unzip compressed EMS before dumping
– User must manually decompress files if using dumplog on non-PG machine.
23
‒ OPC-TDM
‒ CTISVR
‒ EAGTPIM
‒ JGW
‒ VRUPIM
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
OPC Capture
How is OPC Capture different from traditional component logs?
Also: OPCconfig, Pgreg, OPCXferData
Not a Quick Process! 24
Debug Level 1 opc_120301_170156.ems(.gz)
10101000100101100010 Dumplog
OPCdata_opc_0301_170156.cap
111111111111111111111111 Merge Debug
Missing lots of trace messages!
Free Disk
Space > 5GB
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
OPC Capture
Where are the OPC Capture files?
25
PG Service (Re)Start, OPC State Transfer
Triggers dated folder creation containing:
• Capture data
• OPC Config Export
• Registry Export
• State Transfer (optional)
Timestamps must match!
Non-matching OPCdata = Capture File Wrap
Root Cause Unlikely!
M
K
D
I
R
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
OPC Capture Pop Quiz!
If OPC asserted at 6:39PM on February 15th, which folder will have the capture files that include the assertion?
A.
B. C.
D.
Dated Folder is created when OPC starts, so 18:39:44 will have the capture data AFTER the assertion. The folder from the PREVIOUS start is correct.
But what if the Capture Files have wrapped & timestamps don’t match?
26
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
OPC Capture
Hard-Coded 2GB Capture Limit Per Dated Folder
– OPCdata_opc_xxxx_yyyyyy.cap files limit ~10MB each. Wrapping = Failure.
How can we baseline the Capture Files?
7.2(5) & 7.5(1) through 8.5(1): ‘exit_opc’
– Intrusive / downtime required
– Could clear / delay condition recurrence
8.5(2) & Later: ‘newcap’
– Does not kill OPC, but WILL delay requests from agent desktop (i.e. consult calls) for 1-2 seconds! These requests typically have a 60 second timer, so 1-2 seconds will not impact
– Perform during low call volume / agent activity
27
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
OPC Capture
Where are the OPC Capture files?
28
PG Service (Re)Start, OPC State Transfer
Triggers dated folder creation containing:
• Capture data
• OPC Config Export
• Registry Export
• State Transfer (optional)
Timestamps must match!
Non-matching OPCdata = Capture File Wrap
Root Cause Unlikely!
M
K
D
I
R
NewCap in
8.5(2) or later
Updated!
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
The Truth About Minidumps
Minidumps: Quick, limited information. Root cause HIGHLY unlikely!
Full crash dump: Complete process snapshot
– Larger file size: 40MB vs 40KB
– Takes few seconds longer to write…but worth it
RCA Rule #1 – If you see a mdmp, enable the full dump! – Minidump does NOT guarantee root cause
– Risk producing multiple minidumps VS single full crash dump
Always Collect The Following – .mdmp file
– <process>.exe file – Needed to confirm exact build info to match binaries when debugging
– <process>.pdb file – Symbol file containing debug info
29
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
The Truth About Minidumps
30
Registry Value: EMSGenerateSmallMemoryDump
– Default: 0x20000001 Minidump
– Recommended: 0x20000002 Full crash dump
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
Network Optimization
NIC Teaming Not Supported
– Load balancing can negatively affect critical HW I/O continuity
Binding Order
– Public NIC should be listed first followed by Private NIC
Speed And Duplex Settings
– Always full duplex (exceptions with some peripherals, e.g. Avaya)
– Statically set and must match on both the NIC and Switch Port, respectively
– Use Auto/Auto ONLY when running Gigabit NIC/Switch
Best Practices tech tip
– http://www.cisco.com/en/US/products/sw/custcosw/ps1001/products_tech_note09186a00808160f4.shtml
You’ve Heard It A Thousand Times…+1
31
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
Network Optimization
TCP/IP Offloading, Potential Loss of Server Network Connection
– Offloads some TCP/IP processing to network adapters
– Causes serious issues with UCCE & CVP components
– Enabled by default with Windows 2003 SP2, 2008
– http://www.cisco.com/en/US/ts/fn/632/fn63215.html
– http://docwiki.cisco.com/wiki/Contact_Center_Networking:_Offload,_Receive_Side_Scaling_and_Chimney Always disable in all possible places (NIC properties, NIC drivers, Windows registry)
Microsoft TCP/IP Stack Logic Affects MDS TCP Characteristics (RTT)
– http://www.cisco.com/en/US/ts/fn/620/fn62508.html
Important! Urgent! Please Read! Nota Bene!
32
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
Network Optimization
Intermittent Silent Monitoring Disconnections, Failures – 10:15:17.517 552 SilentMonitorService Thd(1292) CUDPPort(008AFB90)::ReceiveData, recvfrom,
Blocking call interrupted by WSACancelBlockingCall. (WSAEINTR), Error Number(10004).
Lost Heartbeats, Test-Other-Side (ToS) Seen in MDS – ra-mds Trace: EMT 10: Missing heartbeat, countdown is 5, last received seqno 269795
– pg1b-mds Trace: Sending TOS request: sequence = 1. pg1b-mds Trace: Received TOS response: sequence=1 status=ENABLED.
Logger Sync issues during business day, then clearing up at night
– Recovery keys different between A-side and B-side
http://www.cisco.com/en/US/ts/fn/632/fn63215.html
33
Validate All Windows 2003 Servers: netsh int ip show offload
Validate All Windows 2008 Servers: netsh in tcp show global
+ NIC Drivers!
+ NIC Properties!
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
Updated Platform Support
8.5(2) adds support for Windows 2008 R2 SP1 (64-Bit Only)
– Enhanced Native Security = no need for UCCE Installer OS Hardening
– Unsupported PGs on 2008: Aspect, Spectrum, Symposium (Ref: Cisco ICM ACD Matrix)
9.0(1) requires Windows 2008 R2 SP1, SQL Server 2008 R2 SP1
OS architecture change = UCCE process windows no longer appear
Title bar indicators gone with the wind(ows).
34
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
Updated Platform Support
OS architecture change = UCCE process windows no longer appear
Title bar indicators gone with the wind(ows).
35
EMSmon
Usage Syntax:
emsmon <cust_instance> <node_name> <process> {system}
Note: ‘system’ argument is optional and only
needed if connecting to remote server
Example (local):
EMSMON ucce RouterA rtr
Example (remote):
EMSMON ucce PG1A pim1 UCCEPG1A
*Max of 1 local and 5 remote connections
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
Updated Platform Support
OS architecture change = UCCE process windows no longer appear
Title bar indicators gone with the wind(ows).
36
EMSmon
Usage Syntax:
emsmon <cust_instance> <node_name> <process> {system}
Note: ‘system’ argument is optional and only
needed if connecting to remote server
Example (local):
EMSMON ucce RouterA rtr
Example (remote):
EMSMON ucce PG1A pim1 UCCEPG1A
*Max of 1 local and 5 remote connections
? ? ?
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
Enter…The Diagnostic Framework Portico
37
Services Installed Locally
Not Bold = No Processes Running
Direct Link
Process Title Bar
Blue: Parentheses status
change within 10 min
Uptime
Red < 10 minutes
10 < Yellow < 30
Smart refresh
Remembers
current location
Bookmark
Your Servers
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
Diagnostic Framework Portico
Single-system “Support Tool” – More than just Process Status!
Set tracing for single process
38
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
Diagnostic Framework Portico
Single-system “Support Tool” – More than just Process Status!
Set tracing for single process
39
… and verify
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
Diagnostic Framework Portico
What if a trace is set using “old” tools?
For example, RTRTrace:
40
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
Diagnostic Framework Portico
Level 99 = Custom (Does Not Match Defined Trace Levels)
Set a trace level 0-3 to overwrite using SetTraceLevel
41
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
Diagnostic Framework Portico
Dump logs for a single process
42
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
Diagnostic Framework API
How does it work?
Other devices also have their own Web Services for similar functions
Isn’t there a tool to bring this all together?
43
RoggerA RoggerB
UCM PUB UCM SUB
IP IVR 1 IP IVR 2
HTTPS
HTTPS
PG1A PG1B
ADS 1 ADS 2
HTTPS
RTMT
RTMT
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
Analysis Manager
Tool within the CUCM Real-Time Monitoring Tool (RTMT)
Node, Trace template configuration steps required
Set and collect traces in real-time or scheduled
Collect system environment info
Supported Products:
– UCM – IOS Gateways
– UCCE – Unity Connection
– UCCX – Unified Presence
– CVP
Solution Tool Option #1
44
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
Analysis Manager
How does it work?
45
RoggerA RoggerB
UCM PUB UCM SUB
IP IVR 1 IP IVR 2
HTTPS
HTTPS
PG1A PG1B
ADS 1 ADS 2
HTTPS
RTMT
RTMT
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
Analysis Manager
How does it work?
46
RoggerA RoggerB
UCM PUB UCM SUB
IP IVR 1 IP IVR 2
PG1A PG1B
ADS 1 ADS 2
ICM
UCM
UCCX
Node Type Nodes
Node Group
Trace Template RTMT
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
Analysis Manager
Add Nodes
47
Node Type
Node
Node Group
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
Analysis Manager
Create Trace Template
48
Select
Processes
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
Analysis Manager
Create Trace Template
49
Set Trace
Levels Level 0 – Default / Error
Level 1 – Warning
Level 2 – Informational
Level 3 – Debug
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
Analysis Manager
Schedule Periodic Collection
Trace level applied at Start Time
Set back to original at End Time
50
Select Node or
Node Group
Set Duration
Set Interval
Select Trace Template
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
Analysis Manager
So this supports all of the core “Cisco Unified” contact center products, right?
– Analysis Manager does NOT support CUIC, Finesse, MediaSense, or SocialMiner
What about the other products that use RTMT? Is this version compatible?
What if I have multiple ICM instances?
– Analysis Manager does not support multiple ICM instances.
Let’s check out Option #2!!
51
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
System CLI
Solution-wide tool
– System Mode – run commands on multiple devices
– Easily Add/Remove devices.
Syntax similar to Cisco IOS gateway CLI
– Without Tab auto-complete
Capture, debug, trace from:
– All CCE servers – IOS Gateways – UCM
– CVP / IP IVR – Presence – CUIC
Low Priority Tasks Throttled – Reduced Priority, Low CPU threshold
– Will only use the IDLE CPU on the system. Max of 20 concurrent requests
– No timeouts on the client; controlled by the server.
– Load risk still applies to IOS devices.
Solution Tool Option #2
52
Multi-Instance Support
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
System CLI
How does it work?
53
RoggerA RoggerB
UCM PUB UCM SUB
IP IVR 1 IP IVR 2
HTTPS
HTTPS
PG1A PG1B
ADS 1 ADS 2
HTTPS
RTMT
RTMT
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
System CLI
How does it work?
54
RoggerA RoggerB
UCM PUB UCM SUB
PG1A PG1B
ADS 1 ADS 2
PG1A PG1B ADS
IP IVR 1 IP IVR 2 CUIC PUB CUIC SUB
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
System CLI
How does it work?
55
RoggerA RoggerB
UCM PUB UCM SUB
IP IVR 1 IP IVR 2
CUIC PUB CUIC SUB
PG1A PG1B ADS
ICM
UCM
UCCX
CUIC
Device Type Devices
CLI
Device List
Location differs
with/without
CVP in
deployment
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
System CLI Deployments Without CVP
How does it work?
56
RoggerA RoggerB
UCM PUB UCM SUB
IP IVR 1 IP IVR 2
CUIC PUB CUIC SUB
PG1A PG1B ADS
ICM
UCM
UCCX
CUIC
Device Type Devices
CLI
Device List
devices.csv
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
System CLI Deployments Without CVP
Devices.csv must be present for CLI to use System Mode (without CVP)
– Otherwise, the CLI will run as a single server tool
Create from Devices-sample.csv (included with every UCCE installation)
57
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
System CLI Deployments Without CVP
58
<server or IP>, <description>, <device type>, <group>, <username>, <password>
10.10.10.30, RoggerA, ICM, ucce-sideA, VMLOAD\Administrator, cvpcisco
10.10.10.31, PG1A, ICM, ucce-sideA, VMLOAD\Administrator, cvpcisco
Required
Recommended
ProTip: If System CLI will be run only on ADS, then devices.csv is only required on ADS.
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
System CLI Deployments Without CVP
How does it work?
59
RoggerA RoggerB
UCM PUB UCM SUB
IP IVR 1 IP IVR 2
CUIC PUB CUIC SUB
PG1A PG1B ADS
ICM
UCM
UCCX
CUIC
Device Type Devices
CLI
Device List
devices.csv
Group
“All ICM”
Device Type
Trace Template
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
System CLI
How does it work?
60
RoggerA RoggerB
UCM PUB UCM SUB
CUIC PUB CUIC SUB
PG1A PG1B ADS
ICM
UCM
CUIC
Devices
CLI
Device List
devices.csv
Group
“All ICM”
Device Type
Device Type
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
System CLI Deployments With CVP
How does it work?
61
RoggerA RoggerB
UCM PUB UCM SUB
CVP 1 CVP 2
CUIC PUB CUIC SUB
PG1A PG1B ADS
ICM
UCM
CVP
CUIC
Devices
Device List
Group
“All ICM”
CVP
OAMP
CVP Remote
Operations
Manager CLI Device Type
Device Type
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
System CLI Deployments With CVP
Devices Managed from CVP OAMP
62
Encrypted!
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
System CLI
Initial Login is to single machine only (CVP OAMP or a UCCE server)
Enter System Mode to connect to solution devices (defined in OAMP or local CSV) admin:system
Initializing system mode …
Retrieving device list. This process may take a few minutes to complete.
. . .
Unable to connect to 10.10.10.130:7890(icm)
. . . . .
Once in System Mode, commands will be run against all devices. admin(system):show version
Retrieving [version] data from device [uccergra30] ProductType [icm] …
ICM 8.5(2) BuildNumber=37931 ES=0 PatchInstallerVer=465
Retrieving [version] data from device [uccepg1a31] ProductType [icm] …
ICM 8.5(2) BuildNumber=37931 ES=0 PatchInstallerVer=465
Useful Commands
63
Success
Fail
Success
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
System CLI
Commands can still be limited to a single device… admin(system):show debug server 10.10.10.30 Retrieving [debug] data from device [10.10.10.30] ProductType [icm] …
Component: icm:Router A, subcomponent: icm:rtr Trace level = 0
…Device Type… admin(system):show devices devicetype icm Device host name: uccergra30 Port: 7890 Product type: ICM Group: ucce-sidea Device host name: uccepg1a31 Port: 7890 Product type: ICM Group: ucce-sidea
64
…Group… admin(system):debug level 1 group ucce-sidea
Component: icm:Logger A, subcomponent: icm:hlgr
Debug level is set successfully
…repeats for all Logger A processes
Component: cvp:CallServer, subcomponent: cvp:MSGBUS
Debug level is set successfully
…repeats for all CVP processes
Component: icm:CTI Server 1A, subcomponent: icm:ctisvr
Debug level is set successfully
…repeats for all CTI Server 1A processes
Component: icm:PG1A, subcomponent: icm:jgw1
Debug level is set successfully
…repeats for all PG1A processes
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
System CLI
…or on-the-fly combination of devices admin(system):show config server 10.10.10.30|10.10.10.31 redirect dir c:\temp Retrieving [configuration] data from device [10.10.10.30] ProductType [icm] … Fetching configuration resource for component DumpCfg Fetching configuration resource for component ExportICMCfg Fetching configuration resource for component ConfigExport Fetching configuration resource for component Registry Retrieving [configuration] data from device [10.10.10.31] ProductType [icm] … Fetching configuration resource for component Registry Output is saved to “c:\temp\clioutput0.zip”
65
File Download
Specify Save
Location
Rogger
DumpCfg,
DB Export
Peripheral Gateway
Registry Export Only
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
System CLI
Best Of All, The Game-Changer
admin(system):show tech-support Retrieving [Every Single Piece Of] data from
device [Your Entire Deployment]
ProductType [ICM, UCM, CVP, IOS, UCCX, CUIC, ETC!] …
Output is saved to
“C:\Cisco\CVP\wsm\CLI\download\clioutput1.zip”
66
Instant 24-hour Snapshot
Produces zip of ALL defined server component
logs (including OPC Capture), ICM config,
dumpcfg, versions, registries,
OS info, PerfMon, debug levels, etc.
For >/<24 hours, add:
“absdatetime = MM-DD-YYYY:hh:mm”
Organized By Device
EVERYTHING TAC/DE would need
to start investigation.
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
System CLI Quick-Reference Guide
http://docwiki.cisco.com/wiki/Unified_System_CLI_Quick-Reference_Guide
+ Command Examples!
+ Troubleshooting! 67
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
System CLI Limitations
CSCub56442 – CVP 9.0 does not allow Remote Operations on client Windows
– Use CVP 8.5 installer for Remote Operations in interim. Fixed in 10.0.
No Device Types exist for Finesse (CSCue98453) or MediaSense (CSCue98487)
– CLI does not speak the same language.
– Can get basic info by using device type CUIC (common platform), but not too useful: Device is marked offline due to unreachable device or invalid user name or password [10.10.10.38:8443(cuic)]
Please run "system init" command to retry.
Product Version List:
Active Server Version: 9.0.1.10000-2
Cisco Finesse
9.0.1.10000-2
Finesse has basic website for logs:
– http://<Primary>/finesse/logs/webservices/
MediaSense has a more advanced Serviceability portal for tracing, services, etc:
– http://<Primary>/oraservice/
SocialMiner not integrated, but enhancement filed too: CSCuf85862 68
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
System CLI Scripting With Batch Files
Samples in C:\icm\serviceability\wsccli\sample
Bat file can contain commonly-used commands
– Set tracing on CVP
– Collect logs from all ICM devices
– Endless possibilities
Bat file can be run manually or automatically with Windows Scheduler
Bring quick resolution to an ongoing issue!
69
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
Diagnostic Framework Match-up
70
Ability to schedule periodic collections
GUI
No multi-instance support
Requires UCM
Thick-Client
No Configuration Needed
Browser Accessible
Quick Single-Process Transactions
No multi-instance support
Limited Scope (Single Device)
Positives Positives
Negatives
Negatives
Positives
Multi-instance support
System Mode
ANY command can be scheduled
No support for OEMs, Finesse & MediaSense
…No GUI?
Negatives
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
CCE Web Administration
CCE 9.0 adds Web Administration (Apache Tomcat 7 webapp)
“Deployment Type Aware”: Used for Precision Routing in UCCE, more in PCCE
72
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
CCE Web Administration
Not to be confused with Web Setup (added in 8.0)!
73
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
CCE Web Administration
Limited management gadgets for UCCE
UCCE Deployment Types
No change to
Configuration
Manager
74
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
CCE Web Administration
Plenty of management gadgets
PCCE Deployment Types
75
Consolidated
Configuration
Manager
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
CCE Web Administration
Plenty of management gadgets
PCCE Deployment Types
76
Consolidated
Configuration
Manager
Packaged CCE is the SAME code as Unified CCE!
Same base installer, same maintenance releases
Same components, same serviceability tools
Only difference = Deployment Type!
More Info: http://www.cisco.com/en/US/products/ps12586/index.html
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
CCE Web Administration
Important Detail:
If a PCCE customer decides they have outgrown the PCCE capacity, they can change the deployment type to UCCE.
BUT, they must know it is a one-way path.
– Moving from UCCE to PCCE is not supported!
77
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
CCE Web Administration
Deployment Type validates system before unlocking simplified PCCE Gadgets
78
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
CCE Web Administration
Packaged CCE limits configuration items – finally!
PCCE Deployment Types
79
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
CCE Web Administration
Limited error descriptions with
standard tools.
Example: Administrator must
trace references in order to
delete a Route.
New tools provide
better detail why a
config change was
not allowed.
Applies to PQs, ECC
variable sizing, etc. MUCH easier for Administrators!
80
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
CCE Web Administration Problem Scenario
Problem Statement: Login attempts to CCE WebAdmin intermittently hang or fail with “server not responding” error
Business Impact: Administrators and Supervisors unable to access PCCE Web Administration console for agent management. Queues are not properly staffed to handle incoming call volume and agents must be adjusted.
Version:
– CCE 9.0(2)
Troubleshooting Performed:
– Restarting the Apache Tomcat service temporarily resolves the issue
Login Timeouts
81
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
CCE Web Administration Problem Scenario
Check the IE9 Developer Tools Console (F12) for script errors
Timeouts from the API are causing UI issues!
WebAdmin = Tomcat WebApp. Increase tracing to see API timeouts.
– Edit /icm/tomcat/conf/logging.properties: com.cisco.ccbu.cce.security.ccead.CCEADConfig.level = FINE
– Stop/Start Tomcat service
82
, check Tomcat logs , reproduce login issue
Verify
Task
Manager
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
CCE Web Administration Problem Scenario
tomcat7-stderr.date.log shows failed login attempt. Feb 15, 2013 4:55:57 PM com.cisco.ccbu.cce.security.ccead.CCEADConfig checkConfigDomainUser
FINE: Could not find admin123@ad in any domain.
Feb 15, 2013 4:55:57 PM com.cisco.ccbu.cce.security.ccead.CCEADConfig getAdministratorPrincipal
FINE: Authentication as administrator failed for admin123@ad
AD authentication is failing!
How does Tomcat authenticate?
Next Step: Enable sADlib debugging:
– Set HKLM\SOFTWARE\Cisco Systems, Inc.\ICM\SystemSettings\Debug_Sadlib to “Yes”
– Restart Tomcat. sADlib output will write to C:\temp\sadlib.log.
– Negligible production impact, but log will grow without purging. Watch disk space, set registry key back to “No” when finished
Time to learn how sADlib works! 83
Tomcat sADlib.dll Windows
API Active
Directory
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
CCE Web Administration Problem Scenario
sADlib recognizes two domain controllers: AD.cisco.com & ADB.cisco.com
1. Search for UPN [email protected] in ADB (arbitrary) 11:05:09 Find External Trusted Domains
11:05:09 Searching for (&(objectCategory=person)(objectClass=user) ([email protected])) in DC=ADB,DC=CISCO,DC=COM
11:05:09 Execute search -- Initialize the search
11:05:09 Get each row from the result table
No hits mentioned. Does the search ever even finish?
– Nope! No hit in ADB, but no failover to AD!
AD delay issues cause sADlib search to hang
– Defensive fix CSCue95184 adds a 5 second timeout to handle this AD condition Available in 9.0(4), 9.0(3) ES3 (link in slide notes)
Fix applied, onward!
84
AD ADB
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
CCE Web Administration Problem Scenario
2. Search for UPN [email protected] in AD with defensive fix 11:05:15 Execute search -- Initialize the search
11:05:15 Get each row from the result table
11:05:20 Search Over for User
3. Search without suffix: admin123 11:05:20 Execute search -- Initialize the search
11:05:20 Get each row from the result table
11:05:20 SadStatus::addError User or Group not found DC=AD,DC=CISCO,DC=COM
11:05:20 Search Over for User
4. Search for sAMAccountName admin123 (pre-Windows 2000 name) 11:05:20 The search for user info is completed
11:05:20 0 "SadMan::findUserInfo" "SadMan::searchB (using dn) Found dnDomain:
OU=USERS, DC=AD,DC=CISCO,DC=COM userName: admin123“
85
AD ADB
No results!
5 second delay = timeout fix!
No results!
No timeout! AD better now?
Found!
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
CCE Web Administration Problem Scenario
Check if user is a member of the CCE Security Groups (config or setup) 11:05:20 0 "SadGroup::getAdAttrib" "threadId: 4583 sAMAccountName"
11:05:20 SadIcm::isIcmSGMember dnUser: CN=admin123, OU=USERS, DC=ad,DC=cisco,DC=com
The user is in the Config group, so login will be permitted. 11:05:21 SadIcmInstance::IsUserMemberOfConfigSgEx user is a member of Config. usserId: CN=admin123,OU=USERS,DC=ad,DC=cisco,DC=com
With the timeout fix in place, the user finally logged in – but only after 12 seconds!
– 11:05:09 11:05:21
– This stresses the importance of a rock-solid AD infrastructure
– Staging Guide references MS tools to ensure AD health and connectivity (dcdiag, netdiag) http://www.cisco.com/en/US/products/sw/custcosw/ps1844/prod_installation_guides_list.html
86
AD ADB
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
CCE Web Administration
Also...Configuring Deployment Type unlocks Congestion Control feature
What purpose does Congestion Control serve? 87
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
Congestion Control
Prior to 9.0
88
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
What is Congestion Control?
Central Controller Guardian
– Keeps the system running close to rated capacity under extreme overload conditions
– Deployment Type sets system capacity (in CPS at Router)
Satisfactory service to a limited percentage of calls rather than degraded service to ALL calls
– Call never reaches routing script! “Handled” at Routing Client.
– What are the “handling” options?
89
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
Congestion Control Treatment Options
Treat Call with Dialed Number Default Label [1]
Treat call with Routing Client Default Label [2]
Treat call with System Default Label [3]
Terminate call with a Dialog Fail or RouteEnd [4]
Treat call with Release Message to Routing Client [5]
90
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
Congestion Control Configuration
1. Set Deployment Type & Enable in Web Administration
2. Select Treatment Mode
– “Use System Congestion Control” = Global mode set in Web Administration gadget
OR
– Set on a per-routing client basis
**All PGs must be at 9.0 for Congestion Control to work properly**
Reference 9.0 SRND for details and upgrade options 91
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
Congestion Levels and Thresholds
System capacity is adjusted based on real time conditions. Assume system configured for 100 CPS (avg)
Onset/Abate allow short spikes when hovering near configured capacity
Prevents high rate of level oscillation & alarming when approaching limits
NOT adjustable. Defined by system Deployment Type
Level 1
Level1Onset
110% of Capacity
L1Reduction
10% Handled
Level1Abate
90% of Capacity
Level 2
Level2Onset
130% of Capacity
Level2Reduction
30% Handled
Level2Abate
100% of Capacity
Level 3
Level3Onset
150% of Capacity
Level3Reduction
30 to 100% Handled
Level3Abatement
100% of Capacity
92
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
Congestion Control
Protects the system from a sudden overload
Existing “committed” calls protected
Restricts capacity by “handling” excess calls
Options:
– Redirect using Label
– Reject/Release
93
Level 1
• Triggered at 110% volume
• 10% reduction in new calls
Level 2
• Triggered at 130% volume
• 30% reduction in new calls
Level 3
• Triggered at 150% volume
• 30-100% reduction in new calls
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
Congestion Control
PG Startup shows current Congestion Control settings
PG1A-opc Trace: ADDED 1 Peripheral.
PG1A-opc Trace: Routing Client congestion control data: Treatmentmode = 0 label = 12345 rcid = 500
Logs also show Congestion level transition
pg1a-opc trace: congestion trigger received: congestion level=3 reduction per=100
Notifications via Event Viewer & SNMP
94
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
Congestion Control
“System Capacity Real Time Data” CUIC report, multiple views:
– Congestion Information, Reduction Percentage
– Key Performance Indicators
95
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
Congestion Control Recap
Assess the system
Deployment Type
Set the Deployment
Type
Select the Congestion Treatment
Mode
- System Level
Select the Congestion Treatment
Mode
- Routing Client
Optionally
Override the system
capacity settings if required.
Setup call treatment
system
Enable the Congestion
Control
Steps to Deployment
System Protected
against overload
System Unprotected
from overload
96
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
Precision Routing
Traditional Skill Group Routing
97
Sales
Support
Returns
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
Precision Routing
Attribute Routing
– Simpler scripting
– Multidimensional
– Multi-step
98
Attribute Value
English 9
Sales 8
Returns 5
Boston True
Attribute Value
English 7
Spanish 8
Support 10
Baltimore True
Attribute Value
English 9
Sales 5
Returns 8
Boston True
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
Precision Routing
Attribute Routing
– Simpler scripting
– Multidimensional
– Multi-step
99
Attribute Value
English 9
Sales 8
Returns 6
Boston True
Attribute Value
English 7
Sales 2
Returns 10
Baltimore True
Attribute Value
English 9
Sales 5
Returns 8
Boston True
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
Precision Routing Problem Scenario
Problem Statement: Unable to create new Precision Queues
Business Impact: Unable to complete implementation of new design, causing delays to rollout of Precision Routing.
Versions:
– CCE 9.0(3)
Troubleshooting Performed:
– Checked documentation and on-line help for limit: Total Skill Groups and Precision Queues per agent: 15
15 = Packaged CCE size limit, but same concept applies to UCCE (higher limit)
Agents & Attributes
100
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
Precision Routing Problem Scenario
What is seen in the Web Administration when trying to add new PQ?
– Improved error messages within UI eliminate need for checking logs
– Well, *almost* eliminate. Logs show offending Agent ID
One agent is preventing us from creating a new Precision Queue?!
– System is smart! No registry keys/DB hacks/etc to get around this limit!
– How is this agent skilled?
101
15:57:07:982 Dis-uaw Trace: [client: Tomcat7(upcc)] Update of central controller failed, Message: Precision
Queue 5017 (Sales) exceeds the limit of 15 combined Skill Groups/Precision Queues for Agent 5023
(5000.666), (Precision Queues : 16 Skill Groups: 0), LastUpdateKey: 0.
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
Precision Routing Problem Scenario
This agent only has 10 attributes!
We should have room still, up to 15, right?
– Not quite. Attempting to add an 11th attribute also fails. Attributes != Precision Queues
102
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
Precision Routing Problem Scenario
103
Agent has 4 Attributes
Assume there is a single Precision Queue match for each Attribute
Additional Precision Queues may also trigger those same Attributes
In PCCE, this agent can belong to 7 additional PQs (15 total)
Attribute
English
French
Italian
Spanish
English French Italian Spanish
Europe Canada USA
Attribute
English
French
Italian
Spanish
Attribute
French
Italian
Attribute
English
French
LATAM
Attribute
English
Spanish
8 PQs from
4 Attributes!
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
Quick Word on APIs
APIs available in CCE 9.0 for:
– Precision Queues – Deployment Type – Bucket Intervals
– Attributes – Congestion Control – AD Domains for Login
Developers Guide: http://www.cisco.com/en/US/products/sw/custcosw/ps1844/products_ programming_reference_guides_list.html
Enables powerful integrations across components within the solution
List, Get, Create, Update, Delete – REST
104
Agent’s Attributes
Fluent in English
Certified to sell Home Insurance
HR Database
API
Now certified to sell Auto Insurance
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
Virtualization
7.5(3) – Client Administrator Workstations, certain Peripheral Gateways (PGs)
8.0(2) – Expanded virtualization support on UCS B & C Series
– Central Controller, CVP, CUIC, Multichannel, etc.
– Launched http://docwiki.cisco.com/wiki/Virtualization_for_Unified_CCE
– OVA Templates introduced with VM resources/reservations
– OS/Application “Direct Install” still very manual
9.0(1) – Automated Cloning procedure released
– Packaged CCE Golden Template Automation Process
– Creates reusable images, decreases time to deploy
– Utilizes VMware PowerCLI, customer-specific data sheet XLS
How is Golden Template different from Direct Install?
Evolution of CCE on VMware
105
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
Unified Computing System (UCS) Platform
UCS C260M2 Server
– Local Storage
Two for redundancy
VMware ESXi
VMware vCenter
– New or existing
Local Storage No SAN!
VMware ESXi VMware ESXi
106
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
Installation Options
Direct Install
– Ideal for customers deploying themselves, lab environments, etc.
107
VMware ESXi VMware ESXi
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
Installation Options
Direct Install
– Ideal for customers deploying themselves, lab environments, etc.
Golden Template Automation
– Ideal for partners repeatedly performing PCCE deployments
108
VMware ESXi VMware ESXi
Partner Lab
Customer 1, 2, 3, …, N
VMware ESXi
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
Virtualization – Automation Procedure
Create Golden Template
Configure Destination
Servers
(Side A & B)
Customization
Spreadsheet
Start Cloning Scripts
Post Cloning Tasks
109
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
Virtualization – Golden Template Creation
Once for each component
Created on “Source” host
Will be cloned to customer environment
No configuration!
Download OVA Create VM
from the OVA
Install
Windows
Install Anti-
Virus
Enable Remote
Desktop
Install VMware
tools
Convert VM to
Golden Template
Install
Application
UCOS-based Applications
110
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
Virtualization – Destination Staging
2. Configure the
Destination servers
Golden Templates
(Source Host)
• RAID
• ESXi
• NIC config
• vCenter
• Etc…
ESXi hosts must be added to vCenter with IP address, not FQDN!
Packaged CCE scripts cannot find ESXi hosts by FQDN
VMware ESXi (A-side)
VMware ESXi (B-side)
111
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
Virtualization – Customization
3. Complete the Automation
Spreadsheet
VMware ESXi (A-side)
VMware ESXi (B-side)
4. Use vSphere PowerCLI to run
DeployVM.PS1 passing the
Automation Spreadsheet and
destination vCenter as parameters
112
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
Virtualization – Cloning
Cloning script creates customer VMs based off source Golden Templates
113
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
Virtualization – Cloning
VMware ESXi VMware ESXi
VMware ESXi
UCOS
CUIC A
Windows
Data
Server A
Windows
Call
Server A
UCOS
Finesse A
Windows
CVP 1A
Windows
CVP 2A
Windows
CVP
OAMP
UCOS
UCM Pub
UCOS
UCM Sub1
UCOS
CUIC B
Windows
Data
Server B
Windows
Call
Server B
UCOS
Finesse B
Windows
CVP 1B
Windows
CVP 2B
Windows
CVP
Reporting
UCOS
UCM Sub1
114
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
Virtualization – Post-Cloning
VMware ESXi VMware ESXi
VMware ESXi
UCOS
CUIC A
Windows
Data
Server A
Windows
Call
Server A
UCOS
Finesse A
Windows
CVP 1A
Windows
CVP 2A
Windows
CVP
OAMP
UCOS
UCM Pub
UCOS
UCM Sub1
UCOS
CUIC B
Windows
Data
Server B
Windows
Call
Server B
UCOS
Finesse B
Windows
CVP 1B
Windows
CVP 2B
Windows
CVP
Reporting
UCOS
UCM Sub1
5. Complete customer-specific configuration
– UCOS PlatformConfig XML FLP
– CCE Domain Manager, Web Setup, Base Config Import
115
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
Virtualization Problem Scenario
Problem Statement: Golden Template cloning fails
Business Impact: Unable to deploy new Packaged CCE customer environment using Golden Templates. Schedule will be heavily impacted if Direct Install procedure must be used.
Versions:
– PCCE 9.0(3)
– VMware ESXi 5.0
– VMware PowerCLI 5.0, 32-bit
Troubleshooting Performed:
– Confirmed ESXi hosts are added to vCenter using IP address
– Confirmed spreadsheet entry matches the VM name, but the PowerCLI will not find it
Golden Template Cloning Confusion
116
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
Virtualization Problem Scenario
What error does the PowerCLI give? Quite a few, actually: Invalid Golden Template Name
Template does not exist
Resource validation failed
Errors related to:
– Source/destination host
– Source/dest datastore
– Template
– VM Name
Work backwards – what does the Automation Spreadsheet say?
117
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
Virtualization Problem Scenario
PowerCLI looking for VM template named “CCE-Staged-W2K8-SP1-8G”
This VM template exists on the source vCenter
Cannot “Edit Settings” and view the properties of a template (like with a VM), but can check the .vmtx config file in the datastore browser.
118
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
Virtualization Problem Scenario
Hmmm…something looks different.
– The file names do not match the folder name!
119
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
Virtualization Problem Scenario
Hmmm…something looks different.
– The file names do not match the folder name!
Download the .vmtx config file and open it in your favorite text editor 120
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
Virtualization Problem Scenario
Parts of vmtx config file also do not match the template name shown in vCenter
This template was renamed in vCenter, which does not change the datastore files
So can we just update our Automation Spreadsheet with the original name?
– No. From the PCCE Install/Config Guide:
121
Original: CCE Staged W2K8_SP1-8G
Renamed: CCE-Staged-W2K8-SP1-8G
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
Virtualization Problem Scenario
To properly rename: Convert to VM > Clone > Convert to Template
– http://communities.vmware.com/message/1568820
– http://kb.vmware.com/kb/1029513
Convert original problematic template to VM
Clone original problematic VM to new VM
122
Convert new VM to Template
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
Virtualization Problem Scenario
Run DeployVM.ps1 cloning script
123
Original Golden Template
New Golden Template
Cloned CCE VM
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
Virtualization Problem Scenario
Run DeployVM.ps1 cloning script
124
Original Golden Template
New Golden Template
Cloned CCE VM
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
Virtualization – Useful Links
Cisco Packaged CCE Automation: Setup & Install Summary Video-on-Demand
– http://www.youtube.com/watch?v=3IqtBiIbecc&list=UULUuIagjnt1XDoc9rfRUUkA&index=1
DeployVM.ps1 script is released with Packaged CCE, but concepts can be applied to any VMware deployment.
– BRKCCT-2052 - UCCE Deployments on Cisco UCS Platforms Best Practices Oops! Tuesday, Jun 25, 8:00 AM - 10:00 AM has passed!
Catch the recording on CiscoLive365.com!
Main Unified Communications virtualization wiki:
– http://docwiki.cisco.com/wiki/Unified_Communications_Virtualization
Important sub-section for ESXi networking:
– http://docwiki.cisco.com/wiki/UCS_Network_Configuration_for_UCCE #Network_Requirements_for_UCS_C_Series_Servers
– Answers common questions for mapping Public/Private/Mgmt networks
125
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
Troubleshooting Unified CCE
Serviceability
– New Tools
– Old Tools with New Tricks
Real Customer Scenarios
– TCP Offload Anomalies
– Web Administration
– Congestion Control
– Precision Routing
– Virtualization
BONUS Multimedia & Solution Approach to Issue Resolution slides in Appendix!
Time for Q&A!
Summary
126
Diagnostic
Framework
Attribute
English
French
Italian
Spanish
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
Troubleshooting Unified CCE
Keep This Presentation as a Reference!
Homework: Implement CLI in your environment & USE IT!
Have a safe trip home.
Core Message
128
Thank You
Serviceability Tools
Automated Improvements
Troubleshooting
Smarter
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
Maximize your Cisco Live experience with your
free Cisco Live 365 account. Download session
PDFs, view sessions on-demand and participate in
live activities throughout the year. Click the Enter
Cisco Live 365 button in your Cisco Live portal to
log in.
Complete Your Online Session Evaluation
Give us your feedback and you could win fabulous prizes. Winners announced daily.
Receive 20 Cisco Daily Challenge points for each session evaluation you complete.
Complete your session evaluation online now through either the mobile app or internet kiosk stations.
129
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
Cisco Collaboration Community and User Group
Discuss strategy, market trends, IT architecture, product features, user adoption, and more with peers
contact centers, social software, telepresence, unified communications, and web conferencing
Interact with Cisco Product Managers, Technical Marketing Engineers and Services Consultants
Hear announcements and see demos of new solutions
Join the Collaboration User Group
– Influence product direction
– Participate in advisory groups & early product releases
– VIP perks & exclusive roadmap sessions at Cisco Live
Connect with Peers and Cisco
130
Visit Collaboration Booth - Cisco Campus:
Explore the Collaboration Community
Join the Collaboration User Group (CUG)
CUG Members: get gift ticket & badge ribbon
www.cisco.com/go/joinconversation
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
Multimedia Integration Combining Email, Chat, and Voice
133
High-level Flow
MR
PG
Agent
PG
GED-188 (CTI) ‒ Agent
availability
for Task
GED-125 (MR) ‒ New Task
‒ Do This With
Task
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
Multimedia Integration
Key Differences from Voice:
Email/Chat Media Routing Domains (MRD) A collection of skill groups and services associated with a specific communication channel. – UCCE Scripting and Media Routing Guide
Concurrency How many tasks can an agent handle in an MRD?
Interruptibility Can one task be interrupted by another task? Concept of “Available” vs. “ICM Available”
Combining Email, Chat, and Voice
134
Email MRD
Sales Support Billing
I.E. Agent “Available” for email but not “ICM Available”
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
Interruptibility & Handle Time
Problem Statement: Agent Productivity confusion.
Business Impact: Agents paid based on activity efficiency. Reported numbers are unclear to supervisors.
Versions:
– UCCE 8.5(3)
– EIM/WIM 4.4(1)
Troubleshooting Performed:
– Verified interruptibility settings for Chat and Email MRDs are correct.
– Verified concurrent task limits defined in EIM/WIM.
– Paid attention to this crash course!
Combining Email, Chat, and Voice
135
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
Interruptibility & Handle Time
136
Media Routing Domain Email Chat
Concurrent Task Limit 3 2
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
Interruptibility & Handle Time
137
Media Routing Domain Email Chat
Concurrent Task Limit 3 2
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
Interruptibility & Handle Time
138
Media Routing Domain Email Chat
Concurrent Task Limit 3 2
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
Interruptibility & Handle Time
139
Media Routing Domain Email Chat
Concurrent Task Limit 3 2
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
Interruptibility & Handle Time
140
Media Routing Domain Email Chat
Concurrent Task Limit 3 2
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
Interruptibility & Handle Time
141
Interrupted!
Media Routing Domain Email Chat
Concurrent Task Limit 3 2
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
Interruptibility & Handle Time
142
Media Routing Domain Email Chat
Concurrent Task Limit 3 2
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
Interruptibility & Handle Time
143
Interrupted!
Media Routing Domain Email Chat
Concurrent Task Limit 3 2
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
Interruptibility & Handle Time
144
Interrupted!
Media Routing Domain Email Chat
Concurrent Task Limit 3 2
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
Interruptibility & Handle Time How does UCCE see all of this?
145
External Agent Assignment Service MR PIM
MR PIM Finished!
MR
PG
Agent
PG
GED-125
18:38:19:693 pg2A-pim2 Trace: Application->PG:
Message = NEW_TASK; Length = 102 bytes
DialogueID = (10) Hex 0000000a
SendSeqNo = (1) Hex 00000001
MRDomainID = (5003) Hex 0000138b
PreviousTask = -1:-1:-1
PreferredAgent = Undefined
Service = (0) Hex 00000000
CiscoReserved = (0) Hex 00000000
ScriptSelector: 8000
ECC Variable Name: user.cim.activity.id
Value: 1372
18:38:19:693 pg2A-pim2 Trace: PG->Application:
Message = DO_THIS_WITH_TASK; Length = 121
bytes
DialogueID = (10) Hex 0000000a
SendSeqNo = (1) Hex 00000001
IcmTaskID = 150083:310: 1
SkillGroup = (5036) Hex 000013ac
Service = Undefined
Agent = (5003) Hex 0000138b
AgentInfo: 1003
Label:
ECC Variable Name: user.cim.activity.id
Value: 1372
Router CallKey
EIM/WIM AgentID
Agent STID
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
Interruptibility & Handle Time
146
Listener CTI Server
Pre-Call Indicator, Links The RouterCallKey & DialedNumber
Indicators are generally sent from an integrated application
– EXCEPT DEVICE_TARGET_PRE_CALL_IND which is sent from Router.
Events are calculations that occur within OPC for interval reporting & All Event Clients
Agent
PG
GED-188
18:38:19:693 cg1A-ctisvr Trace: DEVICE_TARGET_PRE_CALL_IND - Instrument= RouterCallKey=150083 310
NetworkTargetID=0
18:38:19:709 cg1A-ctisvr Trace: SvSkTargID=-1(-1) SkGroupSkTarID=5036(531)
18:38:19:709 cg1A-ctisvr Trace: AgSkTargID=5003 ANI= CED=
18:38:19:709 cg1A-ctisvr SESSION 2: MsgType:AGENT_PRE_CALL_EVENT (MonitorID:0 NumNamedVariables:2
NumNamedArrays:0 ServiceNumber:N/A
18:38:19:709 cg1A-ctisvr SESSION 2: ServiceID:N/A SkillGroupNumber:531 SkillGroupID:5036
SkillGroupPriority:0 MRDID:5003
18:38:19:709 cg1A-ctisvr SESSION 2: AgentInstrument:"" RouterCallKeyDay:150083 RouterCallKeyCallID:310
18:38:19:709 cg1A-ctisvr SESSION 2: RouterCallKeySequenceNum:1 DialedNumber:"8000" )
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
Interruptibility & Handle Time
147
Listener CTI Server
Email Offered & Started
Handle Time Starts Counting
Agent
PG
GED-188
18:38:19:724 cg1A-ctisvr SESSION 3: MsgType:OFFER_TASK_IND (InvokeID:0x2c767 MRDID:5003 ICMAgentID:5003
18:38:19:724 cg1A-ctisvr SESSION 3: TaskID:150083/310/1ARMSkillGroupID:5036 ARMServiceID:N/A )
18:38:19:724 cg1A-ctisvr Trace: OFFER_TASK_EVENT: PeripherlID=5000 PeripherlType=4 MRDomainID=5003
18:38:19:724 cg1A-ctisvr Trace: TaskID=150083:310:1 ICMAgentID=5003 SkillGroupID=5036
18:38:19:724 cg1A-ctisvr Trace: ServiceID=5617416 ApplicationRouted
18:38:25:537 cg1A-ctisvr SESSION 3: MsgType:START_TASK_IND (InvokeID:0x2c768 MRDID:5003 ICMAgentID:5003
18:38:25:537 cg1A-ctisvr SESSION 3: TaskID:150083/310/1ARMSkillGroupID:5036 ARMServiceID:N/A )
18:38:25:537 cg1A-ctisvr Trace: START_TASK_EVENT: PeripherlID=5000 PeripherlType=4 MRDomainID=5003
18:38:25:537 cg1A-ctisvr Trace: TaskID=150083:310:1 ICMAgentID=5003 SkillGroupID=5036
18:38:25:537 cg1A-ctisvr Trace: ServiceID=5617416 ApplicationRouted
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
Interruptibility & Handle Time
148
Listener CTI Server
Agent “Active” In Multimedia SG
Agent Can Still Receive A Second Task
– NumTasks=1, MaxTaskLimit=2
Agent
PG
GED-188
18:38:25:537 cg1A-ctisvr Trace: AGENT_EVENT: ID=21 Periph=5000 Ext= Inst= Sig=
18:38:25:537 cg1A-ctisvr Trace: SGState=ACTIVE SGDuration=0 OverallState=ACTIVE OverallDuration=0 Reason=-1
18:38:25:537 cg1A-ctisvr Trace: MRDID=5003 NumTasks=1 MaxTaskLimit=2 AgtMode=1 AgtAvailStatus=1
ICMAgentID=5003
18:38:25:537 cg1A-ctisvr Trace: SkTgtID=5003 SkGrpNo=0x213 SkGrpID=5036 NumLines=0 CurLine=-1
ClientStatus=0x0
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
Interruptibility & Handle Time
149
Listener CTI Server
Agent Gets Second Email (Pre_Call, Offer) and Reads It
– Handle Time “pauses” for original email and starts for new email
Second Email Completed, Original Email Resumed
Agent
PG
GED-188
18:40:33:632 cg1A-ctisvr Trace: PAUSE_TASK_EVENT: PeripherlID=5000 PeripherlType=4 MRDomainID=5003
18:40:33:632 cg1A-ctisvr Trace: TaskID=150083:310:1 ICMAgentID=5003 Reason=5000
18:40:33:647 cg1A-ctisvr Trace: START_TASK_EVENT: PeripherlID=5000 PeripherlType=4 MRDomainID=5003
18:40:33:647 cg1A-ctisvr Trace: TaskID=150083:311:1 ICMAgentID=5003 SkillGroupID=5036
18:40:33:647 cg1A-ctisvr Trace: ServiceID=5617416 ApplicationRouted
18:40:39:929 cg1A-ctisvr Trace: END_TASK_EVENT: PeripherlID=5000 PeripherlType=4 MRDomainID=5003
18:40:39:929 cg1A-ctisvr Trace: TaskID=150083:311:1 PreviousTaskID=-1:-1:-1
18:40:39:929 cg1A-ctisvr Trace: ICMAgentID=5003
18:40:42:788 cg1A-ctisvr Trace: RESUME_TASK_EVENT: PeripherlID=5000 PeripherlType=4 MRDomainID=5003
18:40:42:788 cg1A-ctisvr Trace: TaskID=150083:310:1 ICMAgentID=5003 Reason=5000
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
Interruptibility & Handle Time
150
Listener CTI Server
Original Email Interrupted By Chat
What are these MRDs? Where can we find their names? USE bxb_awdb select * from t_Media_Routing_Domain
Agent
PG
GED-188
18:40:49:710 cg1A-ctisvr SESSION 3: MsgType:AGENT_INTERRUPT_ADVISORY_EVENT (InvokeID:0x2 MRDID:5001
InterruptingMRDID:5002 ICMAgentID:5003 )
18:40:49:710 cg1A-ctisvr SESSION 3: MsgType:AGENT_INTERRUPT_ADVISORY_EVENT (InvokeID:0x2 MRDID:5003
InterruptingMRDID:5002 ICMAgentID:5003 )
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
Interruptibility & Handle Time
151
Listener CTI Server
Agent Reserved For Chat SG; Interrupted For Email SG
Offer, Start, End Task Indicators/Events and state transitions mimic Email
Note: Handle time for chat increments for all running sessions.
18:40:49:710 cg1A-ctisvr Trace: AGENT_EVENT: ID=21 Periph=5000 Ext= Inst= Sig=
18:40:49:710 cg1A-ctisvr Trace: SGState=RESERVED SGDuration=0 OverallState=RESERVED OverallDuration=0 Reason=-1
18:40:49:710 cg1A-ctisvr Trace: MRDID=5002 NumTasks=1 MaxTaskLimit=1 AgtMode=1 AgtAvailStatus=0 ICMAgentID=5003
18:40:49:710 cg1A-ctisvr Trace: SkTgtID=5003 SkGrpNo=0x21d SkGrpID=5039 NumLines=0 CurLine=-1 ClientStatus=0x0
18:40:49:725 cg1A-ctisvr SESSION 3: MsgType:AGENT_INTERRUPT_ACCEPTED_IND (InvokeID:0x2 MRDID:5001 ICMAgentID:5003 )
18:40:49:725 cg1A-ctisvr SESSION 3: MsgType:AGENT_INTERRUPT_ACCEPTED_IND (InvokeID:0x2 MRDID:5003 ICMAgentID:5003 )
18:40:49:725 cg1A-ctisvr Trace: AGENT_EVENT: ID=21 Periph=5000 Ext= Inst= Sig=
18:40:49:725 cg1A-ctisvr Trace: SGState=INTERRUPTED SGDuration=0 OverallState=INTERRUPTED OverallDuration=0 Reason=0
18:40:49:725 cg1A-ctisvr Trace: MRDID=5003 NumTasks=1 MaxTaskLimit=2 AgtMode=1 AgtAvailStatus=0 ICMAgentID=5003
18:40:49:725 cg1A-ctisvr Trace: SkTgtID=5003 SkGrpNo=0x213 SkGrpID=5036 NumLines=0 CurLine=-1 ClientStatus=0x0
Agent
PG
GED-188
MRD
ID
Media
5001 Outbound
5002 Chat
5003 Inbound
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
Interruptibility & Handle Time
152
How Can a Failure Event Be Mapped to a Task/Activity?
Use DialogueID to Correlate
Agent
PG
18:38:19:693 pg2A-pim2 Trace: Application->PG:
Message = NEW_TASK; Length = 102 bytes
DialogueID = (10) Hex 0000000a SendSeqNo = (1) Hex 00000001
MRDomainID = (5003) Hex 0000138b
PreviousTask = -1:-1:-1
PreferredAgent = Undefined
Service = (0) Hex 00000000
CiscoReserved = (0) Hex 00000000
ScriptSelector: 8000
ECC Variable Name: user.cim.activity.id
Value: 1372
18:38:19:693 pg2A-pim2 Trace: PG->Application:
Message = DO_THIS_WITH_TASK; Length = 121 bytes
DialogueID = (10) Hex 0000000a SendSeqNo = (1) Hex 00000001
IcmTaskID = 150083:310: 1
SkillGroup = (5036) Hex 000013ac
Service = Undefined
Agent = (5003) Hex 0000138b
AgentInfo: 1003
Label:
ECC Variable Name: user.cim.activity.id
Value: 1372
18:38:19:693 pg2A-pim2 Trace: PG->Application:
Message = NEW_TASK_FAILURE_EVENT; Length = 12 bytes
DialogueID = (10) Hex 0000000a SendSeqNo = (1) Hex 00000001
ReasonCode = (209) Hex 000000d1
MR
PG
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
Communicate, Collaborate, Succeed Resolving Problems with a Solution-based Approach
154
Accurately Confirming Critical Items Helps With…
Baseline(s)
Accurate troubleshooting
Improving feedback and collaboration
Reducing cycles, guess work, relying on luck, etc.
“Trust, but verify.” ~Ronald Reagan “What changes have you made?”
“None!”
“What version are you running?” “9.b(7c)”
“Who is affected?” “Everyone!”
“Did you look at the logs?” “Yep.”
“Did you enable tracing?” “Yes.”
“What and when did this happen?” “Don’t ask me.”
“It’s been 10 minutes. When are you going to fix this?!”
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
Communicate, Collaborate, Succeed
155
Clarify Problem
Description
Determine Possible Causes
Investigate
Test Likely Causes
Problem
Solution
Think
Beyond the
Fix…
15
5
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
Communicate, Collaborate, Succeed
156
Emotions
Politics
Fear
Technical
•Brokering Reasonable Expectations
•Fair and Balanced Approach
Situational Analysis
•Formulate Baseline
•Deliver Logs, Recordings, Screenshots
Technical Analysis
•Clear Expectations
•Formulate Next Steps, Workarounds Action Plan
•Alignment and Utilization Resource
Management
•Collaborate on Big Picture Solution
Approach
15
6
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
Communicate, Collaborate, Succeed
157
•Establishes respect
•Acts as a pressure valve
•Shows you care
Listen
•Ask questions to re-enforce what you were told
•Echo and summarize your understanding
Understand
•Log Validation
•Reproduce
•Test Cause & Effect
•Remember, “Trust, but verify.”
Check And Assess Action Plan
•Avoid transactional dialog
•“Tell me, I will forget. Show me, I may remember. Involve me and I will understand.”
Collaborate
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
Solution Problem Isolation
Problem Statement: Agents getting stuck in RESERVED state for 30 seconds, not receiving the call being routed to them.
Business Impact: Calls being rerouted introduce delays, exceed SLAs. This is affecting 20% of calls, causing financial penalties for the call center.
Versions:
– UCCE 8.5(3)
– CVP 8.5(1)
– CUCM 8.5(1)
Troubleshooting Performed: Isolated issue to single site.
What’s the call flow?
158
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
Solution Problem Isolation Narrowing Down The Problem
159
Call comes in to CVP from Voice Gateway
Caller requests to speak with an agent
VVVV
PG PG
Aug 30 2011 15:20:52.360 +0530: %CVP_8_0_SIP-7-CALL: {Thrd=DATAI.11} NEW CALL with guid=1A1935C8100001326975748653ED69C3 legid=8D03395A-D22311E0-9734A76F-421B7E24 dn=61401211 ani=+918946861782 uui=null calldate=Tue Aug 30 15:20:52 IST 2011 video=false
cachecallcontext = false is_postcallsurvey = false RouterCallKey = null RouterCallKeyDay = null RouterCallKeySequenceNumber = null
Aug 30 2011 15:26:12.031 +0530: %CVP_8_0_ICM-7-CALL: {Thrd=pool-1-thread-34-ICM-16843525} CALLGUID =
1A1935C8100001326975748653ED69C3, DLGID = 401702 [IVR_LEG] - Publishing ,, [ICM_RUN_SCRIPT_RESULT], dialogueId=401702,
sendSeqNo=3, invokedId=2, resultCode=true, ced=-1, newTransaction=false, CallContext:, user.microapp.caller_input: 1,
user.microapp.error_code: 0, user.microapp.metadata: N|000|01|00|00|019390|GS,Server,V, user.microapp.FromExtVXML[0]: +918946861782,
user.microapp.FromExtVXML[1]: hin,, LEGID = , DNIS = 10582222217939, ANI = sip:[email protected]:5060
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
Solution Problem Isolation Narrowing Down The Problem
160
Router sends a Connect message for extension 19797
CVP receives the Connect message
VVVV
PG PG
15:26:16:045 ra-rtr Trace: (401693 401702 17939 : 0 0) DeviceTargetPreCall_V11: CID=(149990,405494), PerID=5000, SGSTID=7037, AGSTID=8294,
NetworkCallFlags=0, InvokeID=1830299, GREET=
15:26:16:045 ra-rtr Trace: (401693 401702 17939 : 0 0) Connect: CID=(149990,405494), EventSelect=0, ServiceType=0, RCID=5005, Label(s)=19797
Aug 30 2011 15:26:15.922 +0530: %CVP_8_0_ICM-7-CALL: {Thrd=pool-1-thread-459-ICM-16843638} CALLGUID =
1A1935C8100001326975748653ED69C3, DLGID = 401693 [SIP_LEG] - Publishing ,, [MsgBus:CONNECT], ssId=SYS_SIP2, labelType=NORMAL,
label=19797, correlationId=null, command=null, rckey=405494, rcday=149990, rcseq=0, CallContext:, CallVariable1: 8946861782,
CallVariable2: NA|NA, CallVariable3: 1A1935C8100001326975748653ED69C3, etc...…..
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
15:26:16:045 ra-rtr Trace: (401693 401702 17939 : 0 0) DeviceTargetPreCall_V11: CID=(149990,405494), PerID=5000, SGSTID=7037, AGSTID=8294,
NetworkCallFlags=0, InvokeID=1830299, GREET=
15:26:16:045 ra-rtr Trace: (401693 401702 17939 : 0 0) Connect: CID=(149990,405494), EventSelect=0, ServiceType=0, RCID=5005, Label(s)=19797
Solution Problem Isolation Narrowing Down The Problem
161
Router sends a Connect message for extension 19797
CVP receives the Connect message
VVVV
PG PG
15:26:16:045 ra-rtr Trace: (401693 401702 17939 : 0 0) DeviceTargetPreCall_V11: CID=(149990,405494), PerID=5000, SGSTID=7037, AGSTID=8294,
NetworkCallFlags=0, InvokeID=1830299, GREET=
15:26:16:045 ra-rtr Trace: (401693 401702 17939 : 0 0) Connect: CID=(149990,405494), EventSelect=0, ServiceType=0, RCID=5005, Label(s)=19797
Aug 30 2011 15:26:15.922 +0530: %CVP_8_0_ICM-7-CALL: {Thrd=pool-1-thread-459-ICM-16843638} CALLGUID =
1A1935C8100001326975748653ED69C3, DLGID = 401693 [SIP_LEG] - Publishing ,, [MsgBus:CONNECT], ssId=SYS_SIP2, labelType=NORMAL,
label=19797, correlationId=null, command=null, rckey=405494, rcday=149990, rcseq=0, CallContext:, CallVariable1: 8946861782,
CallVariable2: NA|NA, CallVariable3: 1A1935C8100001326975748653ED69C3, etc...…..
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
Solution Problem Isolation Narrowing Down The Problem
162
Use InvokeID or RCK to match PreCall message in CTI Server logs
Agent was AVAILABLE in call’s destination SG just before this PreCall
VVVV
PG PG
15:26:15:723 cg1B-ctisvr Trace: DEVICE_TARGET_PRE_CALL_IND - Instrument=19797 RouterCallKey=149990 405494 15:26:15:723 cg1B-ctisvr Trace: SvSkTargID=-1(-1) SkGroupSkTarID=7037(617)
15:26:15:723 cg1B-ctisvr Trace: AgSkTargID=8294 ANI=+918946861782 CED=-1
15:26:15:723 cg1B-ctisvr Trace: CallTypeID=5535 PreCallInvokeID=1830299
15:26:15:504 cg1B-ctisvr Trace: AGENT_EVENT: ID=20415 Periph=5000 Ext=19797 Inst=19797 Sig=CTIOSServer
15:26:15:504 cg1B-ctisvr Trace: SkgState=AVAILABLE SkgDuration=0 OverallState=AVAILABLE OverallDuration=0 Reason=0
15:26:15:504 cg1B-ctisvr Trace: MRDID=1 NumTasks=0 MaxTaskLimit=1 AgtMode=1 AGTAvailabilityStatus=1 ICMAgtID=8294
15:26:15:504 cg1B-ctisvr Trace: SkTgtID=8294 SkGrpNo=0x269 SkGrpID=7037 NumLines=1 CurLine=0 ClientStatus=0x1
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
Solution Problem Isolation Narrowing Down The Problem
163
Use InvokeID or RCK to match PreCall message in CTI Server logs
Agent was RESERVED in call’s destination SG concurrent with PreCall
VVVV
PG PG
15:26:15:723 cg1B-ctisvr Trace: DEVICE_TARGET_PRE_CALL_IND - Instrument=19797 RouterCallKey=149990 405494 15:26:15:723 cg1B-ctisvr Trace: SvSkTargID=-1(-1) SkGroupSkTarID=7037(617)
15:26:15:723 cg1B-ctisvr Trace: AgSkTargID=8294 ANI=+918946861782 CED=-1
15:26:15:723 cg1B-ctisvr Trace: CallTypeID=5535 PreCallInvokeID=1830299
15:26:15:723 cg1B-ctisvr Trace: AGENT_EVENT: ID=20415 Periph=5000 Ext=19797 Inst=19797 Sig=CTIOSServer
15:26:15:723 cg1B-ctisvr Trace: SkgState=RESERVED SkgDuration=0 OverallState=RESERVED OverallDuration=0 Reason=0
15:26:15:723 cg1B-ctisvr Trace: MRDID=1 NumTasks=1 MaxTaskLimit=1 AgtMode=1 AGTAvailabilityStatus=0 ICMAgtID=8294
15:26:15:723 cg1B-ctisvr Trace: SkTgtID=8294 SkGrpNo=0x269 SkGrpID=7037 NumLines=1 CurLine=0 ClientStatus=0x1
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
Solution Problem Isolation Narrowing Down The Problem
164
UCCE and CUCM notify CVP that call has been Answered
But the Agent PG is never notified that this call arrived at device 19797.
What does a normal call look like? What do we expect to see?
VVVV
PG PG
Aug 30 2011 15:26:16.172 +0530: %CVP_8_0_ICM-7-CALL: {Thrd=pool-1-thread-284-ICM-16843644} CALLGUID =
1A1935C8100001326975748653ED69C3, DLGID = 401693 [SIP_LEG] - Publishing ,, [ICM_EVENT_REPORT], dialogueId=401693, sendSeqNo=3,
eventId=ANSWER, causeCode=NONE,, LEGID = 8D03395A-D22311E0-9734A76F-421B7E24, DNIS = 61401211, ANI = +918946861782
Aug 30 2011 15:26:16.172 +0530: %_UserCB-6-com.dynamicsoft.DsLibs.DsUALibs.DsSipLlApi.LlSM.client.UserCB: SIP/2.0 200 OK Via: SIP/2.0/UDP 172.18.150.61:5060;branch=z9hG4bKI5Y12NeaMEV6dImqSWb3TA~~8933838
To: "--CVP_8_0_1_0_0_0_1440" <sip:[email protected]>;tag=A269BB6C-77D
From UCCE
From CUCM
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
Solution Problem Isolation Narrowing Down The Problem
165
For a NORMAL call, we should see after the Reservation:
– CSTA_DELIVERED
– CSTA_ESTABLISHED
– RTP_STARTED / RTP_STOPPED
– CSTA_CONNECTION_CLEARED
– Agent transition to NOT_READY state (or wrap-up etc)
None of these messages are seen for the call in question.
VVVV
PG PG
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
Solution Problem Isolation Narrowing Down The Problem
166
PGs internal timer expires in 30 seconds, reverting agent to AVAILABLE
Agent Reserved, PG waiting, CVP Answered. So what’s left?
VVVV
PG PG
15:26:45:945 PG1A-pim1 Trace: RemovePreRoutedCall: Head Inst: N19797 CreatedByPreCall: T PeriphCID: -1 Queue Count: 0
RouterCallKey=(149990/405494)
15:26:45:945 PG1A-pim1 Trace: ProcessReservedTimeout: No call arrived to match PreCall message. netTargID=100780 Ext=19797
AgentID=20415 RTRCallKey=149990.405494 DtAborts=1
15:26:45:945 PG1A-pim1 Trace: SendDeviceTargetAbortInd: agentSkillTargetID = 8294
15:26:45:945 PG1A-pim1 Trace: SetPrevalentAgentState: (Agent 20415 ext 19797) newAgentState=AS_AVAILABLE(3) reasoncode=0
stateDuration=0 currentLineHandle=-1 lineType=LT_UNKNOWN(3) newLineState=LS_UNKNOWN(5) devString=
15:26:45:945 PG1A-pim1 Trace: SetPrevalentState: newState=AS_AVAILABLE(3) reasoncode=0
15:26:45:945 PG1A-pim1 Trace: CancelAgentTimer: type = agentTimerReserved
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
Solution Problem Isolation Narrowing Down The Problem
167
CUCM logs show agent going off-hook and on-hook repeatedly.
CUCM rejects routed call while off-hook
VVVV
PG PG
15:26:15.135 |StationInit: (0002392) SoftKeyEvent softKeyEvent=2(NewCall) lineInstance=1 callReference=0.|6,100,50,1.4164601^
15:26:16.252 |StationInit: (0002392) SoftKeyEvent softKeyEvent=9(EndCall) lineInstance=1 callReference=110085009.|6,100,50,1.4164606^
15:26:17.372 |StationInit: (0002392) SoftKeyEvent softKeyEvent=2(NewCall) lineInstance=1 callReference=0.|6,100,50,1.4164651^
15:26:18.490 |StationInit: (0002392) SoftKeyEvent softKeyEvent=9(EndCall) lineInstance=1 callReference=110085041.|6,100,50,1.4164655^
15:26:19.522 |StationInit: (0002392) SoftKeyEvent softKeyEvent=2(NewCall) lineInstance=1 callReference=0.|6,100,50,1.4164719^
15:26:20.609 |StationInit: (0002392) SoftKeyEvent softKeyEvent=9(EndCall) lineInstance=1 callReference=110085084.|6,100,50,1.4164725^
15:26:17.740 |LineControl(2351) - 1 calls, 0 CiReq, busyTrigger=1, maxCall=2|6,100,57,1.399447^172.18.150.120^*
15:26:17.740 |//SIP/SIPTcp/wait_SdlSPISignal: Outgoing SIP TCP message to 172.18.150.120 on port 37776 index 13 [1449373,NET]
SIP/2.0 486 Busy here
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
15:26:15.135 |StationInit: (0002392) SoftKeyEvent softKeyEvent=2(NewCall) lineInstance=1 callReference=0.|6,100,50,1.4164601^
15:26:16.252 |StationInit: (0002392) SoftKeyEvent softKeyEvent=9(EndCall) lineInstance=1 callReference=110085009.|6,100,50,1.4164606^
15:26:17.372 |StationInit: (0002392) SoftKeyEvent softKeyEvent=2(NewCall) lineInstance=1 callReference=0.|6,100,50,1.4164651^
15:26:18.490 |StationInit: (0002392) SoftKeyEvent softKeyEvent=9(EndCall) lineInstance=1 callReference=110085041.|6,100,50,1.4164655^
15:26:19.522 |StationInit: (0002392) SoftKeyEvent softKeyEvent=2(NewCall) lineInstance=1 callReference=0.|6,100,50,1.4164719^
15:26:20.609 |StationInit: (0002392) SoftKeyEvent softKeyEvent=9(EndCall) lineInstance=1 callReference=110085084.|6,100,50,1.4164725^
Solution Problem Isolation Narrowing Down The Problem
168
CUCM logs show agent going off-hook and on-hook repeatedly.
CUCM rejects routed call while off-hook
VVVV
PG PG
15:26:17.740 |LineControl(2351) - 1 calls, 0 CiReq, busyTrigger=1, maxCall=2|6,100,57,1.399447^172.18.150.120^*
15:26:17.740 |//SIP/SIPTcp/wait_SdlSPISignal: Outgoing SIP TCP message to 172.18.150.120 on port 37776 index 13 [1449373,NET] SIP/2.0
486 Busy here
Hit the Floor!
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
15:26:15.135 |StationInit: (0002392) SoftKeyEvent softKeyEvent=2(NewCall) lineInstance=1 callReference=0.|6,100,50,1.4164601^
15:26:16.252 |StationInit: (0002392) SoftKeyEvent softKeyEvent=9(EndCall) lineInstance=1 callReference=110085009.|6,100,50,1.4164606^
15:26:17.372 |StationInit: (0002392) SoftKeyEvent softKeyEvent=2(NewCall) lineInstance=1 callReference=0.|6,100,50,1.4164651^
15:26:18.490 |StationInit: (0002392) SoftKeyEvent softKeyEvent=9(EndCall) lineInstance=1 callReference=110085041.|6,100,50,1.4164655^
15:26:19.522 |StationInit: (0002392) SoftKeyEvent softKeyEvent=2(NewCall) lineInstance=1 callReference=0.|6,100,50,1.4164719^
15:26:20.609 |StationInit: (0002392) SoftKeyEvent softKeyEvent=9(EndCall) lineInstance=1 callReference=110085084.|6,100,50,1.4164725^
Solution Problem Isolation Narrowing Down The Problem
169
CUCM logs show agent going off-hook and on-hook repeatedly.
CUCM rejects routed call while off-hook
VVVV
PG PG
15:26:17.740 |LineControl(2351) - 1 calls, 0 CiReq, busyTrigger=1, maxCall=2|6,100,57,1.399447^172.18.150.120^*
15:26:17.740 |//SIP/SIPTcp/wait_SdlSPISignal: Outgoing SIP TCP message to 172.18.150.120 on port 37776 index 13 [1449373,NET] SIP/2.0
486 Busy here
Hit the Floor!
© 2013 Cisco and/or its affiliates. All rights reserved. BRKCCT-3005 Cisco Public
Solution Problem Isolation Narrowing Down The Problem
170
CUCM logs show agent going off-hook and on-hook repeatedly.
– Agents do not have enough time in between calls
– Agent Desk Settings issue, possible staffing issue
ACTION PLAN:
– After-Call Wait Time should be increased to meet agent needs.
– Call volume should be evaluated to align with agent staffing.
VVVV
PG PG