90
Send document comments to [email protected] Americas Headquarters Cisco Systems, Inc. 170 West Tasman Drive San Jose, CA 95134-1706 USA http://www.cisco.com Tel: 408 526-4000 800 553-NETS (6387) Fax: 408 527-0883 Cisco UCS Troubleshooting Guide June 25, 2009 Revised August 24, 2010 Text Part Number: OL-20137-04

Cisco UCS Troubleshooting Guide

Embed Size (px)

Citation preview

Page 1: Cisco UCS Troubleshooting Guide

Send document comments to ucs -doc feedback@c i sco .com

Cisco UCS Troubleshooting GuideJune 25, 2009 Revised August 24, 2010

Americas HeadquartersCisco Systems, Inc.170 West Tasman DriveSan Jose, CA 95134-1706 USAhttp://www.cisco.comTel: 408 526-4000

800 553-NETS (6387)Fax: 408 527-0883

Text Part Number: OL-20137-04

Page 2: Cisco UCS Troubleshooting Guide

Send document comments to ucs -doc feedback@c i sco .com

THE SPECIFICATIONS AND INFORMATION REGARDING THE PRODUCTS IN THIS MANUAL ARE SUBJECT TO CHANGE WITHOUT NOTICE. ALL STATEMENTS, INFORMATION, AND RECOMMENDATIONS IN THIS MANUAL ARE BELIEVED TO BE ACCURATE BUT ARE PRESENTED WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED. USERS MUST TAKE FULL RESPONSIBILITY FOR THEIR APPLICATION OF ANY PRODUCTS.

THE SOFTWARE LICENSE AND LIMITED WARRANTY FOR THE ACCOMPANYING PRODUCT ARE SET FORTH IN THE INFORMATION PACKET THAT SHIPPED WITH THE PRODUCT AND ARE INCORPORATED HEREIN BY THIS REFERENCE. IF YOU ARE UNABLE TO LOCATE THE SOFTWARE LICENSE OR LIMITED WARRANTY, CONTACT YOUR CISCO REPRESENTATIVE FOR A COPY.

The Cisco implementation of TCP header compression is an adaptation of a program developed by the University of California, Berkeley (UCB) as part of UCB’s public domain version of the UNIX operating system. All rights reserved. Copyright © 1981, Regents of the University of California.

NOTWITHSTANDING ANY OTHER WARRANTY HEREIN, ALL DOCUMENT FILES AND SOFTWARE OF THESE SUPPLIERS ARE PROVIDED “AS IS” WITH ALL FAULTS. CISCO AND THE ABOVE-NAMED SUPPLIERS DISCLAIM ALL WARRANTIES, EXPRESSED OR IMPLIED, INCLUDING, WITHOUT LIMITATION, THOSE OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OR ARISING FROM A COURSE OF DEALING, USAGE, OR TRADE PRACTICE.

IN NO EVENT SHALL CISCO OR ITS SUPPLIERS BE LIABLE FOR ANY INDIRECT, SPECIAL, CONSEQUENTIAL, OR INCIDENTAL DAMAGES, INCLUDING, WITHOUT LIMITATION, LOST PROFITS OR LOSS OR DAMAGE TO DATA ARISING OUT OF THE USE OR INABILITY TO USE THIS MANUAL, EVEN IF CISCO OR ITS SUPPLIERS HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.

Cisco and the Cisco Logo are trademarks of Cisco Systems, Inc. and/or its affiliates in the U.S. and other countries. A listing of Cisco's trademarks can be found at www.cisco.com/go/trademarks. Third party trademarks mentioned are the property of their respective owners. The use of the word partner does not imply a partnership relationship between Cisco and any other company. (1005R)

Any Internet Protocol (IP) addresses and phone numbers used in this document are not intended to be actual addresses and phone numbers. Any examples, command display output, network topology diagrams, and other figures included in the document are shown for illustrative purposes only. Any use of actual IP addresses or phone numbers in illustrative content is unintentional and coincidental.

Cisco UCS Troubleshooting Guide © 2010 Cisco Systems, Inc. All rights reserved.

Page 3: Cisco UCS Troubleshooting Guide

OL-20137-04

C O N T E N T S

Preface v

Audience v

Organization v

Related Documentation v

Obtaining Documentation, Obtaining Support, and Security Guidelines vi

Introduction 1-1

Troubleshooting Information in Cisco UCS Manager GUI 1-1

Troubleshooting Information in Cisco UCS Manager CLI 1-2

Faults 1-2

Fault Severities 1-3

Fault States 1-3

Fault Types 1-4

Properties of Faults 1-4

Lifecycle of Faults 1-5

Faults in Cisco UCS Manager GUI 1-5

Faults in Cisco UCS Manager CLI 1-6

Fault Collection Policy 1-6

Fault Documentation 1-6

Events 1-6

Properties of Events 1-7

Events in Cisco UCS Manager GUI 1-7

Events in Cisco UCS Manager CLI 1-7

Core Files 1-7

Core Files in Cisco UCS Manager GUI 1-8

Core Files in Cisco UCS Manager CLI 1-8

Core File Exporter 1-8

Audit Log 1-8

Properties of Audit Log Entries 1-9

Audit Log in Cisco UCS Manager GUI 1-9

Audit Log in Cisco UCS Manager CLI 1-9

System Event Log 1-10

SEL File 1-10

SEL Policy 1-10

iCisco UCS Troubleshooting Guide

Page 4: Cisco UCS Troubleshooting Guide

Contents

Syslog 1-10

Syslog Configuration 1-11

Syslog Location 1-11

Syslog Entry Format 1-11

Syslog Entry Severities 1-11

Syslog Entry Parameters 1-12

Syslog Services 1-13

Finite State Machine 2-1

Overview of the FSM 2-1

FSM Stage Names 2-2

FSM in Cisco UCS Manager GUI 2-2

FSM in Cisco UCS Manager CLI 2-3

General Troubleshooting Steps and Solutions 3-1

Guidelines for Troubleshooting 3-1

Gathering Information about a Cisco UCS Instance 3-2

Executing the show tech-support Command 3-2

Troubleshooting Issues with Cisco UCS 4-1

Troubleshooting Boot Issues 4-1

Reboot Warning Does Not Display 4-1

Server Does Not Boot from OS Installed on eUSB 4-2

Troubleshooting KVM Issues 4-3

BadFieldException When Launching KVM Viewer 4-3

KVM Console Failure 4-4

KVM Fails to Open 4-4

Troubleshooting VM Issues 4-5

Currently connected network interface x uses Distributed Virtual Switch (uusid: y) which is accessed on the host via a switch that has no free ports 4-5

Troubleshooting Cisco UCS Manager Issues 4-6

Fatal error: event sequencing is skewed 4-6

HDD Metrics Not Updated in Cisco UCS Manager GUI 4-7

Cisco UCS Manager Reports More Disks in Server than Total Slots Available 4-7

Troubleshooting Fabric Interconnect Issues 4-8

Recovering a Fabric Interconnect from the Boot Loader Prompt 4-9

Troubleshooting Cisco UCS Faults 5-1

Troubleshooting SAN Boot and SAN Connectivity Issues 6-1

SAN Connectivity Checklist 6-1

iiCisco UCS Troubleshooting Guide

OL-20137-04

Page 5: Cisco UCS Troubleshooting Guide

Contents

SAN Array Configuration Checklist 6-2

Recommended Solutions for Issues During SAN Boot 6-2

Troubleshooting Server Hardware Issues 7-1

Diagnostics Button and LEDs 7-1

DIMM Memory Issues 7-2

Known Issues 7-2

Types of DIMM Errors 7-3

Troubleshooting DIMM Errors 7-4

Recommended Solutions for DIMM Issues 7-6

CPU Issues 7-7

Troubleshooting CPU Issues 7-8

Recommended Solutions for CPU Issues 7-8

Disk Drive and RAID Issues 7-9

How to Determine Which RAID Controller Is in Your Server 7-9

How to Disable Quiet Boot 7-10

How To Launch Option ROM-Based Controller Utilities 7-11

For More Information 7-11

Moving a RAID Cluster 7-11

Adapter Issues 7-12

Known Issues 7-13

Troubleshooting Adapter Errors 7-13

Recommended Solutions 7-14

Gathering Information Before Calling Support 7-14

Related Documents 7-15

Cisco UCS Error Messages 8-1

Overview 8-1

Information and Warning Messages 8-2

FSM Messages 8-19

I N D E X

iiiCisco UCS Troubleshooting Guide

OL-20137-04

Page 6: Cisco UCS Troubleshooting Guide

Contents

ivCisco UCS Troubleshooting Guide

OL-20137-04

Page 7: Cisco UCS Troubleshooting Guide

Send document comments to ucs -doc feedback@c i sco .com

Preface

This preface describes the audience and organization of the Cisco UCS Manager Troubleshooting Guide. It also provides information on how to obtain related documentation.

AudienceThis troubleshooting guide is designed for users who are responsible for the following:

• Implementing Cisco UCS instances

• Managing Cisco UCS instances

OrganizationThis troubleshooting guide is organized into the following chapters:

• Chapter 1, “Introduction”

• Chapter 2, “Finite State Machine”

• Chapter 3, “General Troubleshooting Steps and Solutions”

• Chapter 4, “Troubleshooting Issues with Cisco UCS”

• Chapter 5, “Troubleshooting Cisco UCS Faults”

• Chapter 6, “Troubleshooting SAN Boot and SAN Connectivity Issues”

• Chapter 7, “Troubleshooting Server Hardware Issues”

• Chapter 8, “Cisco UCS Error Messages”

Related DocumentationRoadmaps that list all documentation for Cisco Unified Computing System (Cisco UCS) are available at the following URL:

http://www.cisco.com/en/US/products/ps10477/products_documentation_roadmaps_list.html

vCisco UCS Troubleshooting Guide

OL-20137-04

Page 8: Cisco UCS Troubleshooting Guide

Send document comments to ucs -doc feedback@c i sco .com

Preface

Obtaining Documentation, Obtaining Support, and Security Guidelines

For information on obtaining documentation, obtaining support, providing documentation feedback, security guidelines, and also recommended aliases and general Cisco documents, see the monthly What’s New in Cisco Product Documentation, which also lists all new and revised Cisco technical documentation, at:

http://www.cisco.com/en/US/docs/general/whatsnew/whatsnew.html

viCisco UCS Troubleshooting Guide

OL-20137-04

Page 9: Cisco UCS Troubleshooting Guide

Send document comments to ucs -doc feedback@c i sco .com

OL-20137-04

C H A P T E R 1

Introduction

This chapter provides an overview of where to find faults, events, and other information that can help you troubleshoot issues with Cisco Unified Computing System (Cisco UCS).

This chapter contains the following sections:

• Troubleshooting Information in Cisco UCS Manager GUI, page 1-1

• Troubleshooting Information in Cisco UCS Manager CLI, page 1-2

• Faults, page 1-2

• Events, page 1-6

• Core Files, page 1-7

• Audit Log, page 1-8

• System Event Log, page 1-10

• Syslog, page 1-10

Troubleshooting Information in Cisco UCS Manager GUICisco UCS Manager GUI provides several tabs and other areas that you can use to find troubleshooting information for a Cisco UCS instance. For example, you can view faults and events for specific objects or for all objects in the system.

The Admin tab in the Navigation pane provides access to faults, events, core files, and other information that can help you troubleshoot issues.

If you select Faults, Events and Audit Log in the Filter field on the Admin tab, Cisco UCS Manager GUI limits the tree browser so that you can only access the following:

• The faults for all components in the system

• The events for all components in the system

• The audit log for the system

• Any core files created by the fabric interconnects in the system

• The fault collection and core file export settings

1-1Cisco UCS Troubleshooting Guide

Page 10: Cisco UCS Troubleshooting Guide

Send document comments to ucs -doc feedback@c i sco .com

Chapter 1 IntroductionTroubleshooting Information in Cisco UCS Manager CLI

Troubleshooting Information in Cisco UCS Manager CLICisco UCS Manager CLI includes several show commands that you can execute to find troubleshooting information for a Cisco UCS instance. These show commands are scope-aware. For example, if you execute a show fault command from the top scope, it displays all faults in the system. However, if you scope to a specific object, the show fault command only displays faults related to that object.

FaultsIn Cisco UCS, a fault is a mutable object that is managed by Cisco UCS Manager. Each fault represents a failure in the Cisco UCS instance or an alarm threshold that has been raised. During the lifecycle of a fault, it can change from one state or severity to another.

Each fault includes information about the operational state of the affected object at the time the fault was raised. If the fault is transitional and the failure is resolved, the object transitions to a functional state.

A fault remains in Cisco UCS Manager until the fault is cleared and deleted according to the settings in the fault collection policy.

You can view all faults in a Cisco UCS instance from either Cisco UCS Manager CLI or Cisco UCS Manager GUI. You can also configure the fault collection policy to determine how a Cisco UCS instance collects and retains faults.

Note All Cisco UCS faults can be trapped by SNMP.

This section includes the following topics:

• Fault Severities, page 1-3

• Fault States, page 1-3

• Properties of Faults, page 1-4

• Lifecycle of Faults, page 1-5

• Faults in Cisco UCS Manager GUI, page 1-5

• Faults in Cisco UCS Manager CLI, page 1-6

• Fault Collection Policy, page 1-6

• Fault Documentation, page 1-6

1-2Cisco UCS Troubleshooting Guide

OL-20137-04

Page 11: Cisco UCS Troubleshooting Guide

Send document comments to ucs -doc feedback@c i sco .com

Chapter 1 IntroductionFaults

Fault SeveritiesA fault raised in a Cisco UCS instance can transition through more than one severity during its lifecycle. Table 1-1 describes the fault severities that you may encounter.

Fault StatesA fault raised in a Cisco UCS instance transitions through more than one state during its lifecycle. Table 1-2 describes the possible fault states in alphabetical order.

Table 1-1 Fault Severities in Cisco UCS

Severity Description

Critical A service-affecting condition that requires immediate corrective action. For example, this severity could indicate that the managed object is out of service and its capability must be restored.

Major A service-affecting condition that requires urgent corrective action. For example, this severity could indicate a severe degradation in the capability of the managed object and that its full capability must be restored.

Minor A nonservice-affecting fault condition that requires corrective action to prevent a more serious fault from occurring. For example, this severity could indicate that the detected alarm condition is not degrading the capacity of the managed object.

Warning A potential or impending service-affecting fault that has no significant effects in the system. Action should be taken to further diagnose, if necessary, and correct the problem to prevent it from becoming a more serious service-affecting fault.

Condition An informational message about a condition, possibly independently insignificant.

Info A basic notification or informational message, possibly independently insignificant.

Table 1-2 Fault States in Cisco UCS

State Description

Cleared The condition has been resolved and the fault has been cleared.

Flapping A fault was raised, cleared, and then raised again within a short time interval, known as the flap interval.

Soaking A fault was raised and then cleared within a short time interval, known as the flap interval. Since this may be a flapping condition, the fault severity remains at its original active value, but this state indicates the condition that raised the fault has cleared.

1-3Cisco UCS Troubleshooting Guide

OL-20137-04

Page 12: Cisco UCS Troubleshooting Guide

Send document comments to ucs -doc feedback@c i sco .com

Chapter 1 IntroductionFaults

Fault TypesA fault raised in a Cisco UCS instance can be one of the types described in Table 1-3.

Properties of FaultsCisco UCS Manager provides detailed information about each fault raised in a Cisco UCS instance. Table 1-4 describes the fault properties that you can view in Cisco UCS Manager CLI or Cisco UCS Manager GUI.

Table 1-3 Types of Faults in Cisco UCS

Type Description

fsm An FSM task has failed to complete successfully, or Cisco UCS Manager is retrying one of the stages of the FSM.

equipment Cisco UCS Manager has detected that a physical component is inoperable or has another functional issue.

server Cisco UCS Manager cannot complete a server task, such as associating a service profile with a server.

configuration Cisco UCS Manager cannot successfully configure a component.

environment Cisco UCS Manager has detected a power problem, thermal problem, voltage problem, or loss of CMOS settings.

management Cisco UCS Manager has detected a serious management issue, such as one of the following:

• Critical services could not be started

• The primary switch could not be identified

• Components in the instance includes incompatible firmware versions

connectivity Cisco UCS Manager has detected a connectivity problem, such as an unreachable adapter.

network Cisco UCS Manager has detected a network issue, such as a link down.

operational Cisco UCS Manager has detected an operational problem, such as a log capacity issue or a failed server discovery.

Table 1-4 Fault Properties

Property Name Description

Severity The current severity level of the fault. This can be any of the severities described in Table 1-1 on page 1-3.

Last Transition The day and time on which the severity for the fault last changed. If the severity has not changed since the fault was raised, this property displays the original creation date.

Affected Object The component that is affected by the condition that raised the fault.

Description A description of the fault.

1-4Cisco UCS Troubleshooting Guide

OL-20137-04

Page 13: Cisco UCS Troubleshooting Guide

Send document comments to ucs -doc feedback@c i sco .com

Chapter 1 IntroductionFaults

Lifecycle of FaultsFaults in Cisco UCS are stateful. Only one instance of a given fault can exist on each object. If the same fault occurs a second time, Cisco UCS increases the number of occurrences by one.

A fault has the following lifecycle:

1. A condition occurs in the system and Cisco UCS raises a fault. This is the active state.

2. When the fault is alleviated, it enters a flapping or soaking interval that is designed to prevent flapping. Flapping occurs when a fault is raised and cleared several times in rapid succession. During the flapping interval the fault retains its severity for the length of time specified in the fault collection policy.

3. If the condition re-occurs during the flapping interval, the fault returns to the active state. If the condition does not reoccur during the flapping interval, the fault is cleared.

4. The cleared fault enters the retention interval. This interval ensures that the fault reaches the attention of an administrator, even if the condition that caused the fault has been alleviated, and that the fault is not deleted prematurely. The retention interval retains the cleared fault for the length of time specified in the fault collection policy.

5. If the condition reoccurs during the retention interval, the fault returns to the active state. If the condition does not reoccur, the fault is deleted.

Faults in Cisco UCS Manager GUIIf you want to view faults for a single object in the system, navigate to that object in Cisco UCS Manager GUI and click the Faults tab in the Work pane. If you want to view faults for all objects in the system, navigate to the Faults node on the Admin tab under Faults, Events and Audit Log.

In addition, you can also view a summary of all faults in a Cisco UCS instance in the Fault Summary area in the upper left of Cisco UCS Manager GUI. This area provides a summary of all faults that have occurred in the Cisco UCS instance.

ID The unique identifier assigned to the fault.

Type The type of fault that has been raised. This can be any of the types described in Table 1-3 on page 1-4.

Cause The unique identifier associated with the condition that caused the fault.

Created at The day and time when the fault occurred.

Code The unique identifier assigned to the fault.

Number of Occurrences The number of times the event that raised the fault occurred.

Original Severity The severity assigned to the fault the first time it occurred.

Previous Severity If the severity has changed, this is the previous severity.

Highest Severity The highest severity encountered for this issue.

Table 1-4 Fault Properties (continued)

Property Name Description

1-5Cisco UCS Troubleshooting Guide

OL-20137-04

Page 14: Cisco UCS Troubleshooting Guide

Send document comments to ucs -doc feedback@c i sco .com

Chapter 1 IntroductionEvents

Each fault severity is represented by a different icon. The number below each icon indicates how many faults of that severity have occurred in the system. If you click an icon, Cisco UCS Manager GUI opens the Faults tab in the Work area and displays the details of all faults of that severity.

Faults in Cisco UCS Manager CLIIf you want to view faults for all objects in the system, execute the show fault command from the top-level scope. If you want to view faults for a specific object, scope to that object and then execute the show fault command.

If you want to view all available details about a fault, execute the show fault detail command.

Fault Collection PolicyThe fault collection policy controls the lifecycle of a fault in the Cisco UCS instance, including the length of time that each fault remains in the flapping and retention intervals.

Tip For information on how to configure the fault collection policy, see the Cisco UCS configuration guides, accessible through the Cisco UCS B-Series Servers Documentation Roadmap.

Fault DocumentationDocumentation about Cisco UCS faults, including explanations and recommended actions, is available in the Cisco UCS Faults Reference.

EventsIn Cisco UCS, an event is an immutable object that is managed by Cisco UCS Manager. Each event represents a non-persistent condition in the Cisco UCS instance. After Cisco UCS Manager creates and logs an event, the event does not change. For example, if you power on a server, Cisco UCS Manager creates and logs an event for the beginning and the end of that request.

You can view events for a single object, or you can view all events in a Cisco UCS instance from either the Cisco UCS Manager CLI or the Cisco UCS Manager GUI. Events remain in Cisco UCS until the event log fills up. When the log is full, Cisco UCS Manager purges the log and all events in it.

This section includes the following topics:

• Properties of Events, page 1-7

• Events in Cisco UCS Manager GUI, page 1-7

• Events in Cisco UCS Manager CLI, page 1-7

1-6Cisco UCS Troubleshooting Guide

OL-20137-04

Page 15: Cisco UCS Troubleshooting Guide

Send document comments to ucs -doc feedback@c i sco .com

Chapter 1 IntroductionCore Files

Properties of EventsCisco UCS Manager provides detailed information about each event created and logged in a Cisco UCS instance. Table 1-5 describes the fault properties that you can view in Cisco UCS Manager CLI or Cisco UCS Manager GUI.

Events in Cisco UCS Manager GUIIf you want to view events for a single object in the system, navigate to that object in Cisco UCS Manager GUI and click the Events tab in the Work pane. If you want to view events for all objects in the system, navigate to the Events node on the Admin tab under Faults, Events and Audit Log.

Events in Cisco UCS Manager CLIIf you want to view events for all objects in the system, execute the show event command from the top-level scope. If you want to view events for a specific object, scope to that object and then execute the show event command.

If you want to view all available details about an event, execute the show event detail command.

Core FilesCritical failures in Cisco UCS Manager and some Cisco UCS components, such as a fabric interconnect or an I/O module, can cause the system to create a core file. Each core file contains a large amount of data about the system and the component at the time of the failure.

Cisco UCS Manager manages the core files from all components. You can configure Cisco UCS Manager to export a copy of a core file to a location on an external TFTP server as soon as that core file is created.

This section includes the following topics:

• Core Files in Cisco UCS Manager GUI, page 1-8

• Core Files in Cisco UCS Manager CLI, page 1-8

Table 1-5 Event Properties

Property Name Description

Affected Object The component that created the event.

Description A description of the event.

Cause The unique identifier associated with the event.

Created at The day and time when the event was created.

User The type of user that created the event. This can be one of the following:

• admin

• internal

• blank

Code The unique identifier assigned to the event.

1-7Cisco UCS Troubleshooting Guide

OL-20137-04

Page 16: Cisco UCS Troubleshooting Guide

Send document comments to ucs -doc feedback@c i sco .com

Chapter 1 IntroductionAudit Log

• Core File Exporter, page 1-8

Core Files in Cisco UCS Manager GUIYou can find out if a component in the Cisco UCS instance generated a core file by navigating to the Core Files node on the Admin tab under Faults, Events and Audit Log.

Core Files in Cisco UCS Manager CLIYou can find out if a component in the Cisco UCS instance generated a core file by executing the following commands:

scope monitoring

scope sysdebug

show cores

Core File ExporterIf you enable the Core File Exporter, you can configure Cisco UCS Manager to export core files as soon as they occur to a specified location on the network through TFTP. This functionality allows you to export the tar file with the contents of the core file to the location specified.

Tip For information on how to enable the exporter, see the Cisco UCS configuration guides, accessible through the Cisco UCS B-Series Servers Documentation Roadmap.

Audit LogThe audit log records actions performed by users in Cisco UCS Manager, including direct and indirect actions. Each entry in the audit log represents a single, non-persistent action. For example, if a user logs in, logs out, or creates, modifies, or deletes an object such as a service profile, Cisco UCS Manager adds an entry to the audit log for that action.

You can view audit log entries in Cisco UCS Manager CLI, Cisco UCS Manager GUI, or the output from the show tech-support command.

This section includes the following topics:

• Properties of Audit Log Entries, page 1-9

• Audit Log in Cisco UCS Manager GUI, page 1-9

• Audit Log in Cisco UCS Manager CLI, page 1-9

1-8Cisco UCS Troubleshooting Guide

OL-20137-04

Page 17: Cisco UCS Troubleshooting Guide

Send document comments to ucs -doc feedback@c i sco .com

Chapter 1 IntroductionAudit Log

Properties of Audit Log EntriesCisco UCS Manager provides detailed information about each entry in the audit log. Table 1-6 describes the fault properties that you can view in Cisco UCS Manager CLI or Cisco UCS Manager GUI.

Audit Log in Cisco UCS Manager GUIIf you want to view the audit log, navigate to the Audit Log node on the Admin tab under Faults, Events and Audit Log.

Audit Log in Cisco UCS Manager CLIIf you want to view the audit log, execute the show audit-logs command:

scope security

show audit-logs

Table 1-6 Audit Log Entry Properties

Property Name Description

ID The unique identifier associated with the audit log message.

Affected Object The component affected by the user action.

Severity The current severity level of the user action associated with the audit log message. These are the same severities used for faults, as described in Table 1-1 on page 1-3.

Trigger The user role associated with the user that raised the message.

User The type of user that created the event. This can be one of the following:

• admin

• internal

• blank

Indication The action indicated by the audit log message. This can be one of the following:

• creation—A component was added to the system.

• modification—An existing component was changed.

Description A description of the user action.

1-9Cisco UCS Troubleshooting Guide

OL-20137-04

Page 18: Cisco UCS Troubleshooting Guide

Send document comments to ucs -doc feedback@c i sco .com

Chapter 1 IntroductionSystem Event Log

System Event LogThe system event log (SEL) resides on the CIMC in NVRAM. It records most server-related events, such as over and under voltage, temperature events, fan events, events from BIOS,and so on. The SEL is mainly used for troubleshooting purposes.

Tip For more information about the SEL, including how to view the SEL for each server and configure the SEL policy, see the Cisco UCS configuration guides accessible through the Cisco UCS B-Series Servers Documentation Roadmap.

This section includes the following topics:

• SEL File, page 1-10

• SEL Policy, page 1-10

SEL FileThe SEL file is approximately 40KB in size, and no further events can be recorded when it is full. It must be cleared before additional events can be recorded.

SEL PolicyYou can use the SEL policy to backup the SEL to a remote server, and optionally clear the SEL after a backup operation occurs. Backup operations can be triggered based on specific actions, or they can occur at regular intervals. You can also manually backup or clear the SEL.

Cisco UCS Manager automatically generates the SEL backup file, according to the settings in the SEL policy. The filename format is sel-SystemName-ChassisID-ServerID-ServerSerialNumber-Timestamp

For example, a file name could be sel-UCS-A-ch01-serv01-QCI12522939-20091121160736.

SyslogSyslog provides a central point for collecting and processing system logs that you can use to troubleshoot and audit the Cisco UCS instance. Cisco UCS Manager relies on the DCOS syslog mechanism and API and on the syslog feature of the primary fabric interconnect to collect and process the syslog entries.

This section includes the following topics:

• Syslog Configuration, page 1-11

• Syslog Location, page 1-11

• Syslog Entry Format, page 1-11

• Syslog Entry Parameters, page 1-12

• Syslog Services, page 1-13

1-10Cisco UCS Troubleshooting Guide

OL-20137-04

Page 19: Cisco UCS Troubleshooting Guide

Send document comments to ucs -doc feedback@c i sco .com

Chapter 1 IntroductionSyslog

Syslog ConfigurationCisco UCS Manager manages and configures the syslog collectors for the Cisco UCS instance and deploys the configuration to the fabric interconnect or fabric interconnects. This configuration affects all syslog entries generated in the Cisco UCS instance by DCOS or by Cisco UCS Manager.

Syslog LocationYou can configure Cisco UCS Manager to do one or more of the following with the syslog and syslog entries:

• Display the syslog entries in the console or on the monitor

• Store the syslog entries in a file

• Forward the syslog entries to up to three external log collectors where the syslog for the Cisco UCS instance is stored

Syslog Entry FormatEach syslog entry generated by a Cisco UCS component is formatted as follows:

Year month date hh:mm:ss hostname %facility-severity-MNEMONIC description

Example: 2007 Nov 1 14:07:58 excal-113 %MODULE-5-MOD_OK: Module 1 is online

Syslog Entry SeveritiesA syslog entry is assigned a Cisco UCS severity by Cisco UCS Manager. Table 1-7 shows how the Cisco UCS severities map to the syslog severities.

Table 1-7 Syslog Entry Severities in Cisco UCS

Cisco UCS Severity Syslog Severity

Critical CRIT

Major ERR

Minor ERR

Warning WARNING

Info INFO

1-11Cisco UCS Troubleshooting Guide

OL-20137-04

Page 20: Cisco UCS Troubleshooting Guide

Send document comments to ucs -doc feedback@c i sco .com

Chapter 1 IntroductionSyslog

Syslog Entry ParametersTable 1-8 describes the information contained in each syslog entry.

Table 1-8 Syslog Message Content

Name Description

Facility The logging facility that generated and sent the syslog entry. The facilities are broad categories that are represented by integers. These sources can be one of the following standard Linux facilities:

• local0

• local1

• local2

• local3

• local4

• local5

• local6

• local7

Severity The severity of the event, alert, or issue that caused the syslog entry to be generated. The severity can be one of the following:

• emergencies

• critical

• alerts

• errors

• warnings

• information

• notifications

• debugging

Hostname The hostname included in the syslog entry depends upon the component where the entry originated, as follows:

• Fabric interconnect or Cisco UCS Manager, the hostname of the Cisco UCS instance

• All other components, the hostname associated with the VIF

Timestamp The date and time when the syslog entry was generated.

Message A description of the event, alert, or issue that caused the syslog entry to be generated.

1-12Cisco UCS Troubleshooting Guide

OL-20137-04

Page 21: Cisco UCS Troubleshooting Guide

Send document comments to ucs -doc feedback@c i sco .com

Chapter 1 IntroductionSyslog

Syslog ServicesThe following Cisco UCS components use the DCOS syslog services to generate syslog entries for system information and alerts:

• I/O module—Runs syslogd. All syslog entries are sent to the fabric interconnect to which it is connected.

• CIMC—All syslog entries are sent to the primary fabric interconnect in a cluster configuration.

• Adapter—Runs Nic-Tools/Syslog. All syslog entries are sent to both fabric interconnects.

• Cisco UCS Manager—Logs its own syslog entries according to the syslog configuration.

1-13Cisco UCS Troubleshooting Guide

OL-20137-04

Page 22: Cisco UCS Troubleshooting Guide

Send document comments to ucs -doc feedback@c i sco .com

Chapter 1 IntroductionSyslog

1-14Cisco UCS Troubleshooting Guide

OL-20137-04

Page 23: Cisco UCS Troubleshooting Guide

Send document comments to ucs -doc feedback@c i sco .com

OL-20137-04

C H A P T E R 2

Finite State Machine

This chapter provides an overview of the finite state machine (FSM) and how to use it to troubleshoot issues with Cisco UCS.

This chapter contains the following sections:

• Overview of the FSM, page 2-1

• FSM Stage Names, page 2-2

• FSM in Cisco UCS Manager GUI, page 2-2

• FSM in Cisco UCS Manager CLI, page 2-3

Overview of the FSMAn FSM is a workflow model, similar to a flow chart, that is composed of the following:

• A finite number of stages (states)

• Transitions between those stages

• Operations

The current stage in an FSM is determined by past stages and the operations performed to transition between the stages. A transition from one stage to another is dependent on the success or failure of an operation.

Cisco UCS Manager uses FSM tasks that run in the Data Management Engine (DME) to manage end points in the UCS object model, including the following:

• Physical components (chassis, I/O module, servers)

• Logical components (LAN cloud, policies)

• Workflows (server discovery, service profile management, downloads, upgrades, backups)

The DME manages the FSM stages and transitions, and instructs the Application Gateway (AG) to perform operations on the managed end points. Therefore, each stage can be considered to be an interaction between the DME, AG, and managed end point. The AGs do the real work in interacting with managed end points, such as the CIMC, adapter, or I/O module.

When all FSM stages have run successfully, Cisco UCS considers the FSM to be successful.

If the FSM encounters an error or timeout at a stage, the FSM retries that stage at scheduled intervals. When the retry count has been reached for that stage, the FSM stops and Cisco UCS Manager declares the change to have failed. If an FSM task fails, Cisco UCS Manager raises faults and alarms.

2-1Cisco UCS Troubleshooting Guide

Page 24: Cisco UCS Troubleshooting Guide

Send document comments to ucs -doc feedback@c i sco .com

Chapter 2 Finite State MachineFSM Stage Names

Multiple FSM tasks can be associated to an end point. However, only one FSM task at a time can run. Additional FSM tasks for the same end point are placed in a queue and are scheduled to be run when the previous FSM task either successfully completes or fails.

You can view the FSM details for a particular end point to determine if a task succeeded or failed. You can also use the FSM to troubleshoot any failures.

FSM Stage NamesFSM stage names are usually constructed using the following notation

FsmObjectWorkflowOperationWhere-is-it-executed

where:

• Object is the object the FSM is running, such as Blade/Chassis.

• Workflow is the overall task being performed by the FSM, such as Discover or Association.

• Operation is the task being performed at a particular stage, such as Pnuos-Config.

• Where-is-it-executed is generally “”, or “A” or “B” or “Local” or “Peer”. If not specified, it is executed on the managingInst node.

Each FSM stage name has a prefix that identifies the FSM and a suffix that identifies a stage within the FSM. The prefix notation is FsmObjectWorkflow and the suffix notation is OperationWhere-is-it-executed. For example, if the FSM name is FsmComputeBladeDiscoverCimcInventory:

• The prefix is FsmComputeBladeDiscover

• The suffix is CimcInventory

FSM in Cisco UCS Manager GUICisco UCS Manager GUI displays FSM information for an end point on the FSM tab for that end point. You can use the FSM tab to monitor the progress and status of the current FSM task and view a list of the pending FSM tasks.

The information about a current FSM task in Cisco UCS Manager GUI is dynamic and changes as the task progresses. You can view the following information about the current FSM task:

• Which FSM task is being executed

• Current state of that task

• Time and status of the previously completed task

• Any remote invocation error codes returned while processing the task

• Progress of the current task

If you want to view the FSM task for an end point that supports FSM, navigate to the end point in the Navigation pane and click the FSM tab in the Work pane.

2-2Cisco UCS Troubleshooting Guide

OL-20137-04

Page 25: Cisco UCS Troubleshooting Guide

Send document comments to ucs -doc feedback@c i sco .com

Chapter 2 Finite State MachineFSM in Cisco UCS Manager CLI

FSM in Cisco UCS Manager CLICisco UCS Manager CLI can display FSM information for an end point when you are in the command mode for that end point.

You can use the show fsm status command in the appropriate mode to view the current FSM task for an end point. The information displayed about a current FSM task in the CLI is static. You must re-enter the command to see progress updates. The following example displays information about the current FSM task for the server in chassis 1, slot 6:

UCS-A# scope server 1/6UCS-A /chassis/server # show fsm statusSlot: 6Server: sys/chassis-1/blade-6 FSM 1: Remote Result: Not Applicable Remote Error Code: None Remote Error Description: Status: Discover Blade Boot Wait Previous Status: Discover Blade Boot Wait Timestamp: 2006-01-26T23:31:36 Try: 0 Flags: 0 Progress (%): 33 Current Task: Waiting for system reset on server 1/6 (FSM-STAGE:sam:dme:ComputeBladeDiscover:BladeBootWait)

You can use the show fsm task command in the appropriate mode to view all pending tasks in the FSM queue. The following example displays FSM task queue for the server in chassis 1, slot 6:

UCS-A# scope server 1/6UCS-A /chassis/server # show fsm task

FSM Task: Item ID Completion FSM Flags ---------------- -------- ------------ --------- Powercycle 1154858 Scheduled BiosRecovery 1154860 Scheduled

2-3Cisco UCS Troubleshooting Guide

OL-20137-04

Page 26: Cisco UCS Troubleshooting Guide

Send document comments to ucs -doc feedback@c i sco .com

Chapter 2 Finite State MachineFSM in Cisco UCS Manager CLI

2-4Cisco UCS Troubleshooting Guide

OL-20137-04

Page 27: Cisco UCS Troubleshooting Guide

Send document comments to ucs -doc feedback@c i sco .com

OL-20137-04

C H A P T E R 3

General Troubleshooting Steps and Solutions

This chapter provides some steps and possible solutions that you can implement when you troubleshoot issues with Cisco Unified Computing System (Cisco UCS).

This chapter contains the following sections:

• Guidelines for Troubleshooting, page 3-1

• Gathering Information about a Cisco UCS Instance, page 3-2

Guidelines for TroubleshootingWhen you troubleshoot issues with Cisco UCS Manager or a component that it manages, we recommend that you follow the guidelines in Table 3-1:

Table 3-1 Troubleshooting Guidelines

Guideline Description

Check the release notes to see if the issue is a known problem.

The release notes are accessible through the Cisco UCS B-Series Servers Documentation Roadmap.

Take screenshots of the fault or error message dialog box, the FSM for the component, and other relevant areas.

These screenshots provide visual cues about the state of Cisco UCS Manager when the problem occurred. If your computer does not have software to take screenshots, check the documentation for your operating system, as it may include this functionality.

Record the steps that you took directly before the issue occurred.

If you have access to screen or keystroke recording software, repeat the steps you took and record what occurs in Cisco UCS Manager.

If you do not have access to that type of software, repeat the steps you took and make detailed notes of the steps and what happens in Cisco UCS Manager after each step.

Execute the show tech-support command. The information about the current state of the Cisco UCS instance is very helpful to Cisco support and frequently provides the information needed to identify the source of the problem.

3-1Cisco UCS Troubleshooting Guide

Page 28: Cisco UCS Troubleshooting Guide

Send document comments to ucs -doc feedback@c i sco .com

Chapter 3 General Troubleshooting Steps and SolutionsGathering Information about a Cisco UCS Instance

Gathering Information about a Cisco UCS InstanceWhen you encounter an issue that requires troubleshooting or a request for assistance to Cisco Technical Support, you need to collect as much information as possible about the affected Cisco UCS instance.

This section includes the following topics:

• Executing the show tech-support Command, page 3-2

Executing the show tech-support CommandThe show tech-support command outputs information about the Cisco UCS instance that you can send to Cisco support to obtain assistance troubleshooting the issue or issues. Cisco UCS Manager stores the output of the show tech support command in a file.

Note You can only execute the show tech-support command in Cisco UCS Manager CLI.

Command or Action Purpose

Step 1 Fabric-A# connect local-mgmt {a | b} Enters local management mode.

Step 2 Fabric-A(local-mgmt) # show tech-support {chassis chassis-id {all | cimc slot [adapter adapter-id] | iom iom-id} | ucsm} [brief | detail]

Example:

Fabric-A(local-mgmt)# show tech-support chassis 1 all detail

Outputs information about the selected objects in a file that you can send to Cisco support.

Step 3 Fabric-A (local-mgmt) # copy workspace:techsupport/filename.tar {scp | ftp}: user_name@IP_address

Enter username’s password: password

Example:

Fabric-A (local-mgmt) # copy workspace:techsupport/20100624161130_SAM-FCS_UCSM.tar scp://[email protected]/[email protected]'s password: 20100624161130_SAM-FCS_UCSM.tar 100% 13MB 13.3MB/s 00:01 Fabric-A (local-mgmt) #

Copies the output file to an external location through SCP or FTP.

The SCP and FTP commands require an absolute path for the target location. The path to your home directory cannot include special symbols, such as ‘~’.

3-2Cisco UCS Troubleshooting Guide

OL-20137-04

Page 29: Cisco UCS Troubleshooting Guide

Send document comments to ucs -doc feedback@c i sco .com

OL-20137-04

C H A P T E R 4

Troubleshooting Issues with Cisco UCS

This chapter provides some steps and possible solutions that you can implement when you troubleshoot issues with Cisco Unified Computing System (Cisco UCS).

This chapter contains the following sections:

• Troubleshooting Boot Issues, page 4-1

• Troubleshooting KVM Issues, page 4-3

• Troubleshooting VM Issues, page 4-5

• Troubleshooting Cisco UCS Manager Issues, page 4-6

• Troubleshooting Fabric Interconnect Issues, page 4-8

Troubleshooting Boot IssuesThis section includes the following topics:

• Reboot Warning Does Not Display, page 4-1

• Server Does Not Boot from OS Installed on eUSB, page 4-2

Reboot Warning Does Not Display

Problem

The system fails to produce a reboot warning listing dependencies.

Possible Cause

This problem can be caused by changes to a vNIC template or a vHBA template. Reboot warnings occur when the back-end returns a list of dependencies. When you update the template type for a vNIC or vHBA template and make changes to any boot-related properties without applying changes between steps, back-end systems are not triggered to return a list of dependencies.

4-1Cisco UCS Troubleshooting Guide

Page 30: Cisco UCS Troubleshooting Guide

Send document comments to ucs -doc feedback@c i sco .com

Chapter 4 Troubleshooting Issues with Cisco UCSTroubleshooting Boot Issues

Recommended Action

Take the following actions in Cisco UCS Manager GUI:

Step 1 In the vNIC template or vHBA template, do the following:

a. Change the template type from Inital Template to Updating Template.

b. Click Save Changes.

Step 2 Make any additional changes to reboot-related values and Save.

A reboot warning and list of dependencies are displayed.

Server Does Not Boot from OS Installed on eUSB

Problem

The eUSB embedded inside the Cisco UCS server includes an operating system. However, the server does not boot from that operating system.

Possible Cause

This problem can occur when, after associating the server with the service profile, the eUSB is not at the top of the actual boot order for the server.

Recommended Action

To ensure that the server boots from the operating system on the eUSB, take the following actions in Cisco UCS Manager GUI:

Procedure

Step 1 On the Servers tab, do the following to verify the boot policy configuration:

a. Navigate to the service profile associated with the server.

b. On the Boot Order tab, ensure that Local Disk is configured as the first device in the boot policy.

Step 2 On the Equipment tab, do the following to verify the actual boot order for the server:

a. Navigate to the server.

b. On the General tab, expand the Boot Order Details area and verify that the eUSB is listed as the first device on the Actual Boot Order tab.

For example, the first device should be VM eUSB DISK.

Step 3 If the eUSB is not the first device in the actual boot order, do the following:

a. On the General tab for the server, click the following links in the Actions area:

– KVM Console to launch the KVM console

– Boot Server to boot the server

4-2Cisco UCS Troubleshooting Guide

OL-20137-04

Page 31: Cisco UCS Troubleshooting Guide

Send document comments to ucs -doc feedback@c i sco .com

Chapter 4 Troubleshooting Issues with Cisco UCSTroubleshooting KVM Issues

b. In the KVM console, while the server is booting, press F2 to enter the BIOS setup.

c. In the BIOS utility, navigate to the Boot Options tab.

d. Choose Hard Disk Order.

e. Configure Boot Option #1 to the eUSB.

For example, set this option to VM eUSB DISK.

f. Press F10 to save and exit.

Troubleshooting KVM IssuesThis section includes the following topics:

• BadFieldException When Launching KVM Viewer, page 4-3

• KVM Console Failure, page 4-4

• KVM Fails to Open, page 4-4

BadFieldException When Launching KVM Viewer

Problem

A BadFieldException displays when the KVM viewer is launched.

Possible Cause

This problem can be occur because Java Web Start disables the cache by default when it is used in conjunction with an application that uses native libraries.

Recommended Action

If you see this issue, take the following action:

Procedure

Step 1 Choose Start > Control Panel > Java.

Step 2 Choose the General tab.

Step 3 In the Temporary Internet Files area, click Settings.

Step 4 Check the Keep temporary files on my computer check box.

Step 5 Click OK.

4-3Cisco UCS Troubleshooting Guide

OL-20137-04

Page 32: Cisco UCS Troubleshooting Guide

Send document comments to ucs -doc feedback@c i sco .com

Chapter 4 Troubleshooting Issues with Cisco UCSTroubleshooting KVM Issues

KVM Console Failure

Problem

The KVM Console fails to launch and the JRE displays the following message:

Unable to launch the application.

Possible Cause

This problem can be caused if several KVM Consoles are launched simultaneously.

Recommended Action

If you see this issue, take the following action:

Procedure

Step 1 If possible, close one or more of your open KVM Consoles.

Step 2 Relaunch the KVM Console.

KVM Fails to Open

Problem

The first time you attempt to open it on a server, the KVM fails to launch.

Possible Cause

This problem can be caused by a JRE version incompatibility.

Recommended Action

If you see this issue, take the following action:

Procedure

Step 1 Upgrade to JRE 1.6_11.

Step 2 Reboot the server.

Step 3 Launch the KVM Console.

4-4Cisco UCS Troubleshooting Guide

OL-20137-04

Page 33: Cisco UCS Troubleshooting Guide

Send document comments to ucs -doc feedback@c i sco .com

Chapter 4 Troubleshooting Issues with Cisco UCSTroubleshooting VM Issues

Troubleshooting VM IssuesThis section includes the following topic:

• Currently connected network interface x uses Distributed Virtual Switch (uusid: y) which is accessed on the host via a switch that has no free ports, page 4-5

Currently connected network interface x uses Distributed Virtual Switch (uusid: y) which is accessed on the host via a switch that has no free ports

Problem

The following error displays:

Currently connected network interface x uses Distributed Virtual Switch (uusid:y) which is accessed on the host via a switch that has no free ports.

Possible Cause

This problem can be caused by one of the following issues:

• After powering off or migrating a VM from one host to another, the vSphere Server fails to recompute the numPortsAvailable property in the hostProxySwitch object.

• The cumulative number of vNICs for the VMs powered on an ESX host matches or exceeds the number of dynamic nVINCs configured in the server’s service profile.

• After migrating a VM from one data-store to another data-store on the same server, the server incorrectly detects an increase in the number of DVS ports being used by all of the VMs powered on the host.

Recommended Action

If you see this issue, take the following actions:

Procedure

Step 1 Identify the cause of the error.

Step 2 If the error resulted from powering off a VM, or from migrating a VM from one host to another, do the following:

a. Migrate a second VM from the ESX host to another system.

b. When a second port is made available, do one of the following:

– Power on a VM.

– Migrate a VM back to the ESX host.

4-5Cisco UCS Troubleshooting Guide

OL-20137-04

Page 34: Cisco UCS Troubleshooting Guide

Send document comments to ucs -doc feedback@c i sco .com

Chapter 4 Troubleshooting Issues with Cisco UCSTroubleshooting Cisco UCS Manager Issues

Step 3 If the error resulted from migrating a VM instance from one data-store to another data-store on the same server, do the following:

a. Shut down all of the VMs on the ESX host.

b. Retry the migration.

Troubleshooting Cisco UCS Manager IssuesThis section includes the following topics:

• Fatal error: event sequencing is skewed, page 4-6

• HDD Metrics Not Updated in Cisco UCS Manager GUI, page 4-7

• Cisco UCS Manager Reports More Disks in Server than Total Slots Available, page 4-7

Fatal error: event sequencing is skewed

Problem

After coming back from sleep mode, the following message appears:

Fatal error: event sequencing is skewed.

Possible Cause

This problem can be caused if Cisco UCS Manager GUI was not shut down before the computer went to sleep. Since the JRE does not have a sleep detection mechanism, the system is unable to retrack all of the messages received before it went into sleep mode. After multiple retries, this event sequencing error is logged.

Note Always shut down Cisco UCS Manager GUI before putting your computer to sleep.

Recommended Action

Step 1 In Cisco UCS Manager GUI, if a Connection Error dialog box is displayed, click one of the following:

• Re-login to log back in to Cisco UCS Manager GUI

• Exit to exit Cisco UCS Manager GUI

4-6Cisco UCS Troubleshooting Guide

OL-20137-04

Page 35: Cisco UCS Troubleshooting Guide

Send document comments to ucs -doc feedback@c i sco .com

Chapter 4 Troubleshooting Issues with Cisco UCSTroubleshooting Cisco UCS Manager Issues

HDD Metrics Not Updated in Cisco UCS Manager GUI

Problem

After hot-swapping, removing, or adding a hard drive, updated HDD metrics do not display in Cisco UCS Manager GUI.

Possible Cause

This problem can be caused because Cisco UCS Manager only gathers HDD metrics during a system boot. If a hard drive is added or removed after a system boot, Cisco UCS Manager GUI does not update the HDD metrics.

Recommended Action

Step 1 Reboot the server.

Cisco UCS Manager Reports More Disks in Server than Total Slots Available

Problem

Cisco UCS Manager reports that a server has more disks than the total disk slots available in the server. For example, Cisco UCS Manager reports three disks for a server with two disk slots as follows:

RAID Controller 1: Local Disk 1: Product Name: 73GB 6Gb SAS 15K RPM SFF HDD/hot plug/drive sled mounted PID: A03-D073GC2 Serial: D3B0P99001R9 Presence: Equipped Local Disk 2: Product Name: Presence: Equipped Size (MB): Unknown Local Disk 5: Product Name: 73GB 6Gb SAS 15K RPM SFF HDD/hot plug/drive sled mounted Serial: D3B0P99001R9 HW Rev: 0 Size (MB): 70136

Possible Cause

This problem is typically caused by a communication failure between Cisco UCS Manager and the server that reports the inaccurate information.

4-7Cisco UCS Troubleshooting Guide

OL-20137-04

Page 36: Cisco UCS Troubleshooting Guide

Send document comments to ucs -doc feedback@c i sco .com

Chapter 4 Troubleshooting Issues with Cisco UCSTroubleshooting Fabric Interconnect Issues

Recommended Action

Step 1 Decommission the server.

Step 2 Recommission the server.

Troubleshooting Fabric Interconnect IssuesThis section includes the following topic:

• Recovering a Fabric Interconnect from the Boot Loader Prompt, page 4-9

4-8Cisco UCS Troubleshooting Guide

OL-20137-04

Page 37: Cisco UCS Troubleshooting Guide

Send document comments to ucs -doc feedback@c i sco .com

Chapter 4 Troubleshooting Issues with Cisco UCSTroubleshooting Fabric Interconnect Issues

Recovering a Fabric Interconnect from the Boot Loader Prompt

Problem

The fabric interconnect fails to start.

Possible Cause

This problem can be caused by one of the following issues, which require you to use the boot loader prompt to recover the fabric interconnect:

• Kickstart image is corrupted or non-functional for other reasons

• File system on the bootflash memory is corrupted

Recommended Action

If you see this issue, take the following action.

Before You Begin

1. Verify the following physical connections on the fabric interconnect:

– A console port on the first fabric interconnect is physically connected to a computer terminal or console server.

– The management Ethernet port (mgmt0) is connected to an external hub, switch, or router.

– For a cluster configuration, the L1 ports on both fabric interconnects are directly connected to each other and the L2 ports on both fabric interconnects are directly connected to each other.

2. Verify that the console port parameters on the computer terminal (or console server) attached to the console port are as follows:

– 9600 baud

– 8 data bits

– No parity

– 1 stop bit

3. Identify and download the following firmware images and place them on an SCP server:

– Kickstart

– System

– Cisco UCS Manager

4. Collect the following information:

– Full path and filenames for the firmware images

– Hostname of the SCP server

– Username and password you can use to access the files on the SCP server

4-9Cisco UCS Troubleshooting Guide

OL-20137-04

Page 38: Cisco UCS Troubleshooting Guide

Send document comments to ucs -doc feedback@c i sco .com

Chapter 4 Troubleshooting Issues with Cisco UCSTroubleshooting Fabric Interconnect Issues

Procedure

Command or Action Purpose and Information

Step 1 Connect to the console port. —

Step 2 Power cycle the fabric interconnect:

1. Turn off the power to the fabric interconnect.

2. Turn on the power to the fabric interconnect.

You can see the power on self-test messages as the fabric interconnect boots.

Step 3 In the console, press one of the following key combinations as it boots to get the loader prompt:

• Ctrl+l

• Ctrl+Shift+r

You may need to press the selected key combination multiple times before your screen displays the loader prompt.

Step 4 loader > set mac mac_address (Optional) Sets the MAC address for the fabrice interconnect.

Step 5 loader > set ip ip_address netmask Sets the IP address and netmask for the fabric interconnect.

Step 6 loader > set gw gateway Sets the gateway for the fabric interconnect.

Step 7 loader > boot scp://scp_server_ip/ path_relative_to_scp_root/kickstart_filename

Specifies the kickstart image to use for the fabric interconnect and takes you to the kickstart prompt.

Step 8 switch(boot)# init system Initializes the fabric interconnect and creates the partitions in the bootflash.

Step 9 switch(boot)# config t Enters configuration mode.

Step 10 switch(config)# int mgmt 0 Enters management 0 interface mode.

Step 11 switch(config)# ip address ip_address netmask Configures the IP address and netmask for the fabric interconnect.

Step 12 switch(config)# no shut Ensures no shutdown and administratively starts the interface.

Step 13 switch(config)# ip default-gateway gateway Configures the default gateway for the fabric interconnect.

Step 14 switch(config)# end Completes the configuration of the IP address for the fabric interconnect.

Step 15 switch(boot)# copy scp: bootflash:

Example:

switch(boot)# copy scp: bootflash:Enter source filename: /ucs/images/ucs-6100-k9-kickstart.4.1.3.N2.1.3.1.binEnter hostname for the scp server: 192.168.10.10Enter username: [email protected]'s password:ucs-6100-k9-kickstart.4.1.3.N2.1.3.1.bin 100% 21MB 4.1MB/s 00:05

Copies the kickstart image to bootflash.

As shown in the example, you are prompted for additional information, such as a path relative to the SCP root, then provided with a status message.

This example uses a .bin file. Under some circumstances, Cisco Technical Support may provide you with a .gbin file for the new kickstart image.

4-10Cisco UCS Troubleshooting Guide

OL-20137-04

Page 39: Cisco UCS Troubleshooting Guide

Send document comments to ucs -doc feedback@c i sco .com

Chapter 4 Troubleshooting Issues with Cisco UCSTroubleshooting Fabric Interconnect Issues

Step 16 switch(boot)# copy scp: bootflash:

Example:

switch(boot)# copy scp: bootflash:Enter source filename: /ucs/images/ucs-6100-k9-system.4.1.3.N2.1.3.1.binEnter hostname for the scp server: 192.168.10.10Enter username: [email protected]'s password:ucs-6100-k9-system.4.1.3.N2.1.3.1.bin 100% 189MB 6.5MB/s 00:29

Copies the system image to bootflash.

As shown in the example, you are prompted for additional information, such as a path relative to the SCP root, then provided with a status message.

This example uses a .bin file. Under some circumstances, Cisco Technical Support may provide you with a .gbin file for the new system image.

Step 17 switch(boot)# copy scp: bootflash:

Example:

switch(boot)# copy scp: bootflash:Enter source filename: /ucs/images/ucs-manager-k9.1.3.1.binEnter hostname for the scp server: 192.168.10.10Enter username: [email protected]'s password:ucs-manager-k9.1.3.1.bin 100% 131MB 8.8MB/s 00:15

Copies the management image to bootflash.

As shown in the example, you are prompted for additional information, such as a path relative to the SCP root, then provided with a status message.

This example uses a .bin file. Under some circumstances, Cisco Technical Support may provide you with a .gbin file for the new system image.

Step 18 Perform a dir on the bootflash to ensure that the files were copied correctly.

Step 19 switch(boot)# copy bootflash:management_image bootflash:nuova-sim-mgmt-nsg.0.1.0.001.bin

Example:

switch(boot)# copy bootflash:ucs-manager-k9.1.3.1.bin bootflash:nuova-sim-mgmt-nsg.0.1.0.001.bin

Copies the management image in the bootflash and renames the management image to ensure that it is Cisco UCS Manager compliant.

The ‘nuova-sim-mgmt-nsg.0.1.0.001.bin’ line makes the management image Cisco UCS Manager compliant.

Step 20 switch(boot)# load bootflash:system_image

Example:

switch(boot)# load bootflash: ucs-6100-k9-system.4.1.3.N2.1.3.1.bin

Loads the system image from bootflash.

Command or Action Purpose and Information

4-11Cisco UCS Troubleshooting Guide

OL-20137-04

Page 40: Cisco UCS Troubleshooting Guide

Send document comments to ucs -doc feedback@c i sco .com

Chapter 4 Troubleshooting Issues with Cisco UCSTroubleshooting Fabric Interconnect Issues

Step 21 Power cycle the fabric interconnect:

1. Turn off the power to the fabric interconnect.

2. Turn on the power to the fabric interconnect.

You can see the power on self-test messages as the fabric interconnect boots.

Step 22 Upgrade the following firmware to the release of Cisco UCS that you just copied into bootflash;

• Kickstart

• System

• Cisco UCS Manager

Ensures that the fabric interconnect boots correctly in the future. If you do not perform this step, you may have to reboot from the loader prompt again.

Follow the instructions in the appropriate upgrade guide for this release. Assuming that you are upgrading to the same release as other components in the Cisco UCS instance, you do not have to upgrade the other components, such as the IOMs or servers.

Command or Action Purpose and Information

4-12Cisco UCS Troubleshooting Guide

OL-20137-04

Page 41: Cisco UCS Troubleshooting Guide

Send document comments to ucs -doc feedback@c i sco .com

OL-20137-04

C H A P T E R 5

Troubleshooting Cisco UCS Faults

Cisco UCS faults provide information that you can use to troubleshoot issues with a Cisco UCS instance. Documentation about these faults, including explanations and recommended actions, are available in the Cisco UCS Faults Reference.

5-1Cisco UCS Troubleshooting Guide

Page 42: Cisco UCS Troubleshooting Guide

Send document comments to ucs -doc feedback@c i sco .com

Chapter 5 Troubleshooting Cisco UCS Faults

5-2Cisco UCS Troubleshooting Guide

OL-20137-04

Page 43: Cisco UCS Troubleshooting Guide

Send document comments to ucs -doc feedback@c i sco .com

OL-20137-04

C H A P T E R 6

Troubleshooting SAN Boot and SAN Connectivity Issues

This chapter contains information about how to troubleshoot SAN boot and SAN connectivity issues, including the Storage Area Network (SAN) configuration in Cisco UCS Manager and the SAN array.

This chapter contains the following sections:

• SAN Connectivity Checklist, page 6-1

• SAN Array Configuration Checklist, page 6-2

• Recommended Solutions for Issues During SAN Boot, page 6-2

SAN Connectivity ChecklistA problem with connectivity to the SAN array can cause issues with the SAN boot. If other solutions do not resolve your issue, consider the following:

• Are the Fibre Channel uplink ports configured in Cisco UCS Manager?

• Do the numbers assigned to the Virtual Storage Area Networks (VSANs) in Cisco UCS Manager match those configured in the Fibre Channel switch?

• Is N-Port ID Virtualization (NPIV) enabled on the Fibre Channel switch?

• Is the Cisco UCS fabric interconnect logged into the Fibre Channel switch? The Fibre Channel switch displays the fabric interconnect as an NPIV device. For example, you can use the show fcns data command on a Multilayer Director Switch (MDS).

• Is the World Wide Name (WWN) in the correct format in Cisco UCS Manager?

• Have you upgraded the server adapters to use the latest firmware?

• Have you verified the SAN boot and SAN boot target configuration in the boot policy included with the service profile associated with the server?

• Do the vNIC and vHBA names in the boot policy match those in the vHBA assigned to the service profile?

• Is the array Active/Passive?

• Are you booting to the active controller on the array?

• Is the array configured correctly? For example, have you verified the items in the “SAN Array Configuration Checklist” section on page 6-2.

6-1Cisco UCS Troubleshooting Guide

Page 44: Cisco UCS Troubleshooting Guide

Send document comments to ucs -doc feedback@c i sco .com

Chapter 6 Troubleshooting SAN Boot and SAN Connectivity IssuesSAN Array Configuration Checklist

SAN Array Configuration ChecklistA misconfiguration or other issue with the SAN array can cause issues with the SAN boot. If other solutions do not resolve your issue, verify the following basic configurations in the SAN array:

• Has the host been acknowledged or registered by the array?

• Is the array configured to allow the host to access the Logical Unit Number (LUN)? For example, is LUN security or LUN masking configured?

• Did you correctly configure the LUN allocation with the World Wide Port Name (WWPN) assigned in the Cisco UCS instance? If you assign and configure with a World Wide Node Name (WWNN), you could encounter issues.

• Did you map the backed LUN of the array to the same LUN number configured in the Cisco UCS boot policy?

Recommended Solutions for Issues During SAN BootTable 6-1 contains a list of guidelines and recommended solutions that can assist you in troubleshooting a SAN boot issue. If an attempt to boot from a SAN array fails, we recommend that you implement these solutions. These suggested solutions are in addition to those that are described in the “SAN Connectivity Checklist” section on page 6-1 and the “SAN Array Configuration Checklist” section on page 6-2.

Table 6-1 SAN Boot Issues

Issue Recommended Solution

SAN boot fails intermittently. Verify the configuration of the SAN boot target in the boot policy included in the service profile. For example, make sure that the SAN boot target includes a valid WWPN.

Server tries to boot from local disk instead of SAN.

Verify that the configured boot order in the service profile has SAN as the first boot device.

If the boot order in the service profile is correct, verify that the actual boot order for the server includes SAN as the first boot device.

If the actual boot order is not correct, reboot the server.

Server cannot boot from SAN even though boot order is correct.

For Windows and Linux, verify that the boot LUN is numbered as 0 to ensure that LUN is mounted as the first disk from which the server boots.

For ESX, if more than one LUN is presented, verify that the boot LUN is the lowest numbered LUN.

6-2Cisco UCS Troubleshooting Guide

OL-20137-04

Page 45: Cisco UCS Troubleshooting Guide

Send document comments to ucs -doc feedback@c i sco .com

OL-20137-04

C H A P T E R 7

Troubleshooting Server Hardware Issues

This chapter contains information about how to troubleshoot hardware issues not specific to a given model of UCS B-Series server.

This chapter contains the following sections:

• Diagnostics Button and LEDs, page 7-1

• DIMM Memory Issues, page 7-2

• CPU Issues, page 7-7

• Disk Drive and RAID Issues, page 7-9

• Adapter Issues, page 7-12

• Gathering Information Before Calling Support, page 7-14

• Related Documents, page 7-15

Diagnostics Button and LEDsAt blade start-up, POST diagnostics test the CPUs, DIMMs, HDDs and adapter cards, and any failure notifications are sent to UCS Manager. You can view these notification in the System Error Log or in the output of the show tech-support command. If errors are found, an amber diagnostic LED lights up next to the failed component. During run time, the blade BIOS, component drivers, and OS all monitor for hardware faults and lights up the amber diagnostic LED for a component if an uncorrectable error or correctable errors (such as a host ECC error) over the allowed threshold occur.

LED states are saved, and if you remove the blade from the chassis, the LED values persist for up to 10 minutes. Pressing the LED diagnostics button on the motherboard causes the LEDs that currently show a component fault to light for up to 30 seconds for easier component identification. LED fault values are reset when the blade is reinserted into the chassis and booted, and the process begins from its start.

If DIMM insertion errors are detected, they may cause the blade discovery to fail and errors are reported in the server POST information, viewable using the UCS Manager GUI or CLI. UCS blade servers require specific rules to be followed when populating DIMMs in a blade server, and the rules depend on the blade server model. Refer to the documentation for a specific blade server for those rules.

HDD status LEDs are on the front face of the HDD. Faults on the CPU, DIMMs, or adapter cards also cause the server health LED to light solid Amber for minor error conditions or blinking Amber for critical error conditions.

7-1Cisco UCS Troubleshooting Guide

Page 46: Cisco UCS Troubleshooting Guide

Send document comments to ucs -doc feedback@c i sco .com

Chapter 7 Troubleshooting Server Hardware IssuesDIMM Memory Issues

DIMM Memory IssuesA problem with DIMM memory can cause a server to fail to boot or run below its capabilities. If DIMM issues are suspected, consider the following:

• DIMMs tested, qualified, and sold by Cisco are the only DIMMs supported on your system. Third-party DIMMs are not supported, and if they are present, Cisco support will ask you to replace them with Cisco DIMMs before continuing to troubleshoot a problem.

• Is the malfunctioning DIMM supported on that model of server? Refer to the blade server’s installation and service note to verify you are using the correct combination of server, CPU and DIMM.

• Is the malfunctioning DIMM seated correctly in the slot?

• Cisco servers have either a required or recommended order for installing DIMMs. Refer to the blade server’s installation and service note to verify you are adding the DIMMs appropriately for a given server type.

• Most DIMMs are sold in matched pairs, meaning that they are intended to be added two at a time paired with each other. Splitting the pairs may cause memory problems.

• If DIMMs with a maximum speed lower than those previously installed are added to a system, all DIMMs in a server will run at the slower speed or may not work at all. It is best if all DIMMs in a server are of the same type.

• The number and size of DIMMs should be the same for all CPUs in a server, mismatching DIMM configurations may damage system performance.

Known IssuesPlease rule out the following known issues before contacting Cisco Technical support with DIMM-related issues:

Cisco UCS Manager GUI incorrectly reports bad DIMMs.

Cisco UCS Manager GUI may incorrectly report “inoperable memory” when the UCS CLI indicates no failures. This has been observed when running Cisco UCS Manager version 1.0(1e).

Upgrade to Cisco UCS Manager version 1.0(2d) or later. If that is not possible, to confirm memory is okay, please use the following CLI commands where x=chassis# and y=server# and z=memory array ID#:

• scope server x/y -> show memory detail

• scope server x/y -> show memory-array detail -> will give you memory-array ID

• scope server x/y -> scope memory-array z -> show stats history memory-array-env-stats detail

Correctable DIMM error reporting in Cisco UCS Manager will not go away until BMC is rebooted.

Correctable DIMM errors report a DIMM as “Degraded” in Cisco UCS Manager, but the DIMMs will still be available to the blade’s OS.

To correct this, use the following commands to clear the SEL logs from the BMC, then reboot the BMC of the affected blade, or just remove and reseat the blade server from the chassis.

SAM-FCS-A# scope server x/ySAM-FCS-A /chassis/server # scope bmcSAM-FCS-A /chassis/server/bmc # resetSAM-FCS-A /chassis/server/bmc* # commit-buffer

7-2Cisco UCS Troubleshooting Guide

OL-20137-04

Page 47: Cisco UCS Troubleshooting Guide

Send document comments to ucs -doc feedback@c i sco .com

Chapter 7 Troubleshooting Server Hardware IssuesDIMM Memory Issues

Cisco UCS Manager incorrect report of effective memory.

When running Cisco UCS Manager version 1.0(1e), Cisco UCS Manager may misread the SMBIOS table, and may not be able to read it without a server reboot.

Upgrade to Cisco UCS Manager version 1.2(0) or later.

Memory misreported in Cisco UCS Manager.

Memory Arrays show more memory sockets than are physically on the system board.

Upgrade to Cisco UCS Manager version 1.0(2j) or later.

A single DIMM can cause other DIMMs to get marked as bad. POST fails.

The UCS blade will not complete its boot cycle, and FSM will stay stuck at 54%.

Upgrade to Cisco UCS Manager version 1.2.(1b) or later.

Types of DIMM ErrorsThe BIOS in the UCS blades can detect and report two different types of DIMM errors.

Correctable DIMM Errors

DIMMs with correctable errors are not disabled and are available for the OS to use. The Total Memory and Effective Memory will be the same (taking memory mirroring into account). These correctable errors will be reported in UCS Manager as “Degraded.”

If you see a correctable error reported matching the information above, you may be able to correct the problem by resetting the BMC instead of re-seating or resetting the blade server. To do this, use the following UCS Manager CLI commands:

UCS1-A# scope server x/yUCS1-A /chassis/server # scope bmcUCS1-A /chassis/server/bmc # resetUCS1-A /chassis/server/bmc* # commit-buffer

Resetting the BMC will not impact the OS running in the blade.

Uncorrectable DIMM Errors

DIMMs with uncorrectable errors will be disabled and the server’s OS will not see that memory. Additionally, if a DIMM or DIMMs fail while the system is up the OS could crash unexpectedly. UCS Manager will be show the DIMMs are “Inoperable” in the case of uncorrectable DIMM errors. While these errors are not correctable via software it may be possible to identify a bad DIMM and remove it to allow the blade to boot. An example of this type of error is when the BIOS fails to pass the POST due to one or more bad DIMMs.

In situations where BIOS POST failures occur due to suspected memory issues and the particular DIMMs or DIMM slots are not identifiable the following procedure can be used to further isolate a particular failed part:

1. Remove all DIMMs from the system

2. Install a single DIMM (preferably a tested good DIMM) or a DIMM pair in the first usable slot for the first processor (minimum requirement for POST success), that is, DIMM slot A1 on a B200 blade. Refer to the published memory population rules to determine which slot to use.

7-3Cisco UCS Troubleshooting Guide

OL-20137-04

Page 48: Cisco UCS Troubleshooting Guide

Send document comments to ucs -doc feedback@c i sco .com

Chapter 7 Troubleshooting Server Hardware IssuesDIMM Memory Issues

3. Re-attempt to boot the system.

If BIOS POST is still unsuccessful, you may repeat steps 1-3, using a different DIMM in step 2.

If BIOS POST is successful, and the blade can associate to a Service Profile, continue adding memory following the population rules for that model of server. If the system can successfully pass BIOS POST in some memory configurations but not others that information can be used to help isolate the source of the problem.

Troubleshooting DIMM ErrorsTo determine the type of DIMM errors being experienced using the UCS Manager GUI, in the navigation pane expand the desired chassis and select the desired server, then from the Inventory list select the Memory tab. Memory errors on that server are indicated on the screen. You can also check Memory Environmental Statistics under Statistics -> Chart and expand the relevant Memory array.

You can check memory information via the CLI using the following commands:

scope server x/y -> show memory detail scope server x/y -> show memory-array detail scope server x/y -> scope memory-array x -> show stats history memory-array-env-stats detail

Confirm that the amount of memory seen from the OS point-of-view matches what is listed for the server’s associated service profile. Can the OS see all the memory or just part of the memory? If possible, run a memory diagnostic tool from the OS.

The top of Figure 7-1 shows a DIMM that is correctly inserted and latched. Unless there happens to be a small bit of dust blocking one of the contacts, this DIMM should function correctly. The next view below it shows a DIMM mismatched with the key for its slot; the DIMM cannot be inserted in this orientation and needs to be rotated to fit into the slot. In the next illustration down, the left side seems

7-4Cisco UCS Troubleshooting Guide

OL-20137-04

Page 49: Cisco UCS Troubleshooting Guide

Send document comments to ucs -doc feedback@c i sco .com

Chapter 7 Troubleshooting Server Hardware IssuesDIMM Memory Issues

to be correctly seated and the latch is fully connected, but the right side is just barely touching the slot and the latch is not seated into the notch on the DIMM. In the bottom view, the left is again fully inserted and seated, and the right side is partially inserted and incompletely latched.

Figure 7-1 Checking DIMM Insertion

279121

7-5Cisco UCS Troubleshooting Guide

OL-20137-04

Page 50: Cisco UCS Troubleshooting Guide

Send document comments to ucs -doc feedback@c i sco .com

Chapter 7 Troubleshooting Server Hardware IssuesDIMM Memory Issues

Recommended Solutions for DIMM IssuesTable 7-1 contains a list of guidelines and recommended solutions that can assist you in troubleshooting DIMM issues. These suggested solutions include those that are described in the “Known Issues” section on page 7-2 and the “Troubleshooting DIMM Errors” section on page 7-4.

Table 7-1 DIMM Issues

Issue Recommended Solution

DIMM is not recognized Verify that the DIMM is in a slot supporting an active CPU.

Verify that the DIMM is sourced from Cisco. Third-party memory is not supported in UCS systems.

DIMM will not fit in slot Verify that the DIMM is supported on that server model

Verify that the DIMM is oriented correctly in the slot. DIMMs and their slots are keyed and will only seat in one of the two possible orientations

DIMM is reported as bad in the SEL, POST or LEDs

DIMM is reported as inoperable in the UCS Manager GUI or CLI

Verify that the DIMM is supported on that server model.

Verify that the DIMM is populated in its slot according to the population rules for that server model.

Verify that the DIMM is seated fully and correctly in its slot. Reseat it to assure a good contact and rerun POST.

Verify that the DIMM is the issue by trying it in a slot that is known to be functioning correctly.

Verify that the slot for the DIMM is not damaged by trying a DIMM that is known to be functioning correctly in the slot.

Upgrade to UCS Manager version 1.2.(1b) or later.

Reset the BMC.

7-6Cisco UCS Troubleshooting Guide

OL-20137-04

Page 51: Cisco UCS Troubleshooting Guide

Send document comments to ucs -doc feedback@c i sco .com

Chapter 7 Troubleshooting Server Hardware IssuesCPU Issues

CPU IssuesAll UCS servers support 1-2 or 1-4 CPUs. A problem with a CPU can cause a server to fail to boot, run very slowly, or cause serious data loss or corruption. If CPU issues are suspected, consider the following:

• All CPUs in a server should be the same type, running at the same speed and with the same number and size DIMMs populated.

• If the CPU was recently replaced or upgraded, be certain the new CPU is compatible with the server and that a BIOS supporting the CPU was installed. Refer to the server’s documentation for a list of supported Cisco models and product IDs, and only use CPUs supplied by Cisco. BIOS version information can be found in the release notes for a software release.

• When replacing a CPU, great care must be taken to correctly thermally bond the CPU and the heat sink. An overheating CPU will produce fault messages visible in UCS Manager. It will also throttle back on performance in order to prevent damage to itself.

• Check the baffles and airflow for all servers in a chassis if CPU overheating is suspected. Airflow problems in adjacent servers have been known to cause improper CPU cooling in a server.

• CPU speed and memory speed should match. If they do not match, the server runs at the slower of the two speeds.

• In the event of a failed CPU, the remaining active CPU or CPUs will not have access to memory assigned to the failed CPU.

DIMM is reported as degraded in the GUI or CLI, or is running slower than expected

Reset the BMC.

Reseat the blade server in the chassis.

Verify that all DIMMs can rung at the same speed. If a slower DIMM is added to a system that had used faster DIMMs, all DIMMs on a server will run at the slower speed.

DIMM is reported as overheating Verify that the DIMM is seated fully and correctly in its slot. Reseat it to assure a good contact and rerun POST.

Verify that all empty HDD bays, server slots, and power supply bays use blanking covers to assure that the air flows as designed.

Verify that the server air baffles are installed to assure that the air flows as designed.

Verify that any needed CPU air blockers are installed to assure that the air flows as designed. (B440 servers use these for un used CPU slots.)

Table 7-1 DIMM Issues (continued)

Issue Recommended Solution

7-7Cisco UCS Troubleshooting Guide

OL-20137-04

Page 52: Cisco UCS Troubleshooting Guide

Send document comments to ucs -doc feedback@c i sco .com

Chapter 7 Troubleshooting Server Hardware IssuesTroubleshooting CPU Issues

Troubleshooting CPU IssuesTo determine the type of CPU errors being experienced using the UCS Manager GUI, in the navigation pane expand the desired chassis and select the desired server, then in the Inventory window select the CPU tab. CPU errors on that server will be indicated on the screen.

You can check CPU information via the CLI using the following commands:

scope server x/y -> show cpuscope server x/y -> show biosscope server x/y -> show cimc

Recommended Solutions for CPU IssuesTable 7-2 contains a list of guidelines and recommended solutions that can assist you in troubleshooting CPU issues.

Table 7-2 DIMM Issues

Issue Recommended Solution

CPU will not fit in slot Verify that the CPU is supported on that server model.

Verify that the CPU is oriented correctly in the slot. DIMMs and their slots are keyed and will only seat in one of the two possible orientations

DIMM is reported as bad in the SEL, POST or LEDs

DIMM is reported as inoperable in the UCS Manager GUI or CLI

Verify that the DIMM is supported on that server model.

Verify that the DIMM is populated in its slot according to the population rules for that server model.

Verify that the DIMM is seated fully and correctly in its slot. Reseat it to assure a good contact and rerun POST.

Verify that the DIMM is the issue by trying it in a slot that is known to be functioning correctly.

Verify that the slot for the DIMM is not damaged by trying a DIMM that’s known to be functioning correctly in the slot.

Upgrade to UCS Manager version 1.2.(1b) or later.

Reset the BMC.

7-8Cisco UCS Troubleshooting Guide

OL-20137-04

Page 53: Cisco UCS Troubleshooting Guide

Send document comments to ucs -doc feedback@c i sco .com

Chapter 7 Troubleshooting Server Hardware IssuesDisk Drive and RAID Issues

Disk Drive and RAID IssuesA problem with the disk drive or RAID controller can cause a server to fail to boot, or cause serious data loss or corruption. If drive issues are suspected, consider the following:

• Regular use of OS tools to detect and correct drive problems like bad sectors is encouraged. UCS Manager cannot correct anything the server’s OS will not correct better and faster.

• Each disk drive has an Activity LED indicating an outstanding I/O operation to the drive, and a health LED that turns solid Amber if there is a drive fault detected. Drive faults may be detected in the BIOS POST, and SEL messages may contain important information to finding these problems.

• Disk drives are the only major component that can be removed from the server without removing the blade from the system chassis. Just because something can be done doesn’t mean it should be done. Removing one or more disk drives from an active, commissioned server for any length of time may have serious impacts on the RAID controller, UCS Manager service profile, and on the drives themselves. Always decommission a server before removing a disk drive.

• Disk drives are available in several sizes, and if performance is slow because the drive is full or there are issues with the drive that the OS can not solve, you may have no choice but to back up the drive contents and install a larger or new hard drive. UCS servers are like most servers in this respect.

How to Determine Which RAID Controller Is in Your ServerThe B-Series servers can be ordered or configured with a number of RAID controller options:

• UCS B200 and B250 servers have an Intel ICH10R onboard SATA controller on the motherboard. This controller supports RAID 0 and 1 for up to two SATA drives. This controller must be enabled in UCS Manager before you can use RAID, and all RAID options are configurable from UCS Manager.

DIMM is reported as degraded in the GUI or CLI

Reset the BMC.

Reseat the blade server in the chassis.

Verify that all DIMMs are running at the same speed. If a slower DIMM is added to a system that had used faster DIMMs, all DIMMs on a server run at the slower speed.

DIMM is reported as overheating Verify that the DIMM is seated fully and correctly in its slot. Reseat it to assure a good contact and rerun POST.

Verify that all empty HDD bays, server slots, and power supply bays use blanking covers to assure that the air flows as designed.

Verify that the server air baffles are installed to assure that the air flows as designed.

Table 7-2 DIMM Issues

Issue Recommended Solution

7-9Cisco UCS Troubleshooting Guide

OL-20137-04

Page 54: Cisco UCS Troubleshooting Guide

Send document comments to ucs -doc feedback@c i sco .com

Chapter 7 Troubleshooting Server Hardware IssuesDisk Drive and RAID Issues

• UCS B440 servers have an LSI MegaRAID controller card (model varies by server). Depending on the license key installed, these cards provide RAID 0, 1, 5, 6, 10, 50, and 60 support for up to four SAS or SATA drives.

If you do not have a record of which option is used in the server, you can disable quiet boot and read the on-screen messages that are displayed during system bootup.

• Information about the models of RAID controller installed are displayed as part of the verbose boot. You are also prompted to press Ctrl-H to launch configuration utilities for those cards. See also How to Disable Quiet Boot, page 7-10.

• If no models of card are displayed after you disable quiet boot but there is a RAID configuration, your server is using the onboard ICH10R controller. You are also prompted to press Ctrl-M to launch the configuration utilities for this controller (see Figure 7-2). See also How To Launch Option ROM-Based Controller Utilities, page 7-11.

Figure 7-2 Startup Screen for the ICH10R Controller Configuration Utilities

How to Disable Quiet Boot To disable quiet boot so that the controller information and the prompts for the option ROM-based LSI utilities are displayed during bootup, follow these steps:

Step 1 Boot the server and watch for the F2 prompt during bootup.

Step 2 Press F2 when prompted to enter the BIOS Setup utility.

Step 3 On the Main page of the BIOS Setup utility, set Quiet Boot to Disabled. This allows non-default messages, prompts, and POST messages to display during bootup instead of the Cisco logo screen.

Step 4 Press F10 to save your changes and exit the utility.

7-10Cisco UCS Troubleshooting Guide

OL-20137-04

Page 55: Cisco UCS Troubleshooting Guide

Send document comments to ucs -doc feedback@c i sco .com

Chapter 7 Troubleshooting Server Hardware IssuesDisk Drive and RAID Issues

How To Launch Option ROM-Based Controller UtilitiesTo alter the RAID configurations on your hard drives, you can use your host-based utilities that you install on top of your host OS, or you can use the LSI option ROM-based utilities that are installed on the server.

When you boot the server and you have quite boot disabled (see How to Disable Quiet Boot, page 7-10), information about your controller is displayed along with the prompts for the key combination to launch the option ROM-based utilities for your controller.

Watch for the prompt for your controller during verbose boot:

• The prompt for LSI controller card utility is Ctrl-H.

• The prompt for the onboard Intel ICH10R controller utility is Ctrl-M.

For More InformationThe LSI utilities have help documentation for more information about using the utilities.

For basic information on RAID and how to use the LSI utilities, see the documentation at LSI.com:

• LSI MegaRAID SAS Software User’s Guide (for LSI MegaRAID)

http://www.lsi.com/DistributionSystem/AssetDocument/80-00156-01_RevH_SAS_SW_UG.pdf

• LSI Fusion-MPT Device Management User’s Guide (for LSI 3081E)

http://www.lsi.com/DistributionSystem/AssetDocument/documentation/insight_center/tech_trends/fusionmpt/FusionMPT_DevMgrUG.pdf

• LSI Integrated RAID for SAS User’s Guide (for LSI 1064)

http://www.lsi.com/DistributionSystem/AssetDocument/files/docs/techdocs/storage_stand_prod/sas/ir_sas_ug.pdf

Moving a RAID Cluster This section explains how to set a server to recognize a RAID array created on another server. This procedure is useful when upgrading from an M1 server to an M2 server, or any other time data on a RAID array will need to be moved between servers. An array which created on some other server and not recognized on its current server is referred to as a foreign array. A Native array is an array which is ‘Active’ and recognized on a given server.

Step 1 Keep both the starting and destination servers for the RAID cluster in the associated state and shut them down. The Service Profiles for both servers must have an identical Local Disk Configuration Policy.

Note When using this procedure during an M1 to M2 upgrade or direct replacement within a slot, the destination server will not be associated yet or have a disk policy, but instead will inherit the same policies as the starting server when it is inserted into the same slot the starting server was in.

Step 2 Once they are powered-off, physically move the drives in the array to the destination server. If you are changing servers but keeping the drives in the same slot, at this point, insert the new server into the slot of the original server.

Step 3 Connect the KVM dongle and a monitor, keyboard, and mouse to the destination server.

7-11Cisco UCS Troubleshooting Guide

OL-20137-04

Page 56: Cisco UCS Troubleshooting Guide

Send document comments to ucs -doc feedback@c i sco .com

Chapter 7 Troubleshooting Server Hardware IssuesAdapter Issues

Step 4 Boot the destination server, using the power switch on the front of the server. If necessary, disable quiet boot and boot again. (See How to Disable Quiet Boot, page 7-10.)

Step 5 Wait for the LSI Configuration Utility banner.

Step 6 Enter the LSI Configuration Utility by selecting Ctrl-C.

Step 7 Select the SAS Adapter used in your server from the SAS Adapter List. In needed, refer to the “How to Determine Which RAID Controller Is in Your Server” section on page 7-9.

Step 8 Select RAID Properties. This brings up the View Array screen.

Step 9 Select Manage Array. This brings up the Manage Array screen.

Step 10 Select ‘Activate Array’ from the Manage Array screen. Once the activation is complete, the RAID status will change to ‘Optimal’.

Step 11 Select the ‘Synchronize Array’ option from the Manage Array screen.

Step 12 Wait for the Mirror Synchronization to complete, and monitor the progress bar that comes up. Please note that the time to complete the Synchronization can vary depending upon the size of the disks in the RAID array.

Step 13 Once the Mirror Synchronization is complete, press the ESC key several times to go back through each of the windows (one at a time) and then exit the LSI Configuration Utility, selecting the “reboot” option to implement the changes.

Adapter IssuesA problem with the Ethernet or FCoE adapter can cause a server to fail to connect to the network and make it unreachable from UCS Manager. Mechanically, all adapters are unique Cisco designs and non- Cisco adapters are not available, although it could be possible to counterfeit one. If adapter issues are suspected, consider the following:

• Is the adapter a genuine Cisco Adapter? Where and from whom was it purchased?

• Is the adapter type supported in the software release you are using? The Internal Dependencies table in the UCS Manager 1.3(1) Release Notes provides minimum and recommended software versions for all adapters.

• Has the appropriate firmware for the adapter been loaded on the server? In software release versions 1.0(1) through 1.3(1), the UCS Manager version and the adapter firmware version need to match. To update the Cisco UCS software and firmware, refer to the appropriate Upgrading Cisco UCS from Release ____ to Release 1.3(1) document for your installation.

• If the software version update was incomplete, and the firmware version no longer matches the UCS Manager version, try updating the Adapter firmware as described in the Cisco UCS Manager CLI Configuration Guide’s “Managing Firmware“ chapter.

• To deploy two UCS M81KR Virtual Interface Cards on the UCS B250 Extended Memory Blade Server running ESX 4.0 you must upgrade to patch 5 (ESX4.0u1p5) or later release of ESX 4.0.

• If you are migrating from one adapter type to another, as well as making sure the needed drivers for the new adapter type are available, make sure that you update the Service profile to match the new adapter type, and configure services appropriate to that adapter type.

• If you are using dual adapters, be aware there are certain restrictions on the combinations supported. The following combinations are supported:

7-12Cisco UCS Troubleshooting Guide

OL-20137-04

Page 57: Cisco UCS Troubleshooting Guide

Send document comments to ucs -doc feedback@c i sco .com

Chapter 7 Troubleshooting Server Hardware IssuesAdapter Issues

Known IssuesThere are a number of known issues and open bugs with Adapters called out in the Release Notes documentation, refer to the document for your software release. The following is a persistent known condition:

(CSCtd32884 and CSC71310) The type of adapter in a server impacts the maximum MTU supported. Network MTU above the maximums may cause the packet to be dropped for the following adapters:

• The Cisco UCS CNA M71KR adapter supports a maximum MTU of 9216.

• The Cisco UCS 82598KR-CI adapter supports a maximum MTU of 14000.

Troubleshooting Adapter ErrorsThe Link LED on the front of the server is off if the adapter is cannot establish even one network link, and green if one or more links are active. Adapter errors are also reported in the LEDs on the motherboard discussed in the “Diagnostics Button and LEDs” section on page 7-1.

To determine the type of adapter errors being experienced using the UCS Manager GUI, in the navigation pane expand the desired chassis and select the desired server, then in the Inventory window select the Interface Cards tab. Adapter errors on that server will be indicated on the screen.

You can check Adapter state information via the CLI using the following commands:

scope server x/y -> show adapter scope server x/y -> show adapter detail

Server Dual card same type Dual card mixed type

UCS B250 All M71KR-Q or -E + M81KR M72KR-Q or -E + M81KR

UCS B440 All except 82598KR-CI M72KR-Q or -E + M81KR

7-13Cisco UCS Troubleshooting Guide

OL-20137-04

Page 58: Cisco UCS Troubleshooting Guide

Send document comments to ucs -doc feedback@c i sco .com

Chapter 7 Troubleshooting Server Hardware IssuesGathering Information Before Calling Support

Recommended SolutionsTable 7-3 contains a list of guidelines and recommended solutions that can assist you in troubleshooting DIMM issues. These suggested solutions include those that are described in the “Known Issues” section on page 7-2 and the “Troubleshooting DIMM Errors” section on page 7-4.

Gathering Information Before Calling SupportIf you cannot isolate the issue to a particular component, please consider the following questions as they may be helpful when contacting Cisco Technical Assistance Center (TAC).

1. Was the blade working before the problem occurred? Did the problem occur while the blade was running with a service profile associated?

2. Was this a newly inserted blade?

3. Was this blade assembled on-site or did it arrive assembled from Cisco?

4. Has the memory been re-seated?

5. Was the blade powered down or moved from one slot to another slot?

6. Have there been any recent UCS Manager upgrades and if so was the BIOS upgraded as well?

Table 7-3 Adapter Issues

Issue Recommended Solution

Adapter is reported as bad in the SEL, POST or LEDs

Adapter is reported as inoperable in the UCS Manager GUI or CLI

Verify that the adapter is supported on that server model.

Verify that the adapter has the required firmware version to work with your version of UCS Manager.

Verify that the adapter is seated fully and correctly in its slot on the motherboard and in its midplane connections. Reseat it to assure a good contact, reinsert the server, and rerun POST.

Verify that the Adapter is the issue by trying it in a server that is known to be functioning correctly and that uses the same adapter type.

Adapter is reported as degraded in the GUI or CLI

Reseat the blade server in the chassis.

Adapter is reported as overheating Verify that the adapter is seated fully and correctly in its slot. Reseat it to assure a good contact and rerun POST.

Verify that all empty HDD bays, server slots, and power supply bays use blanking covers to assure that the air flows as designed.

Verify that the server air baffles are installed to assure that the air flows as designed.

7-14Cisco UCS Troubleshooting Guide

OL-20137-04

Page 59: Cisco UCS Troubleshooting Guide

Send document comments to ucs -doc feedback@c i sco .com

Chapter 7 Troubleshooting Server Hardware IssuesRelated Documents

When contacting Cisco TAC for UCS issues it is always important to capture the tech-support output from the UCS Manager instance and chassis in question (for example):

UCS1-A# connect local-mgmt UCS1-A(local-mgmt)# show tech chassis X all detail <--where X is the chassis having the blade with the DIMM issue.UCS1-A(local-mgmt)# show tech ucsm detail

Related DocumentsIndividual server models are documented in the Cisco UCS Blade Server Installation and Service Notes.

7-15Cisco UCS Troubleshooting Guide

OL-20137-04

Page 60: Cisco UCS Troubleshooting Guide

Send document comments to ucs -doc feedback@c i sco .com

Chapter 7 Troubleshooting Server Hardware IssuesRelated Documents

7-16Cisco UCS Troubleshooting Guide

OL-20137-04

Page 61: Cisco UCS Troubleshooting Guide

Send document comments to ucs -doc feedback@c i sco .com

OL-20137-04

C H A P T E R 8

Cisco UCS Error Messages

This chapter contains a list of error messages that you may see in a Cisco UCS instance. This chapter contains the following sections:

• Overview, page 8-1

• Information and Warning Messages, page 8-2

• FSM Messages, page 8-19

This chapter does not contain information about Cisco UCS faults. Documentation about faults, including explanations and recommended actions, are available in the Cisco UCS Faults Reference.

OverviewCisco UCS error messages typically display in Cisco UCS Manager GUI and Cisco UCS Manager CLI.

These error messages are specific to the action that a user is performing or the object that a user is configuring or administering. These messages can be the following:

• Informational messages, providing assistance and tips about the action being performed

• Warning messages, providing information about system errors related to an object, such as a user account or service profile, that the user is configuring or administering

• FSM status messages, providing information about the status of an FSM stage

Many error messages contain one or more variables. What information that Cisco UCS Manager uses to replace these variables depends upon the context in which you see the message. Some messages can be generated by more than one type of error.

8-1Cisco UCS Troubleshooting Guide

Page 62: Cisco UCS Troubleshooting Guide

Send document comments to ucs -doc feedback@c i sco .com

Chapter 8 Cisco UCS Error MessagesInformation and Warning Messages

Information and Warning MessagesFailed to get the current time.

An snprintf process failed.

Create-only and naming properties cannot be modified after creation (class=%s, property=%s).

Cannot execute SAM pin command (exit code %d).

Cannot execute SAM unpin command (exit code %d).

Failed to execute reload all command (exit code %d).

A lower privilege user cannot delete a session of higher privilege user.

Provider order numbers must be unique.

Failed to get aaa:RadiusEp object.

Failed to get aaa:AuthRealm object.

RADIUS providers cannot be deleted while authentication realm is set to use RADIUS.

The order of RADIUS providers cannot be changed while authentication realm is set to use RADIUS.

The system does not allow more than %d providers.

TACACS providers cannot be deleted while authentication is set to use TACACS.

The order of TACACS providers cannot be changed while authentication realm is set to use TACACS.

Failed to get aaa:LdapEpMo object.

LDAP providers cannot be deleted while authentication is set to use LDAP.

The order of LDAP providers cannot be changed while authentication realm is set to use LDAP.

Failed to get aaa:Ep object.

Duplicate user MO.

User %s was not found.

Failed to cast to UserMo, user %s.

Failed to create UserLoginMo, user %s.

Failed to get shadow password entry.

Failed to get aaa::Realm object.

Failed to get aaa::UserEp object.

The SSH key format is invalid.

8-2Cisco UCS Troubleshooting Guide

OL-20137-04

Page 63: Cisco UCS Troubleshooting Guide

Send document comments to ucs -doc feedback@c i sco .com

Chapter 8 Cisco UCS Error MessagesInformation and Warning Messages

Expiration date must be set to expire the user account.

Failed to convert date into internal format.

Valid year value is between 1970 and 2037.

User account expiration can be set only for future.

Cannot delete locale ’%s’ with a reference in user account ’%s’.

System does not allow more than %d locales.

Failed to cast to AAA organization MO.

Referred Organization MO does not exist.

Cannot delete role ’%s’ with a reference in user account ’%s’.

System does not allow more than %d roles.

Role name ’%s’ is a reserved word.

’%s’ role should have at least one privilege.

System does not allow more than %d total privileges across all roles.

Role %s cannot be deleted from user %s.

Role %s was not found.

locale %s was not found.

User %s cannot delete itself.

System does not allow more than %d users.

User name ’%s’ is a reserved word.

%s account cannot be modified by a lower privileged account.

System does not allow deleting %s account.

%s account does not expire.

System does not allow modifying roles for %s account.

System does not allow modifying locales for %s account.

’%s’ role cannot be deleted.

’%s’ role cannot be modified, priv %d.

User %s’s privileges (admin, aaa, fault or operations) and locale assignment are incompatible.

User %s’s privileges and locale assignment are incompatible.

Admin privilege can be assigned to a user only by another user with admin privilege.

8-3Cisco UCS Troubleshooting Guide

OL-20137-04

Page 64: Cisco UCS Troubleshooting Guide

Send document comments to ucs -doc feedback@c i sco .com

Chapter 8 Cisco UCS Error MessagesInformation and Warning Messages

A contact must be specified.

%s email address must be specified.

%s email address must contain an ’@’

%s email address must contain name before ’@’

%s email address must contain hostname after ’@’

%s email address must contain only one ’@’.

Must specify a contact phone number.

The contact phone number must be in international format (e.g., +1-800-123-4567).

A contact address must be specified.

An SMTP server address must be specified.

Number of Call Home profiles must not exceed %d.

Cannot delete default profiles.

The format of a full-text profile must be full-text.

The format of short-text profile must be short-text.

The format of the Cisco TAC profile must be XML.

Only Cisco TAC alert is allowed with built-in Cisco TAC profile.

Cannot obtain callhome::SourceMo!

Cannot find callhome::SmtpMo!

Cannot cast to callhome::EpMo!

%s

%d

%d

The chassis decommission is in progress; wait for it to complete.

Cannot mark %s out of service.

Port %s is already allocated to %s.

Slot identity is being established. Try again later.

The number of VSANs on a fabric interconnect cannot exceed %d.

SwitchId property needs to be set to the ID of corresponding fabric interconnect domain, MO id:%u, required id:%u.

Explicit deletion of FC Port VSAN object is prohibited; create under a different VSAN.

8-4Cisco UCS Troubleshooting Guide

OL-20137-04

Page 65: Cisco UCS Troubleshooting Guide

Send document comments to ucs -doc feedback@c i sco .com

Chapter 8 Cisco UCS Error MessagesInformation and Warning Messages

The default VSAN object was not found.

The default cannot be deleted.

Configuration for FC Port %u/%u does not exist for fabric interconnect %s.

VSAN %u (of network %s) and %u (of network %s) - both cannot share the FCoE VLAN %u.

Networks %s and %s - both cannot share same VSAN id (%u) with different FCoE VLAN ids (%u and %u respectively).

Logical Server EP managed object is NULL.

Only one VSAN per port supported.

Number of VLANs on a fabric interconnect cannot exceed %d.

Port-Channel %u contains %u ports, maximum allowed is %u.

Number of port-channels defined: %u, maximum allowed is %u.

VNET %d is already configured as inband management VNET.

Internal error: 1G speed is invalid for the port %u/%u.

Internal error: cannot create fabric interconnect configuration %u/%u MO.

Internal error: cannot create fabric interconnect configuration port-channel %u MO.

Default VLAN cannot be deleted.

Circuit can be reset only on virtualized adapters.

ClusterState request failed, state not initialized.

Motherboard of blade %s was not found.

Motherboard of blade %d/%d was not found.

Blade %s.

No adaptor %s in blade %s.

Motherboard of blade %s.

Blade %s was not found.

Cannot find blade %d/%d.

Empty FRU information; cannot create Chassis/IOM objects.

Fabric interconnect:%u slot:%u port:%u not a serverport.

Fabric interconnect:%u slot:%u port:%u is not operationally up.

No chassis ID is available.

No fabric interconnect %d.

8-5Cisco UCS Troubleshooting Guide

OL-20137-04

Page 66: Cisco UCS Troubleshooting Guide

Send document comments to ucs -doc feedback@c i sco .com

Chapter 8 Cisco UCS Error MessagesInformation and Warning Messages

FE OFFLINE: Cannot find chassis %d.

FE FW UPGRADE: Cannot find chassis %d.

Default VSAN object not found.

Chassis %d was not found.

Server %d/%d was not found.

Server facing FE port %d/%d/%d was not found.

Cannot find chassis %d.

Profile ’%s’ is currently used by some Virtual Machines; cannot delete.

Not able to find sysdebug::CoreFileRepositoryMo.

Failed to generate key ring.

GenerateKey failed.

Failed to generate self signed certificate.

Failed to verify certificate with private key.

Failed to store certificate.

Failed to read certificate file.

Failed to open certificate file.

Failed to generate certificate signing request.

GetCsrFilename Error.

Failed to setReq for certificate signing request.

GenerateCSR failed.

Failed to set key ring for HTTPS service.

Key ring exists, modulus cannot be changed after creation.

Verify certificate error: %s.

Modulus must be set before creating certificate request for key ring %s.

Incorrect MO access.

System does not allow more than %d key rings.

Cannot delete a key ring %s that is in use.

Trustpoint cannot be set for default key ring.

Trustpoint must be set before adding a certificate for key ring %s.

Trustpoint %s does not exist setfor key ring %s.

Failed to verify certificate chain, error: %s.

8-6Cisco UCS Troubleshooting Guide

OL-20137-04

Page 67: Cisco UCS Troubleshooting Guide

Send document comments to ucs -doc feedback@c i sco .com

Chapter 8 Cisco UCS Error MessagesInformation and Warning Messages

Failed to get fingerprint(s).

You cannot delete a trustpoint that is in use.

Failed to get PKI:Ep object.

Cannot cast to sysdebug::LogControlEpMo

Failed to get PKI Ep MO.

Key ring %s does not exist.

Key ring %s certificate is not set.

Invalid WS-Management MO.

Invalid HTTPS/HTTP MO.

Port is in use or it is a reserved port.

Failed to set HTTP port.

Failed to set HTTPS port.

A hostname must be specified to configure external an Syslog server.

The system does not allow more than %d SNMP users.

A system user exists with the same name; choose a different name for the SNMP user.

The privacy password must be a minimum of %d characters.

Privacy password strength check: %s.

The privacy password must be set before enabling AES use.

Only one instance of SNMP MO can exist.

All strings must be specified to configure SNMP service.

SNMP MO is missing.

The system cannot create more than %d SNMP trap hosts.

A community string must be set to configure an SNMP trap host.

Only one instance of DNS MO can exist.

Failed in retrieving DNS MO.

DNS MO is missing.

System does not allow more than %d DNS providers.

Only one instance of DateTime MO can exist.

Failed in retrieving DateTime MO.

Invalid timezone.

DateTime MO is missing.

8-7Cisco UCS Troubleshooting Guide

OL-20137-04

Page 68: Cisco UCS Troubleshooting Guide

Send document comments to ucs -doc feedback@c i sco .com

Chapter 8 Cisco UCS Error MessagesInformation and Warning Messages

System does not allow more than %d NTP providers.

Failed to get comm::SvcEp object.

Failed to restart HTTP server.

Configured object (%s) not found.

Failing explicitly for PROPSETROLLBACK

Failing explicitly for NEWOBJROLLBACK

Failing explicitly for DELOBJROLLBACK

Failing explicitly for CREOBJDELOBJ

Failing explicitly for DELOBJCREOBJ

Image %s is currently being installed.

Image %s is currently installed and in use.

Image %s is currently installed or in use.

DistImage MO %s does not exist in Management Information Tree.

Image for Vendor %s, Model %s and Version %s not found. Download and try again.

Unable to send replication/sync error message.

Invalid leader state %d.

Invalid member state %d.

Failed to templatize "%s" to org "%s" template "%s"

Organization "%s" is unresolvable.

Server "%s" is unresolvable.

Failed to instantiate template "%s" to org "%s" server "%s"

Failed to clone "%s" to org "%s" server "%s"

A DVS named ’%s’ already exists under given vCenter.

A DVS named ’%s’ is under process of deletion, wait until deletion completes.

vCenter ’%s’ and ’%s’ cannot have same host ’%s’

Deletion of vCenter ’%s’ with same host ’%s’ in in progress, wait until it gets deleted.

Internal error: Singleton managing external VM management not found.

Internal error: Extension key not found.

Internal error: Extension key not set.

Resource allocation error: Cannot claim any extension key resource for this DVS.

8-8Cisco UCS Troubleshooting Guide

OL-20137-04

Page 69: Cisco UCS Troubleshooting Guide

Send document comments to ucs -doc feedback@c i sco .com

Chapter 8 Cisco UCS Error MessagesInformation and Warning Messages

Internal error: Unable to create deletion task for DVS %s.

Maximum ports per DVS cannot exceed %u; you must reduce max-ports property of port-profiles.

Only one certificate is allowed at this point.

Certificate file name must not be empty.

Invalid certificate file location.

Cannot change extension key while there are pending DVS deletions.

Cannot change extension key while in use.

Storage controller %d/%d was not found.

Chassis %d: fan module %d was not found.

No fabric extender in chassis %d slot %d.

No blade was found in chassis %d slot %d.

Unknown type %d.

No adaptor %u was found in blade %u chassis %u.

Chassis %d: fan module %d: fan %d was not found.

No CMC in chassis %d slot %d.

Failed to assign VIF ID.

Fabric interconnect port ’%s’ was not found.

MUX server port was not found.

Blade %d/%d was not found or is out of service.

An IP address, netmask, and gateway must be configured for out-of-band configuration.

A virtual network, IP address, netmask, and gateway must be configured for inband configuration.

A virtual IP address must be configured for virtual IP configuration.

Chassis was marked for decommission, decommission is in progress.

Org ’%s’ cannot be deleted.

Org ’%s’ cannot be deleted. No locale access.

Root level org cannot be created.

FC COS cannot be applied to the vNIC.

Only FC COS can be applied to the vHBA.

Root org was not found.

8-9Cisco UCS Troubleshooting Guide

OL-20137-04

Page 70: Cisco UCS Troubleshooting Guide

Send document comments to ucs -doc feedback@c i sco .com

Chapter 8 Cisco UCS Error MessagesInformation and Warning Messages

Only Ethernet best-effort class can match against ’any’ COS value.

Two classes cannot have the same COS value (%d).

Only one class can have a no-drop policy.

Only one class can have a multicast-optimize policy.

Only one class can have a strict bandwidth priority policy.

At least one (enabled) class should have non-zero weight.

Unable to find Peer fabric interconnect object.

Unable to find Management Controller object for peer network element.

Unable to find Management Interface object for peer network element.

Cannot assign the same IP address for both fabric interconnects.

The netmask for both fabric interconnects must be the same.

Out-of-band IP address and virtual IP address must be different.

Adaptor unit [%s] is not contained by BladeMo parent.

Blade server [%s] is not contained by ChassisMo parent.

Network "%s" does not exist.

Invalid vHBA name; the provided name is reserved for FC node.

Dynamic vNIC [%s] cannot be modified.

Invalid prefix of profile name.

Maximum number of port-profiles cannot exceed %u.

Address %s is already assigned to %s.

Pooled address is unknown.

Bad address block range definition collision.

Block definition is too large. Size cannot exceed %u.

Bad address block range definition.

Address is already assigned to %s.

Inconsistent pool definition - the IP address and default gateway must be in same network.

Bad address block range - not in the management subnet.

Block cannot span multiple octets.

Cannot resolve %s: class "%s" empty prefix.

Cannot resolve %s: class "%s" RN has no components.

8-10Cisco UCS Troubleshooting Guide

OL-20137-04

Page 71: Cisco UCS Troubleshooting Guide

Send document comments to ucs -doc feedback@c i sco .com

Chapter 8 Cisco UCS Error MessagesInformation and Warning Messages

Cannot resolve %s: class "%s" does not have a naming definition.

Cannot resolve %s: class "%d" not found.

Blade is already marked out-of-service, ch: %u, slot: %u.

Cannot perform migration when blade is not in service, %s.

Blade is still physically present; removal is not permitted, ch: %u, slot: %u.

Diagnostics image is not available.

No available virtual interfaces.

No available virtual interface found: cannot claim.

Failed to allocate virtual interface.

Failed to allocate host interface.

Unexpected class %s.

Old PN binding %s was not found.

New PN binding %s was not found.

The same profile (%s) cannot be applied to multiple virtual switches under a common organization.

Internal error: Cannot find Ethernet LAN cloud object.

Threshold policy [%s] cannot contain stats class ID [%d].

Threshold class [%s] cannot contain threshold definition for property [%d].

Unable to resolve property %d on MO %s.

Threshold definition [%s] cannot contain threshold for this property type [%s].

Threshold policy [%s] cannot be created under %s.

Default threshold policy [%s] cannot be deleted.

FindDependencies not over-ridden: (policy=%s)

Could not find vNIC ’%s’ for image path ’%s’.

Management IP of fabric interconnect A and virtual IP are not in same subnet.

The virtual IP address is a network broadcast address.

Management IP of fabric interconnect B and virtual IP are not in same subnet.

UCS Manager is running in standalone mode. Cannot configure virtual IP.

XML exception: %s

FAILED TO PARSE STREAM: %s

FAILED TO LOAD: NO ROOT IN PARSED FILE %s

8-11Cisco UCS Troubleshooting Guide

OL-20137-04

Page 72: Cisco UCS Troubleshooting Guide

Send document comments to ucs -doc feedback@c i sco .com

Chapter 8 Cisco UCS Error MessagesInformation and Warning Messages

Cannot create temporary directory.

Cannot remove the temporary file.

Cannot remove the temporary dir after backupdone.

Cannot remove the temporary file after backupdone.

Either the hostname or the remote filename is missing.

Username is required.

Chassis %u is still physically present, so removal is not permitted.

Out of chassis IDs.

Internal error. Installable was deleted while installation was in progress %s.

Internal error. Invalid type for cimcUpdateDataCb

Internal error. CIMC Installable was deleted while installation was in progress %s.

Internal error. Invalid type for cimcFwUpdateSuccessCb

Internal error. Invalid type for cimcFwUpdateFailCb

Internal error. Updatable object was not found.

Internal error. Management Controller was not found.

LS has FW policy. Modify the policy to execute update.

Unable to find CIMC Image for vendor %s, Model %s and version %s.

Invalid adaptor type to update from network.

LS has Host FW policy. Modify the policy to execute update.

Adaptor Installable not found for %s,%s,%s.

Internal error. Installable was deleted while installation was in progress: %s,%s,%s.

Unable to find IOM Image For Vendor %s, Model %s and version %s.

Invalid parent object for version update.

Invalid object for version update.

Unable to find BootUnit MO for BIOS.

Unable to find UCS Manager image for version %s.

Compatibility check failed.

Internal error. Management Installable was deleted while installation was in progress.

Internal error. Kernel Installable was deleted while installation was in progress.

8-12Cisco UCS Troubleshooting Guide

OL-20137-04

Page 73: Cisco UCS Troubleshooting Guide

Send document comments to ucs -doc feedback@c i sco .com

Chapter 8 Cisco UCS Error MessagesInformation and Warning Messages

Internal error. System Installable was deleted while installation was in progress.

Unable to find fabric interconnect kernel image for Vendor %s, Model %s and Version %s

Unable to find fabric interconnect software image for Vendor %s, Model %s and Version %s.

Internal error. Invalid type for cimcSuccessCb

Internal error. Invalid type for cimcFailCb

Internal error. Bootdefinition object not found.

Server Profile has firmware policy. Modify the policy to execute activate.

Invalid startup version %s for CIMC (%s); Current running=%s, backup=%s.

Invalid startup version %s for IOM (%s); Current running=%s, backup=%s.

Invalid adaptor type to activate from network.

LS has Host FW policy. Modify the policy to execute activate.

Invalid startup version %s for adaptor (%s); Current running=%s, backup=%s.

Unable to find installable MO for version %s.

Unknown parent object for version activate.

Root is null.

Root is not method.

lResponse is null.

DcosPers::load():DISCARDING %s[%s]: ALREADY EXISTS

DcosPers::load():DISCARDING %s[%s]: PARENT NOT FOUND (%s)

Version conflict found during replication.

Unknown Message type in Response: %d.

ERROR: Connection request received by Replicator.

Unable to create NVRAM DIR: %s

Unable to create Flash DIR: %s

FAILED TO STORE RECORDS INTO DB %d

FLASH WRITE FAILED. Check for space.

NO MEMORY.

FAILED TO CREATE/OPEN SQLITE DB: %s

CREATE TABLE FAILED

8-13Cisco UCS Troubleshooting Guide

OL-20137-04

Page 74: Cisco UCS Troubleshooting Guide

Send document comments to ucs -doc feedback@c i sco .com

Chapter 8 Cisco UCS Error MessagesInformation and Warning Messages

UNABLE TO SET PAGE SIZE

UNABLE TO SET MAX PAGES

UNABLE TO SET CACHE SIZE

UNABLE TO SET SYNCHRONOUS FLAG

UNABLE TO SET LOCK_MODE FLAG

SQL error while preparing insert statement.

SQL error while preparing delete statement.

SQL error while preparing select statement.

SQL error while preparing beginTx statement.

SQL error while preparing commitTx statement.

SQL error while Deleting record.

SQL error while Writing to database.

SQL error while Storing database version %d.

Bind failed for key: %d.

Error while Reading key %lld: %d.

Failed to delete All records %d.

Failed to prepare SelectAll statement.

Failed to Read record from SelectAll statement %d.

FAILED TO LOAD: NO ROOT IN DB

FAILED TO LOAD: NO CONFIG FOUND

FAILED TO OPEN %s for writing.

Invalid operation received %d.

Unable to open Flash DB %s.

FAILED TO LOAD: NO CONFIG FOUND %s

FAILED TO READ FILE %s

FAILED TO CREATE DIRECTORY %s

No cmd string for id = %u

No view string for id = %u

FAILED TO INITIALIZE CLI EVENT THREAD

FAILED TO LOAD SAMCLID LIBRARY

FAILED TO LOAD SAMVSH LIBRARY

8-14Cisco UCS Troubleshooting Guide

OL-20137-04

Page 75: Cisco UCS Troubleshooting Guide

Send document comments to ucs -doc feedback@c i sco .com

Chapter 8 Cisco UCS Error MessagesInformation and Warning Messages

FAILED TO INITIALIZE DCOS CLI

Warning: Recovering from dead SAM CLI daemon condition

No interface meta object for id = %u

Clause meta object for id = %u is not a Primitive object

No ActionProp meta object for id = %u

ActionProp object for id = %u is not a PropProp object

Clause meta object for id = %u is not a CommandKeyword object

No const adapter meta object for id = %u

No MO meta object for id = %u

No action meta object for id = %u

No layout prop meta object for id = %u

No formatter layout prop meta object for id = %u

No command meta object for id = %u

No syntax elem filter object for id = %u

No condition meta object for id = %u

No containment meta object for id = %u

String assign function called for bitmask type: %u

No type adapter meta object for id = %u

No type meta object for id = %u

No function meta object for id = %u

No layout section meta object for id = %u

No layout meta object for id = %u

No mode meta object for id = %u

No property meta object for id = %u

No model prop meta object for model prop %u, prop name = %s

Buffer is too large to stringify for prop name = %s (size=%u)

No clause meta object for id = %u

No parse filter meta object for id = %u

Clause meta object for id = %u is not a CommandVar object

No compound argument parser meta object for id = %u

No mode argument parser meta object for id = %u

8-15Cisco UCS Troubleshooting Guide

OL-20137-04

Page 76: Cisco UCS Troubleshooting Guide

Send document comments to ucs -doc feedback@c i sco .com

Chapter 8 Cisco UCS Error MessagesInformation and Warning Messages

No ArgParser object for id = %u

Attempt to set UUID to illegal value (uuid=%s).

Duplicate RemoteUser MO.

Failed to create RemoteUserMo, user %.s

Failed to cast to RemoteUserMo, user %s.

Password must be minimum %d characters.

Password strength check: %s.

Argument ’%s’ not found in method ’%s’ argument table.

Deserialization failed for ID %d.

No such Method %u.

Failed to get factory object %s.

Failed to construct object %s.

No class named %s.

Configuration not found for DN: %s.

Configuration not found.

Non-existing argument %d.

Class Mismatch; cannot cast.

[socket=%d] I/O Error %u

Invalid State:

abort: retries exhausted %d.

Exception encountered during processing: "%s" [%d] %s

Bulk abort: bulk retries exhausted %d.

Possible integrity problem: will retry.

Child %s of class %s is already attached. dn[%s]

Same object is already attached %s[%s]

Null RN.

Instance ID was not found.

Same object is already attached [%s].

Child %s[%s | %s](%p) of %s[%s] is already attached [%s | %s | %p].

Child %s is not MO; cannot attach.

Child %s is already attached.

8-16Cisco UCS Troubleshooting Guide

OL-20137-04

Page 77: Cisco UCS Troubleshooting Guide

Send document comments to ucs -doc feedback@c i sco .com

Chapter 8 Cisco UCS Error MessagesInformation and Warning Messages

Child %s is not configured; cannot attach.

Class=%s : prop=%s : ftype=%d

Id=%u

No such filter type: ’%s’

No such filter type: ’%u’

%s: number of components exceeds MAX.

%s NO PREFIX %p

%s : FSM STAGE %s : NOTIFICATION RECEIVED: IN WRONG STATE %s(%d)

%s : FSM STAGE %s : TIMEDOUT NOTIFICATION RECEIVED

%s : FSM STAGE %s : END-POINT UNAVAILABLE

%s : FSM STAGE %s : NON-RETRIABLE AVAILABILITY FAIL

%s : FSM STAGE %s : GENERAL FAIL

%s : FSM STAGE %s : STALE FAILED STIMULUS

%s : FSM STAGE %s : MO NOT FOUND

%s : UNKNOWN FSM STAGE %d: FAILED STIMULUS DISCARDED

%s : FSM STAGE %s : STALE STIMULUS: STIM STAMP: %llu : MO STAMP %llu

No propval meta object for id = %u

Error: call to localtime_r() returned ’%s’ - while converting date value ’%llu’

aInProp == NULL!

No iterator

Unable to create BASE DB DIR: %s

FAILED TO LOAD: UNABLE TO OPEN DB

Exception during load. Quitting

Cannot delete object of class:%s

Cannot change oper props, class:%s

Admin implicit props cannot be modified, class=%s, prop=%s

PreAuthCb: Invalid MO %s access

Cannot create non-creatable object of class:%s

Cannot change oper props

Admin implicit props cannot be modified, prop=%s

User is not privileged to config MO, class %s, MO mask: 0x%llx, Config mask: 0x%llx

8-17Cisco UCS Troubleshooting Guide

OL-20137-04

Page 78: Cisco UCS Troubleshooting Guide

Send document comments to ucs -doc feedback@c i sco .com

Chapter 8 Cisco UCS Error MessagesInformation and Warning Messages

User does not have org access to config MO, class %s

DO NOT USE STRING MUTATORS ON MOS.

Non-existing property %d

Failed to restore naming

RN inconsistent with naming properties.

Could not format RN.

RN is empty and not formatted.

[%p] SINGLE NAME IS EMPTY: "%s" mutable? %d

No naming meta

Returning empty RN

DN missing: cannot automatically resolve parent: recursive containment.

DN missing: cannot automatically resolve parent: non-unique containment path.

Parent not found %s.

Cannot create; object already exists.

Incompatible with class %s; cannot apply config; config rn is %s.

Request to create and delete object of class %s ; mod mask 0x%x

MO factory returned NULL for class: %s.

Child is not creatable: %s.

Child is not concrete: %s.

Object not found: object of class %s with RN %s.

Cannot contain: %s.

RN IS EMPTY: %s["%s" | "%s"]

Naming problem: object of class %s.

%s: Cannot explicitly create: object of class %s with RN %s, DN is %s.

%s: Cannot contain: object of class %s with RN %s, DN is %s.

Child %s cannot be added to deleted object.

Unknown managed object class %s

No such class %u

Cannot construct: possibly abstract class %s

Config factory of class %s failed

Config child of classId %d failed, class not found

8-18Cisco UCS Troubleshooting Guide

OL-20137-04

Page 79: Cisco UCS Troubleshooting Guide

Send document comments to ucs -doc feedback@c i sco .com

Chapter 8 Cisco UCS Error MessagesFSM Messages

Cannot contain MO class %s; %s

Unknown property value %s, name %s, class %s [%s]

Unknown property name %s, value %s

Prop ’%s’ not found in class ’%s’ property table

Config child of class %s failed, rn = %s

PARENT AND CHILD ARE THE SAME

DN IS EMPTY

No class meta for classId = %d

No prop meta %u

Class Mismatch (%u); cannot cast to (%u)

Unexpected mod: %d.

Cannot mark no_mod as created.

Cannot mark mod_clear as created.

Cannot mark unattached as deleted or removed.

Cannot mark created as unattached.

Cannot mark modified as created or unattached.

Cannot mark created as deleted or removed or un-attached.

Cannot mark deleted as created or unattached.

Class %s was not found.

FSM MessagesPathReset:Disable: CLASS MISMATCH(%s); fabric:LocaleEXPECTED: SKIPPING!

PathReset:Enable: CLASS MISMATCH(%s); fabric:LocaleEXPECTED: SKIPPING!

CircuitReset:EnableA: CLASS MISMATCH(%s); dcx:Vc EXPECTED: SKIPPING!

CircuitReset:EnableB: CLASS MISMATCH(%s); dcx:Vc EXPECTED: SKIPPING!

CircuitReset:DisableA: CLASS MISMATCH(%s); dcx:Vc EXPECTED: SKIPPING!

CircuitReset:DisableB: CLASS MISMATCH(%s); dcx:Vc EXPECTED: SKIPPING!

ResetFcPersBinding:Execute: CLASS MISMATCH(%s); compute:Blade EXPECTED: SKIPPING!

ActivateAdaptor:Activate: CLASS MISMATCH(%s); compute:Blade EXPECTED: SKIPPING!

ActivateAdaptor:BladePowerOn: CLASS MISMATCH(%s); compute:Blade EXPECTED: SKIPPING!

8-19Cisco UCS Troubleshooting Guide

OL-20137-04

Page 80: Cisco UCS Troubleshooting Guide

Send document comments to ucs -doc feedback@c i sco .com

Chapter 8 Cisco UCS Error MessagesFSM Messages

ActivateAdaptor:Reset: CLASS MISMATCH(%s); compute:Blade EXPECTED: SKIPPING!

Associate:BiosImgUpdate: CLASS MISMATCH(%s); compute:Blade EXPECTED: SKIPPING!

Associate:BiosPostCompletion: CLASS MISMATCH(%s); compute:Blade EXPECTED: SKIPPING!

Associate:BladeBootHost: CLASS MISMATCH(%s); compute:Blade EXPECTED: SKIPPING!

Associate:BladeBootPnuos: CLASS MISMATCH(%s); compute:Blade EXPECTED: SKIPPING!

Associate:BladePowerOn: CLASS MISMATCH(%s); compute:Blade EXPECTED: SKIPPING!

Associate:ConfigUserAccess: CLASS MISMATCH(%s); compute:Blade EXPECTED: SKIPPING!

Associate:ConfigUuid: CLASS MISMATCH(%s); compute:Blade EXPECTED: SKIPPING!

Associate:HbaImgUpdate: CLASS MISMATCH(%s); compute:Blade EXPECTED: SKIPPING!

Associate:HostOSConfig: CLASS MISMATCH(%s); compute:Blade EXPECTED: SKIPPING!

Associate:HostOSIdent: CLASS MISMATCH(%s); compute:Blade EXPECTED: SKIPPING!

Associate:HostOSPolicy: CLASS MISMATCH(%s); ls:AgentPolicy EXPECTED: SKIPPING!

Associate:NicConfigHostOS: CLASS MISMATCH(%s); compute:Blade EXPECTED: SKIPPING!

Associate:NicConfigPnuOS: CLASS MISMATCH(%s); compute:Blade EXPECTED: SKIPPING!

Associate:NicImgUpdate: CLASS MISMATCH(%s); compute:Blade EXPECTED: SKIPPING!

Associate:NicUnconfigPnuOS: CLASS MISMATCH(%s); compute:Blade EXPECTED: SKIPPING!

Associate:PnuOSConfig: CLASS MISMATCH(%s); compute:Blade EXPECTED: SKIPPING!

Associate:PnuOSIdent: CLASS MISMATCH(%s); compute:Blade EXPECTED: SKIPPING!

Associate:PnuOSLocalDiskConfig: CLASS MISMATCH(%s); compute:Blade EXPECTED: SKIPPING!

Associate:PnuOSPolicy: CLASS MISMATCH(%s); ls:AgentPolicy EXPECTED: SKIPPING!

Associate:PreSanitize: CLASS MISMATCH(%s); compute:Blade EXPECTED: SKIPPING!

Associate:PrepareForBoot: CLASS MISMATCH(%s); compute:Blade EXPECTED: SKIPPING!

Associate:Sanitize: CLASS MISMATCH(%s); compute:Blade EXPECTED: SKIPPING!

Associate:StorageCtlrImgUpdate: CLASS MISMATCH(%s); compute:Blade EXPECTED: SKIPPING!

Associate:SwConfigHostOSLocal: CLASS MISMATCH(%s); fabric:Locale EXPECTED: SKIPPING!

Associate:SwConfigHostOSPeer: CLASS MISMATCH(%s); fabric:Locale EXPECTED: SKIPPING!

Associate:SwConfigPnuOSLocal: CLASS MISMATCH(%s); fabric:Locale EXPECTED: SKIPPING!

8-20Cisco UCS Troubleshooting Guide

OL-20137-04

Page 81: Cisco UCS Troubleshooting Guide

Send document comments to ucs -doc feedback@c i sco .com

Chapter 8 Cisco UCS Error MessagesFSM Messages

Associate:SwConfigPnuOSPeer: CLASS MISMATCH(%s); fabric:Locale EXPECTED: SKIPPING!

Associate:SwUnconfigPnuOSLocal: CLASS MISMATCH(%s); fabric:Locale EXPECTED: SKIPPING!

Associate:SwUnconfigPnuOSPeer: CLASS MISMATCH(%s); fabric:Locale EXPECTED: SKIPPING!

Associate:activateAdaptorNwFw: CLASS MISMATCH(%s); compute:Blade EXPECTED: SKIPPING!

Associate:activateICIMCFw: CLASS MISMATCH(%s); compute:Blade EXPECTED: SKIPPING!

Associate:hagHostOSConnect: CLASS MISMATCH(%s); compute:Blade EXPECTED: SKIPPING!

Associate:hagPnuOSConnect: CLASS MISMATCH(%s); compute:Blade EXPECTED: SKIPPING!

Associate:hagPnuOSDisconnect: CLASS MISMATCH(%s); compute:Blade EXPECTED: SKIPPING!

Associate:resetICIMC: CLASS MISMATCH(%s); compute:Blade EXPECTED: SKIPPING!

Associate:updateAdaptorNwFw: CLASS MISMATCH(%s); compute:Blade EXPECTED: SKIPPING!

Associate:updateICIMCFw: CLASS MISMATCH(%s); compute:Blade EXPECTED: SKIPPING!

Associate:waitForAdaptorNwFwUpdate: CLASS MISMATCH(%s); compute:Blade EXPECTED: SKIPPING!

Associate:waitForICIMCFwUpdate: CLASS MISMATCH(%s); compute:Blade EXPECTED: SKIPPING!

BiosRecovery:PreSanitize: CLASS MISMATCH(%s); compute:Blade EXPECTED: SKIPPING!

BiosRecovery:Sanitize: CLASS MISMATCH(%s); compute:Blade EXPECTED: SKIPPING!

BiosRecovery:SetupVmediaLocal: CLASS MISMATCH(%s); compute:Blade EXPECTED: SKIPPING!

BiosRecovery:SetupVmediaPeer: CLASS MISMATCH(%s); compute:Blade EXPECTED: SKIPPING!

CmosReset:PreSanitize: CLASS MISMATCH(%s); compute:Blade EXPECTED: SKIPPING!

CmosReset:Sanitize: CLASS MISMATCH(%s); compute:Blade EXPECTED: SKIPPING!

Diag:BiosPostCompletion: CLASS MISMATCH(%s); compute:Blade EXPECTED: SKIPPING!

Diag:BladeBoot: CLASS MISMATCH(%s); compute:Blade EXPECTED: SKIPPING!

Diag:BladePowerOn: CLASS MISMATCH(%s); compute:Blade EXPECTED: SKIPPING!

Diag:BladeReadSmbios: CLASS MISMATCH(%s); compute:Blade EXPECTED: SKIPPING!

Diag:cimcInventory: CLASS MISMATCH(%s); compute:Blade EXPECTED: SKIPPING!

Diag:cimcPresence: CLASS MISMATCH(%s); compute:Blade EXPECTED: SKIPPING!

8-21Cisco UCS Troubleshooting Guide

OL-20137-04

Page 82: Cisco UCS Troubleshooting Guide

Send document comments to ucs -doc feedback@c i sco .com

Chapter 8 Cisco UCS Error MessagesFSM Messages

Diag:ConfigFeLocal: CLASS MISMATCH(%s); fabric:Locale EXPECTED: SKIPPING!

Diag:ConfigFePeer: CLASS MISMATCH(%s); fabric:Locale EXPECTED: SKIPPING!

Diag:ConfigSol: CLASS MISMATCH(%s); compute:Blade EXPECTED: SKIPPING!

Diag:ConfigUserAccess: CLASS MISMATCH(%s); compute:Blade EXPECTED: SKIPPING!

Diag:DebugWait: CLASS MISMATCH(%s); compute:Blade EXPECTED: SKIPPING!

Diag:DisableServerConnSwA: CLASS MISMATCH(%s); fabric:Locale EXPECTED: SKIPPING!

Diag:DisableServerConnSwB: CLASS MISMATCH(%s); fabric:Locale EXPECTED: SKIPPING!

Diag:EnableServerConnSwA: CLASS MISMATCH(%s); fabric:Locale EXPECTED: SKIPPING!

Diag:EnableServerConnSwB: CLASS MISMATCH(%s); fabric:Locale EXPECTED: SKIPPING!

Diag:HostConnect: CLASS MISMATCH(%s); compute:Blade EXPECTED: SKIPPING!

Diag:HostDisconnect: CLASS MISMATCH(%s); compute:Blade EXPECTED: SKIPPING!

Diag:HostIdent: CLASS MISMATCH(%s); compute:Blade EXPECTED: SKIPPING!

Diag:HostPolicy: CLASS MISMATCH(%s); ls:AgentPolicy EXPECTED: SKIPPING!

Diag:NicConfig: CLASS MISMATCH(%s); compute:Blade EXPECTED: SKIPPING!

Diag:NicInventory: CLASS MISMATCH(%s); compute:Blade EXPECTED: SKIPPING!

Diag:NicPresence: CLASS MISMATCH(%s); compute:Blade EXPECTED: SKIPPING!

Diag:NicUnconfig: CLASS MISMATCH(%s); compute:Blade EXPECTED: SKIPPING!

Diag:PollMemoryTestStatus: CLASS MISMATCH(%s); compute:Blade EXPECTED: SKIPPING!

Diag:RemoveVMediaPeer: CLASS MISMATCH(%s); compute:Blade EXPECTED: SKIPPING!

Diag:RestoreConfigFeLocal: CLASS MISMATCH(%s); fabric:Locale EXPECTED: SKIPPING!

Diag:RestoreConfigFePeer: CLASS MISMATCH(%s); fabric:Locale EXPECTED: SKIPPING!

Diag:StartMemoryTestStatus: CLASS MISMATCH(%s); compute:Blade EXPECTED: SKIPPING!

Diag:SwConfigLocal: CLASS MISMATCH(%s); fabric:Locale EXPECTED: SKIPPING!

Diag:SwConfigPeer: CLASS MISMATCH(%s); fabric:Locale EXPECTED: SKIPPING!

Diag:SwUnconfigLocal: CLASS MISMATCH(%s); fabric:Locale EXPECTED: SKIPPING!

Diag:SwUnconfigPeer: CLASS MISMATCH(%s); fabric:Locale EXPECTED: SKIPPING!

Diag:UnconfigSol: CLASS MISMATCH(%s); compute:Blade EXPECTED: SKIPPING!

Diag:UnconfigUserAccess: CLASS MISMATCH(%s); compute:Blade EXPECTED: SKIPPING!

Disassociate:BiosPostCompletion: CLASS MISMATCH(%s); compute:Blade EXPECTED: SKIPPING!

Disassociate:BladeBootPnuos: CLASS MISMATCH(%s); compute:Blade EXPECTED:

8-22Cisco UCS Troubleshooting Guide

OL-20137-04

Page 83: Cisco UCS Troubleshooting Guide

Send document comments to ucs -doc feedback@c i sco .com

Chapter 8 Cisco UCS Error MessagesFSM Messages

SKIPPING!

Disassociate:BladePowerOn: CLASS MISMATCH(%s); compute:Blade EXPECTED: SKIPPING!

Disassociate:BladeShutdown: CLASS MISMATCH(%s); compute:Blade EXPECTED: SKIPPING!

Disassociate:ConfigUserAccess: CLASS MISMATCH(%s); compute:Blade EXPECTED: SKIPPING!

Disassociate:NicConfigPnuOS: CLASS MISMATCH(%s); compute:Blade EXPECTED: SKIPPING!

Disassociate:NicUnconfigHostOS: CLASS MISMATCH(%s); compute:Blade EXPECTED: SKIPPING!

Disassociate:NicUnconfigPnuOS: CLASS MISMATCH(%s); compute:Blade EXPECTED: SKIPPING!

Disassociate:PnuOSIdent: CLASS MISMATCH(%s); compute:Blade EXPECTED: SKIPPING!

Disassociate:PnuOSPolicy: CLASS MISMATCH(%s); ls:AgentPolicy EXPECTED: SKIPPING!

Disassociate:PnuOSScrub: CLASS MISMATCH(%s); compute:Blade EXPECTED: SKIPPING!

Disassociate:PnuOSUnconfig: CLASS MISMATCH(%s); compute:Blade EXPECTED: SKIPPING!

Disassociate:PreSanitize: CLASS MISMATCH(%s); compute:Blade EXPECTED: SKIPPING!

Disassociate:Sanitize: CLASS MISMATCH(%s); compute:Blade EXPECTED: SKIPPING!

Disassociate:SwConfigPnuOSLocal: CLASS MISMATCH(%s); fabric:Locale EXPECTED: SKIPPING!

Disassociate:SwConfigPnuOSPeer: CLASS MISMATCH(%s); fabric:Locale EXPECTED: SKIPPING!

Disassociate:SwUnconfigHostOSLocal: CLASS MISMATCH(%s); fabric:Locale EXPECTED: SKIPPING!

Disassociate:SwUnconfigHostOSPeer: CLASS MISMATCH(%s); fabric:Locale EXPECTED: SKIPPING!

Disassociate:SwUnconfigPnuOSLocal: CLASS MISMATCH(%s); fabric:Locale EXPECTED: SKIPPING!

Disassociate:SwUnconfigPnuOSPeer: CLASS MISMATCH(%s); fabric:Locale EXPECTED: SKIPPING!

Disassociate:UnconfigUuid: CLASS MISMATCH(%s); compute:Blade EXPECTED: SKIPPING!

Disassociate:hagPnuOSConnect: CLASS MISMATCH(%s); compute:Blade EXPECTED: SKIPPING!

Disassociate:hagPnuOSDisconnect: CLASS MISMATCH(%s); compute:Blade EXPECTED: SKIPPING!

Discover:BiosPostCompletion: CLASS MISMATCH(%s); compute:Blade EXPECTED: SKIPPING!

Discover:BladeBootPnuos: CLASS MISMATCH(%s); compute:Blade EXPECTED: SKIPPING!

8-23Cisco UCS Troubleshooting Guide

OL-20137-04

Page 84: Cisco UCS Troubleshooting Guide

Send document comments to ucs -doc feedback@c i sco .com

Chapter 8 Cisco UCS Error MessagesFSM Messages

Discover:BladePowerOn: CLASS MISMATCH(%s); compute:Blade EXPECTED: SKIPPING!

Discover:BladeReadSmbios: CLASS MISMATCH(%s); compute:Blade EXPECTED: SKIPPING!

Discover:cimcInventory: CLASS MISMATCH(%s); compute:Blade EXPECTED: SKIPPING!

Discover:cimcPresence: CLASS MISMATCH(%s); compute:Blade EXPECTED: SKIPPING!

Discover:cimcShutdownDiscovered: CLASS MISMATCH(%s); compute:Blade EXPECTED: SKIPPING!

Discover:ConfigFeLocal: CLASS MISMATCH(%s); fabric:Locale EXPECTED: SKIPPING!

Discover:ConfigFePeer: CLASS MISMATCH(%s); fabric:Locale EXPECTED: SKIPPING!

Discover:ConfigUserAccess: CLASS MISMATCH(%s); compute:Blade EXPECTED: SKIPPING!

Discover:NicConfigPnuOS: CLASS MISMATCH(%s); compute:Blade EXPECTED: SKIPPING!

Discover:NicInventory: CLASS MISMATCH(%s); compute:Blade EXPECTED: SKIPPING!

Discover:NicPresence: CLASS MISMATCH(%s); compute:Blade EXPECTED: SKIPPING!

Discover:NicUnconfigPnuOS: CLASS MISMATCH(%s); compute:Blade EXPECTED: SKIPPING!

Discover:PnuOSIdent: CLASS MISMATCH(%s); compute:Blade EXPECTED: SKIPPING!

Discover:PnuOSPolicy: CLASS MISMATCH(%s); ls:AgentPolicy EXPECTED: SKIPPING!

Discover:PnuOSScrub: CLASS MISMATCH(%s); compute:Blade EXPECTED: SKIPPING!

Discover:PreSanitize: CLASS MISMATCH(%s); compute:Blade EXPECTED: SKIPPING!

Discover:Sanitize: CLASS MISMATCH(%s); compute:Blade EXPECTED: SKIPPING!

Discover:SetupVmediaLocal: CLASS MISMATCH(%s); compute:Blade EXPECTED: SKIPPING!

Discover:SetupVmediaPeer: CLASS MISMATCH(%s); compute:Blade EXPECTED: SKIPPING!

Discover:SwConfigPnuOSLocal: CLASS MISMATCH(%s); fabric:Locale EXPECTED: SKIPPING!

Discover:SwConfigPnuOSPeer: CLASS MISMATCH(%s); fabric:Locale EXPECTED: SKIPPING!

Discover:SwUnconfigPnuOSLocal: CLASS MISMATCH(%s); fabric:Locale EXPECTED: SKIPPING!

Discover:SwUnconfigPnuOSPeer: CLASS MISMATCH(%s); fabric:Locale EXPECTED: SKIPPING!

Discover:hagConnect: CLASS MISMATCH(%s); compute:Blade EXPECTED: SKIPPING!

Discover:hagDisconnect: CLASS MISMATCH(%s); compute:Blade EXPECTED: SKIPPING!

Hardreset:PreSanitize: CLASS MISMATCH(%s); compute:Blade EXPECTED: SKIPPING!

Hardreset:Sanitize: CLASS MISMATCH(%s); compute:Blade EXPECTED: SKIPPING!

Powercycle:PreSanitize: CLASS MISMATCH(%s); compute:Blade EXPECTED: SKIPPING!

8-24Cisco UCS Troubleshooting Guide

OL-20137-04

Page 85: Cisco UCS Troubleshooting Guide

Send document comments to ucs -doc feedback@c i sco .com

Chapter 8 Cisco UCS Error MessagesFSM Messages

Powercycle:Sanitize: CLASS MISMATCH(%s); compute:Blade EXPECTED: SKIPPING!

Softreset:PreSanitize: CLASS MISMATCH(%s); compute:Blade EXPECTED: SKIPPING!

Softreset:Sanitize: CLASS MISMATCH(%s); compute:Blade EXPECTED: SKIPPING!

SwConnUpd:A: CLASS MISMATCH(%s); compute:Blade EXPECTED: SKIPPING!

SwConnUpd:B: CLASS MISMATCH(%s); compute:Blade EXPECTED: SKIPPING!

UpdateAdaptor:BladePowerOff: CLASS MISMATCH(%s); compute:Blade EXPECTED: SKIPPING!

UpdateAdaptor:BladePowerOn: CLASS MISMATCH(%s); compute:Blade EXPECTED: SKIPPING!

UpdateAdaptor:PollUpdateStatus: CLASS MISMATCH(%s); compute:Blade EXPECTED: SKIPPING!

UpdateAdaptor:UpdateRequest: CLASS MISMATCH(%s); compute:Blade EXPECTED: SKIPPING!

RemoveChassis:UnIdentifyLocal: CLASS MISMATCH(%s); equipment:IOCard EXPECTED: SKIPPING!

RemoveChassis:UnIdentifyPeer: CLASS MISMATCH(%s); equipment:IOCard EXPECTED: SKIPPING!

FeConn:ConfigureEndPoint: CLASS MISMATCH(%s); equipment:Chassis EXPECTED: SKIPPING!

FeConn:ConfigureSwMgmtEndPoint: CLASS MISMATCH(%s); fabric:Locale EXPECTED: SKIPPING!

FeConn:ConfigureVifNs: CLASS MISMATCH(%s); equipment:IOCard EXPECTED: SKIPPING!

FeConn:DiscoverChassis: CLASS MISMATCH(%s); equipment:Chassis EXPECTED: SKIPPING!

FeConn:EnableChassis: CLASS MISMATCH(%s); equipment:Chassis EXPECTED: SKIPPING!

FePresence:Identify: CLASS MISMATCH(%s); equipment:Chassis EXPECTED: SKIPPING!

SwitchMode:SwConfigLocal: CLASS MISMATCH(%s); fabric:LanCloud EXPECTED: SKIPPING!

SwitchMode:SwConfigPeer: CLASS MISMATCH(%s); fabric:LanCloud EXPECTED: SKIPPING!

ActivateCIMC:Activate: CLASS MISMATCH(%s); compute:Blade EXPECTED: SKIPPING!

ActivateCIMC:Reset: CLASS MISMATCH(%s); compute:Blade EXPECTED: SKIPPING!

ActivateIOM:Activate: CLASS MISMATCH(%s); equipment:IOCard EXPECTED: SKIPPING!

ActivateIOM:Reset: CLASS MISMATCH(%s); equipment:IOCard EXPECTED: SKIPPING!

ExtMgmtIfConfig:Primary: CLASS MISMATCH(%s); compute:Blade EXPECTED: SKIPPING!

ExtMgmtIfConfig:Secondary: CLASS MISMATCH(%s); compute:Blade EXPECTED: SKIPPING!

SysConfig:Primary: CLASS MISMATCH(%s); top:System EXPECTED: SKIPPING!

SysConfig:Secondary: CLASS MISMATCH(%s); top:System EXPECTED: SKIPPING!

8-25Cisco UCS Troubleshooting Guide

OL-20137-04

Page 86: Cisco UCS Troubleshooting Guide

Send document comments to ucs -doc feedback@c i sco .com

Chapter 8 Cisco UCS Error MessagesFSM Messages

UpdateCIMC:PollUpdateStatus: CLASS MISMATCH(%s); compute:Blade EXPECTED: SKIPPING!

UpdateCIMC:UpdateRequest: CLASS MISMATCH(%s); compute:Blade EXPECTED: SKIPPING!

UpdateIOM:PollUpdateStatus: CLASS MISMATCH(%s); equipment:IOCard EXPECTED: SKIPPING!

UpdateIOM:UpdateRequest: CLASS MISMATCH(%s); equipment:IOCard EXPECTED: SKIPPING!

UpdateSwitch:updateLocal: CLASS MISMATCH(%s); firmware:BootDefinition EXPECTED: SKIPPING!

UpdateSwitch:updateRemote: CLASS MISMATCH(%s); firmware:BootDefinition EXPECTED: SKIPPING!

SwMgmtInbandIfConfig:Switch: CLASS MISMATCH(%s); mgmt:If EXPECTED: SKIPPING!

SwMgmtOobIfConfig:Switch: CLASS MISMATCH(%s); mgmt:If EXPECTED: SKIPPING!

Deploy:UpdateConnectivity: CLASS MISMATCH(%s); sw:AccessDomain EXPECTED: SKIPPING!

Deploy:UpdateConnectivity: CLASS MISMATCH(%s); sw:EthLanBorder EXPECTED: SKIPPING!

Deploy:UpdateConnectivity: CLASS MISMATCH(%s); sw:FcSanBorder EXPECTED: SKIPPING!

Deploy:UpdateConnectivity: CLASS MISMATCH(%s); sw:UtilityDomain EXPECTED: SKIPPING!

Export:Execute: CLASS MISMATCH(%s); sysdebug:Core EXPECTED: SKIPPING!

Global:Local: CLASS MISMATCH(%s); sysfile:Repository EXPECTED: SKIPPING!

Global:Peer: CLASS MISMATCH(%s); sysfile:Repository EXPECTED: SKIPPING!

Single:Execute: CLASS MISMATCH(%s); sysfile:Instance EXPECTED: SKIPPING!

8-26Cisco UCS Troubleshooting Guide

OL-20137-04

Page 87: Cisco UCS Troubleshooting Guide

Send document comments to ucs -doc feedback@c i sco .com

OL-20137-04

I N D E X

A

Admin tab 1-1

associating service profiles 6-2, 7-6, 7-8, 7-14

Audience ii-v

audit log

about 1-8

Cisco UCS Manager CLI 1-9

Cisco UCS Manager GUI 1-9

C

Cisco UCS Manager CLI

audit log 1-9

core files 1-8

events 1-7

faults 1-6

FSM 2-3

troubleshooting 1-2

Cisco UCS Manager GUI

audit log 1-9

core files 1-8

events 1-7

faults 1-5

FSM 2-2

troubleshooting 1-1

cleared 1-3

CLI

audit log 1-9

core files 1-8

events 1-7

faults 1-6

FSM 2-3

troubleshooting 1-2

condition 1-3

configuration fault type 1-4

connectivity fault type 1-4

Core File Exporter 1-8

core files

about 1-7

Cisco UCS Manager CLI 1-8

Cisco UCS Manager GUI 1-8

exporting 1-8

critical 1-3

D

Documentation 1-6

documentation, faults 1-6

E

environment fault type 1-4

equipment fault type 1-4

error messages 8-1

event log 1-6

events

about 1-6

Cisco UCS Manager CLI 1-7

Cisco UCS Manager GUI 1-7

properties 1-7, 1-9

purging 1-6

exporting

core files 1-8

show tech-support 3-2

IN-1Cisco UCS Troubleshooting Guide

Page 88: Cisco UCS Troubleshooting Guide

Send document comments to ucs -doc feedback@c i sco .com

Index

F

fault collection policy 1-2, 1-6

faults

about 1-2

active state 1-5

Cisco UCS Manager CLI 1-6

Cisco UCS Manager GUI 1-5

documentation 1-6

fault collection policy 1-6

flapping 1-5

lifecycle 1-5

properties 1-4

retention 1-5

service profiles 5-1

severities 1-3

states 1-3

types 1-4

Faults tab 1-5

Fault Summary area 1-5

fault types

configuration 1-4

connectivity 1-4

environment 1-4

equipment 1-4

fsm 1-4

management 1-4

network 1-4

operational 1-4

server 1-4

finite state machine

See FSM

flapping 1-3

flapping interval 1-5

FSM

about 2-1

Cisco UCS Manager CLI 2-3

Cisco UCS Manager GUI 2-2

dynamic view 2-2

IN-2Cisco UCS Troubleshooting Guide

purpose 2-2

static view 2-3

fsm

fault type 1-4

G

GUI

audit log 1-9

core files 1-8

events 1-7

faults 1-5

FSM 2-2

troubleshooting 1-1

guidelines

security ii-vi

service profiles 6-2, 7-6, 7-8, 7-14

troubleshooting 3-1

I

info 1-3

interval

flapping 1-5

retention 1-5

soaking 1-5

L

lifecycle, faults 1-5

logs

audit 1-8

event 1-6

syslog 1-10

system event 1-10

OL-20137-04

Page 89: Cisco UCS Troubleshooting Guide

Send document comments to ucs -doc feedback@c i sco .com

Index

M

major 1-3

management fault type 1-4

minor 1-3

N

network fault type 1-4

O

obtaining

documentation ii-vi

support ii-vi

operational fault type 1-4

P

policies

SEL policy 1-10

policies, fault 1-6

properties

events 1-7, 1-9

faults 1-4

purging events 1-6

R

related documentation ii-v

retention interval 1-5

S

scope monitoring 1-8

scope security 1-9

scope sysdebug 1-8

security guidelines ii-vi

OL-20137-04

SEL

about 1-10

policy 1-10

server fault type 1-4

service profiles

associating 6-2, 7-6, 7-8, 7-14

faults 5-1

severities

cleared 1-3

condition 1-3

critical 1-3

info 1-3

major 1-3

minor 1-3

syslog 1-11

warning 1-3

show audit-logs 1-9

show cores 1-8

show event 1-7

show event detail 1-7

show fault 1-6

show fault detail 1-6

show tech-support 3-2

soaking 1-3

soaking interval 1-5

states

flapping 1-3

soaking 1-3

support ii-vi

syslog

about 1-10

entry format 1-11

severities 1-11

system event log 1-10

W

warning 1-3

IN-3Cisco UCS Troubleshooting Guide

Page 90: Cisco UCS Troubleshooting Guide

Send document comments to ucs -doc feedback@c i sco .com

Index

IN-4Cisco UCS Troubleshooting Guide

OL-20137-04