NimbleOS 5.3.1.100 Releases Notes · CRITICAL Arrays must be running NimbleOS 5.0.6.0 or later to update to NimbleOS 5.3.1.100. An extended data services outage may occur with MS

NimbleOS 5.3.1.100 Releases Notes

Published January, 2021Version 5.3.1.100.

Legal Notices

Copyright © 2021 by Hewlett Packard Enterprise Development LP

Notices

The information contained herein is subject to change without notice. The only warranties for Hewlett Packard Enterprise productsand services are set forth in the express warranty statements accompanying such products and services. Nothing herein should beconstrued as constituting an additional warranty. Hewlett Packard Enterprise shall not be liable for technical or editorial errors oromissions contained herein.

Confidential computer software. Valid license from Hewlett Packard Enterprise required for possession, use, or copying. Consistentwith FAR 12.211 and 12.212, Commercial Computer Software, Computer Software Documentation, and Technical Data for Com-mercial Items are licensed to the U.S. Government under vendor's standard commercial license.

Links to third-party websites take you outside the Hewlett Packard Enterprise website. Hewlett Packard Enterprise has no controlover and is not responsible for information outside the Hewlett Packard Enterprise website.

Acknowledgments

Intel®, Itanium®, Pentium®, Intel Inside®, and the Intel Inside logo are trademarks of Intel Corporation in the United States and othercountries.

Microsoft® and Windows® are either registered trademarks or trademarks of Microsoft Corporation in the United States and/orother countries.

Adobe® and Acrobat® are trademarks of Adobe Systems Incorporated.

Java® and Oracle® are registered trademarks of Oracle and/or its affiliates.

UNIX® is a registered trademark of The Open Group.

All third-party marks are property of their respective owners.

Publication Date

Friday January 8, 2021 10:31:39

Document ID

vdc1608920921298

Support

All documentation and knowledge base articles are available on HPE InfoSight at https://infosight.hpe.com. To register forHPE InfoSight, click the Create Account link on the main page.

Email: [email protected]

For all other general support contact information, go to https://www.hpe.com/us/en/services/nimble-storage.html.

2

https://infosight.hpe.commailto:[email protected]://www.hpe.com/us/en/services/nimble-storage.html

Contents

NimbleOS 5.3.1.100.................................................................................................................4Important Update Note............................................................................................................................................................................................................4

Special Notes..................................................................................................................................................................................................................................4

New Features in 5.3.1.100.....................................................................................................................................................................................................8

Recent Release Features.........................................................................................................................................................................................................8

Documentation..........................................................................................................................................................................................................................10

Verified Update Paths...........................................................................................................................................................................................................11

Known Critical Issues..............................................................................................................................................................................................................15

Resolved Critical Issues.........................................................................................................................................................................................................22

Resolved Issues.........................................................................................................................................................................................................................22

Known Issues..............................................................................................................................................................................................................................22

3

NimbleOS 5.3.1.100

5.3.1.100Version:

Friday January 8, 2021 10:31:39Revision:

The release notes describe the major changes, fixes, and known issues for this release of the NimbleOS. They do not includeall individual fixes and internal changes.

For technical support, contact HPE Nimble Storage Support at:

mailto:[email protected] (877-364-6253), option 2.

Important Update NoteUpdating NimbleOS can involve an update to component firmware on the standby controller. This can cause an email alertand automated case indicating "Standby Controller Not Available" when the firmware update process takes longer than fiveminutes. This is expected behavior and does not affect data services. At the end of the software update, you can check statusof both controllers in the Web UI under Manage > Hardware. One controller will be ACTIVE and the other STANDBY undernormal operating conditions following a successful software update.

All third-party software notices can be found on HPE InfoSight (https://infosight.hpe.com) on the Resources > Documentationpage:

https://infosight.hpe.com/resources/nimble/docs

The Documentation page also includes the General Terms and Conditions document. You can display this document byperforming the following steps:

1 In the navigation pane on the HPE InfoSight Documentation page, scroll through the Document Type list and selectSupport Policy.

2 In the page that appears, select General Terms and Conditions. This document opens in a browser tab.

Special Notes

DescriptionNote

HPE Nimble Storage continues to qualify configurations between releases. TheValidated Configuration Matrix provides information about validated configurationsand is updated frequently. It is a good practice to check your system configurationagainst this online tool. The Validated Configuration Matrix tool is available onHPE InfoSight:

https://infosight.hpe.com/resources/nimble/validated-configuration-matrix

CRITICAL

Arrays must be running NimbleOS 5.0.6.0 or later to update to NimbleOS 5.3.1.100.CRITICAL

An extended data services outage may occur with MS iSCSI initiator and Intel NICsusing the built-in Windows driver e1q60x64.sys (version 11.0.5.21/11.0.5.22).

If you encounter this problem, please update your system to use the latest Windowsdriver.

CRITICAL

Important Update Note 4

mailto:[email protected]://infosight.hpe.comhttps://infosight.hpe.com/resources/nimble/docshttps://infosight.hpe.com/resources/nimble/validated-configuration-matrix

DescriptionNote

A service outage may occur on Windows 2012 R2 hosts using Emulex or BroadcomFibre Channel HBAs with firmware/driver prior to 11.2. Update the Emulex orBroadcom firmware/driver to 11.2 or later

CRITICAL

Due to a known Red Hat Enterprise Linux bug 1002727, while running virtualizedin VMware ESX, manually rebooting the active controller in presence of heavy IOsusing the reboot --controller command on a Fibre Channel array may trigger anincorrect retry initiated by RHEL guests running the following kernel versions:

• 6.4 and earlier• 6.5 without the patch• 7.0 without the patch

This incorrect retry logic may lead to unexpected application behavior. In theseenvironments, we recommend the failover command instead.

CRITICAL

Due to a known Red Hat Enterprise Linux bug 3550561, unexpected applicationbehavior may occur on RHEL 7.5 hosts with kernel-3.10.0-862.3.2.el7 or derivativesusing Emulex FC FCoE HBAs (lpfc driver) and raw devices. To avoid this issue:

• If running RHEL 7.6, update to kernel-3.10.0-957.el7 or later.• If running RHEL 7.5z, update to kernel-3.10.0-862.25.3.el7 or later.

CRITICAL

As outlined in the current Validated Configuration Matrix, HPE Nimble Storagefully supports Windows guest operating systems on Microsoft Hyper-V, includingVirtual Fibre Channel (VFC) connectivity and multipathing with HPE Nimble StorageDSM and VSS support. However, Linux guest operating systems running in Hyper-V VFC configurations are not qualified.

Running Red Hat Linux guest operating systems with the “Linux Integration Ser-vices” kit installed, or with hv_storvsc drivers in such configurations can lead toRed Hat bug 1364282, which can cause an unexpected service outage.

CRITICAL

Starting with NimbleOS 5.3.1.0, additional Unit Attention (UA) codes for iSCSI havebeen added to notify hosts to take appropriate actions for array side changes.INQUIRY_DATA_HAS_CHANGED has been added to allow the host(s) to ac-count for management IP address changes. CAPACITY_DATA_HAS_CHANGEDhas been added to allow the host(s) to account for volume size changes.

Important

Special Notes 5

DescriptionNote

After performing an update to NimbleOS software version 5.3.x, the updatedNimble vCenter plugin may not be deployed to vCenter. As a result, the vSphereclient will not list the Nimble array group correctly. Instead, a 404 NOT FOUNDerror is displayed, indicating that it could not download the proper resources toload the page. In addition, the Undeploy plugin task will continue to show as runningin the Task Console for several hours.

To resolve this issue:

Unregister the vCenter plugin from the HPE Nimble Storage array usingeither the CLI or UI.

1

Log in to the vCenter as \root and run the following commands:

(service-control --stop vsphere-ui;rm -vfr /etc/vmware/vsphere-ui/vc-pack-ages/vsphere-client-serenity/com.nimblestor-age*;service-control --start vsphere-ui) &

(service-control --stop vsphere-client;rm -vfr /etc/vmware/vsphere-client/vc-pack-ages/vsphere-client-serenity/com.nimblestor-age*;service-control --start vsphere-client) &

This will stop the vSphere UI service, cleanup any existing Nimble pluginpackages, then restart the vSphere UI service.

2

Re-register the vCenter plugin with the HPE Nimble Storage array usingeither the CLI or UI.

3

Log out and log in to the vSphere client to verify that the plugin is de-ployed.

4

This issue is tracked under software defect PRT-495.

Important

Starting with NimbleOS 5.1.1.0, the size of the software package now exceeds 2GB, which may lead to lengthier software download times. Previously, the sizes ofthe NimbleOS 5.0.x download packages were approximately 1.6 GB, and NimbleOS4.x packages were approximately 900 MB.

Important

After completing the NimbleOS update for array groups configured for SynchronousReplication, download the corresponding version of the Synchronous ReplicationWitness software, and update the witness host.

Important

Microsoft Offload Data Transfer (ODX) is not supported if the destination volumehas synchronous replication enabled.

Important

As of vSphere 7.0, VMware has discontinued the flex client. Consequently, the HPENimble Storage vCenter Plugin no longer supports the flex plugin for vCenter 7.0.

Important

Special Notes 6

https://infosight.hpe.com/org/urn%3Ainfosight%3A3d132a16-b52d-4040-a655-b3a9c3647505/resources/nimble/softwarehttps://infosight.hpe.com/org/urn%3Ainfosight%3A3d132a16-b52d-4040-a655-b3a9c3647505/resources/nimble/software

DescriptionNote

You can enable deduplication for CS1000, CS3000, CS5000, CS7000, CS700, andCS500 arrays on a volume only if the corresponding storage pool has a Flash toDisk Ratio (FDR) greater than 4%. To calculate the FDR, obtain the "Total arraycapacity (MiB)" and "Total array cache capacity (MiB)" values by using the HPENimble Storage CLI command poolpool_name. This command returns the Poolcapacity (MiB), which is the "Total array capacity (MiB)", and the Pool cache ca-pacity (MiB), which is the "Total array cache capacity (MIB)".

Then perform the following calculation:

FDR = "Total array cache capacity (MiB)"/"Total array capacity (MiB)" * 100

If the array has sufficient capability for deduplication, the pool --info commandwill also show a value for dedupe capacity (MiB).

Note On the HF20H, HF20, HF40, and HF60 platforms, pool --info displays "N/A"as the value for dedupe capacity (MiB). This because you can enable deduplicationfor the entire array.

Important

For connections to the NimbleOS GUI, you must have port 5392 open for the GroupManagement IP address and both diagnostic IP addresses.

Important

Numerous host integration toolkits are supported in NimbleOS 5.3.1.100. It isstrongly recommended that they be installed on all Windows, Linux, and VMwarehosts. For more information about supported toolkits, refer to the Validated Con-figuration Matrix, which is available on HPE Nimble Storage InfoSight:


Important

HPE Nimble Storage recommends that you update to HPE Nimble Storage WindowsToolkit (NWT) 7.0.1 or later if you are using Microsoft VSS Synchronization andNimbleOS 5.1.4.200 or later.

Using application consistent snapshots with earlier versions of NWT and NimbleOS5.1.4.100 may result in the following error messages:

• In the host's VSS requestor log (C:\ProgramData\Nimble Storage\Logs\VssRe-questor.log):

PID:1996 TID:5752 ERR reqcommon. cpp:683 Request-Status=QueryStatus(), Function=pAsync->QuerySta-tus(), Error=VSS_E_PROVIDER_VETO, rc=SystemError,ca=ContactSupport

• In the Windows event viewer:

event id 4100: EndPrepareSnapshots method: failedto find LUN s/n on connected ar-rays. Make sure that the Nimble array version iscompatible with this version of Nimble WindowsToolkit.

event id 4170: Nimble VSS provider is not compati-ble with the current version of the Nimble arraysoftware(). Install appropriate version of theNimble VSS provider.

NWT 7.0.1 resolves this issue.

Important

Special Notes 7


DescriptionNote

HPE Nimble Storage Connection Manager (NCM) for VMware 7.0 is signed byVMware for ESXi 7.x. It can be installed through the VMware Update Manager oresxcli command without the --no-sig-check flag.

See the NCM for VMware Release Notes 7.0 or later and the latest VMware Integra-tion Guide for further details.

To locate the latest version of the guide, log in to HPE InfoSight. Choose Re-sources > Nimble Storage Documentation. In the left pane, click IntegrationGuide, then click Connection Manager (NCM) for VMware. From the list displayed,choose the version of the guide that you want.

Important

Various timeout values affect HPE Nimble Storage targets from Windows/Linuxhosts. Before you update the NimbleOS, install the HPE Nimble Storage WindowsToolkit (NWT) or HPE Nimble Storage Linux Toolkit (NLT) on the host or tune thetimeout values. Timeout details for various operating systems can be found onHPE InfoSight under Resources > Documentation. From the HPE Nimble StorageDocumentation page, locate the article you want.

The following Knowledge Base articles and Integration Guides explain how toconfigure and verify host timeout settings for the major supported operating sys-tems (OS):

• For Windows, refer to KB-000052: Windows Host Disk Timeout Values.

In the context of Microsoft Windows, the following article should also be con-sidered:

KB-000246 MPIO Timeout Parameters for MSDSM and NimbleDSM in Windows2012 R2

• For VMware, refer to the Common Tasks and Best Practices > Host TimeoutValues section of the VMware Integration Guide.

• For Linux, refer to KB-000304: Linux Host Disk Timeout Values.

Important

vVol VMs cannot be claimed after they are deleted from the downstream array.

A vVol VM can be protected and may subsequently be replicated to a downstreamarray (as configured in the storage policy). In the case where this vVol VM wasdeleted, a supported “claim” workflow allows us to claim this vVol VM on thedownstream array.

Due to validation failures in the vCenter, this workflow is not supported if it isperformed in an environment where the vCenter version is 6.5 or later.

See the following VMware DCPN Ticket Reference:

https://dcpn.force.com/TechnicalRequestCaseRedesignPart-ner?Id=5000H00001JRKhf

Important

New Features in 5.3.1.100There are no new features introduced with NimbleOS 5.3.1.100.

Recent Release FeaturesRecently released features in NimbleOS 5.3.x include:

New Features in 5.3.1.100 8

https://infosight.hpe.com/resources/nimble/docshttps://infosight.hpe.com/resources/nimble/docshttps://infosight.hpe.com/InfoSight/media/cms/active/sup_KB-000052_Windows_Host_Disk_Timeout_Values.whz/index.htmlhttps://infosight.nimblestorage.com/InfoSight/media/kb/active/htr1455131726154.whzhttps://infosight.nimblestorage.com/InfoSight/media/kb/active/htr1455131726154.whzhttps://infosight.hpe.com/InfoSight/media/kb/active/htr1455131700640.whz/index.htmlhttps://dcpn.force.com/TechnicalRequestCaseRedesignPartner?Id=5000H00001JRKhfhttps://dcpn.force.com/TechnicalRequestCaseRedesignPartner?Id=5000H00001JRKhf

25Gb NICs

Adds support for dual-ported Ethernet cards with 25Gb speeds.

Syhchronous Replication Performance Improvements

Provides performance improvements during re-synchronization of synchronously replicated volumes.

Single-click Upgrades of dHCI Service Pack for ProLiant Gen 10

Provides a single-click upgrade process for ProLiant Gen 10 Service Packs in dHCI environments.

dHCI: Allow User to Assign iLO to Separate Network/VLAN (Split Management)

Allows ProLiant iLOs to be configured on separate management networks during dHCI setup.

Report Windows Host Information to the Array

Allows HPE Nimble Host Utilities to post support information to the array .

External KMIP Integration

NimbleOS now supports using an external key manager to manage the master encryption key. External key management isdone via the industry standard KMIP protocol, and any KMIP supporting key manager may be used.

Recent Release Features 9

DocumentationThese Release Notes and other user documentation are available on HPE InfoSight:


You can manually reach the documentation page by logging onto HPE InfoSight and selecting Resources > Nimble Storage >Documentation.

Document Search Interface

There are several methods you can use to locate the documents you need.

The Nimble Storage Documentation page provides a search interface that allows you to search for information across alldocumentation, including support and knowledge base articles, best practices, solutions and integration guides, productdocumentation, and configuration matrices.

To go directly to a document, use the navigation pane on the left side of the Nimble Storage Documentation page. Thenavigation pane organizes documents into categories, including:

• Document Type• Nimble Software and Solutions• Software Version• Integration• Platform

You can use the page scroll bar to move up and down the navigation pane.

Third-Party Software Notices

All third-part software notices can be found in the Documentation Portal on HPE InfoSight.

Here are the steps to manually access the third-party software notices.

1 Log in to HPE InfoSight (https://infosight.hpe.com) .

2 From the menu, select Resources Nimble Documentation .

3 In the left navigation pane of the Documentation Portal, scroll through the Document Type section and selectSupport Policy.

4 From the list of documents, select General Terms and Conditions. The document opens in a new browser tab.

Core User Documentation

The following is the core user documentation for NimbleOS:

• GUI Administration Guide• CLI Administration Guide• SNMP Reference• Command Reference• REST API Reference

If you are using an HPE Nimble Storage dHCI-enabled array, you should also check the dHCI Deployment Guides and GettingStarted Guide.

Workflow Documents

There are several workflow guides that contain procedures you can perform using either the CLI or the GUI. Each workflowguide covers a specific, frequently performed task related to HPE Nimble Storage products. Each task described by a workflowdocument is explained in detail in the GUI Administration Guide and the CLI Administration Guide.

Documentation 10


Hardware

Documentation for all hardware components is available on HPE InfoSight. Click the Hardware Guide link in the DocumentType category. Hardware documentation includes array and expansion shelf installation quick start guides, installation,upgrade, and replacement guides, and comprehensive hardware guides.

Host Integration Guides

Host Integration Guides are available from HPE InfoSight. To locate these documents on the HPE InfoSight Documentationpage, scroll down the navigation pane to the section called Integration Guide.

Note A single Host Integration Guide supports multiple version of NimbleOS and the companion Integration Toolkit softwarepackages. The version number listed on the guide might be different from the version numbers of the NimbleOS and Toolkitsoftware packages that it supports.

Verified Update Paths

Table 1: From Versions 5.x

From Versions 5.x

To VersionFrom Version

5.3.1.1005.3.1.0

5.3.1.1005.3.0.0

5.3.1.1005.2.1.400

5.3.1.1005.2.1.300

5.3.1.1005.2.1.200

5.3.1.1005.2.1.100

5.3.1.1005.2.1.0

5.3.1.1005.1.4.200

5.3.1.1005.1.4.100

5.3.1.1005.1.4.0

5.3.1.1005.1.3.100

5.3.1.1005.1.3.0

5.3.1.1005.1.2.100

5.3.1.1005.1.2.0

5.3.1.1005.1.1.0

5.3.1.1005.0.10.0

5.3.1.1005.0.9.100

5.3.1.1005.0.9.0

5.3.1.1005.0.8.100

5.3.1.1005.0.8.0

5.3.1.1005.0.7.300

Verified Update Paths 11

From Versions 5.x


5.3.1.1005.0.7.200

5.3.1.1005.0.7.100

5.3.1.1005.0.7.0

5.3.1.1005.0.6.0

5.0.10.05.0.5.200

5.0.10.05.0.5.0

5.0.10.05.0.4.0

5.0.10.05.0.3.100

5.0.10.05.0.3.0

5.0.10.05.0.2.0

5.0.10.05.0.1.100

5.0.10.05.0.1.0


From Versions 4.x


5.0.10.04.5.6.0

5.0.10.04.5.5.0

5.0.10.04.5.4.0

5.0.10.04.5.3.0

5.0.10.04.5.2.0

5.0.10.04.5.1.0

5.0.10.04.5.0.0

5.0.10.04.4.1.0

5.0.10.04.4.0.0

5.0.10.04.3.1.0

5.0.10.04.3.0.0

5.0.10.04.2.1.0

5.0.10.04.2.0.0

5.0.10.04.1.0.0



From 3.x Versions


5.0.10.03.9.3.0

5.0.10.03.9.2.0

5.0.10.03.9.1.0

5.0.10.03.9.0.0

5.0.10.03.8.1.0

5.0.10.03.8.0.0

5.0.10.03.7.0.0

5.0.10.03.6.2.0

5.0.10.03.6.1.0

5.0.10.03.6.0.0

5.0.10.03.5.4.0

5.0.10.03.5.3.0

5.0.10.03.5.2.0

5.0.10.03.5.0.0

5.0.10.03.4.1.0

5.0.10.03.4.0.0

5.0.10.03.3.0.0

5.0.10.03.2.1.0

5.0.10.03.1.0.0


From 2.0.x VersionsFrom 2.1.x VersionsFrom 2.2.x, 2.3.x Versions

To VersionFrom VersionTo VersionFrom VersionTo VersionFrom Version

2.1.9.12.0.8.02.3.18.02.1.9.14.5.6.02.3.18.0

2.1.9.12.0.7.02.3.18.02.1.9.04.5.6.02.3.16.0

2.1.9.12.0.6.*2.3.18.02.1.8.04.5.6.02.3.15.0

2.1.9.12.0.5.02.2.9.02.1.7.04.5.6.02.3.14.0

2.1.9.12.0.4.02.2.9.02.1.6.04.5.6.02.3.12.*

2.2.9.02.1.5.04.5.6.02.3.9.*

2.2.9.02.1.4.04.5.6.02.3.8.0

2.2.9.02.1.3.04.5.6.02.3.7.0

2.2.9.02.1.2.04.5.6.02.3.6.0


From 2.0.x VersionsFrom 2.1.x VersionsFrom 2.2.x, 2.3.x Versions


2.1.9.12.1.1.04.5.6.02.3.4.0

2.1.9.12.1.0.04.5.6.02.3.3.0

4.5.6.02.3.2.1

4.5.6.02.3.2.0

4.5.6.02.3.1.0

3.9.3.02.2.11.0

3.9.3.02.2.10.0

3.9.3.02.2.9.0

3.9.3.02.2.7.*

3.9.3.02.2.6.0

3.9.3.02.2.5.*

2.2.11.02.2.3.*

2.2.11.02.2.2.0

2.2.11.02.2.1.0

2.2.11.02.2.0.0


From 1.0.x VersionsFrom 1.3, 1.2, 1.1 VersionsFrom 1.4.x Versions


Contact Support1.0.7.*1.4.6.01.3.*.*2.1.9.11.4.12.0

Contact Support1.0.6.*1.4.6.01.2.*.*2.1.9.11.4.11.0

1.2.2.01.1.*.*2.1.9.11.4.10.0

2.1.9.11.4.9.0

2.1.9.11.4.8.0

2.1.9.11.4.7.0

1.4.12.01.4.*.*


Known Critical Issues

Known Critical Issues in NimbleOS version 5.3.1.100

WorkaroundDescriptionTitleComponentID

Contact HPE Nimble StorageSupport.

On Data Service startup followedby a failover, during the recov-ery process, it is possible to hita checksum mismatch. The DataService restarts when the mis-match is encountered. The ser-vice will make three attempts torestart, before disabling the ser-vice.

Data Service dis-abled on array dueto NVRAM Check-sum mismatch

Data ServiceAS-115397

Contact HPE Nimble StorageSupport

In rare instances, the Data Ser-vice may restart unexpectedlywhile performing Volume Man-agement checksum computa-tions.

Data Service mayrestart unexpected-ly


When planning to remove amember array from group,schedule a planned maintenancewindow and place all ESX hostsinto maintenance mode to mini-mize impact to availability. ESXtypically resumes connection tovVol datastores, and reconnectsto VMs, after a period of 15-30minutes automatically without amanual intervention.

Scaled vVol environments with500 vVol VDI VMs or more than5000 Nimble vVol volumes mayexperience IO disruption whenremoving a member array fromgroup. Symptom of problemwould appear as vVol datastoresbeing (inaccessible). Virtual Ma-chine status would also appearas (inaccessible).

Removing memberarray from multi-ar-ray group maycause IO disruptionto scaled vVol envi-ronments


Not applicableVolume moves transfer datafrom one Nimble array to anoth-er. During this move, if the hostsends I/O to the incorrect array,the I/O needs to be forwardedto the correct array owning thedata. This results in higher thanusual I/O latency. To avoid thisissue in VMware environments,the Nimble Connection Managerfor VMware needs to be installedon all hosts accessing the vol-ume.

Volume move mayresult in latency ifNimble ConnectionManager is not in-stalled


Not applicableThe Data service may restartunexpectedly when a checksumverification failure occurs.

Data service mayrestart unexpected-ly



When the Data Service detectsa metadata inconsistency, theservice may restart repeatedlyand hosts could experience un-expected application behavior.

Data Servicerestarts when de-tecting metadatainconsistency


Known Critical Issues 15



Not applicableWhile committing internaltransactions, Data Service mayhit a rare race condition. To re-cover from this Data Servicemight restart

Data service mayrestart due to arace condition


Not applicableDuring snapshot replication ofa dedupe-enabled volume, thedownstream array file systemmay restart due to an out-of-memory condition.

Snapshot replica-tion of deduplica-tion-enabled vol-umes may lead toFile System restart


Not applicableA disruption in network connec-tions can cause Data Service torestart unexpectedly.

Data Service mayunexpectedlyrestart


Review network devices toidentify and reduce networkchecksum errors.

In environments with frequentnetwork checksum errors, inter-ruptions of Replication and DataService restarts may occur.

Replication interrup-tions and Data Ser-vice restarts mayoccur due to net-work errors


Not applicableData Services may unexpectedlyrestart due to out of memorycondition while running snapreplication on dedupe enabledvolumes with high compressionratios.



Not applicableWhen attempting to perform apool merge operation, if thereare a large number of volumesthat must be striped across thepool, and one of the arrays hasa large number of pendingdeletes, then it is possible for theoperation to fail due to the DataService being overloaded.Symptoms of this behavior areif the pool merge operationhangs for several minutes andreturns the following message:The request could not be under-stood by the server.

Pool merge failsdue to too manypending deletes


Not applicableData Service may restart whenoperations of volume migrationare stuck waiting for ownershiptransfer completion.

Data Service mayrestart unexpected-ly during volumemigration





Not applicableThe Data service may restartwhen Bin Migration is going on.This can happen when followingactivities happen together: 1. Binmigration is occurring for a vol-ume. 2. Group Management ser-vice restart (because of any rea-son). 3. Group Management ser-vice unable to re-sync with Dataservice after &nb-sp; restart. This canlead to Data service restart butit will not impact bin migrationas after Data service restart binmigration will resumed automat-ically.

Rare race conditionbetween Data ser-vice and GroupManagement ser-vice cause Dataservice restart


Not applicableIf the Data Service is restartedduring a volume move, it wouldresult in incorrect Bin Migrationstate to be recovered, causinganother Data service restart.This is a rare event since it re-quires snapshots to be taken aswell as a Data Service restartduring a volume move.

Data Service restartduring volumemove may causethe service torestart again.



Counters used for space account-ing may cause validation to fail.The array may see a blip inspace usage because of whichwrites may be denied and inworst case, volume may go of-fline.

Writes may be de-nied due to coun-ters in space ac-counting


Not applicableOn new array installations, whencreating the first vVol datastoreon the host, there is a possibilitythat the datastore is inaccessibleinitially. However, it will becomeaccessible within 5 minutes.

Delay for first vVoldatastore becomingaccessible

Host IntegrationAS-100561


HPE Nimble Storage has identi-fied a rare firmware defect in asubset of drives which can, un-der certain write intensiveworkloads, cause the array tounder-perform.

Performance affect-ing firmware defectin a subset of 6TBdrives.

PlatformAS-94961




If after 20 minutes the controllersensors do not report goodstate, please contact HPE Nim-ble Storage Support for assis-tance.

During boot up due to a knownIntel defect the controller sen-sors may report missing for aperiod of time in the array alerts.After about 15-20 minutes, itreturns to a valid state and thesensors should report validreadings again.

Controller sensorsmissing forAFxx/HFxx arrays

PlatformAS-86764

Not applicableDuring internal file operations,processes may be waiting for alock to be released. If the waittime exceeds 30 seconds, a ser-vice health check may restartthe Data service to recover.

Data service mayrestart during whenfile operation time-out is exceeded

PlatformAS-86099

Not applicableThe Data Service on the arraymay restart when timeout forinternal communication betweenarray controllers has been ex-ceeded. The service restarts torestore the communication.

Data Service mayrestart due tohealth check failure

PlatformAS-104517

Contact HPE Nimble StorageSupport to review disks for re-placement.

In rare instances, the Data Ser-vice may restart if the array hasmultiple bad drives which makeIO handling very slow.

Data Service mayrestart if the arrayhas multiple baddrives

PlatformAS-108793

Not applicableNimble Drive Error Recovery(NDER) is activated for drivesfailing I/O in an attempt to recov-er the drive. In rare instances,the process may surpass iSCSIhost timeout values, causinghost I/O inaccessibility.

NDER process maylead to host recon-nects

PlatformAS-96053

Not applicableThe current thermal policy hasa limitation where individualPCIe components are not moni-tored on card-by-card basis. Asa result, the high level tempera-ture policy that is current imple-mented, is sometimes incapableof regulating the temperaturesof individual cards.

Limited thermalmonitoring policyfor PCIe compo-nents

PlatformAS-107489

Contact HPE Nimble Support forassistance in restoring array toActive/Standby status.

In rare instances the intercon-nect between controllers may bedown following a controller re-seat.

Array remains insolo/stale aftercontroller reseat.

PlatformAS-51053




Verify that the software updatepackage has been downloadedto the system and then attemptthe software update again usingthe software --resume_updatecommand from the CLI.

During software update fromNimbleOS 3.x versions to laterversion, a failure to extract thesoftware update package mayincorrectly return the errormessage /tmp is out of space.even if there is still space in thedirectory.

Incorrect softwareupdate error mes-sage /tmp is out ofspace.

PlatformAS-93456

Not applicableThe Data Service on the arraymay restart if a Task Manage-ment Function request to aborta command is received while thecommand is still being complet-ed. The service restarts to recov-er from the condition.


SANAS-112859


The Event service may restartunexpectedly when attemptingto load a trusted certificate dueto an issue which impacts certifi-cate management. The servicemay not restart after encounter-ing the issue.

Event service mayrestart unexpected-ly

SecurityAS-115780

Not applicableGroup Management service mayrestart due to a race conditionbetween threads when a volumeaccess control list is removed.The service restarts to recoverfrom the condition.

Group Managementservice may restartdue to race condi-tion

System Manage-ment

AS-110030

A Manual Group Leader Failoverwill be required to restore FibreChannel connectivity to thehosts.

An Automatic Failover (AFO) ofthe Group Management Serviceswill not be initiated if all FibreChannel (FC) interfaces on theGroup Leader array fail on bothcontrollers.

No AutomaticFailover in theevent the host los-es all FC connectivi-ty to an array

System Manage-ment

AS-94737

A failover can be initiated in or-der to restart the Group Manage-ment Service. You may alsocontact HPE Nimble StorageSupport to restart the servicemanually.

When performing a controllerupgrade to a high-end model,the object limits will still showthe lower limits if the GroupManagement Service is notrestarted.

Group ManagementService must berestarted to unlockadditional volumelimits after con-troller upgrade

System Manage-ment

AS-65615




Remove the VLAN tagging fromthe management network toprevent network connectivityissues in Peer PersistenceGroup.

When the array group is config-ured for Peer Persistence andthe management subnet is con-figured with VLAN tagging, thecharacter limit for the interfacecreated for the Secondary Man-agement IP will be exceeded.This can cause this SecondaryManagement IP to be unreach-able.

Backup GroupLeader may experi-ence network con-nectivity issues inPeer Persistencegroup

System Manage-ment

AS-110947

Not applicableThe Group Management servicemay restart during an arrayshutdown while processingREST request. No user opera-tions are impacted because thearray is already in the middle ofa shutdown. The shutdown pro-ceeds normally.

Group Managementservice may restartduring array shut-down

System Manage-ment

AS-61614


The Snapshot rate limit - Num-ber of snapshots that can betaken for a given minute - is in-troduced in 4.x for distributingsnapshot load to the system. Ifa group surpasses this snapshotrate limit in previous NimbleOSreleases, the software update to4.x is still allowed. However, agroup merge is prevented if thesource or destination array sur-passes the snapshot rate limit.The following error message canbe seen during the group mergevalidation: INFO: Rate of snap-shots in the merged groupwould be greater than support-ed snapshot rate. This could oc-cur if even if the source array(the array being added to theexisting group) has 0 volumesor volume collections.

Group merge fails ifone of the groupsviolates the snap-shot rate limit

System Manage-ment

AS-69136

Use unique name when creatingsnapshot collection.

If an attempt is made to createa snapshot collection with samethe name of an existing snap-shot collection, an error indicat-ing object already exists will bereported and the Group Manage-ment service may restart. Theservice will recover upon restart.

Group Managementservice may restartdue to duplicatesnapshot collectionname

System Manage-ment

AS-94113




Not applicableA system management processcan restart when the system isunder heavy load. The systemrecovers automatically. The Dataservice is not affected.

Group ManagementService restarts un-der heavy load

System Manage-ment

AS-100254

Not applicableThe Array Management servicemay restart due to a race condi-tion encountered during serviceshutdown. The service will recov-er after the restart.

Array Managementservice restart dur-ing service shut-down

System Manage-ment

AS-98124

Contact HPE Nimble StorageSupport for assistance in deter-mining the cause of the failure.

If a software update precheckfails, in some cases it will returnonly the failure status withoutproviding additional informationabout the cause of the failure.

Software precheckfailures returngeneric error mes-sage

System Manage-ment

AS-87736

Not applicableGroup Management may be-come unavailable temporarilywhen a large number of restoresare performed on a volume.

Group ManagementServices unavail-able temporarilywhen performingvolume restore

System Manage-ment

AS-98194

Not applicableIn rare occurrences, a customer-initiated reboot may cause akernel reboot on the active con-troller. This will cause a longerreboot cycle.

Graceful shutdowntakes longer thanexpected

System Manage-ment

AS-95169

Not applicableController may reboot unexpect-edly if there is high memory uti-lization for the java and/or jettyprocesses on the array.

Controller may un-expectedly restartdue to high memo-ry utilization

System Manage-ment

AS-84499

Refresh the page to restorebutton functionality.

After performing a bulk updateoperation from the Manage >Data Storage > Volumes >volumename > Data Protec-tion tab in the GUI, the pagefooter may not update and pre-vious button may be unavailable.

Page footer in GUImay fail to updateafter bulk updateoperation

System Manage-ment

AS-97968

Use volume collections to checksyncRep volumes, or use otherfilters to meet the needs

There is currently no way to fil-ter volumes using SynchronousReplication within the array GUI.

Unable to Filter vol-umes using Syn-chronous Replica-tion

System Manage-ment

AS-92379


Resolved Critical Issues

Resolved Critical Issues in NimbleOS version 5.3.1.100


Perform a software update toNimbleOS version 5.3.1.100 orlater to address this issue.

In rare circumstances, an issuewith NVRAM could go undetect-ed and result in an instance ofextended data unavailability.

NimbleOS 5.3.1.0may not be able toproperly detectcertain NVRAMfailure scenarios


Resolved IssuesThere are no resolved issues in NimbleOS version 5.3.1.100

Known Issues

Known Issues in NimbleOS version 5.3.1.100



When trying to allocate datapages for many large FingerprintIndex Hash Table indirectblocks, the process may fail dueto insufficient memory. As a re-sult, the Data Service mayrestart to recover the condition.

Data Servicerestarts unexpect-edly while allocat-ing data pages


Not applicableThe Data Service may restartunexpectedly during a read op-eration when decryption anddecompression fails.



Not applicableIn rare instances, the Data Ser-vice may restart when internaldatabase communication be-tween services is not available.The service restarts to restorethe communication betweenservices.

Data Service mayrestart unexpected-ly due to internaldatabase communi-cation


Not applicableA volume that has a chain of atleast three branches, commonin replication or snapshot re-store workflows, may triggerunexpected Data Servicerestarts upon branches beingtrimmed and deleted.

Data Service mayrestart duringbranch point dele-tion


Not applicableA lost wake-up to an operationmay cause the operation block-ing the checkpoint which resultsin Data Service restart.



Resolved Critical Issues 22



If RAID is degraded for an ex-tended period and Data Servicerestarts occur, contact HPENimble Storage to assess adjust-ing allocated buffer resources.

When RAID is degraded, IOneeds to be reconstructed byreading from multiple disks, andan internal buffer may exhaustits allocated resources. In rarecases when multiple disks aredegraded, the Data Service mayrestart unexpectedly.

Data Service mayrestart unexpected-ly when RAID is indegraded mode


Reducing the load on the arrayis the only known workaroundfor this issue.

On CS2XX or CS3XX, the arraymay run out of data pages whenunder a heavy load. This will re-sult in an unexpected Data Ser-vice restart.

Data Service mayrestart unexpected-ly for CS2XX andCS3XX arrays whenunder a heavy load


Not applicableIn rare instances during offsetcalculation, the File System ser-vice may restart when attempt-ing to read data for replication.The service restarts to recoverfrom the condition.

File system servicemay restart on up-stream array duringreplication orresync of syn-chronous replica-tion.


Not applicableSoftware update to NimbleOS5.2.1.0 and later triggers an up-date of on-disk data structures.Under certain rare conditions,this update could potentiallytake a long time and result in arestart of the Data Service. Thismay cause a momentary delayin IO but should not result in anysignificant disruption, as the da-ta services daemon is designedto restart efficiently before anyIO timeout can occur.



Data service will be available af-ter restart.

During NimbleOS metadatasync, in rare instances, the dataservice may restart unexpected-ly. The metadata sync operationitself wont be affected and therestart will reset the race condi-tion; the data service will stabi-lize after the restart.

Data ServiceRestart due to arace condition dur-ing metadata sync


Known Issues 23



Not applicableUnder certain conditions, theData Service may restart whena large number of internal index-es merge within a short time-span and when some of theseindexes need to merge multipletimes in the same checkpoint. Inthis case, there can be lock con-tention between operations re-sponsible for picking what tomerge and operations process-ing the merge, leading to healthcheck timeout.

Data Service mayrestart unexpected-ly with VM healthcheck failure due tolock contention.


Not applicableUnder rare circumstances, inter-nal threads in the Data Servicecan enter a deadlock when con-tending on the same set of re-sources. When the processhealth monitor detects thisdeadlock, it will force the DataService process to restart in or-der to recover from this situa-tion.

Deadlock in DataService processmay cause a restart



The Data Service may restartunexpectedly if the VolumeManager health check exceedsexpected time out. The servicerestarts to clear the condition.

Data Service mayrestart due to vol-ume managerhealth check failure


Not applicableAs part of the file system check-point process, internal indexesare required to merge to disk.The health check process usesa heartbeat mechanism to deter-mine if the merge is makingprogress. Under some circum-stances, the process performingthe merge of a batch of updatesto the index fails to heartbeatwithin a deadline. When thathappens, the Data Service mayrestart with a health check fail-ure.

Data Service mayrestart unexpected-ly with health checkfailure when inter-nal index datastructures take toolong to merge


Known Issues 24



Not applicableThe data services daemon peri-odically synchronizes its persis-tent in-memory (NVRAM)structures with its other objectson flash. This process is requiredto complete within a reasonableamount of time. Under certainrare conditions, this process canget delayed and trigger a restartof the data services daemon.This can cause a momentarydelay in IO which the daemonrestarts. There should not beany disruption of IO, since thedaemon is designed to restartwithin any IO timeout limits.



Not applicableThe File System Service mayunexpectedly restart because itis not able to allocate memory.This can happen when the pro-cess in charge of evicting dataout of cache cannot evict fastenough, so the filesystem is un-able to allocate memory for newincoming data.

File System Servicemay restart due tomemory exhaustion


The array group will recover it-self following the Data Servicerestart.

The Data Service may restartunexpectedly on an array groupwith Storage Class Memory andSynchronous Replication config-ured, during an automaticfailover (AFO) event. This occurswhen invalidating index data inmemory exceeds the timeout of30 seconds.

Data Service mayrestart duringmetadata invalida-tion


Not applicableUnder certain circumstances, anunexpected Data Service restartmay be triggered with a genericAsynchronous Task Managerhealth check failure. The DataService restarts and resumesoperation normally, withoutcausing loss of access from theconnected hosts.

Unexpected DataService Restart


Known Issues 25



Not applicableIn certain multi-array setupwhere replication is in progressmight encounter a situationwhere peer array is goingthrough a software upgrade andis actually unreachable by otherarray. The other array is howev-er unaware of it because somenetwork device has intervenedthe reachability status of thepeer array. The other arrayneeds to perform certain opera-tions which has a timeout associ-ated with it and undergoes asystem restart to recover.

Multi-array setupwith replicationmight experiencefile system restartswhen peer array isunreachable


Not applicableIn one of the code paths for theprocess that handles fingerprintmapping, the merge status is notproperly set. This may lead toan unexpected Data Servicerestart.

Data Servicerestarts when oneof the processes in-correctly sets its re-turn status.


Not applicableThe Group Data service mayrestart unexpectedly when outof memory condition is encoun-tered. The service restarts toclear the condition.

Group Data Servicemay restart unex-pectedly


Not applicableWhile the Data service is process-ing Update Entries in back-ground, one of these back-ground operations may exceedthe expected timeout and theservice will restart to recover.



Not Applicable, the Data Servicewill resume normal operationafter restart.

Generation delete loads a largenumber of ondisk metadatablocks which may prevent blockindex operation checkpoint fromfinishing. This causes the volumemanager health check to failwhich results in Data Servicerestart.

Data Service mayrestart due to vol-ume managerhealth check failureduring generationdeletion


Known Issues 26



Not applicableWhen a read op finds partial da-ta in-core, it issues a media readto get remaining data. By thetime, media read returns, the in-core data is synced, and tree isreopened for deletion. The readdoes not expect tree to be indelete state and causes the DataService to restart.

Data Service mayrestart unexpected-ly due to race condi-tion


Contact Nimble Storage Support.When flash cache Garbage Col-lection copies forward live dataof a fragmented segment, itcould consume more buffersthan provisioned and cause theData Service to restart to recov-er.

Data Service mayrestart when run-ning out of buffers


Not applicable.Due to issues in communicatingwith the partner array duringreplication, there are few caseswhere the operation is not ableto make progress. As a result,the file system may restart tocorrect this condition.

File system restartto recover fromstalled replication


Not applicableWhile committing internaltransactions, Data Service mayhit a rare race condition. To re-cover from this Data Servicemay restart.

Data service mayrestart due to arace condition


Not applicableIf IO to disk are slow or fail, theData Service may restart to tryto recover the condition.

Data Service restartdue to slow disk IOor disk IO failure


Not applicableIn rare cases, Data Service mayrestart during snapshot replica-tion due to the failure of check-sum algorithms to detect allnetwork errors.

Data Service mayrestart due to net-work errors


Known Issues 27



Pause replication until the net-work link used for replication isstabilized.

During replication, a rare racecondition between adding anddeleting internal recurring taskscan cause a Data Service restart.Arrays that hit this conditiontypically experience a singleservice restart that automatical-ly recovers the condition. How-ever, if the network link used byreplication is unstable, frequentreplication disconnects dramati-cally increase the likelihood ofencountering multiple Data Ser-vice restarts due to this condi-tion.

Data Service mayrestart during inter-nal task scheduling


Not applicableDue to a race condition, the DataService may restart during agraceful shutdown causing unex-pected Data Services restartmessages to be generated. Thisshould not cause any I/O impactbecause the Data Service is al-ready in the process of shuttingdown.

Data Service canrestart unexpected-ly during shutdownprocess


Enabling encryption will resolvethe issue

Disabling encryption mightcause NVRAM data to fill upwith data waiting for masterpassphrase, which can causeSynchronous Replicated vol-umes to go out of sync.

Disabling encryp-tion may causeSynchronous Repli-cated volumes toremain out of sync


Not applicableThe Data Service may restartunexpectedly if Fibre Channelsession termination exceedstimeout to complete. The servicerestarts to recover from thecondition.



Perform one of the followingoptions to avoid/mitigate the is-sue: 1. Reduce overall workloadon the array. 2. Reduce copy of-fload workload on the array.Contact HPE Nimble StorageSupport if further assistance isneeded.

Copy offload optimizations fordeduplication were added inNimbleOS versions 5.1.x andlater. The changes may lead toincreased latency when copy of-fload workflows are added to thearray. This is due to an increasein update entries requiring pro-cessing.

Arrays may experi-ence increased la-tency during copyoffload workloadoperations


Known Issues 28



Retry operation after a few min-utes to reassign array to pool.

Assigning an array to a pool im-mediately after unassigning itfrom the same pool will fail withthe following error - Failed toassign arrays to the pool: A ser-vice is not running or is notreachable

Unassigning andreassigning array toa pool within 5 min-utes will fail


Remove downstream replica us-ing the correct steps orderedbelow. 1. Claim the repli-ca volume. 2. Delete all of thesnapshots for the replica vol-ume. 3. Delete the replica vol-ume.

The service may restart whenremoving the downstream repli-ca using the steps below. 1. Deletion of all snapshots forthe replica volume. 2. Claim thereplica volume 3. Delete thereplica volume

Very rare race be-tween Vol claim(with all snapshotsmarked for dele-tion) and space re-calculation on repli-ca downstream vol-ume


Not applicableThe volume manager will decre-ment bin and child volume folderentries during volume deletion.In rare occurrences, anotherthread may decrement the childvolume entry and not the binentry, causing the Data serviceto restart.

Data Service mayrestart on down-stream array due torace condition dur-ing volume deletion


Reduce the frequency of volumeclone create and delete opera-tions to minimize the likelihoodof occurrence.

Aggressive volume creation anddeletion may cause one of thetree data structures to hit maxi-mum children it can have. Thislimit can occur on volume cre-ation because volume deletiondestroys the children asyn-chronously. The Data Service willrestart when this limit has beenreached.

Aggressive Volumecreation and dele-tion may result inData Service restart


Not applicableUnder certain circumstances, anaggressive internal assert maylead to unexpected restart of theData Service.

Aggressive assertmay lead to unex-pected restart ofthe Data Service


Not applicableThe Group Management servicecommunicates with the VSS ser-vice when creating VSS snap-shots. If this communication isnot able to be completed withinthe expected timeframe, theservice restarts to recover.

Group Managementservice may restartwhen attempting tocreate VSS snap-shot


Contact HPE Nimble StorageSupport to increase the memorylimit for the service.

In some instances the cimserverservice will exceed its memorylimit and no longer be able tostart.

Cimserver stopswhen memory limitis exceeded.


Known Issues 29



Contact Nimble Storage Supportto identify the reason for thepartition filling up and to cleanup the log partition.

Log files generated by processesin the Nimble Operating systemsare rotated and archived usingcertain parameters. Under someconditions, if processes log morethan normal, this can result inthe log partitions being filled up,leading to other issues with crit-ical processes.

Log partition mayfill up

PlatformAS-50033

Please contact HPE NimbleStorage Support

In rare incidents, controllers donot power on following powercycle.

Controller does notpower on followinga power cycle.

PlatformAS-100088

The controller reboot should re-store SAS HBA to normal state.HPE Nimble Support may con-tact customer to collect addition-al diagnostics if required.

When the SAS HBA detectsfaulty states, to recover, the ar-ray needs to reset the SASHBA’s firmware. The SAS HBAfirmware reset can block diskI/Os significantly longer than ourHigh Availability monitoringtimeouts allow. Instead, a con-troller reboot is triggered imme-diately if this state is detected,resulting in an unexpectedtakeover event.

Unexpected con-troller takeover dueto incorrect state ofthe SAS HBA

PlatformAS-33725

Referencing the correspondingExpansion Shelf Quick StartGuide, correct the hardware ca-bling issue.

If a shelf is cabled incorrectlyand added to an array, the rawcapacity calculation could failleading to multiple restarts ofthe Data Service.

Incorrect shelf ca-bling can cause aseries of unexpect-ed Data Servicerestarts

PlatformAS-67242

1. Reseat one of the power sup-plies to clear the hang condition.2. Or update NimbleOS to ver-sion 3.6.0.0 or later to minimizethe number of internal com-mands that can trigger this is-sue.

Under certain circumstances, in-ternal commands may cause theintegrated circuit to hang whichcauses both power supplies ofthe head shelf to appear asmissing. The problem does notimpact the power supplys abilityto deliver power to the array.

Both power sup-plies showing up asmissing

PlatformAS-53621

Not applicableIn certain conditions, I/Ostracked for certain operationsmay total zero, which triggers acheck exception. The reason forthe incorrect value is variabletruncation. The service restartsto recover from the condition.

File System servicemay be restart ontruncated variableread

PlatformAS-114564

Known Issues 30



Not applicableThe Enclosure Managementservice may unexpectedlyrestart due to a race conditionwhich is resulting from drivestatus which has already beenmarked removed by the DataService. The restart of the ser-vice clears the condition.

Enclosure Manage-ment service mayunexpectedlyrestart due to drivestatus race condi-tion

PlatformAS-95294

Not applicableThe Data Service may restartunexpectedly when it detects aninternal check error. The restartof the service clears the condi-tion.

Data Service mayunexpectedlyrestart

PlatformAS-90850

Contact HPE Nimble StorageSupport to identify the faultydrive for replacement.

Data Service may become latentand restart when the array hasa faulty drive with a lot of medi-um read and IO timeout errors.

Data Service mayrestart if the arrayhas an faulty drivewith a high numberof medium read er-rors

PlatformAS-110036

The restart of the service willclear the low memory condition,no further action is needed.

Data Service may restart in therare case when a controller islow on memory.

Data Service mayrestart if a con-troller is low onmemory

PlatformAS-99567

Please contact HPE NimbleStorage Support so the problem-atic drive can be replaced.

In rare instances due to slow IOto one of the disks, the DataService may hit a timeout andrestart unexpectedly. This issuemay occur during a NimbleOSsoftware update.

Data Servicerestarts due to slowIO to one of thedisks

PlatformAS-90096

Not applicableThe Data Service may restartdue to a transient resource allo-cation failure. This happenswhen the service cannot com-plete a disk IO due to transientmemory allocation failure. Thisdoes not cause a service outageas Data service continues normal-ly after a restart.

Data Service restartdue to resource allo-cation failure.

PlatformAS-103802

Not applicableOn very rare occasions of HBAerror handling, interrupts couldbe turned off long enough forthe NVRAM driver to timeoutwhile trying to complete directmemory access operations. Thecontroller recovers after a re-boot is initiated by the kernel.

Kernel panics whileprocessing host busadapter errors

PlatformAS-46629

Known Issues 31



Not applicableThe Enclosure Managementservice may unexpectedlyrestart when it detects an inter-nal check error. The servicerestart clears the condition.

Enclosure Manage-ment Service mayrestart unexpected-ly

PlatformAS-105053

Add the disk from the CLI usingthe disk add command and out-put from disk list: 1. Run disk --list 2. Note the slot number, andshelf location for the disk la-beled foreign. 3. Add the disk:disk --add --array --shelf_location Note: the --force optionmay be required Contact HPENimble Storage Support if thedisk does not move to resynchro-nizing state after completing thecommands.

Disk will report foreign for diskstate in the GUI Hardware pageor in the output of disk --listfrom the CLI. This typically oc-curs if diagnostic data may nothave been removed after test-ing.

Replacement diskreports foreign fordisk state inGUI/CLI

PlatformAS-99428

Review the array configurationmatrix for the array model: https://infos-ight.hpe.com/InfoSight/media/lo-cal/active/34/CSxxx%20Con-fig%20Matrix.pdf Removeany additional cache from theexpansion shelf that exceeds themax cache limit based on arraymodel.

ES2 and AFS2 expansionshelves contain additional slotsfor upgrading cache capacity ofthe array. Older array modelshave a maximum cache limit thatcan be handled by the array. IfES2 or AFS2 expansion shelvesare added to an array and thecache exceeds the max cachelimit for the array type, the dataservice may restart due to run-ning out of data pages.

Data service mayrestart if maximumcache exceeded forCS215, CS235,CS300, CS500,CS700 arrays

PlatformAS-93296

Not applicableIn rare cases, an SSD may reachits endurance limit but continuesto pass Nimble Drive Error Re-covery algorithm. This causes anever-ending process of off-lin-ing and on-lining the drive. Thismay occur for Intel and ToshibaSSDs.

SSD has reached itsendurance limit(wear leveling) butthe disk is notmarked failed.

PlatformAS-91522

No workaround is required. Thearray will recover itself automat-ically by restarting the DataService.

On rare occasions, the arraygroups Data Service may fail toinitialize if a shelf state changeoccurs simultaneously.

Delay with DataService startingduring shelf statechange

PlatformAS-101570

Known Issues 32



Not applicableIn a rare scenario, EnclosureManagement service may occa-sionally restart upon a drivefailure in system partitions. Therestart of the service clears thecondition.

Enclosure Manage-ment service occa-sionally restarts up-on a drive failure

PlatformAS-86100

Not applicableThe Group Management servicemay restart if an error whileopening interconnect-ips fileoccurs. The service restarts torecover from the condition.

Group Managementservice may restartunexpectedly

PlatformAS-112701

Not applicableThe Group Management Servicemay restart unexpectedly dueto receive buffer exhaustion onthe management network inter-face. No visible impact has beenreported because of this issue.

Unexpected GroupManagement Ser-vice restart due toreceive buffer ex-haustion

PlatformAS-98885

Not applicableFile System service may restartunexpectedly when a healthcheck exceeds timeout for vol-ume manager on the array. Theservice restarts to recover fromthe condition.

File System Servicemay restart unex-pectedly

PlatformAS-92471

Please contact HPE NimbleStorage Support to resolve thisissue. This issue is resolved inNimbleOS version 5.0.2.0 andlater.

An attempt to create new VMsin a VMWare vVol environmentfails after updating to NimbleOS4.x.x.x. This occurs when a sys-tem partition within the array isrunning out of space.

VM creation fails inVMWare VVol envi-ronment after up-date to NimbleOS4.x.x.x

PlatformAS-76174

After confirming the array hasreturned to Active/Standby sta-tus, stop the hung task and runthe Nimble Add ProStack servertask again.

If an array failover occurs whileNimble Add ProStack server taskis running from vCenter, theprocess may hang and not com-plete.

Nimble AddProStack servertask may hang ifarray failover oc-curs

dHCIAS-97685

Known Issues 33



Free up /var disk space bydeleting unwanted log files(/var/log/EMU/mili/mili2d.log)and rebooting the host.

As part of the dHCI updateworkflow which involves the ES-Xi server update, some com-mands need to be run on theserver. Running these com-mands fails with error Error: Ageneral system error occurred:Internal error. if the /var is full.This is a known issue with 6.7builds running with Emulexdriver which has been fixed aspart of ESX 6.7 U2. https://docs.vmware.com/en/VMware-vSphere/6.7/rn/vsphere-esxi-67u2-release-notes.html#re-solvedissues

dHCI update work-flow can fail if /varmountpoint on thehost is full.

dHCIAS-103315

The only way to resume the up-date would be to failover to theoriginal group leader array andthen resuming the updatethrough the dHCI vCenter plug-in.

When the dHCI unified updatefeature is used to update thedHCI stack, the update will failif a Group Leader Failover oc-curs during the process.

An in-progess dHCIupdate fails if thegroup leader failsover

dHCIAS-103247

Not applicableWhen an ESXi server is added todHCI cluster, the update pagedoes not get updated to includethe newly added ESXi server.This refresh happens every 4hours. After the next refresh, thenew ESXi servers version will beincluded and accounted for onthe update page.

The update pageon the dHCI plugintakes 4 hours to re-fresh

dHCIAS-103769

Not applicableThe Data service will wait up to300 seconds for host responseto iSCSI commands. If host doesnot abort commands that takemore than 300 seconds to com-plete, the Data service willrestart.

Data Service mayrestart when hostdoes not issueabort to timed outcommand

SANAS-108946

Not applicable.During a controller reboot, dueto resource contention betweennew Fibre Channel (FC) connec-tion attempts and shutdown ofthe FC module, the Data Serviceon the array may restart unex-pectedly.

Data Service restartduring shuttingdown FC service onthe standby con-troller

SANAS-100197

Known Issues 34



Increase NCMs configuration forminimum number of connectionsper volume from 2 to 4 or high-er.

The NCM (Nimble ConnectionManager) handles the iSCSIconnection counts. Today, theNCM is designed to use a FULLMESH connection mechanism.However, when there are veryfew initiators and target ports,the total connection count willbe minimal. The NCM will defaultto a minimum of 2 connectionsper volume. Increasing the num-ber of connections to 4 or moremay produce the maximumthroughput.

Poor iSCSI perfor-mance seen whenminimum connec-tion count is used

SANAS-111926

Identify the offline volumes orstale targets and initiate a hostside cleanup/rescan to avoid re-peated login attempts and fail-ure to the offline or stale targets.

Logins to CHAP authenticationenabled volumes could leak asmall amount of memory thesize of CHAP username. Over aperiod of days and weeks, thisleak can result in one or more ofthe NimbleOS processes runningout of memory. As a result, Nim-bleOS services may restart unex-pectedly. The issue is exacerbat-ed by repeated failed login at-tempts to offline volumes orstale targets.

NimbleOS servicesmay restart unex-pectedly due toslow leak withCHAP logins

SANAS-109412

Not applicableWhen the Data Service is in pro-cess of shutdown, the servicemay restart due to race condi-tion when shutdown threads donot wait or abort pending opera-tion threads.

Data Service mayrestart due to racecondition

SANAS-64790

Not applicableWhen the active controller isbeing shutdown, the Data Ser-vice runs into an internal errorcondition that causes the serviceto restart unexpectedly. Sincethe process is already beingshutdown, there is no impact touser data availability.

The Data Servicerestarts unexpect-edly during shut-down

SANAS-98042

Known Issues 35



Not applicableUnder certain conditions, theData Service on the GroupLeader array may restart unex-pectedly while removing mem-ber array. This is due to a racecondition when processing SCSIRTPG (REPORT TARGET PORTGROUPS) commands. The ser-vice should stabilize on its ownshortly following the restart.

Data Service mayrestart unexpected-ly while removingmember array

SANAS-101325

Not applicableThe Data Service or SCSI HighAvailability Service may restartdue to race condition encoun-tered during process shutdown.

Service may restartdue to race condi-tion

SANAS-89753

Not applicableGroup Management Service mayrestart unexpectedly as onethread has taken a ReadWritelock which has another writerthread, which is waiting forScale-Out Database (SODB)transaction to be completed.The service restarts due to theSODB transaction exceeding theexpected timeout.

Group ManagementService may restartdue to race condi-tion

System Manage-ment

AS-103766

Not applicableA race condition may occur whileupdating records in the ScaleOut database when two threadsattempt to update the samerecord.

Group Managementservice may restartdue to race condi-tion

System Manage-ment

AS-110367

Not applicableWhen created with the RESTAPI, by default the cloned vol-ume inherits the ACL from theparent volume. There is no wayto set or change the ACL for theclone at the time of creationthrough the REST API. Clientsshould make two REST calls, oneto create the clone and then oneto change the ACL to the de-sired value. In the future, wecould change the REST API tonot set the ACL by default andallow the client to set the ACLat the time of the clone creationto the desired value. This onlyapplies to volumes that haveagent_type set to none

REST API Clone ofvolume does notsupport settingACL at time ofclone creation

System Manage-ment

AS-54302

Known Issues 36



Not applicableDue to the multi-threaded na-ture of the Alerts and AlarmsService, it is possible for anevent with a later id to be post-ed prior to an event with an ear-lier id. Therefore it will have anearlier timestamp even thoughits ID is higher. This can causethe IDs to appear out of order.However, the alarms in the listare ordered correctly by times-tamp.

Alarm IDs in alarmlist may appear outof order.

System Manage-ment

AS-105431

After confirming that the config-uration sync has completed thealarms can be deleted or ac-knowledged through the GUI.

Alarms describing a delayedconfiguration synchronizationto an array may not be clearedproperly after an extended net-work outage and remain activein the GUI.

Alarms for Configu-ration Synchroniza-tion may remain af-ter sync has com-pleted

System Manage-ment

AS-99427

Not applicableAPI calls occur between theEvent service and Group Man-agement service when an alarmis cleared. The calls will lookupthe alarm ID and onset event inthe Scale Out Database (SODB).If the alarm is cleared before theonset event lookup completesand is not found during thesecalls, the service may restartunexpectedly.

Event service mayrestart unexpected-ly

System Manage-ment

AS-83604

The problematic alarm can bemanually deleted through theCLI command: alarm --delete

Due to a race condition it ispossible for an alarm to remainuncleared on the system evenwhen the alarm condition it isreporting is no longer the case.This can happen if the onsetalert and recovery alert weregenerated at close to the sametime.

Alarm may notclear as expected

System Manage-ment

AS-94398

Clear the alarm manually fromthe Command Line Interface: 1.Use alarm --list command to de-termine the alarm ID. 2. Usealarm --delete com-mand to delete the alarm.

An alarm onset should be setprior to discovery request aspart of alarm processing. Insome cases the discovery re-quest will appear first resultingin the alarm not being cleared.

Configuration syn-chronization recov-ery alarm is notcleared

System Manage-ment

AS-87604

Known Issues 37



Not applicableThe eventd process may restartunexpectedly in systems withlarger configurations due to ex-ceeding memory limit. The ser-vice will recover after the restart.

Eventd processmay restart due toexceeding memorylimit

System Manage-ment

AS-99679

Not applicableThe Event service may restartunexpectedly due to a memoryaccess issue. The restart will notimpact data connections to thearray and the process will recov-er after the restart.

The Event servicemay restart unex-pectedly

System Manage-ment

AS-85608

Not applicableThe Eventd process may restartunexpectedly due to a raredeadlock condition between itsthreads. The restart of the pro-cess will clear the condition.

Eventd processmay restart unex-pectedly

System Manage-ment

AS-103567

Not applicableAlerts and Alarms processingservice may restart unexpected-ly when certain operation sur-pass the designated healthcheck timeout. The process willstabilize following the restart.

Alerts and Alarmsprocessing servicemay restart unex-pectedly

System Manage-ment

AS-50821

Not applicableEvent Service may restart whenmemory limit is reached. Theservice will restart to recoverfrom the condition.

Event Servicerestarts due tomemory

System Manage-ment

AS-84502

Not applicableThe Eventd process may infre-quently crash, due to a bug in anexternal library used by the DNSresolver. The restart will not im-pact data connections to the ar-ray and the process will recoverafter the restart.

Eventd processmay restart inter-mittently

System Manage-ment

AS-46024

The recipient can create an in-bound transport rule to appenda disclaimer to the messagesfrom the problematic sender.The disclaimer will append theexpected CR-LF combination tothe message so that it can bedelivered. (This disclaimer mayconsist of a single charactersuch as a period or a dash.)

When the array sends an alertvia SMTP, the message may berejected by the server indicating550 5.6.11 SMTPSEND.BareLine-feedsAreIllegal. This can occurif the receiving server does notsupport BDAT command forSMTP chunking

Alerts sent viaSMTP may be re-jected

System Manage-ment

AS-72902

Known Issues 38



Not applicableInternal database maintenanceprocessing may exceed the ex-pected health check timeout.When this occurs, the ArrayManagement service may restartto recover from the condition.

Array Managementservice may restartunexpectedly

System Manage-ment

AS-110436

Not applicableIf a user tries to create a newuser account, but the user does-nt have the privilege to do so,the user creation will fail. Howev-er, an audit log entry is not cre-ated.

No Audit Log entryis created if userdoes not have theprivilege to createuser

System Manage-ment

AS-71090

The Group Management servicewill recover after the crash. Theworkaround would be to nolonger use an excessively longusername, and reduce the user-name to under 255 characters.

An excessively long usernameof over 255 characters will ex-ceed a character limit within thearray groups auditing frame-work. This would cause theGroup Management service torestart unexpectedly.

Group ManagementService restarts dueto excessively longusername

System Manage-ment

AS-82919

Not applicableEvery night, when inactive keysfor deleted encrypted volumeare deleted by the array, it cre-ates an audit log entry with rootas the user performing the ac-tion.

Purge inactive en-crypted keys ap-pears in audit log.

System Manage-ment

AS-96966

Not applicableThe Array Management Serviceprocess may restart after run-ning for a long time due to amemory leak issue. This cancause a short interruption todata service, but it will resumeonce the process has restarted.

Array ManagementService restart

System Manage-ment

AS-116083

Not applicableThe Group Management Serviceon the array may restart unex-pectedly when a SOAP timeoutis encountered after trying tocreate a volume. This occurs dueto a race condition where thearray attempts to delete thevolume after the creation at-tempt fails.

Group ManagementService restarts fol-lowing volume cre-ation timeout

System Manage-ment

AS-104640

Known Issues 39




The GUI relies on REST queriesto obtain FC session information.In environments where there arehigh number of sessions to thearray, the queries can take along time to complete. This willresult in slower GUI response forthe array.

GUI performancemay be slow for Fi-bre Channel (FC)arrays with highsession count

System Manage-ment

AS-112416

Not applicableConnection to the Scale-OutDatabase can not be reinitializedwhen a transaction is inprogress. The Group Manage-ment Service will restart to re-store connection.

Group Managementservice may restartwhen connectionsto Scale-OutDatabase exceedsthreshold value

System Manage-ment

AS-109127

Not applicableDuring the startup process forGroup Management, key valuemetadata is loaded from theScale Out Database (SODB) intomemory. The Group Manage-ment Service may restart due toexceeding the expected time tocomplete startup if there is alarge amount of metadata toload.

Group Managementservice may restartduring startup

System Manage-ment

AS-112158

Not applicableUnder certain circumstances, theArray Management Service maydetect operations that are stillrunning during a planned shut-down. This may result in an un-expected service restart prior tothe array completing shutdown.

During a plannedshutdown, the Ar-ray ManagementService may restart

System Manage-ment

AS-115493

Not applicableIf there is a network connectivityissue between the witness andGroup Leader array, the groupstatus CLI output will update theFailover Mode from Automaticto Manual until the connectionis reestablished. It also displaysthe Witness Status as N/A asopposed to Unreachable.

group --status CLIoutput shows incor-rect Failover Modeduring networkconnectivity issues

System Manage-ment

AS-99704

Known Issues 40



Not applicableDuring software update of arraygroup with Peer Persistenceconfiguration, AutomaticSwitchover (ASO) is disabled. Ifan array goes down (both con-trollers down) during the soft-ware update process, due to apower failure or other unexpect-ed event, hosts could lose accessto data until the failed array re-covers, or a manual switchoverof the affected volumes is done.

Peer PersistenceAutomaticSwitchover (ASO)is disabled duringsoftware update

System Manage-ment

AS-94649

No workaround is needed. Theservice recovers on its own.

In Automatic Switchover environ-ments, in rare instances, networkisolation of the Group Leaderand Backup Group Leader maycause the service that handlesthe automatic switchovers torestart unexpectedly.

Network isolationof the Group Lead-er and BackupGroup Leader arraymay lead to Auto-matic Switchoverservice restarts

System Manage-ment

AS-94683

This restart is non-disruptive tothe data on the array, and theArray Management recovers af-ter the restart occurs.

The arrays database system maybecome unavailable for a limitedtime when there is a failure insetting up the Backup GroupLeader. When attempting todiscover a new Backup GroupLeader, the Array ManagementService may restart due to a racecondition.

Array ManagementService restartsduring BackupGroup Leader dis-covery

System Manage-ment

AS-98953

Not applicableThe array group CLI commandwith limits option (group --list-limits) displays numeric internalidentifiers as part of the informa-tion listed for the volume infor-mation. These numeric identi-fiers are used by the array onlyand can be ignored.

Group limits com-mand lists internalidentifiers

System Manage-ment

AS-101342

Not applicableThe Array Management servicemay restart unexpectedly dueto an a memory allocation failurewhen attempting to synchronizeconfiguration with a member ar-ray. The restart of the ArrayManagement service clears thesituation.

Array Managementservice restarts dueto memory alloca-tion issue

System Manage-ment

AS-104812

Known Issues 41



Please contact HPE NimbleSupport.

Following a power outage, it ispossible that the Backup GroupLeader is not assigned to thegroup. This may occur if theSODB database does not startdue to an SSH key issue.

Backup GroupLeader is not as-signed due to pow-er outage

System Manage-ment

AS-99702

If array is queried for statistics,then its frequency should be re-duced. This is specially relevantfor the setup where script orsome monitoring tool does statsquery frequently in a loop.

There is a memory leak in thestat (statistics) component ofthe Array Management service.Magnitude of the memory leakis proportional to the frequencyof stats query.

Array Managementservice restart dueto memory alloca-tion exception

System Manage-ment

AS-98297

The workaround would be todelete the alarm using the alarm--delete CLI.

When alarm for number of nicsin array net config reaching 120is triggered, it doesnt getcleared even when the numberof nics goes down.

Max limit of 120nics in netconfigalarm does not getcleared

System Manage-ment

AS-87749

Not applicableThe Group Management servicemay restart while collectingstatistics from member array.This can occur when the requestfrom the group leader to mem-ber array exceeds timeout,causing the service to restart torecover.

Group Managementservice may restartwhile collectingmember arraystatistics

System Manage-ment

AS-93469


The Event service may restartunexpectedly if an invalid net-work interface entry is found inthe Scale Out Database (SODB).

Event Service mayrestart due to in-valid database en-try

System Manage-ment

AS-103275

Manually delete the alarm fromthe command line interface. 1.Login to CLI 2. Find the alarm IDusing: alarm --list 3. Delete thealarm using the ID alarm --delete

If an IP address is migrated fromone interface port to another,the alarm that was generatedindicating the original interfacedown may not be cleared.

Alarm not clearedafter IP addressmoves to differentinterface

System Manage-ment

AS-92515

Known Issues 42



The only workaround is to im-prove the quality of the networkconnection so that the loss ofTCP packets is minimized.

Service threads within the groupmanagement API handler arerestricted to run for no morethan 5 minutes. In a very lossynetwork, TCP throughput canbe throttled to almost nothingdue to retransmission timeoutback-off. In one observed case,where the API response wasabout 190 KB, the connectionmanaged to send only about 90KB before the 5 minute timeoutoccurred and caused the GroupManagement service restart.

A lossy networkcauses Group Man-agement servicerestart

System Manage-ment

AS-73595

Not applicableDue to an issue in POCO library,in certain cases, causing theGroup Management service torestart. The service restarts torecover from the condition.

During shutdownthe Group Manage-ment service mayunexpectedlyrestart

System Manage-ment

AS-112702

Not applicableArray Management Service mayrestart unexpectedly when try-ing to start or stop the GroupManagement Service. The sys-tem recovers automatically afterthe Array Management Serviceis restarted

Array ManagementService restartedwhile starting/stop-ping Group Manage-ment Services

System Manage-ment

AS-74556

Not applicableThere is a --force switch avail-able when deleting a perfor-mance policy via the HPE Nim-ble Storage Array CLI. This