EMC Isilon Customer Troubleshooting Guide: How to · PDF file3 - EMC Isilon Customer Troubleshooting Guide: How to Add a Drive to a Node We appreciate your help in improving this document

For links to all Isilon customer troubleshooting guides, visit the Customer Troubleshooting - Isilon Info Hub.

We appreciate your help in improving this document. Submit your feedback at http://bit.ly/isi-docfeedback._________________

___________________________

1 - EMC Isilon Customer Troubleshooting Guide: Hardware – Add a Drive to a Node

Abstract

This guide helps you to troubleshoot problems with replacing a drive in a node.

March 20, 2018

EMC ISILON CUSTOMER TROUBLESHOOTING GUIDE

HOW TO ADD A DRIVE TO A NODE

OneFS 7.2 - 8.1.1

http://bit.ly/isi-docfeedback

https://community.emc.com/docs/DOC-49017




___________________________


Contents and overview

Note Follow all of these steps, in order, until you reach a resolution.

1. Follow these

steps.

2. Perform

troubleshooting

steps in order.

3. Appendixes

Page 3 Before you begin

Appendix A If you need further assistance

Page 4 Start troubleshooting

Appendix B How to use this flowchart

Page 5 Add the drive

Appendix C Example isi devices list and isi devices output

Appendix D Example isi job status output with FlexProtect job running

Appendix E Example isi job status output with FlexProtect job completed

Appendix F Example isi devices list and isi devices output showing a

PREPARING or a HEALTHY drive

Appendix G Example output from the isi devices -a add -d :<bay#> command






___________________________


Before you begin

CAUTION!If the node, subnet, or pool that you are working on goes down during the course of

troubleshooting and you do not have any other way to connect to the cluster , you could

experience data unavailability.

Therefore, make sure that you have more than one way to connect to the cluster before

you start this troubleshooting process. The best method is to have a serial console

connection available. This way, if you are unable to connect through the network, you

will still be able to connect to the cluster physically.

For specific requirements and instructions for making a physical connection to the

cluster, see article 304071 on the Dell EMC Customer Support website.

Before you begin troubleshooting, confirm that you can connect through either another

subnet or pool, or that you have physical access to the cluster.

Configure screen logging through SSH

We recommend that you configure screen logging to log all session input and output during your troubleshooting session .

This log file can be shared with Isilon Technical Support , if you require assistance at any point during troubleshooting.

Note: The screen session capability does not work in OneFS 7.1.0.6 and 7.1.1.2. If you are running either of these versions,

you can configure logging by using your local SSH client 's logging feature.

1. Open an SSH connection to the cluster and log in by using the root account .

Note: If the cluster is in compliance mode, use the compadmin account to log in. All compadmin commands must be

preceded by the sudo prefix.

2. Change the directory to /ifs/data/Isilon_Support by running the following command:

cd /ifs/data/Isilon_Support

3. Run the following command to capture all input and output from the session:

screen -L

This will create a file named screenlog.0 that will be appended to during your session.

4. Perform troubleshooting.




https://support.emc.com/kb/304071



___________________________


Start troubleshooting

Does the output

indicate that any of the

drive bays are in

SMARTFAIL, REPLACE,

EMPTY, or NEW

status?

Go to Page 5

SMARTFAIL

IntroductionStart troubleshooting here. If you need

help to understand the flowchart

conventions used in this guide, see

Appendix B: How to use this flowchart.

If you have not done so already, log in to

the cluster and configure screen logging

through SSH, as described on page 3.

Note Self encrypting drives (SEDs) can stay in

the PREPARING state for an hour or

longer. This behavior is normal when

formatting a drive. Do not interrupt the

PREPARING state because doing so locks

the drive.

Go to Page 6

REPLACE,

EMPTY, or

NEW

Start

__________

To determine the drive health, run the

following command for your version of OneFS:

OneFS 8.0.0 - 8.1.1

isi devices list

OneFS 7.2

isi devices

See Appendix C for example output






___________________________


Add the drive

Verify whether the FlexProtect job is running by running the following command:

isi job status

In the Running and queued jobs section, find the FlexProtect job.

See Appendix D for example output.

Is the

FlexProtect

job running?

Allow the FlexProtect job to finish.

After the job is successfully

completed, the drive bay status

changes to REPLACE.

See Appendix E for example output

of a successfully completed

FlexProtect job and the subsequent

REPLACE status.

Yes

Return to Page 4

No

Page

5

You could have arrived here from:

Page 4 – Start troubleshooting

Page 6 – Add the drive (2)

Note the page number that you

are currently on.

Upload log files and contact Isilon Technical

Support, as instructed in Appendix A.

_________________________

_____________________

__________

After the FlexProtect job is

successfully completed, return to

page 4 to continue troubleshooting.

__________






___________________________


Add the drive (2)

Look for drives that are in a down or soft_fail status by

running the following command:

isi_group_info

See the example output at the bottom of this page.

Page

6



Does the output

indicate that any devices are

in a down or soft_failed

status?

Return to Page 5

Yes

Go to Page 7

No

Cluster-1# isi group info

efs.gmp.group info: <5,859>: { 1:0-5,7-11, 2:0-4,6-10,12-13, 5:0-11, smb: 1-2,5, nfs: 1-2,5, hdfs:

1-2,5, swift: 1-2,5, all_enabled_protocols: 1-2,5 }






___________________________


Add the drive (3)

Replace the drive by consulting the drive replacement

guide for the particular type of node.

See this search string for a list of the available drive

replacement documents.

Page

7



Go to Page 8YesIs the drive

successfully added?


are currently on.



No

To add the new drive, run the following command on the affected

node, where <bay#> is the number of the bay and <node#> is the number of the node to which the drive was added:

OneFS 8.0 – 8.1.1 isi devices drive add <bay#> --node-lnn <node#>

OneFS 7.2

isi devices -a add -d :<bay#>

See Appendix G for example output to determine whether the drive is successfully added.




https://support.emc.com/search/?text=how%20to%20replace%20a%20drive&product_id=15204&AlloftheseWrds=replace%2C%20drive&searchLang=en_US&adv=y%20



___________________________


Add the drive (4)

Does the output

indicate that the drive

status is PREPARING or

HEALTHY?

Page

8




Note Self encrypting drives (SEDs)

can stay in the PREPARING

state for an hour or longer.

This is normal behavior when

formatting a drive. Do not

interrupt the PREPARING state

because this will lock the drive.

Go to Page 9No

A drive status of PREPARING or

HEALTHY indicates that the drive

replacement is successful.

Yes

Return the non-working drive to EMC

Isilon Technical Support by using the

provided return label that is located under

the shipping label of the new drive.

End troubleshooting

_____________________

______________________

__________

To determine the drive health, run the following

command for your version of OneFS:

OneFS 8.0.0 - 8.1.1

isi devices list

OneFS 7.2

isi devices

In the output, look for the bay number and the

status of PREPARING or HEALTHY

See Appendix F for example output






___________________________


Add the drive (5)

Page

9



Does the output

indicate that the drive

status is EMPTY or

REPLACE?


are currently on.



No

Determine whether your workflow

allows you to power cycle the node

now, without affecting production.

Yes

Are you able to

power cycle the node to

finish troubleshooting,

without affecting

production?

No

Unable to continue troubleshooting.

Isilon Technical Support suggests scheduling

an outage window to complete the drive

installation.

When you are able to continue troubleshooting,

return to page 10 of this guide.

Go to Page 10

Yes

_______






___________________________


Add the drive (6)

Page

10



Power cycle the node.

After the node reboots, try to add the drive again by running the following

command on the affected node, where <node#> is the number of the

node, and <bay#> is the number of the bay that the drive was added to:

OneFS 8.0 – 8.1.1

Isi devices drive add <bay#> –node-lnn <node#>

OneFS 7.2

isi devices -a add -d <node#>:<bay#>

See Appendix G for example output that indicates the drive replacement is

successful.

Did the drive add

operation succeed?


are currently on.



No

Return to Page 8Yes






___________________________


Contact EMC Isilon Technical Support

If you need to contact Isilon Technical Support during troubleshooting, reference the page or step that you need help with.

This information and the log file will help Isilon Technical Support staff resolve your case more quickly.

Appendix A: If you need further assistance

Upload node log files and the screen log file to EMC Isilon Technical Support

1. When troubleshooting is complete, type exit to end your screen session.

2. Gather and upload the node log set and include the SSH screen log file by using the command appropriate for your

method of uploading files. If you are not sure which method to use, use FTP.

ESRS:

isi_gather_info --esrs --local-only -f /ifs/data/Isilon_Support/screenlog.0

FTP:

isi_gather_info --ftp --local-only -f /ifs/data/Isilon_Support/screenlog.0

HTTP:

isi_gather_info --http --local-only -f /ifs/data/Isilon_Support/screenlog.0

SMTP:

isi_gather_info --email --local-only -f /ifs/data/Isilon_Support/screenlog.0

SupportIQ:

Copy and paste the following command.

Note: When you copy and paste the command into the command-line interface, it will appear on multiple lines (exactly

as it appears on the page), but when you press Enter, the command will run as it should.

isi_gather_info --local-only -f /ifs/data/Isilon_Support/screenlog.0 --noupload \

--symlink /var/crash/SupportIQ/upload/ftp

3. If you receive a message that the upload was unsuccessful , refer to article 16759 on the EMC Online Support site for

directions on how to upload files over FTP.___________




http://www.emc.com/contact/contact-us.htm




___________________________


Decision diamondYes No

Process stepProcess step with command:

command xyz

Go to Page #

Page

# Note Provides context and additional

information. Sometimes a note is linked

to a process step with a colored dot.

CAUTION!Caution boxes warn that

a particular step needs

to be performed with

great care, to prevent

serious consequences.

End point Document ShapeCalls out supporting documentation

for a process step. When possible,

these shapes contain links to the

reference document.

Sometimes linked to a process step

with a colored dot.

Optional process step

Directional arrows indicate

the path through the

process flow.

IntroductionDescribes what the section helps you to

accomplish.


Page # - Page title

Appendix B: How to use this flowchart







___________________________



Page 4 – Start Troubleshooting

Example isi devices list output

Cluster-2# isi devices list

Lnn Location Device Lnum State Serial

-----------------------------------------------------

2 Bay 1 /dev/da1 12 HEALTHY JPW9K0HD277SAL

2 Bay 2 /dev/da2 10 HEALTHY JPW9K0HD271NKL

2 Bay 3 /dev/da3 9 HEALTHY JPW9K0HZ2KJUYL

2 Bay 4 /dev/da4 8 HEALTHY JPW9K0HZ236JUL

2 Bay 5 /dev/da5 7 HEALTHY JPW9K0J82S0MBL

2 Bay 6 /dev/da6 6 HEALTHY JPW9K0HD277SHL

2 Bay 7 /dev/da7 13 HEALTHY JPW9K0HZ2LG0DL

2 Bay 8 /dev/da8 4 HEALTHY JPW9K0HZ1ZH9UL

2 Bay 9 /dev/da9 3 HEALTHY JPW9K0J82SSGPL

2 Bay 10 /dev/da10 2 HEALTHY JPW9K0HD237TJL

2 Bay 11 /dev/da11 1 HEALTHY JPW9K0HZ2K4M2L

2 Bay 12 /dev/da12 0 HEALTHY JPW9K0HZ236GLL

-----------------------------------------------------

Total: 12

Example output for OneFS 7.2.x is continued on the next page.

Appendix C: Example isi devices list output for OneFS 8.0 – 8.1.1






___________________________


Appendix C: Example isi devices output for OneFS 7.2.x (2)



Example isi devices output

Cluster-1# isi devices

Node 1, [ATTN], [SED Node]

Bay 1 Lnum 34 [HEALTHY] SN:Z298KBBF0000921364ED /dev/da1

Bay 2 Lnum 33 [HEALTHY] SN:Z298H54B00009410BT4C /dev/da2

Bay 3 Lnum 17 [HEALTHY] SN:Z298KARS0000C306EMY3 /dev/da18

Bay 4 Lnum 16 [HEALTHY] SN:Z298JZCA000094042SHE /dev/da19

Bay 5 Lnum 32 [HEALTHY] SN:Z298JXYC0000C3060CU0 /dev/da3

Bay 6 Lnum 15 [HEALTHY] SN:Z298KBJX0000C250BUY4 /dev/da20

Bay 7 Lnum 14 [HEALTHY] SN:Z298K0PR00009301VEAY /dev/da21

Bay 8 Lnum 13 [HEALTHY] SN:Z298JY3J0000C3063VXX /dev/da22

Bay 9 Lnum 31 [HEALTHY] SN:Z298JY81000093078AQ8 /dev/da4

Bay 10 Lnum 12 [HEALTHY] SN:Z298JYL90000C2490VXP /dev/da23

Bay 11 Lnum 11 [HEALTHY] SN:Z298KB7Z0000C305AF8J /dev/da24

Bay 12 Lnum 10 [HEALTHY] SN:Z298JYB800009252UFP1 /dev/da25

Bay 13 Lnum 30 [HEALTHY] SN:Z298KB2A0000C2510H0Q /dev/da5

Bay 14 Lnum 29 [HEALTHY] SN:Z298JYFY00009301VE79 /dev/da6

Bay 15 Lnum 28 [HEALTHY] SN:Z298JYB300009304H7KS /dev/da7

Bay 16 Lnum 9 [HEALTHY] SN:Z298KAWP0000S114Z726 /dev/da26

Bay 17 Lnum 27 [HEALTHY] SN:Z298KBCF0000C2510J1K /dev/da8

Bay 18 Lnum N/A [EMPTY] SN:N/A N/A

Bay 19 Lnum 26 [HEALTHY] SN:Z298K0AX0000C3159JV0 /dev/da9

Bay 20 Lnum 8 [HEALTHY] SN:Z298KBFS0000C2516P72 /dev/da27

Bay 21 Lnum 25 [HEALTHY] SN:Z298K06E00009252U1EU /dev/da10

Bay 22 Lnum 24 [HEALTHY] SN:Z298KB3R00009302X2V7 /dev/da11

Bay 23 Lnum 7 [HEALTHY] SN:Z298JYFR000094042SHP /dev/da28

Bay 24 Lnum 6 [HEALTHY] SN:Z298K97700009328BY46 /dev/da29

Bay 25 Lnum 5 [HEALTHY] SN:Z298KBDM0000C31532VC /dev/da30

Bay 26 Lnum 4 [HEALTHY] SN:Z298JYE400009313D75E /dev/da31

Bay 27 Lnum 3 [HEALTHY] SN:Z298JY4G0000C2516K68 /dev/da32

Bay 28 Lnum 23 [HEALTHY] SN:Z298KB250000C315CUYS /dev/da12

Bay 29 Lnum 2 [HEALTHY] SN:Z298JYGH00009314VE8N /dev/da33

Bay 30 Lnum 22 [HEALTHY] SN:Z298KBD50000C24982KW /dev/da13

Bay 31 Lnum 1 [HEALTHY] SN:Z298K0RE0000C3060DP0 /dev/da34

Bay 32 Lnum 21 [HEALTHY] SN:Z298JXZH0000C251CN9W /dev/da14

Bay 33 Lnum 0 [HEALTHY] SN:Z298KBL60000C314FPMQ /dev/da35

Bay 34 Lnum 20 [HEALTHY] SN:Z298KB1T0000C250GLZG /dev/da15

Bay 35 Lnum 19 [HEALTHY] SN:Z298JYDA00009315GEZ1 /dev/da16

Bay 36 Lnum 18 [HEALTHY] SN:Z298JYRH00009412TWLC /dev/da17






___________________________


Appendix D: Example isi job status output with FlexProtect job

running


Page 5 – Add the drive

Example isi job status output with FlexProtect job running

In this example, the drive in bay 36 was SmartFailed. The output shows that the FlexProtect job is running. See the bold

text in the output:

Cluster-1# isi job status

The job engine can temporarily only run FlexProtect and FlexProtectLin jobs.

Coordinator: 1

Connected: True

Disconnected Nodes: -

Down or Read-Only Nodes: False

Statistics Ready: True

Cluster Is Degraded: True

Run Jobs When Degraded: False

Running and queued jobs:

ID Type State Impact Pri Phase Running Time

----------------------------------------------------------

562 FlexProtect Running Medium 1 2/6 3s

----------------------------------------------------------

Total: 1

Recent finished jobs:

ID Type State Time

-----------------------------------------------------

552 WormQueue Succeeded 2015-12-06T02:00:05

553 ShadowStoreProtect Succeeded 2015-12-06T04:00:13









-----------------------------------------------------

Total: 10

Continued on the next page.________






___________________________


Appendix D: Example isi job status output with FlexProtect job

running (2)


Appendix D: Example isi job status with

FlexProtect job running

Example isi job status output with FlexProtect job running

To confirm that the bay status is SMARTFAIL, run isi devices. See the bold text in the output:






(truncated)








(truncated)


Bay 36 Lnum 18 [SMARTFAIL] SN:Z298JYRH00009412TWLC /dev/da17






___________________________


Appendix E: Example isi job status output with FlexProtect job

completed


Page 5 – Add the drive

Example isi job status output with FlexProtect job completed

To confirm the FlexProtect job is running, run isi devices list (OneFS 8.0-8.1.1) or isi devices (OneFS

7.2). See the bolded line in the output:

Cluster-1# isi job status

The job engine is running.

No running or queued jobs.

Recent finished jobs:

ID Type State Time

-----------------------------------------------------










562 FlexProtect Succeeded 2015-12-09T02:07:15

-----------------------------------------------------

Total: 10

To confirm the drive is ready to replace, run isi devices list (OneFS 8.0-8.1.1) or isi devices (OneFS

7.2). See the bolded line in the output:






(truncated)








(truncated)


Bay 36 Lnum 18 [REPLACE] SN:Z298JYRH00009412TWLC /dev/da17






___________________________


Appendix F: Example isi devices output showing a PREPARING or a

HEALTHY drive



Example isi devices output showing a PREPARING drive

To confirm the drive is formatting, run isi devices. See the bold text in the output:






(truncated)








(truncated)


Bay 36 Lnum 18 [PREPARING] SN:Z298JYRH00009412TWLC /dev/da17

Example isi devices output showing a HEALTHY drive

To confirm the drive is ready to replace, run isi devices. See the bold text in the output:






(truncated)








(truncated)


Bay 36 Lnum 18 [HEALTHY] SN:Z298JYRH00009412TWLC /dev/da17






___________________________


Appendix G: Example output from the

isi devices -a add -d :<bay#> command




Example output from the isi devices -a add -d :<bay#> command

Example output when reading a reinserted, or stopfailed, drive:

Cluster-1# isi devices -a add -d :24

Initiating add on bay 24

. . . . . . . . . . .

!! The add operation succeeded. A healthy drive was found in bay 24.

Example output when adding a blank drive:



. . . . . . . . . . .

!! The add operation failed. An unrecognized drive was found in bay

!! 36. To format this drive and add it to the file system, run 'isi

!! devices -a format' on this drive and follow the instructions

!! provided.

Cluster-1# isi devices -a format -d :36

!! This drive does not appear to be pre-formatted with OneFS.

!! Formatting this drive will destroy any existing data and add it to

!! OneFS. Do you wish to continue formatting this drive? (yes,

!! [cancel])

>>> yes

'format' drive action initiated. This process could take a few

minutes. To monitor the progress of this action, run 'isi devices'.

When the drive appears green, the 'format' and purposing actions have

completed successfully.

Example output when the drive add operation fails, or when there is no drive detected:



. . . . . . . . . . .

!! The add operation failed. No drive was found in bay 18. Verify that

!! a functional drive is physically present in the bay and try again.

_____________________

______________________






___________________________


Copyright © 2018 Dell Inc. or its subsidiaries. All rights reserved.

Dell believes the information in this publication is accurate as of its publication date. The information is subject to change without notice.

THE INFORMATION IN THIS PUBLICATION IS PROVIDED AS-IS. DELL MAKES NO REPRESENTATIONS OR WARRANTIES OF ANY KIND WITH RESPECT TO THE INFORMATION IN THIS PUBLICATION, AND SPECIFICALLY DISCLAIMS IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. USE, COPYING, AND DISTRIBUTION OF ANY DELL SOFTWARE DESCRIBED IN THIS PUBLICATION REQUIRES AN APPLICABLE SOFTWARE LICENSE.

Dell, EMC, and other trademarks are trademarks of Dell Inc. or its subsidiaries. Other trademarks may be the property of their respective owners.

EMC CorporationHopkinton, Massachusetts 01748-91031-508-435-1000 in North America 1-866-464-7381www.EMC.com




https://www.emc.com

Documents

EMC Isilon Customer Troubleshooting Guide: How to · PDF file3 - EMC Isilon Customer Troubleshooting Guide: How to Add a Drive to a Node We appreciate your help in improving this document