EMC Isilon Customer Troubleshooting Guide: SyncIQ Failures · troubleshooting and you do not have any other way to connect to the cluster, you could experience data unavailability

For links to all Isilon customer troubleshooting guides, visit the Customer Troubleshooting - Isilon Info Hub.

We appreciate your help in improving this document. Submit your feedback at http://bit.ly/isilon-docfeedback.

1 - EMC Isilon Customer Troubleshooting Guide: SyncIQ Failures

______________________________________________

Abstract This guide helps you identify and address common issues that can cause SyncIQ to fail.

July 2, 2019

EMC ISILON CUSTOMER TROUBLESHOOTING GUIDE

SYNCIQ FAILURES

OneFS 7.2 - 8.1.0

http://bit.ly/isilon-docfeedback

https://community.emc.com/docs/DOC-49017




______________________________________________

Contents and overview

Before you beginPage 3

Appendix A If you need further assistance

Start troubleshootingPage 4

Sync fails within the first five minutesPage 5

Sync fails after the first five minutesPage 10

SyncIQ error connecting to daemonPage 12

Note Follow all of these steps, in order, until you reach a resolution.

1. Follow these steps.

2. Perform troubleshooting steps in order.

3. Appendixes

Appendix BHow to use this flowchart

Appendix EExample output

Appendix DExample output

Appendix CExample output






______________________________________________

Before you begin

CAUTION!If the node, subnet, or pool that you are working on goes down during the course of troubleshooting and you do not have any other way to connect to the cluster, you could experience data unavailability.

Therefore, make sure that you have more than one way to connect to the cluster before you start this troubleshooting process. The best method is to have a serial console connection available. This way, if you are unable to connect through the network, you will still be able to connect to the cluster physically.

For specific requirements and instructions for making a physical connection to the cluster, see article 304071 on the Online Support site.

Before you begin troubleshooting, confirm that you can connect through either another subnet or pool, or that you have physical access to the cluster.

Configure screen logging through SSHWe recommend that you configure screen logging to log all session input and output during your troubleshooting session. This log file can be shared with Isilon Technical Support, if you require assistance at any point during troubleshooting.

1. Open an SSH connection to the clsuter and log in by using the root account.

Note: If the cluster is in compliance mode, use the compadmin account to log in. All compadmin commands must be preceded by the sudo prefix.

2. Change the directory to / i f s/ dat a/ I si l on_Suppor t by running the following command:

cd / i f s/ dat a/ I si l on_Suppor t

3. Run the following command to capture all input and output from the session:

scr een - L

This will create a file named scr eenl og. 0 that will be appended to during your session.

4. Perform troubleshooting.



https://support.emc.com/kb/304071




______________________________________________

Start troubleshooting

Start

IntroductionStart troubleshooting here. For an overview of the conventions used in this flowchart, see Appendix B: How to use this flowchart.

If you have not done so already, log in to the cluster and configure screen logging through SSH, as described on page 3.

Capture the error for the failing policy as follows:

1. Obtain the SyncIQ job ID by running the following command on the source cluster, where <pol i cy- name> is the name of theSyncIQ policy. See Appendix C for example output.

i si sync r epor t s l i st - - pol i cy- name=<pol i cy- name> - - sor t j ob_i d

2. View the report by running the following command, where <pol i cy- name> is the name of the SyncIQ policy and <j obI D> is the job ID you obtained in step 1. The output of the command lists the error. See Appendix C for example output:

i si sync r epor t s vi ew - - poi l cy=<pol i cy- name> <j obI D> | l ess

In the report, find the error that relates to the failure. Note the start and end times for the policy. See Appendix C for example output.

Did the policy fail within the first five

minutes of starting, or after the first five minutes?

Go to Page 5 Go to Page 10

Afterthe first

five minutes

Withinthe first

five minutes

__________

__________

__________






______________________________________________

Sync fails within the first five minutes

Error TablesFind your error in the tables on this page and the next

few pages. Follow the instructions for your error.

Page 5

You could have arrived here from:

- Page 4 - Start troubleshooting

Error Action

FAI LED ASSERTI ON

Unabl e t o updat e met adat a ( i node changes) . . . Unabl e t o open Li n <l i n>: No such f i l e or di r ect or y

One of the following oper at i on not per mi t t ed er r or s:- unabl e t o del et e- f ai l ed t o move- unabl e t o r ename

Er r or openi ng l i nmap <pol i cyI D>: No such f i l e or di r ect or y

Cl osi ng swor ker ( X) : Br oken pi pe/ Wor k i t em X has been r est ar t ed t oo many t i mes. / SyncI Q pol i cy f ai l ed

Fai l ed t o di f f _r ange, Fai l ed t o di f f f i l e, I nput / out put er r or , Fai l ur e due t o f i l e syst em

er r or s: Mi ssi ng hi gher snapshot ver si on of a f i l e

SyncI Q i s unabl e t o connect t o a r esour ce on t he t ar get cl ust er .

orSour ce node coul d not connect t o t ar get cl ust er .

Contact Isilon Technical Support for assistance.

If your error is not on this page, go to Page 6



http://www.emc.com/contact/contact-us.htm




______________________________________________

Sync fails within the first five minutes (2)

Page 6


- Page 5 - Sync fails within the first five minutes

Error Action

SyncI Q schedul er f ai l ed t o st ar t pol i cy. No nodes i n cl ust er have ext er nal I Ps

orSyncI Q schedul er f ai l ed t o st ar t pol i cy. Pol i cy <name> has an i nval i d subnet : pool

r est r i ct i on of subnet 0: pool 1.

Error with these three phrases together:

- Unabl e t o del et e- unl i nk of Li n 0 f ai l ed- No such f i l e or di r ect or y

These two errors together:

- Fai l ed t o smkchdi r f d t o t ar get di r ect or y

- Oper at i on Not Per mi t t ed

SyncI Q det ect ed a pr obl em wi t h pol i cy conf i gur at i on. Pol i cy r oot pat h

di r ect or y <di r ect or y> does not mat ch bet ween ol d and new snapshot s


Go to the following article to resolve your issue. If the problem persists, contact Isilon Technical Support.

OneFS: SyncIQ policy fails to run with "failed to start policy" error, article 471903


OneFS: SyncIQ policy fails with error: Unable to delete [path]: unlink of Lin 0 failed: No such file or directory, article 471906


OneFS 7.2: SyncIQ fails with error "Failed to smkchdirfd to target directory" and "Operation not permitted," article 469809


OneFS: SyncIQ fails with error: SyncIQ detected a problem with policy configuration. Policy root path directory does not

match between old and new snapshots, article 471898






______________________________________________


Page 7


- Page 6 - Sync fails within the first five minutes (2)

Error Action

SyncI Q f ai l ed t o t ake a snapshot on sour ce cl ust er .

Can' t f i nd l at est snapi d <I D>

SyncI Q f ai l ed t o t ake a snapshot on sour ce cl ust er . Unabl e t o del et e snapshot f r om pr evi ous r un: 44

SyncI Q f ai l ed t o t ake a snapshot on sour ce cl ust er . Fai l ed t o open and l ock

new snapi d XXX: i f s_snap_cr eat e_l ock_l ease( ) f ai l ed:

Oper at i on not per mi t t ed.

Fai l ed t o open: <f i l e pat h> ( l i n xxxxxxxxx) : Per mi ssi on deni ed



OneFS: SyncIQ policy fails with error: SyncIQ failed to take a snapshot on source cluster. Can't find latest snapid XXX,

article 463816


OneFS: Sync fails with error: SyncIQ failed to take a snapshot on source cluster. Unable to delete snapshot from previous

run, article 469839


OneFS: Sync fails with error: SyncIQ failed to take a snapshot on source cluster. Failed to open and lock new snapid XXX:

ifs_snap_create_lock_lease() failed: Operation not permitted, article 471907


Isilon OneFS: A SyncIQ job fails with "permission denied" error because AVScan has quarantined a file, article 477492






______________________________________________


Page 8



Error Action

bad checksum I nput / out put er r or due t o net wor k WAN accel er at or

I ni t i al or di f f sync er r or " Fai l ed t o open di r 0"

Pr i mar y aut hent i cat i on f ai l s

Fai l ed t o move . t mp- wor ki ng- di r



Isilon OneFS: A SyncIQ job fails with "bad checksum: Input/output error", article 477493


Isilon OneFS: A SyncIQ job fails with error "Failed to open dir 0" article 477494


SyncIQ job fails with error Primary authentication fails, article 464224


Isilon OneFS: SyncIQ Error when source directory is overwritten or deleted and re-created: Failed to move

.tmp-working-dir, article 463450

SyncI Q pol i cy f ai l ed. Unexpect ed non- I / O er r or pr eceded by


All syncIQ jobs fail with "Unexpected non-I/O error," preceded by "msg_handshake failed" error,

article 466618






______________________________________________


Page 9



Error Action

( pol i cy name: <name>) SyncI Q det ect ed a pr obl em wi t h pol i cy conf i gur at i on. Pol i cy has

f or ce_i nt er f ace set , but t he sysct l net . i net . i p. choose_i f a_by_i psr c i s not set

SyncI Q er r or connect i ng t o daemon ( bandwi dt h, t hr ot t l e, pwor ker )


OneFS: SyncIQ policy fails with error: SyncIQ detected a problem with policy configuration. Policy has

force_interface set, but the sysctl net.inet.ip.choose_ifa_by_ipsrc is not set,

article 463852

Go to Page 12.

If your error is not listed in the table, contact Isilon Technical

Support for assistance.







______________________________________________

Sync fails after the first five minutes

Page 10


- Page 4 - Start troubleshooting

Error TablesFind your error in the tables on this page and the next

few pages. Follow the instructions for your error.

Error Action

FAI LED ASSERTI ON

Unabl e t o updat e met adat a ( i node changes) . . . Unabl e t o open Li n <l i n>: No such f i l e or

di r ect or y

Contact Isilon Technical Support for assistance.

Cl osi ng swor ker ( X) : Br oken pi pe/ Wor k i t em X has been r est ar t ed t oo many t i mes. / SyncI Q

pol i cy f ai l ed








______________________________________________

Sync fails after the first five minutes (2)

Page 11


- Page 10 - Sync fails after the first five minutes

Error Action

SyncI Q f ai l ed t o t ake a snapshot on sour ce cl ust er .

Fai l ed t o open and l ock new snapi d XXX: i f s_snap_cr eat e_l ock_l ease( ) f ai l ed:

Oper at i on not per mi t t ed.

These two errors together:

- Fai l ed t o smkchdi r f d t o t ar get di r ect or y

- Oper at i on Not Per mi t t ed

bad checksum I nput / out put er r or due t o net wor k WAN accel er at or

SyncI Q er r or connect i ng t o daemon ( bandwi dt h, t hr ot t l e, pwor ker )


OneFS: Sync fails with error: SyncIQ failed to take a snapshot on source cluster. Failed to open and lock new

snapid XXX: ifs_snap_create_lock_lease() failed: Operation not permitted, article 471907


OneFS 7.0 -7.1: SyncIQ fails with error "Failed to smkchdirfd to target directory" and "Operation not

permitted", article 469809


Isilon OneFS: A SyncIQ job fails with "bad checksum: Input/output error",

article 477493

Go to Page 12.

If your error is not listed in the table, contact Isilon Technical

Support for assistance.







______________________________________________

SyncIQ error connecting to daemon

Page 12


- Page 9 - Sync fails within the first five minutes (5)- Page 11 - Sync fails after the first five minutes (2)

Read the following notes about this error. Then continue on to troubleshooting the error.

Notes about this error

Error: SyncI Q er r or connect i ng t o daemon ( bandwi dt h, t hr ot t l e, pwor ker )

This error occurs:When the sync fails at any time during the sync job.

This error appears: In the sync policy report on the OneFS web administration interface or command-line interface and in the / var / l og/ i si _mi gr at e. l og file.

Cause of this error: The source pool does not include node 1. The bandwidth/throttle daemon cannot be reached because it always runs on node 1.

Example of error:2012- 12- 27T18: 10: 37- 06: 00 <3. 3> cl ust er 1- 8( i d8) i si _mi gr at e[ 11771] : coor d[ pol i cy1] : si q_cr eat e_al er t : t ype: 11 ( pol i cy name: pol i cy1 t ar get : cl ust er 1. company. com) SyncI Q er r or connect i ng t o daemon ( bandwi dt h, t hr ot t l e, pwor ker ) . Pl ease ver i f y al l SyncI Q daemons ar e r unni ng. Unabl e t o connect t o t hr ot t l e host f or l ast 1080 seconds

Go to Page 13

______________________________________________________________________________






______________________________________________

SyncIQ error connecting to daemon (2)

Page 13


- Page 12 - SyncIQ error connecting to daemon

Check whether the policy has a source pool restriction by running the following command on the source cluster, where <pol i cy- name> is

the name of the policy.See Appendix D for example output.

i si sync pol i ci es vi ew <pol i cy- name>

In the output, note whether a Source Subnet and Source Pool are listed. These identify the source pool restriction.

Go to Page 14

Does the policy have a source pool

restriction?

No

Go to Page 15Yes

__________






______________________________________________


Page 14


- Page 13 - SyncIQ error connecting to daemon (2)

Does the policy have bandwidth/throttle rules

configured?

Note the page number that you are currently on.

Upload log files and contact Isilon Technical Support, as instructed in

Appendix A.

No

Yes

Use the OneFS web administration interface to add a source pool restriction as follows:

1. Click Data Protection > SyncIQ > Policies.2. For the policy that you want to set the restriction on, click View/Edit.3. Click Edit Policy.4. In the Source Cluster section, in the Restrict Source Nodes

section, select the radio button for Run the policy only on nodes in the specified subnet and pool.

5. Select a subnet and pool from the drop-down list.6. Click Save Changes.

IMPORTANT!Make sure that the front-end network ports on all of the nodes in the source pool restriction

can see each other on the LAN.

Go to Page 15






______________________________________________


Page 15


- Page 13 - SyncIQ error connecting to daemon (2)- Page 14 - SyncIQ error connecting to daemon (3)

Run the following command to view a list of the network pools within a groupnet or subnet:

i si net wor k pool s l i st

Extract the gr oupnet : subnet value for the desired pool name from the output, for example, gr oupnet 1. subnet 3 for pool 5, and provide it as an input for the following command to check

the nodes within that pool.

i si net wor k pool s vi ew <gr oupnet _name>. <subnet _name>. <pool _name>

See Appendix E for example output of both commands.

Is node 1 listed under Ifaces?

Go to Page 16

Go to Page 17Yes

No

_________

______________________________________________________________________________






______________________________________________


Page 16



Is it acceptable for your workflow to add node 1 to the

source pool?



Appendix A.

No

Add node 1 to the source pool using the OneFS web administration interface. For instructions, see the OneFS Web

Administration Guide for your version of OneFS.

Yes

Run the following command to try to run the SyncIQ job again, where <pol i cy- name> is the name of the failed policy:

i si sync j obs st ar t <pol i cy- name>

Does the sync complete successfully

without errors?

Yes

End troubleshooting

Go to Page 17No







______________________________________________


Page 17


- Page 15 - SyncIQ error connecting to daemon (4)- Page 16 - SyncIQ error connecting to daemon (5)

On the source cluster, increase the logging level by running the following command, where <pol i cy- name> is the name of the sync policy:

i si sync pol i ci es modi f y <pol i cy- name> - - l og- l evel =t r ace

Start the policy by running the following command, where <pol i cy- name> is the name of the policy:

i si sync j obs st ar t <pol i cy- name>

Let the policy run until it fails.

Go to Page 18

______________________________________________________________________________






______________________________________________


Page 18



Upload logs and data to Isilon Technical Support as follows:

i si _gat her _i nf o - f / var / cr ash/ i si _mi gr *

On the source cluster, set the logging level back to normal by running the following command, where <pol i cy- name> is the name of the policy:

i si sync pol i ci es modi f y <pol i cy- name> - - l og- l evel =not i ce

CAUTION!This step is very important. Failure to do this may cause boot flash drives to degrade prematurely.



Appendix A.






______________________________________________

Appendix A: If you need further assistance

Contact Isilon Technical SupportIf you need to contact Isilon Technical Support during troubleshooting, reference the page or step that you need help with. This information and the log file will help Isilon Technical Support staff resolve your case more quickly.

Upload node log files and the screen log file to Isilon Technical Support1. When troubleshooting is complete, type exi t to end your screen session.

2. Gather and upload the node log set and include the SSH screen log file by using the command appropriate for your method of uploading files. If you are not sure which method to use, use FTP.

ESRS: i si _gat her _i nf o - - esr s - - l ocal - onl y - f / i f s/ dat a/ I si l on_Suppor t / scr eenl og. 0

FTP: i si _gat her _i nf o - - f t p - - l ocal - onl y - f / i f s/ dat a/ I si l on_Suppor t / scr eenl og. 0

HTTP: i si _gat her _i nf o - - ht t p - - l ocal - onl y - f / i f s/ dat a/ I si l on_Suppor t / scr eenl og. 0

SupportIQ:Copy and paste the following command.Note: When you copy and paste the command into the command-line interface, it will appear on multiple lines (exactly as it appears on the page), but when you press Enter, the command will run as it should.

i si _gat her _i nf o - - l ocal - onl y - f / i f s/ dat a/ I si l on_Suppor t / scr eenl og. 0 - - noupl oad \- - syml i nk / var / cr ash/ Suppor t I Q/ upl oad/ f t p

3. If you receive a message that the upload was unsuccessful, refer to article 304567 for directions on how to upload files over FTP.

____________




https://support.emc.com/kb/304567




______________________________________________

Decision diamondYes No

Process stepProcess step with command:

command xyz

Go to Page #

Page # Note

Provides context and additional information. Sometimes a note is linked to a process step with a colored dot.

CAUTION!Caution boxes warn that a particular step needs to be performed with great care, to prevent serious consequences.

End pointDocument ShapeCalls out supporting documentation for a process step. When possible, these shapes contain links to the reference document.Sometimes linked to a process step with a colored dot.

Optional process step

IntroductionDescribes what the section helps you to accomplish.

Appendix B: How to use this flowchart


- Page 4 - Start Troubleshooting

Directional arrows indicate the path through

the process flow.






______________________________________________

Appendix C: Example output


- Page 4 - Start Troubleshooting

Example outputcl ust er - 1# i si sync r epor t s l i st - - pol i cy- name=pol i cy1 - - sor t j ob_i dPol i cy Name Job I D St ar t Ti me End Ti me Act i on St at e - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -pol i cy1 1 2016- 02- 02T11: 06: 30 2016- 02- 02T11: 06: 38 r un f i ni shedpol i cy1 2 2016- 02- 02T11: 06: 44 2016- 02- 02T11: 06: 53 r un f i ni shedpol i cy1 3 2016- 02- 02T11: 18: 37 2016- 02- 02T11: 18: 41 r un f ai l ed

Example outputcl ust er - 1# i si sync r epor t s vi ew - - pol i cy=dot t est 1 | l ess Pol i cy Name: dot t est Job I D: 1 St ar t Ti me: 2016- 02- 02T17: 27: 55 End Ti me: 2016- 02- 02T17: 33: 10 Act i on: r un St at e: f ai l ed I D: 1- dot t est Pol i cy I D: a12345678b901c23456abc78912d34dc Sync Type: i nval i d Dur at i on: 5m12s Er r or s: No node on sour ce cl ust er was abl e t o connect t o t ar get cl ust er . , Sour ce node coul d not connect t o t ar get cl ust er .<truncated>






______________________________________________

Appendix D: Example output



A source pool restriction can be found by running the following commands. The bold lines in the example output identify the

restrictions. In these examples, the restrictions are subnet 1: pool 0.

Example outputcl ust er - 1# i si sync pol i es vi ew pol i cy1 I D: 1234567891234a5a67890f 1234ba8cab Name: pol i cy1 Pat h: / i f s/ dat a/ backup Act i on: sync Enabl ed: No Tar get : cl ust er . company. com Descr i pt i on: Check I nt egr i t y: YesSour ce I ncl ude Di r ect or i es: -Sour ce Excl ude Di r ect or i es: - Sour ce Subnet : subnet 1 Sour ce Pool : pool 0<truncated>






______________________________________________

Appendix E: Example output



Example outputi si net wor k pool s l i st gr oupnet 1. subnet 3

I D SC Zone Al l ocat i on Met hod- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -gr oupnet 1. subnet 3. pool 5 dat a. company. com st at i cgr oupnet 1. subnet 3. pool 7 dat a. company. com dynami c- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Example outputi si net wor k pool s vi ew gr oupnet 1. subnet 3. pool 5 to display the nodes within pool5

I D: gr oupnet 0. subnet 3. pool 5Gr oupnet : gr oupnet 1subnet : subnet 3Name: pool 5Rul es: -Access Zone: zone3Al l ocat i on Met hod: st at i cAggr egat i on Mode: l acpSC Suspended Nodes: -Descr i pt i on: -I f aces: 1: ext - 2, 2: ext - 2, 3: ext - 2I P Ranges: 203. 0. 223. 12- 203. 0. 223. 22- - - - - - - - - - -<truncated>






______________________________________________

Copyright © 2019 Dell Inc. or its subsidiaries. All rights reserved.

Dell believes the information in this publication is accurate as of its publication date. The information is subject to change without notice.

THE INFORMATION IN THIS PUBLICATION IS PROVIDED "AS-IS." DELL MAKES NO REPRESENTATIONS OR WARRANTIES OF ANY KIND WITH RESPECT TO THE INFORMATION IN THIS PUBLICATION, AND SPECIFICALLY DISCLAIMS IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. USE, COPYING, AND DISTRIBUTION OF ANY DELL SOFTWARE DESCRIBED IN THIS PUBLICATION REQUIRES AN APPLICABLE SOFTWARE LICENSE.

Dell, EMC, and other trademarks are trademarks of Dell Inc. or its subsidiaries. Other trademarks may be the property of their respective owners.

EMC CorporationHopkinton, Massachusetts 01748-91031-508-435-1000 in North America 1-866-464-7381www.EMC.com



https://www.emc.com

Documents

EMC Isilon Customer Troubleshooting Guide: SyncIQ Failures · troubleshooting and you do not have any other way to connect to the cluster, you could experience data unavailability