EMC Isilon Customer Troubleshooting Guide: Troubleshoot a ......How do you keep time? Page 8 Note Follow all of these steps, in order, until you reach a resolution. 1. Follow these

1 - EMC Isilon Customer Troubleshooting Guide: Troubleshoot a Time Drift on Your

Isilon Cluster


Isilon Cluster

For links to all Isilon customer troubleshooting guides, visit the Customer Troubleshooting - Isilon Info Hub.

We appreciate your help in improving this document. Submit your feedback at http://bit.ly/isilon-docfeedback.

______________________________________________________

______________________________________

Abstract

This troubleshooting guide helps you address time skews, or drifts, on your Isilon cluster.

January 18, 2019

EMC ISILON CUSTOMER TROUBLESHOOTING GUIDE

TROUBLESHOOT A TIME DRIFT ON YOUR ISILON CLUSTER

OneFS 7.1.0 - 8.1.0


Isilon Cluster


Isilon Cluster



______________________________________________________

______________________________________

Contents and overview

Before you begin

Page 3

Before you begin

Page 3

Appendix A

If you need further assistance

Appendix A

If you need further assistance

Start troubleshooting

Page 4


Page 4

Set the time on a new node

Page 5


Page 5

Determine which nodes are chimer nodes

Page 7


Page 7

How do you keep time?

Page 8


Page 8

Note Follow all of these steps, in order, until you reach a resolution.

1. Follow these

steps.

2. Perform

troubleshooting

steps in order.

3. Appendixes

Appendix B

How to use this flowchart

Appendix B

How to use this flowchart

NTP

Page 9

NTP

Page 9

SMB Time

Page 29

SMB Time

Page 29

Appendix C

Peering table

Appendix C

Peering table


Isilon Cluster


Isilon Cluster



______________________________________________________

______________________________________

Configure screen logging through SSH

We recommend that you configure screen logging to log all session input and output during your troubleshooting session.

This log file can be shared with Isilon Technical Support, if you require assistance at any point during troubleshooting.

Note: The screen session capability does not work in OneFS 7.1.0.6 and 7.1.1.2. If you are running either of these versions,

you can configure logging by using your local SSH client's logging feature.

1. Open an SSH connection to the cluster and log in by using the root account.

Note: If the cluster is in compliance mode, use the compadmin account to log in. All compadmin commands must be

preceded by the sudo prefix.

2. Change the directory to /ifs/data/Isilon_Support by running the following command:

cd /ifs/data/Isilon_Support

3. Run the following command to capture all input and output from the session:

screen -L

This will create a file named screenlog.0 that will be appended to during your session.

4. Perform troubleshooting.

Before you begin

CAUTION!If the node, subnet, or pool that you are working on goes down during the course of

troubleshooting and you do not have any other way to connect to the cluster, you could

experience data unavailability.

Therefore, make sure that you have more than one way to connect to the cluster before

you start this troubleshooting process. The best method is to have a serial console

connection available. This way, if you are unable to connect through the network, you

will still be able to connect to the cluster physically.

For specific requirements and instructions for making a physical connection to the

cluster, see article 304071 on the EMC Online Support site.

Before you begin troubleshooting, confirm that you can connect through either another

subnet or pool, or that you have physical access to the cluster.

CAUTION!If the node, subnet, or pool that you are working on goes down during the course of

troubleshooting and you do not have any other way to connect to the cluster, you could

experience data unavailability.

Therefore, make sure that you have more than one way to connect to the cluster before

you start this troubleshooting process. The best method is to have a serial console

connection available. This way, if you are unable to connect through the network, you

will still be able to connect to the cluster physically.

For specific requirements and instructions for making a physical connection to the

cluster, see article 304071 on the EMC Online Support site.

Before you begin troubleshooting, confirm that you can connect through either another

subnet or pool, or that you have physical access to the cluster.

https://support.emc.com/kb/304071



Isilon Cluster


Isilon Cluster



______________________________________________________

______________________________________


Is the following error

present in the log files:

frequency error -500 PPM

exceeds tolerance

500 PPM"?

Do you have a new

node that is showing

the wrong time?

Start

IntroductionStart troubleshooting here. For an overview

of the conventions used in this flowchart, see

Appendix B: How to use this flowchart.

IntroductionStart troubleshooting here. For an overview

of the conventions used in this flowchart, see

Appendix B: How to use this flowchart.

If you have not done so already, log in to

the cluster and configure screen logging

through SSH, as described on page 3.

If you have not done so already, log in to

the cluster and configure screen logging

through SSH, as described on page 3.

Go to Page 5Go to Page 5Yes

No

Refer toFrequency error -500 PPM

exceeds tolerance 500

PPM" error, article 471912

Refer toFrequency error -500 PPM

exceeds tolerance 500

PPM" error, article 471912

Go to Page 8Go to Page 8

No

Yes




Isilon Cluster


Isilon Cluster



______________________________________________________

______________________________________


Reset the time on the recently added node by running the following command, where <LNN> is the logical node number of

the recently added node, and <timeIP> is the domain name or IP address of one of your external time sourcess:

isi_for_array -n <LNN> "killall ntpd && cat /dev/null > /var/crash/ntp.drift && ntpdate -ub <timeIP>"

For example:

isi_for_array -n 7 "killall ntpd && cat /dev/null > /var/crash/ntp.drift && ntpdate -ub 10.11.120.13"

Page

5

You could have arrived here from:

• Page 4 - Start troubleshooting



Wait at least five minutes.

Refresh the time on the cluster by running the following command

with the same variables as the previous step:

isi_for_array -n <LNN> "ntpdate -ub <timeIP>"


Note Resetting the time on the cluster system clock helps to verify whether the

NTP internal clock can build a meaningful, stable drift value. The drift

value is how much the clock drifts per unit (parts per million) compared to

the reference clock. It takes at least 24 hours for a drift value to build, and

it usually takes about three days for a drift value to settle into a steady

state where the value changes by only very small amounts hour to hour.


Isilon Cluster


Isilon Cluster



______________________________________________________

______________________________________

Set the time on a new node (2)

Page

6


• Page 5 - Set the time on a new node


• Page 5 - Set the time on a new node

Validate that the recently added node is using a chimer node as its source of time by running the following

command, where <LNN> is the logical node number of the recently added node:

isi_for_array -n <LNN> "ntpq -np"

The output will appear similar to the following, the asterisk (*) indicates the "peer", or source of time, for that node:

cluster-1: remote refid st t when poll reach delay offset jitter

cluster-1: ==============================================================================

cluster-1: +10.11.12.13 10.12.123.13 3 u 889 1024 377 0.098 -2.139 0.013

cluster-1: *192.11.12.13 10.12.123.13 3 u 901 1024 377 0.112 -2.559 0.015

cluster-1: +10.11.12.14 10.12.123.13 3 u 890 1024 377 0.089 -2.652 0.010



Isilon Cluster


Isilon Cluster



______________________________________________________

______________________________________


Page

7


• Page 10 - NTP - External time sources

• Page 18 - Validate time on the non-chimer nodes (2)




By default, a cluster has three chimer nodes. Chimer nodes are the three lowest logical node numbers (LNNs) in the

cluster. If a chimer node goes offline, the node with the next-lowest LNN takes its place.

In a 3-node cluster, there is only one chimer node, the one with the lowest LNN.

In a large cluster with multiple external time sources, there might be only two chimer nodes (if configured as such).

You can determine which nodes are the chimer nodes by following these steps.

Run the following command:

isi_for_array -sQX "grep -B1 server /etc/ntp.conf" | awk '/These are the cluster/,/^$/'

The output will display the IP addresses of the chimer nodes and will appear similar to the following:

# NTP servers. These are the cluster's chimer nodes:

server 10.12.13.100 iburst true



Run the following command to display the node IP addresses and their corresponding LNNs:

isi_nodes "%{internal} %{name}"

The output will appear similar to the following:

10.12.13.101 cluster-1

10.12.13.102 cluster-2

10.12.13.103 cluster-3

10.12.13.104 cluster-4

10.12.13.105 cluster-5

Using the output from the previous steps, determine the IP

addresses of the chimer nodes and note the LNN for each. In

the example above, the chimer nodes are LNNs 1, 2, and 4.


______________________________________________________________

____________________________________________________________________________________

____________________________________________________________________________________

____________________________________________________________________________________

____________________________________________________________________________________


Isilon Cluster


Isilon Cluster



______________________________________________________

______________________________________


Page

8





Which method are

you using to sync time

to the cluster: NTP or

SMB time?

Note: SMB time is not available in OneFS 7.2 and later.

Dell EMC Isilon suggests that you use NTP to sync time to

the cluster. SMB time is not a reliable method for syncing

time on your Isilon cluster.

For more information, see OneFS 6.5 and later: Time on

the cluster is not synchronized with the Active Directory

domain controller, article 463674.

Note: SMB time is not available in OneFS 7.2 and later.

Dell EMC Isilon suggests that you use NTP to sync time to

the cluster. SMB time is not a reliable method for syncing

time on your Isilon cluster.

For more information, see OneFS 6.5 and later: Time on

the cluster is not synchronized with the Active Directory

domain controller, article 463674.

SMB time

Go to Page 9Go to Page 9NTP

Are you able to

switch to NTP for

syncing your cluster

time?


No

Yes Go to Page 9Go to Page 9




Isilon Cluster


Isilon Cluster



______________________________________________________

______________________________________

NTP

Page

9


• Page 8 - How do you keep time?



Configure NTP by following the

instructions in this article:

OneFS 6.5 and later: Time on the

cluster is not synchronized with the

Active Directory domain controller,

article 463674.

Configure NTP by following the

instructions in this article:

OneFS 6.5 and later: Time on the

cluster is not synchronized with the

Active Directory domain controller,

article 463674.

Note It can take 23 hours for time to be

synchronized on all nodes in the

cluster. The chimer nodes

synchronize their times to the external

time sources, and the non-chimer

nodes in the cluster synchronize their

times to the chimer nodes.

Did switching

to NTP resolve

the issue?

End troubleshootingYes

Are your external

time sources providing the

correct, current time, or an

intentionally incorrect

time?

No

Note the page number

that you are currently on.

Upload log files and contact

Isilon Technical Support, as

instructed in Appendix A.



Upload log files and contact

Isilon Technical Support, as


Intentionally

incorrect

time


Correct time




Isilon Cluster


Isilon Cluster



______________________________________________________

______________________________________

NTP - External time sources

Page

10


• Page 9 - NTP

• Page 16 - NTP - Chimer node time incorrect (5)

Identify the external time sources by running the following command:

isi_ntp_config list

The output will appear similar to the following, with all external sources listed:

NTP Server Setting:

time.isilon.com

Nodes excluded from contacting external NTP servers:

No nodes are excluded from contacting external NTP servers.

Chimers: 3

NTP Auth Keyfile:

No Keyfile is configured.

For each external time source, run the following command to obtain all of the

host IP addresses that the server name resolves to, where <externalTime> is

the server name obtained in the previous step:

host <externalTime>

For example: host time.isilon.com


time.isilon.com has address 10.11.12.20

time.isilon.com has address 10.11.12.21

Record this information as you will need to refer to it later in this guide.


____________________________________________________________________________

______________________


Isilon Cluster


Isilon Cluster



______________________________________________________

______________________________________

NTP - Compare chimer node time to known good time

Page

11


• Page 7 - Determine which nodes are

chimer nodes


• Page 7 - Determine which nodes are

chimer nodes

Open a new browser window and browse

to an authoritative, good time source, for

example: www.time.gov.

Open a new browser window and browse

to an authoritative, good time source, for

example: www.time.gov.

Check the time reported by the chimer nodes by

running the following command on the cluster, where

<LNNs> is a comma-separated list of the logical node

numbers of all of the chimer nodes:

isi_for_array -n <LNNs> date

For example:

isi_for_array - 1,2,4 date

Compare the time for each chimer node to

the authoritative, good time source, and

verify whether the chimer node time is

correct to within one second.

Is the time for each

chimer node correct to

within one second of the

authoritative, good time

source?



Yes

No

http://www.time.gov

http://www.time.gov


Isilon Cluster


Isilon Cluster



______________________________________________________

______________________________________

NTP - Chimer node time is incorrect

Page

12


• Page 11 - NTP - Compare chimer node time to known good time

• Page 15 - Chimer node time is incorrect (4)

• Page 16 - Chimer node time is incorrect (5)

Note Resetting the time on the cluster system clock helps to verify whether the

NTP internal clock can build a meaningful, stable drift value. The drift

value is how much the clock drifts per unit (parts per million) compared to

the reference clock. It takes at least 24 hours for a drift value to build, and

it usually takes about three days for a drift value to settle into a steady

state where the value changes by only very small amounts hour to hour.

Reset the time on the recently added node by running the following command, where <LNN> is the logical node number of

the recently added node, and <timeIP> is the domain name or IP address of one of your external time sourcess:

isi_for_array -n <LNN> "killall ntpd && cat /dev/null > /var/crash/ntp.drift && ntpdate -ub <timeIP>"

For example:

isi_for_array -n 7 "killall ntpd && cat /dev/null > /var/crash/ntp.drift && ntpdate -ub 10.11.120.13"

Wait at least five minutes.

Refresh the time on the cluster by running the following command

with the same variables as the previous step:

isi_for_array -n <LNN> "ntpdate -ub <timeIP>"


________________________________________________________________________________________________________

______________________________________________________________________

______________________________________________________________________


Isilon Cluster


Isilon Cluster



______________________________________________________

______________________________________

NTP - Chimer node time is incorrect (2)

Page

13


• Page 12 - NTP - Chimer node time is incorrect


• Page 12 - NTP - Chimer node time is incorrect

Run the following command, where <LNNs> is a

comma-separated list of the logical node

numbers of the chimer nodes:

isi_for_array -n <LNNs> "ntpq -np"

See Appendix C for example output.

Does .STEP.

appear in the

refid column?

This should be a transient condition.

Wait five minutes and try again.

If this reoccurs more than three times,

choose a different source of time.

Yes


No

____________________


Isilon Cluster


Isilon Cluster



______________________________________________________

______________________________________


Page

14


• Page 13 - NTP - Chimer node time is incorrect (2)



For all chimer nodes,

does an asterisk appear in

front of one of the time sources,

in either the remote or the

refid column?

No

The chimer nodes are

correct. Now we will check

the non-chimer nodes.Yes




Isilon Cluster


Isilon Cluster



______________________________________________________

______________________________________


Page

15





Is an asterisk in front

of .LOCL. in either the

remote or refid column

on any chimer node?

No

Yes




Upload log files and

contact Isilon Technical

Support, as instructed in

Appendix A.






Appendix A.

Yes

No

Is this your

second time on

this page?No

A "fudge" source is a clock that

gets its time from the system

clock or some other clock that is

not providing time from a

standards-based NTP time

reference (such as GPS, CDMA,

or a network reference time

server such as time.nist.gov).

Yes

Is your external

time source a

"fudge" source?Yes

Is this your

second time on

this page?


No

Note If this is your second time on this page, note the

page number that you are currently on. Upload

log files and contact Isilon Technical Support as







Isilon Cluster


Isilon Cluster



______________________________________________________

______________________________________


Page

16





How many

external time

sources do you

have?

Two or more

One









Note the page number that you

are currently on.

Upload log files and contact Isilon Technical

Support, as instructed in Appendix A.


are currently on.



Is this a

3-node

cluster?Yes

Do you have

exactly 3

chimers?

No

No

Yes

Reduce the chimer count to two

by running:

isi_ntp_config chimers 2


Note For chimer count, see the output of

isi_ntp_config list that you ran on page 10.

Note For chimer count, see the output of

isi_ntp_config list that you ran on page 10.


Isilon Cluster


Isilon Cluster



______________________________________________________

______________________________________

NTP - Validate time on the non-chimer nodes

Page

17


• Page 11 - NTP - Compare chimer node time to known good time


• Page 19 - NTP - Validate time on the non-chimer nodes (3)

Refresh the time on the cluster by running the following

command, where <timeSource> is the domain name or IP

address of one of the external time sources (preferably the

same one you used previously):

isi_for_array -s "ntpdate -ub <timeSource>"

Validate that the non-chimer nodes (all nodes except the chimers)

are using the chimer nodes as their source of time, by running the

following command, where <LNNs> is a comma-separated list of

the chimer node logical node numbers:

isi_for_array -sXx <LNNs> "ntpq -np && echo"

See the box on this page for example output.


Example outputcluster-1# isi_for_array -sxX 1,2,4 "ntpq -np && echo"


cluster-3: ==============================================================================

cluster-3: +10.11.12.13 10.12.111.13 3 u 889 1024 377 0.098 -2.139 0.013

cluster-3: *192.11.12.13 10.12.111.13 3 u 901 1024 377 0.112 -2.559 0.015

cluster-3: +10.11.12.14 10.12.111.13 3 u 890 1024 377 0.089 -2.652 0.010

cluster-3:


cluster-5: ==============================================================================

cluster-5: +10.11.12.13 10.12.111.13 3 u 1 64 377 0.074 0.018 0.013

cluster-5: *192.11.12.13 10.12.111.13 3 u 41 64 377 0.087 -0.580 0.005

cluster-5: +10.11.12.14 10.12.111.13 3 u 44 64 377 0.069 -0.396 0.007

cluster-5:


cluster-6: ==============================================================================

cluster-6: +10.11.12.13 10.12.111.13 3 u 39 64 377 0.072 -1.341 0.016

cluster-6: *192.11.12.13 10.12.111.13 3 u 1 64 377 0.082 -1.926 0.038

cluster-6: +10.11.12.14 10.12.111.13 3 u 26 64 377 0.111 -1.713 0.020

______________________________________________________________________________________________________

______________________________________________________________________________

______________________________________________________________________________________________


Isilon Cluster


Isilon Cluster



______________________________________________________

______________________________________

NTP - Validate time on the non-chimer nodes (2)

Page

18


• Page 6 - Set the time on a new node (2)

• Page 17 - NTP - Validate the time on the non-chimer nodes

In the remote column for each node, check whether an

asterisk (*) appears next to one of the cluster chimer

nodes (chimer nodes were identified on page 7).

In the remote column for each node, check whether an

asterisk (*) appears next to one of the cluster chimer

nodes (chimer nodes were identified on page 7).

For each

non-chimer node,

does an asterisk

appear in front of a

chimer node?



No

Yes

________________________________________________________________

______________________________________________________________________________________________


Isilon Cluster


Isilon Cluster



______________________________________________________

______________________________________


Page

19


• Page 18 - NTP - Validate time on the non-chimer

nodes (2)


nodes (7)

Reset the time on each non-chimer node that lacks an asterisk, by running the following command, where <LNNs> is a

comma-separated list of the LNNs of all the non-chimer nodes that lack an asterisk, and <chimerIP> is the IP address

of the asterisked chimer node:

isi_for_array -n <LNNs> "killall ntpd && cat /dev/null > /var/crash/ntp.drift && ntpdate -ub <chimerIP>"

Wait 20 minutes.

Validate that the non-chimer nodes are using the chimer nodes as

their source of time, by running the following command, where

<LNNs> is a comma-separated list of the chimer node logical node

numbers (this command excludes the chimer nodes from the output):

isi_for_array -sXx <LNNs> "ntpq -np && echo"

See page 17 for example output.

In the remote column for each node, check whether an asterisk (*)

appears next to one of the cluster chimer nodes (chimer nodes are

identified on page 7).

In the remote column for each node, check whether an asterisk (*)

appears next to one of the cluster chimer nodes (chimer nodes are

identified on page 7).

For each

non-chimer node,

does an asterisk appear

in front of a chimer node

in the remote

column?






Appendix A.






Appendix A.

Go to Page 20Go to Page 20YesNo

____________

_______________________________________

________

_______________________________________

________

_______________________________________

________

_______________________________________

________


Isilon Cluster


Isilon Cluster



______________________________________________________

______________________________________


Page

20



nodes (2)


nodes (3)

Refresh the time on the cluster by running the following

command, where <timeSource> is the domain name or IP

address of one of the external time sources (preferably the

same one you used previously):

isi_for_array -s "ntpdate -ub <timeSource>"

Wait 24 hours.

Compare the time on each node to the time on the external time

source by running the following command, where <timeSource>

is the domain name or IP address of one of the external time

sources (preferably the same one you used previously):

isi _for_array -s "ntpdate -qub <timeSource>"

See the note box on page 21 for example output.


____________

_______________________________________

________

_______________________________________

________

_______________________________________

________

_______________________________________

________


Isilon Cluster


Isilon Cluster



______________________________________________________

______________________________________


Page

21





Your output should appear similar to the example

output in the box at the bottom of this page.

Check the offset for each node. It should be

within ± 0.500 seconds.

Is the

offset within ±

0.500 seconds for

each node?


Yes

Go to Page 23Go to Page 23No

Example outputcluster-1# isi_for_array -s "ntpdate -qub 10.11.12.13"

cluster-1: server 10.11.12.13, stratum 2, offset 0.001176, delay 0.10016

cluster-1: 17 Sep 15:19:45 ntpdate[22860]: step time server 10.254.140.21 offset 0.001176 sec





cluster-5: server 10.11.12.13, stratum 2, offset -0.002581, delay 0.10712

cluster-5: 17 Sep 15:19:47 ntpdate[62312]: step time server 10.254.140.21 offset -0.002581 sec

cluster-6: server 10.11.12.13, stratum 2, offset -0.001351, delay 0.10710

cluster-6: 17 Sep 15:19:48 ntpdate[3141]: step time server 10.254.140.21 offset -0.001351 sec

Note If this is your second time on this

page, note the page number that you

are currently on. Upload log files and

contact Isilon Technical Support as


Note If this is your second time on this

page, note the page number that you

are currently on. Upload log files and

contact Isilon Technical Support as


______________________________________________________________________________________________

______________________________________________________________________________________________

______________________________________________________________________________________________


Isilon Cluster


Isilon Cluster



______________________________________________________

______________________________________


Page

22





Wait 48 hours.

Run the following command, where <timeSource> is the

domain name or IP address of one of the external time

sources (preferably the same one you used previously):

isi_for_array -s "ntpdate -qub <timeSource>"


Note If this is your second time on

this page, note the page

number that you are currently

on. Upload log files and contact

Isilon Technical Support as


Note If this is your second time on

this page, note the page

number that you are currently

on. Upload log files and contact

Isilon Technical Support as


Check the offset for each node. It

should be within ± 0.500 seconds.

Is the

offset within ±

0.500 seconds for

each node?

End troubleshooting

Yes

No Go to Page 23Go to Page 23

____________

____________________________________________________________________________________________________________________________________________________________________________________________

______________________________________________________________________________________________


Isilon Cluster


Isilon Cluster



______________________________________________________

______________________________________


Page

23





Check the peer status of the problem node to determine whether an asterisk appears in front of a chimer

node, by running the following command, where <LNNs> is a comma-separated list of the LNNs of all the

nodes that did not have an offset within ± 0.500 seconds:

isi_for_array -n <LNNs> "ntpq -np"

The output will appear similar to this:


cluster-3: ==============================================================================

cluster-3: +10.11.12.13 10.12.111.13 3 u 889 1024 377 0.098 -2.139 0.013

cluster-3: *192.11.12.111 10.12.111.13 3 u 901 1024 377 0.112 -2.559 0.015

cluster-3: +10.11.12.14 10.12.111.13 3 u 890 1024 377 0.089 -2.652 0.010

For each node,

does an asterisk appear in

front of a chimer node in the

remote column?



Yes

No

______________________________________________________________________________________________

______________________________________________________________________________________________

________________________________________________________________________________________________


Isilon Cluster


Isilon Cluster



______________________________________________________

______________________________________


Page

24



nodes (7)



nodes (7)

Check the frequency value (also know as "drift") for each node. Run the following command, where <LNNs> is a

comma-separated list of the LNNs of all the nodes that did not have an offset within ± 0.500 seconds:

isi_for_array -n <LNNs> "ntpq -c rv"


cluster-3: assID=0 status=0664 leap_none, sync_ntp, 6 events, event_peer/strat_chg,

cluster-3: version="ntpd 4.2.4p4-o Thu Jan 30 04:27:18 UTC 2014 (1)",

cluster-3: processor="amd64", system="Isilon OneFS/v7.1.0.1", leap=00, stratum=3,

cluster-3: precision=-22, rootdelay=74.681, rootdispersion=17.897, peer=60382,

cluster-3: refid=10.12.111.13,

cluster-3: reftime=d7b25e37.038aafea Wed, Sep 3 2014 20:42:47.013, poll=6,

cluster-3: clock=d7b25ea2.4d449a13 Wed, Sep 3 2014 20:44:34.301, state=4,

cluster-3: offset=-0.326, frequency=26.973, jitter=0.061, noise=0.052,

cluster-3: stability=0.008, tai=0

The frequency value for each node should NOT be equal to

exactly 500 or -500 (it should be between -500 and 500). Go

down one of these branches for each node:

For each node that is equal to 500 or -500, go to page 25.

For each node that is not equal to 500 or -500, go to page 27.

CAUTION!Remember to go

down the proper

branch for each

node. You might

need to go down

different branches

for different nodes.

Go to Page 27Go to Page 27 Go to Page 25Go to Page 25

Not equal to

500 or -500

Equal to

500 or -500

____________

____________


Isilon Cluster


Isilon Cluster



______________________________________________________

______________________________________


Page

25



nodes (8)



nodes (8)

Reset the time on the nodes whose frequency is equal to -500 or 500, where <LNNs> is a comma-separated list of the

LNNs of all the nodes whose frequency is equal to -500 or 500, and <chimerIP> is the IP address of a chimer node

(see page 7 for instructions on how to determine the chimer nodes):

isi_for_array -n <LNNs> "killall ntpd && cat /dev/null > /var/crash/ntp.drift && ntpdate -ub <chimerIP>"

Wait at least 5 minutes.

Refresh the time on the nodes by running the following command, where

<LNNs> is a comma-separated list of the LNNs of all the same nodes as

in the previous step (the nodes with frequency equal to -500 or 500), and

<chimerIP> is the same chimer IP address used in the previous step:

isi_for_array -n <LNNs> "ntpdate -ub <chimerIP>"

Wait 24 hours.


__________


Isilon Cluster


Isilon Cluster



______________________________________________________

______________________________________


Page

26



nodes (9)



nodes (9)

Check the offset by running the following command, where <LNNs> is a comma-

separated list of the LNNs of the same nodes used in the previous two steps (nodes

with frequency equal to -500 or 500), and <timeSource> is one of the external

time sources (preferably the same one you used previously):

isi_for_array -n <LNNs> "ntpdate -qub <timeSource>"

Is the offset within

± 0.500 seconds?


are currently on.




are currently on.




No

Yes


Isilon Cluster


Isilon Cluster



______________________________________________________

______________________________________


Page

27



nodes (8)



nodes (8)

Refresh the time on all of the nodes whose frequency is not equal to -500

or 500 by running the following command, where <LNNs> is a comma-

separated list of the LNNs of all the nodes that did not have a frequency

of 500 or -500, and <chimerIP> is the IP address of a chimer node (see

page 7 for instructions on how to determine the chimer nodes):

isi_for_array -n <LNNs> "ntpdate -ub <chimerIP>"

Wait 24 hours.

Check the offset by running the following command, where

<timeSource> is the domain name or IP address of one of the

external time sources (preferably the same one you used previously):

isi_for_array -s "ntpdate -qub <timeSource>"


Is the offset

for each node

within ± 0.500

seconds?



are currently on.




are currently on.



No

Yes

__________

____________


Isilon Cluster


Isilon Cluster



______________________________________________________

______________________________________


Page

28



nodes (11)



nodes (11)

Check the frequency value (also known as "drift") again. Run the

following command, where <LNNs> is a comma-separated list of the

LNNs of all the nodes that did not have an offset within ± 0.500 seconds:

isi_for_array -n <LNNs> "ntpq -c rv"

See the example output on page 23.

For each node, is the

frequency value equal to

exactly 500 or -500?


are currently on.




are currently on.



Yes

Go to Page 22Go to Page 22No

____________


Isilon Cluster


Isilon Cluster



______________________________________________________

______________________________________

SMB time

Page

29





IMPORTANTThis is a temporary workaround only. This

procedure synchronizes the time

temporarily, but the cluster will slide back out

of sync fairly frequently - possibly every few

days, maybe as often as every few hours.

Choose an arbitrary IP address in the 168.154.1.<xxx> range that no one is

using, where <xxx> is a number between 1 and 254. Then, ping the IP address to

confirm that no one is on the other end:

ping -c 1 169.154.1.<xxx>

The response should appear similar to the following:

ping -c 1 169.154.1.1

PING 169.154.1.1 (169.154.1.1): 56 data bytes

--- 169.154.1.1 ping statistics ---

1 packets transmitted, 0 packets received, 100.0% packet loss

Did you

receive the desired

response?

Choose a different IP

address.No


Yes


Isilon Cluster


Isilon Cluster



______________________________________________________

______________________________________

SMB time (2)

Page

30


• Page 29 - SMB time


• Page 29 - SMB time

Add the IP address to the list of NTP servers, by running

the following command, where <IP> is the IP address:

isi_ntp_config add server <IP>

Wait five minutes.

Delete the IP address from the list of NTP servers by running

the following command, where <IP> is the IP address that

you added to the list of NTP servers in the previous step:

isi_ntp_config delete server <IP>

SMB time is enabled and it will attempt to find an Active

Directory (AD) server to synchronize against.

Is the time

synchronized

correctly against an

AD server?


are currently on.




are currently on.



No

Repeat as needed for

all NTP servers that

need to be validated.

End troubleshooting

Yes


Isilon Cluster


Isilon Cluster



______________________________________________________

______________________________________

Contact Isilon Technical Support

If you need to contact Isilon Technical Support during troubleshooting, reference the page or step that you need help with.

This information and the log file will help Isilon Technical Support staff resolve your case more quickly.

Contact Isilon Technical Support

If you need to contact Isilon Technical Support during troubleshooting, reference the page or step that you need help with.

This information and the log file will help Isilon Technical Support staff resolve your case more quickly.

Appendix A: If you need further assistance

Upload node log files and the screen log file to Isilon Technical Support

1. When troubleshooting is complete, in the command-line interface, type exit to end your screen session.

2. Gather and upload the node log set and include the SSH screen log file by using the command appropriate for your

method of uploading files. If you are not sure which method to use, use FTP.

ESRS:

isi_gather_info --esrs --local-only -f /ifs/data/Isilon_Support/screenlog.0

FTP:

isi_gather_info --ftp --local-only -f /ifs/data/Isilon_Support/screenlog.0

HTTP:

isi_gather_info --http --local-only -f /ifs/data/Isilon_Support/screenlog.0

SMTP:

isi_gather_info --email --local-only -f /ifs/data/Isilon_Support/screenlog.0

SupportIQ:

Copy and paste the following command.

Note: When you copy and paste the command into the command-line interface, it will appear on multiple lines (exactly

as it appears on the page), but when you press Enter, the command will run as it should.

isi_gather_info --local-only -f /ifs/data/Isilon_Support/screenlog.0 --noupload \

--symlink /var/crash/SupportIQ/upload/ftp

3. If you receive a message that the upload was unsuccessful, refer to article 304567 on the EMC Online Support site for

directions on how to upload files over FTP.

______________________

Upload node log files and the screen log file to Isilon Technical Support

1. When troubleshooting is complete, in the command-line interface, type exit to end your screen session.

2. Gather and upload the node log set and include the SSH screen log file by using the command appropriate for your

method of uploading files. If you are not sure which method to use, use FTP.

ESRS:

isi_gather_info --esrs --local-only -f /ifs/data/Isilon_Support/screenlog.0

FTP:

isi_gather_info --ftp --local-only -f /ifs/data/Isilon_Support/screenlog.0

HTTP:

isi_gather_info --http --local-only -f /ifs/data/Isilon_Support/screenlog.0

SMTP:

isi_gather_info --email --local-only -f /ifs/data/Isilon_Support/screenlog.0

SupportIQ:

Copy and paste the following command.

Note: When you copy and paste the command into the command-line interface, it will appear on multiple lines (exactly

as it appears on the page), but when you press Enter, the command will run as it should.

isi_gather_info --local-only -f /ifs/data/Isilon_Support/screenlog.0 --noupload \

--symlink /var/crash/SupportIQ/upload/ftp

3. If you receive a message that the upload was unsuccessful, refer to article 304567 on the EMC Online Support site for

directions on how to upload files over FTP.

___________

http://www.emc.com/contact/contact-us.htm

http://www.emc.com/contact/contact-us.htm




Isilon Cluster


Isilon Cluster



______________________________________________________

______________________________________

Decision diamondYes No

Process stepProcess step with command:

command xyz

Go to Page #

Page

# Note Provides context and additional

information. Sometimes a note is linked

to a process step with a colored dot.

CAUTION!Caution boxes warn that

a particular step needs

to be performed with

great care, to prevent

serious consequences.

End point Document ShapeCalls out supporting documentation

for a process step. When possible,

these shapes contain links to the

reference document.

Sometimes linked to a process step

with a colored dot.

Optional process step

Directional arrows indicate

the path through the

process flow.

IntroductionDescribes what the section helps you to

accomplish.


• Page 4 - Start Troubleshooting


• Page 4 - Start Troubleshooting

Appendix B: How to use this flowchart


Isilon Cluster


Isilon Cluster



______________________________________________________

______________________________________





Appendix C: Peering table

Example output

The asterisk indicates the "peer", or source of time, for each node.

cluster-1# isi_for_array -n 1,2,4 "ntpq -np"


cluster-4: ==============================================================================

cluster-4: +10.11.12.13 10.12.111.13 3 u 474 1024 277 0.059 -5.573 0.041

cluster-4: 10.11.12.13 10.12.111.13 3 u 434 1024 377 0.058 -0.180 0.019

cluster-4: 127.127.1.1 .LOCL. 13 l 2 64 377 0.000 0.000 0.000

cluster-4: *10.11.120.120 10.225.225.225 2 u 471 1024 377 74.846 -0.001 0.052


cluster-1: ==============================================================================

cluster-1: *10.11.120.120 10.254.225.252 2 u 817 1024 277 13.966 0.267 0.024

cluster-1: +192.11.12.130 10.12.111.13 3 u 790 1024 377 0.068 0.499 0.063

cluster-1: +192.11.12.130 10.12.111.13 3 u 805 1024 377 0.060 0.663 0.049

cluster-1: 127.127.1.1 .LOCL. 9 l 40 64 377 0.000 0.000 0.000


cluster-2: ==============================================================================

cluster-2: +10.254.140.21 10.225.225.225 2 u 955 1024 277 75.740 -5.071 0.057

cluster-2: *192.11.12.130 10.11.120.120 3 u 834 1024 377 0.083 -0.440 0.078

cluster-2: 192.11.12.130 10.12.111.13 3 u 880 1024 377 0.061 0.165 0.007

cluster-2: 127.127.1.1 .LOCL. 11 l 36 64 377 0.000 0.000 0.000

Notes on asterisks and remote and refid columns

The asterisk (*) indicates which time source is the "peer," or source of time, for each node.

The asterisk itself is located to the left of the remote column, but your chosen time source may be in either the

remote or refid column.

The remote column identifies the time source from which the node gets its time. The refid column identifies the

time source from which the source listed in the remote column gets its time.

Every chimer node should have an asterisk in front of one of your chosen time sources, in either the remote or

refid column.

The asterisk should NOT be in front of the IP address for .LOCL. in the refid column.

In the example on this page, node 2 illustrates the asterisk in front of your external time source in the refid column.

For more information about how to read the peering table, see the next page.______________________


Isilon Cluster


Isilon Cluster



______________________________________________________

______________________________________


• Page 33 - Appendix C: Peering table


• Page 33 - Appendix C: Peering table

Appendix C: Peering table (2)

The peering table appears in the output of an ntpq -p command. The tables in this appendix describe the

various features of the peering table.

Variable Description

symbol single-character code indicating current value of the select field of the peer status word

remote host name (or IP number) of peer

refid association ID or IP of the time source of the next higher stratum

st stratum

t u: unicast, b: broadcast, l: local

when sec/min/hr since last received packet

poll poll interval (log2 s)

reach reach shift register (octal)

delay roundtrip delay in milliseconds

offset offset in milliseconds

jitter phase noise value - the change of the rate of change in the source clock

The following table describes what each column in the peering table means:

Variable Description

symbol single-character code indicating current value of the select field of the peer status word

remote host name (or IP number) of peer

refid association ID or IP of the time source of the next higher stratum

st stratum

t u: unicast, b: broadcast, l: local

when sec/min/hr since last received packet

poll poll interval (log2 s)

reach reach shift register (octal)

delay roundtrip delay in milliseconds

offset offset in milliseconds

jitter phase noise value - the change of the rate of change in the source clock

The following table describes what each column in the peering table means:

Symbol Description

spaceThe peer is discarded as unreachable, synchronized to this server (synch loop), or

exhibiting significant synchronization distance.

x The peer is discarded by the intersection algorithm as a falseticker.

.The peer is discarded as not among the first ten peers sorted by synchronization

distance and so is probably a poor candidate for further consideration.

- The peer is discarded by the clustering algorithm as an outlier.

+ The peer is a survivor and a candidate for the combining algorithm.

#The peer is a survivor, but not among the first six peers sorted by synchronization

distance. If the association is ephemeral, it may be demobilized to conserve resources.

*The peer has been declared the system peer and lends its variables to the system

variables.

o

The peer has been declared the system peer and lends its variables to the system

variables. However, the actual system synchronization is derived from a pulse -per-

second (PPS) signal, either indirectly via the PPS reference clock driver or directly via

kernel interface.

Meaning

reject

falsetick

excess

outlier

candidate

selected

peer

pps.peer

The following table describes each character that might precede a peer name in the peering table:

Symbol Description

spaceThe peer is discarded as unreachable, synchronized to this server (synch loop), or

exhibiting significant synchronization distance.

x The peer is discarded by the intersection algorithm as a falseticker.

.The peer is discarded as not among the first ten peers sorted by synchronization

distance and so is probably a poor candidate for further consideration.

- The peer is discarded by the clustering algorithm as an outlier.

+ The peer is a survivor and a candidate for the combining algorithm.

#The peer is a survivor, but not among the first six peers sorted by synchronization

distance. If the association is ephemeral, it may be demobilized to conserve resources.

*The peer has been declared the system peer and lends its variables to the system

variables.

o

The peer has been declared the system peer and lends its variables to the system

variables. However, the actual system synchronization is derived from a pulse -per-

second (PPS) signal, either indirectly via the PPS reference clock driver or directly via

kernel interface.

Meaning

reject

falsetick

excess

outlier

candidate

selected

peer

pps.peer

The following table describes each character that might precede a peer name in the peering table:


Isilon Cluster


Isilon Cluster



______________________________________________________

______________________________________

Copyright © 2019 Dell Inc. or its subsidiaries. All rights reserved.

Dell believes the information in this publication is accurate as of its publication date. The information is subject to change without notice.

THE INFORMATION IN THIS PUBLICATION IS PROVIDED “AS-IS.“ DELL MAKES NO REPRESENTATIONS OR WARRANTIES OF ANY KIND WITH RESPECT TO THE INFORMATION IN THIS PUBLICATION, AND SPECIFICALLY DISCLAIMS IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. USE, COPYING, AND DISTRIBUTION OF ANY DELL SOFTWARE DESCRIBED IN THIS PUBLICATION REQUIRES AN APPLICABLE SOFTWARE LICENSE.

Dell, EMC, and other trademarks are trademarks of Dell Inc. or its subsidiaries. Other trademarks may be the property of their respective owners.

EMC CorporationHopkinton, Massachusetts 01748-91031-508-435-1000 in North America 1-866-464-7381www.EMC.com

Copyright © 2019 Dell Inc. or its subsidiaries. All rights reserved.

Dell believes the information in this publication is accurate as of its publication date. The information is subject to change without notice.

THE INFORMATION IN THIS PUBLICATION IS PROVIDED “AS-IS.“ DELL MAKES NO REPRESENTATIONS OR WARRANTIES OF ANY KIND WITH RESPECT TO THE INFORMATION IN THIS PUBLICATION, AND SPECIFICALLY DISCLAIMS IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. USE, COPYING, AND DISTRIBUTION OF ANY DELL SOFTWARE DESCRIBED IN THIS PUBLICATION REQUIRES AN APPLICABLE SOFTWARE LICENSE.

Dell, EMC, and other trademarks are trademarks of Dell Inc. or its subsidiaries. Other trademarks may be the property of their respective owners.

EMC CorporationHopkinton, Massachusetts 01748-91031-508-435-1000 in North America 1-866-464-7381www.EMC.com

https://www.emc.com

https://www.emc.com

Documents

EMC Isilon Customer Troubleshooting Guide: Troubleshoot a ......How do you keep time? Page 8 Note Follow all of these steps, in order, until you reach a resolution. 1. Follow these