Debugging Distributed Systems - Devoxx Belgium 2016 [Extended]

Donny Nadolny, PagerDuty#Devoxx #distsys

Debugging Distributed SystemsDonny Nadolny

PagerDuty



What is ZooKeeper

• Distributed system for building distributed systems

• Small in-memory filesystem


ZooKeeper API

• create directory

• create file (ZooKeeper term: “node”)

• atomically update a file

• watch a file for changes

• create “ephemeral” file (goes away when client does)

• create sequential file (concurrent attempts to create are ordered)


ZooKeeper at PagerDuty

• Distributed locking

• Consistent, highly available


Current Talk: Debugging Distributed SystemsFor Cassandra Consistency Issues, See:









ZK 3

ZK 1 ZK 2

DC-A

DC-C

DC-B

24 ms

24 ms 3 ms

… over a WAN


ZooKeeper Overview


The Failure

• Network trouble, one follower falls behind

• ZooKeeper gets stuck - leader still up

1

2

DB

Siz

e


The Failure

• Network trouble, one follower falls behind

• ZooKeeper gets stuck - leader still up

2

DB

Siz

e

1

2

1.51


Recovery

• Restart all nodes

• Restart leader

2

DB

Siz

e

1

2

1.51

3 3


First Hint

• Leader logs: “Toobusytosnap,skipping”


Fault Injection

• Disk slow? let’s test:•sshfsdonny@some_server:/home/donny/mnt

• Similar failure profile


Fault Injection

• Disk slow? let’s test:•sshfsdonny@some_server:/home/donny/mnt

• Similar failure profile

• Re-examine disk latency… nope, was a red herring


Health Checks

• First warning: application monitoring

• High-level application checks are good because they catch many problems, but don’t tell you the cause

• Monitoring ZooKeeper: used ruok


Deep Health Checks

• Added deep health check:

• write to one ZooKeeper key

• read from ZooKeeper key


"LearnerHandler-/123.45.67.89:45874"prio=10tid=0x00000000024bb800nid=0x3d0drunnable[0x00007fe6c3193000]java.lang.Thread.State:RUNNABLEatjava.net.SocketOutputStream.socketWrite0(NativeMethod)atjava.net.SocketOutputStream.socketWrite(SocketOutputStream.java:113)…atorg.apache.jute.BinaryOutputArchive.writeBuffer(BinaryOutputArchive.java:118)…atorg.apache.jute.BinaryOutputArchive.writeRecord(BinaryOutputArchive.java:123)atorg.apache.zookeeper.server.DataTree.serializeNode(DataTree.java:1115)-locked<0x00000000d4cd9e28>(aorg.apache.zookeeper.server.DataNode)atorg.apache.zookeeper.server.DataTree.serializeNode(DataTree.java:1130)…atorg.apache.zookeeper.server.ZKDatabase.serializeSnapshot(ZKDatabase.java:467)atorg.apache.zookeeper.server.quorum.LearnerHandler.run(LearnerHandler.java:493)

The Stack Trace

1

2

3


Threads (Leader)

Request processors

Learner handler (one per follower)

Client requests


🔒🔒

Threads (Leader)

Request processors


Client requests

🔒🔓🔓


Threads (Leader)

Request processors


Client requests

🔒


Threads (Leader)

Request processors


Client requests

🔒

🔒


Threads (Leader)

Request processors


Client requests

🔒

🔒 🔒


void serializeNode(OutputArchive output, String path) { DataNode node = getNode(path); String[] children = {}; synchronized (node) { output.writeString(path, "path"); output.writeRecord(node, "node"); children = node.getChildren(); } for (String child : children) { serializeNode(output, path + "/" + child); }}

Write Snapshot Code (simplified)

Blocking network write


ZooKeeper Heartbeat

• Why didn’t a follower take over?

• restart all nodes - cluster recovers

• restart leader - cluster recovers

• ZK heartbeat: message from leader to follower

• follower gets heartbeat, everything is fine

• follower doesn’t get heartbeat: start an election


Threads (Leader)

Request processors


Client requests

🔒

🔒 🔒


Threads (Leader)

Request processors


Client requests

Quorum Peer

Followers

❤ ❤ ❤

🔒

🔒 🔒


TCP


Follower LeaderESTABLISHED ESTABLISHED

Packet 1

ACK

… SYN, SYN-ACK, ACK …

TCP Data Transmission


Follower LeaderESTABLISHED ESTABLISHED

Packet 1



ESTABLISHED ESTABLISHEDPacket 1

Packet 1~200ms


Follower Leader



Packet 1~200ms

Packet 1 ~200ms


Follower Leader



Packet 1~200ms

Packet 1 ~200ms

~400msPacket 1


Follower Leader



Packet 1~200ms

Packet 1 ~200ms

~400msPacket 1

~800msPacket 1


Follower Leader



Packet 1~200ms

Packet 1 ~200ms

~400msPacket 1

~800ms

~

120sec

Packet 1

Packet 1 120sec

CLOSED

15 retries…


Follower Leader


TCP Retransmission (Linux Defaults)

• Retransmission timeout (RTO) is based on latency

• TCP_RTO_MIN = 200 ms

• TCP_RTO_MAX = 2 minutes

• /proc/sys/net/ipv4/tcp_retries2 = 15 retries

• 0.2 + 0.2 + 0.4 + 0.8 + … + 120 = 924.8 seconds (15.5 mins)



Packet 1~200ms

Packet 1 ~200ms

~400msPacket 1

~800ms

~

120sec

Packet 1

Packet 1 120sec

CLOSED

15.5 mins (or more)

…


Follower Leader


Timeline1. Network trouble begins - packet loss / latency2. Follower falls behind, restarts, requests snapshot3. Leader begins to send snapshot4. Snapshot transfer stalls5. Follower ZooKeeper restarts, attempts to close connection 6. Network heals 7. … Leader still stuck


Timeline1. Network trouble begins - packet loss / latency2. Follower falls behind, restarts, requests snapshot3. Leader begins to send snapshot4. Snapshot transfer stalls5. Follower ZooKeeper restarts, attempts to close connection6. Network heals7. … Leader still stuck


ESTABLISHED ESTABLISHED

FIN/ACK

FIN

ACK

LAST_ACK

CLOSED

TIME_WAIT

CLOSED

60 seconds

FIN_WAIT1

TCP Close Connection

Follower Leader



CLOSED~1m40s

FIN_WAIT1 FINFINFIN

FIN

FIN

8 retries ~


Follower Leader



CLOSED~1m40s

FIN_WAIT1 FIN Packet 1

CLOSED~15.5 mins


Follower Leader



CLOSED~1m40s


CLOSEDRST


Follower Leader


06:51:47iptables:WARN:IN=eth0OUT=MAC=00:0d:12:34:56:78:12:34:56:78:12:34:56:78SRC=<leader_ip>DST=<follower_ip>LEN=54TOS=0x00PREC=0x00TTL=44ID=36370DFPROTO=TCPSPT=3888DPT=36416WINDOW=227RES=0x00ACKPSHURGP=0

syslog - Dropped Packets on Follower



CLOSED~1m40s



Blocked by iptablesX

Follower Leader

XX


iptablesiptables-AINPUT-mstate--stateESTABLISHED,RELATED-jACCEPT

iptables-AINPUT-ptcp--dport80-jACCEPT

... more rules to accept connections …

iptables-AINPUT-jDROP


iptablesiptables-AINPUT-mstate--stateESTABLISHED,RELATED-jACCEPT

iptables-AINPUT-ptcp--dport80-jACCEPT

... more rules to accept connections …

iptables-AINPUT-jDROP

But: iptables connections != netstat connections


conntrack Timeouts

• From linux/net/netfilter/nf_conntrack_proto_tcp.c:

• [TCP_CONNTRACK_LAST_ACK] = 30 SECS


Follower Leader

CLOSED

~51.2s

FIN_WAIT1 FINFINFIN

FIN

FIN~25.6s

kernel TCPconntrackLAST_ACK

30s

30s

30s

30s

CLOSED

~12.8s

30s

~81.2s~102.4s



The Full Story

• Packet loss

• Follower falls behind, requests snapshot

• (Packet loss continues) follower closes connection

• Follower conntrack forgets connection

• Leader now stuck for ~15 mins, even if network heals


(Alternative: kill the follower)

Reproducing (1/3) - Setup

• Follower falls behind:tcqdiscadddeveth0rootnetemdelay500ms100msloss35%

• Wait for a few minutes


Reproducing (2/3) - Request Snapshot

• Remove latency / packet loss:tcqdiscdeldeveth0rootnetem

• Restrict bandwidth:tcqdiscadddeveth0handle1:roothtbdefault11tcclassadddeveth0parent1:classid1:1htbrate100kbpstcclassadddeveth0parent1:1classid1:11htbrate100kbps

• Restart follower ZooKeeper process


Reproducing (3/3) - Close Connection

• Block traffic to leader:iptables-AOUTPUT-ptcp-d<leaderip>-jDROP

• Remove bandwidth restriction:tcqdiscdeldeveth0root

• Kill follower ZooKeeper process, kernel tries to close connection

• Monitor conntrack status, wait for entry to disappear, ~80 seconds:conntrack-L|grep<leaderip>

• Allow traffic to leader:iptables-DOUTPUT-ptcp-d<leaderip>-jDROP


IPsec


Follower Leader

ESP (UDP)

ESP (UDP)IPsec

TCP dataIPsec TCP data

IPsec


IPsec Phase 1

IPsec Phase 2

TCP data

IPsec - Establish Connection

Follower Leader


TCP data

IPsec - Dropped Packets

TCP data

IPsec Phase 1

IPsec Phase 2

Follower Leader


IPsec Heartbeat

IPsec - Heartbeat

TCP data

TCP data

IPsec Phase 1

IPsec Phase 2

Follower Leader


Lessons


Lesson 1

• Don’t lock and block

• TCP can block for a really long time

• Interfaces / abstract methods make analysis harder


Lesson 2

• Automate debug info collection (stack trace, heap dump, transaction logs, etc)


Lesson 3

• Application/dependency checks should be deep health checks!

• Leader/follower heartbeats should be deep health checks!


Questions?Link: “Network issues can cause cluster to hang due to near-deadlock”https://issues.apache.org/jira/browse/ZOOKEEPER-2201

https://issues.apache.org/jira/browse/ZOOKEEPER-2201


“Mess With The Network” Cheat Sheet#addlatencytcqdiscadddeveth0rootnetemdelay500ms100msloss25%

#removelatencytcqdiscdeldeveth0rootnetem

#restrictbandwidthtcqdiscadddeveth0handle1:roothtbdefault11tcclassadddeveth0parent1:classid1:1htbrate100kbpstcclassadddeveth0parent1:1classid1:11htbrate100kbps

#removebandwidthrestrictiontcqdiscdeldeveth0root#tip:whendoinglatency/loss/bandwidthrestriction:#run"sleep60&&<tcdeletecommand>&disown"incaseyoulosesshaccess

#capturepackets,thenopenlocallyinwiresharktcpdump-n"srchost123.45.67.89ordsthost123.45.67.89"-ieth0-s65535-w/tmp/packet.dump

iptables-AOUTPUT-ptcp--dport4444-jDROP#blocktrafficiptables-DOUTPUT-ptcp--dport4444-jDROP#allowtraffic#canuseINPUT/OUTPUTchainforincoming/outgoingtraffic#otheroptions:--dport<destport>,--sport<srcport>,-s<sourceip>,-d<destip>

#configuredatabase/applicationlocaldatadirectorytobe/mnt,[email protected]:/tmp/data/mnt#alternative:nbd(networkblockdevice)

netstat-peanut#networkconnections,regularkernelviewconntrack-L#networkconnections,iptablesview

Software

Debugging Distributed Systems - Devoxx Belgium 2016 [Extended]