Upload
datastax-academy
View
1.531
Download
1
Embed Size (px)
Citation preview
Cassandra Meetup
Monitoring C* Health at ScaleJason Cacciatore @jasoncac
How do we assess health ?
● Node Level - dmesg errors - gossip status - thresholds (heap, disk usage, …)
● Cluster Level – ring aggregate
Scope
● Production + Test● Hundreds of clusters● Over 9,000 nodes
Current Architecture
So… what’s the problem ?● Cron system for monitoring is
problematic○ No state, just snapshot
● Stream processor is a better fit
Mantis● A reactive stream processing system that runs on Apache Mesos
○ Cloud native○ Provides a flexible functional programming model○ Supports job autoscaling○ Deep integration into Netflix ecosystem
● Current Scale○ ~350 jobs○ 8 Million Messages/sec processed○ 250 Gb/sec data processed
● QConNY 2015 talk on Netflix Mantis
Modeled as Mantis Jobs
Resources
Real-time Dashboard
How can I try this ?
● Priam - JMXNodeTool● Stream Processor - Spark
THANK YOU
C* Gossip: the good, the bad and the ugly
Minh Do @timiblossom
What is Gossip Protocol or Gossip?● A peer-to-peer communication protocol in a
distributed system● Inspired by the form of gossip seen in
human social networks● Nodes spread out information to
whichever peers they can contact● Used in C* primarily as a membership
protocol and information sharing
Gossip flow in C*● At start, Gossiper loads seed
addresses from configuration file into the gossip list● doShadowRound on seeds● Every 1s, gossip up to 3 nodes
from the peers: a random peer, a seed peer, and unreachable peer
C* Gossip round in 3 stages
How Gossip Helps C*?● Discover cluster topology (DC, Rack)● Discover token owners● Figure out peer statuses:
○ moving○ leaving/left○ normal○ down○ bootstrapping
● Exchange Schema version● Share Load (used disk space)/Severity (CPU)● Share Release version/Net version
What Gossip does not do for C*?● Detect crashes in Thrift or Native servers
● Manage cluster (need Priam or OpsCenter)
● Collect performance metrics: latencies, RPS, JVM stats, network stats, etc.
● Give C* admins a good sleep.
Gossip race conditionMost of Gossip issues and bugs are caused by incorrect code logics in handling race conditions
● The larger the cluster the higher chance of having race condition
● There are several C* components running in different threads that can affect gossip status:
○ Gossiper○ FailureDetector○ Snitches○ StorageService○ InboundTcpConnection○ OutboundTcpConnection
Pain - Gossip Can Inflict on C*● CASSANDRA-6125 An example to show race condition● CASSANDRA-10298 Gossiper does not clean out
metadata on a dead peer properly to cause a dead peer to stay in ring forever
● CASSANDRA-10371 Dead nodes remain in gossip to prevent a replacement due to FailureDetector unable to evict a down node
● CASSANDRA-8072 Unable to gossip to any seeds● CASSANDRA-8336 Shut down issue
that peers resurrect a down node
Pain - Gossip Can Inflict on C*, cont.● CASSANDRA-8072 and CASSANDRA-7292
Problems on reusing IP of a dead node on a new node● CASSANDRA-10969 Long running cluster (over 1yr) has
restarting issue ● CASSANDRA-8768 Upgrading issue to newer version ● CASSANDRA-10321 Gossip to dead nodes caused CPU
usage to be 100%● A lemon node or AWS network issue to cause one node
not to see the other to display a confusing gossip view
What can we do?
● Rolling restart C* cluster once in awhile● On AWS, when there is an gossip issue,
try reboot. If still have bad gossip view,, replace with a new instance
● Node assassination (unsafe and need a repair/clean-up)
● Monitor network activities to take pre-emptive actions
● Search community for the issues reported in the system logs
● Fix it yourself
● Pray
THANK YOU
References● https://www.cs.cornell.edu/~asdas/research/dsn02-swim.pdf● https://wiki.apache.org/cassandra/ArchitectureGossip
Cassandra Tickler@chriskalan
When does repair fall down?
● Running LCS on an old version of C* ● Space issues● Repair gets stuck
Solution - Cassandra Tickler
Solution - Cassandra Tickler
Solution - Cassandra Tickler
https://github.com/ckalantzis/cassTickler
THANK YOU