Upload
jkni
View
1.716
Download
0
Embed Size (px)
Citation preview
Testing Cassandra Guarantees under Diverse Failure Modes with JepsenJoel Knighton
@joelknighton
DataStax
#CassandraSummit
Who I am
Mathematician
Software hobbyist
Logic enthusiast
Former DataStax Intern
DataStax Cassandra Developer
What I Do
Deconstruct
Formalize
Communicate
Prove
Automate
How We Test #1
Unit Testsant test
in-tree
How We Test #2
Distributed Testsnosetests
On GitHub – available at riptano/cassandra-dtest
Why You’re Here
JepsenKyle Kingsbury (aphyr)https://aphyr.com/tags/jepsen
What Jepsen Is
A blog series about distributed systems behavior
A talk series about distributed systems behavior
A Clojure library to test the behavior of distributed systems
A collection of tests written using those libraries
What We Hope
Jepsen
💘Cassandra
What I Did
Jepsen Testslein test
On GitHub – available at riptano/jepsen
A Test Incarnate
{:name …
:os …
:db …
:client …
:generator …
:conductors {:nemesis …}
:checker …}
names the results
prepares the os
configures/starts/stops the db
interacts with the db
instructions on how to interact
interacts with the environment
looks at and assesses test run
What You Need
One machine to run the tests
+
n machines to run Cassandra
How A Test Runs
lein testos
n1
n2
n3
n4
n5
How A Test Runs
lein testdb
n1
n2
n3
n4
n5
How A Test Runs
lein testclient 1client 2client 3client 4client 5nemesis
n1
n2
n3
n4
n5
readwrite 3
start nemesiswrite 4
readstop nemesis
write 1cas 2 -> 3
…
How A Test Runs
lein testchecker
1 – read2 – write 3 1 – read 0n – start nemesis2 – write timed-out3 – write 4n – started nemesis3 – wrote 44 – read4 – read 4n – stop nemesis0 – write 11 – cas 2 -> 3n – stopped nemesis…
valid?
Latency
Single Test Deep-Dive
lein test :only
cassandra.collections.set-test/
cql-set-isolate-node-decommission
Single Test Name
Test name used to label folder where test results, logs, and history will be stored with timestamp
cassandra cql set isolate node decommission
Single Test Nodes
[:n1 :n2 :n3 :n4 :n5]
Single Test Net
net/iptables
(drop! ;use iptables to drop packets)
(heal! ;flush iptables)
Single Test OS
debian/os(setup! ;adjust hostfile
;update package manager;install base packages like curl, iptables, etc.
;make sure network is healed)(teardown!)
Single Test DB
cassandra.core/db(setup! ;shutdown and wipe Cassandra if running
;install, configure, and start Cassandra)(teardown! ;shutdown and wipe Cassandra)
(log-files ;return path to log files)
Single Test Client
cql-set-client(setup! ;driver connect to all nodes
;create schema)(invoke! ;add? Run CQL to add to set, handle errors
;read? Read value of CQL set, handle errors)(teardown! ;disconnect driver)
Single Test Generator
(gen/phases
(->> (adds)
(gen/stagger 1/10)
(gen/delay 1/2)
std-gen)
(read-once))
Single Test Conductors
{:nemesis (nemesis/partition-random-node)
:decommissioner (c/decommissioner)}
What a Conductor Is
It’s just a client
Single Test Checker
checker/set(check ;look at history of run
;find ok or uncertain adds
;compare these to final read
;return map with validity and
;ok, lost, unexpected, recovered)
Invariants We Test
Do CQL collections (maps, sets) merge cleanly when add-only?
Do counters merge to accurately reflect increments/decrements?
Does LWT in a single datacenter allow us linearizability?
Do materialized views converge to matching the base table?
Do batch writes eventually get applied atomically?
Failures We Consider
How does this work under a variety of network partitions?
What about with node crashes?
Even if nodes are flushing and compacting?
And when nodes are being bootstrapped?
Or decommissioned?
While clocks drift?
How We Run
Start the Docker container
Install Java driver, Cassaforte, clj-ssh, and Jepsen
Use environment variables to point to build under test
Run lein test with any desired selectors and profiles
Tunable Options
Should we make a best-effort attempt to scale test length?
Should we enable commitlog compression, the coordinator batchlog on materialized views, or hinted handoff?
Is a different compaction strategy or phi value in the failure detector appropriate for this test?
Should we install from a tagged release, a URL pointing to a tarball, or a local tarball?
Should we leave Cassandra running after the test?
What We’ve FoundIssues with counter undercounting/overcounting (#10143)
Decommission race conditions causing gossip problems (#10231)
Write durability violations when recovering commitlog (#9851)
Problems with merging of collections (#10001)
Batchlog replay failures after decommission/crash (#10068)
Incorrect asserts in counter write-path when timestamps collide
A variety of materialized view issues during development
Work We Shared
Minor Jepsen fixes/features (Jepsen PRs #58, 59, 62)
Docker images to run Jepsen tests (Docker Hub: tjake/jepsen)
Multibox Vagrant configurations to run Jepsen tests (on GitHub)
Upstream library fixes (clj-ssh PR #36)
Cassandra Jepsen tests (on GitHub)
Available on CassCI (on cassci.datastax.com)
Jepsen on CassCI
Lessons I Learned
Tests verifying invariants under failures are valuable and practical
These tests can and should be a part of regular development
Testing complex systems is hard, but there are low-hanging fruit
Jepsen provides one readily available way to accomplish this goal
Considering invariants against a recorded test run is effective
Invariants should be explicit and carefully considered in design
Thanks
Jake Luciani
DataStax
The Cassandra community
Kyle Kingsbury
QUESTIONS?TLA+ • TLC • TLAPS • Clojure
Formal Methods • Jepsen CRDTs • Cassandra • GossipConsistency Models • Alloy
Model Checking • Testing
@joelknighton#CassandraSummit