Speaker: Jimmy Mardell, Senior Software Engineer at Spotify Video: http://www.youtube.com/watch?v=NQXxKzfv7Zo&list=PLqcm6qE9lgKLoYaakl3YwIWP4hmGsHm5e&index=17 All systems at Spotify have to deal with huge amounts of data. Playlists in particular is a unique challenge. We need to store more than one billion playlists, and make them accessible for not only the playlist owner but also subscribers. Furthermore, we need to handle concurrent changes to collaborative playlists and offline scenarios. The devised solution treats every playlist as a versioned object. We use Cassandra to store these objects in an efficient way, allowing fast read- and write queries. The road there was not pain free however. I will talk about the data model we ended up using, and lessons learned along the way.
Citation preview
1. Playlists at Spotify Using Cassandra to store version
controlled objects at large scale Jimmy Mrdell #CassandraEU October
18, 2013
2. Intro About me Jimmy Mrdell Software Engineer 3 years at
Spotify #CassandraEU 2
3. Intro About Spotify 24 million active users 6 million paying
subscribers 4 000 servers in 4 data centers Over 1 billion
playlists created #CassandraEU 3
4. #CassandraEU Intro Contents Why version control? Playlists
at Spotify Cassandra data model Lessons learned 4
5. Why version control? #CassandraEU What is version control?
Version control is the management of changes to documents
(Wikipedia) Stand-alone (most common) GIT, Subversion etc Embedded
Google Docs 5
6. Why version control? Embedded usage Collaborative editing
Undo functionality Performance Business logic depends on document
history #CassandraEU 6
7. Playlists at Spotify Playlists #CassandraEU 7
8. Playlists at Spotify #CassandraEU 8
9. Playlists at Spotify Playlist challenges More than 1 billion
playlists >40 000 requests/second at peak Offline mode
Concurrent changes #CassandraEU 9
10. Playlists at Spotify Playlist client-server Every playlist
is a version controlled object All playlists are synced on login
Fetch all new changes #CassandraEU 10
11. Playlists at Spotify Playlist client-server Local queue of
playlist modifications Clients optimistically accept changes - fast
UI Queue flushed to server when possible Offline changes Fault
tolerant #CassandraEU 11
12. #CassandraEU Playlists at Spotify 12 Playlist version
control 3,038f...: REM(from=2, len=1) A C 2,19ca...: MOV(from=2,
to=1, len=1) A C B 1,4ed2...: ADD(ix=0, track=A,B,C) A B C 0,ROOT
Representation of a playlist in the backend
13. #CassandraEU Playlists at Spotify Playlist branching
Concurrent changes Offline A B 13
14. #CassandraEU Playlists at Spotify Playlist branching merge
Concurrent changes Offline Conflict resolution Operational
Transformation Clients oblivious of branches B A A B 14
15. Cassandra data model Cassandra data model #CassandraEU
15
16. Cassandra data model Cassandra at Spotify Playlist first
system to use Cassandra Now we use it a lot... Started with
Cassandra 0.7 Using limited set of Cassandra features No super
columns No CQL #CassandraEU 16
17. Cassandra data model Planning a data model Start with the
queries! Three common playlist queries SYNC: Get all changes since
a particular revision GET: Get the most recent snapshot APPEND:
Add/move/delete tracks #CassandraEU 17
18. #CassandraEU Cassandra data model Playlist data model CF
playlist_change Row key spotify:user:spotify:playlist:
3ZgmfR6lsnCwdZUan8EA 1,4ed2... parent=0,ROOT op=ADD(ix=0,
track=A,B,C) 2,19ca... parent=1,4ed2... op=MOV(from=2, to=1, len=1)
3,038f... parent=2,19ca op=REM(from=2, len=1) 18
20. Cassandra data model Playlists in Cassandra Which revision
is the latest? Changes with no children Multiple heads possible!
Heads may appear anywhere within the row #CassandraEU 20
21. #CassandraEU Cassandra data model Playlist data model CF
playlist_change Row key spotify:user:spotify:playlist:
3ZgmfR6lsnCwdZUan8EA 1,4ed2... prnt=0,ROOT op=... CF playlist_head
2,19ca... prnt=1,4ed2... op=... 3,038f... prnt=2,19ca op=... Row
key spotify:user:spotify:playlist: 3ZgmfR6lsnCwdZUan8EA 3,038f...
21
24. Cassandra data model Playlist heads playlist_head is a
small CF Fits in RAM 95% of playlist request only read from
playlist_head Most playlists are already up-to-date #CassandraEU
24
25. Cassandra data model Playlist snapshots playlist_change
works well when syncing playlists Not so well for fetching new
playlists Snapshot cache #CassandraEU 25
27. Cassandra data model Updating playlists Validate change
Locate snapshot Client may append to old version Update all tables
playlist_head last #CassandraEU 27
28. Cassandra data model Cassandra consistency levels
Replication factor 3 All writes using CL_QUORUM Reads from
playlist_head CL_QUORUM Reads from playlist_change and
playlist_snapshot CL_ONE but may fallback to CL_QUORUM #CassandraEU
28
30. Lessons learned Optimizations Leveled compaction Improved
performance a lot Compression Not as impressive CRC checks
#CassandraEU 30
31. Lessons learned Optimizations Trusted Linux page cache to
ensure playlist_head kept in RAM Didnt work Tried Cassandra row
cache NO! mlock to the rescue #CassandraEU 31
32. Lessons learned #CassandraEU An enterprise ready solution
bash# while true; do vmtouch -m 10000000000 -l *head* & sleep
10m kill %vmtouch done 32
33. Lessons learned No moving parts Flash disks are awesome
Reduced size of cluster from 60 to 30 nodes Thanks FusionIO! IOPS
no longer the bottleneck #CassandraEU 33
34. Lessons learned Tombstone hell Noticed requests to
playlist_head took several seconds Huh? Every change causes a value
to be deleted in playlist_head playlist_head is essentially a queue
Well-known anti-pattern #CassandraEU 34
35. Lessons learned Tombstone hell We had rows with >500,000
tombstones Solution: major compaction Relatively fast since
playlist_head is in RAM #CassandraEU 35
36. Lessons learned And more... Large rows in playlist_change
Modify version graph Reduce amount of requests Group playlists by
owner Sounds interesting? Were hiring! #CassandraEU 36
