Playlists at Spotify

October 18, 2013#CassandraEU

Playlists at Spotify

Using Cassandra to store version controlled objects at large scale

Jimmy Mårdell <[email protected]>

mailto:[email protected]

mailto:[email protected]

#CassandraEUIntro

About me

• Jimmy Mårdell• Software Engineer• 3 years at Spotify

2

#CassandraEUIntro

About Spotify

• 24 million active users– 6 million paying subscribers

• 4 000 servers in 4 data centers•Over 1 billion playlists created

3

#CassandraEUIntro

Contents

4

•Why version control?•Playlists at Spotify•Cassandra data model•Lessons learned

#CassandraEUWhy version control?

What is version control?

• “Version control is the management of changes to documents” (Wikipedia)• Stand-alone (most common)– GIT, Subversion etc

• Embedded– Google Docs

5

#CassandraEUWhy version control?

Embedded usage

• Collaborative editing• Undo functionality• Performance• Business logic depends on document history

6

#CassandraEUPlaylists at Spotify

Playlists

7

#CassandraEUPlaylists at Spotify 8

#CassandraEUPlaylists at Spotify 9

Playlist challenges

•More than 1 billion playlists• >40 000 requests/second at peak•Offline mode• Concurrent changes


Playlist client-server

• Every playlist is a version controlled object• All playlists are synced on login– Fetch all new changes

10


Playlist client-server

• Local queue of playlist modifications– Clients optimistically accept changes - fast UI

•Queue flushed to server when possible– Offline changes– Fault tolerant

11


Playlist version control

12

1,4ed2...: ADD(ix=0, track=A,B,C)

2,19ca...: MOV(from=2, to=1, len=1)

3,038f...: REM(from=2, len=1)

0,ROOT

ABC

AC B

AC

Representation of a playlist in the backend

2,19ca...: MOV(from=2, to=1, len=1)2,19ca...: MOV(from=2, to=1, len=1)


Playlist branching

• Concurrent changes– Offline

13

BA


Playlist branching

• Concurrent changes– Offline

• Conflict resolution– Operational Transformation

• Clients oblivious of branches

14

B

B’

A

A’

merge

#CassandraEUCassandra data model

Cassandra data model

15


Cassandra at Spotify

• Playlist first system to use Cassandra– Now we use it a lot...

• Started with Cassandra 0.7• Using limited set of Cassandra features– No super columns– No CQL

16


Planning a data model

• Start with the queries!• Three common playlist queries– SYNC: Get all changes since a particular revision– GET: Get the most recent snapshot– APPEND: Add/move/delete tracks

17


Playlist data model

18

Row key 1,4ed2... 2,19ca... 3,038f...

spotify:user:spotify:playlist:3ZgmfR6lsnCwdffZUan8EA

parent=0,ROOTop=ADD(ix=0, track=A,B,C)

parent=1,4ed2...op=MOV(from=2, to=1, len=1)

parent=2,19caop=REM(from=2, len=1)

CF playlist_change


Playlist data model

19

Row key 1,4ed2... 2,19ca... 3,038f...


parent=0,ROOTop=ADD(ix=0, track=A,B,C)

parent=1,4ed2...op=MOV(from=2, to=1, len=1)

parent=2,19caop=REM(from=2, len=1)

CF playlist_change

Row key 1,8a20... 2,b783... 2,dd07... 3,39ef... 3,5a9c... 4,03fc...

spotify:user:yarin:playlist:4Pj4dCOEEYWDixfYyJwxEf

prnt=0,ROOTop=...

prnt=1,8a20...op=...

prnt=1,8a20...op=...

prnt=2,dd07...op=...

prnt=2,b783...op=...

prnt=2,39ef...prnt=3,5a9c...


Playlists in Cassandra

•Which revision is the latest?– Changes with no children

•Multiple heads possible!– Heads may appear anywhere within the row

20


Playlist data model

21

Row key 1,4ed2... 2,19ca... 3,038f...


prnt=0,ROOTop=...

prnt=1,4ed2...op=...

prnt=2,19caop=...

CF playlist_change

Row key 3,038f...


CF playlist_head


Playlist data model

22

Row key 1,4ed2... 2,19ca... 3,038f...


prnt=0,ROOTop=...


prnt=2,19caop=...

CF playlist_change

Row key 1,8a20... 2,b783... 2,dd07...


prnt=0,ROOTop=...

prnt=1,8a20...op=...

prnt=1,8a20...op=...

Row key 3,038f...


CF playlist_head

Row key 2,b783... 2,dd07...



Playlist data model

23

Row key 1,4ed2... 2,19ca... 3,038f...


prnt=0,ROOTop=...


prnt=2,19caop=...

CF playlist_change

Row key 1,8a20. 2,b783. 2,dd07. 3,39ef. 3,5a9c. 4,03fc.


prt=0,ROOTop=...

prnt=1,8a20op=...

prnt=1,8a20op=...

prnt=2,dd07op=...

prnt=2,b783op=...

prnt=2,39efprnt=3,5a9c

Row key 3,038f...


CF playlist_head

Row key 4,03fc...



Playlist heads

• playlist_head is a small CF– Fits in RAM

• 95% of playlist request only read from playlist_head– Most playlists are already up-to-date

24


Playlist snapshots

• playlist_change works well when syncing playlists• Not so well for fetching new playlists– Snapshot cache

25


Playlist data model

26

Row key 1,4ed2... 2,19ca... 3,038f...


prnt=0,ROOTop=...


prnt=2,19caop=...

CF playlist_change

Row key 1,8a20... 2,b783... 2,dd07...


prnt=0,ROOTop=...

prnt=1,8a20...op=...

prnt=1,8a20...op=...

Row key cache


version=3,038f...contents=A,C

CF playlist_snapshot

Row key cache


version=2,b783...contents=...


Updating playlists

• Validate change– Locate snapshot– Client may append to old version

• Update all tables– playlist_head last

27


Cassandra consistency levels

• Replication factor 3• All writes using CL_QUORUM• Reads from playlist_head – CL_QUORUM

• Reads from playlist_change and playlist_snapshot– CL_ONE but may fallback to CL_QUORUM

28

#CassandraEULessons learned

Lessons learned

29

#CassandraEULessons learned 30

•Leveled compaction

– Improved performance a lot•Compression

–Not as impressive

–CRC checks

Optimizations


Optimizations

• Trusted Linux page cache to ensure playlist_head kept in RAM– Didn’t work

• Tried Cassandra row cache– NO!

•mlock to the rescue

31


An enterprise ready solution

bash# while true; do vmtouch -m 10000000000 -l *head* & sleep 10m kill %vmtouchdone

32


No moving parts

• Flash disks are awesome• Reduced size of cluster from 60 to 30 nodes– Thanks FusionIO!

• IOPS no longer the bottleneck

33


Tombstone hell

• Noticed requests to playlist_head took several seconds– Huh?

• Every change causes a value to be deleted in playlist_head• playlist_head is essentially a queue– Well-known anti-pattern

34


Tombstone hell

•We had rows with >500,000 tombstones• Solution: major compaction– Relatively fast since playlist_head is in RAM

35


And more...

• Large rows in playlist_change– Modify version graph

• Reduce amount of requests– Group playlists by owner

Sounds interesting? We’re hiring!

36

Questions?

Playlists at Spotify - Using Cassandra to store version controlled objects

Text of Playlists at Spotify - Using Cassandra to store version controlled objects

Spotify presentations

Success Stories Spotify - storage.googleapis.comSpotify turned up the volume just days later with Snap Ads in Snapchat’s “NYE Live” Story to promote their “Party” playlists

Why Do You Need Spotify? · Spotify Spotify Ltd. > (382) Reviews Hide Related Q Search Browse Discover Radio Playlists Inbox Friends Settings Daft Punk Get Lucky - Radio Edit . MYHoney:):)

Personalized Playlists at Spotify

Gorillaz - Plastic Beach Album Review | Share Spotify Playlists at Share My Play Lists

LATIN PLAYLISTS - TouchTunes...LATIN PLAYLISTS LATINPLAYLISTEN V.052016 A variety of Latin themed playlists are made available to offer easy access to the music from the hottest artists

Spotify - youthteachingadults.ca · Spotify is an app that lets you listen to music for free. Spotify. Spotify Find the Spotify app Ask your adult learner to find the Spotify app

Getting Your Songs on Spotify Playlists€¦ · Though these kinds of playlists DO feature music from unsigned artists, you need to work your way there one step at a time, lay the

Playlists: October – December 2008

Party Playlists.docx · Web viewHere are our Spotify playlists from previous dance parties: Spanish Family Dance Party Animal Family Dance Party Hat Day Family Dance Party Silly Family

Recommending Music Curators: A Neural Style-Aware Approachtracks (e.g., 40 million on Apple Music and 30 million on Spotify), playlists have become an essential feature of many music

Spotify Guest

All Playlists

Spotify Apps

Instructional Strategies Playlists FINAL - TeacherPageInstructional Strategies Playlists Instructional Strategies Playlists for Teachers movement and discourse playlist rehearsal and

Top 10 wedding songs on spotify playlists in australia

Dayton Audio Last Revised: 11/2/2016 · 11/2/2016 · China, create playlists that you can share with your friends. Spotify is a digital music service that gives you access to millions

Getting Your Songs on Spotify Playlists - CD Baby · 2021. 7. 19. · Collaborative playlists Anyone on Spotify can opt to create a playlist that’s “collaborative,” meaning

SUGGESTED PLAYLISTS & MUSIC LIBRARY

Hecuba2: Cassandra Operations Made Easy (Radovan Zvoncek, Spotify) | C* Summit 2016

2,451 ROADIES · road trip by the numbers. 3. time. zones. 8 . states. 2,451. miles traveled. 5 . roadies. 232 . stops. minutes spent on the road. songs on our spotify . playlists

A Selection of Playlists for March - Spotify

Agile spotify

Spotify Perspectec Rating...Spotify An Attractively Priced Target: CLTV Growth Driven by Hardware and Playlists For Perspectec Subscribers Only 1 Rating BUY 12-Month Price Target $185

THE #1 JUKEBOX MOBILE APP - touchtunes.com · regular hangouts. Playlists Made Easy Manage you playlists in one place and import them into the TouchTunes app. Import playlists from

Cassandra Exports as a Trivially Parallelizable Problem (Emilio Del Tessandoro, Spotify) | Cassandra Summit 2016

Spotify Places

A Hierarchical Self-Attentive Model for Recommending User ...people.tamu.edu/~jwang713/pubs/AttList-CIKM2019.pdfExamples include music playlists on Spotify, collections of ... neural

The White Stripes - Under Great White Northern Lights Album Review | Share Spotify Playlists at Share My Play Lists

Technology

Playlists at Spotify - Using Cassandra to store version controlled objects

Text of Playlists at Spotify - Using Cassandra to store version controlled objects