Upload
ivayr-farah-netto
View
475
Download
2
Tags:
Embed Size (px)
DESCRIPTION
At 8tracks.com we believe on finding the best tool for the job. In the quest of finding a great persistence solution to power our "Explore" system one of our engineers stumbled upon redis.io. A few years later, we run 25 different Redis instances powering several features for our 10 million users. From Tag Browsing, to Autocomplete Search, persisting Player sessions and our News Feed. Our 4 engineers team eats the redis documentation for breakfast. Come join me in some redis love while I tell you why we're such big fans of zsets, set intersections and some other nice features that help us power 12.5 million hours of streaming per month.
Citation preview
redis love at 8tracks
@nettofarah
8tracks
8tracks
stats
• users: 11,351,961!
• public playlists: 1,545,586 (3,5 mi total)!
• uploaded tracks: 28,185,644
not a sysadmin talk
getting sudo
http://devopsreactions.tumblr.com/post/83704083404/getting-sudo
the right tool for the job
http://peopletakingpictureswithipads.tumblr.com/
RIGHT TOOL
http://peopletakingpictureswithipads.tumblr.com/
FOR THE JOB!
http://peopletakingpictureswithipads.tumblr.com/
engineering effort machine effort
engineering effort machine effort
engineering effort machine effort
engineering effort machine effort
When development tries to blame operations for the outage
http://devopsreactions.tumblr.com/post/84309007419/when-development-tries-to-blame-operations-for-the
R.T.F.J. = engineer <3 + machine <3
key-value store
• open source
• in memory (most of the times)
• data-structure server
data types• strings
• lists
• sets
• sorted sets
• hashes
just like that data structures class
except it is COOL!
you can turn this
into this
next next next next
head
tail
or this
next next next
things to keep in mind
• all in memory (but not restricted to)
• no joins or iterating over data (but not restricted to)
• you should be able to rebuild data
why?
• sometimes basic data structures is all you need
• some problems are hard to fit in a relational way
redis @ 8tracks = <3sidekiq
feature toggles
caching
player (moved to couchdb, then to mysql)
autocomplete (moved to SOLR, then to ElasticSearch)
tag browsing!
news feed
listening sessions
rate limiter
awesome documentation
sidekiq
what’s indie?tag
artistplaylist
1 SELECT name !! ! FROM tags !! ! WHERE name !! ! LIKE 'ind%' LIMIT 10;!!2 SELECT name !! ! FROM mixes !! ! WHERE name !! ! LIKE 'ind%' LIMIT 10;!!3 SELECT login !! ! FROM users !! ! WHERE login !! ! LIKE 'ind%' LIMIT 10;!!4 SELECT name !! FROM artists !! WHERE name !! LIKE 'ind%' LIMIT 10;!
you could do
1 (SELECT name FROM tags WHERE name LIKE 'ind%' LIMIT 10)!2 UNION ALL!3 (SELECT name FROM mixes WHERE name LIKE 'ind%' LIMIT 10);!
the problems
• cross-type search
• I don’t want 10 of each, I want the 10 most relevant ones
• can’t select different column counts
not a fair comparison
people don’t use sql databases for search. they use SOLR (or elasticsearch)
sometimes basic data structures!
is all you need
sorted sets
• score
• add, remove, or update log(n)
• elements in order, fast existence test, fast access to elements in the middle
breaking it down
indie =>
probably irrelevant
relevant
i !! ! ! ! ! ! ! in !!
! ! ! ! ! ! ! ind!! ! ! ! ! ! ! indi!! ! ! ! ! ! ! indie
our collection
The indie summer Indiana Industrial
one zset per n-gram
ind [ ][ ][ ]
indi
indie
which is the same asZADD ind, <score>,
ZADD indi, <score>,
ZADD indie, <score>,
searching
1 ind = $r.zevrange('ind', 0, 10)!!
2 indie = $r.zevrange('indie', 0, 10)!
tag browsing
how?
tags mixes
taggings
a single tag1 SELECT m.* FROM mixes m!2 INNER JOIN taggings tg!3 ON tg.mix_id = m.id!4 WHERE tg.tag_id = 28!5 LIMIT 10;!
chill
two tags1 SELECT m.* FROM mixes m!2 INNER JOIN taggings tg!3 ON tg.mix_id = m.id!4 INNER JOIN taggings tg2!5 ON tg2.mix_id = m.id!6 WHERE tg.tag_id = 28!7 AND tg2.tag_id = 12!8 LIMIT 10;! chill
acoustic
adding sort1 SELECT m.* FROM mixes m!2 INNER JOIN taggings tg!3 ON tg.mix_id = m.id!4 INNER JOIN taggings tg2!5 ON tg2.mix_id = m.id!6 WHERE tg.tag_id = 28!7 AND tg2.tag_id = 12!8 ORDER BY m.first_published_at!9 LIMIT 10;!
ZINTERSTORE
http://redis.io/commands/zinterstore
zinterstore
Time complexity: O(N*K)+O(M*log(M))
intersects two sets, then store them in a new different set
we’ll get to this
chill
acoustic
chill acoustic
+
chill acoustic
1 sets = ['chill', 'acoustic']!2 cache_key = 'chill:acoustic:cache'!3 $r.zinterstore(cache_key, sets, :weights => [1, 1])!
chill
chill acoustic
popular
1 sets = ['popular', 'chill', 'acoustic']!2 cache_key = 'popular:chill:acoustic:cache'!3 $r.zinterstore(cache_key, sets, :weights => [1, 0, 0])!
back to the BIG O thing
O(N*K)+O(M*log(M))
- N: smallest input - K: number of sorted sets - M: number of elements of the resulting set
cool stuff worth checking out
• pipelining
• lua scripting
• replication + persistence
lessons
• it consumes a LOT of memory (duuuh)
• sentinel for failure recovery
• DO NOT USE “keys” in production
• duplication is tricky
gracias!
thank you!
obrigado!