Mongo à la Resque

Mongo à la ResqueNicolas Fouché

http:// .com http://about.me/nfo

Say “rescue”

or “resquioux” in French

Resque

One man:

Resque

One language:

Resque

One dependency:

Redis

Remote Dictionary Server

“memcached on steroids”

Initial release was March 2009

Redis

Redis is an open source, advanced key-value store. It is often referred to as a data structure server since keys can contain strings, hashes, lists, sets and sorted sets.

http://redis.io

Redis

In RAM

Latency: 10ns VS 1ms = 1,000,000ms

≃ 5000 Gets/s

≃ 5000 Sets/s

Redis

Virtual memory

keys in memory

values as needed in memory

replication

Redis

well written ANSI C

and clients in all languages

Redis

keep in mind the complexity of each operation

Resque

parent / child fork process

Resque

handles Unix signals

=> god / monit / bluepill friendly

Resque

Sinatra web UI, known as resque-web

Queues

Priority:

$ QUEUES=high,low rake resque:work

Queues

Consume all queues:

$ QUEUES=* rake resque:work

Enqueue jobs

Resque.enqueue(Archive, @repo.id, branch)

Workers

class Archive @queue = :file_serve def self.perform(repo_id, branch = nil) repo = Repository.find(repo_id) repo.create_archive(branch || "master") end end

Persistence

Jobs are stored as JSON

{ "class": "Archive", "args": [ 44, "masterbrew" ] }

Failure backends

redis (for resque-web)

hoptoad

and others (like exceptional)

Don’t like Ruby ?

Alternate implementations

https://github.com/defunkt/resque/wiki/alternate-implementations

resque-mongo

Translating queriesatomic operations

$set - set a particular value

$unset - set a particular value

$inc - increment a particular value by a certain amount

$push - append a value to an array

$pushAll - append several values to an array

$pull - remove a value(s) from an existing array

$pullAll - remove several value(s) from an existing array

$bit - bitwis

Translating queriesPush a job

redis.sadd(:queues, queue)redis.rpush("queue:#{queue}", encode(item))

mongo << {:queue => queue, :item => item, :date => Time.now}mongo_queues.update( {:queue => queue }, {'$inc' => {:count => 1}})

Translating queries

Pop a job

redis.lpop("queue:#{queue}")

doc = mongo.find_and_modify( :query => {:queue => queue}, :sort => [[:date, 1]], :remove => true) mongo_queues.update( {:queue => queue }, {'$inc' => {:count => -1}})

Translating queries

queue size

redis.llen("queue:#{queue}").to_i mongo_queues.find_one(:queue => queue)['count']

Translating queriesPeek a job

list_range("queue:#{queue}", start, count)

start, count = [start, count].map { |n| Integer(n) } res = mongo.find(:queue => queue).sort([:date, 1]).skip(start).limit(count).to_a res.collect! { |doc| doc['item'] } count == 1 ? res.first : res

Translating queries

Remove queue

redis.srem(:queues, queue) redis.del("queue:#{queue}")

mongo.remove({:queue => queue}) mongo_queues.remove({:queue => queue})

Mongo to the rescue ?

Sometimes you just know MongoDB better than Redis

Most of query features of Redis can be translated in Mongo queries

But for Redis addicts, MongoDB has some secrets

Used to MongoDB ?

Used to MongoDB ?[lol@cats ~]# bin/mongo MongoDB shell version: 1.6.4 connecting to: test > show dbs admin local monque > use monque switched to db monque > db.getCollectionNames(); [ "delayed_queue", "failures", "job_groups", "monque", "queues", "schedules", "schedules_changed", "stats", "system.indexes", "workers" ] > db.workers.count(); 40 > db.workers.findOne(); { "_id" : ObjectId("4c863c6c89cb535954000001"), "started" : "Tue Sep 07 2010 13:21:48 GMT+0000 (UTC)", "worker" : "myserver:22868:facebook*" }

Used to MongoDB ?require 'rubygems' require 'mongo' include Mongo db = Connection.new.db('sample-db') coll = db.collection('test') coll.remove 3.times do |i| coll.insert({'a' => i+1}) end puts "There are #{coll.count()} records. Here they are:"coll.find().each { |doc| puts doc.inspect }

Really used to MongoDB ?

New features

Go take a look in this JSON !

Search failed jobs

What’s happening right now ?

Use the mongo console !

Listen on queues by prefix

Queues prefix:

$ QUEUES=email*,analytics* rake resque:work

Plugins

resque-mongo-groups

resque-mongo-scheduler

resque-mongo-groups

by Catalin Bordianu, aka @omikronn

resque-mongo-groups

Resque.enqueue( EatBreakfast, :with => ['eggs', 'bacon'], :group_id => 'meals:' + Time.now.utc.strftime('%Y%m%d') )

resque-mongo-groups

class EatBreakfast extend Resque::Plugins::Groups::TrackedJob end

resque-mongo-groups

def self.perform # ... self.class.atomic_op_on_complete = { '$inc' => { 'eggs' => 1, 'bacon' => 1 } } # ... end

resque-mongo-groups

Resque.group_stats("meals:20110316") { '_id' => 'meals:20110316', 'total' => 4, 'delayed' => 2, 'completed' => 3, 'failed' => 1, 'exceptions' => ["#<TooMuchEggs: beware !>"] 'custom' => {:eggs => 12, :bacon => 1, :potatoes => 3} }

resque-mongo-groups

gem "resque-mongo-groups"

resque-mongo-scheduler

Resque.enqueue_at( 1.hour.from_now, Eat, :what => 'leftovers' )

resque-mongo-scheduler

breakfast: cron: "0 8 * * *" class: Eat args: "bacon" description: Breakfast

gem "resque-mongo-scheduler", :require => "resque-scheduler"

resque-mongo-scheduler

resque-mongo ⚡ resque

You cannot use resque and resque-mongo in the same project

resque-mongo ♥ resque

migration to resque-mongo => free

resque-mongo ♥ resqueStill have to migrate from

to

resque-mongo ♥ resque

easy to merge the last changes from resque to resque-mongo

easy to port existing plugins

Get the gem !

gem install nfo-resque-mongo

The maintainer still did not answer to my ownership requests :(

and use it with Bundler

gem "nfo-resque-mongo", :require => "resque"

Get the source!

https://github.com/nfo/resque-mongo

Help needed !Add exclusive features to resque-web

Migrate existing plugins

Create new plugins

Promote the gem

Or simply use the gem, you ‘ll surely send pull requests

That’s all folks

Credits for Flickr images: acezebragirl4j, kiwitime, fulbert05, josh_exell, benheine, tudacee