Redis: Persistence Power

Redis: Persistence PowerNick Quaranto / @qrush / nick@quaran.to

Tuesday, August 10, 2010

What is Redis?

“advanced key-value store”

REmote DIctionary Server

data structures server

YOUR APP

the basics

persist data as you think of it

in memory, sync to disk in background

ridiculously fast

master-slave replication

keys = strings, value = data structures

http://try.redis-db.com

use it: redis-cli

% ./redis-cli SET user:1:name qrushOK

% ./redis-cli GET user:1:name "qrush"

use it: redis-rb% gem install redis% irb -rubygems -rredis

>> $redis = Redis.new=> #<Redis client v2.0.3 connected...

>> $redis.set "user:1:name", "qrush"=> "OK"

>> $redis.get "user:1:name"=> "qrush"

FEATURE SWITCHESSTRINGS

based onhttp://github.com/blog/677http://github.com/bvandenbos/redis_feature_control

Rediswitch.features << :super_secretRediswitch.features << :payment_gatewayRediswitch.features << :twitter

if Rediswitch.enabled?(:twitter) # post to twitterelse # failwhale ahoy!end

begin # take some moneyrescue PaymentGateway::TotallyDown => ohno Rediswitch.disable(:payment_gateway) # notify the troopsend

class Rediswitch def self.enabled?(feature) $redis.exists(feature) end

def self.enable(feature) $redis.incr(feature) end def self.disable(feature) $redis.del(feature) endend

feature switch lessons

the real win: no-deploy configuration

fast enough to be transparent

next step: separate users into buckets with sets

http://github.com/jamesgolick/rollout

RATE LIMITERSTRINGS

soon to be in place athttp://hoptoadapp.com

class Choker def restrict? track count_for > 60 endend

class Choker def count_for $memcache.get(key, true).to_i endend

class Choker def track if !$memcache.get(key, true) $memcache.add(key, "0", 1.minute.from_now, true) end $memcache.incr(key) endend

class Choker def track if !$redis.exists(key) $redis.setex(key, 60, 0) end $redis.incr(key) endend

class Choker def count_for $redis.get(key).to_i endend

rate limiter lessons

expire semantics are changing in redis 2.2

benchmark the crap out of it

could use a sorted set instead of strings

API USAGE LOGGINGSTRINGSSORTED SETS

based offhttp://www.production-hacks.com/2010/07/10/redis-api-access-logger/

# one way to do itclass ActionHit < ActiveRecord::Base # t.string :controller # t.string :action # t.integer :counterend

class UserHit < ActiveRecord::Base # t.string :controller_action # t.integer :user_id # t.integer :counterend

# for all controllers{ "statuses#update" => 1410, "users#create" => 931, "home#index" => 2936}

# users hitting an action{ "101" => 42, "102" => 13, "103" => 34}

class StatusesController < ApplicationController def update $redis.incr "statuses#update" $redis.incr "statuses#update:#{user.id}" endend

class StatusesController < ApplicationController def update key = "statuses#update" $redis.zincrby "actions", 1, key $redis.zincrby "users:#{key}", 1, user.id endend

# hits for a specific user>> $redis.zscore "users:statuses#update", 1001=> 42 # list all the controller actions, sorted>> $redis.zrevrange "actions", 0, -1, :with_scores => true

=> ["home#index", "2936", "statuses#update", "1410", "users#create", "931"]

api usage logging lessons

sorted set = high score list

bad at historical usage, trends

good for a simple heartbeat or pulse

JOB QUEUELISTS

based onhttp://github.com/defunkt/resque

class Staple @queue = :default

def self.perform(post_id, tempfile) # complex image resizing, cropping endend

class Post < ActiveRecord::Base after_save :process_with_stapler

def process_with_stapler Resque.enqueue(Staple, self.id, @tempfile) endend

module Resque extend self

def push(queue, item) redis.rpush "q:#{queue}", encode(item) end

def pop(queue) decode redis.lpop("q:#{queue}") endend

class Resque::Worker def work loop do if job = Resque.pop(queue) job.perform else sleep 5 end end endend

module Resque extend self def bpop(queue) decode redis.blpop("q:#{queue}") endend

class Resque::Worker def work loop do job = Resque.bpop(queue) job.perform end endend

job queue lessons

guaranteed atomic actions, no row locking

blocking commands simplify daemons

many more queue commands in redis itself!

GLOBAL ERRORSSETSMULTI/EXEC

a new feature athttp://hoptoadapp.com

# text :globals, :default => '', :null => false

class Project < ActiveRecord::Base def has_global?(name) @globals ||= globals.gsub(/,/,' ').split @globals.include?(name) endend

# MORE TABLES!!!!

class Global < ActiveRecord::Base belongs_to :projectend

class Project < ActiveRecord::Base has_many :globalsend

# Project#global_errors

["MySQL::Error", "MemCache::Error", "Net::HTTPFatalError"]

class Project < ActiveRecord::Base def global_key "project-#{id}-globals" end def has_global?(name) $redis.sismember(global_key, name) end end

class Project < ActiveRecord::Base after_save :save_globals

def save_globals $redis.del global_key @globals.each do |g| $redis.sadd global_key, g end endend

[Mysql::Error, MemCache::Error, Net::HTTPFatalError]

SISMEMBER SISMEMBER

[Mysql::Error]

[Mysql::Error, OpenURI::HTTPError]

SISMEMBER

SADDSISMEMBER

class Project < ActiveRecord::Base after_save :save_globals

def save_globals $redis.multi do $redis.del global_key @globals.each do |g| $redis.sadd global_key, g end end endend

global error lessons

avoid joins for simple data

consider race conditions

use append-only file (AOF)

MULTIPLAYER NOTEPADPUB/SUB

based onhttp://github.com/laktek/realie

# usage: ruby pub.rb room username

data = {"user" => ARGV[1]}loop do msg = STDIN.gets $redis.publish ARGV[0], data.merge('msg' => msg.strip).to_jsonend

# sub.rb

$redis = Redis.new(:timeout => 0)$redis.subscribe('rubyonrails', 'rubymidwest') do |on| on.message do |room, msg| data = JSON.parse(msg) puts "##{room} - [#{data['user']}]: #{data['msg']}" endend

% ruby pub.rb rubymidwest qrushi give up, i hate markdown

% ruby sub.rb#rubymidwest - [qrush]: i give up, i hate markdown#rubyonrails - [railsn00b]: undefined method posts_path? wtf?#rubymidwest - [turbage]: seriously.

multiplayer notepad lessons

combine with other data structures

can subscribe to channels via patterns

concurrency in ruby is hard

use eventmachine! (or node.js)

more to learn

know your data! (via @antirez)

command reference on the wiki

active IRC, mailing list

AKASENTAI.comredis in the cloud

Thanks!http://redis.io @qrushhttp://rediscookbook.com http://scr.bi/redispower

BONUS ROUND!I prepared way too many examples. Jackpot!

URL SHORTENERSTRINGS

based onhttp://github.com/mattmatt/relink

require 'sinatra'require 'redis_url'

post '/' do RedisUrl.new(params[:url]).saveend

class RedisUrl attr_accessor :url, :id def initialize(url) @url = url @id = seed # unique string algorithm end

def save $redis.set("relink.url|#{@id}", @url) $redis.set("relink.url.rev|#{@url}", @id) endend

get %r{/(.+)} do |url| u = RedisUrl.find(url) if u u.clicked redirect u.url else status 404 endend

class RedisUrl def self.find(id) u = $redis.get("relink.url|#{id}") if u redis_url = RedisUrl.new(u) redis_url.id = id redis_url end end def clicked $redis.incr("relink.url.clicks|#{@id}") endend

url shortener lessons

common pattern: namespacing

incr/decr assumes value is an integer

wrap behavior into ActiveRecord-like objects

next step: store URLs in a list

LIVEDEBUGGINGLISTS

based onhttp://github.com/quirkey/redisk

def after_save begin # make request to external service rescue Exception => ex logger.error "this shouldn't ever happen!" logger.error ex logger.error ex.backtrace endend

# config/initializers/logger.rb

require 'redisk'path = "#{Rails.env}.log"config.logger = Redisk::Logger.new(path)

class Redisk::IO def write(string) redis.rpush "#{name}:_list", string end def self.readlines(name) redis.lrange("#{name}:_list", 0, -1) endend

live debugging lessons

enables real-time data about your system

dump serialized/marshalled data fast

run the redis instance on a different box

dive deeper: hummingbird

COUNTING DOWNLOADSSTRINGSSORTED SETSHASHES

based onhttp://github.com/rubygems/gemcutter

# bad idea, dude

class Download < ActiveRecord::Base belongs_to :rubygemend

class Rubygem < ActiveRecord::Base has_many :downloadsend

class Download def self.incr(rubygem) $redis.incr("all") $redis.incr(rubygem) $redis.zincrby("today", 1, rubygem) endend

class Download def self.rollover(version) $redis.rename "today", "yesterday"

dls = Hash[*$redis.zrange("yesterday", 0, -1, :with_scores => true)] dls.each do |key, score| $redis.hincrby key, Date.today, score Rubygem.find_by_name(key).increment!(:downloads, score) end endend

get "/api/v1/downloads/rails.json" do $redis.hgetall("rails").to_jsonend

# returns...{ "2010-07-09" => 1908, "2010-07-10" => 1032, "2010-07-11" => 1091,}

counting downloads lessons

hybrid approach does work!

redis is really not for search

test your migration away from SQL

Redis: Persistence Power

Documents

Persistence of Power, Elites and Institutions

Redis : Play buzz uses Redis

Introducing redis

Power to the People: Redis Lua Scripts

RedisConf17 - Building Large High Performance Redis Databases with Redis Enterprise

Redis in Kenshoo microservices - Redis Labs · Redis in Kenshoo microservices Jesque based delayed priority queue system Presented by Dima Vizelman. A Redis Use-Case Performance Aggregation

Applied Redis

Redis Labs Enterprise Cluster DocumentationLabs... · Redis Labs Enterprise Cluster Documentation Page 5 Built-in data persistence, backups and replication RLEC enables the use of

The Power of Patience and Persistence

Troubleshooting redis

IBM Power Systems solution for Redis

Redis Presentation

Redis persistence in practice

Redis Beyond

Redis / Redis Labs Overview and then busting 4 myths … · Redis / Redis Labs Overview and then busting 4 myths about Redis Yiftach Shoolman Co-Founder & CTO Home of Redis

Redis for search - Dvir Volk, Redis Labs

Gromov-Hausdorﬀ stable signatures for shapes using persistence · d X ,F use persistence algorithm to obtain a persistence diagram. • This increases discrimination power! •

Redis as a Reliable Work Queue - Percona Redis as a... · Redis Reliable Queueing Service Producer ... Bronto’s Reliable Queue implementation ! Redis running on Intel(R) Xeon(R)

Amazon · Amazon ElastiCache for Redis ElastiCache for Redis User Guide Table of Contents What Is ElastiCache for Redis

Redis tutoring