Can railsscale

Can Rails (finally) scale?Experience at Ask.fm

Tuesday, September 24, 13

CanRailsScale.com


Ask.fm

• 80M registered users

• 200M monthly unique users

• 400M page-views a day

• ~1M users online


Ask.fm

• 600 questions a second

• 20k requests / second

• 50k likes / minute


Ask.fm

• ~300 ruby on rails nodes

• 16 Cores

• 16Gb RAM

• 24 passengers


Team

10 developers2 Ruby developers


Proposed topics

• Persistence Layer

• Application Layer

• Infrastructure

• Code scalability

• Rails Tools Scalability

• Misc


Proposed topics



• Infrastructure



• Misc


Persistence Layer


Persistence Layer Issues

• Connection pooling

• Datasource switching

• High-Availability

• Redis






• Redis


Connection Pool


Connection Pool


Connection Pool


Connection Pool






• Redis


Datasource switching

Database

ActiveRecord

db_charmer*

Data Access Logic

Business Logic

* https://github.com/kovyrin/db-charmerTuesday, September 24, 13

https://github.com/kovyrin/db-charmer


Example

production: bar: adapter: mysql username: blah host: blah.local database: blah

foo: adapter: mysql username: foo host: foo.local database: foo

class Foo < ActiveRecord::Base db_magic :connection => :fooend

class Bar < ActiveRecord::Base db_magic :connection => :bar, :slave => :bar_slaveend






• Redis


High-Availability

Database

ActiveRecord

db_charmer*

Data Access Logic

Business Logic

* https://github.com/kovyrin/db-charmerTuesday, September 24, 13



• Multiple datasources

• Master - Slave

• Sharding


• Read & Write

• Read

• None

High-Availability


Sharding types

• Modulo function

• Data-dictionary


Example (1)

question_shard_244_247: <<: *question_shard_host_base_srvb71 database: shard244_247 shards: 244..247

question_shard_248_251: <<: *question_shard_host_base_srvb71 database: shard248_251 shards: 248..251


Example (2)

class User < ActiveRecord::Base

def questions(access_mode=:read) on_question_shard(self, access_mode) end

end

user.questions().answered.latest

user.questions(:read_write).build({...})


Example (3)def on_question_shard(access_mode=:read_write, &block) accessor = shard_accessor(access_mode)

question_data_source.shard_for(accessor, nil, &block)end

def shard_for_key(accessor) raise ShardAccessError, 'accessor missing' unless accessor

raise ShardAccessError, 'access denied' unless @shard_access.shard_accessible?(accessor.key, accessor.access_mode)

shard_connection_config( data_source_name(accessor.key) )end






• Redis


Redis Sharding

• ShardMap

• HashRing

• DataDictionary


Proposed topics



• Infrastructure



• Misc


Application Layer

• Ruby threads

• Configuration

• Queue

• Uploads

• Production Log

• Statistics


Application Layer

• Ruby threads

• Configuration

• Queue

• Uploads

• Production Log

• Statistics


Ruby threads

• Multithreading in Ruby sucks :)


Ruby threads

• Parallel DB calls

• Third-party services

• Any service :)


Application Layer

• Ruby threads

• Configuration

• Queue

• Uploads

• Production Log

• Statistics


Configuration

• database.yml

• email.yml

• redis.yml

• ...

• system.yml

• ...


Centralized configuration


Centralized configuration

if defined?(PhusionPassenger) PhusionPassenger.on_event(:starting_worker_process) do |forked| if forked Rlt::Configuration::UpdateWatcher::start(askfm_config.store) else end endelse Rlt::Configuration::UpdateWatcher::start(askfm_config.store)end


Application Layer

• Ruby threads

• Configuration

• Queue

• Uploads

• Production Log

• Statistics


Queue

• Resque


Application Layer

• Ruby threads

• Configuration

• Queue

• Uploads

• Production Log

• Statistics


Upload

• Goliath* application (non-blocking ruby web server)

• 4 servers

• 2.5M Uploads a day

* https://github.com/postrank-labs/goliath/


https://github.com/postrank-labs/goliath/

https://github.com/postrank-labs/goliath/

Application Layer

• Ruby threads

• Configuration

• Queue

• Uploads

• Production Log

• Statistics


Graylog


Application Layer

• Ruby threads

• Configuration

• Queue

• Uploads

• Production Log

• Statistics


Statistics

• Business metrics

• Application metrics

• Performance metrics

• Errors


Statistics


Statistics


Proposed topics



• Infrastructure



• Misc


Infrastructure

• Memory leaks

• Timeout


Proposed topics



• Infrastructure



• Misc


Code scalability

• MVC

• ActiveRecord

• Tests

• DB Migrations


Proposed topics



• Infrastructure



• Misc


Rails tools scalability

• NewRelic

• Exceptional.io

• Google Analytics

• Capistrano


Proposed topics



• Infrastructure



• Misc


GC Tunning


Ruby 2.0


jRuby


Event Machine


Technology

Can railsscale