Porting Rails apps for High Availability

SystemsMarcelo Correia Pinheiro

@salizzar

$ whoami

Programmed with ASP, PHP, C#, Java, Python, Ruby, etc etc

Twitter: @salizzar

http://salizzar.net/ (always under construction)

Locaweb PaaS/IaaS Engineering Team (Domain Registration)

TopicsCommon Architecture

Application Modularity

Useful Gems

Database Replication

Session Management

App Distribution and Deployment

Configuration Management

Load Balancing

Conclusion

Common Architecture

One machine running HTTP Server, App and Database (sometimes in a dedicated server)

Production config files in application GIT repo (all of us use git, alright?)

Deploy via Capistrano / Vlad / * ?

Sufficient for small apps with no perspective to grow absurdly or have monitored growing

Grants HA? Hmmm... what do you do if your database corrupts journaling? (MongoDB REAL CASE)

Common Architecture

Think about it and the possibility of a nightmare due for a bad db query, zombie processes and unexpected errors that will frozen your app

What to do? Look at your app with a Distributed System perspective

Application Modularity

Rails apps, in most cases, can be splitted to two or more small gems or apps (web services, checkout systems, asynch processing, etc)

Application Modularity

You write good code?

If yes, creating gems with it is EASY.

How to detect: first DRY while splitting a app

How to create: $ bundle gem <asdfg>

How to learn more: Creating Your Own Gem in www.rubygems.org

If not, refactoring!

Application Modularity

Breaking your Rails app in smaller apps:

increase future improvements

first step to make distribution via packaging more easy

consequently, more scalable

Look at ActiveController::Metal / sinatra / rack-api / goliath / cramp

Application Modularity

Be careful:

using multiple gems

coupling Rails features inside your models (Rails.logger)

Useful Gems

Hypermedia

Asynchronous Processing (resque)

Webservers

Monitoring

Useful Gems

JSON for the win

Hypermedia-based REST API’s

ROAR: https://github.com/apotonick/roar

RestModel: https://github.com/rodrigues/rest_model

Useful Gems# -*- encoding: UTF-8 -*-

class Domain  attr_accessor :id, :creation_date, :expiration_dateend

module DomainRepresenter  include Roar::Representer::JSON,          Roar::Representer::Feature::Hypermedia

  property :id, :creation_date, :expiration_dateend

request = RestClient.get 'http://an.app/domains/example.com'domain = Domain.new.extend DomainRepresenterdomain.from_json request.body

Useful Gems

# -*- encoding: UTF-8 -*-

class Domain < RestModel  property :id,               type: String  property :creation_date,    type: Date  property :expiration_date,  type: Date  property :provisioning_id,  type: Integerend

request = RestClient.get 'http://an.app/domains/example.com'domain = Domain.from_source(request.body).first

Useful GemsAsynchronous Processing

Resque

resque-scheduler

resque-retry

resque-lock

resque-logger

Useful Gems

resque-scheduler:

https://github.com/bvandenbos/resque-scheduler

cron-based Resque extension

Useful Gems

Incluir config de cron

# resque_scheduler.yml

appointment_notification_enqueuer:  cron:   "0 4 * * *"  class:  AppointmentNotificationEnqueuer  queue:  appointment_notification_enqueuer  args:    email:  true    sms:    true  description: Notify today appointments to customers

free_slot_monitor:  cron:   "0/15 * * * *"  class:  FreeSlotMonitor  queue:  free_slot_monitor  description:  Check for free slots in agenda

Useful Gems# -*- encoding: UTF-8 -*-

class AppointmentNotificationEnqueuer  @queue = :appointment_notification_enqueuer

  def self.perform(args)    appointments = Appointment.today    factory = NotificationFactory.new args    appointments.each do |appointment|      factory.enqueue appointment.id    end  endend

Useful Gemsresque-retry:

https://github.com/lantins/resque-retry

redis backed

retry count times until reach a limit

retry on all or specific exceptions

retry functionality on Resque Web Server

Useful Gems# -*- encoding: UTF-8 -*-

class EmailNotificationWorker  extend Resque::Plugins::Retry

  @queue = :email_notification_enqueuer  @retry_limit = 4  @retry_delay = 300

  def self.perform(args)    appointment = Appointment.find id: args['id']

    mailer = MailNotification.new    mailer.notify_appointment appointment  endend

Useful Gems

Useful Gems

resque-lock:

https://github.com/defunkt/resque-lock

Grants only one instance of a job running in a time

Useful Gems

# -*- encoding: UTF-8 -*-

class SmsNotificationWorker  extend Resque::Plugins::Lock

  def self.perform(args)    appointment = Appointment.find id: args['id']

    sender = SmsNotification.new    sender.deliver_appointment appointment  endend

Useful Gems

resque-logger:

https://github.com/salizzar/resque-logger

Provides a logger for each Resque worker based on queue name

Log Driven Development, please!

Useful Gems

# config/initializers/resque.rb

log_path = Rails.root.join 'log'

config = {  folder:     log_path,  class_name: Logger,  class_args: [ 'daily', 1.gigabyte ],  level:      Logger::INFO,  formatter:  Logger::Formatter.new,}

Resque.logger = config

Useful Gems# -*- encoding: UTF-8 -*-

class EmailNotificationWorker  extend Resque::Plugins::Retry, Resque::Plugins::Logger

  @queue = :email_notification_enqueuer  @retry_limit = 4  @retry_delay = 300

  def self.perform(args)    appointment = Appointment.find id: args['id']

    logger.info(“trying to notify #{appointment.id}”)

    mailer = MailNotification.new    mailer.notify_appointment appointment  endend

Useful GemsWeb Servers:

Thin

https://github.com/macournoyer/thin

EventMachine powered

Killer combo with Nginx

Unicorn

http://unicorn.bogomips.org

Pre-fork based

Great with low-latency network

Useful GemsMonitoring

God

https://github.com/mojombo/god

Several memory leaks at beginning, very stable today

Largely used to manage Resque workers and Thin clusters

Bluepill

https://github.com/arya/bluepill

More simple than God, less features

Some issues with signal handling

Useful Gems

If memory usage is a problem, use Monit.

http://mmonit.com/monit

VERY stable

HIGH expansible

EASY configuration

Database ReplicationRelational Database Replication is not trivial

Be a friend of a sysadm

CAP Theorem: YOU NEED TO KNOW

No magic: look to your architecture

Traditional replication strategies:

master / slave for consistency

master for write / slave for read

Database Replication

My experience:

Relational Databases: no choice

MongoDB: replica set

Redis: master / slave

Database ReplicationFor MongoDB:

MongoMapper

https://github.com/jnunemaker/mongomapper

Largely used, stable

Mongoid

https://github.com/mongoid/mongoid

ActiveRecord-like

Database Replication

brotip #1342: check if your MongoDB client detects when master server goes down automagically (MongoMapper NOT does it)

Suggested keynote: MongoDB em Produção, dá pra confiar?

http://www.slideshare.net/deferraz/mongodb-em-producao

Session Management Given a problem, use the right tool PLEASE

memcached

http://memcached.org

distributed memory object caching system

key/value storage

stores arbitrary data

can be clustered

Session Management

repcached

http://repcached.lab.klab.org

patch inside memcached to add data replication

offers session sharing between memcached servers

Session Managementavailable gems for memcache

memcache-client

https://github.com/mperham/memcache-client

First Ruby client, written by Mike Perham

memcached

https://github.com/evan/memcached

Ruby SWIG wrapper for libmemcached C library

dalli

https://github.com/mperham/dalli

Major refactoring from memcache-client

App Distribution and Deployment

Capistrano? Hnmm... talk with a sysadm about it. It works but lacks security severely (SSH, GCC in production is a invite for attack)

Sysadms knows process monitoring tools, software distribution and server configuration better than us

Be a friend of a sysadm

Or LEARN A LOT to be one :)

App Distribution and Deployment

Build self-packaged apps

$ man dh_make (for Debian based distros)

Hard or too boring? Maybe, but effort compensates

$ dh_make --native --simple --indep --packagename <yourapp> --copyright blank

$ debuild

# dpkg -i <yourapp>_<version>_<arch>.deb

App Distribution and Deployment

Generating a distro-based package with your app grants fast update/rollback to sysadms

# apt-get update && apt-get install <yourapp>

# dpkg -i <your_app_package>

If you want to automate builds:

Setup a repository server

Use Bricklayer and be happy

Configuration Management

Keeping production configs in your app git repo is not always a good idea

Getting config files over control is VERY important to avoid a messy environment

Standardization is king in server configuration

Configuration Management

At this time, three choices:

Puppet

Chef

cfengine

Configuration Management

Puppet

https://github.com/puppetlabs/puppet

Written in Ruby

Large community adoption

Great number of recipes

Works well

Configuration Management

Chef

https://github.com/opscode/chef

Written in Ruby

Created as a Puppet alternative in order to solve some problems with it

Large community adoption

Works good

Configuration Management

cfengine

http://cfengine.com

Written in C

Largely adopted by sysadms

HARD to learn

F****** ROBUST (a single master node can manage 4K servers)

WTF is Load Balancing?

A network methodology to distribute workload across multiple computers in a transparent way for users

Can be proved by /hard|soft/ware

Alternatives:

LVS with keepalived

HAProxy with heartbeat

Load Balancing

Load BalancingLVS (Linux Virtual Server)

http://www.linuxvirtualserver.org

Robust, high scalable solution

Need to share HTTP sessions? It’s the choice

Built-in Linux Kernel module since 2.4.28*, runs in layer-4 (transport layer)

keepalived

http://www.keepalived.org

Largely used with LVS by sysadms

Uses Virtual Router Redundancy Protocol (vrrp) - RFC

Load BalancingHAProxy

http://haproxy.1wt.eu

Generic solution

Works if session sharing between servers is not required

Runs in layer-7 (application layer), computationally more expensive

heartbeat

http://www.linux-ha.org/wiki/Heartbeat

Based on message exchange between clients in a cluster to detect failures

ConclusionPorting Rails apps to a HA environment is a not trivial challenge, but nothing insanely hard

Forces you to think in application architecture, design, distribution, modularity

brotip #915435: do it before coding

Restricts you in order to make server correctly managed

Implies you to be prudent on a deploy

“it sucks at all” = bad excuse, my friend

App tunning later, without fear :)

In fact, in HA deploys are more faster

Questions?

No fear, answers are free as beer :)

Thank you!