Embed Size (px)
DESCRIPTION
Citation preview
zerodowntime
deploysfor Rails
photo by Chrishttpwwwflickrcompeoplechrisinplymouth
class ApparentlyHarmlessMigration lt ActiveRecordMigration def selfup remove_column users notes endend
class CompletelySafeMigration lt ActiveRecordMigration def selfup remove_column users notes end
def selfdown add_column users notes text endend
class CompletelySafeMigration lt ActiveRecordMigration def selfup remove_column users notes end
def selfdown add_column users notes text endend
PGError ERROR column notes does not exist
INSERT INTO users(email password notes)
Zero downtime
Challenge accepted
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Testable code +Test coverage
TDD
Easy maintenance DRY
Zero downtime deploys Hot compatibility
Testable code +Test coverage
TDD
Easy maintenance DRY
Zero downtime deploys Hot compatibility
Testable code +Test coverage
TDD
Easy maintenance DRY
Zero downtime deploys Hot compatibility
Hot compatibility
bull Run concurrent versions of your application
bull That are compatible between each other
Hot compatibility
v1 v2
Hot compatibility
v1 v2
writes to `notes` drop column `notes`
Hot compatibility
v1 v1rsquo v2
writes to `notes` drop column `notes`stop writing to `notes`
Hot compatibility
v1 v1rsquo v2
writes to `notes` drop column `notes`stop writing to `notes`
Hot compatibility
v1 v1rsquo v2
writes to `notes` drop column `notes`stop writing to `notes`
Hot compatibility
v1 v1rsquo v2
writes to `notes` drop column `notes`stop writing to `notes`
Hot compatibility
v1 v1rsquo v2
class User def selfcolumns superreject |c| cname == notes endend
writes to `notes` drop column `notes`stop writing to `notes`
Self dependency
Self dependency
user loads form
Self dependency
new form is deployed
Self dependency
user submits the form
POST loginusername=billmicrosoftcomamppassword=q1w2e3
Self dependency
server blows up
Self dependency
bull Make controllers compatible
Self dependency
bull Make controllers compatible
class AuthController lt ApplicationController
protected
def filtered_params paramsduptap do |p| pmerge(email =gt pdelete(username)) end endend
Assets - Rails 2
Server 1
GET dashboard
GET stylesheetsalljs
200
200
Assets - Rails 2
Server 1 Server 2
GET dashboard
GET stylesheetsalljs
200
404
Assets - Rails 2
bull httpsgithubcomddollarasset-resource
Asset pipeline - Rails 3
v1
GET dashboard
GET assetsapplication-v1js
200
200
Asset pipeline - Rails 3
GET dashboard
GET assetsapplication-v1js
200
404
v1 v2
Asset pipeline - Rails 3
Asset pipeline - Rails 3
bull When updating assets keep the existing version around
Asset pipeline - Rails 3
bull When updating assets keep the existing version around
bull Wersquore still in need of tooling to help cleaning up old assets
Asset pipeline - Rails 3
bull When updating assets keep the existing version around
bull Wersquore still in need of tooling to help cleaning up old assets
rake assetscleanoutdated
Migration strategies
Migration strategies
bull Read-only models
Migration strategies
bull Read-only models
class Role lt ActiveRecordBase def readonly true endend
Migration strategies
bull Read-only models
bull Ignore cached columns
Migration strategies
bull Read-only models
bull Ignore cached columns
class User def selfcolumns superreject |c| cname == notes endend
Migration strategies
bull Read-only models
bull Ignore cached columns
bull Know your database
Migration strategies
bull Read-only models
bull Ignore cached columns
bull Know your database
bull Beware of O(n) db locks
MyISAM
MyISAM
InnoDB
InnoDB
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
InnoDB
1 Create a new table with the same structure
2 Apply changes to new table
3 Copy data over
4 Acquire lock on original table
5 Sync newupdateddeleted rows
6 Swap tables
7 Release lock
InnoDB
1 Create a new table with the same structure
2 Apply changes to new table
3 Copy data over
4 Acquire lock on original table
5 Sync newupdateddeleted rows
6 Swap tables
7 Release lock
InnoDBhttpsgithubcomfreelstable_migrator
InnoDBhttpsgithubcomfreelstable_migrator
class AddStuffToMyBigTable lt TableMigration
migrates users
change_table do |t| tinteger foo null =gt false default =gt 0 tstring bar tindex foo name =gt index_for_foo endend
InnoDBhttpsgithubcomfreelstable_migrator
bull Requires acts_as_paranoid
bull Pretty oldstale codebase
bull Check forks for Rails 3 compatibility
Postgres
bull Table locks on
bull Adding updating or removing columns
bull Adding or removing indexes
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time if you avoid default values and constraints
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
can be done without any locking
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time
Postgres
bull Adding fields with defaults
Postgres
bull Adding fields with defaults
before_save assign_defaults
def assign_defaults selfadmin ||= falseend
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
CREATE INDEX CONCURRENTLY users_email_index ON users(email)
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
bull Outside a transaction
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
bull Outside a transaction
class AddIndexToUsers lt ActiveRecordMigration def selfup return if indexes(users)map(ampname)include(users_email_index) add_index users email name =gt users_email_index end
def selfdown remove_index users name =gt users_email_index endend
What about renaming
v1 v2
reads and writes to `foo`
reads and writes to `bar`
What about renaming
v1 v2
reads and writes to `foo`
reads and writes to `bar`
v1rsquo
add column `bar`+
reads from `foo`writes to both`foo` and `bar`
v1rsquorsquo
populate `bar`where needed
What about renaming
v2
reads and writes to `bar`
v2rsquo
ignore `foo`
v3
drop column `foo`
NoSQL
NoSQL
bull Schema migrations O(n) locks are just part of hot compatibility
NoSQL
bull Schema migrations O(n) locks are just part of hot compatibility
bull You still need to put effort into data migration
Workflow
Workflow
bull Make sure pull requests are addressing hot compatibility
Workflow
bull Make sure pull requests are addressing hot compatibility
bull Work after code is merged
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Production
Production
bull Run multiple server processes
Production
bull Run multiple server processes
bull Cycle them individually or in batches
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
bull Cycle them gracefully
Cycling techniques
bull Above the stack
bull Within the stack
bull Platform
Above the stack
bull Use your favorite web server
bull Setup a load balancer above them route around servers that are cycling
Above the stack
bull PROTIP default setup wonrsquot work
bull Configure load balancer to poll for your application health
class HealthController lt ApplicationController def index if Fileexists(Railsrootjoin(MAINTENANCE)) render text =gt Forget me status =gt 503 else render text =gt Im alive end endend
Cycling + Load balancer
bounce serversrm MAINTENANCE
remove server from rotation
touch MAINTENANCE
new servers come online
add server to rotation
Server Load balancer
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy where to check for the health
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
do not let your web server queue requests
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy to run the health check specified above
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
every 2s
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
stop routing to it once the health check failed once
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
and bring it back once it worked once
ELB
$ elb-configure-healthcheck MyLoadBalancer --target HTTP5000health --interval 2 --unhealthy-threshold 1 --healthy-threshold 1
Above the stack
Above the stackbull Minimal impact
Above the stackbull Minimal impact
bull Extended availability comes for free
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
bull Another component you have to track and maintain
Within the stack
Within the stack
Unicorn
bull Replaces your web server
bull Bounces gracefully
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Within the stack
Within the stackbull Easy to setup and deploy
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
bull Doesnrsquot address availability issues
Platform
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 1s bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 5s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 1s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ heroku labsenable preboot
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 starting for 5s bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 up for 1s bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 5s bundle exec thin start -p $PORT
BETA
Heroku
bull Migrations
Heroku
bull Migrations
$ git push pre-production master$ heroku run rake dbmigrate --remote pre-production$ git push production master
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Future
Future
bull Asset pipeline
Future
bull Asset pipeline
bull Mysql tooling
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
bull Can databases
Future
Future
bull Hot compatibility beyond zero downtime deploys
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
class ApparentlyHarmlessMigration lt ActiveRecordMigration def selfup remove_column users notes endend
class CompletelySafeMigration lt ActiveRecordMigration def selfup remove_column users notes end
def selfdown add_column users notes text endend
class CompletelySafeMigration lt ActiveRecordMigration def selfup remove_column users notes end
def selfdown add_column users notes text endend
PGError ERROR column notes does not exist
INSERT INTO users(email password notes)
Zero downtime
Challenge accepted
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Testable code +Test coverage
TDD
Easy maintenance DRY
Zero downtime deploys Hot compatibility
Testable code +Test coverage
TDD
Easy maintenance DRY
Zero downtime deploys Hot compatibility
Testable code +Test coverage
TDD
Easy maintenance DRY
Zero downtime deploys Hot compatibility
Hot compatibility
bull Run concurrent versions of your application
bull That are compatible between each other
Hot compatibility
v1 v2
Hot compatibility
v1 v2
writes to `notes` drop column `notes`
Hot compatibility
v1 v1rsquo v2
writes to `notes` drop column `notes`stop writing to `notes`
Hot compatibility
v1 v1rsquo v2
writes to `notes` drop column `notes`stop writing to `notes`
Hot compatibility
v1 v1rsquo v2
writes to `notes` drop column `notes`stop writing to `notes`
Hot compatibility
v1 v1rsquo v2
writes to `notes` drop column `notes`stop writing to `notes`
Hot compatibility
v1 v1rsquo v2
class User def selfcolumns superreject |c| cname == notes endend
writes to `notes` drop column `notes`stop writing to `notes`
Self dependency
Self dependency
user loads form
Self dependency
new form is deployed
Self dependency
user submits the form
POST loginusername=billmicrosoftcomamppassword=q1w2e3
Self dependency
server blows up
Self dependency
bull Make controllers compatible
Self dependency
bull Make controllers compatible
class AuthController lt ApplicationController
protected
def filtered_params paramsduptap do |p| pmerge(email =gt pdelete(username)) end endend
Assets - Rails 2
Server 1
GET dashboard
GET stylesheetsalljs
200
200
Assets - Rails 2
Server 1 Server 2
GET dashboard
GET stylesheetsalljs
200
404
Assets - Rails 2
bull httpsgithubcomddollarasset-resource
Asset pipeline - Rails 3
v1
GET dashboard
GET assetsapplication-v1js
200
200
Asset pipeline - Rails 3
GET dashboard
GET assetsapplication-v1js
200
404
v1 v2
Asset pipeline - Rails 3
Asset pipeline - Rails 3
bull When updating assets keep the existing version around
Asset pipeline - Rails 3
bull When updating assets keep the existing version around
bull Wersquore still in need of tooling to help cleaning up old assets
Asset pipeline - Rails 3
bull When updating assets keep the existing version around
bull Wersquore still in need of tooling to help cleaning up old assets
rake assetscleanoutdated
Migration strategies
Migration strategies
bull Read-only models
Migration strategies
bull Read-only models
class Role lt ActiveRecordBase def readonly true endend
Migration strategies
bull Read-only models
bull Ignore cached columns
Migration strategies
bull Read-only models
bull Ignore cached columns
class User def selfcolumns superreject |c| cname == notes endend
Migration strategies
bull Read-only models
bull Ignore cached columns
bull Know your database
Migration strategies
bull Read-only models
bull Ignore cached columns
bull Know your database
bull Beware of O(n) db locks
MyISAM
MyISAM
InnoDB
InnoDB
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
InnoDB
1 Create a new table with the same structure
2 Apply changes to new table
3 Copy data over
4 Acquire lock on original table
5 Sync newupdateddeleted rows
6 Swap tables
7 Release lock
InnoDB
1 Create a new table with the same structure
2 Apply changes to new table
3 Copy data over
4 Acquire lock on original table
5 Sync newupdateddeleted rows
6 Swap tables
7 Release lock
InnoDBhttpsgithubcomfreelstable_migrator
InnoDBhttpsgithubcomfreelstable_migrator
class AddStuffToMyBigTable lt TableMigration
migrates users
change_table do |t| tinteger foo null =gt false default =gt 0 tstring bar tindex foo name =gt index_for_foo endend
InnoDBhttpsgithubcomfreelstable_migrator
bull Requires acts_as_paranoid
bull Pretty oldstale codebase
bull Check forks for Rails 3 compatibility
Postgres
bull Table locks on
bull Adding updating or removing columns
bull Adding or removing indexes
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time if you avoid default values and constraints
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
can be done without any locking
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time
Postgres
bull Adding fields with defaults
Postgres
bull Adding fields with defaults
before_save assign_defaults
def assign_defaults selfadmin ||= falseend
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
CREATE INDEX CONCURRENTLY users_email_index ON users(email)
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
bull Outside a transaction
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
bull Outside a transaction
class AddIndexToUsers lt ActiveRecordMigration def selfup return if indexes(users)map(ampname)include(users_email_index) add_index users email name =gt users_email_index end
def selfdown remove_index users name =gt users_email_index endend
What about renaming
v1 v2
reads and writes to `foo`
reads and writes to `bar`
What about renaming
v1 v2
reads and writes to `foo`
reads and writes to `bar`
v1rsquo
add column `bar`+
reads from `foo`writes to both`foo` and `bar`
v1rsquorsquo
populate `bar`where needed
What about renaming
v2
reads and writes to `bar`
v2rsquo
ignore `foo`
v3
drop column `foo`
NoSQL
NoSQL
bull Schema migrations O(n) locks are just part of hot compatibility
NoSQL
bull Schema migrations O(n) locks are just part of hot compatibility
bull You still need to put effort into data migration
Workflow
Workflow
bull Make sure pull requests are addressing hot compatibility
Workflow
bull Make sure pull requests are addressing hot compatibility
bull Work after code is merged
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Production
Production
bull Run multiple server processes
Production
bull Run multiple server processes
bull Cycle them individually or in batches
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
bull Cycle them gracefully
Cycling techniques
bull Above the stack
bull Within the stack
bull Platform
Above the stack
bull Use your favorite web server
bull Setup a load balancer above them route around servers that are cycling
Above the stack
bull PROTIP default setup wonrsquot work
bull Configure load balancer to poll for your application health
class HealthController lt ApplicationController def index if Fileexists(Railsrootjoin(MAINTENANCE)) render text =gt Forget me status =gt 503 else render text =gt Im alive end endend
Cycling + Load balancer
bounce serversrm MAINTENANCE
remove server from rotation
touch MAINTENANCE
new servers come online
add server to rotation
Server Load balancer
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy where to check for the health
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
do not let your web server queue requests
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy to run the health check specified above
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
every 2s
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
stop routing to it once the health check failed once
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
and bring it back once it worked once
ELB
$ elb-configure-healthcheck MyLoadBalancer --target HTTP5000health --interval 2 --unhealthy-threshold 1 --healthy-threshold 1
Above the stack
Above the stackbull Minimal impact
Above the stackbull Minimal impact
bull Extended availability comes for free
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
bull Another component you have to track and maintain
Within the stack
Within the stack
Unicorn
bull Replaces your web server
bull Bounces gracefully
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Within the stack
Within the stackbull Easy to setup and deploy
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
bull Doesnrsquot address availability issues
Platform
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 1s bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 5s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 1s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ heroku labsenable preboot
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 starting for 5s bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 up for 1s bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 5s bundle exec thin start -p $PORT
BETA
Heroku
bull Migrations
Heroku
bull Migrations
$ git push pre-production master$ heroku run rake dbmigrate --remote pre-production$ git push production master
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Future
Future
bull Asset pipeline
Future
bull Asset pipeline
bull Mysql tooling
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
bull Can databases
Future
Future
bull Hot compatibility beyond zero downtime deploys
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
class CompletelySafeMigration lt ActiveRecordMigration def selfup remove_column users notes end
def selfdown add_column users notes text endend
class CompletelySafeMigration lt ActiveRecordMigration def selfup remove_column users notes end
def selfdown add_column users notes text endend
PGError ERROR column notes does not exist
INSERT INTO users(email password notes)
Zero downtime
Challenge accepted
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Testable code +Test coverage
TDD
Easy maintenance DRY
Zero downtime deploys Hot compatibility
Testable code +Test coverage
TDD
Easy maintenance DRY
Zero downtime deploys Hot compatibility
Testable code +Test coverage
TDD
Easy maintenance DRY
Zero downtime deploys Hot compatibility
Hot compatibility
bull Run concurrent versions of your application
bull That are compatible between each other
Hot compatibility
v1 v2
Hot compatibility
v1 v2
writes to `notes` drop column `notes`
Hot compatibility
v1 v1rsquo v2
writes to `notes` drop column `notes`stop writing to `notes`
Hot compatibility
v1 v1rsquo v2
writes to `notes` drop column `notes`stop writing to `notes`
Hot compatibility
v1 v1rsquo v2
writes to `notes` drop column `notes`stop writing to `notes`
Hot compatibility
v1 v1rsquo v2
writes to `notes` drop column `notes`stop writing to `notes`
Hot compatibility
v1 v1rsquo v2
class User def selfcolumns superreject |c| cname == notes endend
writes to `notes` drop column `notes`stop writing to `notes`
Self dependency
Self dependency
user loads form
Self dependency
new form is deployed
Self dependency
user submits the form
POST loginusername=billmicrosoftcomamppassword=q1w2e3
Self dependency
server blows up
Self dependency
bull Make controllers compatible
Self dependency
bull Make controllers compatible
class AuthController lt ApplicationController
protected
def filtered_params paramsduptap do |p| pmerge(email =gt pdelete(username)) end endend
Assets - Rails 2
Server 1
GET dashboard
GET stylesheetsalljs
200
200
Assets - Rails 2
Server 1 Server 2
GET dashboard
GET stylesheetsalljs
200
404
Assets - Rails 2
bull httpsgithubcomddollarasset-resource
Asset pipeline - Rails 3
v1
GET dashboard
GET assetsapplication-v1js
200
200
Asset pipeline - Rails 3
GET dashboard
GET assetsapplication-v1js
200
404
v1 v2
Asset pipeline - Rails 3
Asset pipeline - Rails 3
bull When updating assets keep the existing version around
Asset pipeline - Rails 3
bull When updating assets keep the existing version around
bull Wersquore still in need of tooling to help cleaning up old assets
Asset pipeline - Rails 3
bull When updating assets keep the existing version around
bull Wersquore still in need of tooling to help cleaning up old assets
rake assetscleanoutdated
Migration strategies
Migration strategies
bull Read-only models
Migration strategies
bull Read-only models
class Role lt ActiveRecordBase def readonly true endend
Migration strategies
bull Read-only models
bull Ignore cached columns
Migration strategies
bull Read-only models
bull Ignore cached columns
class User def selfcolumns superreject |c| cname == notes endend
Migration strategies
bull Read-only models
bull Ignore cached columns
bull Know your database
Migration strategies
bull Read-only models
bull Ignore cached columns
bull Know your database
bull Beware of O(n) db locks
MyISAM
MyISAM
InnoDB
InnoDB
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
InnoDB
1 Create a new table with the same structure
2 Apply changes to new table
3 Copy data over
4 Acquire lock on original table
5 Sync newupdateddeleted rows
6 Swap tables
7 Release lock
InnoDB
1 Create a new table with the same structure
2 Apply changes to new table
3 Copy data over
4 Acquire lock on original table
5 Sync newupdateddeleted rows
6 Swap tables
7 Release lock
InnoDBhttpsgithubcomfreelstable_migrator
InnoDBhttpsgithubcomfreelstable_migrator
class AddStuffToMyBigTable lt TableMigration
migrates users
change_table do |t| tinteger foo null =gt false default =gt 0 tstring bar tindex foo name =gt index_for_foo endend
InnoDBhttpsgithubcomfreelstable_migrator
bull Requires acts_as_paranoid
bull Pretty oldstale codebase
bull Check forks for Rails 3 compatibility
Postgres
bull Table locks on
bull Adding updating or removing columns
bull Adding or removing indexes
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time if you avoid default values and constraints
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
can be done without any locking
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time
Postgres
bull Adding fields with defaults
Postgres
bull Adding fields with defaults
before_save assign_defaults
def assign_defaults selfadmin ||= falseend
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
CREATE INDEX CONCURRENTLY users_email_index ON users(email)
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
bull Outside a transaction
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
bull Outside a transaction
class AddIndexToUsers lt ActiveRecordMigration def selfup return if indexes(users)map(ampname)include(users_email_index) add_index users email name =gt users_email_index end
def selfdown remove_index users name =gt users_email_index endend
What about renaming
v1 v2
reads and writes to `foo`
reads and writes to `bar`
What about renaming
v1 v2
reads and writes to `foo`
reads and writes to `bar`
v1rsquo
add column `bar`+
reads from `foo`writes to both`foo` and `bar`
v1rsquorsquo
populate `bar`where needed
What about renaming
v2
reads and writes to `bar`
v2rsquo
ignore `foo`
v3
drop column `foo`
NoSQL
NoSQL
bull Schema migrations O(n) locks are just part of hot compatibility
NoSQL
bull Schema migrations O(n) locks are just part of hot compatibility
bull You still need to put effort into data migration
Workflow
Workflow
bull Make sure pull requests are addressing hot compatibility
Workflow
bull Make sure pull requests are addressing hot compatibility
bull Work after code is merged
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Production
Production
bull Run multiple server processes
Production
bull Run multiple server processes
bull Cycle them individually or in batches
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
bull Cycle them gracefully
Cycling techniques
bull Above the stack
bull Within the stack
bull Platform
Above the stack
bull Use your favorite web server
bull Setup a load balancer above them route around servers that are cycling
Above the stack
bull PROTIP default setup wonrsquot work
bull Configure load balancer to poll for your application health
class HealthController lt ApplicationController def index if Fileexists(Railsrootjoin(MAINTENANCE)) render text =gt Forget me status =gt 503 else render text =gt Im alive end endend
Cycling + Load balancer
bounce serversrm MAINTENANCE
remove server from rotation
touch MAINTENANCE
new servers come online
add server to rotation
Server Load balancer
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy where to check for the health
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
do not let your web server queue requests
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy to run the health check specified above
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
every 2s
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
stop routing to it once the health check failed once
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
and bring it back once it worked once
ELB
$ elb-configure-healthcheck MyLoadBalancer --target HTTP5000health --interval 2 --unhealthy-threshold 1 --healthy-threshold 1
Above the stack
Above the stackbull Minimal impact
Above the stackbull Minimal impact
bull Extended availability comes for free
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
bull Another component you have to track and maintain
Within the stack
Within the stack
Unicorn
bull Replaces your web server
bull Bounces gracefully
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Within the stack
Within the stackbull Easy to setup and deploy
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
bull Doesnrsquot address availability issues
Platform
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 1s bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 5s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 1s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ heroku labsenable preboot
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 starting for 5s bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 up for 1s bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 5s bundle exec thin start -p $PORT
BETA
Heroku
bull Migrations
Heroku
bull Migrations
$ git push pre-production master$ heroku run rake dbmigrate --remote pre-production$ git push production master
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Future
Future
bull Asset pipeline
Future
bull Asset pipeline
bull Mysql tooling
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
bull Can databases
Future
Future
bull Hot compatibility beyond zero downtime deploys
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
class CompletelySafeMigration lt ActiveRecordMigration def selfup remove_column users notes end
def selfdown add_column users notes text endend
PGError ERROR column notes does not exist
INSERT INTO users(email password notes)
Zero downtime
Challenge accepted
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Testable code +Test coverage
TDD
Easy maintenance DRY
Zero downtime deploys Hot compatibility
Testable code +Test coverage
TDD
Easy maintenance DRY
Zero downtime deploys Hot compatibility
Testable code +Test coverage
TDD
Easy maintenance DRY
Zero downtime deploys Hot compatibility
Hot compatibility
bull Run concurrent versions of your application
bull That are compatible between each other
Hot compatibility
v1 v2
Hot compatibility
v1 v2
writes to `notes` drop column `notes`
Hot compatibility
v1 v1rsquo v2
writes to `notes` drop column `notes`stop writing to `notes`
Hot compatibility
v1 v1rsquo v2
writes to `notes` drop column `notes`stop writing to `notes`
Hot compatibility
v1 v1rsquo v2
writes to `notes` drop column `notes`stop writing to `notes`
Hot compatibility
v1 v1rsquo v2
writes to `notes` drop column `notes`stop writing to `notes`
Hot compatibility
v1 v1rsquo v2
class User def selfcolumns superreject |c| cname == notes endend
writes to `notes` drop column `notes`stop writing to `notes`
Self dependency
Self dependency
user loads form
Self dependency
new form is deployed
Self dependency
user submits the form
POST loginusername=billmicrosoftcomamppassword=q1w2e3
Self dependency
server blows up
Self dependency
bull Make controllers compatible
Self dependency
bull Make controllers compatible
class AuthController lt ApplicationController
protected
def filtered_params paramsduptap do |p| pmerge(email =gt pdelete(username)) end endend
Assets - Rails 2
Server 1
GET dashboard
GET stylesheetsalljs
200
200
Assets - Rails 2
Server 1 Server 2
GET dashboard
GET stylesheetsalljs
200
404
Assets - Rails 2
bull httpsgithubcomddollarasset-resource
Asset pipeline - Rails 3
v1
GET dashboard
GET assetsapplication-v1js
200
200
Asset pipeline - Rails 3
GET dashboard
GET assetsapplication-v1js
200
404
v1 v2
Asset pipeline - Rails 3
Asset pipeline - Rails 3
bull When updating assets keep the existing version around
Asset pipeline - Rails 3
bull When updating assets keep the existing version around
bull Wersquore still in need of tooling to help cleaning up old assets
Asset pipeline - Rails 3
bull When updating assets keep the existing version around
bull Wersquore still in need of tooling to help cleaning up old assets
rake assetscleanoutdated
Migration strategies
Migration strategies
bull Read-only models
Migration strategies
bull Read-only models
class Role lt ActiveRecordBase def readonly true endend
Migration strategies
bull Read-only models
bull Ignore cached columns
Migration strategies
bull Read-only models
bull Ignore cached columns
class User def selfcolumns superreject |c| cname == notes endend
Migration strategies
bull Read-only models
bull Ignore cached columns
bull Know your database
Migration strategies
bull Read-only models
bull Ignore cached columns
bull Know your database
bull Beware of O(n) db locks
MyISAM
MyISAM
InnoDB
InnoDB
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
InnoDB
1 Create a new table with the same structure
2 Apply changes to new table
3 Copy data over
4 Acquire lock on original table
5 Sync newupdateddeleted rows
6 Swap tables
7 Release lock
InnoDB
1 Create a new table with the same structure
2 Apply changes to new table
3 Copy data over
4 Acquire lock on original table
5 Sync newupdateddeleted rows
6 Swap tables
7 Release lock
InnoDBhttpsgithubcomfreelstable_migrator
InnoDBhttpsgithubcomfreelstable_migrator
class AddStuffToMyBigTable lt TableMigration
migrates users
change_table do |t| tinteger foo null =gt false default =gt 0 tstring bar tindex foo name =gt index_for_foo endend
InnoDBhttpsgithubcomfreelstable_migrator
bull Requires acts_as_paranoid
bull Pretty oldstale codebase
bull Check forks for Rails 3 compatibility
Postgres
bull Table locks on
bull Adding updating or removing columns
bull Adding or removing indexes
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time if you avoid default values and constraints
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
can be done without any locking
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time
Postgres
bull Adding fields with defaults
Postgres
bull Adding fields with defaults
before_save assign_defaults
def assign_defaults selfadmin ||= falseend
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
CREATE INDEX CONCURRENTLY users_email_index ON users(email)
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
bull Outside a transaction
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
bull Outside a transaction
class AddIndexToUsers lt ActiveRecordMigration def selfup return if indexes(users)map(ampname)include(users_email_index) add_index users email name =gt users_email_index end
def selfdown remove_index users name =gt users_email_index endend
What about renaming
v1 v2
reads and writes to `foo`
reads and writes to `bar`
What about renaming
v1 v2
reads and writes to `foo`
reads and writes to `bar`
v1rsquo
add column `bar`+
reads from `foo`writes to both`foo` and `bar`
v1rsquorsquo
populate `bar`where needed
What about renaming
v2
reads and writes to `bar`
v2rsquo
ignore `foo`
v3
drop column `foo`
NoSQL
NoSQL
bull Schema migrations O(n) locks are just part of hot compatibility
NoSQL
bull Schema migrations O(n) locks are just part of hot compatibility
bull You still need to put effort into data migration
Workflow
Workflow
bull Make sure pull requests are addressing hot compatibility
Workflow
bull Make sure pull requests are addressing hot compatibility
bull Work after code is merged
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Production
Production
bull Run multiple server processes
Production
bull Run multiple server processes
bull Cycle them individually or in batches
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
bull Cycle them gracefully
Cycling techniques
bull Above the stack
bull Within the stack
bull Platform
Above the stack
bull Use your favorite web server
bull Setup a load balancer above them route around servers that are cycling
Above the stack
bull PROTIP default setup wonrsquot work
bull Configure load balancer to poll for your application health
class HealthController lt ApplicationController def index if Fileexists(Railsrootjoin(MAINTENANCE)) render text =gt Forget me status =gt 503 else render text =gt Im alive end endend
Cycling + Load balancer
bounce serversrm MAINTENANCE
remove server from rotation
touch MAINTENANCE
new servers come online
add server to rotation
Server Load balancer
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy where to check for the health
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
do not let your web server queue requests
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy to run the health check specified above
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
every 2s
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
stop routing to it once the health check failed once
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
and bring it back once it worked once
ELB
$ elb-configure-healthcheck MyLoadBalancer --target HTTP5000health --interval 2 --unhealthy-threshold 1 --healthy-threshold 1
Above the stack
Above the stackbull Minimal impact
Above the stackbull Minimal impact
bull Extended availability comes for free
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
bull Another component you have to track and maintain
Within the stack
Within the stack
Unicorn
bull Replaces your web server
bull Bounces gracefully
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Within the stack
Within the stackbull Easy to setup and deploy
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
bull Doesnrsquot address availability issues
Platform
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 1s bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 5s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 1s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ heroku labsenable preboot
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 starting for 5s bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 up for 1s bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 5s bundle exec thin start -p $PORT
BETA
Heroku
bull Migrations
Heroku
bull Migrations
$ git push pre-production master$ heroku run rake dbmigrate --remote pre-production$ git push production master
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Future
Future
bull Asset pipeline
Future
bull Asset pipeline
bull Mysql tooling
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
bull Can databases
Future
Future
bull Hot compatibility beyond zero downtime deploys
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
PGError ERROR column notes does not exist
INSERT INTO users(email password notes)
Zero downtime
Challenge accepted
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Testable code +Test coverage
TDD
Easy maintenance DRY
Zero downtime deploys Hot compatibility
Testable code +Test coverage
TDD
Easy maintenance DRY
Zero downtime deploys Hot compatibility
Testable code +Test coverage
TDD
Easy maintenance DRY
Zero downtime deploys Hot compatibility
Hot compatibility
bull Run concurrent versions of your application
bull That are compatible between each other
Hot compatibility
v1 v2
Hot compatibility
v1 v2
writes to `notes` drop column `notes`
Hot compatibility
v1 v1rsquo v2
writes to `notes` drop column `notes`stop writing to `notes`
Hot compatibility
v1 v1rsquo v2
writes to `notes` drop column `notes`stop writing to `notes`
Hot compatibility
v1 v1rsquo v2
writes to `notes` drop column `notes`stop writing to `notes`
Hot compatibility
v1 v1rsquo v2
writes to `notes` drop column `notes`stop writing to `notes`
Hot compatibility
v1 v1rsquo v2
class User def selfcolumns superreject |c| cname == notes endend
writes to `notes` drop column `notes`stop writing to `notes`
Self dependency
Self dependency
user loads form
Self dependency
new form is deployed
Self dependency
user submits the form
POST loginusername=billmicrosoftcomamppassword=q1w2e3
Self dependency
server blows up
Self dependency
bull Make controllers compatible
Self dependency
bull Make controllers compatible
class AuthController lt ApplicationController
protected
def filtered_params paramsduptap do |p| pmerge(email =gt pdelete(username)) end endend
Assets - Rails 2
Server 1
GET dashboard
GET stylesheetsalljs
200
200
Assets - Rails 2
Server 1 Server 2
GET dashboard
GET stylesheetsalljs
200
404
Assets - Rails 2
bull httpsgithubcomddollarasset-resource
Asset pipeline - Rails 3
v1
GET dashboard
GET assetsapplication-v1js
200
200
Asset pipeline - Rails 3
GET dashboard
GET assetsapplication-v1js
200
404
v1 v2
Asset pipeline - Rails 3
Asset pipeline - Rails 3
bull When updating assets keep the existing version around
Asset pipeline - Rails 3
bull When updating assets keep the existing version around
bull Wersquore still in need of tooling to help cleaning up old assets
Asset pipeline - Rails 3
bull When updating assets keep the existing version around
bull Wersquore still in need of tooling to help cleaning up old assets
rake assetscleanoutdated
Migration strategies
Migration strategies
bull Read-only models
Migration strategies
bull Read-only models
class Role lt ActiveRecordBase def readonly true endend
Migration strategies
bull Read-only models
bull Ignore cached columns
Migration strategies
bull Read-only models
bull Ignore cached columns
class User def selfcolumns superreject |c| cname == notes endend
Migration strategies
bull Read-only models
bull Ignore cached columns
bull Know your database
Migration strategies
bull Read-only models
bull Ignore cached columns
bull Know your database
bull Beware of O(n) db locks
MyISAM
MyISAM
InnoDB
InnoDB
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
InnoDB
1 Create a new table with the same structure
2 Apply changes to new table
3 Copy data over
4 Acquire lock on original table
5 Sync newupdateddeleted rows
6 Swap tables
7 Release lock
InnoDB
1 Create a new table with the same structure
2 Apply changes to new table
3 Copy data over
4 Acquire lock on original table
5 Sync newupdateddeleted rows
6 Swap tables
7 Release lock
InnoDBhttpsgithubcomfreelstable_migrator
InnoDBhttpsgithubcomfreelstable_migrator
class AddStuffToMyBigTable lt TableMigration
migrates users
change_table do |t| tinteger foo null =gt false default =gt 0 tstring bar tindex foo name =gt index_for_foo endend
InnoDBhttpsgithubcomfreelstable_migrator
bull Requires acts_as_paranoid
bull Pretty oldstale codebase
bull Check forks for Rails 3 compatibility
Postgres
bull Table locks on
bull Adding updating or removing columns
bull Adding or removing indexes
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time if you avoid default values and constraints
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
can be done without any locking
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time
Postgres
bull Adding fields with defaults
Postgres
bull Adding fields with defaults
before_save assign_defaults
def assign_defaults selfadmin ||= falseend
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
CREATE INDEX CONCURRENTLY users_email_index ON users(email)
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
bull Outside a transaction
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
bull Outside a transaction
class AddIndexToUsers lt ActiveRecordMigration def selfup return if indexes(users)map(ampname)include(users_email_index) add_index users email name =gt users_email_index end
def selfdown remove_index users name =gt users_email_index endend
What about renaming
v1 v2
reads and writes to `foo`
reads and writes to `bar`
What about renaming
v1 v2
reads and writes to `foo`
reads and writes to `bar`
v1rsquo
add column `bar`+
reads from `foo`writes to both`foo` and `bar`
v1rsquorsquo
populate `bar`where needed
What about renaming
v2
reads and writes to `bar`
v2rsquo
ignore `foo`
v3
drop column `foo`
NoSQL
NoSQL
bull Schema migrations O(n) locks are just part of hot compatibility
NoSQL
bull Schema migrations O(n) locks are just part of hot compatibility
bull You still need to put effort into data migration
Workflow
Workflow
bull Make sure pull requests are addressing hot compatibility
Workflow
bull Make sure pull requests are addressing hot compatibility
bull Work after code is merged
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Production
Production
bull Run multiple server processes
Production
bull Run multiple server processes
bull Cycle them individually or in batches
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
bull Cycle them gracefully
Cycling techniques
bull Above the stack
bull Within the stack
bull Platform
Above the stack
bull Use your favorite web server
bull Setup a load balancer above them route around servers that are cycling
Above the stack
bull PROTIP default setup wonrsquot work
bull Configure load balancer to poll for your application health
class HealthController lt ApplicationController def index if Fileexists(Railsrootjoin(MAINTENANCE)) render text =gt Forget me status =gt 503 else render text =gt Im alive end endend
Cycling + Load balancer
bounce serversrm MAINTENANCE
remove server from rotation
touch MAINTENANCE
new servers come online
add server to rotation
Server Load balancer
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy where to check for the health
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
do not let your web server queue requests
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy to run the health check specified above
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
every 2s
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
stop routing to it once the health check failed once
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
and bring it back once it worked once
ELB
$ elb-configure-healthcheck MyLoadBalancer --target HTTP5000health --interval 2 --unhealthy-threshold 1 --healthy-threshold 1
Above the stack
Above the stackbull Minimal impact
Above the stackbull Minimal impact
bull Extended availability comes for free
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
bull Another component you have to track and maintain
Within the stack
Within the stack
Unicorn
bull Replaces your web server
bull Bounces gracefully
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Within the stack
Within the stackbull Easy to setup and deploy
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
bull Doesnrsquot address availability issues
Platform
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 1s bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 5s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 1s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ heroku labsenable preboot
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 starting for 5s bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 up for 1s bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 5s bundle exec thin start -p $PORT
BETA
Heroku
bull Migrations
Heroku
bull Migrations
$ git push pre-production master$ heroku run rake dbmigrate --remote pre-production$ git push production master
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Future
Future
bull Asset pipeline
Future
bull Asset pipeline
bull Mysql tooling
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
bull Can databases
Future
Future
bull Hot compatibility beyond zero downtime deploys
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
INSERT INTO users(email password notes)
Zero downtime
Challenge accepted
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Testable code +Test coverage
TDD
Easy maintenance DRY
Zero downtime deploys Hot compatibility
Testable code +Test coverage
TDD
Easy maintenance DRY
Zero downtime deploys Hot compatibility
Testable code +Test coverage
TDD
Easy maintenance DRY
Zero downtime deploys Hot compatibility
Hot compatibility
bull Run concurrent versions of your application
bull That are compatible between each other
Hot compatibility
v1 v2
Hot compatibility
v1 v2
writes to `notes` drop column `notes`
Hot compatibility
v1 v1rsquo v2
writes to `notes` drop column `notes`stop writing to `notes`
Hot compatibility
v1 v1rsquo v2
writes to `notes` drop column `notes`stop writing to `notes`
Hot compatibility
v1 v1rsquo v2
writes to `notes` drop column `notes`stop writing to `notes`
Hot compatibility
v1 v1rsquo v2
writes to `notes` drop column `notes`stop writing to `notes`
Hot compatibility
v1 v1rsquo v2
class User def selfcolumns superreject |c| cname == notes endend
writes to `notes` drop column `notes`stop writing to `notes`
Self dependency
Self dependency
user loads form
Self dependency
new form is deployed
Self dependency
user submits the form
POST loginusername=billmicrosoftcomamppassword=q1w2e3
Self dependency
server blows up
Self dependency
bull Make controllers compatible
Self dependency
bull Make controllers compatible
class AuthController lt ApplicationController
protected
def filtered_params paramsduptap do |p| pmerge(email =gt pdelete(username)) end endend
Assets - Rails 2
Server 1
GET dashboard
GET stylesheetsalljs
200
200
Assets - Rails 2
Server 1 Server 2
GET dashboard
GET stylesheetsalljs
200
404
Assets - Rails 2
bull httpsgithubcomddollarasset-resource
Asset pipeline - Rails 3
v1
GET dashboard
GET assetsapplication-v1js
200
200
Asset pipeline - Rails 3
GET dashboard
GET assetsapplication-v1js
200
404
v1 v2
Asset pipeline - Rails 3
Asset pipeline - Rails 3
bull When updating assets keep the existing version around
Asset pipeline - Rails 3
bull When updating assets keep the existing version around
bull Wersquore still in need of tooling to help cleaning up old assets
Asset pipeline - Rails 3
bull When updating assets keep the existing version around
bull Wersquore still in need of tooling to help cleaning up old assets
rake assetscleanoutdated
Migration strategies
Migration strategies
bull Read-only models
Migration strategies
bull Read-only models
class Role lt ActiveRecordBase def readonly true endend
Migration strategies
bull Read-only models
bull Ignore cached columns
Migration strategies
bull Read-only models
bull Ignore cached columns
class User def selfcolumns superreject |c| cname == notes endend
Migration strategies
bull Read-only models
bull Ignore cached columns
bull Know your database
Migration strategies
bull Read-only models
bull Ignore cached columns
bull Know your database
bull Beware of O(n) db locks
MyISAM
MyISAM
InnoDB
InnoDB
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
InnoDB
1 Create a new table with the same structure
2 Apply changes to new table
3 Copy data over
4 Acquire lock on original table
5 Sync newupdateddeleted rows
6 Swap tables
7 Release lock
InnoDB
1 Create a new table with the same structure
2 Apply changes to new table
3 Copy data over
4 Acquire lock on original table
5 Sync newupdateddeleted rows
6 Swap tables
7 Release lock
InnoDBhttpsgithubcomfreelstable_migrator
InnoDBhttpsgithubcomfreelstable_migrator
class AddStuffToMyBigTable lt TableMigration
migrates users
change_table do |t| tinteger foo null =gt false default =gt 0 tstring bar tindex foo name =gt index_for_foo endend
InnoDBhttpsgithubcomfreelstable_migrator
bull Requires acts_as_paranoid
bull Pretty oldstale codebase
bull Check forks for Rails 3 compatibility
Postgres
bull Table locks on
bull Adding updating or removing columns
bull Adding or removing indexes
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time if you avoid default values and constraints
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
can be done without any locking
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time
Postgres
bull Adding fields with defaults
Postgres
bull Adding fields with defaults
before_save assign_defaults
def assign_defaults selfadmin ||= falseend
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
CREATE INDEX CONCURRENTLY users_email_index ON users(email)
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
bull Outside a transaction
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
bull Outside a transaction
class AddIndexToUsers lt ActiveRecordMigration def selfup return if indexes(users)map(ampname)include(users_email_index) add_index users email name =gt users_email_index end
def selfdown remove_index users name =gt users_email_index endend
What about renaming
v1 v2
reads and writes to `foo`
reads and writes to `bar`
What about renaming
v1 v2
reads and writes to `foo`
reads and writes to `bar`
v1rsquo
add column `bar`+
reads from `foo`writes to both`foo` and `bar`
v1rsquorsquo
populate `bar`where needed
What about renaming
v2
reads and writes to `bar`
v2rsquo
ignore `foo`
v3
drop column `foo`
NoSQL
NoSQL
bull Schema migrations O(n) locks are just part of hot compatibility
NoSQL
bull Schema migrations O(n) locks are just part of hot compatibility
bull You still need to put effort into data migration
Workflow
Workflow
bull Make sure pull requests are addressing hot compatibility
Workflow
bull Make sure pull requests are addressing hot compatibility
bull Work after code is merged
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Production
Production
bull Run multiple server processes
Production
bull Run multiple server processes
bull Cycle them individually or in batches
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
bull Cycle them gracefully
Cycling techniques
bull Above the stack
bull Within the stack
bull Platform
Above the stack
bull Use your favorite web server
bull Setup a load balancer above them route around servers that are cycling
Above the stack
bull PROTIP default setup wonrsquot work
bull Configure load balancer to poll for your application health
class HealthController lt ApplicationController def index if Fileexists(Railsrootjoin(MAINTENANCE)) render text =gt Forget me status =gt 503 else render text =gt Im alive end endend
Cycling + Load balancer
bounce serversrm MAINTENANCE
remove server from rotation
touch MAINTENANCE
new servers come online
add server to rotation
Server Load balancer
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy where to check for the health
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
do not let your web server queue requests
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy to run the health check specified above
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
every 2s
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
stop routing to it once the health check failed once
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
and bring it back once it worked once
ELB
$ elb-configure-healthcheck MyLoadBalancer --target HTTP5000health --interval 2 --unhealthy-threshold 1 --healthy-threshold 1
Above the stack
Above the stackbull Minimal impact
Above the stackbull Minimal impact
bull Extended availability comes for free
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
bull Another component you have to track and maintain
Within the stack
Within the stack
Unicorn
bull Replaces your web server
bull Bounces gracefully
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Within the stack
Within the stackbull Easy to setup and deploy
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
bull Doesnrsquot address availability issues
Platform
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 1s bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 5s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 1s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ heroku labsenable preboot
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 starting for 5s bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 up for 1s bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 5s bundle exec thin start -p $PORT
BETA
Heroku
bull Migrations
Heroku
bull Migrations
$ git push pre-production master$ heroku run rake dbmigrate --remote pre-production$ git push production master
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Future
Future
bull Asset pipeline
Future
bull Asset pipeline
bull Mysql tooling
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
bull Can databases
Future
Future
bull Hot compatibility beyond zero downtime deploys
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
Zero downtime
Challenge accepted
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Testable code +Test coverage
TDD
Easy maintenance DRY
Zero downtime deploys Hot compatibility
Testable code +Test coverage
TDD
Easy maintenance DRY
Zero downtime deploys Hot compatibility
Testable code +Test coverage
TDD
Easy maintenance DRY
Zero downtime deploys Hot compatibility
Hot compatibility
bull Run concurrent versions of your application
bull That are compatible between each other
Hot compatibility
v1 v2
Hot compatibility
v1 v2
writes to `notes` drop column `notes`
Hot compatibility
v1 v1rsquo v2
writes to `notes` drop column `notes`stop writing to `notes`
Hot compatibility
v1 v1rsquo v2
writes to `notes` drop column `notes`stop writing to `notes`
Hot compatibility
v1 v1rsquo v2
writes to `notes` drop column `notes`stop writing to `notes`
Hot compatibility
v1 v1rsquo v2
writes to `notes` drop column `notes`stop writing to `notes`
Hot compatibility
v1 v1rsquo v2
class User def selfcolumns superreject |c| cname == notes endend
writes to `notes` drop column `notes`stop writing to `notes`
Self dependency
Self dependency
user loads form
Self dependency
new form is deployed
Self dependency
user submits the form
POST loginusername=billmicrosoftcomamppassword=q1w2e3
Self dependency
server blows up
Self dependency
bull Make controllers compatible
Self dependency
bull Make controllers compatible
class AuthController lt ApplicationController
protected
def filtered_params paramsduptap do |p| pmerge(email =gt pdelete(username)) end endend
Assets - Rails 2
Server 1
GET dashboard
GET stylesheetsalljs
200
200
Assets - Rails 2
Server 1 Server 2
GET dashboard
GET stylesheetsalljs
200
404
Assets - Rails 2
bull httpsgithubcomddollarasset-resource
Asset pipeline - Rails 3
v1
GET dashboard
GET assetsapplication-v1js
200
200
Asset pipeline - Rails 3
GET dashboard
GET assetsapplication-v1js
200
404
v1 v2
Asset pipeline - Rails 3
Asset pipeline - Rails 3
bull When updating assets keep the existing version around
Asset pipeline - Rails 3
bull When updating assets keep the existing version around
bull Wersquore still in need of tooling to help cleaning up old assets
Asset pipeline - Rails 3
bull When updating assets keep the existing version around
bull Wersquore still in need of tooling to help cleaning up old assets
rake assetscleanoutdated
Migration strategies
Migration strategies
bull Read-only models
Migration strategies
bull Read-only models
class Role lt ActiveRecordBase def readonly true endend
Migration strategies
bull Read-only models
bull Ignore cached columns
Migration strategies
bull Read-only models
bull Ignore cached columns
class User def selfcolumns superreject |c| cname == notes endend
Migration strategies
bull Read-only models
bull Ignore cached columns
bull Know your database
Migration strategies
bull Read-only models
bull Ignore cached columns
bull Know your database
bull Beware of O(n) db locks
MyISAM
MyISAM
InnoDB
InnoDB
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
InnoDB
1 Create a new table with the same structure
2 Apply changes to new table
3 Copy data over
4 Acquire lock on original table
5 Sync newupdateddeleted rows
6 Swap tables
7 Release lock
InnoDB
1 Create a new table with the same structure
2 Apply changes to new table
3 Copy data over
4 Acquire lock on original table
5 Sync newupdateddeleted rows
6 Swap tables
7 Release lock
InnoDBhttpsgithubcomfreelstable_migrator
InnoDBhttpsgithubcomfreelstable_migrator
class AddStuffToMyBigTable lt TableMigration
migrates users
change_table do |t| tinteger foo null =gt false default =gt 0 tstring bar tindex foo name =gt index_for_foo endend
InnoDBhttpsgithubcomfreelstable_migrator
bull Requires acts_as_paranoid
bull Pretty oldstale codebase
bull Check forks for Rails 3 compatibility
Postgres
bull Table locks on
bull Adding updating or removing columns
bull Adding or removing indexes
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time if you avoid default values and constraints
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
can be done without any locking
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time
Postgres
bull Adding fields with defaults
Postgres
bull Adding fields with defaults
before_save assign_defaults
def assign_defaults selfadmin ||= falseend
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
CREATE INDEX CONCURRENTLY users_email_index ON users(email)
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
bull Outside a transaction
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
bull Outside a transaction
class AddIndexToUsers lt ActiveRecordMigration def selfup return if indexes(users)map(ampname)include(users_email_index) add_index users email name =gt users_email_index end
def selfdown remove_index users name =gt users_email_index endend
What about renaming
v1 v2
reads and writes to `foo`
reads and writes to `bar`
What about renaming
v1 v2
reads and writes to `foo`
reads and writes to `bar`
v1rsquo
add column `bar`+
reads from `foo`writes to both`foo` and `bar`
v1rsquorsquo
populate `bar`where needed
What about renaming
v2
reads and writes to `bar`
v2rsquo
ignore `foo`
v3
drop column `foo`
NoSQL
NoSQL
bull Schema migrations O(n) locks are just part of hot compatibility
NoSQL
bull Schema migrations O(n) locks are just part of hot compatibility
bull You still need to put effort into data migration
Workflow
Workflow
bull Make sure pull requests are addressing hot compatibility
Workflow
bull Make sure pull requests are addressing hot compatibility
bull Work after code is merged
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Production
Production
bull Run multiple server processes
Production
bull Run multiple server processes
bull Cycle them individually or in batches
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
bull Cycle them gracefully
Cycling techniques
bull Above the stack
bull Within the stack
bull Platform
Above the stack
bull Use your favorite web server
bull Setup a load balancer above them route around servers that are cycling
Above the stack
bull PROTIP default setup wonrsquot work
bull Configure load balancer to poll for your application health
class HealthController lt ApplicationController def index if Fileexists(Railsrootjoin(MAINTENANCE)) render text =gt Forget me status =gt 503 else render text =gt Im alive end endend
Cycling + Load balancer
bounce serversrm MAINTENANCE
remove server from rotation
touch MAINTENANCE
new servers come online
add server to rotation
Server Load balancer
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy where to check for the health
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
do not let your web server queue requests
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy to run the health check specified above
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
every 2s
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
stop routing to it once the health check failed once
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
and bring it back once it worked once
ELB
$ elb-configure-healthcheck MyLoadBalancer --target HTTP5000health --interval 2 --unhealthy-threshold 1 --healthy-threshold 1
Above the stack
Above the stackbull Minimal impact
Above the stackbull Minimal impact
bull Extended availability comes for free
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
bull Another component you have to track and maintain
Within the stack
Within the stack
Unicorn
bull Replaces your web server
bull Bounces gracefully
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Within the stack
Within the stackbull Easy to setup and deploy
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
bull Doesnrsquot address availability issues
Platform
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 1s bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 5s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 1s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ heroku labsenable preboot
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 starting for 5s bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 up for 1s bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 5s bundle exec thin start -p $PORT
BETA
Heroku
bull Migrations
Heroku
bull Migrations
$ git push pre-production master$ heroku run rake dbmigrate --remote pre-production$ git push production master
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Future
Future
bull Asset pipeline
Future
bull Asset pipeline
bull Mysql tooling
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
bull Can databases
Future
Future
bull Hot compatibility beyond zero downtime deploys
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Testable code +Test coverage
TDD
Easy maintenance DRY
Zero downtime deploys Hot compatibility
Testable code +Test coverage
TDD
Easy maintenance DRY
Zero downtime deploys Hot compatibility
Testable code +Test coverage
TDD
Easy maintenance DRY
Zero downtime deploys Hot compatibility
Hot compatibility
bull Run concurrent versions of your application
bull That are compatible between each other
Hot compatibility
v1 v2
Hot compatibility
v1 v2
writes to `notes` drop column `notes`
Hot compatibility
v1 v1rsquo v2
writes to `notes` drop column `notes`stop writing to `notes`
Hot compatibility
v1 v1rsquo v2
writes to `notes` drop column `notes`stop writing to `notes`
Hot compatibility
v1 v1rsquo v2
writes to `notes` drop column `notes`stop writing to `notes`
Hot compatibility
v1 v1rsquo v2
writes to `notes` drop column `notes`stop writing to `notes`
Hot compatibility
v1 v1rsquo v2
class User def selfcolumns superreject |c| cname == notes endend
writes to `notes` drop column `notes`stop writing to `notes`
Self dependency
Self dependency
user loads form
Self dependency
new form is deployed
Self dependency
user submits the form
POST loginusername=billmicrosoftcomamppassword=q1w2e3
Self dependency
server blows up
Self dependency
bull Make controllers compatible
Self dependency
bull Make controllers compatible
class AuthController lt ApplicationController
protected
def filtered_params paramsduptap do |p| pmerge(email =gt pdelete(username)) end endend
Assets - Rails 2
Server 1
GET dashboard
GET stylesheetsalljs
200
200
Assets - Rails 2
Server 1 Server 2
GET dashboard
GET stylesheetsalljs
200
404
Assets - Rails 2
bull httpsgithubcomddollarasset-resource
Asset pipeline - Rails 3
v1
GET dashboard
GET assetsapplication-v1js
200
200
Asset pipeline - Rails 3
GET dashboard
GET assetsapplication-v1js
200
404
v1 v2
Asset pipeline - Rails 3
Asset pipeline - Rails 3
bull When updating assets keep the existing version around
Asset pipeline - Rails 3
bull When updating assets keep the existing version around
bull Wersquore still in need of tooling to help cleaning up old assets
Asset pipeline - Rails 3
bull When updating assets keep the existing version around
bull Wersquore still in need of tooling to help cleaning up old assets
rake assetscleanoutdated
Migration strategies
Migration strategies
bull Read-only models
Migration strategies
bull Read-only models
class Role lt ActiveRecordBase def readonly true endend
Migration strategies
bull Read-only models
bull Ignore cached columns
Migration strategies
bull Read-only models
bull Ignore cached columns
class User def selfcolumns superreject |c| cname == notes endend
Migration strategies
bull Read-only models
bull Ignore cached columns
bull Know your database
Migration strategies
bull Read-only models
bull Ignore cached columns
bull Know your database
bull Beware of O(n) db locks
MyISAM
MyISAM
InnoDB
InnoDB
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
InnoDB
1 Create a new table with the same structure
2 Apply changes to new table
3 Copy data over
4 Acquire lock on original table
5 Sync newupdateddeleted rows
6 Swap tables
7 Release lock
InnoDB
1 Create a new table with the same structure
2 Apply changes to new table
3 Copy data over
4 Acquire lock on original table
5 Sync newupdateddeleted rows
6 Swap tables
7 Release lock
InnoDBhttpsgithubcomfreelstable_migrator
InnoDBhttpsgithubcomfreelstable_migrator
class AddStuffToMyBigTable lt TableMigration
migrates users
change_table do |t| tinteger foo null =gt false default =gt 0 tstring bar tindex foo name =gt index_for_foo endend
InnoDBhttpsgithubcomfreelstable_migrator
bull Requires acts_as_paranoid
bull Pretty oldstale codebase
bull Check forks for Rails 3 compatibility
Postgres
bull Table locks on
bull Adding updating or removing columns
bull Adding or removing indexes
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time if you avoid default values and constraints
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
can be done without any locking
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time
Postgres
bull Adding fields with defaults
Postgres
bull Adding fields with defaults
before_save assign_defaults
def assign_defaults selfadmin ||= falseend
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
CREATE INDEX CONCURRENTLY users_email_index ON users(email)
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
bull Outside a transaction
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
bull Outside a transaction
class AddIndexToUsers lt ActiveRecordMigration def selfup return if indexes(users)map(ampname)include(users_email_index) add_index users email name =gt users_email_index end
def selfdown remove_index users name =gt users_email_index endend
What about renaming
v1 v2
reads and writes to `foo`
reads and writes to `bar`
What about renaming
v1 v2
reads and writes to `foo`
reads and writes to `bar`
v1rsquo
add column `bar`+
reads from `foo`writes to both`foo` and `bar`
v1rsquorsquo
populate `bar`where needed
What about renaming
v2
reads and writes to `bar`
v2rsquo
ignore `foo`
v3
drop column `foo`
NoSQL
NoSQL
bull Schema migrations O(n) locks are just part of hot compatibility
NoSQL
bull Schema migrations O(n) locks are just part of hot compatibility
bull You still need to put effort into data migration
Workflow
Workflow
bull Make sure pull requests are addressing hot compatibility
Workflow
bull Make sure pull requests are addressing hot compatibility
bull Work after code is merged
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Production
Production
bull Run multiple server processes
Production
bull Run multiple server processes
bull Cycle them individually or in batches
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
bull Cycle them gracefully
Cycling techniques
bull Above the stack
bull Within the stack
bull Platform
Above the stack
bull Use your favorite web server
bull Setup a load balancer above them route around servers that are cycling
Above the stack
bull PROTIP default setup wonrsquot work
bull Configure load balancer to poll for your application health
class HealthController lt ApplicationController def index if Fileexists(Railsrootjoin(MAINTENANCE)) render text =gt Forget me status =gt 503 else render text =gt Im alive end endend
Cycling + Load balancer
bounce serversrm MAINTENANCE
remove server from rotation
touch MAINTENANCE
new servers come online
add server to rotation
Server Load balancer
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy where to check for the health
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
do not let your web server queue requests
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy to run the health check specified above
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
every 2s
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
stop routing to it once the health check failed once
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
and bring it back once it worked once
ELB
$ elb-configure-healthcheck MyLoadBalancer --target HTTP5000health --interval 2 --unhealthy-threshold 1 --healthy-threshold 1
Above the stack
Above the stackbull Minimal impact
Above the stackbull Minimal impact
bull Extended availability comes for free
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
bull Another component you have to track and maintain
Within the stack
Within the stack
Unicorn
bull Replaces your web server
bull Bounces gracefully
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Within the stack
Within the stackbull Easy to setup and deploy
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
bull Doesnrsquot address availability issues
Platform
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 1s bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 5s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 1s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ heroku labsenable preboot
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 starting for 5s bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 up for 1s bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 5s bundle exec thin start -p $PORT
BETA
Heroku
bull Migrations
Heroku
bull Migrations
$ git push pre-production master$ heroku run rake dbmigrate --remote pre-production$ git push production master
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Future
Future
bull Asset pipeline
Future
bull Asset pipeline
bull Mysql tooling
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
bull Can databases
Future
Future
bull Hot compatibility beyond zero downtime deploys
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
Testable code +Test coverage
TDD
Easy maintenance DRY
Zero downtime deploys Hot compatibility
Testable code +Test coverage
TDD
Easy maintenance DRY
Zero downtime deploys Hot compatibility
Testable code +Test coverage
TDD
Easy maintenance DRY
Zero downtime deploys Hot compatibility
Hot compatibility
bull Run concurrent versions of your application
bull That are compatible between each other
Hot compatibility
v1 v2
Hot compatibility
v1 v2
writes to `notes` drop column `notes`
Hot compatibility
v1 v1rsquo v2
writes to `notes` drop column `notes`stop writing to `notes`
Hot compatibility
v1 v1rsquo v2
writes to `notes` drop column `notes`stop writing to `notes`
Hot compatibility
v1 v1rsquo v2
writes to `notes` drop column `notes`stop writing to `notes`
Hot compatibility
v1 v1rsquo v2
writes to `notes` drop column `notes`stop writing to `notes`
Hot compatibility
v1 v1rsquo v2
class User def selfcolumns superreject |c| cname == notes endend
writes to `notes` drop column `notes`stop writing to `notes`
Self dependency
Self dependency
user loads form
Self dependency
new form is deployed
Self dependency
user submits the form
POST loginusername=billmicrosoftcomamppassword=q1w2e3
Self dependency
server blows up
Self dependency
bull Make controllers compatible
Self dependency
bull Make controllers compatible
class AuthController lt ApplicationController
protected
def filtered_params paramsduptap do |p| pmerge(email =gt pdelete(username)) end endend
Assets - Rails 2
Server 1
GET dashboard
GET stylesheetsalljs
200
200
Assets - Rails 2
Server 1 Server 2
GET dashboard
GET stylesheetsalljs
200
404
Assets - Rails 2
bull httpsgithubcomddollarasset-resource
Asset pipeline - Rails 3
v1
GET dashboard
GET assetsapplication-v1js
200
200
Asset pipeline - Rails 3
GET dashboard
GET assetsapplication-v1js
200
404
v1 v2
Asset pipeline - Rails 3
Asset pipeline - Rails 3
bull When updating assets keep the existing version around
Asset pipeline - Rails 3
bull When updating assets keep the existing version around
bull Wersquore still in need of tooling to help cleaning up old assets
Asset pipeline - Rails 3
bull When updating assets keep the existing version around
bull Wersquore still in need of tooling to help cleaning up old assets
rake assetscleanoutdated
Migration strategies
Migration strategies
bull Read-only models
Migration strategies
bull Read-only models
class Role lt ActiveRecordBase def readonly true endend
Migration strategies
bull Read-only models
bull Ignore cached columns
Migration strategies
bull Read-only models
bull Ignore cached columns
class User def selfcolumns superreject |c| cname == notes endend
Migration strategies
bull Read-only models
bull Ignore cached columns
bull Know your database
Migration strategies
bull Read-only models
bull Ignore cached columns
bull Know your database
bull Beware of O(n) db locks
MyISAM
MyISAM
InnoDB
InnoDB
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
InnoDB
1 Create a new table with the same structure
2 Apply changes to new table
3 Copy data over
4 Acquire lock on original table
5 Sync newupdateddeleted rows
6 Swap tables
7 Release lock
InnoDB
1 Create a new table with the same structure
2 Apply changes to new table
3 Copy data over
4 Acquire lock on original table
5 Sync newupdateddeleted rows
6 Swap tables
7 Release lock
InnoDBhttpsgithubcomfreelstable_migrator
InnoDBhttpsgithubcomfreelstable_migrator
class AddStuffToMyBigTable lt TableMigration
migrates users
change_table do |t| tinteger foo null =gt false default =gt 0 tstring bar tindex foo name =gt index_for_foo endend
InnoDBhttpsgithubcomfreelstable_migrator
bull Requires acts_as_paranoid
bull Pretty oldstale codebase
bull Check forks for Rails 3 compatibility
Postgres
bull Table locks on
bull Adding updating or removing columns
bull Adding or removing indexes
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time if you avoid default values and constraints
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
can be done without any locking
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time
Postgres
bull Adding fields with defaults
Postgres
bull Adding fields with defaults
before_save assign_defaults
def assign_defaults selfadmin ||= falseend
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
CREATE INDEX CONCURRENTLY users_email_index ON users(email)
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
bull Outside a transaction
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
bull Outside a transaction
class AddIndexToUsers lt ActiveRecordMigration def selfup return if indexes(users)map(ampname)include(users_email_index) add_index users email name =gt users_email_index end
def selfdown remove_index users name =gt users_email_index endend
What about renaming
v1 v2
reads and writes to `foo`
reads and writes to `bar`
What about renaming
v1 v2
reads and writes to `foo`
reads and writes to `bar`
v1rsquo
add column `bar`+
reads from `foo`writes to both`foo` and `bar`
v1rsquorsquo
populate `bar`where needed
What about renaming
v2
reads and writes to `bar`
v2rsquo
ignore `foo`
v3
drop column `foo`
NoSQL
NoSQL
bull Schema migrations O(n) locks are just part of hot compatibility
NoSQL
bull Schema migrations O(n) locks are just part of hot compatibility
bull You still need to put effort into data migration
Workflow
Workflow
bull Make sure pull requests are addressing hot compatibility
Workflow
bull Make sure pull requests are addressing hot compatibility
bull Work after code is merged
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Production
Production
bull Run multiple server processes
Production
bull Run multiple server processes
bull Cycle them individually or in batches
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
bull Cycle them gracefully
Cycling techniques
bull Above the stack
bull Within the stack
bull Platform
Above the stack
bull Use your favorite web server
bull Setup a load balancer above them route around servers that are cycling
Above the stack
bull PROTIP default setup wonrsquot work
bull Configure load balancer to poll for your application health
class HealthController lt ApplicationController def index if Fileexists(Railsrootjoin(MAINTENANCE)) render text =gt Forget me status =gt 503 else render text =gt Im alive end endend
Cycling + Load balancer
bounce serversrm MAINTENANCE
remove server from rotation
touch MAINTENANCE
new servers come online
add server to rotation
Server Load balancer
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy where to check for the health
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
do not let your web server queue requests
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy to run the health check specified above
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
every 2s
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
stop routing to it once the health check failed once
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
and bring it back once it worked once
ELB
$ elb-configure-healthcheck MyLoadBalancer --target HTTP5000health --interval 2 --unhealthy-threshold 1 --healthy-threshold 1
Above the stack
Above the stackbull Minimal impact
Above the stackbull Minimal impact
bull Extended availability comes for free
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
bull Another component you have to track and maintain
Within the stack
Within the stack
Unicorn
bull Replaces your web server
bull Bounces gracefully
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Within the stack
Within the stackbull Easy to setup and deploy
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
bull Doesnrsquot address availability issues
Platform
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 1s bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 5s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 1s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ heroku labsenable preboot
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 starting for 5s bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 up for 1s bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 5s bundle exec thin start -p $PORT
BETA
Heroku
bull Migrations
Heroku
bull Migrations
$ git push pre-production master$ heroku run rake dbmigrate --remote pre-production$ git push production master
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Future
Future
bull Asset pipeline
Future
bull Asset pipeline
bull Mysql tooling
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
bull Can databases
Future
Future
bull Hot compatibility beyond zero downtime deploys
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
Testable code +Test coverage
TDD
Easy maintenance DRY
Zero downtime deploys Hot compatibility
Testable code +Test coverage
TDD
Easy maintenance DRY
Zero downtime deploys Hot compatibility
Hot compatibility
bull Run concurrent versions of your application
bull That are compatible between each other
Hot compatibility
v1 v2
Hot compatibility
v1 v2
writes to `notes` drop column `notes`
Hot compatibility
v1 v1rsquo v2
writes to `notes` drop column `notes`stop writing to `notes`
Hot compatibility
v1 v1rsquo v2
writes to `notes` drop column `notes`stop writing to `notes`
Hot compatibility
v1 v1rsquo v2
writes to `notes` drop column `notes`stop writing to `notes`
Hot compatibility
v1 v1rsquo v2
writes to `notes` drop column `notes`stop writing to `notes`
Hot compatibility
v1 v1rsquo v2
class User def selfcolumns superreject |c| cname == notes endend
writes to `notes` drop column `notes`stop writing to `notes`
Self dependency
Self dependency
user loads form
Self dependency
new form is deployed
Self dependency
user submits the form
POST loginusername=billmicrosoftcomamppassword=q1w2e3
Self dependency
server blows up
Self dependency
bull Make controllers compatible
Self dependency
bull Make controllers compatible
class AuthController lt ApplicationController
protected
def filtered_params paramsduptap do |p| pmerge(email =gt pdelete(username)) end endend
Assets - Rails 2
Server 1
GET dashboard
GET stylesheetsalljs
200
200
Assets - Rails 2
Server 1 Server 2
GET dashboard
GET stylesheetsalljs
200
404
Assets - Rails 2
bull httpsgithubcomddollarasset-resource
Asset pipeline - Rails 3
v1
GET dashboard
GET assetsapplication-v1js
200
200
Asset pipeline - Rails 3
GET dashboard
GET assetsapplication-v1js
200
404
v1 v2
Asset pipeline - Rails 3
Asset pipeline - Rails 3
bull When updating assets keep the existing version around
Asset pipeline - Rails 3
bull When updating assets keep the existing version around
bull Wersquore still in need of tooling to help cleaning up old assets
Asset pipeline - Rails 3
bull When updating assets keep the existing version around
bull Wersquore still in need of tooling to help cleaning up old assets
rake assetscleanoutdated
Migration strategies
Migration strategies
bull Read-only models
Migration strategies
bull Read-only models
class Role lt ActiveRecordBase def readonly true endend
Migration strategies
bull Read-only models
bull Ignore cached columns
Migration strategies
bull Read-only models
bull Ignore cached columns
class User def selfcolumns superreject |c| cname == notes endend
Migration strategies
bull Read-only models
bull Ignore cached columns
bull Know your database
Migration strategies
bull Read-only models
bull Ignore cached columns
bull Know your database
bull Beware of O(n) db locks
MyISAM
MyISAM
InnoDB
InnoDB
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
InnoDB
1 Create a new table with the same structure
2 Apply changes to new table
3 Copy data over
4 Acquire lock on original table
5 Sync newupdateddeleted rows
6 Swap tables
7 Release lock
InnoDB
1 Create a new table with the same structure
2 Apply changes to new table
3 Copy data over
4 Acquire lock on original table
5 Sync newupdateddeleted rows
6 Swap tables
7 Release lock
InnoDBhttpsgithubcomfreelstable_migrator
InnoDBhttpsgithubcomfreelstable_migrator
class AddStuffToMyBigTable lt TableMigration
migrates users
change_table do |t| tinteger foo null =gt false default =gt 0 tstring bar tindex foo name =gt index_for_foo endend
InnoDBhttpsgithubcomfreelstable_migrator
bull Requires acts_as_paranoid
bull Pretty oldstale codebase
bull Check forks for Rails 3 compatibility
Postgres
bull Table locks on
bull Adding updating or removing columns
bull Adding or removing indexes
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time if you avoid default values and constraints
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
can be done without any locking
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time
Postgres
bull Adding fields with defaults
Postgres
bull Adding fields with defaults
before_save assign_defaults
def assign_defaults selfadmin ||= falseend
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
CREATE INDEX CONCURRENTLY users_email_index ON users(email)
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
bull Outside a transaction
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
bull Outside a transaction
class AddIndexToUsers lt ActiveRecordMigration def selfup return if indexes(users)map(ampname)include(users_email_index) add_index users email name =gt users_email_index end
def selfdown remove_index users name =gt users_email_index endend
What about renaming
v1 v2
reads and writes to `foo`
reads and writes to `bar`
What about renaming
v1 v2
reads and writes to `foo`
reads and writes to `bar`
v1rsquo
add column `bar`+
reads from `foo`writes to both`foo` and `bar`
v1rsquorsquo
populate `bar`where needed
What about renaming
v2
reads and writes to `bar`
v2rsquo
ignore `foo`
v3
drop column `foo`
NoSQL
NoSQL
bull Schema migrations O(n) locks are just part of hot compatibility
NoSQL
bull Schema migrations O(n) locks are just part of hot compatibility
bull You still need to put effort into data migration
Workflow
Workflow
bull Make sure pull requests are addressing hot compatibility
Workflow
bull Make sure pull requests are addressing hot compatibility
bull Work after code is merged
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Production
Production
bull Run multiple server processes
Production
bull Run multiple server processes
bull Cycle them individually or in batches
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
bull Cycle them gracefully
Cycling techniques
bull Above the stack
bull Within the stack
bull Platform
Above the stack
bull Use your favorite web server
bull Setup a load balancer above them route around servers that are cycling
Above the stack
bull PROTIP default setup wonrsquot work
bull Configure load balancer to poll for your application health
class HealthController lt ApplicationController def index if Fileexists(Railsrootjoin(MAINTENANCE)) render text =gt Forget me status =gt 503 else render text =gt Im alive end endend
Cycling + Load balancer
bounce serversrm MAINTENANCE
remove server from rotation
touch MAINTENANCE
new servers come online
add server to rotation
Server Load balancer
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy where to check for the health
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
do not let your web server queue requests
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy to run the health check specified above
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
every 2s
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
stop routing to it once the health check failed once
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
and bring it back once it worked once
ELB
$ elb-configure-healthcheck MyLoadBalancer --target HTTP5000health --interval 2 --unhealthy-threshold 1 --healthy-threshold 1
Above the stack
Above the stackbull Minimal impact
Above the stackbull Minimal impact
bull Extended availability comes for free
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
bull Another component you have to track and maintain
Within the stack
Within the stack
Unicorn
bull Replaces your web server
bull Bounces gracefully
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Within the stack
Within the stackbull Easy to setup and deploy
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
bull Doesnrsquot address availability issues
Platform
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 1s bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 5s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 1s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ heroku labsenable preboot
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 starting for 5s bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 up for 1s bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 5s bundle exec thin start -p $PORT
BETA
Heroku
bull Migrations
Heroku
bull Migrations
$ git push pre-production master$ heroku run rake dbmigrate --remote pre-production$ git push production master
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Future
Future
bull Asset pipeline
Future
bull Asset pipeline
bull Mysql tooling
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
bull Can databases
Future
Future
bull Hot compatibility beyond zero downtime deploys
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
Testable code +Test coverage
TDD
Easy maintenance DRY
Zero downtime deploys Hot compatibility
Hot compatibility
bull Run concurrent versions of your application
bull That are compatible between each other
Hot compatibility
v1 v2
Hot compatibility
v1 v2
writes to `notes` drop column `notes`
Hot compatibility
v1 v1rsquo v2
writes to `notes` drop column `notes`stop writing to `notes`
Hot compatibility
v1 v1rsquo v2
writes to `notes` drop column `notes`stop writing to `notes`
Hot compatibility
v1 v1rsquo v2
writes to `notes` drop column `notes`stop writing to `notes`
Hot compatibility
v1 v1rsquo v2
writes to `notes` drop column `notes`stop writing to `notes`
Hot compatibility
v1 v1rsquo v2
class User def selfcolumns superreject |c| cname == notes endend
writes to `notes` drop column `notes`stop writing to `notes`
Self dependency
Self dependency
user loads form
Self dependency
new form is deployed
Self dependency
user submits the form
POST loginusername=billmicrosoftcomamppassword=q1w2e3
Self dependency
server blows up
Self dependency
bull Make controllers compatible
Self dependency
bull Make controllers compatible
class AuthController lt ApplicationController
protected
def filtered_params paramsduptap do |p| pmerge(email =gt pdelete(username)) end endend
Assets - Rails 2
Server 1
GET dashboard
GET stylesheetsalljs
200
200
Assets - Rails 2
Server 1 Server 2
GET dashboard
GET stylesheetsalljs
200
404
Assets - Rails 2
bull httpsgithubcomddollarasset-resource
Asset pipeline - Rails 3
v1
GET dashboard
GET assetsapplication-v1js
200
200
Asset pipeline - Rails 3
GET dashboard
GET assetsapplication-v1js
200
404
v1 v2
Asset pipeline - Rails 3
Asset pipeline - Rails 3
bull When updating assets keep the existing version around
Asset pipeline - Rails 3
bull When updating assets keep the existing version around
bull Wersquore still in need of tooling to help cleaning up old assets
Asset pipeline - Rails 3
bull When updating assets keep the existing version around
bull Wersquore still in need of tooling to help cleaning up old assets
rake assetscleanoutdated
Migration strategies
Migration strategies
bull Read-only models
Migration strategies
bull Read-only models
class Role lt ActiveRecordBase def readonly true endend
Migration strategies
bull Read-only models
bull Ignore cached columns
Migration strategies
bull Read-only models
bull Ignore cached columns
class User def selfcolumns superreject |c| cname == notes endend
Migration strategies
bull Read-only models
bull Ignore cached columns
bull Know your database
Migration strategies
bull Read-only models
bull Ignore cached columns
bull Know your database
bull Beware of O(n) db locks
MyISAM
MyISAM
InnoDB
InnoDB
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
InnoDB
1 Create a new table with the same structure
2 Apply changes to new table
3 Copy data over
4 Acquire lock on original table
5 Sync newupdateddeleted rows
6 Swap tables
7 Release lock
InnoDB
1 Create a new table with the same structure
2 Apply changes to new table
3 Copy data over
4 Acquire lock on original table
5 Sync newupdateddeleted rows
6 Swap tables
7 Release lock
InnoDBhttpsgithubcomfreelstable_migrator
InnoDBhttpsgithubcomfreelstable_migrator
class AddStuffToMyBigTable lt TableMigration
migrates users
change_table do |t| tinteger foo null =gt false default =gt 0 tstring bar tindex foo name =gt index_for_foo endend
InnoDBhttpsgithubcomfreelstable_migrator
bull Requires acts_as_paranoid
bull Pretty oldstale codebase
bull Check forks for Rails 3 compatibility
Postgres
bull Table locks on
bull Adding updating or removing columns
bull Adding or removing indexes
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time if you avoid default values and constraints
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
can be done without any locking
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time
Postgres
bull Adding fields with defaults
Postgres
bull Adding fields with defaults
before_save assign_defaults
def assign_defaults selfadmin ||= falseend
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
CREATE INDEX CONCURRENTLY users_email_index ON users(email)
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
bull Outside a transaction
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
bull Outside a transaction
class AddIndexToUsers lt ActiveRecordMigration def selfup return if indexes(users)map(ampname)include(users_email_index) add_index users email name =gt users_email_index end
def selfdown remove_index users name =gt users_email_index endend
What about renaming
v1 v2
reads and writes to `foo`
reads and writes to `bar`
What about renaming
v1 v2
reads and writes to `foo`
reads and writes to `bar`
v1rsquo
add column `bar`+
reads from `foo`writes to both`foo` and `bar`
v1rsquorsquo
populate `bar`where needed
What about renaming
v2
reads and writes to `bar`
v2rsquo
ignore `foo`
v3
drop column `foo`
NoSQL
NoSQL
bull Schema migrations O(n) locks are just part of hot compatibility
NoSQL
bull Schema migrations O(n) locks are just part of hot compatibility
bull You still need to put effort into data migration
Workflow
Workflow
bull Make sure pull requests are addressing hot compatibility
Workflow
bull Make sure pull requests are addressing hot compatibility
bull Work after code is merged
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Production
Production
bull Run multiple server processes
Production
bull Run multiple server processes
bull Cycle them individually or in batches
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
bull Cycle them gracefully
Cycling techniques
bull Above the stack
bull Within the stack
bull Platform
Above the stack
bull Use your favorite web server
bull Setup a load balancer above them route around servers that are cycling
Above the stack
bull PROTIP default setup wonrsquot work
bull Configure load balancer to poll for your application health
class HealthController lt ApplicationController def index if Fileexists(Railsrootjoin(MAINTENANCE)) render text =gt Forget me status =gt 503 else render text =gt Im alive end endend
Cycling + Load balancer
bounce serversrm MAINTENANCE
remove server from rotation
touch MAINTENANCE
new servers come online
add server to rotation
Server Load balancer
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy where to check for the health
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
do not let your web server queue requests
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy to run the health check specified above
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
every 2s
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
stop routing to it once the health check failed once
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
and bring it back once it worked once
ELB
$ elb-configure-healthcheck MyLoadBalancer --target HTTP5000health --interval 2 --unhealthy-threshold 1 --healthy-threshold 1
Above the stack
Above the stackbull Minimal impact
Above the stackbull Minimal impact
bull Extended availability comes for free
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
bull Another component you have to track and maintain
Within the stack
Within the stack
Unicorn
bull Replaces your web server
bull Bounces gracefully
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Within the stack
Within the stackbull Easy to setup and deploy
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
bull Doesnrsquot address availability issues
Platform
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 1s bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 5s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 1s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ heroku labsenable preboot
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 starting for 5s bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 up for 1s bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 5s bundle exec thin start -p $PORT
BETA
Heroku
bull Migrations
Heroku
bull Migrations
$ git push pre-production master$ heroku run rake dbmigrate --remote pre-production$ git push production master
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Future
Future
bull Asset pipeline
Future
bull Asset pipeline
bull Mysql tooling
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
bull Can databases
Future
Future
bull Hot compatibility beyond zero downtime deploys
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
Hot compatibility
bull Run concurrent versions of your application
bull That are compatible between each other
Hot compatibility
v1 v2
Hot compatibility
v1 v2
writes to `notes` drop column `notes`
Hot compatibility
v1 v1rsquo v2
writes to `notes` drop column `notes`stop writing to `notes`
Hot compatibility
v1 v1rsquo v2
writes to `notes` drop column `notes`stop writing to `notes`
Hot compatibility
v1 v1rsquo v2
writes to `notes` drop column `notes`stop writing to `notes`
Hot compatibility
v1 v1rsquo v2
writes to `notes` drop column `notes`stop writing to `notes`
Hot compatibility
v1 v1rsquo v2
class User def selfcolumns superreject |c| cname == notes endend
writes to `notes` drop column `notes`stop writing to `notes`
Self dependency
Self dependency
user loads form
Self dependency
new form is deployed
Self dependency
user submits the form
POST loginusername=billmicrosoftcomamppassword=q1w2e3
Self dependency
server blows up
Self dependency
bull Make controllers compatible
Self dependency
bull Make controllers compatible
class AuthController lt ApplicationController
protected
def filtered_params paramsduptap do |p| pmerge(email =gt pdelete(username)) end endend
Assets - Rails 2
Server 1
GET dashboard
GET stylesheetsalljs
200
200
Assets - Rails 2
Server 1 Server 2
GET dashboard
GET stylesheetsalljs
200
404
Assets - Rails 2
bull httpsgithubcomddollarasset-resource
Asset pipeline - Rails 3
v1
GET dashboard
GET assetsapplication-v1js
200
200
Asset pipeline - Rails 3
GET dashboard
GET assetsapplication-v1js
200
404
v1 v2
Asset pipeline - Rails 3
Asset pipeline - Rails 3
bull When updating assets keep the existing version around
Asset pipeline - Rails 3
bull When updating assets keep the existing version around
bull Wersquore still in need of tooling to help cleaning up old assets
Asset pipeline - Rails 3
bull When updating assets keep the existing version around
bull Wersquore still in need of tooling to help cleaning up old assets
rake assetscleanoutdated
Migration strategies
Migration strategies
bull Read-only models
Migration strategies
bull Read-only models
class Role lt ActiveRecordBase def readonly true endend
Migration strategies
bull Read-only models
bull Ignore cached columns
Migration strategies
bull Read-only models
bull Ignore cached columns
class User def selfcolumns superreject |c| cname == notes endend
Migration strategies
bull Read-only models
bull Ignore cached columns
bull Know your database
Migration strategies
bull Read-only models
bull Ignore cached columns
bull Know your database
bull Beware of O(n) db locks
MyISAM
MyISAM
InnoDB
InnoDB
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
InnoDB
1 Create a new table with the same structure
2 Apply changes to new table
3 Copy data over
4 Acquire lock on original table
5 Sync newupdateddeleted rows
6 Swap tables
7 Release lock
InnoDB
1 Create a new table with the same structure
2 Apply changes to new table
3 Copy data over
4 Acquire lock on original table
5 Sync newupdateddeleted rows
6 Swap tables
7 Release lock
InnoDBhttpsgithubcomfreelstable_migrator
InnoDBhttpsgithubcomfreelstable_migrator
class AddStuffToMyBigTable lt TableMigration
migrates users
change_table do |t| tinteger foo null =gt false default =gt 0 tstring bar tindex foo name =gt index_for_foo endend
InnoDBhttpsgithubcomfreelstable_migrator
bull Requires acts_as_paranoid
bull Pretty oldstale codebase
bull Check forks for Rails 3 compatibility
Postgres
bull Table locks on
bull Adding updating or removing columns
bull Adding or removing indexes
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time if you avoid default values and constraints
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
can be done without any locking
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time
Postgres
bull Adding fields with defaults
Postgres
bull Adding fields with defaults
before_save assign_defaults
def assign_defaults selfadmin ||= falseend
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
CREATE INDEX CONCURRENTLY users_email_index ON users(email)
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
bull Outside a transaction
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
bull Outside a transaction
class AddIndexToUsers lt ActiveRecordMigration def selfup return if indexes(users)map(ampname)include(users_email_index) add_index users email name =gt users_email_index end
def selfdown remove_index users name =gt users_email_index endend
What about renaming
v1 v2
reads and writes to `foo`
reads and writes to `bar`
What about renaming
v1 v2
reads and writes to `foo`
reads and writes to `bar`
v1rsquo
add column `bar`+
reads from `foo`writes to both`foo` and `bar`
v1rsquorsquo
populate `bar`where needed
What about renaming
v2
reads and writes to `bar`
v2rsquo
ignore `foo`
v3
drop column `foo`
NoSQL
NoSQL
bull Schema migrations O(n) locks are just part of hot compatibility
NoSQL
bull Schema migrations O(n) locks are just part of hot compatibility
bull You still need to put effort into data migration
Workflow
Workflow
bull Make sure pull requests are addressing hot compatibility
Workflow
bull Make sure pull requests are addressing hot compatibility
bull Work after code is merged
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Production
Production
bull Run multiple server processes
Production
bull Run multiple server processes
bull Cycle them individually or in batches
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
bull Cycle them gracefully
Cycling techniques
bull Above the stack
bull Within the stack
bull Platform
Above the stack
bull Use your favorite web server
bull Setup a load balancer above them route around servers that are cycling
Above the stack
bull PROTIP default setup wonrsquot work
bull Configure load balancer to poll for your application health
class HealthController lt ApplicationController def index if Fileexists(Railsrootjoin(MAINTENANCE)) render text =gt Forget me status =gt 503 else render text =gt Im alive end endend
Cycling + Load balancer
bounce serversrm MAINTENANCE
remove server from rotation
touch MAINTENANCE
new servers come online
add server to rotation
Server Load balancer
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy where to check for the health
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
do not let your web server queue requests
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy to run the health check specified above
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
every 2s
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
stop routing to it once the health check failed once
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
and bring it back once it worked once
ELB
$ elb-configure-healthcheck MyLoadBalancer --target HTTP5000health --interval 2 --unhealthy-threshold 1 --healthy-threshold 1
Above the stack
Above the stackbull Minimal impact
Above the stackbull Minimal impact
bull Extended availability comes for free
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
bull Another component you have to track and maintain
Within the stack
Within the stack
Unicorn
bull Replaces your web server
bull Bounces gracefully
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Within the stack
Within the stackbull Easy to setup and deploy
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
bull Doesnrsquot address availability issues
Platform
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 1s bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 5s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 1s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ heroku labsenable preboot
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 starting for 5s bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 up for 1s bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 5s bundle exec thin start -p $PORT
BETA
Heroku
bull Migrations
Heroku
bull Migrations
$ git push pre-production master$ heroku run rake dbmigrate --remote pre-production$ git push production master
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Future
Future
bull Asset pipeline
Future
bull Asset pipeline
bull Mysql tooling
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
bull Can databases
Future
Future
bull Hot compatibility beyond zero downtime deploys
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
Hot compatibility
v1 v2
Hot compatibility
v1 v2
writes to `notes` drop column `notes`
Hot compatibility
v1 v1rsquo v2
writes to `notes` drop column `notes`stop writing to `notes`
Hot compatibility
v1 v1rsquo v2
writes to `notes` drop column `notes`stop writing to `notes`
Hot compatibility
v1 v1rsquo v2
writes to `notes` drop column `notes`stop writing to `notes`
Hot compatibility
v1 v1rsquo v2
writes to `notes` drop column `notes`stop writing to `notes`
Hot compatibility
v1 v1rsquo v2
class User def selfcolumns superreject |c| cname == notes endend
writes to `notes` drop column `notes`stop writing to `notes`
Self dependency
Self dependency
user loads form
Self dependency
new form is deployed
Self dependency
user submits the form
POST loginusername=billmicrosoftcomamppassword=q1w2e3
Self dependency
server blows up
Self dependency
bull Make controllers compatible
Self dependency
bull Make controllers compatible
class AuthController lt ApplicationController
protected
def filtered_params paramsduptap do |p| pmerge(email =gt pdelete(username)) end endend
Assets - Rails 2
Server 1
GET dashboard
GET stylesheetsalljs
200
200
Assets - Rails 2
Server 1 Server 2
GET dashboard
GET stylesheetsalljs
200
404
Assets - Rails 2
bull httpsgithubcomddollarasset-resource
Asset pipeline - Rails 3
v1
GET dashboard
GET assetsapplication-v1js
200
200
Asset pipeline - Rails 3
GET dashboard
GET assetsapplication-v1js
200
404
v1 v2
Asset pipeline - Rails 3
Asset pipeline - Rails 3
bull When updating assets keep the existing version around
Asset pipeline - Rails 3
bull When updating assets keep the existing version around
bull Wersquore still in need of tooling to help cleaning up old assets
Asset pipeline - Rails 3
bull When updating assets keep the existing version around
bull Wersquore still in need of tooling to help cleaning up old assets
rake assetscleanoutdated
Migration strategies
Migration strategies
bull Read-only models
Migration strategies
bull Read-only models
class Role lt ActiveRecordBase def readonly true endend
Migration strategies
bull Read-only models
bull Ignore cached columns
Migration strategies
bull Read-only models
bull Ignore cached columns
class User def selfcolumns superreject |c| cname == notes endend
Migration strategies
bull Read-only models
bull Ignore cached columns
bull Know your database
Migration strategies
bull Read-only models
bull Ignore cached columns
bull Know your database
bull Beware of O(n) db locks
MyISAM
MyISAM
InnoDB
InnoDB
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
InnoDB
1 Create a new table with the same structure
2 Apply changes to new table
3 Copy data over
4 Acquire lock on original table
5 Sync newupdateddeleted rows
6 Swap tables
7 Release lock
InnoDB
1 Create a new table with the same structure
2 Apply changes to new table
3 Copy data over
4 Acquire lock on original table
5 Sync newupdateddeleted rows
6 Swap tables
7 Release lock
InnoDBhttpsgithubcomfreelstable_migrator
InnoDBhttpsgithubcomfreelstable_migrator
class AddStuffToMyBigTable lt TableMigration
migrates users
change_table do |t| tinteger foo null =gt false default =gt 0 tstring bar tindex foo name =gt index_for_foo endend
InnoDBhttpsgithubcomfreelstable_migrator
bull Requires acts_as_paranoid
bull Pretty oldstale codebase
bull Check forks for Rails 3 compatibility
Postgres
bull Table locks on
bull Adding updating or removing columns
bull Adding or removing indexes
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time if you avoid default values and constraints
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
can be done without any locking
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time
Postgres
bull Adding fields with defaults
Postgres
bull Adding fields with defaults
before_save assign_defaults
def assign_defaults selfadmin ||= falseend
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
CREATE INDEX CONCURRENTLY users_email_index ON users(email)
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
bull Outside a transaction
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
bull Outside a transaction
class AddIndexToUsers lt ActiveRecordMigration def selfup return if indexes(users)map(ampname)include(users_email_index) add_index users email name =gt users_email_index end
def selfdown remove_index users name =gt users_email_index endend
What about renaming
v1 v2
reads and writes to `foo`
reads and writes to `bar`
What about renaming
v1 v2
reads and writes to `foo`
reads and writes to `bar`
v1rsquo
add column `bar`+
reads from `foo`writes to both`foo` and `bar`
v1rsquorsquo
populate `bar`where needed
What about renaming
v2
reads and writes to `bar`
v2rsquo
ignore `foo`
v3
drop column `foo`
NoSQL
NoSQL
bull Schema migrations O(n) locks are just part of hot compatibility
NoSQL
bull Schema migrations O(n) locks are just part of hot compatibility
bull You still need to put effort into data migration
Workflow
Workflow
bull Make sure pull requests are addressing hot compatibility
Workflow
bull Make sure pull requests are addressing hot compatibility
bull Work after code is merged
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Production
Production
bull Run multiple server processes
Production
bull Run multiple server processes
bull Cycle them individually or in batches
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
bull Cycle them gracefully
Cycling techniques
bull Above the stack
bull Within the stack
bull Platform
Above the stack
bull Use your favorite web server
bull Setup a load balancer above them route around servers that are cycling
Above the stack
bull PROTIP default setup wonrsquot work
bull Configure load balancer to poll for your application health
class HealthController lt ApplicationController def index if Fileexists(Railsrootjoin(MAINTENANCE)) render text =gt Forget me status =gt 503 else render text =gt Im alive end endend
Cycling + Load balancer
bounce serversrm MAINTENANCE
remove server from rotation
touch MAINTENANCE
new servers come online
add server to rotation
Server Load balancer
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy where to check for the health
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
do not let your web server queue requests
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy to run the health check specified above
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
every 2s
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
stop routing to it once the health check failed once
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
and bring it back once it worked once
ELB
$ elb-configure-healthcheck MyLoadBalancer --target HTTP5000health --interval 2 --unhealthy-threshold 1 --healthy-threshold 1
Above the stack
Above the stackbull Minimal impact
Above the stackbull Minimal impact
bull Extended availability comes for free
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
bull Another component you have to track and maintain
Within the stack
Within the stack
Unicorn
bull Replaces your web server
bull Bounces gracefully
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Within the stack
Within the stackbull Easy to setup and deploy
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
bull Doesnrsquot address availability issues
Platform
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 1s bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 5s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 1s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ heroku labsenable preboot
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 starting for 5s bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 up for 1s bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 5s bundle exec thin start -p $PORT
BETA
Heroku
bull Migrations
Heroku
bull Migrations
$ git push pre-production master$ heroku run rake dbmigrate --remote pre-production$ git push production master
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Future
Future
bull Asset pipeline
Future
bull Asset pipeline
bull Mysql tooling
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
bull Can databases
Future
Future
bull Hot compatibility beyond zero downtime deploys
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
Hot compatibility
v1 v2
writes to `notes` drop column `notes`
Hot compatibility
v1 v1rsquo v2
writes to `notes` drop column `notes`stop writing to `notes`
Hot compatibility
v1 v1rsquo v2
writes to `notes` drop column `notes`stop writing to `notes`
Hot compatibility
v1 v1rsquo v2
writes to `notes` drop column `notes`stop writing to `notes`
Hot compatibility
v1 v1rsquo v2
writes to `notes` drop column `notes`stop writing to `notes`
Hot compatibility
v1 v1rsquo v2
class User def selfcolumns superreject |c| cname == notes endend
writes to `notes` drop column `notes`stop writing to `notes`
Self dependency
Self dependency
user loads form
Self dependency
new form is deployed
Self dependency
user submits the form
POST loginusername=billmicrosoftcomamppassword=q1w2e3
Self dependency
server blows up
Self dependency
bull Make controllers compatible
Self dependency
bull Make controllers compatible
class AuthController lt ApplicationController
protected
def filtered_params paramsduptap do |p| pmerge(email =gt pdelete(username)) end endend
Assets - Rails 2
Server 1
GET dashboard
GET stylesheetsalljs
200
200
Assets - Rails 2
Server 1 Server 2
GET dashboard
GET stylesheetsalljs
200
404
Assets - Rails 2
bull httpsgithubcomddollarasset-resource
Asset pipeline - Rails 3
v1
GET dashboard
GET assetsapplication-v1js
200
200
Asset pipeline - Rails 3
GET dashboard
GET assetsapplication-v1js
200
404
v1 v2
Asset pipeline - Rails 3
Asset pipeline - Rails 3
bull When updating assets keep the existing version around
Asset pipeline - Rails 3
bull When updating assets keep the existing version around
bull Wersquore still in need of tooling to help cleaning up old assets
Asset pipeline - Rails 3
bull When updating assets keep the existing version around
bull Wersquore still in need of tooling to help cleaning up old assets
rake assetscleanoutdated
Migration strategies
Migration strategies
bull Read-only models
Migration strategies
bull Read-only models
class Role lt ActiveRecordBase def readonly true endend
Migration strategies
bull Read-only models
bull Ignore cached columns
Migration strategies
bull Read-only models
bull Ignore cached columns
class User def selfcolumns superreject |c| cname == notes endend
Migration strategies
bull Read-only models
bull Ignore cached columns
bull Know your database
Migration strategies
bull Read-only models
bull Ignore cached columns
bull Know your database
bull Beware of O(n) db locks
MyISAM
MyISAM
InnoDB
InnoDB
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
InnoDB
1 Create a new table with the same structure
2 Apply changes to new table
3 Copy data over
4 Acquire lock on original table
5 Sync newupdateddeleted rows
6 Swap tables
7 Release lock
InnoDB
1 Create a new table with the same structure
2 Apply changes to new table
3 Copy data over
4 Acquire lock on original table
5 Sync newupdateddeleted rows
6 Swap tables
7 Release lock
InnoDBhttpsgithubcomfreelstable_migrator
InnoDBhttpsgithubcomfreelstable_migrator
class AddStuffToMyBigTable lt TableMigration
migrates users
change_table do |t| tinteger foo null =gt false default =gt 0 tstring bar tindex foo name =gt index_for_foo endend
InnoDBhttpsgithubcomfreelstable_migrator
bull Requires acts_as_paranoid
bull Pretty oldstale codebase
bull Check forks for Rails 3 compatibility
Postgres
bull Table locks on
bull Adding updating or removing columns
bull Adding or removing indexes
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time if you avoid default values and constraints
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
can be done without any locking
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time
Postgres
bull Adding fields with defaults
Postgres
bull Adding fields with defaults
before_save assign_defaults
def assign_defaults selfadmin ||= falseend
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
CREATE INDEX CONCURRENTLY users_email_index ON users(email)
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
bull Outside a transaction
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
bull Outside a transaction
class AddIndexToUsers lt ActiveRecordMigration def selfup return if indexes(users)map(ampname)include(users_email_index) add_index users email name =gt users_email_index end
def selfdown remove_index users name =gt users_email_index endend
What about renaming
v1 v2
reads and writes to `foo`
reads and writes to `bar`
What about renaming
v1 v2
reads and writes to `foo`
reads and writes to `bar`
v1rsquo
add column `bar`+
reads from `foo`writes to both`foo` and `bar`
v1rsquorsquo
populate `bar`where needed
What about renaming
v2
reads and writes to `bar`
v2rsquo
ignore `foo`
v3
drop column `foo`
NoSQL
NoSQL
bull Schema migrations O(n) locks are just part of hot compatibility
NoSQL
bull Schema migrations O(n) locks are just part of hot compatibility
bull You still need to put effort into data migration
Workflow
Workflow
bull Make sure pull requests are addressing hot compatibility
Workflow
bull Make sure pull requests are addressing hot compatibility
bull Work after code is merged
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Production
Production
bull Run multiple server processes
Production
bull Run multiple server processes
bull Cycle them individually or in batches
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
bull Cycle them gracefully
Cycling techniques
bull Above the stack
bull Within the stack
bull Platform
Above the stack
bull Use your favorite web server
bull Setup a load balancer above them route around servers that are cycling
Above the stack
bull PROTIP default setup wonrsquot work
bull Configure load balancer to poll for your application health
class HealthController lt ApplicationController def index if Fileexists(Railsrootjoin(MAINTENANCE)) render text =gt Forget me status =gt 503 else render text =gt Im alive end endend
Cycling + Load balancer
bounce serversrm MAINTENANCE
remove server from rotation
touch MAINTENANCE
new servers come online
add server to rotation
Server Load balancer
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy where to check for the health
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
do not let your web server queue requests
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy to run the health check specified above
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
every 2s
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
stop routing to it once the health check failed once
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
and bring it back once it worked once
ELB
$ elb-configure-healthcheck MyLoadBalancer --target HTTP5000health --interval 2 --unhealthy-threshold 1 --healthy-threshold 1
Above the stack
Above the stackbull Minimal impact
Above the stackbull Minimal impact
bull Extended availability comes for free
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
bull Another component you have to track and maintain
Within the stack
Within the stack
Unicorn
bull Replaces your web server
bull Bounces gracefully
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Within the stack
Within the stackbull Easy to setup and deploy
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
bull Doesnrsquot address availability issues
Platform
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 1s bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 5s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 1s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ heroku labsenable preboot
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 starting for 5s bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 up for 1s bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 5s bundle exec thin start -p $PORT
BETA
Heroku
bull Migrations
Heroku
bull Migrations
$ git push pre-production master$ heroku run rake dbmigrate --remote pre-production$ git push production master
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Future
Future
bull Asset pipeline
Future
bull Asset pipeline
bull Mysql tooling
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
bull Can databases
Future
Future
bull Hot compatibility beyond zero downtime deploys
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
Hot compatibility
v1 v1rsquo v2
writes to `notes` drop column `notes`stop writing to `notes`
Hot compatibility
v1 v1rsquo v2
writes to `notes` drop column `notes`stop writing to `notes`
Hot compatibility
v1 v1rsquo v2
writes to `notes` drop column `notes`stop writing to `notes`
Hot compatibility
v1 v1rsquo v2
writes to `notes` drop column `notes`stop writing to `notes`
Hot compatibility
v1 v1rsquo v2
class User def selfcolumns superreject |c| cname == notes endend
writes to `notes` drop column `notes`stop writing to `notes`
Self dependency
Self dependency
user loads form
Self dependency
new form is deployed
Self dependency
user submits the form
POST loginusername=billmicrosoftcomamppassword=q1w2e3
Self dependency
server blows up
Self dependency
bull Make controllers compatible
Self dependency
bull Make controllers compatible
class AuthController lt ApplicationController
protected
def filtered_params paramsduptap do |p| pmerge(email =gt pdelete(username)) end endend
Assets - Rails 2
Server 1
GET dashboard
GET stylesheetsalljs
200
200
Assets - Rails 2
Server 1 Server 2
GET dashboard
GET stylesheetsalljs
200
404
Assets - Rails 2
bull httpsgithubcomddollarasset-resource
Asset pipeline - Rails 3
v1
GET dashboard
GET assetsapplication-v1js
200
200
Asset pipeline - Rails 3
GET dashboard
GET assetsapplication-v1js
200
404
v1 v2
Asset pipeline - Rails 3
Asset pipeline - Rails 3
bull When updating assets keep the existing version around
Asset pipeline - Rails 3
bull When updating assets keep the existing version around
bull Wersquore still in need of tooling to help cleaning up old assets
Asset pipeline - Rails 3
bull When updating assets keep the existing version around
bull Wersquore still in need of tooling to help cleaning up old assets
rake assetscleanoutdated
Migration strategies
Migration strategies
bull Read-only models
Migration strategies
bull Read-only models
class Role lt ActiveRecordBase def readonly true endend
Migration strategies
bull Read-only models
bull Ignore cached columns
Migration strategies
bull Read-only models
bull Ignore cached columns
class User def selfcolumns superreject |c| cname == notes endend
Migration strategies
bull Read-only models
bull Ignore cached columns
bull Know your database
Migration strategies
bull Read-only models
bull Ignore cached columns
bull Know your database
bull Beware of O(n) db locks
MyISAM
MyISAM
InnoDB
InnoDB
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
InnoDB
1 Create a new table with the same structure
2 Apply changes to new table
3 Copy data over
4 Acquire lock on original table
5 Sync newupdateddeleted rows
6 Swap tables
7 Release lock
InnoDB
1 Create a new table with the same structure
2 Apply changes to new table
3 Copy data over
4 Acquire lock on original table
5 Sync newupdateddeleted rows
6 Swap tables
7 Release lock
InnoDBhttpsgithubcomfreelstable_migrator
InnoDBhttpsgithubcomfreelstable_migrator
class AddStuffToMyBigTable lt TableMigration
migrates users
change_table do |t| tinteger foo null =gt false default =gt 0 tstring bar tindex foo name =gt index_for_foo endend
InnoDBhttpsgithubcomfreelstable_migrator
bull Requires acts_as_paranoid
bull Pretty oldstale codebase
bull Check forks for Rails 3 compatibility
Postgres
bull Table locks on
bull Adding updating or removing columns
bull Adding or removing indexes
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time if you avoid default values and constraints
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
can be done without any locking
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time
Postgres
bull Adding fields with defaults
Postgres
bull Adding fields with defaults
before_save assign_defaults
def assign_defaults selfadmin ||= falseend
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
CREATE INDEX CONCURRENTLY users_email_index ON users(email)
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
bull Outside a transaction
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
bull Outside a transaction
class AddIndexToUsers lt ActiveRecordMigration def selfup return if indexes(users)map(ampname)include(users_email_index) add_index users email name =gt users_email_index end
def selfdown remove_index users name =gt users_email_index endend
What about renaming
v1 v2
reads and writes to `foo`
reads and writes to `bar`
What about renaming
v1 v2
reads and writes to `foo`
reads and writes to `bar`
v1rsquo
add column `bar`+
reads from `foo`writes to both`foo` and `bar`
v1rsquorsquo
populate `bar`where needed
What about renaming
v2
reads and writes to `bar`
v2rsquo
ignore `foo`
v3
drop column `foo`
NoSQL
NoSQL
bull Schema migrations O(n) locks are just part of hot compatibility
NoSQL
bull Schema migrations O(n) locks are just part of hot compatibility
bull You still need to put effort into data migration
Workflow
Workflow
bull Make sure pull requests are addressing hot compatibility
Workflow
bull Make sure pull requests are addressing hot compatibility
bull Work after code is merged
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Production
Production
bull Run multiple server processes
Production
bull Run multiple server processes
bull Cycle them individually or in batches
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
bull Cycle them gracefully
Cycling techniques
bull Above the stack
bull Within the stack
bull Platform
Above the stack
bull Use your favorite web server
bull Setup a load balancer above them route around servers that are cycling
Above the stack
bull PROTIP default setup wonrsquot work
bull Configure load balancer to poll for your application health
class HealthController lt ApplicationController def index if Fileexists(Railsrootjoin(MAINTENANCE)) render text =gt Forget me status =gt 503 else render text =gt Im alive end endend
Cycling + Load balancer
bounce serversrm MAINTENANCE
remove server from rotation
touch MAINTENANCE
new servers come online
add server to rotation
Server Load balancer
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy where to check for the health
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
do not let your web server queue requests
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy to run the health check specified above
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
every 2s
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
stop routing to it once the health check failed once
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
and bring it back once it worked once
ELB
$ elb-configure-healthcheck MyLoadBalancer --target HTTP5000health --interval 2 --unhealthy-threshold 1 --healthy-threshold 1
Above the stack
Above the stackbull Minimal impact
Above the stackbull Minimal impact
bull Extended availability comes for free
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
bull Another component you have to track and maintain
Within the stack
Within the stack
Unicorn
bull Replaces your web server
bull Bounces gracefully
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Within the stack
Within the stackbull Easy to setup and deploy
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
bull Doesnrsquot address availability issues
Platform
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 1s bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 5s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 1s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ heroku labsenable preboot
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 starting for 5s bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 up for 1s bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 5s bundle exec thin start -p $PORT
BETA
Heroku
bull Migrations
Heroku
bull Migrations
$ git push pre-production master$ heroku run rake dbmigrate --remote pre-production$ git push production master
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Future
Future
bull Asset pipeline
Future
bull Asset pipeline
bull Mysql tooling
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
bull Can databases
Future
Future
bull Hot compatibility beyond zero downtime deploys
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
Hot compatibility
v1 v1rsquo v2
writes to `notes` drop column `notes`stop writing to `notes`
Hot compatibility
v1 v1rsquo v2
writes to `notes` drop column `notes`stop writing to `notes`
Hot compatibility
v1 v1rsquo v2
writes to `notes` drop column `notes`stop writing to `notes`
Hot compatibility
v1 v1rsquo v2
class User def selfcolumns superreject |c| cname == notes endend
writes to `notes` drop column `notes`stop writing to `notes`
Self dependency
Self dependency
user loads form
Self dependency
new form is deployed
Self dependency
user submits the form
POST loginusername=billmicrosoftcomamppassword=q1w2e3
Self dependency
server blows up
Self dependency
bull Make controllers compatible
Self dependency
bull Make controllers compatible
class AuthController lt ApplicationController
protected
def filtered_params paramsduptap do |p| pmerge(email =gt pdelete(username)) end endend
Assets - Rails 2
Server 1
GET dashboard
GET stylesheetsalljs
200
200
Assets - Rails 2
Server 1 Server 2
GET dashboard
GET stylesheetsalljs
200
404
Assets - Rails 2
bull httpsgithubcomddollarasset-resource
Asset pipeline - Rails 3
v1
GET dashboard
GET assetsapplication-v1js
200
200
Asset pipeline - Rails 3
GET dashboard
GET assetsapplication-v1js
200
404
v1 v2
Asset pipeline - Rails 3
Asset pipeline - Rails 3
bull When updating assets keep the existing version around
Asset pipeline - Rails 3
bull When updating assets keep the existing version around
bull Wersquore still in need of tooling to help cleaning up old assets
Asset pipeline - Rails 3
bull When updating assets keep the existing version around
bull Wersquore still in need of tooling to help cleaning up old assets
rake assetscleanoutdated
Migration strategies
Migration strategies
bull Read-only models
Migration strategies
bull Read-only models
class Role lt ActiveRecordBase def readonly true endend
Migration strategies
bull Read-only models
bull Ignore cached columns
Migration strategies
bull Read-only models
bull Ignore cached columns
class User def selfcolumns superreject |c| cname == notes endend
Migration strategies
bull Read-only models
bull Ignore cached columns
bull Know your database
Migration strategies
bull Read-only models
bull Ignore cached columns
bull Know your database
bull Beware of O(n) db locks
MyISAM
MyISAM
InnoDB
InnoDB
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
InnoDB
1 Create a new table with the same structure
2 Apply changes to new table
3 Copy data over
4 Acquire lock on original table
5 Sync newupdateddeleted rows
6 Swap tables
7 Release lock
InnoDB
1 Create a new table with the same structure
2 Apply changes to new table
3 Copy data over
4 Acquire lock on original table
5 Sync newupdateddeleted rows
6 Swap tables
7 Release lock
InnoDBhttpsgithubcomfreelstable_migrator
InnoDBhttpsgithubcomfreelstable_migrator
class AddStuffToMyBigTable lt TableMigration
migrates users
change_table do |t| tinteger foo null =gt false default =gt 0 tstring bar tindex foo name =gt index_for_foo endend
InnoDBhttpsgithubcomfreelstable_migrator
bull Requires acts_as_paranoid
bull Pretty oldstale codebase
bull Check forks for Rails 3 compatibility
Postgres
bull Table locks on
bull Adding updating or removing columns
bull Adding or removing indexes
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time if you avoid default values and constraints
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
can be done without any locking
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time
Postgres
bull Adding fields with defaults
Postgres
bull Adding fields with defaults
before_save assign_defaults
def assign_defaults selfadmin ||= falseend
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
CREATE INDEX CONCURRENTLY users_email_index ON users(email)
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
bull Outside a transaction
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
bull Outside a transaction
class AddIndexToUsers lt ActiveRecordMigration def selfup return if indexes(users)map(ampname)include(users_email_index) add_index users email name =gt users_email_index end
def selfdown remove_index users name =gt users_email_index endend
What about renaming
v1 v2
reads and writes to `foo`
reads and writes to `bar`
What about renaming
v1 v2
reads and writes to `foo`
reads and writes to `bar`
v1rsquo
add column `bar`+
reads from `foo`writes to both`foo` and `bar`
v1rsquorsquo
populate `bar`where needed
What about renaming
v2
reads and writes to `bar`
v2rsquo
ignore `foo`
v3
drop column `foo`
NoSQL
NoSQL
bull Schema migrations O(n) locks are just part of hot compatibility
NoSQL
bull Schema migrations O(n) locks are just part of hot compatibility
bull You still need to put effort into data migration
Workflow
Workflow
bull Make sure pull requests are addressing hot compatibility
Workflow
bull Make sure pull requests are addressing hot compatibility
bull Work after code is merged
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Production
Production
bull Run multiple server processes
Production
bull Run multiple server processes
bull Cycle them individually or in batches
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
bull Cycle them gracefully
Cycling techniques
bull Above the stack
bull Within the stack
bull Platform
Above the stack
bull Use your favorite web server
bull Setup a load balancer above them route around servers that are cycling
Above the stack
bull PROTIP default setup wonrsquot work
bull Configure load balancer to poll for your application health
class HealthController lt ApplicationController def index if Fileexists(Railsrootjoin(MAINTENANCE)) render text =gt Forget me status =gt 503 else render text =gt Im alive end endend
Cycling + Load balancer
bounce serversrm MAINTENANCE
remove server from rotation
touch MAINTENANCE
new servers come online
add server to rotation
Server Load balancer
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy where to check for the health
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
do not let your web server queue requests
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy to run the health check specified above
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
every 2s
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
stop routing to it once the health check failed once
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
and bring it back once it worked once
ELB
$ elb-configure-healthcheck MyLoadBalancer --target HTTP5000health --interval 2 --unhealthy-threshold 1 --healthy-threshold 1
Above the stack
Above the stackbull Minimal impact
Above the stackbull Minimal impact
bull Extended availability comes for free
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
bull Another component you have to track and maintain
Within the stack
Within the stack
Unicorn
bull Replaces your web server
bull Bounces gracefully
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Within the stack
Within the stackbull Easy to setup and deploy
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
bull Doesnrsquot address availability issues
Platform
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 1s bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 5s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 1s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ heroku labsenable preboot
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 starting for 5s bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 up for 1s bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 5s bundle exec thin start -p $PORT
BETA
Heroku
bull Migrations
Heroku
bull Migrations
$ git push pre-production master$ heroku run rake dbmigrate --remote pre-production$ git push production master
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Future
Future
bull Asset pipeline
Future
bull Asset pipeline
bull Mysql tooling
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
bull Can databases
Future
Future
bull Hot compatibility beyond zero downtime deploys
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
Hot compatibility
v1 v1rsquo v2
writes to `notes` drop column `notes`stop writing to `notes`
Hot compatibility
v1 v1rsquo v2
writes to `notes` drop column `notes`stop writing to `notes`
Hot compatibility
v1 v1rsquo v2
class User def selfcolumns superreject |c| cname == notes endend
writes to `notes` drop column `notes`stop writing to `notes`
Self dependency
Self dependency
user loads form
Self dependency
new form is deployed
Self dependency
user submits the form
POST loginusername=billmicrosoftcomamppassword=q1w2e3
Self dependency
server blows up
Self dependency
bull Make controllers compatible
Self dependency
bull Make controllers compatible
class AuthController lt ApplicationController
protected
def filtered_params paramsduptap do |p| pmerge(email =gt pdelete(username)) end endend
Assets - Rails 2
Server 1
GET dashboard
GET stylesheetsalljs
200
200
Assets - Rails 2
Server 1 Server 2
GET dashboard
GET stylesheetsalljs
200
404
Assets - Rails 2
bull httpsgithubcomddollarasset-resource
Asset pipeline - Rails 3
v1
GET dashboard
GET assetsapplication-v1js
200
200
Asset pipeline - Rails 3
GET dashboard
GET assetsapplication-v1js
200
404
v1 v2
Asset pipeline - Rails 3
Asset pipeline - Rails 3
bull When updating assets keep the existing version around
Asset pipeline - Rails 3
bull When updating assets keep the existing version around
bull Wersquore still in need of tooling to help cleaning up old assets
Asset pipeline - Rails 3
bull When updating assets keep the existing version around
bull Wersquore still in need of tooling to help cleaning up old assets
rake assetscleanoutdated
Migration strategies
Migration strategies
bull Read-only models
Migration strategies
bull Read-only models
class Role lt ActiveRecordBase def readonly true endend
Migration strategies
bull Read-only models
bull Ignore cached columns
Migration strategies
bull Read-only models
bull Ignore cached columns
class User def selfcolumns superreject |c| cname == notes endend
Migration strategies
bull Read-only models
bull Ignore cached columns
bull Know your database
Migration strategies
bull Read-only models
bull Ignore cached columns
bull Know your database
bull Beware of O(n) db locks
MyISAM
MyISAM
InnoDB
InnoDB
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
InnoDB
1 Create a new table with the same structure
2 Apply changes to new table
3 Copy data over
4 Acquire lock on original table
5 Sync newupdateddeleted rows
6 Swap tables
7 Release lock
InnoDB
1 Create a new table with the same structure
2 Apply changes to new table
3 Copy data over
4 Acquire lock on original table
5 Sync newupdateddeleted rows
6 Swap tables
7 Release lock
InnoDBhttpsgithubcomfreelstable_migrator
InnoDBhttpsgithubcomfreelstable_migrator
class AddStuffToMyBigTable lt TableMigration
migrates users
change_table do |t| tinteger foo null =gt false default =gt 0 tstring bar tindex foo name =gt index_for_foo endend
InnoDBhttpsgithubcomfreelstable_migrator
bull Requires acts_as_paranoid
bull Pretty oldstale codebase
bull Check forks for Rails 3 compatibility
Postgres
bull Table locks on
bull Adding updating or removing columns
bull Adding or removing indexes
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time if you avoid default values and constraints
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
can be done without any locking
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time
Postgres
bull Adding fields with defaults
Postgres
bull Adding fields with defaults
before_save assign_defaults
def assign_defaults selfadmin ||= falseend
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
CREATE INDEX CONCURRENTLY users_email_index ON users(email)
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
bull Outside a transaction
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
bull Outside a transaction
class AddIndexToUsers lt ActiveRecordMigration def selfup return if indexes(users)map(ampname)include(users_email_index) add_index users email name =gt users_email_index end
def selfdown remove_index users name =gt users_email_index endend
What about renaming
v1 v2
reads and writes to `foo`
reads and writes to `bar`
What about renaming
v1 v2
reads and writes to `foo`
reads and writes to `bar`
v1rsquo
add column `bar`+
reads from `foo`writes to both`foo` and `bar`
v1rsquorsquo
populate `bar`where needed
What about renaming
v2
reads and writes to `bar`
v2rsquo
ignore `foo`
v3
drop column `foo`
NoSQL
NoSQL
bull Schema migrations O(n) locks are just part of hot compatibility
NoSQL
bull Schema migrations O(n) locks are just part of hot compatibility
bull You still need to put effort into data migration
Workflow
Workflow
bull Make sure pull requests are addressing hot compatibility
Workflow
bull Make sure pull requests are addressing hot compatibility
bull Work after code is merged
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Production
Production
bull Run multiple server processes
Production
bull Run multiple server processes
bull Cycle them individually or in batches
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
bull Cycle them gracefully
Cycling techniques
bull Above the stack
bull Within the stack
bull Platform
Above the stack
bull Use your favorite web server
bull Setup a load balancer above them route around servers that are cycling
Above the stack
bull PROTIP default setup wonrsquot work
bull Configure load balancer to poll for your application health
class HealthController lt ApplicationController def index if Fileexists(Railsrootjoin(MAINTENANCE)) render text =gt Forget me status =gt 503 else render text =gt Im alive end endend
Cycling + Load balancer
bounce serversrm MAINTENANCE
remove server from rotation
touch MAINTENANCE
new servers come online
add server to rotation
Server Load balancer
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy where to check for the health
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
do not let your web server queue requests
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy to run the health check specified above
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
every 2s
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
stop routing to it once the health check failed once
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
and bring it back once it worked once
ELB
$ elb-configure-healthcheck MyLoadBalancer --target HTTP5000health --interval 2 --unhealthy-threshold 1 --healthy-threshold 1
Above the stack
Above the stackbull Minimal impact
Above the stackbull Minimal impact
bull Extended availability comes for free
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
bull Another component you have to track and maintain
Within the stack
Within the stack
Unicorn
bull Replaces your web server
bull Bounces gracefully
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Within the stack
Within the stackbull Easy to setup and deploy
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
bull Doesnrsquot address availability issues
Platform
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 1s bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 5s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 1s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ heroku labsenable preboot
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 starting for 5s bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 up for 1s bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 5s bundle exec thin start -p $PORT
BETA
Heroku
bull Migrations
Heroku
bull Migrations
$ git push pre-production master$ heroku run rake dbmigrate --remote pre-production$ git push production master
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Future
Future
bull Asset pipeline
Future
bull Asset pipeline
bull Mysql tooling
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
bull Can databases
Future
Future
bull Hot compatibility beyond zero downtime deploys
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
Hot compatibility
v1 v1rsquo v2
writes to `notes` drop column `notes`stop writing to `notes`
Hot compatibility
v1 v1rsquo v2
class User def selfcolumns superreject |c| cname == notes endend
writes to `notes` drop column `notes`stop writing to `notes`
Self dependency
Self dependency
user loads form
Self dependency
new form is deployed
Self dependency
user submits the form
POST loginusername=billmicrosoftcomamppassword=q1w2e3
Self dependency
server blows up
Self dependency
bull Make controllers compatible
Self dependency
bull Make controllers compatible
class AuthController lt ApplicationController
protected
def filtered_params paramsduptap do |p| pmerge(email =gt pdelete(username)) end endend
Assets - Rails 2
Server 1
GET dashboard
GET stylesheetsalljs
200
200
Assets - Rails 2
Server 1 Server 2
GET dashboard
GET stylesheetsalljs
200
404
Assets - Rails 2
bull httpsgithubcomddollarasset-resource
Asset pipeline - Rails 3
v1
GET dashboard
GET assetsapplication-v1js
200
200
Asset pipeline - Rails 3
GET dashboard
GET assetsapplication-v1js
200
404
v1 v2
Asset pipeline - Rails 3
Asset pipeline - Rails 3
bull When updating assets keep the existing version around
Asset pipeline - Rails 3
bull When updating assets keep the existing version around
bull Wersquore still in need of tooling to help cleaning up old assets
Asset pipeline - Rails 3
bull When updating assets keep the existing version around
bull Wersquore still in need of tooling to help cleaning up old assets
rake assetscleanoutdated
Migration strategies
Migration strategies
bull Read-only models
Migration strategies
bull Read-only models
class Role lt ActiveRecordBase def readonly true endend
Migration strategies
bull Read-only models
bull Ignore cached columns
Migration strategies
bull Read-only models
bull Ignore cached columns
class User def selfcolumns superreject |c| cname == notes endend
Migration strategies
bull Read-only models
bull Ignore cached columns
bull Know your database
Migration strategies
bull Read-only models
bull Ignore cached columns
bull Know your database
bull Beware of O(n) db locks
MyISAM
MyISAM
InnoDB
InnoDB
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
InnoDB
1 Create a new table with the same structure
2 Apply changes to new table
3 Copy data over
4 Acquire lock on original table
5 Sync newupdateddeleted rows
6 Swap tables
7 Release lock
InnoDB
1 Create a new table with the same structure
2 Apply changes to new table
3 Copy data over
4 Acquire lock on original table
5 Sync newupdateddeleted rows
6 Swap tables
7 Release lock
InnoDBhttpsgithubcomfreelstable_migrator
InnoDBhttpsgithubcomfreelstable_migrator
class AddStuffToMyBigTable lt TableMigration
migrates users
change_table do |t| tinteger foo null =gt false default =gt 0 tstring bar tindex foo name =gt index_for_foo endend
InnoDBhttpsgithubcomfreelstable_migrator
bull Requires acts_as_paranoid
bull Pretty oldstale codebase
bull Check forks for Rails 3 compatibility
Postgres
bull Table locks on
bull Adding updating or removing columns
bull Adding or removing indexes
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time if you avoid default values and constraints
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
can be done without any locking
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time
Postgres
bull Adding fields with defaults
Postgres
bull Adding fields with defaults
before_save assign_defaults
def assign_defaults selfadmin ||= falseend
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
CREATE INDEX CONCURRENTLY users_email_index ON users(email)
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
bull Outside a transaction
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
bull Outside a transaction
class AddIndexToUsers lt ActiveRecordMigration def selfup return if indexes(users)map(ampname)include(users_email_index) add_index users email name =gt users_email_index end
def selfdown remove_index users name =gt users_email_index endend
What about renaming
v1 v2
reads and writes to `foo`
reads and writes to `bar`
What about renaming
v1 v2
reads and writes to `foo`
reads and writes to `bar`
v1rsquo
add column `bar`+
reads from `foo`writes to both`foo` and `bar`
v1rsquorsquo
populate `bar`where needed
What about renaming
v2
reads and writes to `bar`
v2rsquo
ignore `foo`
v3
drop column `foo`
NoSQL
NoSQL
bull Schema migrations O(n) locks are just part of hot compatibility
NoSQL
bull Schema migrations O(n) locks are just part of hot compatibility
bull You still need to put effort into data migration
Workflow
Workflow
bull Make sure pull requests are addressing hot compatibility
Workflow
bull Make sure pull requests are addressing hot compatibility
bull Work after code is merged
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Production
Production
bull Run multiple server processes
Production
bull Run multiple server processes
bull Cycle them individually or in batches
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
bull Cycle them gracefully
Cycling techniques
bull Above the stack
bull Within the stack
bull Platform
Above the stack
bull Use your favorite web server
bull Setup a load balancer above them route around servers that are cycling
Above the stack
bull PROTIP default setup wonrsquot work
bull Configure load balancer to poll for your application health
class HealthController lt ApplicationController def index if Fileexists(Railsrootjoin(MAINTENANCE)) render text =gt Forget me status =gt 503 else render text =gt Im alive end endend
Cycling + Load balancer
bounce serversrm MAINTENANCE
remove server from rotation
touch MAINTENANCE
new servers come online
add server to rotation
Server Load balancer
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy where to check for the health
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
do not let your web server queue requests
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy to run the health check specified above
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
every 2s
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
stop routing to it once the health check failed once
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
and bring it back once it worked once
ELB
$ elb-configure-healthcheck MyLoadBalancer --target HTTP5000health --interval 2 --unhealthy-threshold 1 --healthy-threshold 1
Above the stack
Above the stackbull Minimal impact
Above the stackbull Minimal impact
bull Extended availability comes for free
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
bull Another component you have to track and maintain
Within the stack
Within the stack
Unicorn
bull Replaces your web server
bull Bounces gracefully
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Within the stack
Within the stackbull Easy to setup and deploy
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
bull Doesnrsquot address availability issues
Platform
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 1s bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 5s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 1s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ heroku labsenable preboot
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 starting for 5s bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 up for 1s bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 5s bundle exec thin start -p $PORT
BETA
Heroku
bull Migrations
Heroku
bull Migrations
$ git push pre-production master$ heroku run rake dbmigrate --remote pre-production$ git push production master
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Future
Future
bull Asset pipeline
Future
bull Asset pipeline
bull Mysql tooling
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
bull Can databases
Future
Future
bull Hot compatibility beyond zero downtime deploys
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
Hot compatibility
v1 v1rsquo v2
class User def selfcolumns superreject |c| cname == notes endend
writes to `notes` drop column `notes`stop writing to `notes`
Self dependency
Self dependency
user loads form
Self dependency
new form is deployed
Self dependency
user submits the form
POST loginusername=billmicrosoftcomamppassword=q1w2e3
Self dependency
server blows up
Self dependency
bull Make controllers compatible
Self dependency
bull Make controllers compatible
class AuthController lt ApplicationController
protected
def filtered_params paramsduptap do |p| pmerge(email =gt pdelete(username)) end endend
Assets - Rails 2
Server 1
GET dashboard
GET stylesheetsalljs
200
200
Assets - Rails 2
Server 1 Server 2
GET dashboard
GET stylesheetsalljs
200
404
Assets - Rails 2
bull httpsgithubcomddollarasset-resource
Asset pipeline - Rails 3
v1
GET dashboard
GET assetsapplication-v1js
200
200
Asset pipeline - Rails 3
GET dashboard
GET assetsapplication-v1js
200
404
v1 v2
Asset pipeline - Rails 3
Asset pipeline - Rails 3
bull When updating assets keep the existing version around
Asset pipeline - Rails 3
bull When updating assets keep the existing version around
bull Wersquore still in need of tooling to help cleaning up old assets
Asset pipeline - Rails 3
bull When updating assets keep the existing version around
bull Wersquore still in need of tooling to help cleaning up old assets
rake assetscleanoutdated
Migration strategies
Migration strategies
bull Read-only models
Migration strategies
bull Read-only models
class Role lt ActiveRecordBase def readonly true endend
Migration strategies
bull Read-only models
bull Ignore cached columns
Migration strategies
bull Read-only models
bull Ignore cached columns
class User def selfcolumns superreject |c| cname == notes endend
Migration strategies
bull Read-only models
bull Ignore cached columns
bull Know your database
Migration strategies
bull Read-only models
bull Ignore cached columns
bull Know your database
bull Beware of O(n) db locks
MyISAM
MyISAM
InnoDB
InnoDB
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
InnoDB
1 Create a new table with the same structure
2 Apply changes to new table
3 Copy data over
4 Acquire lock on original table
5 Sync newupdateddeleted rows
6 Swap tables
7 Release lock
InnoDB
1 Create a new table with the same structure
2 Apply changes to new table
3 Copy data over
4 Acquire lock on original table
5 Sync newupdateddeleted rows
6 Swap tables
7 Release lock
InnoDBhttpsgithubcomfreelstable_migrator
InnoDBhttpsgithubcomfreelstable_migrator
class AddStuffToMyBigTable lt TableMigration
migrates users
change_table do |t| tinteger foo null =gt false default =gt 0 tstring bar tindex foo name =gt index_for_foo endend
InnoDBhttpsgithubcomfreelstable_migrator
bull Requires acts_as_paranoid
bull Pretty oldstale codebase
bull Check forks for Rails 3 compatibility
Postgres
bull Table locks on
bull Adding updating or removing columns
bull Adding or removing indexes
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time if you avoid default values and constraints
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
can be done without any locking
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time
Postgres
bull Adding fields with defaults
Postgres
bull Adding fields with defaults
before_save assign_defaults
def assign_defaults selfadmin ||= falseend
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
CREATE INDEX CONCURRENTLY users_email_index ON users(email)
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
bull Outside a transaction
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
bull Outside a transaction
class AddIndexToUsers lt ActiveRecordMigration def selfup return if indexes(users)map(ampname)include(users_email_index) add_index users email name =gt users_email_index end
def selfdown remove_index users name =gt users_email_index endend
What about renaming
v1 v2
reads and writes to `foo`
reads and writes to `bar`
What about renaming
v1 v2
reads and writes to `foo`
reads and writes to `bar`
v1rsquo
add column `bar`+
reads from `foo`writes to both`foo` and `bar`
v1rsquorsquo
populate `bar`where needed
What about renaming
v2
reads and writes to `bar`
v2rsquo
ignore `foo`
v3
drop column `foo`
NoSQL
NoSQL
bull Schema migrations O(n) locks are just part of hot compatibility
NoSQL
bull Schema migrations O(n) locks are just part of hot compatibility
bull You still need to put effort into data migration
Workflow
Workflow
bull Make sure pull requests are addressing hot compatibility
Workflow
bull Make sure pull requests are addressing hot compatibility
bull Work after code is merged
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Production
Production
bull Run multiple server processes
Production
bull Run multiple server processes
bull Cycle them individually or in batches
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
bull Cycle them gracefully
Cycling techniques
bull Above the stack
bull Within the stack
bull Platform
Above the stack
bull Use your favorite web server
bull Setup a load balancer above them route around servers that are cycling
Above the stack
bull PROTIP default setup wonrsquot work
bull Configure load balancer to poll for your application health
class HealthController lt ApplicationController def index if Fileexists(Railsrootjoin(MAINTENANCE)) render text =gt Forget me status =gt 503 else render text =gt Im alive end endend
Cycling + Load balancer
bounce serversrm MAINTENANCE
remove server from rotation
touch MAINTENANCE
new servers come online
add server to rotation
Server Load balancer
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy where to check for the health
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
do not let your web server queue requests
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy to run the health check specified above
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
every 2s
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
stop routing to it once the health check failed once
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
and bring it back once it worked once
ELB
$ elb-configure-healthcheck MyLoadBalancer --target HTTP5000health --interval 2 --unhealthy-threshold 1 --healthy-threshold 1
Above the stack
Above the stackbull Minimal impact
Above the stackbull Minimal impact
bull Extended availability comes for free
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
bull Another component you have to track and maintain
Within the stack
Within the stack
Unicorn
bull Replaces your web server
bull Bounces gracefully
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Within the stack
Within the stackbull Easy to setup and deploy
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
bull Doesnrsquot address availability issues
Platform
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 1s bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 5s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 1s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ heroku labsenable preboot
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 starting for 5s bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 up for 1s bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 5s bundle exec thin start -p $PORT
BETA
Heroku
bull Migrations
Heroku
bull Migrations
$ git push pre-production master$ heroku run rake dbmigrate --remote pre-production$ git push production master
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Future
Future
bull Asset pipeline
Future
bull Asset pipeline
bull Mysql tooling
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
bull Can databases
Future
Future
bull Hot compatibility beyond zero downtime deploys
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
Self dependency
Self dependency
user loads form
Self dependency
new form is deployed
Self dependency
user submits the form
POST loginusername=billmicrosoftcomamppassword=q1w2e3
Self dependency
server blows up
Self dependency
bull Make controllers compatible
Self dependency
bull Make controllers compatible
class AuthController lt ApplicationController
protected
def filtered_params paramsduptap do |p| pmerge(email =gt pdelete(username)) end endend
Assets - Rails 2
Server 1
GET dashboard
GET stylesheetsalljs
200
200
Assets - Rails 2
Server 1 Server 2
GET dashboard
GET stylesheetsalljs
200
404
Assets - Rails 2
bull httpsgithubcomddollarasset-resource
Asset pipeline - Rails 3
v1
GET dashboard
GET assetsapplication-v1js
200
200
Asset pipeline - Rails 3
GET dashboard
GET assetsapplication-v1js
200
404
v1 v2
Asset pipeline - Rails 3
Asset pipeline - Rails 3
bull When updating assets keep the existing version around
Asset pipeline - Rails 3
bull When updating assets keep the existing version around
bull Wersquore still in need of tooling to help cleaning up old assets
Asset pipeline - Rails 3
bull When updating assets keep the existing version around
bull Wersquore still in need of tooling to help cleaning up old assets
rake assetscleanoutdated
Migration strategies
Migration strategies
bull Read-only models
Migration strategies
bull Read-only models
class Role lt ActiveRecordBase def readonly true endend
Migration strategies
bull Read-only models
bull Ignore cached columns
Migration strategies
bull Read-only models
bull Ignore cached columns
class User def selfcolumns superreject |c| cname == notes endend
Migration strategies
bull Read-only models
bull Ignore cached columns
bull Know your database
Migration strategies
bull Read-only models
bull Ignore cached columns
bull Know your database
bull Beware of O(n) db locks
MyISAM
MyISAM
InnoDB
InnoDB
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
InnoDB
1 Create a new table with the same structure
2 Apply changes to new table
3 Copy data over
4 Acquire lock on original table
5 Sync newupdateddeleted rows
6 Swap tables
7 Release lock
InnoDB
1 Create a new table with the same structure
2 Apply changes to new table
3 Copy data over
4 Acquire lock on original table
5 Sync newupdateddeleted rows
6 Swap tables
7 Release lock
InnoDBhttpsgithubcomfreelstable_migrator
InnoDBhttpsgithubcomfreelstable_migrator
class AddStuffToMyBigTable lt TableMigration
migrates users
change_table do |t| tinteger foo null =gt false default =gt 0 tstring bar tindex foo name =gt index_for_foo endend
InnoDBhttpsgithubcomfreelstable_migrator
bull Requires acts_as_paranoid
bull Pretty oldstale codebase
bull Check forks for Rails 3 compatibility
Postgres
bull Table locks on
bull Adding updating or removing columns
bull Adding or removing indexes
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time if you avoid default values and constraints
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
can be done without any locking
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time
Postgres
bull Adding fields with defaults
Postgres
bull Adding fields with defaults
before_save assign_defaults
def assign_defaults selfadmin ||= falseend
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
CREATE INDEX CONCURRENTLY users_email_index ON users(email)
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
bull Outside a transaction
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
bull Outside a transaction
class AddIndexToUsers lt ActiveRecordMigration def selfup return if indexes(users)map(ampname)include(users_email_index) add_index users email name =gt users_email_index end
def selfdown remove_index users name =gt users_email_index endend
What about renaming
v1 v2
reads and writes to `foo`
reads and writes to `bar`
What about renaming
v1 v2
reads and writes to `foo`
reads and writes to `bar`
v1rsquo
add column `bar`+
reads from `foo`writes to both`foo` and `bar`
v1rsquorsquo
populate `bar`where needed
What about renaming
v2
reads and writes to `bar`
v2rsquo
ignore `foo`
v3
drop column `foo`
NoSQL
NoSQL
bull Schema migrations O(n) locks are just part of hot compatibility
NoSQL
bull Schema migrations O(n) locks are just part of hot compatibility
bull You still need to put effort into data migration
Workflow
Workflow
bull Make sure pull requests are addressing hot compatibility
Workflow
bull Make sure pull requests are addressing hot compatibility
bull Work after code is merged
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Production
Production
bull Run multiple server processes
Production
bull Run multiple server processes
bull Cycle them individually or in batches
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
bull Cycle them gracefully
Cycling techniques
bull Above the stack
bull Within the stack
bull Platform
Above the stack
bull Use your favorite web server
bull Setup a load balancer above them route around servers that are cycling
Above the stack
bull PROTIP default setup wonrsquot work
bull Configure load balancer to poll for your application health
class HealthController lt ApplicationController def index if Fileexists(Railsrootjoin(MAINTENANCE)) render text =gt Forget me status =gt 503 else render text =gt Im alive end endend
Cycling + Load balancer
bounce serversrm MAINTENANCE
remove server from rotation
touch MAINTENANCE
new servers come online
add server to rotation
Server Load balancer
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy where to check for the health
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
do not let your web server queue requests
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy to run the health check specified above
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
every 2s
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
stop routing to it once the health check failed once
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
and bring it back once it worked once
ELB
$ elb-configure-healthcheck MyLoadBalancer --target HTTP5000health --interval 2 --unhealthy-threshold 1 --healthy-threshold 1
Above the stack
Above the stackbull Minimal impact
Above the stackbull Minimal impact
bull Extended availability comes for free
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
bull Another component you have to track and maintain
Within the stack
Within the stack
Unicorn
bull Replaces your web server
bull Bounces gracefully
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Within the stack
Within the stackbull Easy to setup and deploy
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
bull Doesnrsquot address availability issues
Platform
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 1s bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 5s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 1s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ heroku labsenable preboot
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 starting for 5s bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 up for 1s bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 5s bundle exec thin start -p $PORT
BETA
Heroku
bull Migrations
Heroku
bull Migrations
$ git push pre-production master$ heroku run rake dbmigrate --remote pre-production$ git push production master
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Future
Future
bull Asset pipeline
Future
bull Asset pipeline
bull Mysql tooling
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
bull Can databases
Future
Future
bull Hot compatibility beyond zero downtime deploys
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
Self dependency
user loads form
Self dependency
new form is deployed
Self dependency
user submits the form
POST loginusername=billmicrosoftcomamppassword=q1w2e3
Self dependency
server blows up
Self dependency
bull Make controllers compatible
Self dependency
bull Make controllers compatible
class AuthController lt ApplicationController
protected
def filtered_params paramsduptap do |p| pmerge(email =gt pdelete(username)) end endend
Assets - Rails 2
Server 1
GET dashboard
GET stylesheetsalljs
200
200
Assets - Rails 2
Server 1 Server 2
GET dashboard
GET stylesheetsalljs
200
404
Assets - Rails 2
bull httpsgithubcomddollarasset-resource
Asset pipeline - Rails 3
v1
GET dashboard
GET assetsapplication-v1js
200
200
Asset pipeline - Rails 3
GET dashboard
GET assetsapplication-v1js
200
404
v1 v2
Asset pipeline - Rails 3
Asset pipeline - Rails 3
bull When updating assets keep the existing version around
Asset pipeline - Rails 3
bull When updating assets keep the existing version around
bull Wersquore still in need of tooling to help cleaning up old assets
Asset pipeline - Rails 3
bull When updating assets keep the existing version around
bull Wersquore still in need of tooling to help cleaning up old assets
rake assetscleanoutdated
Migration strategies
Migration strategies
bull Read-only models
Migration strategies
bull Read-only models
class Role lt ActiveRecordBase def readonly true endend
Migration strategies
bull Read-only models
bull Ignore cached columns
Migration strategies
bull Read-only models
bull Ignore cached columns
class User def selfcolumns superreject |c| cname == notes endend
Migration strategies
bull Read-only models
bull Ignore cached columns
bull Know your database
Migration strategies
bull Read-only models
bull Ignore cached columns
bull Know your database
bull Beware of O(n) db locks
MyISAM
MyISAM
InnoDB
InnoDB
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
InnoDB
1 Create a new table with the same structure
2 Apply changes to new table
3 Copy data over
4 Acquire lock on original table
5 Sync newupdateddeleted rows
6 Swap tables
7 Release lock
InnoDB
1 Create a new table with the same structure
2 Apply changes to new table
3 Copy data over
4 Acquire lock on original table
5 Sync newupdateddeleted rows
6 Swap tables
7 Release lock
InnoDBhttpsgithubcomfreelstable_migrator
InnoDBhttpsgithubcomfreelstable_migrator
class AddStuffToMyBigTable lt TableMigration
migrates users
change_table do |t| tinteger foo null =gt false default =gt 0 tstring bar tindex foo name =gt index_for_foo endend
InnoDBhttpsgithubcomfreelstable_migrator
bull Requires acts_as_paranoid
bull Pretty oldstale codebase
bull Check forks for Rails 3 compatibility
Postgres
bull Table locks on
bull Adding updating or removing columns
bull Adding or removing indexes
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time if you avoid default values and constraints
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
can be done without any locking
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time
Postgres
bull Adding fields with defaults
Postgres
bull Adding fields with defaults
before_save assign_defaults
def assign_defaults selfadmin ||= falseend
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
CREATE INDEX CONCURRENTLY users_email_index ON users(email)
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
bull Outside a transaction
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
bull Outside a transaction
class AddIndexToUsers lt ActiveRecordMigration def selfup return if indexes(users)map(ampname)include(users_email_index) add_index users email name =gt users_email_index end
def selfdown remove_index users name =gt users_email_index endend
What about renaming
v1 v2
reads and writes to `foo`
reads and writes to `bar`
What about renaming
v1 v2
reads and writes to `foo`
reads and writes to `bar`
v1rsquo
add column `bar`+
reads from `foo`writes to both`foo` and `bar`
v1rsquorsquo
populate `bar`where needed
What about renaming
v2
reads and writes to `bar`
v2rsquo
ignore `foo`
v3
drop column `foo`
NoSQL
NoSQL
bull Schema migrations O(n) locks are just part of hot compatibility
NoSQL
bull Schema migrations O(n) locks are just part of hot compatibility
bull You still need to put effort into data migration
Workflow
Workflow
bull Make sure pull requests are addressing hot compatibility
Workflow
bull Make sure pull requests are addressing hot compatibility
bull Work after code is merged
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Production
Production
bull Run multiple server processes
Production
bull Run multiple server processes
bull Cycle them individually or in batches
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
bull Cycle them gracefully
Cycling techniques
bull Above the stack
bull Within the stack
bull Platform
Above the stack
bull Use your favorite web server
bull Setup a load balancer above them route around servers that are cycling
Above the stack
bull PROTIP default setup wonrsquot work
bull Configure load balancer to poll for your application health
class HealthController lt ApplicationController def index if Fileexists(Railsrootjoin(MAINTENANCE)) render text =gt Forget me status =gt 503 else render text =gt Im alive end endend
Cycling + Load balancer
bounce serversrm MAINTENANCE
remove server from rotation
touch MAINTENANCE
new servers come online
add server to rotation
Server Load balancer
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy where to check for the health
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
do not let your web server queue requests
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy to run the health check specified above
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
every 2s
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
stop routing to it once the health check failed once
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
and bring it back once it worked once
ELB
$ elb-configure-healthcheck MyLoadBalancer --target HTTP5000health --interval 2 --unhealthy-threshold 1 --healthy-threshold 1
Above the stack
Above the stackbull Minimal impact
Above the stackbull Minimal impact
bull Extended availability comes for free
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
bull Another component you have to track and maintain
Within the stack
Within the stack
Unicorn
bull Replaces your web server
bull Bounces gracefully
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Within the stack
Within the stackbull Easy to setup and deploy
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
bull Doesnrsquot address availability issues
Platform
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 1s bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 5s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 1s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ heroku labsenable preboot
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 starting for 5s bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 up for 1s bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 5s bundle exec thin start -p $PORT
BETA
Heroku
bull Migrations
Heroku
bull Migrations
$ git push pre-production master$ heroku run rake dbmigrate --remote pre-production$ git push production master
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Future
Future
bull Asset pipeline
Future
bull Asset pipeline
bull Mysql tooling
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
bull Can databases
Future
Future
bull Hot compatibility beyond zero downtime deploys
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
Self dependency
new form is deployed
Self dependency
user submits the form
POST loginusername=billmicrosoftcomamppassword=q1w2e3
Self dependency
server blows up
Self dependency
bull Make controllers compatible
Self dependency
bull Make controllers compatible
class AuthController lt ApplicationController
protected
def filtered_params paramsduptap do |p| pmerge(email =gt pdelete(username)) end endend
Assets - Rails 2
Server 1
GET dashboard
GET stylesheetsalljs
200
200
Assets - Rails 2
Server 1 Server 2
GET dashboard
GET stylesheetsalljs
200
404
Assets - Rails 2
bull httpsgithubcomddollarasset-resource
Asset pipeline - Rails 3
v1
GET dashboard
GET assetsapplication-v1js
200
200
Asset pipeline - Rails 3
GET dashboard
GET assetsapplication-v1js
200
404
v1 v2
Asset pipeline - Rails 3
Asset pipeline - Rails 3
bull When updating assets keep the existing version around
Asset pipeline - Rails 3
bull When updating assets keep the existing version around
bull Wersquore still in need of tooling to help cleaning up old assets
Asset pipeline - Rails 3
bull When updating assets keep the existing version around
bull Wersquore still in need of tooling to help cleaning up old assets
rake assetscleanoutdated
Migration strategies
Migration strategies
bull Read-only models
Migration strategies
bull Read-only models
class Role lt ActiveRecordBase def readonly true endend
Migration strategies
bull Read-only models
bull Ignore cached columns
Migration strategies
bull Read-only models
bull Ignore cached columns
class User def selfcolumns superreject |c| cname == notes endend
Migration strategies
bull Read-only models
bull Ignore cached columns
bull Know your database
Migration strategies
bull Read-only models
bull Ignore cached columns
bull Know your database
bull Beware of O(n) db locks
MyISAM
MyISAM
InnoDB
InnoDB
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
InnoDB
1 Create a new table with the same structure
2 Apply changes to new table
3 Copy data over
4 Acquire lock on original table
5 Sync newupdateddeleted rows
6 Swap tables
7 Release lock
InnoDB
1 Create a new table with the same structure
2 Apply changes to new table
3 Copy data over
4 Acquire lock on original table
5 Sync newupdateddeleted rows
6 Swap tables
7 Release lock
InnoDBhttpsgithubcomfreelstable_migrator
InnoDBhttpsgithubcomfreelstable_migrator
class AddStuffToMyBigTable lt TableMigration
migrates users
change_table do |t| tinteger foo null =gt false default =gt 0 tstring bar tindex foo name =gt index_for_foo endend
InnoDBhttpsgithubcomfreelstable_migrator
bull Requires acts_as_paranoid
bull Pretty oldstale codebase
bull Check forks for Rails 3 compatibility
Postgres
bull Table locks on
bull Adding updating or removing columns
bull Adding or removing indexes
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time if you avoid default values and constraints
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
can be done without any locking
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time
Postgres
bull Adding fields with defaults
Postgres
bull Adding fields with defaults
before_save assign_defaults
def assign_defaults selfadmin ||= falseend
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
CREATE INDEX CONCURRENTLY users_email_index ON users(email)
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
bull Outside a transaction
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
bull Outside a transaction
class AddIndexToUsers lt ActiveRecordMigration def selfup return if indexes(users)map(ampname)include(users_email_index) add_index users email name =gt users_email_index end
def selfdown remove_index users name =gt users_email_index endend
What about renaming
v1 v2
reads and writes to `foo`
reads and writes to `bar`
What about renaming
v1 v2
reads and writes to `foo`
reads and writes to `bar`
v1rsquo
add column `bar`+
reads from `foo`writes to both`foo` and `bar`
v1rsquorsquo
populate `bar`where needed
What about renaming
v2
reads and writes to `bar`
v2rsquo
ignore `foo`
v3
drop column `foo`
NoSQL
NoSQL
bull Schema migrations O(n) locks are just part of hot compatibility
NoSQL
bull Schema migrations O(n) locks are just part of hot compatibility
bull You still need to put effort into data migration
Workflow
Workflow
bull Make sure pull requests are addressing hot compatibility
Workflow
bull Make sure pull requests are addressing hot compatibility
bull Work after code is merged
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Production
Production
bull Run multiple server processes
Production
bull Run multiple server processes
bull Cycle them individually or in batches
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
bull Cycle them gracefully
Cycling techniques
bull Above the stack
bull Within the stack
bull Platform
Above the stack
bull Use your favorite web server
bull Setup a load balancer above them route around servers that are cycling
Above the stack
bull PROTIP default setup wonrsquot work
bull Configure load balancer to poll for your application health
class HealthController lt ApplicationController def index if Fileexists(Railsrootjoin(MAINTENANCE)) render text =gt Forget me status =gt 503 else render text =gt Im alive end endend
Cycling + Load balancer
bounce serversrm MAINTENANCE
remove server from rotation
touch MAINTENANCE
new servers come online
add server to rotation
Server Load balancer
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy where to check for the health
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
do not let your web server queue requests
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy to run the health check specified above
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
every 2s
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
stop routing to it once the health check failed once
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
and bring it back once it worked once
ELB
$ elb-configure-healthcheck MyLoadBalancer --target HTTP5000health --interval 2 --unhealthy-threshold 1 --healthy-threshold 1
Above the stack
Above the stackbull Minimal impact
Above the stackbull Minimal impact
bull Extended availability comes for free
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
bull Another component you have to track and maintain
Within the stack
Within the stack
Unicorn
bull Replaces your web server
bull Bounces gracefully
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Within the stack
Within the stackbull Easy to setup and deploy
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
bull Doesnrsquot address availability issues
Platform
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 1s bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 5s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 1s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ heroku labsenable preboot
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 starting for 5s bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 up for 1s bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 5s bundle exec thin start -p $PORT
BETA
Heroku
bull Migrations
Heroku
bull Migrations
$ git push pre-production master$ heroku run rake dbmigrate --remote pre-production$ git push production master
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Future
Future
bull Asset pipeline
Future
bull Asset pipeline
bull Mysql tooling
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
bull Can databases
Future
Future
bull Hot compatibility beyond zero downtime deploys
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
Self dependency
user submits the form
POST loginusername=billmicrosoftcomamppassword=q1w2e3
Self dependency
server blows up
Self dependency
bull Make controllers compatible
Self dependency
bull Make controllers compatible
class AuthController lt ApplicationController
protected
def filtered_params paramsduptap do |p| pmerge(email =gt pdelete(username)) end endend
Assets - Rails 2
Server 1
GET dashboard
GET stylesheetsalljs
200
200
Assets - Rails 2
Server 1 Server 2
GET dashboard
GET stylesheetsalljs
200
404
Assets - Rails 2
bull httpsgithubcomddollarasset-resource
Asset pipeline - Rails 3
v1
GET dashboard
GET assetsapplication-v1js
200
200
Asset pipeline - Rails 3
GET dashboard
GET assetsapplication-v1js
200
404
v1 v2
Asset pipeline - Rails 3
Asset pipeline - Rails 3
bull When updating assets keep the existing version around
Asset pipeline - Rails 3
bull When updating assets keep the existing version around
bull Wersquore still in need of tooling to help cleaning up old assets
Asset pipeline - Rails 3
bull When updating assets keep the existing version around
bull Wersquore still in need of tooling to help cleaning up old assets
rake assetscleanoutdated
Migration strategies
Migration strategies
bull Read-only models
Migration strategies
bull Read-only models
class Role lt ActiveRecordBase def readonly true endend
Migration strategies
bull Read-only models
bull Ignore cached columns
Migration strategies
bull Read-only models
bull Ignore cached columns
class User def selfcolumns superreject |c| cname == notes endend
Migration strategies
bull Read-only models
bull Ignore cached columns
bull Know your database
Migration strategies
bull Read-only models
bull Ignore cached columns
bull Know your database
bull Beware of O(n) db locks
MyISAM
MyISAM
InnoDB
InnoDB
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
InnoDB
1 Create a new table with the same structure
2 Apply changes to new table
3 Copy data over
4 Acquire lock on original table
5 Sync newupdateddeleted rows
6 Swap tables
7 Release lock
InnoDB
1 Create a new table with the same structure
2 Apply changes to new table
3 Copy data over
4 Acquire lock on original table
5 Sync newupdateddeleted rows
6 Swap tables
7 Release lock
InnoDBhttpsgithubcomfreelstable_migrator
InnoDBhttpsgithubcomfreelstable_migrator
class AddStuffToMyBigTable lt TableMigration
migrates users
change_table do |t| tinteger foo null =gt false default =gt 0 tstring bar tindex foo name =gt index_for_foo endend
InnoDBhttpsgithubcomfreelstable_migrator
bull Requires acts_as_paranoid
bull Pretty oldstale codebase
bull Check forks for Rails 3 compatibility
Postgres
bull Table locks on
bull Adding updating or removing columns
bull Adding or removing indexes
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time if you avoid default values and constraints
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
can be done without any locking
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time
Postgres
bull Adding fields with defaults
Postgres
bull Adding fields with defaults
before_save assign_defaults
def assign_defaults selfadmin ||= falseend
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
CREATE INDEX CONCURRENTLY users_email_index ON users(email)
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
bull Outside a transaction
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
bull Outside a transaction
class AddIndexToUsers lt ActiveRecordMigration def selfup return if indexes(users)map(ampname)include(users_email_index) add_index users email name =gt users_email_index end
def selfdown remove_index users name =gt users_email_index endend
What about renaming
v1 v2
reads and writes to `foo`
reads and writes to `bar`
What about renaming
v1 v2
reads and writes to `foo`
reads and writes to `bar`
v1rsquo
add column `bar`+
reads from `foo`writes to both`foo` and `bar`
v1rsquorsquo
populate `bar`where needed
What about renaming
v2
reads and writes to `bar`
v2rsquo
ignore `foo`
v3
drop column `foo`
NoSQL
NoSQL
bull Schema migrations O(n) locks are just part of hot compatibility
NoSQL
bull Schema migrations O(n) locks are just part of hot compatibility
bull You still need to put effort into data migration
Workflow
Workflow
bull Make sure pull requests are addressing hot compatibility
Workflow
bull Make sure pull requests are addressing hot compatibility
bull Work after code is merged
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Production
Production
bull Run multiple server processes
Production
bull Run multiple server processes
bull Cycle them individually or in batches
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
bull Cycle them gracefully
Cycling techniques
bull Above the stack
bull Within the stack
bull Platform
Above the stack
bull Use your favorite web server
bull Setup a load balancer above them route around servers that are cycling
Above the stack
bull PROTIP default setup wonrsquot work
bull Configure load balancer to poll for your application health
class HealthController lt ApplicationController def index if Fileexists(Railsrootjoin(MAINTENANCE)) render text =gt Forget me status =gt 503 else render text =gt Im alive end endend
Cycling + Load balancer
bounce serversrm MAINTENANCE
remove server from rotation
touch MAINTENANCE
new servers come online
add server to rotation
Server Load balancer
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy where to check for the health
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
do not let your web server queue requests
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy to run the health check specified above
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
every 2s
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
stop routing to it once the health check failed once
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
and bring it back once it worked once
ELB
$ elb-configure-healthcheck MyLoadBalancer --target HTTP5000health --interval 2 --unhealthy-threshold 1 --healthy-threshold 1
Above the stack
Above the stackbull Minimal impact
Above the stackbull Minimal impact
bull Extended availability comes for free
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
bull Another component you have to track and maintain
Within the stack
Within the stack
Unicorn
bull Replaces your web server
bull Bounces gracefully
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Within the stack
Within the stackbull Easy to setup and deploy
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
bull Doesnrsquot address availability issues
Platform
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 1s bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 5s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 1s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ heroku labsenable preboot
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 starting for 5s bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 up for 1s bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 5s bundle exec thin start -p $PORT
BETA
Heroku
bull Migrations
Heroku
bull Migrations
$ git push pre-production master$ heroku run rake dbmigrate --remote pre-production$ git push production master
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Future
Future
bull Asset pipeline
Future
bull Asset pipeline
bull Mysql tooling
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
bull Can databases
Future
Future
bull Hot compatibility beyond zero downtime deploys
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
Self dependency
server blows up
Self dependency
bull Make controllers compatible
Self dependency
bull Make controllers compatible
class AuthController lt ApplicationController
protected
def filtered_params paramsduptap do |p| pmerge(email =gt pdelete(username)) end endend
Assets - Rails 2
Server 1
GET dashboard
GET stylesheetsalljs
200
200
Assets - Rails 2
Server 1 Server 2
GET dashboard
GET stylesheetsalljs
200
404
Assets - Rails 2
bull httpsgithubcomddollarasset-resource
Asset pipeline - Rails 3
v1
GET dashboard
GET assetsapplication-v1js
200
200
Asset pipeline - Rails 3
GET dashboard
GET assetsapplication-v1js
200
404
v1 v2
Asset pipeline - Rails 3
Asset pipeline - Rails 3
bull When updating assets keep the existing version around
Asset pipeline - Rails 3
bull When updating assets keep the existing version around
bull Wersquore still in need of tooling to help cleaning up old assets
Asset pipeline - Rails 3
bull When updating assets keep the existing version around
bull Wersquore still in need of tooling to help cleaning up old assets
rake assetscleanoutdated
Migration strategies
Migration strategies
bull Read-only models
Migration strategies
bull Read-only models
class Role lt ActiveRecordBase def readonly true endend
Migration strategies
bull Read-only models
bull Ignore cached columns
Migration strategies
bull Read-only models
bull Ignore cached columns
class User def selfcolumns superreject |c| cname == notes endend
Migration strategies
bull Read-only models
bull Ignore cached columns
bull Know your database
Migration strategies
bull Read-only models
bull Ignore cached columns
bull Know your database
bull Beware of O(n) db locks
MyISAM
MyISAM
InnoDB
InnoDB
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
InnoDB
1 Create a new table with the same structure
2 Apply changes to new table
3 Copy data over
4 Acquire lock on original table
5 Sync newupdateddeleted rows
6 Swap tables
7 Release lock
InnoDB
1 Create a new table with the same structure
2 Apply changes to new table
3 Copy data over
4 Acquire lock on original table
5 Sync newupdateddeleted rows
6 Swap tables
7 Release lock
InnoDBhttpsgithubcomfreelstable_migrator
InnoDBhttpsgithubcomfreelstable_migrator
class AddStuffToMyBigTable lt TableMigration
migrates users
change_table do |t| tinteger foo null =gt false default =gt 0 tstring bar tindex foo name =gt index_for_foo endend
InnoDBhttpsgithubcomfreelstable_migrator
bull Requires acts_as_paranoid
bull Pretty oldstale codebase
bull Check forks for Rails 3 compatibility
Postgres
bull Table locks on
bull Adding updating or removing columns
bull Adding or removing indexes
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time if you avoid default values and constraints
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
can be done without any locking
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time
Postgres
bull Adding fields with defaults
Postgres
bull Adding fields with defaults
before_save assign_defaults
def assign_defaults selfadmin ||= falseend
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
CREATE INDEX CONCURRENTLY users_email_index ON users(email)
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
bull Outside a transaction
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
bull Outside a transaction
class AddIndexToUsers lt ActiveRecordMigration def selfup return if indexes(users)map(ampname)include(users_email_index) add_index users email name =gt users_email_index end
def selfdown remove_index users name =gt users_email_index endend
What about renaming
v1 v2
reads and writes to `foo`
reads and writes to `bar`
What about renaming
v1 v2
reads and writes to `foo`
reads and writes to `bar`
v1rsquo
add column `bar`+
reads from `foo`writes to both`foo` and `bar`
v1rsquorsquo
populate `bar`where needed
What about renaming
v2
reads and writes to `bar`
v2rsquo
ignore `foo`
v3
drop column `foo`
NoSQL
NoSQL
bull Schema migrations O(n) locks are just part of hot compatibility
NoSQL
bull Schema migrations O(n) locks are just part of hot compatibility
bull You still need to put effort into data migration
Workflow
Workflow
bull Make sure pull requests are addressing hot compatibility
Workflow
bull Make sure pull requests are addressing hot compatibility
bull Work after code is merged
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Production
Production
bull Run multiple server processes
Production
bull Run multiple server processes
bull Cycle them individually or in batches
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
bull Cycle them gracefully
Cycling techniques
bull Above the stack
bull Within the stack
bull Platform
Above the stack
bull Use your favorite web server
bull Setup a load balancer above them route around servers that are cycling
Above the stack
bull PROTIP default setup wonrsquot work
bull Configure load balancer to poll for your application health
class HealthController lt ApplicationController def index if Fileexists(Railsrootjoin(MAINTENANCE)) render text =gt Forget me status =gt 503 else render text =gt Im alive end endend
Cycling + Load balancer
bounce serversrm MAINTENANCE
remove server from rotation
touch MAINTENANCE
new servers come online
add server to rotation
Server Load balancer
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy where to check for the health
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
do not let your web server queue requests
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy to run the health check specified above
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
every 2s
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
stop routing to it once the health check failed once
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
and bring it back once it worked once
ELB
$ elb-configure-healthcheck MyLoadBalancer --target HTTP5000health --interval 2 --unhealthy-threshold 1 --healthy-threshold 1
Above the stack
Above the stackbull Minimal impact
Above the stackbull Minimal impact
bull Extended availability comes for free
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
bull Another component you have to track and maintain
Within the stack
Within the stack
Unicorn
bull Replaces your web server
bull Bounces gracefully
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Within the stack
Within the stackbull Easy to setup and deploy
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
bull Doesnrsquot address availability issues
Platform
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 1s bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 5s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 1s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ heroku labsenable preboot
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 starting for 5s bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 up for 1s bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 5s bundle exec thin start -p $PORT
BETA
Heroku
bull Migrations
Heroku
bull Migrations
$ git push pre-production master$ heroku run rake dbmigrate --remote pre-production$ git push production master
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Future
Future
bull Asset pipeline
Future
bull Asset pipeline
bull Mysql tooling
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
bull Can databases
Future
Future
bull Hot compatibility beyond zero downtime deploys
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
Self dependency
bull Make controllers compatible
Self dependency
bull Make controllers compatible
class AuthController lt ApplicationController
protected
def filtered_params paramsduptap do |p| pmerge(email =gt pdelete(username)) end endend
Assets - Rails 2
Server 1
GET dashboard
GET stylesheetsalljs
200
200
Assets - Rails 2
Server 1 Server 2
GET dashboard
GET stylesheetsalljs
200
404
Assets - Rails 2
bull httpsgithubcomddollarasset-resource
Asset pipeline - Rails 3
v1
GET dashboard
GET assetsapplication-v1js
200
200
Asset pipeline - Rails 3
GET dashboard
GET assetsapplication-v1js
200
404
v1 v2
Asset pipeline - Rails 3
Asset pipeline - Rails 3
bull When updating assets keep the existing version around
Asset pipeline - Rails 3
bull When updating assets keep the existing version around
bull Wersquore still in need of tooling to help cleaning up old assets
Asset pipeline - Rails 3
bull When updating assets keep the existing version around
bull Wersquore still in need of tooling to help cleaning up old assets
rake assetscleanoutdated
Migration strategies
Migration strategies
bull Read-only models
Migration strategies
bull Read-only models
class Role lt ActiveRecordBase def readonly true endend
Migration strategies
bull Read-only models
bull Ignore cached columns
Migration strategies
bull Read-only models
bull Ignore cached columns
class User def selfcolumns superreject |c| cname == notes endend
Migration strategies
bull Read-only models
bull Ignore cached columns
bull Know your database
Migration strategies
bull Read-only models
bull Ignore cached columns
bull Know your database
bull Beware of O(n) db locks
MyISAM
MyISAM
InnoDB
InnoDB
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
InnoDB
1 Create a new table with the same structure
2 Apply changes to new table
3 Copy data over
4 Acquire lock on original table
5 Sync newupdateddeleted rows
6 Swap tables
7 Release lock
InnoDB
1 Create a new table with the same structure
2 Apply changes to new table
3 Copy data over
4 Acquire lock on original table
5 Sync newupdateddeleted rows
6 Swap tables
7 Release lock
InnoDBhttpsgithubcomfreelstable_migrator
InnoDBhttpsgithubcomfreelstable_migrator
class AddStuffToMyBigTable lt TableMigration
migrates users
change_table do |t| tinteger foo null =gt false default =gt 0 tstring bar tindex foo name =gt index_for_foo endend
InnoDBhttpsgithubcomfreelstable_migrator
bull Requires acts_as_paranoid
bull Pretty oldstale codebase
bull Check forks for Rails 3 compatibility
Postgres
bull Table locks on
bull Adding updating or removing columns
bull Adding or removing indexes
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time if you avoid default values and constraints
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
can be done without any locking
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time
Postgres
bull Adding fields with defaults
Postgres
bull Adding fields with defaults
before_save assign_defaults
def assign_defaults selfadmin ||= falseend
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
CREATE INDEX CONCURRENTLY users_email_index ON users(email)
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
bull Outside a transaction
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
bull Outside a transaction
class AddIndexToUsers lt ActiveRecordMigration def selfup return if indexes(users)map(ampname)include(users_email_index) add_index users email name =gt users_email_index end
def selfdown remove_index users name =gt users_email_index endend
What about renaming
v1 v2
reads and writes to `foo`
reads and writes to `bar`
What about renaming
v1 v2
reads and writes to `foo`
reads and writes to `bar`
v1rsquo
add column `bar`+
reads from `foo`writes to both`foo` and `bar`
v1rsquorsquo
populate `bar`where needed
What about renaming
v2
reads and writes to `bar`
v2rsquo
ignore `foo`
v3
drop column `foo`
NoSQL
NoSQL
bull Schema migrations O(n) locks are just part of hot compatibility
NoSQL
bull Schema migrations O(n) locks are just part of hot compatibility
bull You still need to put effort into data migration
Workflow
Workflow
bull Make sure pull requests are addressing hot compatibility
Workflow
bull Make sure pull requests are addressing hot compatibility
bull Work after code is merged
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Production
Production
bull Run multiple server processes
Production
bull Run multiple server processes
bull Cycle them individually or in batches
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
bull Cycle them gracefully
Cycling techniques
bull Above the stack
bull Within the stack
bull Platform
Above the stack
bull Use your favorite web server
bull Setup a load balancer above them route around servers that are cycling
Above the stack
bull PROTIP default setup wonrsquot work
bull Configure load balancer to poll for your application health
class HealthController lt ApplicationController def index if Fileexists(Railsrootjoin(MAINTENANCE)) render text =gt Forget me status =gt 503 else render text =gt Im alive end endend
Cycling + Load balancer
bounce serversrm MAINTENANCE
remove server from rotation
touch MAINTENANCE
new servers come online
add server to rotation
Server Load balancer
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy where to check for the health
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
do not let your web server queue requests
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy to run the health check specified above
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
every 2s
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
stop routing to it once the health check failed once
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
and bring it back once it worked once
ELB
$ elb-configure-healthcheck MyLoadBalancer --target HTTP5000health --interval 2 --unhealthy-threshold 1 --healthy-threshold 1
Above the stack
Above the stackbull Minimal impact
Above the stackbull Minimal impact
bull Extended availability comes for free
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
bull Another component you have to track and maintain
Within the stack
Within the stack
Unicorn
bull Replaces your web server
bull Bounces gracefully
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Within the stack
Within the stackbull Easy to setup and deploy
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
bull Doesnrsquot address availability issues
Platform
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 1s bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 5s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 1s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ heroku labsenable preboot
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 starting for 5s bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 up for 1s bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 5s bundle exec thin start -p $PORT
BETA
Heroku
bull Migrations
Heroku
bull Migrations
$ git push pre-production master$ heroku run rake dbmigrate --remote pre-production$ git push production master
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Future
Future
bull Asset pipeline
Future
bull Asset pipeline
bull Mysql tooling
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
bull Can databases
Future
Future
bull Hot compatibility beyond zero downtime deploys
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
Self dependency
bull Make controllers compatible
class AuthController lt ApplicationController
protected
def filtered_params paramsduptap do |p| pmerge(email =gt pdelete(username)) end endend
Assets - Rails 2
Server 1
GET dashboard
GET stylesheetsalljs
200
200
Assets - Rails 2
Server 1 Server 2
GET dashboard
GET stylesheetsalljs
200
404
Assets - Rails 2
bull httpsgithubcomddollarasset-resource
Asset pipeline - Rails 3
v1
GET dashboard
GET assetsapplication-v1js
200
200
Asset pipeline - Rails 3
GET dashboard
GET assetsapplication-v1js
200
404
v1 v2
Asset pipeline - Rails 3
Asset pipeline - Rails 3
bull When updating assets keep the existing version around
Asset pipeline - Rails 3
bull When updating assets keep the existing version around
bull Wersquore still in need of tooling to help cleaning up old assets
Asset pipeline - Rails 3
bull When updating assets keep the existing version around
bull Wersquore still in need of tooling to help cleaning up old assets
rake assetscleanoutdated
Migration strategies
Migration strategies
bull Read-only models
Migration strategies
bull Read-only models
class Role lt ActiveRecordBase def readonly true endend
Migration strategies
bull Read-only models
bull Ignore cached columns
Migration strategies
bull Read-only models
bull Ignore cached columns
class User def selfcolumns superreject |c| cname == notes endend
Migration strategies
bull Read-only models
bull Ignore cached columns
bull Know your database
Migration strategies
bull Read-only models
bull Ignore cached columns
bull Know your database
bull Beware of O(n) db locks
MyISAM
MyISAM
InnoDB
InnoDB
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
InnoDB
1 Create a new table with the same structure
2 Apply changes to new table
3 Copy data over
4 Acquire lock on original table
5 Sync newupdateddeleted rows
6 Swap tables
7 Release lock
InnoDB
1 Create a new table with the same structure
2 Apply changes to new table
3 Copy data over
4 Acquire lock on original table
5 Sync newupdateddeleted rows
6 Swap tables
7 Release lock
InnoDBhttpsgithubcomfreelstable_migrator
InnoDBhttpsgithubcomfreelstable_migrator
class AddStuffToMyBigTable lt TableMigration
migrates users
change_table do |t| tinteger foo null =gt false default =gt 0 tstring bar tindex foo name =gt index_for_foo endend
InnoDBhttpsgithubcomfreelstable_migrator
bull Requires acts_as_paranoid
bull Pretty oldstale codebase
bull Check forks for Rails 3 compatibility
Postgres
bull Table locks on
bull Adding updating or removing columns
bull Adding or removing indexes
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time if you avoid default values and constraints
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
can be done without any locking
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time
Postgres
bull Adding fields with defaults
Postgres
bull Adding fields with defaults
before_save assign_defaults
def assign_defaults selfadmin ||= falseend
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
CREATE INDEX CONCURRENTLY users_email_index ON users(email)
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
bull Outside a transaction
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
bull Outside a transaction
class AddIndexToUsers lt ActiveRecordMigration def selfup return if indexes(users)map(ampname)include(users_email_index) add_index users email name =gt users_email_index end
def selfdown remove_index users name =gt users_email_index endend
What about renaming
v1 v2
reads and writes to `foo`
reads and writes to `bar`
What about renaming
v1 v2
reads and writes to `foo`
reads and writes to `bar`
v1rsquo
add column `bar`+
reads from `foo`writes to both`foo` and `bar`
v1rsquorsquo
populate `bar`where needed
What about renaming
v2
reads and writes to `bar`
v2rsquo
ignore `foo`
v3
drop column `foo`
NoSQL
NoSQL
bull Schema migrations O(n) locks are just part of hot compatibility
NoSQL
bull Schema migrations O(n) locks are just part of hot compatibility
bull You still need to put effort into data migration
Workflow
Workflow
bull Make sure pull requests are addressing hot compatibility
Workflow
bull Make sure pull requests are addressing hot compatibility
bull Work after code is merged
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Production
Production
bull Run multiple server processes
Production
bull Run multiple server processes
bull Cycle them individually or in batches
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
bull Cycle them gracefully
Cycling techniques
bull Above the stack
bull Within the stack
bull Platform
Above the stack
bull Use your favorite web server
bull Setup a load balancer above them route around servers that are cycling
Above the stack
bull PROTIP default setup wonrsquot work
bull Configure load balancer to poll for your application health
class HealthController lt ApplicationController def index if Fileexists(Railsrootjoin(MAINTENANCE)) render text =gt Forget me status =gt 503 else render text =gt Im alive end endend
Cycling + Load balancer
bounce serversrm MAINTENANCE
remove server from rotation
touch MAINTENANCE
new servers come online
add server to rotation
Server Load balancer
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy where to check for the health
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
do not let your web server queue requests
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy to run the health check specified above
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
every 2s
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
stop routing to it once the health check failed once
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
and bring it back once it worked once
ELB
$ elb-configure-healthcheck MyLoadBalancer --target HTTP5000health --interval 2 --unhealthy-threshold 1 --healthy-threshold 1
Above the stack
Above the stackbull Minimal impact
Above the stackbull Minimal impact
bull Extended availability comes for free
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
bull Another component you have to track and maintain
Within the stack
Within the stack
Unicorn
bull Replaces your web server
bull Bounces gracefully
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Within the stack
Within the stackbull Easy to setup and deploy
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
bull Doesnrsquot address availability issues
Platform
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 1s bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 5s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 1s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ heroku labsenable preboot
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 starting for 5s bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 up for 1s bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 5s bundle exec thin start -p $PORT
BETA
Heroku
bull Migrations
Heroku
bull Migrations
$ git push pre-production master$ heroku run rake dbmigrate --remote pre-production$ git push production master
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Future
Future
bull Asset pipeline
Future
bull Asset pipeline
bull Mysql tooling
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
bull Can databases
Future
Future
bull Hot compatibility beyond zero downtime deploys
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
Assets - Rails 2
Server 1
GET dashboard
GET stylesheetsalljs
200
200
Assets - Rails 2
Server 1 Server 2
GET dashboard
GET stylesheetsalljs
200
404
Assets - Rails 2
bull httpsgithubcomddollarasset-resource
Asset pipeline - Rails 3
v1
GET dashboard
GET assetsapplication-v1js
200
200
Asset pipeline - Rails 3
GET dashboard
GET assetsapplication-v1js
200
404
v1 v2
Asset pipeline - Rails 3
Asset pipeline - Rails 3
bull When updating assets keep the existing version around
Asset pipeline - Rails 3
bull When updating assets keep the existing version around
bull Wersquore still in need of tooling to help cleaning up old assets
Asset pipeline - Rails 3
bull When updating assets keep the existing version around
bull Wersquore still in need of tooling to help cleaning up old assets
rake assetscleanoutdated
Migration strategies
Migration strategies
bull Read-only models
Migration strategies
bull Read-only models
class Role lt ActiveRecordBase def readonly true endend
Migration strategies
bull Read-only models
bull Ignore cached columns
Migration strategies
bull Read-only models
bull Ignore cached columns
class User def selfcolumns superreject |c| cname == notes endend
Migration strategies
bull Read-only models
bull Ignore cached columns
bull Know your database
Migration strategies
bull Read-only models
bull Ignore cached columns
bull Know your database
bull Beware of O(n) db locks
MyISAM
MyISAM
InnoDB
InnoDB
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
InnoDB
1 Create a new table with the same structure
2 Apply changes to new table
3 Copy data over
4 Acquire lock on original table
5 Sync newupdateddeleted rows
6 Swap tables
7 Release lock
InnoDB
1 Create a new table with the same structure
2 Apply changes to new table
3 Copy data over
4 Acquire lock on original table
5 Sync newupdateddeleted rows
6 Swap tables
7 Release lock
InnoDBhttpsgithubcomfreelstable_migrator
InnoDBhttpsgithubcomfreelstable_migrator
class AddStuffToMyBigTable lt TableMigration
migrates users
change_table do |t| tinteger foo null =gt false default =gt 0 tstring bar tindex foo name =gt index_for_foo endend
InnoDBhttpsgithubcomfreelstable_migrator
bull Requires acts_as_paranoid
bull Pretty oldstale codebase
bull Check forks for Rails 3 compatibility
Postgres
bull Table locks on
bull Adding updating or removing columns
bull Adding or removing indexes
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time if you avoid default values and constraints
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
can be done without any locking
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time
Postgres
bull Adding fields with defaults
Postgres
bull Adding fields with defaults
before_save assign_defaults
def assign_defaults selfadmin ||= falseend
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
CREATE INDEX CONCURRENTLY users_email_index ON users(email)
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
bull Outside a transaction
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
bull Outside a transaction
class AddIndexToUsers lt ActiveRecordMigration def selfup return if indexes(users)map(ampname)include(users_email_index) add_index users email name =gt users_email_index end
def selfdown remove_index users name =gt users_email_index endend
What about renaming
v1 v2
reads and writes to `foo`
reads and writes to `bar`
What about renaming
v1 v2
reads and writes to `foo`
reads and writes to `bar`
v1rsquo
add column `bar`+
reads from `foo`writes to both`foo` and `bar`
v1rsquorsquo
populate `bar`where needed
What about renaming
v2
reads and writes to `bar`
v2rsquo
ignore `foo`
v3
drop column `foo`
NoSQL
NoSQL
bull Schema migrations O(n) locks are just part of hot compatibility
NoSQL
bull Schema migrations O(n) locks are just part of hot compatibility
bull You still need to put effort into data migration
Workflow
Workflow
bull Make sure pull requests are addressing hot compatibility
Workflow
bull Make sure pull requests are addressing hot compatibility
bull Work after code is merged
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Production
Production
bull Run multiple server processes
Production
bull Run multiple server processes
bull Cycle them individually or in batches
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
bull Cycle them gracefully
Cycling techniques
bull Above the stack
bull Within the stack
bull Platform
Above the stack
bull Use your favorite web server
bull Setup a load balancer above them route around servers that are cycling
Above the stack
bull PROTIP default setup wonrsquot work
bull Configure load balancer to poll for your application health
class HealthController lt ApplicationController def index if Fileexists(Railsrootjoin(MAINTENANCE)) render text =gt Forget me status =gt 503 else render text =gt Im alive end endend
Cycling + Load balancer
bounce serversrm MAINTENANCE
remove server from rotation
touch MAINTENANCE
new servers come online
add server to rotation
Server Load balancer
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy where to check for the health
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
do not let your web server queue requests
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy to run the health check specified above
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
every 2s
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
stop routing to it once the health check failed once
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
and bring it back once it worked once
ELB
$ elb-configure-healthcheck MyLoadBalancer --target HTTP5000health --interval 2 --unhealthy-threshold 1 --healthy-threshold 1
Above the stack
Above the stackbull Minimal impact
Above the stackbull Minimal impact
bull Extended availability comes for free
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
bull Another component you have to track and maintain
Within the stack
Within the stack
Unicorn
bull Replaces your web server
bull Bounces gracefully
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Within the stack
Within the stackbull Easy to setup and deploy
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
bull Doesnrsquot address availability issues
Platform
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 1s bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 5s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 1s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ heroku labsenable preboot
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 starting for 5s bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 up for 1s bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 5s bundle exec thin start -p $PORT
BETA
Heroku
bull Migrations
Heroku
bull Migrations
$ git push pre-production master$ heroku run rake dbmigrate --remote pre-production$ git push production master
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Future
Future
bull Asset pipeline
Future
bull Asset pipeline
bull Mysql tooling
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
bull Can databases
Future
Future
bull Hot compatibility beyond zero downtime deploys
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
Assets - Rails 2
Server 1 Server 2
GET dashboard
GET stylesheetsalljs
200
404
Assets - Rails 2
bull httpsgithubcomddollarasset-resource
Asset pipeline - Rails 3
v1
GET dashboard
GET assetsapplication-v1js
200
200
Asset pipeline - Rails 3
GET dashboard
GET assetsapplication-v1js
200
404
v1 v2
Asset pipeline - Rails 3
Asset pipeline - Rails 3
bull When updating assets keep the existing version around
Asset pipeline - Rails 3
bull When updating assets keep the existing version around
bull Wersquore still in need of tooling to help cleaning up old assets
Asset pipeline - Rails 3
bull When updating assets keep the existing version around
bull Wersquore still in need of tooling to help cleaning up old assets
rake assetscleanoutdated
Migration strategies
Migration strategies
bull Read-only models
Migration strategies
bull Read-only models
class Role lt ActiveRecordBase def readonly true endend
Migration strategies
bull Read-only models
bull Ignore cached columns
Migration strategies
bull Read-only models
bull Ignore cached columns
class User def selfcolumns superreject |c| cname == notes endend
Migration strategies
bull Read-only models
bull Ignore cached columns
bull Know your database
Migration strategies
bull Read-only models
bull Ignore cached columns
bull Know your database
bull Beware of O(n) db locks
MyISAM
MyISAM
InnoDB
InnoDB
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
InnoDB
1 Create a new table with the same structure
2 Apply changes to new table
3 Copy data over
4 Acquire lock on original table
5 Sync newupdateddeleted rows
6 Swap tables
7 Release lock
InnoDB
1 Create a new table with the same structure
2 Apply changes to new table
3 Copy data over
4 Acquire lock on original table
5 Sync newupdateddeleted rows
6 Swap tables
7 Release lock
InnoDBhttpsgithubcomfreelstable_migrator
InnoDBhttpsgithubcomfreelstable_migrator
class AddStuffToMyBigTable lt TableMigration
migrates users
change_table do |t| tinteger foo null =gt false default =gt 0 tstring bar tindex foo name =gt index_for_foo endend
InnoDBhttpsgithubcomfreelstable_migrator
bull Requires acts_as_paranoid
bull Pretty oldstale codebase
bull Check forks for Rails 3 compatibility
Postgres
bull Table locks on
bull Adding updating or removing columns
bull Adding or removing indexes
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time if you avoid default values and constraints
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
can be done without any locking
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time
Postgres
bull Adding fields with defaults
Postgres
bull Adding fields with defaults
before_save assign_defaults
def assign_defaults selfadmin ||= falseend
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
CREATE INDEX CONCURRENTLY users_email_index ON users(email)
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
bull Outside a transaction
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
bull Outside a transaction
class AddIndexToUsers lt ActiveRecordMigration def selfup return if indexes(users)map(ampname)include(users_email_index) add_index users email name =gt users_email_index end
def selfdown remove_index users name =gt users_email_index endend
What about renaming
v1 v2
reads and writes to `foo`
reads and writes to `bar`
What about renaming
v1 v2
reads and writes to `foo`
reads and writes to `bar`
v1rsquo
add column `bar`+
reads from `foo`writes to both`foo` and `bar`
v1rsquorsquo
populate `bar`where needed
What about renaming
v2
reads and writes to `bar`
v2rsquo
ignore `foo`
v3
drop column `foo`
NoSQL
NoSQL
bull Schema migrations O(n) locks are just part of hot compatibility
NoSQL
bull Schema migrations O(n) locks are just part of hot compatibility
bull You still need to put effort into data migration
Workflow
Workflow
bull Make sure pull requests are addressing hot compatibility
Workflow
bull Make sure pull requests are addressing hot compatibility
bull Work after code is merged
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Production
Production
bull Run multiple server processes
Production
bull Run multiple server processes
bull Cycle them individually or in batches
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
bull Cycle them gracefully
Cycling techniques
bull Above the stack
bull Within the stack
bull Platform
Above the stack
bull Use your favorite web server
bull Setup a load balancer above them route around servers that are cycling
Above the stack
bull PROTIP default setup wonrsquot work
bull Configure load balancer to poll for your application health
class HealthController lt ApplicationController def index if Fileexists(Railsrootjoin(MAINTENANCE)) render text =gt Forget me status =gt 503 else render text =gt Im alive end endend
Cycling + Load balancer
bounce serversrm MAINTENANCE
remove server from rotation
touch MAINTENANCE
new servers come online
add server to rotation
Server Load balancer
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy where to check for the health
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
do not let your web server queue requests
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy to run the health check specified above
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
every 2s
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
stop routing to it once the health check failed once
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
and bring it back once it worked once
ELB
$ elb-configure-healthcheck MyLoadBalancer --target HTTP5000health --interval 2 --unhealthy-threshold 1 --healthy-threshold 1
Above the stack
Above the stackbull Minimal impact
Above the stackbull Minimal impact
bull Extended availability comes for free
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
bull Another component you have to track and maintain
Within the stack
Within the stack
Unicorn
bull Replaces your web server
bull Bounces gracefully
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Within the stack
Within the stackbull Easy to setup and deploy
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
bull Doesnrsquot address availability issues
Platform
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 1s bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 5s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 1s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ heroku labsenable preboot
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 starting for 5s bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 up for 1s bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 5s bundle exec thin start -p $PORT
BETA
Heroku
bull Migrations
Heroku
bull Migrations
$ git push pre-production master$ heroku run rake dbmigrate --remote pre-production$ git push production master
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Future
Future
bull Asset pipeline
Future
bull Asset pipeline
bull Mysql tooling
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
bull Can databases
Future
Future
bull Hot compatibility beyond zero downtime deploys
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
Assets - Rails 2
bull httpsgithubcomddollarasset-resource
Asset pipeline - Rails 3
v1
GET dashboard
GET assetsapplication-v1js
200
200
Asset pipeline - Rails 3
GET dashboard
GET assetsapplication-v1js
200
404
v1 v2
Asset pipeline - Rails 3
Asset pipeline - Rails 3
bull When updating assets keep the existing version around
Asset pipeline - Rails 3
bull When updating assets keep the existing version around
bull Wersquore still in need of tooling to help cleaning up old assets
Asset pipeline - Rails 3
bull When updating assets keep the existing version around
bull Wersquore still in need of tooling to help cleaning up old assets
rake assetscleanoutdated
Migration strategies
Migration strategies
bull Read-only models
Migration strategies
bull Read-only models
class Role lt ActiveRecordBase def readonly true endend
Migration strategies
bull Read-only models
bull Ignore cached columns
Migration strategies
bull Read-only models
bull Ignore cached columns
class User def selfcolumns superreject |c| cname == notes endend
Migration strategies
bull Read-only models
bull Ignore cached columns
bull Know your database
Migration strategies
bull Read-only models
bull Ignore cached columns
bull Know your database
bull Beware of O(n) db locks
MyISAM
MyISAM
InnoDB
InnoDB
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
InnoDB
1 Create a new table with the same structure
2 Apply changes to new table
3 Copy data over
4 Acquire lock on original table
5 Sync newupdateddeleted rows
6 Swap tables
7 Release lock
InnoDB
1 Create a new table with the same structure
2 Apply changes to new table
3 Copy data over
4 Acquire lock on original table
5 Sync newupdateddeleted rows
6 Swap tables
7 Release lock
InnoDBhttpsgithubcomfreelstable_migrator
InnoDBhttpsgithubcomfreelstable_migrator
class AddStuffToMyBigTable lt TableMigration
migrates users
change_table do |t| tinteger foo null =gt false default =gt 0 tstring bar tindex foo name =gt index_for_foo endend
InnoDBhttpsgithubcomfreelstable_migrator
bull Requires acts_as_paranoid
bull Pretty oldstale codebase
bull Check forks for Rails 3 compatibility
Postgres
bull Table locks on
bull Adding updating or removing columns
bull Adding or removing indexes
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time if you avoid default values and constraints
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
can be done without any locking
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time
Postgres
bull Adding fields with defaults
Postgres
bull Adding fields with defaults
before_save assign_defaults
def assign_defaults selfadmin ||= falseend
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
CREATE INDEX CONCURRENTLY users_email_index ON users(email)
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
bull Outside a transaction
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
bull Outside a transaction
class AddIndexToUsers lt ActiveRecordMigration def selfup return if indexes(users)map(ampname)include(users_email_index) add_index users email name =gt users_email_index end
def selfdown remove_index users name =gt users_email_index endend
What about renaming
v1 v2
reads and writes to `foo`
reads and writes to `bar`
What about renaming
v1 v2
reads and writes to `foo`
reads and writes to `bar`
v1rsquo
add column `bar`+
reads from `foo`writes to both`foo` and `bar`
v1rsquorsquo
populate `bar`where needed
What about renaming
v2
reads and writes to `bar`
v2rsquo
ignore `foo`
v3
drop column `foo`
NoSQL
NoSQL
bull Schema migrations O(n) locks are just part of hot compatibility
NoSQL
bull Schema migrations O(n) locks are just part of hot compatibility
bull You still need to put effort into data migration
Workflow
Workflow
bull Make sure pull requests are addressing hot compatibility
Workflow
bull Make sure pull requests are addressing hot compatibility
bull Work after code is merged
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Production
Production
bull Run multiple server processes
Production
bull Run multiple server processes
bull Cycle them individually or in batches
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
bull Cycle them gracefully
Cycling techniques
bull Above the stack
bull Within the stack
bull Platform
Above the stack
bull Use your favorite web server
bull Setup a load balancer above them route around servers that are cycling
Above the stack
bull PROTIP default setup wonrsquot work
bull Configure load balancer to poll for your application health
class HealthController lt ApplicationController def index if Fileexists(Railsrootjoin(MAINTENANCE)) render text =gt Forget me status =gt 503 else render text =gt Im alive end endend
Cycling + Load balancer
bounce serversrm MAINTENANCE
remove server from rotation
touch MAINTENANCE
new servers come online
add server to rotation
Server Load balancer
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy where to check for the health
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
do not let your web server queue requests
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy to run the health check specified above
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
every 2s
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
stop routing to it once the health check failed once
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
and bring it back once it worked once
ELB
$ elb-configure-healthcheck MyLoadBalancer --target HTTP5000health --interval 2 --unhealthy-threshold 1 --healthy-threshold 1
Above the stack
Above the stackbull Minimal impact
Above the stackbull Minimal impact
bull Extended availability comes for free
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
bull Another component you have to track and maintain
Within the stack
Within the stack
Unicorn
bull Replaces your web server
bull Bounces gracefully
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Within the stack
Within the stackbull Easy to setup and deploy
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
bull Doesnrsquot address availability issues
Platform
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 1s bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 5s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 1s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ heroku labsenable preboot
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 starting for 5s bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 up for 1s bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 5s bundle exec thin start -p $PORT
BETA
Heroku
bull Migrations
Heroku
bull Migrations
$ git push pre-production master$ heroku run rake dbmigrate --remote pre-production$ git push production master
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Future
Future
bull Asset pipeline
Future
bull Asset pipeline
bull Mysql tooling
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
bull Can databases
Future
Future
bull Hot compatibility beyond zero downtime deploys
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
Asset pipeline - Rails 3
v1
GET dashboard
GET assetsapplication-v1js
200
200
Asset pipeline - Rails 3
GET dashboard
GET assetsapplication-v1js
200
404
v1 v2
Asset pipeline - Rails 3
Asset pipeline - Rails 3
bull When updating assets keep the existing version around
Asset pipeline - Rails 3
bull When updating assets keep the existing version around
bull Wersquore still in need of tooling to help cleaning up old assets
Asset pipeline - Rails 3
bull When updating assets keep the existing version around
bull Wersquore still in need of tooling to help cleaning up old assets
rake assetscleanoutdated
Migration strategies
Migration strategies
bull Read-only models
Migration strategies
bull Read-only models
class Role lt ActiveRecordBase def readonly true endend
Migration strategies
bull Read-only models
bull Ignore cached columns
Migration strategies
bull Read-only models
bull Ignore cached columns
class User def selfcolumns superreject |c| cname == notes endend
Migration strategies
bull Read-only models
bull Ignore cached columns
bull Know your database
Migration strategies
bull Read-only models
bull Ignore cached columns
bull Know your database
bull Beware of O(n) db locks
MyISAM
MyISAM
InnoDB
InnoDB
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
InnoDB
1 Create a new table with the same structure
2 Apply changes to new table
3 Copy data over
4 Acquire lock on original table
5 Sync newupdateddeleted rows
6 Swap tables
7 Release lock
InnoDB
1 Create a new table with the same structure
2 Apply changes to new table
3 Copy data over
4 Acquire lock on original table
5 Sync newupdateddeleted rows
6 Swap tables
7 Release lock
InnoDBhttpsgithubcomfreelstable_migrator
InnoDBhttpsgithubcomfreelstable_migrator
class AddStuffToMyBigTable lt TableMigration
migrates users
change_table do |t| tinteger foo null =gt false default =gt 0 tstring bar tindex foo name =gt index_for_foo endend
InnoDBhttpsgithubcomfreelstable_migrator
bull Requires acts_as_paranoid
bull Pretty oldstale codebase
bull Check forks for Rails 3 compatibility
Postgres
bull Table locks on
bull Adding updating or removing columns
bull Adding or removing indexes
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time if you avoid default values and constraints
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
can be done without any locking
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time
Postgres
bull Adding fields with defaults
Postgres
bull Adding fields with defaults
before_save assign_defaults
def assign_defaults selfadmin ||= falseend
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
CREATE INDEX CONCURRENTLY users_email_index ON users(email)
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
bull Outside a transaction
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
bull Outside a transaction
class AddIndexToUsers lt ActiveRecordMigration def selfup return if indexes(users)map(ampname)include(users_email_index) add_index users email name =gt users_email_index end
def selfdown remove_index users name =gt users_email_index endend
What about renaming
v1 v2
reads and writes to `foo`
reads and writes to `bar`
What about renaming
v1 v2
reads and writes to `foo`
reads and writes to `bar`
v1rsquo
add column `bar`+
reads from `foo`writes to both`foo` and `bar`
v1rsquorsquo
populate `bar`where needed
What about renaming
v2
reads and writes to `bar`
v2rsquo
ignore `foo`
v3
drop column `foo`
NoSQL
NoSQL
bull Schema migrations O(n) locks are just part of hot compatibility
NoSQL
bull Schema migrations O(n) locks are just part of hot compatibility
bull You still need to put effort into data migration
Workflow
Workflow
bull Make sure pull requests are addressing hot compatibility
Workflow
bull Make sure pull requests are addressing hot compatibility
bull Work after code is merged
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Production
Production
bull Run multiple server processes
Production
bull Run multiple server processes
bull Cycle them individually or in batches
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
bull Cycle them gracefully
Cycling techniques
bull Above the stack
bull Within the stack
bull Platform
Above the stack
bull Use your favorite web server
bull Setup a load balancer above them route around servers that are cycling
Above the stack
bull PROTIP default setup wonrsquot work
bull Configure load balancer to poll for your application health
class HealthController lt ApplicationController def index if Fileexists(Railsrootjoin(MAINTENANCE)) render text =gt Forget me status =gt 503 else render text =gt Im alive end endend
Cycling + Load balancer
bounce serversrm MAINTENANCE
remove server from rotation
touch MAINTENANCE
new servers come online
add server to rotation
Server Load balancer
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy where to check for the health
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
do not let your web server queue requests
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy to run the health check specified above
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
every 2s
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
stop routing to it once the health check failed once
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
and bring it back once it worked once
ELB
$ elb-configure-healthcheck MyLoadBalancer --target HTTP5000health --interval 2 --unhealthy-threshold 1 --healthy-threshold 1
Above the stack
Above the stackbull Minimal impact
Above the stackbull Minimal impact
bull Extended availability comes for free
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
bull Another component you have to track and maintain
Within the stack
Within the stack
Unicorn
bull Replaces your web server
bull Bounces gracefully
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Within the stack
Within the stackbull Easy to setup and deploy
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
bull Doesnrsquot address availability issues
Platform
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 1s bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 5s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 1s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ heroku labsenable preboot
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 starting for 5s bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 up for 1s bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 5s bundle exec thin start -p $PORT
BETA
Heroku
bull Migrations
Heroku
bull Migrations
$ git push pre-production master$ heroku run rake dbmigrate --remote pre-production$ git push production master
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Future
Future
bull Asset pipeline
Future
bull Asset pipeline
bull Mysql tooling
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
bull Can databases
Future
Future
bull Hot compatibility beyond zero downtime deploys
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
Asset pipeline - Rails 3
GET dashboard
GET assetsapplication-v1js
200
404
v1 v2
Asset pipeline - Rails 3
Asset pipeline - Rails 3
bull When updating assets keep the existing version around
Asset pipeline - Rails 3
bull When updating assets keep the existing version around
bull Wersquore still in need of tooling to help cleaning up old assets
Asset pipeline - Rails 3
bull When updating assets keep the existing version around
bull Wersquore still in need of tooling to help cleaning up old assets
rake assetscleanoutdated
Migration strategies
Migration strategies
bull Read-only models
Migration strategies
bull Read-only models
class Role lt ActiveRecordBase def readonly true endend
Migration strategies
bull Read-only models
bull Ignore cached columns
Migration strategies
bull Read-only models
bull Ignore cached columns
class User def selfcolumns superreject |c| cname == notes endend
Migration strategies
bull Read-only models
bull Ignore cached columns
bull Know your database
Migration strategies
bull Read-only models
bull Ignore cached columns
bull Know your database
bull Beware of O(n) db locks
MyISAM
MyISAM
InnoDB
InnoDB
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
InnoDB
1 Create a new table with the same structure
2 Apply changes to new table
3 Copy data over
4 Acquire lock on original table
5 Sync newupdateddeleted rows
6 Swap tables
7 Release lock
InnoDB
1 Create a new table with the same structure
2 Apply changes to new table
3 Copy data over
4 Acquire lock on original table
5 Sync newupdateddeleted rows
6 Swap tables
7 Release lock
InnoDBhttpsgithubcomfreelstable_migrator
InnoDBhttpsgithubcomfreelstable_migrator
class AddStuffToMyBigTable lt TableMigration
migrates users
change_table do |t| tinteger foo null =gt false default =gt 0 tstring bar tindex foo name =gt index_for_foo endend
InnoDBhttpsgithubcomfreelstable_migrator
bull Requires acts_as_paranoid
bull Pretty oldstale codebase
bull Check forks for Rails 3 compatibility
Postgres
bull Table locks on
bull Adding updating or removing columns
bull Adding or removing indexes
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time if you avoid default values and constraints
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
can be done without any locking
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time
Postgres
bull Adding fields with defaults
Postgres
bull Adding fields with defaults
before_save assign_defaults
def assign_defaults selfadmin ||= falseend
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
CREATE INDEX CONCURRENTLY users_email_index ON users(email)
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
bull Outside a transaction
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
bull Outside a transaction
class AddIndexToUsers lt ActiveRecordMigration def selfup return if indexes(users)map(ampname)include(users_email_index) add_index users email name =gt users_email_index end
def selfdown remove_index users name =gt users_email_index endend
What about renaming
v1 v2
reads and writes to `foo`
reads and writes to `bar`
What about renaming
v1 v2
reads and writes to `foo`
reads and writes to `bar`
v1rsquo
add column `bar`+
reads from `foo`writes to both`foo` and `bar`
v1rsquorsquo
populate `bar`where needed
What about renaming
v2
reads and writes to `bar`
v2rsquo
ignore `foo`
v3
drop column `foo`
NoSQL
NoSQL
bull Schema migrations O(n) locks are just part of hot compatibility
NoSQL
bull Schema migrations O(n) locks are just part of hot compatibility
bull You still need to put effort into data migration
Workflow
Workflow
bull Make sure pull requests are addressing hot compatibility
Workflow
bull Make sure pull requests are addressing hot compatibility
bull Work after code is merged
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Production
Production
bull Run multiple server processes
Production
bull Run multiple server processes
bull Cycle them individually or in batches
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
bull Cycle them gracefully
Cycling techniques
bull Above the stack
bull Within the stack
bull Platform
Above the stack
bull Use your favorite web server
bull Setup a load balancer above them route around servers that are cycling
Above the stack
bull PROTIP default setup wonrsquot work
bull Configure load balancer to poll for your application health
class HealthController lt ApplicationController def index if Fileexists(Railsrootjoin(MAINTENANCE)) render text =gt Forget me status =gt 503 else render text =gt Im alive end endend
Cycling + Load balancer
bounce serversrm MAINTENANCE
remove server from rotation
touch MAINTENANCE
new servers come online
add server to rotation
Server Load balancer
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy where to check for the health
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
do not let your web server queue requests
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy to run the health check specified above
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
every 2s
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
stop routing to it once the health check failed once
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
and bring it back once it worked once
ELB
$ elb-configure-healthcheck MyLoadBalancer --target HTTP5000health --interval 2 --unhealthy-threshold 1 --healthy-threshold 1
Above the stack
Above the stackbull Minimal impact
Above the stackbull Minimal impact
bull Extended availability comes for free
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
bull Another component you have to track and maintain
Within the stack
Within the stack
Unicorn
bull Replaces your web server
bull Bounces gracefully
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Within the stack
Within the stackbull Easy to setup and deploy
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
bull Doesnrsquot address availability issues
Platform
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 1s bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 5s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 1s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ heroku labsenable preboot
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 starting for 5s bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 up for 1s bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 5s bundle exec thin start -p $PORT
BETA
Heroku
bull Migrations
Heroku
bull Migrations
$ git push pre-production master$ heroku run rake dbmigrate --remote pre-production$ git push production master
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Future
Future
bull Asset pipeline
Future
bull Asset pipeline
bull Mysql tooling
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
bull Can databases
Future
Future
bull Hot compatibility beyond zero downtime deploys
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
Asset pipeline - Rails 3
Asset pipeline - Rails 3
bull When updating assets keep the existing version around
Asset pipeline - Rails 3
bull When updating assets keep the existing version around
bull Wersquore still in need of tooling to help cleaning up old assets
Asset pipeline - Rails 3
bull When updating assets keep the existing version around
bull Wersquore still in need of tooling to help cleaning up old assets
rake assetscleanoutdated
Migration strategies
Migration strategies
bull Read-only models
Migration strategies
bull Read-only models
class Role lt ActiveRecordBase def readonly true endend
Migration strategies
bull Read-only models
bull Ignore cached columns
Migration strategies
bull Read-only models
bull Ignore cached columns
class User def selfcolumns superreject |c| cname == notes endend
Migration strategies
bull Read-only models
bull Ignore cached columns
bull Know your database
Migration strategies
bull Read-only models
bull Ignore cached columns
bull Know your database
bull Beware of O(n) db locks
MyISAM
MyISAM
InnoDB
InnoDB
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
InnoDB
1 Create a new table with the same structure
2 Apply changes to new table
3 Copy data over
4 Acquire lock on original table
5 Sync newupdateddeleted rows
6 Swap tables
7 Release lock
InnoDB
1 Create a new table with the same structure
2 Apply changes to new table
3 Copy data over
4 Acquire lock on original table
5 Sync newupdateddeleted rows
6 Swap tables
7 Release lock
InnoDBhttpsgithubcomfreelstable_migrator
InnoDBhttpsgithubcomfreelstable_migrator
class AddStuffToMyBigTable lt TableMigration
migrates users
change_table do |t| tinteger foo null =gt false default =gt 0 tstring bar tindex foo name =gt index_for_foo endend
InnoDBhttpsgithubcomfreelstable_migrator
bull Requires acts_as_paranoid
bull Pretty oldstale codebase
bull Check forks for Rails 3 compatibility
Postgres
bull Table locks on
bull Adding updating or removing columns
bull Adding or removing indexes
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time if you avoid default values and constraints
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
can be done without any locking
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time
Postgres
bull Adding fields with defaults
Postgres
bull Adding fields with defaults
before_save assign_defaults
def assign_defaults selfadmin ||= falseend
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
CREATE INDEX CONCURRENTLY users_email_index ON users(email)
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
bull Outside a transaction
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
bull Outside a transaction
class AddIndexToUsers lt ActiveRecordMigration def selfup return if indexes(users)map(ampname)include(users_email_index) add_index users email name =gt users_email_index end
def selfdown remove_index users name =gt users_email_index endend
What about renaming
v1 v2
reads and writes to `foo`
reads and writes to `bar`
What about renaming
v1 v2
reads and writes to `foo`
reads and writes to `bar`
v1rsquo
add column `bar`+
reads from `foo`writes to both`foo` and `bar`
v1rsquorsquo
populate `bar`where needed
What about renaming
v2
reads and writes to `bar`
v2rsquo
ignore `foo`
v3
drop column `foo`
NoSQL
NoSQL
bull Schema migrations O(n) locks are just part of hot compatibility
NoSQL
bull Schema migrations O(n) locks are just part of hot compatibility
bull You still need to put effort into data migration
Workflow
Workflow
bull Make sure pull requests are addressing hot compatibility
Workflow
bull Make sure pull requests are addressing hot compatibility
bull Work after code is merged
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Production
Production
bull Run multiple server processes
Production
bull Run multiple server processes
bull Cycle them individually or in batches
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
bull Cycle them gracefully
Cycling techniques
bull Above the stack
bull Within the stack
bull Platform
Above the stack
bull Use your favorite web server
bull Setup a load balancer above them route around servers that are cycling
Above the stack
bull PROTIP default setup wonrsquot work
bull Configure load balancer to poll for your application health
class HealthController lt ApplicationController def index if Fileexists(Railsrootjoin(MAINTENANCE)) render text =gt Forget me status =gt 503 else render text =gt Im alive end endend
Cycling + Load balancer
bounce serversrm MAINTENANCE
remove server from rotation
touch MAINTENANCE
new servers come online
add server to rotation
Server Load balancer
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy where to check for the health
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
do not let your web server queue requests
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy to run the health check specified above
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
every 2s
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
stop routing to it once the health check failed once
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
and bring it back once it worked once
ELB
$ elb-configure-healthcheck MyLoadBalancer --target HTTP5000health --interval 2 --unhealthy-threshold 1 --healthy-threshold 1
Above the stack
Above the stackbull Minimal impact
Above the stackbull Minimal impact
bull Extended availability comes for free
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
bull Another component you have to track and maintain
Within the stack
Within the stack
Unicorn
bull Replaces your web server
bull Bounces gracefully
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Within the stack
Within the stackbull Easy to setup and deploy
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
bull Doesnrsquot address availability issues
Platform
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 1s bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 5s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 1s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ heroku labsenable preboot
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 starting for 5s bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 up for 1s bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 5s bundle exec thin start -p $PORT
BETA
Heroku
bull Migrations
Heroku
bull Migrations
$ git push pre-production master$ heroku run rake dbmigrate --remote pre-production$ git push production master
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Future
Future
bull Asset pipeline
Future
bull Asset pipeline
bull Mysql tooling
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
bull Can databases
Future
Future
bull Hot compatibility beyond zero downtime deploys
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
Asset pipeline - Rails 3
bull When updating assets keep the existing version around
Asset pipeline - Rails 3
bull When updating assets keep the existing version around
bull Wersquore still in need of tooling to help cleaning up old assets
Asset pipeline - Rails 3
bull When updating assets keep the existing version around
bull Wersquore still in need of tooling to help cleaning up old assets
rake assetscleanoutdated
Migration strategies
Migration strategies
bull Read-only models
Migration strategies
bull Read-only models
class Role lt ActiveRecordBase def readonly true endend
Migration strategies
bull Read-only models
bull Ignore cached columns
Migration strategies
bull Read-only models
bull Ignore cached columns
class User def selfcolumns superreject |c| cname == notes endend
Migration strategies
bull Read-only models
bull Ignore cached columns
bull Know your database
Migration strategies
bull Read-only models
bull Ignore cached columns
bull Know your database
bull Beware of O(n) db locks
MyISAM
MyISAM
InnoDB
InnoDB
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
InnoDB
1 Create a new table with the same structure
2 Apply changes to new table
3 Copy data over
4 Acquire lock on original table
5 Sync newupdateddeleted rows
6 Swap tables
7 Release lock
InnoDB
1 Create a new table with the same structure
2 Apply changes to new table
3 Copy data over
4 Acquire lock on original table
5 Sync newupdateddeleted rows
6 Swap tables
7 Release lock
InnoDBhttpsgithubcomfreelstable_migrator
InnoDBhttpsgithubcomfreelstable_migrator
class AddStuffToMyBigTable lt TableMigration
migrates users
change_table do |t| tinteger foo null =gt false default =gt 0 tstring bar tindex foo name =gt index_for_foo endend
InnoDBhttpsgithubcomfreelstable_migrator
bull Requires acts_as_paranoid
bull Pretty oldstale codebase
bull Check forks for Rails 3 compatibility
Postgres
bull Table locks on
bull Adding updating or removing columns
bull Adding or removing indexes
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time if you avoid default values and constraints
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
can be done without any locking
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time
Postgres
bull Adding fields with defaults
Postgres
bull Adding fields with defaults
before_save assign_defaults
def assign_defaults selfadmin ||= falseend
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
CREATE INDEX CONCURRENTLY users_email_index ON users(email)
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
bull Outside a transaction
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
bull Outside a transaction
class AddIndexToUsers lt ActiveRecordMigration def selfup return if indexes(users)map(ampname)include(users_email_index) add_index users email name =gt users_email_index end
def selfdown remove_index users name =gt users_email_index endend
What about renaming
v1 v2
reads and writes to `foo`
reads and writes to `bar`
What about renaming
v1 v2
reads and writes to `foo`
reads and writes to `bar`
v1rsquo
add column `bar`+
reads from `foo`writes to both`foo` and `bar`
v1rsquorsquo
populate `bar`where needed
What about renaming
v2
reads and writes to `bar`
v2rsquo
ignore `foo`
v3
drop column `foo`
NoSQL
NoSQL
bull Schema migrations O(n) locks are just part of hot compatibility
NoSQL
bull Schema migrations O(n) locks are just part of hot compatibility
bull You still need to put effort into data migration
Workflow
Workflow
bull Make sure pull requests are addressing hot compatibility
Workflow
bull Make sure pull requests are addressing hot compatibility
bull Work after code is merged
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Production
Production
bull Run multiple server processes
Production
bull Run multiple server processes
bull Cycle them individually or in batches
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
bull Cycle them gracefully
Cycling techniques
bull Above the stack
bull Within the stack
bull Platform
Above the stack
bull Use your favorite web server
bull Setup a load balancer above them route around servers that are cycling
Above the stack
bull PROTIP default setup wonrsquot work
bull Configure load balancer to poll for your application health
class HealthController lt ApplicationController def index if Fileexists(Railsrootjoin(MAINTENANCE)) render text =gt Forget me status =gt 503 else render text =gt Im alive end endend
Cycling + Load balancer
bounce serversrm MAINTENANCE
remove server from rotation
touch MAINTENANCE
new servers come online
add server to rotation
Server Load balancer
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy where to check for the health
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
do not let your web server queue requests
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy to run the health check specified above
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
every 2s
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
stop routing to it once the health check failed once
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
and bring it back once it worked once
ELB
$ elb-configure-healthcheck MyLoadBalancer --target HTTP5000health --interval 2 --unhealthy-threshold 1 --healthy-threshold 1
Above the stack
Above the stackbull Minimal impact
Above the stackbull Minimal impact
bull Extended availability comes for free
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
bull Another component you have to track and maintain
Within the stack
Within the stack
Unicorn
bull Replaces your web server
bull Bounces gracefully
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Within the stack
Within the stackbull Easy to setup and deploy
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
bull Doesnrsquot address availability issues
Platform
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 1s bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 5s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 1s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ heroku labsenable preboot
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 starting for 5s bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 up for 1s bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 5s bundle exec thin start -p $PORT
BETA
Heroku
bull Migrations
Heroku
bull Migrations
$ git push pre-production master$ heroku run rake dbmigrate --remote pre-production$ git push production master
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Future
Future
bull Asset pipeline
Future
bull Asset pipeline
bull Mysql tooling
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
bull Can databases
Future
Future
bull Hot compatibility beyond zero downtime deploys
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
Asset pipeline - Rails 3
bull When updating assets keep the existing version around
bull Wersquore still in need of tooling to help cleaning up old assets
Asset pipeline - Rails 3
bull When updating assets keep the existing version around
bull Wersquore still in need of tooling to help cleaning up old assets
rake assetscleanoutdated
Migration strategies
Migration strategies
bull Read-only models
Migration strategies
bull Read-only models
class Role lt ActiveRecordBase def readonly true endend
Migration strategies
bull Read-only models
bull Ignore cached columns
Migration strategies
bull Read-only models
bull Ignore cached columns
class User def selfcolumns superreject |c| cname == notes endend
Migration strategies
bull Read-only models
bull Ignore cached columns
bull Know your database
Migration strategies
bull Read-only models
bull Ignore cached columns
bull Know your database
bull Beware of O(n) db locks
MyISAM
MyISAM
InnoDB
InnoDB
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
InnoDB
1 Create a new table with the same structure
2 Apply changes to new table
3 Copy data over
4 Acquire lock on original table
5 Sync newupdateddeleted rows
6 Swap tables
7 Release lock
InnoDB
1 Create a new table with the same structure
2 Apply changes to new table
3 Copy data over
4 Acquire lock on original table
5 Sync newupdateddeleted rows
6 Swap tables
7 Release lock
InnoDBhttpsgithubcomfreelstable_migrator
InnoDBhttpsgithubcomfreelstable_migrator
class AddStuffToMyBigTable lt TableMigration
migrates users
change_table do |t| tinteger foo null =gt false default =gt 0 tstring bar tindex foo name =gt index_for_foo endend
InnoDBhttpsgithubcomfreelstable_migrator
bull Requires acts_as_paranoid
bull Pretty oldstale codebase
bull Check forks for Rails 3 compatibility
Postgres
bull Table locks on
bull Adding updating or removing columns
bull Adding or removing indexes
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time if you avoid default values and constraints
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
can be done without any locking
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time
Postgres
bull Adding fields with defaults
Postgres
bull Adding fields with defaults
before_save assign_defaults
def assign_defaults selfadmin ||= falseend
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
CREATE INDEX CONCURRENTLY users_email_index ON users(email)
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
bull Outside a transaction
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
bull Outside a transaction
class AddIndexToUsers lt ActiveRecordMigration def selfup return if indexes(users)map(ampname)include(users_email_index) add_index users email name =gt users_email_index end
def selfdown remove_index users name =gt users_email_index endend
What about renaming
v1 v2
reads and writes to `foo`
reads and writes to `bar`
What about renaming
v1 v2
reads and writes to `foo`
reads and writes to `bar`
v1rsquo
add column `bar`+
reads from `foo`writes to both`foo` and `bar`
v1rsquorsquo
populate `bar`where needed
What about renaming
v2
reads and writes to `bar`
v2rsquo
ignore `foo`
v3
drop column `foo`
NoSQL
NoSQL
bull Schema migrations O(n) locks are just part of hot compatibility
NoSQL
bull Schema migrations O(n) locks are just part of hot compatibility
bull You still need to put effort into data migration
Workflow
Workflow
bull Make sure pull requests are addressing hot compatibility
Workflow
bull Make sure pull requests are addressing hot compatibility
bull Work after code is merged
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Production
Production
bull Run multiple server processes
Production
bull Run multiple server processes
bull Cycle them individually or in batches
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
bull Cycle them gracefully
Cycling techniques
bull Above the stack
bull Within the stack
bull Platform
Above the stack
bull Use your favorite web server
bull Setup a load balancer above them route around servers that are cycling
Above the stack
bull PROTIP default setup wonrsquot work
bull Configure load balancer to poll for your application health
class HealthController lt ApplicationController def index if Fileexists(Railsrootjoin(MAINTENANCE)) render text =gt Forget me status =gt 503 else render text =gt Im alive end endend
Cycling + Load balancer
bounce serversrm MAINTENANCE
remove server from rotation
touch MAINTENANCE
new servers come online
add server to rotation
Server Load balancer
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy where to check for the health
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
do not let your web server queue requests
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy to run the health check specified above
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
every 2s
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
stop routing to it once the health check failed once
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
and bring it back once it worked once
ELB
$ elb-configure-healthcheck MyLoadBalancer --target HTTP5000health --interval 2 --unhealthy-threshold 1 --healthy-threshold 1
Above the stack
Above the stackbull Minimal impact
Above the stackbull Minimal impact
bull Extended availability comes for free
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
bull Another component you have to track and maintain
Within the stack
Within the stack
Unicorn
bull Replaces your web server
bull Bounces gracefully
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Within the stack
Within the stackbull Easy to setup and deploy
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
bull Doesnrsquot address availability issues
Platform
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 1s bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 5s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 1s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ heroku labsenable preboot
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 starting for 5s bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 up for 1s bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 5s bundle exec thin start -p $PORT
BETA
Heroku
bull Migrations
Heroku
bull Migrations
$ git push pre-production master$ heroku run rake dbmigrate --remote pre-production$ git push production master
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Future
Future
bull Asset pipeline
Future
bull Asset pipeline
bull Mysql tooling
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
bull Can databases
Future
Future
bull Hot compatibility beyond zero downtime deploys
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
Asset pipeline - Rails 3
bull When updating assets keep the existing version around
bull Wersquore still in need of tooling to help cleaning up old assets
rake assetscleanoutdated
Migration strategies
Migration strategies
bull Read-only models
Migration strategies
bull Read-only models
class Role lt ActiveRecordBase def readonly true endend
Migration strategies
bull Read-only models
bull Ignore cached columns
Migration strategies
bull Read-only models
bull Ignore cached columns
class User def selfcolumns superreject |c| cname == notes endend
Migration strategies
bull Read-only models
bull Ignore cached columns
bull Know your database
Migration strategies
bull Read-only models
bull Ignore cached columns
bull Know your database
bull Beware of O(n) db locks
MyISAM
MyISAM
InnoDB
InnoDB
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
InnoDB
1 Create a new table with the same structure
2 Apply changes to new table
3 Copy data over
4 Acquire lock on original table
5 Sync newupdateddeleted rows
6 Swap tables
7 Release lock
InnoDB
1 Create a new table with the same structure
2 Apply changes to new table
3 Copy data over
4 Acquire lock on original table
5 Sync newupdateddeleted rows
6 Swap tables
7 Release lock
InnoDBhttpsgithubcomfreelstable_migrator
InnoDBhttpsgithubcomfreelstable_migrator
class AddStuffToMyBigTable lt TableMigration
migrates users
change_table do |t| tinteger foo null =gt false default =gt 0 tstring bar tindex foo name =gt index_for_foo endend
InnoDBhttpsgithubcomfreelstable_migrator
bull Requires acts_as_paranoid
bull Pretty oldstale codebase
bull Check forks for Rails 3 compatibility
Postgres
bull Table locks on
bull Adding updating or removing columns
bull Adding or removing indexes
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time if you avoid default values and constraints
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
can be done without any locking
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time
Postgres
bull Adding fields with defaults
Postgres
bull Adding fields with defaults
before_save assign_defaults
def assign_defaults selfadmin ||= falseend
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
CREATE INDEX CONCURRENTLY users_email_index ON users(email)
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
bull Outside a transaction
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
bull Outside a transaction
class AddIndexToUsers lt ActiveRecordMigration def selfup return if indexes(users)map(ampname)include(users_email_index) add_index users email name =gt users_email_index end
def selfdown remove_index users name =gt users_email_index endend
What about renaming
v1 v2
reads and writes to `foo`
reads and writes to `bar`
What about renaming
v1 v2
reads and writes to `foo`
reads and writes to `bar`
v1rsquo
add column `bar`+
reads from `foo`writes to both`foo` and `bar`
v1rsquorsquo
populate `bar`where needed
What about renaming
v2
reads and writes to `bar`
v2rsquo
ignore `foo`
v3
drop column `foo`
NoSQL
NoSQL
bull Schema migrations O(n) locks are just part of hot compatibility
NoSQL
bull Schema migrations O(n) locks are just part of hot compatibility
bull You still need to put effort into data migration
Workflow
Workflow
bull Make sure pull requests are addressing hot compatibility
Workflow
bull Make sure pull requests are addressing hot compatibility
bull Work after code is merged
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Production
Production
bull Run multiple server processes
Production
bull Run multiple server processes
bull Cycle them individually or in batches
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
bull Cycle them gracefully
Cycling techniques
bull Above the stack
bull Within the stack
bull Platform
Above the stack
bull Use your favorite web server
bull Setup a load balancer above them route around servers that are cycling
Above the stack
bull PROTIP default setup wonrsquot work
bull Configure load balancer to poll for your application health
class HealthController lt ApplicationController def index if Fileexists(Railsrootjoin(MAINTENANCE)) render text =gt Forget me status =gt 503 else render text =gt Im alive end endend
Cycling + Load balancer
bounce serversrm MAINTENANCE
remove server from rotation
touch MAINTENANCE
new servers come online
add server to rotation
Server Load balancer
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy where to check for the health
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
do not let your web server queue requests
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy to run the health check specified above
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
every 2s
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
stop routing to it once the health check failed once
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
and bring it back once it worked once
ELB
$ elb-configure-healthcheck MyLoadBalancer --target HTTP5000health --interval 2 --unhealthy-threshold 1 --healthy-threshold 1
Above the stack
Above the stackbull Minimal impact
Above the stackbull Minimal impact
bull Extended availability comes for free
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
bull Another component you have to track and maintain
Within the stack
Within the stack
Unicorn
bull Replaces your web server
bull Bounces gracefully
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Within the stack
Within the stackbull Easy to setup and deploy
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
bull Doesnrsquot address availability issues
Platform
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 1s bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 5s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 1s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ heroku labsenable preboot
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 starting for 5s bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 up for 1s bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 5s bundle exec thin start -p $PORT
BETA
Heroku
bull Migrations
Heroku
bull Migrations
$ git push pre-production master$ heroku run rake dbmigrate --remote pre-production$ git push production master
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Future
Future
bull Asset pipeline
Future
bull Asset pipeline
bull Mysql tooling
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
bull Can databases
Future
Future
bull Hot compatibility beyond zero downtime deploys
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
Migration strategies
Migration strategies
bull Read-only models
Migration strategies
bull Read-only models
class Role lt ActiveRecordBase def readonly true endend
Migration strategies
bull Read-only models
bull Ignore cached columns
Migration strategies
bull Read-only models
bull Ignore cached columns
class User def selfcolumns superreject |c| cname == notes endend
Migration strategies
bull Read-only models
bull Ignore cached columns
bull Know your database
Migration strategies
bull Read-only models
bull Ignore cached columns
bull Know your database
bull Beware of O(n) db locks
MyISAM
MyISAM
InnoDB
InnoDB
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
InnoDB
1 Create a new table with the same structure
2 Apply changes to new table
3 Copy data over
4 Acquire lock on original table
5 Sync newupdateddeleted rows
6 Swap tables
7 Release lock
InnoDB
1 Create a new table with the same structure
2 Apply changes to new table
3 Copy data over
4 Acquire lock on original table
5 Sync newupdateddeleted rows
6 Swap tables
7 Release lock
InnoDBhttpsgithubcomfreelstable_migrator
InnoDBhttpsgithubcomfreelstable_migrator
class AddStuffToMyBigTable lt TableMigration
migrates users
change_table do |t| tinteger foo null =gt false default =gt 0 tstring bar tindex foo name =gt index_for_foo endend
InnoDBhttpsgithubcomfreelstable_migrator
bull Requires acts_as_paranoid
bull Pretty oldstale codebase
bull Check forks for Rails 3 compatibility
Postgres
bull Table locks on
bull Adding updating or removing columns
bull Adding or removing indexes
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time if you avoid default values and constraints
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
can be done without any locking
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time
Postgres
bull Adding fields with defaults
Postgres
bull Adding fields with defaults
before_save assign_defaults
def assign_defaults selfadmin ||= falseend
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
CREATE INDEX CONCURRENTLY users_email_index ON users(email)
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
bull Outside a transaction
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
bull Outside a transaction
class AddIndexToUsers lt ActiveRecordMigration def selfup return if indexes(users)map(ampname)include(users_email_index) add_index users email name =gt users_email_index end
def selfdown remove_index users name =gt users_email_index endend
What about renaming
v1 v2
reads and writes to `foo`
reads and writes to `bar`
What about renaming
v1 v2
reads and writes to `foo`
reads and writes to `bar`
v1rsquo
add column `bar`+
reads from `foo`writes to both`foo` and `bar`
v1rsquorsquo
populate `bar`where needed
What about renaming
v2
reads and writes to `bar`
v2rsquo
ignore `foo`
v3
drop column `foo`
NoSQL
NoSQL
bull Schema migrations O(n) locks are just part of hot compatibility
NoSQL
bull Schema migrations O(n) locks are just part of hot compatibility
bull You still need to put effort into data migration
Workflow
Workflow
bull Make sure pull requests are addressing hot compatibility
Workflow
bull Make sure pull requests are addressing hot compatibility
bull Work after code is merged
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Production
Production
bull Run multiple server processes
Production
bull Run multiple server processes
bull Cycle them individually or in batches
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
bull Cycle them gracefully
Cycling techniques
bull Above the stack
bull Within the stack
bull Platform
Above the stack
bull Use your favorite web server
bull Setup a load balancer above them route around servers that are cycling
Above the stack
bull PROTIP default setup wonrsquot work
bull Configure load balancer to poll for your application health
class HealthController lt ApplicationController def index if Fileexists(Railsrootjoin(MAINTENANCE)) render text =gt Forget me status =gt 503 else render text =gt Im alive end endend
Cycling + Load balancer
bounce serversrm MAINTENANCE
remove server from rotation
touch MAINTENANCE
new servers come online
add server to rotation
Server Load balancer
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy where to check for the health
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
do not let your web server queue requests
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy to run the health check specified above
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
every 2s
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
stop routing to it once the health check failed once
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
and bring it back once it worked once
ELB
$ elb-configure-healthcheck MyLoadBalancer --target HTTP5000health --interval 2 --unhealthy-threshold 1 --healthy-threshold 1
Above the stack
Above the stackbull Minimal impact
Above the stackbull Minimal impact
bull Extended availability comes for free
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
bull Another component you have to track and maintain
Within the stack
Within the stack
Unicorn
bull Replaces your web server
bull Bounces gracefully
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Within the stack
Within the stackbull Easy to setup and deploy
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
bull Doesnrsquot address availability issues
Platform
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 1s bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 5s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 1s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ heroku labsenable preboot
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 starting for 5s bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 up for 1s bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 5s bundle exec thin start -p $PORT
BETA
Heroku
bull Migrations
Heroku
bull Migrations
$ git push pre-production master$ heroku run rake dbmigrate --remote pre-production$ git push production master
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Future
Future
bull Asset pipeline
Future
bull Asset pipeline
bull Mysql tooling
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
bull Can databases
Future
Future
bull Hot compatibility beyond zero downtime deploys
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
Migration strategies
bull Read-only models
Migration strategies
bull Read-only models
class Role lt ActiveRecordBase def readonly true endend
Migration strategies
bull Read-only models
bull Ignore cached columns
Migration strategies
bull Read-only models
bull Ignore cached columns
class User def selfcolumns superreject |c| cname == notes endend
Migration strategies
bull Read-only models
bull Ignore cached columns
bull Know your database
Migration strategies
bull Read-only models
bull Ignore cached columns
bull Know your database
bull Beware of O(n) db locks
MyISAM
MyISAM
InnoDB
InnoDB
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
InnoDB
1 Create a new table with the same structure
2 Apply changes to new table
3 Copy data over
4 Acquire lock on original table
5 Sync newupdateddeleted rows
6 Swap tables
7 Release lock
InnoDB
1 Create a new table with the same structure
2 Apply changes to new table
3 Copy data over
4 Acquire lock on original table
5 Sync newupdateddeleted rows
6 Swap tables
7 Release lock
InnoDBhttpsgithubcomfreelstable_migrator
InnoDBhttpsgithubcomfreelstable_migrator
class AddStuffToMyBigTable lt TableMigration
migrates users
change_table do |t| tinteger foo null =gt false default =gt 0 tstring bar tindex foo name =gt index_for_foo endend
InnoDBhttpsgithubcomfreelstable_migrator
bull Requires acts_as_paranoid
bull Pretty oldstale codebase
bull Check forks for Rails 3 compatibility
Postgres
bull Table locks on
bull Adding updating or removing columns
bull Adding or removing indexes
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time if you avoid default values and constraints
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
can be done without any locking
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time
Postgres
bull Adding fields with defaults
Postgres
bull Adding fields with defaults
before_save assign_defaults
def assign_defaults selfadmin ||= falseend
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
CREATE INDEX CONCURRENTLY users_email_index ON users(email)
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
bull Outside a transaction
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
bull Outside a transaction
class AddIndexToUsers lt ActiveRecordMigration def selfup return if indexes(users)map(ampname)include(users_email_index) add_index users email name =gt users_email_index end
def selfdown remove_index users name =gt users_email_index endend
What about renaming
v1 v2
reads and writes to `foo`
reads and writes to `bar`
What about renaming
v1 v2
reads and writes to `foo`
reads and writes to `bar`
v1rsquo
add column `bar`+
reads from `foo`writes to both`foo` and `bar`
v1rsquorsquo
populate `bar`where needed
What about renaming
v2
reads and writes to `bar`
v2rsquo
ignore `foo`
v3
drop column `foo`
NoSQL
NoSQL
bull Schema migrations O(n) locks are just part of hot compatibility
NoSQL
bull Schema migrations O(n) locks are just part of hot compatibility
bull You still need to put effort into data migration
Workflow
Workflow
bull Make sure pull requests are addressing hot compatibility
Workflow
bull Make sure pull requests are addressing hot compatibility
bull Work after code is merged
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Production
Production
bull Run multiple server processes
Production
bull Run multiple server processes
bull Cycle them individually or in batches
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
bull Cycle them gracefully
Cycling techniques
bull Above the stack
bull Within the stack
bull Platform
Above the stack
bull Use your favorite web server
bull Setup a load balancer above them route around servers that are cycling
Above the stack
bull PROTIP default setup wonrsquot work
bull Configure load balancer to poll for your application health
class HealthController lt ApplicationController def index if Fileexists(Railsrootjoin(MAINTENANCE)) render text =gt Forget me status =gt 503 else render text =gt Im alive end endend
Cycling + Load balancer
bounce serversrm MAINTENANCE
remove server from rotation
touch MAINTENANCE
new servers come online
add server to rotation
Server Load balancer
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy where to check for the health
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
do not let your web server queue requests
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy to run the health check specified above
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
every 2s
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
stop routing to it once the health check failed once
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
and bring it back once it worked once
ELB
$ elb-configure-healthcheck MyLoadBalancer --target HTTP5000health --interval 2 --unhealthy-threshold 1 --healthy-threshold 1
Above the stack
Above the stackbull Minimal impact
Above the stackbull Minimal impact
bull Extended availability comes for free
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
bull Another component you have to track and maintain
Within the stack
Within the stack
Unicorn
bull Replaces your web server
bull Bounces gracefully
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Within the stack
Within the stackbull Easy to setup and deploy
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
bull Doesnrsquot address availability issues
Platform
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 1s bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 5s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 1s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ heroku labsenable preboot
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 starting for 5s bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 up for 1s bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 5s bundle exec thin start -p $PORT
BETA
Heroku
bull Migrations
Heroku
bull Migrations
$ git push pre-production master$ heroku run rake dbmigrate --remote pre-production$ git push production master
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Future
Future
bull Asset pipeline
Future
bull Asset pipeline
bull Mysql tooling
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
bull Can databases
Future
Future
bull Hot compatibility beyond zero downtime deploys
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
Migration strategies
bull Read-only models
class Role lt ActiveRecordBase def readonly true endend
Migration strategies
bull Read-only models
bull Ignore cached columns
Migration strategies
bull Read-only models
bull Ignore cached columns
class User def selfcolumns superreject |c| cname == notes endend
Migration strategies
bull Read-only models
bull Ignore cached columns
bull Know your database
Migration strategies
bull Read-only models
bull Ignore cached columns
bull Know your database
bull Beware of O(n) db locks
MyISAM
MyISAM
InnoDB
InnoDB
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
InnoDB
1 Create a new table with the same structure
2 Apply changes to new table
3 Copy data over
4 Acquire lock on original table
5 Sync newupdateddeleted rows
6 Swap tables
7 Release lock
InnoDB
1 Create a new table with the same structure
2 Apply changes to new table
3 Copy data over
4 Acquire lock on original table
5 Sync newupdateddeleted rows
6 Swap tables
7 Release lock
InnoDBhttpsgithubcomfreelstable_migrator
InnoDBhttpsgithubcomfreelstable_migrator
class AddStuffToMyBigTable lt TableMigration
migrates users
change_table do |t| tinteger foo null =gt false default =gt 0 tstring bar tindex foo name =gt index_for_foo endend
InnoDBhttpsgithubcomfreelstable_migrator
bull Requires acts_as_paranoid
bull Pretty oldstale codebase
bull Check forks for Rails 3 compatibility
Postgres
bull Table locks on
bull Adding updating or removing columns
bull Adding or removing indexes
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time if you avoid default values and constraints
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
can be done without any locking
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time
Postgres
bull Adding fields with defaults
Postgres
bull Adding fields with defaults
before_save assign_defaults
def assign_defaults selfadmin ||= falseend
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
CREATE INDEX CONCURRENTLY users_email_index ON users(email)
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
bull Outside a transaction
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
bull Outside a transaction
class AddIndexToUsers lt ActiveRecordMigration def selfup return if indexes(users)map(ampname)include(users_email_index) add_index users email name =gt users_email_index end
def selfdown remove_index users name =gt users_email_index endend
What about renaming
v1 v2
reads and writes to `foo`
reads and writes to `bar`
What about renaming
v1 v2
reads and writes to `foo`
reads and writes to `bar`
v1rsquo
add column `bar`+
reads from `foo`writes to both`foo` and `bar`
v1rsquorsquo
populate `bar`where needed
What about renaming
v2
reads and writes to `bar`
v2rsquo
ignore `foo`
v3
drop column `foo`
NoSQL
NoSQL
bull Schema migrations O(n) locks are just part of hot compatibility
NoSQL
bull Schema migrations O(n) locks are just part of hot compatibility
bull You still need to put effort into data migration
Workflow
Workflow
bull Make sure pull requests are addressing hot compatibility
Workflow
bull Make sure pull requests are addressing hot compatibility
bull Work after code is merged
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Production
Production
bull Run multiple server processes
Production
bull Run multiple server processes
bull Cycle them individually or in batches
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
bull Cycle them gracefully
Cycling techniques
bull Above the stack
bull Within the stack
bull Platform
Above the stack
bull Use your favorite web server
bull Setup a load balancer above them route around servers that are cycling
Above the stack
bull PROTIP default setup wonrsquot work
bull Configure load balancer to poll for your application health
class HealthController lt ApplicationController def index if Fileexists(Railsrootjoin(MAINTENANCE)) render text =gt Forget me status =gt 503 else render text =gt Im alive end endend
Cycling + Load balancer
bounce serversrm MAINTENANCE
remove server from rotation
touch MAINTENANCE
new servers come online
add server to rotation
Server Load balancer
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy where to check for the health
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
do not let your web server queue requests
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy to run the health check specified above
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
every 2s
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
stop routing to it once the health check failed once
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
and bring it back once it worked once
ELB
$ elb-configure-healthcheck MyLoadBalancer --target HTTP5000health --interval 2 --unhealthy-threshold 1 --healthy-threshold 1
Above the stack
Above the stackbull Minimal impact
Above the stackbull Minimal impact
bull Extended availability comes for free
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
bull Another component you have to track and maintain
Within the stack
Within the stack
Unicorn
bull Replaces your web server
bull Bounces gracefully
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Within the stack
Within the stackbull Easy to setup and deploy
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
bull Doesnrsquot address availability issues
Platform
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 1s bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 5s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 1s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ heroku labsenable preboot
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 starting for 5s bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 up for 1s bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 5s bundle exec thin start -p $PORT
BETA
Heroku
bull Migrations
Heroku
bull Migrations
$ git push pre-production master$ heroku run rake dbmigrate --remote pre-production$ git push production master
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Future
Future
bull Asset pipeline
Future
bull Asset pipeline
bull Mysql tooling
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
bull Can databases
Future
Future
bull Hot compatibility beyond zero downtime deploys
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
Migration strategies
bull Read-only models
bull Ignore cached columns
Migration strategies
bull Read-only models
bull Ignore cached columns
class User def selfcolumns superreject |c| cname == notes endend
Migration strategies
bull Read-only models
bull Ignore cached columns
bull Know your database
Migration strategies
bull Read-only models
bull Ignore cached columns
bull Know your database
bull Beware of O(n) db locks
MyISAM
MyISAM
InnoDB
InnoDB
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
InnoDB
1 Create a new table with the same structure
2 Apply changes to new table
3 Copy data over
4 Acquire lock on original table
5 Sync newupdateddeleted rows
6 Swap tables
7 Release lock
InnoDB
1 Create a new table with the same structure
2 Apply changes to new table
3 Copy data over
4 Acquire lock on original table
5 Sync newupdateddeleted rows
6 Swap tables
7 Release lock
InnoDBhttpsgithubcomfreelstable_migrator
InnoDBhttpsgithubcomfreelstable_migrator
class AddStuffToMyBigTable lt TableMigration
migrates users
change_table do |t| tinteger foo null =gt false default =gt 0 tstring bar tindex foo name =gt index_for_foo endend
InnoDBhttpsgithubcomfreelstable_migrator
bull Requires acts_as_paranoid
bull Pretty oldstale codebase
bull Check forks for Rails 3 compatibility
Postgres
bull Table locks on
bull Adding updating or removing columns
bull Adding or removing indexes
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time if you avoid default values and constraints
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
can be done without any locking
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time
Postgres
bull Adding fields with defaults
Postgres
bull Adding fields with defaults
before_save assign_defaults
def assign_defaults selfadmin ||= falseend
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
CREATE INDEX CONCURRENTLY users_email_index ON users(email)
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
bull Outside a transaction
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
bull Outside a transaction
class AddIndexToUsers lt ActiveRecordMigration def selfup return if indexes(users)map(ampname)include(users_email_index) add_index users email name =gt users_email_index end
def selfdown remove_index users name =gt users_email_index endend
What about renaming
v1 v2
reads and writes to `foo`
reads and writes to `bar`
What about renaming
v1 v2
reads and writes to `foo`
reads and writes to `bar`
v1rsquo
add column `bar`+
reads from `foo`writes to both`foo` and `bar`
v1rsquorsquo
populate `bar`where needed
What about renaming
v2
reads and writes to `bar`
v2rsquo
ignore `foo`
v3
drop column `foo`
NoSQL
NoSQL
bull Schema migrations O(n) locks are just part of hot compatibility
NoSQL
bull Schema migrations O(n) locks are just part of hot compatibility
bull You still need to put effort into data migration
Workflow
Workflow
bull Make sure pull requests are addressing hot compatibility
Workflow
bull Make sure pull requests are addressing hot compatibility
bull Work after code is merged
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Production
Production
bull Run multiple server processes
Production
bull Run multiple server processes
bull Cycle them individually or in batches
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
bull Cycle them gracefully
Cycling techniques
bull Above the stack
bull Within the stack
bull Platform
Above the stack
bull Use your favorite web server
bull Setup a load balancer above them route around servers that are cycling
Above the stack
bull PROTIP default setup wonrsquot work
bull Configure load balancer to poll for your application health
class HealthController lt ApplicationController def index if Fileexists(Railsrootjoin(MAINTENANCE)) render text =gt Forget me status =gt 503 else render text =gt Im alive end endend
Cycling + Load balancer
bounce serversrm MAINTENANCE
remove server from rotation
touch MAINTENANCE
new servers come online
add server to rotation
Server Load balancer
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy where to check for the health
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
do not let your web server queue requests
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy to run the health check specified above
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
every 2s
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
stop routing to it once the health check failed once
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
and bring it back once it worked once
ELB
$ elb-configure-healthcheck MyLoadBalancer --target HTTP5000health --interval 2 --unhealthy-threshold 1 --healthy-threshold 1
Above the stack
Above the stackbull Minimal impact
Above the stackbull Minimal impact
bull Extended availability comes for free
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
bull Another component you have to track and maintain
Within the stack
Within the stack
Unicorn
bull Replaces your web server
bull Bounces gracefully
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Within the stack
Within the stackbull Easy to setup and deploy
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
bull Doesnrsquot address availability issues
Platform
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 1s bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 5s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 1s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ heroku labsenable preboot
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 starting for 5s bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 up for 1s bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 5s bundle exec thin start -p $PORT
BETA
Heroku
bull Migrations
Heroku
bull Migrations
$ git push pre-production master$ heroku run rake dbmigrate --remote pre-production$ git push production master
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Future
Future
bull Asset pipeline
Future
bull Asset pipeline
bull Mysql tooling
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
bull Can databases
Future
Future
bull Hot compatibility beyond zero downtime deploys
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
Migration strategies
bull Read-only models
bull Ignore cached columns
class User def selfcolumns superreject |c| cname == notes endend
Migration strategies
bull Read-only models
bull Ignore cached columns
bull Know your database
Migration strategies
bull Read-only models
bull Ignore cached columns
bull Know your database
bull Beware of O(n) db locks
MyISAM
MyISAM
InnoDB
InnoDB
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
InnoDB
1 Create a new table with the same structure
2 Apply changes to new table
3 Copy data over
4 Acquire lock on original table
5 Sync newupdateddeleted rows
6 Swap tables
7 Release lock
InnoDB
1 Create a new table with the same structure
2 Apply changes to new table
3 Copy data over
4 Acquire lock on original table
5 Sync newupdateddeleted rows
6 Swap tables
7 Release lock
InnoDBhttpsgithubcomfreelstable_migrator
InnoDBhttpsgithubcomfreelstable_migrator
class AddStuffToMyBigTable lt TableMigration
migrates users
change_table do |t| tinteger foo null =gt false default =gt 0 tstring bar tindex foo name =gt index_for_foo endend
InnoDBhttpsgithubcomfreelstable_migrator
bull Requires acts_as_paranoid
bull Pretty oldstale codebase
bull Check forks for Rails 3 compatibility
Postgres
bull Table locks on
bull Adding updating or removing columns
bull Adding or removing indexes
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time if you avoid default values and constraints
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
can be done without any locking
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time
Postgres
bull Adding fields with defaults
Postgres
bull Adding fields with defaults
before_save assign_defaults
def assign_defaults selfadmin ||= falseend
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
CREATE INDEX CONCURRENTLY users_email_index ON users(email)
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
bull Outside a transaction
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
bull Outside a transaction
class AddIndexToUsers lt ActiveRecordMigration def selfup return if indexes(users)map(ampname)include(users_email_index) add_index users email name =gt users_email_index end
def selfdown remove_index users name =gt users_email_index endend
What about renaming
v1 v2
reads and writes to `foo`
reads and writes to `bar`
What about renaming
v1 v2
reads and writes to `foo`
reads and writes to `bar`
v1rsquo
add column `bar`+
reads from `foo`writes to both`foo` and `bar`
v1rsquorsquo
populate `bar`where needed
What about renaming
v2
reads and writes to `bar`
v2rsquo
ignore `foo`
v3
drop column `foo`
NoSQL
NoSQL
bull Schema migrations O(n) locks are just part of hot compatibility
NoSQL
bull Schema migrations O(n) locks are just part of hot compatibility
bull You still need to put effort into data migration
Workflow
Workflow
bull Make sure pull requests are addressing hot compatibility
Workflow
bull Make sure pull requests are addressing hot compatibility
bull Work after code is merged
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Production
Production
bull Run multiple server processes
Production
bull Run multiple server processes
bull Cycle them individually or in batches
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
bull Cycle them gracefully
Cycling techniques
bull Above the stack
bull Within the stack
bull Platform
Above the stack
bull Use your favorite web server
bull Setup a load balancer above them route around servers that are cycling
Above the stack
bull PROTIP default setup wonrsquot work
bull Configure load balancer to poll for your application health
class HealthController lt ApplicationController def index if Fileexists(Railsrootjoin(MAINTENANCE)) render text =gt Forget me status =gt 503 else render text =gt Im alive end endend
Cycling + Load balancer
bounce serversrm MAINTENANCE
remove server from rotation
touch MAINTENANCE
new servers come online
add server to rotation
Server Load balancer
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy where to check for the health
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
do not let your web server queue requests
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy to run the health check specified above
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
every 2s
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
stop routing to it once the health check failed once
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
and bring it back once it worked once
ELB
$ elb-configure-healthcheck MyLoadBalancer --target HTTP5000health --interval 2 --unhealthy-threshold 1 --healthy-threshold 1
Above the stack
Above the stackbull Minimal impact
Above the stackbull Minimal impact
bull Extended availability comes for free
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
bull Another component you have to track and maintain
Within the stack
Within the stack
Unicorn
bull Replaces your web server
bull Bounces gracefully
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Within the stack
Within the stackbull Easy to setup and deploy
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
bull Doesnrsquot address availability issues
Platform
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 1s bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 5s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 1s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ heroku labsenable preboot
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 starting for 5s bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 up for 1s bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 5s bundle exec thin start -p $PORT
BETA
Heroku
bull Migrations
Heroku
bull Migrations
$ git push pre-production master$ heroku run rake dbmigrate --remote pre-production$ git push production master
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Future
Future
bull Asset pipeline
Future
bull Asset pipeline
bull Mysql tooling
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
bull Can databases
Future
Future
bull Hot compatibility beyond zero downtime deploys
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
Migration strategies
bull Read-only models
bull Ignore cached columns
bull Know your database
Migration strategies
bull Read-only models
bull Ignore cached columns
bull Know your database
bull Beware of O(n) db locks
MyISAM
MyISAM
InnoDB
InnoDB
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
InnoDB
1 Create a new table with the same structure
2 Apply changes to new table
3 Copy data over
4 Acquire lock on original table
5 Sync newupdateddeleted rows
6 Swap tables
7 Release lock
InnoDB
1 Create a new table with the same structure
2 Apply changes to new table
3 Copy data over
4 Acquire lock on original table
5 Sync newupdateddeleted rows
6 Swap tables
7 Release lock
InnoDBhttpsgithubcomfreelstable_migrator
InnoDBhttpsgithubcomfreelstable_migrator
class AddStuffToMyBigTable lt TableMigration
migrates users
change_table do |t| tinteger foo null =gt false default =gt 0 tstring bar tindex foo name =gt index_for_foo endend
InnoDBhttpsgithubcomfreelstable_migrator
bull Requires acts_as_paranoid
bull Pretty oldstale codebase
bull Check forks for Rails 3 compatibility
Postgres
bull Table locks on
bull Adding updating or removing columns
bull Adding or removing indexes
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time if you avoid default values and constraints
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
can be done without any locking
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time
Postgres
bull Adding fields with defaults
Postgres
bull Adding fields with defaults
before_save assign_defaults
def assign_defaults selfadmin ||= falseend
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
CREATE INDEX CONCURRENTLY users_email_index ON users(email)
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
bull Outside a transaction
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
bull Outside a transaction
class AddIndexToUsers lt ActiveRecordMigration def selfup return if indexes(users)map(ampname)include(users_email_index) add_index users email name =gt users_email_index end
def selfdown remove_index users name =gt users_email_index endend
What about renaming
v1 v2
reads and writes to `foo`
reads and writes to `bar`
What about renaming
v1 v2
reads and writes to `foo`
reads and writes to `bar`
v1rsquo
add column `bar`+
reads from `foo`writes to both`foo` and `bar`
v1rsquorsquo
populate `bar`where needed
What about renaming
v2
reads and writes to `bar`
v2rsquo
ignore `foo`
v3
drop column `foo`
NoSQL
NoSQL
bull Schema migrations O(n) locks are just part of hot compatibility
NoSQL
bull Schema migrations O(n) locks are just part of hot compatibility
bull You still need to put effort into data migration
Workflow
Workflow
bull Make sure pull requests are addressing hot compatibility
Workflow
bull Make sure pull requests are addressing hot compatibility
bull Work after code is merged
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Production
Production
bull Run multiple server processes
Production
bull Run multiple server processes
bull Cycle them individually or in batches
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
bull Cycle them gracefully
Cycling techniques
bull Above the stack
bull Within the stack
bull Platform
Above the stack
bull Use your favorite web server
bull Setup a load balancer above them route around servers that are cycling
Above the stack
bull PROTIP default setup wonrsquot work
bull Configure load balancer to poll for your application health
class HealthController lt ApplicationController def index if Fileexists(Railsrootjoin(MAINTENANCE)) render text =gt Forget me status =gt 503 else render text =gt Im alive end endend
Cycling + Load balancer
bounce serversrm MAINTENANCE
remove server from rotation
touch MAINTENANCE
new servers come online
add server to rotation
Server Load balancer
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy where to check for the health
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
do not let your web server queue requests
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy to run the health check specified above
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
every 2s
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
stop routing to it once the health check failed once
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
and bring it back once it worked once
ELB
$ elb-configure-healthcheck MyLoadBalancer --target HTTP5000health --interval 2 --unhealthy-threshold 1 --healthy-threshold 1
Above the stack
Above the stackbull Minimal impact
Above the stackbull Minimal impact
bull Extended availability comes for free
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
bull Another component you have to track and maintain
Within the stack
Within the stack
Unicorn
bull Replaces your web server
bull Bounces gracefully
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Within the stack
Within the stackbull Easy to setup and deploy
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
bull Doesnrsquot address availability issues
Platform
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 1s bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 5s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 1s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ heroku labsenable preboot
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 starting for 5s bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 up for 1s bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 5s bundle exec thin start -p $PORT
BETA
Heroku
bull Migrations
Heroku
bull Migrations
$ git push pre-production master$ heroku run rake dbmigrate --remote pre-production$ git push production master
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Future
Future
bull Asset pipeline
Future
bull Asset pipeline
bull Mysql tooling
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
bull Can databases
Future
Future
bull Hot compatibility beyond zero downtime deploys
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
Migration strategies
bull Read-only models
bull Ignore cached columns
bull Know your database
bull Beware of O(n) db locks
MyISAM
MyISAM
InnoDB
InnoDB
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
InnoDB
1 Create a new table with the same structure
2 Apply changes to new table
3 Copy data over
4 Acquire lock on original table
5 Sync newupdateddeleted rows
6 Swap tables
7 Release lock
InnoDB
1 Create a new table with the same structure
2 Apply changes to new table
3 Copy data over
4 Acquire lock on original table
5 Sync newupdateddeleted rows
6 Swap tables
7 Release lock
InnoDBhttpsgithubcomfreelstable_migrator
InnoDBhttpsgithubcomfreelstable_migrator
class AddStuffToMyBigTable lt TableMigration
migrates users
change_table do |t| tinteger foo null =gt false default =gt 0 tstring bar tindex foo name =gt index_for_foo endend
InnoDBhttpsgithubcomfreelstable_migrator
bull Requires acts_as_paranoid
bull Pretty oldstale codebase
bull Check forks for Rails 3 compatibility
Postgres
bull Table locks on
bull Adding updating or removing columns
bull Adding or removing indexes
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time if you avoid default values and constraints
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
can be done without any locking
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time
Postgres
bull Adding fields with defaults
Postgres
bull Adding fields with defaults
before_save assign_defaults
def assign_defaults selfadmin ||= falseend
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
CREATE INDEX CONCURRENTLY users_email_index ON users(email)
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
bull Outside a transaction
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
bull Outside a transaction
class AddIndexToUsers lt ActiveRecordMigration def selfup return if indexes(users)map(ampname)include(users_email_index) add_index users email name =gt users_email_index end
def selfdown remove_index users name =gt users_email_index endend
What about renaming
v1 v2
reads and writes to `foo`
reads and writes to `bar`
What about renaming
v1 v2
reads and writes to `foo`
reads and writes to `bar`
v1rsquo
add column `bar`+
reads from `foo`writes to both`foo` and `bar`
v1rsquorsquo
populate `bar`where needed
What about renaming
v2
reads and writes to `bar`
v2rsquo
ignore `foo`
v3
drop column `foo`
NoSQL
NoSQL
bull Schema migrations O(n) locks are just part of hot compatibility
NoSQL
bull Schema migrations O(n) locks are just part of hot compatibility
bull You still need to put effort into data migration
Workflow
Workflow
bull Make sure pull requests are addressing hot compatibility
Workflow
bull Make sure pull requests are addressing hot compatibility
bull Work after code is merged
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Production
Production
bull Run multiple server processes
Production
bull Run multiple server processes
bull Cycle them individually or in batches
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
bull Cycle them gracefully
Cycling techniques
bull Above the stack
bull Within the stack
bull Platform
Above the stack
bull Use your favorite web server
bull Setup a load balancer above them route around servers that are cycling
Above the stack
bull PROTIP default setup wonrsquot work
bull Configure load balancer to poll for your application health
class HealthController lt ApplicationController def index if Fileexists(Railsrootjoin(MAINTENANCE)) render text =gt Forget me status =gt 503 else render text =gt Im alive end endend
Cycling + Load balancer
bounce serversrm MAINTENANCE
remove server from rotation
touch MAINTENANCE
new servers come online
add server to rotation
Server Load balancer
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy where to check for the health
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
do not let your web server queue requests
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy to run the health check specified above
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
every 2s
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
stop routing to it once the health check failed once
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
and bring it back once it worked once
ELB
$ elb-configure-healthcheck MyLoadBalancer --target HTTP5000health --interval 2 --unhealthy-threshold 1 --healthy-threshold 1
Above the stack
Above the stackbull Minimal impact
Above the stackbull Minimal impact
bull Extended availability comes for free
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
bull Another component you have to track and maintain
Within the stack
Within the stack
Unicorn
bull Replaces your web server
bull Bounces gracefully
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Within the stack
Within the stackbull Easy to setup and deploy
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
bull Doesnrsquot address availability issues
Platform
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 1s bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 5s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 1s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ heroku labsenable preboot
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 starting for 5s bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 up for 1s bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 5s bundle exec thin start -p $PORT
BETA
Heroku
bull Migrations
Heroku
bull Migrations
$ git push pre-production master$ heroku run rake dbmigrate --remote pre-production$ git push production master
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Future
Future
bull Asset pipeline
Future
bull Asset pipeline
bull Mysql tooling
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
bull Can databases
Future
Future
bull Hot compatibility beyond zero downtime deploys
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
MyISAM
MyISAM
InnoDB
InnoDB
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
InnoDB
1 Create a new table with the same structure
2 Apply changes to new table
3 Copy data over
4 Acquire lock on original table
5 Sync newupdateddeleted rows
6 Swap tables
7 Release lock
InnoDB
1 Create a new table with the same structure
2 Apply changes to new table
3 Copy data over
4 Acquire lock on original table
5 Sync newupdateddeleted rows
6 Swap tables
7 Release lock
InnoDBhttpsgithubcomfreelstable_migrator
InnoDBhttpsgithubcomfreelstable_migrator
class AddStuffToMyBigTable lt TableMigration
migrates users
change_table do |t| tinteger foo null =gt false default =gt 0 tstring bar tindex foo name =gt index_for_foo endend
InnoDBhttpsgithubcomfreelstable_migrator
bull Requires acts_as_paranoid
bull Pretty oldstale codebase
bull Check forks for Rails 3 compatibility
Postgres
bull Table locks on
bull Adding updating or removing columns
bull Adding or removing indexes
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time if you avoid default values and constraints
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
can be done without any locking
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time
Postgres
bull Adding fields with defaults
Postgres
bull Adding fields with defaults
before_save assign_defaults
def assign_defaults selfadmin ||= falseend
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
CREATE INDEX CONCURRENTLY users_email_index ON users(email)
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
bull Outside a transaction
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
bull Outside a transaction
class AddIndexToUsers lt ActiveRecordMigration def selfup return if indexes(users)map(ampname)include(users_email_index) add_index users email name =gt users_email_index end
def selfdown remove_index users name =gt users_email_index endend
What about renaming
v1 v2
reads and writes to `foo`
reads and writes to `bar`
What about renaming
v1 v2
reads and writes to `foo`
reads and writes to `bar`
v1rsquo
add column `bar`+
reads from `foo`writes to both`foo` and `bar`
v1rsquorsquo
populate `bar`where needed
What about renaming
v2
reads and writes to `bar`
v2rsquo
ignore `foo`
v3
drop column `foo`
NoSQL
NoSQL
bull Schema migrations O(n) locks are just part of hot compatibility
NoSQL
bull Schema migrations O(n) locks are just part of hot compatibility
bull You still need to put effort into data migration
Workflow
Workflow
bull Make sure pull requests are addressing hot compatibility
Workflow
bull Make sure pull requests are addressing hot compatibility
bull Work after code is merged
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Production
Production
bull Run multiple server processes
Production
bull Run multiple server processes
bull Cycle them individually or in batches
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
bull Cycle them gracefully
Cycling techniques
bull Above the stack
bull Within the stack
bull Platform
Above the stack
bull Use your favorite web server
bull Setup a load balancer above them route around servers that are cycling
Above the stack
bull PROTIP default setup wonrsquot work
bull Configure load balancer to poll for your application health
class HealthController lt ApplicationController def index if Fileexists(Railsrootjoin(MAINTENANCE)) render text =gt Forget me status =gt 503 else render text =gt Im alive end endend
Cycling + Load balancer
bounce serversrm MAINTENANCE
remove server from rotation
touch MAINTENANCE
new servers come online
add server to rotation
Server Load balancer
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy where to check for the health
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
do not let your web server queue requests
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy to run the health check specified above
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
every 2s
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
stop routing to it once the health check failed once
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
and bring it back once it worked once
ELB
$ elb-configure-healthcheck MyLoadBalancer --target HTTP5000health --interval 2 --unhealthy-threshold 1 --healthy-threshold 1
Above the stack
Above the stackbull Minimal impact
Above the stackbull Minimal impact
bull Extended availability comes for free
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
bull Another component you have to track and maintain
Within the stack
Within the stack
Unicorn
bull Replaces your web server
bull Bounces gracefully
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Within the stack
Within the stackbull Easy to setup and deploy
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
bull Doesnrsquot address availability issues
Platform
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 1s bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 5s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 1s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ heroku labsenable preboot
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 starting for 5s bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 up for 1s bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 5s bundle exec thin start -p $PORT
BETA
Heroku
bull Migrations
Heroku
bull Migrations
$ git push pre-production master$ heroku run rake dbmigrate --remote pre-production$ git push production master
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Future
Future
bull Asset pipeline
Future
bull Asset pipeline
bull Mysql tooling
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
bull Can databases
Future
Future
bull Hot compatibility beyond zero downtime deploys
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
MyISAM
InnoDB
InnoDB
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
InnoDB
1 Create a new table with the same structure
2 Apply changes to new table
3 Copy data over
4 Acquire lock on original table
5 Sync newupdateddeleted rows
6 Swap tables
7 Release lock
InnoDB
1 Create a new table with the same structure
2 Apply changes to new table
3 Copy data over
4 Acquire lock on original table
5 Sync newupdateddeleted rows
6 Swap tables
7 Release lock
InnoDBhttpsgithubcomfreelstable_migrator
InnoDBhttpsgithubcomfreelstable_migrator
class AddStuffToMyBigTable lt TableMigration
migrates users
change_table do |t| tinteger foo null =gt false default =gt 0 tstring bar tindex foo name =gt index_for_foo endend
InnoDBhttpsgithubcomfreelstable_migrator
bull Requires acts_as_paranoid
bull Pretty oldstale codebase
bull Check forks for Rails 3 compatibility
Postgres
bull Table locks on
bull Adding updating or removing columns
bull Adding or removing indexes
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time if you avoid default values and constraints
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
can be done without any locking
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time
Postgres
bull Adding fields with defaults
Postgres
bull Adding fields with defaults
before_save assign_defaults
def assign_defaults selfadmin ||= falseend
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
CREATE INDEX CONCURRENTLY users_email_index ON users(email)
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
bull Outside a transaction
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
bull Outside a transaction
class AddIndexToUsers lt ActiveRecordMigration def selfup return if indexes(users)map(ampname)include(users_email_index) add_index users email name =gt users_email_index end
def selfdown remove_index users name =gt users_email_index endend
What about renaming
v1 v2
reads and writes to `foo`
reads and writes to `bar`
What about renaming
v1 v2
reads and writes to `foo`
reads and writes to `bar`
v1rsquo
add column `bar`+
reads from `foo`writes to both`foo` and `bar`
v1rsquorsquo
populate `bar`where needed
What about renaming
v2
reads and writes to `bar`
v2rsquo
ignore `foo`
v3
drop column `foo`
NoSQL
NoSQL
bull Schema migrations O(n) locks are just part of hot compatibility
NoSQL
bull Schema migrations O(n) locks are just part of hot compatibility
bull You still need to put effort into data migration
Workflow
Workflow
bull Make sure pull requests are addressing hot compatibility
Workflow
bull Make sure pull requests are addressing hot compatibility
bull Work after code is merged
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Production
Production
bull Run multiple server processes
Production
bull Run multiple server processes
bull Cycle them individually or in batches
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
bull Cycle them gracefully
Cycling techniques
bull Above the stack
bull Within the stack
bull Platform
Above the stack
bull Use your favorite web server
bull Setup a load balancer above them route around servers that are cycling
Above the stack
bull PROTIP default setup wonrsquot work
bull Configure load balancer to poll for your application health
class HealthController lt ApplicationController def index if Fileexists(Railsrootjoin(MAINTENANCE)) render text =gt Forget me status =gt 503 else render text =gt Im alive end endend
Cycling + Load balancer
bounce serversrm MAINTENANCE
remove server from rotation
touch MAINTENANCE
new servers come online
add server to rotation
Server Load balancer
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy where to check for the health
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
do not let your web server queue requests
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy to run the health check specified above
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
every 2s
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
stop routing to it once the health check failed once
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
and bring it back once it worked once
ELB
$ elb-configure-healthcheck MyLoadBalancer --target HTTP5000health --interval 2 --unhealthy-threshold 1 --healthy-threshold 1
Above the stack
Above the stackbull Minimal impact
Above the stackbull Minimal impact
bull Extended availability comes for free
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
bull Another component you have to track and maintain
Within the stack
Within the stack
Unicorn
bull Replaces your web server
bull Bounces gracefully
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Within the stack
Within the stackbull Easy to setup and deploy
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
bull Doesnrsquot address availability issues
Platform
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 1s bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 5s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 1s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ heroku labsenable preboot
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 starting for 5s bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 up for 1s bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 5s bundle exec thin start -p $PORT
BETA
Heroku
bull Migrations
Heroku
bull Migrations
$ git push pre-production master$ heroku run rake dbmigrate --remote pre-production$ git push production master
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Future
Future
bull Asset pipeline
Future
bull Asset pipeline
bull Mysql tooling
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
bull Can databases
Future
Future
bull Hot compatibility beyond zero downtime deploys
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
InnoDB
InnoDB
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
InnoDB
1 Create a new table with the same structure
2 Apply changes to new table
3 Copy data over
4 Acquire lock on original table
5 Sync newupdateddeleted rows
6 Swap tables
7 Release lock
InnoDB
1 Create a new table with the same structure
2 Apply changes to new table
3 Copy data over
4 Acquire lock on original table
5 Sync newupdateddeleted rows
6 Swap tables
7 Release lock
InnoDBhttpsgithubcomfreelstable_migrator
InnoDBhttpsgithubcomfreelstable_migrator
class AddStuffToMyBigTable lt TableMigration
migrates users
change_table do |t| tinteger foo null =gt false default =gt 0 tstring bar tindex foo name =gt index_for_foo endend
InnoDBhttpsgithubcomfreelstable_migrator
bull Requires acts_as_paranoid
bull Pretty oldstale codebase
bull Check forks for Rails 3 compatibility
Postgres
bull Table locks on
bull Adding updating or removing columns
bull Adding or removing indexes
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time if you avoid default values and constraints
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
can be done without any locking
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time
Postgres
bull Adding fields with defaults
Postgres
bull Adding fields with defaults
before_save assign_defaults
def assign_defaults selfadmin ||= falseend
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
CREATE INDEX CONCURRENTLY users_email_index ON users(email)
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
bull Outside a transaction
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
bull Outside a transaction
class AddIndexToUsers lt ActiveRecordMigration def selfup return if indexes(users)map(ampname)include(users_email_index) add_index users email name =gt users_email_index end
def selfdown remove_index users name =gt users_email_index endend
What about renaming
v1 v2
reads and writes to `foo`
reads and writes to `bar`
What about renaming
v1 v2
reads and writes to `foo`
reads and writes to `bar`
v1rsquo
add column `bar`+
reads from `foo`writes to both`foo` and `bar`
v1rsquorsquo
populate `bar`where needed
What about renaming
v2
reads and writes to `bar`
v2rsquo
ignore `foo`
v3
drop column `foo`
NoSQL
NoSQL
bull Schema migrations O(n) locks are just part of hot compatibility
NoSQL
bull Schema migrations O(n) locks are just part of hot compatibility
bull You still need to put effort into data migration
Workflow
Workflow
bull Make sure pull requests are addressing hot compatibility
Workflow
bull Make sure pull requests are addressing hot compatibility
bull Work after code is merged
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Production
Production
bull Run multiple server processes
Production
bull Run multiple server processes
bull Cycle them individually or in batches
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
bull Cycle them gracefully
Cycling techniques
bull Above the stack
bull Within the stack
bull Platform
Above the stack
bull Use your favorite web server
bull Setup a load balancer above them route around servers that are cycling
Above the stack
bull PROTIP default setup wonrsquot work
bull Configure load balancer to poll for your application health
class HealthController lt ApplicationController def index if Fileexists(Railsrootjoin(MAINTENANCE)) render text =gt Forget me status =gt 503 else render text =gt Im alive end endend
Cycling + Load balancer
bounce serversrm MAINTENANCE
remove server from rotation
touch MAINTENANCE
new servers come online
add server to rotation
Server Load balancer
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy where to check for the health
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
do not let your web server queue requests
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy to run the health check specified above
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
every 2s
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
stop routing to it once the health check failed once
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
and bring it back once it worked once
ELB
$ elb-configure-healthcheck MyLoadBalancer --target HTTP5000health --interval 2 --unhealthy-threshold 1 --healthy-threshold 1
Above the stack
Above the stackbull Minimal impact
Above the stackbull Minimal impact
bull Extended availability comes for free
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
bull Another component you have to track and maintain
Within the stack
Within the stack
Unicorn
bull Replaces your web server
bull Bounces gracefully
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Within the stack
Within the stackbull Easy to setup and deploy
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
bull Doesnrsquot address availability issues
Platform
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 1s bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 5s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 1s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ heroku labsenable preboot
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 starting for 5s bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 up for 1s bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 5s bundle exec thin start -p $PORT
BETA
Heroku
bull Migrations
Heroku
bull Migrations
$ git push pre-production master$ heroku run rake dbmigrate --remote pre-production$ git push production master
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Future
Future
bull Asset pipeline
Future
bull Asset pipeline
bull Mysql tooling
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
bull Can databases
Future
Future
bull Hot compatibility beyond zero downtime deploys
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
InnoDB
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
InnoDB
1 Create a new table with the same structure
2 Apply changes to new table
3 Copy data over
4 Acquire lock on original table
5 Sync newupdateddeleted rows
6 Swap tables
7 Release lock
InnoDB
1 Create a new table with the same structure
2 Apply changes to new table
3 Copy data over
4 Acquire lock on original table
5 Sync newupdateddeleted rows
6 Swap tables
7 Release lock
InnoDBhttpsgithubcomfreelstable_migrator
InnoDBhttpsgithubcomfreelstable_migrator
class AddStuffToMyBigTable lt TableMigration
migrates users
change_table do |t| tinteger foo null =gt false default =gt 0 tstring bar tindex foo name =gt index_for_foo endend
InnoDBhttpsgithubcomfreelstable_migrator
bull Requires acts_as_paranoid
bull Pretty oldstale codebase
bull Check forks for Rails 3 compatibility
Postgres
bull Table locks on
bull Adding updating or removing columns
bull Adding or removing indexes
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time if you avoid default values and constraints
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
can be done without any locking
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time
Postgres
bull Adding fields with defaults
Postgres
bull Adding fields with defaults
before_save assign_defaults
def assign_defaults selfadmin ||= falseend
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
CREATE INDEX CONCURRENTLY users_email_index ON users(email)
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
bull Outside a transaction
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
bull Outside a transaction
class AddIndexToUsers lt ActiveRecordMigration def selfup return if indexes(users)map(ampname)include(users_email_index) add_index users email name =gt users_email_index end
def selfdown remove_index users name =gt users_email_index endend
What about renaming
v1 v2
reads and writes to `foo`
reads and writes to `bar`
What about renaming
v1 v2
reads and writes to `foo`
reads and writes to `bar`
v1rsquo
add column `bar`+
reads from `foo`writes to both`foo` and `bar`
v1rsquorsquo
populate `bar`where needed
What about renaming
v2
reads and writes to `bar`
v2rsquo
ignore `foo`
v3
drop column `foo`
NoSQL
NoSQL
bull Schema migrations O(n) locks are just part of hot compatibility
NoSQL
bull Schema migrations O(n) locks are just part of hot compatibility
bull You still need to put effort into data migration
Workflow
Workflow
bull Make sure pull requests are addressing hot compatibility
Workflow
bull Make sure pull requests are addressing hot compatibility
bull Work after code is merged
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Production
Production
bull Run multiple server processes
Production
bull Run multiple server processes
bull Cycle them individually or in batches
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
bull Cycle them gracefully
Cycling techniques
bull Above the stack
bull Within the stack
bull Platform
Above the stack
bull Use your favorite web server
bull Setup a load balancer above them route around servers that are cycling
Above the stack
bull PROTIP default setup wonrsquot work
bull Configure load balancer to poll for your application health
class HealthController lt ApplicationController def index if Fileexists(Railsrootjoin(MAINTENANCE)) render text =gt Forget me status =gt 503 else render text =gt Im alive end endend
Cycling + Load balancer
bounce serversrm MAINTENANCE
remove server from rotation
touch MAINTENANCE
new servers come online
add server to rotation
Server Load balancer
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy where to check for the health
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
do not let your web server queue requests
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy to run the health check specified above
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
every 2s
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
stop routing to it once the health check failed once
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
and bring it back once it worked once
ELB
$ elb-configure-healthcheck MyLoadBalancer --target HTTP5000health --interval 2 --unhealthy-threshold 1 --healthy-threshold 1
Above the stack
Above the stackbull Minimal impact
Above the stackbull Minimal impact
bull Extended availability comes for free
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
bull Another component you have to track and maintain
Within the stack
Within the stack
Unicorn
bull Replaces your web server
bull Bounces gracefully
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Within the stack
Within the stackbull Easy to setup and deploy
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
bull Doesnrsquot address availability issues
Platform
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 1s bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 5s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 1s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ heroku labsenable preboot
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 starting for 5s bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 up for 1s bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 5s bundle exec thin start -p $PORT
BETA
Heroku
bull Migrations
Heroku
bull Migrations
$ git push pre-production master$ heroku run rake dbmigrate --remote pre-production$ git push production master
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Future
Future
bull Asset pipeline
Future
bull Asset pipeline
bull Mysql tooling
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
bull Can databases
Future
Future
bull Hot compatibility beyond zero downtime deploys
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
InnoDB
1 Create a new table with the same structure
2 Apply changes to new table
3 Copy data over
4 Acquire lock on original table
5 Sync newupdateddeleted rows
6 Swap tables
7 Release lock
InnoDB
1 Create a new table with the same structure
2 Apply changes to new table
3 Copy data over
4 Acquire lock on original table
5 Sync newupdateddeleted rows
6 Swap tables
7 Release lock
InnoDBhttpsgithubcomfreelstable_migrator
InnoDBhttpsgithubcomfreelstable_migrator
class AddStuffToMyBigTable lt TableMigration
migrates users
change_table do |t| tinteger foo null =gt false default =gt 0 tstring bar tindex foo name =gt index_for_foo endend
InnoDBhttpsgithubcomfreelstable_migrator
bull Requires acts_as_paranoid
bull Pretty oldstale codebase
bull Check forks for Rails 3 compatibility
Postgres
bull Table locks on
bull Adding updating or removing columns
bull Adding or removing indexes
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time if you avoid default values and constraints
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
can be done without any locking
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time
Postgres
bull Adding fields with defaults
Postgres
bull Adding fields with defaults
before_save assign_defaults
def assign_defaults selfadmin ||= falseend
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
CREATE INDEX CONCURRENTLY users_email_index ON users(email)
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
bull Outside a transaction
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
bull Outside a transaction
class AddIndexToUsers lt ActiveRecordMigration def selfup return if indexes(users)map(ampname)include(users_email_index) add_index users email name =gt users_email_index end
def selfdown remove_index users name =gt users_email_index endend
What about renaming
v1 v2
reads and writes to `foo`
reads and writes to `bar`
What about renaming
v1 v2
reads and writes to `foo`
reads and writes to `bar`
v1rsquo
add column `bar`+
reads from `foo`writes to both`foo` and `bar`
v1rsquorsquo
populate `bar`where needed
What about renaming
v2
reads and writes to `bar`
v2rsquo
ignore `foo`
v3
drop column `foo`
NoSQL
NoSQL
bull Schema migrations O(n) locks are just part of hot compatibility
NoSQL
bull Schema migrations O(n) locks are just part of hot compatibility
bull You still need to put effort into data migration
Workflow
Workflow
bull Make sure pull requests are addressing hot compatibility
Workflow
bull Make sure pull requests are addressing hot compatibility
bull Work after code is merged
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Production
Production
bull Run multiple server processes
Production
bull Run multiple server processes
bull Cycle them individually or in batches
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
bull Cycle them gracefully
Cycling techniques
bull Above the stack
bull Within the stack
bull Platform
Above the stack
bull Use your favorite web server
bull Setup a load balancer above them route around servers that are cycling
Above the stack
bull PROTIP default setup wonrsquot work
bull Configure load balancer to poll for your application health
class HealthController lt ApplicationController def index if Fileexists(Railsrootjoin(MAINTENANCE)) render text =gt Forget me status =gt 503 else render text =gt Im alive end endend
Cycling + Load balancer
bounce serversrm MAINTENANCE
remove server from rotation
touch MAINTENANCE
new servers come online
add server to rotation
Server Load balancer
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy where to check for the health
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
do not let your web server queue requests
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy to run the health check specified above
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
every 2s
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
stop routing to it once the health check failed once
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
and bring it back once it worked once
ELB
$ elb-configure-healthcheck MyLoadBalancer --target HTTP5000health --interval 2 --unhealthy-threshold 1 --healthy-threshold 1
Above the stack
Above the stackbull Minimal impact
Above the stackbull Minimal impact
bull Extended availability comes for free
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
bull Another component you have to track and maintain
Within the stack
Within the stack
Unicorn
bull Replaces your web server
bull Bounces gracefully
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Within the stack
Within the stackbull Easy to setup and deploy
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
bull Doesnrsquot address availability issues
Platform
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 1s bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 5s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 1s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ heroku labsenable preboot
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 starting for 5s bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 up for 1s bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 5s bundle exec thin start -p $PORT
BETA
Heroku
bull Migrations
Heroku
bull Migrations
$ git push pre-production master$ heroku run rake dbmigrate --remote pre-production$ git push production master
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Future
Future
bull Asset pipeline
Future
bull Asset pipeline
bull Mysql tooling
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
bull Can databases
Future
Future
bull Hot compatibility beyond zero downtime deploys
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
InnoDB
1 Create a new table with the same structure
2 Apply changes to new table
3 Copy data over
4 Acquire lock on original table
5 Sync newupdateddeleted rows
6 Swap tables
7 Release lock
InnoDBhttpsgithubcomfreelstable_migrator
InnoDBhttpsgithubcomfreelstable_migrator
class AddStuffToMyBigTable lt TableMigration
migrates users
change_table do |t| tinteger foo null =gt false default =gt 0 tstring bar tindex foo name =gt index_for_foo endend
InnoDBhttpsgithubcomfreelstable_migrator
bull Requires acts_as_paranoid
bull Pretty oldstale codebase
bull Check forks for Rails 3 compatibility
Postgres
bull Table locks on
bull Adding updating or removing columns
bull Adding or removing indexes
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time if you avoid default values and constraints
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
can be done without any locking
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time
Postgres
bull Adding fields with defaults
Postgres
bull Adding fields with defaults
before_save assign_defaults
def assign_defaults selfadmin ||= falseend
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
CREATE INDEX CONCURRENTLY users_email_index ON users(email)
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
bull Outside a transaction
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
bull Outside a transaction
class AddIndexToUsers lt ActiveRecordMigration def selfup return if indexes(users)map(ampname)include(users_email_index) add_index users email name =gt users_email_index end
def selfdown remove_index users name =gt users_email_index endend
What about renaming
v1 v2
reads and writes to `foo`
reads and writes to `bar`
What about renaming
v1 v2
reads and writes to `foo`
reads and writes to `bar`
v1rsquo
add column `bar`+
reads from `foo`writes to both`foo` and `bar`
v1rsquorsquo
populate `bar`where needed
What about renaming
v2
reads and writes to `bar`
v2rsquo
ignore `foo`
v3
drop column `foo`
NoSQL
NoSQL
bull Schema migrations O(n) locks are just part of hot compatibility
NoSQL
bull Schema migrations O(n) locks are just part of hot compatibility
bull You still need to put effort into data migration
Workflow
Workflow
bull Make sure pull requests are addressing hot compatibility
Workflow
bull Make sure pull requests are addressing hot compatibility
bull Work after code is merged
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Production
Production
bull Run multiple server processes
Production
bull Run multiple server processes
bull Cycle them individually or in batches
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
bull Cycle them gracefully
Cycling techniques
bull Above the stack
bull Within the stack
bull Platform
Above the stack
bull Use your favorite web server
bull Setup a load balancer above them route around servers that are cycling
Above the stack
bull PROTIP default setup wonrsquot work
bull Configure load balancer to poll for your application health
class HealthController lt ApplicationController def index if Fileexists(Railsrootjoin(MAINTENANCE)) render text =gt Forget me status =gt 503 else render text =gt Im alive end endend
Cycling + Load balancer
bounce serversrm MAINTENANCE
remove server from rotation
touch MAINTENANCE
new servers come online
add server to rotation
Server Load balancer
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy where to check for the health
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
do not let your web server queue requests
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy to run the health check specified above
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
every 2s
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
stop routing to it once the health check failed once
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
and bring it back once it worked once
ELB
$ elb-configure-healthcheck MyLoadBalancer --target HTTP5000health --interval 2 --unhealthy-threshold 1 --healthy-threshold 1
Above the stack
Above the stackbull Minimal impact
Above the stackbull Minimal impact
bull Extended availability comes for free
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
bull Another component you have to track and maintain
Within the stack
Within the stack
Unicorn
bull Replaces your web server
bull Bounces gracefully
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Within the stack
Within the stackbull Easy to setup and deploy
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
bull Doesnrsquot address availability issues
Platform
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 1s bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 5s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 1s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ heroku labsenable preboot
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 starting for 5s bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 up for 1s bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 5s bundle exec thin start -p $PORT
BETA
Heroku
bull Migrations
Heroku
bull Migrations
$ git push pre-production master$ heroku run rake dbmigrate --remote pre-production$ git push production master
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Future
Future
bull Asset pipeline
Future
bull Asset pipeline
bull Mysql tooling
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
bull Can databases
Future
Future
bull Hot compatibility beyond zero downtime deploys
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
InnoDBhttpsgithubcomfreelstable_migrator
InnoDBhttpsgithubcomfreelstable_migrator
class AddStuffToMyBigTable lt TableMigration
migrates users
change_table do |t| tinteger foo null =gt false default =gt 0 tstring bar tindex foo name =gt index_for_foo endend
InnoDBhttpsgithubcomfreelstable_migrator
bull Requires acts_as_paranoid
bull Pretty oldstale codebase
bull Check forks for Rails 3 compatibility
Postgres
bull Table locks on
bull Adding updating or removing columns
bull Adding or removing indexes
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time if you avoid default values and constraints
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
can be done without any locking
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time
Postgres
bull Adding fields with defaults
Postgres
bull Adding fields with defaults
before_save assign_defaults
def assign_defaults selfadmin ||= falseend
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
CREATE INDEX CONCURRENTLY users_email_index ON users(email)
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
bull Outside a transaction
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
bull Outside a transaction
class AddIndexToUsers lt ActiveRecordMigration def selfup return if indexes(users)map(ampname)include(users_email_index) add_index users email name =gt users_email_index end
def selfdown remove_index users name =gt users_email_index endend
What about renaming
v1 v2
reads and writes to `foo`
reads and writes to `bar`
What about renaming
v1 v2
reads and writes to `foo`
reads and writes to `bar`
v1rsquo
add column `bar`+
reads from `foo`writes to both`foo` and `bar`
v1rsquorsquo
populate `bar`where needed
What about renaming
v2
reads and writes to `bar`
v2rsquo
ignore `foo`
v3
drop column `foo`
NoSQL
NoSQL
bull Schema migrations O(n) locks are just part of hot compatibility
NoSQL
bull Schema migrations O(n) locks are just part of hot compatibility
bull You still need to put effort into data migration
Workflow
Workflow
bull Make sure pull requests are addressing hot compatibility
Workflow
bull Make sure pull requests are addressing hot compatibility
bull Work after code is merged
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Production
Production
bull Run multiple server processes
Production
bull Run multiple server processes
bull Cycle them individually or in batches
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
bull Cycle them gracefully
Cycling techniques
bull Above the stack
bull Within the stack
bull Platform
Above the stack
bull Use your favorite web server
bull Setup a load balancer above them route around servers that are cycling
Above the stack
bull PROTIP default setup wonrsquot work
bull Configure load balancer to poll for your application health
class HealthController lt ApplicationController def index if Fileexists(Railsrootjoin(MAINTENANCE)) render text =gt Forget me status =gt 503 else render text =gt Im alive end endend
Cycling + Load balancer
bounce serversrm MAINTENANCE
remove server from rotation
touch MAINTENANCE
new servers come online
add server to rotation
Server Load balancer
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy where to check for the health
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
do not let your web server queue requests
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy to run the health check specified above
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
every 2s
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
stop routing to it once the health check failed once
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
and bring it back once it worked once
ELB
$ elb-configure-healthcheck MyLoadBalancer --target HTTP5000health --interval 2 --unhealthy-threshold 1 --healthy-threshold 1
Above the stack
Above the stackbull Minimal impact
Above the stackbull Minimal impact
bull Extended availability comes for free
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
bull Another component you have to track and maintain
Within the stack
Within the stack
Unicorn
bull Replaces your web server
bull Bounces gracefully
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Within the stack
Within the stackbull Easy to setup and deploy
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
bull Doesnrsquot address availability issues
Platform
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 1s bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 5s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 1s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ heroku labsenable preboot
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 starting for 5s bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 up for 1s bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 5s bundle exec thin start -p $PORT
BETA
Heroku
bull Migrations
Heroku
bull Migrations
$ git push pre-production master$ heroku run rake dbmigrate --remote pre-production$ git push production master
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Future
Future
bull Asset pipeline
Future
bull Asset pipeline
bull Mysql tooling
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
bull Can databases
Future
Future
bull Hot compatibility beyond zero downtime deploys
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
InnoDBhttpsgithubcomfreelstable_migrator
class AddStuffToMyBigTable lt TableMigration
migrates users
change_table do |t| tinteger foo null =gt false default =gt 0 tstring bar tindex foo name =gt index_for_foo endend
InnoDBhttpsgithubcomfreelstable_migrator
bull Requires acts_as_paranoid
bull Pretty oldstale codebase
bull Check forks for Rails 3 compatibility
Postgres
bull Table locks on
bull Adding updating or removing columns
bull Adding or removing indexes
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time if you avoid default values and constraints
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
can be done without any locking
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time
Postgres
bull Adding fields with defaults
Postgres
bull Adding fields with defaults
before_save assign_defaults
def assign_defaults selfadmin ||= falseend
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
CREATE INDEX CONCURRENTLY users_email_index ON users(email)
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
bull Outside a transaction
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
bull Outside a transaction
class AddIndexToUsers lt ActiveRecordMigration def selfup return if indexes(users)map(ampname)include(users_email_index) add_index users email name =gt users_email_index end
def selfdown remove_index users name =gt users_email_index endend
What about renaming
v1 v2
reads and writes to `foo`
reads and writes to `bar`
What about renaming
v1 v2
reads and writes to `foo`
reads and writes to `bar`
v1rsquo
add column `bar`+
reads from `foo`writes to both`foo` and `bar`
v1rsquorsquo
populate `bar`where needed
What about renaming
v2
reads and writes to `bar`
v2rsquo
ignore `foo`
v3
drop column `foo`
NoSQL
NoSQL
bull Schema migrations O(n) locks are just part of hot compatibility
NoSQL
bull Schema migrations O(n) locks are just part of hot compatibility
bull You still need to put effort into data migration
Workflow
Workflow
bull Make sure pull requests are addressing hot compatibility
Workflow
bull Make sure pull requests are addressing hot compatibility
bull Work after code is merged
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Production
Production
bull Run multiple server processes
Production
bull Run multiple server processes
bull Cycle them individually or in batches
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
bull Cycle them gracefully
Cycling techniques
bull Above the stack
bull Within the stack
bull Platform
Above the stack
bull Use your favorite web server
bull Setup a load balancer above them route around servers that are cycling
Above the stack
bull PROTIP default setup wonrsquot work
bull Configure load balancer to poll for your application health
class HealthController lt ApplicationController def index if Fileexists(Railsrootjoin(MAINTENANCE)) render text =gt Forget me status =gt 503 else render text =gt Im alive end endend
Cycling + Load balancer
bounce serversrm MAINTENANCE
remove server from rotation
touch MAINTENANCE
new servers come online
add server to rotation
Server Load balancer
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy where to check for the health
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
do not let your web server queue requests
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy to run the health check specified above
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
every 2s
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
stop routing to it once the health check failed once
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
and bring it back once it worked once
ELB
$ elb-configure-healthcheck MyLoadBalancer --target HTTP5000health --interval 2 --unhealthy-threshold 1 --healthy-threshold 1
Above the stack
Above the stackbull Minimal impact
Above the stackbull Minimal impact
bull Extended availability comes for free
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
bull Another component you have to track and maintain
Within the stack
Within the stack
Unicorn
bull Replaces your web server
bull Bounces gracefully
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Within the stack
Within the stackbull Easy to setup and deploy
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
bull Doesnrsquot address availability issues
Platform
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 1s bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 5s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 1s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ heroku labsenable preboot
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 starting for 5s bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 up for 1s bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 5s bundle exec thin start -p $PORT
BETA
Heroku
bull Migrations
Heroku
bull Migrations
$ git push pre-production master$ heroku run rake dbmigrate --remote pre-production$ git push production master
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Future
Future
bull Asset pipeline
Future
bull Asset pipeline
bull Mysql tooling
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
bull Can databases
Future
Future
bull Hot compatibility beyond zero downtime deploys
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
InnoDBhttpsgithubcomfreelstable_migrator
bull Requires acts_as_paranoid
bull Pretty oldstale codebase
bull Check forks for Rails 3 compatibility
Postgres
bull Table locks on
bull Adding updating or removing columns
bull Adding or removing indexes
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time if you avoid default values and constraints
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
can be done without any locking
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time
Postgres
bull Adding fields with defaults
Postgres
bull Adding fields with defaults
before_save assign_defaults
def assign_defaults selfadmin ||= falseend
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
CREATE INDEX CONCURRENTLY users_email_index ON users(email)
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
bull Outside a transaction
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
bull Outside a transaction
class AddIndexToUsers lt ActiveRecordMigration def selfup return if indexes(users)map(ampname)include(users_email_index) add_index users email name =gt users_email_index end
def selfdown remove_index users name =gt users_email_index endend
What about renaming
v1 v2
reads and writes to `foo`
reads and writes to `bar`
What about renaming
v1 v2
reads and writes to `foo`
reads and writes to `bar`
v1rsquo
add column `bar`+
reads from `foo`writes to both`foo` and `bar`
v1rsquorsquo
populate `bar`where needed
What about renaming
v2
reads and writes to `bar`
v2rsquo
ignore `foo`
v3
drop column `foo`
NoSQL
NoSQL
bull Schema migrations O(n) locks are just part of hot compatibility
NoSQL
bull Schema migrations O(n) locks are just part of hot compatibility
bull You still need to put effort into data migration
Workflow
Workflow
bull Make sure pull requests are addressing hot compatibility
Workflow
bull Make sure pull requests are addressing hot compatibility
bull Work after code is merged
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Production
Production
bull Run multiple server processes
Production
bull Run multiple server processes
bull Cycle them individually or in batches
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
bull Cycle them gracefully
Cycling techniques
bull Above the stack
bull Within the stack
bull Platform
Above the stack
bull Use your favorite web server
bull Setup a load balancer above them route around servers that are cycling
Above the stack
bull PROTIP default setup wonrsquot work
bull Configure load balancer to poll for your application health
class HealthController lt ApplicationController def index if Fileexists(Railsrootjoin(MAINTENANCE)) render text =gt Forget me status =gt 503 else render text =gt Im alive end endend
Cycling + Load balancer
bounce serversrm MAINTENANCE
remove server from rotation
touch MAINTENANCE
new servers come online
add server to rotation
Server Load balancer
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy where to check for the health
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
do not let your web server queue requests
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy to run the health check specified above
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
every 2s
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
stop routing to it once the health check failed once
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
and bring it back once it worked once
ELB
$ elb-configure-healthcheck MyLoadBalancer --target HTTP5000health --interval 2 --unhealthy-threshold 1 --healthy-threshold 1
Above the stack
Above the stackbull Minimal impact
Above the stackbull Minimal impact
bull Extended availability comes for free
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
bull Another component you have to track and maintain
Within the stack
Within the stack
Unicorn
bull Replaces your web server
bull Bounces gracefully
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Within the stack
Within the stackbull Easy to setup and deploy
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
bull Doesnrsquot address availability issues
Platform
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 1s bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 5s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 1s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ heroku labsenable preboot
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 starting for 5s bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 up for 1s bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 5s bundle exec thin start -p $PORT
BETA
Heroku
bull Migrations
Heroku
bull Migrations
$ git push pre-production master$ heroku run rake dbmigrate --remote pre-production$ git push production master
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Future
Future
bull Asset pipeline
Future
bull Asset pipeline
bull Mysql tooling
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
bull Can databases
Future
Future
bull Hot compatibility beyond zero downtime deploys
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
Postgres
bull Table locks on
bull Adding updating or removing columns
bull Adding or removing indexes
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time if you avoid default values and constraints
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
can be done without any locking
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time
Postgres
bull Adding fields with defaults
Postgres
bull Adding fields with defaults
before_save assign_defaults
def assign_defaults selfadmin ||= falseend
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
CREATE INDEX CONCURRENTLY users_email_index ON users(email)
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
bull Outside a transaction
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
bull Outside a transaction
class AddIndexToUsers lt ActiveRecordMigration def selfup return if indexes(users)map(ampname)include(users_email_index) add_index users email name =gt users_email_index end
def selfdown remove_index users name =gt users_email_index endend
What about renaming
v1 v2
reads and writes to `foo`
reads and writes to `bar`
What about renaming
v1 v2
reads and writes to `foo`
reads and writes to `bar`
v1rsquo
add column `bar`+
reads from `foo`writes to both`foo` and `bar`
v1rsquorsquo
populate `bar`where needed
What about renaming
v2
reads and writes to `bar`
v2rsquo
ignore `foo`
v3
drop column `foo`
NoSQL
NoSQL
bull Schema migrations O(n) locks are just part of hot compatibility
NoSQL
bull Schema migrations O(n) locks are just part of hot compatibility
bull You still need to put effort into data migration
Workflow
Workflow
bull Make sure pull requests are addressing hot compatibility
Workflow
bull Make sure pull requests are addressing hot compatibility
bull Work after code is merged
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Production
Production
bull Run multiple server processes
Production
bull Run multiple server processes
bull Cycle them individually or in batches
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
bull Cycle them gracefully
Cycling techniques
bull Above the stack
bull Within the stack
bull Platform
Above the stack
bull Use your favorite web server
bull Setup a load balancer above them route around servers that are cycling
Above the stack
bull PROTIP default setup wonrsquot work
bull Configure load balancer to poll for your application health
class HealthController lt ApplicationController def index if Fileexists(Railsrootjoin(MAINTENANCE)) render text =gt Forget me status =gt 503 else render text =gt Im alive end endend
Cycling + Load balancer
bounce serversrm MAINTENANCE
remove server from rotation
touch MAINTENANCE
new servers come online
add server to rotation
Server Load balancer
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy where to check for the health
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
do not let your web server queue requests
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy to run the health check specified above
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
every 2s
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
stop routing to it once the health check failed once
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
and bring it back once it worked once
ELB
$ elb-configure-healthcheck MyLoadBalancer --target HTTP5000health --interval 2 --unhealthy-threshold 1 --healthy-threshold 1
Above the stack
Above the stackbull Minimal impact
Above the stackbull Minimal impact
bull Extended availability comes for free
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
bull Another component you have to track and maintain
Within the stack
Within the stack
Unicorn
bull Replaces your web server
bull Bounces gracefully
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Within the stack
Within the stackbull Easy to setup and deploy
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
bull Doesnrsquot address availability issues
Platform
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 1s bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 5s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 1s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ heroku labsenable preboot
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 starting for 5s bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 up for 1s bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 5s bundle exec thin start -p $PORT
BETA
Heroku
bull Migrations
Heroku
bull Migrations
$ git push pre-production master$ heroku run rake dbmigrate --remote pre-production$ git push production master
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Future
Future
bull Asset pipeline
Future
bull Asset pipeline
bull Mysql tooling
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
bull Can databases
Future
Future
bull Hot compatibility beyond zero downtime deploys
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time if you avoid default values and constraints
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
can be done without any locking
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time
Postgres
bull Adding fields with defaults
Postgres
bull Adding fields with defaults
before_save assign_defaults
def assign_defaults selfadmin ||= falseend
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
CREATE INDEX CONCURRENTLY users_email_index ON users(email)
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
bull Outside a transaction
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
bull Outside a transaction
class AddIndexToUsers lt ActiveRecordMigration def selfup return if indexes(users)map(ampname)include(users_email_index) add_index users email name =gt users_email_index end
def selfdown remove_index users name =gt users_email_index endend
What about renaming
v1 v2
reads and writes to `foo`
reads and writes to `bar`
What about renaming
v1 v2
reads and writes to `foo`
reads and writes to `bar`
v1rsquo
add column `bar`+
reads from `foo`writes to both`foo` and `bar`
v1rsquorsquo
populate `bar`where needed
What about renaming
v2
reads and writes to `bar`
v2rsquo
ignore `foo`
v3
drop column `foo`
NoSQL
NoSQL
bull Schema migrations O(n) locks are just part of hot compatibility
NoSQL
bull Schema migrations O(n) locks are just part of hot compatibility
bull You still need to put effort into data migration
Workflow
Workflow
bull Make sure pull requests are addressing hot compatibility
Workflow
bull Make sure pull requests are addressing hot compatibility
bull Work after code is merged
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Production
Production
bull Run multiple server processes
Production
bull Run multiple server processes
bull Cycle them individually or in batches
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
bull Cycle them gracefully
Cycling techniques
bull Above the stack
bull Within the stack
bull Platform
Above the stack
bull Use your favorite web server
bull Setup a load balancer above them route around servers that are cycling
Above the stack
bull PROTIP default setup wonrsquot work
bull Configure load balancer to poll for your application health
class HealthController lt ApplicationController def index if Fileexists(Railsrootjoin(MAINTENANCE)) render text =gt Forget me status =gt 503 else render text =gt Im alive end endend
Cycling + Load balancer
bounce serversrm MAINTENANCE
remove server from rotation
touch MAINTENANCE
new servers come online
add server to rotation
Server Load balancer
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy where to check for the health
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
do not let your web server queue requests
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy to run the health check specified above
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
every 2s
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
stop routing to it once the health check failed once
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
and bring it back once it worked once
ELB
$ elb-configure-healthcheck MyLoadBalancer --target HTTP5000health --interval 2 --unhealthy-threshold 1 --healthy-threshold 1
Above the stack
Above the stackbull Minimal impact
Above the stackbull Minimal impact
bull Extended availability comes for free
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
bull Another component you have to track and maintain
Within the stack
Within the stack
Unicorn
bull Replaces your web server
bull Bounces gracefully
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Within the stack
Within the stackbull Easy to setup and deploy
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
bull Doesnrsquot address availability issues
Platform
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 1s bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 5s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 1s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ heroku labsenable preboot
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 starting for 5s bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 up for 1s bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 5s bundle exec thin start -p $PORT
BETA
Heroku
bull Migrations
Heroku
bull Migrations
$ git push pre-production master$ heroku run rake dbmigrate --remote pre-production$ git push production master
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Future
Future
bull Asset pipeline
Future
bull Asset pipeline
bull Mysql tooling
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
bull Can databases
Future
Future
bull Hot compatibility beyond zero downtime deploys
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time if you avoid default values and constraints
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
can be done without any locking
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time
Postgres
bull Adding fields with defaults
Postgres
bull Adding fields with defaults
before_save assign_defaults
def assign_defaults selfadmin ||= falseend
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
CREATE INDEX CONCURRENTLY users_email_index ON users(email)
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
bull Outside a transaction
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
bull Outside a transaction
class AddIndexToUsers lt ActiveRecordMigration def selfup return if indexes(users)map(ampname)include(users_email_index) add_index users email name =gt users_email_index end
def selfdown remove_index users name =gt users_email_index endend
What about renaming
v1 v2
reads and writes to `foo`
reads and writes to `bar`
What about renaming
v1 v2
reads and writes to `foo`
reads and writes to `bar`
v1rsquo
add column `bar`+
reads from `foo`writes to both`foo` and `bar`
v1rsquorsquo
populate `bar`where needed
What about renaming
v2
reads and writes to `bar`
v2rsquo
ignore `foo`
v3
drop column `foo`
NoSQL
NoSQL
bull Schema migrations O(n) locks are just part of hot compatibility
NoSQL
bull Schema migrations O(n) locks are just part of hot compatibility
bull You still need to put effort into data migration
Workflow
Workflow
bull Make sure pull requests are addressing hot compatibility
Workflow
bull Make sure pull requests are addressing hot compatibility
bull Work after code is merged
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Production
Production
bull Run multiple server processes
Production
bull Run multiple server processes
bull Cycle them individually or in batches
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
bull Cycle them gracefully
Cycling techniques
bull Above the stack
bull Within the stack
bull Platform
Above the stack
bull Use your favorite web server
bull Setup a load balancer above them route around servers that are cycling
Above the stack
bull PROTIP default setup wonrsquot work
bull Configure load balancer to poll for your application health
class HealthController lt ApplicationController def index if Fileexists(Railsrootjoin(MAINTENANCE)) render text =gt Forget me status =gt 503 else render text =gt Im alive end endend
Cycling + Load balancer
bounce serversrm MAINTENANCE
remove server from rotation
touch MAINTENANCE
new servers come online
add server to rotation
Server Load balancer
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy where to check for the health
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
do not let your web server queue requests
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy to run the health check specified above
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
every 2s
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
stop routing to it once the health check failed once
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
and bring it back once it worked once
ELB
$ elb-configure-healthcheck MyLoadBalancer --target HTTP5000health --interval 2 --unhealthy-threshold 1 --healthy-threshold 1
Above the stack
Above the stackbull Minimal impact
Above the stackbull Minimal impact
bull Extended availability comes for free
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
bull Another component you have to track and maintain
Within the stack
Within the stack
Unicorn
bull Replaces your web server
bull Bounces gracefully
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Within the stack
Within the stackbull Easy to setup and deploy
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
bull Doesnrsquot address availability issues
Platform
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 1s bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 5s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 1s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ heroku labsenable preboot
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 starting for 5s bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 up for 1s bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 5s bundle exec thin start -p $PORT
BETA
Heroku
bull Migrations
Heroku
bull Migrations
$ git push pre-production master$ heroku run rake dbmigrate --remote pre-production$ git push production master
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Future
Future
bull Asset pipeline
Future
bull Asset pipeline
bull Mysql tooling
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
bull Can databases
Future
Future
bull Hot compatibility beyond zero downtime deploys
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
can be done without any locking
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time
Postgres
bull Adding fields with defaults
Postgres
bull Adding fields with defaults
before_save assign_defaults
def assign_defaults selfadmin ||= falseend
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
CREATE INDEX CONCURRENTLY users_email_index ON users(email)
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
bull Outside a transaction
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
bull Outside a transaction
class AddIndexToUsers lt ActiveRecordMigration def selfup return if indexes(users)map(ampname)include(users_email_index) add_index users email name =gt users_email_index end
def selfdown remove_index users name =gt users_email_index endend
What about renaming
v1 v2
reads and writes to `foo`
reads and writes to `bar`
What about renaming
v1 v2
reads and writes to `foo`
reads and writes to `bar`
v1rsquo
add column `bar`+
reads from `foo`writes to both`foo` and `bar`
v1rsquorsquo
populate `bar`where needed
What about renaming
v2
reads and writes to `bar`
v2rsquo
ignore `foo`
v3
drop column `foo`
NoSQL
NoSQL
bull Schema migrations O(n) locks are just part of hot compatibility
NoSQL
bull Schema migrations O(n) locks are just part of hot compatibility
bull You still need to put effort into data migration
Workflow
Workflow
bull Make sure pull requests are addressing hot compatibility
Workflow
bull Make sure pull requests are addressing hot compatibility
bull Work after code is merged
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Production
Production
bull Run multiple server processes
Production
bull Run multiple server processes
bull Cycle them individually or in batches
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
bull Cycle them gracefully
Cycling techniques
bull Above the stack
bull Within the stack
bull Platform
Above the stack
bull Use your favorite web server
bull Setup a load balancer above them route around servers that are cycling
Above the stack
bull PROTIP default setup wonrsquot work
bull Configure load balancer to poll for your application health
class HealthController lt ApplicationController def index if Fileexists(Railsrootjoin(MAINTENANCE)) render text =gt Forget me status =gt 503 else render text =gt Im alive end endend
Cycling + Load balancer
bounce serversrm MAINTENANCE
remove server from rotation
touch MAINTENANCE
new servers come online
add server to rotation
Server Load balancer
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy where to check for the health
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
do not let your web server queue requests
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy to run the health check specified above
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
every 2s
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
stop routing to it once the health check failed once
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
and bring it back once it worked once
ELB
$ elb-configure-healthcheck MyLoadBalancer --target HTTP5000health --interval 2 --unhealthy-threshold 1 --healthy-threshold 1
Above the stack
Above the stackbull Minimal impact
Above the stackbull Minimal impact
bull Extended availability comes for free
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
bull Another component you have to track and maintain
Within the stack
Within the stack
Unicorn
bull Replaces your web server
bull Bounces gracefully
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Within the stack
Within the stackbull Easy to setup and deploy
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
bull Doesnrsquot address availability issues
Platform
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 1s bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 5s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 1s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ heroku labsenable preboot
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 starting for 5s bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 up for 1s bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 5s bundle exec thin start -p $PORT
BETA
Heroku
bull Migrations
Heroku
bull Migrations
$ git push pre-production master$ heroku run rake dbmigrate --remote pre-production$ git push production master
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Future
Future
bull Asset pipeline
Future
bull Asset pipeline
bull Mysql tooling
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
bull Can databases
Future
Future
bull Hot compatibility beyond zero downtime deploys
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
can be done without any locking
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time
Postgres
bull Adding fields with defaults
Postgres
bull Adding fields with defaults
before_save assign_defaults
def assign_defaults selfadmin ||= falseend
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
CREATE INDEX CONCURRENTLY users_email_index ON users(email)
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
bull Outside a transaction
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
bull Outside a transaction
class AddIndexToUsers lt ActiveRecordMigration def selfup return if indexes(users)map(ampname)include(users_email_index) add_index users email name =gt users_email_index end
def selfdown remove_index users name =gt users_email_index endend
What about renaming
v1 v2
reads and writes to `foo`
reads and writes to `bar`
What about renaming
v1 v2
reads and writes to `foo`
reads and writes to `bar`
v1rsquo
add column `bar`+
reads from `foo`writes to both`foo` and `bar`
v1rsquorsquo
populate `bar`where needed
What about renaming
v2
reads and writes to `bar`
v2rsquo
ignore `foo`
v3
drop column `foo`
NoSQL
NoSQL
bull Schema migrations O(n) locks are just part of hot compatibility
NoSQL
bull Schema migrations O(n) locks are just part of hot compatibility
bull You still need to put effort into data migration
Workflow
Workflow
bull Make sure pull requests are addressing hot compatibility
Workflow
bull Make sure pull requests are addressing hot compatibility
bull Work after code is merged
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Production
Production
bull Run multiple server processes
Production
bull Run multiple server processes
bull Cycle them individually or in batches
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
bull Cycle them gracefully
Cycling techniques
bull Above the stack
bull Within the stack
bull Platform
Above the stack
bull Use your favorite web server
bull Setup a load balancer above them route around servers that are cycling
Above the stack
bull PROTIP default setup wonrsquot work
bull Configure load balancer to poll for your application health
class HealthController lt ApplicationController def index if Fileexists(Railsrootjoin(MAINTENANCE)) render text =gt Forget me status =gt 503 else render text =gt Im alive end endend
Cycling + Load balancer
bounce serversrm MAINTENANCE
remove server from rotation
touch MAINTENANCE
new servers come online
add server to rotation
Server Load balancer
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy where to check for the health
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
do not let your web server queue requests
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy to run the health check specified above
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
every 2s
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
stop routing to it once the health check failed once
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
and bring it back once it worked once
ELB
$ elb-configure-healthcheck MyLoadBalancer --target HTTP5000health --interval 2 --unhealthy-threshold 1 --healthy-threshold 1
Above the stack
Above the stackbull Minimal impact
Above the stackbull Minimal impact
bull Extended availability comes for free
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
bull Another component you have to track and maintain
Within the stack
Within the stack
Unicorn
bull Replaces your web server
bull Bounces gracefully
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Within the stack
Within the stackbull Easy to setup and deploy
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
bull Doesnrsquot address availability issues
Platform
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 1s bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 5s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 1s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ heroku labsenable preboot
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 starting for 5s bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 up for 1s bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 5s bundle exec thin start -p $PORT
BETA
Heroku
bull Migrations
Heroku
bull Migrations
$ git push pre-production master$ heroku run rake dbmigrate --remote pre-production$ git push production master
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Future
Future
bull Asset pipeline
Future
bull Asset pipeline
bull Mysql tooling
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
bull Can databases
Future
Future
bull Hot compatibility beyond zero downtime deploys
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
Postgres
bull O(n) locks on
bull Adding updating or removing columns
bull Adding or removing indexes
constant time
Postgres
bull Adding fields with defaults
Postgres
bull Adding fields with defaults
before_save assign_defaults
def assign_defaults selfadmin ||= falseend
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
CREATE INDEX CONCURRENTLY users_email_index ON users(email)
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
bull Outside a transaction
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
bull Outside a transaction
class AddIndexToUsers lt ActiveRecordMigration def selfup return if indexes(users)map(ampname)include(users_email_index) add_index users email name =gt users_email_index end
def selfdown remove_index users name =gt users_email_index endend
What about renaming
v1 v2
reads and writes to `foo`
reads and writes to `bar`
What about renaming
v1 v2
reads and writes to `foo`
reads and writes to `bar`
v1rsquo
add column `bar`+
reads from `foo`writes to both`foo` and `bar`
v1rsquorsquo
populate `bar`where needed
What about renaming
v2
reads and writes to `bar`
v2rsquo
ignore `foo`
v3
drop column `foo`
NoSQL
NoSQL
bull Schema migrations O(n) locks are just part of hot compatibility
NoSQL
bull Schema migrations O(n) locks are just part of hot compatibility
bull You still need to put effort into data migration
Workflow
Workflow
bull Make sure pull requests are addressing hot compatibility
Workflow
bull Make sure pull requests are addressing hot compatibility
bull Work after code is merged
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Production
Production
bull Run multiple server processes
Production
bull Run multiple server processes
bull Cycle them individually or in batches
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
bull Cycle them gracefully
Cycling techniques
bull Above the stack
bull Within the stack
bull Platform
Above the stack
bull Use your favorite web server
bull Setup a load balancer above them route around servers that are cycling
Above the stack
bull PROTIP default setup wonrsquot work
bull Configure load balancer to poll for your application health
class HealthController lt ApplicationController def index if Fileexists(Railsrootjoin(MAINTENANCE)) render text =gt Forget me status =gt 503 else render text =gt Im alive end endend
Cycling + Load balancer
bounce serversrm MAINTENANCE
remove server from rotation
touch MAINTENANCE
new servers come online
add server to rotation
Server Load balancer
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy where to check for the health
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
do not let your web server queue requests
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy to run the health check specified above
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
every 2s
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
stop routing to it once the health check failed once
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
and bring it back once it worked once
ELB
$ elb-configure-healthcheck MyLoadBalancer --target HTTP5000health --interval 2 --unhealthy-threshold 1 --healthy-threshold 1
Above the stack
Above the stackbull Minimal impact
Above the stackbull Minimal impact
bull Extended availability comes for free
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
bull Another component you have to track and maintain
Within the stack
Within the stack
Unicorn
bull Replaces your web server
bull Bounces gracefully
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Within the stack
Within the stackbull Easy to setup and deploy
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
bull Doesnrsquot address availability issues
Platform
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 1s bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 5s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 1s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ heroku labsenable preboot
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 starting for 5s bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 up for 1s bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 5s bundle exec thin start -p $PORT
BETA
Heroku
bull Migrations
Heroku
bull Migrations
$ git push pre-production master$ heroku run rake dbmigrate --remote pre-production$ git push production master
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Future
Future
bull Asset pipeline
Future
bull Asset pipeline
bull Mysql tooling
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
bull Can databases
Future
Future
bull Hot compatibility beyond zero downtime deploys
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
Postgres
bull Adding fields with defaults
Postgres
bull Adding fields with defaults
before_save assign_defaults
def assign_defaults selfadmin ||= falseend
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
CREATE INDEX CONCURRENTLY users_email_index ON users(email)
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
bull Outside a transaction
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
bull Outside a transaction
class AddIndexToUsers lt ActiveRecordMigration def selfup return if indexes(users)map(ampname)include(users_email_index) add_index users email name =gt users_email_index end
def selfdown remove_index users name =gt users_email_index endend
What about renaming
v1 v2
reads and writes to `foo`
reads and writes to `bar`
What about renaming
v1 v2
reads and writes to `foo`
reads and writes to `bar`
v1rsquo
add column `bar`+
reads from `foo`writes to both`foo` and `bar`
v1rsquorsquo
populate `bar`where needed
What about renaming
v2
reads and writes to `bar`
v2rsquo
ignore `foo`
v3
drop column `foo`
NoSQL
NoSQL
bull Schema migrations O(n) locks are just part of hot compatibility
NoSQL
bull Schema migrations O(n) locks are just part of hot compatibility
bull You still need to put effort into data migration
Workflow
Workflow
bull Make sure pull requests are addressing hot compatibility
Workflow
bull Make sure pull requests are addressing hot compatibility
bull Work after code is merged
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Production
Production
bull Run multiple server processes
Production
bull Run multiple server processes
bull Cycle them individually or in batches
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
bull Cycle them gracefully
Cycling techniques
bull Above the stack
bull Within the stack
bull Platform
Above the stack
bull Use your favorite web server
bull Setup a load balancer above them route around servers that are cycling
Above the stack
bull PROTIP default setup wonrsquot work
bull Configure load balancer to poll for your application health
class HealthController lt ApplicationController def index if Fileexists(Railsrootjoin(MAINTENANCE)) render text =gt Forget me status =gt 503 else render text =gt Im alive end endend
Cycling + Load balancer
bounce serversrm MAINTENANCE
remove server from rotation
touch MAINTENANCE
new servers come online
add server to rotation
Server Load balancer
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy where to check for the health
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
do not let your web server queue requests
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy to run the health check specified above
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
every 2s
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
stop routing to it once the health check failed once
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
and bring it back once it worked once
ELB
$ elb-configure-healthcheck MyLoadBalancer --target HTTP5000health --interval 2 --unhealthy-threshold 1 --healthy-threshold 1
Above the stack
Above the stackbull Minimal impact
Above the stackbull Minimal impact
bull Extended availability comes for free
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
bull Another component you have to track and maintain
Within the stack
Within the stack
Unicorn
bull Replaces your web server
bull Bounces gracefully
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Within the stack
Within the stackbull Easy to setup and deploy
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
bull Doesnrsquot address availability issues
Platform
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 1s bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 5s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 1s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ heroku labsenable preboot
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 starting for 5s bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 up for 1s bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 5s bundle exec thin start -p $PORT
BETA
Heroku
bull Migrations
Heroku
bull Migrations
$ git push pre-production master$ heroku run rake dbmigrate --remote pre-production$ git push production master
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Future
Future
bull Asset pipeline
Future
bull Asset pipeline
bull Mysql tooling
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
bull Can databases
Future
Future
bull Hot compatibility beyond zero downtime deploys
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
Postgres
bull Adding fields with defaults
before_save assign_defaults
def assign_defaults selfadmin ||= falseend
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
CREATE INDEX CONCURRENTLY users_email_index ON users(email)
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
bull Outside a transaction
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
bull Outside a transaction
class AddIndexToUsers lt ActiveRecordMigration def selfup return if indexes(users)map(ampname)include(users_email_index) add_index users email name =gt users_email_index end
def selfdown remove_index users name =gt users_email_index endend
What about renaming
v1 v2
reads and writes to `foo`
reads and writes to `bar`
What about renaming
v1 v2
reads and writes to `foo`
reads and writes to `bar`
v1rsquo
add column `bar`+
reads from `foo`writes to both`foo` and `bar`
v1rsquorsquo
populate `bar`where needed
What about renaming
v2
reads and writes to `bar`
v2rsquo
ignore `foo`
v3
drop column `foo`
NoSQL
NoSQL
bull Schema migrations O(n) locks are just part of hot compatibility
NoSQL
bull Schema migrations O(n) locks are just part of hot compatibility
bull You still need to put effort into data migration
Workflow
Workflow
bull Make sure pull requests are addressing hot compatibility
Workflow
bull Make sure pull requests are addressing hot compatibility
bull Work after code is merged
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Production
Production
bull Run multiple server processes
Production
bull Run multiple server processes
bull Cycle them individually or in batches
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
bull Cycle them gracefully
Cycling techniques
bull Above the stack
bull Within the stack
bull Platform
Above the stack
bull Use your favorite web server
bull Setup a load balancer above them route around servers that are cycling
Above the stack
bull PROTIP default setup wonrsquot work
bull Configure load balancer to poll for your application health
class HealthController lt ApplicationController def index if Fileexists(Railsrootjoin(MAINTENANCE)) render text =gt Forget me status =gt 503 else render text =gt Im alive end endend
Cycling + Load balancer
bounce serversrm MAINTENANCE
remove server from rotation
touch MAINTENANCE
new servers come online
add server to rotation
Server Load balancer
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy where to check for the health
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
do not let your web server queue requests
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy to run the health check specified above
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
every 2s
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
stop routing to it once the health check failed once
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
and bring it back once it worked once
ELB
$ elb-configure-healthcheck MyLoadBalancer --target HTTP5000health --interval 2 --unhealthy-threshold 1 --healthy-threshold 1
Above the stack
Above the stackbull Minimal impact
Above the stackbull Minimal impact
bull Extended availability comes for free
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
bull Another component you have to track and maintain
Within the stack
Within the stack
Unicorn
bull Replaces your web server
bull Bounces gracefully
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Within the stack
Within the stackbull Easy to setup and deploy
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
bull Doesnrsquot address availability issues
Platform
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 1s bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 5s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 1s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ heroku labsenable preboot
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 starting for 5s bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 up for 1s bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 5s bundle exec thin start -p $PORT
BETA
Heroku
bull Migrations
Heroku
bull Migrations
$ git push pre-production master$ heroku run rake dbmigrate --remote pre-production$ git push production master
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Future
Future
bull Asset pipeline
Future
bull Asset pipeline
bull Mysql tooling
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
bull Can databases
Future
Future
bull Hot compatibility beyond zero downtime deploys
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
CREATE INDEX CONCURRENTLY users_email_index ON users(email)
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
bull Outside a transaction
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
bull Outside a transaction
class AddIndexToUsers lt ActiveRecordMigration def selfup return if indexes(users)map(ampname)include(users_email_index) add_index users email name =gt users_email_index end
def selfdown remove_index users name =gt users_email_index endend
What about renaming
v1 v2
reads and writes to `foo`
reads and writes to `bar`
What about renaming
v1 v2
reads and writes to `foo`
reads and writes to `bar`
v1rsquo
add column `bar`+
reads from `foo`writes to both`foo` and `bar`
v1rsquorsquo
populate `bar`where needed
What about renaming
v2
reads and writes to `bar`
v2rsquo
ignore `foo`
v3
drop column `foo`
NoSQL
NoSQL
bull Schema migrations O(n) locks are just part of hot compatibility
NoSQL
bull Schema migrations O(n) locks are just part of hot compatibility
bull You still need to put effort into data migration
Workflow
Workflow
bull Make sure pull requests are addressing hot compatibility
Workflow
bull Make sure pull requests are addressing hot compatibility
bull Work after code is merged
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Production
Production
bull Run multiple server processes
Production
bull Run multiple server processes
bull Cycle them individually or in batches
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
bull Cycle them gracefully
Cycling techniques
bull Above the stack
bull Within the stack
bull Platform
Above the stack
bull Use your favorite web server
bull Setup a load balancer above them route around servers that are cycling
Above the stack
bull PROTIP default setup wonrsquot work
bull Configure load balancer to poll for your application health
class HealthController lt ApplicationController def index if Fileexists(Railsrootjoin(MAINTENANCE)) render text =gt Forget me status =gt 503 else render text =gt Im alive end endend
Cycling + Load balancer
bounce serversrm MAINTENANCE
remove server from rotation
touch MAINTENANCE
new servers come online
add server to rotation
Server Load balancer
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy where to check for the health
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
do not let your web server queue requests
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy to run the health check specified above
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
every 2s
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
stop routing to it once the health check failed once
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
and bring it back once it worked once
ELB
$ elb-configure-healthcheck MyLoadBalancer --target HTTP5000health --interval 2 --unhealthy-threshold 1 --healthy-threshold 1
Above the stack
Above the stackbull Minimal impact
Above the stackbull Minimal impact
bull Extended availability comes for free
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
bull Another component you have to track and maintain
Within the stack
Within the stack
Unicorn
bull Replaces your web server
bull Bounces gracefully
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Within the stack
Within the stackbull Easy to setup and deploy
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
bull Doesnrsquot address availability issues
Platform
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 1s bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 5s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 1s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ heroku labsenable preboot
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 starting for 5s bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 up for 1s bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 5s bundle exec thin start -p $PORT
BETA
Heroku
bull Migrations
Heroku
bull Migrations
$ git push pre-production master$ heroku run rake dbmigrate --remote pre-production$ git push production master
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Future
Future
bull Asset pipeline
Future
bull Asset pipeline
bull Mysql tooling
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
bull Can databases
Future
Future
bull Hot compatibility beyond zero downtime deploys
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
CREATE INDEX CONCURRENTLY users_email_index ON users(email)
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
bull Outside a transaction
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
bull Outside a transaction
class AddIndexToUsers lt ActiveRecordMigration def selfup return if indexes(users)map(ampname)include(users_email_index) add_index users email name =gt users_email_index end
def selfdown remove_index users name =gt users_email_index endend
What about renaming
v1 v2
reads and writes to `foo`
reads and writes to `bar`
What about renaming
v1 v2
reads and writes to `foo`
reads and writes to `bar`
v1rsquo
add column `bar`+
reads from `foo`writes to both`foo` and `bar`
v1rsquorsquo
populate `bar`where needed
What about renaming
v2
reads and writes to `bar`
v2rsquo
ignore `foo`
v3
drop column `foo`
NoSQL
NoSQL
bull Schema migrations O(n) locks are just part of hot compatibility
NoSQL
bull Schema migrations O(n) locks are just part of hot compatibility
bull You still need to put effort into data migration
Workflow
Workflow
bull Make sure pull requests are addressing hot compatibility
Workflow
bull Make sure pull requests are addressing hot compatibility
bull Work after code is merged
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Production
Production
bull Run multiple server processes
Production
bull Run multiple server processes
bull Cycle them individually or in batches
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
bull Cycle them gracefully
Cycling techniques
bull Above the stack
bull Within the stack
bull Platform
Above the stack
bull Use your favorite web server
bull Setup a load balancer above them route around servers that are cycling
Above the stack
bull PROTIP default setup wonrsquot work
bull Configure load balancer to poll for your application health
class HealthController lt ApplicationController def index if Fileexists(Railsrootjoin(MAINTENANCE)) render text =gt Forget me status =gt 503 else render text =gt Im alive end endend
Cycling + Load balancer
bounce serversrm MAINTENANCE
remove server from rotation
touch MAINTENANCE
new servers come online
add server to rotation
Server Load balancer
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy where to check for the health
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
do not let your web server queue requests
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy to run the health check specified above
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
every 2s
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
stop routing to it once the health check failed once
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
and bring it back once it worked once
ELB
$ elb-configure-healthcheck MyLoadBalancer --target HTTP5000health --interval 2 --unhealthy-threshold 1 --healthy-threshold 1
Above the stack
Above the stackbull Minimal impact
Above the stackbull Minimal impact
bull Extended availability comes for free
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
bull Another component you have to track and maintain
Within the stack
Within the stack
Unicorn
bull Replaces your web server
bull Bounces gracefully
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Within the stack
Within the stackbull Easy to setup and deploy
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
bull Doesnrsquot address availability issues
Platform
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 1s bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 5s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 1s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ heroku labsenable preboot
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 starting for 5s bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 up for 1s bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 5s bundle exec thin start -p $PORT
BETA
Heroku
bull Migrations
Heroku
bull Migrations
$ git push pre-production master$ heroku run rake dbmigrate --remote pre-production$ git push production master
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Future
Future
bull Asset pipeline
Future
bull Asset pipeline
bull Mysql tooling
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
bull Can databases
Future
Future
bull Hot compatibility beyond zero downtime deploys
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
bull Outside a transaction
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
bull Outside a transaction
class AddIndexToUsers lt ActiveRecordMigration def selfup return if indexes(users)map(ampname)include(users_email_index) add_index users email name =gt users_email_index end
def selfdown remove_index users name =gt users_email_index endend
What about renaming
v1 v2
reads and writes to `foo`
reads and writes to `bar`
What about renaming
v1 v2
reads and writes to `foo`
reads and writes to `bar`
v1rsquo
add column `bar`+
reads from `foo`writes to both`foo` and `bar`
v1rsquorsquo
populate `bar`where needed
What about renaming
v2
reads and writes to `bar`
v2rsquo
ignore `foo`
v3
drop column `foo`
NoSQL
NoSQL
bull Schema migrations O(n) locks are just part of hot compatibility
NoSQL
bull Schema migrations O(n) locks are just part of hot compatibility
bull You still need to put effort into data migration
Workflow
Workflow
bull Make sure pull requests are addressing hot compatibility
Workflow
bull Make sure pull requests are addressing hot compatibility
bull Work after code is merged
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Production
Production
bull Run multiple server processes
Production
bull Run multiple server processes
bull Cycle them individually or in batches
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
bull Cycle them gracefully
Cycling techniques
bull Above the stack
bull Within the stack
bull Platform
Above the stack
bull Use your favorite web server
bull Setup a load balancer above them route around servers that are cycling
Above the stack
bull PROTIP default setup wonrsquot work
bull Configure load balancer to poll for your application health
class HealthController lt ApplicationController def index if Fileexists(Railsrootjoin(MAINTENANCE)) render text =gt Forget me status =gt 503 else render text =gt Im alive end endend
Cycling + Load balancer
bounce serversrm MAINTENANCE
remove server from rotation
touch MAINTENANCE
new servers come online
add server to rotation
Server Load balancer
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy where to check for the health
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
do not let your web server queue requests
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy to run the health check specified above
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
every 2s
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
stop routing to it once the health check failed once
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
and bring it back once it worked once
ELB
$ elb-configure-healthcheck MyLoadBalancer --target HTTP5000health --interval 2 --unhealthy-threshold 1 --healthy-threshold 1
Above the stack
Above the stackbull Minimal impact
Above the stackbull Minimal impact
bull Extended availability comes for free
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
bull Another component you have to track and maintain
Within the stack
Within the stack
Unicorn
bull Replaces your web server
bull Bounces gracefully
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Within the stack
Within the stackbull Easy to setup and deploy
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
bull Doesnrsquot address availability issues
Platform
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 1s bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 5s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 1s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ heroku labsenable preboot
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 starting for 5s bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 up for 1s bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 5s bundle exec thin start -p $PORT
BETA
Heroku
bull Migrations
Heroku
bull Migrations
$ git push pre-production master$ heroku run rake dbmigrate --remote pre-production$ git push production master
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Future
Future
bull Asset pipeline
Future
bull Asset pipeline
bull Mysql tooling
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
bull Can databases
Future
Future
bull Hot compatibility beyond zero downtime deploys
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
Postgres
bull Adding fields with defaults
bull Create indexes concurrently
bull Outside a transaction
class AddIndexToUsers lt ActiveRecordMigration def selfup return if indexes(users)map(ampname)include(users_email_index) add_index users email name =gt users_email_index end
def selfdown remove_index users name =gt users_email_index endend
What about renaming
v1 v2
reads and writes to `foo`
reads and writes to `bar`
What about renaming
v1 v2
reads and writes to `foo`
reads and writes to `bar`
v1rsquo
add column `bar`+
reads from `foo`writes to both`foo` and `bar`
v1rsquorsquo
populate `bar`where needed
What about renaming
v2
reads and writes to `bar`
v2rsquo
ignore `foo`
v3
drop column `foo`
NoSQL
NoSQL
bull Schema migrations O(n) locks are just part of hot compatibility
NoSQL
bull Schema migrations O(n) locks are just part of hot compatibility
bull You still need to put effort into data migration
Workflow
Workflow
bull Make sure pull requests are addressing hot compatibility
Workflow
bull Make sure pull requests are addressing hot compatibility
bull Work after code is merged
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Production
Production
bull Run multiple server processes
Production
bull Run multiple server processes
bull Cycle them individually or in batches
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
bull Cycle them gracefully
Cycling techniques
bull Above the stack
bull Within the stack
bull Platform
Above the stack
bull Use your favorite web server
bull Setup a load balancer above them route around servers that are cycling
Above the stack
bull PROTIP default setup wonrsquot work
bull Configure load balancer to poll for your application health
class HealthController lt ApplicationController def index if Fileexists(Railsrootjoin(MAINTENANCE)) render text =gt Forget me status =gt 503 else render text =gt Im alive end endend
Cycling + Load balancer
bounce serversrm MAINTENANCE
remove server from rotation
touch MAINTENANCE
new servers come online
add server to rotation
Server Load balancer
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy where to check for the health
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
do not let your web server queue requests
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy to run the health check specified above
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
every 2s
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
stop routing to it once the health check failed once
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
and bring it back once it worked once
ELB
$ elb-configure-healthcheck MyLoadBalancer --target HTTP5000health --interval 2 --unhealthy-threshold 1 --healthy-threshold 1
Above the stack
Above the stackbull Minimal impact
Above the stackbull Minimal impact
bull Extended availability comes for free
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
bull Another component you have to track and maintain
Within the stack
Within the stack
Unicorn
bull Replaces your web server
bull Bounces gracefully
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Within the stack
Within the stackbull Easy to setup and deploy
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
bull Doesnrsquot address availability issues
Platform
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 1s bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 5s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 1s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ heroku labsenable preboot
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 starting for 5s bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 up for 1s bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 5s bundle exec thin start -p $PORT
BETA
Heroku
bull Migrations
Heroku
bull Migrations
$ git push pre-production master$ heroku run rake dbmigrate --remote pre-production$ git push production master
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Future
Future
bull Asset pipeline
Future
bull Asset pipeline
bull Mysql tooling
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
bull Can databases
Future
Future
bull Hot compatibility beyond zero downtime deploys
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
What about renaming
v1 v2
reads and writes to `foo`
reads and writes to `bar`
What about renaming
v1 v2
reads and writes to `foo`
reads and writes to `bar`
v1rsquo
add column `bar`+
reads from `foo`writes to both`foo` and `bar`
v1rsquorsquo
populate `bar`where needed
What about renaming
v2
reads and writes to `bar`
v2rsquo
ignore `foo`
v3
drop column `foo`
NoSQL
NoSQL
bull Schema migrations O(n) locks are just part of hot compatibility
NoSQL
bull Schema migrations O(n) locks are just part of hot compatibility
bull You still need to put effort into data migration
Workflow
Workflow
bull Make sure pull requests are addressing hot compatibility
Workflow
bull Make sure pull requests are addressing hot compatibility
bull Work after code is merged
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Production
Production
bull Run multiple server processes
Production
bull Run multiple server processes
bull Cycle them individually or in batches
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
bull Cycle them gracefully
Cycling techniques
bull Above the stack
bull Within the stack
bull Platform
Above the stack
bull Use your favorite web server
bull Setup a load balancer above them route around servers that are cycling
Above the stack
bull PROTIP default setup wonrsquot work
bull Configure load balancer to poll for your application health
class HealthController lt ApplicationController def index if Fileexists(Railsrootjoin(MAINTENANCE)) render text =gt Forget me status =gt 503 else render text =gt Im alive end endend
Cycling + Load balancer
bounce serversrm MAINTENANCE
remove server from rotation
touch MAINTENANCE
new servers come online
add server to rotation
Server Load balancer
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy where to check for the health
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
do not let your web server queue requests
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy to run the health check specified above
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
every 2s
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
stop routing to it once the health check failed once
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
and bring it back once it worked once
ELB
$ elb-configure-healthcheck MyLoadBalancer --target HTTP5000health --interval 2 --unhealthy-threshold 1 --healthy-threshold 1
Above the stack
Above the stackbull Minimal impact
Above the stackbull Minimal impact
bull Extended availability comes for free
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
bull Another component you have to track and maintain
Within the stack
Within the stack
Unicorn
bull Replaces your web server
bull Bounces gracefully
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Within the stack
Within the stackbull Easy to setup and deploy
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
bull Doesnrsquot address availability issues
Platform
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 1s bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 5s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 1s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ heroku labsenable preboot
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 starting for 5s bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 up for 1s bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 5s bundle exec thin start -p $PORT
BETA
Heroku
bull Migrations
Heroku
bull Migrations
$ git push pre-production master$ heroku run rake dbmigrate --remote pre-production$ git push production master
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Future
Future
bull Asset pipeline
Future
bull Asset pipeline
bull Mysql tooling
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
bull Can databases
Future
Future
bull Hot compatibility beyond zero downtime deploys
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
What about renaming
v1 v2
reads and writes to `foo`
reads and writes to `bar`
v1rsquo
add column `bar`+
reads from `foo`writes to both`foo` and `bar`
v1rsquorsquo
populate `bar`where needed
What about renaming
v2
reads and writes to `bar`
v2rsquo
ignore `foo`
v3
drop column `foo`
NoSQL
NoSQL
bull Schema migrations O(n) locks are just part of hot compatibility
NoSQL
bull Schema migrations O(n) locks are just part of hot compatibility
bull You still need to put effort into data migration
Workflow
Workflow
bull Make sure pull requests are addressing hot compatibility
Workflow
bull Make sure pull requests are addressing hot compatibility
bull Work after code is merged
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Production
Production
bull Run multiple server processes
Production
bull Run multiple server processes
bull Cycle them individually or in batches
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
bull Cycle them gracefully
Cycling techniques
bull Above the stack
bull Within the stack
bull Platform
Above the stack
bull Use your favorite web server
bull Setup a load balancer above them route around servers that are cycling
Above the stack
bull PROTIP default setup wonrsquot work
bull Configure load balancer to poll for your application health
class HealthController lt ApplicationController def index if Fileexists(Railsrootjoin(MAINTENANCE)) render text =gt Forget me status =gt 503 else render text =gt Im alive end endend
Cycling + Load balancer
bounce serversrm MAINTENANCE
remove server from rotation
touch MAINTENANCE
new servers come online
add server to rotation
Server Load balancer
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy where to check for the health
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
do not let your web server queue requests
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy to run the health check specified above
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
every 2s
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
stop routing to it once the health check failed once
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
and bring it back once it worked once
ELB
$ elb-configure-healthcheck MyLoadBalancer --target HTTP5000health --interval 2 --unhealthy-threshold 1 --healthy-threshold 1
Above the stack
Above the stackbull Minimal impact
Above the stackbull Minimal impact
bull Extended availability comes for free
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
bull Another component you have to track and maintain
Within the stack
Within the stack
Unicorn
bull Replaces your web server
bull Bounces gracefully
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Within the stack
Within the stackbull Easy to setup and deploy
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
bull Doesnrsquot address availability issues
Platform
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 1s bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 5s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 1s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ heroku labsenable preboot
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 starting for 5s bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 up for 1s bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 5s bundle exec thin start -p $PORT
BETA
Heroku
bull Migrations
Heroku
bull Migrations
$ git push pre-production master$ heroku run rake dbmigrate --remote pre-production$ git push production master
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Future
Future
bull Asset pipeline
Future
bull Asset pipeline
bull Mysql tooling
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
bull Can databases
Future
Future
bull Hot compatibility beyond zero downtime deploys
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
What about renaming
v2
reads and writes to `bar`
v2rsquo
ignore `foo`
v3
drop column `foo`
NoSQL
NoSQL
bull Schema migrations O(n) locks are just part of hot compatibility
NoSQL
bull Schema migrations O(n) locks are just part of hot compatibility
bull You still need to put effort into data migration
Workflow
Workflow
bull Make sure pull requests are addressing hot compatibility
Workflow
bull Make sure pull requests are addressing hot compatibility
bull Work after code is merged
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Production
Production
bull Run multiple server processes
Production
bull Run multiple server processes
bull Cycle them individually or in batches
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
bull Cycle them gracefully
Cycling techniques
bull Above the stack
bull Within the stack
bull Platform
Above the stack
bull Use your favorite web server
bull Setup a load balancer above them route around servers that are cycling
Above the stack
bull PROTIP default setup wonrsquot work
bull Configure load balancer to poll for your application health
class HealthController lt ApplicationController def index if Fileexists(Railsrootjoin(MAINTENANCE)) render text =gt Forget me status =gt 503 else render text =gt Im alive end endend
Cycling + Load balancer
bounce serversrm MAINTENANCE
remove server from rotation
touch MAINTENANCE
new servers come online
add server to rotation
Server Load balancer
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy where to check for the health
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
do not let your web server queue requests
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy to run the health check specified above
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
every 2s
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
stop routing to it once the health check failed once
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
and bring it back once it worked once
ELB
$ elb-configure-healthcheck MyLoadBalancer --target HTTP5000health --interval 2 --unhealthy-threshold 1 --healthy-threshold 1
Above the stack
Above the stackbull Minimal impact
Above the stackbull Minimal impact
bull Extended availability comes for free
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
bull Another component you have to track and maintain
Within the stack
Within the stack
Unicorn
bull Replaces your web server
bull Bounces gracefully
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Within the stack
Within the stackbull Easy to setup and deploy
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
bull Doesnrsquot address availability issues
Platform
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 1s bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 5s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 1s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ heroku labsenable preboot
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 starting for 5s bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 up for 1s bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 5s bundle exec thin start -p $PORT
BETA
Heroku
bull Migrations
Heroku
bull Migrations
$ git push pre-production master$ heroku run rake dbmigrate --remote pre-production$ git push production master
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Future
Future
bull Asset pipeline
Future
bull Asset pipeline
bull Mysql tooling
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
bull Can databases
Future
Future
bull Hot compatibility beyond zero downtime deploys
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
NoSQL
NoSQL
bull Schema migrations O(n) locks are just part of hot compatibility
NoSQL
bull Schema migrations O(n) locks are just part of hot compatibility
bull You still need to put effort into data migration
Workflow
Workflow
bull Make sure pull requests are addressing hot compatibility
Workflow
bull Make sure pull requests are addressing hot compatibility
bull Work after code is merged
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Production
Production
bull Run multiple server processes
Production
bull Run multiple server processes
bull Cycle them individually or in batches
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
bull Cycle them gracefully
Cycling techniques
bull Above the stack
bull Within the stack
bull Platform
Above the stack
bull Use your favorite web server
bull Setup a load balancer above them route around servers that are cycling
Above the stack
bull PROTIP default setup wonrsquot work
bull Configure load balancer to poll for your application health
class HealthController lt ApplicationController def index if Fileexists(Railsrootjoin(MAINTENANCE)) render text =gt Forget me status =gt 503 else render text =gt Im alive end endend
Cycling + Load balancer
bounce serversrm MAINTENANCE
remove server from rotation
touch MAINTENANCE
new servers come online
add server to rotation
Server Load balancer
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy where to check for the health
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
do not let your web server queue requests
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy to run the health check specified above
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
every 2s
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
stop routing to it once the health check failed once
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
and bring it back once it worked once
ELB
$ elb-configure-healthcheck MyLoadBalancer --target HTTP5000health --interval 2 --unhealthy-threshold 1 --healthy-threshold 1
Above the stack
Above the stackbull Minimal impact
Above the stackbull Minimal impact
bull Extended availability comes for free
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
bull Another component you have to track and maintain
Within the stack
Within the stack
Unicorn
bull Replaces your web server
bull Bounces gracefully
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Within the stack
Within the stackbull Easy to setup and deploy
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
bull Doesnrsquot address availability issues
Platform
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 1s bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 5s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 1s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ heroku labsenable preboot
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 starting for 5s bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 up for 1s bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 5s bundle exec thin start -p $PORT
BETA
Heroku
bull Migrations
Heroku
bull Migrations
$ git push pre-production master$ heroku run rake dbmigrate --remote pre-production$ git push production master
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Future
Future
bull Asset pipeline
Future
bull Asset pipeline
bull Mysql tooling
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
bull Can databases
Future
Future
bull Hot compatibility beyond zero downtime deploys
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
NoSQL
bull Schema migrations O(n) locks are just part of hot compatibility
NoSQL
bull Schema migrations O(n) locks are just part of hot compatibility
bull You still need to put effort into data migration
Workflow
Workflow
bull Make sure pull requests are addressing hot compatibility
Workflow
bull Make sure pull requests are addressing hot compatibility
bull Work after code is merged
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Production
Production
bull Run multiple server processes
Production
bull Run multiple server processes
bull Cycle them individually or in batches
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
bull Cycle them gracefully
Cycling techniques
bull Above the stack
bull Within the stack
bull Platform
Above the stack
bull Use your favorite web server
bull Setup a load balancer above them route around servers that are cycling
Above the stack
bull PROTIP default setup wonrsquot work
bull Configure load balancer to poll for your application health
class HealthController lt ApplicationController def index if Fileexists(Railsrootjoin(MAINTENANCE)) render text =gt Forget me status =gt 503 else render text =gt Im alive end endend
Cycling + Load balancer
bounce serversrm MAINTENANCE
remove server from rotation
touch MAINTENANCE
new servers come online
add server to rotation
Server Load balancer
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy where to check for the health
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
do not let your web server queue requests
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy to run the health check specified above
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
every 2s
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
stop routing to it once the health check failed once
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
and bring it back once it worked once
ELB
$ elb-configure-healthcheck MyLoadBalancer --target HTTP5000health --interval 2 --unhealthy-threshold 1 --healthy-threshold 1
Above the stack
Above the stackbull Minimal impact
Above the stackbull Minimal impact
bull Extended availability comes for free
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
bull Another component you have to track and maintain
Within the stack
Within the stack
Unicorn
bull Replaces your web server
bull Bounces gracefully
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Within the stack
Within the stackbull Easy to setup and deploy
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
bull Doesnrsquot address availability issues
Platform
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 1s bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 5s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 1s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ heroku labsenable preboot
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 starting for 5s bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 up for 1s bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 5s bundle exec thin start -p $PORT
BETA
Heroku
bull Migrations
Heroku
bull Migrations
$ git push pre-production master$ heroku run rake dbmigrate --remote pre-production$ git push production master
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Future
Future
bull Asset pipeline
Future
bull Asset pipeline
bull Mysql tooling
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
bull Can databases
Future
Future
bull Hot compatibility beyond zero downtime deploys
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
NoSQL
bull Schema migrations O(n) locks are just part of hot compatibility
bull You still need to put effort into data migration
Workflow
Workflow
bull Make sure pull requests are addressing hot compatibility
Workflow
bull Make sure pull requests are addressing hot compatibility
bull Work after code is merged
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Production
Production
bull Run multiple server processes
Production
bull Run multiple server processes
bull Cycle them individually or in batches
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
bull Cycle them gracefully
Cycling techniques
bull Above the stack
bull Within the stack
bull Platform
Above the stack
bull Use your favorite web server
bull Setup a load balancer above them route around servers that are cycling
Above the stack
bull PROTIP default setup wonrsquot work
bull Configure load balancer to poll for your application health
class HealthController lt ApplicationController def index if Fileexists(Railsrootjoin(MAINTENANCE)) render text =gt Forget me status =gt 503 else render text =gt Im alive end endend
Cycling + Load balancer
bounce serversrm MAINTENANCE
remove server from rotation
touch MAINTENANCE
new servers come online
add server to rotation
Server Load balancer
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy where to check for the health
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
do not let your web server queue requests
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy to run the health check specified above
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
every 2s
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
stop routing to it once the health check failed once
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
and bring it back once it worked once
ELB
$ elb-configure-healthcheck MyLoadBalancer --target HTTP5000health --interval 2 --unhealthy-threshold 1 --healthy-threshold 1
Above the stack
Above the stackbull Minimal impact
Above the stackbull Minimal impact
bull Extended availability comes for free
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
bull Another component you have to track and maintain
Within the stack
Within the stack
Unicorn
bull Replaces your web server
bull Bounces gracefully
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Within the stack
Within the stackbull Easy to setup and deploy
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
bull Doesnrsquot address availability issues
Platform
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 1s bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 5s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 1s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ heroku labsenable preboot
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 starting for 5s bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 up for 1s bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 5s bundle exec thin start -p $PORT
BETA
Heroku
bull Migrations
Heroku
bull Migrations
$ git push pre-production master$ heroku run rake dbmigrate --remote pre-production$ git push production master
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Future
Future
bull Asset pipeline
Future
bull Asset pipeline
bull Mysql tooling
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
bull Can databases
Future
Future
bull Hot compatibility beyond zero downtime deploys
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
Workflow
Workflow
bull Make sure pull requests are addressing hot compatibility
Workflow
bull Make sure pull requests are addressing hot compatibility
bull Work after code is merged
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Production
Production
bull Run multiple server processes
Production
bull Run multiple server processes
bull Cycle them individually or in batches
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
bull Cycle them gracefully
Cycling techniques
bull Above the stack
bull Within the stack
bull Platform
Above the stack
bull Use your favorite web server
bull Setup a load balancer above them route around servers that are cycling
Above the stack
bull PROTIP default setup wonrsquot work
bull Configure load balancer to poll for your application health
class HealthController lt ApplicationController def index if Fileexists(Railsrootjoin(MAINTENANCE)) render text =gt Forget me status =gt 503 else render text =gt Im alive end endend
Cycling + Load balancer
bounce serversrm MAINTENANCE
remove server from rotation
touch MAINTENANCE
new servers come online
add server to rotation
Server Load balancer
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy where to check for the health
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
do not let your web server queue requests
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy to run the health check specified above
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
every 2s
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
stop routing to it once the health check failed once
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
and bring it back once it worked once
ELB
$ elb-configure-healthcheck MyLoadBalancer --target HTTP5000health --interval 2 --unhealthy-threshold 1 --healthy-threshold 1
Above the stack
Above the stackbull Minimal impact
Above the stackbull Minimal impact
bull Extended availability comes for free
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
bull Another component you have to track and maintain
Within the stack
Within the stack
Unicorn
bull Replaces your web server
bull Bounces gracefully
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Within the stack
Within the stackbull Easy to setup and deploy
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
bull Doesnrsquot address availability issues
Platform
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 1s bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 5s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 1s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ heroku labsenable preboot
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 starting for 5s bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 up for 1s bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 5s bundle exec thin start -p $PORT
BETA
Heroku
bull Migrations
Heroku
bull Migrations
$ git push pre-production master$ heroku run rake dbmigrate --remote pre-production$ git push production master
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Future
Future
bull Asset pipeline
Future
bull Asset pipeline
bull Mysql tooling
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
bull Can databases
Future
Future
bull Hot compatibility beyond zero downtime deploys
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
Workflow
bull Make sure pull requests are addressing hot compatibility
Workflow
bull Make sure pull requests are addressing hot compatibility
bull Work after code is merged
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Production
Production
bull Run multiple server processes
Production
bull Run multiple server processes
bull Cycle them individually or in batches
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
bull Cycle them gracefully
Cycling techniques
bull Above the stack
bull Within the stack
bull Platform
Above the stack
bull Use your favorite web server
bull Setup a load balancer above them route around servers that are cycling
Above the stack
bull PROTIP default setup wonrsquot work
bull Configure load balancer to poll for your application health
class HealthController lt ApplicationController def index if Fileexists(Railsrootjoin(MAINTENANCE)) render text =gt Forget me status =gt 503 else render text =gt Im alive end endend
Cycling + Load balancer
bounce serversrm MAINTENANCE
remove server from rotation
touch MAINTENANCE
new servers come online
add server to rotation
Server Load balancer
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy where to check for the health
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
do not let your web server queue requests
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy to run the health check specified above
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
every 2s
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
stop routing to it once the health check failed once
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
and bring it back once it worked once
ELB
$ elb-configure-healthcheck MyLoadBalancer --target HTTP5000health --interval 2 --unhealthy-threshold 1 --healthy-threshold 1
Above the stack
Above the stackbull Minimal impact
Above the stackbull Minimal impact
bull Extended availability comes for free
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
bull Another component you have to track and maintain
Within the stack
Within the stack
Unicorn
bull Replaces your web server
bull Bounces gracefully
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Within the stack
Within the stackbull Easy to setup and deploy
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
bull Doesnrsquot address availability issues
Platform
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 1s bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 5s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 1s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ heroku labsenable preboot
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 starting for 5s bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 up for 1s bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 5s bundle exec thin start -p $PORT
BETA
Heroku
bull Migrations
Heroku
bull Migrations
$ git push pre-production master$ heroku run rake dbmigrate --remote pre-production$ git push production master
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Future
Future
bull Asset pipeline
Future
bull Asset pipeline
bull Mysql tooling
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
bull Can databases
Future
Future
bull Hot compatibility beyond zero downtime deploys
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
Workflow
bull Make sure pull requests are addressing hot compatibility
bull Work after code is merged
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Production
Production
bull Run multiple server processes
Production
bull Run multiple server processes
bull Cycle them individually or in batches
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
bull Cycle them gracefully
Cycling techniques
bull Above the stack
bull Within the stack
bull Platform
Above the stack
bull Use your favorite web server
bull Setup a load balancer above them route around servers that are cycling
Above the stack
bull PROTIP default setup wonrsquot work
bull Configure load balancer to poll for your application health
class HealthController lt ApplicationController def index if Fileexists(Railsrootjoin(MAINTENANCE)) render text =gt Forget me status =gt 503 else render text =gt Im alive end endend
Cycling + Load balancer
bounce serversrm MAINTENANCE
remove server from rotation
touch MAINTENANCE
new servers come online
add server to rotation
Server Load balancer
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy where to check for the health
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
do not let your web server queue requests
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy to run the health check specified above
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
every 2s
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
stop routing to it once the health check failed once
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
and bring it back once it worked once
ELB
$ elb-configure-healthcheck MyLoadBalancer --target HTTP5000health --interval 2 --unhealthy-threshold 1 --healthy-threshold 1
Above the stack
Above the stackbull Minimal impact
Above the stackbull Minimal impact
bull Extended availability comes for free
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
bull Another component you have to track and maintain
Within the stack
Within the stack
Unicorn
bull Replaces your web server
bull Bounces gracefully
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Within the stack
Within the stackbull Easy to setup and deploy
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
bull Doesnrsquot address availability issues
Platform
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 1s bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 5s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 1s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ heroku labsenable preboot
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 starting for 5s bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 up for 1s bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 5s bundle exec thin start -p $PORT
BETA
Heroku
bull Migrations
Heroku
bull Migrations
$ git push pre-production master$ heroku run rake dbmigrate --remote pre-production$ git push production master
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Future
Future
bull Asset pipeline
Future
bull Asset pipeline
bull Mysql tooling
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
bull Can databases
Future
Future
bull Hot compatibility beyond zero downtime deploys
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Production
Production
bull Run multiple server processes
Production
bull Run multiple server processes
bull Cycle them individually or in batches
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
bull Cycle them gracefully
Cycling techniques
bull Above the stack
bull Within the stack
bull Platform
Above the stack
bull Use your favorite web server
bull Setup a load balancer above them route around servers that are cycling
Above the stack
bull PROTIP default setup wonrsquot work
bull Configure load balancer to poll for your application health
class HealthController lt ApplicationController def index if Fileexists(Railsrootjoin(MAINTENANCE)) render text =gt Forget me status =gt 503 else render text =gt Im alive end endend
Cycling + Load balancer
bounce serversrm MAINTENANCE
remove server from rotation
touch MAINTENANCE
new servers come online
add server to rotation
Server Load balancer
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy where to check for the health
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
do not let your web server queue requests
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy to run the health check specified above
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
every 2s
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
stop routing to it once the health check failed once
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
and bring it back once it worked once
ELB
$ elb-configure-healthcheck MyLoadBalancer --target HTTP5000health --interval 2 --unhealthy-threshold 1 --healthy-threshold 1
Above the stack
Above the stackbull Minimal impact
Above the stackbull Minimal impact
bull Extended availability comes for free
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
bull Another component you have to track and maintain
Within the stack
Within the stack
Unicorn
bull Replaces your web server
bull Bounces gracefully
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Within the stack
Within the stackbull Easy to setup and deploy
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
bull Doesnrsquot address availability issues
Platform
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 1s bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 5s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 1s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ heroku labsenable preboot
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 starting for 5s bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 up for 1s bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 5s bundle exec thin start -p $PORT
BETA
Heroku
bull Migrations
Heroku
bull Migrations
$ git push pre-production master$ heroku run rake dbmigrate --remote pre-production$ git push production master
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Future
Future
bull Asset pipeline
Future
bull Asset pipeline
bull Mysql tooling
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
bull Can databases
Future
Future
bull Hot compatibility beyond zero downtime deploys
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Production
Production
bull Run multiple server processes
Production
bull Run multiple server processes
bull Cycle them individually or in batches
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
bull Cycle them gracefully
Cycling techniques
bull Above the stack
bull Within the stack
bull Platform
Above the stack
bull Use your favorite web server
bull Setup a load balancer above them route around servers that are cycling
Above the stack
bull PROTIP default setup wonrsquot work
bull Configure load balancer to poll for your application health
class HealthController lt ApplicationController def index if Fileexists(Railsrootjoin(MAINTENANCE)) render text =gt Forget me status =gt 503 else render text =gt Im alive end endend
Cycling + Load balancer
bounce serversrm MAINTENANCE
remove server from rotation
touch MAINTENANCE
new servers come online
add server to rotation
Server Load balancer
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy where to check for the health
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
do not let your web server queue requests
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy to run the health check specified above
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
every 2s
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
stop routing to it once the health check failed once
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
and bring it back once it worked once
ELB
$ elb-configure-healthcheck MyLoadBalancer --target HTTP5000health --interval 2 --unhealthy-threshold 1 --healthy-threshold 1
Above the stack
Above the stackbull Minimal impact
Above the stackbull Minimal impact
bull Extended availability comes for free
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
bull Another component you have to track and maintain
Within the stack
Within the stack
Unicorn
bull Replaces your web server
bull Bounces gracefully
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Within the stack
Within the stackbull Easy to setup and deploy
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
bull Doesnrsquot address availability issues
Platform
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 1s bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 5s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 1s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ heroku labsenable preboot
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 starting for 5s bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 up for 1s bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 5s bundle exec thin start -p $PORT
BETA
Heroku
bull Migrations
Heroku
bull Migrations
$ git push pre-production master$ heroku run rake dbmigrate --remote pre-production$ git push production master
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Future
Future
bull Asset pipeline
Future
bull Asset pipeline
bull Mysql tooling
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
bull Can databases
Future
Future
bull Hot compatibility beyond zero downtime deploys
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Production
Production
bull Run multiple server processes
Production
bull Run multiple server processes
bull Cycle them individually or in batches
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
bull Cycle them gracefully
Cycling techniques
bull Above the stack
bull Within the stack
bull Platform
Above the stack
bull Use your favorite web server
bull Setup a load balancer above them route around servers that are cycling
Above the stack
bull PROTIP default setup wonrsquot work
bull Configure load balancer to poll for your application health
class HealthController lt ApplicationController def index if Fileexists(Railsrootjoin(MAINTENANCE)) render text =gt Forget me status =gt 503 else render text =gt Im alive end endend
Cycling + Load balancer
bounce serversrm MAINTENANCE
remove server from rotation
touch MAINTENANCE
new servers come online
add server to rotation
Server Load balancer
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy where to check for the health
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
do not let your web server queue requests
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy to run the health check specified above
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
every 2s
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
stop routing to it once the health check failed once
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
and bring it back once it worked once
ELB
$ elb-configure-healthcheck MyLoadBalancer --target HTTP5000health --interval 2 --unhealthy-threshold 1 --healthy-threshold 1
Above the stack
Above the stackbull Minimal impact
Above the stackbull Minimal impact
bull Extended availability comes for free
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
bull Another component you have to track and maintain
Within the stack
Within the stack
Unicorn
bull Replaces your web server
bull Bounces gracefully
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Within the stack
Within the stackbull Easy to setup and deploy
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
bull Doesnrsquot address availability issues
Platform
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 1s bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 5s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 1s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ heroku labsenable preboot
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 starting for 5s bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 up for 1s bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 5s bundle exec thin start -p $PORT
BETA
Heroku
bull Migrations
Heroku
bull Migrations
$ git push pre-production master$ heroku run rake dbmigrate --remote pre-production$ git push production master
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Future
Future
bull Asset pipeline
Future
bull Asset pipeline
bull Mysql tooling
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
bull Can databases
Future
Future
bull Hot compatibility beyond zero downtime deploys
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Production
Production
bull Run multiple server processes
Production
bull Run multiple server processes
bull Cycle them individually or in batches
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
bull Cycle them gracefully
Cycling techniques
bull Above the stack
bull Within the stack
bull Platform
Above the stack
bull Use your favorite web server
bull Setup a load balancer above them route around servers that are cycling
Above the stack
bull PROTIP default setup wonrsquot work
bull Configure load balancer to poll for your application health
class HealthController lt ApplicationController def index if Fileexists(Railsrootjoin(MAINTENANCE)) render text =gt Forget me status =gt 503 else render text =gt Im alive end endend
Cycling + Load balancer
bounce serversrm MAINTENANCE
remove server from rotation
touch MAINTENANCE
new servers come online
add server to rotation
Server Load balancer
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy where to check for the health
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
do not let your web server queue requests
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy to run the health check specified above
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
every 2s
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
stop routing to it once the health check failed once
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
and bring it back once it worked once
ELB
$ elb-configure-healthcheck MyLoadBalancer --target HTTP5000health --interval 2 --unhealthy-threshold 1 --healthy-threshold 1
Above the stack
Above the stackbull Minimal impact
Above the stackbull Minimal impact
bull Extended availability comes for free
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
bull Another component you have to track and maintain
Within the stack
Within the stack
Unicorn
bull Replaces your web server
bull Bounces gracefully
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Within the stack
Within the stackbull Easy to setup and deploy
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
bull Doesnrsquot address availability issues
Platform
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 1s bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 5s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 1s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ heroku labsenable preboot
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 starting for 5s bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 up for 1s bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 5s bundle exec thin start -p $PORT
BETA
Heroku
bull Migrations
Heroku
bull Migrations
$ git push pre-production master$ heroku run rake dbmigrate --remote pre-production$ git push production master
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Future
Future
bull Asset pipeline
Future
bull Asset pipeline
bull Mysql tooling
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
bull Can databases
Future
Future
bull Hot compatibility beyond zero downtime deploys
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
Meanwhile in production$ thin -C configserveryml restartStopping server on 00005000 Sending QUIT signal to process 3463 Stopping server on 00005001 Sending QUIT signal to process 3464 Stopping server on 00005002 Sending QUIT signal to process 3465 Starting server on 00005000 Starting server on 00005001 Starting server on 00005002
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Production
Production
bull Run multiple server processes
Production
bull Run multiple server processes
bull Cycle them individually or in batches
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
bull Cycle them gracefully
Cycling techniques
bull Above the stack
bull Within the stack
bull Platform
Above the stack
bull Use your favorite web server
bull Setup a load balancer above them route around servers that are cycling
Above the stack
bull PROTIP default setup wonrsquot work
bull Configure load balancer to poll for your application health
class HealthController lt ApplicationController def index if Fileexists(Railsrootjoin(MAINTENANCE)) render text =gt Forget me status =gt 503 else render text =gt Im alive end endend
Cycling + Load balancer
bounce serversrm MAINTENANCE
remove server from rotation
touch MAINTENANCE
new servers come online
add server to rotation
Server Load balancer
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy where to check for the health
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
do not let your web server queue requests
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy to run the health check specified above
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
every 2s
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
stop routing to it once the health check failed once
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
and bring it back once it worked once
ELB
$ elb-configure-healthcheck MyLoadBalancer --target HTTP5000health --interval 2 --unhealthy-threshold 1 --healthy-threshold 1
Above the stack
Above the stackbull Minimal impact
Above the stackbull Minimal impact
bull Extended availability comes for free
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
bull Another component you have to track and maintain
Within the stack
Within the stack
Unicorn
bull Replaces your web server
bull Bounces gracefully
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Within the stack
Within the stackbull Easy to setup and deploy
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
bull Doesnrsquot address availability issues
Platform
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 1s bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 5s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 1s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ heroku labsenable preboot
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 starting for 5s bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 up for 1s bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 5s bundle exec thin start -p $PORT
BETA
Heroku
bull Migrations
Heroku
bull Migrations
$ git push pre-production master$ heroku run rake dbmigrate --remote pre-production$ git push production master
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Future
Future
bull Asset pipeline
Future
bull Asset pipeline
bull Mysql tooling
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
bull Can databases
Future
Future
bull Hot compatibility beyond zero downtime deploys
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Production
Production
bull Run multiple server processes
Production
bull Run multiple server processes
bull Cycle them individually or in batches
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
bull Cycle them gracefully
Cycling techniques
bull Above the stack
bull Within the stack
bull Platform
Above the stack
bull Use your favorite web server
bull Setup a load balancer above them route around servers that are cycling
Above the stack
bull PROTIP default setup wonrsquot work
bull Configure load balancer to poll for your application health
class HealthController lt ApplicationController def index if Fileexists(Railsrootjoin(MAINTENANCE)) render text =gt Forget me status =gt 503 else render text =gt Im alive end endend
Cycling + Load balancer
bounce serversrm MAINTENANCE
remove server from rotation
touch MAINTENANCE
new servers come online
add server to rotation
Server Load balancer
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy where to check for the health
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
do not let your web server queue requests
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy to run the health check specified above
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
every 2s
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
stop routing to it once the health check failed once
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
and bring it back once it worked once
ELB
$ elb-configure-healthcheck MyLoadBalancer --target HTTP5000health --interval 2 --unhealthy-threshold 1 --healthy-threshold 1
Above the stack
Above the stackbull Minimal impact
Above the stackbull Minimal impact
bull Extended availability comes for free
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
bull Another component you have to track and maintain
Within the stack
Within the stack
Unicorn
bull Replaces your web server
bull Bounces gracefully
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Within the stack
Within the stackbull Easy to setup and deploy
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
bull Doesnrsquot address availability issues
Platform
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 1s bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 5s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 1s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ heroku labsenable preboot
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 starting for 5s bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 up for 1s bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 5s bundle exec thin start -p $PORT
BETA
Heroku
bull Migrations
Heroku
bull Migrations
$ git push pre-production master$ heroku run rake dbmigrate --remote pre-production$ git push production master
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Future
Future
bull Asset pipeline
Future
bull Asset pipeline
bull Mysql tooling
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
bull Can databases
Future
Future
bull Hot compatibility beyond zero downtime deploys
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
Production
Production
bull Run multiple server processes
Production
bull Run multiple server processes
bull Cycle them individually or in batches
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
bull Cycle them gracefully
Cycling techniques
bull Above the stack
bull Within the stack
bull Platform
Above the stack
bull Use your favorite web server
bull Setup a load balancer above them route around servers that are cycling
Above the stack
bull PROTIP default setup wonrsquot work
bull Configure load balancer to poll for your application health
class HealthController lt ApplicationController def index if Fileexists(Railsrootjoin(MAINTENANCE)) render text =gt Forget me status =gt 503 else render text =gt Im alive end endend
Cycling + Load balancer
bounce serversrm MAINTENANCE
remove server from rotation
touch MAINTENANCE
new servers come online
add server to rotation
Server Load balancer
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy where to check for the health
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
do not let your web server queue requests
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy to run the health check specified above
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
every 2s
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
stop routing to it once the health check failed once
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
and bring it back once it worked once
ELB
$ elb-configure-healthcheck MyLoadBalancer --target HTTP5000health --interval 2 --unhealthy-threshold 1 --healthy-threshold 1
Above the stack
Above the stackbull Minimal impact
Above the stackbull Minimal impact
bull Extended availability comes for free
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
bull Another component you have to track and maintain
Within the stack
Within the stack
Unicorn
bull Replaces your web server
bull Bounces gracefully
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Within the stack
Within the stackbull Easy to setup and deploy
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
bull Doesnrsquot address availability issues
Platform
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 1s bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 5s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 1s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ heroku labsenable preboot
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 starting for 5s bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 up for 1s bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 5s bundle exec thin start -p $PORT
BETA
Heroku
bull Migrations
Heroku
bull Migrations
$ git push pre-production master$ heroku run rake dbmigrate --remote pre-production$ git push production master
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Future
Future
bull Asset pipeline
Future
bull Asset pipeline
bull Mysql tooling
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
bull Can databases
Future
Future
bull Hot compatibility beyond zero downtime deploys
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
Production
bull Run multiple server processes
Production
bull Run multiple server processes
bull Cycle them individually or in batches
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
bull Cycle them gracefully
Cycling techniques
bull Above the stack
bull Within the stack
bull Platform
Above the stack
bull Use your favorite web server
bull Setup a load balancer above them route around servers that are cycling
Above the stack
bull PROTIP default setup wonrsquot work
bull Configure load balancer to poll for your application health
class HealthController lt ApplicationController def index if Fileexists(Railsrootjoin(MAINTENANCE)) render text =gt Forget me status =gt 503 else render text =gt Im alive end endend
Cycling + Load balancer
bounce serversrm MAINTENANCE
remove server from rotation
touch MAINTENANCE
new servers come online
add server to rotation
Server Load balancer
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy where to check for the health
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
do not let your web server queue requests
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy to run the health check specified above
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
every 2s
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
stop routing to it once the health check failed once
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
and bring it back once it worked once
ELB
$ elb-configure-healthcheck MyLoadBalancer --target HTTP5000health --interval 2 --unhealthy-threshold 1 --healthy-threshold 1
Above the stack
Above the stackbull Minimal impact
Above the stackbull Minimal impact
bull Extended availability comes for free
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
bull Another component you have to track and maintain
Within the stack
Within the stack
Unicorn
bull Replaces your web server
bull Bounces gracefully
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Within the stack
Within the stackbull Easy to setup and deploy
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
bull Doesnrsquot address availability issues
Platform
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 1s bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 5s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 1s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ heroku labsenable preboot
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 starting for 5s bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 up for 1s bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 5s bundle exec thin start -p $PORT
BETA
Heroku
bull Migrations
Heroku
bull Migrations
$ git push pre-production master$ heroku run rake dbmigrate --remote pre-production$ git push production master
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Future
Future
bull Asset pipeline
Future
bull Asset pipeline
bull Mysql tooling
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
bull Can databases
Future
Future
bull Hot compatibility beyond zero downtime deploys
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
Production
bull Run multiple server processes
bull Cycle them individually or in batches
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
bull Cycle them gracefully
Cycling techniques
bull Above the stack
bull Within the stack
bull Platform
Above the stack
bull Use your favorite web server
bull Setup a load balancer above them route around servers that are cycling
Above the stack
bull PROTIP default setup wonrsquot work
bull Configure load balancer to poll for your application health
class HealthController lt ApplicationController def index if Fileexists(Railsrootjoin(MAINTENANCE)) render text =gt Forget me status =gt 503 else render text =gt Im alive end endend
Cycling + Load balancer
bounce serversrm MAINTENANCE
remove server from rotation
touch MAINTENANCE
new servers come online
add server to rotation
Server Load balancer
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy where to check for the health
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
do not let your web server queue requests
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy to run the health check specified above
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
every 2s
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
stop routing to it once the health check failed once
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
and bring it back once it worked once
ELB
$ elb-configure-healthcheck MyLoadBalancer --target HTTP5000health --interval 2 --unhealthy-threshold 1 --healthy-threshold 1
Above the stack
Above the stackbull Minimal impact
Above the stackbull Minimal impact
bull Extended availability comes for free
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
bull Another component you have to track and maintain
Within the stack
Within the stack
Unicorn
bull Replaces your web server
bull Bounces gracefully
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Within the stack
Within the stackbull Easy to setup and deploy
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
bull Doesnrsquot address availability issues
Platform
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 1s bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 5s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 1s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ heroku labsenable preboot
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 starting for 5s bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 up for 1s bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 5s bundle exec thin start -p $PORT
BETA
Heroku
bull Migrations
Heroku
bull Migrations
$ git push pre-production master$ heroku run rake dbmigrate --remote pre-production$ git push production master
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Future
Future
bull Asset pipeline
Future
bull Asset pipeline
bull Mysql tooling
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
bull Can databases
Future
Future
bull Hot compatibility beyond zero downtime deploys
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
bull Cycle them gracefully
Cycling techniques
bull Above the stack
bull Within the stack
bull Platform
Above the stack
bull Use your favorite web server
bull Setup a load balancer above them route around servers that are cycling
Above the stack
bull PROTIP default setup wonrsquot work
bull Configure load balancer to poll for your application health
class HealthController lt ApplicationController def index if Fileexists(Railsrootjoin(MAINTENANCE)) render text =gt Forget me status =gt 503 else render text =gt Im alive end endend
Cycling + Load balancer
bounce serversrm MAINTENANCE
remove server from rotation
touch MAINTENANCE
new servers come online
add server to rotation
Server Load balancer
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy where to check for the health
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
do not let your web server queue requests
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy to run the health check specified above
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
every 2s
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
stop routing to it once the health check failed once
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
and bring it back once it worked once
ELB
$ elb-configure-healthcheck MyLoadBalancer --target HTTP5000health --interval 2 --unhealthy-threshold 1 --healthy-threshold 1
Above the stack
Above the stackbull Minimal impact
Above the stackbull Minimal impact
bull Extended availability comes for free
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
bull Another component you have to track and maintain
Within the stack
Within the stack
Unicorn
bull Replaces your web server
bull Bounces gracefully
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Within the stack
Within the stackbull Easy to setup and deploy
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
bull Doesnrsquot address availability issues
Platform
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 1s bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 5s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 1s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ heroku labsenable preboot
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 starting for 5s bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 up for 1s bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 5s bundle exec thin start -p $PORT
BETA
Heroku
bull Migrations
Heroku
bull Migrations
$ git push pre-production master$ heroku run rake dbmigrate --remote pre-production$ git push production master
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Future
Future
bull Asset pipeline
Future
bull Asset pipeline
bull Mysql tooling
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
bull Can databases
Future
Future
bull Hot compatibility beyond zero downtime deploys
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
Production
bull Run multiple server processes
bull Cycle them individually or in batches
bull Watch out for capacity
bull Cycle them gracefully
Cycling techniques
bull Above the stack
bull Within the stack
bull Platform
Above the stack
bull Use your favorite web server
bull Setup a load balancer above them route around servers that are cycling
Above the stack
bull PROTIP default setup wonrsquot work
bull Configure load balancer to poll for your application health
class HealthController lt ApplicationController def index if Fileexists(Railsrootjoin(MAINTENANCE)) render text =gt Forget me status =gt 503 else render text =gt Im alive end endend
Cycling + Load balancer
bounce serversrm MAINTENANCE
remove server from rotation
touch MAINTENANCE
new servers come online
add server to rotation
Server Load balancer
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy where to check for the health
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
do not let your web server queue requests
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy to run the health check specified above
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
every 2s
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
stop routing to it once the health check failed once
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
and bring it back once it worked once
ELB
$ elb-configure-healthcheck MyLoadBalancer --target HTTP5000health --interval 2 --unhealthy-threshold 1 --healthy-threshold 1
Above the stack
Above the stackbull Minimal impact
Above the stackbull Minimal impact
bull Extended availability comes for free
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
bull Another component you have to track and maintain
Within the stack
Within the stack
Unicorn
bull Replaces your web server
bull Bounces gracefully
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Within the stack
Within the stackbull Easy to setup and deploy
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
bull Doesnrsquot address availability issues
Platform
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 1s bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 5s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 1s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ heroku labsenable preboot
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 starting for 5s bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 up for 1s bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 5s bundle exec thin start -p $PORT
BETA
Heroku
bull Migrations
Heroku
bull Migrations
$ git push pre-production master$ heroku run rake dbmigrate --remote pre-production$ git push production master
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Future
Future
bull Asset pipeline
Future
bull Asset pipeline
bull Mysql tooling
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
bull Can databases
Future
Future
bull Hot compatibility beyond zero downtime deploys
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
Cycling techniques
bull Above the stack
bull Within the stack
bull Platform
Above the stack
bull Use your favorite web server
bull Setup a load balancer above them route around servers that are cycling
Above the stack
bull PROTIP default setup wonrsquot work
bull Configure load balancer to poll for your application health
class HealthController lt ApplicationController def index if Fileexists(Railsrootjoin(MAINTENANCE)) render text =gt Forget me status =gt 503 else render text =gt Im alive end endend
Cycling + Load balancer
bounce serversrm MAINTENANCE
remove server from rotation
touch MAINTENANCE
new servers come online
add server to rotation
Server Load balancer
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy where to check for the health
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
do not let your web server queue requests
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy to run the health check specified above
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
every 2s
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
stop routing to it once the health check failed once
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
and bring it back once it worked once
ELB
$ elb-configure-healthcheck MyLoadBalancer --target HTTP5000health --interval 2 --unhealthy-threshold 1 --healthy-threshold 1
Above the stack
Above the stackbull Minimal impact
Above the stackbull Minimal impact
bull Extended availability comes for free
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
bull Another component you have to track and maintain
Within the stack
Within the stack
Unicorn
bull Replaces your web server
bull Bounces gracefully
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Within the stack
Within the stackbull Easy to setup and deploy
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
bull Doesnrsquot address availability issues
Platform
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 1s bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 5s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 1s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ heroku labsenable preboot
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 starting for 5s bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 up for 1s bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 5s bundle exec thin start -p $PORT
BETA
Heroku
bull Migrations
Heroku
bull Migrations
$ git push pre-production master$ heroku run rake dbmigrate --remote pre-production$ git push production master
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Future
Future
bull Asset pipeline
Future
bull Asset pipeline
bull Mysql tooling
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
bull Can databases
Future
Future
bull Hot compatibility beyond zero downtime deploys
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
Above the stack
bull Use your favorite web server
bull Setup a load balancer above them route around servers that are cycling
Above the stack
bull PROTIP default setup wonrsquot work
bull Configure load balancer to poll for your application health
class HealthController lt ApplicationController def index if Fileexists(Railsrootjoin(MAINTENANCE)) render text =gt Forget me status =gt 503 else render text =gt Im alive end endend
Cycling + Load balancer
bounce serversrm MAINTENANCE
remove server from rotation
touch MAINTENANCE
new servers come online
add server to rotation
Server Load balancer
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy where to check for the health
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
do not let your web server queue requests
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy to run the health check specified above
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
every 2s
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
stop routing to it once the health check failed once
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
and bring it back once it worked once
ELB
$ elb-configure-healthcheck MyLoadBalancer --target HTTP5000health --interval 2 --unhealthy-threshold 1 --healthy-threshold 1
Above the stack
Above the stackbull Minimal impact
Above the stackbull Minimal impact
bull Extended availability comes for free
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
bull Another component you have to track and maintain
Within the stack
Within the stack
Unicorn
bull Replaces your web server
bull Bounces gracefully
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Within the stack
Within the stackbull Easy to setup and deploy
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
bull Doesnrsquot address availability issues
Platform
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 1s bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 5s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 1s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ heroku labsenable preboot
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 starting for 5s bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 up for 1s bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 5s bundle exec thin start -p $PORT
BETA
Heroku
bull Migrations
Heroku
bull Migrations
$ git push pre-production master$ heroku run rake dbmigrate --remote pre-production$ git push production master
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Future
Future
bull Asset pipeline
Future
bull Asset pipeline
bull Mysql tooling
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
bull Can databases
Future
Future
bull Hot compatibility beyond zero downtime deploys
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
Above the stack
bull PROTIP default setup wonrsquot work
bull Configure load balancer to poll for your application health
class HealthController lt ApplicationController def index if Fileexists(Railsrootjoin(MAINTENANCE)) render text =gt Forget me status =gt 503 else render text =gt Im alive end endend
Cycling + Load balancer
bounce serversrm MAINTENANCE
remove server from rotation
touch MAINTENANCE
new servers come online
add server to rotation
Server Load balancer
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy where to check for the health
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
do not let your web server queue requests
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy to run the health check specified above
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
every 2s
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
stop routing to it once the health check failed once
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
and bring it back once it worked once
ELB
$ elb-configure-healthcheck MyLoadBalancer --target HTTP5000health --interval 2 --unhealthy-threshold 1 --healthy-threshold 1
Above the stack
Above the stackbull Minimal impact
Above the stackbull Minimal impact
bull Extended availability comes for free
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
bull Another component you have to track and maintain
Within the stack
Within the stack
Unicorn
bull Replaces your web server
bull Bounces gracefully
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Within the stack
Within the stackbull Easy to setup and deploy
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
bull Doesnrsquot address availability issues
Platform
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 1s bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 5s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 1s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ heroku labsenable preboot
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 starting for 5s bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 up for 1s bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 5s bundle exec thin start -p $PORT
BETA
Heroku
bull Migrations
Heroku
bull Migrations
$ git push pre-production master$ heroku run rake dbmigrate --remote pre-production$ git push production master
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Future
Future
bull Asset pipeline
Future
bull Asset pipeline
bull Mysql tooling
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
bull Can databases
Future
Future
bull Hot compatibility beyond zero downtime deploys
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
Cycling + Load balancer
bounce serversrm MAINTENANCE
remove server from rotation
touch MAINTENANCE
new servers come online
add server to rotation
Server Load balancer
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy where to check for the health
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
do not let your web server queue requests
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy to run the health check specified above
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
every 2s
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
stop routing to it once the health check failed once
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
and bring it back once it worked once
ELB
$ elb-configure-healthcheck MyLoadBalancer --target HTTP5000health --interval 2 --unhealthy-threshold 1 --healthy-threshold 1
Above the stack
Above the stackbull Minimal impact
Above the stackbull Minimal impact
bull Extended availability comes for free
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
bull Another component you have to track and maintain
Within the stack
Within the stack
Unicorn
bull Replaces your web server
bull Bounces gracefully
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Within the stack
Within the stackbull Easy to setup and deploy
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
bull Doesnrsquot address availability issues
Platform
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 1s bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 5s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 1s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ heroku labsenable preboot
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 starting for 5s bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 up for 1s bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 5s bundle exec thin start -p $PORT
BETA
Heroku
bull Migrations
Heroku
bull Migrations
$ git push pre-production master$ heroku run rake dbmigrate --remote pre-production$ git push production master
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Future
Future
bull Asset pipeline
Future
bull Asset pipeline
bull Mysql tooling
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
bull Can databases
Future
Future
bull Hot compatibility beyond zero downtime deploys
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy where to check for the health
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
do not let your web server queue requests
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy to run the health check specified above
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
every 2s
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
stop routing to it once the health check failed once
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
and bring it back once it worked once
ELB
$ elb-configure-healthcheck MyLoadBalancer --target HTTP5000health --interval 2 --unhealthy-threshold 1 --healthy-threshold 1
Above the stack
Above the stackbull Minimal impact
Above the stackbull Minimal impact
bull Extended availability comes for free
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
bull Another component you have to track and maintain
Within the stack
Within the stack
Unicorn
bull Replaces your web server
bull Bounces gracefully
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Within the stack
Within the stackbull Easy to setup and deploy
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
bull Doesnrsquot address availability issues
Platform
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 1s bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 5s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 1s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ heroku labsenable preboot
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 starting for 5s bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 up for 1s bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 5s bundle exec thin start -p $PORT
BETA
Heroku
bull Migrations
Heroku
bull Migrations
$ git push pre-production master$ heroku run rake dbmigrate --remote pre-production$ git push production master
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Future
Future
bull Asset pipeline
Future
bull Asset pipeline
bull Mysql tooling
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
bull Can databases
Future
Future
bull Hot compatibility beyond zero downtime deploys
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
HAProxy
tell HAProxy where to check for the health
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
do not let your web server queue requests
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy to run the health check specified above
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
every 2s
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
stop routing to it once the health check failed once
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
and bring it back once it worked once
ELB
$ elb-configure-healthcheck MyLoadBalancer --target HTTP5000health --interval 2 --unhealthy-threshold 1 --healthy-threshold 1
Above the stack
Above the stackbull Minimal impact
Above the stackbull Minimal impact
bull Extended availability comes for free
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
bull Another component you have to track and maintain
Within the stack
Within the stack
Unicorn
bull Replaces your web server
bull Bounces gracefully
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Within the stack
Within the stackbull Easy to setup and deploy
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
bull Doesnrsquot address availability issues
Platform
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 1s bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 5s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 1s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ heroku labsenable preboot
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 starting for 5s bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 up for 1s bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 5s bundle exec thin start -p $PORT
BETA
Heroku
bull Migrations
Heroku
bull Migrations
$ git push pre-production master$ heroku run rake dbmigrate --remote pre-production$ git push production master
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Future
Future
bull Asset pipeline
Future
bull Asset pipeline
bull Mysql tooling
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
bull Can databases
Future
Future
bull Hot compatibility beyond zero downtime deploys
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
HAProxy
do not let your web server queue requests
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
tell HAProxy to run the health check specified above
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
every 2s
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
stop routing to it once the health check failed once
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
and bring it back once it worked once
ELB
$ elb-configure-healthcheck MyLoadBalancer --target HTTP5000health --interval 2 --unhealthy-threshold 1 --healthy-threshold 1
Above the stack
Above the stackbull Minimal impact
Above the stackbull Minimal impact
bull Extended availability comes for free
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
bull Another component you have to track and maintain
Within the stack
Within the stack
Unicorn
bull Replaces your web server
bull Bounces gracefully
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Within the stack
Within the stackbull Easy to setup and deploy
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
bull Doesnrsquot address availability issues
Platform
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 1s bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 5s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 1s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ heroku labsenable preboot
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 starting for 5s bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 up for 1s bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 5s bundle exec thin start -p $PORT
BETA
Heroku
bull Migrations
Heroku
bull Migrations
$ git push pre-production master$ heroku run rake dbmigrate --remote pre-production$ git push production master
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Future
Future
bull Asset pipeline
Future
bull Asset pipeline
bull Mysql tooling
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
bull Can databases
Future
Future
bull Hot compatibility beyond zero downtime deploys
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
HAProxy
tell HAProxy to run the health check specified above
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
every 2s
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
stop routing to it once the health check failed once
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
and bring it back once it worked once
ELB
$ elb-configure-healthcheck MyLoadBalancer --target HTTP5000health --interval 2 --unhealthy-threshold 1 --healthy-threshold 1
Above the stack
Above the stackbull Minimal impact
Above the stackbull Minimal impact
bull Extended availability comes for free
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
bull Another component you have to track and maintain
Within the stack
Within the stack
Unicorn
bull Replaces your web server
bull Bounces gracefully
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Within the stack
Within the stackbull Easy to setup and deploy
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
bull Doesnrsquot address availability issues
Platform
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 1s bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 5s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 1s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ heroku labsenable preboot
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 starting for 5s bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 up for 1s bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 5s bundle exec thin start -p $PORT
BETA
Heroku
bull Migrations
Heroku
bull Migrations
$ git push pre-production master$ heroku run rake dbmigrate --remote pre-production$ git push production master
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Future
Future
bull Asset pipeline
Future
bull Asset pipeline
bull Mysql tooling
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
bull Can databases
Future
Future
bull Hot compatibility beyond zero downtime deploys
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
HAProxy
every 2s
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
stop routing to it once the health check failed once
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
and bring it back once it worked once
ELB
$ elb-configure-healthcheck MyLoadBalancer --target HTTP5000health --interval 2 --unhealthy-threshold 1 --healthy-threshold 1
Above the stack
Above the stackbull Minimal impact
Above the stackbull Minimal impact
bull Extended availability comes for free
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
bull Another component you have to track and maintain
Within the stack
Within the stack
Unicorn
bull Replaces your web server
bull Bounces gracefully
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Within the stack
Within the stackbull Easy to setup and deploy
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
bull Doesnrsquot address availability issues
Platform
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 1s bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 5s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 1s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ heroku labsenable preboot
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 starting for 5s bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 up for 1s bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 5s bundle exec thin start -p $PORT
BETA
Heroku
bull Migrations
Heroku
bull Migrations
$ git push pre-production master$ heroku run rake dbmigrate --remote pre-production$ git push production master
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Future
Future
bull Asset pipeline
Future
bull Asset pipeline
bull Mysql tooling
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
bull Can databases
Future
Future
bull Hot compatibility beyond zero downtime deploys
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
stop routing to it once the health check failed once
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
and bring it back once it worked once
ELB
$ elb-configure-healthcheck MyLoadBalancer --target HTTP5000health --interval 2 --unhealthy-threshold 1 --healthy-threshold 1
Above the stack
Above the stackbull Minimal impact
Above the stackbull Minimal impact
bull Extended availability comes for free
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
bull Another component you have to track and maintain
Within the stack
Within the stack
Unicorn
bull Replaces your web server
bull Bounces gracefully
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Within the stack
Within the stackbull Easy to setup and deploy
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
bull Doesnrsquot address availability issues
Platform
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 1s bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 5s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 1s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ heroku labsenable preboot
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 starting for 5s bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 up for 1s bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 5s bundle exec thin start -p $PORT
BETA
Heroku
bull Migrations
Heroku
bull Migrations
$ git push pre-production master$ heroku run rake dbmigrate --remote pre-production$ git push production master
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Future
Future
bull Asset pipeline
Future
bull Asset pipeline
bull Mysql tooling
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
bull Can databases
Future
Future
bull Hot compatibility beyond zero downtime deploys
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
HAProxy
option httpchk GET health
listen myapp 9000 server myapp-1 10015000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 10025000 maxconn 1 check inter 2000 fall 1 rise 1
and bring it back once it worked once
ELB
$ elb-configure-healthcheck MyLoadBalancer --target HTTP5000health --interval 2 --unhealthy-threshold 1 --healthy-threshold 1
Above the stack
Above the stackbull Minimal impact
Above the stackbull Minimal impact
bull Extended availability comes for free
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
bull Another component you have to track and maintain
Within the stack
Within the stack
Unicorn
bull Replaces your web server
bull Bounces gracefully
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Within the stack
Within the stackbull Easy to setup and deploy
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
bull Doesnrsquot address availability issues
Platform
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 1s bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 5s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 1s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ heroku labsenable preboot
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 starting for 5s bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 up for 1s bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 5s bundle exec thin start -p $PORT
BETA
Heroku
bull Migrations
Heroku
bull Migrations
$ git push pre-production master$ heroku run rake dbmigrate --remote pre-production$ git push production master
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Future
Future
bull Asset pipeline
Future
bull Asset pipeline
bull Mysql tooling
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
bull Can databases
Future
Future
bull Hot compatibility beyond zero downtime deploys
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
ELB
$ elb-configure-healthcheck MyLoadBalancer --target HTTP5000health --interval 2 --unhealthy-threshold 1 --healthy-threshold 1
Above the stack
Above the stackbull Minimal impact
Above the stackbull Minimal impact
bull Extended availability comes for free
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
bull Another component you have to track and maintain
Within the stack
Within the stack
Unicorn
bull Replaces your web server
bull Bounces gracefully
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Within the stack
Within the stackbull Easy to setup and deploy
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
bull Doesnrsquot address availability issues
Platform
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 1s bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 5s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 1s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ heroku labsenable preboot
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 starting for 5s bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 up for 1s bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 5s bundle exec thin start -p $PORT
BETA
Heroku
bull Migrations
Heroku
bull Migrations
$ git push pre-production master$ heroku run rake dbmigrate --remote pre-production$ git push production master
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Future
Future
bull Asset pipeline
Future
bull Asset pipeline
bull Mysql tooling
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
bull Can databases
Future
Future
bull Hot compatibility beyond zero downtime deploys
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
Above the stack
Above the stackbull Minimal impact
Above the stackbull Minimal impact
bull Extended availability comes for free
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
bull Another component you have to track and maintain
Within the stack
Within the stack
Unicorn
bull Replaces your web server
bull Bounces gracefully
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Within the stack
Within the stackbull Easy to setup and deploy
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
bull Doesnrsquot address availability issues
Platform
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 1s bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 5s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 1s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ heroku labsenable preboot
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 starting for 5s bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 up for 1s bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 5s bundle exec thin start -p $PORT
BETA
Heroku
bull Migrations
Heroku
bull Migrations
$ git push pre-production master$ heroku run rake dbmigrate --remote pre-production$ git push production master
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Future
Future
bull Asset pipeline
Future
bull Asset pipeline
bull Mysql tooling
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
bull Can databases
Future
Future
bull Hot compatibility beyond zero downtime deploys
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
Above the stackbull Minimal impact
Above the stackbull Minimal impact
bull Extended availability comes for free
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
bull Another component you have to track and maintain
Within the stack
Within the stack
Unicorn
bull Replaces your web server
bull Bounces gracefully
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Within the stack
Within the stackbull Easy to setup and deploy
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
bull Doesnrsquot address availability issues
Platform
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 1s bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 5s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 1s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ heroku labsenable preboot
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 starting for 5s bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 up for 1s bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 5s bundle exec thin start -p $PORT
BETA
Heroku
bull Migrations
Heroku
bull Migrations
$ git push pre-production master$ heroku run rake dbmigrate --remote pre-production$ git push production master
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Future
Future
bull Asset pipeline
Future
bull Asset pipeline
bull Mysql tooling
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
bull Can databases
Future
Future
bull Hot compatibility beyond zero downtime deploys
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
Above the stackbull Minimal impact
bull Extended availability comes for free
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
bull Another component you have to track and maintain
Within the stack
Within the stack
Unicorn
bull Replaces your web server
bull Bounces gracefully
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Within the stack
Within the stackbull Easy to setup and deploy
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
bull Doesnrsquot address availability issues
Platform
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 1s bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 5s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 1s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ heroku labsenable preboot
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 starting for 5s bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 up for 1s bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 5s bundle exec thin start -p $PORT
BETA
Heroku
bull Migrations
Heroku
bull Migrations
$ git push pre-production master$ heroku run rake dbmigrate --remote pre-production$ git push production master
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Future
Future
bull Asset pipeline
Future
bull Asset pipeline
bull Mysql tooling
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
bull Can databases
Future
Future
bull Hot compatibility beyond zero downtime deploys
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
bull Another component you have to track and maintain
Within the stack
Within the stack
Unicorn
bull Replaces your web server
bull Bounces gracefully
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Within the stack
Within the stackbull Easy to setup and deploy
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
bull Doesnrsquot address availability issues
Platform
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 1s bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 5s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 1s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ heroku labsenable preboot
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 starting for 5s bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 up for 1s bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 5s bundle exec thin start -p $PORT
BETA
Heroku
bull Migrations
Heroku
bull Migrations
$ git push pre-production master$ heroku run rake dbmigrate --remote pre-production$ git push production master
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Future
Future
bull Asset pipeline
Future
bull Asset pipeline
bull Mysql tooling
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
bull Can databases
Future
Future
bull Hot compatibility beyond zero downtime deploys
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
Above the stackbull Minimal impact
bull Extended availability comes for free
bull Complicated setup
bull Another component you have to track and maintain
Within the stack
Within the stack
Unicorn
bull Replaces your web server
bull Bounces gracefully
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Within the stack
Within the stackbull Easy to setup and deploy
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
bull Doesnrsquot address availability issues
Platform
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 1s bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 5s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 1s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ heroku labsenable preboot
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 starting for 5s bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 up for 1s bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 5s bundle exec thin start -p $PORT
BETA
Heroku
bull Migrations
Heroku
bull Migrations
$ git push pre-production master$ heroku run rake dbmigrate --remote pre-production$ git push production master
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Future
Future
bull Asset pipeline
Future
bull Asset pipeline
bull Mysql tooling
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
bull Can databases
Future
Future
bull Hot compatibility beyond zero downtime deploys
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
Within the stack
Within the stack
Unicorn
bull Replaces your web server
bull Bounces gracefully
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Within the stack
Within the stackbull Easy to setup and deploy
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
bull Doesnrsquot address availability issues
Platform
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 1s bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 5s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 1s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ heroku labsenable preboot
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 starting for 5s bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 up for 1s bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 5s bundle exec thin start -p $PORT
BETA
Heroku
bull Migrations
Heroku
bull Migrations
$ git push pre-production master$ heroku run rake dbmigrate --remote pre-production$ git push production master
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Future
Future
bull Asset pipeline
Future
bull Asset pipeline
bull Mysql tooling
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
bull Can databases
Future
Future
bull Hot compatibility beyond zero downtime deploys
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
Within the stack
Unicorn
bull Replaces your web server
bull Bounces gracefully
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Within the stack
Within the stackbull Easy to setup and deploy
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
bull Doesnrsquot address availability issues
Platform
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 1s bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 5s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 1s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ heroku labsenable preboot
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 starting for 5s bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 up for 1s bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 5s bundle exec thin start -p $PORT
BETA
Heroku
bull Migrations
Heroku
bull Migrations
$ git push pre-production master$ heroku run rake dbmigrate --remote pre-production$ git push production master
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Future
Future
bull Asset pipeline
Future
bull Asset pipeline
bull Mysql tooling
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
bull Can databases
Future
Future
bull Hot compatibility beyond zero downtime deploys
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
Unicorn
bull Replaces your web server
bull Bounces gracefully
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Within the stack
Within the stackbull Easy to setup and deploy
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
bull Doesnrsquot address availability issues
Platform
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 1s bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 5s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 1s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ heroku labsenable preboot
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 starting for 5s bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 up for 1s bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 5s bundle exec thin start -p $PORT
BETA
Heroku
bull Migrations
Heroku
bull Migrations
$ git push pre-production master$ heroku run rake dbmigrate --remote pre-production$ git push production master
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Future
Future
bull Asset pipeline
Future
bull Asset pipeline
bull Mysql tooling
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
bull Can databases
Future
Future
bull Hot compatibility beyond zero downtime deploys
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Within the stack
Within the stackbull Easy to setup and deploy
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
bull Doesnrsquot address availability issues
Platform
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 1s bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 5s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 1s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ heroku labsenable preboot
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 starting for 5s bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 up for 1s bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 5s bundle exec thin start -p $PORT
BETA
Heroku
bull Migrations
Heroku
bull Migrations
$ git push pre-production master$ heroku run rake dbmigrate --remote pre-production$ git push production master
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Future
Future
bull Asset pipeline
Future
bull Asset pipeline
bull Mysql tooling
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
bull Can databases
Future
Future
bull Hot compatibility beyond zero downtime deploys
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
Unicorn
63480 00 14 2549796 unicorn_rails master -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Within the stack
Within the stackbull Easy to setup and deploy
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
bull Doesnrsquot address availability issues
Platform
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 1s bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 5s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 1s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ heroku labsenable preboot
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 starting for 5s bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 up for 1s bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 5s bundle exec thin start -p $PORT
BETA
Heroku
bull Migrations
Heroku
bull Migrations
$ git push pre-production master$ heroku run rake dbmigrate --remote pre-production$ git push production master
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Future
Future
bull Asset pipeline
Future
bull Asset pipeline
bull Mysql tooling
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
bull Can databases
Future
Future
bull Hot compatibility beyond zero downtime deploys
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -USR2 63480
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Within the stack
Within the stackbull Easy to setup and deploy
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
bull Doesnrsquot address availability issues
Platform
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 1s bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 5s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 1s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ heroku labsenable preboot
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 starting for 5s bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 up for 1s bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 5s bundle exec thin start -p $PORT
BETA
Heroku
bull Migrations
Heroku
bull Migrations
$ git push pre-production master$ heroku run rake dbmigrate --remote pre-production$ git push production master
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Future
Future
bull Asset pipeline
Future
bull Asset pipeline
bull Mysql tooling
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
bull Can databases
Future
Future
bull Hot compatibility beyond zero downtime deploys
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Within the stack
Within the stackbull Easy to setup and deploy
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
bull Doesnrsquot address availability issues
Platform
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 1s bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 5s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 1s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ heroku labsenable preboot
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 starting for 5s bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 up for 1s bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 5s bundle exec thin start -p $PORT
BETA
Heroku
bull Migrations
Heroku
bull Migrations
$ git push pre-production master$ heroku run rake dbmigrate --remote pre-production$ git push production master
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Future
Future
bull Asset pipeline
Future
bull Asset pipeline
bull Mysql tooling
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
bull Can databases
Future
Future
bull Hot compatibility beyond zero downtime deploys
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
Unicorn
63480 00 14 2549796 unicorn_rails master (old) -c configunicornrb 63489 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63490 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63491 00 02 2549796 unicorn_rails worker[2] -c configunicornrb 63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Within the stack
Within the stackbull Easy to setup and deploy
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
bull Doesnrsquot address availability issues
Platform
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 1s bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 5s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 1s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ heroku labsenable preboot
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 starting for 5s bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 up for 1s bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 5s bundle exec thin start -p $PORT
BETA
Heroku
bull Migrations
Heroku
bull Migrations
$ git push pre-production master$ heroku run rake dbmigrate --remote pre-production$ git push production master
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Future
Future
bull Asset pipeline
Future
bull Asset pipeline
bull Mysql tooling
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
bull Can databases
Future
Future
bull Hot compatibility beyond zero downtime deploys
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
$ kill -QUIT 63480
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Within the stack
Within the stackbull Easy to setup and deploy
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
bull Doesnrsquot address availability issues
Platform
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 1s bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 5s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 1s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ heroku labsenable preboot
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 starting for 5s bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 up for 1s bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 5s bundle exec thin start -p $PORT
BETA
Heroku
bull Migrations
Heroku
bull Migrations
$ git push pre-production master$ heroku run rake dbmigrate --remote pre-production$ git push production master
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Future
Future
bull Asset pipeline
Future
bull Asset pipeline
bull Mysql tooling
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
bull Can databases
Future
Future
bull Hot compatibility beyond zero downtime deploys
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
Unicorn
63521 00 14 2549796 unicorn_rails master -c configunicornrb 63535 00 02 2549796 unicorn_rails worker[0] -c configunicornrb 63536 00 02 2549796 unicorn_rails worker[1] -c configunicornrb 63537 00 02 2549796 unicorn_rails worker[2] -c configunicornrb
Within the stack
Within the stackbull Easy to setup and deploy
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
bull Doesnrsquot address availability issues
Platform
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 1s bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 5s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 1s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ heroku labsenable preboot
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 starting for 5s bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 up for 1s bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 5s bundle exec thin start -p $PORT
BETA
Heroku
bull Migrations
Heroku
bull Migrations
$ git push pre-production master$ heroku run rake dbmigrate --remote pre-production$ git push production master
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Future
Future
bull Asset pipeline
Future
bull Asset pipeline
bull Mysql tooling
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
bull Can databases
Future
Future
bull Hot compatibility beyond zero downtime deploys
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
Within the stack
Within the stackbull Easy to setup and deploy
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
bull Doesnrsquot address availability issues
Platform
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 1s bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 5s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 1s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ heroku labsenable preboot
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 starting for 5s bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 up for 1s bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 5s bundle exec thin start -p $PORT
BETA
Heroku
bull Migrations
Heroku
bull Migrations
$ git push pre-production master$ heroku run rake dbmigrate --remote pre-production$ git push production master
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Future
Future
bull Asset pipeline
Future
bull Asset pipeline
bull Mysql tooling
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
bull Can databases
Future
Future
bull Hot compatibility beyond zero downtime deploys
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
Within the stackbull Easy to setup and deploy
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
bull Doesnrsquot address availability issues
Platform
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 1s bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 5s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 1s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ heroku labsenable preboot
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 starting for 5s bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 up for 1s bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 5s bundle exec thin start -p $PORT
BETA
Heroku
bull Migrations
Heroku
bull Migrations
$ git push pre-production master$ heroku run rake dbmigrate --remote pre-production$ git push production master
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Future
Future
bull Asset pipeline
Future
bull Asset pipeline
bull Mysql tooling
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
bull Can databases
Future
Future
bull Hot compatibility beyond zero downtime deploys
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
bull Doesnrsquot address availability issues
Platform
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 1s bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 5s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 1s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ heroku labsenable preboot
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 starting for 5s bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 up for 1s bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 5s bundle exec thin start -p $PORT
BETA
Heroku
bull Migrations
Heroku
bull Migrations
$ git push pre-production master$ heroku run rake dbmigrate --remote pre-production$ git push production master
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Future
Future
bull Asset pipeline
Future
bull Asset pipeline
bull Mysql tooling
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
bull Can databases
Future
Future
bull Hot compatibility beyond zero downtime deploys
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
bull Doesnrsquot address availability issues
Platform
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 1s bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 5s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 1s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ heroku labsenable preboot
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 starting for 5s bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 up for 1s bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 5s bundle exec thin start -p $PORT
BETA
Heroku
bull Migrations
Heroku
bull Migrations
$ git push pre-production master$ heroku run rake dbmigrate --remote pre-production$ git push production master
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Future
Future
bull Asset pipeline
Future
bull Asset pipeline
bull Mysql tooling
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
bull Can databases
Future
Future
bull Hot compatibility beyond zero downtime deploys
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
Within the stackbull Easy to setup and deploy
bull Optimal performance to cycle the server
bull Changes your application stack
bull Doesnrsquot address availability issues
Platform
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 1s bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 5s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 1s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ heroku labsenable preboot
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 starting for 5s bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 up for 1s bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 5s bundle exec thin start -p $PORT
BETA
Heroku
bull Migrations
Heroku
bull Migrations
$ git push pre-production master$ heroku run rake dbmigrate --remote pre-production$ git push production master
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Future
Future
bull Asset pipeline
Future
bull Asset pipeline
bull Mysql tooling
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
bull Can databases
Future
Future
bull Hot compatibility beyond zero downtime deploys
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
Platform
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 1s bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 5s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 1s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ heroku labsenable preboot
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 starting for 5s bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 up for 1s bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 5s bundle exec thin start -p $PORT
BETA
Heroku
bull Migrations
Heroku
bull Migrations
$ git push pre-production master$ heroku run rake dbmigrate --remote pre-production$ git push production master
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Future
Future
bull Asset pipeline
Future
bull Asset pipeline
bull Mysql tooling
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
bull Can databases
Future
Future
bull Hot compatibility beyond zero downtime deploys
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 1s bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 5s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 1s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ heroku labsenable preboot
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 starting for 5s bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 up for 1s bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 5s bundle exec thin start -p $PORT
BETA
Heroku
bull Migrations
Heroku
bull Migrations
$ git push pre-production master$ heroku run rake dbmigrate --remote pre-production$ git push production master
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Future
Future
bull Asset pipeline
Future
bull Asset pipeline
bull Mysql tooling
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
bull Can databases
Future
Future
bull Hot compatibility beyond zero downtime deploys
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 1s bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 5s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 1s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ heroku labsenable preboot
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 starting for 5s bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 up for 1s bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 5s bundle exec thin start -p $PORT
BETA
Heroku
bull Migrations
Heroku
bull Migrations
$ git push pre-production master$ heroku run rake dbmigrate --remote pre-production$ git push production master
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Future
Future
bull Asset pipeline
Future
bull Asset pipeline
bull Mysql tooling
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
bull Can databases
Future
Future
bull Hot compatibility beyond zero downtime deploys
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 1s bundle exec thin start -p $PORT
$ git push heroku master
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 5s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 1s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ heroku labsenable preboot
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 starting for 5s bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 up for 1s bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 5s bundle exec thin start -p $PORT
BETA
Heroku
bull Migrations
Heroku
bull Migrations
$ git push pre-production master$ heroku run rake dbmigrate --remote pre-production$ git push production master
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Future
Future
bull Asset pipeline
Future
bull Asset pipeline
bull Mysql tooling
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
bull Can databases
Future
Future
bull Hot compatibility beyond zero downtime deploys
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
Heroku
Process State Command ------- --------------- -------------------------------web1 starting for 5s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 1s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ heroku labsenable preboot
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 starting for 5s bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 up for 1s bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 5s bundle exec thin start -p $PORT
BETA
Heroku
bull Migrations
Heroku
bull Migrations
$ git push pre-production master$ heroku run rake dbmigrate --remote pre-production$ git push production master
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Future
Future
bull Asset pipeline
Future
bull Asset pipeline
bull Mysql tooling
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
bull Can databases
Future
Future
bull Hot compatibility beyond zero downtime deploys
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 1s bundle exec thin start -p $PORT
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ heroku labsenable preboot
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 starting for 5s bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 up for 1s bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 5s bundle exec thin start -p $PORT
BETA
Heroku
bull Migrations
Heroku
bull Migrations
$ git push pre-production master$ heroku run rake dbmigrate --remote pre-production$ git push production master
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Future
Future
bull Asset pipeline
Future
bull Asset pipeline
bull Mysql tooling
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
bull Can databases
Future
Future
bull Hot compatibility beyond zero downtime deploys
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ heroku labsenable preboot
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 starting for 5s bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 up for 1s bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 5s bundle exec thin start -p $PORT
BETA
Heroku
bull Migrations
Heroku
bull Migrations
$ git push pre-production master$ heroku run rake dbmigrate --remote pre-production$ git push production master
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Future
Future
bull Asset pipeline
Future
bull Asset pipeline
bull Mysql tooling
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
bull Can databases
Future
Future
bull Hot compatibility beyond zero downtime deploys
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ heroku labsenable preboot
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 starting for 5s bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 up for 1s bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 5s bundle exec thin start -p $PORT
BETA
Heroku
bull Migrations
Heroku
bull Migrations
$ git push pre-production master$ heroku run rake dbmigrate --remote pre-production$ git push production master
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Future
Future
bull Asset pipeline
Future
bull Asset pipeline
bull Mysql tooling
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
bull Can databases
Future
Future
bull Hot compatibility beyond zero downtime deploys
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 starting for 5s bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 up for 1s bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 5s bundle exec thin start -p $PORT
BETA
Heroku
bull Migrations
Heroku
bull Migrations
$ git push pre-production master$ heroku run rake dbmigrate --remote pre-production$ git push production master
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Future
Future
bull Asset pipeline
Future
bull Asset pipeline
bull Mysql tooling
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
bull Can databases
Future
Future
bull Hot compatibility beyond zero downtime deploys
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 starting for 5s bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 up for 1s bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 5s bundle exec thin start -p $PORT
BETA
Heroku
bull Migrations
Heroku
bull Migrations
$ git push pre-production master$ heroku run rake dbmigrate --remote pre-production$ git push production master
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Future
Future
bull Asset pipeline
Future
bull Asset pipeline
bull Mysql tooling
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
bull Can databases
Future
Future
bull Hot compatibility beyond zero downtime deploys
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 starting for 5s bundle exec thin start -p $PORT
$ git push heroku master
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 up for 1s bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 5s bundle exec thin start -p $PORT
BETA
Heroku
bull Migrations
Heroku
bull Migrations
$ git push pre-production master$ heroku run rake dbmigrate --remote pre-production$ git push production master
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Future
Future
bull Asset pipeline
Future
bull Asset pipeline
bull Mysql tooling
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
bull Can databases
Future
Future
bull Hot compatibility beyond zero downtime deploys
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 10h bundle exec thin start -p $PORTweb1 up for 1s bundle exec thin start -p $PORT
BETA
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 5s bundle exec thin start -p $PORT
BETA
Heroku
bull Migrations
Heroku
bull Migrations
$ git push pre-production master$ heroku run rake dbmigrate --remote pre-production$ git push production master
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Future
Future
bull Asset pipeline
Future
bull Asset pipeline
bull Mysql tooling
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
bull Can databases
Future
Future
bull Hot compatibility beyond zero downtime deploys
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
Heroku
Process State Command ------- --------------- -------------------------------web1 up for 5s bundle exec thin start -p $PORT
BETA
Heroku
bull Migrations
Heroku
bull Migrations
$ git push pre-production master$ heroku run rake dbmigrate --remote pre-production$ git push production master
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Future
Future
bull Asset pipeline
Future
bull Asset pipeline
bull Mysql tooling
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
bull Can databases
Future
Future
bull Hot compatibility beyond zero downtime deploys
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
Heroku
bull Migrations
Heroku
bull Migrations
$ git push pre-production master$ heroku run rake dbmigrate --remote pre-production$ git push production master
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Future
Future
bull Asset pipeline
Future
bull Asset pipeline
bull Mysql tooling
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
bull Can databases
Future
Future
bull Hot compatibility beyond zero downtime deploys
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
Heroku
bull Migrations
$ git push pre-production master$ heroku run rake dbmigrate --remote pre-production$ git push production master
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Future
Future
bull Asset pipeline
Future
bull Asset pipeline
bull Mysql tooling
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
bull Can databases
Future
Future
bull Hot compatibility beyond zero downtime deploys
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
Agenda
bull Introduction
bull Development practices
bull Production environment setup
bull Future
bull Conclusion
Future
Future
bull Asset pipeline
Future
bull Asset pipeline
bull Mysql tooling
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
bull Can databases
Future
Future
bull Hot compatibility beyond zero downtime deploys
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
Future
Future
bull Asset pipeline
Future
bull Asset pipeline
bull Mysql tooling
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
bull Can databases
Future
Future
bull Hot compatibility beyond zero downtime deploys
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
Future
bull Asset pipeline
Future
bull Asset pipeline
bull Mysql tooling
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
bull Can databases
Future
Future
bull Hot compatibility beyond zero downtime deploys
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
Future
bull Asset pipeline
bull Mysql tooling
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
bull Can databases
Future
Future
bull Hot compatibility beyond zero downtime deploys
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
bull Can databases
Future
Future
bull Hot compatibility beyond zero downtime deploys
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
Future
bull Asset pipeline
bull Mysql tooling
bull Can Rails help making the development of hot compatible applications easier
bull Can databases
Future
Future
bull Hot compatibility beyond zero downtime deploys
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
Future
Future
bull Hot compatibility beyond zero downtime deploys
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
Future
bull Hot compatibility beyond zero downtime deploys
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
Conclusion
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
Conclusion
bull Zero downtime is a pain in the ass
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
Conclusion
bull Zero downtime is a pain in the ass
bull Hot compatibility to the rescue
bull Make sure your production env is ready
Thank youhttppedroherokucom
ped
Thank youhttppedroherokucom
ped
