Zero downtime deploys for Rails apps

  • Published on
    15-Jan-2015

  • View
    11.839

  • Download
    2

DESCRIPTION

 

Transcript

<ul><li> 1. zero deploysdowntime for Rails photo by Chris http://www.flickr.com/people/chrisinplymouth/</li></ul><p> 2. class ApparentlyHarmlessMigration &lt; ActiveRecord::Migrationdef self.upremove_column :users, :notesendend 3. class CompletelySafeMigration &lt; ActiveRecord::Migrationdef self.upremove_column :users, :notesenddef self.downadd_column :users, :notes, :textendend 4. class CompletelySafeMigration &lt; ActiveRecord::Migrationdef self.upremove_column :users, :notesenddef self.downadd_column :users, :notes, :textendend 5. PGError: ERROR: column "notes" does not exist 6. INSERT INTO users(email, password, ..., notes) 7. Zero downtime?Challenge accepted 8. Agenda Introduction Development practices Production environment setup Future Conclusion 9. Testable code + TDDTest coverageEasy maintenanceDRYZero downtime deploys Hot compatibility 10. Testable code + TDDTest coverageEasy maintenanceDRYZero downtime deploys Hot compatibility 11. Testable code + TDDTest coverageEasy maintenanceDRYZero downtime deploys Hot compatibility 12. Hot compatibility Run concurrent versions of yourapplication That are compatible between each other 13. Hot compatibilityv1 v2 14. Hot compatibilityv1 v2writes to `notes` drop column `notes` 15. Hot compatibilityv1v1 v2writes to `notes`stop writing to `notes` drop column `notes` 16. Hot compatibilityv1v1 v2writes to `notes`stop writing to `notes` drop column `notes` 17. Hot compatibilityv1v1 v2writes to `notes`stop writing to `notes` drop column `notes` 18. Hot compatibilityv1v1 v2writes to `notes`stop writing to `notes` drop column `notes` 19. Hot compatibilityv1v1v2writes to `notes`stop writing to `notes`drop column `notes` class User def self.columns super.reject { |c| c.name == "notes" } end end 20. Self dependency 21. Self dependencyuser loads form 22. Self dependencynew form is deployed 23. Self dependencyuser submits the form POST /login username=bill@microsoft.com&amp;password=q1w2e3 24. Self dependencyserver blows up 25. Self dependency Make controllers compatible: 26. Self dependency Make controllers compatible:class AuthController &lt; ApplicationControllerprotecteddef filtered_paramsparams.dup.tap do |p|p.merge!(:email =&gt; p.delete(:username))endendend 27. Assets - Rails 2Server 1GET /dashboard200GET /stylesheets/all.js200 28. Assets - Rails 2Server 1 Server 2GET /dashboard200GET /stylesheets/all.js404 29. Assets - Rails 2 https://github.com/ddollar/asset-resource 30. Asset pipeline - Rails 3v1GET /dashboard200GET /assets/application-{v1}.js200 31. Asset pipeline - Rails 3v1 v2GET /dashboard200GET /assets/application-{v1}.js404 32. Asset pipeline - Rails 3 33. Asset pipeline - Rails 3 When updating assets, keep the existingversion around 34. Asset pipeline - Rails 3 When updating assets, keep the existingversion around Were still in need of tooling to helpcleaning up old assets 35. Asset pipeline - Rails 3 When updating assets, keep the existingversion around Were still in need of tooling to helpcleaning up old assetsrake assets:clean:outdated 36. Migration strategies 37. Migration strategies Read-only models 38. Migration strategies Read-only models class Role &lt; ActiveRecord::Base def readonly? true end end 39. Migration strategies Read-only models Ignore cached columns 40. Migration strategies Read-only models Ignore cached columns class User def self.columns super.reject { |c| c.name == "notes" } end end 41. Migration strategies Read-only models Ignore cached columns Know your database 42. Migration strategies Read-only models Ignore cached columns Know your database Beware of O(n) db locks! 43. MyISAM 44. MyISAM 45. InnoDB 46. InnoDB O(n) locks on: Adding, updating or removing columns Adding or removing indexes 47. InnoDB1. Create a new table with the same structure2. Apply changes to new table3. Copy data over4. Acquire lock on original table5. Sync new/updated/deleted rows6. Swap tables7. Release lock 48. InnoDB1. Create a new table with the same structure2. Apply changes to new table3. Copy data over4. Acquire lock on original table5. Sync new/updated/deleted rows6. Swap tables7. Release lock 49. InnoDBhttps://github.com/freels/table_migrator 50. InnoDBhttps://github.com/freels/table_migratorclass AddStuffToMyBigTable &lt; TableMigrationmigrates :userschange_table do |t|t.integer :foo, :null =&gt; false, :default =&gt; 0t.string :bart.index :foo, :name =&gt; index_for_fooendend 51. InnoDBhttps://github.com/freels/table_migrator Requires acts_as_paranoid Pretty old/stale codebase Check forks for Rails 3 compatibility 52. Postgres Table locks on: Adding, updating or removing columns Adding or removing indexes 53. Postgres O(n) locks on: Adding, updating or removing columns Adding or removing indexes 54. Postgres O(n) locks on:constant time if you avoid default values and constraints Adding, updating or removing columns Adding or removing indexes 55. Postgres O(n) locks on: constant time Adding, updating or removing columns Adding or removing indexes 56. Postgres O(n) locks on: Adding, updating or removing columns Adding or removing indexes can be done without any locking 57. Postgres O(n) locks on: Adding, updating or removing columns Adding or removing indexesconstant time 58. Postgres Adding elds with defaults 59. Postgres Adding elds with defaults before_save :assign_defaults def assign_defaults self.admin ||= false end 60. Postgres Adding elds with defaults Create indexes concurrently 61. Postgres Adding elds with defaults Create indexes concurrentlyCREATE INDEX CONCURRENTLY users_email_index ON users(email); 62. Postgres Adding elds with defaults Create indexes concurrently Outside a transaction! 63. Postgres Adding elds with defaults Create indexes concurrently Outside a transaction!class AddIndexToUsers &lt; ActiveRecord::Migrationdef self.upreturn if indexes("users").map(&amp;:name).include?("users_email_index")add_index :users, :email, :name =&gt; "users_email_index"enddef self.downremove_index :users, :name =&gt; "users_email_index"endend 64. What about renaming?v1 v2reads and writes reads and writesto `foo` to `bar` 65. What about renaming?v1v1 v1 v2reads and writes add column `bar` populate `bar` reads and writesto `foo` +where needed to `bar` reads from `foo`writes to both`foo` and `bar` 66. What about renaming? v2v2 v3 reads and writes ignore `foo` drop column `foo` to `bar` 67. NoSQL 68. NoSQL Schema migrations / O(n) locks are justpart of hot compatibility 69. NoSQL Schema migrations / O(n) locks are justpart of hot compatibility You still need to put effort into datamigration 70. Workow 71. Workow Make sure pull requests are addressinghot compatibility 72. Workow Make sure pull requests are addressinghot compatibility Work after code is merged 73. Meanwhile, in production...$ thin -C config/server.yml restartStopping server on 0.0.0.0:5000 ...Sending QUIT signal to process 3463 ...Stopping server on 0.0.0.0:5001 ...Sending QUIT signal to process 3464 ...Stopping server on 0.0.0.0:5002 ...Sending QUIT signal to process 3465 ...Starting server on 0.0.0.0:5000 ...Starting server on 0.0.0.0:5001 ...Starting server on 0.0.0.0:5002 ... 74. Meanwhile, in production...$ thin -C config/server.yml restartStopping server on 0.0.0.0:5000 ...Sending QUIT signal to process 3463 ...Stopping server on 0.0.0.0:5001 ...Sending QUIT signal to process 3464 ...Stopping server on 0.0.0.0:5002 ...Sending QUIT signal to process 3465 ...Starting server on 0.0.0.0:5000 ...Starting server on 0.0.0.0:5001 ...Starting server on 0.0.0.0:5002 ... 75. Meanwhile, in production...$ thin -C config/server.yml restartStopping server on 0.0.0.0:5000 ...Sending QUIT signal to process 3463 ...Stopping server on 0.0.0.0:5001 ...Sending QUIT signal to process 3464 ...Stopping server on 0.0.0.0:5002 ...Sending QUIT signal to process 3465 ...Starting server on 0.0.0.0:5000 ...Starting server on 0.0.0.0:5001 ...Starting server on 0.0.0.0:5002 ... 76. Meanwhile, in production...$ thin -C config/server.yml restartStopping server on 0.0.0.0:5000 ...Sending QUIT signal to process 3463 ...Stopping server on 0.0.0.0:5001 ...Sending QUIT signal to process 3464 ...Stopping server on 0.0.0.0:5002 ...Sending QUIT signal to process 3465 ...Starting server on 0.0.0.0:5000 ...Starting server on 0.0.0.0:5001 ...Starting server on 0.0.0.0:5002 ... 77. Meanwhile, in production...$ thin -C config/server.yml restartStopping server on 0.0.0.0:5000 ...Sending QUIT signal to process 3463 ...Stopping server on 0.0.0.0:5001 ...Sending QUIT signal to process 3464 ...Stopping server on 0.0.0.0:5002 ...Sending QUIT signal to process 3465 ...Starting server on 0.0.0.0:5000 ...Starting server on 0.0.0.0:5001 ...Starting server on 0.0.0.0:5002 ... 78. Agenda Introduction Development practices Production environment setup Future Conclusion 79. Production 80. Production Run multiple server processes 81. Production Run multiple server processes Cycle them individually, or in batches 82. Production Run multiple server processes Cycle them individually, or in batches Watch out for capacity! 83. Production Run multiple server processes Cycle them individually, or in batches Watch out for capacity! Cycle them gracefully 84. Cycling techniques Above the stack Within the stack Platform 85. Above the stack Use your favorite web server Setup a load balancer above them, routearound servers that are cycling 86. Above the stack PROTIP: default setup wont work Congure load balancer to poll for yourapplication health:class HealthController &lt; ApplicationControllerdef indexif File.exists?(Rails.root.join("MAINTENANCE"))render :text =&gt; "Forget me!", :status =&gt; 503elserender :text =&gt; "Im alive!"endendend 87. Cycling + Load balancer Server Load balancer touch MAINTENANCE bounce servers remove server from rotation rm MAINTENANCEnew servers come online add server to rotation 88. HAProxyoption httpchk GET /healthlisten myapp :9000server myapp-1 1.0.0.1:5000 maxconn 1 check inter 2000 fall 1 rise 1server myapp-2 1.0.0.2:5000 maxconn 1 check inter 2000 fall 1 rise 1 89. HAProxytell HAProxy where to check for the healthoption httpchk GET /healthlisten myapp :9000server myapp-1 1.0.0.1:5000 maxconn 1 check inter 2000 fall 1 rise 1server myapp-2 1.0.0.2:5000 maxconn 1 check inter 2000 fall 1 rise 1 90. HAProxyoption httpchk GET /healthlisten myapp :9000server myapp-1 1.0.0.1:5000 maxconn 1 check inter 2000 fall 1 rise 1server myapp-2 1.0.0.2:5000 maxconn 1 check inter 2000 fall 1 rise 1 do not let your web server queue requests 91. HAProxyoption httpchk GET /healthlisten myapp :9000server myapp-1 1.0.0.1:5000 maxconn 1 check inter 2000 fall 1 rise 1server myapp-2 1.0.0.2:5000 maxconn 1 check inter 2000 fall 1 rise 1 tell HAProxy to run the health check specied above 92. HAProxyoption httpchk GET /healthlisten myapp :9000server myapp-1 1.0.0.1:5000 maxconn 1 check inter 2000 fall 1 rise 1server myapp-2 1.0.0.2:5000 maxconn 1 check inter 2000 fall 1 rise 1 every 2s 93. HAProxyoption httpchk GET /healthlisten myapp :9000server myapp-1 1.0.0.1:5000 maxconn 1 check inter 2000 fall 1 rise 1server myapp-2 1.0.0.2:5000 maxconn 1 check inter 2000 fall 1 rise 1stop routing to it once the health check failed once 94. HAProxyoption httpchk GET /healthlisten myapp :9000server myapp-1 1.0.0.1:5000 maxconn 1 check inter 2000 fall 1 rise 1server myapp-2 1.0.0.2:5000 maxconn 1 check inter 2000 fall 1 rise 1 and bring it back once it worked once 95. ELB$ elb-configure-healthcheck MyLoadBalancer --target "HTTP:5000/health" --interval 2 --unhealthy-threshold 1 --healthy-threshold 1 96. Above the stack 97. Above the stack Minimal impact 98. Above the stack Minimal impact Extended availabilitycomes for free 99. Above the stack Minimal impact Complicated setup Extended availabilitycomes for free 100. Above the stack Minimal impact Complicated setup Extended availability Another component youcomes for freehave to track and maintain 101. Within the stack 102. Within the stack 103. Unicorn Replaces your web server Bounces gracefully 104. Unicorn63480 0.0 1.4 2549796 unicorn_rails master -c config/unicorn.rb63489 0.0 0.2 2549796 unicorn_rails worker[0] -c config/unicorn.rb63490 0.0 0.2 2549796 unicorn_rails worker[1] -c config/unicorn.rb63491 0.0 0.2 2549796 unicorn_rails worker[2] -c config/unicorn.rb 105. Unicorn$ kill -USR2 6348063480 0.0 1.4 2549796 unicorn_rails master -c config/unicorn.rb63489 0.0 0.2 2549796 unicorn_rails worker[0] -c config/unicorn.rb63490 0.0 0.2 2549796 unicorn_rails worker[1] -c config/unicorn.rb63491 0.0 0.2 2549796 unicorn_rails worker[2] -c config/unicorn.rb 106. Unicorn$ kill -USR2 6348063480 0.0 1.4 2549796 unicorn_rails master (old) -c config/unicorn.rb63489 0.0 0.2 2549796 unicorn_rails worker[0] -c config/unicorn.rb63490 0.0 0.2 2549796 unicorn_rails worker[1] -c config/unicorn.rb63491 0.0 0.2 2549796 unicorn_rails worker[2] -c config/unicorn.rb63521 0.0 1.4 2549796 unicorn_rails master -c config/unicorn.rb63535 0.0 0.2 2549796 unicorn_rails worker[0] -c config/unicorn.rb63536 0.0 0.2 2549796 unicorn_rails worker[1] -c config/unicorn.rb63537 0.0 0.2 2549796 unicorn_rails worker[2] -c config/unicorn.rb 107. Unicorn63480 0.0 1.4 2549796 unicorn_rails master (old) -c config/unicorn.rb63489 0.0 0.2 2549796 unicorn_rails worker[0] -c config/unicorn.rb63490 0.0 0.2 2549796 unicorn_rails worker[1] -c config/unicorn.rb63491 0.0 0.2 2549796 unicorn_rails worker[2] -c config/unicorn.rb63521 0.0 1.4 2549796 unicorn_rails master -c config/unicorn.rb63535 0.0 0.2 2549796 unicorn_rails worker[0] -c config/unicorn.rb63536 0.0 0.2 2549796 unicorn_rails worker[1] -c config/unicorn.rb63537 0.0 0.2 2549796 unicorn_rails worker[2] -c config/unicorn.rb 108. Unicorn$ kill -QUIT 6348063480 0.0 1.4 2549796 unicorn_rails master (old) -c config/unicorn.rb63489 0.0 0.2 2549796 unicorn_rails worker[0] -c config/unicorn.rb63490 0.0 0.2 2549796 unicorn_rails worker[1] -c config/unicorn.rb63491 0.0 0.2 2549796 unicorn_rails worker[2] -c config/unicorn.rb63521 0.0 1.4 2549796 unicorn_rails master -c config/unicorn.rb63535 0.0 0.2 2549796 unicorn_rails worker[0] -c config/unicorn.rb63536 0.0 0.2 2549796 unicorn_rails worker[1] -c config/unicorn.rb63537 0.0 0.2 2549796 unicorn_rails worker[2] -c config/unicorn.rb 109. Unicorn$ kill -QUIT 6348063521 0.0 1.4 2549796 unicorn_rails master -c config/unicorn.rb63535 0.0 0.2 2549796 unicorn_rails worker[0] -c config/unicorn.rb63536 0.0 0.2 2549796 unicorn_rails worker[1] -c config/unicorn.rb63537 0.0 0.2 2549796 unicorn_rails worker[2] -c config/unicorn.rb 110. Unicorn63521 0.0 1.4 2549796 unicorn_rails master -c config/unicorn.rb63535 0.0 0.2 2549796 unicorn_rails worker[0] -c config/unicorn.rb63536 0.0 0.2 2549796 unicorn_rails worker[1] -c config/unicorn.rb63537 0.0 0.2 2549796 unicorn_rails worker[2] -c config/unicorn.rb 111. Within the stack 112....</p>

Recommended

View more >