View
3.020
Download
0
Category
Preview:
Citation preview
File Upload 2015@choonkeat
choonkeat.com jollygoodcode.com
How’s that predefined styles doing for you?
has_attached_file :asset, styles: { thumb: "100x100#"}
How’s that predefined styles doing for you?
has_attached_file :asset, styles: { thumb: "100x100#", medium: "320x>"}
How’s that predefined styles doing for you?
has_attached_file :asset, styles: { thumb: "100x100#", medium: "320x>", large: "1024x>"}
How’s that predefined styles doing for you?
has_attached_file :asset, styles: { thumb: "100x100#", medium: "320x>", big: "640x>", large: "1024x>"}
Names starting to lose meaning…
How’s that predefined styles doing for you?
has_attached_file :asset, styles: { thumb: "100x100#", thumb2x: "200x200#", medium: "320x>", medium2x: "640x>", big: "640x>", big2x: "1280x>", large: "1024x>", large2x: "2048x>"}
How’s that predefined styles doing for you?
has_attached_file :asset, styles: { thumb: "100x100#", thumb2x: "200x200#", medium: "320x>", medium2x: "640x>", big: "640x>", big2x: "1280x>", large: "1024x>", large2x: "2048x>"}
Reprocess all the production files, each time, we make changes.
404 while rake runs or do at midnight?
How’s that transformation juggling doing for you?
How’s that transformation juggling doing for you?
class MyUploader < CarrierWave::Uploader::Base version :thumb do process resize_to_fill: [280, 280] end
version :small_thumb, from_version: :thumb do process resize_to_fill: [20, 20] endend
How’s that transformation juggling doing for you?
class MyUploader < CarrierWave::Uploader::Base version :thumb do process resize_to_fill: [280, 280] end
version :small_thumb, from_version: :thumb do process resize_to_fill: [20, 20] endend
Did your users wait in the foreground or background?
How’s that file path config doing for you?
How’s that file path config doing for you?
class Avatar < ActiveRecord::Base has_attached_file :image, url: '/system/:class/:attachment/:id/:hash-:style.:extension', hash_secret: Rails.application.secrets.paperclipend
How’s that file path config doing for you?
class Avatar < ActiveRecord::Base has_attached_file :image, url: '/system/:class/:attachment/:id/:hash-:style.:extension', hash_secret: Rails.application.secrets.paperclipend
You sure this is the format?
Or will they need to change?
How’s that file path config doing for you?
class Avatar < ActiveRecord::Base has_attached_file :image, url: '/system/:class/:attachment/:id/:hash-:style.:extension', hash_secret: Rails.application.secrets.paperclipend
How’s that file path config doing for you?
class Avatar < ActiveRecord::Base has_attached_file :image, url: '/system/:class/:attachment/:id/:hash-:style.:extension', hash_secret: Rails.application.secrets.paperclipend
How’s that file path config doing for you?
CarrierWave.configure do |config| config.permissions = 0666 config.directory_permissions = 0777 config.storage = :fileend
How’s that file path config doing for you?
CarrierWave.configure do |config| config.permissions = 0666 config.directory_permissions = 0777 config.storage = :fileend
Did you configure your MySQL data file in your app too?
How’s that file path config doing for you?
class Avatar < ActiveRecord::Base self.table = { name: "avatars", data: "/var/lib/mysql/data/avatars.MYD", index: "/var/lib/mysql/data/avatars.MYI" }end
Did you configure your MySQL data file in your app too?
How’s that form validation error dance doing for you?
How’s that form validation error dance doing for you?
• User chooses a file
• Submit & wait for file to upload ⌛…
• Validation error: “Username is already taken!”
• Re-render form
How’s that form validation error dance doing for you?
• User chooses a file
• Submit & wait for file to upload ⌛…
• Validation error: “Username is already taken!”
• Re-render form
Where dat file go?
How’s that form validation error dance doing for you?
http://stackoverflow.com/questions/5198602/not-losing-paperclip-attachment-when-model-cannot-be-saved-due-to-validation-err
Closed: Won’t Fix
How’s that form validation error dance doing for you?
http://stackoverflow.com/questions/5198602/not-losing-paperclip-attachment-when-model-cannot-be-saved-due-to-validation-err
How’s that form validation error dance doing for you?
http://stackoverflow.com/questions/5198602/not-losing-paperclip-attachment-when-model-cannot-be-saved-due-to-validation-err
Answer: use CarrierWave
How’s that form validation error dance doing for you?
How’s that form validation error dance doing for you?
“Is there a standard approach? This seems like a very common use case.”
How’s the schema pollution doing for you?
How’s the schema pollution doing for you?
class StoreMetadata < ActiveRecord::Migration def change add_column :users, :profile_image_filename, :string add_column :users, :profile_image_size, :integer add_column :users, :profile_image_content_type, :string endend
How’s the schema pollution doing for you?
class StoreMetadata < ActiveRecord::Migration def change add_column :users, :profile_image_filename, :string add_column :users, :profile_image_size, :integer add_column :users, :profile_image_content_type, :string endend
How’s that multiple files doing for you?
How’s that multiple files doing for you?
class Post < ActiveRecord::Base has_many :images, dependent: :destroyend
How’s that multiple files doing for you?
class Post < ActiveRecord::Base has_many :images, dependent: :destroyend
class Image < ActiveRecord::Base belongs_to :post attachment :fileend
How’s that multiple files doing for you?
class Post < ActiveRecord::Base has_many :images, dependent: :destroyend
class Image < ActiveRecord::Base belongs_to :post attachment :fileend
How’s that multiple files doing for you?
class Post < ActiveRecord::Base has_many :images, dependent: :destroyend
class Image < ActiveRecord::Base belongs_to :post attachment :fileend
Is this what you want or just what you’re accustomed to?
How’s Amazon Lambda doing for you?
• User chooses a file
• Submit & wait for file to upload ⌛…
• Success!
• Render page with thumbnail…
How many thumbnails - 404?
How’s Amazon Lambda doing for you?
• User chooses a file
• Submit & wait for file to upload ⌛…
• Success!
• Render page with thumbnail…
Direct upload to AWS?
Cancel form submit - delete files & thumbnails?
Deep integration & assumption
How’s Dragonfly doing for you?
http://markevans.github.io/dragonfly/
http://thecooperreview.com/10-tricks-appear-smart-meetings/
10 Tricks to Appear Smart During Meetings
Take a step back
Take a step back• We want to store a bunch of attributes in a model
• e.g. Title, Body, Tags, Photo
Take a step back{photo}{title}{body}
Take a step back<img src={photo}><h1>{title}</h1>{body}
Take a step back• Why should photo be a disproportionately
complicated attribute in my Article model?
• stored file path
• conversion
• background job
• aws config
• clean up on delete
Take a step back• Why should photo be a disproportionately
complicated attribute in my Article model?
• stored file path
• conversion
• background job
• validation error dance
• aws config
Take a step back• Frankly photo_url is best; least intrusive
Take a step back• Frankly photo_url is best; least intrusive
• Problems
• Remote url 404? (not exclusive to your app)
• Asking users to give us a URL is a hard sell
• Need to render other sizes
• Filter by meta data
Take a step back• Frankly photo_url is best; least intrusive
• Problems
• Remote url 404? (not exclusive to your app)
• Asking users to give us a URL is a hard sell
• Need to render other sizes
• Filter by meta data
Take a step back• Frankly photo_url is best; least intrusive
• Problems
• Remote url 404? (not exclusive to your app)
• Asking users to give us a URL is a hard sell
• Need to render other sizes
• Filter by meta data
Take a step back• Frankly photo_url is best; least intrusive
• Solutions
• Exclusive server for your app
• Upload to that server
• On-the-fly resize based on URL
• Store url with meta data: photo_json instead?
Just add server
• PostgreSQL, MySQL
• Redis
• Memcached
• SMTP server (Mail)
You are already
Generic server to do specialised work
Not specific to your business logic
• Not a new pattern
• Mostly commercial services
• Maybe it has to be Free & Open Source to become a default pattern
Image server
Want• Move the “concern” out of my app
• photo is a regular attribute
• configure my app & forget it exist
Want• Move the “concern” out of my app
• photo is a regular attribute
• configure my app & forget it exist What would my app look
like in a better world?
My app: Bare minimumcreate_table "users" do |t| t.string "name" t.json "avatar" t.json "photos"end
My app: Bare minimumcreate_table "users" do |t| t.string "name" t.json "avatar" t.json "photos" # multiple files in a columnend
My app: Bare minimumImage serverRails appBrowser
{ avatar: #<File..> }
My app: Bare minimumImage serverRails appBrowser
{ avatar: #<File..> }
{“path”:“x.jpg”, “geometry”:“200x600”}
#<File..>
My app: Bare minimumImage serverRails appBrowser
{“path”:“x.jpg”, “geometry”:“200x600”}
#<File..>
user.avatar={“path”: “x.jpg”…}user.save
<img src=“x.jpg”>
{ avatar: #<File..> }
My app: Bare minimumImage serverRails appBrowser
{“path”:“x.jpg”, “geometry”:“200x600”}
<img src=“x.jpg”>
#<File..>
GET x.jpg
#<File..>
user.avatar={“path”: “x.jpg”…}user.save
{ avatar: #<File..> }
My app: Bare minimum• Browser render <file> field; regular form submit
• Receive binary param, uploads to attache server and stores the json response instead
• Your app render <img src> requesting for image in certain size, e.g. http://example/200x/file.png
Image serverRails appBrowser#<File..>
{“path”:“x.jpg”, “geometry”:“200x600”}
Progressive Enhancement
Image serverRails appBrowser#<File..>
{“path”:“x.jpg”, “geometry”:“200x600”}
Progressive Enhancement
{ avatar: {“path”:“x.jpg”, … }
{ avatar: {“path”:“x.jpg”, … }
Image serverRails appBrowser#<File..>
{“path”:“x.jpg”, “geometry”:“200x600”}
user.avatar={“path”: “x.jpg”…}user.save
<img src=“x.jpg”>
Progressive Enhancement
• Browser render <file> field; regular form submit
• Receive binary param, uploads to attache server and stores the json response instead
• Your app render <img src> requesting for image in certain size, e.g. http://example/200x/file.png
Progressive Enhancement
• Browser render <file> field; regular form submit
• Receive binary param, uploads to attache server and stores the json response instead
• Your app render <img src> requesting for image in certain size, e.g. http://example/200x/file.png
• JS upload directly to attache server; “Direct upload” in AWS parlance
• No binary in params; receive and store json attribute
Progressive Enhancement
• When after_update & after_destroy removes obsolete file from attache via delete API
• Just use Ruby; just use your framework
• Pre-render multiple sizes
• fetch the urls, server will generate and cache
• Validation
• validating a regular json attribute
How do I…
• Move the “concern” out of my app
• photo is a regular attribute
• configure my app & forget it exist
Want (cont’d)
Want (cont’d)• Move the “concern” out of my app
• photo is a regular attribute
• configure my app & forget it exist
• Separate, standalone server
• Minimal / zero ongoing administration
Want (cont’d)• Move the “concern” out of my app
• photo is a regular attribute
• configure my app & forget it exist
• Separate, standalone server
• Minimal / zero ongoing administration
How does this server work?
Attache File Server• HTTP server with simple APIs
• upload
• download
• delete
• Rack app + ImageMagick
• Go? Node? C++? PHP?
• GraphicsMagick? MyResizer.bash?
• Uploaded files are stored locally
• Resize local file on-the-fly
• configurable pool size to limit concurrent resizing
• Sync upload to cloud storage
• 2 hop problem vs complexity
• Fixed local storage, purge LRU (zero maintenance)
• Spin up new fresh servers anytime…
Attache File Server
• When requested file does not exist locally
• fetch from cloud storage & write locally
• resume operations
Attache File Server
• Remove obsolete file is “best effort”
• If photo delete failed, do you Error 500 or stop the world?
• OCDs can schedule rake task remove dangling files?
Attache File Server
• Caching in production
• Browser → CDN → Varnish → Disk → Cloud
Attache File Server
https://github.com/choonkeat/attache_api
ATTACHE_URL=http://localhost:9292 ATTACHE_SECRET_KEY=topsecretrake
Compatibility Check
tus.io
• Open protocol for resumable uploads built on HTTP
• Perfect for mobile apps
• Rack middleware implemented in choonkeat/attache#10
Responsive Images with Client Hints
http://blog.imgix.com/2015/10/13/next-generation-responsive-images-with-client.html
Dragonfly & refile
Dragonfly & refile
• tldr: we can fuss over implementation, but it is mostly about architecture
Dragonfly & refile• Rack middleware in your Rails app by default
• performing on-the-fly image resize 😱
• Rack standalone end point
• Dragonfly.app - upload, download, delete
• Refile::App - upload, download, delete
• Downloads are unthrottled
Dragonfly & refile• BEFORE: Rails integrate with AWS
• AFTER: Rails integrate with AWS + Rack app
• Maintain identical AWS config in both apps
• Rails app couldn’t shed the “concern”
• Multiple images still require multiple models
refile
class Post < ActiveRecord::Base has_many :images, dependent: :destroy accepts_attachments_for :images, attachment: :fileend
class Image < ActiveRecord::Base belongs_to :post attachment :fileend
refile
class Post < ActiveRecord::Base has_many :images, dependent: :destroy accepts_attachments_for :images, attachment: :fileend
class Image < ActiveRecord::Base belongs_to :post attachment :fileend
“Note it must be possible to persist images given only the associated post and a file. There must not be any other validations or constraints which prevent images from being saved”
i.e. Must be pure dummy wrapper; no validations here
refile download
https://github.com/refile/refile/blob/master/lib/refile/app.rb
get "/:token/:backend/:processor/:id/:filename" do halt 404 unless download_allowed? stream_file processor.call(file)end
refile download
https://github.com/refile/refile/blob/master/lib/refile/app.rb
get "/:token/:backend/:processor/:id/:filename" do halt 404 unless download_allowed? stream_file processor.call(file)end
How many ImageMagick can you run in parallel?
has_many :images?
refile upload
https://github.com/refile/refile/blob/master/lib/refile/app.rb
post "/:backend" do halt 404 unless upload_allowed? tempfile = request.params.fetch("file").fetch(:tempfile) file = backend.upload(tempfile) content_type :json { id: file.id }.to_jsonend
def file file = backend.get(params[:id]) unless file.exists? log_error("Could not find attachment by id: #{params[:id]}") halt 404 end file.downloadend
post "/:backend" do halt 404 unless upload_allowed? tempfile = request.params.fetch("file").fetch(:tempfile) file = backend.upload(tempfile) content_type :json { id: file.id }.to_jsonend
def file file = backend.get(params[:id]) unless file.exists? log_error("Could not find attachment by id: #{params[:id]}") halt 404 end file.downloadend
refile upload
https://github.com/refile/refile/blob/master/lib/refile/app.rb
2 hops problem when uploading and downloading
• 3mb file becomes 6mb each way
• #create and #show becomes 12mb process
• has_many :images?
refile• CarrierWave-styled named processors (e.g. :fill, :thumb) vs
passing through syntax to underlying ImageMagick
• personally prefer leveraging off existing knowledge
• instead of manually configured syntax sugar
• “2 hop problem” however provide higher consistency when running multiple refile servers
• upload-here-download-there problem
• (considering to perform 2-hop upload instead of async)
refile• Can upload to S3 direct and/or upload to refile
• Redundancy interesting, but prefer less concern in Rails app
• Concept of cache and store to manage “dangling file problem” is worth considering
• Download urls are signed-only (this practice should be adopted)
• can partly counter motivation to abuse “dangling file problem” (aka free file hosting, whee!)
Questions?Attache server https://github.com/choonkeat/attache
Gem for Rails https://github.com/choonkeat/attache_rails
Demo https://attache-demo.herokuapp.com/
Recommended