File Upload 2015

File Upload 2015@choonkeat

choonkeat.com jollygoodcode.com

https://twitter.com/choonkeat

http://choonkeat.com

http://jollygoodcode.com

How’s that predefined styles doing for you?

has_attached_file :asset, styles: { thumb: "100x100#"}


has_attached_file :asset, styles: { thumb: "100x100#", medium: "320x>"}


has_attached_file :asset, styles: { thumb: "100x100#", medium: "320x>", large: "1024x>"}


has_attached_file :asset, styles: { thumb: "100x100#", medium: "320x>", big: "640x>", large: "1024x>"}

Names starting to lose meaning…


has_attached_file :asset, styles: { thumb: "100x100#", thumb2x: "200x200#", medium: "320x>", medium2x: "640x>", big: "640x>", big2x: "1280x>", large: "1024x>", large2x: "2048x>"}


has_attached_file :asset, styles: { thumb: "100x100#", thumb2x: "200x200#", medium: "320x>", medium2x: "640x>", big: "640x>", big2x: "1280x>", large: "1024x>", large2x: "2048x>"}

Reprocess all the production files, each time, we make changes.

404 while rake runs or do at midnight?

How’s that transformation juggling doing for you?


class MyUploader < CarrierWave::Uploader::Base version :thumb do process resize_to_fill: [280, 280] end

version :small_thumb, from_version: :thumb do process resize_to_fill: [20, 20] endend


class MyUploader < CarrierWave::Uploader::Base version :thumb do process resize_to_fill: [280, 280] end

version :small_thumb, from_version: :thumb do process resize_to_fill: [20, 20] endend

Did your users wait in the foreground or background?

How’s that file path config doing for you?


class Avatar < ActiveRecord::Base has_attached_file :image, url: '/system/:class/:attachment/:id/:hash-:style.:extension', hash_secret: Rails.application.secrets.paperclipend



You sure this is the format?

Or will they need to change?






CarrierWave.configure do |config| config.permissions = 0666 config.directory_permissions = 0777 config.storage = :fileend


CarrierWave.configure do |config| config.permissions = 0666 config.directory_permissions = 0777 config.storage = :fileend

Did you configure your MySQL data file in your app too?


class Avatar < ActiveRecord::Base self.table = { name: "avatars", data: "/var/lib/mysql/data/avatars.MYD", index: "/var/lib/mysql/data/avatars.MYI" }end

Did you configure your MySQL data file in your app too?

How’s that form validation error dance doing for you?


• User chooses a file

• Submit & wait for file to upload ⌛…

• Validation error: “Username is already taken!”

• Re-render form




• Validation error: “Username is already taken!”

• Re-render form

Where dat file go?


http://stackoverflow.com/questions/5198602/not-losing-paperclip-attachment-when-model-cannot-be-saved-due-to-validation-err

Closed: Won’t Fix







Answer: use CarrierWave




“Is there a standard approach? This seems like a very common use case.”

How’s the schema pollution doing for you?


class StoreMetadata < ActiveRecord::Migration def change add_column :users, :profile_image_filename, :string add_column :users, :profile_image_size, :integer add_column :users, :profile_image_content_type, :string endend


class StoreMetadata < ActiveRecord::Migration def change add_column :users, :profile_image_filename, :string add_column :users, :profile_image_size, :integer add_column :users, :profile_image_content_type, :string endend

How’s that multiple files doing for you?


class Post < ActiveRecord::Base has_many :images, dependent: :destroyend



class Image < ActiveRecord::Base belongs_to :post attachment :fileend







Is this what you want or just what you’re accustomed to?

How’s Amazon Lambda doing for you?



• Success!

• Render page with thumbnail…

How many thumbnails - 404?

How’s Amazon Lambda doing for you?



• Success!

• Render page with thumbnail…

Direct upload to AWS?

Cancel form submit - delete files & thumbnails?

Deep integration & assumption

How’s Dragonfly doing for you?

http://markevans.github.io/dragonfly/

http://markevans.github.io/dragonfly/

How’s refile doing for you?

https://github.com/refile/refile

https://github.com/refile/refile

http://thecooperreview.com/10-tricks-appear-smart-meetings/

10 Tricks to Appear Smart During Meetings

http://thecooperreview.com/10-tricks-appear-smart-meetings/

Take a step back

Take a step back• We want to store a bunch of attributes in a model

• e.g. Title, Body, Tags, Photo

Take a step back{photo}{title}{body}

Take a step back<img src={photo}><h1>{title}</h1>{body}

Take a step back• Why should photo be a disproportionately

complicated attribute in my Article model?

• stored file path

• conversion

• background job

• aws config

• clean up on delete

Take a step back• Why should photo be a disproportionately

complicated attribute in my Article model?

• stored file path

• conversion

• background job

• validation error dance

• aws config

Take a step back• Frankly photo_url is best; least intrusive


• Problems

• Remote url 404? (not exclusive to your app)

• Asking users to give us a URL is a hard sell

• Need to render other sizes

• Filter by meta data


• Problems






• Problems






• Solutions

• Exclusive server for your app

• Upload to that server

• On-the-fly resize based on URL

• Store url with meta data: photo_json instead?

Just add server

• PostgreSQL, MySQL

• Redis

• Memcached

• SMTP server (Mail)

You are already

Generic server to do specialised work

Not specific to your business logic

• Not a new pattern

• Mostly commercial services

• Maybe it has to be Free & Open Source to become a default pattern

Image server

Want• Move the “concern” out of my app

• photo is a regular attribute

• configure my app & forget it exist

Want• Move the “concern” out of my app


• configure my app & forget it exist What would my app look

like in a better world?

My app: Bare minimumcreate_table "users" do |t| t.string "name" t.json "avatar" t.json "photos"end

My app: Bare minimumcreate_table "users" do |t| t.string "name" t.json "avatar" t.json "photos" # multiple files in a columnend

My app: Bare minimumImage serverRails appBrowser

{ avatar: #<File..> }



{“path”:“x.jpg”, “geometry”:“200x600”}

#<File..>



#<File..>

user.avatar={“path”: “x.jpg”…}user.save

<img src=“x.jpg”>





#<File..>

GET x.jpg

#<File..>



My app: Bare minimum• Browser render <file> field; regular form submit

• Receive binary param, uploads to attache server and stores the json response instead

• Your app render <img src> requesting for image in certain size, e.g. http://example/200x/file.png

Image serverRails appBrowser#<File..>


Progressive Enhancement




{ avatar: {“path”:“x.jpg”, … }

{ avatar: {“path”:“x.jpg”, … }






• Browser render <file> field; regular form submit




• Browser render <file> field; regular form submit



• JS upload directly to attache server; “Direct upload” in AWS parlance

• No binary in params; receive and store json attribute


• When after_update & after_destroy removes obsolete file from attache via delete API

• Just use Ruby; just use your framework

• Pre-render multiple sizes

• fetch the urls, server will generate and cache

• Validation

• validating a regular json attribute

How do I…

• Move the “concern” out of my app



Want (cont’d)

Want (cont’d)• Move the “concern” out of my app



• Separate, standalone server

• Minimal / zero ongoing administration

Want (cont’d)• Move the “concern” out of my app



• Separate, standalone server

• Minimal / zero ongoing administration

How does this server work?

Attache File Server• HTTP server with simple APIs

• upload

• download

• delete

• Rack app + ImageMagick

• Go? Node? C++? PHP?

• GraphicsMagick? MyResizer.bash?

• Uploaded files are stored locally

• Resize local file on-the-fly

• configurable pool size to limit concurrent resizing

• Sync upload to cloud storage

• 2 hop problem vs complexity

• Fixed local storage, purge LRU (zero maintenance)

• Spin up new fresh servers anytime…

Attache File Server

• When requested file does not exist locally

• fetch from cloud storage & write locally

• resume operations

Attache File Server

• Remove obsolete file is “best effort”

• If photo delete failed, do you Error 500 or stop the world?

• OCDs can schedule rake task remove dangling files?

Attache File Server

• Caching in production

• Browser → CDN → Varnish → Disk → Cloud

Attache File Server

Demohttps://attache-demo.herokuapp.com/

https://attache-demo.herokuapp.com/

https://github.com/choonkeat/attache_api

ATTACHE_URL=http://localhost:9292 ATTACHE_SECRET_KEY=topsecretrake

Compatibility Check

https://github.com/choonkeat/attache_api

tus.io

• Open protocol for resumable uploads built on HTTP

• Perfect for mobile apps

• Rack middleware implemented in choonkeat/attache#10

https://github.com/choonkeat/attache/pull/10

Responsive Images with Client Hints

http://blog.imgix.com/2015/10/13/next-generation-responsive-images-with-client.html

http://blog.imgix.com/2015/10/13/next-generation-responsive-images-with-client.html

SmartCrop

https://github.com/jwagner/smartcrop.js/

https://github.com/jwagner/smartcrop.js/

Dragonfly & refile

Dragonfly & refile

• tldr: we can fuss over implementation, but it is mostly about architecture

Dragonfly & refile• Rack middleware in your Rails app by default

• performing on-the-fly image resize 😱

• Rack standalone end point

• Dragonfly.app - upload, download, delete

• Refile::App - upload, download, delete

• Downloads are unthrottled

Dragonfly & refile• BEFORE: Rails integrate with AWS

• AFTER: Rails integrate with AWS + Rack app

• Maintain identical AWS config in both apps

• Rails app couldn’t shed the “concern”

• Multiple images still require multiple models

refile

class Post < ActiveRecord::Base has_many :images, dependent: :destroy accepts_attachments_for :images, attachment: :fileend


refile

class Post < ActiveRecord::Base has_many :images, dependent: :destroy accepts_attachments_for :images, attachment: :fileend


“Note it must be possible to persist images given only the associated post and a file. There must not be any other validations or constraints which prevent images from being saved”

i.e. Must be pure dummy wrapper; no validations here

refile download

https://github.com/refile/refile/blob/master/lib/refile/app.rb

get "/:token/:backend/:processor/:id/:filename" do halt 404 unless download_allowed? stream_file processor.call(file)end


refile download


get "/:token/:backend/:processor/:id/:filename" do halt 404 unless download_allowed? stream_file processor.call(file)end

How many ImageMagick can you run in parallel?

has_many :images?


refile upload


post "/:backend" do halt 404 unless upload_allowed? tempfile = request.params.fetch("file").fetch(:tempfile) file = backend.upload(tempfile) content_type :json { id: file.id }.to_jsonend

def file file = backend.get(params[:id]) unless file.exists? log_error("Could not find attachment by id: #{params[:id]}") halt 404 end file.downloadend


post "/:backend" do halt 404 unless upload_allowed? tempfile = request.params.fetch("file").fetch(:tempfile) file = backend.upload(tempfile) content_type :json { id: file.id }.to_jsonend

def file file = backend.get(params[:id]) unless file.exists? log_error("Could not find attachment by id: #{params[:id]}") halt 404 end file.downloadend

refile upload


2 hops problem when uploading and downloading

• 3mb file becomes 6mb each way

• #create and #show becomes 12mb process

• has_many :images?

refile• CarrierWave-styled named processors (e.g. :fill, :thumb) vs

passing through syntax to underlying ImageMagick

• personally prefer leveraging off existing knowledge

• instead of manually configured syntax sugar

• “2 hop problem” however provide higher consistency when running multiple refile servers

• upload-here-download-there problem

• (considering to perform 2-hop upload instead of async)

refile• Can upload to S3 direct and/or upload to refile

• Redundancy interesting, but prefer less concern in Rails app

• Concept of cache and store to manage “dangling file problem” is worth considering

• Download urls are signed-only (this practice should be adopted)

• can partly counter motivation to abuse “dangling file problem” (aka free file hosting, whee!)

Questions?Attache server https://github.com/choonkeat/attache

Gem for Rails https://github.com/choonkeat/attache_rails

Demo https://attache-demo.herokuapp.com/

https://github.com/choonkeat/attache

https://github.com/choonkeat/attache_rails

https://attache-demo.herokuapp.com/

Software

File Upload 2015