Implementing a Fileserver with Nginx

and Luatechtalk@ferretAndrii Gakhov27.02.2017

Current ArchitectureRestful API server that communicates with clients using application/json

Every client is authenticated by a dedicated 3rd party authentication server and authorized by the API server

End users get benefits from the API via client-side frontend application

The ProblemAt some point, end users request the ability to manage files using the same client-side application and, consequently, same API.

allow to process multipart/form-data requests (that will be proxied from the form on the client-side application)

extract and handle file metadata

provide file storage and access

The SolutionObviously, such functionality is out of the scope for the API and the natural decision is to split it across 2 applications:

API is used for meta file management

Fileserver takes care about actual files upload/download

Instead of using Flask/Django etc., we have decided to implements such functionality in Nginx with Lua scripting

Nginx and LuaNginx is a free, open-source, high-performance HTTP server and reverse proxy, as well as an IMAP/POP3 proxy server.Lua is a powerful, efficient, lightweight, embeddable scripting language. It supports procedural programming, object-oriented programming, functional programming, data-driven programming, and data description.

The Lua module for Nginx ngx_http_lua_module embeds Lua into Nginx and by leveraging Nginx's subrequests, allows the integration of Lua threads into the Nginx event model.

Nginx and LuaNginx object is available in Lua as ngx and server variables are accessible as ngx.var.{name}, requested GET arguments - as ngx.var.arg_{name}, but unfortunately POST variables Nginx doesn't parse into variables automatically.

The various *_by_lua_* configuration directives serve as gateways to the Lua API within the nginx.conf. We use *_by_lua_block to embed Lua code into Nginx as the most notable way to do so.

Nginx and Luaheader_filter_by_lua_block - uses Lua code to define an output header filter (e.g. for overriding or adding a response header).

access_by_lua_block - acts as an access phase handler and executes Lua code for every request. as with other access phase handlers, access_by_lua will not run in subrequests.

content_by_lua_block - acts as a "content handler" and executes Lua code for every request.

Architecture / Uploadlocation = /_upload {

limit_except POST { deny all; }

if ($http_content_type !~ "multipart/form-data")

{

return 406;

}

lua_need_request_body off;

client_body_buffer_size 200M;

client_max_body_size 200M;

default_type 'application/json';

...

}

Architecture / Downloadlocation ~ ^/_download/(.*)$ {

set $path /$1;

default_type 'application/json';

limit_except GET { deny all; }

...

}

User Authentication function authenticate_user(access_token) local params = { method = ngx.HTTP_GET, args = { access_token = access_token } } local res = ngx.location.capture( "/_auth/access/check", params) if not res or res.status ~= 200 then return nil end

return cjson.decode(res.body) end

access_by_lua_block {

local cjson = require("cjson")

local access_token = ngx.var.arg_access_token if not access_token then return_http_forbidden("Forbidden") end

-- authenticate user local credentials = authenticate_user(access_token)

-- if user can't be resolved, return 403 Forbidden if not credentials or not credentials.data.user.id then return_http_forbidden("Forbidden") end}

Upload-- Extract POST variables in a streaming way-- and store file object in a temporary-- file (form_data.file.filepath)-- All.other form variables are in form_data-- (we expect them to be small) local helpers = require("utils")

form_data, err = helpers.streaming_multipart_form() if form_data == nil then return_http_internal_server_error(err) end

data = { status = "not_ready", title = form.title.value, filename = form.file.filename,}local params = { method = ngx.HTTP_POST, args = {access_token = access_token}, body = cjson.encode(data)}

local res = ngx.location.capture("/_api/v1/file", params)if not res then return_http_bad_gateway("No metadata")end

local create_metadata_resp = res.bodyif res and res.status ~= 201 then ngx.status = res.status ngx.print(create_metadata_resp) ngx.exit(res.status)end

local file_metadata = cjson.decode(create_metadata_resp)

Important part to make it work is to specify client_body_buffer_size equal to client_max_body_size.

Otherwise, if the client body is bigger than client_body_buffer_size, the nginx variable $request_body will be empty.

Uploadlocal filex = require("pl.file")local file_fullpath = "/storage/files/" .. file_metadata.hash .. file_metadata.path

-- ensure that subdirectories exist (if not, create)-- store tmp file to its permanent positionis_moved, err = filex.move( form.file.fullpath, file_fullpath)-- make it available for downloadif is_moved then local params = { method = ngx.HTTP_PUT, args = {access_token = access_token}, body = cjson.encode({status = "ready"}) } ngx.location.capture( "/_api/v1/file/" .. file_metadata.id, params)end

-- provide some headers with metadatangx.header["X-File-ID"] = file_metadata.idngx.status = ngx.HTTP_CREATEDngx.print(create_metadata_resp)ngx.exit(ngx.HTTP_CREATED)

Since we need to manage access to the file, we use access policy of files’ metadata instead of the files themselves.

We create metadata in any case (“register the file”), but allow to download it only if file has status “ready”, meaning it was successfully created at the specified location.

Downloadlocal search_by_path = { filters = { path = ngx.var.path, status = "ready" }, size = 1}local params = { method = ngx.HTTP_POST, args = {access_token = access_token}, body = cjson.encode(search_by_path)}

local res = ngx.location.capture( "/_api/v1/search/files", params)if not res then return_http_bad_gateway("Search error")end

local found = cjson.decode(res.body)if found.total < 1 then return_http_not_found("File Not Found")end

local file_metadata = found.results[1]

ngx.header["X-File-Name"] = file_metadata.namengx.header["X-File-Path"] = file_metadata.pathngx.header["X-File-ID"] = file_metadata.id

ngx.req.set_uri( "/" .. file_metadata.hash .. file_metadata.path)ngx.exec("@download_file", download_params)

As soon as the file has been found, its metadata provides us with all necessary information about the location and we are ready to respond it to the user.

Sometimes people recommend to read such file in Lua and return it to the user with ngx.print that is a bad idea for big files (Lua virtual machine will just crash).

Downloadlocation @download_file { internal;

root /storage/files/; try_files $uri =404;

header_filter_by_lua_block { ngx.header["Cache-Control"] = "no-cache" ngx.header["Content-Disposition"] = "attachment; filename=\"" .. ngx.header["X-File-Name"] .. "\"" }}

The @download_file location is quite simple, but additionally we want to play with response headers to provide a real filename for download (on our filesystem all files are stored with unique generated names).

It is an internal named location (to prevent unauthorized access) that just serves requested static files from the desired directory.

How to UseUploadcurl -XPOST https://files.example.com/_upload?access_token={SOME_TOKEN} \ --form file=@/tmp/file.pdf\ --form title="Example title"\ -H "Content-Type: multipart/form-data"

Downloadcurl -XGET https://files.example.com/_download/723533/2338342189083057604.pdf?access_token={SOME_TOKEN}

Thank youRead it in the web:

https://www.datacrucis.com/research/implementing-api-based-fileserver-with-nginx-and-lua.html