Embed Size (px)
Citation preview
atx node.js
August Meetup
background jobsin an async world
Cody Stoltman@particlebanana
sponsors
who are you?
Cody Stoltman@particlebanana
co-founder of Treeline
Sails.js core team
maintainer of Waterline ORM
what’s a background job?
browser /foo/bar
does some stuff
normal request/response lifecycle
browser /foo/bar
does some stuff
slow function
long running process lifecycle
browser /foo/bar
does some stuff
slow function
this isn’t rubynode is async
we have callbacks
what does blocking mean?
browser 1
traditionally
browser 2
server
do something
browser 1
with node.js
browser 2
server
do something
browser 3
browser /foo/bar
does some stuff
slow fuction
how can we handle that function?
• callbacks• just ignore the callback
• over engineer something related to process spawning
why this isn’t ideal
we aren’t blocking so the server can keep responding
but we have to wait until the process is done before responding to the client’s request
how can we handle that function?
• callbacks• just ignore the callback
• over engineer something related to process spawning
browser /foo/bar
does some stuff
slow function
why this isn’t ideal
the client get’s data back but we have no clue as to the success/failure of the task.
we could add logs but we have no way to give the client feedback or to re-try.
how can we handle that function?
• callbacks• just ignore the callback
• over engineer something related to process spawning
let’s learn from our synchronous, threaded brothers and sisters
whats the answer then?
how can we handle that function?
• remote queues
remote queue
queue worker
slow function
server
does some stuff
job
browser
job
why this is better
We can observe our queue
We can scale out horizontally and add capacity as needed
Our queue can be preserved so no jobs are abandoned
scalable remote queue
server server server server
worker worker worker worker
load balancer
queue
back to that event loop thing
call stack libuv api
callback queue
event loop
.on()
handler
handler ()
lets see what blocking actually means
call stack libuv api
callback queue
event loop
.on()
handler
handler ()
delayFn ()
handler handler handler
handler ()
so synchronous code blocks the event loop, don’t do that
what happens if we don’t block it?
call stack libuv api
callback queue
event loop
.on()
handler
handler()
sendEmail() queue
cbhandler handler
cb()handler()
sendEmail() queue
cb
remote queues are good because they let you do more, faster
so more work in the event loop means less concurrency
when should you move a function into a background job?
if for any reason it will block the event loop
it will prevent the response from returning in a timely manner
node modules
node-resquegithub.com/taskrabbit/node-resque
kuegithub.com/Automattic/kue
bullgithub.com/OptimalBits/bull
most queues are built on redis
you could also build on a traditional message queue
summary
remote queues in node same as anywhere else
actually better because you can do more on a single cpu simultaneously(example: sending emails)
scheduled tasks
you need a database backed scheduler
someone should write one
agenda (mongo only)github.com/rschmukler/agenda
until then
demo
thanks!
Phillip Roberts (@latentflip)github.com/latentflip/loupe
Evan Tahler (@evantahler)blog.evantahler.com
atx node.js
August Meetup
