FirehoseBuilding a real-time data stream with NodeJS

Luca Grulla@ForwardLondon, 2/8/2011

Introduction

Or what we wanted to build

We like to have a (near) real-time view of the data stream

Resilient

Scalable

Accurate(but not necessarily precise)

Verse and Chorus

Or how we built it

Stack

• NodeJS (0.4.9)• CoffeeScript• ZeroMQ

Architecture

Interlude

Or joys and pitfalls of building a tail library for node

Living the asynchronous dream

Living the asynchronous dream

• Error manifests itself only at high throughput (7MB/secs)

• The few implementations you can find around are wrong (including Log.io – we are planning to contribute back)

Breathing in an asynchronous way

Learnings

• You need to think asynchronous to write asynchronous code

• Dispatch internal events if you need to keep control of the flow

• “Internal queues” to scale(up to a certain limit)

Performance

• Beyond 5MB we start to fill the internal queue• We are limited by IO, not from CPU/memory• Consistent throughput

Questions ?

References

• npm install tail• https://github.com/forward/node-tail