Transitioning your architecture to scale

Transitioning your architecture to scale

David Tinker - CTO BrandsEye

@david_tinker#scaleconf

15 April 2013

What is BrandsEye?

• We monitor online conversation (Twitter, Facebook, G+, websites, blogs, etc.) about brands (e.g. ABSA, Gautrain) and derive insights from the data

• We process between 1m and 3m tweets and other brand mentions per day

• Relevancy, sentiment analysis, country and language and other variables all automated and crowd sourced

• Clients use our web app to see the results and explore the data

Simplified BrandsEye Schema

Account

urititleextractcountrylanguageetc.matched phrasesbrand sentiments

Mentionnameparent

Brand

querybrand

Search Phrase

BrandsEye in June 2011• Single Java application (ear file) for everything• Mention collection, mention processing & client web app

• All accessing MySQL directly using Hibernate & JDBC

• Single MySQL server at Rackspace• Separate database per client account

• 2 app servers at Rackspace (Apache & JBoss)• Client web app load balanced

• Single instance for mention collection and processing

• Appropriate architecture when BE started (2006)• It was website mentions and not tweets back then and 20/day was a lot!

• Quick to get to market

Problems in 2011

• Fragile• Any problem with mention processing or MySQL would stop mention

collection and lead to missed tweets (and grumpy clients)

• Re-deploying the app (e.g. for client web app change) would interrupt mention collection and processing

• Hard to recover from mention processing problems as mentions were not stored prior to processing

• Any change to any part of the application risked breaking some other part of it (no tests)

• Hard to change MySQL schema as all parts of the application needed to be changed and manually tested

• If our MySQL server dies we are down for a while and will lose data

• Slow• Everything used MySQL and our server (16 cores, 72G RAM) was taking

strain

• Poor latency - long time for a mention to appear in a client account

Solution & Challenges

• Separate out and decouple mention collection, mention processing, the client app and the database

• Had to keep the business running at the same time

• Small team (1.5 - 4 developers) and limited budget so big bang new architecture not an option

=> Incremental approach which is still in progress!

The major components and technologies are described on the following slides with details on what worked and what didn’t work for us

Simplified Architecture

ChickenMention Store

AnalyticsPublic API

Account Mention Data

PostgreSQL master+slave

BeefFeedproxy

Mention Collector

Redis Mongo

RabbitMQ

Account Mention Data

PostgreSQL master+slave

BrandsEyeJavascript app

JSONHTTP

PorkMention

ProcessingPipeline

Redis

MashAccount Meta

Data

PgSQL

3rd party apps

AMQP

JDBC

MentionsGravyThe BrandsEye

Crowd

MySQL

Mentions

AMQP

Sunday Lunch

Mention Flow

FeedproxyMention Collector RabbitMQ

PorkMention

ProcessingPipeline

ChickenMention Store

AnalyticsPublic API

Account Mention Data

PostgreSQL master+slave

GravyThe BrandsEye

Crowd

Rater (starving student)

Feedproxy (Mention Collector)

• Java app deployed on 2 virtual servers• Collects mentions from many different sources using Redis sorted sets

for efficient polling and de-duping

• Buffers mentions in a MongoDB capped collection as JSON messages

• Writes them to a RabbitMQ queue on a remote server for processing

• Can replay mentions from a point in the past to recover from processing failures

• Has a web UI to display status and stats

Feedproxy (2)

• Redis• Redis is a semi-persistent key/value store with sets, lists etc.

• Perfect for this application

• Uses clever fork trick with copy-on-write to save to disk periodically

• The data fits in memory easily and its not terribly bad if we lose the most recent 2 minutes or so

• Lightning fast and uses hardly any CPU

• As easy as using in memory data structures but the app continues where it left off after a re-deploy

• Have to watch out for “leaks” in Redis data structures

• Clustered version not available yet but you can do replication

Feedproxy (3)

• MongoDB• JSON data store with indexing, querying and so on

• Uses memory mapped files for everything and relies on the OS

• Has capped collections (ring buffer) with fixed size

• Capped collection uses a lot of IO once it fills up even though we only use one index

• Gets “swapped out” so occasional retrieval of old mentions takes a long time - Mongo not so good on a machine doing other stuff as well

• “Expensive” way of buffering our mentions

• Have written a replacement but its not in production yet

Feedproxy (4)

• RabbitMQ• RabbitMQ is a message broker

• You setup exchanges which distribute messages to queues for consumers to process

• Initially used it to buffer mentions on the mention collector server and a RabbitMQ shovel to get those to the RabbitMQ instance on the processing machine

• The shovel got “stuck” sometimes (every couple of weeks)

• Rabbit’s memory usage climbs linearly with the number of messages in its queues regardless of queue durability settings + it stops accepting messages when a “high watermark” of memory usage is reached

• Cannot “replay” mentions from a point in time in the past

• Not good as a durable store for messages

Chicken API & Mention Store

ChickenMention Store

AnalyticsPublic API

Account Mention Data

PostgreSQL master+slave

BeefFeedproxy

Mention Collector

Redis Mongo

RabbitMQ

Account Mention Data

PostgreSQL master+slave

BrandsEyeJavascript app

JSONHTTP

PorkMention

ProcessingPipeline

Redis

MashAccount Meta

Data

PgSQL

3rd party apps

AMQP

JDBC

MentionsGravyThe BrandsEye

Crowd

MySQL

Mentions

AMQP

Chicken API & Mention Store

• Provides a REST API to access mentions and analytics• Written using Grails, a Groovy+Java Ruby on Rails clone for Spring stack

• The only app with access to the account mention databases

• Translates our high level mention filter language into SQL

• Has good set of functional tests

• BrandsEye customers can use the API to build their own apps

• Supports multiple different mention stores with a single API

• We are busy transitioning from a single MySQL server to several PostgreSQL clusters

• Keeps stats in Redis

• Uses Apache SOLR for full text search (likely to be replaced with PostgreSQL)

• Stateless (will be load balanced soon)

Chicken API & Mention Store• Online documentation

• We use semantic versioning• Major: Increment on breaking API changes

• Minor: Increment when new functionality added

• Patch: Increment for bug fixes

The Book of ChickenWelcome to the BrandsEye API, v1.17.10

https://api.brandseye.com/rest/accounts/BESC27AA/mentions?filter=Published inthelast month and Language isnt 'en'

select id, title, extract ... from mention join ... where published_date >= ? and

language <> ?Can also do groupby etc.

Why PostgreSQL?• Synchronous replication since 9.1• Transaction on master only commits when slave has received the log

records

• Keeps pair of servers exactly in sync

• Replication done over dedicated gigabit network link

• Read-only queries can go to master or slave, writes only to master

• We wrote an app (running on 3rd machine) to monitor the pair and promote the slave / disable replication as needed (uses Hetzner failover IP addresses to make the dead machine inaccessible)

• MySQL now also has something similar but it all seems a bit ropey

• Other reasons• PostgreSQL has good full text search and arrays

• MySQL query planner doesn’t handle some simple subqueries + inexplicably fails to use indexes for others

• We need to stick with RDBMS as we do lots of ad-hoc queries

Why Grails?• Quick to build apps• Groovy has syntax similar to Ruby and Python but runs on the JVM and

integrates seamlessly with Java code

• Most Java code is also valid Groovy code so great for a team with Java background

• Grails is very much like Rails but using familiar Java stuff (Spring, Hibernate etc.)

• Lots of plugins and they are easy to write

• Performance is available• Need something fast? Just write that little bit in Java

• We won’t have to switch stacks (e.g. from Ruby to Java) at some point for performance reasons

Mash Account Meta Data

ChickenMention Store

AnalyticsPublic API

Account Mention Data

PostgreSQL master+slave

BeefFeedproxy

Mention Collector

Redis Mongo

RabbitMQ

Account Mention Data

PostgreSQL master+slave

BrandsEyeJavascript app

JSONHTTP

PorkMention

ProcessingPipeline

Redis

MashAccount Meta

Data

PgSQL

3rd party apps

AMQP

JDBC

MentionsGravyThe BrandsEye

Crowd

MySQL

Mentions

AMQP

Mash Account Meta Data

• Provides a REST API for account meta data• Grails app using MySQL (moving to PostgreSQL)

• Brands, search phrases, processing rules etc.

• Notifies client applications of changes via RabbitMQ topic exchange

• Client apps typically cache the data until it changes or a timeout expires

• Most client apps use a Java library which handles the caching and maps the JSON to a data model

• Simple way to distribute the information to many de-coupled apps

Pork Mention Processor

ChickenMention Store

AnalyticsPublic API

Account Mention Data

PostgreSQL master+slave

BeefFeedproxy

Mention Collector

Redis Mongo

RabbitMQ

Account Mention Data

PostgreSQL master+slave

BrandsEyeJavascript app

JSONHTTP

PorkMention

ProcessingPipeline

Redis

MashAccount Meta

Data

PgSQL

3rd party apps

AMQP

JDBC

MentionsGravyThe BrandsEye

Crowd

MySQL

Mentions

AMQP

Pork Mention Processor• Consumes mentions (JSON messages) from

RabbitMQ queues• Grails app with basic UI for monitoring

• Annotates mentions with extra information (relevancy, sentiment, country, language etc.) using machine learning and other techniques

• Applies automated processing rules

• Sends & receives mentions from the BrandsEye crowd

• Writes mentions to Chicken (Mention Store)

• ACK mention if all good, otherwise NACK and re-process

• Lots of batching and shared models for performance and rate limiting reasons

• Groovy closures and other features result in compact maintainable code

• Small amount of performance centric code in Java + machine learning libraries

• Can process 1m+ mentions/hour, mostly waiting for Chicken

• Keeps stats in Redis

RabbitMQ

• Good• ACK/NACK model is very convenient for development

• Can limit number of un-ACKed messages allowed to control app memory usage

• Exchange types and routing keys allow for flexible setup

• Easy to duplicate messages on the fly (e.g. for debugging in live)

• Nice admin console

• Limitations• Cannot cluster queues so even for clustered rabbit losing a machine

loses everything in its queues

• Memory usage climbs rapidly if you aren’t consuming messages

Beef Client Web App

• Grails + Javascript using Backbone, Handlebars etc.• Communicates with Chicken using the same API we offer to clients

• Maintains brands, phrases etc. by communicating with Mash

• All REST using JSON

• Makes it possible for us to refactor all of the backend apps (e.g. change database schemas) so long as we keep the API the same

Monitoring

• We use Wormly for server and app alerts

• Each app has a simple web console that can be checked for “ERROR”

• Logs are aggregated using Graylog2

SCM and Packaging• We use Git & Bitbucket• Same as Github but much cheaper for a small team with many repos

• Master must always be deployable, we don’t use branches much

• Apps are packaged as executable war files• Servlet container (Jetty) embedded in war

• java -jar crackling.war and it will come up listening on its port

• Built on the target machine (no CI yet ...)

• Apache or Nginx in front

• Good documentation• Each app has a README.md describing its purpose, how to build and

run/publish it, API endpoints, dependencies (and how to install them) etc.

Hetzner & AWS vs Rackspace

• Price• Rackspace is at least 6x more expensive that Hetzner for a similar

(supposedly better quality) machine

• Rackspace cloud servers are also much more expensive than Amazon servers (6x or more) and they charge full price even when your server isn’t running

• So we are moving towards “more machines that can fail” instead of “a few really reliable machines”

• Other factors• Hetzner give failover IPs that you can change with an API for

implementing HA stuff

• Servers have 2 NICs so you can create private nets for replication

• Traffic is free (great for backups) up to 10000G/month

• We need physical hardware for at least the database servers for performance

Tech Summary

• Redis• Fast solid software, lots of use cases for data sets that fit in memory

• Watch out for leaks

• MongoDB• Capped collections slower than you expect

• Really wants to be the only thing installed on the server

• MySQL• Not very good at optimizing queries or using indexes, dodgy replication

• PostgreSQL• Synchronous replication is cool

• Advanced query optimizer

• RabbitMQ• Great for short term routing of messages, watch out for memory usage

Conclusion

• We can now easily scale BrandsEye to handle any number of clients and volume of mentions

• All of this is still in progress

• We have lots of other little apps not described here interacting using the same tech

• We have done a lot of work with chef for our new servers but its not handling everything yet

Questions?

$100 discount to Scaleconf people!