Scaling RabbitMQ to 11

Gavin M. Roy @crad - Surge 2011

Scaling RabbitMQto 11



INTERACTIVE POLLS• http://pollev.com/gmr

• SMS: 22333

2

http://pollev.com/gmr

http://pollev.com/gmr


myYearbook.com• #1 Comscore Teen Destination, Top Trafficked Sites in US

• Use of RabbitMQ dates back to 2009

• Multiple clusters for different purposes

• Browser <-> Backend Message Bus

• Messaging, Event Processing

3


DISASTER PORN


ABOUT AMPQ• Platform, Vendor Neutral Messaging format

• 0.9.1 finalized in November 2008

• JP Morgan Chase, Red Hat, Rabbit Technologies, iMatrix,

IONA Technologies, Cisco Systems, Envoy Technologies,

Twist Process Innovations

• Members now include Bank of America, Barclays Banks,

Microsoft Corporation, Novell, VMWare

• 1.0 Final: October 2011

5


ABOUT RABBITMQ• Born in 2006, Released in 2007, Acquired in 2010

• Written in Erlang/OTP

• Talks AMQP 0.8 and 0.9.1

• Features include:

• Clustering, Active/Active HA Queues

• Management UI and API

• Plugin Architecture

6


ABOUT YOU & RABBITMQ


RABBITMQ& AMQPCONCEPTS


PUBLISHERS


CONSUMERS


QUEUES• Durability: Queue survives reboot

• Auto-Delete: Delete the queue when consumer goes away

• Exclusive: Only one consumer may consume

• x-expires: Auto-Delete after given duration

• x-message-ttl: Auto-discard a message after ttl

11


MESSAGES• Have client specified properties:

• Content type, Encoding, Timestamp, App-Id, User-Id,

Headers

• Delivery Mode:

• 1: Non-Persistent

• 2: Persistent

12


MESSAGE ROUTINGIt’s not just about Queues.


ROUTING KEYIn addition to the exchange, determines what is done with a

message.


CC & BCCMore routing options per message, but violates AMQP.


HEADERS EXCHANGEMore on this later.


EXCHANGES


FANOUT• 1:N Message Delivery Pattern

• No Routing Keys Involved

• Queues bound to a fanout exchange get all messages sent to

the exchange

18


DIRECT• 1:1 Message Delivery Pattern

• Routing Keys are direct, no wildcarding

• All queues bound to a routing key get the message

19


TOPIC EXCHANGES• Pattern Matching in Routing Keys

• Publish one message type to Surge.Sessions, another to

Surge.Mentions

• Period delimiter, * stays within the period scope, # does not

• Example patterns: *.Mentions, Surge.*, #

• Slower than Fanout and Direct

20


HEADERS EXCHANGES• Matches on Basic.Properties headers table values to the

arguments table specified when binding a queue

• Use x-match when declaring exchange to specify any or all

headers must match

• Slow implementation in Erlang

21


PLUGIN EXCHANGES• Consistent Hash Exchange: Round-robin distribution to queues

• External: RPC Exchange Plugin

• Script: Similar in intent to External

• Riak

• Last-Value Cache

• Recent History Exchange: Keeps last 20 messages routed

• Global Fanout: Sends to all queues, regardless of routing key

22


ONE MORE THING...Exchange-to-Exchange Bindings


CLUSTERS


STATS, DISK AND RAM NODES


WHERE DOES THE QUEUE LIVE?


CROSS-NODE PUBLISHING & CONSUMING


MAX CLUSTER SIZE~32 notes is a realistic maximum node count for a cluster


SCALING ISSUES?




WHAT TO WATCH



RABBITMQ

Y U NO TAKE MESSAGE?


TCP BACKPRESSUREMUCH?


MEMORY MANAGEMENT


[{RABBIT, [{VM_MEMORY_HIGH_WATERMARK, 0.4}]}].

40%


MESSAGE PERSISTENCE50-70% performance penalty compared to non-persisted

messages. * http://www.rabbitmq.com/faq.html#performance-persistent

http://www.rabbitmq.com/faq.html#performance-persistent

http://www.rabbitmq.com/faq.html#performance-persistent


CONSUMINGAck, Nack and Reject vs Basic.Consume(no_ack=True)



Server Client

Basic.DeliverBasic.[Ack/Nack/Reject]

Basic.[Ack/Nack/Reject]Ok



BASIC.QOS

• Prefetch Size and Count

• How many messages the

client wants the RabbitMQ

to pre-deliver to the client

• Doesn’t work in no_ack


TOO MANY SHORT LIVED CONNECTIONS


APACHE + MOD_PHP


CONNECTION PERSISTENCE


100 SERVERS125 BACKENDS1 REQUEST/S12,500 CONN/S


AMQP CONNECTION SEQUENCE

Client Server

Protocol Header FrameConnection.Start

Connection.StartOk

Connection.TuneOkConnection.Tune

Connection.OpenOkConnection.Open

Channel.OpenChannel.OpenOk


685 BYTES PER8.16 MB/S IN STARTUP @12,500 CONN/S


VORPAL BUNNY


VORPAL BUNNY• Light-weight PHP client for RabbitMQ

• Used the experimental JSON-RPC Channel plugin

• Handles the overhead of the HTTP, JSON-RPC to AMQP

analog

• Fire and forget message publishing

• Can not be transparently load balanced

48


TOO MANY CONNECTIONS


VELOCITY TOO HIGH FOR ONE NODE?


MEMORYUTILIZATION DISPROPORTIONATE TO QUEUE DEPTHS


CLUSTERNODE FAILURES


UP TO 60 SECONDS TO DETECT


MESSAGES DROPPED ON THE FLOOR*


* 2.6.0 ADDEDACTIVE/ACTIVE QUEUES IN CLUSTERS


ACTIVE/PASSIVE FAILOVER?Recommendation is to use Pacemaker and DRBD or shared

storage for Mnesia databases.


RECOVERY?


CLUSTERING ALTERNATIVES


Scaling RabbitMQ to 11

FEDERATION

• Exchange Plugin

• Provides ability to route all

messages to a x-federation

exchange to another broker


SHOVEL PLUGIN

• From Queue on one broker to an

exchange on another broker.

• Effective for cross-cluster data

broadcasting


TRENDING & MONITORING• RabbitMQ Message Velocity Rates Gotcha

• Management API Plugin exposes RESTful API including

monitoring data points

• 2.6.1 allows monitoring tag for users to allow data access

without full admin access

• Queue.Declare(passive=True) gets queue depths as well

• Plug in to Graphite, Nagios,or your favorite tool

61


PHOTO CREDITS• “skateboard kids 7”

© 2007 Volker Neumann

http://flickr.com/prawnpie

• “methodcn contest”

©2009 Erin O’Hara

http://flickr.com/erinohara73/

• USPS Truck Accident

Valley City Fire Department

http://www.valleycityfire.com

62







QUESTIONS?

Documents

Scaling RabbitMQ to 11