Scaling RabbitMQ to 11

Gavin M. Roy @crad - Surge 2011

Scaling RabbitMQto 11

INTERACTIVE POLLS• http://pollev.com/gmr

• SMS: 22333

myYearbook.com• #1 Comscore Teen Destination, Top Trafficked Sites in US

• Use of RabbitMQ dates back to 2009

• Multiple clusters for different purposes

• Browser <-> Backend Message Bus

• Messaging, Event Processing

DISASTER PORN

ABOUT AMPQ• Platform, Vendor Neutral Messaging format

• 0.9.1 finalized in November 2008

• JP Morgan Chase, Red Hat, Rabbit Technologies, iMatrix,

IONA Technologies, Cisco Systems, Envoy Technologies,

Twist Process Innovations

• Members now include Bank of America, Barclays Banks,

Microsoft Corporation, Novell, VMWare

• 1.0 Final: October 2011

ABOUT RABBITMQ• Born in 2006, Released in 2007, Acquired in 2010

• Written in Erlang/OTP

• Talks AMQP 0.8 and 0.9.1

• Features include:

• Clustering, Active/Active HA Queues

• Management UI and API

• Plugin Architecture

ABOUT YOU & RABBITMQ

RABBITMQ& AMQPCONCEPTS

PUBLISHERS

CONSUMERS

QUEUES• Durability: Queue survives reboot

• Auto-Delete: Delete the queue when consumer goes away

• Exclusive: Only one consumer may consume

• x-expires: Auto-Delete after given duration

• x-message-ttl: Auto-discard a message after ttl

MESSAGES• Have client specified properties:

• Content type, Encoding, Timestamp, App-Id, User-Id,

Headers

• Delivery Mode:

• 1: Non-Persistent

• 2: Persistent

MESSAGE ROUTINGIt’s not just about Queues.

ROUTING KEYIn addition to the exchange, determines what is done with a

message.

CC & BCCMore routing options per message, but violates AMQP.

HEADERS EXCHANGEMore on this later.

EXCHANGES

FANOUT• 1:N Message Delivery Pattern

• No Routing Keys Involved

• Queues bound to a fanout exchange get all messages sent to

the exchange

DIRECT• 1:1 Message Delivery Pattern

• Routing Keys are direct, no wildcarding

• All queues bound to a routing key get the message

TOPIC EXCHANGES• Pattern Matching in Routing Keys

• Publish one message type to Surge.Sessions, another to

Surge.Mentions

• Period delimiter, * stays within the period scope, # does not

• Example patterns: *.Mentions, Surge.*, #

• Slower than Fanout and Direct

HEADERS EXCHANGES• Matches on Basic.Properties headers table values to the

arguments table specified when binding a queue

• Use x-match when declaring exchange to specify any or all

headers must match

• Slow implementation in Erlang

PLUGIN EXCHANGES• Consistent Hash Exchange: Round-robin distribution to queues

• External: RPC Exchange Plugin

• Script: Similar in intent to External

• Riak

• Last-Value Cache

• Recent History Exchange: Keeps last 20 messages routed

• Global Fanout: Sends to all queues, regardless of routing key

ONE MORE THING...Exchange-to-Exchange Bindings

CLUSTERS

STATS, DISK AND RAM NODES

WHERE DOES THE QUEUE LIVE?

CROSS-NODE PUBLISHING & CONSUMING

MAX CLUSTER SIZE~32 notes is a realistic maximum node count for a cluster

SCALING ISSUES?

WHAT TO WATCH

RABBITMQ

Y U NO TAKE MESSAGE?

TCP BACKPRESSUREMUCH?

MEMORY MANAGEMENT

[{RABBIT, [{VM_MEMORY_HIGH_WATERMARK, 0.4}]}].

MESSAGE PERSISTENCE50-70% performance penalty compared to non-persisted

messages. * http://www.rabbitmq.com/faq.html#performance-persistent

CONSUMINGAck, Nack and Reject vs Basic.Consume(no_ack=True)

Server Client

Basic.DeliverBasic.[Ack/Nack/Reject]

Basic.[Ack/Nack/Reject]Ok

BASIC.QOS

• Prefetch Size and Count

• How many messages the

client wants the RabbitMQ

to pre-deliver to the client

• Doesn’t work in no_ack

TOO MANY SHORT LIVED CONNECTIONS

APACHE + MOD_PHP

CONNECTION PERSISTENCE

100 SERVERS125 BACKENDS1 REQUEST/S12,500 CONN/S

AMQP CONNECTION SEQUENCE

Client Server

Protocol Header FrameConnection.Start

Connection.StartOk

Connection.TuneOkConnection.Tune

Connection.OpenOkConnection.Open

Channel.OpenChannel.OpenOk

685 BYTES PER8.16 MB/S IN STARTUP @12,500 CONN/S

VORPAL BUNNY

VORPAL BUNNY• Light-weight PHP client for RabbitMQ

• Used the experimental JSON-RPC Channel plugin

• Handles the overhead of the HTTP, JSON-RPC to AMQP

analog

• Fire and forget message publishing

• Can not be transparently load balanced

TOO MANY CONNECTIONS

VELOCITY TOO HIGH FOR ONE NODE?

MEMORYUTILIZATION DISPROPORTIONATE TO QUEUE DEPTHS

CLUSTERNODE FAILURES

UP TO 60 SECONDS TO DETECT

MESSAGES DROPPED ON THE FLOOR*

* 2.6.0 ADDEDACTIVE/ACTIVE QUEUES IN CLUSTERS

ACTIVE/PASSIVE FAILOVER?Recommendation is to use Pacemaker and DRBD or shared

storage for Mnesia databases.

RECOVERY?

CLUSTERING ALTERNATIVES

Scaling RabbitMQ to 11

FEDERATION

• Exchange Plugin

• Provides ability to route all

messages to a x-federation

exchange to another broker

SHOVEL PLUGIN

• From Queue on one broker to an

exchange on another broker.

• Effective for cross-cluster data

broadcasting

TRENDING & MONITORING• RabbitMQ Message Velocity Rates Gotcha

• Management API Plugin exposes RESTful API including

monitoring data points

• 2.6.1 allows monitoring tag for users to allow data access

without full admin access

• Queue.Declare(passive=True) gets queue depths as well

• Plug in to Graphite, Nagios,or your favorite tool

PHOTO CREDITS• “skateboard kids 7”

http://flickr.com/prawnpie

• “methodcn contest”

http://flickr.com/erinohara73/

• USPS Truck Accident

Valley City Fire Department

http://www.valleycityfire.com

QUESTIONS?

Scaling RabbitMQ to 11

Documents

Zenoss Service Dynamics RPS 4.2.5 SP743 Release …...1. RabbitMQ ZenPack Zenoss recommends installation of the RabbitMQ ZenPack (version 1.0.7+) for monitoring of the Zenoss RabbitMQ

Operational and Scaling Wins at Workday · 2019. 2. 26. · Cassandra rabbitmq zookeeper Contrail API. Key drivers for architectural evolution Downtime 0 upgrade Provide upgrade path

Learning RabbitMQ - Sample Chapter

RabbitMQ for Perl mongers

MicroservicePlumbing With RabbitMQ€¦ · • .NET Core • RabbitMQ • Docker • Vue.JS. RabbitMQOverview What is RabbitMQ? @JSONROW. What is AMQP? Advanced Message Queue Protocol

Integrating PostgreSql with RabbitMQ

building a scalable geofence platform based on RabbitMQ ......RabbitMQ GEO FENCES generic production-grade message broker custom exchanges - x-ingress: special "shovel" rabbitmq app

Rabbitmq Boot System

Scaling applications with RabbitMQ at SunshinePHP

PHP, RabbitMQ, and You

RabbitMQ fairly-indepth

SCALING INSTAGRAM INFRA › system › files › presentation-slides › qcon20… · RabbitMQ PostgreSQL Cassandra Celery Other Services Django. STORAGE VS. COMPUTING • Storage:

RabbitMQ in Sprayer

RabbitMQ + CouchDB = Awesome

Reactive Streams and RabbitMQ

RabbitMQ Oxford Geek Night

RabbitMQ Hands On

AMQP and Beyond - RabbitMQ rabbitmq-jsonrpc-channel rabbitmq-jsonrpc rfc4627_jsonrpc rabbitmq-mochiweb mochiweb rabbit_stomp rabbithub rabbitmq-bql amqp_client rabbitmq-shovel rabbitmq-toke

BECS RabbitMQ Connector Documentation · BECS RabbitMQ Connector Documentation Page 3 of 14 3. After installation (if standard package does not include RabbitMQ Connector), is necessary

Integrating RabbitMQ with PHP