48
Messaging Standards and Systems AMQP & RabbitMQ Gavin M. Roy VP of Architecture AWeber Communications Twitter: @Crad POSSCON April 14, 2015

Messaging Standards and Systems - AMQP & RabbitMQ

  • Upload
    posscon

  • View
    152

  • Download
    6

Embed Size (px)

Citation preview

Page 1: Messaging Standards and Systems - AMQP & RabbitMQ

Messaging Standards and Systems

AMQP & RabbitMQGavin M. RoyVP of ArchitectureAWeber CommunicationsTwitter: @Crad

POSSCON April 14, 2015

Page 2: Messaging Standards and Systems - AMQP & RabbitMQ

About MeVP of Architecture

AWeber Communications

Blame me for pika, rabbitpy, pamqp, and a handful of

RabbitMQ plugins

Blog: https://gavinroy.com

Github: https://github.com/gmr

Book: http://manning.com/roy

Page 3: Messaging Standards and Systems - AMQP & RabbitMQ

Advanced Message Queueing Protocol

❖ Open Standard

❖ Platform and Vendor Neutral

❖ Multiple Versions

Page 4: Messaging Standards and Systems - AMQP & RabbitMQ

AMQP Timeline

2003

2005

2006

2008

2011Initial Specification

Working Group formed

AMQP 0-8

AMQP 0-9-1

OASISAMQP 1.0

Page 5: Messaging Standards and Systems - AMQP & RabbitMQ

AMQP Working Group❖ JPMorgan Chase

❖ Cisco Systems

❖ IONA Technologies

❖ iMatrix

❖ RedHat

❖ TWIST

❖ Bank of America

❖ Barclays

❖ Credit Suisse

❖ Deutsche Börse Systems

❖ Goldman Sachs

❖ HCL Technologies

❖ Progress Software

❖ IIT Software

❖ INETCO Systems Ltd.

❖ Informatica Corporation

❖ Microsoft Corporation

❖ my-Channels

❖ Novell

❖ Solace Systems

❖ Tervala, Inc.

❖ VMWare

❖ WSO2

Page 6: Messaging Standards and Systems - AMQP & RabbitMQ

Competing AMQP Standards

≠0-8&

0-9-11.0

Page 7: Messaging Standards and Systems - AMQP & RabbitMQ

AMQP 0-9-1

❖ Currently has wider support than AMQP 1.0

❖ Multiple Broker Implementations: RabbitMQ, Apache Qpid, and SwiftMQ to name a few

❖ Specifies a Model and Protocol

Page 8: Messaging Standards and Systems - AMQP & RabbitMQ

Native AMQP 0-8/0-9-1 Clients

C

Clojure

Cobol

Common Lisp

Delphi

Erlang

Go

Groovy

Haskell

Java

JavaScript

.NET

OCaml

Perl

PHP

Python

Ruby

Scala

Page 9: Messaging Standards and Systems - AMQP & RabbitMQ

❖ Exchange Receives and route messages

❖ Message Queue Stores messages until they can be consumed

❖ Binding Defines the relationship between an Exchange and Queue and provides routing criteria

Advanced Message Queueing Model

X

Exchange

BindingMessage Queue

Page 10: Messaging Standards and Systems - AMQP & RabbitMQ

Routing Keys

❖ Provided when publishing a message

❖ Compared against binding keys by exchanges

❖ Ideally provide context to the message:

❖ Connote the type of the message

❖ Categorize the content in the message

❖ Specify the type of consumer that should receive it

Page 11: Messaging Standards and Systems - AMQP & RabbitMQ

Advanced Message Queueing Protocol

❖ Compact, binary frame format wire protocol

❖ Bi-directional RPC

❖ Commands consist of Classes and Methods:

❖ Example Request: Queue.Declare(name=“foo”)

❖ Response: Queue.DeclareOk(messages=0, consumers=0)

Page 12: Messaging Standards and Systems - AMQP & RabbitMQ

Common AMQP Terms

❖ Broker A server that implements AMQP

❖ Producer or Publisher A client application that sends messages to a broker

❖ Consumer A client application that reads messages from a queue

Page 13: Messaging Standards and Systems - AMQP & RabbitMQ

Publishers and Consumers

CP

Page 14: Messaging Standards and Systems - AMQP & RabbitMQ

Multiple Publishers

CP

P

P

Page 15: Messaging Standards and Systems - AMQP & RabbitMQ

Multiple Consumers

C

C

C

P

Page 16: Messaging Standards and Systems - AMQP & RabbitMQ

AMQP Messages

❖ Comprised of 3 or more frames:

❖ Method FrameBasic.Publish, Basic.Deliver, etc

❖ Content Header with body size & message properties timestamp, message-id, app-id, etc

❖ n Body Frames with the opaque message payload

Page 17: Messaging Standards and Systems - AMQP & RabbitMQ

AMQP 0-9-1 Issues & Gotchas

❖ Ambiguous

❖ Authentication

❖ Asynchronous

❖ Connection Negotiation

❖ Exceptions!

Page 18: Messaging Standards and Systems - AMQP & RabbitMQ

RabbitMQ

❖ Open Source (MPL)

❖ Written in Erlang/OTP

❖ Developed/Maintained by Pivotal

❖ Multi-Protocol AMQP 0-9-1 & 1.0, MQTT, STOMP, XMPP, HTTP, Web-STOMP & More

❖ Roots in AMQP 0-8/0-9-1

Page 19: Messaging Standards and Systems - AMQP & RabbitMQ

Who Uses It?Agora Games

Chef

Google AdMob

Instagram

MeetMe

Mercado Libre

Mozilla

NASA

New York Times

National Science Foundation

Openstack

Rapportive

Reddit

Soundcloud

(and many more)

Page 20: Messaging Standards and Systems - AMQP & RabbitMQ

Why Use RabbitMQ?

❖ Create loosely-coupled applications

❖ Communicate across applications or platforms

❖ Tap into pre-existing message flows for new purposes

❖ Scale-out clustering for growth, throughput, and HA

❖ Federation for WAN latencies and network partitions

❖ Extensible plugin-in architecture

Page 21: Messaging Standards and Systems - AMQP & RabbitMQ

RabbitMQ Extensions to AMQPAuthentication Failure

Exchange to Exchange Bindings

Delivery Confirmations

Basic.Nack

Consumer cancellations

Consumer priorities

Dead Letter Exchanges

Alternate Exchanges

Connection blocking

Message CC & BCC Routing

Queue Length Limits

Per Queue Message TTL

Per Message TTL

Queue TTL

Message User ID Validation

Auto-delete exchanges

Page 22: Messaging Standards and Systems - AMQP & RabbitMQ

RabbitMQ Clustering

❖ LAN Only

❖ Adds highly-available queues

❖ Is cohesive, publish and consume from any node

❖ Leverages native Erlang clustering and communication

❖ Has multiple strategies for dealing with network partitions

❖ Manually configured via configuration or command line*

Page 23: Messaging Standards and Systems - AMQP & RabbitMQ

RabbitMQ Federation

❖ Provides loosely coupled inter-node/cluster RabbitMQ communication

❖ Connect multiple RabbitMQ clusters across data centers or the Internet

❖ Can be applied to both exchanges and queues

Cluster A

X

Cluster B

X

Cluster C

X

Page 24: Messaging Standards and Systems - AMQP & RabbitMQ

RabbitMQ Pluginsamqp1_0 Adds AMQP 1.0 support to RabbitMQ

auth-backend-amqp AMQP RPC for Authentication

auth-backend-ldap Authenticate via an external LDAP server

autocluster-consul Automatically create clusters using Consul

msg_store_eleveldb_index LevelDB storage for queue messages

presence-exchange Publishes messages upon binding changes

rabbitmq-topTop like view of RabbitMQ processes in the management UI

sharding Scale out RabbitMQ with automatic queue sharding

(and many more)

Page 25: Messaging Standards and Systems - AMQP & RabbitMQ

Message Routing

Page 26: Messaging Standards and Systems - AMQP & RabbitMQ

Built-In Exchange Types❖ Direct

String matching on the routing key

❖ FanoutNo routing key, messages delivered to all bound queues

❖ TopicPattern matching in the routing key

❖ HeadersNo routing key, value matching in the headers property

Page 27: Messaging Standards and Systems - AMQP & RabbitMQ

Topic Exchange Binding KeysRouting Key: namespace.delimited.keys

#Receive all messages

namespace.# Receive all messages in namespace

namespace.delimited.* Receive all namespace.delimited messages

namespace.*.keys Receive all namespace messages ending with keys

Page 28: Messaging Standards and Systems - AMQP & RabbitMQ

Exchange to Exchange Binding

X

X

X

Queue

Queue

Queue

Queue

Queue

Queue

Messages published into a topic exchange are routed to two other exchanges and a queue

Messages published into the consistent-hashing exchange are distributed amongst three queues

Messages published into a direct exchange routed to two queues

P

Page 29: Messaging Standards and Systems - AMQP & RabbitMQ

Alternate Exchanges

P X

UnroutableMessagesX

Unroutable messages are sent to an alternate exchange

Page 30: Messaging Standards and Systems - AMQP & RabbitMQ

Dead Letter Exchanges

P X

Reject Queue

Queue C

X

Consumer rejects the message without requeue

Dead-Letter Exchange routes the message

Page 31: Messaging Standards and Systems - AMQP & RabbitMQ

Exchange PluginsConsistent Hashing Distribute messages via hashed value of routing key

Event Publishes messages on AMQP events such as queue creation

Random Distribute messages across all bound queues randomly

PostgreSQL LISTEN Subscribes to and publishes PostgreSQL notifications

Recent History Sends the last n messages to any newly bound queue

Reverse TopicAllows for routing patterns at publish time and not via binding

Riak StorageStores messages published through the exchange into Riak

Script Exchange Calls out to external scripts for message routing

Page 32: Messaging Standards and Systems - AMQP & RabbitMQ

Performance Considerations

Page 33: Messaging Standards and Systems - AMQP & RabbitMQ

Publishing Performance Scale

Persisted Messages

Transactions

HA Queues

Publisher confirms

No guarantees

Notification on failure

Alternate exchanges

HA Queues w/ Transactions

Page 34: Messaging Standards and Systems - AMQP & RabbitMQ

Persisted Messages IO Model

Disk

Queue

Message Queue

Queue

Messages with delivery-mode 2 sent into RabbitMQ are persisted to disk

Pointers to the message are placed in the queue data structures

When the message is no longer in a queue, it is removed from disk

Page 35: Messaging Standards and Systems - AMQP & RabbitMQ

HA Queues & Performance

Queue

Queue

Queue

P

RabbitMQ Cluster

A publisher sends to any node in a RabbitMQ cluster

The RabbitMQ servers in the cluster synchronize the state of the message in the queue

The message that was published is put in the queue and is stored on each server

Page 36: Messaging Standards and Systems - AMQP & RabbitMQ

Consumer Performance Scale

Getting Messages

Consuming and using Transactions

Consume with Acknowledgements

Consuming with "No Ack Mode" enabled

Consuming with Acknowledgements and QoS > 1

Page 37: Messaging Standards and Systems - AMQP & RabbitMQ

Operational Concerns

Page 38: Messaging Standards and Systems - AMQP & RabbitMQ

Trending & Monitoring

❖ RabbitMQ Management Plugin provides internal stats

❖ Queue depths, connection counts, throughput, memory usage, etc

❖ Monitor with common tools such as Nagios or Sensu to services such as Boundary and NewRelic

❖ Stream based monitoring with Riemann for anomaly detection

Page 39: Messaging Standards and Systems - AMQP & RabbitMQ

Thoughts on Configuration❖ Use configuration management!

❖ Even for exchanges, queues, and bindings if possible

❖ Helpful for disaster recovery

❖ Use RabbitMQ’s Policies when possible

❖ Exchanges and queues are immutable

❖ Deleting and redeclaring for changes can be disruptive

Page 40: Messaging Standards and Systems - AMQP & RabbitMQ

Sample Usage Patterns

Page 41: Messaging Standards and Systems - AMQP & RabbitMQ

Evolve Tightly-Coupled Applications

Application DB

Page 42: Messaging Standards and Systems - AMQP & RabbitMQ

Decoupling Database Writes

Application C DB

Page 43: Messaging Standards and Systems - AMQP & RabbitMQ

Listen for Database Notifications

X Queue

Page 44: Messaging Standards and Systems - AMQP & RabbitMQ

Multi-Purposed Messages

C

C

C

P

Queue

DB

Cloud

GraphiteQueue

Queue

X

Page 45: Messaging Standards and Systems - AMQP & RabbitMQ

RPC

C / PC / P

P / C

C / PResponse Queue

X

X

WorkerQueue

Front-end web app receives a request

Internet

App publishes RPC request

One of the subscribed worker consumers receives the RPC requestAfter processing the request,

the worker publishes the replyApp receives the reply from the response queue

App replies to client

Page 46: Messaging Standards and Systems - AMQP & RabbitMQ

Delayed Messages

WARNING: This pattern is a hack and can cause undesirable situations should the wait queue overwhelm the system!

P X

X

Wait Queue

Ready Queue C

The wait queue is declared with a message TTL and a dead-letter exchangeMessages that will be delivered

later are routed to a wait queue

When a message times out due to the TTL, it is published to the DLX

The internally re-published message is routed to the queue with active consumers

Page 47: Messaging Standards and Systems - AMQP & RabbitMQ

HA RabbitMQ with FederationUpstream Cluster A

Queue A

Queue B

Upstream Cluster B

Queue A

Queue B

Downstream Cluster A

Queue A

Queue B

Downstream Cluster B

Queue A

Queue B

HA Proxy 1

HA Proxy 2

P

P

P

C A

C B

Page 48: Messaging Standards and Systems - AMQP & RabbitMQ

Questions?