Distributed messaging with AMQP

Distributed MessagingAMQP, RabbitMQ (Quick demo)

Tuesday, 10 September 13

Event LoggingCurrent architecture & problems

Event Logging

ServerClient

Client

MongoDB

Problem

ServerClient

Client

65MongoDB

Problem

ServerClient

Client

MongoDB

Problem

ServerClient

Client

MongoDB

Problem

ServerClient

Client

MongoDB

Possible Solutions

• Server tweaks: TCP conns, ulimit, etc

• Multi-master model

• Localised queue

• ... and quite a few more

Possible Solutions

• Server tweaks: TCP conns, ulimit, etc

• Multi-master model

• Localised queue

• ... and quite a few more

Localised Queues

• Fast (no TCP latency)

• Network partition tolerant

• Modular & scalable

• Reliable & Durable

Why Localise?

ServerClient

Client

65MongoDB

Why Localise?

ServerClient

Client

65MongoDB

Why Localise?

ServerClient

Client

65MongoDB

Why Localise?

ServerClient

Client

65MongoDB

Why Localise?

ServerClient

Client

65MongoDB

Client

Queue Criteria

• Reliable

• Durable

• Scalable

AMQPAdvanced Message Queueing Protocol

Why AMQP?

• Stable and mature

• Designed for SX markets

• Widely supported (good interop)

• Modular, Fast & Flexible

Service ModelWire

AMQP Overview

Advanced Message Queueing Protocol

Service Model Wire

AMQP Overview

➡ Model/component spec

‣ Exchange

‣ Message Queue

‣ Binding

AMQ Model Wire

AMQP Overview

‣ Exchange

‣ Message Queue

‣ Binding

AMQ Model Wire

AMQP Overview

➡ Function layer

‣ Transactions, Exchange, Queues, etc

➡ Transport layer

‣ Protocol format

‣ Data rep/framing

‣ Multiplexing, heart-beating, etc

AMQ Model Wire

AMQP Overview

➡ Function layer

➡ Transport layer

‣ Protocol format

‣ Exchange

‣ Message Queue

‣ Binding

ExchangeQueuesBindingAMQ Model

AMQ Model

QueuesExchange Binding

AMQ Model

Virtual Host

AMQ Model

Virtual Host

AMQ Model

Publisher Consumer

Virtual Host

QueuesExchange

Binding

AMQ Model

Publisher Consumer

Virtual Host

QueuesExchange

Binding

AMQ Model

Router

Message

Publisher Consumer

Virtual Host

QueuesExchange

Binding

AMQ Model

Router

Message

Publisher Consumer

Virtual Host

QueuesExchange

Binding

AMQ Model

Router

Message

Publisher Consumer

Virtual Host

QueuesExchange

Binding

AMQ Model

Router

?Message

Publisher Consumer

Virtual Host

QueuesExchange

Binding

AMQ Model

Message

Router

Publisher Consumer

Virtual Host

QueuesExchange

Binding

AMQ Model

Message

Router

Publisher

➡ Queue: ‣ critical_error_queue

➡ Exchange: ‣ errors_exchange

➡ Filter:‣ *.error.critical

Consumer

Virtual Host

QueuesExchange

Binding

AMQ Model

Message

Router

Publisher Consumer

AMQP Service Model

‣ routes messages based on criteria‣ doesn’t store messages‣ can inspect message content‣ can be created at runtime consumers

AMQP Service Model

‣ store messages‣ named‣ bound-able to exchange‣ criteria‣ can be created at runtime by consumers

AMQP Service Model

‣ creates a relationship between queues and exchanges‣ contains criteria and properties‣ can be created at runtime by consumers

AMQP Service Model

Publisher Consumer

AMQP Service Model

Publisher Consumer

‣ can create exchanges and queues

Exchange

AMQP Service Model

FanoutTopicHeadersSystem

Exchange

AMQP Service Model

Types(routing algo)

Fanout Topic Headers System

Exchange

AMQP Service Model

Types(routing algo)

Direct

Fanout Topic Headers System

Exchange

AMQP Service Model

Types(routing algo)

Direct

Exchange

AMQP Service ModelDirect Type

Exchange‘events’

Message

event.user.click

event.user.view

event.user.share

Message

event.user.click

event.user.view

event.user.share

user_click_queue

Binding

AMQP Service Model

➡Queue: ‣ user_click_queue

Direct Type

Message

event.user.click

event.user.view

event.user.share

user_click_queue

Binding

AMQP Service Model

➡Exchange: ‣ events

Direct Type

Message

event.user.click

event.user.view

event.user.share

user_click_queue

Binding

AMQP Service Model

➡Routing Key:‣ event.user.click

Direct Type

Message

event.user.click

event.user.view

event.user.share

user_click_queue

Binding

AMQP Service Model

➡Routing Key:‣ event.user.click

Direct Type

Message

event.user.click

event.user.view

event.user.share

PHP Script

user_click_queue

Message

event.user.click

event.user.view

event.user.share

PHP Script

user_click_queue

ConsumerPublisher/Producer

Message

event.user.click

event.user.view

event.user.share

user_click_queue

user_view_queue

user_share_queue

Message

event.user.click

event.user.view

event.user.share

user_click_queue

user_view_queue

user_share_queue

Message

AMQP Service ModelWhat if...

Message

event.user.click

event.user.view

event.user.share

user_interaction_queue

user_shares_queue

Message

event.user.click

event.user.view

event.user.share

Message

event.user.click

event.user.view

event.user.share

user_shares_queue

AMQP Service ModelTopic Exchange

Binding

Message

event.user.click

event.user.view

event.user.share

Binding

➡Queue: ‣ user_interaction_queue

➡Routing Key:‣ event.user.*

Message

event.user.click

event.user.view

event.user.share

Binding

➡Routing Key:‣ event.user.*

Message

event.user.click

event.user.view

event.user.share

PHP Script

user_shares_queue

Binding

Message

event.user.click

event.user.view

event.user.share

user_shares_queue

Binding

➡Routing Key:‣ event.user.share

Message

event.user.click

event.user.view

event.user.share

user_shares_queue

Binding

➡Routing Key:‣ event.user.share

Message

event.user.click

event.user.view

event.user.share

PHP Script

Message

event.user.click

event.user.view

event.user.share

user_shares_queue

Message

event.user.*

event.user.share

Message

event.user.click

event.user.view

event.user.share

user_shares_queue

Message

Can be done at runtime

event.user.*

event.user.share

Message

event.user.click

event.user.view

event.user.share

user_shares_queue

Message

Click vs Share Conversion?

event.user.*

event.user.share

Message

event.user.click

event.user.view

event.user.share

user_shares_queue

Message

Click vs Share Conversion?

event.user.*

event.user.share

Messageclick_share_queue

Message

event.user.shareevent.user.click

Message

event.user.click

event.user.view

event.user.share

user_shares_queueMessage

live_view_queue

event.user.*

event.user.share

*.*.view

Message

Messageevent.staff.view

MessageMessage

Staff vs user views?

AMQ Model Wire

AMQP Overview

➡ Function layer

➡ Transport layer

‣ Protocol format

‣ Exchange

‣ Message Queue

‣ Binding

‣ Exchange

‣ Message Queue

‣ Binding

AMQ Model Wire

AMQP Overview

➡ Function layer

➡ Transport layer

‣ Protocol format

Wire Protocol

‣ Datagram specification‣ Protocol header‣ Payload format‣ Data fields specification‣ Content framing‣ Heartbeat frames

‣ Channel multiplexing‣ Visibility guarantee‣ Channel closure‣ Content synchronisation‣ Content ordering

Wire ProtocolMethod Payloads

Virtual Host

QueuesExchange

Binding

Summary

Message

Router

Publisher Consumer

Revisiting event loggingA proposed solution

Event Logging Revisited

Client

MongoDB

Client

ClientMQ

MongoDB

MQClient

Client

MongoDB

MQClient

Client

MongoDB

MQClient

Client

MongoDB

Client

Message

event.user.click

event.user.view

event.user.share

user_shares_queueMessage

live_view_queue

event.user.*

event.user.share

*.*.view

Message

Messageevent.staff.view

MessageMessage

Remember this?

MongoDB

MQClient

Client

ReportingSQL

Notification

RabbitMQAn AMQP Implementation

Why RabbitMQ?

• Mature (born 2006)

• Acquired by SpringSource (now part of VMWare) in 2010

• Used by companies like Nokia, Google, Mozilla, vFabric, SecondLife, OpenStack

Overview

• Written in Erlang/OTP

• Supports other protocols too - STOMP, MQTT

• Many library supports - Python, PHP, Ruby, .NET

• RESTful API is available

Hands on Demo

Coming soon...RabbitMQ in Depth (sort of)

Distributed messaging with AMQP

Technology

The Future of Messaging: RabbitMQ and AMQP

Hybrid Messaging with MQ Light, MQ's Beta Support for AMQP, and IBM Bluemix

Next-Generation AMQP Messaging Performance, Architectures ... · PDF fileNext-Generation AMQP Messaging Performance, Architectures, and Ecosystems with Red Hat ... -- your code/logic

B - AMQP Programming Guide · Eurex Clearing Eurex Clearing Messaging Interfaces Connectivity B: AMQP Programming Guide

Website Monitoring with Distributed Messages/Tasks Processing (AMQP & RabbitMQ) on Django

Distributed messaging through Kafka

Rabbitmq, amqp Intro - Messaging Patterns

Spring AMQP · The Spring AMQP project applies core Spring concepts to the development of AMQP-based messaging solutions. We provide a “template” as a high-level abstraction for

AMQP Advanced Message Queuing Protocol - RabbitMQ - Messaging that

Kafka: a Distributed Messaging System for Log Processing

AMQP Setup & Internals · 1.1.4 AMQP The Advanced Message Queuing Protocol (AMQP) constitutes the preferred transport layer for delivering messages. AMQP is an open standard with

A Study on IoT Messaging Protocols and it’s · 2019-03-20 · AMQP is different from MQTT based on their components, AMQP broker is composed of two components, namely ex-change

Architecture | The Future of Messaging: RabbitMQ and AMQP | Eberhard Wolff

Cisco Voice Messaging€¦ · messaging systems share a messaging backbone that spans all locations. Message delivery from the distributed messaging systems occurs via the messaging

Www.cs.wisc.edu/Condor Messaging, AMQP, and Condor Huh? What? Why? Todd Tannenbaum, UW-Madison (with much thanks to John O’Hara!)

Messaging Technologies for the Industrial Internet and …€¦ · Messaging Technologies for the Industrial Internet and ... A Comparison Between DDS, AMQP, MQTT, JMS, ... high performance

Using AMQP, and RHM (messaging) to solve real · PDF fileUsing AMQP, and RHM (messaging) ... before they can start, these include routing, replay, IVQ, LVQ ... (there is more than

Messaging with amqp and rabbitmq

(Implemented in C++) AMQP Messaging Broker · Overriding the default message group identifier for the broker ..... 69. vii Introduction Qpid provides two AMQP messaging brokers: •

Messaging - zeromqzeromq.wdfiles.com/local--files/area:whitepapers/messaging-2010-02... · Why bother? We've spent years making messaging Working on protocols like AMQP and RestMS