Download pdf - Distributed messaging with AMQP

Distributed MessagingAMQP, RabbitMQ (Quick demo)

Tuesday, 10 September 13

Event LoggingCurrent architecture & problems


Event Logging

ServerClient

Client

Client

MongoDB


Problem

ServerClient

Client

Client

80

110

65MongoDB


Problem

ServerClient

Client

Client

80

110

65

255

MongoDB


Problem

ServerClient

Client

Client

0

0

0

0

MongoDB


Problem

ServerClient

Client

Client

0

0

0

0

MongoDB


Possible Solutions

• Server tweaks: TCP conns, ulimit, etc

• Multi-master model

• Localised queue

• ... and quite a few more


Possible Solutions

• Server tweaks: TCP conns, ulimit, etc

• Multi-master model

• Localised queue

• ... and quite a few more


Localised Queues

• Fast (no TCP latency)

• Network partition tolerant

• Modular & scalable

• Reliable & Durable


Why Localise?

ServerClient

Client

Client

80

110

65MongoDB

Q

Q

Q

< 10


Why Localise?

ServerClient

Client

Client

80

110

65MongoDB

Q

Q

Q

0


Why Localise?

ServerClient

Client

Client

80

110

65MongoDB

Q

Q

Q

0


Why Localise?

ServerClient

Client

Client

80

110

65MongoDB

Q

Q

Q

< 10


Why Localise?

ServerClient

Client

Client

80

110

65MongoDB

Q

Q

Q

< 10

Client

Q


Queue Criteria

• Reliable

• Durable

• Scalable


AMQPAdvanced Message Queueing Protocol


Why AMQP?

• Stable and mature

• Designed for SX markets

• Widely supported (good interop)

• Modular, Fast & Flexible


Service ModelWire

AMQP Overview

AMQP

Advanced Message Queueing Protocol


Service Model Wire

AMQP Overview

AMQP


➡ Model/component spec

‣ Exchange

‣ Message Queue

‣ Binding

AMQ Model Wire

AMQP Overview

AMQP



‣ Exchange

‣ Message Queue

‣ Binding

AMQ Model Wire

AMQP Overview

AMQP

➡ Function layer

‣ Transactions, Exchange, Queues, etc

➡ Transport layer

‣ Protocol format

‣ Data rep/framing

‣ Multiplexing, heart-beating, etc


AMQ Model Wire

AMQP Overview

AMQP

➡ Function layer


➡ Transport layer

‣ Protocol format




‣ Exchange

‣ Message Queue

‣ Binding


ExchangeQueuesBindingAMQ Model

AMQ Model


QueuesExchange Binding

AMQ Model

AMQ Model


Virtual Host


AMQ Model

AMQ Model


Virtual Host


AMQ Model

AMQ Model

Publisher Consumer


Virtual Host

QueuesExchange

Binding

AMQ Model

Publisher Consumer


Virtual Host

QueuesExchange

Binding

AMQ Model

Router

Message

Publisher Consumer


Virtual Host

QueuesExchange

Binding

AMQ Model

Router

Message

Publisher Consumer


Virtual Host

QueuesExchange

Binding

AMQ Model

Router

Message

Publisher Consumer


Virtual Host

QueuesExchange

Binding

AMQ Model

Router

?Message

Publisher Consumer


Virtual Host

QueuesExchange

Binding

AMQ Model

Message

Router

?

Publisher Consumer


Virtual Host

QueuesExchange

Binding

AMQ Model

Message

Router

Publisher

➡ Queue: ‣ critical_error_queue

➡ Exchange: ‣ errors_exchange

➡ Filter:‣ *.error.critical

Consumer


Virtual Host

QueuesExchange

Binding

AMQ Model




Message

Router

Publisher Consumer



AMQP Service Model



AMQP Service Model

‣ routes messages based on criteria‣ doesn’t store messages‣ can inspect message content‣ can be created at runtime consumers



AMQP Service Model

‣ store messages‣ named‣ bound-able to exchange‣ criteria‣ can be created at runtime by consumers




AMQP Service Model

‣ creates a relationship between queues and exchanges‣ contains criteria and properties‣ can be created at runtime by consumers





AMQP Service Model






AMQP Service Model

Publisher Consumer






AMQP Service Model

Publisher Consumer




‣ can create exchanges and queues

‣ can create exchanges and queues


Exchange

AMQP Service Model


FanoutTopicHeadersSystem

Exchange

AMQP Service Model

Types(routing algo)


Fanout Topic Headers System

Exchange

AMQP Service Model

Types(routing algo)

Direct


Fanout Topic Headers System

Exchange

AMQP Service Model

Types(routing algo)

Direct


Exchange

AMQP Service ModelDirect Type


Exchange‘events’





Message

Message

Message

event.user.click

event.user.view

event.user.share




Message

Message

Message

event.user.click

event.user.view

event.user.share

user_click_queue



Binding

AMQP Service Model

➡Queue: ‣ user_click_queue

Direct Type

Message

Message

Message

event.user.click

event.user.view

event.user.share

user_click_queue



Binding

AMQP Service Model


➡Exchange: ‣ events

Direct Type

Message

Message

Message

event.user.click

event.user.view

event.user.share

user_click_queue



Binding

AMQP Service Model



➡Routing Key:‣ event.user.click

Direct Type

Message

Message

Message

event.user.click

event.user.view

event.user.share

user_click_queue



Binding

AMQP Service Model



➡Routing Key:‣ event.user.click

Direct Type

Message

Message

Message

event.user.click

event.user.view

event.user.share

PHP Script

PHP Script

PHP Script

user_click_queue




Message

Message

Message

event.user.click

event.user.view

event.user.share

PHP Script

PHP Script

PHP Script

user_click_queue

ConsumerPublisher/Producer




Message

Message

Message

event.user.click

event.user.view

event.user.share

user_click_queue

user_view_queue

user_share_queue




Message

Message

Message

event.user.click

event.user.view

event.user.share

user_click_queue

user_view_queue

user_share_queue

Message

Message

Message



AMQP Service ModelWhat if...

Message

Message

Message

event.user.click

event.user.view

event.user.share

user_interaction_queue

user_shares_queue





user_shares_queue

Message

Message

Message

event.user.click

event.user.view

event.user.share




Message

Message

Message

event.user.click

event.user.view

event.user.share


user_shares_queue



AMQP Service ModelTopic Exchange


Binding

Message

Message

Message

event.user.click

event.user.view

event.user.share





Binding

➡Queue: ‣ user_interaction_queue


➡Routing Key:‣ event.user.*

Message

Message

Message

event.user.click

event.user.view

event.user.share





Binding



➡Routing Key:‣ event.user.*

Message

Message

Message

event.user.click

event.user.view

event.user.share

PHP Script

PHP Script

PHP Script




user_shares_queue

Binding

Message

Message

Message

event.user.click

event.user.view

event.user.share




user_shares_queue

Binding



➡Routing Key:‣ event.user.share

Message

Message

Message

event.user.click

event.user.view

event.user.share




user_shares_queue

Binding



➡Routing Key:‣ event.user.share

Message

Message

Message

event.user.click

event.user.view

event.user.share

PHP Script

PHP Script

PHP Script


Message



Message

Message

Message

event.user.click

event.user.view

event.user.share


user_shares_queue

Message

Message

event.user.*

event.user.share


Message



Message

Message

Message

event.user.click

event.user.view

event.user.share


user_shares_queue

Message

Message

Can be done at runtime

event.user.*

event.user.share


Message



Message

Message

Message

event.user.click

event.user.view

event.user.share


user_shares_queue

Message

Message

Click vs Share Conversion?

event.user.*

event.user.share


Message



Message

Message

Message

event.user.click

event.user.view

event.user.share


user_shares_queue

Message

Message

Click vs Share Conversion?

event.user.*

event.user.share

Messageclick_share_queue

Message

event.user.shareevent.user.click


Message



Message

Message

Message

event.user.click

event.user.view

event.user.share


user_shares_queueMessage

live_view_queue

event.user.*

event.user.share

*.*.view

Message

Messageevent.staff.view

MessageMessage

Staff vs user views?


AMQ Model Wire

AMQP Overview

AMQP

➡ Function layer


➡ Transport layer

‣ Protocol format




‣ Exchange

‣ Message Queue

‣ Binding



‣ Exchange

‣ Message Queue

‣ Binding

AMQ Model Wire

AMQP Overview

AMQP

➡ Function layer


➡ Transport layer

‣ Protocol format




Wire

Wire Protocol


Wire

Wire Protocol

‣ Datagram specification‣ Protocol header‣ Payload format‣ Data fields specification‣ Content framing‣ Heartbeat frames

‣ Channel multiplexing‣ Visibility guarantee‣ Channel closure‣ Content synchronisation‣ Content ordering


Wire ProtocolMethod Payloads


Wire ProtocolMethod Payloads


Virtual Host

QueuesExchange

Binding

Summary




Message

Router

Publisher Consumer


Revisiting event loggingA proposed solution


Event Logging Revisited

Client

Client

Client

MongoDB



Client

Client

ClientMQ

MQ

MQ

MongoDB


MongoDB


MQClient

Client

Client

80

110

65

MQ

MQ

MQ


MongoDB


MQClient

Client

Client

80

110

65

MQ

MQ

MQ


MongoDB


MQClient

Client

Client

80

110

65

MQ

MQ

MQ

MQ


MongoDB


MQ

Client

Client

Client

80

110

65

MQ

MQ

MQ

MQ


Message



Message

Message

Message

event.user.click

event.user.view

event.user.share


user_shares_queueMessage

live_view_queue

event.user.*

event.user.share

*.*.view

Message

Messageevent.staff.view

MessageMessage

Remember this?


MongoDB


MQClient

Client

Client

80

110

65

MQ

MQ

MQ

ReportingSQL

Notification


RabbitMQAn AMQP Implementation


Why RabbitMQ?

• Mature (born 2006)

• Acquired by SpringSource (now part of VMWare) in 2010

• Used by companies like Nokia, Google, Mozilla, vFabric, SecondLife, OpenStack


Overview

• Written in Erlang/OTP

• Supports other protocols too - STOMP, MQTT

• Many library supports - Python, PHP, Ruby, .NET

• RESTful API is available


Hands on Demo


Q&A


Coming soon...RabbitMQ in Depth (sort of)
