Distributed MessagingAMQP, RabbitMQ (Quick demo)
Tuesday, 10 September 13
Event LoggingCurrent architecture & problems
Tuesday, 10 September 13
Event Logging
ServerClient
Client
Client
MongoDB
Tuesday, 10 September 13
Problem
ServerClient
Client
Client
80
110
65MongoDB
Tuesday, 10 September 13
Problem
ServerClient
Client
Client
80
110
65
255
MongoDB
Tuesday, 10 September 13
Problem
ServerClient
Client
Client
0
0
0
0
MongoDB
Tuesday, 10 September 13
Problem
ServerClient
Client
Client
0
0
0
0
MongoDB
Tuesday, 10 September 13
Possible Solutions
• Server tweaks: TCP conns, ulimit, etc
• Multi-master model
• Localised queue
• ... and quite a few more
Tuesday, 10 September 13
Possible Solutions
• Server tweaks: TCP conns, ulimit, etc
• Multi-master model
• Localised queue
• ... and quite a few more
Tuesday, 10 September 13
Localised Queues
• Fast (no TCP latency)
• Network partition tolerant
• Modular & scalable
• Reliable & Durable
Tuesday, 10 September 13
Why Localise?
ServerClient
Client
Client
80
110
65MongoDB
Q
Q
Q
< 10
Tuesday, 10 September 13
Why Localise?
ServerClient
Client
Client
80
110
65MongoDB
Q
Q
Q
0
Tuesday, 10 September 13
Why Localise?
ServerClient
Client
Client
80
110
65MongoDB
Q
Q
Q
0
Tuesday, 10 September 13
Why Localise?
ServerClient
Client
Client
80
110
65MongoDB
Q
Q
Q
< 10
Tuesday, 10 September 13
Why Localise?
ServerClient
Client
Client
80
110
65MongoDB
Q
Q
Q
< 10
Client
Q
Tuesday, 10 September 13
Queue Criteria
• Reliable
• Durable
• Scalable
Tuesday, 10 September 13
AMQPAdvanced Message Queueing Protocol
Tuesday, 10 September 13
Why AMQP?
• Stable and mature
• Designed for SX markets
• Widely supported (good interop)
• Modular, Fast & Flexible
Tuesday, 10 September 13
Service ModelWire
AMQP Overview
AMQP
Advanced Message Queueing Protocol
Tuesday, 10 September 13
Service Model Wire
AMQP Overview
AMQP
Tuesday, 10 September 13
➡ Model/component spec
‣ Exchange
‣ Message Queue
‣ Binding
AMQ Model Wire
AMQP Overview
AMQP
Tuesday, 10 September 13
➡ Model/component spec
‣ Exchange
‣ Message Queue
‣ Binding
AMQ Model Wire
AMQP Overview
AMQP
➡ Function layer
‣ Transactions, Exchange, Queues, etc
➡ Transport layer
‣ Protocol format
‣ Data rep/framing
‣ Multiplexing, heart-beating, etc
Tuesday, 10 September 13
AMQ Model Wire
AMQP Overview
AMQP
➡ Function layer
‣ Transactions, Exchange, Queues, etc
➡ Transport layer
‣ Protocol format
‣ Data rep/framing
‣ Multiplexing, heart-beating, etc
➡ Model/component spec
‣ Exchange
‣ Message Queue
‣ Binding
Tuesday, 10 September 13
ExchangeQueuesBindingAMQ Model
AMQ Model
Tuesday, 10 September 13
QueuesExchange Binding
AMQ Model
AMQ Model
Tuesday, 10 September 13
Virtual Host
QueuesExchange Binding
AMQ Model
AMQ Model
Tuesday, 10 September 13
Virtual Host
QueuesExchange Binding
AMQ Model
AMQ Model
Publisher Consumer
Tuesday, 10 September 13
Virtual Host
QueuesExchange
Binding
AMQ Model
Publisher Consumer
Tuesday, 10 September 13
Virtual Host
QueuesExchange
Binding
AMQ Model
Router
Message
Publisher Consumer
Tuesday, 10 September 13
Virtual Host
QueuesExchange
Binding
AMQ Model
Router
Message
Publisher Consumer
Tuesday, 10 September 13
Virtual Host
QueuesExchange
Binding
AMQ Model
Router
Message
Publisher Consumer
Tuesday, 10 September 13
Virtual Host
QueuesExchange
Binding
AMQ Model
Router
?Message
Publisher Consumer
Tuesday, 10 September 13
Virtual Host
QueuesExchange
Binding
AMQ Model
Message
Router
?
Publisher Consumer
Tuesday, 10 September 13
Virtual Host
QueuesExchange
Binding
AMQ Model
Message
Router
Publisher
➡ Queue: ‣ critical_error_queue
➡ Exchange: ‣ errors_exchange
➡ Filter:‣ *.error.critical
Consumer
Tuesday, 10 September 13
Virtual Host
QueuesExchange
Binding
AMQ Model
➡ Queue: ‣ critical_error_queue
➡ Exchange: ‣ errors_exchange
➡ Filter:‣ *.error.critical
Message
Router
Publisher Consumer
Tuesday, 10 September 13
QueuesExchange Binding
AMQP Service Model
Tuesday, 10 September 13
QueuesExchange Binding
AMQP Service Model
‣ routes messages based on criteria‣ doesn’t store messages‣ can inspect message content‣ can be created at runtime consumers
Tuesday, 10 September 13
QueuesExchange Binding
AMQP Service Model
‣ store messages‣ named‣ bound-able to exchange‣ criteria‣ can be created at runtime by consumers
‣ routes messages based on criteria‣ doesn’t store messages‣ can inspect message content‣ can be created at runtime consumers
Tuesday, 10 September 13
QueuesExchange Binding
AMQP Service Model
‣ creates a relationship between queues and exchanges‣ contains criteria and properties‣ can be created at runtime by consumers
‣ store messages‣ named‣ bound-able to exchange‣ criteria‣ can be created at runtime by consumers
‣ routes messages based on criteria‣ doesn’t store messages‣ can inspect message content‣ can be created at runtime consumers
Tuesday, 10 September 13
QueuesExchange Binding
AMQP Service Model
‣ creates a relationship between queues and exchanges‣ contains criteria and properties‣ can be created at runtime by consumers
‣ store messages‣ named‣ bound-able to exchange‣ criteria‣ can be created at runtime by consumers
‣ routes messages based on criteria‣ doesn’t store messages‣ can inspect message content‣ can be created at runtime consumers
Tuesday, 10 September 13
QueuesExchange Binding
AMQP Service Model
Publisher Consumer
‣ creates a relationship between queues and exchanges‣ contains criteria and properties‣ can be created at runtime by consumers
‣ store messages‣ named‣ bound-able to exchange‣ criteria‣ can be created at runtime by consumers
‣ routes messages based on criteria‣ doesn’t store messages‣ can inspect message content‣ can be created at runtime consumers
Tuesday, 10 September 13
QueuesExchange Binding
AMQP Service Model
Publisher Consumer
‣ creates a relationship between queues and exchanges‣ contains criteria and properties‣ can be created at runtime by consumers
‣ store messages‣ named‣ bound-able to exchange‣ criteria‣ can be created at runtime by consumers
‣ routes messages based on criteria‣ doesn’t store messages‣ can inspect message content‣ can be created at runtime consumers
‣ can create exchanges and queues
‣ can create exchanges and queues
Tuesday, 10 September 13
Exchange
AMQP Service Model
Tuesday, 10 September 13
FanoutTopicHeadersSystem
Exchange
AMQP Service Model
Types(routing algo)
Tuesday, 10 September 13
Fanout Topic Headers System
Exchange
AMQP Service Model
Types(routing algo)
Direct
Tuesday, 10 September 13
Fanout Topic Headers System
Exchange
AMQP Service Model
Types(routing algo)
Direct
Tuesday, 10 September 13
Exchange
AMQP Service ModelDirect Type
Tuesday, 10 September 13
Exchange‘events’
AMQP Service ModelDirect Type
Tuesday, 10 September 13
Exchange‘events’
AMQP Service ModelDirect Type
Message
Message
Message
event.user.click
event.user.view
event.user.share
Tuesday, 10 September 13
Exchange‘events’
AMQP Service ModelDirect Type
Message
Message
Message
event.user.click
event.user.view
event.user.share
user_click_queue
Tuesday, 10 September 13
Exchange‘events’
Binding
AMQP Service Model
➡Queue: ‣ user_click_queue
Direct Type
Message
Message
Message
event.user.click
event.user.view
event.user.share
user_click_queue
Tuesday, 10 September 13
Exchange‘events’
Binding
AMQP Service Model
➡Queue: ‣ user_click_queue
➡Exchange: ‣ events
Direct Type
Message
Message
Message
event.user.click
event.user.view
event.user.share
user_click_queue
Tuesday, 10 September 13
Exchange‘events’
Binding
AMQP Service Model
➡Queue: ‣ user_click_queue
➡Exchange: ‣ events
➡Routing Key:‣ event.user.click
Direct Type
Message
Message
Message
event.user.click
event.user.view
event.user.share
user_click_queue
Tuesday, 10 September 13
Exchange‘events’
Binding
AMQP Service Model
➡Queue: ‣ user_click_queue
➡Exchange: ‣ events
➡Routing Key:‣ event.user.click
Direct Type
Message
Message
Message
event.user.click
event.user.view
event.user.share
PHP Script
PHP Script
PHP Script
user_click_queue
Tuesday, 10 September 13
Exchange‘events’
AMQP Service ModelDirect Type
Message
Message
Message
event.user.click
event.user.view
event.user.share
PHP Script
PHP Script
PHP Script
user_click_queue
ConsumerPublisher/Producer
Tuesday, 10 September 13
Exchange‘events’
AMQP Service ModelDirect Type
Message
Message
Message
event.user.click
event.user.view
event.user.share
user_click_queue
user_view_queue
user_share_queue
Tuesday, 10 September 13
Exchange‘events’
AMQP Service ModelDirect Type
Message
Message
Message
event.user.click
event.user.view
event.user.share
user_click_queue
user_view_queue
user_share_queue
Message
Message
Message
Tuesday, 10 September 13
Exchange‘events’
AMQP Service ModelWhat if...
Message
Message
Message
event.user.click
event.user.view
event.user.share
user_interaction_queue
user_shares_queue
Tuesday, 10 September 13
Exchange‘events’
AMQP Service ModelWhat if...
user_interaction_queue
user_shares_queue
Message
Message
Message
event.user.click
event.user.view
event.user.share
Tuesday, 10 September 13
Exchange‘events’
AMQP Service ModelWhat if...
Message
Message
Message
event.user.click
event.user.view
event.user.share
user_interaction_queue
user_shares_queue
Tuesday, 10 September 13
Exchange‘events’
AMQP Service ModelTopic Exchange
user_interaction_queue
Binding
Message
Message
Message
event.user.click
event.user.view
event.user.share
Tuesday, 10 September 13
Exchange‘events’
AMQP Service ModelTopic Exchange
user_interaction_queue
Binding
➡Queue: ‣ user_interaction_queue
➡Exchange: ‣ events
➡Routing Key:‣ event.user.*
Message
Message
Message
event.user.click
event.user.view
event.user.share
Tuesday, 10 September 13
Exchange‘events’
AMQP Service ModelTopic Exchange
user_interaction_queue
Binding
➡Queue: ‣ user_interaction_queue
➡Exchange: ‣ events
➡Routing Key:‣ event.user.*
Message
Message
Message
event.user.click
event.user.view
event.user.share
PHP Script
PHP Script
PHP Script
Tuesday, 10 September 13
Exchange‘events’
AMQP Service ModelTopic Exchange
user_shares_queue
Binding
Message
Message
Message
event.user.click
event.user.view
event.user.share
Tuesday, 10 September 13
Exchange‘events’
AMQP Service ModelTopic Exchange
user_shares_queue
Binding
➡Queue: ‣ user_interaction_queue
➡Exchange: ‣ events
➡Routing Key:‣ event.user.share
Message
Message
Message
event.user.click
event.user.view
event.user.share
Tuesday, 10 September 13
Exchange‘events’
AMQP Service ModelTopic Exchange
user_shares_queue
Binding
➡Queue: ‣ user_interaction_queue
➡Exchange: ‣ events
➡Routing Key:‣ event.user.share
Message
Message
Message
event.user.click
event.user.view
event.user.share
PHP Script
PHP Script
PHP Script
Tuesday, 10 September 13
Message
Exchange‘events’
AMQP Service ModelTopic Exchange
Message
Message
Message
event.user.click
event.user.view
event.user.share
user_interaction_queue
user_shares_queue
Message
Message
event.user.*
event.user.share
Tuesday, 10 September 13
Message
Exchange‘events’
AMQP Service ModelTopic Exchange
Message
Message
Message
event.user.click
event.user.view
event.user.share
user_interaction_queue
user_shares_queue
Message
Message
Can be done at runtime
event.user.*
event.user.share
Tuesday, 10 September 13
Message
Exchange‘events’
AMQP Service ModelTopic Exchange
Message
Message
Message
event.user.click
event.user.view
event.user.share
user_interaction_queue
user_shares_queue
Message
Message
Click vs Share Conversion?
event.user.*
event.user.share
Tuesday, 10 September 13
Message
Exchange‘events’
AMQP Service ModelTopic Exchange
Message
Message
Message
event.user.click
event.user.view
event.user.share
user_interaction_queue
user_shares_queue
Message
Message
Click vs Share Conversion?
event.user.*
event.user.share
Messageclick_share_queue
Message
event.user.shareevent.user.click
Tuesday, 10 September 13
Message
Exchange‘events’
AMQP Service ModelTopic Exchange
Message
Message
Message
event.user.click
event.user.view
event.user.share
user_interaction_queue
user_shares_queueMessage
live_view_queue
event.user.*
event.user.share
*.*.view
Message
Messageevent.staff.view
MessageMessage
Staff vs user views?
Tuesday, 10 September 13
AMQ Model Wire
AMQP Overview
AMQP
➡ Function layer
‣ Transactions, Exchange, Queues, etc
➡ Transport layer
‣ Protocol format
‣ Data rep/framing
‣ Multiplexing, heart-beating, etc
➡ Model/component spec
‣ Exchange
‣ Message Queue
‣ Binding
Tuesday, 10 September 13
➡ Model/component spec
‣ Exchange
‣ Message Queue
‣ Binding
AMQ Model Wire
AMQP Overview
AMQP
➡ Function layer
‣ Transactions, Exchange, Queues, etc
➡ Transport layer
‣ Protocol format
‣ Data rep/framing
‣ Multiplexing, heart-beating, etc
Tuesday, 10 September 13
Wire
Wire Protocol
Tuesday, 10 September 13
Wire
Wire Protocol
‣ Datagram specification‣ Protocol header‣ Payload format‣ Data fields specification‣ Content framing‣ Heartbeat frames
‣ Channel multiplexing‣ Visibility guarantee‣ Channel closure‣ Content synchronisation‣ Content ordering
Tuesday, 10 September 13
Wire ProtocolMethod Payloads
Tuesday, 10 September 13
Wire ProtocolMethod Payloads
Tuesday, 10 September 13
Virtual Host
QueuesExchange
Binding
Summary
➡ Queue: ‣ critical_error_queue
➡ Exchange: ‣ errors_exchange
➡ Filter:‣ *.error.critical
Message
Router
Publisher Consumer
Tuesday, 10 September 13
Revisiting event loggingA proposed solution
Tuesday, 10 September 13
Event Logging Revisited
Client
Client
Client
MongoDB
Tuesday, 10 September 13
Event Logging Revisited
Client
Client
ClientMQ
MQ
MQ
MongoDB
Tuesday, 10 September 13
MongoDB
Event Logging Revisited
MQClient
Client
Client
80
110
65
MQ
MQ
MQ
Tuesday, 10 September 13
MongoDB
Event Logging Revisited
MQClient
Client
Client
80
110
65
MQ
MQ
MQ
Tuesday, 10 September 13
MongoDB
Event Logging Revisited
MQClient
Client
Client
80
110
65
MQ
MQ
MQ
MQ
Tuesday, 10 September 13
MongoDB
Event Logging Revisited
MQ
Client
Client
Client
80
110
65
MQ
MQ
MQ
MQ
Tuesday, 10 September 13
Message
Exchange‘events’
AMQP Service ModelTopic Exchange
Message
Message
Message
event.user.click
event.user.view
event.user.share
user_interaction_queue
user_shares_queueMessage
live_view_queue
event.user.*
event.user.share
*.*.view
Message
Messageevent.staff.view
MessageMessage
Remember this?
Tuesday, 10 September 13
MongoDB
Event Logging Revisited
MQClient
Client
Client
80
110
65
MQ
MQ
MQ
ReportingSQL
Notification
Tuesday, 10 September 13
RabbitMQAn AMQP Implementation
Tuesday, 10 September 13
Why RabbitMQ?
• Mature (born 2006)
• Acquired by SpringSource (now part of VMWare) in 2010
• Used by companies like Nokia, Google, Mozilla, vFabric, SecondLife, OpenStack
Tuesday, 10 September 13
Overview
• Written in Erlang/OTP
• Supports other protocols too - STOMP, MQTT
• Many library supports - Python, PHP, Ruby, .NET
• RESTful API is available
Tuesday, 10 September 13
Hands on Demo
Tuesday, 10 September 13
Q&A
Tuesday, 10 September 13
Coming soon...RabbitMQ in Depth (sort of)
Tuesday, 10 September 13