Click here to load reader
Upload
javier-arias-losada
View
805
Download
4
Tags:
Embed Size (px)
DESCRIPTION
Introduction to RabbitMQ, Amqp and some messaging patterns. Easy to follow code examples step by step provided. Sample code and escenarios can be found at: https://gist.github.com/javierarilos/9348168
Citation preview
intro and messaging patternsjavier arias losada @javier_arilos
amqp and rabbitmq
introduction to messaging
messaging
● provides asynchronous communications● loosely coupled modules / processes● MoM => Message Oriented Middleware
amqp concepts (i)
amqp is a message oriented wire-level protocol
● message broker: receives and dispatches msgs using AMQP
● connection physical connection (eg: tcp/ip)● channel: allows n clients over one connection
consumerproducer
producer consumer
amqp concepts (ii)
consumerproducer
producer consumer
Clients produce and consume messages.
Exchanges Route and filter messages to queues: binding rules, direct, one-to-one, fanout, topic, headers
Queues buffer messages between producers and consumers
Messages are always: >> sent to exchanges>> consumed from queues
escenario 1
● producing and consuming, simplest routing
examples using Python, pika and rabbitmq, everything should apply to other
languages, libraries and amqp brokers
‘important’ messages must be sent to queue ‘important-jobs’
consumerproducer
source code for the example: https://gist.github.com/javierarilos/9348168
step 1- connect & channel setup
from pika import BlockingConnection, ConnectionParameters conn = BlockingConnection(ConnectionParameters('localhost'))ch = conn.channel()
consumerproducer
Same code for consumer and producer
step 2- declare exchange
Declare an exchange, important parameters:
● name: if exists, nothing is done● type: direct, fanout, pub-sub, headers● durable: must exchange be kept between server restarts?● autodelete: must exchange be deleted when not in use?● internal: is it for internal routing use, or public to clients?
ch.exchange_declare(exchange='important', type='direct')
consumerproducerexchange: importanttype: direct
step 3- declare queue
Declare a queue, important parameters:
● name: if exists, nothing is done● durable: must queue declaration be kept between server restarts?● exclusive: Is this the only connection that can consume from the queue?● autodelete: must queue be deleted when not in use?
ch.queue_declare(queue='important-jobs')
consumerproducerexchange: importanttype: direct
queue: important-jobs
step 4- bind queue and exchange
ch.queue_bind(exchange='important', queue='important-jobs', routing_key='important')
consumerproducerexchange: importanttype: direct
queue: important-jobs
routing_key: important
Binding a queue and exchange:
● establishes a route (exchange => queue)● based on a criteria (exchange type + routing key)
Here: when producer sends a message to ‘important’ exchange with routing key ‘important’, message will be forwarded to queue ‘important-jobs’.
step 5- send the message
ch.basic_publish(exchange='important', routing_key='important', body='new important task')
consumerproducerexchange: importanttype: direct
queue: important-jobs
routing_key: important
Bonus track: default exchange, forwards to a queue, without any exchange declaration:default exchange’s name is empty string = ‘’routing_key is the queue name ‘important-jobs’
ch.basic_publish(exchange=’’, routing_key='important-jobs', body=’def exch important task')
step 6- consume the message
method_frame, header_frame, body = ch.basic_get('important-jobs')print body ch.basic_ack(method_frame.delivery_tag)
consumerproducerexchange: importanttype: direct
queue: important-jobs
routing_key: important
Messages must be acknowledged
escenario 2
● default exchange in more detail● message to two queues, depending on routing key
‘important’ messages must be sent to queues ‘important-jobs’ and ‘traces’
consumerproducer
consumer
step 1- create & bind new queue
ch.queue_declare(queue='traces')ch.queue_bind(exchange='important', queue='traces', routing_key='important')ch.basic_publish(exchange='important', routing_key='important', body='[another task to be handled]')
consumerproducerexchange: importanttype: direct
queue: important-jobs
routing_key: important
consumerrouting_key: important
queue: traces
step 2- consuming from both
consumerproducerexchange: importanttype: direct
queue: important-jobs
consumerrouting_key: important
queue: traces
routing_key: important
● Binding the new ‘traces’ queue to existing ‘important’ exchange does not affect publishing code.
● Consumers and queues may be added dynamically without affecting the producer.
escenario 3● 2 x (binding + routing-key) => to 1 queue● exchange to exchange binding● headers exchange
‘customer’ messages to ‘important’ exchange must be sent to different queues depending on the operation to perform (‘signup’, ‘update’) and to ‘traces’
traces consumerproducer
update consumer
signup consumer
step 1- bind traces, declare q’s
routing_key: customer traces
consumerproducer
update consumer
signup consumer
exchange: importanttype: direct
queue: traces
queue: signup
queue: update
‘traces’ queue is bound to ‘important’ exch with two routing keys: ‘customer’ and ‘important’
ch.queue_bind(exchange='important', queue='traces', routing_key='customer')ch.queue_declare(queue='signup')ch.queue_declare(queue='update')
exchange: importanttype: direct
traces consumerproducer
update consumer
signup consumer
queue: traces
queue: signup
queue: update
exchange: customertype: headers
routing_key: customer
step 2- customer exchange & bind
ch.exchange_declare(exchange='customer', type='headers')ch.exchange_bind(source='important', destination='customer', routing_key='customer')ch.queue_bind(exchange='customer', queue='signup', arguments={'operation': 'signup', 'x-match':'any'})ch.queue_bind(exchange='customer', queue='update', arguments={'operation': 'update', 'x-match':'any'})
step 3- customer signup message
ch.basic_publish(exchange='important', routing_key='customer', body='cust num=25', properties=BasicProperties(headers=s{'operation': 'signup'})) method_frame, header_frame, msg = ch.basic_get('signup')print "msg received from queue 'signup' : ", msgch.basic_ack(method_frame.delivery_tag)
routing_key: customer traces
consumerproducer
update consumer
signup consumer
exchange: importanttype: direct
queue: traces
queue: signup
queue: update
exchange: customertype: headers
msg also routed to traces queue
bonus: topic exchanges
traces consumer
producer
customer consumer
signup consumerexchange: operations
type: topic
queue: traces
queue: signup
queue: update
r_k: #
r_k: *.signup
r_k: customer.*
topic exchanges allow to route messages based on topics. Examples from escenario 3:
○ Producer sends with routing keys: ‘customer.signup’, ‘customer.update’○ ‘traces’ consumer subscribes to ‘#’ that means all routing keys○ ‘signup’ customer consumer subscribes to ‘customer.signup’○ Consumer wanting all customer operations: ‘customer.*’○ Consumer wanting all signup operations: *.signup’
escenario 4● DeadLetter exchanges in RabbitMQ
What will we do with a message RabbitMQ cannot deliver?
(on client rejection, timeout, queue length exceeded)
By default those messages are dropped, we want not to lose them
consumerproducer
step 1- create rejected-jobs
ch.exchange_declare(exchange='rejected-jobs', type='direct')
ch.queue_declare(queue='rejected-jobs')
ch.queue_bind(exchange='rejected-jobs', queue='rejected-jobs', routing_key='important')
consumerproducer
queue: rejected-jobsexchange: rejected-jobstype: direct
step 2- deadlettr important-jobs
ch.queue_delete('important-jobs')ch.queue_declare(queue='important-jobs', arguments={'x-dead-letter-exchange': 'rejected-jobs'})ch.queue_bind(exchange='important', queue='important-jobs', routing_key='important')
consumerproducer
queue: rejected-jobsexchange: rejected-jobstype: direct
exchange: importanttype: direct
routing_key: important
x-dead-letter-exchange: rejected-jobs
step 3- consumer rejects job
ch.basic_publish(exchange='important', routing_key='important', body='[unparseable message]')
method_frame, header_frame, important_job = ch.basic_get('important-jobs')ch.basic_reject(method_frame.delivery_tag, requeue=False)
method_frame, header_frame, rejected_job = ch.basic_get('rejected-jobs')ch.basic_ack(method_frame.delivery_tag)
consumerproducer
queue: rejected-jobsexchange: rejected-jobstype: direct
exchange: importanttype: direct
routing_key: important
x-dead-letter-exchange: rejected-jobs
What are you waiting to start working with RabbitMQ?
It’s fun!
Questions?
Thank you for attending.