48
The Future of Messaging: RabbitMQ and AMQP Eberhard Wolff Architecture & Technology Manager adesso AG, Germany

The Future of Messaging: RabbitMQ and AMQP

Embed Size (px)

Citation preview

Page 1: The Future of Messaging: RabbitMQ and AMQP

The Future of Messaging: RabbitMQ and AMQP

Eberhard Wolff Architecture & Technology Manager

adesso AG, Germany

Page 2: The Future of Messaging: RabbitMQ and AMQP

About me •  Eberhard Wolff •  Architecture & Technology Manager at adesso •  adesso is a leading IT consultancy in Germany •  Speaker •  Author (i.e. first German Spring book) •  Blog: http://ewolff.com •  Twitter: @ewolff •  [email protected]

Page 3: The Future of Messaging: RabbitMQ and AMQP

Overview •  Why Messaging, AMQP and RabbitMQ

•  Basic AMQP

•  Advanced AMQP

•  Advanced Spring-AMQP

Page 4: The Future of Messaging: RabbitMQ and AMQP

RPC •  Predominant approach

– RMI, SOAP Web Services, CORBA, HttpInvoker, Burlap, Hessian

•  Calls remote methods with parameter •  …and waits for response •  Problems:

– Explicitly tells the server what to do i.e. tight coupling

– What about network failures? – What about long latencies?

Page 5: The Future of Messaging: RabbitMQ and AMQP

Why Messaging? •  Decoupling

– Data, no action i.e. receiver can react arbitrarily

– Asynchronous i.e. decoupled by time

•  Reliable – Message can be stored-and-

forwarded – Redelivery until message

processed •  Solves typical problems of

distributed systems

Component

Component Messages

Page 6: The Future of Messaging: RabbitMQ and AMQP

Why Messaging? •  But: Requires different architecture •  Very different from calling remote methods •  Asynchronous

•  AJAX has the same model

•  See for example “Patterns of Enterprise Integration”

Page 7: The Future of Messaging: RabbitMQ and AMQP

Why AMQP? •  Open standard protocol •  Standard wire protocol •  i.e. just one client library – no matter which

implementation you are using •  Less vendor lock in •  Efficient

– Binary wire protocol •  Support in all major languages •  Supported on most OS platforms

Page 8: The Future of Messaging: RabbitMQ and AMQP

What about JMS? •  JMS has been the default for Java messaging

system for 10+ years •  But:

•  Only standardized on the API level •  Less flexible than AMQP

•  Mapping AMQP/JMS is being defined •  There is

git://github.com/pieterh/openamq-jms.git

Page 9: The Future of Messaging: RabbitMQ and AMQP

Why Rabbit? •  Because it has a kewl name •  Numerous protocols supported •  Most popular choice on EC2 •  Foundation for demanding systems e.g.

NASA’s cloud initiative Nebula •  Implemented in Erlang •  Clustering built in •  Currently in 2.4.1 •  Supports AMQP 0.8, 0.9, 0.9.1

Page 10: The Future of Messaging: RabbitMQ and AMQP

Broad Support in RabbitMQ

Page 11: The Future of Messaging: RabbitMQ and AMQP

Broad Support in the JVM Space •  Grails Plug In •  Java Client •  Scala / Lift support

•  We will discuss Spring support in detail •  Spring AMQP project 1.0.0.RC1 •  http://www.springsource.org/spring-amqp

Page 12: The Future of Messaging: RabbitMQ and AMQP

Why Erlang? •  Originally designed for telephone

switches by Ericsson •  Much easier to develop scalable and fault

tolerant systems (by factors) •  See Motorola presentation:

http://www.slideshare.net/Arbow/comparing-cpp-and-erlang-for-motorola-telecoms-software

•  Good tool for reliable and scalable systems

Page 13: The Future of Messaging: RabbitMQ and AMQP

Erlang‘s Model

Light weight

process with state

Monitor

Link to monitor, restart

Light weight

process with state

Light weight

process with state

Messages Messages

Page 14: The Future of Messaging: RabbitMQ and AMQP

Why Erlang? •  Let it crash

–  If a process fails, it can be easily restarted – Different approach to fault tolerance – Otherwise lots of error handling

•  Message Passing in the Core – RabbitMQ is a messaging system…

•  Light-weight process model – Scalabiliy

Page 15: The Future of Messaging: RabbitMQ and AMQP

Very Basic AMQP •  Queues: Store messages •  Queues might be

– Durable: Survive server restarts – Exclusive: For one connection – autoDelete: Deleted if connection closes

•  Queue usually created by consumer •  All resources are dynamic •  Producer sends a message to a Queue •  Let’s see some code…

Page 16: The Future of Messaging: RabbitMQ and AMQP

Spring’s RabbitTemplate •  Send & receive message •  AmqpTemplate: Generic AMQP interface •  RabbitOperations: Rabbit specific interface:

(adds just a callback) •  RabbitTemplate: Implementation •  Spring might provide support for other AMQP

implementations later •  Common interface

Page 17: The Future of Messaging: RabbitMQ and AMQP

Spring’s MessageConverter •  Messages are binary data •  RabbitTemplate uses MessageConverter

to convert between objects and messages •  Can also send binary data if preferred •  Default: SimpleMessageConverter

–  byte[] directly transferred –  String converted with configurable encoding –  Serializable are serialized –  Content type set accordingly

•  JsonMessageConverter converts from / to JSON using Jackson

•  MarshallingMessageConverter converts from / to XML using Spring's OXM mapping

Page 18: The Future of Messaging: RabbitMQ and AMQP

Spring‘s AdminTemplate •  Main purpose: Configure the AMQP

infrastructure •  E.g. create queues

•  AmpqAdmin: Generic AMQP interface •  RabbitAdmin: Rabbit specific

Page 19: The Future of Messaging: RabbitMQ and AMQP

Code ConnectionFactory conFactory = new SingleConnectionFactory("localhost");RabbitTemplate template = new RabbitTemplate(conFactory);RabbitAdmin admin = new RabbitAdmin(conFactory);admin.declareQueue(new Queue("myQueue"));template.convertAndSend("myQueue", "Hi AMQP!");String receive = (String) template.receiveAndConvert("myQueue");Assert.assertEquals("Hi AMQP!", receive);

Page 20: The Future of Messaging: RabbitMQ and AMQP

Basics of AMQP •  Sending messages directly to queues is not

enough •  What about e.g. pub / sub?

•  Exchange: Route messages (stateless) •  Messages are byte-streams •  Example used the default exchange

•  More dynamic, flexible and cleaner than JMS

Page 21: The Future of Messaging: RabbitMQ and AMQP

AMQP  in  a  nutshell  Exchange routes message Stateless Usually created by producer No queue: Message discarded

X

Queues buffer messages Usually created by consumer

Binding binds an Exchange to a Queue

Page 22: The Future of Messaging: RabbitMQ and AMQP

AMQP  in  a  nutshell  

AMQP protocol

Producer and Consumer might be written in Java, C#, Python, Ruby …

X

C

C

P

RabbitMQ AMQP protocol

Page 23: The Future of Messaging: RabbitMQ and AMQP

Exchange: Route Messages •  The type of Exchange defined the routing

algorithm used •  Binding provides selector for routing •  Exchange is addressed by name

•  Some standard types •  Can provide additional ones

X

Page 24: The Future of Messaging: RabbitMQ and AMQP

Fanout Exchange •  Broadcast to all bound queues •  Fast •  Simple

•  amq.fanout is mandatory

•  To broadcast information

X

Page 25: The Future of Messaging: RabbitMQ and AMQP

X

C

C

C

P

Fanout

Fanout Exchange X

Page 26: The Future of Messaging: RabbitMQ and AMQP

Queue fanoutQueue = new Queue("fanoutQueue");admin.declareQueue(fanoutQueue);FanoutExchange fanoutExchange= new FanoutExchange("myFanout");admin.declareExchange(fanoutExchange);admin.declareBinding( BindingBuilder.from(fanoutQueue). to(fanoutExchange));template.setExchange("myFanout");template.convertAndSend("Hi Fanout!");String receive = (String) template.receiveAndConvert("fanoutQueue");Assert.assertEquals("Hi Fanout!", receive);

Page 27: The Future of Messaging: RabbitMQ and AMQP

Direct Exchange •  Routing based on one routing key •  amq.direct and the default Exchange (no

name) always exist

•  To send work orders to a specific worker

X

Page 28: The Future of Messaging: RabbitMQ and AMQP

X

C

C

C

P

Direct Exchange

express

normal

express normal

Direct Exchange

Page 29: The Future of Messaging: RabbitMQ and AMQP

Queue directQueue = new Queue("direct");admin.declareQueue(directQueue);admin.declareBinding(BindingBuilder .from(directQueue) .to(new DirectExchange("amq.direct")) .with("helloKey"));template.setExchange("amq.direct");template.convertAndSend("amq.direct","dropMe", "I will be dropped!");template.convertAndSend("amq.direct","helloKey", "Hi Direct!");Assert.assertEquals("Hi Direct!", template.receiveAndConvert("direct"));Assert.assertNull( template.receiveAndConvert("direct"));

Page 30: The Future of Messaging: RabbitMQ and AMQP

Topic Exchange •  Routing based on routing pattern •  amq.topic is mandatory

•  E.g. for public / subscribe scenarios

X

Page 31: The Future of Messaging: RabbitMQ and AMQP

Topic Exchange  

X C

C

P

Topic Exchange

order.*

invoice.*

order.DE invoice.USD

Page 32: The Future of Messaging: RabbitMQ and AMQP

Headers Exchange •  Routing based on one or more headers and

an expression •  amqp.match is mandatory

•  Complex routing roles

X

Page 33: The Future of Messaging: RabbitMQ and AMQP

Other Features •  Message can be persistent •  Request / response using correlations

possible

•  Redelivery / acknowledgement possible

•  Clustering with e.g. Linux HA possible •  ...or send message through multiple channels

and drop duplicates

Page 34: The Future of Messaging: RabbitMQ and AMQP

More about RabbitMQ and Spring

Page 35: The Future of Messaging: RabbitMQ and AMQP

Configuring Rabbit Resources with Spring

•  Spring enables decoupling of your application code from the underlying infrastructure

•  The container provides the resources

•  The application is simply coded against the API

Page 36: The Future of Messaging: RabbitMQ and AMQP

<bean id="connectionFactory" class="org.sfw.amqp.rabbit.connection.SingleConnectionFactory"> <property name="username" value="guest"/> <property name="password" value="guest"/> <constructor-arg value="localhost" /> </bean>

Configuring a ConnectionFactory

•  Can easily modify configuration options

Page 37: The Future of Messaging: RabbitMQ and AMQP

<bean id="rabbitTemplate" class="org.springframework.amqp.rabbit.core.RabbitTemplate"> <constructor-arg ref="connectionFactory" /> <property name="routingKey" value=”invoice.USD" /> </bean>

Defining a RabbitTemplate Bean •  Provide a reference to the ConnectionFactory •  Optionally provide other references

– MessageConverter – Routing key and exchange to be used if none is

specified

Page 38: The Future of Messaging: RabbitMQ and AMQP

The MessageListener •  So far: Calling receive() on RabbitTemplate •  Needed: Something that is called when a new

message appears •  The API defines this interface for

asynchronous reception of messages

public void onMessage(Message) { // handle the message }

Page 39: The Future of Messaging: RabbitMQ and AMQP

Spring’s MessageListener Container

•  Spring provides lightweight containers to call MessageListeners

•  SimpleMessageListenerContainer •  Advanced scheduling and endpoint

management options available •  i.e. thread pools, concurrent consumers,

transaction handling

Page 40: The Future of Messaging: RabbitMQ and AMQP

Defining a Message Listener Container

•  Every time a new message appears on my.amqp.queue the messageListener is called

<bean class="org.sfw.amqp.rabbit.listener.SimpleMessageListenerContainer"> <property name="connectionFactory" ref="connectionFactory" /> <property name="queueNames" value="my.amqp.queue" /> <property name="messageListener" ref="messageListener" /> </bean>

Page 41: The Future of Messaging: RabbitMQ and AMQP

Spring's message-driven objects •  MessageListener means the receiver

depends on Spring API •  Why not just a POJO? •  MessageListenerAdapter takes a POJO and

makes it a MessageListener •  i.e. calls consume on Bean consumer <bean id="messageListenerAdapter"

class="org.sfw.amqp.rabbit.listener.adapter.MessageListenerAdapter"> <property name="delegate" ref="consumer" /> <property name="defaultListenerMethod" value="consume" /> <property name="messageConverter" ref="jsonMessageConverter" /> </bean>

Page 42: The Future of Messaging: RabbitMQ and AMQP

Easier Using Namespaces

•  Results in the same Spring Beans

<rabbit:listener-container connection-factory="connectionFactory“ message-converter="jsonMessageConverter"> <rabbit:listener ref="consumer" method="consume"

queue-names="my.amqp.queue2" /> </rabbit:listener-container>

Page 43: The Future of Messaging: RabbitMQ and AMQP

Consumer code

•  No dependency on AMQP! •  But: What about the result of the method? •  Send to the Reply-To address given in

message properties with same correlationId as original method

@Componentpublic class Consumer { public String consume(String message) { return …; }}

Page 44: The Future of Messaging: RabbitMQ and AMQP

Client Code

•  Message sent to destination with routing key •  Reply-To set to exclusive, autodelete, non-

durable queue •  Response received through Reply-To

converted and returned •  Easy request-response! •  Beware of potential latency

String response = (String) rabbitOperations.convertSendAndReceive( "my.fanout", "", "test");

Page 45: The Future of Messaging: RabbitMQ and AMQP

Create Environment Using Namespaces

•  ...if you don‘t like API calls

<rabbit:fanout-exchange name="my.fanout2"> <rabbit:bindings> <rabbit:binding queue="my.amqp.queue2" /> </rabbit:bindings> </rabbit:fanout-exchange> <rabbit:queue name="my.amqp.queue2" />

Page 46: The Future of Messaging: RabbitMQ and AMQP

Conclusion: AMQP •  Ubiquitous Messaging •  AMQP: Protocol standard •  Better scalability •  Dynamic resources

Page 47: The Future of Messaging: RabbitMQ and AMQP

Conclusion: Spring AMQP •  Easy to use •  Flexible (e.g. message encoding) •  Allows scalable message handling •  Full support for AMQP and RabbitMQ

Page 48: The Future of Messaging: RabbitMQ and AMQP

More •  http://springsource.org/spring-amqp •  Also a .NET version available •  …and support Spring Integration •  http://blog.springsource.com/2011/04/01/

routing-topologies-for-performance-and-scalability-with-rabbitm

•  Transaction support

•  …and there is very similar JMS support