MixIT 14 - How to choose your message broker

How to choose your message broker

Guillaume Arnaud @guillarnaud

• EBIA ALLIANCE = XEBIA + XEBIALABS + THIGA + UX REPUBLIC

Offtopic

�2

commercial products low latency

benchmarks ESB


Benchmarks

�3


Exchange information !

�4

notifications

RPC

log centralized

dashboard

synchronization

worker

metrics


Once upon a time JMS…

�5

API (!=protocol)

JMS ∈ JEE

100% Java

JMS 2.02001

Transaction

decoupling


JMS: client side

�6

connexion

session

broker

client

clientsessiontcp!transport

buffer

buffer


Beware of wrong asynchronism

�7

connection = new ActiveMQConnectionFactory( "failover:(tcp://localhost:61616)?timeout=1000") .createConnection(); … producer.send(queue,message);


Exit strategy

�8

bounded

ABC

A consumer

queue dead letter queue

B

C

Time To Live

10 seconds

=10 seconds


JMS: queue vs topic

�9

ABC

A

B

C

consumer 1

queue

consumer 2

consumer 3

ABC

A

C

consumer 1

topic

consumer 2

consumer 3

BC

ABC

B

consumer 3


AMQP: le challenger

�10

100% Protocol

interoperability

routing

2006

0.9.1 vs 1.0


broker

AMQP

�11

exchange

queue

publisher

subscriber

subscriberqueue

queue.a.b

queue.a.c

queue.a.*

binding

virtual host


AMQP: topology

�12

exchange

queue

queue

queue

queuefanout

exchange queue.a.b

fanout

direct

exchangequeue.*.#

queue topic

exchangequeue.a.b (ignored)

queueheader1: value1 header2: value2 …

headers


STOMP: light messaging

�13

textual

light

Simple Text Orientated Messaging Protocol

2010 (?)

incompatibility

heartbeat

js + ws

security


STOMP: message structure

�14

COMMAND header1:value1 header2:value2 !Body^@

CONNECT accept-version:1.0,1.2 host:myhost heart-beat: 10,20 !̂@

CONNECTED version:1.2 heart-beat:10,20 !̂@

SEND destination:/queue/a content-type:text/plain !hello ^@

SUBSCRIBE id:0 destination:/queue/a ack:client !̂@

MESSAGE subscription:0 message-id:003 destination: /queue/a content-type:text/plain !hello ^@


MQTT

�15

Device

QoSCompact

2007

Last Known Good

semantic

Highly constrained


MQTT

�16

sender serverpublishQoS 0: at most once delivery

client serverpublish

ack

QoS 1: at least once delivery

QoS 2: exactly one delivery

receiver

receiver

publish

publish

client server

publish

receive

receiverpublish

complete

release

idid

id

id


Kafka

�17

distributed

complexity

MessageSet

ZeroCopy

2011

consumer oriented

ordering

append only

LinkedIn

λ & μ-service architecture


Kafka

�18

producer

consumer consumer

partitionpartition partition partition

serveur serveur

consumer consumer

consumer consumerconsumer groupconsumer group

topic:partition topic:partition

zookeeper


Distribution/high availability

�19

broker 1

broker 2

shared fs/db

load balancing

High Availability!actif/actif!

actif/passif

broker 1

broker 2

broker 3

broker 4

P1

C1

Network of brokers!(a topology example)

failover

Zookeeper!(robust distribution) idempotency


Performance

�20

persistance mandatory ?

filesystemdatabase

distribution

yes

no

batchprefetch

ack

no serialization

headers

specialized queue

flow control


My subjective solutions

�21


Conclusion

�22

decoupling

semantic buffer

routingheartbeat

ordering

batch/prefetch exit strategy

idempotency flow control

Software

MixIT 14 - How to choose your message broker