Zebus Building a P2P Service Bus

Kévin Lovato - @alprema

What is an ESB?

Enterprise Service Bus 101

Application 1

Bus

Application 3 Application 2

InterestingMessage

InterestingMessage InterestingMessage

What we had

QPID (broker) Single instance / single server

Application 1

Application 2 Application 3

Why change?

New requirements

• More resilience

• Better latency

• Better throughput

How it works

Basics

• Communication: Zmq + ProtoBuf

• Discovery: All knowing Directory service

• Storage: Cassandra (Plugable)

Directory

Peer 2 Peer 3

Peer 1 needs to connect to the bus

Startup

Peer 1 Peer 2 Peer 3

Register Peers list + Subscriptions

Startup

Directory

Peer 1 Peer 2 Peer 3

New Peer information

Startup

Directory

Peer 1 Peer 2 Peer 3

Direct communication

Startup

Directory

Persistence

Peer 1 Peer 2 Message

Message persistence

Message copy

Store

Persistence

Peer 1 Peer 2

Message persistence

Message-handling ack

”Delete ”

How we use it

Sending a command

private void SendMessage() { _bus.Send(new SendChatMessageCommand("Hello world", "me")); }

Receiving a command

public class ChatMessageHandler : IMessageHandler<SendChatMessageCommand> { public void Handle(SendChatMessageCommand chatMessageCommand) { Console.WriteLine(chatMessageCommand.Message); } }

Dynamic subscription

private void MakeSubscriptionForMe() { var constraint = Subscription.Matching<Msg>(msg => msg.Author == "me"); _bus.Subscribe(new[] { constraint }, msg => DoTheThing(msg)); }

Configuring the bus

private IBus ConfigureAndStartBus() { return new BusFactory().WithConfiguration("tcp://directory:129", "Prod") .CreateAndStartBus(); }

https://github.com/Abc-Arbitrage/Zebus.Samples for more

Design pitfalls

Mistakes you should avoid when designing a distributed system

Proper bus shutdown

Peer 1 Peer 2

Network Zmq layer Zmq layer

Normal operations

Proper bus shutdown

Peer 1 Peer 2

Network Zmq layer Zmq layer

Shutdown phase

Proper bus shutdown

Peer 1 Peer 2

Network Zmq layer Zmq layer

Shutdown phase

Messages in the Zmq buffers are not guaranteed to be processed

Proper bus shutdown

Peer 1 Peer 2

Solution

EndOfStream message

Use an end of stream handshake before closing streams

Proper bus shutdown

Peer 1 Peer 2

Solution

EndOfStreamAck

Use an end of stream handshake before closing streams

Directory fault tolerance

Register Peers list + Subscriptions

Naive implementation

Peer 1

Directory 1

Directory fault tolerance

The directory became the new SPOF

Directory down

Peer 1

Directory fault tolerance

Peer 1

Solution

Directory 1 Directory 2

Redundant directories, state synchronized using Cassandra

Out of order subscriptions

Peer 1

Directory 1 Directory 2

Naive implementation

Subscriptions update A

Out of order subscriptions

Peer 1

Directory 1 Directory 2

Naive implementation

Subscriptions update B

Out of order subscriptions

Peer 1

Directory 1 Directory 2

Out of order processing

For unknown reasons, Directory 2 does the update first

Out of order subscriptions

Peer 1

Directory 1 Directory 2

Directory 1 catches up and overwrites Directory 2’s update

Out of order processing

Out of order subscriptions

Peer 1

Directory 1 Directory 2

The state is incorrect

Out of order processing

Out of order subscriptions

Peer 1

Directory 1 Directory 2

Solution

Update A with client timestamp

Update B with client timestamp

Out of order subscriptions

Peer 1

Directory 1 Directory 2

Solution

Use the client timestamp to let Cassandra resolve the conflict

Massive subscriptions

Peer 1

Directory 1

Naive implementation

Subscriptions update – 10 subscriptions

Massive subscriptions

Peer 1

Directory 1

Directory DoS

Subscriptions update – 10 000 subscriptions ( ) 10 000 x

Massive subscriptions

Peer 1

Directory 1

Solution

Subscriptions update – few subscriptions ( ) 10 000 x

Use incremental updates to mitigate load Restructure the Directory storage

And the list goes on…

• Fast and resilient persistent storage

And the list goes on…

• Persistence failure handling

And the list goes on…

• Time synchronization between peers

And the list goes on…

• Java / .Net types conversions (Cassandra)

Why did we do it anyway?

• We needed something light and easy to use

• We needed to match the existing bus API

• Well… we probably also underestimated the

madness that awaited us

What can it do?

Up to 140K messages per second ~200M messages transferred per day 2 years in production More than 60K subscriptions ~350 peers in production

Want to give it a try?

https://github.com/Abc-Arbitrage/Zebus

Or build cool stuff with us?

We’re hiring

Questions