RabbitMQ Oxford Geek Night

8/2/2019 RabbitMQ Oxford Geek Night

1/35

1

Databases suck for Messaging

Alexis Richardson

Oxford Geek Night May 2009


2/35

Computers were

meant to get rid

of this

2


3/35

A new kind of fail?

3


4/35

Solution - use a database?

4


5/35

Databases were

meant to get rid

of this too

5


6/35

6

I want to know

when THIS

changes


7/35

Problem - databases suck for messaging

DATA is facts (state) persisted on disk

Databases are great for storing facts and asking questions about them

If you know what to ask them and are willing to keep asking them

INFORMATION is always changing

Networks are great pushing changes (messages) to recipients

Databases are not optimised for this

7


8/35

Social applications store data

8


9/35


9

And theres terabytes of


10/35


10

And theres terabytes of

Enterprises are made of people toSo: the same issues arise


11/35

11

I want to know

when THIS

changes


12/35

Email doesnt scale

12


13/35

Information gets old - real quick

13


14/35

But current

information is

valuable

14


15/35

Example: Flickr

Flickr is a vast database of social objects

Filtered by interest and relationships

So - tell me whats currently relevant

Without sending me more emails....

15


16/35

Polling sucks

16


17/35

Whats going on here? We are trapped in the database world view

17

Example due to Brett Slatkin and Brad Fitzpatrick at Google

http://code.google.com/p/pubsubhubbub/wiki/WhyPollingSucks
http://code.google.com/p/pubsubhubbub/wiki/WhyPollingSuckshttp://code.google.com/p/pubsubhubbub/wiki/WhyPollingSuckshttp://code.google.com/p/pubsubhubbub/wiki/WhyPollingSucks


18/35

How do we defeat this evil?

18


19/35

Can we apply the Hollywood Principle?

19


20/35

Can we apply the Hollywood Principle?

20

Hint: phone calls and SMS dont

travel through databases


21/35

Email push is direct

21

to:blaineblahalexis blaine


22/35

Email push is direct

22

to:blaineblahalexis blaine

blahblahblahblah


23/35

Polling is just reverse spam

23

to:alexis?

????

alexis blaine


24/35

Publishing to a queue (topic) takes the spam burden off the receiver

24

alexis blaine

blahblahblahblahblah

topic: from alexis

PUSH PUSHfilterauth


25/35

And you can use pubsub and queues in all sorts of ways

25


26/35

Databases are not meant to do Pubsub

26

SELECT * FROM queue WHERE done = 0 ORDER BY created LIMIT 1


27/35

A true story

most people would just create a simple "queue" table

and: SELECT * FROM queue WHERE done = 0 ORDER BY created LIMIT 1

concurrency issues on that thing now - inserting into the queue occasionally takelonger than doing the task that needs to be executed synchronously

middle management did not want to have "new third party software" because itwould be too much Operations to learn and manage

so they decided for the time being a MySQL based queue would be sufficient (onlfew million messages/day) and implemented it in PHP/MySQL resulting in lots of dhours for implementation / testing, and more hours because of performance issue

so i think we have spent about the same amount of time as we would to get adescent thing up and running, but now we're stuck with a lame and unscalable mydatabase machines that is dressed as a message broker

27


28/35

Flickr from a Pubsub point of view

Flickr = people publishing to a vast set of streams (photostreams)

Users express interest through subscription

I dont need to see everything - only changes on what I follow

This seems better - but whats missing?

- I am still trapped by the database world view.

- I still poll for changes (thats what RSS does)

- I want the PUSH that email gave me, without the spam.....

28

Ch t i ti f PUBSUB


29/35

Characteristics of PUBSUB

A means of authenticated communication (network transport)

- eg HTTP, OAuth

An addressable place to publish to

- Usually a topic, feed, endpoint or address.

A way to name, enrol, share, and discover these addressable places

- For example [email protected] or - TBD!

A way to deliver and ack delivery (or take responsibility)

The above is in fact a distributed object system

29

H d l bl lik Fli k /T itt /
mailto:[email protected]:[email protected]:[email protected]


30/35

How do you solve a problem like Flickr/Twitter/...

30

DataObjects serialised as

rows in tablesMessages organised

into streams

Interest Filter by query Durable follow

NotificationPull

(polling sucks!) Push

Buffering(non-idempotent)

Put = add row to tableGet = delete top row

PUT and GET aresymmetric

Scale

Overheads tend to

grow indefinitely

Data flows out to

destinations

Database Pubsub/MQ

Y i ht d i if d t it d t f d


31/35

You might need messaging if ... you need to monitor data feeds

31

(CC) Kishore Nagarigari

You might need messaging if you need a message delivered respon


32/35

You might need messaging if ... you need a message delivered respon

32

You might need messaging if you need things done in order


33/35

You might need messaging if ... you need things done in order

33

(CC) David Mach

You might need messaging if you are using the cloud


34/35

You might need messaging if ... you are using the cloud

34

RabbitMQ is a messaging server that just works!


35/35

RabbitMQ is a messaging server that just works!

35

Im in yr serverz,queueing yr messagez

co

Documents

RabbitMQ Oxford Geek Night