Le Streaming d'API : pourquoi et comment transformer vos APIs statiques en données temps réel ?

#DevoxxFR @Audrey_Neveu

The End of PollingWhy and how to transform a REST API

into a Data Streaming API


ABOUT ME

Developer Relations Co-Leader France

@Audrey_Neveu


MODERN TIMES


OUR GOAL


ANIMATION IS THE KEY... BUT WHY?


BECAUSE OF EVOLUTION


USER INTERFACE


KEEP CALM ... AND LEAVE.


REFRESH BUTTON IS EVIL


REAL-TIME USER EXPERIENCE


F.O.M.O (FEAR OF MISSING OUT)


API STREAMING


SOLUTIONS FOR REAL-TIME APPLICATIONS

✓ WebHooks

✓ (Long) Polling

✓ Web-Sockets

✓ Pub/Sub

✓ Server-Sent Events


IT’S ALL ABOUT FREQUENCY ...


… AND USAGE


(Long) Polling


98.5% of polls are wasted

Source:

http://resthooks.org/

POLLING IS NOT A SOLUTIONMADNESS ™(and neither is long polling)


WebHooks


✓ webhooks.org

✓ aka User Defined HTTP Callback

✓ concept / method

WEBHOOKS

✓

http://www.webhooks.org/

http://www.webhooks.org/

HTTP/1.1 201 CreatedLink:<http://subscription-server.com/subscription>;rel="subscription"

@StreamdataIOstreamdataio

POST /eventsource HTTP/1.1Host: subscription-server.comPragma: subscribeCallback: <http://example.com/callback>; method="POST"rel="subscriber"

POST /callback HTTP/1.1Host: example.comLink: <http://subscription-server.com/subscription>;rel="subscription"Content-HMAC: sha1 C+7Hteo/D9vJXQ3UfzxbwnXaijM=Content-Length: 21Content-Type: application/x-www-form-urlencodedpayload=Hello%20world

Consumer API Provider


✓ Define a callback URL ✓ Create a subscription endpoint

- GET /webhook

- POST /webhook

- GET /webhook/{id}

- PUT /webhook/{id}

- DELETE /webhook/{id}

TODO LIST


✓ Implement your webhook queue

- inline HTTP Requests

- SQL-based queue

- AMQP broker

- batch


CONSUMER SETUP


EVENT RECEIVEDPOST /callback HTTP/1.1Host: www.example.comX-Github-Delivery: 72d3162e-cc78-11e3-81ab-4c9367dc0958User-Agent: GitHub-Hookshot/044aaddContent-Type: application/jsonContent-Length: 6615X-GitHub-Event: issues

Payload={ "action": "opened", "issue": { "url": "https://api.github.com/repos/octocat/Hello-World/issues/1347", "number": 1347, ... }, "repository" : { "id": 1296269, "full_name": "octocat/Hello-World", "owner": { "login": "octocat", "id": 1, ... }, ... }, "sender": { "login": "octocat", "id": 1, ... }}


PROS AND CONS

✓ Real-Time updates

✓ Easily consumed

✓ Without dedicated resources

■ Poor user experience

■ Does not work with all clients

■ Manual setup✓ Easily integrated

■ Debugging

✓ (almost) Real-Time updates


KNOWN ISSUES


■ DDoS Attack

■ Missed notification

■ DDoS Attack

■ Deduplication


CHECK LIST


✓ Implement authentication

✓ Expected answer?

✓ Monitor payload size

✓ Handle request number

✓ Handle duplicates

✓ One callback per webhook


DYNAMIC SUBSCRIPTIONS

✓ Restful Webhooks

✓ REST Hooks

✓ PubSubHubbub


PubSubHubbubaka PubSubaka PuSH


RSS / Atom feeds

pubsubhubbub/

Open Protocol

Based on Publish / Subscribe Pattern …

PUBSUB

… and on topics

https://github.com/pubsubhubbub/

https://github.com/pubsubhubbub/


SubscribersHub

PUBSUB

Publishers


Discovery

SUBSCRIBE - PUBLISHERS

Link: <https://my-hub.com/>; rel="hub"Link: <https://my.resource.com>; rel="self"

<link rel="self" href="https://my-resource.com/"><link rel="hub" href="https://my-hub.com/">

Publishers


SUBSCRIBE - SUBSCRIBERS 1/2

POST https://my-hub.com/…hub.mode="subscribe"hub.topic="https://my-resource.com/"hub.callback="http://example.com/callback"hub.secret="my-token"

Subscribers Send subscription request to the Hub


SUBSCRIBE - HUB

GET http://example.com/callback…hub.mode="subscribe"hub.topic="https://my-resource.com/"hub.challenge="a random string"hub.lease_seconds=86400

Hub Verify intent of the subscribers


SUBSCRIBE - SUBSCRIBERS 2/2

HTTP/1.1 200 OKBody:{

hub.challenge: "a random string"}

Subscribers Answer verification request


CONSUMER SETUP


PUBLISH - PUBLISHERS

POST https://my-hub.com/…

hub.mode=publishhub.url=https://my.updated-resource.com

Publishers Ping the Hub


PUBLISH - HUB

POST http://example.com/callbackLink: <https://my-hub.com/>; rel="hub"Link: <https://my.updated-resource.com>; rel="self"

X-Hub-Signature="my-sha1-signature"

Hub (Authenticated) Content DistributionContent Distribution


PUBLISH - SUBSCRIBERS

GET https://my.updated-resource.com

Subscribers Pull the updated resource


{ "object":"page", "entry":[ { "id":"51044240199134611", "time":1447342027, "changes":[ { "field":"leadgen", "value":{ "adgroup_id":0, "ad_id":0, "created_time":1447342026, "leadgen_id":55459717045641545, "page_id":516540199134611, "form_id":551111744595541 } } ] } ]}

EVENT RECEIVED


UNSUBSCRIBE - SUBSCRIBERS 1/2

POST https://my-hub.com/…hub.mode="unsubscribe"hub.topic="https://my-resource.com/"hub.callback="http://example.com/callback"

Subscribers Send unsubscription request to the Hub


UNSUBSCRIBE - HUB

GET http://example.com/callback…hub.mode="unsubscribe"hub.topic="https://my-resource.com/"hub.challenge="a random string"

Hub Verify intent of the subscribers


UNSUBSCRIBE - SUBSCRIBERS 2/2

HTTP/1.1 200 OKBody:{

hub.challenge: "a random string"}

Subscribers Answer verification request


WEBHOOKS PROS AND CONS

✓ (almost) Real-Time updates

✓ Easily consumed

✓ Without dedicated resources

■ Poor user experience

■ Does not work with all clients

■ Manual setup✓ Easily integrated

■ Debugging

■ Need another call to get data


KNOWN ISSUES


■ DDoS Attack

■ Missed notification

■ DDoS Attack

■ Deduplication


WEBHOOK VS PUBSUB


Push Technologies


PUSH TECHNOLOGIES

Server-Sent Events

2008 2006

W3C Specification

Web-Sockets


PUSH TECHNOLOGIES

Text + Binary Text

Server-Sent EventsWeb-Sockets


PROTOCOLS

GET /chat HTTP/1.1Host: example.comUpgrade: websocketConnection: UpgradeSec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ==Sec-WebSocket-Protocol: chatSec-WebSocket-Version: 13

GET /stream HTTP/1.1 1Host: example.comAccept: text/event-stream

Web-Sockets (RFC-6455)

Server-Sent Events

HTTP/1.1 101 Switching ProtocolsUpgrade: websocketConnection: UpgradeSec-WebSocket-Accept: HSmrc0sMlYUkAGmm5OPpG2HaGWk=Sec-WebSocket-Protocol: chatSec-WebSocket-Version: 13


CONFIGURATION



MESSAGES FORMAT

var msg = {

type: "message",

text: "Hello Devoxx

France!",

id: 12345,

date: Date.now()

};

data: Hello Devoxx France!


data: with two lines

data: {"time": "16:34:36", "text": "Hello Devoxx

France!"}

id: 12345

event: foo


retry: 10000



IMPLEMENTATION

var websocket = new WebSocket ('ws://websocketserver/echo');

var eventSource = new EventSource

('http://sseserver/echo');

websocket.onOpen = function(){...

};

eventSource.onopen = function(){...

};

websocket.onMessage = function(e){var data = e.data;var message = data.msg;...

};

eventSource.onMessage = function(e){var message = e.data;...

};

websocket.onError = function(e){...

};

eventSource.onError = function(e){...

};

...

eventSource .addEventListener('foo',function(e){ // do something}, false);

eventSource .addEventListener('bar',function(e){ // do something else}, false);



LOST IN CONNECTION



BROWSER SUPPORT

57. 52. 10. 44. 11. 14. 57. 52. 10. 44. 11. UC.

source : http://caniuse.com/


http://caniuse.com/


MOBILE BROWSER SUPPORT

source : http://caniuse.com/

57. 51. 10.3. all. 56. 57. 51. 10.3. all. 56.


http://caniuse.com/


PERFORMANCES

8s 5s x1.6

8s 6s x1.3

16s 7s x.2.2

source: http://matthiasnehlsen.com/blog/2013/05/01/server-sent-events-vs-websockets/

SSEWeb-Sockets Diff

http://matthiasnehlsen.com/blog/2013/05/01/server-sent-events-vs-websockets/


CHOOSE WISELY


Proxy-as-a-Service

✓ works with any JSON API

✓ streaming based on Server-Sent Events

✓ dynamic cache

✓ incremental updates

STREAMDATA.IO


JSON-PATCH (RFC-6902)

[{"title":"Value 0","price":66,"param1":"1","param2":"22","param3":"33"}, {"title":"Value 1","price":63,"param1":"11","param2":"2","param3":"53"}, {"title":"Value 2","price":85,"param1":"1","param2":"22","param3":"33"}, {"title":"Value 3","price":21,"param1":"31","param2":"12","param3":"4"}, {"title":"Value 4","price":10,"param1":"151","param2":"22","param3":"33"}, {"title":"Value 5","price":6,"param1":"11","param2":"21","param3":"33"}, {"title":"Value 6","price":60,"param1":"11","param2":"222","param3":"33"}]

[{"title":"Value 0","price":66,"param1":"1","param2":"22","param3":"33"}, {"title":"Value 1","price":63,"param1":"11","param2":"2","param3":"53"}, {"title":"Value 2","price":5,"param1":"1","param2":"22","param3":"33"}, {"title":"Value 3","price":21,"param1":"31","param2":"32","param3":"4"}, {"title":"Value 4","price":10,"param1":"151","param2":"22","param3":"33"}, {"title":"Value 5","price":6,"param1":"11","param2":"21","param3":"33"}, {"title":"Value 6","price":60,"param1":"11","param2":"222","param3":"33"}]

[{"op":"replace","path":"/2/price","value":5}, {"op":"replace","path":"/3/param2","value":"32"}]


DEMO


VOTE SERVER-SENT EVENTS!


THANKS!Q&A

Engineering

Le Streaming d'API : pourquoi et comment transformer vos APIs statiques en données temps réel ?