Upload
wallyqs
View
646
Download
2
Embed Size (px)
Citation preview
Simple, Secure and Scalable MessagingFor The Cloud Native EraWaldemar Quevedo /
All Things Open / October 2017
@wallyqs
1 . 1
Agenda
Intro to the NATS project
NATS & NATS Streaming Overview
Demos
2 . 1
NATS is a simple, high performance open sourcemessaging system for cloud native applications.
3 . 1
Facts about NATS
Written by Derek Collison in 2010, originally for CloudFoundry
Initial implementation in Ruby, was then rewritten in Go in 2012
Project used in production environments for ~7 years
NATS Server v1.0.0 milestone reached in July 2017 �
Github Organization: https://github.com/nats-io
4 . 1
NATS is fast
5 . 1
Stats about NATSNATS Server can process above +11M messages per second (w/ nats-bench)
6 . 1
Stats about NATSReliable Low Latency Pub Sub!
Source: http://bravenewgeek.com/benchmarking-message-queue-latency/
7 . 1
NATS ispredictable �
8 . 1
https://twitter.com/ripienaar/status/905299955077787648
9 . 1
https://twitter.com/ngduynd/status/907922573521936384
10 . 1
NATS is simple
11 . 1
Simplicity in NATS
Plain text protocol with few commands → Simple network clients & API
7MB binary with no dependencies → Small Docker image �
Fire & Forget Pub Sub on top of TCP/IP
| PUB | SUB | UNSUB | CONNECT | INFO | MSG | -ERR | +OK | PING | PONG |
telnet demo.nats.io 4222 Connected to demo.nats.io. INFO {"server_id":"Fzwx2ndlHFg3lvVwwdBRSe","tls_required":false,...,"max_payload":1048576} SUB hello 1 +OK PUB hello 5 world +OK MSG hello 1 5 world
12 . 1
Basic API: Go Clientpackage main import ( "log" "github.com/nats-io/go-nats" ) func main(){ nc, err := nats.Connect("nats://127.0.0.1:4222") if err != nil { log.Fatalf("Error: %s", err) } done := make(chan struct{}) nc.Subscribe("hello", func(m *nats.Msg){ log.Printf("[Received] %s", string(m.Data)) done <- struct{}{} }) nc.Publish("hello", []byte("world")) <-done }
13 . 1
Basic API: Ruby Clientrequire 'nats/client' NATS.start do |nc| nc.subscribe("hello") do |msg| puts "[Received] #{msg}" end nc.publish("hello", "world") end
14 . 1
Basic API: Python Clientawait nc.connect() async def handler(msg): print("[Received] {data}".format( data=msg.data.decode())) # Coroutine based subscriber await nc.subscribe("foo", cb=handler) await nc.publish("foo", "bar")
15 . 1
Basic API: Node.js Clientvar nats = require('nats').connect(); // Simple Publisher nats.publish('foo', 'Hello World!'); // Simple Subscriber nats.subscribe('foo', function(msg) { console.log('[Received] ' + msg); });
16 . 1
Basic API: C ClientnatsConnection_Publish(nc,"foo",data,5); natsConnection_Subscribe(&sub,nc,"foo",onMsg, NULL); void onMsg(natsConnection *nc, natsSubscription *sub, natsMsg *msg, void *closure) { printf("[Received] %.*s\n", natsMsg_GetData(msg)); // ... }
17 . 1
Basic API: C# Clientusing NATS.Client; ConnectionFactory cf = new ConnectionFactory(); IConnection c = cf.CreateConnection(); ISyncSubscription sSync = c.SubscribeSync("hello"); // Wait up to 1000 ms for next message Msg m = sSync.NextMessage(1000); c.Publish("hello", Encoding.UTF8.GetBytes("world"));
18 . 1
Basic API: Java ClientConnection nc = Nats.connect(); nc.subscribe("foo", m -> { System.out.printf("Received a message: %s\n", new String(m.getData())); }); nc.publish("foo", "Hello World".getBytes());
19 . 1
Clients of�cially supported by the NATS team �
20 . 1
NATS is resilient
21 . 1
Protection against Slow Consumerspackage main import ( "log" "github.com/nats-io/go-nats" ) func main(){ nc, err := nats.Connect("nats://127.0.0.1:4222") if err != nil { log.Fatalf("Error: %s", err) } n := 0 total := 1000000000 done := make(chan struct{}) nc.Subscribe("hello", func(m *nats.Msg){ n++ if n == total { close(done) } }) for i := 0; i < total; i++ { nc.Publish("hello", []byte("world")) } select { case <-done: } }
22 . 1
Protection against Slow ConsumersServer disconnects client with Slow Consumer error
Client was too slow at processing the received messages.
[33682] 2017/10/19 07:52:51.647917 [INF] Starting nats-server version 0.9.6 [33682] 2017/10/19 07:52:51.648085 [INF] Starting http monitor on 0.0.0.0:8222 [33682] 2017/10/19 07:52:51.648631 [INF] Listening for client connections on 0.0.0.0:4222 [33682] 2017/10/19 07:52:51.648691 [INF] Server is ready [33682] 2017/10/19 07:52:55.376292 [INF] 127.0.0.1:63184 - cid:1 - Slow Consumer Detected [33682] 2017/10/19 07:52:57.424381 [INF] 127.0.0.1:63185 - cid:2 - Slow Consumer Detected [33682] 2017/10/19 07:52:59.478090 [INF] 127.0.0.1:63186 - cid:3 - Slow Consumer Detected [33682] 2017/10/19 07:53:01.541496 [INF] 127.0.0.1:63188 - cid:4 - Slow Consumer Detected
23 . 1
NATS Server ClusterHigh Availability via full-mesh cluster topology
24 . 1
NATS Server ClusterHigh Availability via full-mesh cluster topology
25 . 1
NATS & Delivery Guarantees
The NATS Server is �re & forget, providing at-most once delivery.
This means that if the consumer is not connected it will not receive the message!
26 . 1
Is NATS a Message Broker/Queue?From the book (Kleppmann, O'Reilly)
"a message broker (also known as a message queue), is essentially a kind of databasethat is optimized for handling message streams."
❌
Designing Data-Intensive Applications
27 . 1
Is NATS a Message Broker/Queue?From the book (Kleppmann, O'Reilly)
"By centralizing the data in the broker, these systems can more easily tolerate clientsthat come and go (connect, disconnect, and crash)
"the question of durability is moved to the broker instead
❌
Designing Data-Intensive Applications
28 . 1
Is NATS a Message Broker/Queue?From the book (Kleppmann, O'Reilly)
"Faced with slow consumers, they generally allow unbounded queueing (as opposed todropping messages or backpressure)
❌
Designing Data-Intensive Applications
29 . 1
The NATS Server does none of that!
It's the total opposite.
30 . 1
31 . 1
NATS Streaming
codename: STAN
32 . 1
NATS Streaming
Supports at-least-once delivery model and durability of messages
Rate limiting & message redelivery features
First release in 2016
33 . 1
NATS Streaming: How it worksDesigned as a request/response protocol which uses NATS as its transport.
34 . 1
NATS Streaming: How it worksDesigned as a request/response protocol which uses NATS as its transport.
35 . 1
NATS Streaming: How it worksDesigned as a request/response protocol which uses NATS as its transport.
36 . 1
NATS Streaming: How it worksDesigned as a request/response protocol which uses NATS as its transport.
37 . 1
NATS Streaming ClusteringClustering support in the works!
To be fair, you have to have a very high IQ to understand Rick & Morty clustering
https://github.com/nats-io/nats-streaming-server/issues/316
38 . 1
NATS Streaming: Go ClientPublish and persist a message
Start with last received message
Receive all messages
sc, _ := stan.Connect(clusterID, clientID) sc.Publish("foo", []byte("Hello World"))
sub, err := sc.Subscribe("foo", func(m *stan.Msg) { fmt.Printf("Received a message: %s\n", string(m.Data)) })
sub, err := sc.Subscribe("foo", func(m *stan.Msg) { fmt.Printf("Received a message: %s\n", string(m.Data)) }, stan.DeliverAllAvailable())
39 . 1
NATS Streaming: Ruby Clientrequire 'stan/client' sc = STAN::Client.new # Customize connection to NATS opts = { servers: ["nats://127.0.0.1:4222"] } sc.connect("test-cluster", "client-123", nats: opts) # Simple async subscriber sub = sc.subscribe("foo", start_at: :first) do |msg| puts "Received a message (seq=#{msg.seq}): #{msg.data}" end # Synchronous Publisher, does not return until an ack # has been received from NATS Streaming. sc.publish("foo", "hello world")
40 . 1
NATS Streaming: Python Clientnc = NATS() await nc.connect(io_loop=loop) sc = STAN() await sc.connect("test-cluster", "client-123", nats=nc) await sc.publish("hi", b'hello') async def cb(msg): print("Received a message (seq={}): {}".format(msg.seq, msg.data)) # Subscribe to get all messages since beginning. sub = await sc.subscribe("hi", start_at='first', cb=cb) await sub.unsubscribe()
41 . 1
NATS Streaming: Node.js Clientvar stan = require('node-nats-streaming').connect('test-cluster', 'test'); stan.on('connect', function () { stan.publish('hello', 'world', function(err, guid){ /*...*/ }); var opts = stan.subscriptionOptions().setStartWithLastReceived(); var subscription = stan.subscribe('hello', opts); subscription.on('message', function (msg) { console.log('Received [' + msg.getSequence() + '] ' + msg.getData()); }); });
42 . 1
NATS Streaming: Java ClientStreamingConnectionFactory cf = new StreamingConnectionFactory("test-cluster", "bar"); StreamingConnection sc = cf.createConnection(); sc.publish("foo", "Hello World".getBytes()); Subscription sub = sc.subscribe("foo", new MessageHandler() { public void onMessage(Message m) { System.out.printf("Received a message: %s\n", new String(m.getData())); doneSignal.countDown(); } }, new SubscriptionOptions.Builder().deliverAllAvailable().build());
43 . 1
NATS Streaming: C# Clientvar cf = new StanConnectionFactory(); using (var c = cf.CreateConnection("test-cluster", "appname")) { using (c.Subscribe("foo", (obj, args) => { Console.WriteLine( System.Text.Encoding.UTF8.GetString(args.Message.Data)); })) { c.Publish("foo", System.Text.Encoding.UTF8.GetBytes("hello")); } }
44 . 1
Also of�cially supported by the NATS team �
45 . 1
Remember: There are 2 servers
NATS Server / gnatsd
NATS Streaming Server / nats-streaming-server
NATS Streaming itself uses the NATS Server: Two birds, one stone
https://github.com/nats-io/gnatsd
https://github.com/nats-io/nats-streaming-server
46 . 1
Demos
47 . 1
Congrats!Now you're ready to start using
/ github.com/nats-io @nats_io
https://nats.io/
48 . 1