JEEConf. Vanilla java

Vanilla java or handling 10k req/sec per core

Dmitriy DumanskiyBlynk

Co-Founder / Java / Android

Java blog : http://habrahabr.ru/users/doom369/topicsDOU : http://dou.ua/users/DOOM/articles/

Experience

mGage - 3k req/secmobclix - 6k req/secxxxx - 11k req/sec

Visa

Load : ~2000 req/sec

Twitter : new tweets

Load : ~6000 req/sec

Facebook messenger

Load : 300k req/sec

IoT Market

● 4.4 billion devices right now (70$ b)● 25 billion devices per 2020 (270$ b)

Makers problem

+ = ?

Problem

● Thousands of connections● Keep-alive● Security● Pulling / polling● Simplicity

IoT Protocols

HTTP, MQTT, XMPP, AMQP, CoAP

Minimum packet size

● HTTP - 26 bytes (~70 bytes)● MQTT - 2 bytes● Blynk - 3 bytes

Blocking IO

connection new Socket() new Thread() read / writeServer Socket

Blocking IO

● 256kb stack● 1k threads == 250 MB of RAM● Сontext switching

Non-Blockingnew Socket()

read / write

Selector

Thread

new Socket()

read / write

new Socket()

read / write

Non-Blocking

● Few threads. Usually - n cores x 2.● No context switching● No memory consumption

Non-blocking frameworks● Apache MINA● Grizzly● Xnio● Vert.x● Akka● Netty

Non-blocking frameworks● Apache MINA● Grizzly● Xnio● Vert.x● Akka● Netty

Non-blocking frameworks● Apache MINA● Grizzly● Xnio● Vert.x● Akka● Netty

Non-blocking frameworks● Apache MINA● Grizzly● Xnio● Vert.x● Akka● Netty

Non-blocking frameworks● Apache MINA● Grizzly● Xnio● Vert.x● Akka● Netty

Netty

Netty cons

Netty is hard. 2-3 weeks onboarding

Netty cons

Bad documentation

Netty cons

Too many abstractions

Netty cons

Easy shot in the foot

Netty cons

ByteBufin.readBytes(length) - memory leak

in.readSlice(length) - ok

Netty cons

Still has bugs, 300 issues

Netty pros

Full control

TCP_FASTOPEN, TCP_MD5SIG, TCP_CORK, TCP_KEEPCNT, TCP_KEEPIDLE, TCP_KEEPINTVL,

SPLICE

Netty pros

Supports everything

TCP, UDP, UDT, SCTP, HTTP, HTTPS, HTTP/2, SPDY, Memcached, MQTT, etc

Netty pros

A lot of default Handlers

Netty pros

Fast

Netty pros

Super quick community help

Blynk architecture

https://github.com/blynkkk/blynk-server

Open source

https://github.com/blynkkk/blynk-library

Blynk stats

● ~2000 of hardware always online● ~300 users always online● ~1000 local Blynk Server installations● ~3 Mbps in traffic● ~7 billions requests per month

Blynk load

● 6% CPU with 2500 req/sec on 4 cores● 10k req/sec per core● 90% of requests - 3 handlers● Only 3% requests are HTTP

Why so fast?

No blocking IO

Request

BE

Blocking IO Workers

Response

Batches

Request

BE

Response

Collector1 min batch File System

What is your DB?

try (Writer writer = Files.newBufferedWriter(file)) { writer.write(user.toString());}

No synchronization

Client 1connect

Selector Thread 1assign

Client 2connect

Selector Thread 2assign

Client 2login

Logic Thread 1reregister

No GC pressure

ctx.write(new Response(mId));vs

ByteBuf buf = ctx.alloc().directBuffer(1);buf.writeByte(mId);

ctx.write(buf);

In memory

Set<User> users = JsonParser.parseUsers(dir);

+ 1 min batches

Native transports

● Native epoll● Native OpenSSL

Version 1

Blynk monolith

80$

Version 1. Tech stack

● Vanilla Java 8● Netty (HTTP, WebSockets, TCP/IP)

Version 1. Pros

Simple single jar deploymentjava -jar server.jar

100ms start

Version 1. Pros

Simple backup script

Version 1. Pros

Single JVM, no cross-JVM shared state

Version 1. Pros

Entry level

Version 1. Cons

Single point of failure, single datacenter

Version 1. Cons

Latency is bad for distant locations

Version 1. Cons

Need to drop connections on redeploy

Version 2

ClientGet DNS record EUROPE

USA

ASIAGeo DNS

50$

Version 2

● Improved latency● No common storage anymore

Version 2

USA

Clients

DC 1

EU

Clients

DC 2

ASIA

Clients

DC 3

Version 3

USA

Clients

DC 1

EU

Clients

DB

ASIA

Clients

DC 3

Version 3. DB Cassandra

Node 1

Node 3

Node 2

Node 3

Node 2

Node 1

Node 3

Node 2

DC 1 DC 2

Node 1

15$

Version 3. DB Cassandra

● No typical batches● Minimum requirement 1GB of RAM● Performs poorly on low-end VMs● Much slower than Postgres

Version 3. Final

USA

Clients

DC 1

EU

Clients

Postgres

ASIA

Clients

DC 365$

Version 3. Final

● 60k req/sec for 65$ (25x for current load)● + 10x growth to 20 CPU● Samsung cloud - 1 req/sec per 6$

Q & A