Real-time Insights, powered by Reactive Programming

Real-time Insights powered by Reactive Programming

Jay Phelps | @_jayphelps


Let's talk why


Time is money


127+ million person-hours lost

per year

$500k — $1+ million

per hour average hourly cost of

critical failure

https://devops.com/real-cost-downtime/

Fortune 500


Every second counts


Not just outages...


debugging, testing, and even Information Security too


improving Developer Experience

Senior Software Engineer |

@_jayphelps

Jay Phelps


debugging, testing, and InfoSec


InfoSec

“preventing unauthorized access, use, or disruption of information”


stopping hackers


InfoSec



I'm on it.


We can block exploits using our gateway proxy


...we need to know it's working...


...we want to watch attackers try...


...in real-time


We need real-time insights For debugging, testing, and InfoSec


Logging is the answer


LOG ALL THE THINGSLOG ALL THE THINGSLOG ALL THE THINGSLOG ALL THE THINGSLOG ALL THE THINGS


One little problem...


We have millions of devices...


We have thousands of servers...


We need to process them in real-time


“Netflix is a log generating company that happens to stream movies” - Adrian Cockroft


Massive amount of streaming logs


Reactive Extensions


Rx


The best ideas from the Observer pattern, the Iterator pattern, and functional programming


“lodash for events”




High-level intro


.map(value=>value*10)

.forEach(value=>{

console.log(value);

});

[1,2,3]



.forEach(value=>{

console.log(value);

});

[1,2,3]



.forEach(value=>{

console.log(value);

});

[1,2,3]



.forEach(value=>{

console.log(value);

});

[1,2,3]



.forEach(value=>{

console.log(value);

});

[1,2,3]



.forEach(value=>{

console.log(value);

});

[1,2,3]

//10



.forEach(value=>{

console.log(value);

});

[1,2,3]

//10//20



.forEach(value=>{

console.log(value);

});

[1,2,3]

//10//20//30


[1,2,3]

//10//20//30


.forEach(value=>{

console.log(value);

});


[1,2,3]

//10//20//30


.forEach(value=>{

console.log(value);

});


Observable.of(1,2,3)


.forEach(value=>{

console.log(value);

});




.forEach(value=>{

console.log(value);

});




.subscribe(value=>{

console.log(value);

});




.subscribe(value=>{

console.log(value);

});



//10


.subscribe(value=>{

console.log(value);

});



//10//20


.subscribe(value=>{

console.log(value);

});



//10//20//30


.subscribe(value=>{

console.log(value);

});


Array is a collection


Observable is a collection that arrives over time


Represents a stream


button.addEventListener('click',event=>{

console.log('youclicked!');

});


Observable.fromEvent(button,'click')

.subscribe(event=>{


});

.subscribe(event=>{


});



.debounceTime(500)

.subscribe(event=>{


});



.debounceTime(500)



.debounceTime(500)



.debounceTime(500)



.debounceTime(500)



.debounceTime(500)



.debounceTime(500)


.debounceTime(500)



.debounceTime(500)



.debounceTime(500)



.debounceTime(500)



.debounceTime(500)



.debounceTime(500)



.debounceTime(500)


500 ms


.debounceTime(500)


500 ms


.debounceTime(500)


500 ms


.debounceTime(500)


500 ms


.debounceTime(500)


500 ms


.debounceTime(500)


500 ms


.debounceTime(500)


500 ms


.debounceTime(500)


500 ms


.debounceTime(500)


500 ms 500 ms


.debounceTime(500)


500 ms 500 ms


.debounceTime(500)


500 ms 500 ms


.debounceTime(500)


500 ms 500 ms


.debounceTime(500)


500 ms 500 ms500 ms


.debounceTime(500)


500 ms 500 ms500 ms


.debounceTime(500)


500 ms 500 ms500 ms



.debounceTime(500)


Stream of logs


{

"path":"/something",

"status":500

}

Log message in JSON


import{webSocket}from"rxjs/observable/webSocket";

letstreamOfLogs=webSocket("ws://logs.netflix.com")

.subscribe(msg=>{

console.log(msg);

});


streamOfLogs

.filter(msg=>msg.status!==200)

.subscribe(msg=>{

console.log(msg);

});


streamOfLogs


500 200 404 301


500

streamOfLogs


404


Learning Rx isn't easy


It's worth it!


It's worth it!Jay!!


Case Study

Scaling high volume logs in real-time


We're going to log JSON


{


"status":200

}

Device{

"event":"StartPlay",

"device":"XBox360"

}

Server


Stream them all through a single pipeline


How do you find and process the logs you want


Write a custom job in Java?


Job: a unit of work


{


"status":500,

"duration":135,

//etc...

}

Server log message


streamOfLogs

.filter(event->event.get("status")!=200)

.map(event->

newJSONObject()

.put("path",event.get("path"))

.put("status",event.get("status"))

);


Powerful, but... not ideal


Query Language


SELECTpath,statusWHEREstatus!=200


Just-in-time (JIT) Compilation


SELECTpath,status,durationWHEREstatus!=200




streamOfLogs

.filter(event->event.get("status")!=200)

.map(event->

newJSONObject()

.put("path",event.get("path"))

.put("status",event.get("status"))

);



What's next?


8+ million messages per second, peakm

essa

ges

per s

econ

d

Time


How do we scale this?


Distribute work via autoscaling


A

Source

B

Server

B

Server

50%

50%

Load balancing a job


Chain jobs together


SELECTpath,statusWHEREsource=="API"

Chain jobs together


A

Job

B C

Job Job

Chain jobs together


High-volume, distributed systems have a problem...


Backpressure


“pressure opposed to the desired flow of gases in confined places such as a pipe” - wikipedia.org


A B C

Server Server Server

100 rps 75 rps


A B C


60 sec * 25 rps = 1,500 rpm!

100 rps 75 rps


A B C


100 rps 75 rps

60 sec * 60 min * 25 rps = 90,000 rph!


Buffer or Drop


onBackpressureBuffer

.onBackpressureBuffer()


streamOfLogs

.subscribe(value->{

//somethingexpensive

Thread.sleep(100);

});



streamOfLogs

.subscribe(value->{


Thread.sleep(100);

});



streamOfLogs

.subscribe(value->{


Thread.sleep(100);

});


1 2 3 4 5


[3,4,5]

1 2


onBackpressureDrop

.onBackpressureDrop()

streamOfLogs

.subscribe(value->{


Thread.sleep(100);

});



streamOfLogs

.subscribe(value->{


Thread.sleep(100);

});



streamOfLogs

.subscribe(value->{


Thread.sleep(100);

});



1 2 3 4 5


1 4


Job authors choose which


700+ jobs running 24/7 8+ million events/second

Autoscaling

MantisLow-latency, high throughput stream-processing job platform

http://techblog.netflix.com


We need somewhere to submit and view query results


Query Builder UI


SELECTpath,statusWHEREstatus!=200








Can be extremely high-volume

100k+ rps


We simply can’t render that fast!


Even if we could, it would be bad UX


Performance solutions are often driven by UX


So what do we do?


UI virtualization aka Virtual Table?

VirtualizedNOT Virtualized vs.


We still can’t update that virtual table 100k per second


This is also backpressure


Buffer or Drop


Buffer for 1 second


getWebSocket()

.bufferTime(1000)


Buffer size is unbounded


See what your users actually do


Users want a sample


Batch sampling


Drop after reaching a certain threshold


Batch sampling


Batch sampling


Batch sampling


Batch sampling


letbuffer=getWebSocket()

.bufferTime(1000);

letgate=newBehaviorSubject(true);

letbatchSize=50;

letbatchSizeCounter=0;

letresults=gate$

.switchMap(enabled=>enabled?buffer:Observable.never())

.do(buffer=>{

batchSizeCounter+=buffer.length;

if(batchSizeCounter>=batchSize){

//truncatesthearray,ifit'soverbatchSizebuffer.length=batchSize;

//turnsonthegate,pausingthestream

gate.next(false);

batchSizeCounter=0;

}

});

https://goo.gl/DMOqBA



.bufferTime(1000);


letbatchSize=50;


letresults=gate$


.do(buffer=>{





gate.next(false);

batchSizeCounter=0;

}

});




.bufferTime(1000);


letbatchSize=50;


letresults=gate$


.do(buffer=>{





gate.next(false);

batchSizeCounter=0;

}

});




.bufferTime(1000);


letbatchSize=50;


letresults=gate$


.do(buffer=>{





gate.next(false);

batchSizeCounter=0;

}

});




.bufferTime(1000);


letbatchSize=50;


letresults=gate$


.do(buffer=>{





gate.next(false);

batchSizeCounter=0;

}

});




.bufferTime(1000);


letbatchSize=50;


letresults=gate$


.do(buffer=>{





gate.next(false);

batchSizeCounter=0;

}

});




.bufferTime(1000);


letbatchSize=50;


letresults=gate$


.do(buffer=>{





gate.next(false);

batchSizeCounter=0;

}

});



Works for low-volume queries too


Raven


RavenSweet Christmas


improved debugging, testing, and InfoSec


Netflix loves Rx


Rx is powerful, cross-platform

Thanks!

@_jayphelps