Building Reactive Distributed Systems For Streaming Big Data, Analytics & Machine Learning

1

Building Reactive Distributed Systems For Streaming Big Data, Analytics & ML

Helena Edelson @helenaedelson Reactive Summit 2016

https://twitter.com/helenaedelson

@helenaedelson #reactivesummit

Distributed Systems & Big Data Platform Engineer, Event-Driven systems, Streaming Analytics, Machine Learning, Scala

Committer: FiloDB, Spark Cassandra Connector, Kafka Connect Cassandra

Contributor: Akka (Akka Cluster), prev: Spring Integration

Speaker: Reactive Summit, Kafka Summit, Spark Summit, Strata, QCon, Scala Days, Philly ETE

2

I play with big data

twitter.com/helenaedelson slideshare.net/helenaedelson

linkedin.com/in/helenaedelson github.com/helena

and think about FT


http://www.slideshare.net/helenaedelson

https://www.linkedin.com/in/helenaedelson

http://github.com/helena


If you’re here you’ve taken the red pill


It’s Not Easy Being…

Globally Distributed

Eventually Consistent

Highly Available

While handling TBs of data

Self Healing

Fast

4


Massive event spikes & bursty traffic

Fast producers / slow consumers

Network partitioning & out of sync systems

DC down

Failure before commit

DDOS'ing yourself - no backpressure

Lack of graceful lifecycle handling, i.e. data loss when auto-scaling down

5


Beware the C word

Consistency?

6

Convergence?

@helenaedelson #reactivesummit 7

Complexity, or…


http://www.slideshare.net/palvaro/ricon-keynote-outwards-from-the-middle-of-the-maze/42

The Real Topic ^Other


“Everything fails, all the time”

Start with this premise

Build failure, chaos routing and intelligence into your platform infrastructure

Be fault proactive vs just reactive

9

- Werner Vogels


The matrix is everywhere…

It is a world where humans vs machines need to be directly involved in health of systems, respond to alerts…


Log analysis tools like Splunk perpetuate this illusion…


Self Healing, Intelligent Platforms


Self Healing, Intelligent Platforms

Imagine a world

where engineers designed software

while the machines did triage, failure analysis and took action to correct failures

and learned from it…


Alerting and Failure Response

The New Boilerplate


Building Systems That LearnYou have systems receiving, processing & collecting data. They can provide insight into trends, patterns, and when they start failing, your monitoring triggers alerts to humans.

Alternative

1. Automatically route data only to nodes that can process it.

2. Automatically route failure contexts to event streams with ML algorithms to learn: What is normal? What is an anomaly? When these anomalies occur, what are they indicative of?

3. Automate proactive actions when those anomalies bubble up to a fault-aware layer.

15


Define Failures vs Error Fault Tolerance Pathways & Routing

Failures - more network related, connectivity, …

Errors - application errors, user input from REST requests, config load-time errors…

Escape Loops and Feedback system

Built-in failure detection


I want to keep all my apps healthy, handle errors with a

similar strategies, and failures with a similar strategy…

…and share that data across datacenters with my multi-dc aware kernel to build smarter,

automated systems.


In A Nutshell

Event Stream

Petabytes Multi-DC Aware Mgmt

Kernel

Clustered Apps

Clustered Apps

Clustered AppsClustered

Apps

Clustered Apps

Clustered Apps

Clustered Apps

Clustered Apps


In A Nutshell

Multi-DC Aware Mgmt

Kernel

Clustered Apps

Clustered Apps


Apps

Clustered Apps

Clustered Apps

Clustered Apps

Clustered Apps

Danger Sauce

Event Stream

Petabytes


Multi-DC Aware Mgmt

Kernel

20

X-DC-Aware Kernel

Clustered Apps

Clustered Apps


Apps

Clustered Apps

Clustered Apps

Clustered Apps

Clustered Apps

Event Stream

Petabytes Danger Sauce


I want to

route failure context

(when a node is alive

enough to talk)

to my.failures.fu Kafka topic for

my X-DC kernel to process through my ML streams

to learn about usage,

traffic, memory patterns and

anomalies, to anticipate failures,

intercept and correct

Without Human Intervention


Akka Cluster it. The cluster knows

the node is down. Many options/

mechanisms to send/publish failure

metadata to learning streams.

If it’s a network failure and my node can’t

report to the kernel?

Bring it


I think I’ll start with a

base cluster-aware Akka Extension for

my platform.

Any non-JVM apps can share data

with these via Kafka, ZeroMQ,

Cassandra…

Sew easy.

Might sneak it in without a

PR.


Then I can bake my fault tolerance

strategy & route awareness in every

app & node.

And make it know when to use

Kafka and when to use Akka for

communication.


And all logic and flows for graceful handling of

node lifecycle are baked in to the framework too.

Dude, that’s tight.

Graceful Lifecycle


Akka ClusterNode-aware cluster membership

Gossip protocol

Status of all member nodes in the ring

Tunable, automatic failure detection

No single point of failure or bottleneck

A level of composability above Actors & Actor Hierarchy

26


Akka Cluster has tunable Gossip.I

can partition segments of my node ring

by role, and route data by role or

broadcast.

Tunable Gossip


Akka Cluster has

AdaptiveLoadBalancing logic.

I can route events away from nodes in trouble and

directed to healthier nodes. Many options, highly

configurable, good granularity.

AdaptiveLoadBalancing* Router

Cluster Metrics API


Dynamically Route by Node Health Cluster Metrics API & AdaptiveLBR

Leader

Let’s pummel my cluster with critical data !



Oh look, slow consumers or something in GC

thrash…Leader

Uses EWMA algo to weight decay of health data



Leader

The Cluster leader ‘knew’ the red

node was nearing capacity & routed my traffic to healthy nodes.


Reactive - Proactive

32


Proactive - Parallelized

Akka Cluster on nodes

Cluster LB routers

Node router Actors

Kafka replication & routing across datacenters

33



I drank the cool aid,

put on the costume,

now what…


Embrace Uncertainty


So many opportunities for failure in this statement

Eventually Consistent Across DCs

37


there is no now…

Embrace Asynchrony & Location Transparency

…or where


There Is No Now

39

US-East-1

MirrorMakerEU-west-1

ZK

ZK

Akka micro

services

Akka micro

services

Akka micro

services

FiloDB Cassandra

Akka micro

services

Akka micro

services

Akka micro

services

Spark

Compute Clusters

Analytics/Timeseries/ML Storage Storage Clusters

S3/ Cold

Raw Event Stream

Event Stream

DC-1

DC-2

Akka Cluster-ed apps, partitioned by Role=service Raw Event Replay

FlinkBackpressure


US-East-1


ZK

ZK

Akka micro

services

Akka micro

services

Akka micro

services

FiloDB Cassandra

Akka micro

services

Akka micro

services

Akka micro

services

Spark

Compute Clusters

Analytics/Timeseries/ML Storage Storage Clusters

S3/ Cold

Raw Event Stream

Raw Event Replay

Event Stream

But I Can Replay Future[Then]

DC-1

DC-2

Akka Cluster-ed apps, partitioned by Role=service

Backpressure


Replaying Data from Time (T)

General time - We updated some compute algorithms which run on raw and aggregated data. I need to replay some datasets against the updated algos

Specific time - A node crashed or there was a network partitioning event and we lost data from T1 to T2. I need to replay from T1 to T2 without having to de-dupe data

41


Cassandra Bakes Clustered Ordering Into Data ModelCREATE TABLE timeseries_t ( id text, year int, month int, day int, hour int, PRIMARY KEY ((wsid), year, month, day, hour)) WITH CLUSTERING ORDER BY (year DESC, month DESC, day DESC, hour DESC);

42

Now the read path queries automatically return sorted by most recent. No sorts to do or ordering in your code.

Can also easily bucket in cold storage for faster reads.


US-East-1


ZK

ZK

Compute Clusters

Event Stream

Build Apps Not Clusters?App clusters

vs Infra clusters

Actors & Kafka Streams



Event Stream

Event Stream


Kafka Streams Key Features

Secure stream processing using Kafka security

Elastic and highly scalable

Fault-tolerant

Stateful and stateless computations

44


Kafka Streams Key FeaturesInteractive queries

Time Model & Windowing

Supports late-arriving and out-of-order data

Millisecond processing latency

At-least-once processing guarantees

exactly-once is in progress :)

45


val builder = new KStreamBuilder()

// Consume from topic x in the stream, from any or one DC

val kstream: KStream[K,V] = builder.stream(des, des, “fu.input.topic")

// Do some analytics computations

// Publish to subscribers in all DCs in the stream

kstream.to(“serviceA.questionB.aggregC.topic”, …)

// Start the stream

val streams = new KafkaStreams(builder, props)

streams.start()

// Do some more work then close the stream

streams.close()

Kafka Streams: KStream

46


Kafka Streams: KTableval locations: KTable[UserId, Location] = builder.table(“user-locations-topic”)

val userPrefs: KTable[UserId, Prefs] = builder.table(“user-preferences-topic”)

// Join detailed info from 2 streams as events stream in

val userProfiles: KTable[UserId, UserProfile] = locations.join(userPrefs, (loc, prefs) -> new UserProfile(loc, prefs))

// Compute statistics

val usersPerRegion: KTable[UserId, Long] = userProfiles

.filter((userId, profile) -> profile.age < 30)

.groupBy((userId, profile) -> profile.location)

.count()


val streams = new KafkaStreams(builder, props)

streams.start()

// Called when a stream thread abruptly terminates

// due to an uncaught exception.

streams.setUncaughtExceptionHandler(myErrorHandler)

Kafka Streams DSL Basics

48


Integrating IntelligenceFor Fast Data and Feedback Loops


Translation Layer

50

US-East-1

MirrorMaker

EU-west-1

PubSub Apps

PubSub Apps

PubSub Apps

Compute Clusters

Event StreamDC-1Raw Data

Cassandra

Spark SQLSpark MLLib

Spark Streaming

Kafka


Translation Layer

51

US-East-1

MirrorMaker

EU-west-1

PubSub Apps

PubSub Apps

PubSub Apps

Compute Clusters


Cassandra

To Columnar Format

Reads chunks, translates to spark

rows

Spark MLLib Spark SQL

Spark Streaming

Kafka

FiloDB


Translation Layer

52

US-East-1

MirrorMaker

EU-west-1

PubSub Apps

PubSub Apps

PubSub Apps

Compute Clusters


Cassandra

To Columnar Format

Reads chunks, translates to spark

rows

Spark MLLib Spark SQL

Spark Streaming

Kafka

FiloDB

Faster Ad Hoc

Querying

Faster ML

Feedback

Instead of slowing things down, it makes it faster.


Your Akka Cluster got in my Reactive Distributed Analytics

Database!

Walking with a jar of PB as one does…


FiloDB: Reactive

Scala & SBT

Spark (also written in scala)

Akka actors & Akka Cluster for Coordinators

Futures for IO

Typesafe Config

54


import filodb.spark._

KafkaUtils.createDirectStream[..](..)

.map(_._2)

.map(FuEvent(_))

.foreachRDD { rdd =>

sql.insertIntoFilo(rdd.toDF, “fu_events”, database = Some(“org1Keyspace”))}

55

FiloDB API: Append only, to existing dataset

Write: Kafka, Spark Streaming, Spark SQL & FiloDB


Read: FiloDB & Spark SQL

56


val df1 = sqlContext.read.format(“filodb.spark”) .option(“dataset”, “aggrTable”) .load()

val df2 = sqlContext.filoDataset(“aggrTable”, database = Some(“ks2”))

More typesafe than the dataframe read (df1)


DetermineTrain the model

Train on historic data

Score

Based on Objectives

Proactive Responses

React in real time

Machine Learning


What do I want to predict & what can I learn from the data?

Which attributes in datasets are valuable predictors?

Which algorithms will best uncover valuable patterns?

58

Optimization & Predictive Models


I need fast ML feedback loops, what tools should I pick? Depends…


Read from stored training data into MLLib

60

import org.apache.spark.mllib.stat.Statistics


val df = sqlContext.filoDataset(“table”, database = Some(“ks”))

val seriesX = df.select(“NumX”).map(_.getInt(0).toDouble)

val seriesY = df.select(“NumY”).map(_.getInt(0).toDouble)

val correlation: Double = Statistics.corr(seriesX, seriesY, “pearson”)

Calculate correlation between multiple series of data


Read from the stream to MLLib & store for later work in FiloDB

61

val stream = KafkaUtils.createDirectStream[..](..) .map(transformFunc) .map(LabeledPoint.parse)

val trainingData = sqlContext.filoDataset(“trainings_fu")

val model = new StreamingLinearRegressionWithSGD() .setInitialWeights(Vectors.dense(weights)) .setStepSize(0.2).setNumIterations(25) .trainOn(trainingData)

model .predictOnValues(stream.map(lp => (lp.label, lp.features))) .insertIntoFilo(“predictions_fu")


Parting Thoughts


Don’t take when someone says X doesn’t work at face value.

It may not have been the right choice for their use case

They may not have configured/deployed/wrote code for it properly for their load

All technologies are optimized for a set of things vs every thing

Invest in R&D

63

Cultivate Skepticism


Modular does one thing

Decoupled knows one thing

Reusable & Extendable when not final or private for a reason

64

Keep Everything Separate


Did you just duplicate codez?


Versus duplicating, with disparate strategies, all over your teams & orgs

Build A Reactive / Proactive Failure & Chaos Intelligence System for platform infrastructure


Byzantine Fault Tolerance?

67

Looks like I'll miss

standup


twitter.com/helenaedelson slideshare.net/helenaedelson

github.com/helena

Thanks!


http://www.slideshare.net/helenaedelson

http://github.com/helena

Technology

Building Reactive Distributed Systems For Streaming Big Data, Analytics & Machine Learning