Stream Based Architecture

@_tommichiels_#Devoxx #Streambased

Stream Based Architecture punching and crunching

tom.michiels@archimiddle.com

During this talk I will try to convince you that we need to evolve to stream based architectures

Setting the stageOur actors

Punchers

Crunchers

Birth, First Encounter, The LoveStory

Crunchers life

1950 1970 1990

Machine Code

Imperative

Crunchers life

Declarative

Punchers live

Data models

Hierarchical

Network

RelationalNavigational

Normalisation

Isolate data so that additions, deletions, and modifications can be made in just

one table and then propagated through the rest of the database using the defined foreign keys

1950 1970 1990

Navigational

Punchers life

Relational

The marriage, thx Edgar

Normalisation

Wedding vows

Crunch(calculate)

Punch(store)

One happy family

And then it started to get complicate

Complexity

PerformanceConcurrent users/operations

Performance

Complexity(Re)act to changing world

ComplexityHistorical reasons

ComplexityLegacy

Puncher problems

• Persistence / IO ?•Model ?

Cruncher problems

Complexity

•Abstractions ?•Reuse?

Crunch(calculate)

Punch(store)

Scaling up(centralised)

Crunch(calculate)

Punch(store)

Scaling out(distributed)

Crunch(calculate)

Punch(store)

Scaling out(distributed)

Propagating Through

•File•DB•RPC•Message

Centralised vs distributed

Complexity

Distributed

Centralised

Layers

Crunch(calculate)

Punch(store)

2 vs N tier

Complexity

Distributed

Centralised

2 tier N tier

DataDriven

Complexity

Distributed

Centralised

DataDriven

2 tier N tier

ETLExtract

Transform

Abstraction

Crunchers midlife crisis

1950 1970 1990

Machine Code

ProceduralObject Oriented

Crunchers midlife crisis

AppDriven

Complexity

Distributed

Centralised

AppDriven

2 tier N tier

ORM Hell

Impedance Mismatch

Monolith Hell

Spaghetti integration

Service Oriented

Complexity

Distributed

CentralisedService Oriented

2 tier N tier

Satellite Services

client

Service interface

Data aggregates

O/R mapping

Punchers midlife crisis

•Partitions•Eventual Consistence

First Class Citizens

MicroServices

client

Service

What about reports?

DataLake

Distributed Data Driven

Complexity

Distributed

Centralised

2 tier N tier

Distributed DataDriven

Distributed Data Driven

Complexity

Distributed

Centralised

2 tier N tier

Distributed DataDriven

DataDrivenAppDriven

Service Oriented

How did we end up here!!!!

Shared DB

Messaging

The Divorce(ETL)

Remember our wedding vows

Get us out of this mess!

Immutable

an immutable object is an object whose state cannot be modified after it is created

Stream

10 10 - 10 10 8 vs

is a (ordered) sequence of immutable objects

Stream Computing

10 + 10+ (- 10) = 10

state is derivedintent

Back to Maths

•Associativity•Commutative

Back to Maths

10 + 10+ (- 10) = 10

Binary Operation

Back to Maths

A ∪ B

Binary Operation

Birth of a commutative monoid

(10 + 10)+ (- 10) = ((-10) +10) + 10

Calculation can be partitioned

How we combine intermediate results is not important

Programming with solid base in maths

Examples of streams

•Logfiles•Journals/Ledgers•Atom Feeds

Propagating New Style

•Make stream available•Consumers keep track of consumed events

•Producers Schema on Write

Wedding vows renewed

Propagate immutable events by using streams and use stream crunching as local as possible.

Wedding vows renewed

Not Only ETL

Streams on Micro(Service) Level

•EventSourcing•CQRS

Streams on Micro Level

Streams on Macro level

kappa architecture

Reports vs Dashboards

Fast Data

• BloomFilter

• HyperLogLog

• Count-MinSketch

• MinHash

Stream processing

Smack Stack https://github.com/killrweather/killrweather

Happily Ever After

Stream Based Architecture

Technology

Ontology-based Stream/Sensor Data Modeling

Architecture for Stream OLAP Exploiting SPE and OLAP Engine...we propose a novel architecture for online analytical processing (OLAP) over streams exploiting off-the-shelf stream processing

Stream-based Data Synchronization

Android's New Stream-Based Camera Architecture

Stream Cube: An Architecture for Multi-Dimensional Analysis of Data

Stream Cube: An Architecture for Multi-Dimensional Analysis of …jpei/publications/streamcube-dpd... · 2006-01-16 · STREAM CUBE: AN ARCHITECTURE FOR MULTI-DIMENSIONAL ANALYSIS

LFSR-based Stream Ciphers - Inria · PDF fileChapter 3 LFSR-based Stream Ciphers Inordertominimizethesizeoftheinternalstate,streamciphersdedicatedtolow-costhard-ware implementations

Grid-based Data Stream Processing in e-Science€¦ · Lehrstuhl Informatik III: Datenbanksysteme Grid-based Data Stream Processing in e-Science 1 Grid-based Data Stream Processing

A Flexible IoT Stream Processing Architecture Based on

Cross-stream Project-based Learning

Apache Apex: Stream Processing Architecture and Applications

Stream for Kx · Stream for Kx 1 ` Stream for Kx is a ... quality and on-budget delivery. ... horizontally using a simple architecture and boasting over 20 storage data types, Stream

Semantic Stream Reasoning: Introduction and Architecture ...klusch/i2s/ICM-Hydraulic_AMA-WR-final.pdf · 16.03.2016 Intelligent Condition Monitoring 3 Condition-based maintenance

Value Mapping –Critical Business Architecture Viewpoint · Value Stream –Metamodel 11 Value Stream Value Stream Stage Name: Text Description: Text /Is Start Stage: Yes/No Stakeholder

Fjording The Stream An Architecture for Queries over Streaming Sensor Data

Stream Processor Architecture

Stream Processing Main References: “Comparing Reyes and OpenGL on a Stream Architecture”, 2002 “Polygon Rendering on a Stream Architecture”, 2000 Department

A Scalable Big Stream Cloud Architecture for the Internet of Things

COMPUTER GRAPHICS ON A STREAM ARCHITECTURE

DEM-Based Stream and Watershed Delineation