CLOUD-NATIVE STREAMING AND EVENT-DRIVEN MICROSERVICES

Marius Bogoevici @mariusbogoevici

CLOUD-NATIVE STREAMING AND EVENT-DRIVEN MICROSERVICES

• Use cases: • predictive maintenance, fraud detection, QoS, IoT • enterprise integration meets ‘Big Data’

• Specific requirements • High throughput, low latency • Grouping, ordering, windowing

• Middleware adaptations: beyond JMS, Rabbit MQ • Kafka, Amazon Kinesis, Google Pub/Sub, Azure

Event Hubs • Brokerless: Aeron, ZeroMQ

Spring I/O 2016

CLOUD-NATIVE STREAMING AND EVENT-DRIVEN MICROSERVICES

• Architectural style advantages: • bounded context, development agility

• Process-specific resource tuning: • memory, CPU, instance count

• Messaging vs. HTTP • Discoverability • Availability • Consistency across heterogenous resources

Spring I/O 2016

CLOUD-NATIVE STREAMING AND EVENT-DRIVEN MICROSERVICES

• Modern platforms • Resource management • memory, CPU, instance count

• Scaling up/down • Health monitoring and failover • Routing and load balancing

Spring I/O 2016

ApacheYARNApacheMesos Kubernetes

Spring I/O 2016

dataflow:> stream create demo --definition “http | file”

Goal

Spring Cloud Stream Spring Cloud Data Flow

Spring I/O 2016

Spring Cloud Data Flow

Spring Cloud Stream

Spring Cloud Task

2015

Spring Cloud Stream

• Event-driven microservice framework • Built on battle-tested components • Spring Boot: full-stack standalone

applications • Spring Integration: Messaging, EIP

patterns, connectors • Opinionated primitives • Pluggable middleware abstractions

Spring I/O 2016

Spring Cloud Stream in a nutshell

Spring I/O 2016

…. 10000 ft nutshell

Spring I/O 2016

Durable publish-subscribe

• Fits both streaming and event-driven use cases

Spring I/O 2016

Consumer Groups

• Borrowed from Kafka, applied to all binders • Groups of competing consumers within the pub-sub

destination • Used for scaling and partitioning

Spring I/O 2016

Partitioning

• Required for stateful processing • Outputs specify a partitioning strategy • Inputs consuming from specific partition

Spring I/O 2016

Programming model

@EnableBinding + Binder implementation

Apache Kafka

Gemfire

Spring I/O 2016

DEMO

Spring Cloud Data Flow

• Orchestration: • DSL for Stream topologies • REST API • Shell • UI

• Portable Deployment SPI • OOTB apps for common integration use-

casesSpring I/O

2016

Stream DSL: pipes and filters

Stream definition

Boot Apps build on top of Spring Ecosystem

httpfile = http | file

|

Spring I/O 2016

‘pipe’, or how to communicate Rabbit, Kafka, Gemfire

Spring Cloud Deployer• SPI for deploying applications to modern runtimes

• Local (for testing) • Cloud Foundry • YARN • Kubernetes • Mesos + Marathon

• Different resources supported • Spring Boot Uberjars • Docker Images

• Different locations • File system, HDFS, HTTP, Maven, Docker Hub

Spring I/O 2016

Spring Cloud Data Flow

Pla$orm

Spring I/O 2016

Spring I/O 2016

DEMO

Timeline

• Spring Cloud Stream • 1.0.0.GA May 10, 2016 • 1.1.0.M1 July 2016 • 1.1.0.GA Q3 2016

• Spring Cloud Data Flow • 1.0.0.M3 May 7, 2016 • 1.0.0.RC1 End of May, 2016

Spring I/O 2016

Spring Cloud Stream 1.1.0.M1

• Reactive APIs: Reactor, RxJava • Kafka 0.9 • Schema evolution support

Spring I/O 2016

Q&A Marius Bogoevici

twitter: @mariusbogoevici