Pivoting Spring XD to Spring Cloud Data Flow

Sabby Anandan | Product Manager | @sabbyanandan

business premise

microservices

what it means for data

tools / frameworks

what we learned

how it all comes together

Sense of Urgency

business results

business strategy

Sense of UrgencyOR

Need for Momentum?

business strategy

business results

business strategy

business results

Continuous Delivery

Continuous Delivery

Deliver Continuously1 2

frequent roll out updates, even multiple times a day

gives you more ways to maintain availabilityreduce risks

allows you to make sound technology choices

gives your team more velocity

microservices architecture

business strategy

business results

devops culture

continuous delivery

Casey West Principal Technologist, Cloud Foundry

“how do we get there?”

Continuously Delivered Microservices

Microservicesarchitecture composed of loosely coupled

elements that have bounded contexts

Adrian Cockcroft Battery Ventures

previously: Cloud Architect at Netflix

“what tools can I use?”

Spring

Spring BootAutoConfigure

StartersCLI

Actuator

Dev-Tools

Health Checks

Security

Logging

BUILDING ADAPTIVE APPLICATIONS IS HARD

built on Cloud Foundry

code will be open sourced.

Demo: Spring Boot

Emerging Challenges• separate deployable units (registry, discovery,

versioning) • reliable operationalization (fault tolerance) • too many moving parts (coordination, routing,

load balancing)

Spring

Spring Cloud

Spring Cloud

Hystrix Eureka Ribbon

Zuul Feign

Service Discovery Config Server Control Bus

Zookeeper Distributed TracingConfig Server

Spring Cloud@EnableConfigServer

@EnableEurekaServer

@EnableDiscoveryClient@EnableCircuitBreaker

@EnableHystrix

@EnableAtlas

@EnableTurbine

@EnableZuulProxy

OK, I get it....what’s specical about

microservices?

Microservicesa single application as a suite of small services,

each running in its own process and communicating with lightweight mechanisms, often

an HTTP resource API

James Lewis and Martin Fowler ThoughtWorks

Microservicesa single application as a suite of small services,

each running in its own process and communicating with lightweight

mechanisms, often an an HTTP resource API

OK, I get it....what if I have a lot

of data?

every software company cares about data

Data-driven Microservicescomposition of small standalone executable applications communicating over messaging

middleware

Mark Fisher and Mark Pollack Spring/Pivotal

Data-driven Microservicescomposition of small standalone executable

applications communicating over messaging middleware

operates by consuming from and producing to message queues and data stores

“i need real-time insights from data”

Spring

Developer ToolkitsSpring Integration

enables lightweight messaging within Spring-

based applications and supports integration with

external systems Spring Cloud Stream enables you to create powerful distributed streaming data applications with a minimal amount of coding

Spring Cloud Stream

@EnableBinding(Source.class)

@EnableRXJavaProcessor

@EnableBinding(Sink.class)

@EnableBinding(Processor.class)

http | transform | log

| = ?

http | transform | log

| = Binder

Binders

BUILDING ADAPTIVE APPLICATIONS IS HARD

built on Cloud Foundry

code will be open sourced.

Demo: Spring Cloud Stream

@EnableBinding(Source.class)

Discover OutputChannel

Discover Binder

Discover Type

Converter

Bind Producer

Auto Configure

$ java -jar http-source.jar

$ java -jar transform-processor.jar

@EnableBinding(Processor.class)

Discover Input and Output Channels

Discover Binder

Discover Type

Converter

Bind Producer and

ConsumerAuto

Configure

@EnableBinding(Sink.class)

Discover Input

Channel

Discover Binder

Discover Type

Converter

Bind Consumer

Auto Configure

$ java -jar log-sink.jar

“i have large volumes of asynchronous data

systems”

Spring

Reactive Streams@EnableRXJavaProcessor

10 11 12 13 14 15 16 17 18 19.. ..

10 11 12 13 14 15 16 17 18 19.. ..

i0

i112

17

“i need offline data analysis”

Spring

Batch-processing MicroservicesSpring Batch

enables the development of robust batch applications

vital for the daily operations of enterprise systems

Spring Cloud Task enables you to develop and run short-lived executable data applications locally or in the cloud

@EnableTaskDiscover

DataSourceBind

DataSourceAuto

Configure

$ java -jar timestamp-task.jar

BUILDING ADAPTIVE APPLICATIONS IS HARD

built on Cloud Foundry

code will be open sourced.

Recap

I need real-time insights from data

I need offline data analysis

Spring Cloud Stream

Spring Cloud Task

Stream

Batch

Stream Batch

Data Architecture

Spring

Spring XD

Kodiak: Turnkey Data Integration Platform Sep 2012

Apr 2013

Aug 2013

Jan 2014

Apr 2014

Nov 2014

Dec 2014

Jan 2015

Feb 2015

Batch Jobs (Gaps: Oozie; Flume; Sqoop)

PMML (closed-loop-analytics; IoT), YARN

Kafka and Sqoop Support

Spring XD on CloudFoundry (beta)

Spring XD Launch

Builds on Spring Boot

Spark and Python Support

Spark Streaming, RxJava, Reactor Support

Apr 2015 BDS [HAWQ, GPDB, GemFire, PHD 3.0/HDP]

Flo UI, HDP/Ambari, Lattice Integration Jun 2015

Nov 2015 Flo Composed Jobs/Workflows, Perf. Improvements

Kodiak: Turnkey Data Integration Platform Sep 2012

Apr 2013

Aug 2013

Jan 2014

Apr 2014

Nov 2014

Dec 2014

Jan 2015

Feb 2015

Batch Jobs (Gaps: Oozie; Flume; Sqoop)

PMML (closed-loop-analytics; IoT), YARN

Kafka and Sqoop Support

Spring XD on CloudFoundry (beta)

Spring XD Launch

Builds on Spring Boot

Spark and Python Support

Spark Streaming, RxJava, Reactor Support

Apr 2015 BDS [HAWQ, GPDB, GemFire, PHD 3.0/HDP]

Flo UI, HDP/Ambari, Lattice Integration Jun 2015

Nov 2015 Flo Composed Jobs/Workflows, Perf. Improvements

Not Quiet There• couldn’t scale without undeploying

running stream or batch pipelines • couldn’t do rolling upgrades • couldn’t do continuous deployments

longer running services

short lived processes

Data Microservices Architecture

Stream Batch

Spring XD

Spring Cloud Data Flow

Spring Cloud Data Flow isa cloud native programming and operating model

for composable data microservices on a structured platform

Spring Cloud Data Flow isa cloud native programming and operating model for composable data microservices on a

structured platform

Spring Cloud Data Flow isa cloud native programming and operating model for composable data microservices on a

structured platform@EnableBinding(Source)class Greeter { @InboundChannelAdapter(Source.OUTPUT) String greet() { "hello world" }}

canary deployments

continuous delivery

monitoring

@EnableBinding(Source.class)

@EnableBinding(Processor.class)

@EnableBinding(Sink.class)

Spring Cloud Data Flow isa cloud native programming and operating model for composable data microservices on a

structured platform

http

transform

log

job foo < bar || baz & jaz > bye

Streams

Jobsfoo

bar jaz

baz

bye

Spring Cloud Data Flow isa cloud native programming and operating model for composable data microservices on a

structured platformPrivate and Public PaaS

Polyglot

Multiple Platforms

Service Marketplace

Direct Access to Containers

Metrics, Logging, Monitoring

Dynamic Scaling and Resource Allocation

Integrated Security and Auditing

Spring Cloud Data Flow isa cloud native programming and operating model for composable data microservices on a

structured platformPrivate and Public PaaS

Polyglot

Multiple Platforms

Service Marketplace

Direct Access to Containers

Metrics, Logging, Monitoring

Dynamic Scaling and Resource Allocation

Integrated Security and Auditing

Spring Cloud Data Flow isa cloud native programming and operating model

for composable data microservices on a structured platform

……

Polyglot

Multiple Platforms

……

Direct Access to Containers

Metrics, Logging, Monitoring

Dynamic Scaling and Resource Allocation

Integrated Security and Auditing

YARN

XD Modules

XD Admin

XD Containers

Zoo

Keep

erZo

o Ke

eperAdmin / Flo UI

Shell

CURL

Database

AdminAdmin / Flo UI

Shell

CURL ??X

YARN

Stream/Task Boot Apps

Orchestrate Composable Data Microservices

HOST

Cloud Foundry YARN Local

Spring Cloud Data Flow

µPCF

Spring Cloud Stream/Task AppsSpring Cloud Stream Binders [Rabbit, Kafka, Redis]

BUILDING ADAPTIVE APPLICATIONS IS HARD

built on Cloud Foundry

code will be open sourced.

Demo: Spring Cloud Data Flow

cf push• here is my source code • run it on the cloud for me • i do not care how

yarn push• here is my source code • run it on the hadoop for me • i do not care how

“simple things should be simple, complex things should be possible”

- alan kay

ResourcesSpring Cloud Data Flow: http://cloud.spring.io/spring-cloud-dataflow/ Spring Cloud Data Flow Samples: https://github.com/spring-cloud/spring-cloud-dataflow-samples

Spring Cloud Stream: http://cloud.spring.io/spring-cloud-stream/ Spring Cloud Task: http://cloud.spring.io/spring-cloud-task/