View
133
Download
0
Category
Preview:
Citation preview
Lightbend Fast Data Platform
A Technical Overview
A Bit of History:IT Services vs. Big Data
Big DataServices
Some Overlap: Concerns, Architecture
The Recent Past…
Microservices & Fast Data
Much More Overlap
The Future?
Drilling Down:Monoliths to Microservices
lightbend.com/reactive-microservices-architecture
• Tangled responsibilities, lead to infrequent, “big-bang” deployments
• App lifetimes months to forever!
Monoliths
• Each does one thing, so must be message driven & asynchronous
• Updates easier, deployments frequent • App lifetimes: minutes! to forever
Microservices
• But be careful: • Message overhead is much better than
function calls!
Microservices
OpenSourceCoreForDevelopmentinScala/Java
Produc8onSuiteForOpera8ons/DevOps
Lightbend Reactive Platform
Drilling Down:Big Data
YARN
HDFS
MRjob#1
MRjob#2
Flume Sqoop
DBs
SlaveNode
DiskDiskDiskDiskDisk
NodeMgr
DataNode
Master
ResourceManager
NameNode
Hadoop
YARN
HDFS
MRjob#1
MRjob#2
Flume Sqoop
DBs
SlaveNode
DiskDiskDiskDiskDisk
NodeMgr
DataNode
Master
ResourceManager
NameNode
Hadoop 2013: Embrace Spark
Sparkjob#1
Sparkjob#2
YARN
HDFS
MRjob#1
MRjob#2
Flume Sqoop
DBs
SlaveNode
DiskDiskDiskDiskDisk
NodeMgr
DataNode
Master
ResourceManager
NameNode
Go beyond batch?
Sparkjob#1
Sparkjob#2
The Emergence of Fast Data:(Time Is Money)
Respond to change
IoT
20
Logs
Sockets
Storage
HDFS,S3,…
SQL/NoSQL ES
KafkaStreams
Streaming
Streams
SQL
Batch
…
Streaming
Flink
Fast Data and Microservices; Are they Converging?
• Each [stream app or μservice]: • does one responsibility • ingests unending [data or messages]
Synergies
22
Logs
Sockets
Storage
HDFS,S3,…
SQL/NoSQL ES
KafkaStreams
Streaming
Streams
SQL
Batch
…
Streaming
Flink
• Each [stream app or μservice] must: • operate asynchronously • offer never-ending service
Synergies
23
Logs
Sockets
Storage
HDFS,S3,…
SQL/NoSQL ES
KafkaStreams
Streaming
Streams
SQL
Batch
…
Streaming
Flink
• These architectures are converging: 1.Similar design problems 2.Data becomes dominant problem
Thesis
24
Logs
Sockets
Storage
HDFS,S3,…
SQL/NoSQL ES
KafkaStreams
Streaming
Streams
SQL
Batch
…
Streaming
Flink
Lightbend Fast Data Platform
1. An accelerated on ramp for building streaming data systems, data applications, and other microservices.
Value Three Ways
2. Best practices guidance for solving specific design problems:
- Sample apps - Documentation - Enablement services
Value Three Ways
3. Machine learning-based monitoring and management:
-Keep your systems resilient, scalable, and responsive with minimal user intervention.
Value Three Ways
Lightbend Fast Data Platform V1.0
Internet
Logs
Sockets
DC/OS:OnPremiseorCloud
FDP
KafkaConnect
Storage
HDFS,S3,…
SQL/NoSQL ES
KafkaStreams
CloudHosted
FDP.ai
ClusterAnalysis
MachineLearningMicroservices
ProducDonSuite
MachineLearning
ML
Streaming
Streams
SQL
Batch
…
Streaming
Flink
Intelligent
Man
agem
ent
…Co
nsoles
Lightbend Fast Data Platform V1.0
Internet
Logs
Sockets
DC/OS:OnPremiseorCloud
FDP
KafkaConnect
Storage
HDFS,S3,…
SQL/NoSQL ES
KafkaStreams
CloudHosted
FDP.ai
ClusterAnalysis
MachineLearningMicroservices
ProducDonSuite
MachineLearning
ML
Streaming
Streams
SQL
Batch
…
Streaming
Flink
Intelligent
Man
agem
ent
…Co
nsoles
Lightbend Fast Data Platform V1.0
Internet
Logs
Sockets
DC/OS:OnPremiseorCloud
FDP
KafkaConnect
Storage
HDFS,S3,…
SQL/NoSQL ES
KafkaStreams
CloudHosted
FDP.ai
ClusterAnalysis
MachineLearningMicroservices
ProducDonSuite
MachineLearning
ML
Streaming
Streams
SQL
Batch
…
Streaming
Flink
Intelligent
Man
agem
ent
…Co
nsoles
Lightbend Fast Data Platform V1.0
Internet
Logs
Sockets
DC/OS:OnPremiseorCloud
FDP
KafkaConnect
Storage
HDFS,S3,…
SQL/NoSQL ES
KafkaStreams
CloudHosted
FDP.ai
ClusterAnalysis
MachineLearningMicroservices
ProducDonSuite
MachineLearning
ML
Streaming
Streams
SQL
Batch
…
Streaming
Flink
Intelligent
Man
agem
ent
…Co
nsoles
Service 1
Log & Other Files
Internet
Services
Service 2
Service 3
Services
Services
N * M links ConsumersProducers
Service 1
Log & Other Files
Internet
Services
Service 2
Service 3
Services
Services
N + M links ConsumersProducers
Lightbend Fast Data Platform V1.0
Internet
Logs
Sockets
DC/OS:OnPremiseorCloud
FDP
KafkaConnect
Storage
HDFS,S3,…
SQL/NoSQL ES
KafkaStreams
CloudHosted
FDP.ai
ClusterAnalysis
MachineLearningMicroservices
ProducDonSuite
MachineLearning
ML
Streaming
Streams
SQL
Batch
…
Streaming
Flink
Intelligent
Man
agem
ent
…Co
nsoles
Lightbend Fast Data Platform V1.0
Internet
Logs
Sockets
DC/OS:OnPremiseorCloud
FDP
KafkaConnect
Storage
HDFS,S3,…
SQL/NoSQL ES
KafkaStreams
CloudHosted
FDP.ai
ClusterAnalysis
MachineLearningMicroservices
ProducDonSuite
MachineLearning
ML
Streaming
Streams
SQL
Batch
…
Streaming
Flink
Intelligent
Man
agem
ent
…Co
nsoles
Lightbend Fast Data Platform V1.0
Internet
Logs
Sockets
DC/OS:OnPremiseorCloud
FDP
KafkaConnect
Storage
HDFS,S3,…
SQL/NoSQL ES
KafkaStreams
CloudHosted
FDP.ai
ClusterAnalysis
MachineLearningMicroservices
ProducDonSuite
MachineLearning
ML
Streaming
Streams
SQL
Batch
…
Streaming
Flink
Intelligent
Man
agem
ent
…Co
nsoles
• Low latency? How low? • High volume? How high?
Streaming Tradeoffs (1/3)
38
• Which kinds of data processing & analytics are required?
• How will this processing be done? • Individual processing of events? • Bulk processing of records?
Streaming Tradeoffs (2/3)
39
• Which tools and data sources/sinks must interoperate with your streaming tool?
Streaming Tradeoffs (3/3)
40
41
42
• Low latency • Low volume • Complex flows • Complex Event
Processing
43
• Med. latency • High volume • Data flows, SQL • En masse
processing
44
• Low latency • High volume • Data flows,
correctness • En masse
processing
45
• Low latency • Med. volume
• ETL, “tables” • Data flow or
per event
Lightbend Fast Data Platform V1.0
Internet
Logs
Sockets
DC/OS:OnPremiseorCloud
FDP
KafkaConnect
Storage
HDFS,S3,…
SQL/NoSQL ES
KafkaStreams
CloudHosted
FDP.ai
ClusterAnalysis
MachineLearningMicroservices
ProducDonSuite
MachineLearning
ML
Streaming
Streams
SQL
Batch
…
Streaming
Flink
Intelligent
Man
agem
ent
…Co
nsoles
Lightbend Fast Data Platform V1.0
Internet
Logs
Sockets
DC/OS:OnPremiseorCloud
FDP
KafkaConnect
Storage
HDFS,S3,…
SQL/NoSQL ES
KafkaStreams
CloudHosted
FDP.ai
ClusterAnalysis
MachineLearningMicroservices
ProducDonSuite
MachineLearning
ML
Streaming
Streams
SQL
Batch
…
Streaming
Flink
Intelligent
Man
agem
ent
…Co
nsoles
Kafka & Spark Metrics
Correlated Troubleshooting
Lightbend Fast Data Platform V1.0
Internet
Logs
Sockets
DC/OS:OnPremiseorCloud
FDP
KafkaConnect
Storage
HDFS,S3,…
SQL/NoSQL ES
KafkaStreams
CloudHosted
FDP.ai
ClusterAnalysis
MachineLearningMicroservices
ProducDonSuite
MachineLearning
ML
Streaming
Streams
SQL
Batch
…
Streaming
Flink
Intelligent
Man
agem
ent
…Co
nsoles
• Only metrics are uploaded, not sensitive domain data.
• One less service for you to manage. • …
Why Only Hosted Services?
• … • We can rapidly evolve this service.
without impacting your environment. • You benefit from aggregated knowledge
from all FDP clusters.
Why Only Hosted Services?
Lightbend Fast Data Platform V1.0
Internet
Logs
Sockets
DC/OS:OnPremiseorCloud
FDP
KafkaConnect
Storage
HDFS,S3,…
SQL/NoSQL ES
KafkaStreams
CloudHosted
FDP.ai
ClusterAnalysis
MachineLearningMicroservices
ProducDonSuite
MachineLearning
ML
Streaming
Streams
SQL
Batch
…
Streaming
Flink
Intelligent
Man
agem
ent
…Co
nsoles
• Alpha: Jan 9 • Beta 1: April • Beta 2: July • V1.0: September
Timeline
54
Lightbend Fast Data Platform V1.0
Internet
Logs
Sockets
DC/OS,MarathonOnPremiseorCloud
FDP
KafkaConnect
Storage
HDFS,S3,…
SQL/NoSQL ES
KafkaStreams
CloudHosted
GravitonClusterAnalysis
MachineLearningMicroservices
ProducConSuite
MachineLearning
ML
Streaming
Streams
SQL
Batch
…
Streaming
Flink
Intelligent
Man
agem
ent
…Co
nsoles
For More Information:
lightbend.com/fast-data-platform
For More Information
lightbend.com/ reactive-microservices-architecture
bit.ly/lightbend-fast-data
For More Information
• FDP: lightbend.com/fast-data-platform • Book: bit.ly/lightbend-fast-data
Recommended