Upload
jenn-rawlins
View
244
Download
0
Embed Size (px)
Citation preview
Jennifer Rawlins
Real Time Streaming with Kafka - for the data scientist
About MeJenn Rawlins has been creating software solutions for 19 years. She began her career at Microsoft as an engineer in test, and as an international program manager. This was followed by management and consultant roles, working with VPs and Directors across multiple industries to create custom software solutions. She then changed her focus to software engineering roles.
Recently Jenn has created Big Data solutions using Hadoop, Yarn, Kafka, and Cassandra, writing real time streaming solutions in Java and Scala. Her current focus is a solution in AWS for IoT devices.
AGENDA
❖ Messaging Systems
❖ Kafka
❖ SparkR
❖ Data Processing Pipelines
What is a message queueing systemMessages are sent to a queue. Messages are read from a queue. The queue is independent of the senders or receivers (Publishers/Subscribers or Producers/Consumers). Fast, Predictable, easy to scale.
Cloud solutions
Amazon SQS - Simple Queue Service
Azure service bus
Server Solutions
Kafka
IBM WebSphere MQ
RabbitMQ
ActiveMQ
Windows Servers MSMQ
KafkaLinkedIn uses Apache Kafka as a central publish-subscribe log for integrating data between applications, stream processing, and Hadoop data ingestion.
REAL-TIME STREAMING
1.Data pipelines that reliably get data between systems or applications.
2.Applications to transform or react to streams of data.
Real Time Process streams of records as they occur. Data in, Data out.
Fault Tolerant Store streams of records in a fault-tolerant way.
Highly Scalable (Horizontal) Nodes can be added and removed from a
Kafka Cluster and the cluster will rebalance itself. High Availability begins at 5 Nodes.
Ordering guaranteed within a partition as it was received
Parallel processing of partitioned topics
Multi publisher (producer) - kafka writes message as received to a specific topic, balancing across multiple partitions.
Multi subscriber (consumer) - Partitions assigned to specific subscriber.
Producer
ProducerProduce
rProducerProduce
rProducer
Consumer
ConsumerConsume
rConsumerConsume
rConsumer
KafkaProducer
Consumer
Consumer
Kafka Cluster
Producer
Producer
Consumer
Record consists of a key, a value, and a timestamp. (message)
Topic kafka stores streams of records in categories called topics.
Cluster Kafka is run as a cluster on one or more servers.
Broker The actual server, and synchronization layer between server instances.
Node The logical kafka entity or ‘worker’ on each server.
Publish and subscribe to streams of records. Similar to a message queue or enterprise messaging system.
Publish and Consume streams of records.
Process streams of records efficiently and in real time.
Store streams of records safely in a distributed, replicated cluster. Fault Tolerant.
A Stream is an unbounded, continuously updating data set. A stream is an ordered, replayable, and fault-tolerant sequence of immutable data records.
A Stream DSL is stateful, and is a processor topology.
# Example: a record stream for page view events
1 => {"time":1440557383335, "user_id":1, "url":"/home?user=1"}
5 => {"time":1440557383345, "user_id":5, "url":"/home?user=5"}
2 => {"time":1440557383456, "user_id":2, "url":"/profile?user=2"}
1 => {"time":1440557385365, "user_id":1, "url":"/profile?user=1"}
Typical Use Cases
Message Broker ActiveMQ or RabbitMQ
Website Activity Tracking
Metrics - monitoring
Log Aggregation
Stream Processing
Event Sourcing
Website Activity Tracking
TRACKING - Web Site Activity
Add clicks
page views,
searches,
or other actions users may take
Record of each activity is published to central topics, with one topic per activity type.
Application
Connector
RealTimeProcessor
Application
Application
Connector
Kafka Cluster
DataStore
DataStore
Application
Producers (write)
DataStore
Processor
Real Time
Consumers (read)
Connectors
Stream Processors
User Action
Platforms Spark runs on Hadoop Yarn, Apache Mesos, in Standalone cluster mode, or in the on EC2.
Languages Can be used from Scala, Python, and R shells
Processing optimizes jobs running on Hadoop in memory by 100x, or 10X faster on disk.
R limitationsR is a popular statistical programming language used for data processing and machine learning tasks.
Data Analysis is usually limited to a single thread, and the memory available on a single computer.
Developed at the AMPLab, it was accepted and merged into Spark version 1.4
Provides an R frontend to apache Spark
Uses the Sparks data sources API to read from a variety of sources: Hive(Hadoop), Json Files, Parquet Files.
Uses Spark’s distributed computation engine to run large scale data analysis from the R shell on a cluster: Many Cores, Many Machines.
SparkDataFrame (distributed collection of data organized in named columns) inherit optimizations from the computation engine.
SparkR: R package for Apache Spark
MLib and SparkR
Machine Learning algorithms currently supported:
Generalized Linear Model
Accelerated Failure Time (AFT)
Survival Regression Model
Naive Bayes Model
KMeans Model
SparkR uses MLib to training the model.
Real Time Record Processing
Example Real Time Scenario: Serve up related ads to user that are more likely to be clicked
Kafka Data Stream
Spark StreamingWebsiteUser Clicks Ad Record added to
AdClick TopicAdClick run Ad through model to update predictive score
ApplicationLog Click RecordUse AdClick to find related ads to serve to user using predictive scoring.
Display New Ads to User
Real-time process user data using an R model in a Spark job.
Batch process data from Kafka, Hadoop HDFS, SQL, Cassandra, HBase
Model Training multiple times with SparkR from multiple data sources
Historical Record Batch ProcessingSparkRKafka Data Streams
AdClick
HomePageView
Spark job
AdClick topic: run recent records through model
RSpark & SparkHadoopHive
AdClick model training on historical data
Cassandra
SQL
Pull Topics to create stores of data for many related features
AdView Kafka Topic
Language
Kafka is written in Java
In Kafka the communication between the clients and the servers is done with a simple, high-performance, language agnostic TCP protocol. This protocol is versioned and maintains backwards compatibility with older version. We provide a Java client for Kafka, but clients are available in many languages0
.JavaC/C++PythonGo (AKA golang)Erlang.NETClojureRubyNode.js
Proxy (HTTP REST, etc)Perlstdin/stdoutPHPRustAlternative JavaStormScala DSL Clojure
Kafka http://kafka.apache.org/
Free and Open Source Software under the Apache License
Github code repo: https://github.com/apache/kafka
Confluent http://www.confluent.io/
Open Source offering Consulting, Training, Support, Monitoring Tools
Confluent Docs: http://docs.confluent.io/3.0.0/streams/developer-guide.html
Examples: https://github.com/confluentinc/examples/tree/kafka-0.10.0.0-cp-3.0.0/kafka-streams/src/main/java/io/confluent/examples/streams
ResourcesLinkedIn Story:
https://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying
Benchmarking:
https://engineering.linkedin.com/kafka/benchmarking-apache-kafka-2-million-writes-second-three-cheap-machines
SparkR
https://spark.apache.org/docs/latest/sparkr.html
https://databricks.com/blog/2015/06/09/announcing-sparkr-r-on-spark.html
https://cs.stanford.edu/~matei/papers/2016/sigmod_sparkr.pdf