IoT Austin CUG talk

1© Cloudera, Inc. All rights reserved.

IoT with Spark StreamingAnand Iyer, Senior Product Manager


Spark Streaming• Incoming data stream is represented as DStreams (Discretized Streams)• Stream is broken down into micro-batches• Each micro-batch is an RDD – process using RDD operations• Micro-batches usually 0.5 sec in size


Cloudera customer use case examples – Streaming

• On-line fraud detection

Financial Services

• On-line recommender systems

• Inventory management

Retail

• Incident prediction (sepsis)

Health

• Analysis of ad performance in real-time

Ad tech


Concrete end-to-end IoT Use CaseUsing Spark Streaming with Kafka, HBase & Solr


Proactive maintenance and accident prevention in Railways

• Sensor information continuously streaming in from railway carriages• Goal: Early detection of damage to rail carriage wheels or to railway tracks• Proactively fix issues before they become severe• Prevent derailments, save money and lives• Based on real-world use case, modified to fit the talk


Locomotive Wheel Axle Sensors

Each Sensor Reading Contains:- Unique ID- Locomotive ID- Speed- Temperature- Pressure- Acoustic signals - GPS Co-ordinates- Timestamp- etc

mailto:http://soundbible.com/1697-Train-Wheels-Grinding.html


Identify Damage to locomotive axle or wheelsManifests as sustained increasein sensor readings like temperature,pressure, acoustic noise, etc.


Identify Damage on railway tracksManifests as a sudden spikein sensor readings forpressure or acoustic noise.


Real-Time Detection of Locomotive Wheel Damage

Kafka

- Enrich incoming events with relevant meta-data- Locomotive information from

locomotive ID: type, weight, cargo,etc- Sensor information from Sensor ID:

precise location, type, etc- GPS co-ordinates to location

characteristics such as gradient of track.- Recommend HBase as metadata store.- Use HBase-spark module to fetch data.

- Apply application logic to determine if sensor readings indicate damage- Simple rule based- Complex predictive machine learning

model



Kafka Kafka

https://github.com/harishreedharan/spark-streaming-kafka-output

HDFS



- When an alert is thrown, technician will need to diagnose the event

- Requires visualizing sensor data as a time-series:- Over arbitrary windows of time- Compare with values from prior trips- Software for visualization: http://grafana.org/

- Technician can take appropriate action based on analysis:- Send rail carriage for maintenance- Stop train immediately to prevent accident

Visualize Time-Series Sensor Data

http://grafana.org/

http://grafana.org/


Data Store for Time-Series Data

Ideal solution: Kudu- Time series data entails sequential scans for writes and reads, interspersed with

random seeks

Until Kudu is GA:- Use HBase and model tables for time-series data- OpenTSDB:

- Built on top of HBase- Uses a HBase table schema optimized for time-series data- Simple HTTP API



Kafka Kafka

HDFS


Detecting damage to Railtracks

• They manifest as a sharp spike in sensor readings (pressure, acoustic noise)• Multiple sensors will demonstrate the same spike at the same location (GPS co-

ordinates)• Multiple sensors from multiple trains will give similar readings at the same

location.

How to detect?• Index each sensor reading, in Solr, such that they can be queried by GPS co-

ordinates• When a “spike” is observed, and corresponding alert event is fired, trigger a

search


Detecting damage to Rail tracks• Index each sensor reading, with the Morphlines library• Embed call to Morphlines in your Spark Streaming application• Values can be kept in the index for specified period of time, such as a month. Solr can automatically

purge old documents from the index.

• When a “spike” is observed, and corresponding alert event is fired, trigger a search (manually or programmatically)

• Search for sensor readings at the same GPS co-ordinates as the latest spike.• Filter out irrelevant readings (e.g. readings on the left track, if spike was observed on the right track)• Sort results by time, latest to oldest

• If majority of recent readings show a “spike”, indicative of track damage


Final Architecture

Kafka Kafka

HDFS

Morphlines

HBase-Sp

ark/R

EST


Noteworthy Streaming Constructs


Sliding Window Operations

Example usages: - compute counts of items in latest window of time, such as occurrences of exceptions in

a log or trending hashtags in a tweet stream- Join two streams by matching keys within same window

Note: Provide adequate memory to hold a window’s worth of data

Define operations on data within a sliding window.Window Parameters: - window length- sliding interval


Maintain and update arbitrary stateupdateStateByKey(...)• Define initial state• Provide state update function• Continuously update with new information

Examples: • Running count of words seen in text stream• Per user session state from activity stream

Note: Requires periodic check-pointing to fault-tolerant storage.


Lessons from Production


Use Kafka Direct Connector whenever possible• Better efficiency and performance than Receiver based Connectors

• Automatic back-pressure: steady performance

KafkaSparkDriver

Executor

Executor

Executor

Executor

Receiver

Receiver

SparkDriver

Executor

Executor

Executor

Executor


The challenge with Checkpoints

• Spark checkpoints are java serialized• Upgradeability can be an issue – upgrading the version of Spark or your

application can make checkpointed data unreadable

But long running applications need updates and upgrades!!


Upgrades with Checkpoints

• Most often, all you need to pick up is some previous state – maybe an RDD or some “state”(updateStateByKey), or last processed Kafka offsets

• The solution: Disable Spark Checkpoints

• Use foreachRDD to persist state yourself, to HDFS, in a format your application can understand• E.g. Avro, Protobuf, Parquet…

• For upateStateByKey, generate the new state - then persist


updateStateByKey(…) upcoming improvements

• Time-out: Automatically delete data after a preset number of micro-batches

• Efficient Updates: Only update a subset of the keys

• Callback to persist state during graceful shutdown


Exactly Once SemanticsWhat is it?Given a stream of incoming data, any operator is applied exactly once on each item.

Why is it important? Prevent erroneous processing of data stream. E.g., Double counting of aggregations or throwing of redundant alerts

Spark Streaming provides exactly one semantics for data transformations.However, output operations provide at-least once semantics!!


Exactly Once Semantics with Spark Streaming & Kafka

• Associate a “key” with each value written to external store, that can be used for de-duping

• This key needs to be unique for a given micro-batch

• Kafka Direct Connector provides the following associated with each record, which will be the same for a given micro-batch:Kafka-Partition + start-offset + end-offset• Check out org.apache.spark.streaming.kafka.OffsetRanges and

org.apache.spark.streaming.kafka.HasOffsetRanges


Thank You

Technology

IoT Austin CUG talk