Streaming Architectures for IoT Applications

DataNatives ConferenceBerlin, 19/11/2015

AgendaI. Intro – IoT Use Cases

II. Challenges

III. Effective Architectures for IoT

I. Intro – IoTIoT – More than a buzz word and not that new, either.

- First implementation as early as 1982 in an internet connected Coke vending machine at Carnegie Mellon University

- The rise of RFID, later NFC and other technologies popularised the concept in the late 90’s and 2000’s

- Device’s numbers are exploding, while expected to reach 50 billion devices by 2020

I. Intro – Use CasesIoT turns formerly unintelligent isolated devices into something smart by interconnecting them:- Smart wearables: sport bands, health

sensors etc..- Smart homes: auto-sensing shades, bulbs,

heating- Smart transportation: vehicle and parking

monitoring, dynamic routing, telematics are all powered by IoT devices

- Smart manufacturing: sensors in factories monitor supply chains, temperature and other metricised values

- Smart environment: climate sensors, pollution meters, weather radars etc.

I. Intro – Anatomy of IoTIoT devices consist of two key elements:

I. A network interface – which enables communication with other devices and a data processing centre

II. A computing device – to collect data from sensors

IoT devices are usually small, embedded devices with minimal computing capacity and low energy requirements.Smartphone apps are an exception – usually in NFC/beacon use cases (Airports, Malls etc..)

II. ChallengesSince IoT devices can’t do much on their own, they must rely on a supporting data centre for data submission (unidirectional) and feedback (bidirectional)

Þ Multiple devices will send their data at any time in any order

Þ Devices will expect real time updates in return

Data Synchronisation happens within the data centre

II. Challenges – Time and Order

Data Synchronisation in real time

Þ Data centres must be able to handle unsorted, unordered incoming events.

Þ Data centres must offer instantaneous processing and updates for IoT devices that expect feedback.

This mandates near real time, stream-oriented processing as offered by modern Big Data architectures.

III. Effective ArchitecturesThe fundamental principles of effective data processing architecture for IoT devices are:

1. Asynchronous processing of incoming events (Message Broker)

2. Constant, streamed computation of events (Stream Processing)

3. Near real time delivery of aggregated data sets (Data Store)

III. Message BrokeringMessage Brokers provide an asynchronous Queue for incoming messages. Producers write independently from a Queue from which (multiple) Consumers pick up the messages.

Typical Broker implementations for IoT would be Apache Kafka for high throughput or MQTT compliant Brokers such as RabittMQ, Active MQ and others.

III. Stream ProcessingIncoming events are consumed by a stream processing pipeline that continuously aggregates and computes incoming events.

The result of which is stored in a persistent storage. Typical implementations of Stream Processing Frameworks are: Apache Spark, Apache Flink or Storm.

III. Near Real Time DeliveryIoT Device feedback is achieved by providing either immediate responses or providing a REST interface; both of which are connected to the stream processing’s persistent storage.

The type of storage heavily depends on the use case involved – Caches, relational databases, key value stores and other NoSQL DBs are all valid choices.

III. Long Term StorageIn some use cases, raw device data most be kept in long term storage, either for model calculation or for thorough re-computation.

In such instances, batch processing on top of inexpensive, scalable storage is the weapon of choice. Popular implementations are Spark or Flink on top of Hadoop’s HDFS or linear scalable NoSQL stores such as Cassandra, ElasticSearch or MongoDB

III. Architectural overview

III. Key Learnings1. Decouple IoT event delivery with

asynchronous Message Brokers

2. Continuously process incoming events and update your state in a real time store with a streaming framework

3. Provide a lightweight query interface for that store for your devices

Thank you!Hivemind Technologies AG

Sechtemer Str. 550968 Cologne, GermanyTel. +49 221 29218 400

www.hivemindtechnologies.com