Spark Streaming andExpert Systems

James Nowell24 May 2016

The Problem

Expert Systems

Where Spark Fits

Overview

2

THE PROBLEM

3

Basic IoT devices don’t have much state Inferences can be made with regular conditionals

Simple Streams

4

tizen365.com

Multiple sensors Multiple streams

(sensor fusion!) Partial data Bad data Human entered data

Complex Streams

5

http://www.record-eagle.com/news/lifestyles/traverse-city-s-ice-road-trucker/article_8cb844bd-b613-586d-b7a0-974789984e1d.html

EXPERT SYSTEMS

6

AKA Production Systems, Rule Engines, Inference System CLIPS, Drools Turn a domain expert into code

Mostly deterministicTasks that are simple or procedural for an expert

Fact based reasoningIf TheseFacts then NewFacts

Rete Algorithm

What is an Expert System?

7

Determine the time for the current state

In Practice: Is it moving?

8

Calculate some basic movement information

In Practice: Is it moving?

9

Decide the truck didn’t become a space shipThe converse is also asserted elsewhere

In Practice: Is it moving?

10

Yep, the truck seems to be moving

In Practice: Is it moving?

11

Assert that the thing is, in fact, moving

In Practice: Is it moving?

12

WHERE SPARK FITS

13

Load the statePersistence mechanism varies

Generate a new state with the rules Emit events with the rules

Events are often based on state transitionsSettle the new state first

Persist it allUse a journal model

In an RDD

14

Parallelism App structure similar to any other streaming app Batch is extremely similar to streaming

THE reason to use SparkReprocess your entire corpus with the same code you use for new data

What Spark Buys Us

15

Build the rules at compile timekie-maven-plugin

Maximize condition reuse Use flag events to kill loops and aid debugging Write unit tests

LotsAnd lotsAnd lotsOf unit tests

Write fully fledged “Thing” tests

Things to Know

16

QUESTIONS?

17