IoT Innovation Lab Berlin @relayr - Kay Lerch on Getting basics right for your IoT business (2016-05-04)

www.immobilienscout24.de

Get the basics right for your IoT businessIoT Innovation Lab @ Relayr – 2016-05-04 – Speaker: @KayLerch

Agenda

1.Know the tech stack2.AWS can manage things for you3.AWS is not the only truth4.That’s Apache NiFi5.AWS IoT basic concepts (Recap)6.AWS IoT & Apache NiFi7.Focus on business value8.Focus on innovation

Seite 2Get the basics right for your IoT business | Kay Lerch

Know the tech stackbefore you do business


A reference architecture for IoT

Seite 4Source: WSO2 (IoT Whitepaper)

Value Value

Value

Value

Effort

Effort

Effort

Challenges:• Set the basics right (EFFORT)• Integrate (VALUE) in your current business

/ IT landscape• Be innovative!

Solution:Minimize (EFFORT)• Rely on managed services and toolsMaximize (VALUE)• Have the basics right• rely on established tools• Think over your integration strategyDrive innovation• Think beyond borders• disrupt business with combining

established things in a totally new fashion

Get the basics right for your IoT business | Kay Lerch

AWS can manage those basicswith a managed IoT solution


What AWS (IoT) provides …

Seite 6

Device Registry

TLS Certificates with associated IAM-like policies

Communication over HTTP and MQTT

Message broker with topicsState persistence with shadows

Event processing and forwarding with IoT Rules and Rule actions

Device SDKs in Node.JS, Cand for Arduino Yún


What AWS (IoT) provides …

Seite 7

Event processing and analytics with Kinesis Eventstreams, ML, EMR and Lambda

Data / event stores like S3 or Dynamo DB

API Gateway for public endpoint management

Have your IoT related key metrics in CloudWatch

EC2 as a ground for your (web) applications communicating with devices


AWS IoT core tents

Seite 8

Read this: “Core tenets of IoT” by AWS (as of April 2016)


AWS is not the only truthYou surely have your lovely tools outside AWS


A brief overview of data processing and analysisThat‘s quite a lot of tooling and technology …

Seite 10

Hidden complexities

Data Producers

Data Consumers

EventBroker

(Big) data stores

Data analysis challenges (Realtime) Data Processing & AnalyticsIntegrationchallengesIn

gest

ion

chal

leng

es

Data security challenges


A brief overview of data processing and analysisThat‘s quite a lot of tooling and technology …

Seite 11

Hidden complexities

Data Producers

Data Consumers

EventBroker

(Big) data stores

Data analysis challenges (Realtime) Data Processing & AnalyticsIntegrationchallengesIn

gest

ion

chal

leng

es

Data security challenges


If you want …

a (realtime) big picture of your dataflows an option to overlook lineage of each data element have the flexibility to change things on the fly prioritize data overcome challenges of integrating the variety of

technologies with one overarching solution enforce security and compliance along dataflows rely on extensibility driven by OS community satisfy all those needs and keep your tools get rid of only those tools focused on moving data without

making concessions to overall performance

… then you might love:

Just to let you know …


Value Value

Value

Value

Effort

Effort

Effort

We are here

And a bit of here


That’s Apache NiFi


Seite 14

That‘s Apache NiFiin one page

Apache nifi supports powerful and scalable directed graphs of datarouting, transformation, and system mediation logic.Web-based user interfaceSeamless experience between design, control, feedback, and monitoringHighly configurableLoss tolerant vs guaranteed delivery, Low latency vs high throughput, Dynamic prioritization, Flow can be modified at runtime, Back pressureData ProvenanceTrack dataflow from beginning to endDesigned for extensionBuild your own processors and more, Enables rapid development and effective testingSecureSSL, SSH, HTTPS, encrypted content, etc..., Pluggable role-based authentication/authorization

Source: https://nifi.apache.org/ Get the basics right for your IoT business | Kay Lerch

Seite 15

That‘s Apache NiFiin real and feel

Go to NiFi’s interface and understand:

Processors

Templates

Concept of back pressure

Concept of data prioritization

Provenence Graph


NiFi Cluster

NiFi Cluster Manager (NCM)

JVM

Node (Primary)

NiFi Clustered Architecture

JVM

WebserverProvenanceRepository

ContentRepository

FlowfileRepository

REST-APIAdminUI

Webserver REST-APIAdmin UI

Flow Controller

Cluster Manager

Processor 1Processor 2

Isolated Processor

Controller Service 1Controller Service 2Controller Service n

Hear

tbea

t

Lead

er

elec

tion

Repo

rt ch

ange

Embedded Apache Zookeeper

Node (Slave)JVM

WebserverProvenanceRepository

ContentRepository

FlowfileRepository

REST-APIAdminUI

Flow ControllerProcessor 1Processor 2

Isolated Processor

Controller Service 1Controller Service 2Controller Service n

Hear

tbea

t

Repo

rt ch

ange

Sync

St

ate

Sync

St

ate

AWS IoTBasics (Recap)


Seite 18

AWS IoTThe Shadow

AWS IoT

Thing

Thing Shadow

Rule

Reports State Mirrors State in Shadow

Gets reported state or sets desired state

Propagates desired state

Receives desired state

Fulfi

lls

desir

ed st

ate

Subscribes to particular messages

AWS Services

Some AWS Resource

Routesmessage

TLS1.2

TLS 1.2 Policy


AWS IoTMQTT topics

AWS IoT

Thing Shadow

get

get/accepted

get/rejected

Request state

Get shadow state

Get error

update

update/accepted

update/rejected

update/delta

Update state

Confirmation

Get error

Changedstate

1

2

1

2

3

Thing topics name pattern: $aws/things/thing_name/...


Apache NiFi & AWS IoTNew processors


Seite 21

Apache NiFi & AWS IoTGetIOTMqtt – a MQTT client

AWS IoT

Thing Shadow

updateUpdate state

Establish Connection

Subscribe

Receive state

1

2

3

Flowfile


Seite 22

Apache NiFi & AWS IoTGetIOTMqtt – Configuration


Seite 23

Apache NiFi & AWS IoTGetIOTMqtt – Live demo


Seite 24

Apache NiFi & AWS IoTGetIOTMqtt – Live demo


Seite 25

Apache NiFi & AWS IoTPutIOTMqtt – instruct a „thing“ (but bypass the shadow)

AWS IoT

Thing Shadow

update / delta

Update state

Establish Connection

Publish state

1

2 Flowfile

Flowfile


Seite 26

Apache NiFi & AWS IoTPutIOTMqtt – Configuration


Seite 27

Apache NiFi & AWS IoTPutIOTMqtt – Live demo


Seite 28

Apache NiFi & AWS IoTGetIOTShadow – constantly check last reported state

AWS IoT

Thing Shadow

updateReport state

RequestShadow

Flowfile

Flowfile


Seite 29

Apache NiFi & AWS IoTGetIOTShadow – Configuration


Seite 30

Apache NiFi & AWS IoTPutIOTShadow – instruct a „thing“ over its shadow

AWS IoT

Thing Shadow

update / delta

Desire state

UpdateShadow

Flowfile

Flowfile


Seite 31

Apache NiFi & AWS IoTPutIOTShadow – Configuration


We got the basicsLet’s focus on business value


Homework almost done


Value Value

Value

Value

Effort

Effort

Effort

Challenges:• Set the basics right (EFFORT)• Integrate (VALUE) in your current business• Be innovative!

Solution:Minimize (EFFORT) Rely on managed services and toolsMaximize (VALUE) Have the basics right rely on established tools• Think over your integration strategyDrive innovation• Think beyond borders• disrupt business with combining



Agile integration strategy with AWS Simple Workflows at ImmobilienScout

Legacy / 3rd Party

AWS Simple Workflows (SWF)

AWS

SNS

AWSLambda

AWS

S3

AWS

EC2

AWS

ECS

OnPr

emise

Brok

er

Start Workflow

Put File

Bucket Topic

PublishMessage

Start Workflow

ActivityTask

Function

Function

Trusted Applications / Mobile Apps AWS Flow Framework

Start Workflow

EventTrigger

Send DataInvoke Service

DeciderAppsAW

S Cl

oudw

atc

h

AWS

Clou

dTra

il Logs

Metrics

Alarms

Log Measure

Activity Task

Activity Task

Decision Task

Decision Task

Workflow History

AWS Flow Framework

AWS

***

Any

ActivityTask

DecisionTask

Start Workflow

Trigger Trigger

Notify

APICall

Discover

Decision Task

ServiceApps

AWS Flow Framework

ActivityTask

AWS

IAM

Policies

Manage Permissions

Function

LambdaTask

APICall

IoT Device

AWS

IoT

Topic

PublishMessage

Function

Start Workflow

Rule

Dyna

mo

Start Workflow

Put File

Table

Function

Trigger

Partner

AWS

API

Gate

way

StartWorkflow

Invoke

Function

Route

Service

Lambda Task Decision

Task

Hybrid integration platform aiming for Self-Service integration

We got the basicsLet’s focus on innovation


Homework done!


Value Value

Value

Value

Effort

Effort

Effort

Challenges:• Set the basics right (EFFORT)• Integrate (VALUE) in your current business• Be innovative!

Solution:Minimize (EFFORT) Rely on managed services and toolsMaximize (VALUE) Have the basics right rely on established tools Think over your integration strategyDrive innovation• Think beyond borders• disrupt business with combining



Gain real business value


Value Value

Value

Value

Effort

Effort

Effort

1) Combine values of different worlds to find a breathtaking use case

2) Rely on your basics

3) Build a product around your solution

4) Change the world

Value Value+

Effort Effort Effort+ +

+

=

+

Adopt new things…


Gain valueDemo (Smart home)

Seite 39

+ +


Pick some fancy frontend (VALUE) Pick some fancy “backend” (VALUE)

Rely on your basics (EFFORT)

= HaveSmart Home

Gain valueDemo (VoiceOps)

Seite 40

+ +


Pick some fancy frontend (VALUE) Pick some fancy “backend” (VALUE)

Rely on your basics (EFFORT)

= HaveVoiceOps

Once you got the basics right, it’s up to you to …


www.immobilienscout24.de

Thanks for you attention. Any questions?Contact:Immobilien Scout GmbHAndreasstraße 1010243 Berlin

Kay LerchFon +49 30 24 301-1149

[email protected]

Backup


T-Cobbler

I need some “thing” …

Input & Output units on your breadboard

I need some “thing” …

Control GPIO-Pins with sending or receiving electric impulses

Control an output Control an input

Communicating with AWS IoT over MQTT topics

Alexa Skill AWS Lambda

AWS API Gateway

LambdaPost

Message to Slack

LambdaUpdate Shadow

Rpi 2

AWS IoT

Thing Shadow

Python ScriptGPIO

Rule

Invoke

MQTTPublish

MQTT Receive

Update Shadow on state change

Impulse Sequence

SVC

OutgoingWebhook

Post Message over Slack Web API

433 Mhz Signal

C++ D/AConverter

Post Message if no state changes

AWS Lambda

AWS EC2

LambdaStart / Stop

EC2-Instance

Rpi 2

AWS IoT

Thing Shadow

Python ScriptGPIO

Rule

Invoke

MQTTPublish

Impulse Sequence


433 Mhz Signal

C++ A/DConverter

Start / Stop over AWS SDK

AWS CW

LambdaReact on EC2 state change

Rule

Update Shadow

MQTT Receive

Impulse

On state change

Rpi 2

AWS IoT

Thing Shadow

Python ScriptGPIO

Rule

MQTTPublish

Impulse Sequence


AWS CWAlarmOn Humidity > 30%Metric Metric

Alexa Skill

Read out temperature and humidity

Technology

IoT Innovation Lab Berlin @relayr - Kay Lerch on Getting basics right for your IoT business (2016-05-04)